B3: load_only + sticky policies, capped-trace builder, sweep driver

Three additions land together because B3's whole point is comparing
LMetric against meaningful controls.

- scripts/cache_aware_proxy.py: two new --policy values.
  - load_only: pure min(num_requests) routing, no cache or affinity.
    The B3 control that strips locality so the LMetric-vs-load gap is
    legible.
  - sticky: first turn goes to min-load, subsequent turns ALWAYS
    return to the same instance, even under saturation. The B3
    control that maxes out locality so the hot-spot cost is legible.
- scripts/build_capped_trace.py: per-session turn cap (default 8).
  Generates the session-mass-equalized variant the TODO calls for so
  that hot-spot index can be re-measured with the heavy-tail removed.
- scripts/b3_sweep.sh: orchestrates the 5-cell sweep.
  - GPU_INDICES makes it easy to skip a dead GPU.
  - EXTRA_VLLM_ARGS defaults to --enable-prompt-tokens-details so
    usage.prompt_tokens_details.cached_tokens is populated. vLLM
    0.18.1 omits the field by default and breaks the reuse-decomp
    pipeline; the smoke run surfaced this.
  - Trap kills EngineCore by name in addition to "vllm serve" — the
    parent dies first but the child holds GPU memory. Was the root
    cause of the 89 GB ghost on GPU 0 earlier today.
  - Proxy readiness is a polling loop, not a fixed sleep.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-25 17:54:24 +08:00
parent 763355b825
commit c6b7c3471b
4 changed files with 352 additions and 1 deletions

View File

@@ -301,6 +301,48 @@ def test_settings_has_runtime_knobs(proxy):
s.cache_gate_ratio = saved
def test_pick_instance_load_only_picks_min_num_requests(proxy):
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b"),
_make_inst(proxy, "http://c")]
insts[0].num_requests = 5
insts[1].num_requests = 2
insts[2].num_requests = 8
chosen, idx = proxy.pick_instance_load_only(insts, None, "sess1", 1000, {})
assert idx == 1 and chosen is insts[1]
def test_pick_instance_load_only_ignores_cache_hits(proxy):
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
block_size = proxy.BLOCK_SIZE
prefix = [123] * (block_size * 4)
insts[0].record_prefix(prefix)
insts[0].num_requests = 10
insts[1].num_requests = 0
chosen, idx = proxy.pick_instance_load_only(insts, prefix, None,
len(prefix), {})
assert idx == 1, "load_only must ignore cache hit on inst[0]"
def test_pick_instance_sticky_first_turn_picks_min_load(proxy):
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
insts[0].num_requests = 10
insts[1].num_requests = 2
affinity = {}
chosen, idx = proxy.pick_instance_sticky(insts, None, "sess1", 1000, affinity)
assert idx == 1
assert affinity == {"sess1": 1}
def test_pick_instance_sticky_subsequent_never_breaks(proxy):
"""Once assigned, sticky must never re-route even under massive overload."""
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
affinity = {"sess1": 0}
insts[0].num_requests = 1_000_000
insts[1].num_requests = 0
chosen, idx = proxy.pick_instance_sticky(insts, None, "sess1", 1000, affinity)
assert idx == 0, "sticky must stay even when pinned instance is saturated"
def test_p_offload_penalty_uses_settings_heavy_threshold(proxy):
"""M2: tweaking SETTINGS.heavy_threshold changes the P-offload penalty."""
inst = proxy.InstanceState("http://x")