B3: load_only + sticky policies, capped-trace builder, sweep driver
Three additions land together because B3's whole point is comparing
LMetric against meaningful controls.
- scripts/cache_aware_proxy.py: two new --policy values.
- load_only: pure min(num_requests) routing, no cache or affinity.
The B3 control that strips locality so the LMetric-vs-load gap is
legible.
- sticky: first turn goes to min-load, subsequent turns ALWAYS
return to the same instance, even under saturation. The B3
control that maxes out locality so the hot-spot cost is legible.
- scripts/build_capped_trace.py: per-session turn cap (default 8).
Generates the session-mass-equalized variant the TODO calls for so
that hot-spot index can be re-measured with the heavy-tail removed.
- scripts/b3_sweep.sh: orchestrates the 5-cell sweep.
- GPU_INDICES makes it easy to skip a dead GPU.
- EXTRA_VLLM_ARGS defaults to --enable-prompt-tokens-details so
usage.prompt_tokens_details.cached_tokens is populated. vLLM
0.18.1 omits the field by default and breaks the reuse-decomp
pipeline; the smoke run surfaced this.
- Trap kills EngineCore by name in addition to "vllm serve" — the
parent dies first but the child holds GPU memory. Was the root
cause of the 89 GB ghost on GPU 0 earlier today.
- Proxy readiness is a polling loop, not a fixed sleep.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -301,6 +301,48 @@ def test_settings_has_runtime_knobs(proxy):
|
||||
s.cache_gate_ratio = saved
|
||||
|
||||
|
||||
def test_pick_instance_load_only_picks_min_num_requests(proxy):
|
||||
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b"),
|
||||
_make_inst(proxy, "http://c")]
|
||||
insts[0].num_requests = 5
|
||||
insts[1].num_requests = 2
|
||||
insts[2].num_requests = 8
|
||||
chosen, idx = proxy.pick_instance_load_only(insts, None, "sess1", 1000, {})
|
||||
assert idx == 1 and chosen is insts[1]
|
||||
|
||||
|
||||
def test_pick_instance_load_only_ignores_cache_hits(proxy):
|
||||
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
|
||||
block_size = proxy.BLOCK_SIZE
|
||||
prefix = [123] * (block_size * 4)
|
||||
insts[0].record_prefix(prefix)
|
||||
insts[0].num_requests = 10
|
||||
insts[1].num_requests = 0
|
||||
chosen, idx = proxy.pick_instance_load_only(insts, prefix, None,
|
||||
len(prefix), {})
|
||||
assert idx == 1, "load_only must ignore cache hit on inst[0]"
|
||||
|
||||
|
||||
def test_pick_instance_sticky_first_turn_picks_min_load(proxy):
|
||||
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
|
||||
insts[0].num_requests = 10
|
||||
insts[1].num_requests = 2
|
||||
affinity = {}
|
||||
chosen, idx = proxy.pick_instance_sticky(insts, None, "sess1", 1000, affinity)
|
||||
assert idx == 1
|
||||
assert affinity == {"sess1": 1}
|
||||
|
||||
|
||||
def test_pick_instance_sticky_subsequent_never_breaks(proxy):
|
||||
"""Once assigned, sticky must never re-route even under massive overload."""
|
||||
insts = [_make_inst(proxy, "http://a"), _make_inst(proxy, "http://b")]
|
||||
affinity = {"sess1": 0}
|
||||
insts[0].num_requests = 1_000_000
|
||||
insts[1].num_requests = 0
|
||||
chosen, idx = proxy.pick_instance_sticky(insts, None, "sess1", 1000, affinity)
|
||||
assert idx == 0, "sticky must stay even when pinned instance is saturated"
|
||||
|
||||
|
||||
def test_p_offload_penalty_uses_settings_heavy_threshold(proxy):
|
||||
"""M2: tweaking SETTINGS.heavy_threshold changes the P-offload penalty."""
|
||||
inst = proxy.InstanceState("http://x")
|
||||
|
||||
Reference in New Issue
Block a user