Root cause of the 6P+2D run-to-run collapse (rep1 100%, rep2 56%,
rep3 80%, session-routing 6.6%): not load-shedding, but a consumer
EngineCore crash.
Failure chain observed in the consumer logs:
1. D-pool fills to ~97% (decode-side capacity ceiling, the H1 story)
2. a large request's KV transfer fails: "Mooncake transfer engine
returned -1" (112k-token request, pool full)
3. scheduler fails the request (kv_load_failure_policy=fail)
4. PromptTokenStats.local_cache_hit = num_cached + recomputed -
num_external_computed goes NEGATIVE (external transfer exceeded
cached count)
5. loggers.record() calls Counter.inc(negative) -> prometheus raises
"Counters can only be incremented by non-negative amounts."
6. EngineCore dies -> every subsequent request fails (the cliff:
all successes in the first ~110s, zero after)
This turns ONE failed request into a total config collapse, and is
what made the round-robin 6P+2D reps look randomly variable.
Fix: clamp the three per-source prompt-token counts to >= 0 in
loggers.record() before they hit Counter.inc(). Pure insertion,
revertible via the existing sentinel mechanism. Lets a transfer
failure stay a single failed request instead of killing the engine,
so routing arms can be compared on equal footing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three independent bugs were blocking PD-disagg smoke; each fix is
isolated so the next PD experiment doesn't re-hit them.
1. mb5_launch.sh
- stop_all() also kills mb5_pd_proxy.py (our vendored copy),
not just the upstream filename, and asserts ports 8000-8007 +
PROXY_PORT are free before launching — stale proxies were
silently passing the readiness check.
- Proxy readiness uses a generic "any HTTP response" probe;
mooncake_connector_proxy only exposes /v1/completions so
/v1/models 404 is expected.
2. mb5_pd_proxy.py (vendored from third_party so deploy.sh ships it)
- Force min_tokens=1 on the prefill leg. Clients that set
min_tokens == max_tokens (our replayer does) collide with
vLLM's min_tokens<=max_tokens check after the proxy caps
max_tokens=1.
3. instrument_kv_snapshot.py
- Adds a second patch target: initialize
MooncakeConnectorWorker.bootstrap_server = None in __init__.
vLLM 0.18.1 only sets it under the is_kv_producer branch, so
kv_consumer hits AttributeError as soon as the first remote
prefill request lands.
- apply/revert refactored to iterate over (path, patches) pairs.
plot_kv_pool_timeline.py also handles snapshot files that never
captured a running request (would otherwise IndexError on an empty
stackplot input).
Smoke: 4P+4D × 20 reqs → 20/20 success, mean 3.9s, p99 17s, 8 PIDs
all writing snapshots (601 total), well above the 8C baseline.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The §3.2 H1 (D-pool capacity wall) argument needs system-level evidence,
not just headline latency. This patch lets us record, every ~100 ms,
the exact composition of each vLLM instance's KV pool:
- total / free / used block counts
- for each RUNNING request: blocks held, computed tokens, prompt tokens
- for each WAITING request: prompt tokens, status
Hook: inside Scheduler.schedule() right before the return. Per-request
blocks come from coordinator.single_type_managers[*].req_to_blocks
(vLLM 0.18.1's own per-request bookkeeping; no new tracking layer).
Throttled by MB5_PERIOD_MS env var (default 100 ms = 10 Hz) so a
13-min trace replay produces ~8 k snapshots per instance instead of
~80 k unthrottled.
Output: $MB5_LOG_DIR/mb5_kv_snapshot_pid<pid>.jsonl
(default MB5_LOG_DIR=/tmp). One file per EngineCore PID.
Apply/revert idempotent, same pattern as instrument_mooncake.py.
Markers: # MB5_INSTRUMENT_START / # MB5_INSTRUMENT_END.
Validated on dash1 venv: apply → py_compile ok → revert → py_compile ok.
With this in place we can build the stacked-area "KV pool composition
over time" figure the user asked for: x = wall-clock, y = block count,
colored bands = per-request portions. Comparing 8C colo vs 4P+4D
on the same trace will directly show whether (and when) the D pool
hits its ceiling — turning "PD-disagg is X× worse" into "PD-disagg
is X× worse BECAUSE these specific requests at this specific time
filled the pool and forced this queue depth".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
