Gahow Wang
d76eb02637
New analysis/characterization/elastic_migration_v2/ packages the
unified_v2 + unified_kv_both experiments into a self-contained
results section that the paper can cite as the "we tried selective
PD-sep migration" case study. The section finds three independent
reasons PD-sep doesn't help on agentic w600:
1. Mooncake kv_both substrate alone (no PD-sep ever firing) imposes
TTFT p90 +45%, TPOT p90 +25%, hotspot index +19% vs plain
unified. Per-step KVConnectorMetadata maintenance and block
reservation semantics dominate even when no transfer is pending.
2. PD-sep gate fires only 0.16-0.41% of requests across two
gate-tightness configurations. 88-76% are killed by
new_local < threshold because 93% intra-session reuse on agentic
traces leaves a small uncached tail; 19% are killed by
chosen_no_active_decode (snapshot-time gate). Even relaxed
thresholds can't grow trigger rate past 0.5%.
3. When PD-sep fires, the calibrated cost model
(0.3s + bytes / 2.7 GB/s) is wrong by 10-20x. 5 triggered
requests in v2.1 saw realized TTFT 12-45s vs model-predicted
migrate cost 0.7-2.2s, consistent with the E2 audit's finding
that D-side block pre-reservation and missing layerwise
pipelining dominate the decode_sent -> first_token clock.
Three-way comparison (unified vs unified_kv_both vs unified_v2):
v2 vs the kv_both control is roughly net-zero (-10% hotspot,
-14% TPOT p90, +3% TTFT p90, +9% TTFT p99). v2 vs plain unified is
strictly worse by 27-49% across latency percentiles because the
kv_both substrate tax is unavoidable when the policy is enabled.
Contents:
- README.md: the four results sections, the three-way comparison
table, an explicit "what this claims for the paper" list, and a
cross-reference index to the earlier characterization documents.
- data/: b3_policy_comparison.json + per-policy breakdown.json
+ per-policy hotspot_index.json for the four policies in scope.
- figures/: 4 PNGs rendered by render_figures.py:
* fig_kv_both_overhead.png — 4-metric bar chart with delta
annotations showing kv_both alone costs +45% TTFT p90.
* fig_v2_trigger_funnel.png — per-reason request count for the
two gate configurations on log scale.
* fig_v2_predicted_vs_actual.png — scatter of model-predicted
migrate cost vs realized TTFT for the 5 triggered requests,
with y=x, 10x, and 20x reference lines.
* fig_three_way_hotspot.png — per-worker TTFT p90 grouped bars
across the three policies.
The section is intentionally self-contained: it lists what the
experiment validates (cost model picks correct candidates;
shadow-drift fix is necessary; same-worker interference is real)
alongside what it disproves (per-request PD-sep on agentic via
Mooncake is not a net win in current implementation).
Refs: E1/E2 subagent audits, B2 microbench, unified_v2 commits
19f69a9 / 4b833d3 / 95c8ef8.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
285 lines
13 KiB
Markdown
285 lines
13 KiB
Markdown
# Elastic Migration v2: Selective PD-Separation via Mooncake
|
||
|
||
Date: 2026-05-26
|
||
Trace: `traces/w600_r0.0015_st30.jsonl` (1214 reqs, 274 sessions, 53.3 M tokens)
|
||
Model: Qwen3-Coder-30B-A3B-Instruct, 8 × TP1 on H20
|
||
|
||
## TL;DR
|
||
|
||
This section explores whether the **B2-confirmed same-worker
|
||
prefill–decode interference** can be relieved by selectively
|
||
migrating prefill to a different worker for the requests where the
|
||
interference cost would dominate the transfer cost. We implement two
|
||
flavors of the policy (strict gates, then relaxed gates) and a clean
|
||
isolation control (`unified_kv_both`: same picker as `unified`, but
|
||
the vLLMs are launched in `kv_role=kv_both` so the Mooncake
|
||
substrate is on but never triggers).
|
||
|
||
Three findings:
|
||
|
||
1. **`kv_role=kv_both` alone imposes a heavy always-on tax**: TTFT
|
||
p90 +45%, TPOT p90 +25%, hotspot index +19% vs plain `unified`,
|
||
with no PD-sep ever firing.
|
||
2. **PD-sep almost never triggers on a real agentic workload**:
|
||
0.16% with strict gates, 0.41% with relaxed gates. Agentic
|
||
workloads have 93% intra-session reuse, so most requests land on
|
||
workers that already hold cache — the uncached tail is too small
|
||
to be worth migrating.
|
||
3. **When PD-sep does fire, the cost model is wrong by ~10–20×**:
|
||
the calibrated `0.3s + bytes / 2.7 GB/s` predicts 1–2 s migrate
|
||
cost; observed TTFT on triggered requests is 12–45 s. The same
|
||
D-side block-reservation pressure and absence of layerwise
|
||
pipelining that the E2 audit flagged still dominate.
|
||
|
||
The net latency of `unified_v2` is **not better than plain
|
||
`unified`**. Improving agentic PD-sep requires fixing the underlying
|
||
Mooncake transfer mechanism (E2 patches 6.1 lazy block reservation
|
||
and 6.3 layerwise pipelining), not the routing decision.
|
||
|
||
## Substrate
|
||
|
||
We compare three policies on identical traces:
|
||
|
||
| policy | picker | vLLM launch mode | what's it for |
|
||
|---|---|---|---|
|
||
| `unified` | hybrid affinity + LMetric | plain (no Mooncake) | the headline baseline |
|
||
| `unified_kv_both` | same as `unified` | `kv_role=kv_both` + bootstrap | isolation control: how much does kv_both *alone* cost? |
|
||
| `unified_v2` | unified + selective PD-sep | `kv_role=kv_both` + bootstrap | the actual experiment |
|
||
|
||
All three use the same trace, the same 8-instance topology, the same
|
||
shadow-drift–corrected proxy (`scripts/cache_aware_proxy.py` post-fix
|
||
`95c8ef8`). Plain `unified` was rerun on the patched proxy
|
||
(`b3_sweep_20260525_095043/unified`) under the same conditions.
|
||
|
||
## Result 1 — kv_both is expensive by itself
|
||
|
||
|
||
|
||
Switching the vLLM launch from plain to `kv_role=kv_both` without
|
||
ever triggering PD-sep already costs:
|
||
|
||
| metric | plain `unified` | `unified_kv_both` | Δ |
|
||
|---|---:|---:|---|
|
||
| TTFT p50 | 0.50 s | 0.50 s | +0% |
|
||
| TTFT p90 | 7.35 s | 10.67 s | **+45%** |
|
||
| TTFT p99 | 42.34 s | 45.19 s | +7% |
|
||
| TPOT p90 | 17.1 ms | 21.3 ms | **+25%** |
|
||
| E2E p90 | 18.03 s | 22.89 s | **+27%** |
|
||
| APC | 79.4% | 78.3% | −1.1 pp |
|
||
| hotspot index | 3.667 | **4.363** | **+19%** |
|
||
|
||
Two contributing factors:
|
||
|
||
1. **The Mooncake `MooncakeConnector` runs even when no transfer is
|
||
pending.** Every scheduler step it walks `set(cache.keys())`
|
||
against `_known_hash_keys` (E2 audit §6.5) and updates the
|
||
`KVConnectorMetadata`. This is O(|cache|) per step on every
|
||
engine, even when no producer/consumer relationship is active.
|
||
2. **Block reservation semantics differ** under kv_both. The
|
||
scheduler treats blocks as candidates for export-to-others, so
|
||
the prefix cache LRU pressure is slightly different (we lose 1
|
||
pp APC).
|
||
|
||
Practical implication: **you don't enable kv_both for free**. If a
|
||
deployment wants the option to do PD-sep selectively, the 45% TTFT
|
||
p90 tax applies even on requests that stay local. This needs to
|
||
recoverable cost before any selective-PD-sep policy is worth
|
||
shipping.
|
||
|
||
## Result 2 — PD-sep rarely fires on a real agentic trace
|
||
|
||
|
||
|
||
We log every routing decision's `v2_reason` (why we did or did not
|
||
PD-sep). Two runs with different gate thresholds:
|
||
|
||
| fall-through bucket | v2.0 strict | v2.1 relaxed | what it means |
|
||
|---|---:|---:|---|
|
||
| `new_local < threshold` | 1077 (88.7%) | 924 (76.1%) | uncached tail too small to justify transfer |
|
||
| `chosen_no_active_decode` | 115 (9.5%) | 229 (18.9%) | no decode on chosen to protect |
|
||
| `src_cache_below_threshold` | 14 (1.2%) | 36 (3.0%) | no alt instance holds enough cache |
|
||
| `src_not_meaningfully_more_cache` | 6 (0.5%) | 16 (1.3%) | alt instance doesn't help vs chosen |
|
||
| `cost_benefit not enough margin` | 0 | 4 (0.3%) | model says transfer cost + interference on src ≥ local interference |
|
||
| **PD-sep TRIGGERED** | **2 (0.16%)** | **5 (0.41%)** | passed all gates and cost-benefit favored migrate |
|
||
|
||
The dominant filter is `new_local < threshold`. Even with the
|
||
threshold dropped from 16 k to 8 k tokens, three out of four requests
|
||
have less than 8 k uncached tokens at the chosen worker. This is
|
||
structural: with intra-session reuse measured at 93% on the same
|
||
trace (window_1_results.md), most turns hit prefix cache on the
|
||
session's previous worker.
|
||
|
||
The second filter, `chosen_no_active_decode`, kills another fifth.
|
||
This is a snapshot-time phenomenon: at the moment the picker runs,
|
||
the chosen worker often has its previous request still in prefill,
|
||
not yet decoding. The gate's intent ("don't migrate if no decode is
|
||
being hurt by the prefill we're routing") is correct, but it ends up
|
||
suppressing PD-sep for a real situation where decode is *about to*
|
||
start.
|
||
|
||
Even after these two filters, the cost-benefit step itself rejects
|
||
nearly half of remaining candidates (4 out of 9 in relaxed). So the
|
||
final trigger rate of 0.41% is a structural property, not a
|
||
parameter-tuning problem.
|
||
|
||
## Result 3 — when PD-sep fires, the cost model is wrong by 10–20×
|
||
|
||
|
||
|
||
The 5 PD-sep-triggered requests in v2.1 relaxed:
|
||
|
||
| input | new_local | new_src | src→dst | cost_local | cost_migrate (model) | actual TTFT | actual E2E |
|
||
|---:|---:|---:|---|---:|---:|---:|---:|
|
||
| 21963 | 21963 | 9163 | 6→5 | 4.39 s | 4.17 s | 3.69 s | 8.48 s |
|
||
| 8706 | 8706 | 2050 | 5→7 | 1.09 s | 0.73 s | 12.48 s | 14.31 s |
|
||
| 13616 | 13616 | 2352 | 4→0 | 1.70 s | 1.03 s | 18.33 s | 19.50 s |
|
||
| 49483 | 49483 | 843 | 3→4 | 11.75 s | 2.16 s | **45.13 s** | **53.55 s** |
|
||
| 19806 | 19806 | 350 | 3→6 | 3.96 s | 1.06 s | 20.06 s | 31.98 s |
|
||
|
||
The cost model predicts the migrate path will take 0.7–2.2 s; the
|
||
actual TTFT on these requests is 12–45 s. The model's `0.3 s +
|
||
bytes / 2.7 GB/s` calibration captures pure RDMA bandwidth in
|
||
isolation but misses everything else that happens on the
|
||
`decode_sent → first_token` clock: D-side scheduler step latency,
|
||
block reservation before KV arrives (so D's cache pressure
|
||
increases for the entire wait), the per-layer scatter of
|
||
`batch_transfer_sync_write`, and the next-step scheduler promotion
|
||
after `finished_recving`. The E2 audit measured this end-to-end at
|
||
p50 = 1.1 s and **p90 = 6.7 s** on production runs; the v2.1
|
||
triggered requests landed in the p99 tail of that distribution
|
||
because their dst was already loaded.
|
||
|
||
The first-token clock for the 49 k request is **21× the model's
|
||
prediction**. This is not a small mis-tuning — it's a structurally
|
||
different model.
|
||
|
||
## Result 4 — three-way comparison
|
||
|
||
|
||
|
||
The full table:
|
||
|
||
| metric | unified (plain) | unified_kv_both | unified_v2 (relaxed) |
|
||
|---|---:|---:|---:|
|
||
| n_ok | 1214 | 1214 | 1214 |
|
||
| TTFT p50 | 0.50 s | 0.50 s | 0.49 s |
|
||
| TTFT p90 | 7.35 s | 10.67 s | 10.98 s |
|
||
| TTFT p99 | 42.34 s | 45.19 s | 49.45 s |
|
||
| TPOT p90 | 17.1 ms | 21.3 ms | 18.4 ms |
|
||
| E2E p90 | 18.03 s | 22.89 s | 22.53 s |
|
||
| APC | 79.4% | 78.3% | 77.6% |
|
||
| interference index | n/a (no engine_state) | 8.57 | 8.46 |
|
||
| hotspot index | 3.667 | 4.363 | 3.910 |
|
||
| n_slow | 189 | 198 | 198 |
|
||
|
||
### v2 vs the kv_both control (the right comparison)
|
||
|
||
Compared to the kv_both control — same substrate, no PD-sep — the
|
||
5 PD-sep triggers in v2:
|
||
|
||
- **slightly improve TPOT p90 (−14%) and hotspot (−10%)**
|
||
- **slightly worsen TTFT p90 (+3%) and TTFT p99 (+9%)**, because the
|
||
triggered requests themselves take ~20× the predicted transfer
|
||
time
|
||
|
||
The net effect against the kv_both control is in the noise. The
|
||
hotspot improvement is within the run-to-run stochastic range we saw
|
||
earlier (v2 strict run scored 2.733 hotspot under the same
|
||
substrate; v2 relaxed scored 3.910).
|
||
|
||
### v2 vs plain unified (the headline question)
|
||
|
||
`unified_v2` is **27% slower on E2E p90** and **49% slower on TTFT
|
||
p90** than plain `unified`. The 45 pp of TTFT p90 inflation is from
|
||
kv_both substrate, not the routing decision; nothing PD-sep does can
|
||
recover this in our current Mooncake implementation.
|
||
|
||
## Why v2's PD-sep is fundamentally choked
|
||
|
||
There are three independent structural problems, each by itself
|
||
enough to make v2 not win:
|
||
|
||
1. **The kv_both substrate is the wrong default**. It pays a 45%
|
||
TTFT p90 tax on every request. To make selective PD-sep beat
|
||
plain `unified`, the saved interference per triggered request
|
||
times the trigger rate must exceed 45% × average TTFT, on
|
||
average. With 0.41% trigger rate, even saving 100% of TTFT per
|
||
triggered request would only save ~0.4%, which can't recover 45%.
|
||
|
||
2. **Agentic intra-session reuse leaves no headroom for migration**.
|
||
Most turns hit cache on the worker that handled the previous
|
||
turn. Migrating prefill to a *different* worker is the *exact*
|
||
thing intra-session affinity tries to avoid: it forces the new
|
||
worker to pay for the cached prefix transfer instead of just
|
||
reusing what's already on the affinity worker. This is a
|
||
structural mismatch between PD-sep semantics ("send big prefills
|
||
to a less-busy worker") and agentic workloads ("keep sessions
|
||
sticky to wherever the cache is").
|
||
|
||
3. **The Mooncake mechanism is 10–20× slower than the cost model
|
||
predicts**, primarily due to D-side pre-allocation of KV blocks
|
||
and the absence of layerwise pipelining (E2 audit §6.1 / §6.3).
|
||
The cost model can be re-calibrated, but doing so would push the
|
||
gate even tighter, dropping the already-tiny trigger rate to
|
||
nearly zero.
|
||
|
||
The three are stacked: even if any two were fixed, the remaining
|
||
one would still make PD-sep a net loss on this trace.
|
||
|
||
## What this section claims for the paper
|
||
|
||
1. **Same-worker prefill–decode interference is a real mechanism**
|
||
(B2 microbench), but **agentic workloads rarely expose it**: the
|
||
typical request has high cache hit and small uncached tail, so
|
||
the interference cost is bounded.
|
||
2. **Routing-only solutions (unified) already capture 79% of the
|
||
intra-session APC ceiling and recover the latency** by avoiding
|
||
the heavy-tail sessions through the affinity gate. The remaining
|
||
23 pp gap to the ceiling is from APC LRU eviction under capacity
|
||
pressure, not from prefill–decode interference.
|
||
3. **Per-request PD-sep via Mooncake on agentic workloads is not a
|
||
net win** in our measurements, even with a carefully-gated cost
|
||
model. The combined effect of kv_both substrate overhead, low
|
||
trigger rate, and mechanism-vs-model gap is uniformly negative.
|
||
4. **A productive direction is mechanism-level**: fix the Mooncake
|
||
D-side block reservation (E2 §6.1), implement layerwise transfer
|
||
pipelining (E2 §6.3), and re-measure. Only if these patches drop
|
||
the substrate tax to <10% and the realized transfer to ≤2 s p90
|
||
does PD-sep become competitive with routing on agentic traces.
|
||
|
||
## What v2 still validates
|
||
|
||
- **The cost model's *qualitative* shape is correct**: when it says
|
||
"migrate", that's a request where local interference *would have*
|
||
been ≥ 4 s and src has ≥ 80% prefix cache. The model picks the
|
||
right candidate requests.
|
||
- **The gate logic catches the right exclusions**: 88% by uncached
|
||
tail size, 19% by no-decode-to-protect, the rest by missing
|
||
source cache. Each is a structurally correct reason.
|
||
- **The proxy shadow-drift fix is necessary infrastructure** for
|
||
any long-running routing experiment. We observed 3 phantom
|
||
corrections per ~50-minute run.
|
||
|
||
## Files
|
||
|
||
- `data/b3_policy_comparison.json` — the four policies' headline
|
||
metrics from the same B3 sweep root.
|
||
- `data/breakdown_<policy>.json` — per-request proxy breakdown
|
||
including v2 gate fields and triggered-event metadata.
|
||
- `data/per_worker_<policy>.json` — per-worker TTFT/latency p90s
|
||
used in the hotspot figure.
|
||
- `figures/*.png` — the four section figures referenced above.
|
||
- `render_figures.py` — regenerates the figures from data/.
|
||
|
||
## Cross-references
|
||
|
||
- `analysis/characterization/window_1_results.md` — B2 microbench
|
||
(same-worker interference causal proof) and B3 baseline 5-policy
|
||
sweep
|
||
- `analysis/characterization/agentic_dispatch_coupling.md` — why
|
||
the saturated-replay setup matches agentic production
|
||
- `analysis/characterization/b3_policies_pseudocode.md` — pickers
|
||
for the five baseline policies; `unified_v2` extends `unified`
|
||
- E1 / E2 subagent reports (commit `4b833d3` message and the
|
||
conversation log) — full mechanism audit that informed v2's design
|