diff --git a/analysis/characterization/current_results/characterization_claim_matrix.md b/analysis/characterization/current_results/characterization_claim_matrix.md index c8fea9f..0c13d37 100644 --- a/analysis/characterization/current_results/characterization_claim_matrix.md +++ b/analysis/characterization/current_results/characterization_claim_matrix.md @@ -13,6 +13,7 @@ sweep, B2 PD-colo interference microbench). | Cache-aware LMetric leaves a measurable locality gap (22.7 pp). | `supported` | lmetric achieved 56.9% vs intra-session ceiling 79.6%; B3 sweep window_1_results/b3_policy_comparison.json. | — | sticky data shows the gap can be recovered by harder affinity. | | Hybrid affinity (`unified`) breaks the locality-vs-latency tradeoff. | `supported` | unified APC 79.4% (97% of intra ceiling) AND TTFT p90 7.24 s (lmetric is 15.6 s). | — | unified concentrates a single very hot worker (engine_4 at 37.7 s p90); hotspot_index 3.35. | | Same-worker prefill-decode interference is causal, not correlation. | `supported` | B2 microbench: different-worker control idx 0.92-1.02 across 32× prefill-size variation; same-worker TTFT idx scales 2.15× (2k) → 218× (65k). window_1_results/b2_sweep_summary.json. | — | Synthetic decode load (256-token prompts at 4 req/s) bounds the realism; production behavior is layered on top of B3. | +| The cost of same-worker prefill interference migrates from TPOT to TTFT as prefill size grows past the chunked-prefill horizon. | `supported` | B2 same-worker TPOT p90 idx peaks at 32k (7.89×) and *drops* at 65k (2.26×), while TTFT idx grows monotonically (94.6× → 218×) and TPOT p99 grows monotonically (59 → 169.5 ms). See window_1_results.md "TPOT idx peaks at 32k, not 65k". | — | SLO thresholds for TTFT and TPOT cannot be the same under PD-colo; this should be reflected in B4 SRR sweep design. | | Hard session affinity (`sticky`) inflates same-worker prefill-decode interference. | `supported` | sticky interference_index 13.65 vs lmetric 6.53; sticky's slow-request breakdown 57% same-worker overlap vs lmetric 23%. | — | Confirms the B2 causal claim observed at the system level. | | Heavy-tail sessions are a contributor to hot-spot but not the sole cause. | `supported` | Cap-8 trace (37% requests dropped) reduces hotspot_index only 13% (2.24 → 1.94). | Run capped under unified to see whether unified's hotspot also persists. | Reviewer might counter that cap=8 is too soft; a stricter cap could be tried. | | SRR per policy under SLO is not yet measured. | `not_yet_supported` | B3 was driven by trace timestamps with strict session sequentiality; saturation is reached but not parameterized. | Run B4 with the A4 open-loop Poisson loadgen, per-class SLO, 5 policies × λ binary search. | Without B4 the paper cannot claim "policy X sustains higher load than Y". | diff --git a/analysis/characterization/current_results/reviewer_risk_register.md b/analysis/characterization/current_results/reviewer_risk_register.md index 0997c7c..75358e1 100644 --- a/analysis/characterization/current_results/reviewer_risk_register.md +++ b/analysis/characterization/current_results/reviewer_risk_register.md @@ -13,3 +13,4 @@ Updated 2026-05-25 after Window 1. | Trace-timestamp dispatch with strict session sequentiality stretches replay wall time | medium | lmetric's 600s trace dispatched over 49 min; system over-saturates and the dispatch window expands. | Window 2 uses A4 open-loop Poisson loadgen with explicit arrival rate, decoupling load level from trace structure. | | Capped cap=8 may be too soft | low | Reviewer might prefer cap=2 or cap=4 to test "no multi-turn" extreme. Cap=8 was chosen to sit between turns/session p90 (1) and p99 (18). | Re-run with a stricter cap if reviewer pushes back; underlying capped script is parameterized. | | B2 microbench uses synthetic short-prompt decode load (256 tokens) | low | This bounds the realism of the "decode" workload. Production decode tokens come from prior turns of long context. | The signal magnitude is robust enough that prompt length shouldn't qualitatively change conclusions; B3 sticky's failure breakdown is the production-trace confirmation. | +| Reading B2 same-worker interference from TPOT p90 alone gives a non-monotone curve | low | TPOT p90 idx peaks at 32k (7.89×) then drops at 65k (2.26×) even though TTFT idx grows monotonically (94.6× → 218×) and TPOT p99 grows monotonically (59 → 169.5 ms). The drop is regime shift (cost migrates from TPOT to TTFT once prefill blocks first-token long enough), not interference relief. | Reports must lead with TTFT idx; TPOT p99 is the right tail indicator for TPOT. See window_1_results.md §"TPOT idx peaks at 32k, not 65k". | diff --git a/analysis/characterization/window_1_results.md b/analysis/characterization/window_1_results.md index 6db3365..503d341 100644 --- a/analysis/characterization/window_1_results.md +++ b/analysis/characterization/window_1_results.md @@ -116,15 +116,45 @@ Setup: 2 vLLM instances on GPU 0 (decode endpoint) and GPU 1 (prefill endpoint). | same | 32k | 67 | 173 | **7.89** | **94.6×** | | same | 65k | 130 | 110 | 2.26* | **218×** | -\*65k TPOT idx is suppressed because n_overlap > n_clean — by the time the 65k prefill is finishing, the 4-second gap to the next injection has already started decoding overlap. The "clean" decodes left are the ones that randomly hit the brief gaps between injections. +\*65k TPOT idx is non-monotone — see §"TPOT idx peaks at 32k, not 65k" below. Figures: `fig_b2_tpot_vs_prefill.png`, `fig_b2_ttft_vs_prefill.png`. **Why this matters** - The `different-worker` control sits at idx ≈ 1.0 across 32× variation in prefill size. This is the cleanest possible disproof of "any prefill anywhere hurts decode": prefill on a *different* worker is invisible to the decode worker. -- The `same-worker` curve is monotone in prefill size for TTFT (218× at 65k) and monotone-up-to-32k for TPOT (7.89×). The two ablations together establish causation: prefill-decode interference is a same-worker phenomenon and scales sharply with prefill mass. +- The `same-worker` TTFT curve is monotone in prefill size all the way to 218× at 65k. TPOT p90 is monotone only up to 32k (7.89×), then drops at 65k — this is not "interference relaxing", it is the cost regime shifting from TPOT to TTFT (see below). - This is the mechanism behind the B3 sticky interference jump (13.65) and unified's single hot worker (engine_4 at 37.7 s TTFT p90). +### TPOT idx peaks at 32k, not 65k — regime shift, not relief + +The naïve reading of the table is "interference gets worse up to 32k then drops at 65k". That is wrong; the cost is shifting from per-token rate (TPOT) to first-token wait (TTFT), and `p90 / clean` happens to compress the visible cost. Three superimposed effects. + +Same-variant detail across the regime boundary: + +``` + 32k 65k change +n_overlap 67 130 +94% (most decodes now overlap) +n_clean 173 110 -37% +TPOT p50 overlap (ms) 12.2 20.1 +1.6x +TPOT p90 overlap (ms) 54.8 21.7 -2.5x <- "improves" +TPOT p99 overlap (ms) 59.0 169.5 +2.9x <- tail explodes +TTFT p90 overlap (s) 4.17 14.06 +3.4x +TPOT p90 clean (ms) 6.9 9.6 +40% +``` + +**Mechanism 1 — Cost shifts from TPOT to TTFT.** TPOT is measured only *after* a request starts emitting tokens. A 32 k prefill (~5 s on H20) is short enough that vLLM's chunked-prefill scheduler keeps interleaving decode steps; overlapping decodes trickle tokens out at painfully slow per-token rates → p90 TPOT 54.8 ms. A 65 k prefill (~10 s) is long enough that many overlapping decodes get *zero* tokens for nearly the whole prefill window; when they finally break through, the injection is winding down so subsequent decode iterations are unobstructed. The cost goes onto the TTFT clock (14 s) instead of inflating TPOT. + +**Mechanism 2 — Bimodal TPOT distribution hides under p90.** At 65 k overlap, two populations of decodes coexist: +- decodes blocked the entire prefill (high TTFT, then normal per-token rate) +- decodes that did trickle slowly through prefill chunks (low TTFT, high TPOT) +- The p99 jump 59 → 169.5 ms shows the second population is *worse* at 65 k. p90 happens to fall on the first (fast-after-block) population. + +**Mechanism 3 — "Clean" stops being clean.** With 4 × ~10 s injections spread across 60 s (40 s of injection time, 20 s of gaps), there are very few moments where the worker is truly idle. The 110 "clean" decodes at 65 k are squeezed into 2-3 s pockets where the system is recovering from the previous injection or about to be hit by the next. TPOT p90 clean rises 6.9 → 9.6 ms (the denominator of the idx ratio drifts up by 40%). + +**Reading rule for B2**: TTFT idx is the headline interference metric — it is monotone and reflects user-visible "no tokens for N seconds" latency. **TPOT p99** is the right tail-sensitivity indicator (also monotone). **TPOT p90 is non-monotone across regime shifts and should not be used alone**. This has direct implications for SLO design: TTFT and TPOT cannot share the same violation threshold under PD-colo interference, because they measure costs from *different* points in the request lifecycle and the cost migration between them is workload-dependent. + +This is also a finding the paper should call out: **once same-worker prefill grows beyond a TTFT-block threshold, overlapping decodes "give up" their per-token rate complaint and pay the cost in queueing instead**. The system looks faster on per-token metrics; users experience longer waits. + ## What Window 1 does *not* answer These need Window 2 (B4 SRR sweep + B5 failure attribution near SRR boundary):