Gahow Wang
559faa1e26
The B2 same-worker TPOT p90 idx is non-monotone: 7.89x at 32k drops to 2.26x at 65k. The naive reading is "interference gets weaker for huge prefills"; the actual mechanism is a regime shift, and reading TPOT p90 alone is misleading. Three superimposed effects: 1. Cost migration TPOT -> TTFT. A 32k prefill is short enough that chunked-prefill keeps interleaving decode steps, so overlapping decodes trickle tokens out at painful per-token rates. A 65k prefill is long enough that overlapping decodes are *fully* blocked for ~10s; once they break through, the injection is winding down and subsequent iterations run unobstructed. The cost lands on the TTFT clock (14s) instead of inflating TPOT. 2. Bimodal TPOT distribution. At 65k overlap, decodes split into "blocked entire prefill then normal rate" and "trickled slowly through prefill chunks". p99 sits on the second population and grows 59 -> 169.5 ms; p90 sits on the first and shrinks. 3. "Clean" stops being clean. With 4x ~10s injections in 60s, the 110 "clean" decodes at 65k are squeezed into 2-3s recovery pockets. TPOT p90 clean rises 6.9 -> 9.6 ms (40%), shrinking the denominator of the ratio. window_1_results.md adds a new B2 subsection laying out the mechanism with the per-cell data table and the explicit reading rule: headline interference metric is TTFT idx (monotone); TPOT p99 is the right tail indicator; TPOT p90 alone is unsafe across regime shifts. Direct implication: TTFT and TPOT need separate SLO thresholds under PD-colo, because they measure costs from different points in the request lifecycle and the cost migration between them is workload-dependent. current_results/characterization_claim_matrix.md adds a new supported claim for the cost migration, listed against the existing B2 evidence. current_results/reviewer_risk_register.md adds a low-severity entry warning future readers off TPOT p90 alone. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
21 lines
4.6 KiB
Markdown
21 lines
4.6 KiB
Markdown
# Characterization Claim Matrix
|
||
|
||
Updated 2026-05-25 after Window 1 (B1' KV-footprint + reuse, B3 5-policy
|
||
sweep, B2 PD-colo interference microbench).
|
||
|
||
| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |
|
||
|---|---|---|---|---|
|
||
| Per-session sequentiality is enforced when replayer + proxy carry the new join fields. | `supported` | A1 unix timestamps (t_dispatch/t_first_token/t_finish_unix) and X-Request-Id passthrough; smoke validation 2026-05-25 confirmed 30/30 join coverage. | Use this stack for all Window 2 B4/B5 SRR runs. | Legacy outputs/ without these fields still cannot be re-classified as `online_realistic`. |
|
||
| Agentic workload is long-input / short-output / heavy-tail session mass. | `supported` | Full trace CPU summary (full_trace_summary.json): input p50/p90/p99 = 20k/87.9k/125.5k; top 1% sessions hold 46.5% of input mass. Full trace 2.11M requests, 1.31M sessions. | — | Sample trace (w600) percentiles inherit from this full trace but should not be extrapolated. |
|
||
| KV per request for Qwen3-Coder-30B-A3B is 98304 B/token; p50/p90/p99 footprint = 1.83/8.04/11.49 GiB. | `supported` | window_1_results/kv_footprint_summary.json; computed from model config and full trace input lengths. | — | Assumes bf16; would scale for fp8/int8 quant. |
|
||
| Workload reuse is overwhelmingly intra-session. | `supported` | Real reuse decomposition from w600 lmetric run: intra 93.2%, cross 5.7%, shared 1.1% (window_1_results/lmetric_reuse.json). Theoretical any-vs-intra ceiling gap 0.7 pp. | — | Trace-specific; ChatGPT-style workloads with long system prompts would shift toward shared-prefix. |
|
||
| Theoretical APC ceiling on w600 trace is 79.6% (intra) / 80.3% (any-session). | `supported` | window_1_results/apc_upper_w600.json from block-level trie walk on `hash_ids`. | — | Assumes infinite per-worker cache (no eviction); achieved values must be read as fraction of this ceiling. |
|
||
| Cache-aware LMetric leaves a measurable locality gap (22.7 pp). | `supported` | lmetric achieved 56.9% vs intra-session ceiling 79.6%; B3 sweep window_1_results/b3_policy_comparison.json. | — | sticky data shows the gap can be recovered by harder affinity. |
|
||
| Hybrid affinity (`unified`) breaks the locality-vs-latency tradeoff. | `supported` | unified APC 79.4% (97% of intra ceiling) AND TTFT p90 7.24 s (lmetric is 15.6 s). | — | unified concentrates a single very hot worker (engine_4 at 37.7 s p90); hotspot_index 3.35. |
|
||
| Same-worker prefill-decode interference is causal, not correlation. | `supported` | B2 microbench: different-worker control idx 0.92-1.02 across 32× prefill-size variation; same-worker TTFT idx scales 2.15× (2k) → 218× (65k). window_1_results/b2_sweep_summary.json. | — | Synthetic decode load (256-token prompts at 4 req/s) bounds the realism; production behavior is layered on top of B3. |
|
||
| The cost of same-worker prefill interference migrates from TPOT to TTFT as prefill size grows past the chunked-prefill horizon. | `supported` | B2 same-worker TPOT p90 idx peaks at 32k (7.89×) and *drops* at 65k (2.26×), while TTFT idx grows monotonically (94.6× → 218×) and TPOT p99 grows monotonically (59 → 169.5 ms). See window_1_results.md "TPOT idx peaks at 32k, not 65k". | — | SLO thresholds for TTFT and TPOT cannot be the same under PD-colo; this should be reflected in B4 SRR sweep design. |
|
||
| Hard session affinity (`sticky`) inflates same-worker prefill-decode interference. | `supported` | sticky interference_index 13.65 vs lmetric 6.53; sticky's slow-request breakdown 57% same-worker overlap vs lmetric 23%. | — | Confirms the B2 causal claim observed at the system level. |
|
||
| Heavy-tail sessions are a contributor to hot-spot but not the sole cause. | `supported` | Cap-8 trace (37% requests dropped) reduces hotspot_index only 13% (2.24 → 1.94). | Run capped under unified to see whether unified's hotspot also persists. | Reviewer might counter that cap=8 is too soft; a stricter cap could be tried. |
|
||
| SRR per policy under SLO is not yet measured. | `not_yet_supported` | B3 was driven by trace timestamps with strict session sequentiality; saturation is reached but not parameterized. | Run B4 with the A4 open-loop Poisson loadgen, per-class SLO, 5 policies × λ binary search. | Without B4 the paper cannot claim "policy X sustains higher load than Y". |
|
||
| Failure attribution near SRR boundary is not yet measured. | `not_yet_supported` | B5 protocol exists; no runs. | After B4, rerun each policy at 0.9× / 1.0× / 1.1× of its SRR_max with the same instrumentation, label slow requests. | The current `joined_analysis.label_slow_requests` is the labeler; needs SRR boundaries to point at. |
|