gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
559faa1e26882b1628f8cbca95feec9c734516ac
agentic-kvc/analysis/characterization/current_results/characterization_claim_matrix.md
Gahow Wang 559faa1e26 B2 finding: TPOT idx peaks at 32k, not 65k — cost migrates to TTFT 
The B2 same-worker TPOT p90 idx is non-monotone: 7.89x at 32k drops
to 2.26x at 65k. The naive reading is "interference gets weaker for
huge prefills"; the actual mechanism is a regime shift, and reading
TPOT p90 alone is misleading.

Three superimposed effects:

1. Cost migration TPOT -> TTFT. A 32k prefill is short enough that
   chunked-prefill keeps interleaving decode steps, so overlapping
   decodes trickle tokens out at painful per-token rates. A 65k
   prefill is long enough that overlapping decodes are *fully*
   blocked for ~10s; once they break through, the injection is
   winding down and subsequent iterations run unobstructed. The
   cost lands on the TTFT clock (14s) instead of inflating TPOT.

2. Bimodal TPOT distribution. At 65k overlap, decodes split into
   "blocked entire prefill then normal rate" and "trickled slowly
   through prefill chunks". p99 sits on the second population and
   grows 59 -> 169.5 ms; p90 sits on the first and shrinks.

3. "Clean" stops being clean. With 4x ~10s injections in 60s, the
   110 "clean" decodes at 65k are squeezed into 2-3s recovery
   pockets. TPOT p90 clean rises 6.9 -> 9.6 ms (40%), shrinking
   the denominator of the ratio.

window_1_results.md adds a new B2 subsection laying out the
mechanism with the per-cell data table and the explicit reading
rule: headline interference metric is TTFT idx (monotone); TPOT
p99 is the right tail indicator; TPOT p90 alone is unsafe across
regime shifts. Direct implication: TTFT and TPOT need separate
SLO thresholds under PD-colo, because they measure costs from
different points in the request lifecycle and the cost migration
between them is workload-dependent.

current_results/characterization_claim_matrix.md adds a new
supported claim for the cost migration, listed against the existing
B2 evidence. current_results/reviewer_risk_register.md adds a
low-severity entry warning future readers off TPOT p90 alone.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-26 00:35:45 +08:00

4.6 KiB
Raw Blame History

Characterization Claim Matrix

Updated 2026-05-25 after Window 1 (B1' KV-footprint + reuse, B3 5-policy sweep, B2 PD-colo interference microbench).

Claim Status Supporting Data Needed Next Reviewer Risk
Per-session sequentiality is enforced when replayer + proxy carry the new join fields. supported A1 unix timestamps (t_dispatch/t_first_token/t_finish_unix) and X-Request-Id passthrough; smoke validation 2026-05-25 confirmed 30/30 join coverage. Use this stack for all Window 2 B4/B5 SRR runs. Legacy outputs/ without these fields still cannot be re-classified as online_realistic.
Agentic workload is long-input / short-output / heavy-tail session mass. supported Full trace CPU summary (full_trace_summary.json): input p50/p90/p99 = 20k/87.9k/125.5k; top 1% sessions hold 46.5% of input mass. Full trace 2.11M requests, 1.31M sessions. — Sample trace (w600) percentiles inherit from this full trace but should not be extrapolated.
KV per request for Qwen3-Coder-30B-A3B is 98304 B/token; p50/p90/p99 footprint = 1.83/8.04/11.49 GiB. supported window_1_results/kv_footprint_summary.json; computed from model config and full trace input lengths. — Assumes bf16; would scale for fp8/int8 quant.
Workload reuse is overwhelmingly intra-session. supported Real reuse decomposition from w600 lmetric run: intra 93.2%, cross 5.7%, shared 1.1% (window_1_results/lmetric_reuse.json). Theoretical any-vs-intra ceiling gap 0.7 pp. — Trace-specific; ChatGPT-style workloads with long system prompts would shift toward shared-prefix.
Theoretical APC ceiling on w600 trace is 79.6% (intra) / 80.3% (any-session). supported window_1_results/apc_upper_w600.json from block-level trie walk on hash_ids. — Assumes infinite per-worker cache (no eviction); achieved values must be read as fraction of this ceiling.
Cache-aware LMetric leaves a measurable locality gap (22.7 pp). supported lmetric achieved 56.9% vs intra-session ceiling 79.6%; B3 sweep window_1_results/b3_policy_comparison.json. — sticky data shows the gap can be recovered by harder affinity.
Hybrid affinity (unified) breaks the locality-vs-latency tradeoff. supported unified APC 79.4% (97% of intra ceiling) AND TTFT p90 7.24 s (lmetric is 15.6 s). — unified concentrates a single very hot worker (engine_4 at 37.7 s p90); hotspot_index 3.35.
Same-worker prefill-decode interference is causal, not correlation. supported B2 microbench: different-worker control idx 0.92-1.02 across 32× prefill-size variation; same-worker TTFT idx scales 2.15× (2k) → 218× (65k). window_1_results/b2_sweep_summary.json. — Synthetic decode load (256-token prompts at 4 req/s) bounds the realism; production behavior is layered on top of B3.
The cost of same-worker prefill interference migrates from TPOT to TTFT as prefill size grows past the chunked-prefill horizon. supported B2 same-worker TPOT p90 idx peaks at 32k (7.89×) and drops at 65k (2.26×), while TTFT idx grows monotonically (94.6× → 218×) and TPOT p99 grows monotonically (59 → 169.5 ms). See window_1_results.md "TPOT idx peaks at 32k, not 65k". — SLO thresholds for TTFT and TPOT cannot be the same under PD-colo; this should be reflected in B4 SRR sweep design.
Hard session affinity (sticky) inflates same-worker prefill-decode interference. supported sticky interference_index 13.65 vs lmetric 6.53; sticky's slow-request breakdown 57% same-worker overlap vs lmetric 23%. — Confirms the B2 causal claim observed at the system level.
Heavy-tail sessions are a contributor to hot-spot but not the sole cause. supported Cap-8 trace (37% requests dropped) reduces hotspot_index only 13% (2.24 → 1.94). Run capped under unified to see whether unified's hotspot also persists. Reviewer might counter that cap=8 is too soft; a stricter cap could be tried.
SRR per policy under SLO is not yet measured. not_yet_supported B3 was driven by trace timestamps with strict session sequentiality; saturation is reached but not parameterized. Run B4 with the A4 open-loop Poisson loadgen, per-class SLO, 5 policies × λ binary search. Without B4 the paper cannot claim "policy X sustains higher load than Y".
Failure attribution near SRR boundary is not yet measured. not_yet_supported B5 protocol exists; no runs. After B4, rerun each policy at 0.9× / 1.0× / 1.1× of its SRR_max with the same instrumentation, label slow requests. The current joined_analysis.label_slow_requests is the labeler; needs SRR boundaries to point at.