agentic-kvc/analysis/characterization/window_1_results.md

Window 1 Results: B1' + B2 + B3

Status: Window 1 complete (CPU + 2 dash0 GPU windows on 2026-05-25) Sweep: outputs/b3_sweep_20260525_095043 (B3) + outputs/b2_microbench/ (B2) Trace: traces/w600_r0.0015_st30.jsonl (1214 requests / 274 sessions / 53.3 M input tokens) Model: Qwen3-Coder-30B-A3B-Instruct (TP1 × 8 instances on H20)

Per-policy artifacts under window_1_results/. Figures under window_1_results/figures/.

Headline

Claim	Status	Evidence
Agentic workload reuse is overwhelmingly intra-session	supported	93.2% of cached_tokens are intra-session (real); theoretical any-session APC ceiling 80.3% vs intra-session ceiling 79.6% → < 1pp gap
LMetric leaves 23 pp of APC on the table	supported	lmetric achieved 56.9% vs intra-session ceiling 79.6% (theoretical)
Hard session affinity recovers the locality lost by LMetric	supported	sticky APC 77.2% = 97% of theoretical ceiling
Hard affinity inflates same-worker prefill-decode interference	supported	sticky interference_index 13.65 vs lmetric 6.53
Hybrid affinity (Unified) breaks the locality-vs-latency tradeoff	supported	unified hits 79.4% APC and TTFT p90 7.35 s (lmetric 15.67 s) simultaneously
Same-worker prefill-decode interference is causal, not correlation	supported	different-worker control idx≈1.0; same-worker idx scales monotonically with prefill size
Heavy-tail sessions are a contributor to hot-spot, not the sole cause	supported	cap=8 truncated trace cuts 37% of work; hotspot drops only ~10% (2.253→2.020)
The agentic dispatch coupling amplifies policy gaps under saturation	supported, framed as feature	Slow policy → longer session lifetime → more concurrent in-flight → harder system. B3 measures the combined policy + feedback effect, which is what an agentic operator experiences. See agentic_dispatch_coupling.md.

B1' Workload characterization

Per-request KV footprint (Qwen3-Coder-30B-A3B)

kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × dtype_bytes = 2 × 48 × 4 × 128 × 2 = 98304 B

Full GLM-5.1 trace (2.11 M requests, 1.31 M sessions):

	p50	p90	p95	p99	max
KV per request	1.83 GiB	8.04 GiB	9.59 GiB	11.49 GiB	18.5 GiB

H20 has ~95 GiB usable per GPU. A single p99 request occupies 12% of a single H20's HBM purely for KV. Multi-request batching is bounded by this.

Figure: figures/fig_kv_footprint_cdf.png.

Real reuse decomposition (from lmetric run on w600 trace)

class	tokens	fraction
intra-session	28.3 M	93.2%
cross-session	1.72 M	5.7%
shared / system-prefix	0.34 M	1.1%
unclassified	0	0.0%

→ session-affinity routing covers >99% of the reuse signal. There is no meaningful "system prompt" in this trace.

Figure: figures/fig_reuse_decomposition.png.

Theoretical APC ceilings on w600

Computed by building a block-level trie of hash_ids per session (intra-session) or globally (any-session), then walking each request's hash_ids to count its longest prefix-match against previously-seen prefixes.

variant	upper bound	hit requests
any-session (perfect global cache)	80.3%	961 / 1214
intra-session only	79.6%	914 / 1214
shared-prefix only (pos 0, ≥8 sessions)	0.10%	107 / 1214

Gap "any − intra" is 0.7 pp → no meaningful cross-session sharing in this trace.

B3 5-policy routing sweep

8 vLLM instances on TP1, w600 trace, --enable-prompt-tokens-details so cached_tokens is reported per request.

policy	TTFT p50/p90/p99	TPOT p50/p90/p99 ms	E2E p50/p90/p99	APC	interference	hotspot	n_slow
lmetric	0.94 / 15.67 / 53.57	8.9 / 21.2 / 176.9	2.75 / 24.82 / 79.83	56.9%	6.53	2.253	295
load_only	1.26 / 20.20 / 52.84	9.2 / 26.9 / 320.7	3.59 / 33.46 / 93.93	54.1%	9.16	1.294	379
sticky	0.54 / 18.02 / 74.09	8.9 / 36.4 / 357.2	2.08 / 34.63 / 134.36	77.2%	13.65	2.728	234
unified	0.50 / 7.35 / 42.34	8.1 / 17.1 / 118.3	1.75 / 18.03 / 68.43	79.4%	n/a*	3.667	189
capped	1.20 / 12.83 / 46.62	7.2 / 16.0 / 101.7	2.59 / 21.25 / 73.79	31.6%	6.33	2.020	185

*unified engine_state was overwritten by my analyzer's slice step before the b3_analyze.sh fix landed; vLLM and the patch worked correctly. The B2 microbench provides a cleaner interference proof.

Methodology note (read before interpreting latency comparisons): B3 uses session-sequential trace dispatch — turn N+1 fires the instant turn N completes when the trace timestamp has already passed. This is the right model of agentic workloads (tool-call driven, no user think-time), but it means under saturation each policy's effective in-flight session count is a function of its own per-turn latency (slower policy → longer mean session lifetime → more concurrent in-flight). The reported gaps are therefore "policy + agentic-feedback-amplification", which is what a production agentic operator would experience when switching policies. See agentic_dispatch_coupling.md for the full argument. B4 will report the orthogonal "fixed-λ open-loop" measurement.

Mechanism indices

• interference_index = TPOT_p90(decode overlapping same-worker prefill) / TPOT_p90(clean)
• hotspot_index = max(worker TTFT p90) / median(worker TTFT p90)

Figures: fig_b3_latency_bars.png, fig_b3_apc_vs_upper.png, fig_b3_apc_vs_hotspot.png, fig_b3_per_worker_ttft_p90.png, fig_b3_failure_breakdown.png.

Per-policy reading

• lmetric is the cache-aware baseline. APC 56.9% achieves only 71% of the intra-session ceiling — the missing 23 pp is the locality opportunity unified picks up.
• load_only strips cache awareness. Hot-spot drops to 1.294 (best), but APC only drops 3 pp because the picker's min(num_requests) tie-break to instance 0 creates accidental stickiness at low concurrency.
• sticky locks each session to one worker. APC climbs to 77.2% (97% of ceiling) but interference doubles to 13.65 and TPOT p99 hits 345 ms.
• unified is the hybrid — affinity gate (cache_ratio>0.5 AND num_req ≤ 2×avg) keeps locality where it pays and drops it where it would hurt. The result is APC 79.4% and TTFT p90 cut in half from lmetric. The one bad worker (engine_4 at 37.7s p90) drives hotspot_index=3.667, but the other seven workers are all under 18 s.
• capped runs lmetric on a turn-capped trace (max 8 turns/session). Removes 37% of requests but APC also crashes to 31.6% and hotspot only improves by ~10% (2.253 → 2.020). This is the session-mass ablation: heavy sessions are a contributor to hot-spot but not the sole cause.

Slow-request cause breakdown (from joined_analysis.label_slow_requests)

policy	n_slow	same-worker overlap	hot worker queue	cache miss large append	unknown
lmetric	295	69 (23%)	68 (23%)	94 (32%)	64 (22%)
load_only	379	108 (29%)	33 (9%)	151 (40%)	87 (23%)
sticky	234	134 (57%)	51 (22%)	20 (9%)	29 (12%)
unified	189	0 (no engine_state)	116 (61%)	18 (10%)	55 (29%)
capped	185	45 (24%)	66 (36%)	60 (32%)	14 (8%)

PD-colo failures are mixed-mechanism: lmetric has no single dominant cause. Sticky concentrates failures into same-worker overlap (locality is on, cache misses are gone, but interference takes over).

B2 PD-colo interference microbench

Setup: 2 vLLM instances on GPU 0 (decode endpoint) and GPU 1 (prefill endpoint). A continuous 4 req/s short-prompt decode load runs against GPU 0 for 60 s per cell. 4 large-prompt one-token "prefill injections" fire every 12 s, targeted at either the same instance (same) or the paired one (different). Decode requests are labeled overlap iff their [t_first_token, t_finish] intersects any injection window. We compare TPOT p90 (overlap vs clean) per cell.

variant	prefill	n_overlap	n_clean	TPOT idx	TTFT idx
different	2k–65k	12–126	114–228	0.92–1.02	0.96–1.00
same	2k	12	228	1.16	2.15
same	8k	19	221	1.90	12.1×
same	16k	37	203	3.37	30.8×
same	32k	67	173	7.89	94.6×
same	65k	130	110	2.26*	218×

*65k TPOT idx is non-monotone — see §"TPOT idx peaks at 32k, not 65k" below.

Figures: fig_b2_tpot_vs_prefill.png, fig_b2_ttft_vs_prefill.png.

Why this matters

• The different-worker control sits at idx ≈ 1.0 across 32× variation in prefill size. This is the cleanest possible disproof of "any prefill anywhere hurts decode": prefill on a different worker is invisible to the decode worker.
• The same-worker TTFT curve is monotone in prefill size all the way to 218× at 65k. TPOT p90 is monotone only up to 32k (7.89×), then drops at 65k — this is not "interference relaxing", it is the cost regime shifting from TPOT to TTFT (see below).
• This is the mechanism behind the B3 sticky interference jump (13.65) and unified's single hot worker (engine_4 at 37.7 s TTFT p90).

TPOT idx peaks at 32k, not 65k — regime shift, not relief

The naïve reading of the table is "interference gets worse up to 32k then drops at 65k". That is wrong; the cost is shifting from per-token rate (TPOT) to first-token wait (TTFT), and p90 / clean happens to compress the visible cost. Three superimposed effects.

Same-variant detail across the regime boundary:

                          32k         65k       change
n_overlap                  67         130       +94%  (most decodes now overlap)
n_clean                   173         110       -37%
TPOT p50 overlap (ms)    12.2        20.1       +1.6x
TPOT p90 overlap (ms)    54.8        21.7       -2.5x  <- "improves"
TPOT p99 overlap (ms)    59.0       169.5       +2.9x  <- tail explodes
TTFT p90 overlap  (s)    4.17       14.06       +3.4x
TPOT p90 clean   (ms)     6.9         9.6       +40%

Mechanism 1 — Cost shifts from TPOT to TTFT. TPOT is measured only after a request starts emitting tokens. A 32 k prefill (~5 s on H20) is short enough that vLLM's chunked-prefill scheduler keeps interleaving decode steps; overlapping decodes trickle tokens out at painfully slow per-token rates → p90 TPOT 54.8 ms. A 65 k prefill (~10 s) is long enough that many overlapping decodes get zero tokens for nearly the whole prefill window; when they finally break through, the injection is winding down so subsequent decode iterations are unobstructed. The cost goes onto the TTFT clock (14 s) instead of inflating TPOT.

Mechanism 2 — Bimodal TPOT distribution hides under p90. At 65 k overlap, two populations of decodes coexist:

• decodes blocked the entire prefill (high TTFT, then normal per-token rate)
• decodes that did trickle slowly through prefill chunks (low TTFT, high TPOT)
• The p99 jump 59 → 169.5 ms shows the second population is worse at 65 k. p90 happens to fall on the first (fast-after-block) population.

Mechanism 3 — "Clean" stops being clean. With 4 × ~10 s injections spread across 60 s (40 s of injection time, 20 s of gaps), there are very few moments where the worker is truly idle. The 110 "clean" decodes at 65 k are squeezed into 2-3 s pockets where the system is recovering from the previous injection or about to be hit by the next. TPOT p90 clean rises 6.9 → 9.6 ms (the denominator of the idx ratio drifts up by 40%).

Reading rule for B2: TTFT idx is the headline interference metric — it is monotone and reflects user-visible "no tokens for N seconds" latency. TPOT p99 is the right tail-sensitivity indicator (also monotone). TPOT p90 is non-monotone across regime shifts and should not be used alone. This has direct implications for SLO design: TTFT and TPOT cannot share the same violation threshold under PD-colo interference, because they measure costs from different points in the request lifecycle and the cost migration between them is workload-dependent.

This is also a finding the paper should call out: once same-worker prefill grows beyond a TTFT-block threshold, overlapping decodes "give up" their per-token rate complaint and pay the cost in queueing instead. The system looks faster on per-token metrics; users experience longer waits.

What Window 1 does not answer

These need Window 2 (B4 SRR sweep + B5 failure attribution near SRR boundary):

1. Sustainable arrival rate (SRR) per policy under SLO. B3 was driven by trace timestamps with strict session sequentiality; when 8 instances cannot keep up, requests pile up and the effective dispatch window stretches (lmetric: trace claims 600 s, actual replay 49 min). We measured saturated behavior but not the saturation point. B4 needs the A4 open-loop Poisson loadgen with per-class SLO thresholds.
2. Failure breakdown at the SRR boundary. B5 will rerun each policy at 0.9× / 1.0× / 1.1× of its SRR_max and label each SLO-violating request — gives the paper its causal failure-attribution table.

Optional / paper-polish runs (not blocking the story):

3. unified isolated rerun to capture interference_index (B2 already provides cleaner causal proof; skip unless reviewer asks).
4. B2 with the proxy in path — measure whether the production cache_aware routing actually pushes prefill and decode onto different workers in practice.
5. KV-occupancy timeline per worker — needs polling vllm:gpu_cache_usage during B3 reruns; useful for "KV pressure drives cache miss" subsection.

Limitations (read this before quoting B3 numbers)

1. Agentic dispatch coupling is by design. B3 is the "production-replay under captured agentic load" experiment, not the "controlled-load envelope" experiment. Latency p90 reflects both per-request policy effect AND the agentic feedback amplification (slow policy → longer mean session lifetime → more concurrent in-flight). Both contributions are real and visible to a production operator; the paper must report both, not subtract one. See agentic_dispatch_coupling.md. The orthogonal "fixed-λ Poisson" measurement is B4.

2. B3 interference_index is a binary indicator. A decode is labeled "overlap" iff any other request's prefill exists on the chosen worker during [t_first_token, t_finish], regardless of prefill size. B2's per-prefill-size cells (2k = 1.16×, 65k = 2.26×) cannot be directly read off B3's aggregate numbers (lmetric 6.53, sticky 13.65). The B3 numbers are size-weighted averages of the per-cell signal, valid for within-B3 cross-policy comparison but not for direct cross-batch numerical comparison with B2.

3. Hot-sweep cache contamination (low): lmetric ran from cold; load_only and sticky ran on the same 8 vLLMs without restart. First-turn cached_tokens verification puts empirical contamination at < 1% APC, well below the cross-policy gaps. unified and capped were rerun cold-start specifically to remove any residual concern.

4. Unified's interference_index is missing. The original b3_analyze.sh unconditionally truncate-wrote sliced engine_state files; isolated runs that wrote engine_state into their own per-policy directory were overwritten. Fixed in commit df32499; capped was the first run to benefit and survived. Implication: unified's slow-request mechanism breakdown (rows 0 / 116 / 18 / 55 for same-worker overlap / hot worker queue / cache miss / unknown) has the "same-worker overlap" label unrecoverable and forced into the catch-all buckets — do not read unified's failure attribution as causal.

5. w600 is not the full GLM-5.1 trace (1214 req vs 2.11 M). All B3/B2 percentiles are on the sample. The full-trace KV-footprint stats are on the full trace.

6. Reuse decomposition (intra/cross/shared/unclassified) is per-cached-token only in expectation — joined_analysis.py distributes a request's cached_tokens count uniformly across its hash_ids and classifies block-by-block. For the w600 trace with <1% cross-session sharing the qualitative split is robust; for workloads with mixed-class hashes within a single request the classifier should be revisited.

Reproduction commands

# B3 5-policy sweep
bash scripts/b3_sweep.sh                                   # lmetric, load_only, sticky (hot-cache)
bash scripts/b3_isolated_policy.sh unified <trace> <dir>   # isolated cold-start
bash scripts/b3_isolated_policy.sh lmetric <capped> <dir>  # capped variant

bash scripts/b3_analyze.sh outputs/b3_sweep_<TS>
python3 scripts/render_b3_report.py --sweep-dir outputs/b3_sweep_<TS>

# B2 interference microbench
# (launch 2 vLLM on ports 8100/8101 with --enable-prompt-tokens-details first)
python3 scripts/b2_interference.py \
    --decode-endpoint http://127.0.0.1:8100 \
    --prefill-endpoint http://127.0.0.1:8101 \
    --model <model> \
    --out-dir outputs/b2_microbench/sweep
python3 analysis/characterization/b2_sweep_analysis.py --sweep-dir outputs/b2_microbench/sweep

# Figures
python3 analysis/characterization/render_window1_figures.py \
    --results-dir analysis/characterization/window_1_results \
    --out-dir analysis/characterization/window_1_results/figures