agentic-kvc/analysis/characterization/window_1_results.md

# Window 1 Results: B1' + B2 + B3

Status: Window 1 complete (CPU + 2 dash0 GPU windows on 2026-05-25)
Sweep: `outputs/b3_sweep_20260525_095043` (B3) + `outputs/b2_microbench/` (B2)
Trace: `traces/w600_r0.0015_st30.jsonl` (1214 requests / 274 sessions / 53.3 M input tokens)
Model: Qwen3-Coder-30B-A3B-Instruct (TP1 × 8 instances on H20)

Per-policy artifacts under `window_1_results/`. Figures under `window_1_results/figures/`.

## Headline

| Claim | Status | Evidence |
|---|---|---|
| Agentic workload reuse is overwhelmingly intra-session | **supported** | 93.2% of cached_tokens are intra-session (real); theoretical any-session APC ceiling 80.3% vs intra-session ceiling 79.6% → < 1pp gap |
| LMetric leaves 23 pp of APC on the table | **supported** | lmetric achieved 56.9% vs intra-session ceiling 79.6% (theoretical) |
| Hard session affinity recovers the locality lost by LMetric | **supported** | sticky APC 77.2% = 97% of theoretical ceiling |
| Hard affinity inflates same-worker prefill-decode interference | **supported** | sticky interference_index 13.65 vs lmetric 6.53 |
| Hybrid affinity (Unified) breaks the locality-vs-latency tradeoff | **supported** | unified hits 79.4% APC and TTFT p90 7.35 s (lmetric 15.67 s) simultaneously |
| Same-worker prefill-decode interference is causal, not correlation | **supported** | different-worker control idx≈1.0; same-worker idx scales monotonically with prefill size |
| Heavy-tail sessions are *a* contributor to hot-spot, not the sole cause | **supported** | cap=8 truncated trace cuts 37% of work; hotspot drops only ~10% (2.253→2.020) |
| The agentic dispatch coupling amplifies policy gaps under saturation | **supported, framed as feature** | Slow policy → longer session lifetime → more concurrent in-flight → harder system. B3 measures the combined policy + feedback effect, which is what an agentic operator experiences. See `agentic_dispatch_coupling.md`. |

## B1' Workload characterization

### Per-request KV footprint (Qwen3-Coder-30B-A3B)

`kv_bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × dtype_bytes = 2 × 48 × 4 × 128 × 2 = 98304 B`

Full GLM-5.1 trace (2.11 M requests, 1.31 M sessions):

| | p50 | p90 | p95 | p99 | max |
|---|---:|---:|---:|---:|---:|
| KV per request | 1.83 GiB | 8.04 GiB | 9.59 GiB | **11.49 GiB** | 18.5 GiB |

H20 has ~95 GiB usable per GPU. **A single p99 request occupies 12% of a single H20's HBM** purely for KV. Multi-request batching is bounded by this.

Figure: `figures/fig_kv_footprint_cdf.png`.

### Real reuse decomposition (from lmetric run on w600 trace)

| class | tokens | fraction |
|---|---:|---:|
| intra-session | 28.3 M | **93.2%** |
| cross-session | 1.72 M | 5.7% |
| shared / system-prefix | 0.34 M | 1.1% |
| unclassified | 0 | 0.0% |

→ session-affinity routing covers >99% of the reuse signal. There is no meaningful "system prompt" in this trace.

Figure: `figures/fig_reuse_decomposition.png`.

### Theoretical APC ceilings on w600

Computed by building a block-level trie of `hash_ids` per session (intra-session) or globally (any-session), then walking each request's `hash_ids` to count its longest prefix-match against previously-seen prefixes.

| variant | upper bound | hit requests |
|---|---:|---:|
| any-session (perfect global cache) | **80.3%** | 961 / 1214 |
| intra-session only | **79.6%** | 914 / 1214 |
| shared-prefix only (pos 0, ≥8 sessions) | 0.10% | 107 / 1214 |

Gap "any − intra" is 0.7 pp → no meaningful cross-session sharing in this trace.

## B3 5-policy routing sweep

8 vLLM instances on TP1, w600 trace, `--enable-prompt-tokens-details` so `cached_tokens` is reported per request.

| policy | TTFT p50/p90/p99 | TPOT p50/p90/p99 ms | E2E p50/p90/p99 | **APC** | interference | **hotspot** | n_slow |
|---|---|---|---|---:|---:|---:|---:|
| lmetric | 0.94 / 15.67 / 53.57 | 8.9 / 21.2 / 176.9 | 2.75 / 24.82 / 79.83 | 56.9% | 6.53 | 2.253 | 295 |
| load_only | 1.26 / 20.20 / 52.84 | 9.2 / 26.9 / 320.7 | 3.59 / 33.46 / 93.93 | 54.1% | 9.16 | **1.294** | 379 |
| sticky | 0.54 / 18.02 / 74.09 | 8.9 / 36.4 / 357.2 | 2.08 / 34.63 / 134.36 | 77.2% | **13.65** | 2.728 | 234 |
| **unified** | **0.50 / 7.35 / 42.34** | 8.1 / 17.1 / 118.3 | **1.75 / 18.03 / 68.43** | **79.4%** | n/a* | **3.667** | **189** |
| capped | 1.20 / 12.83 / 46.62 | 7.2 / 16.0 / 101.7 | 2.59 / 21.25 / 73.79 | 31.6% | 6.33 | 2.020 | 185 |

\*unified `engine_state` was overwritten by my analyzer's slice step before the `b3_analyze.sh` fix landed; vLLM and the patch worked correctly. The B2 microbench provides a cleaner interference proof.

> **Methodology note (read before interpreting latency comparisons)**: B3 uses
> session-sequential trace dispatch — turn N+1 fires the instant turn N
> completes when the trace timestamp has already passed. This is the right
> model of agentic workloads (tool-call driven, no user think-time), but it
> means under saturation each policy's effective in-flight session count is
> a function of its own per-turn latency (slower policy → longer mean
> session lifetime → more concurrent in-flight). The reported gaps are
> therefore "policy + agentic-feedback-amplification", which is what a
> production agentic operator would experience when switching policies.
> See `agentic_dispatch_coupling.md` for the full argument. B4 will report
> the orthogonal "fixed-λ open-loop" measurement.

**Mechanism indices**
- `interference_index` = TPOT_p90(decode overlapping same-worker prefill) / TPOT_p90(clean)
- `hotspot_index` = max(worker TTFT p90) / median(worker TTFT p90)

Figures: `fig_b3_latency_bars.png`, `fig_b3_apc_vs_upper.png`,
`fig_b3_apc_vs_hotspot.png`, `fig_b3_per_worker_ttft_p90.png`,
`fig_b3_failure_breakdown.png`.

### Per-policy reading

- **lmetric** is the cache-aware baseline. APC 56.9% achieves only 71% of the intra-session ceiling — the missing 23 pp is the locality opportunity unified picks up.
- **load_only** strips cache awareness. Hot-spot drops to 1.294 (best), but APC only drops 3 pp because the picker's `min(num_requests)` tie-break to instance 0 creates accidental stickiness at low concurrency.
- **sticky** locks each session to one worker. APC climbs to 77.2% (97% of ceiling) but interference doubles to 13.65 and TPOT p99 hits 345 ms.
- **unified** is the hybrid — affinity gate `(cache_ratio>0.5 AND num_req ≤ 2×avg)` keeps locality where it pays and drops it where it would hurt. The result is APC 79.4% **and** TTFT p90 cut in half from lmetric. The one bad worker (engine_4 at 37.7s p90) drives `hotspot_index=3.667`, but the other seven workers are all under 18 s.
- **capped** runs lmetric on a turn-capped trace (max 8 turns/session). Removes 37% of requests but APC also crashes to 31.6% and hotspot only improves by ~10% (2.253 → 2.020). This is the session-mass ablation: heavy sessions are *a* contributor to hot-spot but not the sole cause.

### Slow-request cause breakdown (from `joined_analysis.label_slow_requests`)

| policy | n_slow | same-worker overlap | hot worker queue | cache miss large append | unknown |
|---|---:|---:|---:|---:|---:|
| lmetric | 295 | 69 (23%) | 68 (23%) | 94 (32%) | 64 (22%) |
| load_only | 379 | 108 (29%) | 33 (9%) | 151 (40%) | 87 (23%) |
| sticky | 234 | **134 (57%)** | 51 (22%) | **20 (9%)** | 29 (12%) |
| unified | 189 | 0 (no engine_state) | 116 (61%) | 18 (10%) | 55 (29%) |
| capped | 185 | 45 (24%) | 66 (36%) | 60 (32%) | 14 (8%) |

PD-colo failures are mixed-mechanism: lmetric has no single dominant cause.
Sticky concentrates failures into same-worker overlap (locality is on, cache misses are gone, but interference takes over).

## B2 PD-colo interference microbench

Setup: 2 vLLM instances on GPU 0 (decode endpoint) and GPU 1 (prefill endpoint). A continuous 4 req/s short-prompt decode load runs against GPU 0 for 60 s per cell. 4 large-prompt one-token "prefill injections" fire every 12 s, targeted at either the same instance (`same`) or the paired one (`different`). Decode requests are labeled overlap iff their `[t_first_token, t_finish]` intersects any injection window. We compare TPOT p90 (overlap vs clean) per cell.

| variant | prefill | n_overlap | n_clean | **TPOT idx** | **TTFT idx** |
|---|---:|---:|---:|---:|---:|
| different | 2k–65k | 12–126 | 114–228 | **0.92–1.02** | **0.96–1.00** |
| same | 2k | 12 | 228 | 1.16 | 2.15 |
| same | 8k | 19 | 221 | 1.90 | **12.1×** |
| same | 16k | 37 | 203 | 3.37 | **30.8×** |
| same | 32k | 67 | 173 | **7.89** | **94.6×** |
| same | 65k | 130 | 110 | 2.26* | **218×** |

\*65k TPOT idx is non-monotone — see §"TPOT idx peaks at 32k, not 65k" below.

Figures: `fig_b2_tpot_vs_prefill.png`, `fig_b2_ttft_vs_prefill.png`.

**Why this matters**
- The `different-worker` control sits at idx ≈ 1.0 across 32× variation in prefill size. This is the cleanest possible disproof of "any prefill anywhere hurts decode": prefill on a *different* worker is invisible to the decode worker.
- The `same-worker` TTFT curve is monotone in prefill size all the way to 218× at 65k. TPOT p90 is monotone only up to 32k (7.89×), then drops at 65k — this is not "interference relaxing", it is the cost regime shifting from TPOT to TTFT (see below).
- This is the mechanism behind the B3 sticky interference jump (13.65) and unified's single hot worker (engine_4 at 37.7 s TTFT p90).

### TPOT idx peaks at 32k, not 65k — regime shift, not relief

The naïve reading of the table is "interference gets worse up to 32k then drops at 65k". That is wrong; the cost is shifting from per-token rate (TPOT) to first-token wait (TTFT), and `p90 / clean` happens to compress the visible cost. Three superimposed effects.

Same-variant detail across the regime boundary:

```
                          32k         65k       change
n_overlap                  67         130       +94%  (most decodes now overlap)
n_clean                   173         110       -37%
TPOT p50 overlap (ms)    12.2        20.1       +1.6x
TPOT p90 overlap (ms)    54.8        21.7       -2.5x  <- "improves"
TPOT p99 overlap (ms)    59.0       169.5       +2.9x  <- tail explodes
TTFT p90 overlap  (s)    4.17       14.06       +3.4x
TPOT p90 clean   (ms)     6.9         9.6       +40%
```

**Mechanism 1 — Cost shifts from TPOT to TTFT.** TPOT is measured only *after* a request starts emitting tokens. A 32 k prefill (~5 s on H20) is short enough that vLLM's chunked-prefill scheduler keeps interleaving decode steps; overlapping decodes trickle tokens out at painfully slow per-token rates → p90 TPOT 54.8 ms. A 65 k prefill (~10 s) is long enough that many overlapping decodes get *zero* tokens for nearly the whole prefill window; when they finally break through, the injection is winding down so subsequent decode iterations are unobstructed. The cost goes onto the TTFT clock (14 s) instead of inflating TPOT.

**Mechanism 2 — Bimodal TPOT distribution hides under p90.** At 65 k overlap, two populations of decodes coexist:
- decodes blocked the entire prefill (high TTFT, then normal per-token rate)
- decodes that did trickle slowly through prefill chunks (low TTFT, high TPOT)
- The p99 jump 59 → 169.5 ms shows the second population is *worse* at 65 k. p90 happens to fall on the first (fast-after-block) population.

**Mechanism 3 — "Clean" stops being clean.** With 4 × ~10 s injections spread across 60 s (40 s of injection time, 20 s of gaps), there are very few moments where the worker is truly idle. The 110 "clean" decodes at 65 k are squeezed into 2-3 s pockets where the system is recovering from the previous injection or about to be hit by the next. TPOT p90 clean rises 6.9 → 9.6 ms (the denominator of the idx ratio drifts up by 40%).

**Reading rule for B2**: TTFT idx is the headline interference metric — it is monotone and reflects user-visible "no tokens for N seconds" latency. **TPOT p99** is the right tail-sensitivity indicator (also monotone). **TPOT p90 is non-monotone across regime shifts and should not be used alone**. This has direct implications for SLO design: TTFT and TPOT cannot share the same violation threshold under PD-colo interference, because they measure costs from *different* points in the request lifecycle and the cost migration between them is workload-dependent.

This is also a finding the paper should call out: **once same-worker prefill grows beyond a TTFT-block threshold, overlapping decodes "give up" their per-token rate complaint and pay the cost in queueing instead**. The system looks faster on per-token metrics; users experience longer waits.

## What Window 1 does *not* answer

These need Window 2 (B4 SRR sweep + B5 failure attribution near SRR boundary):

1. **Sustainable arrival rate (SRR) per policy under SLO**. B3 was driven by trace timestamps with strict session sequentiality; when 8 instances cannot keep up, requests pile up and the *effective* dispatch window stretches (lmetric: trace claims 600 s, actual replay 49 min). We measured *saturated* behavior but not the saturation point. B4 needs the A4 open-loop Poisson loadgen with per-class SLO thresholds.
2. **Failure breakdown at the SRR boundary**. B5 will rerun each policy at 0.9× / 1.0× / 1.1× of its SRR_max and label each SLO-violating request — gives the paper its causal failure-attribution table.

Optional / paper-polish runs (not blocking the story):

3. unified isolated rerun to capture `interference_index` (B2 already provides cleaner causal proof; skip unless reviewer asks).
4. B2 with the proxy in path — measure whether the production cache_aware routing actually pushes prefill and decode onto different workers in practice.
5. KV-occupancy timeline per worker — needs polling `vllm:gpu_cache_usage` during B3 reruns; useful for "KV pressure drives cache miss" subsection.

## Limitations (read this before quoting B3 numbers)

1. **Agentic dispatch coupling is by design**. B3 is the
   "production-replay under captured agentic load" experiment, not the
   "controlled-load envelope" experiment. Latency p90 reflects both
   per-request policy effect AND the agentic feedback amplification
   (slow policy → longer mean session lifetime → more concurrent
   in-flight). Both contributions are real and visible to a production
   operator; **the paper must report both, not subtract one**. See
   `agentic_dispatch_coupling.md`. The orthogonal "fixed-λ Poisson"
   measurement is B4.

2. **B3 `interference_index` is a binary indicator**. A decode is
   labeled "overlap" iff *any* other request's prefill exists on the
   chosen worker during `[t_first_token, t_finish]`, regardless of
   prefill size. B2's per-prefill-size cells (2k = 1.16×, 65k = 2.26×)
   cannot be directly read off B3's aggregate numbers (lmetric 6.53,
   sticky 13.65). The B3 numbers are size-weighted averages of the
   per-cell signal, valid for *within-B3 cross-policy* comparison but
   not for direct cross-batch numerical comparison with B2.

3. **Hot-sweep cache contamination (low)**: `lmetric` ran from cold;
   `load_only` and `sticky` ran on the same 8 vLLMs without restart.
   First-turn cached_tokens verification puts empirical contamination
   at < 1% APC, well below the cross-policy gaps. `unified` and
   `capped` were rerun cold-start specifically to remove any residual
   concern.

4. **Unified's `interference_index` is missing**. The original
   `b3_analyze.sh` unconditionally truncate-wrote sliced engine_state
   files; isolated runs that wrote engine_state into their own
   per-policy directory were overwritten. Fixed in commit `df32499`;
   capped was the first run to benefit and survived. **Implication**:
   unified's slow-request mechanism breakdown (rows 0 / 116 / 18 / 55
   for same-worker overlap / hot worker queue / cache miss / unknown)
   has the "same-worker overlap" label *unrecoverable* and forced into
   the catch-all buckets — do not read unified's failure attribution
   as causal.

5. **w600 is not the full GLM-5.1 trace** (1214 req vs 2.11 M). All
   B3/B2 percentiles are on the sample. The full-trace KV-footprint
   stats are on the full trace.

6. **Reuse decomposition (intra/cross/shared/unclassified) is
   per-cached-token only in expectation** — `joined_analysis.py`
   distributes a request's `cached_tokens` count uniformly across its
   `hash_ids` and classifies block-by-block. For the w600 trace with
   <1% cross-session sharing the qualitative split is robust; for
   workloads with mixed-class hashes within a single request the
   classifier should be revisited.

## Reproduction commands

```bash
# B3 5-policy sweep
bash scripts/b3_sweep.sh                                   # lmetric, load_only, sticky (hot-cache)
bash scripts/b3_isolated_policy.sh unified <trace> <dir>   # isolated cold-start
bash scripts/b3_isolated_policy.sh lmetric <capped> <dir>  # capped variant

bash scripts/b3_analyze.sh outputs/b3_sweep_<TS>
python3 scripts/render_b3_report.py --sweep-dir outputs/b3_sweep_<TS>

# B2 interference microbench
# (launch 2 vLLM on ports 8100/8101 with --enable-prompt-tokens-details first)
python3 scripts/b2_interference.py \
    --decode-endpoint http://127.0.0.1:8100 \
    --prefill-endpoint http://127.0.0.1:8101 \
    --model <model> \
    --out-dir outputs/b2_microbench/sweep
python3 analysis/characterization/b2_sweep_analysis.py --sweep-dir outputs/b2_microbench/sweep

# Figures
python3 analysis/characterization/render_window1_figures.py \
    --results-dir analysis/characterization/window_1_results \
    --out-dir analysis/characterization/window_1_results/figures
```