gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
922d79ac95f4b9d2ce08f2aec4de7dd694c970c0
agentic-kvc/analysis/characterization/window_1_results/b3_report.md
Gahow Wang 0c3220cbb8 Window 1 results: combined B1' + B2 + B3 report and artifacts 
analysis/characterization/window_1_results.md is the headline write-up
for Window 1: workload characterization (KV per request, real reuse
decomposition, APC theoretical ceilings), B3 5-policy sweep with
per-policy interpretation, B2 same-vs-different-worker interference
microbench with causal reading, and an explicit list of what Window 1
does *not* answer (deferred to B4 SRR sweep + B5 attribution).

Under window_1_results/:
- 5 raw result JSONs from the B3 sweep, the B2 microbench, the APC
  upper bound, and the KV footprint
- per-policy hotspot_index.json snapshots so render_window1_figures.py
  can plot per-worker TTFT p90 distributions
- 8 PNG figures (figures/) covering the headline claims

Three takeaways the figures pin down:
1) intra-session reuse dominates (93.2%), so session-affinity routing
   is the right primary lever
2) unified hybrid affinity hits 79.4% APC (97% of the 79.6% intra-
   session ceiling) AND cuts TTFT p90 from lmetric's 15.6s to 7.24s
3) B2 different-worker control sits at idx ≈ 1.0 across 32× prefill-
   size variation; same-worker TTFT idx scales 2.15× -> 218×, which
   is the cleanest causal evidence for same-worker prefill-decode
   interference

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-25 23:25:09 +08:00

3.7 KiB
Raw Blame History

B3 Routing Sweep Report

Sweep dir: b3_sweep_20260525_095043 Trace: w600_r0.0015_st30.jsonl (~1.2k reqs, 8 × TP1) Policies present: lmetric, load_only, sticky, unified, capped Policies pending: —

Headline latencies + APC

policy ok/total TTFT p50/p90/p99 (s) TPOT p50/p90/p99 (ms) E2E p50/p90/p99 (s) APC
lmetric 1214/1214 0.94/15.59/52.95 8.9/21.2/175.9 2.75/24.75/79.62 56.9%
load_only 1214/1214 1.25/20.15/52.65 9.2/26.7/320.7 3.58/33.43/93.92 54.1%
sticky 1214/1214 0.54/18.02/71.37 8.9/36.1/345.2 2.08/34.61/133.58 77.2%
unified 1213/1214 0.50/7.24/42.02 8.1/17.1/118.1 1.75/17.89/68.18 79.4%
capped 770/770 1.20/12.76/46.05 7.2/16.0/101.5 2.59/21.24/73.39 31.6%

Mechanism indices

policy interference_index hotspot_index (TTFT p90) intra-session reuse cross-session reuse n_slow
lmetric 6.53 2.24 93.2% 5.7% 295
load_only 9.16 1.14 93.5% 5.3% 379
sticky 13.65 2.35 93.3% 5.5% 234
unified — 3.35 93.1% 5.7% 189
capped 6.33 1.94 91.9% 6.0% 185
  • interference_index = TPOT_p90(decode overlapping same-worker prefill) / TPOT_p90(clean)
  • hotspot_index = max(worker TTFT_p90) / median(worker TTFT_p90)

Slow-request cause breakdown

policy n_slow same-worker overlap hot worker queue cache miss large append high KV unknown
lmetric 295 69 68 94 0 64
load_only 379 108 33 151 0 87
sticky 234 134 51 20 0 29
unified 189 0 116 18 0 55
capped 185 45 66 60 0 14

Policy notes

  • lmetric — cache-aware P_tokens × BS (main baseline)
  • load_only — control: min(num_requests), no cache, no affinity
  • sticky — control: hard session affinity (never break)
  • unified — hybrid affinity + LMetric fallback
  • capped — lmetric on per-session turn-capped trace

Per-policy per-worker TTFT p90 (s)

lmetric

worker TTFT p90 (s)
http://127.0.0.1:8000 28.18
http://127.0.0.1:8001 13.15
http://127.0.0.1:8002 13.82
http://127.0.0.1:8003 14.00
http://127.0.0.1:8004 31.34
http://127.0.0.1:8005 7.87
http://127.0.0.1:8006 14.15
http://127.0.0.1:8007 11.78

load_only

worker TTFT p90 (s)
http://127.0.0.1:8000 22.06
http://127.0.0.1:8001 16.43
http://127.0.0.1:8002 16.81
http://127.0.0.1:8003 23.58
http://127.0.0.1:8004 25.14
http://127.0.0.1:8005 16.08
http://127.0.0.1:8006 23.96
http://127.0.0.1:8007 13.95

sticky

worker TTFT p90 (s)
http://127.0.0.1:8000 12.28
http://127.0.0.1:8001 23.57
http://127.0.0.1:8002 5.20
http://127.0.0.1:8003 55.38
http://127.0.0.1:8004 17.03
http://127.0.0.1:8005 25.49
http://127.0.0.1:8006 36.31
http://127.0.0.1:8007 2.50

unified

worker TTFT p90 (s)
http://127.0.0.1:8000 11.26
http://127.0.0.1:8001 3.61
http://127.0.0.1:8002 16.18
http://127.0.0.1:8003 9.31
http://127.0.0.1:8004 37.73
http://127.0.0.1:8005 18.33
http://127.0.0.1:8006 3.63
http://127.0.0.1:8007 7.77

capped

worker TTFT p90 (s)
http://127.0.0.1:8000 19.77
http://127.0.0.1:8001 15.79
http://127.0.0.1:8002 20.40
http://127.0.0.1:8003 10.54
http://127.0.0.1:8004 9.52
http://127.0.0.1:8005 9.46
http://127.0.0.1:8006 7.38
http://127.0.0.1:8007 9.66