Window 1 results: combined B1' + B2 + B3 report and artifacts
analysis/characterization/window_1_results.md is the headline write-up for Window 1: workload characterization (KV per request, real reuse decomposition, APC theoretical ceilings), B3 5-policy sweep with per-policy interpretation, B2 same-vs-different-worker interference microbench with causal reading, and an explicit list of what Window 1 does *not* answer (deferred to B4 SRR sweep + B5 attribution). Under window_1_results/: - 5 raw result JSONs from the B3 sweep, the B2 microbench, the APC upper bound, and the KV footprint - per-policy hotspot_index.json snapshots so render_window1_figures.py can plot per-worker TTFT p90 distributions - 8 PNG figures (figures/) covering the headline claims Three takeaways the figures pin down: 1) intra-session reuse dominates (93.2%), so session-affinity routing is the right primary lever 2) unified hybrid affinity hits 79.4% APC (97% of the 79.6% intra- session ceiling) AND cuts TTFT p90 from lmetric's 15.6s to 7.24s 3) B2 different-worker control sits at idx ≈ 1.0 across 32× prefill- size variation; same-worker TTFT idx scales 2.15× -> 218×, which is the cleanest causal evidence for same-worker prefill-decode interference Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
114
analysis/characterization/window_1_results/b3_report.md
Normal file
114
analysis/characterization/window_1_results/b3_report.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# B3 Routing Sweep Report
|
||||
|
||||
Sweep dir: `b3_sweep_20260525_095043`
|
||||
Trace: w600_r0.0015_st30.jsonl (~1.2k reqs, 8 × TP1)
|
||||
Policies present: lmetric, load_only, sticky, unified, capped
|
||||
Policies pending: —
|
||||
|
||||
## Headline latencies + APC
|
||||
|
||||
| policy | ok/total | TTFT p50/p90/p99 (s) | TPOT p50/p90/p99 (ms) | E2E p50/p90/p99 (s) | APC |
|
||||
|---|---:|---|---|---|---:|
|
||||
| **lmetric** | 1214/1214 | 0.94/15.59/52.95 | 8.9/21.2/175.9 | 2.75/24.75/79.62 | 56.9% |
|
||||
| **load_only** | 1214/1214 | 1.25/20.15/52.65 | 9.2/26.7/320.7 | 3.58/33.43/93.92 | 54.1% |
|
||||
| **sticky** | 1214/1214 | 0.54/18.02/71.37 | 8.9/36.1/345.2 | 2.08/34.61/133.58 | 77.2% |
|
||||
| **unified** | 1213/1214 | 0.50/7.24/42.02 | 8.1/17.1/118.1 | 1.75/17.89/68.18 | 79.4% |
|
||||
| **capped** | 770/770 | 1.20/12.76/46.05 | 7.2/16.0/101.5 | 2.59/21.24/73.39 | 31.6% |
|
||||
|
||||
## Mechanism indices
|
||||
|
||||
| policy | interference_index | hotspot_index (TTFT p90) | intra-session reuse | cross-session reuse | n_slow |
|
||||
|---|---:|---:|---:|---:|---:|
|
||||
| **lmetric** | 6.53 | 2.24 | 93.2% | 5.7% | 295 |
|
||||
| **load_only** | 9.16 | 1.14 | 93.5% | 5.3% | 379 |
|
||||
| **sticky** | 13.65 | 2.35 | 93.3% | 5.5% | 234 |
|
||||
| **unified** | — | 3.35 | 93.1% | 5.7% | 189 |
|
||||
| **capped** | 6.33 | 1.94 | 91.9% | 6.0% | 185 |
|
||||
|
||||
- **interference_index** = TPOT_p90(decode overlapping same-worker prefill) / TPOT_p90(clean)
|
||||
- **hotspot_index** = max(worker TTFT_p90) / median(worker TTFT_p90)
|
||||
|
||||
## Slow-request cause breakdown
|
||||
|
||||
| policy | n_slow | same-worker overlap | hot worker queue | cache miss large append | high KV | unknown |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| **lmetric** | 295 | 69 | 68 | 94 | 0 | 64 |
|
||||
| **load_only** | 379 | 108 | 33 | 151 | 0 | 87 |
|
||||
| **sticky** | 234 | 134 | 51 | 20 | 0 | 29 |
|
||||
| **unified** | 189 | 0 | 116 | 18 | 0 | 55 |
|
||||
| **capped** | 185 | 45 | 66 | 60 | 0 | 14 |
|
||||
|
||||
## Policy notes
|
||||
|
||||
- **lmetric** — cache-aware P_tokens × BS (main baseline)
|
||||
- **load_only** — control: min(num_requests), no cache, no affinity
|
||||
- **sticky** — control: hard session affinity (never break)
|
||||
- **unified** — hybrid affinity + LMetric fallback
|
||||
- **capped** — lmetric on per-session turn-capped trace
|
||||
|
||||
## Per-policy per-worker TTFT p90 (s)
|
||||
|
||||
### lmetric
|
||||
|
||||
| worker | TTFT p90 (s) |
|
||||
|---|---:|
|
||||
| http://127.0.0.1:8000 | 28.18 |
|
||||
| http://127.0.0.1:8001 | 13.15 |
|
||||
| http://127.0.0.1:8002 | 13.82 |
|
||||
| http://127.0.0.1:8003 | 14.00 |
|
||||
| http://127.0.0.1:8004 | 31.34 |
|
||||
| http://127.0.0.1:8005 | 7.87 |
|
||||
| http://127.0.0.1:8006 | 14.15 |
|
||||
| http://127.0.0.1:8007 | 11.78 |
|
||||
|
||||
### load_only
|
||||
|
||||
| worker | TTFT p90 (s) |
|
||||
|---|---:|
|
||||
| http://127.0.0.1:8000 | 22.06 |
|
||||
| http://127.0.0.1:8001 | 16.43 |
|
||||
| http://127.0.0.1:8002 | 16.81 |
|
||||
| http://127.0.0.1:8003 | 23.58 |
|
||||
| http://127.0.0.1:8004 | 25.14 |
|
||||
| http://127.0.0.1:8005 | 16.08 |
|
||||
| http://127.0.0.1:8006 | 23.96 |
|
||||
| http://127.0.0.1:8007 | 13.95 |
|
||||
|
||||
### sticky
|
||||
|
||||
| worker | TTFT p90 (s) |
|
||||
|---|---:|
|
||||
| http://127.0.0.1:8000 | 12.28 |
|
||||
| http://127.0.0.1:8001 | 23.57 |
|
||||
| http://127.0.0.1:8002 | 5.20 |
|
||||
| http://127.0.0.1:8003 | 55.38 |
|
||||
| http://127.0.0.1:8004 | 17.03 |
|
||||
| http://127.0.0.1:8005 | 25.49 |
|
||||
| http://127.0.0.1:8006 | 36.31 |
|
||||
| http://127.0.0.1:8007 | 2.50 |
|
||||
|
||||
### unified
|
||||
|
||||
| worker | TTFT p90 (s) |
|
||||
|---|---:|
|
||||
| http://127.0.0.1:8000 | 11.26 |
|
||||
| http://127.0.0.1:8001 | 3.61 |
|
||||
| http://127.0.0.1:8002 | 16.18 |
|
||||
| http://127.0.0.1:8003 | 9.31 |
|
||||
| http://127.0.0.1:8004 | 37.73 |
|
||||
| http://127.0.0.1:8005 | 18.33 |
|
||||
| http://127.0.0.1:8006 | 3.63 |
|
||||
| http://127.0.0.1:8007 | 7.77 |
|
||||
|
||||
### capped
|
||||
|
||||
| worker | TTFT p90 (s) |
|
||||
|---|---:|
|
||||
| http://127.0.0.1:8000 | 19.77 |
|
||||
| http://127.0.0.1:8001 | 15.79 |
|
||||
| http://127.0.0.1:8002 | 20.40 |
|
||||
| http://127.0.0.1:8003 | 10.54 |
|
||||
| http://127.0.0.1:8004 | 9.52 |
|
||||
| http://127.0.0.1:8005 | 9.46 |
|
||||
| http://127.0.0.1:8006 | 7.38 |
|
||||
| http://127.0.0.1:8007 | 9.66 |
|
||||
Reference in New Issue
Block a user