Add agentic workload characterization audit scaffold

analysis/characterization/README.md

@@ -0,0 +1,255 @@
+# Characterization Analyzer Runbook
+
+CPU-only scaffold for Batch 0 and Batch 1 in
+`analysis/characterization_todo_for_interns.md`.
+
+This directory has three components:
+
+- `analyze.py`: Batch 0/1 analyzer for trace and per-request metrics.
+- `summarize_runs.py`: CPU-only audit of already completed benchmark
+  directories.
+- `protocols.md`: exact protocol for Batch 2-6 experiments that require fresh
+  GPU runs or additional instrumentation.
+
+The analyzer reads existing trace and metrics artifacts and writes:
+
+```text
+outputs/characterization/<date>/<task_name>/
+├── manifest.json
+├── raw/
+├── summary.json
+├── summary.md
+├── audit.md
+├── session_concurrency.json
+├── session_arrival_stats.json
+├── turn_interval_stats.json
+├── trace_profile.json
+├── invalid_runs.md
+├── workload_summary.json
+├── kv_footprint_summary.json
+├── reuse_decomposition.json
+├── session_skew.json
+├── append_delta_stats.json
+└── figures/
+```
+
+If `matplotlib` is installed, simple PNG/PDF figures are emitted under
+`figures/`. If it is not installed, all JSON/Markdown data artifacts are still
+written.
+
+## Canonical Data Sources
+
+Canonical full traces live on dash0:
+
+- formatted trace: `~/ali-trace/trace-glm5.1-formatted/`
+- raw unformatted trace: `~/ali-trace/trace-glm5.1/`
+
+For the current GLM-5.1 characterization, prefer the compact formatted file:
+
+```text
+~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
+```
+
+Do not pass `051315-051317-raw.jsonl` or the files under
+`~/ali-trace/trace-glm5.1/` directly to this analyzer unless you first convert
+them to the formatted schema. Those raw files are tens to hundreds of GiB and
+contain full prompt payloads rather than the compact characterization schema.
+
+The analyzer is CPU-only. For full trace characterization, either:
+
+- run it on dash0 against the formatted JSONL files without starting any GPU
+  service; or
+- copy/rsync the needed trace files from dash0 to this repository or another
+  local path, then run the analyzer locally.
+
+Only light directory/field inspection is needed on dash0 before choosing which
+trace file to analyze.
+
+The raw unformatted directory is listed as a source option for provenance, but
+this analyzer expects formatted JSONL records. Raw files should be converted to
+the formatted schema before being passed to `--trace`.
+
+## Inputs
+
+Trace JSONL:
+
+- Expected formatted fields: `chat_id`, `parent_chat_id`, `timestamp`,
+  `input_length`, `output_length`, `type`, `turn`, `hash_ids`, optional
+  `session_id`.
+- If `session_id` is absent, sessions are reconstructed from
+  `parent_chat_id` chains.
+- `timestamp` is treated as scheduled trace time, not proof of actual dispatch
+  time.
+
+Metrics JSONL:
+
+- Expected replayer fields: `request_id`, `session_id`, `turn_id`,
+  `trace_timestamp_s`, `input_length`, `output_length`, `cached_tokens`,
+  `latency_s`, `ttft_s`, `tpot_s`, `actual_output_tokens`, `error`.
+- If the metrics file is from the current replayer, it does not include actual
+  dispatch/finish wall-clock timestamps. Batch 0 will therefore mark actual
+  session sequentiality as unavailable and separately report a scheduled
+  estimate from `trace_timestamp_s + latency_s`.
+
+Proxy breakdown:
+
+- Optional JSON/JSONL with fields such as `request_id`, `t_proxy_recv`,
+  `t_first_token`, `t_done`, `cache_hit`, `estimated_new_tokens`,
+  `route_class`, `routed_to`, `policy`.
+- Batch 0 can prove actual per-session in-flight concurrency only when these
+  timing rows can be joined to analyzed requests by `request_id`.
+- Existing proxy breakdown artifacts may not contain `session_id`; without a
+  request-id join to trace/metrics, they can still support append/cache-hit
+  statistics but not per-session concurrency.
+
+Run config:
+
+- Optional JSON, usually `outputs/<run>/config.json`.
+- Used for manifest fields such as `policy`, `time_scale`, and request count
+  when available.
+
+## Commands
+
+Trace-only dry run:
+
+```bash
+python3 analysis/characterization/analyze.py \
+  --trace traces/w600_r0.0015_st30.jsonl \
+  --task-name w600_trace_only \
+  --overwrite
+```
+
+Trace plus replayer metrics:
+
+```bash
+python3 analysis/characterization/analyze.py \
+  --trace traces/w600_r0.0015_st30.jsonl \
+  --metrics outputs/smoke_test/metrics.jsonl \
+  --task-name smoke_trace_metrics \
+  --overwrite
+```
+
+Proxy breakdown append/cache analysis:
+
+```bash
+python3 analysis/characterization/analyze.py \
+  --breakdown outputs/contention_16s_elastic/breakdown.json \
+  --config outputs/contention_16s_elastic/config.json \
+  --task-name contention_breakdown \
+  --overwrite
+```
+
+Full trace on dash0, CPU-only:
+
+```bash
+python3 analysis/characterization/analyze.py \
+  --trace ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
+  --task-name full_trace_characterization \
+  --overwrite
+```
+
+Local run after copying from dash0:
+
+```bash
+rsync -av dash0:~/ali-trace/trace-glm5.1-formatted/<trace-file>.jsonl traces/
+python3 analysis/characterization/analyze.py \
+  --trace traces/<trace-file>.jsonl \
+  --task-name full_trace_characterization \
+  --overwrite
+```
+
+By default the analyzer records file size and mtime but skips full SHA256
+hashing, because canonical raw trace files can be hundreds of GiB. Add
+`--hash-inputs` only when you intentionally want a full file hash.
+
+KV footprint requires a model-specific value:
+
+```bash
+python3 analysis/characterization/analyze.py \
+  --trace traces/w600_r0.0015_st30.jsonl \
+  --kv-bytes-per-token 98304 \
+  --task-name w600_with_kv_estimate \
+  --overwrite
+```
+
+Summarize existing completed runs:
+
+```bash
+python3 analysis/characterization/summarize_runs.py
+```
+
+This writes:
+
+```text
+analysis/characterization/current_results/
+├── run_summaries.json
+├── comparisons.json
+├── claim_matrix.json
+├── reviewer_risk_register.json
+├── current_results.md
+├── characterization_claim_matrix.md
+├── all_figures_index.md
+├── reviewer_risk_register.md
+└── reproduction_commands.sh
+```
+
+## Batch 0 Semantics
+
+The online-serving invariant is:
+
+```text
+Each session has at most one in-flight turn.
+```
+
+The analyzer reports:
+
+- actual interval status from dispatch and finish/error timestamps;
+- scheduled estimate from trace timestamps plus latency when available;
+- per-session max in-flight;
+- session start-time distribution;
+- turn inter-arrival distribution;
+- attempted/completed/error counts and goodput when metrics exist;
+- run classification.
+
+Important limitation: trace timestamps alone cannot prove actual replay
+sequentiality. A run is only classified as `online_realistic` when actual
+per-request dispatch and finish/error timestamps prove
+`max_inflight_per_session <= 1`.
+
+## Batch 1 Semantics
+
+The analyzer reports:
+
+- input/output CDF stats;
+- input/output ratio;
+- KV footprint CDF stats when `--kv-bytes-per-token` is supplied;
+- session skew and top-session contribution;
+- append/uncached token stats when `cached_tokens` or `cache_hit` exists;
+- reuse decomposition when both cached-token fields and `hash_ids` exist.
+
+Reuse decomposition is conservative:
+
+- `intra_session`: cached hash block was seen earlier in the same session;
+- `cross_session`: cached hash block was seen earlier in another session;
+- `shared/system-prefix`: early-position block appears in many sessions;
+- `unclassified`: cached tokens could not be mapped to a previously seen hash
+  block.
+
+If cached-token/cache-hit fields are absent, reuse and append artifacts are
+written with `status: "unavailable"` and list the required fields.
+
+## Limitations
+
+- The script does not run a benchmark, query a live service, touch GPU state,
+  or start any daemon.
+- Request-id joins are exact. If trace, metrics, and proxy artifacts use
+  different request IDs, the unmatched rows are preserved under `raw/`.
+- Actual Batch 0 sequentiality needs actual dispatch and finish/error
+  timestamps. Current `replayer/metrics.py` metrics are not enough by
+  themselves.
+- `kv_bytes_per_token` depends on model architecture, layer count, KV heads,
+  head dimension, and dtype. The analyzer will not guess it.
+- Shared/system-prefix reuse classification is a heuristic based on trace
+  `hash_ids` positions and cross-session frequency. Adjust
+  `--shared-prefix-min-sessions` and `--system-prefix-blocks` if the formatted
+  trace provides a stronger system-prefix marker.

analysis/characterization/current_results/all_figures_index.md

@@ -0,0 +1,10 @@
+# Figures Index
+
+No generated figures are committed by this script. Batch-specific figures should be generated from:
+
+- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.
+- future Batch 2 step-timeline artifacts for interference plots.
+- future Batch 3 per-worker/session artifacts for hot-spot plots.
+- future Batch 4 arrival-rate sweep artifacts for SRR curves.
+
+This file exists so the audit package has a stable placeholder until fresh figures are generated.

analysis/characterization/current_results/characterization_claim_matrix.md

@@ -0,0 +1,11 @@
+# Characterization Claim Matrix
+
+| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |
+|---|---|---|---|---|
+| Batch 0 substrate audit is only partially complete for existing runs. | `partially_supported` | metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts. | Add request dispatch and finish/error timestamps to future replayer/proxy metrics. | Cannot use these runs to prove online per-session sequentiality. |
+| Batch 1 workload shape can be characterized from formatted traces and metrics. | `supported_for_trace_shape` | Full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass. | Add cache-hit joined records for actual reuse decomposition. | Actual cache reuse decomposition needs cached_tokens joined with hash_ids. |
+| Static PD separation is worse than combined in existing 200-request GPU A/B. | `supported_by_existing_artifact` | outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json. | Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology. | Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy. |
+| Elastic transfer-based migration does not improve high-contention 500-request run. | `supported_by_existing_artifact` | outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv. | Attribute whether failure is trigger quality, transfer overhead, or wrong load regime. | Existing metrics lack actual sequentiality proof and per-request transfer waterfall. |
+| PD-colo prefill/decode interference is not yet directly proven by step-level data in this package. | `not_yet_supported` | No decode-step and prefill-overlap timestamp artifact found in summarized runs. | Run Batch 2 controlled same-worker/different-worker injection with step timestamps. | Cannot claim interference as causal without Batch 2. |
+| Session hot-spot residual imbalance is suggested but not fully attributed. | `partially_supported` | gpu_util.csv shows per-GPU mean-util imbalance in existing runs. | Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker. | GPU util imbalance alone is not enough to prove session hot-spot. |
+| SRR is not measured by existing fixed-request runs. | `not_yet_supported` | No arrival-rate sweep artifacts found. | Implement Batch 4 Poisson session-arrival SRR sweep. | Latency-at-one-load cannot support sustainable throughput claim. |

analysis/characterization/current_results/claim_matrix.json

@@ -0,0 +1,51 @@
+[
+  {
+    "claim": "Batch 0 substrate audit is only partially complete for existing runs.",
+    "needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.",
+    "reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.",
+    "status": "partially_supported",
+    "supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts."
+  },
+  {
+    "claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.",
+    "needed_next": "Add cache-hit joined records for actual reuse decomposition.",
+    "reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.",
+    "status": "supported_for_trace_shape",
+    "supporting_data": "Full compact trace CPU summary in full_trace_summary.json: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass."
+  },
+  {
+    "claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.",
+    "needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.",
+    "reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.",
+    "status": "supported_by_existing_artifact",
+    "supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json."
+  },
+  {
+    "claim": "Elastic transfer-based migration does not improve high-contention 500-request run.",
+    "needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.",
+    "reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.",
+    "status": "supported_by_existing_artifact",
+    "supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv."
+  },
+  {
+    "claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.",
+    "needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.",
+    "reviewer_risk": "Cannot claim interference as causal without Batch 2.",
+    "status": "not_yet_supported",
+    "supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs."
+  },
+  {
+    "claim": "Session hot-spot residual imbalance is suggested but not fully attributed.",
+    "needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.",
+    "reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.",
+    "status": "partially_supported",
+    "supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs."
+  },
+  {
+    "claim": "SRR is not measured by existing fixed-request runs.",
+    "needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.",
+    "reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.",
+    "status": "not_yet_supported",
+    "supporting_data": "No arrival-rate sweep artifacts found."
+  }
+]

analysis/characterization/current_results/comparisons.json

@@ -0,0 +1,95 @@
+[
+  {
+    "baseline": "outputs/gpu_ab_combined",
+    "e2e_p50_delta_pct": 40.870329127661,
+    "e2e_p90_delta_pct": 15.206416995091814,
+    "error_count": [
+      2,
+      13
+    ],
+    "gpu_imbalance_ratio": [
+      3.2445157838416265,
+      11.149056603773586
+    ],
+    "gpu_mean_util": [
+      30.541666666666664,
+      12.367081447963802
+    ],
+    "name": "combined_vs_pdsep_200",
+    "request_count": [
+      200,
+      200
+    ],
+    "success_count": [
+      198,
+      187
+    ],
+    "tpot_p90_delta_pct": 1.3481309269699875,
+    "ttft_p50_delta_pct": 98.06752892925572,
+    "ttft_p90_delta_pct": 44.79649177751278,
+    "variant": "outputs/gpu_ab_pdsep",
+    "wall_clock_delta_pct": 142.27736808267244
+  },
+  {
+    "baseline": "outputs/contention_16s_ts10",
+    "e2e_p50_delta_pct": 11.538788125232664,
+    "e2e_p90_delta_pct": -5.080083318118138,
+    "error_count": [
+      2,
+      2
+    ],
+    "gpu_imbalance_ratio": [
+      2.310775410408662,
+      2.600767754318618
+    ],
+    "gpu_mean_util": [
+      23.030492424242425,
+      26.349561403508773
+    ],
+    "name": "contention_baseline_vs_elastic_500",
+    "request_count": [
+      500,
+      500
+    ],
+    "success_count": [
+      498,
+      498
+    ],
+    "tpot_p90_delta_pct": 13.63098996823875,
+    "ttft_p50_delta_pct": 12.433589435386224,
+    "ttft_p90_delta_pct": 13.412576920999959,
+    "variant": "outputs/contention_16s_elastic",
+    "wall_clock_delta_pct": -0.5645626396767849
+  },
+  {
+    "baseline": "outputs/combined_1000req",
+    "e2e_p50_delta_pct": 202.85189980479385,
+    "e2e_p90_delta_pct": 128.274511020719,
+    "error_count": [
+      2,
+      204
+    ],
+    "gpu_imbalance_ratio": [
+      null,
+      null
+    ],
+    "gpu_mean_util": [
+      null,
+      null
+    ],
+    "name": "combined_1000_vs_pdsep_mooncake",
+    "request_count": [
+      1000,
+      1000
+    ],
+    "success_count": [
+      998,
+      796
+    ],
+    "tpot_p90_delta_pct": -34.83638659447109,
+    "ttft_p50_delta_pct": 781.9835547522864,
+    "ttft_p90_delta_pct": 1030.68607857992,
+    "variant": "outputs/exp3_pd_sep_tp1_mooncake",
+    "wall_clock_delta_pct": 119.18997774599991
+  }
+]

analysis/characterization/current_results/current_results.md

@@ -0,0 +1,77 @@
+# Current Characterization Results
+
+Generated: 2026-05-25T06:52:18.096448+00:00
+Git commit: `21ffb3d4f77956d008b1815a3c0d46e0188ac390`
+
+## Canonical Full-Trace CPU Summary
+
+Source: `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`.
+This is CPU-only parsing of the compact formatted trace with session IDs
+reconstructed from `parent_chat_id` chains.
+
+| Metric | Value |
+|---|---:|
+| Requests | 2,114,220 |
+| Sessions | 1,307,276 |
+| Trace span | 7,199.975 s |
+| Input tokens p50/p90/p99 | 20,030 / 87,855 / 125,527 |
+| Output tokens p50/p90/p99 | 80 / 811 / 6,615 |
+| Input/output ratio p50/p90/p99 | 217.8 / 1,204.4 / 4,251.6 |
+| Turns/session p50/p90/p99/max | 1 / 1 / 18 / 3,091 |
+| Session input tokens p50/p90/p99/max | 12,486 / 72,676 / 974,934 / 156,756,974 |
+| Top 1% / 5% / 10% sessions by input-token mass | 46.5% / 66.5% / 74.6% |
+
+Immediate reading: the full trace strongly supports long-input/short-output
+and heavy-tailed session token mass. It does **not** by itself prove online
+sequentiality or actual cache-hit reuse; those require runtime timestamps and
+cache-hit fields.
+
+## Existing Run Summaries
+
+| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |
+|---|---:|---:|---:|---:|---:|---:|
+| outputs/gpu_ab_combined | 198/200 | 1.01/9.36 | 5.05/30.2 | 0.0732 | 30.5 | 3.24 |
+| outputs/gpu_ab_pdsep | 187/200 | 1.99/13.5 | 7.11/34.8 | 0.0742 | 12.4 | 11.1 |
+| outputs/contention_16s_ts10 | 498/500 | 0.826/9.71 | 5.8/51 | 0.103 | 23 | 2.31 |
+| outputs/contention_16s_elastic | 498/500 | 0.929/11 | 6.47/48.4 | 0.117 | 26.3 | 2.6 |
+| outputs/combined_1000req | 998/1000 | 0.393/2.57 | 3.22/28 | 0.113 | n/a | n/a |
+| outputs/exp3_pd_sep_tp1_mooncake | 796/1000 | 3.47/29 | 9.75/63.9 | 0.0739 | n/a | n/a |
+
+## Pairwise Comparisons
+
+| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |
+|---|---:|---:|---:|---:|---:|---:|
+| combined_vs_pdsep_200 | +98.1% | +44.8% | +40.9% | +15.2% | +1.3% | +142.3% |
+| contention_baseline_vs_elastic_500 | +12.4% | +13.4% | +11.5% | -5.1% | +13.6% | -0.6% |
+| combined_1000_vs_pdsep_mooncake | +782.0% | +1030.7% | +202.9% | +128.3% | -34.8% | +119.2% |
+
+## What We Can Say Now
+
+- **partially_supported**: Batch 0 substrate audit is only partially complete for existing runs.
+  Supporting data: metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.
+  Next: Add request dispatch and finish/error timestamps to future replayer/proxy metrics.
+- **supported_for_trace_shape**: Batch 1 workload shape can be characterized from formatted traces and metrics.
+  Supporting data: full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass.
+  Next: add cache-hit joined records for actual reuse decomposition.
+- **supported_by_existing_artifact**: Static PD separation is worse than combined in existing 200-request GPU A/B.
+  Supporting data: outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.
+  Next: Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.
+- **supported_by_existing_artifact**: Elastic transfer-based migration does not improve high-contention 500-request run.
+  Supporting data: outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.
+  Next: Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.
+- **not_yet_supported**: PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.
+  Supporting data: No decode-step and prefill-overlap timestamp artifact found in summarized runs.
+  Next: Run Batch 2 controlled same-worker/different-worker injection with step timestamps.
+- **partially_supported**: Session hot-spot residual imbalance is suggested but not fully attributed.
+  Supporting data: gpu_util.csv shows per-GPU mean-util imbalance in existing runs.
+  Next: Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.
+- **not_yet_supported**: SRR is not measured by existing fixed-request runs.
+  Supporting data: No arrival-rate sweep artifacts found.
+  Next: Implement Batch 4 Poisson session-arrival SRR sweep.
+
+## Main Reviewer Risks
+
+- **high**: Session sequentiality not proven - Add dispatch/finish timestamps and run Batch 0 before SRR claims.
+- **medium**: Legacy PD-sep data may not match final methodology - Use fresh PD matrix for paper-grade claims.
+- **medium**: GPU util is not a sufficient hot-spot proof - Add route-decision and per-worker queue logs for Batch 3.
+- **medium**: Cache reuse decomposition is incomplete without joined hash/cache-hit data - Emit hash_ids/session_id/cached_tokens in the same per-request record.

analysis/characterization/current_results/full_trace_summary.json

@@ -0,0 +1,56 @@
+{
+  "input": {
+    "count": 2114220,
+    "max": 202371,
+    "mean": 33637.38370084476,
+    "p50": 20030.0,
+    "p90": 87855.1000000001,
+    "p95": 104738.0,
+    "p99": 125527.0
+  },
+  "input_output_ratio": {
+    "count": 2108130,
+    "max": 143664.0,
+    "mean": 534.3516074828406,
+    "p50": 217.8,
+    "p90": 1204.3769610389616,
+    "p95": 1814.3478327228322,
+    "p99": 4251.585499999998
+  },
+  "output": {
+    "count": 2114220,
+    "max": 132665,
+    "mean": 444.97059624826176,
+    "p50": 80.0,
+    "p90": 811.0,
+    "p95": 2213.0,
+    "p99": 6614.810000000056
+  },
+  "path": "/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl",
+  "records": 2114220,
+  "session_input_tokens": {
+    "count": 1307276,
+    "max": 156756974,
+    "mean": 54400.77639916896,
+    "p50": 12486.0,
+    "p90": 72676.0,
+    "p95": 108523.25,
+    "p99": 974933.75
+  },
+  "sessions": 1307276,
+  "top_session_input_fraction": {
+    "top10pct": 0.7464402483455778,
+    "top1pct": 0.46456810581415175,
+    "top5pct": 0.6651718740752172
+  },
+  "trace_span_s": 7199.975,
+  "turns_per_session": {
+    "count": 1307276,
+    "max": 3091,
+    "mean": 1.6172713336739908,
+    "p50": 1.0,
+    "p90": 1.0,
+    "p95": 2.0,
+    "p99": 18.0
+  }
+}

analysis/characterization/current_results/main_claim_allowed_runs.md

@@ -0,0 +1,66 @@
+# Main-Claim Allowed Runs
+
+Status: current audit gate
+Date: 2026-05-25
+
+## Allowed For Workload-Shape Claims
+
+These artifacts can support trace/workload characterization claims:
+
+- `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`
+  - Compact formatted full trace.
+  - CPU summary recorded in `full_trace_summary.json`.
+  - Supports long-input/short-output and session token-mass skew claims.
+  - Does not prove runtime cache hits or online sequentiality.
+- `traces/w600_r0.0015_st30.jsonl`
+  - Local sampled trace.
+  - Useful for local dry runs and figure generation.
+  - Not the canonical full-trace source.
+
+## Allowed For Legacy Baseline Sanity Claims
+
+These existing runs can support sanity-level comparisons, but not final
+paper-grade SRR claims:
+
+- `outputs/gpu_ab_combined`
+- `outputs/gpu_ab_pdsep`
+- `outputs/contention_16s_ts10`
+- `outputs/contention_16s_elastic`
+- `outputs/combined_1000req`
+- `outputs/exp3_pd_sep_tp1_mooncake`
+
+Allowed claims:
+
+- Static PD-sep was worse than combined in these existing fixed-request runs.
+- Elastic transfer-based migration did not improve the summarized 500-request
+  high-contention run.
+- GPU-util imbalance exists in these artifacts.
+
+Disallowed claims:
+
+- Online SRR.
+- Per-session sequentiality.
+- Causal attribution of prefill/decode interference.
+- Causal attribution of session hot spots from GPU utilization alone.
+
+## Not Yet Allowed For Main Claims
+
+The following need fresh instrumentation or fresh runs:
+
+- Batch 2 prefill/decode interference.
+- Batch 3 session hot-spot root cause.
+- Batch 4 sustainable request rate.
+- Batch 5 failure attribution near SRR boundary.
+
+## Required Upgrade Before Paper-Grade Claims
+
+Future main-claim runs must include:
+
+- per-request actual dispatch timestamp;
+- per-request finish/error timestamp;
+- route decision and selected worker;
+- per-worker queue delay;
+- per-worker KV occupancy;
+- per-worker APC/cache-hit snapshot;
+- attempted/completed/error/goodput counters;
+- session-causal load generation.

analysis/characterization/current_results/reproduction_commands.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Rebuild this current-results audit package.
+python3 analysis/characterization/summarize_runs.py --output-dir analysis/characterization/current_results --runs outputs/gpu_ab_combined outputs/gpu_ab_pdsep outputs/contention_16s_ts10 outputs/contention_16s_elastic outputs/combined_1000req outputs/exp3_pd_sep_tp1_mooncake
+
+# Example Batch 0/1 local trace analysis.
+python3 analysis/characterization/analyze.py \
+  --trace traces/w600_r0.0015_st30.jsonl \
+  --kv-bytes-per-token 98304 \
+  --task-name w600_local_full_trace \
+  --overwrite
+
+# CPU-only full compact trace summary was computed on dash0 from:
+# /home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
+# Recompute either by running analyze.py on dash0, or by copying that compact
+# formatted JSONL locally. Do not use the 487G raw file directly.

analysis/characterization/current_results/reviewer_risk_register.json

@@ -0,0 +1,26 @@
+[
+  {
+    "evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.",
+    "mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.",
+    "risk": "Session sequentiality not proven",
+    "severity": "high"
+  },
+  {
+    "evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.",
+    "mitigation": "Use fresh PD matrix for paper-grade claims.",
+    "risk": "Legacy PD-sep data may not match final methodology",
+    "severity": "medium"
+  },
+  {
+    "evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.",
+    "mitigation": "Add route-decision and per-worker queue logs for Batch 3.",
+    "risk": "GPU util is not a sufficient hot-spot proof",
+    "severity": "medium"
+  },
+  {
+    "evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.",
+    "mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.",
+    "risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data",
+    "severity": "medium"
+  }
+]

analysis/characterization/current_results/reviewer_risk_register.md

@@ -0,0 +1,8 @@
+# Reviewer Risk Register
+
+| Risk | Severity | Evidence | Mitigation |
+|---|---|---|---|
+| Session sequentiality not proven | `high` | Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps. | Add dispatch/finish timestamps and run Batch 0 before SRR claims. |
+| Legacy PD-sep data may not match final methodology | `medium` | PD matrix scaffold exists separately; some old runs used earlier flags/methodology. | Use fresh PD matrix for paper-grade claims. |
+| GPU util is not a sufficient hot-spot proof | `medium` | Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership. | Add route-decision and per-worker queue logs for Batch 3. |
+| Cache reuse decomposition is incomplete without joined hash/cache-hit data | `medium` | Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts. | Emit hash_ids/session_id/cached_tokens in the same per-request record. |

analysis/characterization/current_results/run_summaries.json

@@ -0,0 +1,720 @@
+[
+  {
+    "apc_summary": {
+      "reason": "apc.txt missing",
+      "status": "unavailable"
+    },
+    "artifact_availability": {
+      "apc_txt": false,
+      "breakdown_json": false,
+      "gpu_util_csv": true,
+      "metrics_jsonl": true,
+      "metrics_summary_json": true
+    },
+    "breakdown_summary": {
+      "reason": "breakdown.json missing",
+      "status": "unavailable"
+    },
+    "error_count": 2,
+    "exists": true,
+    "external_cache_hit_ratio": null,
+    "gpu_summary": {
+      "gpu_count": 8,
+      "max_mean_util_pct": 63.166666666666664,
+      "max_min_ratio": 3.2445157838416265,
+      "mean_util_pct": 30.541666666666664,
+      "min_mean_util_pct": 19.46875,
+      "per_gpu_mean_util_pct": {
+        "0": 29.145833333333332,
+        "1": 20.041666666666668,
+        "2": 25.0,
+        "3": 63.166666666666664,
+        "4": 21.927083333333332,
+        "5": 34.90625,
+        "6": 19.46875,
+        "7": 30.677083333333332
+      },
+      "status": "available",
+      "stddev_across_gpu_mean_util_pct": 13.337857305429534
+    },
+    "latency_stats_s": {
+      "count": 198.0,
+      "mean": 13.01780862723021,
+      "p50": 5.048548387829214,
+      "p90": 30.18109704903327,
+      "p99": 119.01174414204434
+    },
+    "metrics_jsonl_rows": 200,
+    "metrics_summary_available": true,
+    "prefix_cache_hit_ratio": 0.0,
+    "request_count": 200,
+    "run": "outputs/gpu_ab_combined",
+    "session_summary": {
+      "request_cached_tokens": {
+        "count": 200,
+        "max": 0.0,
+        "mean": 0.0,
+        "p50": 0.0,
+        "p90": 0.0,
+        "p95": 0.0,
+        "p99": 0.0
+      },
+      "request_input_tokens": {
+        "count": 200,
+        "max": 111927.0,
+        "mean": 29318.375,
+        "p50": 21376.0,
+        "p90": 81218.19999999998,
+        "p95": 87571.2,
+        "p99": 101619.27999999994
+      },
+      "request_output_tokens": {
+        "count": 200,
+        "max": 5083.0,
+        "mean": 257.675,
+        "p50": 72.0,
+        "p90": 664.3,
+        "p95": 1063.9999999999993,
+        "p99": 3300.1799999999994
+      },
+      "session_count": 145,
+      "session_input_tokens": {
+        "count": 145,
+        "max": 1567423.0,
+        "mean": 40439.137931034486,
+        "p50": 10879.0,
+        "p90": 72438.39999999997,
+        "p95": 106934.39999999988,
+        "p99": 374927.28
+      },
+      "status": "available",
+      "top_session_input_fraction": {
+        "top_10pct": 0.6588294883328288,
+        "top_1pct": 0.33276622595897626,
+        "top_5pct": 0.5534430199490934
+      },
+      "turns_per_session": {
+        "count": 145,
+        "max": 21.0,
+        "mean": 1.3793103448275863,
+        "p50": 1.0,
+        "p90": 1.0,
+        "p95": 2.0,
+        "p99": 10.560000000000002
+      }
+    },
+    "success_count": 198,
+    "tpot_stats_s": {
+      "count": 198.0,
+      "mean": 0.04929455188644857,
+      "p50": 0.03717198147904128,
+      "p90": 0.07317714408040046,
+      "p99": 0.10039294634234945
+    },
+    "ttft_stats_s": {
+      "count": 198.0,
+      "mean": 3.8987488178453984,
+      "p50": 1.0068706551101059,
+      "p90": 9.355209570843726,
+      "p99": 33.855437273858115
+    },
+    "wall_clock_s": 483.543720243033
+  },
+  {
+    "apc_summary": {
+      "reason": "apc.txt missing",
+      "status": "unavailable"
+    },
+    "artifact_availability": {
+      "apc_txt": false,
+      "breakdown_json": false,
+      "gpu_util_csv": true,
+      "metrics_jsonl": true,
+      "metrics_summary_json": true
+    },
+    "breakdown_summary": {
+      "reason": "breakdown.json missing",
+      "status": "unavailable"
+    },
+    "error_count": 13,
+    "exists": true,
+    "external_cache_hit_ratio": null,
+    "gpu_summary": {
+      "gpu_count": 8,
+      "max_mean_util_pct": 26.737556561085974,
+      "max_min_ratio": 11.149056603773586,
+      "mean_util_pct": 12.367081447963802,
+      "min_mean_util_pct": 2.3981900452488687,
+      "per_gpu_mean_util_pct": {
+        "0": 26.737556561085974,
+        "1": 14.44343891402715,
+        "2": 19.036199095022624,
+        "3": 7.4389140271493215,
+        "4": 2.3981900452488687,
+        "5": 8.841628959276019,
+        "6": 11.963800904977376,
+        "7": 8.076923076923077
+      },
+      "status": "available",
+      "stddev_across_gpu_mean_util_pct": 7.15857427072345
+    },
+    "latency_stats_s": {
+      "count": 187.0,
+      "mean": 15.366858840781827,
+      "p50": 7.111906730104238,
+      "p90": 34.77056052000262,
+      "p99": 119.2064151619561
+    },
+    "metrics_jsonl_rows": 200,
+    "metrics_summary_available": true,
+    "prefix_cache_hit_ratio": 0.0,
+    "request_count": 200,
+    "run": "outputs/gpu_ab_pdsep",
+    "session_summary": {
+      "request_cached_tokens": {
+        "count": 200,
+        "max": 0.0,
+        "mean": 0.0,
+        "p50": 0.0,
+        "p90": 0.0,
+        "p95": 0.0,
+        "p99": 0.0
+      },
+      "request_input_tokens": {
+        "count": 200,
+        "max": 111927.0,
+        "mean": 29318.375,
+        "p50": 21376.0,
+        "p90": 81218.19999999998,
+        "p95": 87571.2,
+        "p99": 101619.27999999994
+      },
+      "request_output_tokens": {
+        "count": 200,
+        "max": 5097.0,
+        "mean": 290.18,
+        "p50": 75.5,
+        "p90": 685.8,
+        "p95": 1102.5999999999997,
+        "p99": 3433.6599999999844
+      },
+      "session_count": 145,
+      "session_input_tokens": {
+        "count": 145,
+        "max": 1567423.0,
+        "mean": 40439.137931034486,
+        "p50": 10879.0,
+        "p90": 72438.39999999997,
+        "p95": 106934.39999999988,
+        "p99": 374927.28
+      },
+      "status": "available",
+      "top_session_input_fraction": {
+        "top_10pct": 0.6588294883328288,
+        "top_1pct": 0.33276622595897626,
+        "top_5pct": 0.5534430199490934
+      },
+      "turns_per_session": {
+        "count": 145,
+        "max": 21.0,
+        "mean": 1.3793103448275863,
+        "p50": 1.0,
+        "p90": 1.0,
+        "p95": 2.0,
+        "p99": 10.560000000000002
+      }
+    },
+    "success_count": 187,
+    "tpot_stats_s": {
+      "count": 187.0,
+      "mean": 0.05175559044923285,
+      "p50": 0.04020531923068981,
+      "p90": 0.07416366779122173,
+      "p99": 0.10770365060609463
+    },
+    "ttft_stats_s": {
+      "count": 187.0,
+      "mean": 5.3933139261814524,
+      "p50": 1.9942838260903955,
+      "p90": 13.546015257015824,
+      "p99": 39.30951087013818
+    },
+    "wall_clock_s": 1171.516998933861
+  },
+  {
+    "apc_summary": {
+      "reason": "apc.txt missing",
+      "status": "unavailable"
+    },
+    "artifact_availability": {
+      "apc_txt": false,
+      "breakdown_json": false,
+      "gpu_util_csv": true,
+      "metrics_jsonl": true,
+      "metrics_summary_json": true
+    },
+    "breakdown_summary": {
+      "reason": "breakdown.json missing",
+      "status": "unavailable"
+    },
+    "error_count": 2,
+    "exists": true,
+    "external_cache_hit_ratio": 0.0,
+    "gpu_summary": {
+      "gpu_count": 8,
+      "max_mean_util_pct": 40.095454545454544,
+      "max_min_ratio": 2.310775410408662,
+      "mean_util_pct": 23.030492424242425,
+      "min_mean_util_pct": 17.35151515151515,
+      "per_gpu_mean_util_pct": {
+        "0": 22.243939393939392,
+        "1": 22.798484848484847,
+        "2": 22.09090909090909,
+        "3": 40.095454545454544,
+        "4": 17.484848484848484,
+        "5": 20.225757575757576,
+        "6": 17.35151515151515,
+        "7": 21.953030303030303
+      },
+      "status": "available",
+      "stddev_across_gpu_mean_util_pct": 6.752783259358802
+    },
+    "latency_stats_s": {
+      "count": 498.0,
+      "mean": 21.908606036002105,
+      "p50": 5.798741589998826,
+      "p90": 51.00022796400299,
+      "p99": 201.6967467680006
+    },
+    "metrics_jsonl_rows": 500,
+    "metrics_summary_available": true,
+    "prefix_cache_hit_ratio": 0.0,
+    "request_count": 500,
+    "run": "outputs/contention_16s_ts10",
+    "session_summary": {
+      "request_cached_tokens": {
+        "count": 500,
+        "max": 0.0,
+        "mean": 0.0,
+        "p50": 0.0,
+        "p90": 0.0,
+        "p95": 0.0,
+        "p99": 0.0
+      },
+      "request_input_tokens": {
+        "count": 500,
+        "max": 134865.0,
+        "mean": 29190.97,
+        "p50": 17756.0,
+        "p90": 77365.70000000006,
+        "p95": 87790.89999999997,
+        "p99": 111360.44
+      },
+      "request_output_tokens": {
+        "count": 500,
+        "max": 16427.0,
+        "mean": 404.286,
+        "p50": 75.0,
+        "p90": 863.7000000000004,
+        "p95": 2384.45,
+        "p99": 4928.559999999999
+      },
+      "session_count": 347,
+      "session_input_tokens": {
+        "count": 347,
+        "max": 1929072.0,
+        "mean": 42061.916426512966,
+        "p50": 11342.0,
+        "p90": 68297.20000000004,
+        "p95": 116213.79999999993,
+        "p99": 521898.36000000057
+      },
+      "status": "available",
+      "top_session_input_fraction": {
+        "top_10pct": 0.6824335059780473,
+        "top_1pct": 0.32352388426969025,
+        "top_5pct": 0.579125805000656
+      },
+      "turns_per_session": {
+        "count": 347,
+        "max": 29.0,
+        "mean": 1.440922190201729,
+        "p50": 1.0,
+        "p90": 1.0,
+        "p95": 3.0,
+        "p99": 10.54000000000002
+      }
+    },
+    "success_count": 498,
+    "tpot_stats_s": {
+      "count": 498.0,
+      "mean": 0.07772806444638436,
+      "p50": 0.06774563665097065,
+      "p90": 0.10325221968416468,
+      "p99": 0.503443661631568
+    },
+    "ttft_stats_s": {
+      "count": 498.0,
+      "mean": 3.5127640336366928,
+      "p50": 0.8264882279981975,
+      "p90": 9.71474659699743,
+      "p99": 34.40133859900379
+    },
+    "wall_clock_s": 1472.1331836190002
+  },
+  {
+    "apc_summary": {
+      "line_count": 8,
+      "preview": "inst_0: prefix=45.7% ext=24.7%\ninst_1: prefix=33.1% ext=13.3%\ninst_2: prefix=14.2% ext=0.0%\ninst_3: prefix=50.7% ext=6.3%\ninst_4: prefix=33.6% ext=0.0%\ninst_5: prefix=66.6% ext=9.2%\ninst_6: prefix=42.1% ext=18.8%\ninst_7: prefix=36.7% ext=7.9%",
+      "status": "available"
+    },
+    "artifact_availability": {
+      "apc_txt": true,
+      "breakdown_json": true,
+      "gpu_util_csv": true,
+      "metrics_jsonl": true,
+      "metrics_summary_json": true
+    },
+    "breakdown_summary": {
+      "field_sample": [
+        "cache_hit",
+        "estimated_new_tokens",
+        "input_length",
+        "policy",
+        "request_id",
+        "route_class",
+        "routed_to",
+        "t_done",
+        "t_first_token",
+        "t_proxy_recv"
+      ],
+      "mode_counts": {
+        "HEAVY_COLO": 111,
+        "HEAVY_OFFLOAD": 13,
+        "MEDIUM": 163,
+        "WARM": 213
+      },
+      "route_counts": {
+        "linear": 487
+      },
+      "row_count": 500,
+      "status": "available"
+    },
+    "error_count": 2,
+    "exists": true,
+    "external_cache_hit_ratio": 0.0,
+    "gpu_summary": {
+      "gpu_count": 8,
+      "max_mean_util_pct": 47.54385964912281,
+      "max_min_ratio": 2.600767754318618,
+      "mean_util_pct": 26.349561403508773,
+      "min_mean_util_pct": 18.280701754385966,
+      "per_gpu_mean_util_pct": {
+        "0": 22.410526315789475,
+        "1": 26.57894736842105,
+        "2": 23.54035087719298,
+        "3": 47.54385964912281,
+        "4": 18.96842105263158,
+        "5": 26.403508771929825,
+        "6": 18.280701754385966,
+        "7": 27.07017543859649
+      },
+      "status": "available",
+      "stddev_across_gpu_mean_util_pct": 8.6079068381278
+    },
+    "latency_stats_s": {
+      "count": 498.0,
+      "mean": 21.780466573040204,
+      "p50": 6.467846095998539,
+      "p90": 48.40937389100145,
+      "p99": 196.93401125499804
+    },
+    "metrics_jsonl_rows": 500,
+    "metrics_summary_available": true,
+    "prefix_cache_hit_ratio": 0.0,
+    "request_count": 500,
+    "run": "outputs/contention_16s_elastic",
+    "session_summary": {
+      "request_cached_tokens": {
+        "count": 500,
+        "max": 0.0,
+        "mean": 0.0,
+        "p50": 0.0,
+        "p90": 0.0,
+        "p95": 0.0,
+        "p99": 0.0
+      },
+      "request_input_tokens": {
+        "count": 500,
+        "max": 134865.0,
+        "mean": 29190.97,
+        "p50": 17756.0,
+        "p90": 77365.70000000006,
+        "p95": 87790.89999999997,
+        "p99": 111360.44
+      },
+      "request_output_tokens": {
+        "count": 500,
+        "max": 16427.0,
+        "mean": 395.466,
+        "p50": 75.0,
+        "p90": 809.8000000000013,
+        "p95": 2264.2999999999943,
+        "p99": 4928.559999999999
+      },
+      "session_count": 347,
+      "session_input_tokens": {
+        "count": 347,
+        "max": 1929072.0,
+        "mean": 42061.916426512966,
+        "p50": 11342.0,
+        "p90": 68297.20000000004,
+        "p95": 116213.79999999993,
+        "p99": 521898.36000000057
+      },
+      "status": "available",
+      "top_session_input_fraction": {
+        "top_10pct": 0.6824335059780473,
+        "top_1pct": 0.32352388426969025,
+        "top_5pct": 0.579125805000656
+      },
+      "turns_per_session": {
+        "count": 347,
+        "max": 29.0,
+        "mean": 1.440922190201729,
+        "p50": 1.0,
+        "p90": 1.0,
+        "p95": 3.0,
+        "p99": 10.54000000000002
+      }
+    },
+    "success_count": 498,
+    "tpot_stats_s": {
+      "count": 498.0,
+      "mean": 0.08306021258238128,
+      "p50": 0.06973490711122092,
+      "p90": 0.117326519391297,
+      "p99": 0.5879216773072795
+    },
+    "ttft_stats_s": {
+      "count": 498.0,
+      "mean": 3.757858794331448,
+      "p50": 0.9292503809992922,
+      "p90": 11.017744456999935,
+      "p99": 34.20241540799907
+    },
+    "wall_clock_s": 1463.8220696580029
+  },
+  {
+    "apc_summary": {
+      "reason": "apc.txt missing",
+      "status": "unavailable"
+    },
+    "artifact_availability": {
+      "apc_txt": false,
+      "breakdown_json": false,
+      "gpu_util_csv": false,
+      "metrics_jsonl": true,
+      "metrics_summary_json": true
+    },
+    "breakdown_summary": {
+      "reason": "breakdown.json missing",
+      "status": "unavailable"
+    },
+    "error_count": 2,
+    "exists": true,
+    "external_cache_hit_ratio": null,
+    "gpu_summary": {
+      "reason": "gpu_util.csv missing",
+      "status": "unavailable"
+    },
+    "latency_stats_s": {
+      "count": 998.0,
+      "mean": 15.744418202954922,
+      "p50": 3.220371481962502,
+      "p90": 28.005282410187647,
+      "p99": 204.00479479599744
+    },
+    "metrics_jsonl_rows": 1000,
+    "metrics_summary_available": true,
+    "prefix_cache_hit_ratio": 0.5443149471989938,
+    "request_count": 1000,
+    "run": "outputs/combined_1000req",
+    "session_summary": {
+      "request_cached_tokens": {
+        "count": 1000,
+        "max": 0.0,
+        "mean": 0.0,
+        "p50": 0.0,
+        "p90": 0.0,
+        "p95": 0.0,
+        "p99": 0.0
+      },
+      "request_input_tokens": {
+        "count": 1000,
+        "max": 171427.0,
+        "mean": 31611.14,
+        "p50": 19798.0,
+        "p90": 82584.20000000001,
+        "p95": 93305.64999999997,
+        "p99": 111947.83999999998
+      },
+      "request_output_tokens": {
+        "count": 1000,
+        "max": 41233.0,
+        "mean": 392.579,
+        "p50": 70.0,
+        "p90": 685.8000000000002,
+        "p95": 1834.6999999999975,
+        "p99": 4584.459999999996
+      },
+      "session_count": 671,
+      "session_input_tokens": {
+        "count": 671,
+        "max": 2320064.0,
+        "mean": 47110.49180327869,
+        "p50": 12394.0,
+        "p90": 71875.0,
+        "p95": 112984.5,
+        "p99": 760591.4999999925
+      },
+      "status": "available",
+      "top_session_input_fraction": {
+        "top_10pct": 0.6915149849072194,
+        "top_1pct": 0.36223103627392117,
+        "top_5pct": 0.5945521736957288
+      },
+      "turns_per_session": {
+        "count": 671,
+        "max": 36.0,
+        "mean": 1.4903129657228018,
+        "p50": 1.0,
+        "p90": 1.0,
+        "p95": 3.0,
+        "p99": 12.599999999999909
+      }
+    },
+    "success_count": 998,
+    "tpot_stats_s": {
+      "count": 998.0,
+      "mean": 0.05681003092240363,
+      "p50": 0.04029440155459775,
+      "p90": 0.11343103184021618,
+      "p99": 0.2962149563411783
+    },
+    "ttft_stats_s": {
+      "count": 998.0,
+      "mean": 1.0351265607308877,
+      "p50": 0.39304836583323777,
+      "p90": 2.5655168020166457,
+      "p99": 8.210839052917436
+    },
+    "wall_clock_s": 1340.061015482992
+  },
+  {
+    "apc_summary": {
+      "reason": "apc.txt missing",
+      "status": "unavailable"
+    },
+    "artifact_availability": {
+      "apc_txt": false,
+      "breakdown_json": false,
+      "gpu_util_csv": false,
+      "metrics_jsonl": true,
+      "metrics_summary_json": true
+    },
+    "breakdown_summary": {
+      "reason": "breakdown.json missing",
+      "status": "unavailable"
+    },
+    "error_count": 204,
+    "exists": true,
+    "external_cache_hit_ratio": null,
+    "gpu_summary": {
+      "reason": "gpu_util.csv missing",
+      "status": "unavailable"
+    },
+    "latency_stats_s": {
+      "count": 796.0,
+      "mean": 23.561836449842698,
+      "p50": 9.752956213895231,
+      "p90": 63.928921481827274,
+      "p99": 168.2179710320197
+    },
+    "metrics_jsonl_rows": 1000,
+    "metrics_summary_available": true,
+    "prefix_cache_hit_ratio": 0.0,
+    "request_count": 1000,
+    "run": "outputs/exp3_pd_sep_tp1_mooncake",
+    "session_summary": {
+      "request_cached_tokens": {
+        "count": 1000,
+        "max": 0.0,
+        "mean": 0.0,
+        "p50": 0.0,
+        "p90": 0.0,
+        "p95": 0.0,
+        "p99": 0.0
+      },
+      "request_input_tokens": {
+        "count": 1000,
+        "max": 171427.0,
+        "mean": 31611.14,
+        "p50": 19798.0,
+        "p90": 82584.20000000001,
+        "p95": 93305.64999999997,
+        "p99": 111947.83999999998
+      },
+      "request_output_tokens": {
+        "count": 1000,
+        "max": 41233.0,
+        "mean": 412.176,
+        "p50": 74.0,
+        "p90": 759.7000000000008,
+        "p95": 1834.6999999999975,
+        "p99": 4928.559999999999
+      },
+      "session_count": 671,
+      "session_input_tokens": {
+        "count": 671,
+        "max": 2320064.0,
+        "mean": 47110.49180327869,
+        "p50": 12394.0,
+        "p90": 71875.0,
+        "p95": 112984.5,
+        "p99": 760591.4999999925
+      },
+      "status": "available",
+      "top_session_input_fraction": {
+        "top_10pct": 0.6915149849072194,
+        "top_1pct": 0.36223103627392117,
+        "top_5pct": 0.5945521736957288
+      },
+      "turns_per_session": {
+        "count": 671,
+        "max": 36.0,
+        "mean": 1.4903129657228018,
+        "p50": 1.0,
+        "p90": 1.0,
+        "p95": 3.0,
+        "p99": 12.599999999999909
+      }
+    },
+    "success_count": 796,
+    "tpot_stats_s": {
+      "count": 796.0,
+      "mean": 0.05211825330315642,
+      "p50": 0.06622354774330945,
+      "p90": 0.07391575907026088,
+      "p99": 0.10499966285609896
+    },
+    "ttft_stats_s": {
+      "count": 796.0,
+      "mean": 10.753033336598003,
+      "p50": 3.4666219488717616,
+      "p90": 29.00794132403098,
+      "p99": 81.7531874559354
+    },
+    "wall_clock_s": 2937.2794416199904
+  }
+]

analysis/characterization/protocols.md

@@ -0,0 +1,264 @@
+# Characterization Protocols For Remaining Batches
+
+Status: implementation protocol and audit checklist
+Date: 2026-05-25
+
+This file completes the `analysis/characterization` scaffold for the TODO
+list. It separates what is already implemented from what requires fresh GPU
+runs or new engine/proxy instrumentation.
+
+## Implemented Now
+
+### Batch 0/1 Analyzer
+
+Use:
+
+```bash
+python3 analysis/characterization/analyze.py \
+  --trace traces/w600_r0.0015_st30.jsonl \
+  --kv-bytes-per-token 98304 \
+  --task-name w600_local_full_trace \
+  --overwrite
+```
+
+The analyzer writes:
+
+- `manifest.json`
+- `summary.json`
+- `summary.md`
+- `audit.md`
+- `session_concurrency.json`
+- `session_arrival_stats.json`
+- `turn_interval_stats.json`
+- `trace_profile.json`
+- `workload_summary.json`
+- `kv_footprint_summary.json`
+- `reuse_decomposition.json`
+- `session_skew.json`
+- `append_delta_stats.json`
+
+Limitations:
+
+- Actual online sequentiality requires dispatch and finish/error timestamps.
+  Existing `metrics.jsonl` artifacts generally do not contain these fields.
+- Actual reuse decomposition requires `cached_tokens`/`cache_hit`, `hash_ids`,
+  and `session_id` in the same joinable request record.
+
+### Existing-Run Audit
+
+Use:
+
+```bash
+python3 analysis/characterization/summarize_runs.py
+```
+
+The script writes an audit package under:
+
+```text
+analysis/characterization/current_results/
+```
+
+It summarizes already completed runs and explicitly marks which claims are
+supported, partially supported, or not yet supported.
+
+## Batch 2 Protocol: PD-Colo Prefill/Decode Interference
+
+Purpose:
+
+Prove whether same-worker prefill overlap increases decode TPOT/queue delay.
+
+Required new instrumentation:
+
+- per-request dispatch timestamp
+- per-request finish/error timestamp
+- per decode step timestamp
+- decode step worker id
+- prefill chunk start/end timestamp
+- prefill worker id
+- request/session id associated with each prefill chunk
+
+Required arms:
+
+1. decode-only steady load
+2. decode + same-worker heavy prefill injection
+3. decode + different-worker heavy prefill injection
+4. trace replay with overlap labels
+
+Required sweep:
+
+```text
+uncached_prefill_tokens in {2k, 8k, 16k, 32k, 64k}
+chunked_prefill_size in available engine values
+```
+
+Required outputs:
+
+- `interference_microbench_summary.json`
+- `decode_step_timeseries.csv`
+- `prefill_overlap_events.jsonl`
+- `interference_index.json`
+- TPOT timeline figure with prefill overlays
+- same-worker vs different-worker TPOT boxplot
+
+Pass condition:
+
+```text
+TPOT_p90(overlap_same_worker) / TPOT_p90(no_overlap) > 1
+```
+
+and the effect must be materially weaker in the different-worker control.
+
+## Batch 3 Protocol: Session Hot-Spot Residual Imbalance
+
+Purpose:
+
+Prove whether cache-aware/LMetric still leaves hot workers under
+session-heavy skew.
+
+Required new instrumentation:
+
+- route decision per request
+- chosen worker
+- candidate worker scores
+- cache hit / estimated uncached tokens per candidate
+- per-worker request queue length/delay
+- per-worker decode queue length/delay
+- per-worker KV occupancy
+- per-worker APC/cache-hit snapshot
+
+Required arms:
+
+1. corrected LMetric/cache-aware
+2. load-only routing
+3. hard sticky routing
+4. current Unified hybrid
+5. session-mass capped/equalized replay
+
+Required outputs:
+
+- `worker_balance_summary.json`
+- `session_to_worker_map.json`
+- `session_mass_summary.json`
+- `routing_policy_comparison.json`
+- `hotspot_index.json`
+- per-worker queue delay bar
+- APC vs queue delay scatter
+- top-session contribution bar
+- policy tradeoff plot: APC vs hot-spot index
+
+Pass condition:
+
+LMetric/cache-aware must show measurable residual worker skew, and that skew
+must correlate with session token mass or locality.
+
+GPU utilization alone is not enough for this claim.
+
+## Batch 4 Protocol: Sustainable Request Rate
+
+Purpose:
+
+Measure:
+
+```text
+SRR(SLO) = max arrival rate satisfying SLO in steady state
+```
+
+Required load generator behavior:
+
+- open-loop session arrivals, preferably Poisson
+- session-internal sequentiality
+- warmup window
+- steady-state measurement window
+- explicit attempted/completed/error counters
+
+Provisional SLO:
+
+```text
+TTFT_p90 <= T_ttft
+E2E_p90  <= T_e2e
+TPOT_p90 <= T_tpot
+error_rate <= epsilon
+queue length stable
+KV occupancy stable
+```
+
+Required arms:
+
+1. PD-colo corrected LMetric/cache-aware
+2. static PD-disagg
+3. current Unified hybrid
+4. optional hard sticky
+5. optional load-only
+
+Required outputs:
+
+- `srr_curve.json`
+- `lambda_runs/<lambda>/summary.json`
+- `slo_violation_reason.json`
+- `goodput_vs_arrival_rate.json`
+- SRR bar chart
+- latency vs arrival rate curves
+- goodput vs arrival rate
+- queue/KV stability plot near failure point
+
+Pass condition:
+
+Each policy has a measured max sustainable lambda under the same SLO and
+same session-causal arrival process.
+
+## Batch 5 Protocol: Failure Attribution Near SRR Boundary
+
+Purpose:
+
+Explain why each policy fails near SRR.
+
+Required rates:
+
+```text
+lambda = 0.9 * SRR
+lambda = 1.0 * SRR
+lambda = 1.1 * SRR
+```
+
+Labels for each slow/SLO-violating request:
+
+- same-worker prefill overlap
+- hot worker queue
+- high KV occupancy
+- cache miss / large uncached append
+- transfer wait
+- P queue wait
+- D admission wait
+- unknown
+
+Required outputs:
+
+- `slow_request_attribution.jsonl`
+- `failure_breakdown.json`
+- `case_studies.md`
+- `worker_failure_windows.json`
+- violation cause stacked bar
+- slow request waterfall
+- worker timeline near failure
+
+Pass condition:
+
+The analysis must explain whether PD-colo is limited by interference,
+hot-spot, KV pressure, or a mixture, and whether Unified/PUSH underperforms
+because of trigger quality, transfer cost, target admission, or load regime.
+
+## Batch 6 Protocol: Audit Package
+
+Implemented by `summarize_runs.py` for existing runs and extended by fresh
+Batch 2-5 outputs later.
+
+Required files:
+
+- `characterization_claim_matrix.md`
+- `all_figures_index.md`
+- `reviewer_risk_register.md`
+- `reproduction_commands.sh`
+- `main_claim_allowed_runs.md`
+
+Current package intentionally marks Batch 2/4/5 claims as not yet supported
+until fresh instrumented experiments exist.

analysis/characterization/summarize_runs.py

@@ -0,0 +1,666 @@
+#!/usr/bin/env python3
+"""Summarize existing benchmark artifacts for characterization review.
+
+This is a CPU-only companion to ``analyze.py``. It does not run benchmarks.
+It reads completed output directories and produces an audit-oriented package
+that helps decide which TODO claims are currently supported by existing data
+and which still need fresh GPU runs or additional instrumentation.
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import datetime as dt
+import json
+import math
+import statistics
+import subprocess
+from pathlib import Path
+from typing import Any
+
+
+JsonDict = dict[str, Any]
+
+
+DEFAULT_RUNS = [
+    "outputs/gpu_ab_combined",
+    "outputs/gpu_ab_pdsep",
+    "outputs/contention_16s_ts10",
+    "outputs/contention_16s_elastic",
+    "outputs/combined_1000req",
+    "outputs/exp3_pd_sep_tp1_mooncake",
+]
+
+
+def main() -> None:
+    args = parse_args()
+    out_dir = args.output_dir
+    out_dir.mkdir(parents=True, exist_ok=True)
+
+    run_dirs = [Path(p) for p in (args.runs or DEFAULT_RUNS)]
+    summaries = [summarize_run(path) for path in run_dirs]
+    comparisons = build_comparisons(summaries)
+    claim_matrix = build_claim_matrix(summaries, comparisons)
+    risk_register = build_risk_register(summaries)
+
+    write_json(out_dir / "run_summaries.json", summaries)
+    write_json(out_dir / "comparisons.json", comparisons)
+    write_json(out_dir / "claim_matrix.json", claim_matrix)
+    write_json(out_dir / "reviewer_risk_register.json", risk_register)
+    (out_dir / "current_results.md").write_text(
+        render_current_results(summaries, comparisons, claim_matrix, risk_register),
+        encoding="utf-8",
+    )
+    (out_dir / "characterization_claim_matrix.md").write_text(
+        render_claim_matrix(claim_matrix),
+        encoding="utf-8",
+    )
+    (out_dir / "reviewer_risk_register.md").write_text(
+        render_risk_register(risk_register),
+        encoding="utf-8",
+    )
+    (out_dir / "all_figures_index.md").write_text(
+        render_figures_index(summaries),
+        encoding="utf-8",
+    )
+    (out_dir / "reproduction_commands.sh").write_text(
+        render_reproduction_commands(args, run_dirs),
+        encoding="utf-8",
+    )
+    print(f"Wrote run summary package to {out_dir}")
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Summarize existing characterization-relevant output dirs.",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "--runs",
+        nargs="*",
+        default=None,
+        help="Output directories to summarize. Defaults to a small curated set.",
+    )
+    parser.add_argument(
+        "--output-dir",
+        type=Path,
+        default=Path("analysis/characterization/current_results"),
+        help="Directory for generated review artifacts.",
+    )
+    return parser.parse_args()
+
+
+def summarize_run(path: Path) -> JsonDict:
+    metrics_summary = load_json(path / "metrics.summary.json")
+    metrics_rows = load_jsonl(path / "metrics.jsonl")
+    gpu_summary = summarize_gpu(path / "gpu_util.csv")
+    breakdown_summary = summarize_breakdown(path / "breakdown.json")
+    apc_summary = summarize_apc(path / "apc.txt")
+    return {
+        "run": str(path),
+        "exists": path.exists(),
+        "metrics_summary_available": bool(metrics_summary),
+        "metrics_jsonl_rows": len(metrics_rows),
+        "request_count": first_present(metrics_summary, ["request_count"]),
+        "success_count": first_present(metrics_summary, ["success_count"]),
+        "error_count": first_present(metrics_summary, ["error_count"]),
+        "wall_clock_s": first_present(metrics_summary, ["wall_clock_s"]),
+        "latency_stats_s": metrics_summary.get("latency_stats_s"),
+        "ttft_stats_s": metrics_summary.get("ttft_stats_s"),
+        "tpot_stats_s": metrics_summary.get("tpot_stats_s"),
+        "prefix_cache_hit_ratio": metrics_summary.get("prefix_cache_hit_ratio"),
+        "external_cache_hit_ratio": metrics_summary.get("external_cache_hit_ratio"),
+        "session_summary": summarize_sessions(metrics_rows),
+        "gpu_summary": gpu_summary,
+        "breakdown_summary": breakdown_summary,
+        "apc_summary": apc_summary,
+        "artifact_availability": {
+            "metrics_summary_json": (path / "metrics.summary.json").exists(),
+            "metrics_jsonl": (path / "metrics.jsonl").exists(),
+            "gpu_util_csv": (path / "gpu_util.csv").exists(),
+            "breakdown_json": (path / "breakdown.json").exists(),
+            "apc_txt": (path / "apc.txt").exists(),
+        },
+    }
+
+
+def summarize_sessions(rows: list[JsonDict]) -> JsonDict:
+    if not rows:
+        return {
+            "status": "unavailable",
+            "reason": "metrics.jsonl missing",
+        }
+    sessions: dict[str, JsonDict] = {}
+    input_values = []
+    output_values = []
+    cached_values = []
+    for row in rows:
+        sid = str(row.get("session_id", ""))
+        item = sessions.setdefault(
+            sid,
+            {
+                "turns": 0,
+                "input_tokens": 0.0,
+                "output_tokens": 0.0,
+                "cached_tokens": 0.0,
+            },
+        )
+        inp = to_float(row.get("input_length")) or 0.0
+        out = to_float(row.get("actual_output_tokens")) or to_float(row.get("output_length")) or 0.0
+        cached = to_float(row.get("cached_tokens")) or 0.0
+        item["turns"] += 1
+        item["input_tokens"] += inp
+        item["output_tokens"] += out
+        item["cached_tokens"] += cached
+        input_values.append(inp)
+        output_values.append(out)
+        cached_values.append(cached)
+    per_session_input = [v["input_tokens"] for v in sessions.values()]
+    return {
+        "status": "available",
+        "request_input_tokens": stats(input_values),
+        "request_output_tokens": stats(output_values),
+        "request_cached_tokens": stats(cached_values),
+        "session_count": len(sessions),
+        "turns_per_session": stats([v["turns"] for v in sessions.values()]),
+        "session_input_tokens": stats(per_session_input),
+        "top_session_input_fraction": top_contribution(per_session_input),
+    }
+
+
+def summarize_gpu(path: Path) -> JsonDict:
+    if not path.exists():
+        return {
+            "status": "unavailable",
+            "reason": "gpu_util.csv missing",
+        }
+    values: dict[str, list[float]] = {}
+    with path.open() as handle:
+        reader = csv.DictReader(handle)
+        for row in reader:
+            gpu = str(row.get("gpu", ""))
+            util = to_float(row.get("util_pct"))
+            if gpu and util is not None:
+                values.setdefault(gpu, []).append(util)
+    means = {gpu: statistics.fmean(vals) for gpu, vals in values.items() if vals}
+    if not means:
+        return {
+            "status": "unavailable",
+            "reason": "gpu_util.csv had no util_pct rows",
+        }
+    mean_values = list(means.values())
+    return {
+        "status": "available",
+        "gpu_count": len(means),
+        "per_gpu_mean_util_pct": means,
+        "mean_util_pct": statistics.fmean(mean_values),
+        "stddev_across_gpu_mean_util_pct": statistics.pstdev(mean_values),
+        "max_mean_util_pct": max(mean_values),
+        "min_mean_util_pct": min(mean_values),
+        "max_min_ratio": max(mean_values) / max(min(mean_values), 1e-9),
+    }
+
+
+def summarize_breakdown(path: Path) -> JsonDict:
+    if not path.exists():
+        return {
+            "status": "unavailable",
+            "reason": "breakdown.json missing",
+        }
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except Exception as exc:
+        return {
+            "status": "unavailable",
+            "reason": f"failed to parse breakdown: {exc}",
+        }
+    rows: list[JsonDict]
+    if isinstance(data, list):
+        rows = [r for r in data if isinstance(r, dict)]
+    elif isinstance(data, dict):
+        rows = data.get("records") if isinstance(data.get("records"), list) else []
+        if not rows:
+            rows = [data]
+    else:
+        rows = []
+    mode_counts: dict[str, int] = {}
+    route_counts: dict[str, int] = {}
+    for row in rows:
+        mode = row.get("mode") or row.get("execution_mode") or row.get("route_class")
+        route = row.get("route") or row.get("decision") or row.get("policy")
+        if mode is not None:
+            mode_counts[str(mode)] = mode_counts.get(str(mode), 0) + 1
+        if route is not None:
+            route_counts[str(route)] = route_counts.get(str(route), 0) + 1
+    return {
+        "status": "available",
+        "row_count": len(rows),
+        "mode_counts": mode_counts,
+        "route_counts": route_counts,
+        "field_sample": sorted(rows[0].keys()) if rows else [],
+    }
+
+
+def summarize_apc(path: Path) -> JsonDict:
+    if not path.exists():
+        return {
+            "status": "unavailable",
+            "reason": "apc.txt missing",
+        }
+    text = path.read_text(encoding="utf-8", errors="replace")
+    return {
+        "status": "available",
+        "line_count": len(text.splitlines()),
+        "preview": "\n".join(text.splitlines()[:20]),
+    }
+
+
+def build_comparisons(summaries: list[JsonDict]) -> list[JsonDict]:
+    by_run = {s["run"]: s for s in summaries}
+    pairs = [
+        ("combined_vs_pdsep_200", "outputs/gpu_ab_combined", "outputs/gpu_ab_pdsep"),
+        ("contention_baseline_vs_elastic_500", "outputs/contention_16s_ts10", "outputs/contention_16s_elastic"),
+        ("combined_1000_vs_pdsep_mooncake", "outputs/combined_1000req", "outputs/exp3_pd_sep_tp1_mooncake"),
+    ]
+    out = []
+    for name, base, variant in pairs:
+        if base not in by_run or variant not in by_run:
+            continue
+        out.append(compare_pair(name, by_run[base], by_run[variant]))
+    return out
+
+
+def compare_pair(name: str, base: JsonDict, variant: JsonDict) -> JsonDict:
+    return {
+        "name": name,
+        "baseline": base["run"],
+        "variant": variant["run"],
+        "request_count": [base.get("request_count"), variant.get("request_count")],
+        "success_count": [base.get("success_count"), variant.get("success_count")],
+        "error_count": [base.get("error_count"), variant.get("error_count")],
+        "ttft_p50_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p50"), stat_value(variant, "ttft_stats_s", "p50")),
+        "ttft_p90_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p90"), stat_value(variant, "ttft_stats_s", "p90")),
+        "e2e_p50_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p50"), stat_value(variant, "latency_stats_s", "p50")),
+        "e2e_p90_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p90"), stat_value(variant, "latency_stats_s", "p90")),
+        "tpot_p90_delta_pct": pct_delta(stat_value(base, "tpot_stats_s", "p90"), stat_value(variant, "tpot_stats_s", "p90")),
+        "wall_clock_delta_pct": pct_delta(base.get("wall_clock_s"), variant.get("wall_clock_s")),
+        "gpu_mean_util": [
+            nested(base, ["gpu_summary", "mean_util_pct"]),
+            nested(variant, ["gpu_summary", "mean_util_pct"]),
+        ],
+        "gpu_imbalance_ratio": [
+            nested(base, ["gpu_summary", "max_min_ratio"]),
+            nested(variant, ["gpu_summary", "max_min_ratio"]),
+        ],
+    }
+
+
+def build_claim_matrix(summaries: list[JsonDict], comparisons: list[JsonDict]) -> list[JsonDict]:
+    has_gpu = any((s.get("gpu_summary") or {}).get("status") == "available" for s in summaries)
+    has_metrics = any(s.get("metrics_summary_available") for s in summaries)
+    return [
+        {
+            "claim": "Batch 0 substrate audit is only partially complete for existing runs.",
+            "status": "partially_supported",
+            "supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.",
+            "needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.",
+            "reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.",
+        },
+        {
+            "claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.",
+            "status": "supported_for_trace_shape",
+            "supporting_data": "analysis/characterization/analyze.py outputs workload_summary/session_skew/KV footprint when given trace and kv_bytes_per_token.",
+            "needed_next": "Run on dash0 compact formatted trace for canonical full-trace numbers.",
+            "reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.",
+        },
+        {
+            "claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.",
+            "status": "supported_by_existing_artifact" if has_metrics else "unavailable",
+            "supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.",
+            "needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.",
+            "reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.",
+        },
+        {
+            "claim": "Elastic transfer-based migration does not improve high-contention 500-request run.",
+            "status": "supported_by_existing_artifact" if has_metrics else "unavailable",
+            "supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.",
+            "needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.",
+            "reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.",
+        },
+        {
+            "claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.",
+            "status": "not_yet_supported",
+            "supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs.",
+            "needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.",
+            "reviewer_risk": "Cannot claim interference as causal without Batch 2.",
+        },
+        {
+            "claim": "Session hot-spot residual imbalance is suggested but not fully attributed.",
+            "status": "partially_supported" if has_gpu else "unavailable",
+            "supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs.",
+            "needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.",
+            "reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.",
+        },
+        {
+            "claim": "SRR is not measured by existing fixed-request runs.",
+            "status": "not_yet_supported",
+            "supporting_data": "No arrival-rate sweep artifacts found.",
+            "needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.",
+            "reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.",
+        },
+    ]
+
+
+def build_risk_register(summaries: list[JsonDict]) -> list[JsonDict]:
+    return [
+        {
+            "risk": "Session sequentiality not proven",
+            "severity": "high",
+            "evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.",
+            "mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.",
+        },
+        {
+            "risk": "Legacy PD-sep data may not match final methodology",
+            "severity": "medium",
+            "evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.",
+            "mitigation": "Use fresh PD matrix for paper-grade claims.",
+        },
+        {
+            "risk": "GPU util is not a sufficient hot-spot proof",
+            "severity": "medium",
+            "evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.",
+            "mitigation": "Add route-decision and per-worker queue logs for Batch 3.",
+        },
+        {
+            "risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data",
+            "severity": "medium",
+            "evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.",
+            "mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.",
+        },
+    ]
+
+
+def render_current_results(
+    summaries: list[JsonDict],
+    comparisons: list[JsonDict],
+    claim_matrix: list[JsonDict],
+    risk_register: list[JsonDict],
+) -> str:
+    lines = [
+        "# Current Characterization Results",
+        "",
+        f"Generated: {dt.datetime.now(dt.timezone.utc).isoformat()}",
+        f"Git commit: `{git_commit()}`",
+        "",
+        "## Existing Run Summaries",
+        "",
+        "| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |",
+        "|---|---:|---:|---:|---:|---:|---:|",
+    ]
+    for s in summaries:
+        lines.append(
+            "| {run} | {ok}/{req} | {ttft50}/{ttft90} | {e2e50}/{e2e90} | {tpot90} | {gpu_mean} | {gpu_imb} |".format(
+                run=s["run"],
+                ok=fmt(s.get("success_count")),
+                req=fmt(s.get("request_count")),
+                ttft50=fmt(stat_value(s, "ttft_stats_s", "p50")),
+                ttft90=fmt(stat_value(s, "ttft_stats_s", "p90")),
+                e2e50=fmt(stat_value(s, "latency_stats_s", "p50")),
+                e2e90=fmt(stat_value(s, "latency_stats_s", "p90")),
+                tpot90=fmt(stat_value(s, "tpot_stats_s", "p90")),
+                gpu_mean=fmt(nested(s, ["gpu_summary", "mean_util_pct"])),
+                gpu_imb=fmt(nested(s, ["gpu_summary", "max_min_ratio"])),
+            )
+        )
+    lines.extend([
+        "",
+        "## Pairwise Comparisons",
+        "",
+        "| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |",
+        "|---|---:|---:|---:|---:|---:|---:|",
+    ])
+    for c in comparisons:
+        lines.append(
+            "| {name} | {ttft50} | {ttft90} | {e2e50} | {e2e90} | {tpot90} | {wall} |".format(
+                name=c["name"],
+                ttft50=fmt_pct(c.get("ttft_p50_delta_pct")),
+                ttft90=fmt_pct(c.get("ttft_p90_delta_pct")),
+                e2e50=fmt_pct(c.get("e2e_p50_delta_pct")),
+                e2e90=fmt_pct(c.get("e2e_p90_delta_pct")),
+                tpot90=fmt_pct(c.get("tpot_p90_delta_pct")),
+                wall=fmt_pct(c.get("wall_clock_delta_pct")),
+            )
+        )
+    lines.extend([
+        "",
+        "## What We Can Say Now",
+        "",
+    ])
+    for item in claim_matrix:
+        lines.append(f"- **{item['status']}**: {item['claim']}")
+        lines.append(f"  Supporting data: {item['supporting_data']}")
+        lines.append(f"  Next: {item['needed_next']}")
+    lines.extend([
+        "",
+        "## Main Reviewer Risks",
+        "",
+    ])
+    for item in risk_register:
+        lines.append(f"- **{item['severity']}**: {item['risk']} - {item['mitigation']}")
+    lines.append("")
+    return "\n".join(lines)
+
+
+def render_claim_matrix(items: list[JsonDict]) -> str:
+    lines = [
+        "# Characterization Claim Matrix",
+        "",
+        "| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |",
+        "|---|---|---|---|---|",
+    ]
+    for item in items:
+        lines.append(
+            f"| {item['claim']} | `{item['status']}` | {item['supporting_data']} | {item['needed_next']} | {item['reviewer_risk']} |"
+        )
+    lines.append("")
+    return "\n".join(lines)
+
+
+def render_risk_register(items: list[JsonDict]) -> str:
+    lines = [
+        "# Reviewer Risk Register",
+        "",
+        "| Risk | Severity | Evidence | Mitigation |",
+        "|---|---|---|---|",
+    ]
+    for item in items:
+        lines.append(
+            f"| {item['risk']} | `{item['severity']}` | {item['evidence']} | {item['mitigation']} |"
+        )
+    lines.append("")
+    return "\n".join(lines)
+
+
+def render_figures_index(summaries: list[JsonDict]) -> str:
+    return "\n".join([
+        "# Figures Index",
+        "",
+        "No generated figures are committed by this script. Batch-specific figures should be generated from:",
+        "",
+        "- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.",
+        "- future Batch 2 step-timeline artifacts for interference plots.",
+        "- future Batch 3 per-worker/session artifacts for hot-spot plots.",
+        "- future Batch 4 arrival-rate sweep artifacts for SRR curves.",
+        "",
+        "This file exists so the audit package has a stable placeholder until fresh figures are generated.",
+        "",
+    ])
+
+
+def render_reproduction_commands(args: argparse.Namespace, run_dirs: list[Path]) -> str:
+    runs = " ".join(str(p) for p in run_dirs)
+    return "\n".join([
+        "#!/usr/bin/env bash",
+        "set -euo pipefail",
+        "",
+        "# Rebuild this current-results audit package.",
+        f"python3 analysis/characterization/summarize_runs.py --output-dir {args.output_dir} --runs {runs}",
+        "",
+        "# Example Batch 0/1 local trace analysis.",
+        "python3 analysis/characterization/analyze.py \\",
+        "  --trace traces/w600_r0.0015_st30.jsonl \\",
+        "  --kv-bytes-per-token 98304 \\",
+        "  --task-name w600_local_full_trace \\",
+        "  --overwrite",
+        "",
+    ])
+
+
+def load_json(path: Path) -> JsonDict:
+    if not path.exists():
+        return {}
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except Exception:
+        return {}
+    return data if isinstance(data, dict) else {}
+
+
+def load_jsonl(path: Path) -> list[JsonDict]:
+    if not path.exists():
+        return []
+    rows = []
+    with path.open(encoding="utf-8") as handle:
+        for line in handle:
+            if not line.strip():
+                continue
+            try:
+                row = json.loads(line)
+            except Exception:
+                continue
+            if isinstance(row, dict):
+                rows.append(row)
+    return rows
+
+
+def write_json(path: Path, data: Any) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+
+
+def first_present(row: JsonDict, keys: list[str]) -> Any:
+    for key in keys:
+        if key in row:
+            return row[key]
+    return None
+
+
+def stat_value(run: JsonDict, stat_key: str, value_key: str) -> float | None:
+    stats_obj = run.get(stat_key)
+    if not isinstance(stats_obj, dict):
+        return None
+    return to_float(stats_obj.get(value_key))
+
+
+def nested(row: JsonDict, keys: list[str]) -> Any:
+    cur: Any = row
+    for key in keys:
+        if not isinstance(cur, dict):
+            return None
+        cur = cur.get(key)
+    return cur
+
+
+def pct_delta(base: Any, variant: Any) -> float | None:
+    b = to_float(base)
+    v = to_float(variant)
+    if b is None or v is None or b == 0:
+        return None
+    return (v - b) / b * 100.0
+
+
+def to_float(value: Any) -> float | None:
+    if value is None:
+        return None
+    try:
+        out = float(value)
+    except (TypeError, ValueError):
+        return None
+    return out if math.isfinite(out) else None
+
+
+def stats(values: list[float]) -> JsonDict | None:
+    clean = sorted(float(v) for v in values if math.isfinite(float(v)))
+    if not clean:
+        return None
+    return {
+        "count": len(clean),
+        "mean": statistics.fmean(clean),
+        "p50": percentile(clean, 0.50),
+        "p90": percentile(clean, 0.90),
+        "p95": percentile(clean, 0.95),
+        "p99": percentile(clean, 0.99),
+        "max": clean[-1],
+    }
+
+
+def percentile(values: list[float], q: float) -> float:
+    if len(values) == 1:
+        return values[0]
+    rank = q * (len(values) - 1)
+    lo = int(rank)
+    hi = min(lo + 1, len(values) - 1)
+    frac = rank - lo
+    return values[lo] * (1 - frac) + values[hi] * frac
+
+
+def top_contribution(values: list[float]) -> JsonDict:
+    clean = sorted([v for v in values if math.isfinite(v)], reverse=True)
+    total = sum(clean)
+    if not clean or total <= 0:
+        return {"top_1pct": None, "top_5pct": None, "top_10pct": None}
+
+    def frac(pct: float) -> float:
+        k = max(1, math.ceil(len(clean) * pct))
+        return sum(clean[:k]) / total
+
+    return {
+        "top_1pct": frac(0.01),
+        "top_5pct": frac(0.05),
+        "top_10pct": frac(0.10),
+    }
+
+
+def fmt(value: Any) -> str:
+    num = to_float(value)
+    if num is None:
+        return "n/a"
+    if abs(num - round(num)) < 1e-9 and abs(num) < 1_000_000:
+        return str(int(round(num)))
+    return f"{num:.3g}"
+
+
+def fmt_pct(value: Any) -> str:
+    num = to_float(value)
+    if num is None:
+        return "n/a"
+    return f"{num:+.1f}%"
+
+
+def git_commit() -> str:
+    try:
+        result = subprocess.run(
+            ["git", "rev-parse", "HEAD"],
+            check=True,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.DEVNULL,
+            text=True,
+        )
+    except Exception:
+        return ""
+    return result.stdout.strip()
+
+
+if __name__ == "__main__":
+    main()

analysis/characterization_todo_for_interns.md

@@ -0,0 +1,641 @@
+# Agentic Workload Characterization TODO
+
+Status: execution checklist for interns
+Date: 2026-05-25
+
+## 0. Purpose
+
+We are not starting from the assumption that Unified routing or PUSH
+migration is already the answer.
+
+The first goal is to build a rigorous characterization package that proves:
+
+1. which dimensions make agentic serving different;
+2. where static PD-disaggregation works poorly;
+3. where PD-colocation/cache-aware routing still has residual failure modes;
+4. how these failure modes reduce sustainable request rate under SLO.
+
+Only after these facts are established should we refine the positive system
+design.
+
+Primary system goal:
+
+```text
+maximize sustainable request rate under SLO
+```
+
+Prefill-decode interference and session hot-spot imbalance are mechanisms
+that may reduce SRR. They are not the final metric by themselves.
+
+## 1. Global Delivery Rules
+
+Every task must produce data, figures, and an audit trail. A task is not
+complete if it only produces a written conclusion.
+
+Use this output layout:
+
+```text
+outputs/characterization/<date>/<task_name>/
+├── manifest.json
+├── raw/
+├── summary.json
+├── summary.md
+├── figures/
+└── audit.md
+```
+
+Required fields in `manifest.json`:
+
+```json
+{
+  "git_commit": "",
+  "host": "",
+  "gpu_type": "",
+  "gpu_count": 0,
+  "trace_path": "",
+  "trace_sha256": "",
+  "policy": "",
+  "launch_command": "",
+  "request_limit": null,
+  "time_scale": null,
+  "session_sampling_method": "",
+  "session_sequential": true,
+  "start_time": "",
+  "end_time": ""
+}
+```
+
+Every comparison must report:
+
+- attempted requests
+- completed requests
+- errors / timeouts
+- goodput
+- TTFT p50/p90/p99
+- E2E p50/p90/p99
+- TPOT p50/p90/p99
+- per-worker queue metrics
+- per-worker GPU utilization
+- per-worker KV occupancy if available
+- per-worker APC / cache-hit metrics
+
+Every figure must be reproducible from raw data by a script committed or
+saved alongside the artifact.
+
+## 2. Batch 0: Benchmark Substrate Audit
+
+### Goal
+
+Prove the load generator and trace replay are valid before trusting any
+performance result.
+
+The most important invariant:
+
+```text
+For online agentic serving, each session must have at most one in-flight turn.
+Turn N+1 must not be sent before turn N completes.
+```
+
+### TODO
+
+1. Implement or run an analyzer that reconstructs per-session request
+   intervals:
+   - dispatch timestamp
+   - first-token timestamp
+   - finish timestamp
+   - error / timeout timestamp
+2. Compute max concurrent in-flight turns per session.
+3. Compute session start-time distribution.
+4. Compute turn inter-arrival distribution.
+5. Classify each existing run as one of:
+   - `online_realistic`
+   - `burst_stress`
+   - `synthetic_microbench`
+   - `invalid_for_online_claim`
+6. For any run where session sequentiality is violated, write down exactly
+   which claim it can still support.
+
+### Data Artifacts
+
+- `session_concurrency.json`
+- `session_arrival_stats.json`
+- `turn_interval_stats.json`
+- `trace_profile.json`
+- `invalid_runs.md`
+
+### Figures
+
+- session start-time CDF
+- per-session max in-flight histogram
+- turns per session CDF
+- turn inter-arrival CDF
+
+### Audit Checks
+
+The `audit.md` must answer:
+
+1. Does the main trace satisfy `max_inflight_per_session == 1`?
+2. If not, is the run explicitly labeled as stress or invalid?
+3. Are attempted/completed/error counts included?
+4. Are latency percentiles computed only over successes, and if so, is
+   goodput also reported?
+
+### Pass Criteria
+
+- Main online-serving experiments must have `max_inflight_per_session == 1`.
+- Any violation must be clearly labeled and excluded from SRR claims.
+
+## 3. Batch 1: Workload Characterization
+
+### Goal
+
+Establish agentic workload facts independent of any proposed system.
+
+Required facts:
+
+1. long input, short output;
+2. large per-request KV footprint;
+3. reuse is mostly intra-session;
+4. session token mass is heavy-tailed;
+5. total prompt length and effective uncached prefill work are different.
+
+### TODO
+
+1. Compute input token CDF.
+2. Compute output token CDF.
+3. Compute input/output ratio.
+4. Estimate KV footprint per request:
+
+   ```text
+   kv_bytes_per_request = input_tokens * kv_bytes_per_token
+   ```
+
+5. Decompose reusable KV into:
+   - intra-session reuse
+   - cross-session reuse
+   - shared/system-prefix reuse
+6. Compute session-level skew:
+   - turns per session
+   - cumulative input tokens per session
+   - cumulative output tokens per session
+   - cumulative uncached tokens per session
+   - top-k session contribution
+7. Compute append / effective-prefill distribution:
+
+   ```text
+   uncached_tokens = input_tokens - cached_tokens
+   ```
+
+8. Compare total input length vs uncached tokens.
+
+### Data Artifacts
+
+- `workload_summary.json`
+- `kv_footprint_summary.json`
+- `reuse_decomposition.json`
+- `session_skew.json`
+- `append_delta_stats.json`
+
+### Figures
+
+- input/output token CDF
+- input/output ratio CDF
+- KV footprint CDF
+- reuse decomposition stacked bar
+- turns per session CDF
+- per-session token mass Lorenz curve
+- top-k sessions token contribution bar
+- total input vs uncached tokens scatter
+
+### Audit Checks
+
+The `audit.md` must answer:
+
+1. What are input p50/p90/p99?
+2. What are output p50/p90/p99?
+3. What is the estimated KV footprint p50/p90/p99?
+4. What fraction of reuse is intra-session?
+5. What fraction of total token mass comes from top 1% / 5% sessions?
+6. Are long prompts often small appends after cache reuse?
+
+### Pass Criteria
+
+The batch passes only if these facts can be stated numerically with raw data
+links and plotted figures.
+
+## 4. Batch 2: PD-Colo Prefill-Decode Interference Proof
+
+### Goal
+
+Prove that PD-colocation can suffer from prefill-decode interference under
+high load, and quantify how much this affects TPOT, decode queueing, and SLO.
+
+Hypothesis:
+
+```text
+When heavy uncached prefill overlaps with active decode on the same worker,
+decode TPOT and/or decode queue delay increases.
+```
+
+### TODO
+
+1. Run controlled microbenchmarks:
+   - decode-only steady load;
+   - decode load plus same-worker heavy prefill injection;
+   - decode load plus different-worker heavy prefill injection.
+2. Sweep uncached prefill sizes:
+   - 2k
+   - 8k
+   - 16k
+   - 32k
+   - 64k
+3. If supported, sweep chunked prefill size.
+4. Log timestamps for:
+   - decode steps;
+   - prefill start/end;
+   - prefill chunks;
+   - queue admission;
+   - request completion.
+5. In trace replay, label decode steps by whether they overlap with
+   same-worker prefill.
+6. Compute:
+
+   ```text
+   interference_index =
+     TPOT_p90(decode steps overlapping same-worker prefill)
+     / TPOT_p90(decode steps without same-worker prefill)
+   ```
+
+7. Compare same-worker vs different-worker controls.
+
+### Data Artifacts
+
+- `interference_microbench_summary.json`
+- `decode_step_timeseries.csv`
+- `prefill_overlap_events.jsonl`
+- `interference_index.json`
+- `trace_overlap_summary.json`
+
+### Figures
+
+- TPOT time series with prefill overlap annotation
+- interference index vs uncached prefill size
+- same-worker vs different-worker TPOT boxplot
+- chunk size vs TTFT/TPOT tradeoff
+- trace replay overlap vs non-overlap TPOT comparison
+
+### Audit Checks
+
+The `audit.md` must answer:
+
+1. Is the interference observed on the same worker?
+2. Is the different-worker control significantly weaker?
+3. Does interference grow with uncached prefill size?
+4. Does the phenomenon appear in real trace replay, not only microbench?
+5. Could the result be explained by global load instead of local colocation?
+
+### Pass Criteria
+
+- Same-worker overlap must measurably increase TPOT or decode queue delay.
+- The effect must be weaker or absent in the different-worker control.
+- The effect must be visible in at least one trace replay setting.
+
+## 5. Batch 3: Session Hot-Spot Residual Imbalance Proof
+
+### Goal
+
+Prove that cache-aware/LMetric is a strong baseline but still leaves residual
+hot-worker imbalance due to session skew and locality.
+
+Hypothesis:
+
+```text
+Cache-aware routing preserves locality by attracting future turns to cached
+workers. This is usually good, but heavy-tailed sessions can create hot
+workers whose queue delay/SLO violations are much worse than the median
+worker even when other workers still have headroom.
+```
+
+### TODO
+
+1. Run the same session-causal trace with:
+   - corrected LMetric/cache-aware;
+   - load-only routing;
+   - hard sticky routing;
+   - current Unified hybrid, if available.
+2. For each worker, record:
+   - assigned session count;
+   - cumulative input tokens;
+   - cumulative uncached tokens;
+   - cumulative output tokens;
+   - request queue delay;
+   - decode queue delay;
+   - GPU utilization;
+   - KV occupancy;
+   - APC / cache-hit rate;
+   - SLO violations.
+3. For each session, record:
+   - worker set used;
+   - primary worker;
+   - cumulative token mass;
+   - number of turns;
+   - latency contribution;
+   - whether it appears in slow-request set.
+4. Create a session-mass capped or equalized replay:
+   - cap max session turns or token mass;
+   - rerun LMetric/cache-aware;
+   - compare hot-spot index.
+5. Compute:
+
+   ```text
+   hotspot_index =
+     max_worker_queue_delay_p90 / median_worker_queue_delay_p90
+   ```
+
+6. Compute locality/load tradeoff:
+
+   ```text
+   locality_gain = APC(policy) - APC(load_only)
+   imbalance_cost =
+     max_worker_latency_p90(policy) - median_worker_latency_p90(policy)
+   ```
+
+### Data Artifacts
+
+- `worker_balance_summary.json`
+- `session_to_worker_map.json`
+- `session_mass_summary.json`
+- `routing_policy_comparison.json`
+- `hotspot_index.json`
+- `capped_session_replay_summary.json`
+
+### Figures
+
+- per-worker queue delay bar
+- per-worker token mass bar
+- GPU utilization timeline by worker
+- KV occupancy timeline by worker
+- APC vs queue delay scatter
+- top sessions contribution bar
+- policy tradeoff plot: APC vs hotspot_index
+- original vs session-capped hot-spot comparison
+
+### Audit Checks
+
+The `audit.md` must answer:
+
+1. Does LMetric/cache-aware still show worker-level skew?
+2. Are SLO violations concentrated on hot workers or hot sessions?
+3. Does load-only routing improve balance but reduce APC/locality?
+4. Does hard sticky improve locality but worsen hot-spot/HOL?
+5. Does session-mass capping reduce hot spots?
+
+### Pass Criteria
+
+- LMetric/cache-aware must be shown as strong but imperfect.
+- There must be measurable residual hot-worker imbalance.
+- The imbalance must correlate with session token mass or locality.
+
+## 6. Batch 4: Sustainable Request Rate Sweep
+
+### Goal
+
+Connect interference and hot-spot mechanisms to the final metric:
+
+```text
+SRR(SLO) = max arrival rate satisfying SLO in steady state
+```
+
+### TODO
+
+1. Define provisional SLO thresholds. Use configurable values, for example:
+
+   ```text
+   TTFT_p90 <= T_ttft
+   E2E_p90  <= T_e2e
+   TPOT_p90 <= T_tpot
+   error_rate <= epsilon
+   queue length stable
+   KV occupancy stable
+   ```
+
+2. Implement arrival-rate sweep:
+   - Poisson session arrivals;
+   - session-internal sequentiality;
+   - warmup window;
+   - steady-state measurement window.
+3. For each arrival rate `lambda`, run:
+   - PD-colo cache-aware/LMetric;
+   - static PD-disagg;
+   - current Unified hybrid;
+   - optional hard sticky;
+   - optional load-only.
+4. Find maximum sustainable lambda for each policy.
+5. Report instability reasons:
+   - SLO violation;
+   - queue growth;
+   - KV occupancy growth;
+   - error/timeout growth.
+
+### Data Artifacts
+
+- `srr_curve.json`
+- `lambda_runs/<lambda>/summary.json`
+- `slo_violation_reason.json`
+- `goodput_vs_arrival_rate.json`
+- `stability_summary.json`
+
+### Figures
+
+- SRR bar chart
+- TTFT p90 vs arrival rate
+- E2E p90 vs arrival rate
+- TPOT p90 vs arrival rate
+- goodput vs arrival rate
+- error rate vs arrival rate
+- queue length over time near failure point
+- KV occupancy over time near failure point
+
+### Audit Checks
+
+The `audit.md` must answer:
+
+1. Are session arrivals open-loop and Poisson?
+2. Is session-internal sequentiality enforced?
+3. How long are warmup and steady-state windows?
+4. Is SRR failure persistent rather than transient?
+5. Are completed/requested counts reported at every lambda?
+6. Are policies compared on the same trace and same arrival process?
+
+### Pass Criteria
+
+- Each policy must have a measured SRR under the same SLO.
+- Failure must be attributed to persistent SLO violation, queue growth, KV
+  growth, or error growth.
+- Data must be session-causal.
+
+## 7. Batch 5: Failure Attribution Near SRR Boundary
+
+### Goal
+
+At and around the PD-colo/LMetric failure point, determine whether SLO
+violations are caused by prefill-decode interference, session hot spots, KV
+pressure, cache misses, or other mechanisms.
+
+### TODO
+
+1. Select three arrival rates:
+
+   ```text
+   lambda = 0.9 * SRR
+   lambda = 1.0 * SRR
+   lambda = 1.1 * SRR
+   ```
+
+2. For every slow or SLO-violating request, assign labels:
+   - same-worker prefill overlap;
+   - hot worker queue;
+   - high KV occupancy;
+   - cache miss / large uncached append;
+   - transfer wait;
+   - P queue wait;
+   - D admission wait;
+   - unknown.
+3. Produce per-request waterfall for representative slow requests.
+4. Produce per-worker timeline around failure windows.
+5. Summarize cause distribution.
+
+### Data Artifacts
+
+- `slow_request_attribution.jsonl`
+- `failure_breakdown.json`
+- `case_studies.md`
+- `worker_failure_windows.json`
+
+### Figures
+
+- SLO violation cause stacked bar
+- slow request waterfall
+- worker timeline near failure
+- prefill/decode/KV/queue stacked breakdown
+- failure cause vs arrival rate
+
+### Audit Checks
+
+The `audit.md` must answer:
+
+1. What fraction of slow requests overlap same-worker prefill?
+2. What fraction are on hot workers?
+3. What fraction happen under high KV occupancy?
+4. What fraction are large uncached append requests?
+5. For PD-disagg/Unified migration, how much time is transfer/P queue/D wait?
+6. What remains unexplained?
+
+### Pass Criteria
+
+The batch must answer:
+
+1. Why PD-colo/LMetric hits its SRR limit.
+2. Why static PD-disagg hits its SRR limit.
+3. If Unified/PUSH underperforms, whether the cause is trigger quality, cost
+   model, transfer overhead, wrong load regime, or something else.
+
+## 8. Batch 6: Audit Package
+
+### Goal
+
+Make the whole characterization package reviewable by a strict systems
+reviewer.
+
+### TODO
+
+1. Write a claim matrix:
+
+   ```text
+   claim -> data artifact -> figure -> script -> caveat -> reviewer risk
+   ```
+
+2. Write a figure index:
+   - figure filename;
+   - source data;
+   - generation command;
+   - intended claim.
+3. Write a reviewer risk register:
+   - loadgen validity risks;
+   - trace representativeness risks;
+   - metric bias risks;
+   - implementation-specific risks;
+   - generalization risks.
+4. Write a reproduction script or command list.
+5. Mark experiments that cannot support main claims.
+
+### Final Artifacts
+
+- `characterization_claim_matrix.md`
+- `all_figures_index.md`
+- `reviewer_risk_register.md`
+- `reproduction_commands.sh`
+- `main_claim_allowed_runs.md`
+
+### Audit Checks
+
+The final package must satisfy:
+
+1. Every claim links to raw data.
+2. Every figure can be regenerated.
+3. Every experiment has a manifest.
+4. Every caveat is explicit.
+5. Invalid or stress-only runs are not used for online-serving claims.
+
+## 9. Priority Order
+
+### Priority 1
+
+Do these first:
+
+1. Batch 0: Benchmark Substrate Audit
+2. Batch 1: Workload Characterization
+3. Batch 3: Session Hot-Spot Residual Imbalance Proof
+
+Reason:
+
+These define whether the trace and routing problem are real. Without them,
+SRR sweeps and system experiments are not trustworthy.
+
+### Priority 2
+
+Do these after the substrate and workload facts are stable:
+
+1. Batch 2: PD-Colo Prefill-Decode Interference Proof
+2. Batch 5: Failure Attribution Near SRR Boundary
+
+Reason:
+
+These explain the mechanisms behind SLO/SRR failure and determine what the
+positive system should actually fix.
+
+### Priority 3
+
+Do these after instrumentation and attribution are ready:
+
+1. Batch 4: Sustainable Request Rate Sweep
+2. Batch 6: Audit Package
+
+Reason:
+
+SRR sweeps are expensive. They should run only after trace validity,
+logging, and attribution labels are ready.
+
+## 10. Non-Negotiable Reviewer Rules
+
+1. Do not use session-nonsequential loadgen for online-serving claims.
+2. Do not compare latency percentiles without attempted/completed/error counts.
+3. Do not use APC alone as a success metric.
+4. Do not use average GPU utilization as proof of load balance.
+5. Do not compare policies on different traces unless explicitly labeled.
+6. Do not hide failed requests or timeouts.
+7. Do not claim Unified/PUSH is the answer before failure attribution proves
+   the relevant bottleneck and cost budget.
+8. Treat corrected LMetric/cache-aware PD-colo as the main baseline.
+9. Treat static PD-disagg as an important baseline, not a strawman.
+10. Every result must be reproducible from raw artifacts and commands.