# Characterization Protocols For Remaining Batches Status: implementation protocol and audit checklist Date: 2026-05-25 This file completes the `analysis/characterization` scaffold for the TODO list. It separates what is already implemented from what requires fresh GPU runs or new engine/proxy instrumentation. ## Implemented Now ### Batch 0/1 Analyzer Use: ```bash python3 analysis/characterization/analyze.py \ --trace traces/w600_r0.0015_st30.jsonl \ --kv-bytes-per-token 98304 \ --task-name w600_local_full_trace \ --overwrite ``` The analyzer writes: - `manifest.json` - `summary.json` - `summary.md` - `audit.md` - `session_concurrency.json` - `session_arrival_stats.json` - `turn_interval_stats.json` - `trace_profile.json` - `workload_summary.json` - `kv_footprint_summary.json` - `reuse_decomposition.json` - `session_skew.json` - `append_delta_stats.json` Limitations: - Actual online sequentiality requires dispatch and finish/error timestamps. Existing `metrics.jsonl` artifacts generally do not contain these fields. - Actual reuse decomposition requires `cached_tokens`/`cache_hit`, `hash_ids`, and `session_id` in the same joinable request record. ### Existing-Run Audit Use: ```bash python3 analysis/characterization/summarize_runs.py ``` The script writes an audit package under: ```text analysis/characterization/current_results/ ``` It summarizes already completed runs and explicitly marks which claims are supported, partially supported, or not yet supported. ## Batch 2 Protocol: PD-Colo Prefill/Decode Interference Purpose: Prove whether same-worker prefill overlap increases decode TPOT/queue delay. Required new instrumentation: - per-request dispatch timestamp - per-request finish/error timestamp - per decode step timestamp - decode step worker id - prefill chunk start/end timestamp - prefill worker id - request/session id associated with each prefill chunk Required arms: 1. decode-only steady load 2. decode + same-worker heavy prefill injection 3. decode + different-worker heavy prefill injection 4. trace replay with overlap labels Required sweep: ```text uncached_prefill_tokens in {2k, 8k, 16k, 32k, 64k} chunked_prefill_size in available engine values ``` Required outputs: - `interference_microbench_summary.json` - `decode_step_timeseries.csv` - `prefill_overlap_events.jsonl` - `interference_index.json` - TPOT timeline figure with prefill overlays - same-worker vs different-worker TPOT boxplot Pass condition: ```text TPOT_p90(overlap_same_worker) / TPOT_p90(no_overlap) > 1 ``` and the effect must be materially weaker in the different-worker control. ## Batch 3 Protocol: Session Hot-Spot Residual Imbalance Purpose: Prove whether cache-aware/LMetric still leaves hot workers under session-heavy skew. Required new instrumentation: - route decision per request - chosen worker - candidate worker scores - cache hit / estimated uncached tokens per candidate - per-worker request queue length/delay - per-worker decode queue length/delay - per-worker KV occupancy - per-worker APC/cache-hit snapshot Required arms: 1. corrected LMetric/cache-aware 2. load-only routing 3. hard sticky routing 4. current Unified hybrid 5. session-mass capped/equalized replay Required outputs: - `worker_balance_summary.json` - `session_to_worker_map.json` - `session_mass_summary.json` - `routing_policy_comparison.json` - `hotspot_index.json` - per-worker queue delay bar - APC vs queue delay scatter - top-session contribution bar - policy tradeoff plot: APC vs hot-spot index Pass condition: LMetric/cache-aware must show measurable residual worker skew, and that skew must correlate with session token mass or locality. GPU utilization alone is not enough for this claim. ## Batch 4 Protocol: Sustainable Request Rate Purpose: Measure: ```text SRR(SLO) = max arrival rate satisfying SLO in steady state ``` Required load generator behavior: - open-loop session arrivals, preferably Poisson - session-internal sequentiality - warmup window - steady-state measurement window - explicit attempted/completed/error counters Provisional SLO: ```text TTFT_p90 <= T_ttft E2E_p90 <= T_e2e TPOT_p90 <= T_tpot error_rate <= epsilon queue length stable KV occupancy stable ``` Required arms: 1. PD-colo corrected LMetric/cache-aware 2. static PD-disagg 3. current Unified hybrid 4. optional hard sticky 5. optional load-only Required outputs: - `srr_curve.json` - `lambda_runs//summary.json` - `slo_violation_reason.json` - `goodput_vs_arrival_rate.json` - SRR bar chart - latency vs arrival rate curves - goodput vs arrival rate - queue/KV stability plot near failure point Pass condition: Each policy has a measured max sustainable lambda under the same SLO and same session-causal arrival process. ## Batch 5 Protocol: Failure Attribution Near SRR Boundary Purpose: Explain why each policy fails near SRR. Required rates: ```text lambda = 0.9 * SRR lambda = 1.0 * SRR lambda = 1.1 * SRR ``` Labels for each slow/SLO-violating request: - same-worker prefill overlap - hot worker queue - high KV occupancy - cache miss / large uncached append - transfer wait - P queue wait - D admission wait - unknown Required outputs: - `slow_request_attribution.jsonl` - `failure_breakdown.json` - `case_studies.md` - `worker_failure_windows.json` - violation cause stacked bar - slow request waterfall - worker timeline near failure Pass condition: The analysis must explain whether PD-colo is limited by interference, hot-spot, KV pressure, or a mixture, and whether Unified/PUSH underperforms because of trigger quality, transfer cost, target admission, or load regime. ## Batch 6 Protocol: Audit Package Implemented by `summarize_runs.py` for existing runs and extended by fresh Batch 2-5 outputs later. Required files: - `characterization_claim_matrix.md` - `all_figures_index.md` - `reviewer_risk_register.md` - `reproduction_commands.sh` - `main_claim_allowed_runs.md` Current package intentionally marks Batch 2/4/5 claims as not yet supported until fresh instrumented experiments exist.