PD-sep matrix infrastructure: bench.sh pdsep mode + matrix driver

Adds the experiment harness that gates the empirical claims (C2/C3/C4/C5)
in the PD-sep paper section. Three pieces:

  1. scripts/bench.sh: new --mode pdsep with --pd-ratio P:D, and an
     --eager flag to re-enable --enforce-eager for the cuda-graph
     ablation. pdsep reuses the elastic-mode Mooncake kv_both launch and
     swaps the proxy command from --combined to --prefill/--decode.
     baseline and elastic flows are unchanged.

  2. analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh: matrix
     driver that runs {combined-ca, pdsep-4p4d, pdsep-6p2d} x cudagraph
     x 3 seeds by default (~2 h on dash0). --with-rr adds combined-rr;
     --with-eager doubles to ~5 h with the cuda-graph ablation. Skips
     completed runs, captures per-instance vLLM logs (needed for C3
     step-level KV-utilization mining).

  3. fig_kv_memory_wall.pdf: empirical anchor (star) at REPORT.md §3.3's
     observed 6P+2D 97% KV utilization. The marker lands on the model's
     predicted curve at p90 input, confirming the steady-state analysis.

README updated with the run command, output layout, and the followup
plotters that consume outputs/pd_matrix/.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

analysis/pd_sep_paper_section/README.md

@@ -14,7 +14,8 @@ analysis/pd_sep_paper_section/
 │   ├── plot_workload.py            # C1: input/output CDF + KV reuse decomposition
 │   ├── plot_roofline.py            # C6: prefill roofline at varying cache reuse
 │   ├── plot_routing_lever.py       # C7: routing vs PD-sep as design levers
-│   └── plot_kv_memory_wall.py      # KV mem-wall: the system-level explanation
+│   ├── plot_kv_memory_wall.py      # KV mem-wall model + empirical anchor
+│   └── bench_pd_matrix.sh          # orchestrates the C2/C3/C4/C5 experiment matrix on dash0
 └── figures/
     ├── fig_c1a_io_cdf.pdf          # input/output token CDF (from traces/w600_r0.0015_st30.jsonl)
     ├── fig_c1b_reuse.pdf           # KV reuse decomposition: 79% intra-session
@@ -92,17 +93,51 @@ From repo root:
 
 All four default `--outdir` to `analysis/pd_sep_paper_section/figures`.
 
+## Running the experiment matrix (gating C2/C3/C4/C5)
+
+`bench_pd_matrix.sh` orchestrates the experiments that the unrendered
+claims depend on. It uses the extended `scripts/bench.sh` (now supports
+`--mode pdsep --pd-ratio 4:4|6:2` and `--eager` for the cuda-graph
+ablation; all launchers no longer pin `--enforce-eager`).
+
+On dash0:
+
+```bash
+cd ~/agentic-kv
+# minimal set: 3 configs (combined-ca, pdsep-4p4d, pdsep-6p2d) x 3 seeds
+# = 9 runs ~= 2 h
+bash analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh
+
+# full matrix (adds combined-rr and the eager ablation)
+bash analysis/pd_sep_paper_section/scripts/bench_pd_matrix.sh \
+    --with-rr --with-eager
+```
+
+Each run writes to `outputs/pd_matrix/<config>_<mode>_seed<N>/` with
+`metrics.summary.json`, `breakdown.json`, `apc.txt`, `gpu_util.csv`, and
+per-instance vLLM logs (the latter contain the step-level
+`KV cache: X%` lines needed for the C3 time-series figure).
+
 ## Caveats / open items
 
 - **C7 uses legacy data**. The footer of `fig_c7_routing_lever.pdf` says so:
-  PD-sep numbers come from the random-sampled trace + `--enforce-eager`. Re-run
-  on `traces/w600_r0.0015_st30.jsonl` with cuda-graphs on before paper-grade
-  citation. The plotting code keeps the source numbers in a single `ROWS`
-  table (top of `plot_routing_lever.py`) for a one-line swap.
-- **C2/C3/C4/C5 figures are not produced** because the experiments have not
-  been re-run. The 4h matrix proposed in the prior conversation turn
-  (Combined + RR, Combined + cache-aware, PD-sep 4P+4D, PD-sep 6P+2D, plus
-  eager-vs-cudagraph ablation, ×3 seeds) is the prerequisite.
+  PD-sep numbers come from the random-sampled trace + `--enforce-eager`.
+  After `pd_matrix` lands, swap the four numbers in `plot_routing_lever.py`'s
+  `ROWS` table and re-render.
+- **C2/C3/C4/C5 figures depend on `pd_matrix` outputs**. Followup plotters
+  (TBD) will read `outputs/pd_matrix/*/metrics.summary.json`,
+  `breakdown.json`, and the `KV cache: X%` lines from per-instance logs to
+  produce: bar chart with error bars (C2), KV utilization time-series (C3),
+  TTFT stacked breakdown (C4), 2x2 cuda-graph ablation (C5).
+- **C8 (mined logs)**: rejected — existing PD-sep `outputs/exp3_*` directories
+  have per-request metrics but no per-stage breakdown and no step-level KV
+  utilization. C3/C4 require fresh runs with proxy `/breakdown` collection
+  (already automatic in `bench.sh collect_artifacts()`).
 - **C6 is analytical**, so it is independent of any re-run. The numbers
   match `scripts/compute_roofline.py` (constants are duplicated; if one
   changes, the other must change too).
+- **fig_kv_memory_wall.pdf** is analytical with one empirical anchor (the
+  star marker for REPORT.md §3.3 6P+2D @ 97 % KV utilization). It does not
+  need a re-run, but the empirical anchor's *pinpoint* would be more
+  rigorous from a `pd_matrix` 6P+2D log (KV-utilization time-series rather
+  than the single snapshot).