PD-sep matrix results: C2/C3/C4 figures + empirical mechanism refined

Captures 5 runs from the experiment matrix (combined-ca x3 seeds,
pdsep-4p4d seed1, pdsep-6p2d seed1) on traces/w600_r0.0015_st30.jsonl
with cuda graphs enabled. The headline:

  combined-ca:  TTFT p50 0.91s   success 99.5%
  pdsep-4p4d:   TTFT p50 62.8s   success 52%   (69x worse, half dropped)
  pdsep-6p2d:   TTFT p50 51.1s   success 68%   (56x worse, third dropped)

C2 (fig_c2): headline bars per config with error bars.
C3 (fig_c3): per-instance KV utilization time-series. Both PD-sep
  splits hit the memory wall, but the side differs by P:D ratio --
  4P+4D pins the P-side, 6P+2D pins both sides (D-side back-pressures
  P-side).
C4 (fig_c4): TTFT stacked breakdown. 99% of PD-sep TTFT is P-side
  prefill compute; D-side wait + first token is <=1.2s. The bottleneck
  is P-side prefill queueing, not D-side decode wait as the original
  analytical model assumed.

system_analysis.md gains a Layer 5b that reconciles the analytical
KV-wall model (which considered D-side only) with the empirical
finding that the wall hits whichever side has fewer GPUs, and
co-saturates both at extreme splits via D-side back-pressure.

plot_pd_matrix.py ingests outputs/pd_matrix/* into all four figures.
bench.sh gained AGENTIC_STEP_LOG_DIR hooks for future runs (set during
this work but not used by the current matrix's data).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

analysis/pd_sep_paper_section/system_analysis.md

@@ -160,6 +160,97 @@ The empirical KV occupancy on the 6P+2D run was 97 %
 (`analysis/pd_separation_analysis.md` §3.3) — the model and the
 measurement agree to within the resolution of the steady-state assumption.
 
+## Layer 5b: empirical refinement — the bottleneck side depends on the P:D split
+
+The model above predicts D-side saturation. The 6P+2D run in
+`analysis/pd_separation_analysis.md` §3.3 is consistent with it. But the
+new 4P+4D run (`outputs/pd_matrix/pdsep-4p4d_cudagraph_seed1/`, captured
+during this section's experiment matrix) tells a richer story.
+
+Empirical numbers (combined-ca vs both PD-sep splits, same trace, all
+cudagraph, `figures/fig_c2_pdsep_vs_combined.pdf`):
+
+| metric | combined-ca (N=3) | pdsep-4p4d (N=1) | pdsep-6p2d (N=1) |
+|---|---|---|---|
+| success | 99.5 % | **52 %** (444/850) | **68 %** (574/850) |
+| TTFT p50 | 0.91 s | **62.8 s** (69×) | **51.1 s** (56×) |
+| TTFT p90 | 12.7 s | **491 s** (39×) | **400 s** (31×) |
+| TPOT p90 | 0.027 s | 0.013 s (-52 %) | 0.020 s (-26 %) |
+| E2E p50 | 2.5 s | **65.1 s** (26×) | **53.4 s** (21×) |
+| wall clock | 944 s | 7558 s (8×) | 3693 s (3.9×) |
+
+The per-stage TTFT decomposition (`figures/fig_c4_ttft_stacked.pdf`)
+shows that for both PD-sep splits **>97 % of TTFT is P-side prefill
+compute** (65.6 s / 66.2 s in 4p4d; 43.1 s / 44.3 s in 6p2d). D-side
+wait + first token is at most 1.2 s in either config.
+
+The KV-utilization time-series (`figures/fig_c3_kv_timeseries.pdf`)
+tells the full story:
+
+- **combined-ca**: 8 GPUs oscillate 0–98 %, peaks bursty and short
+- **pdsep-4p4d**: P-instances (orange) pinned at 85–100 % the entire
+  2-hour run; D-instances (red) bounce between 10 % and 50 % — only
+  P side hits the wall
+- **pdsep-6p2d**: **both** sides pinned near 100 % the entire run
+  (per-instance peaks 99–100 % across all 8). P-side fills because D
+  back-pressures (D can't free KV slots fast enough → P can't
+  hand off → P-side KV accumulates).
+
+This refines Layer 5: PD separation hits a memory wall on whichever
+side has fewer GPUs, and at extreme splits it co-saturates both sides
+through D-back-pressure.
+
+### Why P-side fills in 4P+4D
+
+Two effects combine on P:
+
+1. **Compute concentration.** Combined spreads prefill across 8 GPUs;
+   4P+4D over 4. Per-P-GPU prefill load is 2× the per-Combined-GPU load.
+   With chunked prefill, multiple in-flight prefills compete for the
+   scheduler.
+
+2. **KV residency on P.** Mooncake does block-by-block transfer *after*
+   the full prefill completes. Until D pulls and acknowledges every
+   block, the completed-but-not-yet-transferred KV sits in P's pool —
+   on top of all the partially-prefilled in-flight KV. Many concurrent
+   33–132 k contexts overwhelm a single 28 GB pool.
+
+This is the same memory-wall mechanism Layer 5 described, but on the
+*prefill* side. The Layer 5 analytical model in
+`figures/fig_kv_memory_wall.pdf` accounted only for *decode-side* KV
+demand. The full model is:
+
+```
+P-side occupancy = (in_flight_prefills × KV_per_req) / (N_P × pool)
+D-side occupancy = (concurrent_decode × KV_per_req) / (N_D × pool)
+```
+
+Whichever side hits the wall first becomes the back-pressure source.
+4P+4D's P-side fills first because 4 GPUs are doing 8 GPUs' worth of
+prefill. 6P+2D's D-side hits the wall (4× concentration), which then
+back-pressures P (no slots to hand off into) until P also fills. In
+either case, *some* side blocks — which is why PD separation regresses
+across the P:D ratios we tested.
+
+### Updated falsifiable condition
+
+The condition for PD separation to *not* regress is now two-sided:
+
+```
+max(
+  in_flight_prefills × KV_per_req / (N_P × pool),
+  concurrent_decode  × KV_per_req / (N_D × pool)
+) < 1
+```
+
+For chatbot workloads (KV/req ≈ 200 MB), this holds easily on either
+side. For agentic with KV/req ≈ 3.3 GB on average and 10–13 GB at the
+tail, both terms cross 1 well below the chosen N_P or N_D.
+
+Followups: re-render `fig_kv_memory_wall.pdf` to show both P and D
+curves once the 6P+2D run lands, with the empirical P-side and D-side
+peaks marked.
+
 ## Layer 6: the DistServe / Splitwise assumption that silently breaks
 
 To formalize: the regime in which PD separation pays is bounded by both a