# MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)

Persistent record of the phase-interference microbench used to put a
quantitative upper bound on **what PD-disaggregation can buy** under the
chunked-prefill-on baseline. Re-runs append a dated section at the
bottom; the **Summary** block is what gets cited.

---

## Summary (latest)

| Headline | Value |
|---|---|
| Baseline single-stream TPOT (D=1, idle GPU) | **4.8 ms** |
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |

**What MB1 actually measures**:

> During a prefill burst, every ongoing decode stream is essentially
> halted (per-stream effective TPOT is 15×–2000× baseline, scaling with
> prefill size). The **total decode time lost per prefill event is
> `D × T_prefill`** (D concurrent decodes each lose ~T_prefill of useful
> work). For the trace mean (P ≈ 33k tokens, T_prefill ≈ 4.5 s) at D=8
> that's **~36 seconds of decode-equivalent work lost per request**.
> This is the **upper bound on what PD-disaggregation's phase isolation
> could recover** on the decode side.

**⚠ Correction (2026-05-27)**: an earlier version of this README framed
the §3.2 PD-disagg argument as "phase-isolation benefit is capped at
the decode duration of the new request (~50–200 ms), so MB2 transfer
cost dominates". That framing was wrong. The correct accounting is
benefit-per-prefill-event = D × T_prefill (aggregate decode time saved
across all stalled streams), which is **much larger than per-request
transfer cost**. The actual reason static PD-disagg fails in agentic
is **D-side KV pool capacity** (`figs/f4b_pdsep_kv_wall.png`), not a
cost-vs-benefit imbalance on phase isolation. See `RESULTS_SUMMARY.md`
section 4 for the corrected framing.

## Setup

| Component | Value |
|---|---|
| Host | dash1, H20 96 GiB, driver 570.133.20 |
| Venv | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` |
| vLLM | 0.18.1 official wheel (chunked-prefill default-on, V1 engine) |
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
| Launch flags | `--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192` |
| kv_connector | **none** (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2) |

## Method

Adapted from `microbench/interference/driver.py`:

1. Start D streaming decode requests on `/v1/chat/completions` with a
   long max_tokens cap. Discard the first 32 tokens as warmup.
2. After 1 s, inject one prefill-only request with `max_tokens=1` and
   an input of `P` synthetic tokens (uuid-seeded for zero prefix-cache
   reuse). Measure the prefill's TTFT.
3. Bin the *during-prefill* tokens from each decode stream by whether
   their wall-clock falls inside `[prefill_inject_ts, prefill_inject_ts +
   prefill_ttft]`. Report inter-token p50 / p90.
4. Bin a baseline run (D streams, no prefill injection) the same way.

We additionally compute the **effective per-stream TPOT during the
prefill burst** as the single most informative summary:

```
eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)
```

This is the average rate at which each decode stream produces tokens
while a prefill is in flight. Compared to baseline TPOT it gives the
real per-stream throughput penalty (chunked-prefill p50 looks deceptively
fine because most decode-token intervals during the burst are at normal
speed; p90 sees the stall but is itself noisy; the effective TPOT is
the cleanest "average over the whole burst window" number).

## Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192

3 D × 5 P × 3 reps. Aggregated by `analyze_mb1.py`.

| D | P (tok) | base TPOT (ms) | prefill_ttft (ms) | per-stream tokens during | effective TPOT during (ms) | penalty | max PD-disagg benefit per stream (ms) |
|--:|--:|--:|--:|--:|--:|--:|--:|
| 1 |   2 048 | 4.79 |   163 | 4.0 |     41 |     8× |    144 |
| 1 |   8 192 | 4.78 |   584 | 5.0 |    117 |    24× |    560 |
| 1 |  32 768 | 4.78 | 4 515 | 5.0 |    903 |   189× |  4 491 |
| 1 |  65 536 | 4.78 | 15 568 | 5.3 |  2 919 |   610× | 15 542 |
| 1 | 131 072 | 4.78 | 56 765 | 5.7 | 10 017 | 2 094× | 56 738 |
| 4 |   2 048 | 5.62 |   138 | 3.9 |     36 |     6× |    117 |
| 4 |   8 192 | 6.08 |   574 | 4.5 |    128 |    21× |    547 |
| 4 |  32 768 | 6.09 | 4 529 | 11.9 |    381 |    63× |  4 457 |
| 4 |  65 536 | 5.85 | 15 587 | 19.8 |    789 |   135× | 15 471 |
| 4 | 131 072 | 6.27 | 56 697 | 37.4 |  1 517 |   242× | 56 463 |
| 8 |   2 048 | 7.71 |   143 | 4.5 |     32 |     4× |    109 |
| 8 |   8 192 | 7.69 |   583 | 5.1 |    114 |    15× |    544 |
| 8 |  32 768 | 7.42 | 4 520 | 11.7 |    387 |    52× |  4 434 |
| 8 |  65 536 | 7.67 | 15 615 | 20.6 |    757 |    99× | 15 457 |
| 8 | 131 072 | 7.74 | 56 991 | 40.2 |  1 419 |   183× | 56 680 |

**Reading the table**:

- *Baseline TPOT* grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8).
  Multi-stream decoding has small but nonzero contention even without
  prefill.
- *Effective TPOT during* grows mostly with P: a single 8k prefill stalls
  decode for ~580 ms regardless of D, so each stream emits only a handful
  of tokens during that 580 ms window — effective per-stream TPOT
  collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
- *Penalty* is the eff/baseline ratio. Above 50× for P ≥ 32k. Above
  500× for D=1 at P ≥ 65k.
- *Max PD-disagg benefit per stream* = `prefill_ttft − per_stream_tokens
  × baseline_TPOT` ≈ `prefill_ttft` (since interference essentially
  halts decode). This is the entire prefill duration's worth of decode
  time that could in principle be recovered.

**Connecting to the §3.2 PD-disagg argument** (corrected):

PD-disagg's promised phase-isolation benefit is **per prefill event**,
not per request. When a new prefill arrives, it stalls every concurrent
decode stream on the same GPU. The aggregate decode time lost across
those D streams is `D × T_prefill`. PD-disagg moving prefill off-decode-GPU
recovers all of it.

Plugging numbers per prefill event:

| Prefill size | T_prefill | PD-disagg cost (MB2 T_transfer) | PD-disagg benefit (D=8 × T_prefill) | Ratio |
|---:|---:|---:|---:|---:|
| 2k tok (trace lower) | 0.14 s | 8 ms | 1.1 s | 0.7 % |
| 33k tok (trace mean) | 4.5 s | 320 ms | 36 s | 0.9 % |
| 125k tok (~p99) | 57 s | 1.9 s | 456 s | 0.4 % |

On the **phase-isolation axis alone**, PD-disagg wins by 100×–250×.
The reason static PD-disagg nonetheless **fails in agentic** is a
*different* failure mode: the D-side KV pool cannot fit p90+ requests
(p99 = 11.5 GiB; D-instance pool ≈ 38 GiB; 4P+4D halves system-wide
decode capacity → TTFT p50 62×, success rate 99.5% → 52% in colleague's
4P+4D experiment). The structural problem is **capacity** (see
`figs/f4b_pdsep_kv_wall.png`), not transfer-cost vs phase-isolation
trade-off.

## Reproduction

```bash
# vllm pair-free single-instance launch
ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
  bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'

# sweep
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
    --host 127.0.0.1 --port 8000 \
    --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
    --reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'

# pull + analyze
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
    analysis/mb1/summary.csv
.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
  --summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
.venv/bin/python microbench/fresh_setup/plot_mb1.py \
  --mb1 analysis/mb1/breakdown.json \
  --mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
  --mb2-inter analysis/mb2/inter_kvboth_breakdown.json

# teardown
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'
```

## Open questions / next runs

- **Chunk size sensitivity**: this run uses `--max-num-batched-tokens
  8192`. Sarathi-Serve goes smaller (e.g. 1024) and recovers more
  decode interleaving inside each prefill burst. Worth running
  chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
- **Higher D**: 12, 16 streams to see whether the penalty saturates or
  keeps shrinking per-stream.
- **Cross-validate effective_TPOT_during with token-time-series plot**:
  raw per-token timestamps could reveal whether the stall is a few big
  spikes or many small ones (currently inferred from p50/p90 spread).

## Run log

### 2026-05-27 — dash1 GPU 0, chunk_tokens=8192

3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
Figure: `figs/mb1_interference.png`. The figure
`figs/pd_cost_vs_benefit.png` from the original commit `029821c` was
based on the wrong "benefit ≤ decode duration" accounting; **deleted in
the correction commit**.