# MB1 — Prefill–Decode Interference (chunked-prefill on, vLLM 0.18.1 default)

Persistent record of the phase-interference microbench used to put a
quantitative upper bound on **what PD-disaggregation can buy** under the
chunked-prefill-on baseline. Re-runs append a dated section at the
bottom; the **Summary** block is what gets cited.

---

## Summary (latest)

| Headline | Value |
|---|---|
| Baseline single-stream TPOT (D=1, idle GPU) | **4.8 ms** |
| Effective per-stream TPOT during **8k-token** prefill burst (D=8) | **114 ms (≈15× baseline)** |
| Effective per-stream TPOT during **32k-token** prefill burst (D=8) | **388 ms (≈52×)** |
| Effective per-stream TPOT during **131k-token** prefill burst (D=8) | **1419 ms (≈183×)** |
| Maximum PD-disagg benefit per agentic decode | **≤ 50–200 ms** (= decode duration) |

**§3.2 headline (cost vs benefit, this run + MB2)**:

> Under chunked-prefill, every ongoing decode stream is essentially
> **halted while a prefill chunk is in flight** — per-stream effective
> TPOT during the burst is 15× to 2000× baseline, scaling with prefill
> size. PD-disagg can recover this stall, but the recovery is bounded by
> the **decode duration** of the request being protected. For agentic,
> decode is 50–200 ms (tool-call output). MB2 shows PD-disagg pays
> 300 ms – 10 s of KV-transfer cost per request to do that recovery. The
> cost exceeds the benefit ceiling for any per-request KV ≥ ~80 MiB
> (~830 tokens) — well below all agentic operating points. The benefit
> never beats the cost in this workload.

## Setup

| Component | Value |
|---|---|
| Host | dash1, H20 96 GiB, driver 570.133.20 |
| Venv | `/home/admin/cpfs/wjh/agentic-kv-fresh/.venv` |
| vLLM | 0.18.1 official wheel (chunked-prefill default-on, V1 engine) |
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` |
| Launch flags | `--tensor-parallel-size 1 --enable-prefix-caching --gpu-memory-utilization 0.9 --max-model-len 200000 --max-num-batched-tokens 8192` |
| kv_connector | **none** (this measures pure single-GPU phase interference; PD-disagg cost lives in MB2) |

## Method

Adapted from `microbench/interference/driver.py`:

1. Start D streaming decode requests on `/v1/chat/completions` with a
   long max_tokens cap. Discard the first 32 tokens as warmup.
2. After 1 s, inject one prefill-only request with `max_tokens=1` and
   an input of `P` synthetic tokens (uuid-seeded for zero prefix-cache
   reuse). Measure the prefill's TTFT.
3. Bin the *during-prefill* tokens from each decode stream by whether
   their wall-clock falls inside `[prefill_inject_ts, prefill_inject_ts +
   prefill_ttft]`. Report inter-token p50 / p90.
4. Bin a baseline run (D streams, no prefill injection) the same way.

We additionally compute the **effective per-stream TPOT during the
prefill burst** as the single most informative summary:

```
eff_TPOT_during = prefill_ttft_ms / (num_tokens_during_prefill / D)
```

This is the average rate at which each decode stream produces tokens
while a prefill is in flight. Compared to baseline TPOT it gives the
real per-stream throughput penalty (chunked-prefill p50 looks deceptively
fine because most decode-token intervals during the burst are at normal
speed; p90 sees the stall but is itself noisy; the effective TPOT is
the cleanest "average over the whole burst window" number).

## Results — 2026-05-27, dash1 GPU 0, chunk_tokens=8192

3 D × 5 P × 3 reps. Aggregated by `analyze_mb1.py`.

| D | P (tok) | base TPOT (ms) | prefill_ttft (ms) | per-stream tokens during | effective TPOT during (ms) | penalty | max PD-disagg benefit per stream (ms) |
|--:|--:|--:|--:|--:|--:|--:|--:|
| 1 |   2 048 | 4.79 |   163 | 4.0 |     41 |     8× |    144 |
| 1 |   8 192 | 4.78 |   584 | 5.0 |    117 |    24× |    560 |
| 1 |  32 768 | 4.78 | 4 515 | 5.0 |    903 |   189× |  4 491 |
| 1 |  65 536 | 4.78 | 15 568 | 5.3 |  2 919 |   610× | 15 542 |
| 1 | 131 072 | 4.78 | 56 765 | 5.7 | 10 017 | 2 094× | 56 738 |
| 4 |   2 048 | 5.62 |   138 | 3.9 |     36 |     6× |    117 |
| 4 |   8 192 | 6.08 |   574 | 4.5 |    128 |    21× |    547 |
| 4 |  32 768 | 6.09 | 4 529 | 11.9 |    381 |    63× |  4 457 |
| 4 |  65 536 | 5.85 | 15 587 | 19.8 |    789 |   135× | 15 471 |
| 4 | 131 072 | 6.27 | 56 697 | 37.4 |  1 517 |   242× | 56 463 |
| 8 |   2 048 | 7.71 |   143 | 4.5 |     32 |     4× |    109 |
| 8 |   8 192 | 7.69 |   583 | 5.1 |    114 |    15× |    544 |
| 8 |  32 768 | 7.42 | 4 520 | 11.7 |    387 |    52× |  4 434 |
| 8 |  65 536 | 7.67 | 15 615 | 20.6 |    757 |    99× | 15 457 |
| 8 | 131 072 | 7.74 | 56 991 | 40.2 |  1 419 |   183× | 56 680 |

**Reading the table**:

- *Baseline TPOT* grows mildly with D (4.8 ms → 7.7 ms as D goes 1 → 8).
  Multi-stream decoding has small but nonzero contention even without
  prefill.
- *Effective TPOT during* grows mostly with P: a single 8k prefill stalls
  decode for ~580 ms regardless of D, so each stream emits only a handful
  of tokens during that 580 ms window — effective per-stream TPOT
  collapses to 100–130 ms. Larger prefill = more chunks = larger stall.
- *Penalty* is the eff/baseline ratio. Above 50× for P ≥ 32k. Above
  500× for D=1 at P ≥ 65k.
- *Max PD-disagg benefit per stream* = `prefill_ttft − per_stream_tokens
  × baseline_TPOT` ≈ `prefill_ttft` (since interference essentially
  halts decode). This is the entire prefill duration's worth of decode
  time that could in principle be recovered.

Two big caveats for **agentic** application:

1. **Decode is short** (~50–200 ms for tool-call output). The actual
   recoverable benefit per request is bounded by the decode duration,
   not by `prefill_ttft`. If a decode lasts 100 ms and a 5-second prefill
   collides with it, PD-disagg can save at most 100 ms — not 5 s.
2. **PD-disagg pays KV-transfer cost** (MB2: 300 ms – 10 s per request
   for agentic sizes). For any KV ≥ ~80 MiB the cost already exceeds the
   ~100 ms benefit ceiling. Cost > benefit across the whole agentic
   distribution.

## §3.2 cost-vs-benefit figure

`figs/pd_cost_vs_benefit.png` overlays MB1 benefit ceiling (50–200 ms
band, capped by decode duration) on top of MB2 transfer cost curve. The
cost curve crosses the benefit ceiling somewhere around **80 MiB / 830
tokens** of KV — well below the trace mean (192 MiB / 2k tok ≈ trace
mean per request KV, and we know agentic averages 33k tokens, p99
125k). For anything bigger PD-disagg pays more than it can recover.

## Reproduction

```bash
# vllm pair-free single-instance launch
ssh dash1 'GPU=0 PORT=8000 CHUNK_TOKENS=8192 \
  bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh start'

# sweep
ssh dash1 'source /home/admin/cpfs/wjh/agentic-kv-fresh/.venv/bin/activate && \
  python /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_driver.py \
    --host 127.0.0.1 --port 8000 \
    --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
    --decode-batch-sizes 1,4,8 --prefill-tokens 2048,8192,32768,65536,131072 \
    --reps 3 --output-dir /home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results'

# pull + analyze
scp dash1:/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/summary.csv \
    analysis/mb1/summary.csv
.venv/bin/python microbench/fresh_setup/analyze_mb1.py \
  --summary analysis/mb1/summary.csv --out analysis/mb1/breakdown.json
.venv/bin/python microbench/fresh_setup/plot_mb1.py \
  --mb1 analysis/mb1/breakdown.json \
  --mb2-intra analysis/mb2/intra_kvboth_breakdown.json \
  --mb2-inter analysis/mb2/inter_kvboth_breakdown.json

# teardown
ssh dash1 'bash /home/admin/cpfs/wjh/agentic-kv-fresh/scripts/mb1_launch.sh stop'
```

## Open questions / next runs

- **Chunk size sensitivity**: this run uses `--max-num-batched-tokens
  8192`. Sarathi-Serve goes smaller (e.g. 1024) and recovers more
  decode interleaving inside each prefill burst. Worth running
  chunk_tokens ∈ {1024, 2048, 4096, 16384} to map the chunk-size axis.
- **Higher D**: 12, 16 streams to see whether the penalty saturates or
  keeps shrinking per-stream.
- **Cross-validate effective_TPOT_during with token-time-series plot**:
  raw per-token timestamps could reveal whether the stall is a few big
  spikes or many small ones (currently inferred from p50/p90 spread).

## Run log

### 2026-05-27 — dash1 GPU 0, chunk_tokens=8192

3 × 5 × 3 sweep. CSV: `analysis/mb1/summary.csv`. Per-config JSONs on
dash1 at `/home/admin/cpfs/wjh/agentic-kv-fresh/mb1_results/chunk8192/`.
Figures: `figs/mb1_interference.png`, `figs/pd_cost_vs_benefit.png`.