agentic-kvc/REPORT.md

# Milestone Report: Elastic P2P vs PD-Combined Baseline

**Date**: 2026-05-22
**Author**: Gahow Wang
**Status**: Phase 1 complete — baseline + elastic validated, system-level analysis done

---

## 1. Research Question

For agentic LLM workloads (long input, short output, high KV cache reuse), is prefill-decode disaggregation beneficial? If full PD separation hurts (proven in §3), can **selective** disaggregation of only heavy requests improve serving latency while preserving KV cache locality?

## 2. Experimental Setup

### 2.1 Hardware

| Resource | Spec |
|----------|------|
| Machine | dash0 / dash1 (identical config) |
| GPU | 8× NVIDIA H20 96GB HBM, NVLink |
| Network | 4× ConnectX-7 200Gbps RDMA |
| Storage | cpfs shared storage across machines |

### 2.2 Software

| Component | Version | Notes |
|-----------|---------|-------|
| vLLM | 0.18.1 (source in `third_party/vllm/`) | Patched scheduler assert (see `patches/`) |
| Mooncake | 0.3.10 | RDMA-based KV transfer between instances |
| Python | 3.x managed by `uv` | `.venv/` at project root |
| Model | `Qwen3-Coder-30B-A3B-Instruct` | MoE 128 experts top-8, 3B active params |
| Model path | `~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` | Same on dash0 and dash1 |

### 2.3 Workload Trace

| Property | Value |
|----------|-------|
| Source | GLM-5.1 Agentic Coder, production cluster, 2h window |
| Raw trace | `~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl` on dash0 |
| Total requests | 2,114,220 |
| Avg input tokens | 33,600 (p50=20k, p90=88k) |
| Avg output tokens | 445 (p50=80) |
| I/O ratio | 75.6× aggregate |
| Prefill token share | 98% |
| KV reuse (intra-session) | 91% of reusable blocks |
| Theoretical max APC | 71% (infinite cache, single instance) |

**Sampled trace for benchmarks**: `traces/sampled_1000req_seed42.jsonl` (1000 requests, seed=42, preserving session structure). For 200-request ablations: replayer `--request-limit 200`.

### 2.4 Two Configurations Compared

#### Baseline: PD-Combined (8× TP=1 DP=8)

```
8 independent vLLM instances, 1 GPU each, no Mooncake.
All instances do both prefill and decode.
Global scheduler (cache_aware_proxy.py --combined) handles:
  - Session-sticky routing (multi-turn → same instance)
  - Load-aware override (if pinned instance > 2× avg load, redirect)
  - Cache-hit scoring (prefer instance with matching prefix blocks)
```

Launch:
```bash
# On dash0:
for i in $(seq 0 7); do
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \
    vllm serve ~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
        --port $((8000+i)) --tp 1 \
        --enable-prefix-caching --enforce-eager \
        --gpu-memory-utilization 0.9 --max-model-len 200000 \
        > /tmp/ab_base_$i.log 2>&1 &
done

python scripts/cache_aware_proxy.py \
    --combined http://127.0.0.1:800{0..7} --port 9090
```

#### Elastic P2P Offload (8× TP=1 kv_both + selective offload)

```
8 independent vLLM instances, 1 GPU each, all kv_role=kv_both (Mooncake).
Same global scheduler, plus elastic offload logic:
  - Proxy classifies each request: WARM (<5k new), MEDIUM (5-20k), HEAVY (>20k)
  - WARM/MEDIUM: co-located on session-sticky instance (no KV transfer)
  - HEAVY: prefill on a different instance (P), KV via Mooncake RDMA,
    decode on session-sticky instance (D)
  - Cap: max 4 concurrent offloads (MAX_OFFLOAD_INFLIGHT)
  - P instance selection: round-robin with overload skip
```

Launch:
```bash
# On dash1 (or use scripts/launch_elastic_p2p.sh):
for i in $(seq 0 7); do
    VLLM_MOONCAKE_BOOTSTRAP_PORT=$((8998+i)) \
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \
    vllm serve ~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
        --port $((8000+i)) --tp 1 \
        --enable-prefix-caching --enforce-eager \
        --gpu-memory-utilization 0.9 --max-model-len 200000 \
        --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
        > /tmp/ab_elastic_$i.log 2>&1 &
    sleep 2  # stagger to avoid NCCL port collision
done

# Wait for bootstrap servers
for bp in $(seq 8998 9005); do
    until curl -s localhost:$bp/query > /dev/null 2>&1; do sleep 2; done
done

python scripts/cache_aware_proxy.py \
    --combined http://127.0.0.1:800{0..7} \
    --bootstrap-ports 8998,8999,9000,9001,9002,9003,9004,9005 \
    --offload --heavy-threshold 20000 --port 9090
```

### 2.5 Benchmark Parameters

| Parameter | Value |
|-----------|-------|
| Requests | 200 (from sampled 1000-req trace, `--request-limit 200`) |
| Time scale | 20× (compress 2h trace into ~6min) |
| Max inflight sessions | 8 |
| Request timeout | 600s |
| vLLM flags | `--enforce-eager --enable-prefix-caching --max-model-len 200000` |
| GPU memory util | 0.9 |
| Fresh restart | Both configs started from cold (no warm cache) |

### 2.6 Reproducing the Benchmark

```bash
# Activate environment
cd ~/agentic-kv && source .venv/bin/activate

# Ensure sampled trace exists
python scripts/sample_trace.py \
    --input ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
    --output traces/sampled_1000req_seed42.jsonl \
    --target-requests 1000 --seed 42

# Start GPU monitoring (in a separate terminal)
bash scripts/gpu_monitor.sh > outputs/<tag>/gpu_util.csv &

# Run replayer against proxy
python -m replayer \
    --trace traces/sampled_1000req_seed42.jsonl \
    --output outputs/<tag>/metrics.jsonl \
    --endpoint http://localhost:9090 \
    --time-scale 20 --max-inflight-sessions 8 \
    --request-limit 200 -v

# Collect proxy breakdown (elastic only)
curl -s http://localhost:9090/breakdown > outputs/<tag>/breakdown.json

# Collect APC from vLLM logs
for i in $(seq 0 7); do
    grep "Prefix cache hit rate\|External prefix cache hit rate" /tmp/<prefix>_$i.log | tail -2
done
```

## 3. Results

> **Errata (2026-05-22)**: The initial cross-machine A/B (dash0 baseline vs dash1 elastic) reported -44% E2E improvement. Post-hoc analysis revealed the dash0 baseline instances were **not freshly restarted** — residual KV cache from prior experiments caused 2× TTFT inflation. All results below use verified fresh-restart experiments on the same machine.

### 3.1 Fair Comparison (all fresh-restart, same machine dash0, 200 req)

| Config | OK/N | TTFT p50 | TTFT p90 | TPOT p90 | E2E p50 |
|--------|------|----------|----------|----------|---------|
| **Baseline (no Mooncake)** | **198/200** | **1.075s** | **9.384s** | **0.076s** | **5.075s** |
| LMetric routing | 198/200 | 1.099s | 9.392s | 0.073s | 5.205s |
| Elastic P2P (kv_both) | 195/200 | 1.018s | 11.312s | 0.085s | 6.977s |

### 3.2 Per-Class Breakdown

**Baseline (fresh):**

| Class | Count | % | TTFT p50 | TTFT p90 | TPOT p90 |
|-------|-------|---|----------|----------|----------|
| WARM (<5k) | 46 | 23% | 0.137s | 0.262s | 0.061s |
| MEDIUM (5-20k) | 50 | 25% | 0.921s | 1.846s | 0.079s |
| HEAVY (20-50k) | 64 | 32% | 2.660s | 6.278s | 0.076s |
| HEAVY (>50k) | 38 | 19% | 9.587s | 30.415s | 0.102s |

**Elastic P2P (fresh):**

| Class | Count | % | TTFT p50 | TTFT p90 | TPOT p90 |
|-------|-------|---|----------|----------|----------|
| WARM (<5k) | 46 | 23% | 0.142s | 0.279s | 0.072s |
| MEDIUM (5-20k) | 50 | 25% | 0.766s | 1.814s | 0.197s |
| HEAVY (>20k) | 99 | 51% | 6.390s | 22.668s | 0.085s |

### 3.3 Success Rate

| Config | OK | Total | Rate | Failure mode |
|--------|-----|-------|------|-------------|
| Baseline | 198 | 200 | 99.0% | RemoteProtocolError (replayer-side) |
| Elastic P2P | 195 | 200 | 97.5% | 2× RemoteProtocolError + 3× ReadTimeout on >60k |

Elastic's 3 extra errors are D-side KV pull failures: prefill succeeded on P, KV pushed to Mooncake, but D never produced first token (decode scheduler couldn't allocate KV cache space). Prefill timeout fallback (120s → co-located) was never triggered.

### 3.4 Routing Policy: Linear vs LMetric (OSDI'26)

LMetric (`score = P_tokens × BS`) vs linear (`score = ongoing_tokens - α·cache_hit`). Both fresh-restart, same trace.

| Policy | TTFT p50 | TTFT p90 | TPOT p90 | E2E p50 | Delta E2E |
|--------|----------|----------|----------|---------|-----------|
| Linear | 1.086s | 9.432s | 0.077s | 5.423s | — |
| LMetric | 1.099s | 9.392s | 0.073s | 5.205s | **-4.0%** |

LMetric provides modest improvement through better load balancing. Routing policy headroom is limited for this workload.

### 3.5 Errata: Why Prior Cross-Machine A/B Was Invalid

The initial comparison (commit `1e86285`) reported:
```
Baseline (dash0): TTFT50=2.383  E2E50=10.232  ← WRONG (warm instances)
Elastic  (dash1): TTFT50=1.315  E2E50=5.708
Delta:                   -45%          -44%    ← INVALID
```

**Evidence that prior baseline was not fresh:**
1. `inst_7` APC = 68.3% — impossible from 25 cold-start requests (max ~25%)
2. WARM TTFT p90 = 3.327s (fresh = 0.262s, 12.7× gap) — indicates KV cache memory pressure from prior experiments
3. HEAVY TPOT p90 = 0.154s (fresh = 0.076s, 2.0× gap) — heavy prefill-decode interference from full KV cache

The elastic numbers on dash1 were genuinely fresh. The "improvement" was actually comparing fresh elastic against degraded baseline.

## 4. System-Level Analysis

### 4.1 Elastic P2P Does Not Improve Single-Machine Performance

Under fair comparison (same machine, both fresh):

| Metric | Baseline | Elastic | Delta |
|--------|----------|---------|-------|
| TTFT p50 | 1.075s | 1.018s | -5.3% |
| TTFT p90 | 9.384s | 11.312s | +20.5% |
| TPOT p90 | 0.076s | 0.085s | +11.6% |
| E2E p50 | 5.075s | 6.977s | +37.5% |

Elastic is **worse** on all metrics except TTFT p50. Root causes:

**1. Mooncake kv_both memory overhead**

Each instance with `kv_role=kv_both` maintains RDMA buffers + Mooncake bootstrap server, reducing GPU memory available for KV cache. This affects ALL requests (including WARM/MEDIUM that don't use P2P transfer), causing more cache eviction and higher TPOT.

Evidence: MEDIUM TPOT p90 = 0.197s (elastic) vs 0.079s (baseline) — **2.5× worse** despite MEDIUM requests not using P2P at all.

**2. D-side KV pull failures**

3 HEAVY requests completed prefill on P instance successfully but D-side never produced first token. The KV cache on D was too full to allocate space for the transferred blocks. These became 600s timeouts.

**3. P2P overhead without proportional benefit**

The P2P path adds: prefill queue on P (p50=6.3s) + KV transfer + decode start on D (p50=0.8s). For requests where the D instance isn't under heavy prefill load (which is the case on fresh instances), co-located execution is faster.

### 4.2 When Elastic P2P Could Help

Elastic P2P is designed for the scenario where D-instance decode is disrupted by co-located heavy prefill. On fresh instances with 200 requests, this contention is moderate. The benefit may emerge under:
- Higher sustained load (1000+ concurrent requests)
- Longer experiment duration (KV cache fills up, eviction pressure increases)
- Multi-machine deployment (P on a different node, no memory competition)

## 5. Data & Log Locations

### 5.1 Experiment Outputs (on respective machines)

| Directory | Machine | Config | Notes |
|-----------|---------|--------|-------|
| `outputs/ab_baseline/` | dash0 | Combined 8× TP=1 | ~~Initial A/B~~ (INVALIDATED: warm instances) |
| `outputs/ab_elastic/` | dash0 | Elastic P2P cap=4 | ~~Initial A/B~~ (INVALIDATED) |
| `outputs/baseline_stability_fresh/` | dash0 | Combined 8× fresh | **Canonical baseline** (§3.1) |
| `outputs/elastic_stability_*/` | dash0 | Elastic P2P kv_both fresh | **Canonical elastic** (§3.1) |
| `outputs/ab_linear/` | dash0 | Linear policy, 200 req | §3.4 routing policy comparison |
| `outputs/ab_lmetric/` | dash0 | LMetric policy, 200 req | §3.4 routing policy comparison |
| `outputs/gpu_ab_combined/` | local | Combined 8× TP=1 | Earlier run, has gpu_util.csv |
| `outputs/gpu_ab_pdsep/` | local | PD-Sep 4P+4D | Earlier run, has gpu_util.csv |
| `outputs/exp2_combined_tp1_dp8/` | local | Combined 8× TP=1 | 1000 req, cache-aware |
| `outputs/exp3_pd_sep_tp1_mooncake/` | local | PD-Sep 4P+4D Mooncake | 1000 req |

### 5.2 vLLM Instance Logs

| Path pattern | Machine | Config |
|-------------|---------|--------|
| `/tmp/ab_base_$i.log` | dash0 | Baseline instances 0-7 |
| `/tmp/ab_elastic_$i.log` | dash1 | Elastic instances 0-7 |
| `/tmp/lmetric_ab_inst_$i.log` | dash0 | Linear policy instances 0-7 (§3.6) |
| `/tmp/lmetric_inst_$i.log` | dash0 | LMetric policy instances 0-7 (§3.6) |

Logs contain `Prefix cache hit rate` and `External prefix cache hit rate` lines for APC extraction.

### 5.3 Trace Data

| Path | Machine | Description |
|------|---------|-------------|
| `~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl` | dash0 | Full 2h production trace (2.1M requests) |
| `traces/sampled_1000req_seed42.jsonl` | all | Sampled 1000 requests (gitignored, regenerate with `sample_trace.py`) |

### 5.4 Analysis Documents

| File | Content |
|------|---------|
| `analysis/pd_separation_analysis.md` | Main report: PD-Sep vs Combined + Elastic P2P (§5) |
| `analysis/elastic_offload_design.md` | Elastic P2P design rationale |
| `analysis/kv_lifecycle_design.md` | KV cache eviction policy analysis |
| `analysis/adaptive_prefill_offload_design.md` | Initial adaptive offload design (superseded by elastic) |

## 6. Repository Structure

```
agentic-kv/
├── analysis/                    # Research reports and design docs
│   ├── pd_separation_analysis.md    # Main comprehensive report
│   ├── elastic_offload_design.md    # Elastic P2P design
│   ├── kv_lifecycle_design.md       # Cache eviction analysis
│   └── ...
├── replayer/                    # Trace replay framework
│   ├── __main__.py              # CLI entry: python -m replayer
│   ├── replay.py                # Async replayer (session-aware, SSE streaming)
│   ├── trace.py                 # TraceRequest dataclass, session/hash_id handling
│   └── metrics.py               # RequestMetrics, crash-safe JSONL sink
├── scripts/
│   ├── cache_aware_proxy.py     # Global scheduler (combined + PD-sep + elastic offload)
│   ├── sample_trace.py          # Cluster-to-machine trace sampler
│   ├── launch_vllm.sh           # Launch combined TP=8
│   ├── launch_pd_mooncake.sh    # Launch PD-Sep with Mooncake
│   ├── launch_elastic_p2p.sh    # Launch elastic P2P (8× kv_both + offload proxy)
│   ├── run_experiments.sh       # Full experiment matrix (combined/PD-sep)
│   ├── run_benchmark.sh         # Single benchmark run
│   ├── gpu_monitor.sh           # GPU utilization sampler (5s CSV)
│   ├── compute_roofline.py      # Prefill/decode roofline analysis
│   ├── analyze_*.py             # Various analysis scripts
│   └── compare_*.py             # Experiment comparison scripts
├── patches/
│   ├── 0001-fix-kv-transfer-abort-race.patch
│   └── README.md
├── third_party/vllm/            # vLLM 0.18.1 source (with patch applied)
├── outputs/                     # Experiment results (gitignored)
├── traces/                      # Sampled traces (gitignored)
├── TODO.md                      # Original research goals
└── REPORT.md                    # This milestone report
```

## 7. Key Scripts Reference

| Script | What it does | Key flags |
|--------|-------------|-----------|
| `scripts/cache_aware_proxy.py` | Global scheduler + elastic offload proxy | `--combined`, `--offload`, `--policy {linear,lmetric}`, `--heavy-threshold`, `--bootstrap-ports` |
| `scripts/run_lmetric_ab.sh` | A/B: linear vs lmetric routing policy | Runs both experiments with fresh restart |
| `scripts/run_elastic_stability_test.sh` | Elastic vs baseline with full isolation | Fresh start/stop per experiment |
| `scripts/bench.sh` | Standard single-experiment harness | `--tag`, `--mode {baseline,elastic}` |
| `scripts/sample_trace.py` | Sample complete sessions from cluster trace | `--target-requests`, `--seed` |
| `python -m replayer` | Replay trace against vLLM endpoint | `--time-scale`, `--max-inflight-sessions`, `--request-limit` |
| `scripts/gpu_monitor.sh` | Sample nvidia-smi to CSV | Pipe to `outputs/<tag>/gpu_util.csv` |
| `scripts/launch_elastic_p2p.sh` | Launch all 8 kv_both instances + offload proxy | `HEAVY_THRESHOLD`, `MAX_OFFLOAD` env vars |

## 8. Conclusions & Next Steps

### Established findings:
1. Full PD separation is **net negative** for single-machine agentic workloads (KV cache memory wall)
2. Cache-aware session-sticky routing is the **dominant optimization** (+24pp APC, -60% TTFT vs round-robin)
3. **Elastic P2P offload does NOT improve single-machine performance** — Mooncake kv_both memory overhead (+11% TPOT, +37% E2E) outweighs prefill isolation benefit under moderate load (200 req)
4. LMetric (OSDI'26) provides modest **E2E -4%** over linear routing; routing policy headroom is limited
5. **Experimental methodology matters**: warm vs fresh instances cause 2× TTFT difference; all comparisons must use verified fresh restart

### Lessons learned:
- Prior cross-machine A/B (commit `1e86285`) was invalid — warm baseline inflated by 2× due to residual KV cache state
- `kv_role=kv_both` has non-trivial always-on overhead even when P2P transfer is not used
- Experiment isolation (kill all → verify GPU free → fresh start) is critical for reproducibility

### Open problems:
1. Elastic P2P may help under **sustained high load** (KV cache pressure makes co-located interference worse) — needs 1000-req experiment
2. Mooncake kv_both memory overhead quantification and potential lazy initialization
3. Multi-machine elastic (P on different node, no memory competition)
4. Router state accuracy: proxy shadow state vs vLLM-internal exact state (TODO: vLLM → Redis → router)
5. `scripts/bench.sh` standardized harness to prevent future warm-instance mistakes

---

*Updated 2026-05-22. Prior elastic A/B results (commit `1e86285`) invalidated — see §3.5 errata.*