# Milestone Report: Elastic P2P vs PD-Combined Baseline

**Date**: 2026-05-22
**Author**: Gahow Wang
**Status**: Phase 1 complete — baseline + elastic validated, system-level analysis done

---

## 1. Research Question

For agentic LLM workloads (long input, short output, high KV cache reuse), is prefill-decode disaggregation beneficial? If full PD separation hurts (proven in §3), can **selective** disaggregation of only heavy requests improve serving latency while preserving KV cache locality?

## 2. Experimental Setup

### 2.1 Hardware

| Resource | Spec |
|----------|------|
| Machine | dash0 / dash1 (identical config) |
| GPU | 8× NVIDIA H20 96GB HBM, NVLink |
| Network | 4× ConnectX-7 200Gbps RDMA |
| Storage | cpfs shared storage across machines |

### 2.2 Software

| Component | Version | Notes |
|-----------|---------|-------|
| vLLM | 0.18.1 (source in `third_party/vllm/`) | Patched scheduler assert (see `patches/`) |
| Mooncake | 0.3.10 | RDMA-based KV transfer between instances |
| Python | 3.x managed by `uv` | `.venv/` at project root |
| Model | `Qwen3-Coder-30B-A3B-Instruct` | MoE 128 experts top-8, 3B active params |
| Model path | `~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct` | Same on dash0 and dash1 |

### 2.3 Workload Trace

| Property | Value |
|----------|-------|
| Source | GLM-5.1 Agentic Coder, production cluster, 2h window |
| Raw trace | `~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl` on dash0 |
| Total requests | 2,114,220 |
| Avg input tokens | 33,600 (p50=20k, p90=88k) |
| Avg output tokens | 445 (p50=80) |
| I/O ratio | 75.6× aggregate |
| Prefill token share | 98% |
| KV reuse (intra-session) | 91% of reusable blocks |
| Theoretical max APC | 71% (infinite cache, single instance) |

**Sampled trace for benchmarks**: `traces/sampled_1000req_seed42.jsonl` (1000 requests, seed=42, preserving session structure). For 200-request ablations: replayer `--request-limit 200`.

### 2.4 Two Configurations Compared

#### Baseline: PD-Combined (8× TP=1 DP=8)

```
8 independent vLLM instances, 1 GPU each, no Mooncake.
All instances do both prefill and decode.
Global scheduler (cache_aware_proxy.py --combined) handles:
  - Session-sticky routing (multi-turn → same instance)
  - Load-aware override (if pinned instance > 2× avg load, redirect)
  - Cache-hit scoring (prefer instance with matching prefix blocks)
```

Launch:
```bash
# On dash0:
for i in $(seq 0 7); do
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \
    vllm serve ~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
        --port $((8000+i)) --tp 1 \
        --enable-prefix-caching --enforce-eager \
        --gpu-memory-utilization 0.9 --max-model-len 200000 \
        > /tmp/ab_base_$i.log 2>&1 &
done

python scripts/cache_aware_proxy.py \
    --combined http://127.0.0.1:800{0..7} --port 9090
```

#### Elastic P2P Offload (8× TP=1 kv_both + selective offload)

```
8 independent vLLM instances, 1 GPU each, all kv_role=kv_both (Mooncake).
Same global scheduler, plus elastic offload logic:
  - Proxy classifies each request: WARM (<5k new), MEDIUM (5-20k), HEAVY (>20k)
  - WARM/MEDIUM: co-located on session-sticky instance (no KV transfer)
  - HEAVY: prefill on a different instance (P), KV via Mooncake RDMA,
    decode on session-sticky instance (D)
  - Cap: max 4 concurrent offloads (MAX_OFFLOAD_INFLIGHT)
  - P instance selection: round-robin with overload skip
```

Launch:
```bash
# On dash1 (or use scripts/launch_elastic_p2p.sh):
for i in $(seq 0 7); do
    VLLM_MOONCAKE_BOOTSTRAP_PORT=$((8998+i)) \
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \
    vllm serve ~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
        --port $((8000+i)) --tp 1 \
        --enable-prefix-caching --enforce-eager \
        --gpu-memory-utilization 0.9 --max-model-len 200000 \
        --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
        > /tmp/ab_elastic_$i.log 2>&1 &
    sleep 2  # stagger to avoid NCCL port collision
done

# Wait for bootstrap servers
for bp in $(seq 8998 9005); do
    until curl -s localhost:$bp/query > /dev/null 2>&1; do sleep 2; done
done

python scripts/cache_aware_proxy.py \
    --combined http://127.0.0.1:800{0..7} \
    --bootstrap-ports 8998,8999,9000,9001,9002,9003,9004,9005 \
    --offload --heavy-threshold 20000 --port 9090
```

### 2.5 Benchmark Parameters

| Parameter | Value |
|-----------|-------|
| Requests | 200 (from sampled 1000-req trace, `--request-limit 200`) |
| Time scale | 20× (compress 2h trace into ~6min) |
| Max inflight sessions | 8 |
| Request timeout | 600s |
| vLLM flags | `--enforce-eager --enable-prefix-caching --max-model-len 200000` |
| GPU memory util | 0.9 |
| Fresh restart | Both configs started from cold (no warm cache) |

### 2.6 Reproducing the Benchmark

```bash
# Activate environment
cd ~/agentic-kv && source .venv/bin/activate

# Ensure sampled trace exists
python scripts/sample_trace.py \
    --input ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
    --output traces/sampled_1000req_seed42.jsonl \
    --target-requests 1000 --seed 42

# Start GPU monitoring (in a separate terminal)
bash scripts/gpu_monitor.sh > outputs/<tag>/gpu_util.csv &

# Run replayer against proxy
python -m replayer \
    --trace traces/sampled_1000req_seed42.jsonl \
    --output outputs/<tag>/metrics.jsonl \
    --endpoint http://localhost:9090 \
    --time-scale 20 --max-inflight-sessions 8 \
    --request-limit 200 -v

# Collect proxy breakdown (elastic only)
curl -s http://localhost:9090/breakdown > outputs/<tag>/breakdown.json

# Collect APC from vLLM logs
for i in $(seq 0 7); do
    grep "Prefix cache hit rate\|External prefix cache hit rate" /tmp/<prefix>_$i.log | tail -2
done
```

## 3. Results

### 3.1 End-to-End Performance

| Config | OK/N | TTFT p50 | TTFT p90 | TPOT p50 | TPOT p90 | E2E p50 |
|--------|------|----------|----------|----------|----------|---------|
| Baseline (combined) | 198/200 | 2.383s | 27.622s | 0.069s | 0.117s | 10.232s |
| Elastic P2P (cap=4) | 185/196 | **1.315s** | **13.179s** | **0.066s** | **0.075s** | **5.708s** |
| **Delta** | | **-45%** | **-52%** | **-4%** | **-36%** | **-44%** |

### 3.2 KV Cache Hit Ratio

Sampled from vLLM instance logs at end of experiment:

**Baseline** (local prefix cache only):

| Instance | Prefix APC |
|----------|-----------|
| inst_0 | 48.6% |
| inst_3 | 3.8% |
| inst_7 | 68.3% |
| **Std dev** | **~33pp** |

**Elastic** (local prefix + Mooncake external):

| Instance | Prefix APC | External APC | Effective |
|----------|-----------|-------------|-----------|
| inst_0 | 37.8% | 31.6% | 69.4% |
| inst_3 | 36.6% | 34.2% | 70.8% |
| inst_7 | 25.0% | 0.0% | 25.0% |
| **Prefix std** | **~7pp** | | |

Key finding: elastic has **much more uniform** prefix APC across instances (std ~7pp vs ~33pp), and Mooncake external cache adds 30-34pp on active decode instances.

### 3.3 GPU Utilization

| Config | Mean | Min | Max | Imbalance |
|--------|------|-----|-----|-----------|
| Baseline | 28.7% | 20% | 38% | 1.9× |
| Elastic | 15.8% | 7.6% | 30.4% | 3.0× |

### 3.4 Success Rate

| Config | OK | Total | Rate | Failure mode |
|--------|-----|-------|------|-------------|
| Baseline | 198 | 200 | 99.0% | Generic timeout |
| Elastic | 185 | 196 | 94.4% | Mooncake transfer timeout on >60k requests |

### 3.5 Per-Class TTFT Breakdown (Baseline Combined)

| Class | Count | % | Input p50 | TTFT p50 | TTFT p90 |
|-------|-------|---|-----------|----------|----------|
| WARM (<5k) | 46 | 23% | 1,095 | 0.133s | 0.260s |
| MEDIUM (5-20k) | 50 | 25% | 10,879 | 0.873s | 1.808s |
| HEAVY (20-50k) | 64 | 32% | 34,368 | 2.589s | 6.302s |
| HEAVY (>50k) | 38 | 19% | 83,018 | 9.563s | 30.480s |

HEAVY requests (51% of traffic) dominate tail latency. Elastic offloads precisely these.

## 4. System-Level Analysis

### 4.1 Why Elastic Wins Despite Lower GPU Utilization

**Mechanism 1: Eliminating prefill-decode interference (TPOT -36%)**

In combined mode, vLLM chunked prefill interleaves prefill and decode. An 80k-token HEAVY prefill occupies the GPU for seconds, delaying co-resident decode. Elastic routes heavy prefill to a different instance, so the decode pipeline is uninterrupted.

Evidence: TPOT p90 drops from 0.117s (baseline) to 0.075s (elastic).

**Mechanism 2: Better effective cache utilization (TTFT -45%)**

Baseline APC is skewed (3.8%–68.3%) because heavy prefills evict other sessions' cached blocks. Elastic preserves D-instance prefix chains by offloading heavy prefills to P instances. Combined with Mooncake external cache, effective APC reaches ~70% on active instances vs ~40% baseline average.

**Mechanism 3: Faster KV cache turnover**

Lower GPU utilization (15.8% vs 28.7%) is not waste — it reflects that requests complete 44% faster. Less contention → decode finishes faster → KV cache freed sooner → next request starts faster. The same total work completes in 56% of the wall time.

### 4.2 Known Limitation: GPU Load Imbalance

Elastic has 3.0× imbalance (7.6% min vs 30.4% max) vs baseline's 1.9×.

Root causes:
1. **P-instance concentration**: Previous implementation always picked the globally least-loaded instance as P, concentrating P-role work on the same few idle instances.
2. **Session skew**: Some sessions have many turns with large inputs, keeping their pinned instance busy while others go idle.

**Implemented fix** (in latest `cache_aware_proxy.py`): Round-robin P-instance selection with overload skip, replacing `argmin(ongoing_tokens)`. Needs validation in next experiment cycle.

## 5. Data & Log Locations

### 5.1 Experiment Outputs (on respective machines)

| Directory | Machine | Config | Notes |
|-----------|---------|--------|-------|
| `outputs/ab_baseline/` | dash0 | Combined 8× TP=1 | Fair A/B baseline (§3) |
| `outputs/ab_elastic/` | dash1 | Elastic P2P cap=4 | Fair A/B elastic (§3) |
| `outputs/gpu_ab_combined/` | local | Combined 8× TP=1 | Earlier run, has gpu_util.csv |
| `outputs/gpu_ab_pdsep/` | local | PD-Sep 4P+4D | Earlier run, has gpu_util.csv |
| `outputs/exp2_combined_tp1_dp8/` | local | Combined 8× TP=1 | 1000 req, cache-aware |
| `outputs/exp3_pd_sep_tp1_mooncake/` | local | PD-Sep 4P+4D Mooncake | 1000 req |

### 5.2 vLLM Instance Logs

| Path pattern | Machine | Config |
|-------------|---------|--------|
| `/tmp/ab_base_$i.log` | dash0 | Baseline instances 0-7 |
| `/tmp/ab_elastic_$i.log` | dash1 | Elastic instances 0-7 |

Logs contain `Prefix cache hit rate` and `External prefix cache hit rate` lines for APC extraction.

### 5.3 Trace Data

| Path | Machine | Description |
|------|---------|-------------|
| `~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl` | dash0 | Full 2h production trace (2.1M requests) |
| `traces/sampled_1000req_seed42.jsonl` | all | Sampled 1000 requests (gitignored, regenerate with `sample_trace.py`) |

### 5.4 Analysis Documents

| File | Content |
|------|---------|
| `analysis/pd_separation_analysis.md` | Main report: PD-Sep vs Combined + Elastic P2P (§5) |
| `analysis/elastic_offload_design.md` | Elastic P2P design rationale |
| `analysis/kv_lifecycle_design.md` | KV cache eviction policy analysis |
| `analysis/adaptive_prefill_offload_design.md` | Initial adaptive offload design (superseded by elastic) |

## 6. Repository Structure

```
agentic-kv/
├── analysis/                    # Research reports and design docs
│   ├── pd_separation_analysis.md    # Main comprehensive report
│   ├── elastic_offload_design.md    # Elastic P2P design
│   ├── kv_lifecycle_design.md       # Cache eviction analysis
│   └── ...
├── replayer/                    # Trace replay framework
│   ├── __main__.py              # CLI entry: python -m replayer
│   ├── replay.py                # Async replayer (session-aware, SSE streaming)
│   ├── trace.py                 # TraceRequest dataclass, session/hash_id handling
│   └── metrics.py               # RequestMetrics, crash-safe JSONL sink
├── scripts/
│   ├── cache_aware_proxy.py     # Global scheduler (combined + PD-sep + elastic offload)
│   ├── sample_trace.py          # Cluster-to-machine trace sampler
│   ├── launch_vllm.sh           # Launch combined TP=8
│   ├── launch_pd_mooncake.sh    # Launch PD-Sep with Mooncake
│   ├── launch_elastic_p2p.sh    # Launch elastic P2P (8× kv_both + offload proxy)
│   ├── run_experiments.sh       # Full experiment matrix (combined/PD-sep)
│   ├── run_benchmark.sh         # Single benchmark run
│   ├── gpu_monitor.sh           # GPU utilization sampler (5s CSV)
│   ├── compute_roofline.py      # Prefill/decode roofline analysis
│   ├── analyze_*.py             # Various analysis scripts
│   └── compare_*.py             # Experiment comparison scripts
├── patches/
│   ├── 0001-fix-kv-transfer-abort-race.patch
│   └── README.md
├── third_party/vllm/            # vLLM 0.18.1 source (with patch applied)
├── outputs/                     # Experiment results (gitignored)
├── traces/                      # Sampled traces (gitignored)
├── TODO.md                      # Original research goals
└── REPORT.md                    # This milestone report
```

## 7. Key Scripts Reference

| Script | What it does | Key flags |
|--------|-------------|-----------|
| `scripts/cache_aware_proxy.py` | Global scheduler + elastic offload proxy | `--combined`, `--offload`, `--heavy-threshold`, `--bootstrap-ports` |
| `scripts/sample_trace.py` | Sample complete sessions from cluster trace | `--target-requests`, `--seed` |
| `python -m replayer` | Replay trace against vLLM endpoint | `--time-scale`, `--max-inflight-sessions`, `--request-limit` |
| `scripts/gpu_monitor.sh` | Sample nvidia-smi to CSV | Pipe to `outputs/<tag>/gpu_util.csv` |
| `scripts/launch_elastic_p2p.sh` | Launch all 8 kv_both instances + offload proxy | `HEAVY_THRESHOLD`, `MAX_OFFLOAD` env vars |

## 8. Conclusions & Next Steps

### Established findings:
1. Full PD separation is **net negative** for single-machine agentic workloads (KV cache memory wall)
2. Cache-aware session-sticky routing is the **dominant optimization** (+24pp APC, -60% TTFT)
3. Elastic P2P offload achieves **-45% TTFT, -36% TPOT, -44% E2E** by selectively isolating heavy prefills while preserving decode cache locality
4. The GPU utilization paradox (lower util but better performance) is explained by higher per-request efficiency

### Open problems:
1. GPU load imbalance (3.0× in elastic) — round-robin P fix implemented, needs validation
2. Elastic success rate (94.4%) — Mooncake transfer timeouts on >60k requests
3. Scaling to multi-machine (cross-node Mooncake transfers not yet tested)
4. Adaptive offload threshold (fixed 20k may not be optimal for all load levels)

---

*Generated from experiments run on 2026-05-22. Git commit: `1e86285` (A/B results) + subsequent proxy improvements.*