agentic-kvc/REPORT.md

Milestone Report: Elastic P2P vs PD-Combined Baseline

Date: 2026-05-22 Author: Gahow Wang Status: Phase 1 complete — baseline + elastic validated, system-level analysis done

---

1. Research Question

For agentic LLM workloads (long input, short output, high KV cache reuse), is prefill-decode disaggregation beneficial? If full PD separation hurts (proven in §3), can selective disaggregation of only heavy requests improve serving latency while preserving KV cache locality?

2. Experimental Setup

2.1 Hardware

Resource	Spec
Machine	dash0 / dash1 (identical config)
GPU	8× NVIDIA H20 96GB HBM, NVLink
Network	4× ConnectX-7 200Gbps RDMA
Storage	cpfs shared storage across machines

2.2 Software

Component	Version	Notes
vLLM	0.18.1 (source in third_party/vllm/)	Patched scheduler assert (see patches/)
Mooncake	0.3.10	RDMA-based KV transfer between instances
Python	3.x managed by uv	.venv/ at project root
Model	Qwen3-Coder-30B-A3B-Instruct	MoE 128 experts top-8, 3B active params
Model path	~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct	Same on dash0 and dash1

2.3 Workload Trace

Property	Value
Source	GLM-5.1 Agentic Coder, production cluster, 2h window
Raw trace	~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl on dash0
Total requests	2,114,220
Avg input tokens	33,600 (p50=20k, p90=88k)
Avg output tokens	445 (p50=80)
I/O ratio	75.6× aggregate
Prefill token share	98%
KV reuse (intra-session)	91% of reusable blocks
Theoretical max APC	71% (infinite cache, single instance)

Sampled trace for benchmarks: traces/sampled_1000req_seed42.jsonl (1000 requests, seed=42, preserving session structure). For 200-request ablations: replayer --request-limit 200.

2.4 Two Configurations Compared

Baseline: PD-Combined (8× TP=1 DP=8)

8 independent vLLM instances, 1 GPU each, no Mooncake.
All instances do both prefill and decode.
Global scheduler (cache_aware_proxy.py --combined) handles:
  - Session-sticky routing (multi-turn → same instance)
  - Load-aware override (if pinned instance > 2× avg load, redirect)
  - Cache-hit scoring (prefer instance with matching prefix blocks)

Launch:

# On dash0:
for i in $(seq 0 7); do
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \
    vllm serve ~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
        --port $((8000+i)) --tp 1 \
        --enable-prefix-caching --enforce-eager \
        --gpu-memory-utilization 0.9 --max-model-len 200000 \
        > /tmp/ab_base_$i.log 2>&1 &
done

python scripts/cache_aware_proxy.py \
    --combined http://127.0.0.1:800{0..7} --port 9090

Elastic P2P Offload (8× TP=1 kv_both + selective offload)

8 independent vLLM instances, 1 GPU each, all kv_role=kv_both (Mooncake).
Same global scheduler, plus elastic offload logic:
  - Proxy classifies each request: WARM (<5k new), MEDIUM (5-20k), HEAVY (>20k)
  - WARM/MEDIUM: co-located on session-sticky instance (no KV transfer)
  - HEAVY: prefill on a different instance (P), KV via Mooncake RDMA,
    decode on session-sticky instance (D)
  - Cap: max 4 concurrent offloads (MAX_OFFLOAD_INFLIGHT)
  - P instance selection: round-robin with overload skip

Launch:

# On dash1 (or use scripts/launch_elastic_p2p.sh):
for i in $(seq 0 7); do
    VLLM_MOONCAKE_BOOTSTRAP_PORT=$((8998+i)) \
    MASTER_PORT=$((29500+i)) CUDA_VISIBLE_DEVICES=$i \
    vllm serve ~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct \
        --port $((8000+i)) --tp 1 \
        --enable-prefix-caching --enforce-eager \
        --gpu-memory-utilization 0.9 --max-model-len 200000 \
        --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_both"}' \
        > /tmp/ab_elastic_$i.log 2>&1 &
    sleep 2  # stagger to avoid NCCL port collision
done

# Wait for bootstrap servers
for bp in $(seq 8998 9005); do
    until curl -s localhost:$bp/query > /dev/null 2>&1; do sleep 2; done
done

python scripts/cache_aware_proxy.py \
    --combined http://127.0.0.1:800{0..7} \
    --bootstrap-ports 8998,8999,9000,9001,9002,9003,9004,9005 \
    --offload --heavy-threshold 20000 --port 9090

2.5 Benchmark Parameters

Parameter	Value
Requests	200 (from sampled 1000-req trace, --request-limit 200)
Time scale	20× (compress 2h trace into ~6min)
Max inflight sessions	8
Request timeout	600s
vLLM flags	--enforce-eager --enable-prefix-caching --max-model-len 200000
GPU memory util	0.9
Fresh restart	Both configs started from cold (no warm cache)

2.6 Reproducing the Benchmark

# Activate environment
cd ~/agentic-kv && source .venv/bin/activate

# Ensure sampled trace exists
python scripts/sample_trace.py \
    --input ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
    --output traces/sampled_1000req_seed42.jsonl \
    --target-requests 1000 --seed 42

# Start GPU monitoring (in a separate terminal)
bash scripts/gpu_monitor.sh > outputs/<tag>/gpu_util.csv &

# Run replayer against proxy
python -m replayer \
    --trace traces/sampled_1000req_seed42.jsonl \
    --output outputs/<tag>/metrics.jsonl \
    --endpoint http://localhost:9090 \
    --time-scale 20 --max-inflight-sessions 8 \
    --request-limit 200 -v

# Collect proxy breakdown (elastic only)
curl -s http://localhost:9090/breakdown > outputs/<tag>/breakdown.json

# Collect APC from vLLM logs
for i in $(seq 0 7); do
    grep "Prefix cache hit rate\|External prefix cache hit rate" /tmp/<prefix>_$i.log | tail -2
done

3. Results

3.1 End-to-End Performance

Config	OK/N	TTFT p50	TTFT p90	TPOT p50	TPOT p90	E2E p50
Baseline linear	198/200	2.383s	27.622s	0.069s	0.117s	10.232s
Baseline LMetric	198/200	1.099s	9.392s	0.063s	0.073s	5.205s
Elastic P2P (cap=4)	185/196	1.315s	13.179s	0.066s	0.075s	5.708s

Note: "Baseline linear" was run on dash0 during the initial A/B (different machine load conditions). "Baseline LMetric" was run on fresh-restart dash0, same conditions as "Baseline linear (fresh)" below in §3.6.

3.2 KV Cache Hit Ratio

Sampled from vLLM instance logs at end of experiment:

Baseline (local prefix cache only):

Instance	Prefix APC
inst_0	48.6%
inst_3	3.8%
inst_7	68.3%
Std dev	~33pp

Elastic (local prefix + Mooncake external):

Instance	Prefix APC	External APC	Effective
inst_0	37.8%	31.6%	69.4%
inst_3	36.6%	34.2%	70.8%
inst_7	25.0%	0.0%	25.0%
Prefix std	~7pp

Key finding: elastic has much more uniform prefix APC across instances (std ~7pp vs ~33pp), and Mooncake external cache adds 30-34pp on active decode instances.

3.3 GPU Utilization

Config	Mean	Min	Max	Imbalance
Baseline	28.7%	20%	38%	1.9×
Elastic	15.8%	7.6%	30.4%	3.0×

3.4 Success Rate

Config	OK	Total	Rate	Failure mode
Baseline	198	200	99.0%	Generic timeout
Elastic	185	196	94.4%	Mooncake transfer timeout on >60k requests

3.5 Per-Class TTFT Breakdown (Baseline Combined)

Class	Count	%	Input p50	TTFT p50	TTFT p90
WARM (<5k)	46	23%	1,095	0.133s	0.260s
MEDIUM (5-20k)	50	25%	10,879	0.873s	1.808s
HEAVY (20-50k)	64	32%	34,368	2.589s	6.302s
HEAVY (>50k)	38	19%	83,018	9.563s	30.480s

HEAVY requests (51% of traffic) dominate tail latency. Elastic offloads precisely these.

3.6 Routing Policy Comparison: Linear vs LMetric (OSDI'26)

LMetric (Zhang et al., OSDI'26) replaces linear combination score = load - α·cache_hit with hyperparameter-free multiplication score = P_tokens × BS:

• P_tokens = pending prefill tokens on instance + new request's uncached tokens
• BS = batch size (waiting + running request count) + 1

Both experiments: 8× TP=1 fresh-restart instances on dash0, same trace (200 req, time_scale=20).

Policy	OK/N	TTFT p50	TTFT p90	TPOT p90	E2E p50
Linear	198/200	1.086s	9.432s	0.0773s	5.423s
LMetric	198/200	1.099s	9.392s	0.0727s	5.205s
Delta		+1.2%	-0.4%	-5.9%	-4.0%

Per-class breakdown:

Class	Linear TTFT p50	LMetric TTFT p50	Linear TPOT p90	LMetric TPOT p90
WARM (<5k, n=46)	0.143s	0.134s	0.058s	0.061s
MEDIUM (5-20k, n=50)	0.921s	0.809s	0.078s	0.073s
HEAVY (>20k, n=102)	4.875s	4.943s	0.078s	0.074s

APC comparison (prefix cache hit rate per instance):

	Linear	LMetric
Mean	32.5%	30.8%
Std	~22pp	~19pp
Range	3.3%–63.3%	4.9%–67.2%

Analysis: LMetric provides modest improvements in TPOT (-5.9%) and E2E (-4.0%) through better load balancing (the multiplication naturally penalizes overloaded instances). TTFT is unchanged because HEAVY requests dominate and session affinity constrains routing freedom. APC skew is slightly reduced. The improvement is far smaller than elastic P2P offload (-44% E2E), confirming that for agentic workloads, the bottleneck is prefill-decode interference, not routing policy.

Data: outputs/ab_linear/ and outputs/ab_lmetric/ on dash0. Logs: /tmp/lmetric_ab_inst_*.log (linear) and /tmp/lmetric_inst_*.log (LMetric).

4. System-Level Analysis

4.1 Why Elastic Wins Despite Lower GPU Utilization

Mechanism 1: Eliminating prefill-decode interference (TPOT -36%)

In combined mode, vLLM chunked prefill interleaves prefill and decode. An 80k-token HEAVY prefill occupies the GPU for seconds, delaying co-resident decode. Elastic routes heavy prefill to a different instance, so the decode pipeline is uninterrupted.

Evidence: TPOT p90 drops from 0.117s (baseline) to 0.075s (elastic).

Mechanism 2: Better effective cache utilization (TTFT -45%)

Baseline APC is skewed (3.8%–68.3%) because heavy prefills evict other sessions' cached blocks. Elastic preserves D-instance prefix chains by offloading heavy prefills to P instances. Combined with Mooncake external cache, effective APC reaches ~70% on active instances vs ~40% baseline average.

Mechanism 3: Faster KV cache turnover

Lower GPU utilization (15.8% vs 28.7%) is not waste — it reflects that requests complete 44% faster. Less contention → decode finishes faster → KV cache freed sooner → next request starts faster. The same total work completes in 56% of the wall time.

4.2 Known Limitation: GPU Load Imbalance

Elastic has 3.0× imbalance (7.6% min vs 30.4% max) vs baseline's 1.9×.

Root causes:

1. P-instance concentration: Previous implementation always picked the globally least-loaded instance as P, concentrating P-role work on the same few idle instances.
2. Session skew: Some sessions have many turns with large inputs, keeping their pinned instance busy while others go idle.

Implemented fix (in latest cache_aware_proxy.py): Round-robin P-instance selection with overload skip, replacing argmin(ongoing_tokens). Needs validation in next experiment cycle.

5. Data & Log Locations

5.1 Experiment Outputs (on respective machines)

Directory	Machine	Config	Notes
outputs/ab_baseline/	dash0	Combined 8× TP=1	Fair A/B baseline (§3)
outputs/ab_elastic/	dash1	Elastic P2P cap=4	Fair A/B elastic (§3)
outputs/gpu_ab_combined/	local	Combined 8× TP=1	Earlier run, has gpu_util.csv
outputs/gpu_ab_pdsep/	local	PD-Sep 4P+4D	Earlier run, has gpu_util.csv
outputs/ab_linear/	dash0	Linear policy, 200 req	§3.6 routing policy comparison
outputs/ab_lmetric/	dash0	LMetric policy, 200 req	§3.6 routing policy comparison
outputs/exp2_combined_tp1_dp8/	local	Combined 8× TP=1	1000 req, cache-aware
outputs/exp3_pd_sep_tp1_mooncake/	local	PD-Sep 4P+4D Mooncake	1000 req

5.2 vLLM Instance Logs

Path pattern	Machine	Config
/tmp/ab_base_$i.log	dash0	Baseline instances 0-7
/tmp/ab_elastic_$i.log	dash1	Elastic instances 0-7
/tmp/lmetric_ab_inst_$i.log	dash0	Linear policy instances 0-7 (§3.6)
/tmp/lmetric_inst_$i.log	dash0	LMetric policy instances 0-7 (§3.6)

Logs contain Prefix cache hit rate and External prefix cache hit rate lines for APC extraction.

5.3 Trace Data

Path	Machine	Description
~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl	dash0	Full 2h production trace (2.1M requests)
traces/sampled_1000req_seed42.jsonl	all	Sampled 1000 requests (gitignored, regenerate with sample_trace.py)

5.4 Analysis Documents

File	Content
analysis/pd_separation_analysis.md	Main report: PD-Sep vs Combined + Elastic P2P (§5)
analysis/elastic_offload_design.md	Elastic P2P design rationale
analysis/kv_lifecycle_design.md	KV cache eviction policy analysis
analysis/adaptive_prefill_offload_design.md	Initial adaptive offload design (superseded by elastic)

6. Repository Structure

agentic-kv/
├── analysis/                    # Research reports and design docs
│   ├── pd_separation_analysis.md    # Main comprehensive report
│   ├── elastic_offload_design.md    # Elastic P2P design
│   ├── kv_lifecycle_design.md       # Cache eviction analysis
│   └── ...
├── replayer/                    # Trace replay framework
│   ├── __main__.py              # CLI entry: python -m replayer
│   ├── replay.py                # Async replayer (session-aware, SSE streaming)
│   ├── trace.py                 # TraceRequest dataclass, session/hash_id handling
│   └── metrics.py               # RequestMetrics, crash-safe JSONL sink
├── scripts/
│   ├── cache_aware_proxy.py     # Global scheduler (combined + PD-sep + elastic offload)
│   ├── sample_trace.py          # Cluster-to-machine trace sampler
│   ├── launch_vllm.sh           # Launch combined TP=8
│   ├── launch_pd_mooncake.sh    # Launch PD-Sep with Mooncake
│   ├── launch_elastic_p2p.sh    # Launch elastic P2P (8× kv_both + offload proxy)
│   ├── run_experiments.sh       # Full experiment matrix (combined/PD-sep)
│   ├── run_benchmark.sh         # Single benchmark run
│   ├── gpu_monitor.sh           # GPU utilization sampler (5s CSV)
│   ├── compute_roofline.py      # Prefill/decode roofline analysis
│   ├── analyze_*.py             # Various analysis scripts
│   └── compare_*.py             # Experiment comparison scripts
├── patches/
│   ├── 0001-fix-kv-transfer-abort-race.patch
│   └── README.md
├── third_party/vllm/            # vLLM 0.18.1 source (with patch applied)
├── outputs/                     # Experiment results (gitignored)
├── traces/                      # Sampled traces (gitignored)
├── TODO.md                      # Original research goals
└── REPORT.md                    # This milestone report

7. Key Scripts Reference

Script	What it does	Key flags
scripts/cache_aware_proxy.py	Global scheduler + elastic offload proxy	--combined, --offload, --policy {linear,lmetric}, --heavy-threshold, --bootstrap-ports
scripts/run_lmetric_ab.sh	A/B: linear vs lmetric routing policy	Runs both experiments with fresh restart
scripts/sample_trace.py	Sample complete sessions from cluster trace	--target-requests, --seed
python -m replayer	Replay trace against vLLM endpoint	--time-scale, --max-inflight-sessions, --request-limit
scripts/gpu_monitor.sh	Sample nvidia-smi to CSV	Pipe to outputs/<tag>/gpu_util.csv
scripts/launch_elastic_p2p.sh	Launch all 8 kv_both instances + offload proxy	HEAVY_THRESHOLD, MAX_OFFLOAD env vars

8. Conclusions & Next Steps

Established findings:

1. Full PD separation is net negative for single-machine agentic workloads (KV cache memory wall)
2. Cache-aware session-sticky routing is the dominant optimization (+24pp APC, -60% TTFT)
3. Elastic P2P offload achieves -45% TTFT, -36% TPOT, -44% E2E by selectively isolating heavy prefills while preserving decode cache locality
4. The GPU utilization paradox (lower util but better performance) is explained by higher per-request efficiency
5. LMetric (OSDI'26) multiplication-based routing provides modest improvement over linear (E2E -4%, TPOT -6%), confirming that routing policy alone has limited headroom — the bottleneck is prefill-decode interference

Open problems:

1. GPU load imbalance (3.0× in elastic) — round-robin P fix implemented, needs validation
2. Elastic success rate (94.4%) — Mooncake transfer timeouts on >60k requests
3. Scaling to multi-machine (cross-node Mooncake transfers not yet tested)
4. Adaptive offload threshold (fixed 20k may not be optimal for all load levels)
5. Router state accuracy: proxy shadow state vs vLLM-internal exact state (TODO: vLLM → Redis → router pipeline for ablation)

---

Generated from experiments run on 2026-05-22. Git commits: 1e86285 (elastic A/B), 2b0ac70 (phase 1 milestone), subsequent LMetric implementation.