agentic-pd-hybrid/docs/E1_E2_RESULTS_ZH.md

E1 vs E2 Experiment Results — H200 + Driver 570

Status: E1 ✅ complete (2026-05-12 01:48 UTC, wall 1h29min). E2 ⏳ running. Branch: h200-cu130. Trace: outputs/inferact_50sess.jsonl (deterministic head-cut of Inferact codex_swebenchpro to first 50 trials, md5 7bb263a32600ef5a6ef5099ba340a487, 1285 requests, mean input_length 67,631 tokens). Hardware: 4× H200 80GB, driver 570.86.15 (cu12.8 API), Mellanox mlx5_60 RoCE 400 Gb/s NDR. Model: Qwen3-30B-A3B-Instruct-2507 (TP1). Toolchain: vendored SGLang 0.5.10 + cu12.8 nvcc local install (~/cuda-12.8) — see docs/H200_DRIVER570_SETUP_ZH.md.

---

1. Hypotheses being tested

From docs/ONBOARDING_NEXT_AGENT_ZH.md §3.1:

• H1: KVC v2's wins are not just from "1P3D topology + kv-aware policy" — the KVC layer (admission / migration / direct-to-D) contributes meaningfully on top. Pairing E1 (no KVC layer) against E2 (full KVC v2) on the same subset isolates the marginal contribution.
• H2/H3: Enabling real RDMA pushes TTFT p99 down from the reported 1.28s (TCP loopback) toward ~0.7s. Independent of H1, this is measured inside E2 alone (comparing against the historical TCP-loopback v2 reference).

---

2. E1 results — naive 1P3D + kv-aware + RDMA

Configuration: mechanism=pd-disaggregation, policy=kv-aware, 1P3D (GPU0=P, GPU1/2/3=D), --force-rdma --ib-device mlx5_60, --concurrency-limit 32, ts=1.

Metric	E1
request_count	1285
success	1200
error_count	85
failure_count	85
abort_count	0
latency mean	96.34 s
latency p50	93.21 s
latency p90	180.69 s
latency p99	219.46 s
ttft mean	90.48 s
ttft p50	88.62 s
ttft p90	175.13 s
ttft p99	207.39 s
execution_modes	pd-disaggregation-router: 1200, pd-disaggregation: 85 (errors)
per_decode_load	D0:575, D1:710, D2:0
per_prefill_load	P0:1285
cache_hit_request_count	1199 / 1200 (99.9%)

Key observations on E1

1. D2 was never bound to a single session. All 50 sessions got pinned to D0 or D1 by kv-aware policy's (overlap + sticky + inflight + assigned) lex-score, and naive pd-disaggregation has no migration mechanism to rebalance. Effective topology was 1P2D, not 1P3D.
2. Massive queueing. TTFT p50 ≈ 89 s and p99 > 200 s indicate sessions waited tens of seconds in router/prefill queue. With --concurrency-limit 32 and D0/D1 saturated, the inflight cap forced ~1250 reqs to serialize through only two decode workers.
3. 85 failures (6.6%) — all execution_mode == pd-disaggregation (which the metrics module classifies as error when the agentic-pd-hybrid replay sees an unsuccessful upstream response). Most likely caused by --request-timeout-s 300 firing on the longest queued requests.
4. Cache hit 99.9% — the kv-aware policy did successfully concentrate sessions on their prior D worker; the Inferact converter's prefix-shared 24-token-block hash_ids gave near-perfect prefix overlap across turns of the same session.

What E1 establishes

For the same hardware, same trace, same model, naive 1P3D + kv-aware policy is unusable for multi-session agentic workloads:

• session-stickiness without migration leaves a third of compute capacity (1 of 3 decode GPUs) entirely unused
• queueing dominates user-facing latency
• failure rate is 6.6% even with 5 minutes per-request timeout

This is the baseline H1 needs — it shows the KVC layer (E2) has something concrete to improve over.

---

3. E2 results — pending

Background task b0im1d48q, launched 2026-05-12 01:48 UTC. Same subset, full KVC v2 stack (reset-on-success migration, direct-append threshold 8192), RDMA on, all other knobs identical to E1.

Expected differences:

• Direct-to-D fast path engaged for turn≥1 requests → fewer P/D round-trips
• Migration triggered when sessions hit D0/D1 saturation → D2 should see traffic
• Lower TTFT p50 / latency mean
• TTFT p99 still constrained by reseed slow-path (P re-prefill + mooncake transfer)

Will be filled in upon completion.

---

4. Comparison table — pending

To be appended.

---

5. Open questions for the next iteration

• Are the 85 E1 errors all timeouts? request-metrics.jsonl rows with error execution_mode should be sampled to confirm. (Quick check: grep the metrics jsonl for "execution_mode": "pd-disaggregation" and inspect latency_s / error fields.)
• Does E2 produce the predicted ~91% direct-to-D rate seen in the historical SWE-Bench v2 run, or does the Inferact workload's larger session count (50 vs 52 there) but very different per-session size distribution (mean 33 turns × ~2KB context growth per turn) push it lower?
• Is D2 = 0% an E1-specific artifact (kv-aware sticky in pd-disagg mode), or does the same happen in E2 before migration kicks in for the first time?