agentic-pd-hybrid/docs/E1_E2_RESULTS_ZH.md

E1 vs E2 Experiment Results — H200 + Driver 570

Status: E1 ✅ complete (2026-05-12 01:48 UTC, wall 1h29min). E2 ⏳ running. Branch: h200-cu130. Trace: outputs/inferact_50sess.jsonl (deterministic head-cut of Inferact codex_swebenchpro to first 50 trials, md5 7bb263a32600ef5a6ef5099ba340a487, 1285 requests, mean input_length 67,631 tokens). Hardware: 4× H200 80GB, driver 570.86.15 (cu12.8 API), Mellanox mlx5_60 RoCE 400 Gb/s NDR. Model: Qwen3-30B-A3B-Instruct-2507 (TP1). Toolchain: vendored SGLang 0.5.10 + cu12.8 nvcc local install (~/cuda-12.8) — see docs/H200_DRIVER570_SETUP_ZH.md.

---

1. Hypotheses being tested

From docs/ONBOARDING_NEXT_AGENT_ZH.md §3.1:

• H1: KVC v2's wins are not just from "1P3D topology + kv-aware policy" — the KVC layer (admission / migration / direct-to-D) contributes meaningfully on top. Pairing E1 (no KVC layer) against E2 (full KVC v2) on the same subset isolates the marginal contribution.
• H2/H3: Enabling real RDMA pushes TTFT p99 down from the reported 1.28s (TCP loopback) toward ~0.7s. Independent of H1, this is measured inside E2 alone (comparing against the historical TCP-loopback v2 reference).

---

2. E1 results — naive 1P3D + kv-aware + RDMA

Configuration: mechanism=pd-disaggregation, policy=kv-aware, 1P3D (GPU0=P, GPU1/2/3=D), --force-rdma --ib-device mlx5_60, --concurrency-limit 32, ts=1.

Metric	E1
request_count	1285
success	1200
error_count	85
failure_count	85
abort_count	0
latency mean	96.34 s
latency p50	93.21 s
latency p90	180.69 s
latency p99	219.46 s
ttft mean	90.48 s
ttft p50	88.62 s
ttft p90	175.13 s
ttft p99	207.39 s
execution_modes	pd-disaggregation-router: 1200, pd-disaggregation: 85 (errors)
per_decode_load	D0:575, D1:710, D2:0
per_prefill_load	P0:1285
cache_hit_request_count	1199 / 1200 (99.9%)

Key observations on E1

1. D2 was never bound to a single session. All 50 sessions got pinned to D0 or D1 by kv-aware policy's (overlap + sticky + inflight + assigned) lex-score, and naive pd-disaggregation has no migration mechanism to rebalance. Effective topology was 1P2D, not 1P3D.
2. Massive queueing. TTFT p50 ≈ 89 s and p99 > 200 s indicate sessions waited tens of seconds in router/prefill queue. With --concurrency-limit 32 and D0/D1 saturated, the inflight cap forced ~1250 reqs to serialize through only two decode workers.
3. 85 failures (6.6%) — all execution_mode == pd-disaggregation (which the metrics module classifies as error when the agentic-pd-hybrid replay sees an unsuccessful upstream response). Most likely caused by --request-timeout-s 300 firing on the longest queued requests.
4. Cache hit 99.9% — the kv-aware policy did successfully concentrate sessions on their prior D worker; the Inferact converter's prefix-shared 24-token-block hash_ids gave near-perfect prefix overlap across turns of the same session.

What E1 establishes

For the same hardware, same trace, same model, naive 1P3D + kv-aware policy is unusable for multi-session agentic workloads:

• session-stickiness without migration leaves a third of compute capacity (1 of 3 decode GPUs) entirely unused
• queueing dominates user-facing latency
• failure rate is 6.6% even with 5 minutes per-request timeout

This is the baseline H1 needs — it shows the KVC layer (E2) has something concrete to improve over.

---

3. E2 — in progress + an unexpected finding about D2

Background task b0im1d48q, launched 2026-05-12 01:48 UTC. Mid-run snapshot at 16 minutes (33 % POSTs dispatched):

	D0	D1	D2
bindings so far	248	267	0
GPU util (snapshot)	0 %	0 %	0 %
KV pool util (across run)	high	high	empty

D2 receives zero traffic in E2 too, just like E1. This is not the result we expected — H1 predicted that KVC's session-migration mechanism (reset-on-success blacklist with migration_reject_threshold=3) would route around the imbalance E1 showed. It doesn't.

Root cause

KvAwarePolicy.select (policies.py:171-202) scores candidates by 4-tuple lex order (overlap + α·sticky, sticky, -inflight, -assigned). The overlap term dominates: any D that has resident KV blocks matching the incoming request's hash_ids wins position 0.

In the Inferact codex_swebenchpro workload, all 50 sessions begin with identical "permissions instructions" boilerplate (the converter sees this as identical first-block content across trial 0..49). Our hash_id construction (sha256 over the token sequence per 24-token block, see scripts/convert_inferact_to_trace.py) therefore yields identical block hashes across sessions for the first ~50 blocks.

Concretely, when session N's turn 0 lands:

• D0 / D1 already host previous sessions → their state.resident sets include those shared boilerplate hashes → overlap > 0
• D2 has never been admitted → state.resident[D2] is empty → overlap = 0
• D0/D1 tie at position 0; D2 always loses

The migration mechanism never triggers because D0/D1 have ample KV (peak token_usage ~0.86 in v2 historical reports) and never reject admission. No rejects → no (session, D) blacklist accumulation → no migration → D2 stays cold forever.

Implication for H1

H1 is not falsified, but it is qualified: KVC v2 still improves over naive pd-disaggregation on per-request work (direct-to-D fast path skips P→D mooncake transfer for turn≥1 on the same D), but it does not automatically balance load across D workers when the workload has high cross-session prefix overlap. To realise the full theoretical benefit of 1P3D on this workload, the policy needs an explicit cold-D bonus, or a pre-warming step that seeds D2 with shared boilerplate at startup.

Full E2 metrics will be filled in upon completion (ETA ~22 min from snapshot).

---

4. Comparison table — pending

To be appended.

---

5. Open questions for the next iteration

• Are the 85 E1 errors all timeouts? request-metrics.jsonl rows with error execution_mode should be sampled to confirm. (Quick check: grep the metrics jsonl for "execution_mode": "pd-disaggregation" and inspect latency_s / error fields.)
• Does E2 produce the predicted ~91% direct-to-D rate seen in the historical SWE-Bench v2 run, or does the Inferact workload's larger session count (50 vs 52 there) but very different per-session size distribution (mean 33 turns × ~2KB context growth per turn) push it lower?
• Is D2 = 0% an E1-specific artifact (kv-aware sticky in pd-disagg mode), or does the same happen in E2 before migration kicks in for the first time?