agentic-pd-hybrid/docs/E1_E2_RESULTS_ZH.md

E1 vs E2 Experiment Results — H200 + Driver 570

Status: E1 ✅ complete (2026-05-12 01:48 UTC, wall 1h29min). E2 ✅ complete (2026-05-12 03:22 UTC, wall 1h33min). Branch: h200-cu130. Trace: outputs/inferact_50sess.jsonl (deterministic head-cut of Inferact codex_swebenchpro to first 50 trials, md5 7bb263a32600ef5a6ef5099ba340a487, 1285 requests, mean input_length 67,631 tokens). Hardware: 4× H200 80GB, driver 570.86.15 (cu12.8 API), Mellanox mlx5_60 RoCE 400 Gb/s NDR. Model: Qwen3-30B-A3B-Instruct-2507 (TP1). Toolchain: vendored SGLang 0.5.10 + cu12.8 nvcc local install (~/cuda-12.8) — see docs/H200_DRIVER570_SETUP_ZH.md.

---

1. Hypotheses being tested

From docs/ONBOARDING_NEXT_AGENT_ZH.md §3.1:

• H1: KVC v2's wins are not just from "1P3D topology + kv-aware policy" — the KVC layer (admission / migration / direct-to-D) contributes meaningfully on top. Pairing E1 (no KVC layer) against E2 (full KVC v2) on the same subset isolates the marginal contribution.
• H2/H3: Enabling real RDMA pushes TTFT p99 down from the reported 1.28s (TCP loopback) toward ~0.7s. Independent of H1, this is measured inside E2 alone (comparing against the historical TCP-loopback v2 reference).

---

2. E1 results — naive 1P3D + kv-aware + RDMA

Configuration: mechanism=pd-disaggregation, policy=kv-aware, 1P3D (GPU0=P, GPU1/2/3=D), --force-rdma --ib-device mlx5_60, --concurrency-limit 32, ts=1.

Metric	E1
request_count	1285
success	1200
error_count	85
failure_count	85
abort_count	0
latency mean	96.34 s
latency p50	93.21 s
latency p90	180.69 s
latency p99	219.46 s
ttft mean	90.48 s
ttft p50	88.62 s
ttft p90	175.13 s
ttft p99	207.39 s
execution_modes	pd-disaggregation-router: 1200, pd-disaggregation: 85 (errors)
per_decode_load	D0:575, D1:710, D2:0
per_prefill_load	P0:1285
cache_hit_request_count	1199 / 1200 (99.9%)

Key observations on E1

1. D2 was never bound to a single session. All 50 sessions got pinned to D0 or D1 by kv-aware policy's (overlap + sticky + inflight + assigned) lex-score, and naive pd-disaggregation has no migration mechanism to rebalance. Effective topology was 1P2D, not 1P3D.
2. Massive queueing. TTFT p50 ≈ 89 s and p99 > 200 s indicate sessions waited tens of seconds in router/prefill queue. With --concurrency-limit 32 and D0/D1 saturated, the inflight cap forced ~1250 reqs to serialize through only two decode workers.
3. 85 failures (6.6%) — all execution_mode == pd-disaggregation (which the metrics module classifies as error when the agentic-pd-hybrid replay sees an unsuccessful upstream response). Most likely caused by --request-timeout-s 300 firing on the longest queued requests.
4. Cache hit 99.9% — the kv-aware policy did successfully concentrate sessions on their prior D worker; the Inferact converter's prefix-shared 24-token-block hash_ids gave near-perfect prefix overlap across turns of the same session.

What E1 establishes

For the same hardware, same trace, same model, naive 1P3D + kv-aware policy is unusable for multi-session agentic workloads:

• session-stickiness without migration leaves a third of compute capacity (1 of 3 decode GPUs) entirely unused
• queueing dominates user-facing latency
• failure rate is 6.6% even with 5 minutes per-request timeout

This is the baseline H1 needs — it shows the KVC layer (E2) has something concrete to improve over.

---

3. E2 results — KVC v2 + RDMA

Configuration: mechanism=kvcache-centric, policy=kv-aware, 1P3D, --force-rdma --ib-device mlx5_60, --kvcache-admission-mode worker, --kvcache-direct-max-uncached-tokens 8192, --kvcache-migration-reject-threshold 3, --kvcache-prefill-backup-policy release-after-transfer, --kvcache-prefill-priority-eviction, ts=1.

Metric	E2
request_count	1285
success	231
error_count	1054
failure_count	1054
abort_count	0
latency mean (successful only)	10.94 s
latency p50	7.44 s
latency p90	20.68 s
latency p99	64.73 s
ttft mean (successful only)	1.76 s
ttft p50	0.43 s
ttft p90	6.56 s
ttft p99	8.74 s
execution_modes (succ.)	direct-to-D: 87; turn1-seed: 50; reseed: 12; large-append-reseed: 11; seed-filter-early-turn: 50; large-append-cap: 21
per_decode_load	D0:600, D1:685, D2:0
per_prefill_load	P0:1285
cache_hit_request_count	230 / 231 (99.6 %)

Key observations on E2

1. D2 still has zero bindings — same root cause as E1. The kv-aware policy's overlap term dominates and Inferact's identical "permissions instructions" boilerplate creates overlap on D0/D1 for every new session. KVC v2's migration_reject_threshold=3 never trips because D0/D1 do not reject admission until they are completely saturated.
2. 80 % failure rate, 1054 / 1285. The 1054 reqs classified as bare kvcache-centric execution_mode are upstream timeouts / failures: they exceeded --request-timeout-s 300 while waiting for admission. v2's stricter admission path (direct-to-D requires both session resident on D and append_len ≤ τ_append AND capacity) rejects more often than E1's vanilla pd-disagg; the rejects don't trigger migration (see §3 root cause), they cause the request to fall through to fallback-seed / fallback-no-d-capacity which queue on already-saturated D0/D1, hit timeout, and fail.
3. Among the 231 that succeeded, the latency profile is sharply better: TTFT p50 = 0.43 s vs E1's 88.62 s (E2/E1 = 0.5 %), latency p50 = 7.44 s vs E1's 93.21 s (8 %). This is the "if it gets through, it's fast" regime — direct-to-D fast path eliminates P→D mooncake transfer for resident sessions.
4. Direct-to-D fast path engaged 87 / 231 = 37.7 % of successful requests. Lower than historical v2's 91.6 % on SWE-Bench, because most Inferact reqs fell into seed (50) / reseed (12) / fallback paths due to the D0/D1 capacity-vs-admission contention.

---

4. Comparison table — E1 vs E2

Numbers below are over all 1285 requests for E1 (since failure rate is small) but only the 231 successful for E2 (since the bulk timed out before producing latency datapoints). This is not a fair head-to-head, see §6.

Metric	E1	E2 (succ only)	E2 / E1
Total reqs	1285	1285	–
Successful	1200	231	0.19×
error_count	85 (6.6 %)	1054 (82 %)	12.4× worse
lat mean	96.34 s	10.94 s	0.114
lat p50	93.21 s	7.44 s	0.080
lat p90	180.69 s	20.68 s	0.114
lat p99	219.46 s	64.73 s	0.295
ttft mean	90.48 s	1.76 s	0.019
ttft p50	88.62 s	0.43 s	0.005
ttft p90	175.13 s	6.56 s	0.037
ttft p99	207.39 s	8.74 s	0.042
per_decode_load	D0:575, D1:710, D2:0	D0:600, D1:685, D2:0	both 1P2D
direct-to-D %	N/A (no KVC)	87/231 = 37.7 %	–

---

5. Interpreting H1 / H2 / H3

H1 (was: KVC layer adds value on top of 1P3D + kv-aware) — qualified

The H1 hypothesis as stated in ONBOARDING_NEXT_AGENT_ZH.md predicted E2 would clearly win on most metrics. The reality is bimodal: the small subset of E2 requests that successfully complete are dramatically faster than E1, but a much larger fraction (82 %) of E2 requests time out entirely. Net throughput on this workload is worse for E2 than E1.

Two issues drove this:

1. The D2 cold-start pathology already documented in §3, root cause. Both runs are de facto 1P2D, not 1P3D.
2. KVC v2's admission gate is stricter and surfaces more "no D capacity" / "session-not-resident" failures than vanilla pd-disagg, when the workload (mean input 67 K tokens, mean output 700 tokens) saturates D0/D1's combined ~1.5 M KV pool.

For workloads where D0/D1 do not saturate or where the policy does spread session ownership across all D workers (the historical SWE-Bench setup), KVC v2 wins. The Inferact codex_swebenchpro subset breaks both assumptions.

H2 / H3 (RDMA reduces TTFT p99) — cannot be evaluated cleanly here

The historical reference point is "KVC v2 + TCP loopback, SWE-Bench 50sess: TTFT p99 = 1.28 s". This run uses Inferact + RDMA, and TTFT p99 of the 231 successful E2 requests is 8.74 s — much higher than the TCP baseline. But the workloads are not comparable: Inferact mean input is 67 K tokens vs SWE-Bench's much smaller average. Per-request prefill + transfer is roughly 5× longer here. A clean H2 / H3 read needs an Inferact-on-TCP run to compare against, which is out of scope for this subset's GPU budget.

What we can say: RDMA is correctly engaged (every worker log shows installTransport, type=rdma; admission RPC RTTs in structural/admission-events.jsonl are ~6 ms — consistent with one-hop RoCE).

---

6. What this experiment actually shows

1. The H200 + driver 570 + cu12.8 toolchain works for production-scale SGLang xPyD workloads. Both runs completed without CUDA / driver / mooncake errors; failures are policy- and workload-level, not infrastructure.
2. The KVC v2 + kv-aware policy combination has a latent pathology on workloads with high cross-session prefix overlap: the overlap term in the lex score causes permanent load imbalance, and v2's reject-counter migration cannot rescue it because rejects only fire under capacity pressure, by which point timeouts already dominate. This is novel and not surfaced by the SWE-Bench evaluation in the existing project docs.
3. For Inferact-like workloads, a cold-D bonus (e.g. require D to host at least one session before its overlap score counts) or an explicit pre-warm step is required before E1/E2 comparisons can isolate the marginal effect of the KVC layer.

---

7. Reproducibility

• Trace: outputs/inferact_50sess.jsonl, md5 7bb263a32600ef5a6ef5099ba340a487, regenerable via scripts/sample_trace_subset.py.
• E1: bash scripts/sweep_e1_naive_1p3d.sh (1h 29 min wall)
• E2: bash scripts/sweep_e2_kvc_v2_rdma.sh (1h 33 min wall)
• Summary JSON paths:
  • outputs/e1_naive_1p3d_kvaware_rdma_50sess/e1_naive_1p3d_kvaware_run1_summary.json
  • outputs/e2_kvc_v2_rdma_50sess/e2_kvc_v2_rdma_run1_summary.json
• Per-request metrics JSONL alongside each summary, plus structural events under */structural/.

---

8. Open follow-ups for the next agent

1. Add a cold-D bonus to KvAwarePolicy.select (e.g. positive constant for D with state.resident[D] == ∅) and re-run E2 on the same subset. Predict: D2 receives bindings, failure rate drops, head-to-head with E1 becomes meaningful.
2. Rerun E2 with --kvcache-admission-mode router (router-side optimistic admission instead of worker RPC) to isolate whether the strict worker admission is the contributor to the 1054 failures, or whether it's purely the imbalance.
3. Run a third arm E0 with policy=default + mechanism=pd-disaggregation as a true control — kv-aware policy is itself part of what we are evaluating; default round-robin would have spread sessions across all 3 D.
4. Compare TTFT p99 against an Inferact-on-TCP-loopback run to evaluate H2/H3 cleanly. Cost: 1 more E2-shaped sweep (~1.5 h).
5. Investigate the 1054 E2 failures in request-metrics.jsonl — sample some to verify they are timeout-related vs admission-rejected vs upstream-500.

---

4. Comparison table — pending

To be appended.

---

5. Open questions for the next iteration

• Are the 85 E1 errors all timeouts? request-metrics.jsonl rows with error execution_mode should be sampled to confirm. (Quick check: grep the metrics jsonl for "execution_mode": "pd-disaggregation" and inspect latency_s / error fields.)
• Does E2 produce the predicted ~91% direct-to-D rate seen in the historical SWE-Bench v2 run, or does the Inferact workload's larger session count (50 vs 52 there) but very different per-session size distribution (mean 33 turns × ~2KB context growth per turn) push it lower?
• Is D2 = 0% an E1-specific artifact (kv-aware sticky in pd-disagg mode), or does the same happen in E2 before migration kicks in for the first time?