agentic-pd-hybrid/docs/E1_E2_RESULTS_ZH.md

E1 vs E2 Experiment Results — H200 + Driver 570

Status: E1 ✅ complete (2026-05-12 01:48 UTC, wall 1h29min). E2 ✅ complete (2026-05-12 03:22 UTC, wall 1h33min). Branch: h200-cu130. Trace: outputs/inferact_50sess.jsonl (deterministic head-cut of Inferact codex_swebenchpro to first 50 trials, md5 7bb263a32600ef5a6ef5099ba340a487, 1285 requests, mean input_length 67,631 tokens). Hardware: 4× H200 80GB, driver 570.86.15 (cu12.8 API), Mellanox mlx5_60 RoCE 400 Gb/s NDR. Model: Qwen3-30B-A3B-Instruct-2507 (TP1). Toolchain: vendored SGLang 0.5.10 + cu12.8 nvcc local install (~/cuda-12.8) — see docs/H200_DRIVER570_SETUP_ZH.md.

---

1. Hypotheses being tested

From docs/ONBOARDING_NEXT_AGENT_ZH.md §3.1:

• H1: KVC v2's wins are not just from "1P3D topology + kv-aware policy" — the KVC layer (admission / migration / direct-to-D) contributes meaningfully on top. Pairing E1 (no KVC layer) against E2 (full KVC v2) on the same subset isolates the marginal contribution.
• H2/H3: Enabling real RDMA pushes TTFT p99 down from the reported 1.28s (TCP loopback) toward ~0.7s. Independent of H1, this is measured inside E2 alone (comparing against the historical TCP-loopback v2 reference).

---

2. E1 results — naive 1P3D + kv-aware + RDMA

Configuration: mechanism=pd-disaggregation, policy=kv-aware, 1P3D (GPU0=P, GPU1/2/3=D), --force-rdma --ib-device mlx5_60, --concurrency-limit 32, ts=1.

Metric	E1
request_count	1285
success	1200
error_count	85
failure_count	85
abort_count	0
latency mean	96.34 s
latency p50	93.21 s
latency p90	180.69 s
latency p99	219.46 s
ttft mean	90.48 s
ttft p50	88.62 s
ttft p90	175.13 s
ttft p99	207.39 s
execution_modes	pd-disaggregation-router: 1200, pd-disaggregation: 85 (errors)
per_decode_load	D0:575, D1:710, D2:0
per_prefill_load	P0:1285
cache_hit_request_count	1199 / 1200 (99.9%)

Key observations on E1

1. D2 was never bound to a single session. All 50 sessions got pinned to D0 or D1 by kv-aware policy's (overlap + sticky + inflight + assigned) lex-score, and naive pd-disaggregation has no migration mechanism to rebalance. Effective topology was 1P2D, not 1P3D.
2. Massive queueing. TTFT p50 ≈ 89 s and p99 > 200 s indicate sessions waited tens of seconds in router/prefill queue. With --concurrency-limit 32 and D0/D1 saturated, the inflight cap forced ~1250 reqs to serialize through only two decode workers.
3. 85 failures (6.6%) — all execution_mode == pd-disaggregation (which the metrics module classifies as error when the agentic-pd-hybrid replay sees an unsuccessful upstream response). Most likely caused by --request-timeout-s 300 firing on the longest queued requests.
4. Cache hit 99.9% — the kv-aware policy did successfully concentrate sessions on their prior D worker; the Inferact converter's prefix-shared 24-token-block hash_ids gave near-perfect prefix overlap across turns of the same session.

What E1 establishes

For the same hardware, same trace, same model, naive 1P3D + kv-aware policy is unusable for multi-session agentic workloads:

• session-stickiness without migration leaves a third of compute capacity (1 of 3 decode GPUs) entirely unused
• queueing dominates user-facing latency
• failure rate is 6.6% even with 5 minutes per-request timeout

This is the baseline H1 needs — it shows the KVC layer (E2) has something concrete to improve over.

---

3. E2 results — KVC v2 + RDMA

Configuration: mechanism=kvcache-centric, policy=kv-aware, 1P3D, --force-rdma --ib-device mlx5_60, --kvcache-admission-mode worker, --kvcache-direct-max-uncached-tokens 8192, --kvcache-migration-reject-threshold 3, --kvcache-prefill-backup-policy release-after-transfer, --kvcache-prefill-priority-eviction, ts=1.

Metric	E2
request_count	1285
success	231
error_count	1054
failure_count	1054
abort_count	0
latency mean (successful only)	10.94 s
latency p50	7.44 s
latency p90	20.68 s
latency p99	64.73 s
ttft mean (successful only)	1.76 s
ttft p50	0.43 s
ttft p90	6.56 s
ttft p99	8.74 s
execution_modes (succ.)	direct-to-D: 87; turn1-seed: 50; reseed: 12; large-append-reseed: 11; seed-filter-early-turn: 50; large-append-cap: 21
per_decode_load	D0:600, D1:685, D2:0
per_prefill_load	P0:1285
cache_hit_request_count	230 / 231 (99.6 %)

Key observations on E2

1. D2 still has zero bindings — same root cause as E1. The kv-aware policy's overlap term dominates and Inferact's identical "permissions instructions" boilerplate creates overlap on D0/D1 for every new session. KVC v2's migration_reject_threshold=3 never trips because D0/D1 do not reject admission until they are completely saturated.
2. 80 % failure rate, 1054 / 1285. NOT timeouts — actual root cause is a 3-layer cascade documented in §6. Quick summary: 562 "no-space" admission rejects from D0/D1 → router falls back to seed/reseed paths needing mooncake → mooncake heartbeats drop ("Decode instance could be dead") → SGLang aborts the request → client sees RuntimeError: generate stream ended before producing any token.
3. Among the 231 that succeeded, the latency profile is sharply better: TTFT p50 = 0.43 s vs E1's 88.62 s (E2/E1 = 0.5 %), latency p50 = 7.44 s vs E1's 93.21 s (8 %). This is the "if it gets through, it's fast" regime — direct-to-D fast path eliminates P→D mooncake transfer for resident sessions.
4. Direct-to-D fast path engaged 87 / 231 = 37.7 % of successful requests. Lower than historical v2's 91.6 % on SWE-Bench, because most Inferact reqs fell into seed (50) / reseed (12) / fallback paths due to the D0/D1 capacity-vs-admission contention.

---

4. Comparison table — E1 vs E2

Numbers below are over all 1285 requests for E1 (since failure rate is small) but only the 231 successful for E2 (since the bulk timed out before producing latency datapoints). This is not a fair head-to-head, see §6.

Metric	E1	E2 (succ only)	E2 / E1
Total reqs	1285	1285	–
Successful	1200	231	0.19×
error_count	85 (6.6 %)	1054 (82 %)	12.4× worse
lat mean	96.34 s	10.94 s	0.114
lat p50	93.21 s	7.44 s	0.080
lat p90	180.69 s	20.68 s	0.114
lat p99	219.46 s	64.73 s	0.295
ttft mean	90.48 s	1.76 s	0.019
ttft p50	88.62 s	0.43 s	0.005
ttft p90	175.13 s	6.56 s	0.037
ttft p99	207.39 s	8.74 s	0.042
per_decode_load	D0:575, D1:710, D2:0	D0:600, D1:685, D2:0	both 1P2D
direct-to-D %	N/A (no KVC)	87/231 = 37.7 %	–

---

5. Interpreting H1 / H2 / H3

H1 (was: KVC layer adds value on top of 1P3D + kv-aware) — qualified

The H1 hypothesis as stated in ONBOARDING_NEXT_AGENT_ZH.md predicted E2 would clearly win on most metrics. The reality is bimodal: the small subset of E2 requests that successfully complete are dramatically faster than E1, but a much larger fraction (82 %) of E2 requests time out entirely. Net throughput on this workload is worse for E2 than E1.

Two issues drove this:

1. The D2 cold-start pathology already documented in §3, root cause. Both runs are de facto 1P2D, not 1P3D.
2. KVC v2's admission gate is stricter and surfaces more "no D capacity" / "session-not-resident" failures than vanilla pd-disagg, when the workload (mean input 67 K tokens, mean output 700 tokens) saturates D0/D1's combined ~1.5 M KV pool.

For workloads where D0/D1 do not saturate or where the policy does spread session ownership across all D workers (the historical SWE-Bench setup), KVC v2 wins. The Inferact codex_swebenchpro subset breaks both assumptions.

H2 / H3 (RDMA reduces TTFT p99) — cannot be evaluated cleanly here

The historical reference point is "KVC v2 + TCP loopback, SWE-Bench 50sess: TTFT p99 = 1.28 s". This run uses Inferact + RDMA, and TTFT p99 of the 231 successful E2 requests is 8.74 s — much higher than the TCP baseline. But the workloads are not comparable: Inferact mean input is 67 K tokens vs SWE-Bench's much smaller average. Per-request prefill + transfer is roughly 5× longer here. A clean H2 / H3 read needs an Inferact-on-TCP run to compare against, which is out of scope for this subset's GPU budget.

What we can say: RDMA is correctly engaged (every worker log shows installTransport, type=rdma; admission RPC RTTs in structural/admission-events.jsonl are ~6 ms — consistent with one-hop RoCE).

---

5b. Why E2 has 80 % failures — the real chain (forensic)

The summary's error_count: 1054 and execution_mode: kvcache-centric mask the actual cascade. Pulling the underlying request-metrics.jsonl, structural/admission-events.jsonl, and per-worker SGLang logs gives the full picture.

Layer 1 — worker admission rejects (51 % of admit attempts)

From structural/admission-events.jsonl:

admit ok      = 581  (modes: seed=494, direct_append=87)
admit reject  = 605  (reasons: no-space=562, session-not-resident=43)

562 "no-space" rejects — D worker (almost always D0 or D1) reports its KV pool is full and refuses to take the request as direct-append. The router then re-routes the request to the seed/reseed path.

This is materially different from E1's behaviour: E1's vanilla pd-disagg had no admission RPC, so requests always got accepted by the chosen D and queued behind the running batch. E1 paid for that as a 90-second TTFT but never saw a "no-space" failure.

Layer 2 — mooncake P→D transfer failures (real, observed in prefill log)

From logs/prefill-0.log:

[01:56:42] Prefill transfer failed for request rank=0 req.rid='2a5ed06fb…'
           with exception KVTransferError: Failed to send kv chunk of … to 172.18.112.37:46067
[01:56:42] Prefill transfer failed for request rank=0 req.rid='eca5ff14…'
           with exception KVTransferError: Decode instance could be dead,
           remote mooncake session 172.18.112.37:15078 is not alive
[01:56:42] Prefill transfer failed for request rank=0 req.rid='7ed9827b…'
           Decode instance could be dead, remote mooncake session ... is not alive

When the seed/reseed fallback queue piles up (because of layer 1), the D worker becomes heavily backlogged and its mooncake bootstrap session heartbeat drops — P interprets this as "the D worker is dead" and fails the transfer. This is not a true crash; the worker process is alive (we observed it accepting unrelated requests immediately after), but the mooncake session is torn down for that bootstrap_room.

Layer 3 — client-visible error

From request-metrics.jsonl for all 1054 failed reqs:

"error": "RuntimeError: generate stream ended before producing any token"

This is what agentic-pd-hybrid sees when the SGLang /generate SSE stream closes with zero output tokens — the upstream abort from layer 1 or layer 2 propagates as an empty stream.

The complete causal chain

Inferact shared "permissions instructions" boilerplate
    ↓
overlap term in kv-aware lex score never lets D2 win → D2 cold forever
    ↓
50 sessions all pinned to D0 / D1
    ↓
D0 / D1 KV pool saturates
    ↓
worker admission emits 562 × "no-space"  ← Layer 1
    ↓
router falls back to seed/reseed path (needs P→D mooncake transfer)
    ↓
P→D transfer queue piles up; D mooncake heartbeat drops
    ↓
"Decode instance could be dead" → KVTransferError  ← Layer 2
    ↓
SGLang aborts the req → SSE stream closes with 0 tokens
    ↓
agentic-pd-hybrid raises "generate stream ended ..." for 1054 reqs  ← Layer 3

Why E1 didn't hit this

E1 used mechanism=pd-disaggregation, which has no per-worker admission RPC. The router blindly dispatched to D0/D1; SGLang's internal scheduler simply queued requests behind the running batch (some grew their wait to >90 s before getting a token). Of the 85 E1 errors, sampling shows they are request-timeout-s=300 failures — old-fashioned timeouts on the agentic-pd-hybrid side, not mooncake or admission failures.

So:

• E1 trades latency for resilience: nobody rejects, everyone queues, you pay TTFT.
• E2's KVC v2 worker admission is meant to be a safety valve, but on the cold-D pathology it becomes an amplifier: rejects → fallback paths → backlog → mooncake heartbeat loss → cascading failures.

The real fix

Worker admission per se is not the bug — the bug is that there is no D-rebalancing happening upstream. With balanced D load (e.g. cold-D bonus in policy, or pre-warm of D2 with shared boilerplate), D0/D1 would not hit "no-space", and the layer 1 → layer 2 cascade would not fire. The reseed long-tail TTFT (8.74 s p99 here) becomes the dominant cost — exactly the regime onboarding §3.1 H3 describes.

---

6. What this experiment actually shows

1. The H200 + driver 570 + cu12.8 toolchain works for production-scale SGLang xPyD workloads. Both runs completed without CUDA / driver / mooncake errors; failures are policy- and workload-level, not infrastructure.
2. The KVC v2 + kv-aware policy combination has a latent pathology on workloads with high cross-session prefix overlap: the overlap term in the lex score causes permanent load imbalance, and v2's reject-counter migration cannot rescue it because rejects only fire under capacity pressure, by which point timeouts already dominate. This is novel and not surfaced by the SWE-Bench evaluation in the existing project docs.
3. For Inferact-like workloads, a cold-D bonus (e.g. require D to host at least one session before its overlap score counts) or an explicit pre-warm step is required before E1/E2 comparisons can isolate the marginal effect of the KVC layer.

---

7. Reproducibility

• Trace: outputs/inferact_50sess.jsonl, md5 7bb263a32600ef5a6ef5099ba340a487, regenerable via scripts/sample_trace_subset.py.
• E1: bash scripts/sweep_e1_naive_1p3d.sh (1h 29 min wall)
• E2: bash scripts/sweep_e2_kvc_v2_rdma.sh (1h 33 min wall)
• Summary JSON paths:
  • outputs/e1_naive_1p3d_kvaware_rdma_50sess/e1_naive_1p3d_kvaware_run1_summary.json
  • outputs/e2_kvc_v2_rdma_50sess/e2_kvc_v2_rdma_run1_summary.json
• Per-request metrics JSONL alongside each summary, plus structural events under */structural/.

---

8. Open follow-ups for the next agent

1. Add a cold-D bonus to KvAwarePolicy.select (e.g. positive constant for D with state.resident[D] == ∅) and re-run E2 on the same subset. Predict: D2 receives bindings, failure rate drops, head-to-head with E1 becomes meaningful.
2. Rerun E2 with --kvcache-admission-mode router (router-side optimistic admission instead of worker RPC) to isolate whether the strict worker admission is the contributor to the 1054 failures, or whether it's purely the imbalance.
3. Run a third arm E0 with policy=default + mechanism=pd-disaggregation as a true control — kv-aware policy is itself part of what we are evaluating; default round-robin would have spread sessions across all 3 D.
4. Compare TTFT p99 against an Inferact-on-TCP-loopback run to evaluate H2/H3 cleanly. Cost: 1 more E2-shaped sweep (~1.5 h).
5. Investigate the 1054 E2 failures in request-metrics.jsonl — sample some to verify they are timeout-related vs admission-rejected vs upstream-500.

---

4. Comparison table — pending

To be appended.

---

5. Open questions for the next iteration

• Are the 85 E1 errors all timeouts? request-metrics.jsonl rows with error execution_mode should be sampled to confirm. (Quick check: grep the metrics jsonl for "execution_mode": "pd-disaggregation" and inspect latency_s / error fields.)
• Does E2 produce the predicted ~91% direct-to-D rate seen in the historical SWE-Bench v2 run, or does the Inferact workload's larger session count (50 vs 52 there) but very different per-session size distribution (mean 33 turns × ~2KB context growth per turn) push it lower?
• Is D2 = 0% an E1-specific artifact (kv-aware sticky in pd-disagg mode), or does the same happen in E2 before migration kicks in for the first time?