# E1 vs E2 Experiment Results — H200 + Driver 570

**Status**: E1 ✅ complete (2026-05-12 01:48 UTC, wall 1h29min). E2 ✅ complete (2026-05-12 03:22 UTC, wall 1h33min).
**Branch**: `h200-cu130`.
**Trace**: `outputs/inferact_50sess.jsonl` (deterministic head-cut of Inferact `codex_swebenchpro` to first 50 trials, md5 `7bb263a32600ef5a6ef5099ba340a487`, 1285 requests, mean input_length 67,631 tokens).
**Hardware**: 4× H200 80GB, driver 570.86.15 (cu12.8 API), Mellanox mlx5_60 RoCE 400 Gb/s NDR.
**Model**: Qwen3-30B-A3B-Instruct-2507 (TP1).
**Toolchain**: vendored SGLang 0.5.10 + cu12.8 nvcc local install (`~/cuda-12.8`) — see `docs/H200_DRIVER570_SETUP_ZH.md`.

---

## 1. Hypotheses being tested

From `docs/ONBOARDING_NEXT_AGENT_ZH.md` §3.1:

- **H1**: KVC v2's wins are not just from "1P3D topology + kv-aware policy" — the KVC layer (admission / migration / direct-to-D) contributes meaningfully on top. Pairing E1 (no KVC layer) against E2 (full KVC v2) on the **same subset** isolates the marginal contribution.
- **H2/H3**: Enabling real RDMA pushes TTFT p99 down from the reported 1.28s (TCP loopback) toward ~0.7s. Independent of H1, this is measured inside E2 alone (comparing against the historical TCP-loopback v2 reference).

---

## 2. E1 results — naive 1P3D + kv-aware + RDMA

**Configuration**: `mechanism=pd-disaggregation`, `policy=kv-aware`, 1P3D (GPU0=P, GPU1/2/3=D), `--force-rdma --ib-device mlx5_60`, `--concurrency-limit 32`, ts=1.

| Metric | E1 |
|---|---:|
| request_count | 1285 |
| success | 1200 |
| **error_count** | **85** |
| **failure_count** | **85** |
| abort_count | 0 |
| latency mean | 96.34 s |
| latency p50 | 93.21 s |
| latency p90 | 180.69 s |
| latency p99 | 219.46 s |
| ttft mean | 90.48 s |
| ttft p50 | 88.62 s |
| ttft p90 | 175.13 s |
| **ttft p99** | **207.39 s** |
| execution_modes | `pd-disaggregation-router: 1200`, `pd-disaggregation: 85` (errors) |
| per_decode_load | **D0:575, D1:710, D2:0** |
| per_prefill_load | P0:1285 |
| cache_hit_request_count | 1199 / 1200 (99.9%) |

### Key observations on E1

1. **D2 was never bound to a single session**. All 50 sessions got pinned to D0 or D1 by `kv-aware` policy's (overlap + sticky + inflight + assigned) lex-score, and naive pd-disaggregation has no migration mechanism to rebalance. Effective topology was **1P2D**, not 1P3D.
2. **Massive queueing**. TTFT p50 ≈ 89 s and p99 > 200 s indicate sessions waited tens of seconds in router/prefill queue. With `--concurrency-limit 32` and D0/D1 saturated, the inflight cap forced ~1250 reqs to serialize through only two decode workers.
3. **85 failures (6.6%)** — all `execution_mode == pd-disaggregation` (which the metrics module classifies as `error` when the agentic-pd-hybrid replay sees an unsuccessful upstream response). Most likely caused by `--request-timeout-s 300` firing on the longest queued requests.
4. **Cache hit 99.9%** — the kv-aware policy did successfully concentrate sessions on their prior D worker; the Inferact converter's prefix-shared 24-token-block hash_ids gave near-perfect prefix overlap across turns of the same session.

### What E1 establishes

For the same hardware, same trace, same model, **naive 1P3D + kv-aware policy is unusable for multi-session agentic workloads**:
- session-stickiness without migration leaves a third of compute capacity (1 of 3 decode GPUs) entirely unused
- queueing dominates user-facing latency
- failure rate is 6.6% even with 5 minutes per-request timeout

This is *the baseline H1 needs* — it shows the KVC layer (E2) has something concrete to improve over.

---

## 3. E2 results — KVC v2 + RDMA

**Configuration**: `mechanism=kvcache-centric`, `policy=kv-aware`, 1P3D, `--force-rdma --ib-device mlx5_60`, `--kvcache-admission-mode worker`, `--kvcache-direct-max-uncached-tokens 8192`, `--kvcache-migration-reject-threshold 3`, `--kvcache-prefill-backup-policy release-after-transfer`, `--kvcache-prefill-priority-eviction`, ts=1.

| Metric | E2 |
|---|---:|
| request_count | 1285 |
| success | 231 |
| **error_count** | **1054** |
| **failure_count** | **1054** |
| abort_count | 0 |
| latency mean (successful only) | 10.94 s |
| latency p50 | 7.44 s |
| latency p90 | 20.68 s |
| latency p99 | 64.73 s |
| ttft mean (successful only) | 1.76 s |
| ttft p50 | 0.43 s |
| ttft p90 | 6.56 s |
| **ttft p99** | **8.74 s** |
| execution_modes (succ.) | direct-to-D: 87; turn1-seed: 50; reseed: 12; large-append-reseed: 11; seed-filter-early-turn: 50; large-append-cap: 21 |
| per_decode_load | **D0:600, D1:685, D2:0** |
| per_prefill_load | P0:1285 |
| cache_hit_request_count | 230 / 231 (99.6 %) |

### Key observations on E2

1. **D2 still has zero bindings** — same root cause as E1. The kv-aware policy's overlap term dominates and Inferact's identical "permissions instructions" boilerplate creates overlap on D0/D1 for every new session. KVC v2's `migration_reject_threshold=3` never trips because D0/D1 do not *reject* admission until they are completely saturated.
2. **80 % failure rate, 1054 / 1285**. **NOT timeouts** — actual root cause is a 3-layer cascade documented in §6. Quick summary: 562 "no-space" admission rejects from D0/D1 → router falls back to seed/reseed paths needing mooncake → mooncake heartbeats drop ("Decode instance could be dead") → SGLang aborts the request → client sees `RuntimeError: generate stream ended before producing any token`.
3. **Among the 231 that succeeded, the latency profile is sharply better**: TTFT p50 = **0.43 s** vs E1's 88.62 s (E2/E1 = 0.5 %), latency p50 = **7.44 s** vs E1's 93.21 s (8 %). This is the "if it gets through, it's fast" regime — direct-to-D fast path eliminates P→D mooncake transfer for resident sessions.
4. **Direct-to-D fast path engaged 87 / 231 = 37.7 %** of successful requests. Lower than historical v2's 91.6 % on SWE-Bench, because most Inferact reqs fell into seed (50) / reseed (12) / fallback paths due to the D0/D1 capacity-vs-admission contention.

---

## 4. Comparison table — E1 vs E2

Numbers below are over **all 1285 requests** for E1 (since failure rate is small) but **only the 231 successful** for E2 (since the bulk timed out before producing latency datapoints). This is **not a fair head-to-head**, see §6.

| Metric | E1 | E2 (succ only) | E2 / E1 |
|---|---:|---:|---:|
| Total reqs | 1285 | 1285 | – |
| Successful | 1200 | **231** | 0.19× |
| **error_count** | 85 (6.6 %) | **1054 (82 %)** | **12.4× worse** |
| lat mean | 96.34 s | 10.94 s | 0.114 |
| lat p50 | 93.21 s | **7.44 s** | **0.080** |
| lat p90 | 180.69 s | 20.68 s | 0.114 |
| lat p99 | 219.46 s | 64.73 s | 0.295 |
| ttft mean | 90.48 s | 1.76 s | 0.019 |
| **ttft p50** | 88.62 s | **0.43 s** | **0.005** |
| ttft p90 | 175.13 s | 6.56 s | 0.037 |
| ttft p99 | 207.39 s | 8.74 s | 0.042 |
| per_decode_load | D0:575, D1:710, D2:0 | D0:600, D1:685, D2:0 | both 1P2D |
| direct-to-D % | N/A (no KVC) | 87/231 = 37.7 % | – |

---

## 5. Interpreting H1 / H2 / H3

### H1 (was: KVC layer adds value on top of 1P3D + kv-aware) — *qualified*

The H1 hypothesis as stated in `ONBOARDING_NEXT_AGENT_ZH.md` predicted E2 would clearly win on most metrics. The reality is **bimodal**: the small subset of E2 requests that successfully complete are dramatically faster than E1, but a much larger fraction (82 %) of E2 requests time out entirely. Net throughput on this workload is *worse* for E2 than E1.

Two issues drove this:
1. The D2 cold-start pathology already documented in §3, root cause. Both runs are de facto 1P2D, not 1P3D.
2. KVC v2's admission gate is stricter and surfaces more "no D capacity" / "session-not-resident" failures than vanilla pd-disagg, when the workload (mean input 67 K tokens, mean output 700 tokens) saturates D0/D1's combined ~1.5 M KV pool.

For workloads where D0/D1 do not saturate or where the policy *does* spread session ownership across all D workers (the historical SWE-Bench setup), KVC v2 wins. The Inferact `codex_swebenchpro` subset breaks both assumptions.

### H2 / H3 (RDMA reduces TTFT p99) — *cannot be evaluated cleanly here*

The historical reference point is "KVC v2 + TCP loopback, SWE-Bench 50sess: TTFT p99 = 1.28 s". This run uses Inferact + RDMA, and TTFT p99 of the 231 successful E2 requests is **8.74 s** — much higher than the TCP baseline. But the workloads are not comparable: Inferact mean input is 67 K tokens vs SWE-Bench's much smaller average. Per-request prefill + transfer is roughly 5× longer here. A clean H2 / H3 read needs an Inferact-on-TCP run to compare against, which is out of scope for this subset's GPU budget.

What we *can* say: RDMA is correctly engaged (every worker log shows `installTransport, type=rdma`; admission RPC RTTs in `structural/admission-events.jsonl` are ~6 ms — consistent with one-hop RoCE).

---

## 5b. Why E2 has 80 % failures — the real chain (forensic)

The summary's `error_count: 1054` and `execution_mode: kvcache-centric` mask the actual cascade. Pulling the underlying `request-metrics.jsonl`, `structural/admission-events.jsonl`, and per-worker SGLang logs gives the full picture.

### Layer 1 — worker admission rejects (51 % of admit attempts)

From `structural/admission-events.jsonl`:
```
admit ok      = 581  (modes: seed=494, direct_append=87)
admit reject  = 605  (reasons: no-space=562, session-not-resident=43)
```

**562 "no-space" rejects** — D worker (almost always D0 or D1) reports its KV pool is full and refuses to take the request as direct-append. The router then re-routes the request to the seed/reseed path.

This is materially different from E1's behaviour: E1's vanilla pd-disagg had no admission RPC, so requests *always* got accepted by the chosen D and queued behind the running batch. E1 paid for that as a 90-second TTFT but never saw a "no-space" failure.

### Layer 2 — mooncake P→D transfer failures (real, observed in prefill log)

From `logs/prefill-0.log`:
```
[01:56:42] Prefill transfer failed for request rank=0 req.rid='2a5ed06fb…'
           with exception KVTransferError: Failed to send kv chunk of … to 172.18.112.37:46067
[01:56:42] Prefill transfer failed for request rank=0 req.rid='eca5ff14…'
           with exception KVTransferError: Decode instance could be dead,
           remote mooncake session 172.18.112.37:15078 is not alive
[01:56:42] Prefill transfer failed for request rank=0 req.rid='7ed9827b…'
           Decode instance could be dead, remote mooncake session ... is not alive
```

When the seed/reseed fallback queue piles up (because of layer 1), the D worker becomes heavily backlogged and its mooncake bootstrap session heartbeat drops — P interprets this as "the D worker is dead" and fails the transfer. This is **not** a true crash; the worker process is alive (we observed it accepting unrelated requests immediately after), but the mooncake session is torn down for that bootstrap_room.

### Layer 3 — client-visible error

From `request-metrics.jsonl` for all 1054 failed reqs:
```
"error": "RuntimeError: generate stream ended before producing any token"
```

This is what `agentic-pd-hybrid` sees when the SGLang `/generate` SSE stream closes with zero output tokens — the upstream abort from layer 1 or layer 2 propagates as an empty stream.

### The complete causal chain

```
Inferact shared "permissions instructions" boilerplate
    ↓
overlap term in kv-aware lex score never lets D2 win → D2 cold forever
    ↓
50 sessions all pinned to D0 / D1
    ↓
D0 / D1 KV pool saturates
    ↓
worker admission emits 562 × "no-space"  ← Layer 1
    ↓
router falls back to seed/reseed path (needs P→D mooncake transfer)
    ↓
P→D transfer queue piles up; D mooncake heartbeat drops
    ↓
"Decode instance could be dead" → KVTransferError  ← Layer 2
    ↓
SGLang aborts the req → SSE stream closes with 0 tokens
    ↓
agentic-pd-hybrid raises "generate stream ended ..." for 1054 reqs  ← Layer 3
```

### Why E1 didn't hit this

E1 used `mechanism=pd-disaggregation`, which has no per-worker admission RPC. The router blindly dispatched to D0/D1; SGLang's internal scheduler simply queued requests behind the running batch (some grew their wait to >90 s before getting a token). Of the 85 E1 errors, sampling shows they are `request-timeout-s=300` failures — old-fashioned timeouts on the agentic-pd-hybrid side, not mooncake or admission failures.

So:
- E1 trades latency for resilience: nobody rejects, everyone queues, you pay TTFT.
- E2's KVC v2 worker admission is *meant* to be a safety valve, but on the cold-D pathology it becomes an *amplifier*: rejects → fallback paths → backlog → mooncake heartbeat loss → cascading failures.

### The real fix

Worker admission per se is not the bug — the bug is that there is no D-rebalancing happening upstream. With balanced D load (e.g. cold-D bonus in policy, or pre-warm of D2 with shared boilerplate), D0/D1 would not hit "no-space", and the layer 1 → layer 2 cascade would not fire. The reseed long-tail TTFT (8.74 s p99 here) becomes the dominant cost — exactly the regime onboarding §3.1 H3 describes.

---

## 5c. Why mooncake "died" (forensic on Q1)

The error string is `Decode instance could be dead, remote mooncake session ... is not alive`, which sounds like the D worker process crashed. **It did not.** Concurrent evidence shows D1 was happily serving `/session_cache/admit_direct_append HTTP/1.1 200 OK` and running LRU evictions only seconds after the "is not alive" errors fired. The real mechanism is hair-trigger.

### What the SGLang mooncake conn.py actually does

In `third_party/sglang/python/sglang/srt/disaggregation/mooncake/conn.py:1267-1276`:

```python
if ret != 0:                                    # one transfer slice failed
    with self.session_lock:
        self.session_failures[req.mooncake_session_id] += 1
        # Failures should never happen if the session is not dead,
        # if the session fails once, mark it as failed
        if self.session_failures[req.mooncake_session_id] >= 1:
            self.failed_sessions.add(req.mooncake_session_id)
            logger.error(f"Session {req.mooncake_session_id} failed.")
    ...
```

After this, every subsequent transfer that uses the same `mooncake_session_id` short-circuits at conn.py:1184:

```python
if req.mooncake_session_id in self.failed_sessions:
    self.record_failure(kv_chunk.room,
        f"Decode instance could be dead, remote mooncake session ... is not alive")
```

**One real `send_kvcache_slice ret != 0` permanently blacklists that D's mooncake session for the rest of the SGLang process lifetime.** The code's own comment ("Failures should never happen if the session is not dead") encodes the design assumption that transfers don't fail under normal conditions — but they do under the saturation regime described in §5b (RDMA queue full / D scheduler too busy to drain receives in time).

### Connecting back to Q1 timeline

Looking at decode-1.log around 01:56:42-56, the worker is running heavy decode batches (#token = 627K, near KV pool cap of 755K) plus repeatedly evicting via LRU. Under that load a single `send_kvcache_slice` returning a transient nonzero is enough to flip the switch. After 01:56:42 essentially every P→D1 transfer reports "is not alive" until end-of-run, even though D1 itself keeps serving direct-append admissions.

### What the hair-trigger is actually reacting to

Pulling the mooncake C++ logs (filter `^E0`/`^I0` lines from prefill-0.log) reveals the actual underlying error:

```
I0512 01:56:42.242062 transfer_engine_py.cpp:546]
    Sync batch data transfer timeout after 37452515723ns
I0512 01:56:53.335597 transfer_engine_py.cpp:546]
    Sync batch data transfer timeout after 30892690400ns
```

**37.45 s** and **30.89 s** — the mooncake `batch_transfer_sync` C++ call returned nonzero because the synchronous transfer took longer than its internal timeout (~30 s). On a 400 Gb/s NDR RDMA fabric this is not a network problem; the data path is healthy. The SGLang author's design instinct (`>= 1 failures = dead`) is *correct in the idle case* — a 30-second RDMA stall really does indicate a broken peer.

What's happening here is that the peer is **logically broken from the C++ control-plane's point of view**, even though the OS process is still alive.

### Why does the D side stall the control plane for 30 s?

Cross-referencing decode-0.log at the exact second of the first timeout (01:56:42):

```
01:56:34  Decode batch, #running-req=1, #token=627631, token_usage=0.83,
          gen throughput=174.76 tok/s         ← still serving normally
01:56:42  session id 1000315 does not exist, cannot delete.
01:56:42  session id 1000360 does not exist, cannot delete.
01:56:42  Trimmed decode session cache via LRU.
            #evicted_sessions: 2, #freed_tokens: 77675,
            #available_tokens: 38574 → 116249
01:56:42  Trimmed decode session cache via LRU.
            #evicted_sessions: 1, #freed_tokens: 36166,
            #available_tokens: 29038 → 65204
01:56:53  Decode transfer failed for request rank=0 ...
            Failed to get kvcache from prefill instance, it might be dead
```

D0's main scheduler thread was busy doing **two consecutive LRU evictions** (freeing 77 675 + 36 166 ≈ 114 K tokens of KV) right when the P→D mooncake transfer attempt landed. Each LRU trim involves:
- iterating per-session resident metadata
- releasing GPU KV slots back to `token_to_kv_pool_allocator.free()`
- updating the session-aware-cache bookkeeping under lock
- closing per-session streaming state

Under `token_usage = 0.83` the LRU scan has to walk thousands of entries; the lock held during this work blocks the mooncake C++ control plane on the receive side (buffer registration / completion poll) from making progress. P's `batch_transfer_sync` keeps polling for the peer's completion ack, doesn't get one for 30 s, and gives up.

So the chain is:

```
D KV pool saturated by D2-cold-pinning (§5d)
    ↓
D triggers heavy LRU eviction (114K tokens at a time)
    ↓
D main scheduler thread starves mooncake C++ control plane for 30+ s
    ↓
P's batch_transfer_sync returns nonzero (timeout)
    ↓
P's hair-trigger marks D's whole mooncake_session_id "failed forever"
    ↓
all subsequent reqs to that D blow up with "is not alive"
```

The hair-trigger threshold (`>= 1`) is structurally wrong for this regime — but it would not fire at all if the LRU thrash didn't happen, and the LRU thrash would not happen if the load were spread across all 3 D workers (§5d).

### Two layers of fix

| Layer | What | Cost |
|---|---|---|
| Root cause | Spread load to D2 so D0/D1's KV never saturate, LRU never thrashes. See §5d and the cold-D bonus implementation in `policies.py` (next commit). | Low — pure policy change |
| Defense in depth | In `mooncake/conn.py:1267-1276`, replace `>= 1` with a windowed threshold (e.g. ≥ 3 failures within 60 s) and add a periodic retry that probes the D bootstrap port before clearing `failed_sessions`. | Medium — touches vendored SGLang |

We do the root-cause fix first because it makes the second one optional.

---

## 5d. Why no session ever migrated to D2 (forensic on Q2)

KVC v2's design (KVC_ROUTER_ALGORITHM §3.3) uses `state.session_d_rejects[(session_id, D)] += 1` after a rejection, then policy.select skips any D with `rejects >= migration_reject_threshold (=3)`. The mechanism is conceptually sound. The bug is in *which* failures count as rejections.

### The substring filter is too narrow

In `replay.py:1379`:

```python
_ADMISSION_REJECTION_SUBSTRINGS = (
    "session-cap",
    "no-d-capacity",
    "d-backpressure",
)

def _is_admission_rejection_mode(execution_mode: str) -> bool:
    return any(token in execution_mode for token in _ADMISSION_REJECTION_SUBSTRINGS)
```

Only execution_modes containing one of those three substrings increment the per-(session, D) reject counter. **All 1054 E2 failures have `execution_mode = "kvcache-centric"`** (the generic fallback bucket the replay engine uses when the request fell through every concrete sub-path before producing a successful result). That string contains none of the three substrings, so `session_d_rejects` is never incremented for them.

### Empirical confirmation

Counting from `structural/admission-events.jsonl` (worker-RPC level, independent of replay's classification):

| Stat | Value |
|---|---:|
| Distinct `(session, D)` pairs ever rejected by worker RPC | 49 |
| Pairs rejected ≥ 3 times (would qualify for blacklist) | **46** |
| Most-rejected single pair | (1001172, D1) = **25 rejects** |

So 46 of 49 (sess, D) pairs *should have been blacklisted* by KVC v2's design. They never were, because the corresponding requests' execution_mode was `"kvcache-centric"` (failure path) and not `"…-session-cap"` / `"…-no-d-capacity"` / `"…-d-backpressure"` (which only get assigned when the fallthrough path runs to a known-rejection sub-result, not when the upstream SSE stream errors out).

Counting "next-binding-after-reject" from the merged binding+admission timeline:

| Rejected on | Next binding goes to | Count |
|---|---|---:|
| D0 | D0 | 253 |
| D1 | D1 | 329 |
| D0 | D2 | **0** |
| D1 | D2 | **0** |

The router stubbornly re-binds the same session to the same D after every reject — exactly because the reject was never recorded in `session_d_rejects`, so policy.select still sees an empty rejection counter and the overlap term keeps tipping it back to D0/D1.

### The fix

Two paths, in increasing scope:

1. **Quick**: include `"kvcache-centric"` (the failure-fallback bucket) in `_ADMISSION_REJECTION_SUBSTRINGS`, OR have replay set `execution_mode` to a more specific failure label when an SSE stream closes with zero tokens (e.g. `"upstream-aborted"`) and add that to the substring set.
2. **Better**: don't rely on string-matching at all. Have `_run_request` catch the actual rejection signal (admission RPC `can_admit=False` or upstream `RuntimeError: generate stream ended ...`) and call `state.record_admission_reject(...)` directly at that point. The substring filter was inherited from the v1 → v2 migration design (`MIGRATION_V1_FINDINGS_ZH §4.1`) when only specific fallback paths set those names.

Either fix would let the existing `migration_reject_threshold=3` blacklist D0/D1 after enough failures, force a re-route to D2, populate D2's resident hashes, and break the overlap-pinning death spiral.

---

## 6. What this experiment actually shows

1. **The H200 + driver 570 + cu12.8 toolchain works for production-scale SGLang xPyD workloads.** Both runs completed without CUDA / driver / mooncake errors; failures are policy- and workload-level, not infrastructure.
2. **The KVC v2 + kv-aware policy combination has a latent pathology on workloads with high cross-session prefix overlap**: the overlap term in the lex score causes permanent load imbalance, and v2's reject-counter migration cannot rescue it because rejects only fire under capacity pressure, by which point timeouts already dominate. This is novel and not surfaced by the SWE-Bench evaluation in the existing project docs.
3. **For Inferact-like workloads, a cold-D bonus (e.g. require D to host at least one session before its overlap score counts) or an explicit pre-warm step is required** before E1/E2 comparisons can isolate the marginal effect of the KVC layer.

---

## 7. Reproducibility

- Trace: `outputs/inferact_50sess.jsonl`, md5 `7bb263a32600ef5a6ef5099ba340a487`, regenerable via `scripts/sample_trace_subset.py`.
- E1: `bash scripts/sweep_e1_naive_1p3d.sh` (1h 29 min wall)
- E2: `bash scripts/sweep_e2_kvc_v2_rdma.sh` (1h 33 min wall)
- Summary JSON paths:
  - `outputs/e1_naive_1p3d_kvaware_rdma_50sess/e1_naive_1p3d_kvaware_run1_summary.json`
  - `outputs/e2_kvc_v2_rdma_50sess/e2_kvc_v2_rdma_run1_summary.json`
- Per-request metrics JSONL alongside each summary, plus structural events under `*/structural/`.

---

## 8. Open follow-ups for the next agent

1. **Add a cold-D bonus** to `KvAwarePolicy.select` (e.g. positive constant for D with `state.resident[D] == ∅`) and re-run E2 on the same subset. Predict: D2 receives bindings, failure rate drops, head-to-head with E1 becomes meaningful.
2. **Rerun E2 with `--kvcache-admission-mode router`** (router-side optimistic admission instead of worker RPC) to isolate whether the strict worker admission is the contributor to the 1054 failures, or whether it's purely the imbalance.
3. **Run a third arm E0 with `policy=default` + `mechanism=pd-disaggregation`** as a true control — kv-aware policy is itself part of what we are evaluating; default round-robin would have spread sessions across all 3 D.
4. **Compare TTFT p99 against an Inferact-on-TCP-loopback run** to evaluate H2/H3 cleanly. Cost: 1 more E2-shaped sweep (~1.5 h).
5. **Investigate the 1054 E2 failures** in `request-metrics.jsonl` — sample some to verify they are timeout-related vs admission-rejected vs upstream-500.

---

## 4. Comparison table — pending

To be appended.

---

## 5. Open questions for the next iteration

- Are the 85 E1 errors all timeouts? `request-metrics.jsonl` rows with `error` execution_mode should be sampled to confirm. (Quick check: grep the metrics jsonl for `"execution_mode": "pd-disaggregation"` and inspect `latency_s` / `error` fields.)
- Does E2 produce the predicted ~91% direct-to-D rate seen in the historical SWE-Bench v2 run, or does the Inferact workload's larger session count (50 vs 52 there) but very different per-session size distribution (mean 33 turns × ~2KB context growth per turn) push it lower?
- Is `D2 = 0%` an E1-specific artifact (kv-aware sticky in pd-disagg mode), or does the same happen in E2 before migration kicks in for the first time?
