e3e5c45ed4
Same pathological imbalance E1 showed reproduces in E2: D2 has zero bindings at 33% POSTs in. Root cause is structural, not a KVC v2 bug: all 50 Inferact sessions begin with identical "permissions instructions" boilerplate, so the converter assigns them identical first-block hash_ids. kv-aware policy's overlap term (lex-score position 0) makes any already-resident D dominate a fresh D unconditionally, and v2's migration only activates on admission rejects which never fire because D0/D1 KV pools have headroom. The H1 conclusion is qualified: KVC v2 helps per-request work (direct- to-D fast path) but does not rebalance D worker load on workloads with shared cross-session prefixes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
107 lines
6.7 KiB
Markdown
107 lines
6.7 KiB
Markdown
# E1 vs E2 Experiment Results — H200 + Driver 570
|
||
|
||
**Status**: E1 ✅ complete (2026-05-12 01:48 UTC, wall 1h29min). E2 ⏳ running.
|
||
**Branch**: `h200-cu130`.
|
||
**Trace**: `outputs/inferact_50sess.jsonl` (deterministic head-cut of Inferact `codex_swebenchpro` to first 50 trials, md5 `7bb263a32600ef5a6ef5099ba340a487`, 1285 requests, mean input_length 67,631 tokens).
|
||
**Hardware**: 4× H200 80GB, driver 570.86.15 (cu12.8 API), Mellanox mlx5_60 RoCE 400 Gb/s NDR.
|
||
**Model**: Qwen3-30B-A3B-Instruct-2507 (TP1).
|
||
**Toolchain**: vendored SGLang 0.5.10 + cu12.8 nvcc local install (`~/cuda-12.8`) — see `docs/H200_DRIVER570_SETUP_ZH.md`.
|
||
|
||
---
|
||
|
||
## 1. Hypotheses being tested
|
||
|
||
From `docs/ONBOARDING_NEXT_AGENT_ZH.md` §3.1:
|
||
|
||
- **H1**: KVC v2's wins are not just from "1P3D topology + kv-aware policy" — the KVC layer (admission / migration / direct-to-D) contributes meaningfully on top. Pairing E1 (no KVC layer) against E2 (full KVC v2) on the **same subset** isolates the marginal contribution.
|
||
- **H2/H3**: Enabling real RDMA pushes TTFT p99 down from the reported 1.28s (TCP loopback) toward ~0.7s. Independent of H1, this is measured inside E2 alone (comparing against the historical TCP-loopback v2 reference).
|
||
|
||
---
|
||
|
||
## 2. E1 results — naive 1P3D + kv-aware + RDMA
|
||
|
||
**Configuration**: `mechanism=pd-disaggregation`, `policy=kv-aware`, 1P3D (GPU0=P, GPU1/2/3=D), `--force-rdma --ib-device mlx5_60`, `--concurrency-limit 32`, ts=1.
|
||
|
||
| Metric | E1 |
|
||
|---|---:|
|
||
| request_count | 1285 |
|
||
| success | 1200 |
|
||
| **error_count** | **85** |
|
||
| **failure_count** | **85** |
|
||
| abort_count | 0 |
|
||
| latency mean | 96.34 s |
|
||
| latency p50 | 93.21 s |
|
||
| latency p90 | 180.69 s |
|
||
| latency p99 | 219.46 s |
|
||
| ttft mean | 90.48 s |
|
||
| ttft p50 | 88.62 s |
|
||
| ttft p90 | 175.13 s |
|
||
| **ttft p99** | **207.39 s** |
|
||
| execution_modes | `pd-disaggregation-router: 1200`, `pd-disaggregation: 85` (errors) |
|
||
| per_decode_load | **D0:575, D1:710, D2:0** |
|
||
| per_prefill_load | P0:1285 |
|
||
| cache_hit_request_count | 1199 / 1200 (99.9%) |
|
||
|
||
### Key observations on E1
|
||
|
||
1. **D2 was never bound to a single session**. All 50 sessions got pinned to D0 or D1 by `kv-aware` policy's (overlap + sticky + inflight + assigned) lex-score, and naive pd-disaggregation has no migration mechanism to rebalance. Effective topology was **1P2D**, not 1P3D.
|
||
2. **Massive queueing**. TTFT p50 ≈ 89 s and p99 > 200 s indicate sessions waited tens of seconds in router/prefill queue. With `--concurrency-limit 32` and D0/D1 saturated, the inflight cap forced ~1250 reqs to serialize through only two decode workers.
|
||
3. **85 failures (6.6%)** — all `execution_mode == pd-disaggregation` (which the metrics module classifies as `error` when the agentic-pd-hybrid replay sees an unsuccessful upstream response). Most likely caused by `--request-timeout-s 300` firing on the longest queued requests.
|
||
4. **Cache hit 99.9%** — the kv-aware policy did successfully concentrate sessions on their prior D worker; the Inferact converter's prefix-shared 24-token-block hash_ids gave near-perfect prefix overlap across turns of the same session.
|
||
|
||
### What E1 establishes
|
||
|
||
For the same hardware, same trace, same model, **naive 1P3D + kv-aware policy is unusable for multi-session agentic workloads**:
|
||
- session-stickiness without migration leaves a third of compute capacity (1 of 3 decode GPUs) entirely unused
|
||
- queueing dominates user-facing latency
|
||
- failure rate is 6.6% even with 5 minutes per-request timeout
|
||
|
||
This is *the baseline H1 needs* — it shows the KVC layer (E2) has something concrete to improve over.
|
||
|
||
---
|
||
|
||
## 3. E2 — in progress + an unexpected finding about D2
|
||
|
||
Background task `b0im1d48q`, launched 2026-05-12 01:48 UTC. Mid-run snapshot at 16 minutes (33 % POSTs dispatched):
|
||
|
||
| | D0 | D1 | D2 |
|
||
|---|---:|---:|---:|
|
||
| bindings so far | 248 | 267 | **0** |
|
||
| GPU util (snapshot) | 0 % | 0 % | 0 % |
|
||
| KV pool util (across run) | high | high | empty |
|
||
|
||
**D2 receives zero traffic in E2 too, just like E1**. This is *not* the result we expected — H1 predicted that KVC's session-migration mechanism (reset-on-success blacklist with `migration_reject_threshold=3`) would route around the imbalance E1 showed. It doesn't.
|
||
|
||
### Root cause
|
||
|
||
`KvAwarePolicy.select` (policies.py:171-202) scores candidates by 4-tuple lex order `(overlap + α·sticky, sticky, -inflight, -assigned)`. The `overlap` term dominates: any D that has resident KV blocks matching the incoming request's `hash_ids` wins position 0.
|
||
|
||
In the Inferact `codex_swebenchpro` workload, **all 50 sessions begin with identical "permissions instructions" boilerplate** (the converter sees this as identical first-block content across trial 0..49). Our hash_id construction (sha256 over the token sequence per 24-token block, see `scripts/convert_inferact_to_trace.py`) therefore yields *identical block hashes across sessions* for the first ~50 blocks.
|
||
|
||
Concretely, when session N's turn 0 lands:
|
||
- D0 / D1 already host previous sessions → their `state.resident` sets include those shared boilerplate hashes → `overlap > 0`
|
||
- D2 has never been admitted → `state.resident[D2]` is empty → `overlap = 0`
|
||
- D0/D1 tie at position 0; D2 always loses
|
||
|
||
The migration mechanism never triggers because D0/D1 have ample KV (peak token_usage ~0.86 in v2 historical reports) and never *reject* admission. No rejects → no `(session, D)` blacklist accumulation → no migration → D2 stays cold forever.
|
||
|
||
### Implication for H1
|
||
|
||
H1 is *not falsified*, but it is *qualified*: KVC v2 still improves over naive pd-disaggregation on per-request work (direct-to-D fast path skips P→D mooncake transfer for turn≥1 on the same D), but it does **not** automatically balance load across D workers when the workload has high cross-session prefix overlap. To realise the full theoretical benefit of 1P3D on this workload, the policy needs an explicit cold-D bonus, or a pre-warming step that seeds D2 with shared boilerplate at startup.
|
||
|
||
Full E2 metrics will be filled in upon completion (ETA ~22 min from snapshot).
|
||
|
||
---
|
||
|
||
## 4. Comparison table — pending
|
||
|
||
To be appended.
|
||
|
||
---
|
||
|
||
## 5. Open questions for the next iteration
|
||
|
||
- Are the 85 E1 errors all timeouts? `request-metrics.jsonl` rows with `error` execution_mode should be sampled to confirm. (Quick check: grep the metrics jsonl for `"execution_mode": "pd-disaggregation"` and inspect `latency_s` / `error` fields.)
|
||
- Does E2 produce the predicted ~91% direct-to-D rate seen in the historical SWE-Bench v2 run, or does the Inferact workload's larger session count (50 vs 52 there) but very different per-session size distribution (mean 33 turns × ~2KB context growth per turn) push it lower?
|
||
- Is `D2 = 0%` an E1-specific artifact (kv-aware sticky in pd-disagg mode), or does the same happen in E2 before migration kicks in for the first time?
|