v2 exp(d): 5-policy routing under tracets vs thinktime — ranking flip
Extends exp(c) (dispatch ablation, 1 round-robin policy) to the full 5-policy routing comparison, both modes on the SAME ttp trace (807 reqs, fresh vLLM/arm, dash0 8xH20). Confirms exp(c)'s prediction and finds something stronger: the dispatch mode FLIPS which policy wins. - thinktime helps every policy but helps LPWL most (TTFT p90 -40%, E2E mean -31% vs -3..-16% for the rest): tracets bursts punish prefill-spreading. - Ranking flip: tracets -> LPWL only ties unified_ab on TTFT p90 and is 3rd on E2E mean; thinktime -> LPWL is 1st on both (TTFT p90 -31%, best TPOT/balance, zero knobs) vs the tuned unified+A+B. - => benchmark agentic routing with thinktime; tracets' burst artifact erases LPWL's advantage. Caveat n=1: tracets ranking is run-sensitive (does not reproduce dash1 lpwl_5policy_600s.md), the thinktime advantage is the robust signal (appears in both environments). README + grouped-bar fig (figs/exp_d_policy_dispatch.png) + bench_report summaries in results/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
114
v2/exp_d_policy_dispatch/README.md
Normal file
114
v2/exp_d_policy_dispatch/README.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# exp (d) — 5-policy routing under `tracets` vs `thinktime`
|
||||
|
||||
exp (c) showed the **dispatch mode** changes measured performance for a single
|
||||
round-robin policy, and predicted: *"a cache-aware policy (LPWL) would lower the
|
||||
latencies and likely **widen** the thinktime advantage."* exp (d) tests that with
|
||||
the full routing comparison — and finds something stronger: **the dispatch mode
|
||||
flips which policy wins.**
|
||||
|
||||
**Question.** Does the parameter-free LPWL still beat the tuned `unified+A+B`
|
||||
baseline once we benchmark with the *faithful* `thinktime` load instead of the
|
||||
`tracets` burst artifact?
|
||||
|
||||
## Setup
|
||||
|
||||
5 routing policies, each its own **fresh vLLM (cold APC)** on dash0 8×H20,
|
||||
Qwen3-Coder-30B-A3B, via `scripts/b3_isolated_policy.sh`. **Both dispatch modes
|
||||
run on the *same* trace** `traces/w600_r0.0015_st30_first600s_ttp.jsonl` (807
|
||||
reqs, 274 sessions) — the only variable is `REPLAY_DISPATCH_MODE`
|
||||
(`tracets` ignores the `time_to_parent_chat` field, `thinktime` consumes it).
|
||||
Analyzer: `scripts/bench_report.py` (summaries in `results/`).
|
||||
|
||||
- `leastwork` — **LPWL**, parameter-free (`pending_prefill + max(0, input−cache_hit)`)
|
||||
- `unified_ab` — unified hybrid, tuned A+B′ (`of=1.3, lmw=0.01`)
|
||||
- `unified_def` — unified hybrid, defaults (`of=2.0, lmw=0.0`)
|
||||
- `lmetric` — P_tokens × BS, no affinity
|
||||
- `sticky` — hard session affinity
|
||||
|
||||
## Result (ms; `figs/exp_d_policy_dispatch.png`)
|
||||
|
||||
| policy | mode | TTFT p90 | E2E mean | E2E p90 | E2E p99 | TPOT p90 | APC | req-bal |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| **LPWL** | tracets | 11099 | 9827 | 25366 | 93929 | 33 | 0.650 | **1.49×** |
|
||||
| **LPWL** | **thinktime** | **6713** | **6788** | **17635** | 69946 | **18** | 0.676 | 1.94× |
|
||||
| unified+A+B | tracets | 10783 | 8531 | 22063 | 75419 | 21 | 0.667 | 1.54× |
|
||||
| unified+A+B | thinktime | 9736 | 7131 | 18690 | **63788** | 19 | 0.676 | 2.16× |
|
||||
| unified default | tracets | 12997 | 8366 | 22819 | 82257 | 20 | 0.693 | 1.56× |
|
||||
| unified default | thinktime | 11268 | 7975 | 24096 | 72334 | 22 | 0.693 | 2.91× |
|
||||
| LMetric | tracets | 16492 | 10775 | 27791 | 99231 | 39 | 0.495 | 2.19× |
|
||||
| LMetric | thinktime | 15607 | 9902 | 27819 | 73672 | 30 | 0.483 | 2.10× |
|
||||
| sticky | tracets | 15236 | 10139 | 27974 | 82362 | 31 | 0.693 | 2.06× |
|
||||
| sticky | thinktime | 14838 | 8663 | 24966 | 70933 | 24 | 0.694 | 2.48× |
|
||||
|
||||
### Finding 1 — `thinktime` helps every policy, but helps **LPWL the most**
|
||||
|
||||
Per-policy `tracets`→`thinktime` change (negative = thinktime better):
|
||||
|
||||
| policy | ΔTTFT p90 | ΔE2E mean | ΔTPOT p90 |
|
||||
|---|---:|---:|---:|
|
||||
| **LPWL** | **−40%** | **−31%** | **−45%** |
|
||||
| unified+A+B | −10% | −16% | −10% |
|
||||
| unified default | −13% | −5% | +10% |
|
||||
| LMetric | −5% | −8% | −23% |
|
||||
| sticky | −3% | −15% | −23% |
|
||||
|
||||
`tracets` collapses the inter-turn think-time to ~0 (exp c), manufacturing bursts
|
||||
→ peak concurrency → KV pressure → preemption. Those bursts punish exactly the
|
||||
policy that spreads prefill thinly across hosts (LPWL keeps the tightest request
|
||||
balance, 1.49×), because under a burst the spread sacrifices locality without the
|
||||
slack to amortize it. Remove the artifact and LPWL's prefill-aware placement pays.
|
||||
|
||||
### Finding 2 — the dispatch mode **flips the cross-policy ranking**
|
||||
|
||||
- **TTFT p90:** `tracets` → `unified_ab (10.8s) ≈ LPWL (11.1s)` — LPWL only *ties*,
|
||||
even slightly behind. `thinktime` → **LPWL (6.7s)** < unified_ab (9.7s): LPWL is
|
||||
first, **−31%** vs the tuned baseline.
|
||||
- **E2E mean:** `tracets` → unified_def (8.4s) < unified_ab (8.5s) < **LPWL (9.8s)**
|
||||
— LPWL is *3rd, behind both unified variants*. `thinktime` → **LPWL (6.8s)** <
|
||||
unified_ab (7.1s) < unified_def (8.0s): LPWL is **first**.
|
||||
|
||||
So under artificial `tracets` bursts the parameter-free policy looks tied-or-worse;
|
||||
under the faithful `thinktime` load it is the clear winner on TTFT and E2E, at
|
||||
zero knobs and best balance.
|
||||
|
||||
## Conclusion
|
||||
|
||||
**Benchmark agentic routing with `thinktime`. Under it, the parameter-free LPWL is
|
||||
the best of the five policies** — TTFT p90 −31%, E2E mean −5% / p90 −6%, best TPOT,
|
||||
tightest balance vs the *tuned* `unified+A+B` — and the `tracets` burst artifact is
|
||||
precisely what erases that advantage (it even drops LPWL to 3rd on E2E). This both
|
||||
confirms exp (c)'s prediction and is independent evidence for the GPU-hit-first
|
||||
routing story: faithful load rewards keeping the active working set GPU-resident.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **n = 1 per arm.** The `tracets` ranking here does **not** reproduce the earlier
|
||||
dash1 `analysis/lpwl_5policy_600s.md` (which saw LPWL win TTFT p90 −31% *in
|
||||
tracets*); on dash0 `tracets` it is a tie. i.e. **`tracets` rankings are
|
||||
run/harness-sensitive** — the robust signal is the `thinktime` advantage, which
|
||||
appears in *both* environments. Repeat ×3 to bound noise.
|
||||
- LPWL's one persistent weak spot is **E2E p99** (thinktime 69.9s vs unified_ab
|
||||
63.8s) — the structural HEAVY+ >50k decode tail, identical across policies, not
|
||||
routing-fixable (see `lpwl_5policy_600s.md` κ-ablation).
|
||||
- `thinktime` advantage is a capacity-slack effect; under saturation the modes
|
||||
converge (exp c, N=6).
|
||||
|
||||
## Repro
|
||||
```bash
|
||||
# 1. annotate the full trace with time_to_parent_chat (dash0; once)
|
||||
python scripts/add_ttp_streaming.py 051315-051317.jsonl 051315-051317-ttp.jsonl \
|
||||
051315-051317-raw.jsonl
|
||||
# 2. resample (same seed reproduces traces/w600_r0.0015_st30.jsonl + the ttp field;
|
||||
# first600s = timestamp<600 filter)
|
||||
python scripts/sample_trace.py --input 051315-051317-ttp.jsonl \
|
||||
--output traces/w600_r0.0015_st30_ttp.jsonl \
|
||||
--window-seconds 600 --sample-ratio 0.0015 --max-single-turn-ratio 0.30 --seed 42
|
||||
# 3. run both modes x 5 policies (~3.5 h, fresh vLLM/arm)
|
||||
TRACE_FILE=traces/w600_r0.0015_st30_first600s_ttp.jsonl \
|
||||
bash microbench/connector_tax/cache_sweep/run_5policy_both_modes.sh
|
||||
# 4. report + plot
|
||||
python scripts/bench_report.py --root outputs/policy5_600s_thinktime_<date> \
|
||||
--json v2/exp_d_policy_dispatch/results/thinktime.json \
|
||||
leastwork unified_ab unified_def lmetric sticky
|
||||
python v2/exp_d_policy_dispatch/plot.py
|
||||
```
|
||||
Reference in New Issue
Block a user