# Engine-state ablation: real-time state vs router shadow counters

**Question.** The router (`cache_aware_proxy`) routes on **shadow counters** it
maintains itself (incremented at dispatch, reconciled to vLLM `/metrics` only
every 30 s → stale). Does feeding it **real** per-engine state (running/waiting,
KV-used, pending-prefill, `max_prefill_remaining`) change routing decisions,
performance, or the policy ranking?

**Setup.** dash1, 8×H20 (TP=1), Qwen3-Coder-30B-A3B, trace
`w600_r0.0015_st30.jsonl` (1214 reqs / 274 sessions). Each policy run as a
matched pair: `es0` (shadow only) vs `es1` (real-state feed via
`file:///dev/shm/...`, published ~50 ms by a scheduler daemon thread, read by
the proxy via `eff_* = max(shadow, real)`). Only the state source differs.
Driver: `run_full_ablation.sh`; per-cell freshness via `fresh_sampler.py`;
comparison via `cmp_es.py`.

## Result — real-time state is NOT the routing lever

It reshuffles 44–76% of routing decisions but **never beats the champion**;
the cache-affinity champion (`unified+A+B`, es0 p90 **7.62 s**) stays best.

| Policy | how it uses load | inst/session | reroute % | TTFT p90 es0→es1 | mean es0→es1 | verdict |
|---|---|--:|--:|--:|--:|---|
| `sticky` | **once at session birth, then pinned** | 1.00 | 44.5% | 13.42 → **9.95 (−26%)** | 4.13→3.65 | **HELPS** |
| `unified+A+B` | per-req, affinity-dominated | 1.22 | 76.4% | 7.62 → 7.76 (+1.8%) | 3.20→3.24 | wash |
| `v3_AB_lw` | per-req, affinity-dom + migration | ~1.2 | 71.7% | 9.35 → 9.49 (+1.5%) | 3.34→3.58 | wash* |
| `unified_kv_both` | per-req, affinity-dom (same picker) | ~1.2 | 73.6% | 6.45 → 9.28 (+44%) | 3.07→3.49 | worse† |
| `lmetric` | per-req, load×batch | 2.04 | 73.4% | 15.63 → 18.23 (+16.6%) | 5.18→5.80 | HURTS |
| `load_only` | per-req, pure load | 2.22 | 72.7% | 21.79 → 27.69 (+27%) | 6.65→8.42 | HURTS |

\* v3 real-state migration targeting backfired: migrations 26→32, migrated-req
mean TTFT 11.99→18.45 s (+54%). Real state does not rescue migration.
† same picker as `unified`; the 1.8%-vs-44% spread is run-variance (single
pairs) in which reshuffled routes hit hotspots — sign is consistently ≥ neutral.

## Mechanism — the sign is set by reactivity, not "affinity vs not"

- **One-shot placement (`sticky`) → HELPS.** `pick_instance_sticky` is *not* a
  stateless hash: the first turn picks `min(eff_num_requests())` (load), then
  `affinity[session]` pins it for all later turns. State enters at exactly one
  decision per session; real load → better placement that compounds across the
  session, locality preserved, no per-request oscillation.
- **Per-request, affinity-dominated (`unified`/`v3`/`kv_both`) → wash-to-worse.**
  The hybrid picker mostly obeys affinity; only the ~12% fallback fraction
  consults load. Net 0…+44%, never helps.
- **Per-request, pure load (`lmetric`/`load_only`) → HURTS, monotonic in
  load-purity.** Routing on *instantaneous* load induces **herding** (everyone
  piles onto whatever momentarily looks idle → transient overload → tail
  inflation); the stale shadow counter was inadvertently a **dampener**.

## Why the result is trustworthy (not a stale-feed artifact)
The feed was fresh on every es1 cell: age median **25 ms**, **≤92 ms even
during 100k-token prefills**, <0.5 % of samples >2 s stale (and those not
during prefills → reader drops them → shadow fallback). The feared GIL-
starvation of the publisher during big prefills did **not** materialize.

## Implications
1. Don't invest in real-time state for per-request routing — it never wins and
   degrades load-driven policies up to +27 %.
2. The cache-affinity champion is robust to state source; A+B+RaceFix already
   handled the staleness that mattered.
3. **Design insight:** the only place ground-truth state helps is **one-shot
   session placement** (decide well once on real load, then commit) — not
   per-request load polling.
4. All prior shadow-state results stand; the router's approximate state was
   never the bottleneck. Workload skew + affinity discipline are.

## Reproduce
```bash
# per-cell: same proxy, ES=0 (shadow) vs ES=1 (real); see run_v3_trace.sh
MODE=baseline POLICY=unified AB_FLAGS="--overload-factor 1.3 --lmetric-decode-weight 0.01" \
  ES=1 TAG=unified_AB_es1 bash run_v3_trace.sh
# full sweep (waits for the champion es1 marker, then runs the rest):
bash run_full_ablation.sh
# compare a pair:
python cmp_es.py <es0_dir>/unified_v3 <es1_dir>/unified_v3 abl_<tag>_es1.freshness.jsonl
```