agentic-kvc/microbench/connector_tax/layerwise/ES_ABLATION_RESULTS.md

Engine-state ablation: real-time state vs router shadow counters

Question. The router (cache_aware_proxy) routes on shadow counters it maintains itself (incremented at dispatch, reconciled to vLLM /metrics only every 30 s → stale). Does feeding it real per-engine state (running/waiting, KV-used, pending-prefill, max_prefill_remaining) change routing decisions, performance, or the policy ranking?

Setup. dash1, 8×H20 (TP=1), Qwen3-Coder-30B-A3B, trace w600_r0.0015_st30.jsonl (1214 reqs / 274 sessions). Each policy run as a matched pair: es0 (shadow only) vs es1 (real-state feed via file:///dev/shm/..., published ~50 ms by a scheduler daemon thread, read by the proxy via eff_* = max(shadow, real)). Only the state source differs. Driver: run_full_ablation.sh; per-cell freshness via fresh_sampler.py; comparison via cmp_es.py.

Result — real-time state is NOT the routing lever

It reshuffles 44–76% of routing decisions but never beats the champion; the cache-affinity champion (unified+A+B, es0 p90 7.62 s) stays best.

Policy	how it uses load	inst/session	reroute %	TTFT p90 es0→es1	mean es0→es1	verdict
sticky	once at session birth, then pinned	1.00	44.5%	13.42 → 9.95 (−26%)	4.13→3.65	HELPS
unified+A+B	per-req, affinity-dominated	1.22	76.4%	7.62 → 7.76 (+1.8%)	3.20→3.24	wash
v3_AB_lw	per-req, affinity-dom + migration	~1.2	71.7%	9.35 → 9.49 (+1.5%)	3.34→3.58	wash*
unified_kv_both	per-req, affinity-dom (same picker)	~1.2	73.6%	6.45 → 9.28 (+44%)	3.07→3.49	worse†
lmetric	per-req, load×batch	2.04	73.4%	15.63 → 18.23 (+16.6%)	5.18→5.80	HURTS
load_only	per-req, pure load	2.22	72.7%	21.79 → 27.69 (+27%)	6.65→8.42	HURTS

* v3 real-state migration targeting backfired: migrations 26→32, migrated-req mean TTFT 11.99→18.45 s (+54%). Real state does not rescue migration. † same picker as unified; the 1.8%-vs-44% spread is run-variance (single pairs) in which reshuffled routes hit hotspots — sign is consistently ≥ neutral.

Mechanism — the sign is set by reactivity, not "affinity vs not"

• One-shot placement (sticky) → HELPS. pick_instance_sticky is not a stateless hash: the first turn picks min(eff_num_requests()) (load), then affinity[session] pins it for all later turns. State enters at exactly one decision per session; real load → better placement that compounds across the session, locality preserved, no per-request oscillation.
• Per-request, affinity-dominated (unified/v3/kv_both) → wash-to-worse. The hybrid picker mostly obeys affinity; only the ~12% fallback fraction consults load. Net 0…+44%, never helps.
• Per-request, pure load (lmetric/load_only) → HURTS, monotonic in load-purity. Routing on instantaneous load induces herding (everyone piles onto whatever momentarily looks idle → transient overload → tail inflation); the stale shadow counter was inadvertently a dampener.

Why the result is trustworthy (not a stale-feed artifact)

The feed was fresh on every es1 cell: age median 25 ms, ≤92 ms even during 100k-token prefills, <0.5 % of samples >2 s stale (and those not during prefills → reader drops them → shadow fallback). The feared GIL- starvation of the publisher during big prefills did not materialize.

Implications

1. Don't invest in real-time state for per-request routing — it never wins and degrades load-driven policies up to +27 %.
2. The cache-affinity champion is robust to state source; A+B+RaceFix already handled the staleness that mattered.
3. Design insight: the only place ground-truth state helps is one-shot session placement (decide well once on real load, then commit) — not per-request load polling.
4. All prior shadow-state results stand; the router's approximate state was never the bottleneck. Workload skew + affinity discipline are.

Reproduce

# per-cell: same proxy, ES=0 (shadow) vs ES=1 (real); see run_v3_trace.sh
MODE=baseline POLICY=unified AB_FLAGS="--overload-factor 1.3 --lmetric-decode-weight 0.01" \
  ES=1 TAG=unified_AB_es1 bash run_v3_trace.sh
# full sweep (waits for the champion es1 marker, then runs the rest):
bash run_full_ablation.sh
# compare a pair:
python cmp_es.py <es0_dir>/unified_v3 <es1_dir>/unified_v3 abl_<tag>_es1.freshness.jsonl