agentic-kvc/microbench/connector_tax/layerwise/P2_ENGINE_STATE.md

P2: real engine-state feed for migration target selection

Problem: the router (cache_aware_proxy.py) decides migration targets from shadow counters it maintains itself (incremented at dispatch, decremented at completion) and reconciles to vLLM /metrics only every 30 s (_reconcile_loop). So every routing/migration decision is on stale state. Worse, the signal that predicts the ~45% control-plane stall — is the target mid-large-prefill? (a big prefill holds the GIL and starves the mooncake receiver_loop) — isn't visible at all, and /metrics doesn't expose it either.

Fix: vLLM publishes real per-engine state to a shared store ~20 Hz; the router reads ground truth and avoids GIL-stall / capacity-wall targets.

Components (all unit-tested without GPUs)

• engine_state.py — canonical compute_snapshot(scheduler, id), StateWriter, StateReader. Schema per engine: ts, num_running, num_waiting, gpu_blocks_total/free, gpu_kv_used_frac, pending_prefill_tokens, ongoing_decode_tokens, num_prefilling, max_prefill_remaining.
• instrument_engine_state.py — vLLM Scheduler patch (apply/revert markers ES_INSTRUMENT_*): a daemon thread publishes the snapshot every AGENTIC_ENGINE_STATE_PERIOD_MS (50 ms) off the forward hot path. Inlined writer (engine process needs no repo import). Coexists with MB5.
• migration_target.py — pure target scorer: avoid max_prefill_remaining ≥ es_big_prefill_threshold (GIL stall) and gpu_kv_used_frac ≥ es_kv_wall_frac (capacity wall), then rank by cache-richness and real load.
• cache_aware_proxy.WRITEMODE.py — wired: InstanceState.real_state, _engine_state_poll_loop (instance i ← engine_{i}), _real_load/Gate-3 and Mechanism-B now real-state-aware. --engine-state-uri flag; off ⇒ identical to before (shadow only).

Transport (AGENTIC_ENGINE_STATE_URI / --engine-state-uri): file:///dev/shm/agentic_engine_state (default, zero-dep, single-node) or redis://host:port/0 (multi-node; needs redis-py + server — not installed on dash0, so file backend is the working default).

Tests (no GPU)

• compute_snapshot field math (mock scheduler): running/waiting, max_prefill_remaining, pending, decode, kv_used_frac.
• writer→reader round-trip + staleness drop (file backend).
• target scorer: 5 cases incl. avoid GIL-stall target even when its shadow load is lower, real load beats stale shadow, cache-rich wins, avoid KV wall, graceful fallback when feed missing.
• end-to-end: publish 8 engines (one mid-130k-prefill) → proxy inlined reader → target selection avoids it.

Enabling in a GPU run (when free)

1. instrument_engine_state.py --apply on the dash0 venv.
2. export AGENTIC_ENGINE_STATE_URI=file:///dev/shm/agentic_engine_state before the launcher (vLLM instances inherit it; AGENTIC_WORKER_ID=engine_{i} already set by b3_isolated_policy.sh → publishes as engine_{i}).
3. Proxy: EXTRA_PROXY_ARGS="--engine-state-uri file:///dev/shm/agentic_engine_state ...".
4. Revert the patch + rm -rf /dev/shm/agentic_engine_state after.

Status / scope

• Built + unit-tested; NOT yet run against live engines (GPU busy).
• Scoped to migration target selection (the P2 ask). The same real-load signal could also de-stale the base pick_instance_unified_hybrid LMetric fallback (the 8007-hotspot class from UNIFIED_ABLATION) — follow-up.
• TP=1 only (one EngineCore/instance → one publisher/engine_id). TP>1 needs per-rank ids.