# Migration Trigger Validation (unified_v4) — 2026-05-30

Hardware: dash2, 8×H20, Qwen3-Coder-30B-A3B, TP=1, kv_both + DR-fix substrate.
Trace: `w600_r0.0015_st30_first600s.jsonl` (807 reqs, 600s span).
Policy: `unified_v4` = unified hybrid routing + pending-prefill-queue-triggered
session migration (commit `3a6bf5d` on `kzlin-dev` branch).

## Research Question

Does Pillar 2 (hot-triggered session migration) provide measurable benefit on
top of Pillar 1 (affinity-default routing)?

## Experiment Design

| Arm | Policy | Substrate | Trace QPS |
|---|---|---|---|
| unified_1x | unified (affinity-only) | kv_both + DR-fix | ~1.3 (original) |
| unified_v4_1x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~1.3 |
| unified_v4_2x | unified_v4 (affinity + migration) | kv_both + DR-fix | ~2.7 (2× compressed) |

The 2× trace was generated by halving inter-request intervals:
`ts_new = ts_min + (ts_orig - ts_min) / 2`.

## Results

### 1x QPS: unified vs unified_v4

| Metric | unified | unified_v4 | Delta |
|---|---:|---:|---:|
| OK/total | 807/807 | 807/807 | — |
| TTFT mean | 3.990s | 4.142s | +3.8% |
| TTFT p50 | 0.719s | 0.711s | −1.0% |
| TTFT p90 | 11.499s | 12.293s | +6.9% |
| TPOT p90 | 0.024s | 0.022s | −9.3% |
| E2E p50 | 2.265s | 2.293s | +1.2% |
| E2E p90 | 24.507s | 23.955s | −2.3% |
| **Migrations** | **0** | **0** | — |

**Conclusion**: At 1x QPS (~1.3 req/s, ~0.16 req/instance/s), the migration
trigger NEVER fires. The two arms produce statistically identical results.

### 2x QPS: unified_v4 under higher load

| Metric | unified_v4 @ 2x |
|---|---:|
| OK/total | 807/807 |
| TTFT mean | 5.227s |
| TTFT p90 | 15.000s |
| E2E p90 | 39.401s |
| **Migrations** | **4/807 (0.5%)** |

4 migrated requests (all verified via `v3_decode_target_url` in breakdown):

| Session | Input | new_local | src_pp | fleet_median | proj_prefill | Target |
|---|---:|---:|---:|---:|---:|---|
| 1313181 | 22,686 | 22,686 | 13,360 | 6,680 | 5.1s | inst_5 |
| 1310590 | 32,440 | 14,520 | 57,051 | 12,630 | 10.2s | inst_4 |
| 1373431 | 126,340 | 126,340 | 73,385 | 33,294 | 28.5s | inst_4 |
| 1313181 | 60,004 | 17,508 | 19,503 | 3,806 | 5.3s | inst_5 |

## Root Cause Analysis: Why Zero Migrations at 1x

The unified_v4 trigger requires BOTH arms to fire simultaneously:
- **Absolute SLO arm**: `proj_prefill_s(src) > 2.5s` — fires for 41% of eligible requests
- **Relative arm**: `src_pending_prefill > fleet_median × 1.5` — NEVER fires at 1x

The relative arm fails because `pending_prefill_tokens` (the proxy's shadow
counter) is 0 for **95% of routing decisions** at 1x QPS:

| QPS | src_pp > 0 (% of eligible) | Migrations |
|---:|---:|---:|
| 1.3 (1×) | 5% (8/241) | 0 |
| 2.7 (2×) | 24% (62/257) | 4 |

**Mechanism**: `pending_prefill_tokens` reflects previously-dispatched requests
that haven't finished their prefill yet. At 0.16 req/instance/s, each instance
completes its prefill before the next request arrives — the counter is almost
always 0 at decision time. Only under genuine queueing pressure (2× and above)
does the counter become non-zero and the relative arm can fire.

The high TTFT at 1x (~11.5s p90) comes from **compute-bound large prefills**
(single 60k+ token requests inherently need ~9s), NOT from queue depth.

## Interpretation for Paper

1. **The migration mechanism is functionally correct.** At 2x it fires on the
   right signal (src genuinely overloaded relative to fleet) and selects valid
   targets (cooler instances with load gap).

2. **At benchmark scale (8 instances, ~1 QPS), migration is not needed.** The
   affinity-default routing (Pillar 1) already achieves APC ~79% and the
   remaining hot-pin issue is mild (max-worker/median-worker ≈ 3.7×). The
   "dispatch coupling" feedback loop is present but not yet at the catastrophic
   amplification regime.

3. **Migration becomes relevant under scale-out + higher load.** With more
   instances (16–32), session skew concentrates more load per hot instance
   while cold instances sit idle — exactly the condition where `src_pp >
   fleet_median × 1.5` naturally fires. The 1x→2x progression (0%→0.5%
   migration rate) shows the correct scaling direction.

4. **Paper §3.3 framing**: Migration is a **scale-out insurance mechanism**
   that gracefully degrades to no-op under low load. Its value is NOT
   demonstrable at 8-instance single-node benchmark; the argument must rely on
   (a) the mechanism's correctness (this experiment), (b) the substrate's
   net-positive property (commit `ef9e010`), and (c) scale-out projection
   (future: 16+ GPU, multi-node).

## Next Steps

- [ ] **Scale-out validation** (16 GPU, 2 nodes): With more instances and the
  same trace, more sessions compete per-instance → higher pending_prefill →
  migration triggers naturally. This is the strongest evidence path.
- [ ] **3–4× QPS on 8 instances**: Push to saturation to measure migration's
  effect in the catastrophic regime. Risk: may exceed serving capacity (errors).
- [ ] **Threshold sensitivity**: Ablate `v4_rel_hi` (1.5→1.2→1.0) and
  `v4_ttft_slo_s` (2.5→1.5→1.0) to characterize the trigger landscape.

## Reproduction

```bash
# On dash2 (local /tmp, does NOT modify shared NAS):
# 1x QPS
bash /tmp/migration_exp/run_migration_ab.sh   # interleaved unified vs unified_v4

# 2x QPS
python3 -c "
import json
trace_in = '/home/admin/cpfs/wjh/agentic-kv/traces/w600_r0.0015_st30_first600s.jsonl'
rows = [json.loads(l) for l in open(trace_in)]
ts_min = min(r['timestamp'] for r in rows)
for r in rows: r['timestamp'] = ts_min + (r['timestamp'] - ts_min) / 2.0
with open('/tmp/migration_exp/trace_2x.jsonl', 'w') as f:
    for r in rows: f.write(json.dumps(r) + '\n')
"
bash /tmp/migration_exp/run_2x.sh
```

## Data Locations (dash2 /tmp, ephemeral)

| Path | Content |
|---|---|
| `/tmp/migration_exp/outputs/unified_run1/` | Baseline arm (1x) |
| `/tmp/migration_exp/outputs/unified_v4_run1/` | Migration arm (1x) |
| `/tmp/migration_exp/outputs/unified_v4_2x/` | Migration arm (2x) |
| `/tmp/migration_exp/outputs/*/breakdown.json` | Per-request routing decisions with v4_* fields |
| `/tmp/migration_exp/outputs/*/metrics.jsonl` | Per-request latency metrics |