From 2b9eae0d54eccb48c2f2d78fc8ba54dbb82d069b Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sun, 24 May 2026 03:15:32 +0800
Subject: [PATCH] =?UTF-8?q?Report=20=C2=A73.9:=20Unified=20routing=20final?=
 =?UTF-8?q?=20results=20=E2=80=94=20TTFT=20-25%,=20E2E=20-7%?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

850/850, 0 errors. Single argmin(latency) with soft affinity.
116 PUSH_MIGRATE (all with cache, avg 25k tokens), 723 LOCAL.
TPOT p90 +15% tradeoff from kv_both overhead.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 REPORT.md | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/REPORT.md b/REPORT.md
index deb60bb..d6bdf14 100644
--- a/REPORT.md
+++ b/REPORT.md
@@ -356,6 +356,38 @@ The elastic numbers on dash1 were genuinely fresh. The "improvement" was actuall
 
 **Output**: `outputs/eval_direct_rdma_v*/` on dash0.
 
+### 3.9 Unified Routing (Final Design)
+
+Replaced two-phase routing (pick_instance → offload gate) with single `argmin(expected_latency)` per instance:
+
+```
+latency(D) = queue(D) + prefill(D) + transfer(D)
+  - Local cache: prefill = (input - local_hit) / throughput, transfer = 0
+  - PUSH from C: prefill = (input - push_hit) / throughput, transfer = 0.1s
+  - Cold:        prefill = input / throughput, transfer = 0
+```
+
+Session affinity as soft preference: use last instance if its cost ≤ 2× global best.
+
+Only 2 measured parameters: `prefill_throughput=7000 tok/s`, `rdma_overhead=0.1s`.
+
+**Results (eval_unified_v3, 850/850, 0 errors):**
+
+| Metric | Baseline | **Unified v3** | Delta |
+|--------|----------|---------------|-------|
+| TTFT mean | 4.35s | **3.24s** | **-25.5%** |
+| TTFT p50 | 0.95s | **0.78s** | **-17.9%** |
+| TTFT p90 | 12.47s | **7.79s** | **-37.5%** |
+| TPOT p90 | 0.177 | 0.204 | +14.9% |
+| E2E mean | 19.10s | **17.69s** | **-7.4%** |
+| E2E p50 | 6.44s | **5.48s** | **-14.9%** |
+
+**Routing**: 723 LOCAL + 116 PUSH_MIGRATE (13.8%). All 116 pushes had cache (avg 25k tokens) — no cold offloads. The unified cost model naturally avoids cold migration because `cold + RDMA > cold` (RDMA adds overhead without reducing prefill).
+
+**Tradeoff**: TPOT p90 +15% from kv_both background threads + PUSH operations. In exchange: TTFT -38%, E2E -15% at p50.
+
+**Output**: `outputs/eval_unified_v3/` on dash0.
+
 ## 4. System-Level Analysis
 
 ### 4.1 Elastic P2P Does Not Improve Single-Machine Performance