TP=2 DP=4 + hybrid routing: best TTFT at cost of TPOT

TP=2 DP=4 with hybrid routing achieves TTFT p50=0.611s (-43% vs TP=1),
the best TTFT across all tested configurations. But TPOT p90=0.109s
(+51% vs TP=1) due to cross-GPU all-reduce in decode.

Full comparison across 7 configurations shows two Pareto-optimal points:
  TP=1 DP=8 hybrid: best TPOT (0.072s), good TTFT (1.064s)
  TP=2 DP=4 hybrid: best TTFT (0.611s), acceptable TPOT (0.109s)

The choice depends on SLO:
  TTFT-sensitive (interactive) -> TP=2 DP=4
  TPOT-sensitive (streaming)   -> TP=1 DP=8

All PD-Sep configurations are strictly dominated by one of these two.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

scripts/final_all_comparison.py

@@ -0,0 +1,47 @@
+"""Final comparison across ALL tested configurations."""
+import json, csv, statistics, os
+
+def lat(path):
+    rows = [json.loads(l) for l in open(path)]
+    ok = [r for r in rows if not r.get("error")]
+    ttfts = sorted([r["ttft_s"] for r in ok if r.get("ttft_s")])
+    tpots = sorted([r["tpot_s"] for r in ok if r.get("tpot_s") and r["tpot_s"] > 0])
+    lats = sorted([r["latency_s"] for r in ok])
+    p = lambda v, q: v[min(int(q * len(v)), len(v) - 1)] if v else 0
+    return len(ok), len(rows), p(ttfts,.5), p(ttfts,.9), p(tpots,.5), p(tpots,.9), p(lats,.5)
+
+def gpu(path):
+    rows = list(csv.DictReader(open(path)))
+    vals = [float(r["util_pct"]) for r in rows]
+    return statistics.fmean(vals) if vals else 0
+
+print("COMPLETE CONFIGURATION COMPARISON")
+print("All use same 1000-req trace (200 req for GPU tests), 8xH20")
+print("=" * 85)
+fmt = "%-35s %6s %8s %8s %8s %8s %8s"
+print(fmt % ("Config", "OK/N", "TTFT50", "TTFT90", "TPOT50", "TPOT90", "E2E50"))
+print("-" * 85)
+
+configs = [
+    ("gpu_ab_combined", "TP=1 DP=8 old cache-aware"),
+    ("gpu_ab_hybrid", "TP=1 DP=8 hybrid routing"),
+    ("tp2dp4_hybrid", "TP=2 DP=4 hybrid routing  *** NEW"),
+    ("gpu_ab_pdsep", "TP=1 PD-Sep 4P+4D"),
+    ("gpu_ab_6p2d", "TP=1 PD-Sep 6P+2D"),
+    ("adaptive_v2_baseline", "TP=1 kv_both baseline"),
+    ("adaptive_v2_offload", "TP=1 adaptive offload"),
+]
+
+for d, label in configs:
+    mp = "outputs/%s/metrics.jsonl" % d
+    if not os.path.exists(mp):
+        continue
+    ok, n, t50, t90, p50, p90, e50 = lat(mp)
+    print(fmt % (label, "%d/%d" % (ok, n),
+                 "%.3f" % t50, "%.3f" % t90, "%.3f" % p50, "%.3f" % p90, "%.3f" % e50))
+
+print()
+print("KEY TRADEOFF: TP=1 vs TP=2")
+print("  TP=1 DP=8: Better TPOT (0.072), more instances for routing diversity")
+print("  TP=2 DP=4: Better TTFT (0.565), faster prefill, larger KV cache per inst")
+print("  Which is better depends on SLO: TTFT-sensitive -> TP=2, TPOT-sensitive -> TP=1")