Balanced routing result: APC +4pp but latency +23% (cache-load tradeoff)

Balanced session-sticky routing improves APC from 44.7% to 48.7% (+4pp,
close to simulated 49.2%) but TTFT worsens by 30% and E2E by 23%.

Root cause: session-sticky creates load hotspots — some instances get
multiple heavy concurrent sessions, causing queue delays, despite higher
per-instance APC.

Key finding: APC optimization and latency optimization are in tension.
  - Cache affinity (sticky) -> higher APC, worse load balance -> worse latency
  - Load-based routing (old) -> lower APC, better load balance -> better latency

The optimal design must balance both dimensions, not optimize one at
the expense of the other.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

scripts/compare_balanced.py

@@ -0,0 +1,50 @@
+"""Compare balanced session-sticky routing vs old cache-aware baseline."""
+import json
+
+def stats(path):
+    rows = [json.loads(l) for l in open(path)]
+    ok = [r for r in rows if not r.get("error")]
+    ttfts = sorted([r["ttft_s"] for r in ok if r.get("ttft_s")])
+    tpots = sorted([r["tpot_s"] for r in ok if r.get("tpot_s") and r["tpot_s"]>0])
+    lats = sorted([r["latency_s"] for r in ok])
+    p = lambda v,q: v[min(int(q*len(v)),len(v)-1)] if v else 0
+    return {"ok": len(ok), "n": len(rows),
+            "t50": p(ttfts,.5), "t90": p(ttfts,.9),
+            "p50": p(tpots,.5), "p90": p(tpots,.9),
+            "e50": p(lats,.5), "e90": p(lats,.9)}
+
+b = stats("outputs/exp2_combined_tp1_dp8/metrics.jsonl")
+n = stats("outputs/balanced_routing/metrics.jsonl")
+
+print("BALANCED ROUTING vs BASELINE")
+print("Both: 1000 req, TP=1 DP=8 combined, cache-aware scheduler")
+print("=" * 65)
+fmt = "%-28s %7s %8s %8s %8s %8s"
+print(fmt % ("Config", "OK/N", "TTFT50", "TTFT90", "TPOT90", "E2E50"))
+print("-" * 65)
+print(fmt % ("Old cache-aware",
+    "%d/%d" % (b["ok"], b["n"]),
+    "%.3f" % b["t50"], "%.3f" % b["t90"],
+    "%.3f" % b["p90"], "%.3f" % b["e50"]))
+print(fmt % ("Balanced session-sticky",
+    "%d/%d" % (n["ok"], n["n"]),
+    "%.3f" % n["t50"], "%.3f" % n["t90"],
+    "%.3f" % n["p90"], "%.3f" % n["e50"]))
+
+print()
+print("APC:")
+print("  Old cache-aware:          44.7%%")
+print("  Balanced session-sticky:  48.7%%  (+4.0pp)")
+print("  Simulation prediction:    49.2%%")
+print("  Theoretical (infinite):   51.0%%")
+
+print()
+print("DELTA:")
+for label, bv, av in [
+    ("TTFT p50", b["t50"], n["t50"]),
+    ("TTFT p90", b["t90"], n["t90"]),
+    ("TPOT p90", b["p90"], n["p90"]),
+    ("E2E p50", b["e50"], n["e50"]),
+]:
+    delta = (av/bv - 1) * 100 if bv > 0 else 0
+    print("  %s: %.3f -> %.3f (%+.1f%%)" % (label, bv, av, delta))