Balanced session-sticky routing + agentic workload pattern analysis

Routing fix: new sessions placed by cumulative token load (greedy bin
packing) with cache-hit tiebreak. Session affinity for turn 2+.
Replayer now sends X-Session-Id header for proper session tracking.

Agentic workload core patterns (GLM-5.1 trace):
  - 91% of reusable KV is intra-session (not cross-session)
  - Session-sticky routing is THE critical optimization
  - 36% warm requests (1.3k new tokens), 64% cold (17k+)
  - After cache: effective prefill/decode ratio drops from 61.5x to 28.7x
  - Cross-session sharing (system prompt) is only 4.8% of tokens

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

scripts/cache_aware_proxy.py

@@ -63,25 +63,51 @@ class InstanceState:
             self.cached_blocks = set(list(self.cached_blocks)[-100000:])
 
 
+# Cumulative token load per instance (for balanced session placement)
+_inst_cumulative_tokens: list[int] = []
+
+
 def pick_instance(instances: list[InstanceState], token_ids: list[int] | None,
                   session_id: str | None, input_length: int,
                   affinity: dict[str, int]) -> tuple[InstanceState, int]:
-    """Normalized load - cache bonus scoring."""
+    """Session-sticky + KV-size balanced placement.
+
+    Turn 2+: session affinity (sticky to same instance for KV reuse).
+    Turn 1 (new session): place on instance with least cumulative token load
+    (greedy bin packing), with cache-hit tiebreak.
+    """
+    global _inst_cumulative_tokens
+    if not _inst_cumulative_tokens:
+        _inst_cumulative_tokens = [0] * len(instances)
+
+    # Session affinity for turn 2+
     if session_id and session_id in affinity:
         idx = affinity[session_id]
         if idx < len(instances):
             return instances[idx], idx
 
-    avg_load = max(sum(i.ongoing_tokens for i in instances) / len(instances), 1.0)
-    best_idx, best_score = 0, float("inf")
-    for i, inst in enumerate(instances):
-        cache_hit = inst.estimate_cache_hit(token_ids)
-        cache_ratio = cache_hit / input_length if input_length > 0 else 0.0
-        score = inst.ongoing_tokens / avg_load - CACHE_HIT_ALPHA * cache_ratio
-        if score < best_score:
-            best_score = score
+    # New session: balanced placement
+    # Primary: least cumulative tokens (long-term balance)
+    # Secondary: cache hit (tiebreak for prefix reuse)
+    min_load = min(_inst_cumulative_tokens)
+    # Candidates within 10% of min load
+    threshold = min_load + max(min_load * 0.1, 10000)
+    candidates = [i for i in range(len(instances))
+                  if _inst_cumulative_tokens[i] <= threshold]
+
+    if not candidates:
+        candidates = list(range(len(instances)))
+
+    # Among candidates, pick best cache hit
+    best_idx = candidates[0]
+    best_hit = 0
+    for i in candidates:
+        hit = instances[i].estimate_cache_hit(token_ids)
+        if hit > best_hit:
+            best_hit = hit
             best_idx = i
 
+    _inst_cumulative_tokens[best_idx] += input_length
     if session_id:
         affinity[session_id] = best_idx
     return instances[best_idx], best_idx