Agentic workload PD separation analysis with trace-driven benchmarks

Systematic study of prefill-decode disaggregation for agentic LLM workloads
using production GLM-5.1 coder trace (2.1M requests, 71B input tokens).

Key findings:
- Cache-aware routing improves TPOT p90 by 15% and APC from 20.8% to 44.7%
  without PD separation, matching PD-Sep's decode isolation benefit
- PD separation adds +72% TTFT overhead (KV transfer) with no TPOT gain
  when using the same cache-aware scheduler
- Prefill remains compute-bound even at 95% KV cache reuse (AI >1000x
  vs decode AI <2), but absolute FLOPs drop 71% from cache hits
- For agentic MoE workloads, cache-aware routing > PD separation

Infrastructure:
- Trace sampler preserving session structure + hash_ids for prefix sharing
- Async trace replayer with streaming TTFT/TPOT/E2E measurement
- Unified cache-aware + token-level load-balanced global scheduler proxy
  supporting both PD-colocated and PD-disaggregated (Mooncake/RDMA) modes
- vLLM 0.18.1 scheduler patch for KV transfer abort race condition
- Roofline analysis tool for prefill/decode compute characterization

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

replayer/trace.py

@@ -0,0 +1,84 @@
+"""Trace data structures and loader for the Ali agentic-coder trace format.
+
+Trace format (one JSON per line):
+  chat_id, parent_chat_id, timestamp, input_length, output_length,
+  type, turn, hash_ids[]
+
+Sessions are derived from parent_chat_id chains:
+  - parent_chat_id == -1  →  new session root
+  - parent_chat_id >= 0   →  belongs to the same session as the parent
+"""
+
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class TraceRequest:
+    request_id: str
+    session_id: str
+    chat_id: int
+    parent_chat_id: int
+    timestamp_s: float
+    input_length: int
+    output_length: int
+    request_type: str
+    turn_id: int
+    hash_ids: tuple[int, ...]
+
+
+def load_trace(
+    path: Path,
+    *,
+    request_limit: int | None = None,
+) -> list[TraceRequest]:
+    """Load trace and resolve session IDs from parent_chat_id chains."""
+    chat_to_session: dict[int, str] = {}
+    requests: list[TraceRequest] = []
+
+    with path.open("r", encoding="utf-8") as fh:
+        for idx, line in enumerate(fh):
+            if request_limit is not None and len(requests) >= request_limit:
+                break
+            row = json.loads(line)
+            chat_id = int(row["chat_id"])
+            parent_chat_id = int(row["parent_chat_id"])
+
+            if "session_id" in row:
+                session_id = str(row["session_id"])
+            else:
+                session_id = _resolve_session_id(
+                    chat_id, parent_chat_id, chat_to_session,
+                )
+            chat_to_session[chat_id] = session_id
+
+            requests.append(TraceRequest(
+                request_id=f"{session_id}:{row['turn']}:{chat_id}:{idx}",
+                session_id=session_id,
+                chat_id=chat_id,
+                parent_chat_id=parent_chat_id,
+                timestamp_s=float(row["timestamp"]),
+                input_length=int(row["input_length"]),
+                output_length=int(row["output_length"]),
+                request_type=str(row["type"]),
+                turn_id=int(row["turn"]),
+                hash_ids=tuple(int(h) for h in row.get("hash_ids", [])),
+            ))
+
+    return requests
+
+
+def _resolve_session_id(
+    chat_id: int,
+    parent_chat_id: int,
+    chat_to_session: dict[int, str],
+) -> str:
+    if parent_chat_id < 0:
+        session_id = str(chat_id)
+    else:
+        session_id = chat_to_session.get(parent_chat_id, str(parent_chat_id))
+    chat_to_session[chat_id] = session_id
+    return session_id