agentic-kvc/analysis/agentic_pd_unified_story_plan.md

# Agentic PD / Unified Routing Story Plan

Status: draft for review
Date: 2026-05-25

## 0. Goal

This document aligns three threads:

1. `agentic-kv`: vLLM-based PD-colocation, full PD separation, LMetric,
   Unified routing, and elastic migration experiments.
2. `dash0:/home/admin/cpfs/wjh/agentic-kv`: run artifacts and the
   latest PD-separation paper-section scaffold.
3. `~/phd/agentic-pd-hybrid`: SGLang/PPD/KVC experiments, including
   retractions and stricter framing around loadgen validity.

The purpose is to converge on a defensible story and a concrete task plan,
not to force the old Unified routing hypothesis to be true.

## 1. Current Best Framing

### 1.1 Workload premise

Agentic serving is not chatbot serving.

- Requests have long input contexts and short outputs.
- Most reusable KV is intra-session, not cross-session.
- Sessions are multi-turn and causally sequential: turn N+1 cannot be
  faithfully issued before turn N finishes.
- Long-lived sessions create two competing needs:
  - keep cache locality for future turns;
  - avoid pinning all future work to an overloaded worker.

This workload makes cache locality a first-order system objective, but
also makes naive session pinning dangerous.

### 1.2 System premise

PD separation is not a universally good abstraction. It helps only when:

```
saved decode interference > KV transfer + P queue + D queue + KV capacity cost
```

For agentic workloads, that inequality often fails because long-context KV
is large and decode-side KV residency becomes the limiting resource.

### 1.3 Main thesis candidate

Static PD separation is the wrong default for single-node agentic serving.
The stronger baseline is PD-colocation with cache-aware routing. The
interesting open problem is not "separate prefill and decode everywhere",
but:

> how to preserve session-level KV locality while retaining enough routing
> freedom to avoid hot-worker queueing and decode interference.

Unified routing should be framed as an attempt at that problem. The current
experiments show that the migration actuator was too expensive, so the
story should distinguish the principle from the failed mechanism.

## 2. What We Should Align Across Repos

### 2.1 Naming / architecture mapping

Use one taxonomy consistently:

| Name in paper/story | vLLM repo term | SGLang/KVC repo term | Meaning |
|---|---|---|---|
| Replica / PD-colo | combined / PD-colocated | `pd_colo`, SGLang `cache_aware` | all workers do prefill + decode |
| x=0 PD-disagg | full PD separation | `pd_disagg` | every turn goes P then D |
| x=1 / append-prefill-on-D | not implemented as such in vLLM experiments | KVC / PPD-style direct-to-D | turn 1 seeds D; later turns prefill locally on D |
| Elastic migration | Unified PUSH / elastic offload | smart migration / re-pin sessions | move a session or a request away from overloaded worker |
| Hybrid routing | current Unified baseline | PD-colo + soft pin / kv-aware | cache-aware LB plus explicit affinity only when worth it |

Important distinction: vLLM Unified PUSH migration is not the same as PPD
x=1. Unified PUSH still pays cross-instance KV movement for migrated
requests. PPD x=1 tries to avoid P-to-D transfer on later turns by doing
append-prefill directly on the resident D node.

### 2.2 Results that look stable

These are safe to build around:

1. Full/static PD separation is weak for agentic on one node.
   - vLLM evidence: decode-side KV memory wall and transfer overhead.
   - SGLang evidence: x=0 PD-disagg is consistently worse than PD-colo.

2. LMetric/cache-aware routing is a strong baseline.
   - In vLLM, corrected LMetric nearly matches session-sticky because
     `new_tokens = input - cached` creates implicit soft affinity.
   - In SGLang, `cache_aware` is the production-stable baseline and often
     wins or ties.

3. Explicit session pinning is not automatically good.
   - It can improve cache hit rate.
   - It can create head-of-line blocking if sessions grow unevenly.
   - Initial placement matters; mid-session migration is desirable in
     principle but hard in practice.

4. Transfer-based migration is currently too expensive.
   - vLLM experiments: forced/relaxed migration worsened E2E tail.
   - Mooncake path lacks enough overlap/layerwise benefit in current setup.

5. Loadgen validity must be treated as substrate, not detail.
   - `agentic-pd-hybrid` explicitly retracted high-concurrency claims
     where session sequentiality was violated.
   - Future experiments must enforce per-session causality.

### 2.3 Results that need more careful wording

1. "Unified beats LMetric" should not be stated as a strong result yet.
   The latest stable implementation is closer to LMetric plus a high-cache
   affinity gate. Expected gain is small by design.

2. "PD separation is always bad" is too broad.
   The correct claim is conditional: static/full PD separation is net
   negative in the long-context, high-KV-footprint, single-node agentic
   regime we measured.

3. "KVC/PPD wins" is not established for our stack.
   The SGLang repo contains useful PPD framing, but also several retractions:
   high-concurrency wins were affected by loadgen issues, and KVC stability
   was not production-ready in some runs.

4. "Session migration will fix load balance" is still a hypothesis.
   It is valid as a first-principles goal, but the tested vLLM actuator
   did not satisfy the cost budget.

## 3. Proposed Storytelling Outline

### Section A: Why agentic serving is different

Claim:

- Agentic workloads combine long contexts, high intra-session reuse, and
  sequential multi-turn sessions.
- This makes KV cache lifecycle and routing more important than the classic
  prefill/decode kernel dichotomy.

Evidence to use:

- Input/output token CDF.
- KV reuse decomposition: intra-session vs cross-session.
- Session length / context growth examples.
- Per-session sequentiality requirement.

Needed cleanup:

- Use one trace definition and report sampling method.
- Explicitly state whether a trace is online-realistic, synthetic burst,
  or stress test.

### Section B: Why static PD separation fails

Claim:

- The classic roofline premise is true but insufficient.
- Prefill can be compute-bound while static PD separation still loses at
  the system level.

Mechanism:

- PD separation relocates prefill; it does not reduce total prefill work.
- It adds KV transfer.
- It concentrates decode KV residency onto fewer D GPUs.
- Long-context agentic requests hit a decode-side KV memory wall.

Evidence to use:

- `analysis/pd_sep_paper_section/system_analysis.md`
- C1 workload figures.
- C6 roofline figure.
- KV memory wall model.
- Fresh PD matrix once rerun without forced eager mode.

Task implication:

- Complete C2/C3/C4/C5 matrix before making this a paper-grade section.

### Section C: Why cache-aware PD-colo is hard to beat

Claim:

- Cache-aware routing already captures much of the desired session affinity.
- LMetric's cache-adjusted prefill cost gives implicit soft affinity without
  hard pinning.

Mechanism:

- A worker with cached prefix has lower `new_tokens`.
- This naturally attracts later turns unless the worker is sufficiently
  loaded.
- This is exactly the balance we want: preserve locality while retaining
  routing freedom.

Evidence to use:

- Corrected LMetric vs Linear comparison.
- APC distribution.
- PD-colo stability from SGLang/KVC repo.

Task implication:

- Treat LMetric/cache-aware PD-colo as the primary baseline, not round-robin
  or naive sticky.

### Section D: Why Unified migration did not improve over LMetric

Claim:

- Unified's principle was right, but the migration mechanism failed the
  cost budget.

Mechanism:

- At conservative gates, too few requests migrate to change load balance.
- At relaxed gates, migration overhead dominates.
- Cold/heavy requests often cannot benefit from source cache and remain
  colocated.
- Cached migration still pays P-side queue, KV movement, and D admission.
- The cost model initially underestimated cache-attraction feedback and
  queue effects.

Evidence to use:

- Git history: single argmin -> soft affinity -> decode load/hard gate ->
  forced migration -> revert -> hybrid LMetric.
- Approach B / relaxed gate regressions.
- 16-session contention: interference exists, but elastic RDMA made TPOT
  worse and offloaded too few requests.

Task implication:

- Do not revive three-way argmin or aggressive PUSH migration.
- Frame current Unified as hybrid LMetric plus selective affinity.

### Section E: What remains promising

There are two different future paths. They should not be conflated.

Path 1: Conservative, vLLM-ready.

- Stay PD-colocated.
- Use corrected LMetric as base.
- Add only explicit high-cache affinity / tie-break logic where it improves
  stability.
- Improve scheduling: adaptive chunked prefill, decode-priority controls,
  better observability of queue and cache state.

Path 2: Research, PPD-style.

- Turn 1 seeds session on D.
- Later turns do append-prefill on resident D, avoiding P-to-D transfer.
- Dynamic x chooses P vs D based on append size, P queue, D load, and SLO.
- Requires stable implementation and strict loadgen validation.

The paper/story can say: transfer-based migration did not work; append-
prefill-on-resident-D remains a different and potentially better actuator.

## 4. Design Direction Recommendation

### 4.1 Near-term path

Use PD-colo cache-aware as the production baseline and paper baseline.

Implement/validate only low-risk routing improvements:

1. Pure LMetric baseline must stay separate and reproducible.
2. Unified hybrid should be LMetric plus:
   - high-cache explicit affinity;
   - overload escape;
   - deterministic non-degenerate tie-break;
   - route-decision logging.
3. No Mooncake/PUSH migration on the critical comparison path.

This gives a clean statement:

> The best robust single-node policy we have is cache-aware PD-colocation.
> Unified hybrid is a small refinement, not a new disaggregation win.

### 4.2 Research path

If we want a stronger contribution beyond "PD-sep loses", the promising
research direction is:

> session-resident append-prefill with dynamic P/D selection.

This aligns better with PPD than vLLM PUSH migration does.

Key design principle:

- Do not move KV just to run prefill elsewhere unless the future benefit is
  large enough to amortize the transfer.
- Prefer using the worker that already owns the session KV, unless decode
  load or append size makes that choice violate SLO.

## 5. Experiment Plan

### 5.1 Must-have validity checks

For every benchmark:

- Per-session sequentiality enforced.
- Attempted/completed/error counts reported.
- Pair by `(session_id, turn_id)` when comparing arms.
- Report goodput, not only latency of successes.
- Record git commit, launch flags, trace path, request limit, time scale,
  session sampling method, and hardware.

### 5.2 PD separation matrix

Goal: make the static PD-sep negative result paper-grade.

Arms:

- PD-colo cache-aware.
- PD-sep 4P+4D.
- PD-sep 6P+2D.
- Optional: round-robin baseline only as sanity, not main comparison.
- Optional: eager vs cudagraph ablation.

Metrics:

- TTFT/E2E/TPOT p50/p90/p99.
- Goodput and error rate.
- APC mean and per-instance distribution.
- GPU util and decode-side KV occupancy time series.
- TTFT breakdown: prefill, KV transfer, D wait.

Output:

- C2 headline bar with error bars.
- C3 KV utilization time series.
- C4 TTFT stacked breakdown.
- C5 cuda-graph ablation.

### 5.3 LMetric vs Unified hybrid

Goal: determine whether current Unified has any real gain over LMetric.

Arms:

- Pure corrected LMetric.
- Current Unified hybrid.

Run:

- 3-5 paired trials on the same trace.
- No Mooncake/PUSH.
- Same launch flags.

Additional logging:

- Route reason: `lmetric`, `high_cache_affinity`, `overload_escape`,
  `tie_break`.
- Chosen instance load, cache hit, effective new tokens.

Decision rule:

- If gain is within noise, do not oversell Unified as a performance win.
  Keep it as a policy cleanup / safety improvement.

### 5.4 Interference and scheduler experiments

Goal: test whether scheduling is the right actuator after routing saturates.

Arms:

- Different chunked prefill sizes.
- Decode-priority / prefill throttling if available.
- High-concurrency but session-sequential trace.

Metrics:

- TPOT under concurrent heavy prefills.
- TTFT for heavy turns.
- Decode queue delay.
- GPU util timeline.

Expected value:

- If migration is too expensive, reducing prefill interference in-place is
  the most plausible next improvement.

### 5.5 PPD/KVC-style research validation

Goal: separate PPD x=1 from failed x=0/full PD and failed transfer-based
migration.

Arms:

- PD-colo cache-aware.
- x=0 PD-disagg.
- x=1 append-prefill-on-D if implementation is stable.
- Dynamic x if available.

Guardrails:

- Do not use old high-concurrency KVC numbers without the loadgen caveat.
- Do not compare partial successful subsets without goodput.
- Treat SGLang implementation bugs as system results, not hidden noise.

## 6. Task Breakdown

### Track 1: Documentation alignment

Owner task:

- Update `REPORT.md`, `docs/migration-policy-design.md`, and
  `analysis/research_findings.md` so they use the taxonomy in section 2.

Concrete edits:

- Mark single-argmin/PUSH Unified as historical.
- State that current Unified is hybrid LMetric plus high-cache affinity.
- Add mapping to PPD taxonomy: Replica, x=0 PD, x=1 append-prefill.
- Add loadgen validity checklist.

Done when:

- A reviewer can no longer confuse vLLM PUSH migration with PPD x=1.
- LMetric baseline and Unified hybrid are described as separate policies.

### Track 2: Current routing cleanup

Owner task:

- Make current Unified hybrid auditable and minimal.

Concrete edits:

- Remove stale unreachable PUSH code from `scripts/cache_aware_proxy.py`.
- Keep pure `--policy lmetric` untouched.
- Add route-decision fields for Unified hybrid.
- Add tests:
  - pure LMetric remains pure;
  - high-cache affinity triggers only under its intended gate;
  - overload escape works;
  - empty-batch tie-break does not collapse to instance 0.

Done when:

- `pytest tests/test_proxy_pick.py` covers LMetric and Unified separately.
- Bench logs can count how often Unified did something beyond LMetric.

### Track 3: PD-sep paper matrix

Owner task:

- Finish the `analysis/pd_sep_paper_section` missing claims.

Concrete work:

- Run `bench_pd_matrix.sh` on dash0.
- Collect `metrics.summary.json`, `breakdown.json`, `apc.txt`,
  `gpu_util.csv`, and per-instance KV logs.
- Add plotters for C2/C3/C4/C5.
- Replace legacy C7 numbers with matrix outputs.

Done when:

- The PD-sep negative result no longer relies on old `--enforce-eager`
  methodology or single snapshots.

### Track 4: Benchmark substrate validation

Owner task:

- Audit the vLLM replayer and any dash0 loadgen scripts for session
  sequentiality and arrival semantics.

Concrete checks:

- Verify no session has more than one in-flight turn unless explicitly
  configured as a stress test.
- Add an analyzer that reports max concurrent turns per session.
- Report sampled session-start distribution.
- Add goodput and error-rate comparisons to all summary scripts.

Done when:

- We can label each experiment as online-realistic, burst stress, or
  synthetic microbench.

### Track 5: Scheduler/interference path

Owner task:

- Test whether in-place scheduling beats transfer-based migration.

Concrete experiments:

- Chunk size sweep.
- Decode-priority or prefill-throttle sweep.
- 16+ session sequential replay.

Done when:

- We know whether the next performance lever is scheduler policy or routing
  policy.

### Track 6: PPD-style appendix / related design

Owner task:

- Extract the useful `agentic-pd-hybrid` lessons without importing invalid
  claims.

Concrete work:

- Summarize:
  - loadgen bug and retractions;
  - PD-colo as stable baseline;
  - x=0 PD-disagg failure;
  - x=1/append-prefill motivation;
  - dynamic threshold lessons.
- Decide whether this is mainline future work or an appendix framing.

Done when:

- The story can cite PPD-style append-prefill as a distinct future actuator,
  not as evidence that the current Unified migration already works.

## 7. Proposed One-Sentence Story

Agentic serving breaks the classic PD-disaggregation intuition: long-lived
sessions make KV locality dominant, while long contexts make decode-side KV
capacity and transfer costs dominate the gains from isolating prefill; the
robust design is cache-aware PD-colocation with carefully limited session
affinity, and future disaggregation must be dynamic and session-resident
rather than static or transfer-heavy.

## 8. Open Decisions For Review

1. Do we want the main paper contribution to be the negative result
   "static PD separation fails for agentic", or the positive system
   "cache-aware PD-colo / Unified hybrid"?

2. Is PPD-style x=1 append-prefill a future-work section, or do we need to
   implement a minimal stable version before finalizing the story?

3. Should current Unified be presented as a named system if its measured
   improvement over LMetric is small, or should it be framed as an audit of
   why LMetric/cache-aware is already strong?

4. Which trace is the canonical trace for claims: the vLLM trace in
   `agentic-kv`, the GLM-5.1 trace in `agentic-pd-hybrid`, or both with
   explicit regime labels?

5. What is the target venue-style claim: systems negative result,
   workload characterization, or routing/scheduling algorithm?