Gahow Wang
9a81c993ab
the root README + AGENTS.md
Without this, the four docs added on this branch
(AUDIT_AND_ROADMAP, INDEX, BLOCK_LEVEL_EVICTION_DESIGN,
D_TO_P_SYNC_CONTRACT, EVALUATION_PROTOCOL) are reachable
only by listing docs/. This wires them into the two entry
points an agent or collaborator hits first.
README.md changes:
- top-of-page pointer to INDEX_ZH for new collaborators
- pointer to AUDIT_AND_ROADMAP_ZH for project state
- "单元测试 (无 GPU)" section: how to run pytest
- "评测脚本" section: invocations for the two new
analysis scripts
AGENTS.md changes:
- top section "For new collaborators / agents" before
the existing "Environment" block, pointing at INDEX_ZH,
AUDIT_AND_ROADMAP_ZH, the two ready-to-pick-up design
docs, and EVALUATION_PROTOCOL_ZH
- pytest invocation under Environment
119 lines
3.3 KiB
Markdown
119 lines
3.3 KiB
Markdown
# AGENTS.md
|
|
|
|
## For new collaborators / agents
|
|
|
|
Before doing anything else, read [docs/INDEX_ZH.md](docs/INDEX_ZH.md). It points to the
|
|
3 must-read docs and a role-based reading path (new SWE, paper reviewer,
|
|
reproducing student, control-plane reader).
|
|
|
|
Cross-branch progress, weaknesses, and roadmap live in
|
|
[docs/AUDIT_AND_ROADMAP_ZH.md](docs/AUDIT_AND_ROADMAP_ZH.md). It is the single source of truth
|
|
for "what's done, what's broken, what to do next."
|
|
|
|
Two engineering work items are pre-specced and ready to pick up:
|
|
- block-level eviction refactor — [docs/BLOCK_LEVEL_EVICTION_DESIGN_ZH.md](docs/BLOCK_LEVEL_EVICTION_DESIGN_ZH.md)
|
|
- D→P incremental KV sync — [docs/D_TO_P_SYNC_CONTRACT_ZH.md](docs/D_TO_P_SYNC_CONTRACT_ZH.md)
|
|
|
|
Evaluation protocol (paper-quality N, paired CI, stratification,
|
|
baselines) is in [docs/EVALUATION_PROTOCOL_ZH.md](docs/EVALUATION_PROTOCOL_ZH.md).
|
|
|
|
## Environment
|
|
|
|
Use `uv` to manage all python environment. `uv add` to manage deps so that we can `uv sync` to get exactly same runnable environment each time.
|
|
|
|
Algorithm-layer unit tests (no GPU, no SGLang):
|
|
|
|
```bash
|
|
uv sync --group test
|
|
uv run pytest
|
|
```
|
|
|
|
## Goal
|
|
|
|
Build a minimal prototype on top of **SGLang xPyD** to test whether **session-aware / KV-cache-aware P/D routing** can improve **end-to-end latency** for agentic coding workloads.
|
|
|
|
Current setup:
|
|
- SGLang: `v0.5.10`
|
|
- Model: `Qwen3-Coder-30B-A3B-Instruct` (`~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct`)
|
|
- xPyD runs on this single 8-GPU node, so the current constraint is **$x + y \le 8$**
|
|
- Even in local experiments, the implementation should preserve the **P -> D RDMA-style data path** semantics as much as possible; local runs should treat this as a loopback-based stand-in rather than collapsing P/D into a special in-process shortcut
|
|
- Traces:
|
|
- Ali coding agent (`~/ali-trace/trace-qwen3-coder-formatted/041715-041717.jsonl`)
|
|
|
|
---
|
|
|
|
## MVP Scope
|
|
|
|
We only do the following:
|
|
|
|
1. Run **SGLang xPyD** correctly on one machine
|
|
2. Add a **baseline router**
|
|
- `turn1`: default routing
|
|
- `turn2+`: prefer previous `D` node for the same session
|
|
3. Add a **KV-cache-aware routing** policy
|
|
4. Replay traces and compare policies with the same evaluation pipeline
|
|
|
|
Out of scope for now:
|
|
- autoscaling
|
|
- fault tolerance
|
|
- large-scale cluster scheduler
|
|
- production hardening
|
|
- general multi-tenant serving
|
|
|
|
---
|
|
|
|
## What matters
|
|
|
|
Primary metric:
|
|
- **E2E latency**
|
|
|
|
Secondary metrics:
|
|
- TTFT
|
|
- TPOT
|
|
- KV transfer volume
|
|
- cache hit / reuse
|
|
- re-prefill count
|
|
- per-node load
|
|
|
|
Do not optimize TTFT alone if E2E does not improve.
|
|
|
|
---
|
|
|
|
## Development Order
|
|
|
|
Implement in this order:
|
|
|
|
1. **Bring up xPyD**
|
|
2. **Add trace replay + metrics logging**
|
|
3. **Implement sticky-to-D baseline**
|
|
4. **Implement KV-cache-aware routing**
|
|
5. **Analyze gains and failure cases**
|
|
|
|
Do not skip step 2.
|
|
|
|
---
|
|
|
|
## Core Rules
|
|
|
|
### 1. Keep policy separate from mechanism
|
|
- mechanism = how requests / KV / xPyD work
|
|
- policy = how we choose `P` and `D`
|
|
|
|
Do not mix them unless necessary.
|
|
|
|
### 2. Prefer simple, debuggable logic
|
|
Start with simple heuristics before complex scoring.
|
|
|
|
### 3. Log everything needed to explain results
|
|
Each request should log:
|
|
- request id
|
|
- session id
|
|
- turn id
|
|
- assigned P node
|
|
- assigned D node
|
|
- latency
|
|
- whether reuse was expected / observed
|
|
|
|
### 4. Small interfaces only
|
|
Avoid over-abstraction.
|