|
|
6cc1c9332d
|
docs: Phase 14 design doc + benchmark, fix Phase 11/12 honesty
Phase 14 (Flash Attention):
- Design doc: FA2 algorithm, SM120 hardware constraints (FA4 incompatible),
kernel config (BR=BC=64, 32KB smem), GQA mapping, causal tile-skip,
known limitations and optimization roadmap
- Benchmark doc: correctness (9/10 top-1 match, identical to pre-FA baseline),
performance tracking (6.9→10.3→12.9 tok/s across phases), memory savings
analysis, remaining bottleneck breakdown
Phase 11 doc: title corrected from "Paged Attention" to "GPU-Resident KV Cache"
with explicit note that paged allocation was not implemented.
Phase 12 doc: "当前状态" updated from "未实现" to reflect actual state —
iteration-level scheduling implemented + verified (6.0x concurrent speedup),
batched GPU forward explicitly marked as not yet implemented.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-05-22 18:51:29 +08:00 |
|
|
|
d8493bd70f
|
phase 12: implement real continuous batching scheduler
Rewrote engine.rs from scratch:
- Scheduler loop: admit → prefill → decode → finish → check new requests
- Multiple sequences run concurrently (max_batch_size configurable)
- Each sequence has independent GpuKVCache
- Non-blocking try_recv() for new requests during decode iterations
- Dynamic join: new requests enter batch immediately, don't wait for others
Verified with concurrent test (tools/test_concurrent.py):
- 3 concurrent requests: wall_time=3.8s, concurrency_ratio=2.82x ✓
- 5 concurrent requests: wall_time=6.1s, concurrency_ratio=4.04x ✓
- All outputs are coherent and correct
Design doc (docs/12-continuous-batching.md) fully rewritten with:
- Detailed scheduler loop pseudocode
- Data structures (Sequence, Scheduler)
- Acceptance criteria with specific test cases
- Clear separation from Phase 13 (HTTP layer)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-05-22 13:44:26 +08:00 |
|
|
|
7d05ececa0
|
docs: split Phase 12 and Phase 13 into separate design documents
- docs/12-continuous-batching.md: scheduler, sequence management,
batching strategy (currently single-request, expandable)
- docs/13-http-api.md: HTTP server, OpenAI-compatible API,
axum architecture, SSE streaming (TODO)
Phase 12 = WHAT to compute (scheduling decisions)
Phase 13 = HOW to expose it (HTTP protocol layer)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-05-22 13:15:27 +08:00 |
|
