phase 12: implement real continuous batching scheduler

Rewrote engine.rs from scratch:
- Scheduler loop: admit → prefill → decode → finish → check new requests
- Multiple sequences run concurrently (max_batch_size configurable)
- Each sequence has independent GpuKVCache
- Non-blocking try_recv() for new requests during decode iterations
- Dynamic join: new requests enter batch immediately, don't wait for others

Verified with concurrent test (tools/test_concurrent.py):
- 3 concurrent requests: wall_time=3.8s, concurrency_ratio=2.82x ✓
- 5 concurrent requests: wall_time=6.1s, concurrency_ratio=4.04x ✓
- All outputs are coherent and correct

Design doc (docs/12-continuous-batching.md) fully rewritten with:
- Detailed scheduler loop pseudocode
- Data structures (Sequence, Scheduler)
- Acceptance criteria with specific test cases
- Clear separation from Phase 13 (HTTP layer)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-22 13:44:26 +08:00
parent 7d05ececa0
commit d8493bd70f
5 changed files with 348 additions and 100 deletions

View File

@@ -72,7 +72,7 @@ pub async fn chat_completions(
max_tokens: req.max_tokens,
sender: tx,
};
state.engine_sender.lock().unwrap().send(gen_req).unwrap();
state.engine_sender.lock().unwrap().send(gen_req).expect("engine channel closed");
// Now await — no MutexGuards held here
let mut content = String::new();