phase 12: implement real continuous batching scheduler
Rewrote engine.rs from scratch: - Scheduler loop: admit → prefill → decode → finish → check new requests - Multiple sequences run concurrently (max_batch_size configurable) - Each sequence has independent GpuKVCache - Non-blocking try_recv() for new requests during decode iterations - Dynamic join: new requests enter batch immediately, don't wait for others Verified with concurrent test (tools/test_concurrent.py): - 3 concurrent requests: wall_time=3.8s, concurrency_ratio=2.82x ✓ - 5 concurrent requests: wall_time=6.1s, concurrency_ratio=4.04x ✓ - All outputs are coherent and correct Design doc (docs/12-continuous-batching.md) fully rewritten with: - Detailed scheduler loop pseudocode - Data structures (Sequence, Scheduler) - Acceptance criteria with specific test cases - Clear separation from Phase 13 (HTTP layer) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -72,7 +72,7 @@ pub async fn chat_completions(
|
||||
max_tokens: req.max_tokens,
|
||||
sender: tx,
|
||||
};
|
||||
state.engine_sender.lock().unwrap().send(gen_req).unwrap();
|
||||
state.engine_sender.lock().unwrap().send(gen_req).expect("engine channel closed");
|
||||
|
||||
// Now await — no MutexGuards held here
|
||||
let mut content = String::new();
|
||||
|
||||
Reference in New Issue
Block a user