Gahow Wang d8493bd70f phase 12: implement real continuous batching scheduler ...

Rewrote engine.rs from scratch:
- Scheduler loop: admit → prefill → decode → finish → check new requests
- Multiple sequences run concurrently (max_batch_size configurable)
- Each sequence has independent GpuKVCache
- Non-blocking try_recv() for new requests during decode iterations
- Dynamic join: new requests enter batch immediately, don't wait for others

Verified with concurrent test (tools/test_concurrent.py):
- 3 concurrent requests: wall_time=3.8s, concurrency_ratio=2.82x ✓
- 5 concurrent requests: wall_time=6.1s, concurrency_ratio=4.04x ✓
- All outputs are coherent and correct

Design doc (docs/12-continuous-batching.md) fully rewritten with:
- Detailed scheduler loop pseudocode
- Data structures (Sequence, Scheduler)
- Acceptance criteria with specific test cases
- Clear separation from Phase 13 (HTTP layer)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-05-22 13:44:26 +08:00

..

benchmarks

phase 10: add Qwen3-8B benchmark + performance fix

2026-05-22 10:25:33 +08:00

00-roadmap.md

phase 0+1: project scaffold + xserv-cuda crate

2026-05-21 18:40:22 +08:00

01-cuda-ffi.md

docs: add design docs + takeaways for Phase 2 and Phase 3

2026-05-21 20:59:45 +08:00

02-tensor.md

docs: add design docs + takeaways for Phase 2 and Phase 3

2026-05-21 20:59:45 +08:00

03-gemm.md

docs: add design docs + takeaways for Phase 2 and Phase 3

2026-05-21 20:59:45 +08:00

04-transformer-kernels.md

phase 4: transformer core kernels

2026-05-21 21:07:24 +08:00

05-attention.md

phase 5: naive multi-head attention

2026-05-21 21:17:23 +08:00

06-model-loading.md

phase 6+7+8: model loading, BPE tokenizer, GPT-2 inference (Milestone ①)

2026-05-21 22:04:00 +08:00

07-tokenizer.md

phase 6+7+8: model loading, BPE tokenizer, GPT-2 inference (Milestone ①)

2026-05-21 22:04:00 +08:00

08-gpt2.md

phase 6+7+8: model loading, BPE tokenizer, GPT-2 inference (Milestone ①)

2026-05-21 22:04:00 +08:00

09-kv-cache.md

phase 9: KV cache + autoregressive generation

2026-05-21 23:39:41 +08:00

10-qwen3.md

phase 10: Qwen3-8B support (Milestone ②)

2026-05-22 00:46:37 +08:00

11-paged-attention.md

phase 11: GPU-resident KV cache

2026-05-22 11:50:12 +08:00

12-continuous-batching.md

phase 12: implement real continuous batching scheduler

2026-05-22 13:44:26 +08:00

13-http-api.md

docs: split Phase 12 and Phase 13 into separate design documents

2026-05-22 13:15:27 +08:00