Gahow Wang
d52baa0006
- paged_kv_cache: new block-paged KV cache; adds a pinned-host swap pool with
a second BlockAllocator, per-sequence Location {Gpu,Cpu}, and lossless
swap_out/swap_in (block-granular D2H/H2D) for vLLM-style preemption.
bytes_per_block helper exposes per-block cost for VRAM-based sizing.
- decode_graph: CUDA-graph decode path.
- qwen3/gpt2/kv_cache: paged prefill/decode forward + related updates.
- tokenizer/bins: BPE updates, new xserv-chat CLI, bench-qwen3 tweaks.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
23 lines
631 B
Rust
23 lines
631 B
Rust
pub mod config;
|
|
pub mod decode_graph;
|
|
pub mod gpt2;
|
|
pub mod kv_cache;
|
|
pub mod loader;
|
|
pub mod paged_kv_cache;
|
|
pub mod qwen3;
|
|
pub mod sampling;
|
|
|
|
pub use config::ModelConfig;
|
|
pub use decode_graph::{DecodeGraphState, LayerWeightPtrs};
|
|
pub use gpt2::{GPT2, KVCache};
|
|
pub use kv_cache::GpuKVCache;
|
|
pub use paged_kv_cache::{BlockAllocator, Location, PagedKVCache, BLOCK_SIZE};
|
|
pub use qwen3::Qwen3;
|
|
pub use sampling::{SamplingParams, sample};
|
|
|
|
/// Initialize GPU kernel hooks. Called automatically by model constructors,
|
|
/// but safe to call multiple times (idempotent via OnceLock).
|
|
pub fn init_kernels() {
|
|
xserv_kernels::init();
|
|
}
|