Gahow Wang
e207523e21
Custom bandwidth-optimized GEMV kernel for M=1 BF16 decode, replacing cuBLAS which achieves only ~8% bandwidth utilization for tiny M=1 GEMMs. Kernel design (csrc/gemm/gemv.cu): - K-split tiled: TILE_N=128, TILE_K=256, Grid=(N/128, K/256)=512 blocks - High occupancy: 512 blocks / 170 SMs = ~3 blocks/SM - Coalesced memory access: adjacent threads read adjacent columns of W - Shared memory for x vector (avoids redundant global reads) - FP32 accumulation via atomicAdd (K-split partial sums) - Separate fp32→bf16 conversion kernel Integration: - matmul() auto-dispatches to custom GEMV when M==1 && dtype==BF16 - Batched decode (M>1) continues to use cuBLAS - Caching allocator provides FP32 temp buffer (pooled, no per-call malloc) Ablation results (dash5, RTX 5090, Qwen3-8B BF16): | Config | tok/s | vs HF (36) | vs roofline (112) | |--------|-------|-----------|-------------------| | Phase 14 (cuBLAS M=1) | 13.2 | 37% | 12% | | + Custom GEMV (M=1) | 46.6 | 130% | 42% | | Concurrent batch=4 | 28.2 | 78% | — | Single-request throughput now EXCEEDS HuggingFace transformers by 30%. The custom GEMV achieves ~42% of the theoretical roofline (vs 12% before). Note: concurrent batch=4 (28.2 tok/s) is slower than serial (46.6 tok/s) because the per-seq attention/reshape overhead in batched decode outweighs the cuBLAS M=4 benefit when the custom GEMV already handles M=1 efficiently. Engine should prefer serial decode when custom GEMV is available. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
33 lines
1.1 KiB
Rust
33 lines
1.1 KiB
Rust
use std::env;
|
|
|
|
fn main() {
|
|
let cuda_path = env::var("CUDA_HOME")
|
|
.or_else(|_| env::var("CUDA_PATH"))
|
|
.unwrap_or_else(|_| "/usr/local/cuda".to_string());
|
|
|
|
println!("cargo:rustc-link-search=native={cuda_path}/lib64");
|
|
println!("cargo:rustc-link-lib=dylib=cudart");
|
|
println!("cargo:rustc-link-lib=dylib=cublas");
|
|
|
|
cc::Build::new()
|
|
.cuda(true)
|
|
.cudart("shared")
|
|
.flag("-gencode=arch=compute_120,code=sm_120")
|
|
.include("../../csrc")
|
|
.file("../../csrc/gemm/naive.cu")
|
|
.file("../../csrc/gemm/tiled.cu")
|
|
.file("../../csrc/gemm/gemv.cu")
|
|
.file("../../csrc/normalization/rmsnorm.cu")
|
|
.file("../../csrc/normalization/layernorm.cu")
|
|
.file("../../csrc/activation/activations.cu")
|
|
.file("../../csrc/reduce/softmax.cu")
|
|
.file("../../csrc/embedding/embedding.cu")
|
|
.file("../../csrc/embedding/rope.cu")
|
|
.file("../../csrc/attention/causal_mask.cu")
|
|
.file("../../csrc/embedding/transpose.cu")
|
|
.file("../../csrc/attention/flash_attention.cu")
|
|
.compile("xserv_kernels");
|
|
|
|
println!("cargo:rerun-if-changed=../../csrc/");
|
|
}
|