gahow/xserv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase 10: GPU add/mul kernels + BF16 precision analysis

Browse Source 
Kernel additions:
- add_f32/bf16, mul_f32/bf16 CUDA kernels (element-wise, on GPU)
- Refactored activation.rs with dispatch_unary/dispatch_binary helpers
- Qwen3 and GPT-2 now use GPU add/mul instead of CPU round-trips

GPT-2 add_bias also moved to GPU (broadcast via tile + GPU add)

BF16 precision analysis (docs/benchmarks/phase10-qwen3.md):
- Root cause: separate attention kernels materialize BF16 intermediates
  (QK^T→BF16→scale→BF16→mask→BF16→softmax→BF16 vs HF's fused FP32 path)
- HF itself SDPA vs Eager also differs by ~0.125 logit
- xserv vs HF: ~1-2 logit systematic offset, but same top-1 in 84% cases
- Industry standard for BF16: top-5 overlap (we achieve 100%)
- Fix path: Flash Attention (Phase 14) to fuse attention in FP32

Performance: TTFT 138→119ms, TBT 144→137ms (GPU ops faster than CPU)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-22 11:35:26 +08:00
parent 268e40d764
commit be5c64ea8a
6 changed files with 143 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
crates/xserv-model/src/qwen3.rs
View File

@@ -250,27 +250,11 @@ fn repeat_kv(x: &Tensor, n_rep: usize) -> Tensor {
}
fn add_any(a: &Tensor, b: &Tensor) -> Tensor {
assert_eq!(a.shape(), b.shape());
let a_cpu = a.to_device(Device::Cpu);
let b_cpu = b.to_device(Device::Cpu);
let ad = a_cpu.as_slice::<bf16>();
let bd = b_cpu.as_slice::<bf16>();
let r: Vec<bf16> = ad.iter().zip(bd)
.map(|(x, y)| bf16::from_f32(x.to_f32() + y.to_f32()))
.collect();
Tensor::from_slice(&r, a.shape()).to_device(a.device())
xserv_kernels::add(a, b)
}
fn mul_any(a: &Tensor, b: &Tensor) -> Tensor {
assert_eq!(a.shape(), b.shape());
let a_cpu = a.to_device(Device::Cpu);
let b_cpu = b.to_device(Device::Cpu);
let ad = a_cpu.as_slice::<bf16>();
let bd = b_cpu.as_slice::<bf16>();
let r: Vec<bf16> = ad.iter().zip(bd)
.map(|(x, y)| bf16::from_f32(x.to_f32() * y.to_f32()))
.collect();
Tensor::from_slice(&r, a.shape()).to_device(a.device())
xserv_kernels::mul(a, b)
}
pub fn sample_greedy(logits: &Tensor) -> u32 {