gahow/xserv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase 10: GPU add/mul kernels + BF16 precision analysis

Browse Source 
Kernel additions:
- add_f32/bf16, mul_f32/bf16 CUDA kernels (element-wise, on GPU)
- Refactored activation.rs with dispatch_unary/dispatch_binary helpers
- Qwen3 and GPT-2 now use GPU add/mul instead of CPU round-trips

GPT-2 add_bias also moved to GPU (broadcast via tile + GPU add)

BF16 precision analysis (docs/benchmarks/phase10-qwen3.md):
- Root cause: separate attention kernels materialize BF16 intermediates
  (QK^T→BF16→scale→BF16→mask→BF16→softmax→BF16 vs HF's fused FP32 path)
- HF itself SDPA vs Eager also differs by ~0.125 logit
- xserv vs HF: ~1-2 logit systematic offset, but same top-1 in 84% cases
- Industry standard for BF16: top-5 overlap (we achieve 100%)
- Fix path: Flash Attention (Phase 14) to fuse attention in FP32

Performance: TTFT 138→119ms, TBT 144→137ms (GPU ops faster than CPU)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-22 11:35:26 +08:00
parent 268e40d764
commit be5c64ea8a
6 changed files with 143 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

36
crates/xserv-model/src/gpt2.rs
View File

@@ -247,27 +247,33 @@ fn matmul_2d(a: &Tensor, b: &Tensor) -> Tensor {
}
fn add_tensors(a: &Tensor, b: &Tensor) -> Tensor {
assert_eq!(a.shape(), b.shape());
assert_eq!(a.dtype(), DType::F32);
let a_cpu = a.to_device(Device::Cpu);
let b_cpu = b.to_device(Device::Cpu);
let a_data = a_cpu.as_slice::<f32>();
let b_data = b_cpu.as_slice::<f32>();
let sum: Vec<f32> = a_data.iter().zip(b_data).map(|(x, y)| x + y).collect();
Tensor::from_slice(&sum, a.shape()).to_device(a.device())
xserv_kernels::add(a, b)
}
fn add_bias(x: &Tensor, bias: &Tensor) -> Tensor {
// bias: [N], x: [S, N] — broadcast add via reshape
assert_eq!(x.ndim(), 2);
assert_eq!(bias.ndim(), 1);
assert_eq!(x.shape()[1], bias.shape()[0]);
let x_cpu = x.to_device(Device::Cpu);
let b_cpu = bias.to_device(Device::Cpu);
let x_data = x_cpu.as_slice::<f32>();
let b_data = b_cpu.as_slice::<f32>();
let n = bias.shape()[0];
let result: Vec<f32> = x_data.iter().enumerate().map(|(i, &v)| v + b_data[i % n]).collect();
Tensor::from_slice(&result, x.shape()).to_device(x.device())
assert_eq!(x.shape()[1], n);
let rows = x.shape()[0];
// Broadcast: tile bias to [S, N] on CPU, then GPU add
let b_cpu = bias.to_device(Device::Cpu);
match x.dtype() {
DType::F32 => {
let bd = b_cpu.as_slice::<f32>();
let tiled: Vec<f32> = (0..rows).flat_map(|_| bd.iter().copied()).collect();
let b_full = Tensor::from_slice(&tiled, x.shape()).to_device(x.device());
xserv_kernels::add(x, &b_full)
}
DType::BF16 => {
let bd = b_cpu.as_slice::<half::bf16>();
let tiled: Vec<half::bf16> = (0..rows).flat_map(|_| bd.iter().copied()).collect();
let b_full = Tensor::from_slice(&tiled, x.shape()).to_device(x.device());
xserv_kernels::add(x, &b_full)
}
_ => panic!("unsupported dtype"),
}
}
fn split_qkv(qkv: &Tensor, num_heads: usize, head_dim: usize, seq_len: usize) -> (Tensor, Tensor, Tensor) {