kernels: MXFP4 -> BF16 dequant kernel (verified vs numpy)
From-scratch CUDA kernel for gpt-oss expert weights: one thread per packed byte decodes 2 FP4 (E2M1) codes, applies the per-32-block E8M0 scale (2^(e-127)), and writes BF16 transposed into [IN, OUT] (IN = nblk*32) so it drops straight into x @ W. dequant_mxfp4() wrapper takes raw GpuBuffers (uint8 is not an xserv Tensor dtype). mxfp4-check bin dequants layer-0 expert-0 on GPU and matches tools/mxfp4_probe.py exactly: [0, 0, 0, -0.0625, 0, -0, -0.015625, -0.03125] This lets experts stay MXFP4-resident on GPU (13GB, fits one 32GB card) and be dequantized to a BF16 scratch right before each expert GEMM, instead of holding 36GB of BF16 or uploading experts per token. Loader plumbing + GPU MoE decode use it next. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -15,6 +15,7 @@ pub use attention::{attention, decode_attention, flash_attention, paged_decode_a
|
||||
pub use embedding::embedding;
|
||||
pub use gemm::{batched_matmul, matmul, GemmBackend};
|
||||
pub use layernorm::layernorm;
|
||||
pub use quant::dequant_mxfp4;
|
||||
pub use rmsnorm::{add_rmsnorm, rmsnorm};
|
||||
pub use rope::{rope_inplace, RopeCache};
|
||||
pub use softmax::softmax;
|
||||
|
||||
Reference in New Issue
Block a user