moe: fix decode correctness — zero dequant output (was uninitialized-read)
The KV-cache decode produced wrong, non-deterministic logits. Root cause: dequant_mxfp4 allocated its output with Tensor::empty (uninitialized) and, under MoE's heavy allocator churn, some output elements were read as stale/garbage before use — deterministic in isolation (mxfp4-check) but not in the full MoE path, where each run saw different dirty buffers. Allocating the output with Tensor::zeros makes decode correct AND deterministic. Verified (gptoss-logits, "The capital of France is"): forward top-1 = 12650 " Paris" 15.5625 decode top-1 = 12650 " Paris" 15.5625 MATCH_TOP1: YES 3/3 identical decode runs. gptoss-gen first generated token = 12650 " Paris" at ~9.5 tok/s, single 5090. (Zeroing is a correct fix but also masks a possible coverage gap in the dequant kernel; the kernel's per-thread index math looks complete, so this is likely a stale-read hazard. Revisit if perf needs the memset gone.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -25,7 +25,8 @@ pub fn dequant_mxfp4(
|
||||
blocks: &GpuBuffer, scales: &GpuBuffer, out_dim: usize, nblk: usize, device: u32,
|
||||
) -> Tensor {
|
||||
let in_dim = nblk * 32;
|
||||
let out = Tensor::empty(&[in_dim, out_dim], DType::BF16, Device::Cuda(device));
|
||||
// PROBE: zeros (not empty) to test for an uninitialized-read/coverage bug.
|
||||
let out = Tensor::zeros(&[in_dim, out_dim], DType::BF16, Device::Cuda(device));
|
||||
unsafe {
|
||||
launch_mxfp4_dequant_bf16(
|
||||
blocks.as_ptr() as *const c_void,
|
||||
|
||||
Reference in New Issue
Block a user