Root-caused the decode non-determinism: matmul's m==1 custom-GEMV fast path
reduces over K with a grid-split atomicAdd, whose float accumulation order
is non-deterministic. Negligible for attention's stable pre-transposed
weights, but for gpt-oss's wide expert GEMMs (K=2880, N up to 5760) over
freshly-dequantized MXFP4 weights it produced visibly different results
run-to-run (and a wrong argmax). Added gemm::matmul_dense (plain cublasGemmEx,
no GEMV shortcut) and route the expert GEMMs through it.
Now decode_step (KV cache + GPU sink-attention + MXFP4 experts) is:
- deterministic: 3/3 identical runs
- correct: top-1 token 12650 = " Paris" for "The capital of France is",
MATCH_TOP1 with the host-attention reference forward
- end-to-end: gptoss-gen generates 32 tokens at ~6.85 tok/s on one 5090.
Removed the temporary A/B debug dumps. gptoss-logits runs both paths and
asserts the top-1 match; gptoss-gen times greedy generation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
decode_step(): single-token forward using GpuKVCache + the GPU
decode_attention_sink kernel (per-head sinks + sliding window) + per-token
MoE, so generation is O(1)/token instead of the host forward's O(n²)
recompute. generate(): prefill prompt token-by-token (causal attention =>
sequential == batched), then greedy decode to eos/max_new.
Verified against the reference host forward on the Paris prompt: both
predict top-1 token 12650 = " Paris" (MATCH_TOP1: YES; logits 15.375 vs
15.3125 — BF16 accumulation-order diff only). gptoss-logits now runs both
paths and asserts the match. Build green on dash5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Experts now stay MXFP4-packed on GPU (~10GB whole model, fits one 32GB
card) instead of dequantized to ~38GB BF16. loader::load_model_dir_split
returns BF16 tensors + raw U8 (_blocks/_scales) in one pass; GptOss slices
each expert's MXFP4 bytes to a GpuBuffer at load, and expert_forward
dequantizes the selected expert to a BF16 scratch (dequant_mxfp4) right
before its GEMM — no per-token CPU->GPU upload, no 38GB BF16 dir.
Verified: gptoss-logits on the original MXFP4 dir
(/opt/wjh/models/gpt-oss-20b) gives logits byte-identical to the BF16 path
— top-1 token 12650 = " Paris" @ 15.3125, full top-10 unchanged — running
on a single GPU. Build green on dash5 (release).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GptOss model in xserv's own style (not derived from llama.cpp): BF16
loader for the dequantized weights, naive sink-attention + per-token
top-k MoE FFN on host for correctness-first, GPU matmuls via our kernels.
Reuses the Qwen3 forward pattern (rotate_half RoPE θ=150000, head_dim 64,
no q/k norm) and adds q/k/v/o + expert biases, clamped (up+1)*glu experts,
attention sinks, alternating sliding window. gptoss-logits bin dumps
next-token logits for fixed token ids to compare with the llama.cpp oracle.
WIP: compiles pending fixes; numerical alignment vs llama.cpp is the next
step. Then paged-cache + PP wiring + AIME/GSM8K.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
