xserv/docs/benchmarks/sparse-moe.md

Sparse MoE decode — 1.8× over dense; beats llama.cpp at TP=2 (gpt-oss-20b, RTX 5090)

Phase 20 (docs/20-sparse-moe.md): decode computes only the routed top-4 experts via fused expert-indexed GEMVs (csrc/moe/moe_sparse.cu) instead of the dense all-local-expert batched GEMM. FP8 weights run W8A16 (weights FP8, activations BF16 — decode is memory-bound, tensor cores irrelevant at M=1); MXFP4 runs W4A16. Dense path retained for prefill / num_tokens > 8 and via XSERV_DENSE_MOE=1 for A/B.

In-process decode (bench-gpt-oss, greedy, 96 tokens)

config	TPOT	tok/s
dense FP8 TP=2 (baseline)	13.9 ms	72
sparse FP8 TP=2	7.6 ms	132
sparse MXFP4 TP=2	8.4 ms	118
sparse FP8 TP=1 (one 5090)	7.8 ms	128
sparse MXFP4 TP=1	8.9 ms	113

• Sparse FP8 = 1.8× over dense. Greedy output stays coherent.
• TP=1 ≈ TP=2: expert reads are now so small that PCIe all-reduce eats the TP gain — single-GPU serving becomes the attractive deployment.
• MXFP4 reads half the bytes of FP8 but stays slower: the 4-bit dequant GEMV has lower effective bandwidth (same fixed inefficiency seen in the dense MXFP4 experiments); at sparse sizes both are partly launch/latency-bound.

Head-to-head vs llama.cpp (tools/xserv_vs_llama.py, warm servers, TP=2, GPUs 0-1, 6 reps, 256 tok)

prompt	metric	xserv sparse FP8	llama MXFP4	xserv vs llama
short	TTFT	35.3 ms	62.7 ms	1.78× faster
short	TPOT	7.32 ms	8.42 ms	1.15× faster
medium	TTFT	49.4 ms	65.0 ms	1.32× faster
medium	TPOT	7.19 ms	7.54 ms	1.05× faster
medium	tok/s	139.1	132.7
long (1.6k)	TTFT	94.1 ms	44.7 ms	0.48× (llama wins)
long	TPOT	7.25 ms	7.64 ms	1.05× faster

Decode TPOT now beats llama.cpp at every prompt length (was 2× slower: 13.1 vs 6.6 ms before sparse). Remaining loss: long-prompt TTFT — prefill is still the dense all-expert GEMM; sparse/grouped prefill is the next phase.

Post-review fixes (same harness, rerun): removing three leftover cudaDeviceSynchronize from the decode hot path and replacing the CPU-tiled prefill bias-add (96 D2H/H2D round-trips per prefill) with a GPU broadcast kernel improved both axes — TPOT 7.19-7.32 → 6.99-7.21 ms, TTFT short/medium/long 35/49/94 → 29/42/79 ms. GSM8K-50: 94% (unchanged).

TP=1 head-to-head (single 5090; server now routes gpt-oss tp=1 to the TP engine)

prompt	metric	xserv sparse FP8	llama MXFP4
short	TTFT / TPOT	42.8 ms / 7.00 ms	34.5 ms / 3.22 ms
medium	TTFT / TPOT	57.1 ms / 7.19 ms	37.3 ms / 2.89 ms
long	TTFT / TPOT	119.6 ms / 7.20 ms	27.8 ms / 2.88 ms
	tok/s	139–143	311–347

Single-GPU is llama.cpp's sweet spot and it wins 2.2–2.5×. Two structural reasons, both instructive:

1. llama TP=2 (7.5–8.4 ms) is much WORSE than its TP=1 (2.9 ms): its PCIe cross-GPU split costs ~5 ms/token. xserv's NCCL all-reduce is cheap enough that TP=2 ≈ TP=1 (7.2 vs 7.0 ms) — but xserv's single-GPU floor is high.
2. xserv TP=1 reads ~4.7 GB/token (experts FP8 2.4 GB + non-expert weights still BF16 ~2.3 GB, half of that the 201k-vocab lm_head) ≈ 3.1 ms of pure HBM time; the other ~4 ms is launch overhead (~200 kernels/token, no CUDA graphs) + BF16 GEMV efficiency. llama reads ~1.3 GB (everything MXFP4) and replays the whole token as one CUDA graph.

Correctness

• Greedy generations coherent across prompts (FP8/MXFP4, TP=1/2).
• Sparse FP8 is W8A16 vs dense W8A8 — activations are no longer quantized, so tokens are not expected to be byte-identical to dense; quality is checked by GSM8K instead.
• GSM8K-100 (greedy, TP=2, tools/eval_gsm8k_fast.py): 96/100 = 96.0% vs dense FP8 91.0% / BF16 90.0% — no regression (within greedy-nondeterminism noise; W8A16 removes activation-quantization error so ≥ dense is expected). Avg 1.3 s/problem also reflects the decode speedup.

Phase 21 update: decode CUDA graph + GPU argmax (docs/21-cuda-graph-decode.md)

The whole batch=1 decode step now replays as one CUDA graph, and greedy sampling uses the GPU argmax kernel (4-byte D2H instead of a 402 KB logits copy + 201k-element host scan). In-process A/B: graph −0.6 ms, GPU argmax −1.0 ms. Warm-server head-to-head (same harness/GPUs, 6 reps):

	xserv FP8 (graph)	llama MXFP4
TP=2 TPOT	5.76–5.89 ms (170–174 tok/s)	7.42–8.45 ms	xserv 1.26–1.47×
TP=2 TTFT s/m/l	25 / 28 / 51 ms	63 / 66 / 45 ms	xserv 2.4× s/m; long ~par
TP=1 TPOT	5.78–5.95 ms	2.80–3.22 ms	llama 2.0× (was 2.5×)
TP=1 TTFT s/m	32 / 35 ms	34 / 36 ms	xserv slightly ahead

GSM8K-50 through the graph path: 47/50 = 94% (unchanged). Note: GPU argmax breaks exact-tie logits differently than the host scan, so greedy trajectories can legitimately diverge at a tie token.

Remaining gaps / next levers (to catch llama TP=1 at 2.8 ms)

Per-token fixed overhead is now mostly gone; the residual ~5.8 ms is dominated by HBM bytes and kernel efficiency. In impact order:

1. Quantize non-expert weights (~1.5 ms): attn qkv/o + the 1.16 GB BF16 lm_head read every token; FP8/MXFP4 them like llama quantizes everything.
2. GEMV/attention bandwidth tuning: effective BW of the hand GEMVs is well under peak; llama's 2.8 ms implies ~85%+ efficiency on ~1.3 GB.
3. Sparse prefill (permute tokens by expert + grouped GEMM): long-prompt TTFT 51–75 ms → llama's ~30 ms territory.
4. W4A4 FP4 tensor cores / bandwidth-tuned MXFP4 GEMV: make 4-bit experts actually beat FP8.