|
|
c7d0750c32
|
moe(wip): gpt-oss-20b groundwork — config fields, arch doc, MXFP4 tools
Phase 19 start. config.rs: explicit head_dim (gpt-oss=64) + MoE fields
(num_local_experts, num_experts_per_tok, swiglu_limit, sliding_window,
layer_types) with accessors; Qwen3/GPT-2 paths unchanged (fall back to
hidden/num_heads when head_dim absent).
docs/19-moe-gpt-oss.md: architecture + exact HF reference math (router
softmax-after-topk, interleaved clamped (up+1)*glu experts, attention
sinks, alternating sliding window, rotate_half RoPE theta=150000,
head_dim 64), verified tensor layout, MXFP4 dequant plan.
docs/MOE_PROGRESS.md: resume/handoff snapshot.
tools/mxfp4_probe.py: inspect safetensors + validate MXFP4 decode (done).
tools/gptoss_dequant.py: MXFP4 experts -> plain BF16 safetensors dir so
the existing loader reads it (no MXFP4 in Rust for the first pass).
Verified: llama.cpp (dash5, LLM_ARCH_OPENAI_MOE) runs the gpt-oss-20b
MXFP4 GGUF correctly (17*24 -> 408) = the correctness oracle. MXFP4 decode
validated in numpy. Model + GGUF staged on dash5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
2026-05-29 21:01:53 +08:00 |
|
