2 Commits

Author	SHA1	Message	Date
Gahow Wang	c7d0750c32	moe(wip): gpt-oss-20b groundwork — config fields, arch doc, MXFP4 tools ... Phase 19 start. config.rs: explicit head_dim (gpt-oss=64) + MoE fields (num_local_experts, num_experts_per_tok, swiglu_limit, sliding_window, layer_types) with accessors; Qwen3/GPT-2 paths unchanged (fall back to hidden/num_heads when head_dim absent). docs/19-moe-gpt-oss.md: architecture + exact HF reference math (router softmax-after-topk, interleaved clamped (up+1)*glu experts, attention sinks, alternating sliding window, rotate_half RoPE theta=150000, head_dim 64), verified tensor layout, MXFP4 dequant plan. docs/MOE_PROGRESS.md: resume/handoff snapshot. tools/mxfp4_probe.py: inspect safetensors + validate MXFP4 decode (done). tools/gptoss_dequant.py: MXFP4 experts -> plain BF16 safetensors dir so the existing loader reads it (no MXFP4 in Rust for the first pass). Verified: llama.cpp (dash5, LLM_ARCH_OPENAI_MOE) runs the gpt-oss-20b MXFP4 GGUF correctly (17*24 -> 408) = the correctness oracle. MXFP4 decode validated in numpy. Model + GGUF staged on dash5. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-05-29 21:01:53 +08:00
Gahow Wang	057a3c68a3	docs: Phase 19 MoE (gpt-oss-20b) design + progress snapshot ... Architecture + exact HF reference math (router softmax-after-topk, interleaved clamped (up+1)*glu experts, attention sinks, alternating sliding window, head_dim 64, rope 150000), MXFP4 dequant plan, and the correctness-first -> PP -> llama.cpp roadmap. MOE_PROGRESS.md captures live state for resuming after a machine reboot (HF is firewalled here; download via proxy + hf-mirror; gpt-oss-20b not yet downloaded). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-05-29 19:13:23 +08:00