perf: KI-2 FIXED — dim768 bf16 fits batch 32, tok/s 31.5K→40.8K
bf16 mixed precision (fp32 master) solves the v4 dim768 fp32 batch-32 OOM and speeds up the now-compute-bound dim768 GEMMs (dash5 1× RTX 5090 32GB, dim768/18L/24h×32 ffn2048 seq256, steady-state): config batch peak mem tok/s fits 32GB fp32 16 27.2 GB 31.5K yes bf16 16 19.3 GB 35.5K yes (-29% mem / +13% tok/s) fp32 32 — — OOM bf16 32 31.1 GB 40.8K yes (+29% vs fp32-b16) Verified on dash5: fp32 suite green at tight tol + xserv export md5 bit-identical to registry; bf16 looser-tol (loss 1.2e-4, logits p99 6.8e-3, grad 1.0e-2) + 150-step convergence tracks fp32 (3.984 vs 3.988); 2-GPU bf16 DDP at per-rank batch 32 trains cleanly. Mark KI-2 FIXED; fill docs/11 results. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -76,8 +76,23 @@
|
||||
- **dim768 bf16 能跑 per-rank batch 32**(v4 OOM 的触发点)。
|
||||
- 测 dim768 **bf16 vs fp32** 的峰值显存 + steady-state tok/s(预期:显存↓、tok/s↑)。
|
||||
|
||||
## 实测结果(dash5, 8× RTX 5090, sm_120)
|
||||
## 实测结果(dash5, 1× RTX 5090 32GB, sm_120)
|
||||
|
||||
> 见 `docs/known-issues.md` KI-2 的 before→after 表(fp32 batch32 OOM → bf16 batch32 fit;显存 A→B;tok/s A→B)。
|
||||
**闸门 ① fp32 不回归**:全套测试在原紧容差绿(autograd 15 / structural 5 / GEMM 5 / batched==looped / overfit 27/27 / AdamW GPU bit-exact + host 对 torch / checkpoint 逐位 / DDP 2)。**xserv 闭环**:v3 ckpt 用 T12 代码重导 `model.safetensors` 与 registry **md5 逐位一致**(`b04fc9f9a0c9af04c47d9ca649aea12e`)——export/fp32 数值零漂移。
|
||||
|
||||
**闸门 ② bf16 正确性 + 收敛**:
|
||||
- **looser-tol(tests/bf16.rs)**:同 fp32 master 跑 fp32 vs bf16——loss rel `1.2e-4`、logits mean rel `2.0e-3` / p99 `6.8e-3`、grad worst scaled-mean `1.0e-2`,无 NaN,grad 仍 fp32(master 未动)。
|
||||
- **收敛**:dim768 短训 150 步,bf16-b16 loss 轨迹对住 fp32-b16(step50 `4.40` vs `4.40`、step149 `3.984` vs `3.988`),单调降、无发散。
|
||||
|
||||
**闸门 ③ 显存 + 吞吐(dim768/18L/24h×32 ffn2048 seq256, steady-state)**:
|
||||
|
||||
| config | per-rank batch | 峰值显存 | tok/s | fits 32GB? |
|
||||
|---|---|---|---|---|
|
||||
| fp32 | 16 (v4 fallback) | 27.2 GB | 31.5K | ✅ |
|
||||
| **bf16** | 16 | **19.3 GB(−29%)** | **35.5K(+13%)** | ✅ |
|
||||
| fp32 | 32 | — | — | ❌ **OOM** |
|
||||
| **bf16** | **32(甜点区)** | **31.1 GB** | **40.8K** | ✅ **解 OOM** |
|
||||
|
||||
→ 同 batch:bf16 显存 −29% / tok/s +13%;**bf16 解 fp32-batch32 OOM**,batch32 达 40.8K tok/s(+29% vs fp32-b16)。KI-2 标 **FIXED**。
|
||||
</content>
|
||||
</invoke>
|
||||
|
||||
Reference in New Issue
Block a user