gahow/xtrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: T16 grad-accum results — evolution row + README build-journey

Browse Source 
dash5-verified gate numbers: accum=N bit-close to N× big batch (loss
8.5e-8 / grad 3.8e-5), accum=1 bit-identical (0.0), DDP+accum matches
single-GPU (5.7e-7), memory flat (same effective batch 64: 27.7GB big →
7.2GB accum, −74%), xserv closed loop md5-identical + token-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-06-17 23:52:32 +08:00
parent b06b553f99
commit 8bd7db16e1
2 changed files with 9 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
README.md
View File

@@ -50,9 +50,13 @@ Each phase: design doc + implementation + tests + a scoped commit (see [`docs/`]
| **T11** | **device caching allocator** (fixes KI-5) | single-GPU 2.3×; **8-GPU 461K tok/s** |
| **T12** | **bf16 mixed precision** (fp32 master, fixes KI-2) | dim768 OOM solved; −29% mem |
| **T13** | **activation recompute** / checkpointing (fixes KI-3) | dim1024 fits; grads bit-identical |
| **T16** | **gradient accumulation** (`--accum-steps`; DDP all-reduces only at the boundary) | equiv to N× big batch (grad 3.8e-5); same effective-64 batch 27.7GB→7.2GB (−74%) |
The four performance fixes (T10–T13) each removed a real bottleneck — see
[`docs/known-issues.md`](docs/known-issues.md).
[`docs/known-issues.md`](docs/known-issues.md). **Phase 2 (systems-stack depth)**
revisits hand-writing deferred training-stack features; T16 = micro-batch gradient
accumulation ([`docs/15-grad-accum.md`](docs/15-grad-accum.md)), which decouples the
effective batch from activation memory (memory tracks the micro-batch, not N×).
## The scaling study — v0 → v8