test: grad-accum equivalence + accum=1 bit-identity + DDP+accum

- grad_accum.rs: accum=N×B grads bit-close to a single N·B big batch;
  accum_steps=1 bit-identical (max|Δ|==0) to no-accum; real train() loop
  with accum tracks a big-batch baseline over 20 AdamW steps.
- ddp_correctness.rs: world=2 + accum=2 matches a single-GPU big batch of
  the same effective size (loss + cross-rank + vs-baseline).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-17 23:45:40 +08:00
parent 7a03b0054a
commit abe5ceb913
3 changed files with 419 additions and 0 deletions

View File

@@ -84,6 +84,7 @@ fn trains_on_tinystories() {
let tcfg = TrainConfig {
seq_len: 64,
batch_size: 8,
accum_steps: 1,
steps,
schedule: LrSchedule {
max_lr: 3e-3,