Adds ddp_dropout_is_live_and_p0_bit_identical, run via the real launcher
path (DdpContext::init + train_rank). It would have caught the original bug:
- GATE A (world=1, ONE step — the deterministic scope): a p=0 run is
BIT-IDENTICAL (step loss + post-step params) to the no-dropout path.
ops::dropout(p=0) is a clone no-op regardless of training mode. At world=1
the NCCL all-reduce short-circuits; bit-identity holds for a single
forward+backward (the engine's atomicAdd backward order compounds a few
ULP only over many AdamW steps — the known fresh-train md5 caveat — so the
honest bit-identity scope is one step, no optimizer-state compounding).
- GATE A2 (world=2): p=0 matches a separate no-dropout baseline within NCCL's
run-to-run ULP noise (< 1e-6, KI-5 — the all-reduce is not bit-reproducible
on this PCIe box). Enabling dropout=0 doesn't perturb the DDP path beyond it.
- GATE B (world=2): a p=0.2 run's loss trace DIFFERS by > 1e-3 from p=0 —
orders of magnitude above the KI-5 noise floor. On the pre-T21 code the
model stays in eval mode, so p=0.2 would be an identity and the trace would
match p=0 at the noise floor — this gate fails.
- GATE C: model.is_training() == true after the run (direct proof that
train_rank called model.train() and it survived the final-step eval).
- p>0 run is finite (no NaN/Inf).
eval_every < steps so a periodic eval fires mid-run (flipping to eval mode),
exercising the per-step model.train() restore discipline the pilot called out.
Run with --test-threads=1 like the other DDP tests (shared-GPU deadlock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gahow Wang
a447631c4b