Adds ddp_dropout_is_live_and_p0_bit_identical, run via the real launcher
path (DdpContext::init + train_rank). It would have caught the original bug:
- GATE A (world=1, deterministic — all_reduce short-circuits, no NCCL): a
p=0 run is BIT-IDENTICAL (loss trace + final params) to the no-dropout
path. ops::dropout(p=0) is a clone no-op regardless of training mode.
- GATE A2 (world=2): p=0 matches a separate no-dropout baseline within
NCCL's run-to-run ULP noise (< 1e-6, KI-5 — the all-reduce is not
bit-reproducible on this PCIe box). Enabling dropout=0 doesn't perturb
the DDP path beyond that noise floor.
- GATE B (world=2): a p=0.2 run's loss trace DIFFERS by > 1e-3 from p=0 —
orders of magnitude above the KI-5 noise floor. On the pre-T21 code the
model stays in eval mode, so p=0.2 would be an identity and the trace
would match p=0 at the noise floor — this gate fails.
- GATE C: model.is_training() == true after the run (direct proof that
train_rank called model.train() and it survived the final-step eval).
- p>0 run is finite (no NaN/Inf).
eval_every < steps so a periodic eval fires mid-run (flipping to eval mode),
exercising the per-step model.train() restore discipline the pilot called out.
Run with --test-threads=1 like the other DDP tests (shared-GPU deadlock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gahow Wang
c35d3851d2