Adds ddp_dropout_is_live_and_p0_bit_identical, run via the real launcher
path (train_rank, world=2). It would have caught the original bug:
- GATE A: a p=0 DDP run is BIT-IDENTICAL (loss trace + final params) to the
no-dropout path — the regression guard for --dropout 0 (default).
- GATE B: a p=0.2 DDP run's loss trace DIFFERS measurably (>1e-3) from p=0.
On the pre-T21 code the model stays in eval mode, so p=0.2 would be an
identity and the trace would be bit-identical to p=0 — this gate fails.
- GATE C: model.is_training() == true after the run (direct proof that
train_rank called model.train() and it survived the final-step eval).
- p>0 run is finite (no NaN/Inf).
eval_every < steps so a periodic eval fires mid-run (flipping to eval mode),
exercising the per-step model.train() restore discipline the pilot called out.
Run with --test-threads=1 like the other DDP tests (shared-GPU deadlock).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Gahow Wang
1b58bd8626