Gahow Wang 81f3cf59e5 distributed: T21 — wire dropout into the DDP path (--dropout + model.train()) ...

V9-PILOT caught a launcher-level integration gap: T18 wired dropout into
the single-GPU bin/train, but the DDP path never did. train_ddp had no
--dropout flag and never set cfg.dropout, and ddp.rs::train_rank never
called model.train() — so under DDP every forward ran in the default eval
mode and dropout was a silent identity, regardless of config.

Fix, mirroring the single-GPU train/eval discipline:
- train_ddp.rs: add a --dropout <p> flag (default 0 = off, matching the
  prior behavior) and set cfg.dropout from it; log it when on.
- ddp.rs::train_rank: call model.train() at the start of each step (before
  the micro-batch loop). eval_loss() flips the model to eval mode and does
  not restore it, so re-asserting train() each step keeps dropout live
  across eval boundaries.

--dropout 0 (default) is bit-identical to the prior DDP path: cfg.dropout
stays 0 and ops::dropout(p=0) is a clone no-op regardless of training mode.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-18 21:08:17 +08:00

..

src

distributed: T21 — wire dropout into the DDP path (--dropout + model.train())

2026-06-18 21:08:17 +08:00

tests

test: process-per-GPU DDP correctness (ddp_proc.rs)

2026-06-18 17:48:52 +08:00

build.rs

dist: nccl ffi + comm bootstrap

2026-06-15 17:14:56 +08:00

Cargo.toml

dist: ddp all-reduce + sharded batch

2026-06-15 17:15:29 +08:00