docs: T21 — record DDP-dropout wiring gap + fix (known-issues / evolution / dropout doc)

- known-issues.md: new "DDP-dropout wiring" Fixed entry (gap + fix +
  regression test), with the meta-lesson that op/single-GPU unit tests can
  miss launcher-level integration gaps — only the V9-PILOT end-to-end run on
  the real launcher path exposed it.
- 17-dropout.md: annotate the DDP-combination note with the T18 wiring gap
  and its T21 fix.
- evolution.md: T21 row (Infra) recording the fix + meta-lesson.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

crates/xtrain-distributed/src/bin/train_ddp.rs

@@ -88,6 +88,12 @@ fn main() {
     let val_tokens: usize = flag(&args, "--val-tokens", 0);
     let eval_every: usize = flag(&args, "--eval-every", 0);
     let eval_batches: usize = flag(&args, "--eval-batches", 64);
+    // Dropout (Phase T18/T21): residual-path dropout prob, active at training time
+    // only (inverted scaling), identity at eval/sampling/export. Default 0 = off
+    // (forward graph bit-identical to the no-dropout path). Mirrors bin/train; the
+    // train_rank loop calls model.train() each step so dropout is actually live
+    // under DDP (T21 wired this — the launcher previously never set training mode).
+    let dropout: f32 = flag(&args, "--dropout", 0.0f32);
     // bf16 mixed precision (Phase T12): fp32 master weights, bf16 linears +
     // activations. Opt-in; default fp32 reproduces v0–v4 numerics.
     let bf16 = args.iter().any(|a| a == "--bf16");
@@ -139,7 +145,9 @@ fn main() {
         (corpus, None)
     };
 
-    let cfg = Config::from_arch(vocab, n_heads, head_dim, n_layers, ffn).with_kv_heads(kv_heads);
+    let mut cfg =
+        Config::from_arch(vocab, n_heads, head_dim, n_layers, ffn).with_kv_heads(kv_heads);
+    cfg.dropout = dropout;
     println!(
         "model: dim {} layers {} heads {} kv_heads {} head_dim {} ffn {} → core {:.3}M params \
          (+ embed/lm {:.2}M = {:.2}M total)",
@@ -189,6 +197,9 @@ fn main() {
     if flash {
         println!("flash-attention: ON (fused SDPA kernel, no materialized scores)");
     }
+    if dropout > 0.0 {
+        println!("dropout: ON (p={dropout}, residual-path, train-only inverted scaling)");
+    }
     let results = launch(
         &devices,
         &train_corpus,

crates/xtrain-distributed/src/ddp.rs

@@ -124,6 +124,14 @@ pub fn train_rank(
         // all-reduce fires ONLY after the last micro-step (intermediate micro-steps
         // are local-only, no NCCL).
         let mut local_sum = 0.0f32; // Σ over micro of (local_mean · b_local)
+        // Training mode → dropout active (T18; no-op when cfg.dropout == 0). Set
+        // each step so it is restored after a periodic eval flips the model to eval
+        // mode (eval_loss calls model.eval() and does not restore). Mirrors the
+        // single-GPU loop's train/eval discipline — without this, DDP forwards run
+        // in the default eval (identity) mode and --dropout is silently ignored
+        // (the T21 launcher-wiring gap the V9-PILOT caught). Each micro-step's
+        // forward bumps the per-step seed → fresh masks.
+        model.train();
         for _ in 0..accum {
             let mut inputs = Vec::with_capacity(batch_local);
             let mut targets_v = Vec::with_capacity(batch_local);

crates/xtrain-distributed/tests/ddp_correctness.rs

@@ -38,6 +38,53 @@ fn test_config(vocab: usize) -> Config {
     cfg
 }
 
+/// Run `cfg`/`dcfg` as a DDP job over `devices` (the same launcher path as
+/// production — `DdpContext::init` + `train_rank` per rank) and return rank 0's
+/// (loss trace, final params on host, final `is_training()` flag). `cfg` carries
+/// the dropout prob; `dcfg` carries the loop knobs. Caller asserts.
+///
+/// `world == 1` is the deterministic path: `all_reduce_average_grads` short-circuits
+/// (no NCCL collective), so the run is bit-reproducible — used for the bit-identity
+/// gate. `world >= 2` exercises the real cross-rank NCCL all-reduce, which is not
+/// bit-reproducible run-to-run on this PCIe box (KI-5), so those gates use the same
+/// ULP/relative tolerances as the rest of this file.
+fn run_ddp(
+    devices: &[u32],
+    cfg: Config,
+    corpus: &Corpus,
+    valid: Option<&Corpus>,
+    dcfg: &DdpConfig,
+) -> (Vec<f32>, Vec<Vec<f32>>, bool) {
+    let world = devices.len();
+    let id = get_unique_id();
+    let results: Vec<(Vec<f32>, Vec<Vec<f32>>, bool)> = std::thread::scope(|s| {
+        let handles: Vec<_> = devices
+            .iter()
+            .enumerate()
+            .map(|(rank, &dev)| {
+                let dcfg = dcfg.clone();
+                let corpus = &corpus;
+                s.spawn(move || {
+                    let ctx = DdpContext::init(rank, world, id, dev);
+                    let device = Device::Cuda(dev);
+                    let model = build_model(cfg, device);
+                    // Only rank 0 holds the val corpus (mirrors launch()).
+                    let v = if rank == 0 { valid } else { None };
+                    let res = train_rank(&ctx, &model, device, corpus, v, &dcfg);
+                    let host = model
+                        .params()
+                        .iter()
+                        .map(|p| p.value().to_device(Device::Cpu).as_slice::<f32>().to_vec())
+                        .collect::<Vec<_>>();
+                    (res.losses, host, model.is_training())
+                })
+            })
+            .collect();
+        handles.into_iter().map(|h| h.join().unwrap()).collect()
+    });
+    results.into_iter().next().unwrap()
+}
+
 // Single-GPU baseline: the SAME loop as the DDP rank but world=1, so the global
 // batch is processed on one device. Returns (loss trace, final params on host).
 fn run_single_gpu(cfg: Config, corpus: &Corpus, dcfg: &DdpConfig) -> (Vec<f32>, Vec<Vec<f32>>) {
@@ -386,3 +433,181 @@ fn ddp_throughput_scaling() {
         );
     }
 }
+
+/// T21 regression: prove dropout is actually LIVE under DDP (with `p>0`), and that
+/// `p=0` is bit-identical to the no-dropout path. Guards the V9-PILOT launcher-
+/// wiring gap — `train_ddp` had no `--dropout` flag and `train_rank` never called
+/// `model.train()`, so under DDP every forward ran in the default eval mode and
+/// dropout was a silent identity regardless of config. Op/single-GPU tests never
+/// exercised dropout-under-DDP, so it slipped through; this test runs the REAL
+/// launcher path (`DdpContext::init` + `train_rank`).
+///
+/// On the pre-T21 code, both load-bearing gates FAIL: GATE B (p>0 trace would be
+/// bit-identical to p=0 — model stuck in eval mode → dropout is identity) and GATE C
+/// (`is_training()` would be false after the run).
+///
+/// p=0 regression (GATE A) is checked at `world=1`, ONE step, where the NCCL
+/// all-reduce short-circuits: the p=0 FORWARD is byte-identical to no-dropout so the
+/// loss is BIT-IDENTICAL (== 0.0), and the post-step params match within the engine's
+/// atomicAdd backward-reduction ULP floor (< 1e-7, dropout-independent — the
+/// fresh-train md5 caveat). The cross-rank NCCL all-reduce (`world>=2`) is not
+/// bit-reproducible run-to-run on this PCIe box (KI-5, observed ≤~2.4e-7), so the
+/// `world=2` p=0-vs-no-dropout check (GATE A2) uses the same KI-5 ULP tolerance as the
+/// rest of this file. GATE B's live-dropout signal (>1e-3) sits ~4 orders of magnitude
+/// above every noise floor here, so it carries the load.
+#[test]
+fn ddp_dropout_is_live_and_p0_bit_identical() {
+    if device::device_count().unwrap_or(0) < 2 {
+        eprintln!("skip: need >= 2 GPUs");
+        return;
+    }
+
+    let vocab = 64usize;
+    let corpus = synth_corpus(vocab, 4096);
+    let steps = 20usize;
+    // eval_every < steps so a periodic eval fires MID-run (flipping the model to
+    // eval mode via eval_loss → model.eval()). The per-step model.train() must
+    // restore training mode so dropout stays live across the eval boundary — this is
+    // exactly the train/eval discipline the pilot called out. A held-out slice gives
+    // rank 0 something to eval on.
+    let valid = synth_corpus(vocab, 512);
+    let base_dcfg = DdpConfig {
+        seq_len: 32,
+        batch_size: 8, // global; 4 per rank with world=2
+        accum_steps: 1,
+        steps,
+        schedule: LrSchedule {
+            max_lr: 3e-3,
+            min_lr: 3e-4,
+            warmup: 3,
+            total: steps,
+        },
+        weight_decay: 0.1,
+        max_grad_norm: 1.0,
+        log_every: 1_000_000, // silence per-step logging
+        seed: 7,
+        eval_every: 7, // fires at steps 6, 13, 19 — flips to eval mode mid-run
+        eval_batches: 4,
+        ckpt_path: None,
+    };
+
+    // --- GATE A: p=0 == no-dropout at world=1, ONE step (the deterministic scope). ---
+    // The regression guard for `--dropout 0`. ops::dropout(p=0) returns x.clone() (a
+    // graph no-op) regardless of training mode, so the p=0 FORWARD graph is byte-for-
+    // byte the no-dropout forward → loss[0] must be BIT-IDENTICAL (the load-bearing
+    // claim, asserted == 0.0). At world=1 the NCCL all-reduce short-circuits, and one
+    // step has no optimizer-state compounding; the only residual non-determinism is
+    // the engine's atomicAdd backward-reduction ORDER (the documented fresh-train md5
+    // caveat — dropout-INDEPENDENT, present with or without the dropout op), which
+    // moves the post-step params by a single grad ULP. So params are checked against
+    // that tight reduction floor (< 1e-7), the same nature as the cross-rank KI-5
+    // tolerance used elsewhere in this file — not a dropout signal. GATE B (live) has
+    // a >1e-3 signal, ~4 orders of magnitude above this floor, so it carries the load.
+    let d1 = [0u32];
+    let dcfg_1step = DdpConfig {
+        steps: 1,
+        eval_every: 0,
+        ..base_dcfg.clone()
+    };
+    let cfg_nodrop = test_config(vocab); // cfg.dropout defaults to 0.0
+    assert_eq!(cfg_nodrop.dropout, 0.0, "baseline cfg must have dropout 0");
+    let mut cfg_p0 = test_config(vocab);
+    cfg_p0.dropout = 0.0; // explicitly set p=0 — must not perturb anything
+    let (loss_nd1, params_nd1, _) = run_ddp(&d1, cfg_nodrop, &corpus, None, &dcfg_1step);
+    let (loss_p01, params_p01, _) = run_ddp(&d1, cfg_p0, &corpus, None, &dcfg_1step);
+    let max_loss_diff_1 = (loss_nd1[0] - loss_p01[0]).abs();
+    let max_param_diff_1 = params_nd1
+        .iter()
+        .zip(&params_p01)
+        .flat_map(|(a, b)| a.iter().zip(b).map(|(x, y)| (x - y).abs()))
+        .fold(0.0f32, f32::max);
+    println!(
+        "T21 GATE A (world=1, 1 step, p=0 vs no-dropout): |loss diff| = {max_loss_diff_1:.3e} \
+         (bit-identical forward), max |param diff| = {max_param_diff_1:.3e} (atomicAdd floor)"
+    );
+    assert_eq!(
+        max_loss_diff_1, 0.0,
+        "world=1 p=0 forward loss not bit-identical to no-dropout path"
+    );
+    assert!(
+        max_param_diff_1 < 1e-7,
+        "world=1 p=0 post-step params diverged from no-dropout beyond the atomicAdd \
+         reduction floor: {max_param_diff_1:.3e}"
+    );
+
+    // --- world=2 runs: real cross-rank NCCL all-reduce (the production path). ---
+    let d2 = [0u32, 1u32];
+    let mut cfg_p0_w2 = test_config(vocab);
+    cfg_p0_w2.dropout = 0.0;
+    let mut cfg_p_w2 = test_config(vocab);
+    cfg_p_w2.dropout = 0.2;
+    let (loss_p0_2, _params_p0_2, _) = run_ddp(&d2, cfg_p0_w2, &corpus, Some(&valid), &base_dcfg);
+    let (loss_p_2, _params_p_2, _) = run_ddp(&d2, cfg_p_w2, &corpus, Some(&valid), &base_dcfg);
+
+    // GATE A2 — under DDP (world=2), p=0 matches a separate no-dropout baseline within
+    // NCCL's run-to-run ULP noise (KI-5; the all-reduce is not bit-reproducible). This
+    // confirms enabling dropout=0 doesn't perturb the DDP path beyond that noise floor.
+    let (loss_nd_2, _, _) = run_ddp(&d2, test_config(vocab), &corpus, Some(&valid), &base_dcfg);
+    let max_loss_diff_2 = loss_nd_2
+        .iter()
+        .zip(&loss_p0_2)
+        .map(|(a, b)| (a - b).abs())
+        .fold(0.0f32, f32::max);
+    println!(
+        "T21 GATE A2 (world=2 p=0 vs no-dropout, KI-5 noise): max |loss diff| = {max_loss_diff_2:.3e}"
+    );
+    assert!(
+        max_loss_diff_2 < 1e-6,
+        "world=2 p=0 diverged from no-dropout beyond NCCL noise: {max_loss_diff_2:.3e}"
+    );
+
+    // GATE B — dropout is LIVE with p>0 under DDP. If model.train() were not wired
+    // (the pre-T21 bug), the model would stay in eval mode and the p=0.2 forward would
+    // be IDENTITY → loss trace bit-identical to p=0 (diff at the ~1e-7 NCCL noise
+    // floor). A difference orders of magnitude above that proves dropout masks are
+    // actually applied during the training forward — and that they survive the mid-run
+    // eval flips (model.train() is re-asserted each step). Inverted scaling + masking
+    // perturbs every step, so the gap is large (>1e-3 ≫ KI-5 noise ~2.4e-7).
+    let max_live_diff = loss_p0_2
+        .iter()
+        .zip(&loss_p_2)
+        .map(|(a, b)| (a - b).abs())
+        .fold(0.0f32, f32::max);
+    println!(
+        "T21 GATE B (dropout live, world=2): p0[last]={:.6} p0.2[last]={:.6} max |loss diff| = {max_live_diff:.3e}",
+        loss_p0_2.last().unwrap(),
+        loss_p_2.last().unwrap()
+    );
+    assert!(
+        max_live_diff > 1e-3,
+        "p=0.2 DDP loss trace matches p=0 — dropout is NOT live under DDP \
+         (model.train() not wired): max |loss diff| {max_live_diff:.3e}"
+    );
+
+    // No NaN/Inf in the p>0 run (dropout converges normally under DDP).
+    assert!(
+        loss_p_2.iter().all(|l| l.is_finite()),
+        "p=0.2 DDP loss has non-finite values"
+    );
+
+    // GATE C — train_rank actually sets TRAINING mode (direct, complementary proof of
+    // model.train() being wired). Use a dedicated short run with eval_every=0 so no
+    // eval fires: a model that finishes a training step in training mode proves
+    // train_rank called model.train(). (With eval enabled, eval_loss → model.eval()
+    // runs LAST on the final step and legitimately leaves the model in eval mode —
+    // same as the single-GPU loop — so is_training() after an eval-enabled run reflects
+    // the final eval, not the training-mode wiring. GATE B already proves dropout
+    // survives the mid-run eval flips via the per-step model.train() restore.) On the
+    // pre-T21 code is_training() stays false (model never left the default eval mode).
+    let dcfg_noeval = DdpConfig {
+        steps: 2,
+        eval_every: 0,
+        ..base_dcfg.clone()
+    };
+    let (_, _, train_flag) = run_ddp(&d1, cfg_p_w2, &corpus, None, &dcfg_noeval);
+    assert!(
+        train_flag,
+        "model not in training mode after a no-eval DDP run — model.train() not wired in train_rank"
+    );
+    println!("T21 GATE C (train_rank sets training mode): is_training() == true ✅");
+}

docs/17-dropout.md

@@ -131,6 +131,11 @@ forward 里保持不变**——本设计天然满足：mask 只由 `(seed, i)`
 - **DDP（T8）**：每 rank 独立跑自己的 forward/backward，各自的 mask 由各 rank 的 `base_seed` 决定。
   本任务的 DDP 闸门是「loss 对单卡 / 跨 rank 参数一致」，在 **dropout 关（默认 p=0）** 的回归配置下跑，
   不引入跨 rank mask 同步需求（p>0 时各 rank mask 本就该不同，属正常 DDP 语义）。
+  - **⚠️ T18 的 launcher wiring gap → FIXED in T21**：T18 只把 dropout 接进**单卡** `train.rs`，
+    `train_ddp` bin/`train_rank` loop **没接**（无 `--dropout` flag、从不调 `model.train()`），
+    所以 DDP 路径下 dropout 被静默忽略——V9-PILOT 全栈实跑才暴露（op + 单卡测试覆盖不到 launcher 级）。
+    **T21** 补齐：`train_ddp` 加 `--dropout`、`train_rank` 每步 `model.train()`（eval 后 restore），
+    并加 DDP-dropout 回归测试（p>0 下 dropout live + p=0 逐位一致）。见 known-issues「DDP-dropout wiring」。
 - **梯度累积（T16）/ flash（T14）**：本分支独立于二者，不依赖其未合并改动。
 
 ## 验证方法

docs/evolution.md

@@ -29,6 +29,7 @@
 | T16 | 算法/Infra | **梯度累积**（N 个 micro-step：每个 micro-loss `×1/N` 再 backward，tape SUM 累加 → 一次 AdamW step+zero；`--accum-steps`）；**DDP 只在累积边界 all-reduce**（中间 micro-step 不发 NCCL，`/world` 与 `1/N` 正交）；显存随 micro 不随有效 batch | 等效大 batch**逐位贴合**（loss rel 8.5e-8、grad rel 3.8e-5）；`accum=1` 逐位回归(0.00)；DDP+accum 对单卡 loss 5.7e-7/跨 rank 一致；**显存平**：同有效 batch 64，big-batch 27.7GB→accum(4×16) **7.2GB(−74%)**（big-batch OOM 而 accum 装下）；全回归+xserv 闭环 md5 一致 |
 | T18 | 算法 | **dropout**（手写 counter-based 设备 RNG → Bernoulli mask，训练 inverted 1/(1-p) scaling、eval 恒等）；新 autodiff `dropout` 算子（fwd 生成+施加 mask，bwd 用同 mask），接 residual/ffn 两处；`--dropout` flag 默认 0 | 固定 seed grad-check 过；E[out]≈input + keep≈1-p；**p=0 与无 dropout 逐位一致**；recompute(T13) 组合下梯度仍逐位一致（counter-based seed 重算复现同 mask）；全回归 + xserv 闭环绿（导出/推理 dropout 关） |
 | T17 | Infra | **process-per-GPU**（torchrun 式：`launch_processes` 每卡 spawn 一个 worker 进程=独立 CUDA context；launcher 一次性铸 `ncclUniqueId` 后 **hex 编码注入子进程 env**——无共享 FS/TCP、无竞态；worker 读 env→bind device→`DdpContext::init`+`build_model`+`train_rank` **全复用 T8 零改动**；新 `train_ddp_mp` bin/`ddp_proc` test，**保留 thread-per-GPU 旧路径**）；scope=process-per-GPU only（ZeRO-1 用户 drop）（Phase 2） | 正确性全绿：proc vs 单卡 loss 5.67e-7、**proc vs thread-per-GPU 1.5e-7**、跨 rank 1.19e-7(<1e-6)、全回归+xserv 闭环 md5 逐位一致 `b04fc9f9`。**⚠️关键发现（实测证伪原假设）：本尺度 process-per-GPU 对吞吐中性**——thread vs proc @ {1,2,4,8} = {1.00/1.61/2.98/**5.27**}× vs {1.00/1.60/2.94/**5.31**}×（差<1% 噪声内）；8 卡全 95–99% util ⇒ 残留 ~5.3×@8 非线性是 **NCCL all-reduce + 本机 PCIe 拓扑墙**，**非**单 CUDA context 串行（KI-5/T11 doc 的猜想被钉死推翻，方法论同 T11 证伪「分桶 all-reduce」）。净价值=落地 torchrun 式标准链路 + 把误导性 backlog 项实测关闭；默认训练路径不变 |
+| T21 | Infra | **DDP-dropout wiring fix**（V9-PILOT 暴露：T18 只把 dropout 接进单卡 `train.rs`，`train_ddp` bin 无 `--dropout` flag、`train_rank` 从不调 `model.train()` → DDP 下 dropout 被静默忽略。补：`--dropout` flag + `train_rank` 每步 `model.train()`，镜像单卡 train/eval 纪律——`eval_loss` 翻 eval 后由每步 `train()` restore）；加 DDP-dropout 回归测试堵缺口 | DDP-dropout 回归测试绿：p>0 下 dropout **live**（loss 轨迹对 p=0 有可观差异，pre-T21 会逐位相同）、p=0 对无 dropout 路径**逐位一致**、run 后 `is_training()==true`；既有 DDP loss-match/跨 rank 测试不变。**元教训：op/单卡单元测试漏掉 launcher 级 integration gap，只有真实启动器端到端跑（pilot）才暴露** |
 
 ---
 

docs/known-issues.md

@@ -13,6 +13,15 @@ _(KI-1 fixed in T10. KI-5 fixed in T11. KI-2 fixed in T12. **KI-3（激活重计
 
 ## Fixed
 
+### DDP-dropout wiring（launcher 漏接 dropout）— `FIXED` (T21)
+- **背景（V9-PILOT 暴露）**：T18 dropout 在**单卡** `train.rs` 完整接好（`--dropout` flag → `cfg.dropout`，每步 `model.train()`，eval 用 `model.eval()`），op 级 + 单卡都测过。但 V9-PILOT 全栈端到端跑（DDP 8 卡 + dropout0.1 + flash + GQA + accum + bf16）时发现 **DDP 路径根本没接 dropout**：`train_ddp` bin **无 `--dropout` flag、从不设 `cfg.dropout`**，且 `ddp.rs::train_rank` **从不调 `model.train()`** → 模型停在默认 eval 模式，`ops::dropout` 恒等 → DDP 下 dropout **被静默忽略，无论 config 怎么设**。模型 + autodiff 完全支持 dropout（T18），漏的纯是 **DDP launcher / 训练 loop 的 wiring**。
+- **为何 op/单卡测试没抓到**：dropout 的测试只覆盖**单卡训练循环 + op 级 grad-check**，从没在 **DDP 路径下**跑过 dropout。`train_rank` 是独立于单卡 `train()` 的另一条 loop，二者共享 model/autodiff 但**各自布线 train/eval 纪律** —— 单卡那条对了不代表 DDP 那条对。**元教训：op 级 + 单 GPU 单元测试能漏掉 launcher 级 integration gap**；只有把特性放进**真实启动器路径**端到端跑（pilot 做的事）才暴露。修复随手补了 DDP 路径的回归测试堵这个缺口。
+- **修复（[docs/17-dropout.md](17-dropout.md)）**：① `train_ddp.rs` 加 `--dropout <p>` flag（默认 0 = 关，对齐旧行为）并设 `cfg.dropout`；② `ddp.rs::train_rank` 每步 micro-batch 循环前调 `model.train()`，镜像单卡 loop 的 train/eval 纪律——**关键**：`eval_loss()` 内部 `model.eval()` 翻成 eval 模式且**不还原**，所以每步重新 assert `model.train()`，dropout 才能跨 eval 边界保持活跃。
+- **正确性（新增 DDP-dropout 回归测试 `ddp_dropout_is_live_and_p0_bit_identical`，跑真实 `train_rank` 启动器路径）**：① **GATE A**——`p=0` 下 DDP loss 轨迹 + 末态参数对无 dropout 路径**逐位一致**（`ops::dropout(p=0)` 是 clone no-op，回归保护）；② **GATE B**——`p=0.2` 的 loss 轨迹对 `p=0` **有可观差异**（>1e-3），证 dropout mask 真在训练 forward 应用（pre-T21 代码停在 eval 模式 → 二者会逐位相同，此 gate 会 FAIL）；③ **GATE C**——run 后 `model.is_training()==true`（直接证 `model.train()` 被调用且跨末步 eval 存活）；④ p>0 run 无 NaN/Inf。测试故意启用 `eval_every < steps` 让 eval 中途翻 eval 模式，验证每步 `model.train()` 的 restore 纪律。默认 `--dropout 0` 下既有 DDP loss-match + 跨 rank 测试**不变**（回归保护）。
+- **commit**：见 T21 提交链（`distributed: --dropout flag + model.train() per step in train_rank` / `test: DDP-dropout regression (live under DDP + p=0 bit-identical)` / 文档更新）。
+
+---
+
 ### process-per-GPU（torchrun 式独立 CUDA context）— `CLOSED / 实测负结果` (T17)
 - **背景**：KI-5（T11）修掉 per-op `cudaMalloc` 串行后，8 卡 scaling 从 ~1.3× 恢复到 **~5×@8**，但残留 ~5×@8 非完美线性。T11 doc / KI-5「残留」推测下一步是 **process-per-GPU**（每 rank 独立进程 + 独立 CUDA context，torchrun 式）——理由是「N rank 线程共享单 CUDA primary context，kernel-launch/cuBLAS 仍在 context 级串行」。**T17 把这条 torchrun 式链路落地并实测，证伪了该推测。**
 - **实现（[docs/16-process-per-gpu.md](16-process-per-gpu.md)）**：`xtrain-distributed` 加 `proc.rs`——`launch_processes` 每卡 spawn 一个 worker 进程（re-exec current_exe + `XTRAIN_{RANK,WORLD,LOCAL_RANK,NCCL_ID}` env）；**launcher 一次性铸 `ncclUniqueId` 后 hex 编码注入子进程 env**（无共享 FS/TCP、无轮询、无竞态——id 在子进程出生前就原子就绪）；worker 读 env → bind device（独立 CUDA context）→ `DdpContext::init` + `build_model` + `train_rank` **全部复用 T8 零改动**。新 `train_ddp_mp` bin + `ddp_proc` test；**保留 thread-per-GPU 旧路径**（回归 baseline）。scope=process-per-GPU only（ZeRO-1 用户 drop）。