docs: T21 — record DDP-dropout wiring gap + fix (known-issues / evolution / dropout doc)
- known-issues.md: new "DDP-dropout wiring" Fixed entry (gap + fix + regression test), with the meta-lesson that op/single-GPU unit tests can miss launcher-level integration gaps — only the V9-PILOT end-to-end run on the real launcher path exposed it. - 17-dropout.md: annotate the DDP-combination note with the T18 wiring gap and its T21 fix. - evolution.md: T21 row (Infra) recording the fix + meta-lesson. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -29,6 +29,7 @@
|
||||
| T16 | 算法/Infra | **梯度累积**(N 个 micro-step:每个 micro-loss `×1/N` 再 backward,tape SUM 累加 → 一次 AdamW step+zero;`--accum-steps`);**DDP 只在累积边界 all-reduce**(中间 micro-step 不发 NCCL,`/world` 与 `1/N` 正交);显存随 micro 不随有效 batch | 等效大 batch**逐位贴合**(loss rel 8.5e-8、grad rel 3.8e-5);`accum=1` 逐位回归(0.00);DDP+accum 对单卡 loss 5.7e-7/跨 rank 一致;**显存平**:同有效 batch 64,big-batch 27.7GB→accum(4×16) **7.2GB(−74%)**(big-batch OOM 而 accum 装下);全回归+xserv 闭环 md5 一致 |
|
||||
| T18 | 算法 | **dropout**(手写 counter-based 设备 RNG → Bernoulli mask,训练 inverted 1/(1-p) scaling、eval 恒等);新 autodiff `dropout` 算子(fwd 生成+施加 mask,bwd 用同 mask),接 residual/ffn 两处;`--dropout` flag 默认 0 | 固定 seed grad-check 过;E[out]≈input + keep≈1-p;**p=0 与无 dropout 逐位一致**;recompute(T13) 组合下梯度仍逐位一致(counter-based seed 重算复现同 mask);全回归 + xserv 闭环绿(导出/推理 dropout 关) |
|
||||
| T17 | Infra | **process-per-GPU**(torchrun 式:`launch_processes` 每卡 spawn 一个 worker 进程=独立 CUDA context;launcher 一次性铸 `ncclUniqueId` 后 **hex 编码注入子进程 env**——无共享 FS/TCP、无竞态;worker 读 env→bind device→`DdpContext::init`+`build_model`+`train_rank` **全复用 T8 零改动**;新 `train_ddp_mp` bin/`ddp_proc` test,**保留 thread-per-GPU 旧路径**);scope=process-per-GPU only(ZeRO-1 用户 drop)(Phase 2) | 正确性全绿:proc vs 单卡 loss 5.67e-7、**proc vs thread-per-GPU 1.5e-7**、跨 rank 1.19e-7(<1e-6)、全回归+xserv 闭环 md5 逐位一致 `b04fc9f9`。**⚠️关键发现(实测证伪原假设):本尺度 process-per-GPU 对吞吐中性**——thread vs proc @ {1,2,4,8} = {1.00/1.61/2.98/**5.27**}× vs {1.00/1.60/2.94/**5.31**}×(差<1% 噪声内);8 卡全 95–99% util ⇒ 残留 ~5.3×@8 非线性是 **NCCL all-reduce + 本机 PCIe 拓扑墙**,**非**单 CUDA context 串行(KI-5/T11 doc 的猜想被钉死推翻,方法论同 T11 证伪「分桶 all-reduce」)。净价值=落地 torchrun 式标准链路 + 把误导性 backlog 项实测关闭;默认训练路径不变 |
|
||||
| T21 | Infra | **DDP-dropout wiring fix**(V9-PILOT 暴露:T18 只把 dropout 接进单卡 `train.rs`,`train_ddp` bin 无 `--dropout` flag、`train_rank` 从不调 `model.train()` → DDP 下 dropout 被静默忽略。补:`--dropout` flag + `train_rank` 每步 `model.train()`,镜像单卡 train/eval 纪律——`eval_loss` 翻 eval 后由每步 `train()` restore);加 DDP-dropout 回归测试堵缺口 | DDP-dropout 回归测试绿:p>0 下 dropout **live**(loss 轨迹对 p=0 有可观差异,pre-T21 会逐位相同)、p=0 对无 dropout 路径**逐位一致**、run 后 `is_training()==true`;既有 DDP loss-match/跨 rank 测试不变。**元教训:op/单卡单元测试漏掉 launcher 级 integration gap,只有真实启动器端到端跑(pilot)才暴露** |
|
||||
|
||||
---
|
||||
|
||||
|
||||
Reference in New Issue
Block a user