docs: T21 — record DDP-dropout wiring gap + fix (known-issues / evolution / dropout doc)

- known-issues.md: new "DDP-dropout wiring" Fixed entry (gap + fix +
  regression test), with the meta-lesson that op/single-GPU unit tests can
  miss launcher-level integration gaps — only the V9-PILOT end-to-end run on
  the real launcher path exposed it.
- 17-dropout.md: annotate the DDP-combination note with the T18 wiring gap
  and its T21 fix.
- evolution.md: T21 row (Infra) recording the fix + meta-lesson.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-18 21:12:11 +08:00
parent c35d3851d2
commit 24c3da2f42
3 changed files with 15 additions and 0 deletions

View File

@@ -29,6 +29,7 @@
| T16 | 算法/Infra | **梯度累积**N micro-step每个 micro-loss `×1/N` backwardtape SUM 累加 一次 AdamW step+zero`--accum-steps`**DDP 只在累积边界 all-reduce**中间 micro-step 不发 NCCL`/world` `1/N` 正交显存随 micro 不随有效 batch | 等效大 batch**逐位贴合**loss rel 8.5e-8grad rel 3.8e-5`accum=1` 逐位回归(0.00)DDP+accum 对单卡 loss 5.7e-7/ rank 一致**显存平**同有效 batch 64big-batch 27.7GBaccum(4×16) **7.2GB(74%)**big-batch OOM accum 装下全回归+xserv 闭环 md5 一致 |
| T18 | 算法 | **dropout**手写 counter-based 设备 RNG Bernoulli mask训练 inverted 1/(1-p) scalingeval 恒等 autodiff `dropout` 算子fwd 生成+施加 maskbwd 用同 mask residual/ffn 两处`--dropout` flag 默认 0 | 固定 seed grad-check E[out]≈input + keep1-p**p=0 与无 dropout 逐位一致**recompute(T13) 组合下梯度仍逐位一致counter-based seed 重算复现同 mask全回归 + xserv 闭环绿导出/推理 dropout |
| T17 | Infra | **process-per-GPU**torchrun `launch_processes` 每卡 spawn 一个 worker 进程=独立 CUDA contextlauncher 一次性铸 `ncclUniqueId` **hex 编码注入子进程 env**——无共享 FS/TCP无竞态worker envbind device`DdpContext::init`+`build_model`+`train_rank` **全复用 T8 零改动** `train_ddp_mp` bin/`ddp_proc` test**保留 thread-per-GPU 旧路径**scope=process-per-GPU onlyZeRO-1 用户 dropPhase 2 | 正确性全绿proc vs 单卡 loss 5.67e-7、**proc vs thread-per-GPU 1.5e-7**、 rank 1.19e-7(<1e-6)、全回归+xserv 闭环 md5 逐位一致 `b04fc9f9`。**⚠️关键发现实测证伪原假设本尺度 process-per-GPU 对吞吐中性**——thread vs proc @ {1,2,4,8} = {1.00/1.61/2.98/**5.27**}× vs {1.00/1.60/2.94/**5.31**}×<1% 噪声内8 卡全 9599% util 残留 ~5.3×@8 非线性是 **NCCL all-reduce + 本机 PCIe 拓扑墙****** CUDA context 串行KI-5/T11 doc 的猜想被钉死推翻方法论同 T11 证伪分桶 all-reduce」)。净价值=落地 torchrun 式标准链路 + 把误导性 backlog 项实测关闭默认训练路径不变 |
| T21 | Infra | **DDP-dropout wiring fix**V9-PILOT 暴露T18 只把 dropout 接进单卡 `train.rs``train_ddp` bin `--dropout` flag`train_rank` 从不调 `model.train()` DDP dropout 被静默忽略`--dropout` flag + `train_rank` 每步 `model.train()`镜像单卡 train/eval 纪律——`eval_loss` eval 后由每步 `train()` restore DDP-dropout 回归测试堵缺口 | DDP-dropout 回归测试绿p>0 下 dropout **live**loss 轨迹对 p=0 有可观差异pre-T21 会逐位相同、p=0 对无 dropout 路径**逐位一致**、run 后 `is_training()==true`;既有 DDP loss-match/跨 rank 测试不变。**元教训op/单卡单元测试漏掉 launcher 级 integration gap只有真实启动器端到端跑pilot才暴露** |
---