Gahow Wang e625aa05dd dropout: wire into model (residual sites) + train/eval switch + flag (T18) ...

Config.dropout (default 0). TinyTransformer gets a Cell<bool> training switch
(train()/eval()/with_training, default eval = safe) + a Cell<u64> step_seed bumped
once per training forward. forward_batched derives a per-layer block_seed (pure fn
of step_seed×layer) and block_forward derives two per-site seeds, inserting
ops::dropout at the attn and ffn sub-block outputs (before each residual). The
seed is a pure function of (step_seed, layer, site) so the checkpoint (T13)
recompute re-derives the same masks → grads stay exact. p=0 or eval → no dropout
node → graph bit-identical to pre-T18.

train_loop: model.train() per step (restored after eval flips to eval); eval_loss
runs model.eval(). bin/train: --dropout flag → cfg.dropout. Export/sampling run in
eval (default), so exported weights are dropout-free (xserv closed loop unaffected).

Model-level tests (dropout.rs): p=0 bit-identical to no-dropout (logits/loss/grads);
eval(p>0) == p=0 identity; train differs from eval + finite; recompute-with-dropout
grads match non-recompute (fp32 + bf16).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-18 00:05:32 +08:00

..

xtrain-autodiff

dropout: autodiff op + fixed-seed grad-check (T18)

2026-06-18 00:05:32 +08:00

xtrain-cuda

dropout: device RNG kernel + Tensor fwd/bwd (T18)

2026-06-18 00:05:18 +08:00

xtrain-distributed

model: per-block activation recompute (--recompute)

2026-06-17 09:42:42 +08:00

xtrain-model

dropout: wire into model (residual sites) + train/eval switch + flag (T18)

2026-06-18 00:05:32 +08:00

xtrain-optim

perf: make xtrain-cuda a regular dep of xtrain-optim (GPU AdamW)

2026-06-15 16:53:52 +08:00

xtrain-tensor

dropout: device RNG kernel + Tensor fwd/bwd (T18)

2026-06-18 00:05:18 +08:00

xtrain-train

dropout: wire into model (residual sites) + train/eval switch + flag (T18)

2026-06-18 00:05:32 +08:00