dropout: wire into model (residual sites) + train/eval switch + flag (T18)
Config.dropout (default 0). TinyTransformer gets a Cell<bool> training switch (train()/eval()/with_training, default eval = safe) + a Cell<u64> step_seed bumped once per training forward. forward_batched derives a per-layer block_seed (pure fn of step_seed×layer) and block_forward derives two per-site seeds, inserting ops::dropout at the attn and ffn sub-block outputs (before each residual). The seed is a pure function of (step_seed, layer, site) so the checkpoint (T13) recompute re-derives the same masks → grads stay exact. p=0 or eval → no dropout node → graph bit-identical to pre-T18. train_loop: model.train() per step (restored after eval flips to eval); eval_loss runs model.eval(). bin/train: --dropout flag → cfg.dropout. Export/sampling run in eval (default), so exported weights are dropout-free (xserv closed loop unaffected). Model-level tests (dropout.rs): p=0 bit-identical to no-dropout (logits/loss/grads); eval(p>0) == p=0 identity; train differs from eval + finite; recompute-with-dropout grads match non-recompute (fp32 + bf16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -20,6 +20,11 @@ pub struct Config {
|
||||
pub eps: f32,
|
||||
/// RoPE base frequency (theta).
|
||||
pub rope_theta: f32,
|
||||
/// Dropout probability `p` (Phase T18). Applied at the attention/MLP sub-block
|
||||
/// outputs (before each residual add) at TRAINING time, with inverted scaling
|
||||
/// `1/(1-p)`; disabled (identity) at eval. Default `0.0` = no dropout, and the
|
||||
/// forward graph is then bit-identical to the pre-T18 path.
|
||||
pub dropout: f32,
|
||||
}
|
||||
|
||||
impl Config {
|
||||
@@ -36,6 +41,7 @@ impl Config {
|
||||
ffn_hidden: 64,
|
||||
eps: 1e-5,
|
||||
rope_theta: 10000.0,
|
||||
dropout: 0.0,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -60,6 +66,7 @@ impl Config {
|
||||
ffn_hidden,
|
||||
eps: 1e-5,
|
||||
rope_theta: 10000.0,
|
||||
dropout: 0.0,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user