dropout: wire into model (residual sites) + train/eval switch + flag (T18)
Config.dropout (default 0). TinyTransformer gets a Cell<bool> training switch (train()/eval()/with_training, default eval = safe) + a Cell<u64> step_seed bumped once per training forward. forward_batched derives a per-layer block_seed (pure fn of step_seed×layer) and block_forward derives two per-site seeds, inserting ops::dropout at the attn and ffn sub-block outputs (before each residual). The seed is a pure function of (step_seed, layer, site) so the checkpoint (T13) recompute re-derives the same masks → grads stay exact. p=0 or eval → no dropout node → graph bit-identical to pre-T18. train_loop: model.train() per step (restored after eval flips to eval); eval_loss runs model.eval(). bin/train: --dropout flag → cfg.dropout. Export/sampling run in eval (default), so exported weights are dropout-free (xserv closed loop unaffected). Model-level tests (dropout.rs): p=0 bit-identical to no-dropout (logits/loss/grads); eval(p>0) == p=0 identity; train differs from eval + finite; recompute-with-dropout grads match non-recompute (fp32 + bf16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -89,6 +89,9 @@ pub fn train(
|
||||
}
|
||||
let ids = batched_ids_tensor(&inputs, device);
|
||||
let targets = batched_ids_tensor(&targets_v, device);
|
||||
// Training mode → dropout active (T18; no-op when cfg.dropout == 0). Set
|
||||
// each step so it is restored after a periodic eval flips to eval mode.
|
||||
model.train();
|
||||
let loss = model.loss_batched(&ids, &targets, cfg.batch_size);
|
||||
let step_loss = read_scalar(&loss);
|
||||
loss.backward();
|
||||
@@ -169,6 +172,8 @@ pub fn eval_loss(
|
||||
if valid.len() <= seq + 1 {
|
||||
return f32::NAN;
|
||||
}
|
||||
// Eval mode → dropout is identity (T18).
|
||||
model.eval();
|
||||
let n_win = (valid.len() - 1) / seq; // disjoint windows that fit
|
||||
let batches = batches.max(1).min(n_win.max(1));
|
||||
let stride = (n_win / batches).max(1);
|
||||
|
||||
Reference in New Issue
Block a user