ops: embedding/reshape/transpose/split-merge-heads fwd+bwd
Phase T5 structural ops on top of the T4 set, needed to assemble the tiny transformer: - embedding: gather rows by I32 ids (CUDA kernel) / scatter-add backward (atomic, so repeated ids accumulate). csrc/ops/model.cu + ffi. - reshape: contiguous metadata-only view (Tensor::reshape), no kernel. - transpose_3d01: [a,b,c]->[b,a,c] for the multi-head layout (kernel). - autograd nodes: embedding/reshape/transpose_3d01/transpose_2d, plus split_heads (->Vec<Var>) / merge_heads for per-head attention. - tape: Var::zero_grad + set_value so a hand-written GD step can update params and clear grads between steps. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -68,6 +68,19 @@ impl Var {
|
||||
self.0.borrow().grad.clone()
|
||||
}
|
||||
|
||||
/// Clear the accumulated gradient. Call on every parameter between training
|
||||
/// steps so the next `backward` accumulates from zero (grads SUM otherwise).
|
||||
pub fn zero_grad(&self) {
|
||||
self.0.borrow_mut().grad = None;
|
||||
}
|
||||
|
||||
/// Overwrite this node's value tensor in place. Used by the optimizer to
|
||||
/// apply a parameter update (`p ← p − lr·grad`) while keeping the leaf's
|
||||
/// identity stable across steps.
|
||||
pub fn set_value(&self, value: Tensor) {
|
||||
self.0.borrow_mut().value = value;
|
||||
}
|
||||
|
||||
/// Pointer identity, used to dedup nodes during the topological sort.
|
||||
fn id(&self) -> *const RefCell<VarNode> {
|
||||
Rc::as_ptr(&self.0)
|
||||
|
||||
Reference in New Issue
Block a user