post-train: M4 — clipped_pg_loss + scale_rows (GRPO policy-gradient op)
The GRPO (M4) token-level loss op + the one primitive it needs: - scale_rows(x[r,c], s[r]): per-row scale (new ~5-line CUDA kernel). The clipped-PG backward scales each completion token's row of (probs − onehot) by its own per-token coefficient, which cross_entropy_backward's single scalar scale can't express. - clipped_pg_loss(logits, target, logp_old, logp_ref, A, eps, beta): per-token ρ_t = exp(logπθ_t − logp_old_t), L = −mean min(ρA, clip(ρ,1±ε)A) + β·mean KL (k3 estimator), masked to completion tokens. Backward reuses the CE machinery (probs − onehot) + scale_rows. Gates: grad-check the active PG path + the A=0 (KL-only) path; degenerate value checks ε→∞ ⇒ vanilla PG, β=0 ⇒ no KL. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -941,6 +941,31 @@ impl Tensor {
|
||||
dx
|
||||
}
|
||||
|
||||
/// Per-row scale: `out[r,c] = self[r,c] * s[r]`. `self`:[rows,cols] F32,
|
||||
/// `s`:[rows] F32. Used by the GRPO (M4) policy-gradient backward, where each
|
||||
/// completion token's row of `(probs − onehot)` is scaled by its own per-token
|
||||
/// coefficient (the per-token clipped-PG + KL gradient). Forward-only.
|
||||
#[cfg(not(no_cuda))]
|
||||
pub fn scale_rows(&self, s: &Tensor) -> Self {
|
||||
assert_eq!(self.ndim(), 2, "scale_rows requires a 2D tensor");
|
||||
assert_eq!(self.dtype, DType::F32, "scale_rows is F32");
|
||||
assert_eq!(s.dtype, DType::F32, "scale vector is F32");
|
||||
let (rows, cols) = (self.shape[0], self.shape[1]);
|
||||
assert_eq!(s.numel(), rows, "scale vector must have one entry per row");
|
||||
let out = Tensor::zeros(&self.shape, DType::F32, self.device());
|
||||
unsafe {
|
||||
xtrain_cuda::ffi::launch_scale_rows_f32(
|
||||
self.data_ptr() as *const f32,
|
||||
s.data_ptr() as *const f32,
|
||||
out.data_ptr() as *mut f32,
|
||||
rows as i32,
|
||||
cols as i32,
|
||||
std::ptr::null_mut(),
|
||||
);
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
// --- Structural / model ops (the T5 kernels) ---
|
||||
|
||||
/// Reshape to `new_shape` (must keep `numel`). Pure metadata change on a
|
||||
|
||||
Reference in New Issue
Block a user