dist: drop unused import; relax DDP-vs-single-GPU param tolerance
dash5 verify: loss trajectory matches single-GPU to max_rel 1.16e-7 and cross-rank params are bit-identical (0.0), but DDP-vs-single-GPU per-param rel diff is ~2.8e-3 after 20 AdamW steps — expected, since the two differ only in gradient summation order (fp add isn't associative) and that rounding compounds. Bump check (c) 1e-3 -> 1e-2 (a/b stay tight). Also remove an unused DType import. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -156,7 +156,7 @@ where
|
||||
/// one-element device buffer. Used only for the logged/returned loss, so the cost
|
||||
/// (one tiny collective per step) is negligible. Returns the summed value.
|
||||
fn all_reduce_loss(ctx: &DdpContext, local: f32) -> f32 {
|
||||
use xtrain_tensor::{DType, Tensor};
|
||||
use xtrain_tensor::Tensor;
|
||||
if ctx.world == 1 {
|
||||
return local;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user