Gahow Wang
77a82bfeee
Training loop (train_loop.rs): sample batch_size sequences, forward loss + backward (tape SUMs grads), clip_grad_norm with ×1/batch averaging, AdamW step with scheduled lr, zero_grad; logs loss/lr/gnorm/tok-s and checkpoints periodically; returns the loss trace. Checkpoint (checkpoint.rs): flat little-endian dump of params() in order (magic/version/count + per-param ndim/dims/f32 data); load_into validates and overwrites a matching model's params via set_value (exact f32 round-trip). Sampler (sample.rs): autoregressive greedy / temperature generation — re-runs forward on the growing prefix (model is single-sequence, RoPE pos=row). bin/train.rs: end-to-end entry — load tokenizer+corpus, train a tiny 4-layer model for a bounded budget, checkpoint, print samples. no_cuda stub keeps it buildable on a GPU-less host. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
21 lines
725 B
TOML
21 lines
725 B
TOML
[package]
|
|
name = "xtrain-train"
|
|
version.workspace = true
|
|
edition.workspace = true
|
|
|
|
[dependencies]
|
|
xtrain-tensor = { path = "../xtrain-tensor" }
|
|
xtrain-autodiff = { path = "../xtrain-autodiff" }
|
|
xtrain-model = { path = "../xtrain-model" }
|
|
xtrain-optim = { path = "../xtrain-optim" }
|
|
xtrain-cuda = { path = "../xtrain-cuda" }
|
|
# Reuse xserv's from-scratch GPT-2/Qwen BPE (project decision). This relative
|
|
# path resolves on both ~/projects (local) and /opt/wjh/projects (dash5). The
|
|
# crate inherits xserv's workspace for its own deps (serde/regex) — Cargo reads
|
|
# the target package's workspace, not ours.
|
|
xserv-tokenizer = { path = "../../../xserv/crates/xserv-tokenizer" }
|
|
|
|
[[bin]]
|
|
name = "train"
|
|
path = "src/bin/train.rs"
|