data: gpt2 bpe via xserv-tokenizer + TinyStories corpus + lr schedule + grad clip

New xtrain-train crate scaffold. Data pipeline reuses xserv's from-scratch
GPT-2/Qwen BPE via a path-dep (../../../xserv/crates/xserv-tokenizer, resolves
on both ~/projects and dash5 /opt/wjh/projects): Corpus::load tokenizes the
corpus into one id stream and samples fixed-length (input, target) next-token
windows (LCG-seeded, reproducible). Trims a range-downloaded file to whole
stories (<|endoftext|> boundaries).

Also the host-only training math: LrSchedule (linear warmup + cosine decay)
and global L2 grad-norm + clip scale, each with a local unit test.

Corpus: data/tinystories-valid-3mb.txt — first ~3MB of TinyStories-valid
(fetched on dash5 via hf-mirror.com; HF direct unreachable). Substitution
noted: a real TinyStories subset, not the full set.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 16:29:32 +08:00
parent f22429f5b8
commit 7d84a64f5c
9 changed files with 23007 additions and 0 deletions

View File

@@ -0,0 +1,12 @@
//! Training stack (Phase T6): LR schedule, global-norm grad clipping, checkpoint
//! save/load, the GPT-2 BPE data pipeline (reusing xserv's tokenizer), an
//! autoregressive sampler, and the training loop that wires them onto the T5
//! `TinyTransformer` + the hand-written AdamW (`xtrain-optim`).
//!
//! Host-only pieces (LR schedule, grad-norm math) always compile so the crate
//! `cargo check`s on a GPU-less host; everything that touches GPU tensors is
//! gated behind `not(no_cuda)`.
pub mod clip;
pub mod data;
pub mod schedule;