data: gpt2 bpe via xserv-tokenizer + TinyStories corpus + lr schedule + grad clip
New xtrain-train crate scaffold. Data pipeline reuses xserv's from-scratch GPT-2/Qwen BPE via a path-dep (../../../xserv/crates/xserv-tokenizer, resolves on both ~/projects and dash5 /opt/wjh/projects): Corpus::load tokenizes the corpus into one id stream and samples fixed-length (input, target) next-token windows (LCG-seeded, reproducible). Trims a range-downloaded file to whole stories (<|endoftext|> boundaries). Also the host-only training math: LrSchedule (linear warmup + cosine decay) and global L2 grad-norm + clip scale, each with a local unit test. Corpus: data/tinystories-valid-3mb.txt — first ~3MB of TinyStories-valid (fetched on dash5 via hf-mirror.com; HF direct unreachable). Substitution noted: a real TinyStories subset, not the full set. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
12
crates/xtrain-train/src/lib.rs
Normal file
12
crates/xtrain-train/src/lib.rs
Normal file
@@ -0,0 +1,12 @@
|
||||
//! Training stack (Phase T6): LR schedule, global-norm grad clipping, checkpoint
|
||||
//! save/load, the GPT-2 BPE data pipeline (reusing xserv's tokenizer), an
|
||||
//! autoregressive sampler, and the training loop that wires them onto the T5
|
||||
//! `TinyTransformer` + the hand-written AdamW (`xtrain-optim`).
|
||||
//!
|
||||
//! Host-only pieces (LR schedule, grad-norm math) always compile so the crate
|
||||
//! `cargo check`s on a GPU-less host; everything that touches GPU tensors is
|
||||
//! gated behind `not(no_cuda)`.
|
||||
|
||||
pub mod clip;
|
||||
pub mod data;
|
||||
pub mod schedule;
|
||||
Reference in New Issue
Block a user