gahow/xtrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
63 Commits 1 Branch 0 Tags
28801fbfe5907c71d2b10b1577b4349952ac74b4
Commit Graph

4 Commits

Author SHA1 Message Date
Gahow Wang
a78502e0f0 docs: run v3 — TinyStories, dim512, val 1.30 
Per-run design doc docs/runs/03-v3-tinystories-dim512.md (data 245.8M tok full
TinyStories ~0.53 epoch / arch dim512 16L core 67.13M vs total 118.59M, what
changed vs v2 / hyperparams 30000 steps batch 32 seq 256 lr 6e-4→6e-5 warmup
1500 + cosine clip 1.0 single-GPU batched via T10 / results train 10.91→1.40
best val 1.3027 ~26K tok/s / improvement vs v2 1.71→1.30 with side-by-side
samples). Notes v3 validated T10 batched forward at scale and avoided KI-5 by
staying single-GPU; v4 proposal + open levers (KI-2/3/4/5, data ladder).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 03:37:45 +08:00
Gahow Wang
64b2a8c09e run: v3 archive + export (dim512, single-GPU batched, val 1.30) 
v3 trained (30000 steps × batch 32 × seq 256 = 245.8M tok, ~0.53 epoch),
single-GPU batched via T10 (~26K tok/s, ~2.65h). Archived to registry
~/projects/tiny-models/v3-tinystories-dim512/ (xtrain.ckpt + config.json +
model.safetensors BF16 179 tensors + tokenizer.json + RUN.md) and served in
xserv (loads 16L/dim512 qwen3, 2/3 prompts token-match xtrain greedy; 3rd
diverges on BF16 drift as in v1/v2).

best/final val 1.3027 (beats ~1.4 target). val ladder on the same held-out
1M-token set: v0 3.80 / v1 2.58 / v2 1.71 / v3 1.30. T10 (batched forward)
validated at scale (KI-1 root cause = launch-bound, not all-reduce); single-GPU
avoids KI-5. Update docs/runs/README.md comparison table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-16 03:37:36 +08:00
Gahow Wang
bf679f6f1f docs: run v2 — TinyStories, dim384/12L, DDP 4-card (val 1.71) 
Scaling run v2 design doc + comparison-table update. v2 = dim384/12L/12h
SwiGLU ffn1536 (core 28.32M, total 66.92M), trained 4500 steps / ~36.9M
tokens on full TinyStories (reused v1 u16 cache) via NCCL DDP across 4
RTX 5090s. Best val 1.7055 (train 10.89→1.72), a clear jump over v1 2.58
and v0 3.80. Exported to xserv (135 BF16 tensors) and archived in the
dash5 registry; xserv greedy token-matches xtrain on 2/3 fixed prompts
(3rd diverges late under BF16 drift). Records the DDP weak-scaling caveat
(global batch too small → all-reduce dominates) → links docs/known-issues
KI-1; v3 proposal applies KI-1's fix (much larger global batch).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-15 22:38:31 +08:00
Gahow Wang
264660527f docs: run v1 — TinyStories full, dim256 
docs/runs/01-v1-tinystories-dim256.md + docs/runs/README.md comparison table.
v1: full TinyStories train (468.3M tok, u16-cached) + dim256/8L (core 8.39M).
Same-held-out-set val loss v0 3.8050 → v1 2.5847 (−1.22); v1 samples coherent
stories vs v0's "mommy's mommy's mommy" loop; exports + serves token-identical
in xserv. Single RTX 5090, ~25.9 min, ~3310 tok/s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-15 19:09:46 +08:00