xtrain

gahow/xtrain

Fork 0

Commit Graph

Author	SHA1	Message	Date
Gahow Wang	64b2a8c09e	run: v3 archive + export (dim512, single-GPU batched, val 1.30) v3 trained (30000 steps × batch 32 × seq 256 = 245.8M tok, ~0.53 epoch), single-GPU batched via T10 (~26K tok/s, ~2.65h). Archived to registry ~/projects/tiny-models/v3-tinystories-dim512/ (xtrain.ckpt + config.json + model.safetensors BF16 179 tensors + tokenizer.json + RUN.md) and served in xserv (loads 16L/dim512 qwen3, 2/3 prompts token-match xtrain greedy; 3rd diverges on BF16 drift as in v1/v2). best/final val 1.3027 (beats ~1.4 target). val ladder on the same held-out 1M-token set: v0 3.80 / v1 2.58 / v2 1.71 / v3 1.30. T10 (batched forward) validated at scale (KI-1 root cause = launch-bound, not all-reduce); single-GPU avoids KI-5. Update docs/runs/README.md comparison table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-16 03:37:36 +08:00
Gahow Wang	bf679f6f1f	docs: run v2 — TinyStories, dim384/12L, DDP 4-card (val 1.71) Scaling run v2 design doc + comparison-table update. v2 = dim384/12L/12h SwiGLU ffn1536 (core 28.32M, total 66.92M), trained 4500 steps / ~36.9M tokens on full TinyStories (reused v1 u16 cache) via NCCL DDP across 4 RTX 5090s. Best val 1.7055 (train 10.89→1.72), a clear jump over v1 2.58 and v0 3.80. Exported to xserv (135 BF16 tensors) and archived in the dash5 registry; xserv greedy token-matches xtrain on 2/3 fixed prompts (3rd diverges late under BF16 drift). Records the DDP weak-scaling caveat (global batch too small → all-reduce dominates) → links docs/known-issues KI-1; v3 proposal applies KI-1's fix (much larger global batch). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 22:38:31 +08:00
Gahow Wang	264660527f	docs: run v1 — TinyStories full, dim256 docs/runs/01-v1-tinystories-dim256.md + docs/runs/README.md comparison table. v1: full TinyStories train (468.3M tok, u16-cached) + dim256/8L (core 8.39M). Same-held-out-set val loss v0 3.8050 → v1 2.5847 (−1.22); v1 samples coherent stories vs v0's "mommy's mommy's mommy" loop; exports + serves token-identical in xserv. Single RTX 5090, ~25.9 min, ~3310 tok/s. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-15 19:09:46 +08:00

Author

SHA1

Message

Date

Gahow Wang

64b2a8c09e

run: v3 archive + export (dim512, single-GPU batched, val 1.30)

v3 trained (30000 steps × batch 32 × seq 256 = 245.8M tok, ~0.53 epoch),
single-GPU batched via T10 (~26K tok/s, ~2.65h). Archived to registry
~/projects/tiny-models/v3-tinystories-dim512/ (xtrain.ckpt + config.json +
model.safetensors BF16 179 tensors + tokenizer.json + RUN.md) and served in
xserv (loads 16L/dim512 qwen3, 2/3 prompts token-match xtrain greedy; 3rd
diverges on BF16 drift as in v1/v2).

best/final val 1.3027 (beats ~1.4 target). val ladder on the same held-out
1M-token set: v0 3.80 / v1 2.58 / v2 1.71 / v3 1.30. T10 (batched forward)
validated at scale (KI-1 root cause = launch-bound, not all-reduce); single-GPU
avoids KI-5. Update docs/runs/README.md comparison table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-16 03:37:36 +08:00

Gahow Wang

bf679f6f1f

docs: run v2 — TinyStories, dim384/12L, DDP 4-card (val 1.71)

Scaling run v2 design doc + comparison-table update. v2 = dim384/12L/12h
SwiGLU ffn1536 (core 28.32M, total 66.92M), trained 4500 steps / ~36.9M
tokens on full TinyStories (reused v1 u16 cache) via NCCL DDP across 4
RTX 5090s. Best val 1.7055 (train 10.89→1.72), a clear jump over v1 2.58
and v0 3.80. Exported to xserv (135 BF16 tensors) and archived in the
dash5 registry; xserv greedy token-matches xtrain on 2/3 fixed prompts
(3rd diverges late under BF16 drift). Records the DDP weak-scaling caveat
(global batch too small → all-reduce dominates) → links docs/known-issues
KI-1; v3 proposal applies KI-1's fix (much larger global batch).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-15 22:38:31 +08:00

Gahow Wang

264660527f

docs: run v1 — TinyStories full, dim256

docs/runs/01-v1-tinystories-dim256.md + docs/runs/README.md comparison table.
v1: full TinyStories train (468.3M tok, u16-cached) + dim256/8L (core 8.39M).
Same-held-out-set val loss v0 3.8050 → v1 2.5847 (−1.22); v1 samples coherent
stories vs v0's "mommy's mommy's mommy" loop; exports + serves token-identical
in xserv. Single RTX 5090, ~25.9 min, ~3310 tok/s.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-15 19:09:46 +08:00

3 Commits