gahow/xtrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

run: v3 archive + export (dim512, single-GPU batched, val 1.30)

Browse Source 
v3 trained (30000 steps × batch 32 × seq 256 = 245.8M tok, ~0.53 epoch),
single-GPU batched via T10 (~26K tok/s, ~2.65h). Archived to registry
~/projects/tiny-models/v3-tinystories-dim512/ (xtrain.ckpt + config.json +
model.safetensors BF16 179 tensors + tokenizer.json + RUN.md) and served in
xserv (loads 16L/dim512 qwen3, 2/3 prompts token-match xtrain greedy; 3rd
diverges on BF16 drift as in v1/v2).

best/final val 1.3027 (beats ~1.4 target). val ladder on the same held-out
1M-token set: v0 3.80 / v1 2.58 / v2 1.71 / v3 1.30. T10 (batched forward)
validated at scale (KI-1 root cause = launch-bound, not all-reduce); single-GPU
avoids KI-5. Update docs/runs/README.md comparison table.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-06-16 03:37:36 +08:00
parent 9a25616a30
commit 64b2a8c09e
1 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
docs/runs/README.md
View File

@@ -20,9 +20,11 @@ val loss 一栏给的是各版**各自训练 run 报告的 best val**（held-out
| [v0-baseline](../../docs/05-training-loop.md) | TinyStories valid 3MB 切片 (~72 万 tok) | 32 / 4 / 2·16 / 64 | ~41K | 3.26M | **3.8050** | 太小不可用；采样陷入 "mommy's mommy's mommy" 循环 |
| [v1-tinystories-dim256](01-v1-tinystories-dim256.md) | TinyStories **全量 train** (468.3M tok, u16 缓存) | 256 / 8 / 8·32 / 1024 | 8.39M | 34.13M | **2.5847** | 全量数据 + dim256/8L；val 低 1.22，采样连贯成篇；~25.9min/单卡 |
| [v2-tinystories-dim384](02-v2-tinystories-dim384.md) | TinyStories 全量 (复用 v1 缓存, 训 ~36.9M tok) | 384 / 12 / 12·32 / 1536 | 28.32M | 66.92M | **1.7055** | dim384/12L + **DDP 4 卡**；val 比 v1 低 0.88，情节更长；~2.8h/4 卡。⚠️ DDP 弱扩展见 [KI-1](../known-issues.md) |
| [v3-tinystories-dim512](03-v3-tinystories-dim512.md) | TinyStories 全量 (复用 v1 缓存, 训 ~245.8M tok, ~0.53 epoch) | 512 / 16 / 16·32 / 2048 | 67.13M | 118.59M | **1.3027** | dim512/16L + **单卡 batched (T10)**；val 比 v2 低 0.40，带动机/转折的连续叙事；~2.65h/单卡 ~26K tok/s。T10 修 KI-1 根因(launch-bound)，单卡避开 KI-5 |
## 下一档（提案）
- **v3**（待派发）：见 `02-v2-*.md` 末尾 "v3 提案"——先修 KI-1（加大 global batch 恢复 DDP 扩展），
再放大 dim512/16L (~75M core) + 更多步数，TinyStories 接近上限后上更广语料。
- **v4**（待派发）：见 `03-v3-*.md` 末尾 "v4 提案"——放大 dim640–768/20–24L (~130–200M core) +
~600M–1B token，目标 val ~1.0–1.1；多卡需先修 KI-5（分桶 all-reduce），模型变大后启用 KI-2/3
(bf16/重计算)，并按数据阶梯开始广化语料（TinyStories + 通用高质语料）。
</content>