post-train: M1 — verifiable arithmetic task + SFT data generator
First post-training milestone (docs/18). Lands the verifiable task + its data
pipeline, all verified host-side (no CUDA); the SFT run itself reuses the
existing --sft-tsv path on the GPU box.
- task.rs: the shared task spec — two-operand integer arithmetic, answer in
\boxed{N}, with parse_boxed_answer + check_answer (exact-match rule-based
reward). One module reused by M1 (SFT data), M3 (DPO pairs), M4 (GRPO reward).
- gen_arith_task bin: writes arith_sft.tsv (--sft-tsv format) + held-out
arith_eval_prompts.txt (greedy_sample format) + arith_eval_gold.txt; train
deduped, eval disjoint from train.
- data.rs: extract assistant-only masking into a pure, testable sft_row()
(behavior-preserving; single-turn bit-identical to fbf4ac2).
Gate (verified locally, no_cuda): cargo test -p xtrain-train --lib = 9/9 pass
(masking, SFT-target self-consistency over 2000 samples, parser edges, seed
determinism); a 200/50 gen run = clean 2-col TSV, correct gold incl. negatives,
0 train/eval leakage. SFT training run + format-eval pending on dash5.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -282,3 +282,47 @@ gates as flash/GQA.
|
||||
|
||||
> Each milestone is one design+gate cycle; results get appended here (like the run docs) and a
|
||||
> row in `docs/evolution.md` (algorithm/infra dimensions) when it lands.
|
||||
|
||||
## Implementation log
|
||||
|
||||
### M1 — SFT task baseline (infra landed; SFT run pending on the GPU box)
|
||||
|
||||
The verifiable task and its data pipeline are implemented and verified host-side (no CUDA
|
||||
needed). The SFT training run itself is a one-liner over the existing `--sft-tsv` path and
|
||||
runs on dash5.
|
||||
|
||||
**Verifiable task (the spec, in one Rust module — `crates/xtrain-train/src/task.rs`):**
|
||||
|
||||
- Two-operand integer arithmetic, ops `+ − ×`; operands `[0,99]` for `+/−`, `[0,12]` for `×`
|
||||
(modest products); subtraction may be negative.
|
||||
- User turn: `What is A op B?`. SFT target: `A op B = \boxed{N}.` — teaches the answer FORMAT;
|
||||
the checker reads only `\boxed{}`, so arithmetic *correctness* is what M3/M4 improve.
|
||||
- Rule-based reward: `parse_boxed_answer` (takes the LAST `\boxed{int}`) + `check_answer`
|
||||
(exact match vs. gold). This is the single shared checker reused by M3 (pair construction)
|
||||
and M4 (GRPO reward).
|
||||
- Why this task: trivial deterministic checker, freely scalable difficulty, and it directly
|
||||
probes the base model's known arithmetic weakness (v12 SFT failed `12 * 13`).
|
||||
|
||||
**Data generator (`crates/xtrain-train/src/bin/gen_arith_task.rs`, pure host bin):**
|
||||
writes `arith_sft.tsv` (`user<TAB>assistant` for `--sft-tsv`), `arith_eval_prompts.txt`
|
||||
(`greedy_sample --prompts-file` format), and `arith_eval_gold.txt` (parallel gold ints).
|
||||
Train rows are deduped; eval is held out from train (no leakage).
|
||||
|
||||
**Masking made testable:** the assistant-only label masking in `load_sft_tsv_cached` was
|
||||
extracted into a pure `sft_row(prompt_ids, answer_ids)` helper (behavior-preserving — the
|
||||
single-turn path is bit-identical to `fbf4ac2`).
|
||||
|
||||
**Gate (verified locally in `no_cuda` mode):** `cargo test -p xtrain-train --lib` → 9/9 pass,
|
||||
including `sft_row` masks prompt→`-100` / supervises answer, the SFT-target self-consistency
|
||||
invariant (always checker-correct over 2000 samples), parser edge cases, and seed determinism.
|
||||
A 200/50 generation run confirmed clean 2-column TSV, correct gold (incl. negatives), and 0
|
||||
train/eval leakage.
|
||||
|
||||
**Pending on dash5 (needs GPU):**
|
||||
1. generate the dataset: `gen_arith_task --n <N> --eval <M> --seed 1 --out-dir <dir>`;
|
||||
2. SFT from the v12 base: `train <tok> <dir>/arith_sft.tsv --sft-tsv --init-ckpt <v12-base.ckpt>
|
||||
<arch flags> --bf16 --recompute --flash --ckpt <out.ckpt>` → the P0 reference/init checkpoint;
|
||||
3. format eval: `greedy_sample <out.ckpt> <tok> <arch> --prompts-file <dir>/arith_eval_prompts.txt`,
|
||||
then score completions with `check_answer` vs. `arith_eval_gold.txt` (a tiny scorer is a
|
||||
one-off; can fold into M3's pipeline). Baseline pass-rate here is expected to be low — M1 only
|
||||
buys the format; correctness is M3/M4's job.
|
||||
|
||||
Reference in New Issue
Block a user