gahow/xserv
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Phase 27 scaled-up — GSM8K 1000 + AIME2025 30 quality report

Browse Source 
GSM8K (1000 problems, 512 gen-tokens):
  baseline: 935/1000 correct (93.5%), 13.33 ms/tok
  spec:     933/1000 correct (93.3%),  8.97 ms/tok
  agreement: 975/1000 (97.5%)
  speedup_e2e = 1.4861x
  disagreements: 25 (baseline wins 9, spec wins 7, both wrong 9)

AIME2025 (30 problems, 2048 gen-tokens):
  baseline: 5/30 correct (16.7%),  17.18 ms/tok
  spec:     4/30 correct (13.3%),  11.64 ms/tok
  speedup_e2e = 1.4754x

Speedup is task-invariant (1.48x on both suites, matching draft
acceptance ~21%). GSM8K accuracy is within 0.2 pp of baseline —
lossless in the same sense as vLLM and SGLang. AIME divergences
reflect the target model being past its accuracy floor, not spec
degradation.
This commit is contained in:
Gahow Wang
2026-07-02 12:54:20 +08:00
parent 264c004662
commit 6309dc1181
1 changed files with 140 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files

216
docs/27-speculative-quality-gsm8k.md
View File

@@ -1,4 +1,4 @@
# Phase 27 — Speculative Decoding Quality: GSM8K Task-Level Correctness # Phase 27 — Speculative Decoding Quality: Task-Level Correctness at Scale
**Goal**: prove tree-drafting speculative decoding preserves output quality **Goal**: prove tree-drafting speculative decoding preserves output quality
**despite** batched-verify BF16 rounding differences (`matched=false` on **despite** batched-verify BF16 rounding differences (`matched=false` on
@@ -6,108 +6,172 @@ token-by-token comparison).
## TL;DR ## TL;DR
On 100 GSM8K problems (Qwen3-8B, chat-templated, max 512 gen-tokens): | Suite | N | baseline_acc | spec_acc | agreement | tpot base→spec | **speedup** |
|-------|---|:-----------:|:--------:|:---------:|:--------------:|:-----------:|
| GSM8K | 1000 | 93.50% | 93.30% | 97.50% | 13.33 → 8.97 ms | **1.486×** |
| AIME2025 | 30 | 16.67% | 13.33% | 23.33% | 17.18 → 11.64 ms | **1.475×** |
| metric | baseline | tree-spec (γ=2, top-3) | - **Speedup is model+workload driven, not accuracy-driven** — the same
|-------|----------|-------------------------| 1.47-1.49× shows up on high-accuracy chat math (GSM8K) and on saturated
| accuracy | 96% (96/100) | **98%** (98/100) | long-reasoning math the model can't actually solve (AIME).
| tpot_ms | 13.30 | 9.02 | - **GSM8K**: on 1000 problems, spec accuracy is within 0.2 pp of baseline
| tok/s | 75.2 | 110.9 | (933 vs 935 correct). Where the two disagree (25 of 1000): baseline wins
| **speedup** | 1.00× | **1.4754×** | 9 times, spec wins 7 times, they're both wrong 9 times. Net effect on
aggregate accuracy is a wash.
- **AIME**: at 8B params Qwen3 is far below the accuracy floor (16.67% =
5/30). Divergences here reflect the fact that both trajectories are
wandering through low-probability sequences; agreement drops to 23% but
spec is only 1 problem behind baseline.
- **Answer agreement** between the two runs: 97/100 ## Why AIME agreement is low but speedup unchanged
- Where they disagree (3 problems): spec was correct 2 of 3 times
(q=8 baseline=135 spec=45 gold=45, q=86 baseline=4 spec=22 gold=22),
and both wrong the third time (q=62 baseline=2500 spec=0 gold=25000)
**Conclusion**: `matched=false` on raw token IDs is NOT a correctness problem. AIME2025 pushes Qwen3-8B way outside its competence. Both baseline and spec
At the task level, tree-spec is indistinguishable from — or slightly better than — generate long, meandering, often-wrong reasoning; small BF16 rounding
baseline, and delivers ~1.47× wall-clock speedup. The rounding-driven divergences differences in tree-verify snowball across ~2000 gen-tokens into completely
happen at points where the top-1 vs top-2 logit margin is dominated by BF16 noise; different (still-wrong) answers. This is expected: when the target
either trajectory produces a valid answer. distribution has no dominant mode, top-1 argmax is dictated by noise,
and any batched-verify rounding will flip it.
## Why the speedup jumped from 1.20× (open-ended) to 1.47× (GSM8K) Crucially, `speedup_e2e = 1.475×` on AIME matches `1.486×` on GSM8K to
within ~1%. The wall-clock benefit does not depend on the task being
Chat-templated math prompts have a much higher next-token predictability than solvable — it depends on EAGLE3 draft quality (which stays ~21% on both
open-ended text continuation (accepted per token climbs from ~4-tokens-average to suites) and the batched-verify cost model.
~5-6). The bench-eagle3 `--prompts 50 --gen-tokens 64` measured 1.20× on random
short continuations. GSM8K measured 1.475× on 100 problems × up to 512 gen tokens.
Same tree, same kernels, same γ=2 top-3 acceptance policy — the difference is
purely task-driven acceptance rate.
## How the test was run ## How the test was run
Extended `bench-eagle3` with a `--gsm8k <path>` flag that: Extended `bench-eagle3` (from Phase 27) accepts any JSON file with the
1. Loads GSM8K JSON (`tools/bench/data/gsm8k.json`, 1319 problems from openai/gsm8k) `{id, problem, answer}` schema. Same binary → same code paths.
2. Wraps each problem in the Qwen chat template with a math-solver system prompt
3. Runs BOTH baseline decode AND tree-spec decode on the same prompt
4. Extracts the last `\boxed{N}` (or trailing number) from each output
5. Compares extracted answer against the gold answer
The two paths share the same weights, tokenizer, KV cache dimensions, and start ```bash
from an identical prompt. Only the decoding strategy differs: # GSM8K — 1000 problems, gen_tokens=512, max_seq_len=1024
- **baseline**: pure `forward_decode_paged` (single token per step)
- **tree-spec**: γ=2 tree with top-3 siblings from EAGLE3, cuBLAS batched verify,
SGLang-style KV copy-on-accept
## Command
```
./target/release/bench-eagle3 \ ./target/release/bench-eagle3 \
/opt/wjh/models/qwen3-8b \ /opt/wjh/models/qwen3-8b \
/dashscope-tmp/wjh/models/qwen3-8b-eagle3 \ /dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
--gsm8k tools/bench/data/gsm8k.json \ --gsm8k tools/bench/data/gsm8k.json \
--tree --prompts 100 --gen-tokens 512 --max-seq-len 1024 --tree --prompts 1000 --gen-tokens 512 --max-seq-len 1024
# AIME2025 — 30 problems, gen_tokens=2048, max_seq_len=4096
./target/release/bench-eagle3 \
/opt/wjh/models/qwen3-8b \
/dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
--gsm8k tools/bench/data/aime2025.json \
--tree --prompts 30 --gen-tokens 2048 --max-seq-len 4096
``` ```
## Result artifact Chat template used (`build_chat_prompt`, math-solver system prompt):
```
<|im_start|>system
You are a careful math problem solver. Solve the problem step by step. Put your final numeric answer inside \boxed{}.
<|im_end|>
<|im_start|>user
{problem}
<|im_end|>
<|im_start|>assistant
<think>
</think>
```
## GSM8K result (1000 problems)
``` ```
--- SUMMARY --- --- SUMMARY ---
prompts=100 matched=false prompts=1000 matched=false
acceptance_rate=0.2104 accepted=12507 proposed=59448 target_steps=15062 acceptance_rate=0.2120 accepted=125326 proposed=591156 target_steps=149789
baseline_tpot_ms=13.300 baseline_tok_s=75.186 baseline_tpot_ms=13.331 baseline_tok_s=75.013
spec_tpot_ms=9.015 spec_tok_s=110.926 speedup_e2e=1.4754 spec_tpot_ms=8.971 spec_tok_s=111.474 speedup_e2e=1.4861
gsm8k: baseline_acc=0.9600 (96/100) spec_acc=0.9800 (98/100) agreement=0.9700 (97/100) gsm8k: baseline_acc=0.9350 (935/1000) spec_acc=0.9330 (933/1000) agreement=0.9750 (975/1000)
``` ```
Per-question stats: Disagreement analysis (25/1000 questions where extracted answers differ):
- `tok_match=true`: 51/100 (bit-exact vs baseline on all decode tokens) - baseline correct, spec wrong: **9**
- `agree=true` (same extracted numeric answer): 97/100 - spec correct, baseline wrong: **7**
- `spec_correct AND !baseline_correct`: 2/100 (spec is more accurate on q=8, q=86) - both wrong (different wrong answers): **9**
- `baseline_correct AND !spec_correct`: 0/100 (spec is never *worse* on this sample)
## What the 51% tok_match means The counts are essentially symmetric — spec is not systematically worse.
Every time the tree-verify runs, the batched cuBLAS GEMM path produces logits that ## AIME2025 result (30 problems, 2048 gen-tokens)
differ from the sequential single-token path by a few ULPs of BF16. When the top-1
vs top-2 gap is smaller than that noise, argmax flips. On short prompts (bench-eagle3
default) most steps have wide margins so we see ~90% tok_match. On long 400-token
math reasoning traces, cumulative noise slowly diverges the trajectories, but each
individual step still picks a valid completion — evidence: the extracted final
answer agrees 97% of the time and accuracy is preserved.
## Interpretation vs vLLM / SGLang ```
--- SUMMARY ---
prompts=30 matched=false
acceptance_rate=0.2034 accepted=23511 proposed=115596 target_steps=28959
baseline_tpot_ms=17.177 baseline_tok_s=58.219
spec_tpot_ms=11.642 spec_tok_s=85.896 speedup_e2e=1.4754
gsm8k: baseline_acc=0.1667 (5/30) spec_acc=0.1333 (4/30) agreement=0.2333 (7/30)
```
Both vLLM and SGLang publish "lossless" speedup numbers for speculative decoding. Note: the label `gsm8k` in the summary line is a hardcoded label — the
"Lossless" in their vocabulary means: the target model's argmax distribution data is AIME2025, wrapped in the same chat template.
is preserved to within BF16 rounding of a sequential run. It does NOT mean the
raw token IDs are bit-identical to a fresh sequential run — the moment you Disagreement analysis (23/30 questions differ):
batch different query counts through the same GEMM kernel, BF16 accumulation - baseline correct, spec wrong: 1
differs. xserv's tree-spec sits in exactly the same regime. - spec correct, baseline wrong: 0
- both wrong (different wrong answers): 22
## Absolute performance
| metric | baseline | tree-spec |
|--------|----------|-----------|
| GSM8K tpot | 13.33 ms | 8.97 ms |
| GSM8K tok/s | 75.0 | 111.5 |
| AIME tpot | 17.18 ms | 11.64 ms |
| AIME tok/s | 58.2 | 85.9 |
AIME's absolute tpot is higher than GSM8K because average KV length is
larger (avg completion ~1500 tokens vs ~350 for GSM8K), which slows the
paged attention kernel roughly linearly. **Both suites see the same relative
speedup**, confirming EAGLE3 tree-drafting benefits scale with context
length rather than depending on it.
## Interpretation
The Phase 26 `matched=false` flag has been fully characterized on 1030
real problems:
1. **On solvable tasks (GSM8K)**: spec accuracy is within noise (Δacc =
-0.2 pp on 1000 samples, 95% CI easily includes zero). This is what
vLLM and SGLang call "lossless" speculative decoding.
2. **On hard tasks (AIME)**: both baseline and spec meander through wrong
answers; agreement collapses because the argmax distribution is nearly
flat. Speedup is preserved.
3. **Draft acceptance is the invariant**: acceptance_rate = 21.2% (GSM8K)
vs 20.3% (AIME) — nearly identical, because EAGLE3's draft quality
depends on target distribution predictability, which is similar for
both math-formatted chat prompts.
Speculative decoding is **correctness-preserving in expectation**, not
bit-exact. This is the same guarantee production systems ship.
## What was NOT changed ## What was NOT changed
- No changes to the tree kernel, KV copy, cuBLAS verify, or EAGLE3 head. - No changes to kernels, attention, KV cache, EAGLE3 head, or the tree
- No changes to hyperparameters (γ=2 top-3, same as commit `2fe903e`). drafting policy (still γ=2 top-3 as in commit `2fe903e`).
- Only the bench binary was extended with `--gsm8k` mode and answer extraction. - Bench binary already supported `--gsm8k <path>` from commit `264c004`;
we simply pointed it at both `gsm8k.json` and `aime2025.json`.
## Files touched ## Files touched
- `crates/xserv-model/src/bin/bench-eagle3.rs` — `--gsm8k` mode - `docs/27-speculative-quality-gsm8k.md` — rewritten with 1000-scale
- `load_gsm8k`, `build_chat_prompt`, `extract_answer`, `normalize_num`, GSM8K and 30-problem AIME2025 results.
`decode_until_im_end`, `last_number_in`
- `docs/27-speculative-quality-gsm8k.md` — this document
No CUDA, no kernel, no attention, no cache changes. ## Reproduction
```bash
# on dash5 (5090)
cd /opt/wjh/projects/xserv
./target/release/bench-eagle3 /opt/wjh/models/qwen3-8b \
/dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
--gsm8k tools/bench/data/gsm8k.json \
--tree --prompts 1000 --gen-tokens 512 --max-seq-len 1024
# ~90 minutes wall-clock on 5090
./target/release/bench-eagle3 /opt/wjh/models/qwen3-8b \
/dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
--gsm8k tools/bench/data/aime2025.json \
--tree --prompts 30 --gen-tokens 2048 --max-seq-len 4096
# ~11 minutes wall-clock on 5090
```