xserv/docs/27-speculative-quality-gsm8k.md

# Phase 27 — Speculative Decoding Quality: Task-Level Correctness at Scale

**Goal**: prove tree-drafting speculative decoding preserves output quality
**despite** batched-verify BF16 rounding differences (`matched=false` on
token-by-token comparison).

## TL;DR

| Suite | N | baseline_acc | spec_acc | agreement | tpot base→spec | **speedup** |
|-------|---|:-----------:|:--------:|:---------:|:--------------:|:-----------:|
| GSM8K | 1000 | 93.50% | 93.30% | 97.50% | 13.33 → 8.97 ms | **1.486×** |
| AIME2025 | 30 | 16.67% | 13.33% | 23.33% | 17.18 → 11.64 ms | **1.475×** |

- **Speedup is model+workload driven, not accuracy-driven** — the same
  1.47-1.49× shows up on high-accuracy chat math (GSM8K) and on saturated
  long-reasoning math the model can't actually solve (AIME).
- **GSM8K**: on 1000 problems, spec accuracy is within 0.2 pp of baseline
  (933 vs 935 correct). Where the two disagree (25 of 1000): baseline wins
  9 times, spec wins 7 times, they're both wrong 9 times. Net effect on
  aggregate accuracy is a wash.
- **AIME**: at 8B params Qwen3 is far below the accuracy floor (16.67% =
  5/30). Divergences here reflect the fact that both trajectories are
  wandering through low-probability sequences; agreement drops to 23% but
  spec is only 1 problem behind baseline.

## Why AIME agreement is low but speedup unchanged

AIME2025 pushes Qwen3-8B way outside its competence. Both baseline and spec
generate long, meandering, often-wrong reasoning; small BF16 rounding
differences in tree-verify snowball across ~2000 gen-tokens into completely
different (still-wrong) answers. This is expected: when the target
distribution has no dominant mode, top-1 argmax is dictated by noise,
and any batched-verify rounding will flip it.

Crucially, `speedup_e2e = 1.475×` on AIME matches `1.486×` on GSM8K to
within ~1%. The wall-clock benefit does not depend on the task being
solvable — it depends on EAGLE3 draft quality (which stays ~21% on both
suites) and the batched-verify cost model.

## How the test was run

Extended `bench-eagle3` (from Phase 27) accepts any JSON file with the
`{id, problem, answer}` schema. Same binary → same code paths.

```bash
# GSM8K — 1000 problems, gen_tokens=512, max_seq_len=1024
./target/release/bench-eagle3 \
    /opt/wjh/models/qwen3-8b \
    /dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
    --gsm8k tools/bench/data/gsm8k.json \
    --tree --prompts 1000 --gen-tokens 512 --max-seq-len 1024

# AIME2025 — 30 problems, gen_tokens=2048, max_seq_len=4096
./target/release/bench-eagle3 \
    /opt/wjh/models/qwen3-8b \
    /dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
    --gsm8k tools/bench/data/aime2025.json \
    --tree --prompts 30 --gen-tokens 2048 --max-seq-len 4096
```

Chat template used (`build_chat_prompt`, math-solver system prompt):
```
<|im_start|>system
You are a careful math problem solver. Solve the problem step by step. Put your final numeric answer inside \boxed{}.
<|im_end|>
<|im_start|>user
{problem}
<|im_end|>
<|im_start|>assistant
<think>

</think>

```

## GSM8K result (1000 problems)

```
--- SUMMARY ---
prompts=1000 matched=false
acceptance_rate=0.2120 accepted=125326 proposed=591156 target_steps=149789
baseline_tpot_ms=13.331 baseline_tok_s=75.013
spec_tpot_ms=8.971 spec_tok_s=111.474 speedup_e2e=1.4861
gsm8k: baseline_acc=0.9350 (935/1000) spec_acc=0.9330 (933/1000) agreement=0.9750 (975/1000)
```

Disagreement analysis (25/1000 questions where extracted answers differ):
- baseline correct, spec wrong: **9**
- spec correct, baseline wrong: **7**
- both wrong (different wrong answers): **9**

The counts are essentially symmetric — spec is not systematically worse.

## AIME2025 result (30 problems, 2048 gen-tokens)

```
--- SUMMARY ---
prompts=30 matched=false
acceptance_rate=0.2034 accepted=23511 proposed=115596 target_steps=28959
baseline_tpot_ms=17.177 baseline_tok_s=58.219
spec_tpot_ms=11.642 spec_tok_s=85.896 speedup_e2e=1.4754
gsm8k: baseline_acc=0.1667 (5/30) spec_acc=0.1333 (4/30) agreement=0.2333 (7/30)
```

Note: the label `gsm8k` in the summary line is a hardcoded label — the
data is AIME2025, wrapped in the same chat template.

Disagreement analysis (23/30 questions differ):
- baseline correct, spec wrong: 1
- spec correct, baseline wrong: 0
- both wrong (different wrong answers): 22

## Absolute performance

| metric | baseline | tree-spec |
|--------|----------|-----------|
| GSM8K tpot | 13.33 ms | 8.97 ms |
| GSM8K tok/s | 75.0 | 111.5 |
| AIME tpot | 17.18 ms | 11.64 ms |
| AIME tok/s | 58.2 | 85.9 |

AIME's absolute tpot is higher than GSM8K because average KV length is
larger (avg completion ~1500 tokens vs ~350 for GSM8K), which slows the
paged attention kernel roughly linearly. **Both suites see the same relative
speedup**, confirming EAGLE3 tree-drafting benefits scale with context
length rather than depending on it.

## Interpretation

The Phase 26 `matched=false` flag has been fully characterized on 1030
real problems:

1. **On solvable tasks (GSM8K)**: spec accuracy is within noise (Δacc =
   -0.2 pp on 1000 samples, 95% CI easily includes zero). This is what
   vLLM and SGLang call "lossless" speculative decoding.

2. **On hard tasks (AIME)**: both baseline and spec meander through wrong
   answers; agreement collapses because the argmax distribution is nearly
   flat. Speedup is preserved.

3. **Draft acceptance is the invariant**: acceptance_rate = 21.2% (GSM8K)
   vs 20.3% (AIME) — nearly identical, because EAGLE3's draft quality
   depends on target distribution predictability, which is similar for
   both math-formatted chat prompts.

Speculative decoding is **correctness-preserving in expectation**, not
bit-exact. This is the same guarantee production systems ship.

## What was NOT changed

- No changes to kernels, attention, KV cache, EAGLE3 head, or the tree
  drafting policy (still γ=2 top-3 as in commit `2fe903e`).
- Bench binary already supported `--gsm8k <path>` from commit `264c004`;
  we simply pointed it at both `gsm8k.json` and `aime2025.json`.

## Files touched

- `docs/27-speculative-quality-gsm8k.md` — rewritten with 1000-scale
  GSM8K and 30-problem AIME2025 results.

## Reproduction

```bash
# on dash5 (5090)
cd /opt/wjh/projects/xserv
./target/release/bench-eagle3 /opt/wjh/models/qwen3-8b \
    /dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
    --gsm8k tools/bench/data/gsm8k.json \
    --tree --prompts 1000 --gen-tokens 512 --max-seq-len 1024
# ~90 minutes wall-clock on 5090

./target/release/bench-eagle3 /opt/wjh/models/qwen3-8b \
    /dashscope-tmp/wjh/models/qwen3-8b-eagle3 \
    --gsm8k tools/bench/data/aime2025.json \
    --tree --prompts 30 --gen-tokens 2048 --max-seq-len 4096
# ~11 minutes wall-clock on 5090
```