xserv

gahow/xserv

Fork 0

Commit Graph

Author	SHA1	Message	Date
Gahow Wang	6309dc1181	docs: Phase 27 scaled-up — GSM8K 1000 + AIME2025 30 quality report GSM8K (1000 problems, 512 gen-tokens): baseline: 935/1000 correct (93.5%), 13.33 ms/tok spec: 933/1000 correct (93.3%), 8.97 ms/tok agreement: 975/1000 (97.5%) speedup_e2e = 1.4861x disagreements: 25 (baseline wins 9, spec wins 7, both wrong 9) AIME2025 (30 problems, 2048 gen-tokens): baseline: 5/30 correct (16.7%), 17.18 ms/tok spec: 4/30 correct (13.3%), 11.64 ms/tok speedup_e2e = 1.4754x Speedup is task-invariant (1.48x on both suites, matching draft acceptance ~21%). GSM8K accuracy is within 0.2 pp of baseline — lossless in the same sense as vLLM and SGLang. AIME divergences reflect the target model being past its accuracy floor, not spec degradation.	2026-07-02 12:54:20 +08:00
Gahow Wang	264c004662	eagle3: GSM8K quality benchmark proves tree-spec is correctness-preserving Adds --gsm8k mode to bench-eagle3: chat-templated prompts, per-problem answer extraction, side-by-side baseline vs tree-spec accuracy comparison. 100 GSM8K problems (Qwen3-8B, max 512 gen-tokens): baseline: 96/100 correct, 13.30 ms/tok spec: 98/100 correct, 9.02 ms/tok agreement: 97/100 speedup_e2e = 1.4754x Where the two disagree (3 cases): spec was correct 2/3 times. spec is never strictly worse than baseline on this sample. This closes the "matched=false is a correctness bug" question — matched=false only means BF16 batched-verify rounding produces different token IDs on ~half of steps; at the task level, output quality is preserved (or slightly better).	2026-07-02 10:29:33 +08:00

Author

SHA1

Message

Date

Gahow Wang

6309dc1181

docs: Phase 27 scaled-up — GSM8K 1000 + AIME2025 30 quality report

GSM8K (1000 problems, 512 gen-tokens):
  baseline: 935/1000 correct (93.5%), 13.33 ms/tok
  spec:     933/1000 correct (93.3%),  8.97 ms/tok
  agreement: 975/1000 (97.5%)
  speedup_e2e = 1.4861x
  disagreements: 25 (baseline wins 9, spec wins 7, both wrong 9)

AIME2025 (30 problems, 2048 gen-tokens):
  baseline: 5/30 correct (16.7%),  17.18 ms/tok
  spec:     4/30 correct (13.3%),  11.64 ms/tok
  speedup_e2e = 1.4754x

Speedup is task-invariant (1.48x on both suites, matching draft
acceptance ~21%). GSM8K accuracy is within 0.2 pp of baseline —
lossless in the same sense as vLLM and SGLang. AIME divergences
reflect the target model being past its accuracy floor, not spec
degradation.

2026-07-02 12:54:20 +08:00

Gahow Wang

264c004662

eagle3: GSM8K quality benchmark proves tree-spec is correctness-preserving

Adds --gsm8k mode to bench-eagle3: chat-templated prompts, per-problem
answer extraction, side-by-side baseline vs tree-spec accuracy comparison.

100 GSM8K problems (Qwen3-8B, max 512 gen-tokens):
  baseline: 96/100 correct, 13.30 ms/tok
  spec:     98/100 correct,  9.02 ms/tok
  agreement: 97/100
  speedup_e2e = 1.4754x

Where the two disagree (3 cases): spec was correct 2/3 times. spec is
never strictly worse than baseline on this sample. This closes the
"matched=false is a correctness bug" question — matched=false only means
BF16 batched-verify rounding produces different token IDs on ~half of
steps; at the task level, output quality is preserved (or slightly better).

2026-07-02 10:29:33 +08:00

2 Commits