Add t2d bool tensor loading and per-slot top-3 rate tracking to
bench-eagle3 so we can distinguish three failure modes:
- Not covered: target's argmax not in EAGLE's 32k-vocab (upper bound).
- Not top-3: target's argmax not in EAGLE's top-3 (drafting quality).
- Not top-1: target's argmax not EAGLE's argmax (final acceptance rule).
Measured on 50 prompts × 64 tokens γ=2:
d[0]: correct=27%, top-3=42%, covered=98% → EAGLE covers vocab well
but often ranks target
answer below top-1.
d[1]: correct=9%, top-3=17%, covered=100% → recursive draft even
weaker.
Coverage is essentially not a bottleneck (98%+). The bottleneck is
that EAGLE ranks the true target answer only ~27% of the time at slot
0. Top-3 rate (~42%) shows the correct answer is often in EAGLE's
distribution but not the highest-scored candidate.
To exploit the top-3 headroom would require tree-based verify (multiple
candidates per position, tree-aware attention masking). Each candidate
attends only to its own branch, not siblings. Current paged_decode_
attention writes K/V at unique per-batch positions and does not
support tree causal masks.
Speedup formula analysis (from bench-verify-cost):
γ=2: verify_cost=1.11×, round_yield=1.34 → theoretical speedup=1.21×,
observed 1.10× (0.11× lost to EAGLE draft cost + bookkeeping).
γ=4: verify_cost=1.12×, round_yield=1.36 → theoretical=1.21×,
observed 1.02×.
Current numbers are near-optimal given measured acceptance. Further
gains require either tree drafting (unlocks top-K acceptance) or a
better-trained EAGLE head. Neither is a small change.
Gahow Wang
10a98539d0