diff --git a/docs/harness-ablation/stop-a-validation-20260615.md b/docs/harness-ablation/stop-a-validation-20260615.md new file mode 100644 index 0000000..4a9db5e --- /dev/null +++ b/docs/harness-ablation/stop-a-validation-20260615.md @@ -0,0 +1,96 @@ +# Stop-A validation (Phase 3) — 2026-06-15 + +Branch `feat/two-stop`. Stop-A = truncate each probe's replay once the offered +L-C-A of the replayed prefix converges to the full set (pure L-C-A criterion + +C-gate). This note records the CPU calibration and the GPU fidelity check. + +## 1. Calibration (CPU, no serving) + +`scripts/stop_a_calibration.py` on the dash0 0321 10:00–10:10 windows: + +| dim | chat (19239 req, hit≈7%) | coder (2451 req, structured reuse) | +| --- | --- | --- | +| A | ≥0.95 by frac 0.10 | fast | +| L | ≥0.96 from frac 0.05 | 0.05=0.75 (heavy tail) → ≥0.94 by 0.20 | +| **C (slowest)** | noisy, dips (0.50→0.885, 0.55→0.835), stable ≥0.92 only ~0.85 | smooth, stable ≥0.92 by ~0.70 | + +Stop fraction (τ_L=τ_A=0.90, W=3): + +| τ_c | chat | coder | +| --- | --- | --- | +| 0.85 | 0.45 (273s) | 0.45 (255s) | +| 0.90 | 0.70 (423s) | 0.55 (318s) | +| 0.92 | 0.85 (513s) | 0.70 (411s) | + +Findings: +- **C is the slowest dimension in both workloads** — reproduces paper §5.2 / Fig 9. +- **What makes C hard to call converged is signal *noise*, not reuse magnitude.** + Low-reuse chat has a sparse/spiky ideal-hit-length series, so its C similarity + oscillates and is *harder* to stabilize than the structured, higher-reuse coder. + Consequence: a strict τ_c (0.92) gives chat only ~15% saving. A more robust C + feature for the low-reuse regime is future work. + +## 2. GPU fidelity check (Qwen3-30B-A3B, vLLM 0.11.1, H20) + +One full-window run (`adaptive_stop` disabled, `replay_time_scale=1.0`, window +`chat_w20260311_1000`, 0–8k, out=128), then `scripts/stop_a_validate.py` +recomputes each probe's convergence prefix and compares the truncated verdict to +the full verdict — so a single GPU run validates truncation fidelity (no second run). + +Trial result: best feasible `sampling_u=0.078125`, request_rate **2.30 req/s**, +pass_rate 0.973. + +Per-probe verdict (τ=0.9): + +| τ_c | verdict matches | mean replay saved | +| --- | --- | --- | +| 0.85 | 3/4 | 54% | +| 0.90 | 3/4 | 52% | +| 0.92 | 3/4 | 38% | + +The mismatch is the same probe at every τ_c — the feasibility knee `0.08594`: + +``` +thresh full_pass prefix_pass full_feas prefix_feas +0.08594 0.946 0.956–0.961 False True <- mismatch +0.07812 0.973 0.987–0.990 True True +0.06250 0.986 1.000 True True +0.09375 0.268 0.49–0.54 False False +``` + +## 3. Interpretation + +- **Stop-A works and saves ~50% of replay** (vs the full 600 s window) while + preserving 3/4 probe verdicts. (The paper's ~70% is vs a 30-min fixed baseline; + our baseline is the 600 s window, so the percentages are not directly comparable.) +- **The one failure is a boundary false-positive at the feasibility knee.** At + `0.08594` the full window is 0.946 (just below the 0.95 SLO) but the prefix is + 0.956–0.961 (just above): the *second half* of the window degraded — engine-state + drift (KV fill / fragmentation / later-arriving harder requests) that the + *offered* L-C-A cannot see. The C-gate did not help because offered-C had + converged; the divergence is in the measured pass-rate, not in C. +- If Stop-A were enabled, the binary search would accept `0.08594`, overestimating + the peak sustainable rate by one binary step (~10%). + +**This is the boundary jitter we accepted when choosing the pure-L-C-A criterion.** +The data now argues for revisiting the previously-declined **SLO-boundary guard**: +keep replaying while the measured pass-rate is within ±δ of the target, even after +L-C-A converges. It targets exactly this knee case at low extra cost (it only +extends replay on probes sitting on the feasibility boundary). Recommend adding it +as a small Stop-A enhancement before enabling Stop-A in production studies. + +## Repro + +``` +# calibration +PYTHONPATH=src python3 scripts/stop_a_calibration.py \ + --jsonl