aituner/docs/two-stop-summary.md

Two-stop work — summary (feat/two-stop)

Aligns the tuning harness with paper.pdf by implementing and validating the two distinct stopping mechanisms the paper conflates, plus a length-aware SLO and two harness-robustness fixes. 117 unit tests pass.

The two stops (they are different things)

• Stop-A — evaluation sufficiency (per replay / probe): how much trace to replay before a measurement is trustworthy. Criterion: the replayed prefix's offered L-C-A converges to the full set's. Saves per-evaluation GPU time.
• Stop-B — tuning convergence (across iterations): whether any config-improvement opportunity remains; stop if not. Saves iterations.

What was built

Area	Summary	Commit
Unify L-C-A	The prompt's workload_lca_profile is now the canonical 10-dim lca.py vector, not an ad-hoc re-derivation	6f8e3c9
Session-coherent load axis	sampling_u assigned per session (parent_chat_id chain) so thresholding keeps/drops whole multi-turn sessions, preserving intra-session KV reuse (C) across load levels	0f15bbc
Stop-A	lca.find_convergence_prefix (deterministic offered-L-C-A convergence prefix + C-gate: never declare infeasible on a cold cache); spec.AdaptiveStopSpec (default off); worker._adaptive_replay_set truncates replay + writes a certificate	51a9e4a
Stop-A boundary guard	Re-measure the full window when a truncated probe's pass-rate is within boundary_delta of the SLO target (fixes feasibility-knee false positives)	dfc823f
Stop-B authority	harness._stop_authority (mirrors the deterministic validator); study tune honors an LLM should_stop only if the validator agrees, else a bounded veto	a8f9034
Length-aware SLO	linear_ms rule: threshold = intercept_ms + per_token_ms·L_in (e.g. 4s + L_in/8k)	ed2bbe0
Robustness: timeout	HTTP stream now wraps OSError/TimeoutError as HttpClientError — a request exceeding request_timeout_s is a failed request, not a crashed trial	2fcaf80
Robustness: SIGTERM	run_trial installs a SIGTERM handler so killing study tune tears down the engine (and EngineCore workers) instead of orphaning it / leaking GPU memory	b17b213
Tooling	stop_a_calibration.py (CPU convergence curve), stop_a_validate.py (offline truncation-fidelity)	08e53fd, 3af1d84

A subagent code review found no blocking bugs (and independently validated session coherence against a real trace); three minor fixes applied in 43125f4.

Validation results (real GPU runs, dash0 H20)

• Stop-A (stop-a-validation-20260615.md): CPU calibration reproduces the paper's C-slowest ordering; GPU fidelity check on Qwen3-30B-A3B saves ~52% replay at τ_c=0.90 with 3/4 probe verdicts preserved; the one mismatch is a feasibility-knee false positive that the boundary guard fixes — with the guard, best threshold matches full replay exactly while still saving ~38% replay.
• 27B TP sweep (qwen27b-tp-sweep-20260616.md): under the length-aware SLO, dense Qwen3.5-27B per-GPU rises sharply with TP — TP1 0.065 → TP2 0.29 → TP4 ≥0.91 — the opposite of the MoE 30B-A3B (TP1 best per-GPU). TP1 is TPOT-bound.
• Stop-B (stop-b-e2e-20260615.md, stop-b-e2e-27b-20260616.md): the 30B run shows the deterministic validator stop + a premature-LLM-stop veto; the 27B run (real gpt-5.4 loop) shows a genuine improving climb TP1 0.123 → TP2 0.29 → TP4 1.00 req/s/GPU (8.1×), each a correctly-diagnosed gain, then correctly rejecting a TP4 runtime tweak that measured worse (no regression). The SIGTERM fix was validated in practice (clean teardown, no leak).

Open items (next round)

• Harness convergence ablation (NOT yet done on this branch). The paper's harness result — domain-knowledge knob-family rules steering the LLM and cutting iterations — has only qualitative evidence here (the 27B climb shows correct steering) plus older smoke-regime ablations (qwen27b-chat-0-8k-harness-fig18.md: iters-to-best 4→2). A controlled use_harness=true vs false (naive tuner) comparison on the 27B is the missing quantified result.
• Loop efficiency: at scale=1.0 infeasible high-θ probes burn the early_stop_max_elapsed_s=900 cap (a TP4 trial took ~3h). Lower it to ~300s for practical agentic loops.
• dash0 GPUs 0/1 still hold leaked memory (pre-fix orphans) — needs a root nvidia-smi --gpu-reset.
• Deferred: more robust C feature for the low-reuse regime; Stop-C cross-day retune trigger (paper Q1); §7 baselines (SCOOT / naive / community).