Fig18 substrate: real output_length + criterion-A time_scale + Stop-A drain deadline

Replace the out=128 / scale=0.5 ablation substrate with a paper-faithful one:
- Use the trace's real output_length (drop completion_tokens_override=128). The
  0-8k chat window has p50=531 / p99=2436 / max=35168 output tokens, so decode
  (TPOT) becomes the dominant bottleneck instead of an artificial 128-token cap.
- replay_time_scale=0.8775, chosen by criterion-A: binary-search the smallest
  scale whose A-family L-C-A similarity to the real (scale=1.0) arrivals stays
  >= tau (0.90). The old scale=0.5 had sim_A=0.56, distorting the arrival axis
  far below the tau bar used everywhere else. New calibrator:
  scripts/calibrate_time_scale.py.
- Per-probe Stop-A-consistent drain deadline (worker._probe_drain_deadline): the
  wall-clock a *feasible* config needs to drain the LCA-admitted set
  (last_arrival + worst-case TTFT + p99_out * TPOT budget + margin). With real
  outputs decode dominates wall-clock, so the old fixed 320s cap would truncate
  the Stop-A offered window mid-decode. early_stop_max_elapsed_s (1000s) is now a
  hard ceiling; the per-probe deadline governs. The lag cap still cuts overload.

12-iter paired driver (both arms on dash1, removes the dash0/dash1 host confound):
scripts/run_ablation_pair_d1.sh. 115 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-17 17:24:00 +08:00
parent 816765071f
commit 0c23285f39
6 changed files with 197 additions and 10 deletions

View File

@@ -18,7 +18,7 @@ from .engine import build_launch_recipe
from .http_client import HttpClientError, stream_chat_completion, wait_for_server
from .lca import find_convergence_prefix, resolve_length_mode
from .search import ThresholdProbe, binary_search_max_feasible
from .slo import RequestOutcome, evaluate_request, summarize_evaluations
from .slo import RequestOutcome, _rule_threshold_ms, evaluate_request, summarize_evaluations
from .spec import ConfigPatch, SamplingSearchSpec, TrialSpec, load_study_spec, to_jsonable
from .trace import TraceRequest, load_trace_requests, select_requests_for_threshold
@@ -254,6 +254,34 @@ def _ignore_sigterm_if_main() -> None:
pass
def _probe_drain_deadline(
reqs: list[TraceRequest], slo: Any, *, ceiling: float | None
) -> float | None:
"""Stop-A-consistent per-probe drain deadline (wall-clock seconds).
The deadline is the time a *feasible* config needs to drain the admitted set:
the last admitted arrival plus the worst-case TTFT budget plus the p99 output
length times the TPOT budget. A config that cannot finish by this deadline is
genuinely SLO-infeasible, so the clock never pre-empts the LCA-matched offered
window (Stop-A) -- it only fails the unfit. ``ceiling`` is a hard safety cap.
"""
if not reqs or slo.tpot_rule is None:
return ceiling
last_arrival = max(float(r.arrival_s or 0.0) for r in reqs)
inputs = sorted(int(r.prompt_tokens_hint or 0) for r in reqs)
outputs = sorted(int(r.completion_tokens_hint or 0) for r in reqs)
def _p99(xs: list[int]) -> int:
return xs[min(len(xs) - 1, int(0.99 * len(xs)))] if xs else 0
p99_in, p99_out = _p99(inputs), _p99(outputs)
tpot_ms = _rule_threshold_ms(slo.tpot_rule, p99_in)
ttft_ms = _rule_threshold_ms(slo.ttft_rule, p99_in) if slo.ttft_rule is not None else 0.0
margin_s = 30.0
deadline = last_arrival + (ttft_ms + p99_out * tpot_ms) / 1000.0 + margin_s
return min(float(ceiling), deadline) if ceiling else deadline
def _adaptive_replay_set(
selected: list[TraceRequest],
*,
@@ -640,7 +668,9 @@ def run_trial(trial_spec_path: Path) -> dict[str, Any]:
max_concurrency=study.trace.max_concurrency,
target_pass_rate=study.slo.target_pass_rate,
max_lag_s=study.trace.early_stop_max_lag_s,
max_elapsed_s=study.trace.early_stop_max_elapsed_s,
max_elapsed_s=_probe_drain_deadline(
reqs, study.slo, ceiling=study.trace.early_stop_max_elapsed_s
),
evaluate_outcome=lambda outcome: evaluate_request(outcome, study.slo),
drain_inflight_on_early_stop=not restart_after_early_stop,
)