Gahow Wang
d975e57bb5
At replay_time_scale=0.2 the 600s arrival window compresses to 120s, so the inherited 900s wall-clock elapsed cap let overloaded TP1 probes burn ~15min each (the tractability hazard the brief flagged). Scale the caps proportionately to the time axis: early_stop_max_elapsed_s 900->180, early_stop_max_lag_s 120->30. Feasible probes (~120s arrival + drain) finish well inside 180s; overloaded probes die in ~3min. Both configs still differ only in use_harness + study_id. Adds the ablation doc skeleton and a read-only trajectory-extraction helper. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.0 KiB
Harness vs naive agentic tuner — controlled ablation on dense Qwen3.5-27B — 2026-06-16
Branch main. Quantifies the value of the paper's harness (domain-knowledge
knob-family guidance) by running the agentic tuning loop twice on the same
workload, identical in every respect except llm.use_harness:
- Harness ON (
dash0_qwen27b_ablation_harness_on.json, studydash0-qwen27b-ablation-harness-on): the prompt carries theHarnesses:section (ranked bottleneck hypotheses + per-knob-family use-when / procedure / guards, with anactive_nowflag), the loop can emit a deterministic harness-guided first probe, and a Stop-B validator gates the LLM'sshould_stop(an unauthorized stop is vetoed). - Naive OFF (
dash0_qwen27b_ablation_naive_off.json, studydash0-qwen27b-ablation-naive-off):use_harness=false. No harness prompt section, no deterministic guided/stop proposals, and the LLM's ownshould_stopis honored without a validator veto. The prompt still tells the LLM that TP/DP/EP are tunable and gives the full study/SLO/trial-history context — so the difference is purely the harness guidance, this is the paper's "naive agentic tuner."
The two config files differ in exactly two keys (llm.use_harness and
study_id); verified by diff.
Substrate (why these knobs, and the comparability caveat)
This ablation measures the tuning process (proposal path + convergence), not
absolute peak-rate, so a faster replay substrate is used to keep it tractable
(at replay_time_scale=1.0 a single TP4 trial took ~3 h — see
stop-b-e2e-27b-20260616.md).
| knob | value | rationale |
|---|---|---|
trace.replay_time_scale |
0.2 | arrival times are multiplied by 0.2, i.e. the same request set arrives in 1/5 the wall-clock → ~5× higher effective offered load. arrival_s = timestamp * time_scale (trace.py:223). Mild arrival-time compression: the lever the brief prescribes (compress time, do not just cut the elapsed cap). |
search.high |
0.25 | upper bound of the sampling_u binary search |
search.max_probes |
5 | probe budget per trial |
--max-trials |
8 | iteration budget |
| Stop-A | enabled (unchanged) | converged-prefix replay truncation stays on for both runs |
| SLO | length-aware TTFT (4s + L_in/8k) + TPOT ≤ 50 ms | unchanged from base |
| GPUs | CUDA_VISIBLE_DEVICES=2,3,4,5,6,7 |
GPUs 0/1 avoided |
Comparability caveat. Because arrival times are compressed 5×, the absolute
request_rate_per_gpu values are not comparable to the scale=1.0 ground-truth
climb (TP1 0.123 → TP2 0.29 → TP4 1.00). The ablation reads the trajectory
shape (which knob family each iteration tries, whether the incumbent climbs
monotonically, where each run stops) and the relative per-GPU ordering across
topologies — not the absolute numbers.