aituner/docs/harness-ablation/harness-vs-naive-20260616.md

Harness vs naive (use_harness on/off) — convergence ablation — 2026-06-16/17

Controlled ablation of the paper's "harness" (domain-knowledge knob-family steering): the same agentic loop with llm.use_harness=true vs false (= the paper's naive agentic tuner: free-form LLM proposals, no Harnesses: prompt section, no deterministic guided proposals, no Stop-B validator/veto). Same workload, model, SLO, substrate — the only difference is use_harness (configs dash0_qwen27b_ablation_harness_on.json / ..._naive_off.json, verified to differ only in that flag + study_id).

• Model/host: dense Qwen3.5-27B, vLLM 0.11.1, 8×H20 (run on dash0; cpfs shared with dash1).
• Workload: chat 0–8k, length-aware TTFT SLO (4s + L_in/8k) + TPOT ≤ 50 ms, pass ≥ 95%.
• Substrate (process comparison, not absolute peak-rate): replay_time_scale=0.5, completion_tokens_override=128, Stop-A on, search.high=0.25, 6 probes, max-trials 6, --skip-baseline (the low-capacity TP1 auto-baseline is infeasible under this SLO+compression and would trip baseline_all_infeasible; skipping it lets both loops climb from their first proposal).
• This measures the tuning process (which knob family, convergence), not validated peak-rate.

Result

Harness ON — converged to the right answer in 2 iterations

iter	proposer	config	per_gpu	outcome
1	LLM (harness-guided)	TP2	0.247	feasible
2	harness (deterministic)	TP4	0.340	feasible — incumbent
3	harness	TP4 + chunked-prefill + mbt=16384	0.333	worse → rejected
(—)	LLM	should_stop	—	VETOED by validator ("decode TPOT still the bottleneck; adjacent probes weak")
4	LLM	TP2 + DP2	0.194	worse → rejected
(—)	LLM	should_stop	STOP	honored (llm_after_veto_budget)

Incumbent TP4 @ 0.340 req/s/GPU; iters-to-best = 2; no regression (the two worse refinements were correctly not adopted); the premature LLM stop was vetoed once, then honored after the budget.

Naive OFF — wandered in the wrong knob family, never converged

iter	config (TP never changed from 1)	outcome
1	mbt=16384, seqs=128	infeasible (tpot>50, ttft>budget)
2	mbt=32768, seqs=256, prefix-cache off, chunked	infeasible (same)
3	mbt=49152, seqs=384	infeasible (same)
4	mbt=65536, seqs=512	FAILED — engine crash (OOM at huge mbt)
5	mbt=57344, seqs=448	interrupted by a dash0 outage

Incumbent None — no feasible config found in 5 trials. The naive LLM kept tuning runtime knobs (batched-tokens / num-seqs / caching) and never raised TP.

Interpretation (the headline)

The bottleneck here is compute (decode TPOT + prefill queueing). The harness diagnosed it and steered straight to the knob family that adds compute — tensor parallelism — reaching a feasible TP4 @ 0.34 req/s/GPU in 2 iterations, then correctly rejecting weaker refinements and stopping. The naive tuner spent its whole budget on runtime knobs that cannot add compute, never tried raising TP, found zero feasible configs, and even crashed the engine. This is a clean, stark quantification of the harness's value: **right-knob-family steering → fast convergence

• no regression, vs aimless runtime wandering → non-convergence.** It reproduces the paper's Figure-18 finding (harness converges in a few iters; the naive agentic tuner wastes the budget).

Caveats / honesty

• Compressed substrate (scale=0.5, out=128) → the per-GPU numbers are process comparators, not validated peak-rates; the direction/convergence is the result.
• The naive run was interrupted at trial-5 by a dash0 connectivity outage (not by the experiment). The conclusion is already unambiguous (5 trials, never raised TP, all infeasible / one crash); a confirmatory naive-to-completion run on dash1 can remove the asterisk.
• LLM nondeterminism: a single run per arm. The harness arm's deterministic guided proposals (TP4 at iter 2) and validator veto are reproducible; the naive arm's exact path varies but its failure mode (wrong knob family, no TP) is the expected one.
• dash0/dash1 share the cpfs mount, so the run artifacts under .aituner/abl-* are readable/continuable from either host.