# Harness vs naive (use_harness on/off) — convergence ablation — 2026-06-16/17

Controlled ablation of the paper's "harness" (domain-knowledge knob-family steering):
the same agentic loop with `llm.use_harness=true` vs `false` (= the paper's naive
agentic tuner: free-form LLM proposals, no `Harnesses:` prompt section, no
deterministic guided proposals, no Stop-B validator/veto). Same workload, model, SLO,
substrate — the only difference is `use_harness` (configs
`dash0_qwen27b_ablation_harness_on.json` / `..._naive_off.json`, verified to differ
only in that flag + study_id).

- Model/host: dense Qwen3.5-27B, vLLM 0.11.1, 8×H20 (run on dash0; cpfs shared with dash1).
- Workload: chat 0–8k, length-aware TTFT SLO (4s + L_in/8k) + TPOT ≤ 50 ms, pass ≥ 95%.
- Substrate (process comparison, not absolute peak-rate): `replay_time_scale=0.5`,
  `completion_tokens_override=128`, Stop-A on, `search.high=0.25`, 6 probes, max-trials 6,
  **`--skip-baseline`** (the low-capacity TP1 auto-baseline is infeasible under this
  SLO+compression and would trip `baseline_all_infeasible`; skipping it lets both loops
  climb from their first proposal).
- This measures the tuning *process* (which knob family, convergence), not validated
  peak-rate.

## Result

### Harness ON — converged to the right answer in 2 iterations
| iter | proposer | config | per_gpu | outcome |
| --- | --- | --- | --- | --- |
| 1 | LLM (harness-guided) | TP2 | 0.247 | feasible |
| 2 | harness (deterministic) | **TP4** | **0.340** | feasible — incumbent |
| 3 | harness | TP4 + chunked-prefill + mbt=16384 | 0.333 | worse → rejected |
| (—) | LLM | `should_stop` | — | **VETOED** by validator ("decode TPOT still the bottleneck; adjacent probes weak") |
| 4 | LLM | TP2 + DP2 | 0.194 | worse → rejected |
| (—) | LLM | `should_stop` | STOP | honored (`llm_after_veto_budget`) |

Incumbent **TP4 @ 0.340 req/s/GPU**; iters-to-best = 2; no regression (the two worse
refinements were correctly not adopted); the premature LLM stop was vetoed once, then
honored after the budget.

### Naive OFF — wandered in the wrong knob family, never converged
| iter | config (TP never changed from 1) | outcome |
| --- | --- | --- |
| 1 | mbt=16384, seqs=128 | infeasible (`tpot>50`, `ttft>budget`) |
| 2 | mbt=32768, seqs=256, prefix-cache off, chunked | infeasible (same) |
| 3 | mbt=49152, seqs=384 | infeasible (same) |
| 4 | mbt=65536, seqs=512 | **FAILED** — engine crash (OOM at huge mbt) |
| 5 | mbt=57344, seqs=448 | interrupted by a dash0 outage |

Incumbent **None** — no feasible config found in 5 trials. The naive LLM kept tuning
**runtime** knobs (batched-tokens / num-seqs / caching) and **never raised TP**.

## Interpretation (the headline)

The bottleneck here is **compute** (decode TPOT + prefill queueing). The harness
diagnosed it and steered straight to the knob family that adds compute — **tensor
parallelism** — reaching a feasible **TP4 @ 0.34 req/s/GPU in 2 iterations**, then
correctly rejecting weaker refinements and stopping. The naive tuner spent its whole
budget on **runtime knobs that cannot add compute**, never tried raising TP, found
**zero** feasible configs, and even crashed the engine. This is a clean, stark
quantification of the harness's value: **right-knob-family steering → fast convergence
+ no regression, vs aimless runtime wandering → non-convergence.** It reproduces the
paper's Figure-18 finding (harness converges in a few iters; the naive agentic tuner
wastes the budget).

## Caveats / honesty

- Compressed substrate (scale=0.5, out=128) → the per-GPU numbers are *process*
  comparators, not validated peak-rates; the **direction/convergence** is the result.
- The naive run was interrupted at trial-5 by a dash0 connectivity outage (not by the
  experiment). The conclusion is already unambiguous (5 trials, never raised TP, all
  infeasible / one crash). A confirmatory naive-to-completion was attempted on dash1
  (shared cpfs, GPUs free) but **dash1 cannot reach the LLM gateway** (`Connection
  refused` to the codex endpoint — dash0 has outbound access, dash1 does not), so the
  LLM-driven loop only runs on dash0. The full-budget naive run can be completed when
  dash0 is back; it will not change the conclusion.
- LLM nondeterminism: a single run per arm. The harness arm's deterministic guided
  proposals (TP4 at iter 2) and validator veto are reproducible; the naive arm's exact
  path varies but its failure mode (wrong knob family, no TP) is the expected one.
- dash0/dash1 share the cpfs mount, so the run artifacts under `.aituner/abl-*` are
  readable/continuable from either host.