aituner/docs/harness-ablation/harness-vs-naive-20260616.md

# Harness vs naive (use_harness on/off) — convergence ablation — 2026-06-16/17

Controlled ablation of the paper's "harness" (domain-knowledge knob-family steering):
the same agentic loop with `llm.use_harness=true` vs `false` (= the paper's naive
agentic tuner: free-form LLM proposals, no `Harnesses:` prompt section, no
deterministic guided proposals, no Stop-B validator/veto). Same workload, model, SLO,
substrate — the only difference is `use_harness` (configs
`dash0_qwen27b_ablation_harness_on.json` / `..._naive_off.json`, verified to differ
only in that flag + study_id).

- Model: dense Qwen3.5-27B, vLLM 0.11.1, 8×H20 (dash0 and dash1 share the cpfs mount).
- Workload: chat 0–8k, length-aware TTFT SLO (4s + L_in/8k) + TPOT ≤ 50 ms, pass ≥ 95%.
- Substrate (process comparison, not absolute peak-rate): `replay_time_scale=0.5`,
  `completion_tokens_override=128`, Stop-A on, `search.high=0.25`, 6 probes, max-trials 6,
  **`--skip-baseline`** (the low-capacity TP1 auto-baseline is infeasible under this
  SLO+compression and would trip `baseline_all_infeasible`; skipping it lets both loops
  climb from their first proposal).
- This measures the tuning *process* (which knob family, convergence speed, stop
  discipline), not a validated peak-rate.

## Harness ON — converged in 2 iterations, then stopped
| iter | proposer | config | per_gpu | outcome |
| --- | --- | --- | --- | --- |
| 1 | LLM (harness-guided) | TP2 | 0.247 | feasible |
| 2 | harness (deterministic) | **TP4** | **0.340** | feasible — incumbent |
| 3 | harness | TP4 + chunked-prefill + mbt=16384 | 0.333 | worse → rejected |
| (—) | LLM | `should_stop` | — | **VETOED** ("decode TPOT still the bottleneck; adjacent probes weak") |
| 4 | LLM | TP2 + DP2 | 0.194 | worse → rejected |
| (—) | LLM | `should_stop` | **STOP** | honored after veto budget |

Best **TP4 @ 0.340**; iters-to-best = **2**; ran **4 trials then stopped** (Stop-B +
one veto of a premature stop); no regression.

## Naive OFF — nondeterministic; reaches the optimum slowly at best, fails at worst

The naive (free-form) `gpt-5.4` loop behaved very differently across two runs — it has
no harness steering and no stop logic:

**Run A (dash0, interrupted by an outage at trial-5):** kept **TP=1** the whole time and
cycled runtime knobs (`max-num-batched-tokens` 16k→65k, `max-num-seqs`, caching). All
trials **infeasible** (same `tpot>50` + `ttft>budget`), trial-4 **crashed the engine**
(OOM at mbt=65536). Found **no feasible config** in 5 trials — never tried raising TP.

**Run B (dash1, full budget):**
| iter | config | per_gpu | note |
| --- | --- | --- | --- |
| 1 | TP2 | 0.247 | feasible |
| 2 | TP2 + max-num-seqs=32 | 0.218 | worse |
| 3 | TP2 + mbt=12288 | 0.218 | worse |
| 4 | TP2 (re-proposal) | 0.218 | no gain |
| 5 | TP2 + gpu-mem-util=0.85 | 0.218 | worse |
| 6 | **TP4** | **0.340** | reaches the optimum — at the last trial |

Best **TP4 @ 0.340** — the *same* optimum as the harness — but iters-to-best = **6**,
it used the **entire budget with no early stop**, and trials 2–5 were detours (TP2 +
runtime tweaks, all worse than trial-1) before it stumbled onto TP4.

## Comparison

| | Harness ON | Naive OFF (B, dash1) | Naive OFF (A, dash0) |
| --- | --- | --- | --- |
| best per-GPU | 0.340 (TP4) | 0.340 (TP4) | none (failed) |
| iters-to-best | **2** | 6 | — |
| trials used | **4 (stopped)** | 6 (full budget, no stop) | 5 (interrupted) |
| stopped early? | yes (Stop-B + veto) | no | — |
| wasted trials | 2 (post-best refinements) | 4 (TP2+runtime detours) | 5 (runtime-only, infeasible) |
| path to optimum | direct (TP2→TP4) | slow (TP2→runtime detour→TP4) | wrong family (runtime on TP1) |

## Interpretation (honest)

The bottleneck is **compute** (decode TPOT + prefill queueing), which only a
compute-adding knob (**tensor parallelism**) fixes. Findings:

1. **A strong frontier model can sometimes find the right knob unaided** — naive run B
   eventually reached TP4 = 0.34, the same optimum as the harness. This matches the
   paper's own caveat (§7.3): stronger models reduce, but do not remove, the need for
   structured guidance. So the harness's value is **not** "naive always fails."
2. **The harness's value is reliability, speed, and stop discipline.** With the harness:
   converged in **2 iters** and **stopped at 4** (recognized convergence; vetoed a
   premature stop). Naive: **3× slower** to the same answer (6 iters), **never stopped**
   (burned the full budget on detours), and in run A **failed outright** — never tried
   TP, found nothing, crashed the engine. Naive is **nondeterministic and unreliable**;
   the harness is fast, monotone (no regression), and self-terminating.
3. This reproduces the paper's Figure-18 story: the harness converges in a few
   iterations and stops, while the naive agentic tuner wastes the budget (and can fail
   to converge entirely).

## Caveats

- Compressed substrate (scale=0.5, out=128) → per-GPU numbers are *process* comparators,
  not validated peak-rates; the convergence behavior is the result. (The TP4 optimum
  reproduced at 0.340 across the harness run and naive run B, a useful consistency check.)
- One run per arm per host; naive is nondeterministic (runs A and B differ markedly),
  which is itself part of the finding. The harness arm's deterministic guided proposal
  (TP4 at iter 2) and validator veto are reproducible.
- Infra notes: dash0 (LLM-gateway reachable) went down mid-experiment; dash1 shares the
  cpfs and ran the completion. The codex `config.toml` points at a dash0-local proxy
  (`127.0.0.1:11235`); on dash1 the LLM endpoint must be reached directly (set empty
  `*_proxy` env) — see `scripts/run_naive_d1.sh`.