Gahow Wang
e7d1b3ba01
Controlled use_harness on/off on dense 27B (same workload/SLO/substrate, only the flag differs). Harness ON: TP2 -> TP4 (0.34 req/s/GPU) in 2 iters, rejected two worse refinements, premature LLM stop vetoed then honored -> converged, no regression. Naive OFF: kept TP=1 and cranked runtime knobs (mbt 16k->65k, seqs, caching), all 5 trials infeasible (same TPOT/TTFT compute bottleneck), one engine OOM crash, no feasible config found. The bottleneck is compute; the harness steered to the knob family that adds compute (TP) while naive wandered in knobs that cannot. Reproduces the paper's Fig-18 finding. Substrate is compressed (process comparison, not peak-rate); naive run was infra-interrupted at trial-5 (already conclusive). Read from cpfs via dash1. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
75 lines
4.3 KiB
Markdown
75 lines
4.3 KiB
Markdown
# Harness vs naive (use_harness on/off) — convergence ablation — 2026-06-16/17
|
||
|
||
Controlled ablation of the paper's "harness" (domain-knowledge knob-family steering):
|
||
the same agentic loop with `llm.use_harness=true` vs `false` (= the paper's naive
|
||
agentic tuner: free-form LLM proposals, no `Harnesses:` prompt section, no
|
||
deterministic guided proposals, no Stop-B validator/veto). Same workload, model, SLO,
|
||
substrate — the only difference is `use_harness` (configs
|
||
`dash0_qwen27b_ablation_harness_on.json` / `..._naive_off.json`, verified to differ
|
||
only in that flag + study_id).
|
||
|
||
- Model/host: dense Qwen3.5-27B, vLLM 0.11.1, 8×H20 (run on dash0; cpfs shared with dash1).
|
||
- Workload: chat 0–8k, length-aware TTFT SLO (4s + L_in/8k) + TPOT ≤ 50 ms, pass ≥ 95%.
|
||
- Substrate (process comparison, not absolute peak-rate): `replay_time_scale=0.5`,
|
||
`completion_tokens_override=128`, Stop-A on, `search.high=0.25`, 6 probes, max-trials 6,
|
||
**`--skip-baseline`** (the low-capacity TP1 auto-baseline is infeasible under this
|
||
SLO+compression and would trip `baseline_all_infeasible`; skipping it lets both loops
|
||
climb from their first proposal).
|
||
- This measures the tuning *process* (which knob family, convergence), not validated
|
||
peak-rate.
|
||
|
||
## Result
|
||
|
||
### Harness ON — converged to the right answer in 2 iterations
|
||
| iter | proposer | config | per_gpu | outcome |
|
||
| --- | --- | --- | --- | --- |
|
||
| 1 | LLM (harness-guided) | TP2 | 0.247 | feasible |
|
||
| 2 | harness (deterministic) | **TP4** | **0.340** | feasible — incumbent |
|
||
| 3 | harness | TP4 + chunked-prefill + mbt=16384 | 0.333 | worse → rejected |
|
||
| (—) | LLM | `should_stop` | — | **VETOED** by validator ("decode TPOT still the bottleneck; adjacent probes weak") |
|
||
| 4 | LLM | TP2 + DP2 | 0.194 | worse → rejected |
|
||
| (—) | LLM | `should_stop` | STOP | honored (`llm_after_veto_budget`) |
|
||
|
||
Incumbent **TP4 @ 0.340 req/s/GPU**; iters-to-best = 2; no regression (the two worse
|
||
refinements were correctly not adopted); the premature LLM stop was vetoed once, then
|
||
honored after the budget.
|
||
|
||
### Naive OFF — wandered in the wrong knob family, never converged
|
||
| iter | config (TP never changed from 1) | outcome |
|
||
| --- | --- | --- |
|
||
| 1 | mbt=16384, seqs=128 | infeasible (`tpot>50`, `ttft>budget`) |
|
||
| 2 | mbt=32768, seqs=256, prefix-cache off, chunked | infeasible (same) |
|
||
| 3 | mbt=49152, seqs=384 | infeasible (same) |
|
||
| 4 | mbt=65536, seqs=512 | **FAILED** — engine crash (OOM at huge mbt) |
|
||
| 5 | mbt=57344, seqs=448 | interrupted by a dash0 outage |
|
||
|
||
Incumbent **None** — no feasible config found in 5 trials. The naive LLM kept tuning
|
||
**runtime** knobs (batched-tokens / num-seqs / caching) and **never raised TP**.
|
||
|
||
## Interpretation (the headline)
|
||
|
||
The bottleneck here is **compute** (decode TPOT + prefill queueing). The harness
|
||
diagnosed it and steered straight to the knob family that adds compute — **tensor
|
||
parallelism** — reaching a feasible **TP4 @ 0.34 req/s/GPU in 2 iterations**, then
|
||
correctly rejecting weaker refinements and stopping. The naive tuner spent its whole
|
||
budget on **runtime knobs that cannot add compute**, never tried raising TP, found
|
||
**zero** feasible configs, and even crashed the engine. This is a clean, stark
|
||
quantification of the harness's value: **right-knob-family steering → fast convergence
|
||
+ no regression, vs aimless runtime wandering → non-convergence.** It reproduces the
|
||
paper's Figure-18 finding (harness converges in a few iters; the naive agentic tuner
|
||
wastes the budget).
|
||
|
||
## Caveats / honesty
|
||
|
||
- Compressed substrate (scale=0.5, out=128) → the per-GPU numbers are *process*
|
||
comparators, not validated peak-rates; the **direction/convergence** is the result.
|
||
- The naive run was interrupted at trial-5 by a dash0 connectivity outage (not by the
|
||
experiment). The conclusion is already unambiguous (5 trials, never raised TP, all
|
||
infeasible / one crash); a confirmatory naive-to-completion run on dash1 can remove
|
||
the asterisk.
|
||
- LLM nondeterminism: a single run per arm. The harness arm's deterministic guided
|
||
proposals (TP4 at iter 2) and validator veto are reproducible; the naive arm's exact
|
||
path varies but its failure mode (wrong knob family, no TP) is the expected one.
|
||
- dash0/dash1 share the cpfs mount, so the run artifacts under `.aituner/abl-*` are
|
||
readable/continuable from either host.
|