aituner/docs/aituner-harness-summary.md

AITuner Harness Summary

What The Harness Adds

The harness turns each LLM proposal from open-ended config search into a bottleneck-directed decision.

1. Workload profile

   • Extracts L-C-A features from the trace window:
     • L: prompt length percentiles and tail ratio.
     • C: prefix/cache reuse estimates from hash_ids when available.
     • A: request rate, burst ratio, and interarrival variation.
   • These features are injected into the prompt as a structured Harnesses section.

2. Trial diagnostics

   • Reads recent trial result JSON.
   • Summarizes feasible probes, all-infeasible probes, pass rates, request rates, latency percentiles, and failed SLO reason counts.
   • Classifies the active bottleneck as ttft_prefill, decode_tpot, admission_or_queueing, launch_or_memory, or unknown.

3. Knob-family harnesses

   • Maps bottlenecks to a small number of plausible knob families.
   • Current harness families:
     • tensor-parallel-size: long-prompt TTFT/prefill bottlenecks.
     • max-num-batched-tokens: prefill batching or fragmentation, with trust-region guards.
     • max-num-seqs: cache-heavy or admission-limited workloads.
     • enable-chunked-prefill: long-tail prompt blocking.
     • gpu-memory-utilization: memory headroom after topology and batching are stable.
   • Each family has use_when, procedure, guards, and active_now fields.

4. Proposal discipline and early stop

   • The prompt requires the LLM to choose at most one primary knob family unless history proves a coupled change is needed.
   • It must use adjacent legal topology choices and stay inside topology constraints.
   • It receives tested config signatures, so it should not repeat already-tried configs.
   • A deterministic harness stop can now emit should_stop=true before calling the LLM when completed validation evidence says another trial is not justified.

5. Baseline-first loop

   • LLM-driven study tune now evaluates the initial engine config first unless --skip-baseline is passed.
   • This aligns the loop with evaluate-then-search: the first LLM proposal sees measured bottleneck evidence rather than guessing from static config.

What Accelerates Convergence

The speedup comes from reducing wasted proposal families, not from changing the benchmark metric.

1. Topology-before-runtime on prefill bottlenecks

   • For long-prompt, low-cache-reuse windows, the harness activates the TP harness before speculative runtime knobs.
   • Example: qwen27b 0-8k chat reached TP=2, DP=1 at iter 2 under harness replay, while the original run spent iter 2 on DP=2 and iter 3 on DP=4.

2. Guarded stop after validation, not immediately after a strong incumbent

   • If the newest trial is the incumbent and improves per-GPU throughput by at least 1.8x over baseline, the harness requires direct evidence before trying runtime-only tweaks.
   • It does not stop at the first large gain. It requires post-incumbent validation trials across nearby topology/runtime families, and stops only if those trials fail to produce a feasible per-GPU improvement.
   • With the guard, study tune can write a harness-stop-XXXX proposal and exit without spending another GPU trial.

3. All-infeasible plateau detection

   • When recent all-infeasible trials at the same sampling threshold stop improving pass rate and p95 TTFT, the harness blocks repeating the same primary knob family.
   • This prevents continuing a direction such as DP-only scale-out after DP4 and DP8 plateau.
   • Plateau alone does not trigger deterministic early stop; it forces either a different justified family or a later validation/convergence stop.

4. Cleaner early-stop handling

   • Early-stopped probes no longer leave in-flight requests polluting the next probe.
   • Default behavior drains in-flight requests for comparable production runs.
   • Engine relaunch after early stop is available as opt-in for faster smoke studies, but it is not the default because it can change warm-state comparability.

5. Search-high saturation stop

   • If the incumbent's highest measured probe is feasible and is within the configured binary-search resolution of search.high, the harness stops before asking the LLM for another proposal. Individual request failures can be present when the aggregate probe still meets the configured pass-rate SLO.
   • This is not a model-specific threshold. It means the workload search range, not the engine config, is currently the limiting measurement bound.

6. Deterministic first probes

   • After a baseline latency bottleneck, the harness can propose the adjacent legal TP increase before asking the LLM.
   • After a TP incumbent improves per-GPU throughput, the harness keeps that topology and applies a same-topology runtime seed before trying DP/EP or broad runtime changes.

qwen27b 0-8k Evidence

Source: docs/qwen27b-chat-0-8k-harness-fig18.md.

Metric: best-so-far feasible request_rate_per_gpu.

Variant	Iter 1	Iter 2	Iter 3	Iter 4	Iter 5-12
Before harness	0.0350	0.0617	0.0617	0.2025	0.2025
After harness strict replay	0.0350	0.2025	0.2025 stop	0.2025	0.2025

Result:

• Before harness reached the best value at iter 4.
• After harness reached the same value at iter 2 and stopped at iter 3.
• Iterations-to-best improved from 4 to 2, a 2x convergence speedup on this case.
• The harness also avoided eight post-best infeasible runtime-only probes.

Current Risks

• The harness is still prompt-guided for choosing the next non-stop proposal. The deterministic stop path is hard-coded in study tune, but proposal-family blocking is not yet enforced by a separate validator.
• Strong-incumbent stopping is intentionally biased toward fewer GPU trials after validation evidence accumulates. Workloads with very narrow runtime sweet spots may still need a "continue local refinement" exception when the user wants absolute best throughput rather than fastest convergence to a good config.
• Full fresh reruns on large models are expensive. Strict replay is useful for measuring proposal-path improvements when the proposed configs already exist in prior measured runs, but publication-quality claims still need fresh no-relaunch runs when time allows.

Qwen3-30B-A3B Community vLLM Evidence

Source: docs/qwen30b-community-vllm020/harness-early-stop-ablation-20260502.md.

Metric: best-so-far feasible request_rate_per_gpu on the bounded 0-8k chat replay with 128 output tokens and replay_time_scale=0.1.

Initial search.high=0.125 result:

Variant	Iter 1	Iter 2	Iter 3-12
no-harness	1.0333	1.0333	1.0333
harness	1.0333	1.0333 stop	1.0333

Result:

• Both variants found the same best measured config: the default community vLLM launch.
• Harness stopped at iter 2 because the incumbent saturated search.high; no LLM proposal or GPU trial was needed after baseline.
• No-harness spent the full 12-iteration budget: iter 2 was worse per GPU, and iter 3-12 were launch failures.
• This was a measurement-ceiling result, not proof of global optimality.

High search.high=1.0 rerun:

Variant	Iter 1	Iter 2	Iter 3	Iter 4	Iter 5	Iter 6	Iter 7	Iter 8-12
no-harness	2.2000	3.2583	3.2583	3.2583	3.2583	3.3000	3.3500	3.3500
harness-guided-v2	2.3833	3.2583	3.2833	3.3000	3.3000 stop	3.3000	3.3000	3.3000

Result:

• Raising search.high showed that default vLLM was not actually optimal; the prior run was capped by workload range.
• Harness reached the same TP2/runtime config family in 4 iterations instead of 7 by making deterministic first TP and same-topology runtime proposals.
• The single-run best value differs by about 1.5% (3.3000 vs 3.3500) for the same config family, so this should be interpreted as faster convergence to the same region, not an exact single-run throughput win.