85 lines
5.4 KiB
Markdown
85 lines
5.4 KiB
Markdown
# AITuner Harness Summary
|
|
|
|
## What The Harness Adds
|
|
|
|
The harness turns each LLM proposal from open-ended config search into a bottleneck-directed decision.
|
|
|
|
1. Workload profile
|
|
- Extracts L-C-A features from the trace window:
|
|
- L: prompt length percentiles and tail ratio.
|
|
- C: prefix/cache reuse estimates from `hash_ids` when available.
|
|
- A: request rate, burst ratio, and interarrival variation.
|
|
- These features are injected into the prompt as a structured `Harnesses` section.
|
|
|
|
2. Trial diagnostics
|
|
- Reads recent trial result JSON.
|
|
- Summarizes feasible probes, all-infeasible probes, pass rates, request rates, latency percentiles, and failed SLO reason counts.
|
|
- Classifies the active bottleneck as `ttft_prefill`, `decode_tpot`, `admission_or_queueing`, `launch_or_memory`, or unknown.
|
|
|
|
3. Knob-family harnesses
|
|
- Maps bottlenecks to a small number of plausible knob families.
|
|
- Current harness families:
|
|
- `tensor-parallel-size`: long-prompt TTFT/prefill bottlenecks.
|
|
- `max-num-batched-tokens`: prefill batching or fragmentation, with trust-region guards.
|
|
- `max-num-seqs`: cache-heavy or admission-limited workloads.
|
|
- `enable-chunked-prefill`: long-tail prompt blocking.
|
|
- `gpu-memory-utilization`: memory headroom after topology and batching are stable.
|
|
- Each family has `use_when`, `procedure`, `guards`, and `active_now` fields.
|
|
|
|
4. Proposal discipline and early stop
|
|
- The prompt requires the LLM to choose at most one primary knob family unless history proves a coupled change is needed.
|
|
- It must use adjacent legal topology choices and stay inside topology constraints.
|
|
- It receives tested config signatures, so it should not repeat already-tried configs.
|
|
- A deterministic harness stop can now emit `should_stop=true` before calling the LLM when completed validation evidence says another trial is not justified.
|
|
|
|
5. Baseline-first loop
|
|
- LLM-driven `study tune` now evaluates the initial engine config first unless `--skip-baseline` is passed.
|
|
- This aligns the loop with evaluate-then-search: the first LLM proposal sees measured bottleneck evidence rather than guessing from static config.
|
|
|
|
## What Accelerates Convergence
|
|
|
|
The speedup comes from reducing wasted proposal families, not from changing the benchmark metric.
|
|
|
|
1. Topology-before-runtime on prefill bottlenecks
|
|
- For long-prompt, low-cache-reuse windows, the harness activates the TP harness before speculative runtime knobs.
|
|
- Example: qwen27b 0-8k chat reached `TP=2, DP=1` at iter 2 under harness replay, while the original run spent iter 2 on `DP=2` and iter 3 on `DP=4`.
|
|
|
|
2. Guarded stop after validation, not immediately after a strong incumbent
|
|
- If the newest trial is the incumbent and improves per-GPU throughput by at least `1.8x` over baseline, the harness requires direct evidence before trying runtime-only tweaks.
|
|
- It does not stop at the first large gain. It requires post-incumbent validation trials across nearby topology/runtime families, and stops only if those trials fail to produce a feasible per-GPU improvement.
|
|
- With the guard, `study tune` can write a `harness-stop-XXXX` proposal and exit without spending another GPU trial.
|
|
|
|
3. All-infeasible plateau detection
|
|
- When recent all-infeasible trials at the same sampling threshold stop improving pass rate and p95 TTFT, the harness blocks repeating the same primary knob family.
|
|
- This prevents continuing a direction such as DP-only scale-out after DP4 and DP8 plateau.
|
|
- Plateau alone does not trigger deterministic early stop; it forces either a different justified family or a later validation/convergence stop.
|
|
|
|
4. Cleaner early-stop handling
|
|
- Early-stopped probes no longer leave in-flight requests polluting the next probe.
|
|
- Default behavior drains in-flight requests for comparable production runs.
|
|
- Engine relaunch after early stop is available as opt-in for faster smoke studies, but it is not the default because it can change warm-state comparability.
|
|
|
|
## qwen27b 0-8k Evidence
|
|
|
|
Source: `docs/qwen27b-chat-0-8k-harness-fig18.md`.
|
|
|
|
Metric: best-so-far feasible `request_rate_per_gpu`.
|
|
|
|
| Variant | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5-12 |
|
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
|
| Before harness | 0.0350 | 0.0617 | 0.0617 | 0.2025 | 0.2025 |
|
|
| After harness strict replay | 0.0350 | 0.2025 | 0.2025 stop | 0.2025 | 0.2025 |
|
|
|
|
Result:
|
|
|
|
- Before harness reached the best value at iter 4.
|
|
- After harness reached the same value at iter 2 and stopped at iter 3.
|
|
- Iterations-to-best improved from `4` to `2`, a `2x` convergence speedup on this case.
|
|
- The harness also avoided eight post-best infeasible runtime-only probes.
|
|
|
|
## Current Risks
|
|
|
|
- The harness is still prompt-guided for choosing the next non-stop proposal. The deterministic stop path is hard-coded in `study tune`, but proposal-family blocking is not yet enforced by a separate validator.
|
|
- Strong-incumbent stopping is intentionally biased toward fewer GPU trials after validation evidence accumulates. Workloads with very narrow runtime sweet spots may still need a "continue local refinement" exception when the user wants absolute best throughput rather than fastest convergence to a good config.
|
|
- Full fresh reruns on large models are expensive. Strict replay is useful for measuring proposal-path improvements when the proposed configs already exist in prior measured runs, but publication-quality claims still need fresh no-relaunch runs when time allows.
|