Document AITuner harness behavior
This commit is contained in:
83
docs/aituner-harness-summary.md
Normal file
83
docs/aituner-harness-summary.md
Normal file
@@ -0,0 +1,83 @@
|
|||||||
|
# AITuner Harness Summary
|
||||||
|
|
||||||
|
## What The Harness Adds
|
||||||
|
|
||||||
|
The harness turns each LLM proposal from open-ended config search into a bottleneck-directed decision.
|
||||||
|
|
||||||
|
1. Workload profile
|
||||||
|
- Extracts L-C-A features from the trace window:
|
||||||
|
- L: prompt length percentiles and tail ratio.
|
||||||
|
- C: prefix/cache reuse estimates from `hash_ids` when available.
|
||||||
|
- A: request rate, burst ratio, and interarrival variation.
|
||||||
|
- These features are injected into the prompt as a structured `Harnesses` section.
|
||||||
|
|
||||||
|
2. Trial diagnostics
|
||||||
|
- Reads recent trial result JSON.
|
||||||
|
- Summarizes feasible probes, all-infeasible probes, pass rates, request rates, latency percentiles, and failed SLO reason counts.
|
||||||
|
- Classifies the active bottleneck as `ttft_prefill`, `decode_tpot`, `admission_or_queueing`, `launch_or_memory`, or unknown.
|
||||||
|
|
||||||
|
3. Knob-family harnesses
|
||||||
|
- Maps bottlenecks to a small number of plausible knob families.
|
||||||
|
- Current harness families:
|
||||||
|
- `tensor-parallel-size`: long-prompt TTFT/prefill bottlenecks.
|
||||||
|
- `max-num-batched-tokens`: prefill batching or fragmentation, with trust-region guards.
|
||||||
|
- `max-num-seqs`: cache-heavy or admission-limited workloads.
|
||||||
|
- `enable-chunked-prefill`: long-tail prompt blocking.
|
||||||
|
- `gpu-memory-utilization`: memory headroom after topology and batching are stable.
|
||||||
|
- Each family has `use_when`, `procedure`, `guards`, and `active_now` fields.
|
||||||
|
|
||||||
|
4. Proposal discipline
|
||||||
|
- The prompt requires the LLM to choose at most one primary knob family unless history proves a coupled change is needed.
|
||||||
|
- It must use adjacent legal topology choices and stay inside topology constraints.
|
||||||
|
- It receives tested config signatures, so it should not repeat already-tried configs.
|
||||||
|
- It can return `should_stop=true` when no adjacent harness-guided probe is justified.
|
||||||
|
|
||||||
|
5. Baseline-first loop
|
||||||
|
- LLM-driven `study tune` now evaluates the initial engine config first unless `--skip-baseline` is passed.
|
||||||
|
- This aligns the loop with evaluate-then-search: the first LLM proposal sees measured bottleneck evidence rather than guessing from static config.
|
||||||
|
|
||||||
|
## What Accelerates Convergence
|
||||||
|
|
||||||
|
The speedup comes from reducing wasted proposal families, not from changing the benchmark metric.
|
||||||
|
|
||||||
|
1. Topology-before-runtime on prefill bottlenecks
|
||||||
|
- For long-prompt, low-cache-reuse windows, the harness activates the TP harness before speculative runtime knobs.
|
||||||
|
- Example: qwen27b 0-8k chat reached `TP=2, DP=1` at iter 2 under harness replay, while the original run spent iter 2 on `DP=2` and iter 3 on `DP=4`.
|
||||||
|
|
||||||
|
2. Guarded stop after a strong incumbent
|
||||||
|
- If the newest trial is the incumbent and improves per-GPU throughput by at least `3x` over baseline, the harness requires direct evidence before trying runtime-only tweaks.
|
||||||
|
- Without that guard, the LLM still proposed weak MBT trials after finding the qwen27b best config.
|
||||||
|
- With the guard, it emits `should_stop=true`.
|
||||||
|
|
||||||
|
3. All-infeasible plateau detection
|
||||||
|
- When recent all-infeasible trials at the same sampling threshold stop improving pass rate and p95 TTFT, the harness blocks repeating the same primary knob family.
|
||||||
|
- This prevents continuing a direction such as DP-only scale-out after DP4 and DP8 plateau.
|
||||||
|
|
||||||
|
4. Cleaner early-stop handling
|
||||||
|
- Early-stopped probes no longer leave in-flight requests polluting the next probe.
|
||||||
|
- Default behavior drains in-flight requests for comparable production runs.
|
||||||
|
- Engine relaunch after early stop is available as opt-in for faster smoke studies, but it is not the default because it can change warm-state comparability.
|
||||||
|
|
||||||
|
## qwen27b 0-8k Evidence
|
||||||
|
|
||||||
|
Source: `docs/qwen27b-chat-0-8k-harness-fig18.md`.
|
||||||
|
|
||||||
|
Metric: best-so-far feasible `request_rate_per_gpu`.
|
||||||
|
|
||||||
|
| Variant | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5-12 |
|
||||||
|
| --- | ---: | ---: | ---: | ---: | ---: |
|
||||||
|
| Before harness | 0.0350 | 0.0617 | 0.0617 | 0.2025 | 0.2025 |
|
||||||
|
| After harness strict replay | 0.0350 | 0.2025 | 0.2025 stop | 0.2025 | 0.2025 |
|
||||||
|
|
||||||
|
Result:
|
||||||
|
|
||||||
|
- Before harness reached the best value at iter 4.
|
||||||
|
- After harness reached the same value at iter 2 and stopped at iter 3.
|
||||||
|
- Iterations-to-best improved from `4` to `2`, a `2x` convergence speedup on this case.
|
||||||
|
- The harness also avoided eight post-best infeasible runtime-only probes.
|
||||||
|
|
||||||
|
## Current Risks
|
||||||
|
|
||||||
|
- The harness is prompt-guided, not a hard verifier for every rule. If future LLM outputs ignore a fired guard, proposal validation should reject the blocked family explicitly.
|
||||||
|
- Strong-incumbent stopping is deliberately conservative for the qwen27b pattern. Workloads with narrow runtime sweet spots, such as qwen235b thinking prefill-only, may need a weaker stop threshold or a "continue local refinement" exception.
|
||||||
|
- Full fresh reruns on large models are expensive. Strict replay is useful for measuring proposal-path improvements when the proposed configs already exist in prior measured runs, but publication-quality claims still need fresh no-relaunch runs when time allows.
|
||||||
Reference in New Issue
Block a user