aituner/docs/harness-tuning-progress.md

Harness-Guided AITuner Progress

Goal

Improve AITuner convergence for the dash0 internal vLLM + Qwen3.5-27B 0-8k chat study. The prior 12-iteration run can still propose worse configs after finding good ones. The new harness should make config proposals bottleneck-directed and stop spending GPU trials once no adjacent harness-guided probe is justified.

Paper Alignment

• Prompt structure now includes an explicit [Harnesses] section aligned with paper Figure 12.
• The harness uses the paper's L-C-A workload model:
  • L: prompt length percentiles and tail ratio.
  • C: prefix/KV-cache reuse estimated from repeated hash_ids blocks when available.
  • A: request rate, 1-second QPS burst ratio, and interarrival CV.
• Knob rules follow the paper's Figure 13 style:
  • map active bottleneck to a knob family;
  • probe adjacent legal choices;
  • enforce guard conditions to avoid harmful side effects;
  • prefer stopping over weak exploratory proposals after convergence.

Local Implementation Log

• Added src/aituner/harness.py.
  • Builds structured harness context for prompt injection.
  • Adds TP, max-num-seqs, max-num-batched-tokens, chunked-prefill, and memory-utilization harnesses when those knobs are tunable.
  • Extracts compact recent trial diagnostics from result JSON files.
  • Adds a convergence guard based on recent completed trial performance.
• Extended src/aituner/trace.py.
  • summarize_window now reports L-C-A features.
  • TraceRequest now carries optional metadata for hash_ids, turn, parent chat id, and trace type.
• Extended src/aituner/llm.py.
  • Prompt now includes tested config signatures and the structured harness section.
  • Prompt schema now asks for should_stop.
• Extended src/aituner/spec.py.
  • Proposal accepts optional should_stop.
• Extended src/aituner/cli.py.
  • study tune honors should_stop=true by recording the proposal and not launching another GPU trial.
• Extended tests/test_core_flow.py.
  • Prompt includes harness context.
  • Trace summary includes new L-C-A fields.
  • Proposal parsing accepts should_stop.
  • CLI does not launch a trial for a stop proposal.

Local Verification

• python3 -m compileall -q src tests: passed.
• PYTHONPATH=src python3 -m unittest tests.test_core_flow: passed, 59 tests.
• pytest -q and python3 -m pytest -q: not runnable locally because pytest is not installed.

Remote Experiment Log

Pending. Next steps:

1. Commit and push the harness implementation.
2. Pull on dash0 in /home/admin/cpfs/wjh/aituner/aituner.
3. Start a real harness-guided Qwen3.5-27B 0-8k chat tuning run from configs/examples/dash0_qwen27b_tight_slo_run4_0_8k.json.
4. Compare the first few iterations against the prior 12-iteration behavior:
   • best request rate per GPU should improve or reach the known good region in fewer trials;
   • proposals should follow the active bottleneck harness;
   • if the incumbent has converged, the LLM should emit should_stop=true instead of proposing a weak exploratory config.