gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document harness v2 rerun criteria

Browse Source
This commit is contained in:
Gahow Wang
2026-05-12 22:23:12 +08:00
parent e3ed775afd
commit fb6d74a18c
1 changed files with 23 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
docs/harness-ablation/profile-driven-harness-implementation-20260512.md
View File

@@ -81,3 +81,26 @@ Started on `dash0` (`11.73.2.172`) at commit `17e9681`.
- log: `.aituner-tight/logs/qwen27b-profileplanner-harness-20260512.log` - log: `.aituner-tight/logs/qwen27b-profileplanner-harness-20260512.log`
- status at launch check: `trial-0001` baseline is running under AITuner; no manual intervention in the tuning loop. - status at launch check: `trial-0001` baseline is running under AITuner; no manual intervention in the tuning loop.
## V1 Early Stop
The first profile-planner run was stopped before accepting it as evidence. A read-only replay of its completed baseline probe history showed that the planner would choose `max-num-seqs=64` for iter2. That was a diagnosis bug:
- `slo_pass_rate_unrecoverable` is a binary-search early-stop summary, not a bottleneck class.
- The harness was counting that summary as an admission/queueing failure.
- Because this count dominated the real TTFT/TPOT failure counts, the planner selected a concurrency action instead of testing TP.
Fix commit: `e3ed775`.
The fix excludes `slo_pass_rate_unrecoverable` from the admission/queueing failure bucket. With the same baseline probe history, the planner now ranks `ttft_prefill` first and proposes `tensor-parallel-size=2` for iter2.
## V2 Experiment Started
Started on `dash0` (`11.73.2.172`) at commit `e3ed775`.
- tmux session: `qwen27b-profileplanner-v2-harness-20260512`
- spec: `.aituner-tight/specs/dash0-qwen27b-chat-0-8k-ttft4s-tpot25-gpu3skip-12iter-harness-profileplanner-v2-20260512.json`
- study id: `dash0-qwen27b-chat-0-8k-ttft4s-tpot25-gpu3skip-12iter-harness-profileplanner-v2-20260512`
- log: `.aituner-tight/logs/qwen27b-profileplanner-v2-harness-20260512.log`
- monitor: read-only subagent `Wegener`
Acceptance for this run is based on end-to-end trial results, not unit tests. If the first four trials lag the min-prompt no-harness baseline (`0.0650`, `0.1992`, `0.2696`, then failed/NA), the run should be treated as a failed harness iteration and the harness should be optimized again.