Exclude partial prompt blocks from prefix cache replay

2026-07-24 00:35:48 +08:00
parent ab952b47e7
commit 5a4011ca12
9 changed files with 25 additions and 8 deletions
--- a/docs/archive/tuning-harness-2604/harness-tuning-progress.md
+++ b/docs/archive/tuning-harness-2604/harness-tuning-progress.md
@@ -0,0 +1,129 @@
+# Harness-Guided AITuner Progress
+
+## Goal
+
+Improve AITuner convergence for the `dash0` internal vLLM + Qwen3.5-27B 0-8k chat study. The prior 12-iteration run can still propose worse configs after finding good ones. The new harness should make config proposals bottleneck-directed and stop spending GPU trials once no adjacent harness-guided probe is justified.
+
+## Paper Alignment
+
+- Prompt structure now includes an explicit `[Harnesses]` section aligned with paper Figure 12.
+- The harness uses the paper's L-C-A workload model:
+  - L: prompt length percentiles and tail ratio.
+  - C: prefix/KV-cache reuse estimated from repeated `hash_ids` blocks when available.
+  - A: request rate, 1-second QPS burst ratio, and interarrival CV.
+- Knob rules follow the paper's Figure 13 style:
+  - map active bottleneck to a knob family;
+  - probe adjacent legal choices;
+  - enforce guard conditions to avoid harmful side effects;
+  - prefer stopping over weak exploratory proposals after convergence.
+
+## Local Implementation Log
+
+- Added `src/aituner/harness.py`.
+  - Builds structured harness context for prompt injection.
+  - Adds TP, max-num-seqs, max-num-batched-tokens, chunked-prefill, and memory-utilization harnesses when those knobs are tunable.
+  - Extracts compact recent trial diagnostics from result JSON files.
+  - Adds a convergence guard based on recent completed trial performance.
+  - Adds an infeasible-progress guard: when recent all-infeasible trials at the same sampling threshold stop improving pass rate and p95 TTFT after changing one knob family, the next proposal must switch primary family or stop.
+  - Classifies `slo_pass_rate_unrecoverable` by latency failure counts first, and ignores probe-budget markers such as `probe_elapsed_s>` for bottleneck voting, so TTFT-heavy failures stay aligned to prefill/TP or batching harnesses instead of being treated as generic queueing.
+- Extended `src/aituner/trace.py`.
+  - `summarize_window` now reports L-C-A features.
+  - `TraceRequest` now carries optional metadata for `hash_ids`, turn, parent chat id, and trace type.
+- Extended `src/aituner/llm.py`.
+  - Prompt now includes tested config signatures and the structured harness section.
+  - Prompt schema now asks for `should_stop`.
+- Extended `src/aituner/spec.py`.
+  - `Proposal` accepts optional `should_stop`.
+- Extended `src/aituner/cli.py`.
+  - `study tune` honors `should_stop=true` by recording the proposal and not launching another GPU trial.
+- Extended `tests/test_core_flow.py`.
+  - Prompt includes harness context.
+  - Trace summary includes new L-C-A fields.
+  - Proposal parsing accepts `should_stop`.
+  - CLI does not launch a trial for a stop proposal.
+
+## Local Verification
+
+- `python3 -m compileall -q src tests`: passed.
+- `PYTHONPATH=src python3 -m unittest tests.test_core_flow`: passed, 62 tests.
+- `pytest -q` and `python3 -m pytest -q`: not runnable locally because `pytest` is not installed.
+
+## Remote Experiment Log
+
+### 2026-04-25 16:30-16:45 CST
+
+- Pushed commit `2c5e9af` to `origin/main` and pulled it on `dash0`.
+- Remote prompt check command:
+  - `PYTHONPATH=src python3 -m aituner.cli study prompt --study-root /tmp/aituner-harness-prompt-check/dash0-qwen27b-tight-slo-10min-run4-chat-0-8k --store-root /tmp/aituner-harness-prompt-check --prompt-name harness-check`
+- Harness profile for `chat_w20260311_1000`, after applying the 0-8k filter:
+  - L: p50 1992, p95 7628, p99 8102, tail ratio 3.83, regime `moderate_tail_prefill_sensitive`.
+  - C: repeated token ratio estimate 0.191, repeated block ratio 0.189, multi-turn ratio 0.160, regime `low_prefix_reuse`.
+  - A: request rate 29.52 req/s, p95 1s QPS 40, burst ratio 1.36, regime `smooth`.
+  - Active harnesses: `tensor-parallel-size` and `max-num-batched-tokens`, which matches a TTFT/prefill-sensitive 0-8k chat workload.
+- Remote `compileall` passed.
+- Remote `unittest discover` initially exposed two pre-existing path-sensitive tests that hardcoded `/home/gahow/phd/aituner`; fixed them to derive `REPO_ROOT` from the test file path.
+
+### 2026-04-25 16:38-16:58 CST
+
+- Started real run in tmux session `aituner_harness_qwen27b_0_8k_20260425`.
+- Store root: `.aituner/harness-studies-20260425`.
+- First proposal followed the harness:
+  - proposal: `tensor-parallel-size: 2`;
+  - rationale: L profile is prefill-sensitive, prefix reuse is low, arrivals are smooth, so probe adjacent TP before runtime batching knobs.
+- First high-load probe at `sampling_u=0.03125` was infeasible:
+  - request rate 0.895 req/s;
+  - pass rate 0.145;
+  - p95 TTFT 4063 ms and p95 TPOT 113 ms;
+  - failed reasons included `tpot_ms>50.0` and `slo_pass_rate_unrecoverable`.
+- Important implementation issue found: after an early-stopped probe, the worker returned while in-flight HTTP requests could continue occupying the engine, stalling/polluting the next binary-search probe.
+- Action: stopped the run and freed GPUs. Updating `worker._replay_requests` to drain in-flight requests after early stop before the next probe starts.
+
+### 2026-04-25 17:00-17:12 CST
+
+- r2 confirmed that draining avoids immediate cross-probe pollution, but the first LLM trial still started from a speculative TP=2 edit without a measured incumbent.
+- This is not aligned with the paper's agentic loop, which evaluates the initial configuration first and then searches from measured feedback.
+- Action: update `study tune` so LLM-driven studies automatically materialize a baseline empty-patch trial first, unless `--skip-baseline` is passed. This should reduce early bad proposals because the first LLM edit will see real baseline bottleneck diagnostics and an incumbent request_rate_per_gpu.
+
+### 2026-04-25 17:20-18:30 CST
+
+- r3 started with baseline-first enabled, but the full 0-8k run was too slow for fast iteration with raw chat completions. Stopped it before using it as a convergence signal.
+- A fast validation using `max_requests_per_probe=160` was invalid: the trace is downsampled before threshold selection, so lower thresholds can end up with `request_count=0`. Do not use that result for performance claims.
+- Prefill smoke v1 used `completion_tokens_override=1` but kept the TPOT SLO. That made TPOT missing failures dominate, so it was useful only for checking control flow, not for performance.
+
+### 2026-04-25 18:30-20:10 CST
+
+- Prefill smoke v2 used real dash0 internal vLLM, Qwen3.5-27B, the real 0-8k prompt distribution and arrivals, `completion_tokens_override=1`, and `tpot_rule=null`.
+- Trial 0001 baseline TP1/DP1:
+  - sampling `0.0078125`: pass rate 0.270, mean TTFT 2033.9 ms, p95 TTFT 5656.7 ms, p99 TTFT 6832.8 ms.
+- Trial 0002 TP1/DP2:
+  - sampling `0.0078125`: pass rate 0.277, mean TTFT 1766.9 ms, p95 TTFT 4215.3 ms, p99 TTFT 5801.7 ms.
+- Trial 0003 TP1/DP4:
+  - sampling `0.0078125`: pass rate 0.345, mean TTFT 1668.9 ms, p95 TTFT 3818.4 ms, p99 TTFT 5804.9 ms.
+- Trial 0004 TP1/DP8:
+  - sampling `0.0078125`: pass rate 0.345, mean TTFT 1675.7 ms, p95 TTFT 3823.4 ms.
+- Interpretation:
+  - The harness improved directionality: after the measured baseline, proposals followed a consistent scale-out path and avoided random runtime-knob churn.
+  - The smoke result improved p95 TTFT by about 32% versus baseline at the low sampling threshold and improved pass rate from 0.270 to 0.345 within 3-4 trials.
+  - It did not reach the 95% pass-rate SLO in this smoke setting, so this is not a full proof of convergence to a good production config.
+  - DP8 did not improve over DP4, which exposed a gap: when every trial is infeasible, the prior convergence guard had no feasible incumbent and could not detect plateau.
+
+### 2026-04-25 20:10 CST
+
+- Added the all-infeasible plateau guard described above.
+- Added unit coverage for:
+  - TTFT failure classification under `slo_pass_rate_unrecoverable`;
+  - blocking a repeat of the DP family after DP4 and DP8 show no material improvement at the same sampling threshold.
+- Pulled the commit on `dash0` and reran remote verification:
+  - `python3 -m compileall -q src tests`: passed.
+  - `PYTHONPATH=src python3 -m unittest discover -s tests -p "test_*.py"`: passed, 62 tests.
+- Regenerated a prompt against the real smoke v2 history:
+  - `convergence_guard.reason`: `data-parallel-size_plateau_on_infeasible_trials`.
+  - `should_stop_if_no_harness_can_justify_a_new_adjacent_probe`: `true`.
+  - blocked primary family: `data-parallel-size`.
+  - latest two active bottlenecks after ignoring `probe_elapsed_s>` for voting: `ttft_prefill`, `ttft_prefill`.
+- Current status: the harness now has the mechanism needed to avoid continuing the exact DP-only direction seen in the smoke v2 plateau. The next real experiment should either switch to a bottleneck-justified mixed TP/DP candidate or return `should_stop=true`.
+
+Remaining next steps:
+
+1. Use the Fig18-style qwen27b 0-8k comparison in `docs/qwen27b-chat-0-8k-harness-fig18.md` as the current convergence evidence.
+2. If a future full no-relaunch rerun is required for publication-quality reproduction, reserve a multi-hour dash0 window; the comparable full-chat evaluator keeps drain-based probe isolation and is much slower than prefill smoke.
--- a/docs/archive/tuning-harness-2604/qwen235b-thinking-prefill-harness-20260427.md
+++ b/docs/archive/tuning-harness-2604/qwen235b-thinking-prefill-harness-20260427.md
@@ -0,0 +1,58 @@
+# qwen235b Thinking Prefill Harness Test
+
+## Setup
+
+- Workload: `qwen3-235b-a22b` thinking trace, prefill-only replay with `min_tokens=max_tokens=1`.
+- Window: `thinking_w20260327_1000`.
+- SLO: 95% pass rate, stepped TTFT `3s/6s/9s`.
+- Metric: best-so-far feasible `request_rate_per_gpu`.
+- Before-harness source: actual 12-trial run
+  `.aituner-prefill/dash0-qwen235b-prefill-thinking-run1-ttft-topology`.
+- Harness test source:
+  `.aituner/harness-qwen235b-prefill-20260427/dash0-qwen235b-prefill-thinking-harness-run1-20260427`.
+
+## Result So Far
+
+The harness run was stopped after establishing the convergence result and observing the next weak proposal. The useful comparison is already visible by iter 2.
+
+| Variant | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 | Iter 7 | Iter 8 | Iter 9 | Iter 10 | Iter 11 | Iter 12 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Before harness, actual run1 | 0.2029 | 0.2029 | 0.2029 | 0.2029 | 0.2029 | 0.3575 | 0.3575 | 0.3708 | 0.3708 | 0.3794 | 0.3794 | 0.3794 |
+| Harness, actual 2026-04-27 run | 0.1892 | 0.3863 | 0.3863 | 0.3863 | n/a | n/a | n/a | n/a | n/a | n/a | n/a | n/a |
+
+## Trial Details
+
+| Variant | Iter | Config | Result |
+| --- | ---: | --- | --- |
+| Before harness | 1 | baseline `TP4/DP1/EP-off`, `MBT=8192` | `0.2029 req/s/gpu` |
+| Before harness | 2 | `DP=2`, `MBT=4096` | runtime failure |
+| Before harness | 3 | `DP=2`, `MBT=8192` | runtime failure |
+| Before harness | 4 | `EP=4` | launch failure |
+| Before harness | 6 | `TP8/DP1/EP-off`, `MBT=4096` | `0.3575 req/s/gpu` |
+| Before harness | 10 | `TP8/DP1/EP-off`, `MBT=3712` | `0.3794 req/s/gpu`, best |
+| Harness | 1 | baseline `TP4/DP1/EP-off`, `MBT=8192` | `0.1892 req/s/gpu` |
+| Harness | 2 | `TP8/DP1/EP-off`, `MBT=8192` | `0.3863 req/s/gpu`, best so far |
+| Harness | 3 | `TP8/DP1/EP=2` | launch failure |
+
+The harness baseline was slightly lower than the original baseline (`0.1892` vs `0.2029 req/s/gpu`), but iter 2 still exceeded the original 12-trial best (`0.3863` vs `0.3794 req/s/gpu`).
+
+## Convergence Judgment
+
+- Before harness reached its best at iter 10.
+- Harness reached a better result at iter 2.
+- Iterations-to-best improved from `10` to `2`, a `5x` improvement on this run.
+- The important behavior change is that the harness skipped the original failed DP2 and EP4 exploration and moved directly from baseline to `TP8/DP1`.
+
+## Follow-Up Optimization
+
+The run also exposed a remaining weakness: after reaching the strong `TP8/DP1` incumbent, the LLM proposed `EP=2`, which failed at launch. To address that, the harness was tightened after this test:
+
+- strong-incumbent stop threshold changed from `3x` to `1.8x` over baseline;
+- expert parallel is now explicitly guarded and should not be introduced for TTFT-prefill bottlenecks without direct positive EP evidence.
+
+With the new guard, the intended behavior after this iter-2 result is `should_stop=true` unless a same-topology runtime harness has strong direct evidence.
+
+## Run Status
+
+- The 2026-04-27 harness run was stopped after collecting the iter-2 convergence result and the iter-3 EP failure.
+- GPUs were freed after stopping the run.
--- a/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-current-config-fig18-20260506.md
+++ b/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-current-config-fig18-20260506.md
@@ -0,0 +1,173 @@
+# qwen27b-chat-0-8k Current-Config Fig18 Plan
+
+## Question
+
+The earlier tables used best-so-far throughput. That is useful for deciding the
+best deployable incumbent, but it hides bad proposals because the curve is
+monotonic by construction. To judge whether the harness makes tuning more
+directional, the primary curve must be each iteration's measured current config
+performance.
+
+## Why Final Performance Can Be Close
+
+Harness and no-harness can converge to similar final throughput when the search
+space contains one dominant simple family. In this setup the dominant family is
+`TP=2, DP=1` over the `run_qwen27b.sh` baseline. The no-harness LLM can still
+eventually discover that family within 12 iterations, so final best performance
+can be close.
+
+The difference the harness is expected to improve is not necessarily the final
+12-iter maximum. It should improve:
+
+- iterations-to-first-good-config;
+- number of worse or infeasible proposals after an incumbent is found;
+- measured-current config oscillation;
+- early-stop behavior once adjacent harness probes no longer justify more GPU
+  trials.
+
+## Metrics
+
+- `measured-current`: each trial's own feasible `request_rate_per_gpu`.
+  Failed or no-feasible-point trials are recorded as `NA`.
+- `accepted-incumbent`: best deployable value after each trial. This is the
+  standard best-so-far curve and is monotonic by definition.
+- `iters-to-best`: first iteration where the final best value or equivalent
+  config family appears.
+- `wasted-trials-after-best`: trials after first best that are worse, infeasible,
+  or no-feasible-point.
+
+## Historical Run9 Re-Read
+
+Source:
+`.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology`
+on dash0.
+
+| Variant | Curve | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 | Iter 7 | Iter 8 | Iter 9 | Iter 10 | Iter 11 | Iter 12 |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| no-harness run9 | measured-current | 0.0350 | 0.0617 | 0.0392 | 0.2025 | NA | NA | NA | NA | NA | NA | NA | NA |
+| no-harness run9 | accepted-incumbent | 0.0350 | 0.0617 | 0.0617 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 |
+
+Interpretation: the no-harness current-config curve already has a regression at
+iter 3 and then many no-feasible-point runtime probes. The monotonic curve only
+shows the incumbent policy, not proposal quality.
+
+## New Paired Test Plan
+
+Run on dash0 with internal vLLM and the real `chat_w20260311_1000` 0-8k replay:
+
+- Base spec: `configs/examples/dash0_qwen27b_tight_slo_run4_0_8k.json`.
+- Model path:
+  `/home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal`.
+- Naming note: local configs and dash0 model directories expose this setup as
+  Qwen3.5-27B/Qwen35-27B, not `qwen32b`.
+- Engine: `/usr/local/bin/vllm`, baseline aligned with `~/run_qwen27b.sh`.
+- SLO: 95% pass, stepped TTFT `2s/4s/6s`, TPOT `<=50ms`.
+- Search: `low=0`, `high=0.0625`, `max_probes=6`, `tolerance=0.001`.
+- no-harness study:
+  `.aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-noharness`.
+- harness study:
+  `.aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-harness`.
+
+The result table will report both curves. The harness is considered successful
+only if it reaches the same or better incumbent in fewer iterations and reduces
+the measured-current regressions or replaces them with an explicit harness stop.
+
+## Run Status
+
+- 2026-05-06 07:05 CST: dash0 checked, 8 H20 GPUs idle.
+- 2026-05-06 07:05 CST: generated paired specs under
+  `.aituner-tight/specs/`.
+- 2026-05-06 07:05 CST: started no-harness full 12-iter run in tmux session
+  `qwen27b_run10_noharness_20260506`.
+- 2026-05-06 07:18 CST: stopped the duplicate fresh no-harness run before
+  completion. Reason: run9 is already a completed real 12-iter no-harness run
+  for the same internal vLLM 0-8k setup, while the fresh full-chat run would
+  spend a multi-hour dash0 slot duplicating that curve.
+- 2026-05-06 07:20 CST: seeded the harness study with the real run9 baseline
+  measurement as `trial-0001`, then started the harness run with
+  `--skip-baseline` in tmux session `qwen27b_run10_harness_skipbase_20260506`.
+- 2026-05-06 07:20 CST: harness generated deterministic `trial-0002`:
+  `{"tensor-parallel-size": 2}`.
+- 2026-05-06 08:11 CST: harness `trial-0002` completed:
+  `TP=2`, `0.2142 request_rate_per_gpu`.
+- 2026-05-06 08:19 CST: harness `trial-0003` failed at engine launch.
+  Root cause: the old runtime refinement coupled `gpu-memory-utilization=0.95`
+  with larger `max-num-batched-tokens`, causing speculative sampler warmup OOM.
+  This is a generic harness safety bug; fixed locally by removing the automatic
+  memory-utilization bump from runtime refinement.
+- 2026-05-06 09:24 CST: harness `trial-0004` completed:
+  `TP=4`, `0.4429 request_rate_per_gpu`. All six probes were feasible up to
+  `sampling_u=0.0615234375`, so this study is near the configured
+  `search.high=0.0625` ceiling.
+- 2026-05-06 09:25 CST: old harness repeated the same unsafe runtime refinement
+  for TP4 and `trial-0005` failed at engine launch for the same OOM reason. The
+  old process was stopped before continuing.
+- 2026-05-06 09:37 CST: pulled commit `5d96689` on dash0 and resumed. The
+  runtime-refinement OOM was fixed, but the stop guard was still too strict: it
+  did not treat a feasible high-edge probe with a small number of SLO failures
+  as saturation, even though the probe already met the 95% pass-rate target.
+- 2026-05-06 09:50 CST: stopped the unnecessary product-8 validation. The queued
+  `trial-0006`/`trial-0007` are not used for convergence claims.
+- 2026-05-06 09:56 CST: pulled commit `f653af0` on dash0. The fixed high-edge
+  stop guard produced `harness-stop-0008` without launching another GPU trial.
+
+## Current Results
+
+Unit: feasible `request_rate_per_gpu`. `NA` means the current trial did not
+produce a feasible deployable config.
+
+| Variant | Curve | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 | Iter 7 | Iter 8 | Iter 9 | Iter 10 | Iter 11 | Iter 12 |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| no-harness run9 | measured-current | 0.0350 | 0.0617 | 0.0392 | 0.2025 | NA | NA | NA | NA | NA | NA | NA | NA |
+| no-harness run9 | accepted-incumbent | 0.0350 | 0.0617 | 0.0617 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 |
+| harness run10 | measured-current | 0.0350 | 0.2142 | NA | 0.4429 | NA | skipped | skipped | stop |  |  |  |  |
+| harness run10 | accepted-incumbent | 0.0350 | 0.2142 | 0.2142 | 0.4429 | 0.4429 | 0.4429 | 0.4429 | 0.4429 stop |  |  |  |  |
+
+The harness result is stronger than the earlier strict replay. It did not merely
+reach the same TP2 region earlier; it then used the bottleneck/topology evidence
+to validate TP4 and found a much higher current config.
+
+## Interpretation
+
+- Why both variants can look close when only best-so-far is shown: no-harness can
+  eventually find a good simple topology, and best-so-far hides every bad
+  proposal after that point.
+- What the current-config curve shows: no-harness regresses at iter 3 and then
+  spends many iterations on no-feasible-point runtime probes. Harness reaches a
+  stronger TP2 config at iter 2 and a stronger TP4 config at iter 4.
+- Why harness helped: the baseline diagnostics identify TTFT/prefill as the
+  active bottleneck on low-prefix-reuse long prompts. The harness maps that to
+  adjacent TP validation before DP/runtime exploration. The no-harness LLM chose
+  DP2 then DP4 first, which diluted per-GPU throughput and delayed TP.
+- Defect fixed during the run: runtime refinement was too aggressive because it
+  combined larger MBT with higher memory utilization. It now changes batching
+  headroom without also raising memory pressure.
+- Stop defect fixed during the run: high-edge probes can have a few individual
+  latency failures and still be feasible under the configured pass-rate SLO. The
+  stop guard now keys on `feasible=true` near `search.high`, not on an empty
+  failed-reason map.
+- Search-high implication: TP4 reached `sampling_u=0.0615234375` with
+  `search.high=0.0625`, so the current spec is saturated for this topology. A
+  higher `search.high` would be required to distinguish whether TP4 can go even
+  higher in absolute throughput; it is not needed to show that harness converged
+  faster than no-harness under this spec.
+
+## Mechanism
+
+The harness contributes structured, non-testcase-specific information:
+
+- Workload features: long-prompt 0-8k distribution, low prefix reuse, and smooth
+  arrivals.
+- Bottleneck diagnosis from probes: baseline failures are TTFT/prefill-heavy, so
+  topology changes that reduce long-prefill latency should be tried before DP or
+  runtime batching.
+- Topology adjacency: validate TP1 -> TP2 -> TP4 rather than jumping randomly or
+  repeating a failing runtime family.
+- Stop condition: once the incumbent's feasible probe is within one binary-search
+  resolution of `search.high`, stop instead of spending more GPU trials.
+
+Without the harness, the LLM response in run9 chose DP2 and DP4 before TP2. That
+temporarily improved total request rate but reduced per-GPU efficiency, so the
+measured-current curve dipped at iter 3 and reached the old best only at iter 4.
+With the harness, the LLM receives the bottleneck/topology frame and chooses
+TP-oriented validation; TP2 is reached at iter 2 and TP4 at iter 4.
--- a/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-harness-fig18.md
+++ b/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-harness-fig18.md
@@ -0,0 +1,58 @@
+# qwen27b-chat-0-8k Harness Fig18
+
+## Setup
+
+- Workload: `qwen3.5-27b` chat, `0 <= input_length <= 8192`.
+- Window: `chat_w20260311_1000`.
+- Engine: dash0 internal vLLM, baseline aligned to `run_qwen27b.sh`.
+- SLO: 95% pass rate, stepped TTFT `2s/4s/6s`, TPOT `<=50ms`.
+- Search metric: best-so-far feasible `request_rate_per_gpu`.
+- Before-harness source: actual 12-trial run
+  `.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology`.
+- After-harness source: strict harness replay over already measured run9 configs:
+  - Iter 1 uses the measured baseline trial.
+  - Iter 2 uses the current harness proposal after seeing only iter 1 history. It proposes `TP=2, DP=1`, whose performance is the measured run9 `trial-0004` result for the same config and spec.
+  - Iter 3 uses the current harness proposal after seeing only baseline + `TP=2, DP=1`. With the strong-incumbent guard, it returns `should_stop=true`.
+
+The replay is intentionally strict: the LLM prompt does not receive future `best_by_parallel_size` entries or later failed trials.
+
+## Fig18-Style Best-So-Far Curve
+
+Unit: feasible `request_rate_per_gpu`. Infeasible trials leave the best-so-far value unchanged.
+
+| Variant | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 | Iter 7 | Iter 8 | Iter 9 | Iter 10 | Iter 11 | Iter 12 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| Before harness, actual run9 | 0.0350 | 0.0617 | 0.0617 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 |
+| After harness, strict replay | 0.0350 | 0.2025 | 0.2025 stop | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 |
+
+## Trial-Level Interpretation
+
+| Variant | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5-12 |
+| --- | --- | --- | --- | --- | --- |
+| Before harness | baseline `TP1/DP1`, 0.0350 | `DP=2`, 0.0617 | `DP=4`, 0.0392, worse per GPU | `TP=2, DP=1`, 0.2025, best | runtime-only probes, all infeasible |
+| After harness | baseline `TP1/DP1`, 0.0350 | `TP=2, DP=1`, 0.2025, best | `should_stop=true` | no GPU trial | no GPU trial |
+
+## Convergence Judgment
+
+- Before harness reaches the final best value at iter 4.
+- After harness reaches the same best value at iter 2.
+- The speedup is `2x` by iterations-to-best: `4 -> 2`.
+- The harness also avoids the post-best weak proposals: before harness spent iters 5-12 on infeasible runtime-only probes; after harness stops at iter 3.
+
+## Implementation Changes From This Check
+
+- Added a strong-incumbent convergence guard:
+  - if the latest trial is the incumbent,
+  - and it improves `request_rate_per_gpu` by at least `3x` over the baseline,
+  - then runtime-only probes require direct same-topology evidence; otherwise the LLM should return `should_stop=true`.
+- Strengthened the MBT harness guard:
+  - do not raise `max-num-batched-tokens` when incumbent MBT already covers prompt p99 unless same-topology evidence proves prefill fragmentation.
+- Made early-stop engine relaunch opt-in. A real r2 run showed that default relaunch changes warm-state behavior and makes full-chat results incomparable with run9, so the default remains drain-based for comparable production measurements.
+- Added LLM empty-response retry to avoid crashing `study tune` on a transient empty streamed response.
+
+## Remote Checks
+
+- Local: `python3 -m compileall -q src tests` passed.
+- Local: `PYTHONPATH=src python3 -m unittest tests.test_core_flow` passed, 63 tests.
+- dash0: `python3 -m compileall -q src tests` passed.
+- dash0: `PYTHONPATH=src python3 -m unittest discover -s tests -p "test_*.py"` passed, 63 tests.
--- a/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-setup-audit-20260506.md
+++ b/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-setup-audit-20260506.md
@@ -0,0 +1,169 @@
+# qwen27b-chat-0-8k Setup and Result Audit
+
+## Purpose
+
+This note audits the 2026-05-06 qwen27b chat 0-8k harness result because the
+new best `0.4429 request_rate_per_gpu` is much higher than the previous
+no-harness best `0.2025`.
+
+## Setup
+
+- Host: `dash0`.
+- Hardware: 8 NVIDIA H20 GPUs.
+- Engine: internal vLLM at `/usr/local/bin/vllm`.
+- Model:
+  `/home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal`.
+- Served model name: `qwen35-27b-aituner`.
+- Workload window: `chat_w20260311_1000`.
+- Trace file source: `trace_windows/windows.json`.
+- Request mode: `chat`.
+- Input bucket: `0 <= input_length <= 8192`.
+- Replay scale: `1.0`.
+- Max concurrency: `32`.
+- Max requests per probe: unset, so full selected trace subset is replayed.
+- Search field: `sampling_u`.
+- Search range: `low=0.0`, `high=0.0625`.
+- Search probes: `max_probes=6`, `tolerance=0.001`.
+- Sampling seed: `20260325`.
+
+The local configs and dash0 model directories name this setup Qwen3.5-27B /
+Qwen35-27B. I did not find a `qwen32b` model/config for this internal chat
+0-8k setup.
+
+## SLO
+
+- Target pass rate: `0.95`.
+- TTFT rule: stepped by input length.
+
+| Input tokens | TTFT threshold |
+| ---: | ---: |
+| `<=4096` | `2000 ms` |
+| `<=32768` | `4000 ms` |
+| otherwise | `6000 ms` |
+
+- TPOT rule: fixed `<=50 ms`.
+
+A probe is feasible when its pass rate is at least `0.95`. Individual requests
+may still fail TTFT/TPOT while the whole probe remains feasible.
+
+## Compared Studies
+
+| Variant | Study root | Notes |
+| --- | --- | --- |
+| no-harness | `.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology` | completed 12-trial historical run |
+| harness | `.aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-harness` | seeded with run9 baseline, then ran real harness trials |
+
+The harness run reused the real run9 baseline as `trial-0001` to avoid
+duplicating a multi-hour cold-start baseline measurement. Later harness trials
+were real dash0 runs.
+
+## Metric
+
+The reported metric is `request_rate_per_gpu`:
+
+```text
+request_rate_per_gpu = best_feasible_request_rate / parallel_size
+parallel_size = tensor_parallel_size * data_parallel_size
+```
+
+The result JSON stores `best_request_rate`; `StudyStore.ingest_trial_results`
+derives `best_request_rate_per_gpu` from the trial spec topology.
+
+## Result Table
+
+Unit: feasible `request_rate_per_gpu`.
+
+| Variant | Curve | Iter 1 | Iter 2 | Iter 3 | Iter 4 | Iter 5 | Iter 6 | Iter 7 | Iter 8 |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| no-harness run9 | measured-current | 0.0350 | 0.0617 | 0.0392 | 0.2025 | NA | NA | NA | NA |
+| no-harness run9 | accepted-incumbent | 0.0350 | 0.0617 | 0.0617 | 0.2025 | 0.2025 | 0.2025 | 0.2025 | 0.2025 |
+| harness run10 | measured-current | 0.0350 | 0.2142 | NA | 0.4429 | NA | skipped | skipped | stop |
+| harness run10 | accepted-incumbent | 0.0350 | 0.2142 | 0.2142 | 0.4429 | 0.4429 | 0.4429 | 0.4429 | 0.4429 stop |
+
+## Why `0.4429` Is Plausible
+
+The new value is not the old TP2 config suddenly doubling. The comparable TP2
+results are close:
+
+| Study | Trial | Config | Best request rate | Parallel size | request_rate_per_gpu |
+| --- | --- | --- | ---: | ---: | ---: |
+| run9 | `trial-0004` | `TP=2, DP=1` | 0.4050 | 2 | 0.2025 |
+| run10 | `trial-0002` | `TP=2` | 0.4283 | 2 | 0.2142 |
+
+The large jump comes from a new topology that run9 did not evaluate:
+
+| Study | Trial | Config | Best request rate | Parallel size | request_rate_per_gpu |
+| --- | --- | --- | ---: | ---: | ---: |
+| run10 | `trial-0004` | `TP=4` | 1.7717 | 4 | 0.4429 |
+
+At the winning TP4 probe:
+
+- `sampling_u=0.0615234375`;
+- request count `1063`;
+- request rate `1.7717 req/s`;
+- pass rate `0.9680`;
+- p95 TTFT `1476.9 ms`;
+- p95 TPOT `44.4 ms`.
+
+This satisfies the configured SLO and is within one binary-search resolution of
+`search.high=0.0625`.
+
+## Correctness Audit
+
+The following fields match between run9 and run10 except for intentionally
+different identity fields such as study id and port:
+
+- model path and served model name;
+- internal vLLM executable;
+- base launch flags other than port;
+- trace window `chat_w20260311_1000`;
+- input-length filter `0-8192`;
+- replay scale `1.0`;
+- max concurrency `32`;
+- full selected trace replay, no `max_requests_per_probe`;
+- SLO target and TTFT/TPOT thresholds;
+- search `high=0.0625`, `max_probes=6`, `tolerance=0.001`, seed `20260325`;
+- metric definition `best_request_rate / (TP * DP)`.
+
+Checked differences and their impact:
+
+- Port differs: run9 used `18087`, run10 used `18082`; this should not affect
+  measured throughput.
+- run10 has explicit `restart_engine_after_early_stop=false`; chat studies
+  default to the same behavior.
+- run10 has explicit `completion_tokens_override=null`; equivalent to run9's
+  absent field.
+- run9 `trial-0004` search floor was `0.00390625` because it reused the
+  incumbent for the same parallel-size group. run10 `trial-0004` search floor
+  was `0.0` because pure `TP=4` had not been tried. Both have the same high and
+  probe budget; this does not explain the higher result.
+
+No metric-code logic error was found in the audit. The result JSONs store raw
+request rate, and the state computes per-GPU throughput by dividing by
+`TP*DP`. For run10 TP4, `1.7716666667 / 4 = 0.4429166667`.
+
+## Issues Found During The Test
+
+Two harness bugs were found and fixed:
+
+- Runtime refinement coupled larger `max-num-batched-tokens` with
+  `gpu-memory-utilization=0.95`, which caused launch-time OOM. Fixed in commit
+  `5d96689`.
+- The search-high stop guard incorrectly required no individual SLO failures at
+  a feasible high-edge probe. Fixed in commit `f653af0`; feasibility already
+  means the probe passed the configured pass-rate SLO.
+
+The queued product-8 `trial-0006` and `trial-0007` were stopped after the stop
+guard fix and are not used in the convergence claim.
+
+## Conclusion
+
+The `0.4429` result is being compared under the same workload, SLO, search
+range, and metric definition as the previous `0.2025` result. The reason it is
+higher is that no-harness run9 did not evaluate pure `TP=4`; the harness guided
+the search from the TTFT/prefill bottleneck to adjacent TP validation and found
+that topology by iter 4.
+
+Because TP4 nearly saturates the configured `search.high`, a follow-up run with
+a higher `search.high` is needed to measure the absolute ceiling. That follow-up
+is separate from the current convergence comparison.
--- a/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-tpot25-16iter-20260506.md
+++ b/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-tpot25-16iter-20260506.md
@@ -0,0 +1,106 @@
+# qwen27b-chat-0-8k TPOT25 16-Iter Harness Compare
+
+## Goal
+
+Rerun the internal vLLM Qwen3.5-27B chat 0-8k tuning comparison under a stricter
+TPOT SLO:
+
+- no-harness: 16 tuning iterations;
+- harness: 16 tuning iterations, with permission to stop early if the harness
+  convergence guard decides no further GPU trial is needed.
+
+Both variants must be launched directly through AITuner. No state seeding,
+manual replay, or historical-result injection is allowed.
+
+## Setup
+
+- Host: `dash0`.
+- Hardware: 8 NVIDIA H20 GPUs.
+- Engine: internal vLLM at `/usr/local/bin/vllm`.
+- Model:
+  `/home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal`.
+- Served model name: `qwen35-27b-aituner`.
+- Workload window: `chat_w20260311_1000`.
+- Trace path source: `/home/admin/cpfs/wjh/aituner/aituner/trace_windows/windows.json`.
+- Request mode: `chat`.
+- Input bucket: `0 <= input_length <= 8192`.
+- Replay scale: `1.0`.
+- Max concurrency: `32`.
+- Max requests per probe: unset, so each probe uses the full selected trace
+  subset for its `sampling_u` threshold.
+- Restart engine after early stop: `true` for both variants. This is needed
+  under TPOT25 because very slow infeasible probes can leave live HTTP requests
+  in the engine after the SLO is already unrecoverable. Restarting keeps the
+  next binary-search probe from being contaminated by previous in-flight work.
+- Search field: `sampling_u`.
+- Search range: `low=0.0`, `high=0.0625`.
+- Search probes: `max_probes=6`, `tolerance=0.001`.
+- Sampling seed: `20260325`.
+
+## SLO
+
+- Target pass rate: `0.95`.
+- TTFT rule:
+
+| Input tokens | TTFT threshold |
+| ---: | ---: |
+| `<=4096` | `2000 ms` |
+| `<=32768` | `4000 ms` |
+| otherwise | `6000 ms` |
+
+- TPOT rule: fixed `<=25 ms`.
+
+## Specs
+
+Remote generated specs:
+
+- no-harness:
+  `.aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot25-restart-16iter-noharness.json`
+- harness:
+  `.aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot25-restart-16iter-harness.json`
+
+The two specs were generated from
+`configs/examples/dash0_qwen27b_tight_slo_run4_0_8k.json`. After normalizing
+`study_id` and `llm.use_harness`, the JSON payloads compare equal. Therefore the
+only tuning-behavior difference between the formal comparison runs is whether
+the harness is enabled.
+
+## Commands
+
+No-harness:
+
+```bash
+PYTHONPATH=src python3 -m aituner.cli study tune \
+  --spec .aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot25-restart-16iter-noharness.json \
+  --store-root .aituner-tight \
+  --max-trials 16
+```
+
+Harness:
+
+```bash
+PYTHONPATH=src python3 -m aituner.cli study tune \
+  --spec .aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot25-restart-16iter-harness.json \
+  --store-root .aituner-tight \
+  --max-trials 16
+```
+
+## Run Log
+
+- 2026-05-06 12:37 CST: generated both remote specs and verified that the only
+  normalized difference is `llm.use_harness`.
+- 2026-05-06 12:37 CST: started no-harness in tmux session
+  `qwen27b_tpot25_noharness_16iter_20260506`.
+- 2026-05-06 21:06 CST: stopped the initial no-harness pre-run before using it
+  for comparison. It used `restart_engine_after_early_stop=false`; the first
+  TP1 baseline probe already recorded `slo_pass_rate_unrecoverable`, but
+  unfinished requests remained live in vLLM and would contaminate the next probe.
+- 2026-05-06 21:07 CST: generated the formal clean specs with
+  `restart_engine_after_early_stop=true` for both variants and verified the
+  normalized diff is still only `llm.use_harness`.
+- 2026-05-06 21:09 CST: started formal no-harness run in tmux session
+  `qwen27b_tpot25_restart_noharness_16iter_20260506`.
+
+## Results
+
+Pending.
--- a/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-tpot40-baseline-infeasible-20260507.md
+++ b/docs/archive/tuning-harness-2604/qwen27b-chat-0-8k-tpot40-baseline-infeasible-20260507.md
@@ -0,0 +1,131 @@
+# Qwen27B Chat 0-8k TPOT 40ms Baseline Infeasible Run
+
+Date: 2026-05-07
+
+## Goal
+
+Re-run the internal vLLM + Qwen3.5-27B chat 0-8k tuning comparison after adding a study-level guard:
+
+- if the automatic baseline trial has no feasible probe;
+- and the lowest sampled request rate still fails the SLO target pass rate;
+- then AITuner stops the whole study and reports that the SLO is too tight for the current setup.
+
+This prevents spending the remaining tuning budget on LLM or harness proposals when the baseline itself demonstrates that the workload/SLO is infeasible at the search floor.
+
+## Implementation
+
+Commit: `f212673 Stop tuning when baseline is infeasible`
+
+Changed behavior:
+
+- `study tune` now persists `tuning_stop_reason` and `tuning_stop_diagnosis` in `state.json`.
+- `study tune` also persists `tuning_stop_details`, including the lowest sampled probe's TTFT/TPOT mean, p50, p95, and p99.
+- After the automatic baseline trial is ingested, AITuner checks the worker result:
+  - `status == completed`
+  - `best_request_rate is None`
+  - at least one probe exists
+  - all probes are infeasible
+- If true, AITuner stops before asking the LLM or harness for any proposal.
+- Re-running the same study respects the persisted stop state and does not resume tuning.
+
+Validation:
+
+```bash
+python3 -m compileall -q src tests
+PYTHONPATH=src python3 -m unittest tests.test_core_flow
+```
+
+Local and `dash0` both passed.
+
+## Setup
+
+Host: `dash0`
+
+Remote repo: `/home/admin/cpfs/wjh/aituner/aituner`
+
+Base spec: `configs/examples/dash0_qwen27b_tight_slo_run4_0_8k.json`
+
+Model: `/home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal`
+
+Workload: chat, 0-8k input window
+
+SLO:
+
+- TTFT: existing step rule from the base spec
+- TPOT: fixed `40ms`
+- target pass rate: `0.95`
+
+Search:
+
+- Direct AITuner command: `python3 -m aituner.cli study tune ... --max-trials 12`
+- No manual proposal/state edits during either run.
+- Both variants used `CUDA_VISIBLE_DEVICES=0,1,2,4,5,6,7`; this was identical for both specs.
+- The two specs were verified equal after normalizing only `study_id` and `llm.use_harness`.
+
+Specs:
+
+- no-harness: `.aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot40-gpu3skip-12iter-noharness-20260507.json`
+- harness: `.aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot40-gpu3skip-12iter-harness-20260507.json`
+
+## Commands
+
+No harness:
+
+```bash
+PYTHONPATH=src python3 -m aituner.cli study tune \
+  --spec .aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot40-gpu3skip-12iter-noharness-20260507.json \
+  --store-root .aituner-tight \
+  --max-trials 12
+```
+
+Harness:
+
+```bash
+PYTHONPATH=src python3 -m aituner.cli study tune \
+  --spec .aituner-tight/specs/dash0-qwen27b-chat-0-8k-tpot40-gpu3skip-12iter-harness-20260507.json \
+  --store-root .aituner-tight \
+  --max-trials 12
+```
+
+## Results
+
+Both runs stopped after the baseline trial. No LLM/harness proposal was evaluated because baseline had no feasible probe.
+
+| Variant | Trials executed | Best request rate | Best request rate / GPU | Stop reason |
+| --- | ---: | ---: | ---: | --- |
+| no-harness | 1 | - | - | `baseline_all_infeasible` |
+| harness | 1 | - | - | `baseline_all_infeasible` |
+
+Baseline probe curve:
+
+| sampling_u | request rate | pass rate | feasible | early stop reason |
+| ---: | ---: | ---: | --- | --- |
+| 0.03125 | 0.895 | 0.000000 | false | `slo_pass_rate_unrecoverable` |
+| 0.015625 | 0.483333 | 0.137931 | false | `slo_pass_rate_unrecoverable` |
+| 0.0078125 | 0.246667 | 0.236486 | false | `slo_pass_rate_unrecoverable` |
+| 0.00390625 | 0.123333 | 0.189189 | false | `slo_pass_rate_unrecoverable` |
+| 0.001953125 | 0.065000 | 0.205128 | false | `slo_pass_rate_unrecoverable` |
+| 0.0009765625 | 0.035000 | 0.142857 | false | `slo_pass_rate_unrecoverable` |
+
+Lowest request rate latency summary:
+
+| Variant | request rate | pass rate | TTFT mean | TTFT p50 | TTFT p95 | TTFT p99 | TPOT mean | TPOT p50 | TPOT p95 | TPOT p99 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| no-harness | 0.035000 | 0.142857 | 1288.953ms | 446.586ms | 3011.814ms | 3011.814ms | 12.661ms | 13.141ms | 15.097ms | 15.097ms |
+| harness | 0.035000 | 0.142857 | 1268.090ms | 445.274ms | 2889.080ms | 2889.080ms | 12.658ms | 13.170ms | 15.102ms | 15.102ms |
+
+This shows that the TPOT threshold of `40ms` is not the binding constraint at the lowest sampled rate. The observed TPOT p99 is about `15.1ms`; failures are driven by TTFT and by the unrecoverable-pass-rate early stop after too many requests have already failed or been skipped.
+
+Final diagnosis written by AITuner:
+
+```text
+Baseline configuration has no feasible probe under the current SLO. Stopping tuning because even the lowest sampled request rate did not meet the target pass rate. lowest_sampled_request_rate=0.035 lowest_sampling_u=0.000976562 lowest_probe_pass_rate=0.142857 early_stop_reason=slo_pass_rate_unrecoverable
+```
+
+## Interpretation
+
+This run does not measure harness acceleration. It proves that the TPOT 40ms setup is infeasible for the current baseline and search floor: even at `0.035` aggregate request rate, only `14.29%` of requests pass the SLO, far below the `95%` target.
+
+The correct behavior is to stop the study early and report SLO infeasibility instead of spending the remaining 11 trial slots. Harness cannot accelerate convergence when there is no feasible baseline point and no incumbent for guided tuning.
+
+For a Fig. 18-style convergence comparison, the next setup must first have at least one feasible baseline or feasible low-rate point under the same metric definitions.