aituner/docs/harness-ablation/qwen235b-thinking-prefill-ttft-20260510.md

# qwen235b Thinking Prefill Harness Ablation, 2026-05-10

## Setup

- Host: `dash0`
- Engine: internal vLLM at `/usr/local/bin/vllm`
- Model: `/home/admin/resource/model/464482ce.qwen3-235b-a22b/256k-0717`
- Trace window: `thinking_w20260327_1000`
- Request mode: chat, with `completion_tokens_override=1` for prefill-only measurement
- SLO: TTFT-only stepped p95 pass target, target pass rate `0.95`
  - input tokens `<=4096`: `3000 ms`
  - input tokens `<=32768`: `6000 ms`
  - otherwise: `9000 ms`
- Search: `sampling_u` in `[0, 0.125]`, tolerance `0.001`, max probes `6`
- Trial budget: no-harness allowed 12 GPU trials; harness allowed 12 but could stop early
- Store root: `.aituner-prefill`

The two fresh specs were identical except `study_id` and `llm.use_harness`:

- no-harness: `.aituner-prefill/specs/dash0-qwen235b-prefill-thinking-run1-ttft-harness-ablation-12iter-noharness-rerun2-20260510.json`
- harness: `.aituner-prefill/specs/dash0-qwen235b-prefill-thinking-run1-ttft-harness-ablation-12iter-harness-rerun2-20260510.json`

Both runs were launched through `python3 -m aituner.cli study tune`; no proposal or study state was edited manually during tuning.

## Result

The table below is the raw per-iteration performance for a Fig18-style plot. Use this table as `perf[i]`; do not replace missing points with `max(perf[:i+1])`.

Metric: `best_request_rate_per_gpu` from that trial's own `result.json`. `NA` means the proposed config did not produce a feasible point under the SLO, either because the engine/probe failed or because every sampled probe was infeasible.

| Variant | iter1 | iter2 | iter3 | iter4 | iter5 | iter6 | iter7 | iter8 | iter9 | iter10 | iter11 | iter12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness raw `perf[i]` | 0.2029 | NA | NA | 0.3863 | NA | NA | NA | 0.3879 | 0.3892 | 0.3896 | 0.3900 | 0.3900 |
| harness raw `perf[i]` | 0.2029 | NA | 0.3863 | stop | stop | stop | stop | stop | stop | stop | stop | stop |

The raw no-harness curve is therefore not monotonic. The apparent monotonic 12-iter sequence comes only from plotting best-so-far rather than the measured performance of each proposal.

Per-trial details:

| Variant | iter1 | iter2 | iter3 | iter4 | iter5 | iter6 | iter7 | iter8 | iter9 | iter10 | iter11 | iter12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness, per-trial | 0.2029 | - | - | 0.3863 | - | - | - | 0.3879 | 0.3892 | 0.3896 | 0.3900 | 0.3900 |
| harness, per-trial | 0.2029 | - | 0.3863 | stop | stop | stop | stop | stop | stop | stop | stop | stop |

Best-so-far curve, shown only to explain final incumbent selection:

| Variant | iter1 | iter2 | iter3 | iter4 | iter5 | iter6 | iter7 | iter8 | iter9 | iter10 | iter11 | iter12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness | 0.2029 | 0.2029 | 0.2029 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3879 | 0.3892 | 0.3896 | 0.3900 | 0.3900 |
| harness | 0.2029 | 0.2029 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3863 | 0.3863 |

For plotting raw `perf[i]`, the failed/infeasible points should stay missing or be rendered as invalid trials. If a plotting script requires numeric values, use `0` only with an explicit label that this means "no feasible configuration under the configured SLO"; do not forward-fill from the incumbent.

Final best:

| Variant | GPU trials spent | Best trial | Best config summary | Best req/s | Best req/s/GPU | Final vs no-harness |
| --- | ---: | --- | --- | ---: | ---: | ---: |
| no-harness | 12 | `trial-0011`/`trial-0012` | TP8, DP1, EP off, `max-num-batched-tokens` 7936/8064 | 3.1200 | 0.3900 | baseline |
| harness | 3 | `trial-0003` | TP8, DP1, EP off | 3.0900 | 0.3863 | -0.96% |

Harness reached `0.38625 req/s/GPU` at iter3. No-harness first reached the same TP8 family at iter4, then spent eight more GPU trials to move from `0.38625` to `0.39000 req/s/GPU`, an absolute gain of `0.00375 req/s/GPU` or `0.97%`.

## What the Harness Did

The harness did not use a testcase-specific throughput threshold. The stop decision came from the generic search-high saturation rule:

- incumbent highest feasible probe: `sampling_u=0.123046875`
- configured `search.high`: `0.125`
- binary-search resolution: `(0.125 - 0.0) / 2^6 = 0.001953125`
- gap to search high: `0.001953125`

Because the incumbent was feasible and within one configured search resolution of `search.high`, the harness emitted `harness-stop-0004` before launching another GPU trial. This means the current study could no longer measure a materially higher workload without increasing `search.high`; it is not a claim of global engine optimality.

The harness context also made the LLM response more directed after failure:

- After baseline, it exposed the TTFT-only prefill bottleneck and the sharp queueing knee around `sampling_u=0.03515625`.
- The LLM first chose TP4/DP2 to use the idle 4 GPUs while preserving the validated TP4 shard shape. This failed with `connection refused`, matching the no-harness failure family.
- The next harness prompt included that failure, and the LLM switched to TP8/DP1 with EP off, explicitly avoiding the failed DP2 family.
- No-harness inserted an extra EP4 launch-failure trial before reaching TP8/DP1.

## Conclusion

Harness accelerated convergence mainly through early stopping, not by finding a much better final config on this setup. It reduced GPU trials from 12 to 3 while preserving 99.0% of the no-harness final throughput. It also reached the first strong TP8 point one trial earlier than no-harness.

The limitation is that the generic search-high stop guard stopped before local runtime tuning of `max-num-batched-tokens`, which no-harness used to recover a small additional `0.97%`. For this setup, that tradeoff is acceptable if the goal is fast convergence under a fixed measurement ceiling; if the goal is exact final throughput, the next study should raise `search.high` or disable search-high early stop for a local-polish phase.