aituner/docs/harness-ablation/qwen235b-thinking-prefill-ttft-3s6s9s-20260514.md

qwen235b Thinking Prefill Harness Ablation (TTFT 3s/6s/9s)

Date: 2026-05-14 / 2026-05-15

Supersedes: qwen235b-thinking-prefill-ttft-20260510.md (different SLO thresholds).

Setup

• Host: dash0
• Engine: internal vLLM at /usr/local/bin/vllm
• Model: /home/admin/resource/model/464482ce.qwen3-235b-a22b/256k-0717
• Trace window: thinking_w20260327_1000
• Request mode: chat, with completion_tokens_override=1 for prefill-only measurement
• SLO: TTFT-only stepped p95 pass target, target pass rate 0.95
  • input tokens <=4096: 3000 ms
  • input tokens <=32768: 6000 ms
  • otherwise: 9000 ms
• GPU env: CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 (8x H20)
• Baseline topology: TP=4
• LLM: gpt-5.4
• Code: profile-driven harness planner, post GPU-visibility fix (5c2958e+)

Studies

Variant	Study ID	search.high
no-harness	dash0-qwen235b-prefill-thinking-ttft-3s6s9s-12iter-noharness-minprompt-gpt54-20260514	0.125
harness	dash0-qwen235b-prefill-thinking-ttft-3s6s9s-12iter-harness-profileplanner-gpt54-20260514	0.125
harness (high=0.25)	dash0-qwen235b-prefill-thinking-ttft-3s6s9s-high025-12iter-harness-profileplanner-gpt54-20260515	0.25

The harness (high=0.25) run was added to test whether raising search.high lets the harness find a better runtime config after reaching the search ceiling at 0.125.

Result

Raw per-iteration performance for Fig18-style plot. Metric: best_request_rate_per_gpu. NA means the proposed config did not produce a feasible point. fail means engine launch failure. stop means harness stopped before launching another trial.

Variant	iter1	iter2	iter3	iter4	iter5	iter6	iter7	iter8	iter9	iter10	iter11	iter12
no-harness raw perf[i]	0.1804	fail	0.1892	fail	0.1892	0.1804	0.2217	0.2029	0.2029	0.2029	0.1892	0.1804
harness raw perf[i]	0.2029	0.3863	stop	stop	stop	stop	stop	stop	stop	stop	stop	stop
harness (high=0.25) raw perf[i]	0.2029	0.3921	0.3442	0.3921	0.3821	0.3821	0.3821	0.3688	0.3821	0.3821	0.3821	0.3821

Variant	GPU trials	Best iter	Best req/s	Best req/s/GPU	Best config summary
no-harness	12	7	0.8867	0.2217	TP=4, MNS=112, MBT=7168
harness	2 (stop)	2	3.0900	0.3863	TP=8
harness (high=0.25)	12	2	3.1367	0.3921	TP=8

Harness reached +74.2% over no-harness at iter 2. With search.high=0.25, the harness found 0.3921 req/s/GPU (+76.8%).

Incumbent Curve

Best-so-far request rate per GPU after each iteration.

Variant	1	2	3	4	5	6	7	8	9	10	11	12
no-harness	0.1804	0.1804	0.1892	0.1892	0.1892	0.1892	0.2217	0.2217	0.2217	0.2217	0.2217	0.2217
harness	0.2029	0.3863	stop	stop	stop	stop	stop	stop	stop	stop	stop	stop
harness (high=0.25)	0.2029	0.3921	0.3921	0.3921	0.3921	0.3921	0.3921	0.3921	0.3921	0.3921	0.3921	0.3921

Trial Details

No-harness:

Iter	Result / GPU	Incumbent / GPU	Status	Config summary
1	0.1804	0.1804	completed	baseline (TP=4)
2	-	0.1804	launch fail	TP=4, EP=4, MNS=128
3	0.1892	0.1892	completed	MNS=96
4	-	0.1892	launch fail	TP=4, DP=2, EP off, MNS=96
5	0.1892	0.1892	completed	MNS=112
6	0.1804	0.1892	completed	MNS=112, MBT=9216
7	0.2217	0.2217	completed	MNS=112, MBT=7168
8	0.2029	0.2217	completed	MNS=112, MBT=6144
9	0.2029	0.2217	completed	MNS=120, MBT=7168
10	0.2029	0.2217	completed	TP=4, DP=1, EP off, MNS=108, MBT=7168
11	0.1892	0.2217	completed	MNS=112, MBT=7680
12	0.1804	0.2217	completed	MNS=112, MBT=6912

Harness (search.high=0.125):

Iter	Result / GPU	Incumbent / GPU	Status	Config summary
1	0.2029	0.2029	completed	baseline (TP=4)
2	0.3863	0.3863	completed	TP=8
3	-	-	harness stop	search-high saturation (sampling_u=0.123 vs search.high=0.125)

Harness (search.high=0.25):

Iter	Result / GPU	Incumbent / GPU	Status	Config summary
1	0.2029	0.2029	completed	baseline (TP=4)
2	0.3921	0.3921	completed	TP=8
3	0.3442	0.3921	completed	TP=8, chunked-prefill, MBT=32768
4	0.3921	0.3921	completed	TP=8, MBT=12288
5	0.3821	0.3921	completed	TP=8, EP off, MBT=16384
6	0.3821	0.3921	completed	TP=8, EP off, MBT=14336
7	0.3821	0.3921	completed	TP=8, EP off, MBT=10240
8	0.3688	0.3921	completed	TP=8, EP off, MBT=11776
9	0.3821	0.3921	completed	TP=8, EP off, MBT=13312
10	0.3821	0.3921	completed	TP=8, EP off, MBT=7168
11	0.3821	0.3921	completed	TP=8, EP off, MBT=12032
12	0.3821	0.3921	completed	TP=8, EP off, MBT=12800

Interpretation

No-harness never attempted TP=8. It stayed on the TP=4 baseline, encountered two launch failures (EP=4 and DP=2), and spent all remaining trials on runtime knob tuning within the TP=4 family. Its best finding was MNS=112, MBT=7168 at iter 7 (0.2217 req/s/GPU).

Harness identified ttft_prefill as the dominant bottleneck from the baseline trial and immediately proposed TP=8 as the first topology move. This is the correct direction for a prefill-only workload with heavy-tail prompts (p95 ~19.7k tokens, p99 ~30k tokens).

With search.high=0.125, the harness stopped at iter 2 because the incumbent's best feasible sampling_u=0.123 was within one search resolution of search.high. With search.high=0.25, the harness continued for 12 trials but the best remained iter 2 (TP=8, default MBT). The additional 10 trials explored MBT variations on TP=8 but none improved per-GPU throughput. This confirms the 2-trial harness result was already at or near the local optimum.

The gap between harness and no-harness (+76.8%) comes entirely from topology: TP=8 doubles the per-GPU prefill compute bandwidth compared to TP=4, which directly reduces TTFT and allows higher admitted request rates under the stepped TTFT SLO.

Comparison with Previous Run (2026-05-10)

The 2026-05-10 run used different SLO thresholds and is documented in qwen235b-thinking-prefill-ttft-20260510.md. The core finding is consistent: harness finds TP=8 at iter 2-3 while no-harness gets stuck on TP=4 runtime tuning.