gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
48911b658bbf052d70d952d1cdf55ad6b50ba7a5
aituner/docs/harness-ablation/qwen235b-thinking-prefill-ttft-3s6s9s-20260514.md
Gahow Wang 984eb1f325 Document 8-GPU harness ablation results for qwen27b and qwen235b prefill 
Add completed experiment results from dash0 runs after 2026-05-13:
- qwen27b chat 0-8k: harness +118.6% over no-harness (0.2696 vs 0.1233 req/s/GPU)
- qwen235b prefill TTFT 3s/6s/9s: harness +76.8% (0.3921 vs 0.2217 req/s/GPU)

Mark old 7-GPU and pre-5/13 docs as superseded. Update implementation
log with completed run status.
2026-05-16 21:23:16 +08:00

118 lines
6.7 KiB
Markdown
Raw Blame History

# qwen235b Thinking Prefill Harness Ablation (TTFT 3s/6s/9s)
Date: 2026-05-14 / 2026-05-15
Supersedes: `qwen235b-thinking-prefill-ttft-20260510.md` (different SLO thresholds).
## Setup
- Host: `dash0`
- Engine: internal vLLM at `/usr/local/bin/vllm`
- Model: `/home/admin/resource/model/464482ce.qwen3-235b-a22b/256k-0717`
- Trace window: `thinking_w20260327_1000`
- Request mode: chat, with `completion_tokens_override=1` for prefill-only measurement
- SLO: TTFT-only stepped p95 pass target, target pass rate `0.95`
- input tokens `<=4096`: `3000 ms`
- input tokens `<=32768`: `6000 ms`
- otherwise: `9000 ms`
- GPU env: `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7` (8x H20)
- Baseline topology: `TP=4`
- LLM: `gpt-5.4`
- Code: profile-driven harness planner, post GPU-visibility fix (`5c2958e`+)
## Studies
| Variant | Study ID | search.high |
| --- | --- | ---: |
| no-harness | `dash0-qwen235b-prefill-thinking-ttft-3s6s9s-12iter-noharness-minprompt-gpt54-20260514` | 0.125 |
| harness | `dash0-qwen235b-prefill-thinking-ttft-3s6s9s-12iter-harness-profileplanner-gpt54-20260514` | 0.125 |
| harness (high=0.25) | `dash0-qwen235b-prefill-thinking-ttft-3s6s9s-high025-12iter-harness-profileplanner-gpt54-20260515` | 0.25 |
The `harness (high=0.25)` run was added to test whether raising `search.high` lets the harness find a better runtime config after reaching the search ceiling at `0.125`.
## Result
Raw per-iteration performance for Fig18-style plot. Metric: `best_request_rate_per_gpu`. `NA` means the proposed config did not produce a feasible point. `fail` means engine launch failure. `stop` means harness stopped before launching another trial.
| Variant | iter1 | iter2 | iter3 | iter4 | iter5 | iter6 | iter7 | iter8 | iter9 | iter10 | iter11 | iter12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness raw `perf[i]` | 0.1804 | fail | 0.1892 | fail | 0.1892 | 0.1804 | 0.2217 | 0.2029 | 0.2029 | 0.2029 | 0.1892 | 0.1804 |
| harness raw `perf[i]` | 0.2029 | 0.3863 | stop | stop | stop | stop | stop | stop | stop | stop | stop | stop |
| harness (high=0.25) raw `perf[i]` | 0.2029 | 0.3921 | 0.3442 | 0.3921 | 0.3821 | 0.3821 | 0.3821 | 0.3688 | 0.3821 | 0.3821 | 0.3821 | 0.3821 |
| Variant | GPU trials | Best iter | Best req/s | Best req/s/GPU | Best config summary |
| --- | ---: | ---: | ---: | ---: | --- |
| no-harness | 12 | 7 | 0.8867 | 0.2217 | TP=4, MNS=112, MBT=7168 |
| harness | 2 (stop) | 2 | 3.0900 | 0.3863 | TP=8 |
| harness (high=0.25) | 12 | 2 | 3.1367 | **0.3921** | TP=8 |
Harness reached **+74.2%** over no-harness at iter 2. With `search.high=0.25`, the harness found `0.3921 req/s/GPU` (+76.8%).
## Incumbent Curve
Best-so-far request rate per GPU after each iteration.
| Variant | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness | 0.1804 | 0.1804 | 0.1892 | 0.1892 | 0.1892 | 0.1892 | 0.2217 | 0.2217 | 0.2217 | 0.2217 | 0.2217 | 0.2217 |
| harness | 0.2029 | 0.3863 | stop | stop | stop | stop | stop | stop | stop | stop | stop | stop |
| harness (high=0.25) | 0.2029 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 | 0.3921 |
## Trial Details
No-harness:
| Iter | Result / GPU | Incumbent / GPU | Status | Config summary |
| ---: | ---: | ---: | --- | --- |
| 1 | 0.1804 | 0.1804 | completed | baseline (TP=4) |
| 2 | - | 0.1804 | launch fail | TP=4, EP=4, MNS=128 |
| 3 | 0.1892 | 0.1892 | completed | MNS=96 |
| 4 | - | 0.1892 | launch fail | TP=4, DP=2, EP off, MNS=96 |
| 5 | 0.1892 | 0.1892 | completed | MNS=112 |
| 6 | 0.1804 | 0.1892 | completed | MNS=112, MBT=9216 |
| 7 | 0.2217 | 0.2217 | completed | MNS=112, MBT=7168 |
| 8 | 0.2029 | 0.2217 | completed | MNS=112, MBT=6144 |
| 9 | 0.2029 | 0.2217 | completed | MNS=120, MBT=7168 |
| 10 | 0.2029 | 0.2217 | completed | TP=4, DP=1, EP off, MNS=108, MBT=7168 |
| 11 | 0.1892 | 0.2217 | completed | MNS=112, MBT=7680 |
| 12 | 0.1804 | 0.2217 | completed | MNS=112, MBT=6912 |
Harness (`search.high=0.125`):
| Iter | Result / GPU | Incumbent / GPU | Status | Config summary |
| ---: | ---: | ---: | --- | --- |
| 1 | 0.2029 | 0.2029 | completed | baseline (TP=4) |
| 2 | 0.3863 | 0.3863 | completed | TP=8 |
| 3 | - | - | harness stop | search-high saturation (`sampling_u=0.123` vs `search.high=0.125`) |
Harness (`search.high=0.25`):
| Iter | Result / GPU | Incumbent / GPU | Status | Config summary |
| ---: | ---: | ---: | --- | --- |
| 1 | 0.2029 | 0.2029 | completed | baseline (TP=4) |
| 2 | 0.3921 | 0.3921 | completed | TP=8 |
| 3 | 0.3442 | 0.3921 | completed | TP=8, chunked-prefill, MBT=32768 |
| 4 | 0.3921 | 0.3921 | completed | TP=8, MBT=12288 |
| 5 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=16384 |
| 6 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=14336 |
| 7 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=10240 |
| 8 | 0.3688 | 0.3921 | completed | TP=8, EP off, MBT=11776 |
| 9 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=13312 |
| 10 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=7168 |
| 11 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=12032 |
| 12 | 0.3821 | 0.3921 | completed | TP=8, EP off, MBT=12800 |
## Interpretation
No-harness never attempted TP=8. It stayed on the TP=4 baseline, encountered two launch failures (EP=4 and DP=2), and spent all remaining trials on runtime knob tuning within the TP=4 family. Its best finding was `MNS=112, MBT=7168` at iter 7 (`0.2217 req/s/GPU`).
Harness identified `ttft_prefill` as the dominant bottleneck from the baseline trial and immediately proposed TP=8 as the first topology move. This is the correct direction for a prefill-only workload with heavy-tail prompts (p95 ~19.7k tokens, p99 ~30k tokens).
With `search.high=0.125`, the harness stopped at iter 2 because the incumbent's best feasible `sampling_u=0.123` was within one search resolution of `search.high`. With `search.high=0.25`, the harness continued for 12 trials but the best remained iter 2 (`TP=8, default MBT`). The additional 10 trials explored MBT variations on TP=8 but none improved per-GPU throughput. This confirms the 2-trial harness result was already at or near the local optimum.
The gap between harness and no-harness (`+76.8%`) comes entirely from topology: TP=8 doubles the per-GPU prefill compute bandwidth compared to TP=4, which directly reduces TTFT and allows higher admitted request rates under the stepped TTFT SLO.
## Comparison with Previous Run (2026-05-10)
The 2026-05-10 run used different SLO thresholds and is documented in `qwen235b-thinking-prefill-ttft-20260510.md`. The core finding is consistent: harness finds TP=8 at iter 2-3 while no-harness gets stuck on TP=4 runtime tuning.
Reference in New Issue View Git Blame Copy Permalink