# Qwen27B Chat 0-8k Harness Ablation (8-GPU)

Date: 2026-05-13

Supersedes: `qwen27b-chat-0-8k-ttft4s-tpot25-20260510.md` (7-GPU / gpu3skip setup).

## Setup

- Host: `dash0`
- Model: `/home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal`
- Workload: `chat_w20260311_1000`, chat, 0-8k input window
- SLO: TTFT <= 4000ms and TPOT <= 25ms, target pass rate = 0.95
- Trial budget: 12 total tuning iterations per study
- Search: `sampling_u` in `[0, 0.0625]`, tolerance `0.001`, max probes `6`
- Execution: `python3 -m aituner.cli study tune ... --max-trials 12`
- GPU env: `CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7` (8x H20)
- Baseline topology: `TP=1`
- LLM: `gpt-5.4`
- Code: profile-driven harness planner, post GPU-visibility fix (`5c2958e`+)

## Studies

| Variant | Study ID |
| --- | --- |
| no-harness | `dash0-qwen27b-chat-0-8k-ttft4s-tpot25-gpu8-12iter-noharness-minprompt-gpt54-20260513` |
| harness | `dash0-qwen27b-chat-0-8k-ttft4s-tpot25-gpu8-12iter-harness-profileplanner-20260513` |

## Result

Raw per-iteration performance for Fig18-style plot. Metric: `best_request_rate_per_gpu` from that trial's own `result.json`. `NA` means the proposed config did not produce a feasible point. `fail` means engine launch failure.

| Variant | iter1 | iter2 | iter3 | iter4 | iter5 | iter6 | iter7 | iter8 | iter9 | iter10 | iter11 | iter12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness raw `perf[i]` | 0.0650 | fail | fail | 0.0617 | 0.0650 | 0.1233 | 0.1050 | 0.1233 | 0.0650 | 0.0650 | 0.0617 | 0.1233 |
| harness raw `perf[i]` | 0.0650 | 0.1992 | 0.2621 | 0.2056 | 0.1544 | 0.2696 | 0.2621 | 0.2621 | 0.2696 | 0.2621 | 0.2621 | 0.2621 |

| Variant | Best iter | Best request rate | Best request rate / GPU | Best config summary |
| --- | ---: | ---: | ---: | --- |
| no-harness | 6 | 0.1233 | 0.1233 | `enable-prefix-caching=false` |
| harness | 6 | 1.0783 | **0.2696** | `tensor-parallel-size=4`, `max-num-batched-tokens=7680` |

Harness final best is **+118.6%** over no-harness.

## Incumbent Curve

Best-so-far request rate per GPU after each iteration.

| Variant | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| no-harness | 0.0650 | 0.0650 | 0.0650 | 0.0650 | 0.0650 | 0.1233 | 0.1233 | 0.1233 | 0.1233 | 0.1233 | 0.1233 | 0.1233 |
| harness | 0.0650 | 0.1992 | 0.2621 | 0.2621 | 0.2621 | 0.2696 | 0.2696 | 0.2696 | 0.2696 | 0.2696 | 0.2696 | 0.2696 |

## Trial Details

No-harness:

| Iter | Result / GPU | Incumbent / GPU | Status | Config summary |
| ---: | ---: | ---: | --- | --- |
| 1 | 0.0650 | 0.0650 | completed | baseline |
| 2 | - | 0.0650 | launch fail | `gpu-memory-utilization=0.94`, `max-num-batched-tokens=16384` |
| 3 | - | 0.0650 | launch fail | `enable-chunked-prefill=false` |
| 4 | 0.0617 | 0.0650 | completed | `data-parallel-size=2` |
| 5 | 0.0650 | 0.0650 | completed | `block-size=32` |
| 6 | 0.1233 | 0.1233 | completed | `enable-prefix-caching=false` |
| 7 | 0.1050 | 0.1233 | completed | `enable-prefix-caching=false`, `block-size=32` |
| 8 | 0.1233 | 0.1233 | completed | `enable-prefix-caching=false`, `max-num-seqs=32` |
| 9 | 0.0650 | 0.1233 | completed | `enable-prefix-caching=false`, `max-num-batched-tokens=4096` |
| 10 | 0.0650 | 0.1233 | completed | `enable-prefix-caching=false`, `max-num-seqs=16` |
| 11 | 0.0617 | 0.1233 | completed | `data-parallel-size=2`, `enable-prefix-caching=false` |
| 12 | 0.1233 | 0.1233 | completed | `enable-prefix-caching=false` (+ torch compile off) |

Harness:

| Iter | Result / GPU | Incumbent / GPU | Status | Config summary |
| ---: | ---: | ---: | --- | --- |
| 1 | 0.0650 | 0.0650 | completed | baseline |
| 2 | 0.1992 | 0.1992 | completed | `tensor-parallel-size=2` |
| 3 | 0.2621 | 0.2621 | completed | `tensor-parallel-size=4` |
| 4 | 0.2056 | 0.2621 | completed | `tensor-parallel-size=8` |
| 5 | 0.1544 | 0.2621 | completed | `tensor-parallel-size=4`, `data-parallel-size=2` |
| 6 | 0.2696 | 0.2696 | completed | `tensor-parallel-size=4`, `max-num-batched-tokens=7680` |
| 7 | 0.2621 | 0.2696 | completed | `tensor-parallel-size=4`, `enable-chunked-prefill=true`, `max-num-batched-tokens=12288` |
| 8 | 0.2621 | 0.2696 | completed | `tensor-parallel-size=4`, `max-num-batched-tokens=7424` |
| 9 | 0.2696 | 0.2696 | completed | `tensor-parallel-size=4`, `max-num-batched-tokens=7680`, `max-num-seqs=64` |
| 10 | 0.2621 | 0.2696 | completed | `tensor-parallel-size=4`, `max-num-batched-tokens=7680`, `max-num-seqs=56` |
| 11 | 0.2621 | 0.2696 | completed | `tensor-parallel-size=4`, `max-num-batched-tokens=7680`, `max-num-seqs=60` |
| 12 | 0.2621 | 0.2696 | completed | `tensor-parallel-size=4`, `max-num-batched-tokens=7680`, `max-num-seqs=63` |

## Interpretation

No-harness never tested any TP change in 12 trials. It started from TP=1, encountered two early launch failures, then spent all remaining budget on runtime knobs (`enable-prefix-caching`, `block-size`, `max-num-seqs`, `max-num-batched-tokens`). Its best discovery was disabling prefix caching at iter 6, reaching only `0.1233 req/s/GPU`.

Harness systematically explored the TP frontier: iter 2 tested TP=2, iter 3 tested TP=4, iter 4 tested TP=8. The profile-driven planner identified `ttft_prefill` as the ranked bottleneck and proposed increasing TP as the primary relief action. After TP=4 proved best per-GPU, the harness tested TP=4/DP=2 (worse) then shifted to runtime refinement within the TP=4 family, settling on `max-num-batched-tokens=7680` as the marginal improvement.

The result demonstrates that topology exploration is critical for this workload: the no-harness LLM failed to discover TP>1 configurations entirely, while the harness reached the optimal TP=4 topology by iter 3 and refined it by iter 6.

## Comparison with Previous 7-GPU Run

The 7-GPU (`gpu3skip`) run from 2026-05-10 used `CUDA_VISIBLE_DEVICES=0,1,2,4,5,6,7` and is not directly comparable. The harness result on 7-GPU was `0.2742 req/s/GPU` (TP=4, chunked-prefill, MBT=16384). On 8-GPU, the harness found a similar TP=4 optimum at `0.2696 req/s/GPU` with slightly different runtime tuning. The core finding is consistent: harness accelerates topology discovery and significantly outperforms no-harness.