qwen27b-chat-pd-colocation

qwen3.5-27b chat trace, 0~8k input bucket, internal vLLM (/usr/local/bin/vllm), baseline aligned to ~/run_qwen27b.sh, compared by request_rate_per_gpu.

Setup

Hardware: dash0, 8x H20
Model: /home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal
Engine: internal vLLM, PD-colocation baseline from ~/run_qwen27b.sh
Baseline topology: TP=1, DP=1, EP=1
Trace: chat_w20260311_1000
Trace source: trace_windows/traces/chat_w20260311_1000.jsonl
Window duration: 600s (10:00-10:10, 2026-03-11)
Request mode: chat
Input bucket: 0 <= input_length <= 8192
SLO:
- pass target: 95%
- TTFT <= 2000ms for <=4096 input tokens
- TTFT <= 4000ms for <=32768 input tokens
- TTFT <= 6000ms for >32768 input tokens
- TPOT <= 50ms
Search:
- sampling_u in [0, 0.0625]
- max_probes = 6
- 12 trials total
Proposal model: codex / gpt-5.4

Run assets

Study root: /home/admin/cpfs/wjh/aituner/aituner/.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology
State: /home/admin/cpfs/wjh/aituner/aituner/.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology/state.json
Log: /home/admin/cpfs/wjh/aituner/aituner/logs/dash0_qwen27b_tight_slo_run9_0_8k_codex_topology.log
Spec: /home/admin/cpfs/wjh/aituner/aituner/.aituner-tight/specs/dash0_qwen27b_tight_slo_run9_0_8k_codex_topology.json

Best result

Best trial: trial-0004
Best config:
- tensor-parallel-size=2
- data-parallel-size=1
Best sampling_u: 0.013061523438
Best request rate: 0.405 req/s
Best request rate per GPU: 0.2025 req/s/gpu
Best pass rate: 0.9629629629629629

Compared with baseline:

trial-0001: 0.035 req/s, 0.035 req/s/gpu
trial-0004: 0.405 req/s, 0.2025 req/s/gpu
Raw throughput gain: 11.57x
Per-GPU throughput gain: 5.79x

12-trial summary

Trial	Proposed config delta	Result
`trial-0001`	baseline `TP1/DP1`	`0.0350 req/s`, `0.0350 req/s/gpu`, feasible
`trial-0002`	`DP=2`	`0.1233 req/s`, `0.0617 req/s/gpu`, feasible
`trial-0003`	`DP=4`	`0.1567 req/s`, `0.0392 req/s/gpu`, feasible
`trial-0004`	`TP=2, DP=1`	`0.4050 req/s`, `0.2025 req/s/gpu`, feasible, best
`trial-0005`	`trial-0004 + max-num-batched-tokens=16384`	infeasible
`trial-0006`	`trial-0004 + max-num-seqs=24`	infeasible
`trial-0007`	`trial-0004 + max-num-batched-tokens=12288`	infeasible
`trial-0008`	`trial-0004 + block-size=32`	infeasible
`trial-0009`	`trial-0004 + gpu-memory-utilization=0.93`	infeasible
`trial-0010`	`trial-0004 + max-num-seqs=16, max-num-batched-tokens=6144`	infeasible
`trial-0011`	`trial-0004 + enable-prefix-caching=false`	infeasible
`trial-0012`	`trial-0004 + block-size=128`	infeasible

Key insights

The baseline must be the real ~/run_qwen27b.sh TP=1 shape. Under that correct baseline, TP=2, DP=1 is clearly better on both raw throughput and request_rate_per_gpu.
Pure DP scaling helped from DP1 -> DP2, but DP4 already lost per-GPU efficiency. The main win came from TP2, not from adding more replicas.
After topology settled at TP2/DP1, the remaining bottleneck was TTFT tail, not TPOT. Later runtime-only trials generally failed around 0.435 req/s with pass_rate ~= 0.89, while TPOT p95 stayed acceptable and TTFT p95 stayed near 2.5s~3.0s.
For this 0~8k chat bucket, the useful topology search space was small but important. Without per-topology sampling_u search isolation, this result would have been easy to miss.

Current recommendation

Use trial-0004 as the default serving shape for this workload:

tensor-parallel-size=2
data-parallel-size=1
keep the rest of the run_qwen27b.sh baseline unchanged