84 lines
3.8 KiB
Markdown
84 lines
3.8 KiB
Markdown
# qwen27b-chat-pd-colocation
|
|
|
|
qwen3.5-27b `chat` trace, `0~8k` input bucket, internal vLLM (`/usr/local/bin/vllm`), baseline aligned to `~/run_qwen27b.sh`, compared by `request_rate_per_gpu`.
|
|
|
|
## Setup
|
|
|
|
- Hardware: `dash0`, `8x H20`
|
|
- Model: `/home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal`
|
|
- Engine: internal vLLM, PD-colocation baseline from `~/run_qwen27b.sh`
|
|
- Baseline topology: `TP=1, DP=1, EP=1`
|
|
- Trace: `chat_w20260311_1000`
|
|
- Trace source: `trace_windows/traces/chat_w20260311_1000.jsonl`
|
|
- Window duration: `600s` (`10:00-10:10`, `2026-03-11`)
|
|
- Request mode: `chat`
|
|
- Input bucket: `0 <= input_length <= 8192`
|
|
- SLO:
|
|
- pass target: `95%`
|
|
- `TTFT <= 2000ms` for `<=4096` input tokens
|
|
- `TTFT <= 4000ms` for `<=32768` input tokens
|
|
- `TTFT <= 6000ms` for `>32768` input tokens
|
|
- `TPOT <= 50ms`
|
|
- Search:
|
|
- `sampling_u in [0, 0.0625]`
|
|
- `max_probes = 6`
|
|
- `12` trials total
|
|
- Proposal model: `codex / gpt-5.4`
|
|
|
|
## Run assets
|
|
|
|
- Study root: `/home/admin/cpfs/wjh/aituner/aituner/.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology`
|
|
- State: `/home/admin/cpfs/wjh/aituner/aituner/.aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology/state.json`
|
|
- Log: `/home/admin/cpfs/wjh/aituner/aituner/logs/dash0_qwen27b_tight_slo_run9_0_8k_codex_topology.log`
|
|
- Spec: `/home/admin/cpfs/wjh/aituner/aituner/.aituner-tight/specs/dash0_qwen27b_tight_slo_run9_0_8k_codex_topology.json`
|
|
|
|
## Best result
|
|
|
|
- Best trial: `trial-0004`
|
|
- Best config:
|
|
- `tensor-parallel-size=2`
|
|
- `data-parallel-size=1`
|
|
- Best `sampling_u`: `0.013061523438`
|
|
- Best request rate: `0.405 req/s`
|
|
- Best request rate per GPU: `0.2025 req/s/gpu`
|
|
- Best pass rate: `0.9629629629629629`
|
|
|
|
Compared with baseline:
|
|
|
|
- `trial-0001`: `0.035 req/s`, `0.035 req/s/gpu`
|
|
- `trial-0004`: `0.405 req/s`, `0.2025 req/s/gpu`
|
|
- Raw throughput gain: `11.57x`
|
|
- Per-GPU throughput gain: `5.79x`
|
|
|
|
## 12-trial summary
|
|
|
|
| Trial | Proposed config delta | Result |
|
|
| --- | --- | --- |
|
|
| `trial-0001` | baseline `TP1/DP1` | `0.0350 req/s`, `0.0350 req/s/gpu`, feasible |
|
|
| `trial-0002` | `DP=2` | `0.1233 req/s`, `0.0617 req/s/gpu`, feasible |
|
|
| `trial-0003` | `DP=4` | `0.1567 req/s`, `0.0392 req/s/gpu`, feasible |
|
|
| `trial-0004` | `TP=2, DP=1` | `0.4050 req/s`, `0.2025 req/s/gpu`, feasible, best |
|
|
| `trial-0005` | `trial-0004 + max-num-batched-tokens=16384` | infeasible |
|
|
| `trial-0006` | `trial-0004 + max-num-seqs=24` | infeasible |
|
|
| `trial-0007` | `trial-0004 + max-num-batched-tokens=12288` | infeasible |
|
|
| `trial-0008` | `trial-0004 + block-size=32` | infeasible |
|
|
| `trial-0009` | `trial-0004 + gpu-memory-utilization=0.93` | infeasible |
|
|
| `trial-0010` | `trial-0004 + max-num-seqs=16, max-num-batched-tokens=6144` | infeasible |
|
|
| `trial-0011` | `trial-0004 + enable-prefix-caching=false` | infeasible |
|
|
| `trial-0012` | `trial-0004 + block-size=128` | infeasible |
|
|
|
|
## Key insights
|
|
|
|
- The baseline must be the real `~/run_qwen27b.sh` `TP=1` shape. Under that correct baseline, `TP=2, DP=1` is clearly better on both raw throughput and `request_rate_per_gpu`.
|
|
- Pure DP scaling helped from `DP1 -> DP2`, but `DP4` already lost per-GPU efficiency. The main win came from `TP2`, not from adding more replicas.
|
|
- After topology settled at `TP2/DP1`, the remaining bottleneck was `TTFT` tail, not `TPOT`. Later runtime-only trials generally failed around `0.435 req/s` with `pass_rate ~= 0.89`, while `TPOT p95` stayed acceptable and `TTFT p95` stayed near `2.5s~3.0s`.
|
|
- For this `0~8k` chat bucket, the useful topology search space was small but important. Without per-topology `sampling_u` search isolation, this result would have been easy to miss.
|
|
|
|
## Current recommendation
|
|
|
|
Use `trial-0004` as the default serving shape for this workload:
|
|
|
|
- `tensor-parallel-size=2`
|
|
- `data-parallel-size=1`
|
|
- keep the rest of the `run_qwen27b.sh` baseline unchanged
|