gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
98cd6dd81aa3bada09310148c791bda164e0d0d4
aituner/docs/qwen27b-chat-0-8k-current-config-fig18-20260506.md

9.3 KiB
Raw Blame History

qwen27b-chat-0-8k Current-Config Fig18 Plan

Question

The earlier tables used best-so-far throughput. That is useful for deciding the best deployable incumbent, but it hides bad proposals because the curve is monotonic by construction. To judge whether the harness makes tuning more directional, the primary curve must be each iteration's measured current config performance.

Why Final Performance Can Be Close

Harness and no-harness can converge to similar final throughput when the search space contains one dominant simple family. In this setup the dominant family is TP=2, DP=1 over the run_qwen27b.sh baseline. The no-harness LLM can still eventually discover that family within 12 iterations, so final best performance can be close.

The difference the harness is expected to improve is not necessarily the final 12-iter maximum. It should improve:

  • iterations-to-first-good-config;
  • number of worse or infeasible proposals after an incumbent is found;
  • measured-current config oscillation;
  • early-stop behavior once adjacent harness probes no longer justify more GPU trials.

Metrics

  • measured-current: each trial's own feasible request_rate_per_gpu. Failed or no-feasible-point trials are recorded as NA.
  • accepted-incumbent: best deployable value after each trial. This is the standard best-so-far curve and is monotonic by definition.
  • iters-to-best: first iteration where the final best value or equivalent config family appears.
  • wasted-trials-after-best: trials after first best that are worse, infeasible, or no-feasible-point.

Historical Run9 Re-Read

Source: .aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology on dash0.

Variant Curve Iter 1 Iter 2 Iter 3 Iter 4 Iter 5 Iter 6 Iter 7 Iter 8 Iter 9 Iter 10 Iter 11 Iter 12
no-harness run9 measured-current 0.0350 0.0617 0.0392 0.2025 NA NA NA NA NA NA NA NA
no-harness run9 accepted-incumbent 0.0350 0.0617 0.0617 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025

Interpretation: the no-harness current-config curve already has a regression at iter 3 and then many no-feasible-point runtime probes. The monotonic curve only shows the incumbent policy, not proposal quality.

New Paired Test Plan

Run on dash0 with internal vLLM and the real chat_w20260311_1000 0-8k replay:

  • Base spec: configs/examples/dash0_qwen27b_tight_slo_run4_0_8k.json.
  • Model path: /home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal.
  • Naming note: local configs and dash0 model directories expose this setup as Qwen3.5-27B/Qwen35-27B, not qwen32b.
  • Engine: /usr/local/bin/vllm, baseline aligned with ~/run_qwen27b.sh.
  • SLO: 95% pass, stepped TTFT 2s/4s/6s, TPOT <=50ms.
  • Search: low=0, high=0.0625, max_probes=6, tolerance=0.001.
  • no-harness study: .aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-noharness.
  • harness study: .aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-harness.

The result table will report both curves. The harness is considered successful only if it reaches the same or better incumbent in fewer iterations and reduces the measured-current regressions or replaces them with an explicit harness stop.

Run Status

  • 2026-05-06 07:05 CST: dash0 checked, 8 H20 GPUs idle.
  • 2026-05-06 07:05 CST: generated paired specs under .aituner-tight/specs/.
  • 2026-05-06 07:05 CST: started no-harness full 12-iter run in tmux session qwen27b_run10_noharness_20260506.
  • 2026-05-06 07:18 CST: stopped the duplicate fresh no-harness run before completion. Reason: run9 is already a completed real 12-iter no-harness run for the same internal vLLM 0-8k setup, while the fresh full-chat run would spend a multi-hour dash0 slot duplicating that curve.
  • 2026-05-06 07:20 CST: seeded the harness study with the real run9 baseline measurement as trial-0001, then started the harness run with --skip-baseline in tmux session qwen27b_run10_harness_skipbase_20260506.
  • 2026-05-06 07:20 CST: harness generated deterministic trial-0002: {"tensor-parallel-size": 2}.
  • 2026-05-06 08:11 CST: harness trial-0002 completed: TP=2, 0.2142 request_rate_per_gpu.
  • 2026-05-06 08:19 CST: harness trial-0003 failed at engine launch. Root cause: the old runtime refinement coupled gpu-memory-utilization=0.95 with larger max-num-batched-tokens, causing speculative sampler warmup OOM. This is a generic harness safety bug; fixed locally by removing the automatic memory-utilization bump from runtime refinement.
  • 2026-05-06 09:24 CST: harness trial-0004 completed: TP=4, 0.4429 request_rate_per_gpu. All six probes were feasible up to sampling_u=0.0615234375, so this study is near the configured search.high=0.0625 ceiling.
  • 2026-05-06 09:25 CST: old harness repeated the same unsafe runtime refinement for TP4 and trial-0005 failed at engine launch for the same OOM reason. The old process was stopped before continuing.
  • 2026-05-06 09:37 CST: pulled commit 5d96689 on dash0 and resumed. The runtime-refinement OOM was fixed, but the stop guard was still too strict: it did not treat a feasible high-edge probe with a small number of SLO failures as saturation, even though the probe already met the 95% pass-rate target.
  • 2026-05-06 09:50 CST: stopped the unnecessary product-8 validation. The queued trial-0006/trial-0007 are not used for convergence claims.
  • 2026-05-06 09:56 CST: pulled commit f653af0 on dash0. The fixed high-edge stop guard produced harness-stop-0008 without launching another GPU trial.

Current Results

Unit: feasible request_rate_per_gpu. NA means the current trial did not produce a feasible deployable config.

Variant Curve Iter 1 Iter 2 Iter 3 Iter 4 Iter 5 Iter 6 Iter 7 Iter 8 Iter 9 Iter 10 Iter 11 Iter 12
no-harness run9 measured-current 0.0350 0.0617 0.0392 0.2025 NA NA NA NA NA NA NA NA
no-harness run9 accepted-incumbent 0.0350 0.0617 0.0617 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025
harness run10 measured-current 0.0350 0.2142 NA 0.4429 NA skipped skipped stop
harness run10 accepted-incumbent 0.0350 0.2142 0.2142 0.4429 0.4429 0.4429 0.4429 0.4429 stop

The harness result is stronger than the earlier strict replay. It did not merely reach the same TP2 region earlier; it then used the bottleneck/topology evidence to validate TP4 and found a much higher current config.

Interpretation

  • Why both variants can look close when only best-so-far is shown: no-harness can eventually find a good simple topology, and best-so-far hides every bad proposal after that point.
  • What the current-config curve shows: no-harness regresses at iter 3 and then spends many iterations on no-feasible-point runtime probes. Harness reaches a stronger TP2 config at iter 2 and a stronger TP4 config at iter 4.
  • Why harness helped: the baseline diagnostics identify TTFT/prefill as the active bottleneck on low-prefix-reuse long prompts. The harness maps that to adjacent TP validation before DP/runtime exploration. The no-harness LLM chose DP2 then DP4 first, which diluted per-GPU throughput and delayed TP.
  • Defect fixed during the run: runtime refinement was too aggressive because it combined larger MBT with higher memory utilization. It now changes batching headroom without also raising memory pressure.
  • Stop defect fixed during the run: high-edge probes can have a few individual latency failures and still be feasible under the configured pass-rate SLO. The stop guard now keys on feasible=true near search.high, not on an empty failed-reason map.
  • Search-high implication: TP4 reached sampling_u=0.0615234375 with search.high=0.0625, so the current spec is saturated for this topology. A higher search.high would be required to distinguish whether TP4 can go even higher in absolute throughput; it is not needed to show that harness converged faster than no-harness under this spec.

Mechanism

The harness contributes structured, non-testcase-specific information:

  • Workload features: long-prompt 0-8k distribution, low prefix reuse, and smooth arrivals.
  • Bottleneck diagnosis from probes: baseline failures are TTFT/prefill-heavy, so topology changes that reduce long-prefill latency should be tried before DP or runtime batching.
  • Topology adjacency: validate TP1 -> TP2 -> TP4 rather than jumping randomly or repeating a failing runtime family.
  • Stop condition: once the incumbent's feasible probe is within one binary-search resolution of search.high, stop instead of spending more GPU trials.

Without the harness, the LLM response in run9 chose DP2 and DP4 before TP2. That temporarily improved total request rate but reduced per-GPU efficiency, so the measured-current curve dipped at iter 3 and reached the old best only at iter 4. With the harness, the LLM receives the bottleneck/topology frame and chooses TP-oriented validation; TP2 is reached at iter 2 and TP4 at iter 4.