gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
f653af09a871ae7993eed2696d1f3528536cb342
aituner/docs/qwen27b-chat-0-8k-current-config-fig18-20260506.md

7.1 KiB
Raw Blame History

qwen27b-chat-0-8k Current-Config Fig18 Plan

Question

The earlier tables used best-so-far throughput. That is useful for deciding the best deployable incumbent, but it hides bad proposals because the curve is monotonic by construction. To judge whether the harness makes tuning more directional, the primary curve must be each iteration's measured current config performance.

Why Final Performance Can Be Close

Harness and no-harness can converge to similar final throughput when the search space contains one dominant simple family. In this setup the dominant family is TP=2, DP=1 over the run_qwen27b.sh baseline. The no-harness LLM can still eventually discover that family within 12 iterations, so final best performance can be close.

The difference the harness is expected to improve is not necessarily the final 12-iter maximum. It should improve:

  • iterations-to-first-good-config;
  • number of worse or infeasible proposals after an incumbent is found;
  • measured-current config oscillation;
  • early-stop behavior once adjacent harness probes no longer justify more GPU trials.

Metrics

  • measured-current: each trial's own feasible request_rate_per_gpu. Failed or no-feasible-point trials are recorded as NA.
  • accepted-incumbent: best deployable value after each trial. This is the standard best-so-far curve and is monotonic by definition.
  • iters-to-best: first iteration where the final best value or equivalent config family appears.
  • wasted-trials-after-best: trials after first best that are worse, infeasible, or no-feasible-point.

Historical Run9 Re-Read

Source: .aituner-tight/dash0-qwen27b-tight-slo-10min-run9-chat-0-8k-codex-topology on dash0.

Variant Curve Iter 1 Iter 2 Iter 3 Iter 4 Iter 5 Iter 6 Iter 7 Iter 8 Iter 9 Iter 10 Iter 11 Iter 12
no-harness run9 measured-current 0.0350 0.0617 0.0392 0.2025 NA NA NA NA NA NA NA NA
no-harness run9 accepted-incumbent 0.0350 0.0617 0.0617 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025

Interpretation: the no-harness current-config curve already has a regression at iter 3 and then many no-feasible-point runtime probes. The monotonic curve only shows the incumbent policy, not proposal quality.

New Paired Test Plan

Run on dash0 with internal vLLM and the real chat_w20260311_1000 0-8k replay:

  • Base spec: configs/examples/dash0_qwen27b_tight_slo_run4_0_8k.json.
  • Model path: /home/admin/resource/model/464482ce/qwen3.5-27b/256k-0223-internal.
  • Engine: /usr/local/bin/vllm, baseline aligned with ~/run_qwen27b.sh.
  • SLO: 95% pass, stepped TTFT 2s/4s/6s, TPOT <=50ms.
  • Search: low=0, high=0.0625, max_probes=6, tolerance=0.001.
  • no-harness study: .aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-noharness.
  • harness study: .aituner-tight/dash0-qwen27b-tight-slo-10min-run10-chat-0-8k-current-harness.

The result table will report both curves. The harness is considered successful only if it reaches the same or better incumbent in fewer iterations and reduces the measured-current regressions or replaces them with an explicit harness stop.

Run Status

  • 2026-05-06 07:05 CST: dash0 checked, 8 H20 GPUs idle.
  • 2026-05-06 07:05 CST: generated paired specs under .aituner-tight/specs/.
  • 2026-05-06 07:05 CST: started no-harness full 12-iter run in tmux session qwen27b_run10_noharness_20260506.
  • 2026-05-06 07:18 CST: stopped the duplicate fresh no-harness run before completion. Reason: run9 is already a completed real 12-iter no-harness run for the same internal vLLM 0-8k setup, while the fresh full-chat run would spend a multi-hour dash0 slot duplicating that curve.
  • 2026-05-06 07:20 CST: seeded the harness study with the real run9 baseline measurement as trial-0001, then started the harness run with --skip-baseline in tmux session qwen27b_run10_harness_skipbase_20260506.
  • 2026-05-06 07:20 CST: harness generated deterministic trial-0002: {"tensor-parallel-size": 2}.
  • 2026-05-06 08:11 CST: harness trial-0002 completed: TP=2, 0.2142 request_rate_per_gpu.
  • 2026-05-06 08:19 CST: harness trial-0003 failed at engine launch. Root cause: the old runtime refinement coupled gpu-memory-utilization=0.95 with larger max-num-batched-tokens, causing speculative sampler warmup OOM. This is a generic harness safety bug; fixed locally by removing the automatic memory-utilization bump from runtime refinement.
  • 2026-05-06 09:24 CST: harness trial-0004 completed: TP=4, 0.4429 request_rate_per_gpu. All six probes were feasible up to sampling_u=0.0615234375, so this study is near the configured search.high=0.0625 ceiling.
  • 2026-05-06 09:25 CST: old harness repeated the same unsafe runtime refinement for TP4 and trial-0005 failed at engine launch for the same OOM reason. The old process was stopped before continuing.

Current Results

Unit: feasible request_rate_per_gpu. NA means the current trial did not produce a feasible deployable config.

Variant Curve Iter 1 Iter 2 Iter 3 Iter 4 Iter 5 Iter 6 Iter 7 Iter 8 Iter 9 Iter 10 Iter 11 Iter 12
no-harness run9 measured-current 0.0350 0.0617 0.0392 0.2025 NA NA NA NA NA NA NA NA
no-harness run9 accepted-incumbent 0.0350 0.0617 0.0617 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025 0.2025
harness run10 measured-current 0.0350 0.2142 NA 0.4429 NA pending pending pending pending pending pending pending
harness run10 accepted-incumbent 0.0350 0.2142 0.2142 0.4429 0.4429 pending pending pending pending pending pending pending

The harness result is stronger than the earlier strict replay. It did not merely reach the same TP2 region earlier; it then used the bottleneck/topology evidence to validate TP4 and found a much higher current config.

Interpretation

  • Why both variants can look close when only best-so-far is shown: no-harness can eventually find a good simple topology, and best-so-far hides every bad proposal after that point.
  • What the current-config curve shows: no-harness regresses at iter 3 and then spends many iterations on no-feasible-point runtime probes. Harness reaches a stronger TP2 config at iter 2 and a stronger TP4 config at iter 4.
  • Why harness helped: the baseline diagnostics identify TTFT/prefill as the active bottleneck on low-prefix-reuse long prompts. The harness maps that to adjacent TP validation before DP/runtime exploration. The no-harness LLM chose DP2 then DP4 first, which diluted per-GPU throughput and delayed TP.
  • Remaining defect found during the run: runtime refinement was too aggressive because it combined larger MBT with higher memory utilization. This has been fixed so future runtime validation changes batching headroom without also raising memory pressure.