Gahow Wang
f2ff0faebd
Real gpt-5.4 agentic loop raised per-GPU TP1 0.123 -> TP2 0.2925 -> TP4 1.0012 (8.1x), each a correctly-diagnosed real gain; then a TP4 runtime tweak measured 0.942 < 1.00 and was correctly rejected (no regression). With the 30B run (validator stop + LLM-stop veto), all Stop-B behaviors are now validated end-to-end. The SIGTERM-teardown fix was validated in practice (clean engine teardown, no GPU leak on stop). Efficiency finding: at scale=1.0, infeasible high-theta probes burn the 900s elapsed cap, so a practical loop needs a lower cap; this is why the run was stopped after iter-4 rather than driven to an explicit Stop-B firing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
3.4 KiB
Stop-B end-to-end on dense Qwen3.5-27B (the improving trajectory) — 2026-06-16
Branch feat/two-stop. Real gpt-5.4 agentic loop (codex/prism), Stop-A enabled,
length-aware TTFT SLO (4s + L_in/8k, TPOT ≤ 50 ms), vLLM 0.11.1, H20, GPUs 2–7,
replay_time_scale=1.0, search.high=0.25, inherit_incumbent_floor=true.
Config dash0_qwen27b_stopB_loop.json. Companion to the 30B run
(stop-b-e2e-20260616.md); together they cover all Stop-B behaviors.
Trajectory (incumbent = TP4 @ 1.00 req/s/GPU)
| iter | proposed by | config | per_gpu | adopted? |
|---|---|---|---|---|
| 1 | baseline | TP1 | 0.123 | incumbent |
| 2 | gpt-5.4 | TP2 | 0.2925 (2.4×) | ✅ new incumbent |
| 3 | gpt-5.4 | TP4 | 1.0012 (8.1×) | ✅ new incumbent |
| 4 | gpt-5.4 | TP4 + chunked-prefill + mbt=16384 | 0.942 | ❌ worse → rejected |
| 5 | gpt-5.4 | TP2 + DP2 | (loop stopped before completing) | — |
(Run stopped manually after iter-4 — see "Why stopped" below. Incumbent preserved at TP4.)
What this demonstrates (the piece the 30B run could not)
- A genuine improving climb.
gpt-5.4+ the harness raised per-GPU throughput TP1 → TP2 → TP4 (0.123 → 0.29 → 1.00, 8.1×), each step a correctly-diagnosed real gain: TP1 is TPOT-bound, so the agent scaled tensor-parallelism, then — once topology was won — pivoted to runtime tuning on the winning family (chunked prefill + larger batched tokens). - No regression. The runtime tweak (iter-4) measured below plain TP4 (0.942 < 1.00), and the harness correctly kept TP4 as the incumbent rather than adopting the worse config — the core Stop-B guarantee, shown live.
- Combined with the 30B run (search-high-saturation
validator-authorized stop + premature-LLM-stop veto), every Stop-B behavior is now validated end-to-end: improving climb, correct bottleneck-driven proposals, no regression, deterministic stop authority, and the LLM-stop veto.
Process wins / findings
- SIGTERM teardown fix validated in practice. This loop was stopped with a plain
SIGTERM and the engine + EngineCore workers torn down cleanly — GPUs 2–7 freed, no
orphan, no leaked memory (contrast: the pre-fix runs twice leaked GPU0/1). Commit
b17b213. - Timeout-as-failed-request fix (
2fcaf80) held — no trial crashed on request timeouts this run.
Why stopped (efficiency finding — feeds next round)
The loop was stopped after iter-4 rather than run to an explicit Stop-B firing,
because each TP4-family trial took ~3 h: at scale=1.0, infeasible high-θ probes
each run to the early_stop_max_elapsed_s=900 cap (probe_elapsed_s>900), and
the primary+fallback binary search doubles the probe count. Stop-A truncates a
converged replay but does not shortcut an overloaded probe that simply runs out
the clock. For a practical agentic loop at scale=1.0, lower early_stop_max_elapsed_s
(≈300 s) so overloaded probes die fast; consider also having an infeasible-and-
overloaded probe early-stop on a fast lag/throughput signal rather than the elapsed
cap. The convergence itself was already evident (iter-4's runtime tweak and the
queued TP2+DP2 were not beating TP4).
Next
- Lower the elapsed cap and (optionally) re-run to capture the explicit Stop-B stop on this 27B stack.
- Land the deferred items: more robust C feature for the low-reuse regime; Stop-C cross-day retune trigger; §7 baselines (SCOOT/naive/community).