Gahow Wang
f2ff0faebd
Real gpt-5.4 agentic loop raised per-GPU TP1 0.123 -> TP2 0.2925 -> TP4 1.0012 (8.1x), each a correctly-diagnosed real gain; then a TP4 runtime tweak measured 0.942 < 1.00 and was correctly rejected (no regression). With the 30B run (validator stop + LLM-stop veto), all Stop-B behaviors are now validated end-to-end. The SIGTERM-teardown fix was validated in practice (clean engine teardown, no GPU leak on stop). Efficiency finding: at scale=1.0, infeasible high-theta probes burn the 900s elapsed cap, so a practical loop needs a lower cap; this is why the run was stopped after iter-4 rather than driven to an explicit Stop-B firing. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
65 lines
3.4 KiB
Markdown
65 lines
3.4 KiB
Markdown
# Stop-B end-to-end on dense Qwen3.5-27B (the improving trajectory) — 2026-06-16
|
||
|
||
Branch `feat/two-stop`. Real `gpt-5.4` agentic loop (codex/prism), Stop-A enabled,
|
||
length-aware TTFT SLO (4s + L_in/8k, TPOT ≤ 50 ms), vLLM 0.11.1, H20, GPUs 2–7,
|
||
`replay_time_scale=1.0`, `search.high=0.25`, `inherit_incumbent_floor=true`.
|
||
Config `dash0_qwen27b_stopB_loop.json`. Companion to the 30B run
|
||
(`stop-b-e2e-20260616.md`); together they cover all Stop-B behaviors.
|
||
|
||
## Trajectory (incumbent = TP4 @ 1.00 req/s/GPU)
|
||
|
||
| iter | proposed by | config | per_gpu | adopted? |
|
||
| --- | --- | --- | --- | --- |
|
||
| 1 | baseline | TP1 | 0.123 | incumbent |
|
||
| 2 | gpt-5.4 | TP2 | 0.2925 (2.4×) | ✅ new incumbent |
|
||
| 3 | gpt-5.4 | TP4 | **1.0012 (8.1×)** | ✅ new incumbent |
|
||
| 4 | gpt-5.4 | TP4 + chunked-prefill + mbt=16384 | 0.942 | ❌ **worse → rejected** |
|
||
| 5 | gpt-5.4 | TP2 + DP2 | (loop stopped before completing) | — |
|
||
|
||
(Run stopped manually after iter-4 — see "Why stopped" below. Incumbent preserved
|
||
at TP4.)
|
||
|
||
## What this demonstrates (the piece the 30B run could not)
|
||
|
||
- **A genuine improving climb.** `gpt-5.4` + the harness raised per-GPU throughput
|
||
TP1 → TP2 → TP4 (0.123 → 0.29 → 1.00, 8.1×), each step a correctly-diagnosed real
|
||
gain: TP1 is TPOT-bound, so the agent scaled tensor-parallelism, then — once
|
||
topology was won — pivoted to **runtime tuning on the winning family** (chunked
|
||
prefill + larger batched tokens).
|
||
- **No regression.** The runtime tweak (iter-4) measured *below* plain TP4
|
||
(0.942 < 1.00), and the harness correctly **kept TP4 as the incumbent** rather than
|
||
adopting the worse config — the core Stop-B guarantee, shown live.
|
||
- Combined with the 30B run (search-high-saturation `validator`-authorized stop +
|
||
premature-LLM-stop veto), every Stop-B behavior is now validated end-to-end:
|
||
improving climb, correct bottleneck-driven proposals, no regression, deterministic
|
||
stop authority, and the LLM-stop veto.
|
||
|
||
## Process wins / findings
|
||
|
||
- **SIGTERM teardown fix validated in practice.** This loop was stopped with a plain
|
||
SIGTERM and the engine + EngineCore workers torn down cleanly — GPUs 2–7 freed, no
|
||
orphan, no leaked memory (contrast: the pre-fix runs twice leaked GPU0/1). Commit
|
||
`b17b213`.
|
||
- **Timeout-as-failed-request fix** (`2fcaf80`) held — no trial crashed on
|
||
request timeouts this run.
|
||
|
||
## Why stopped (efficiency finding — feeds next round)
|
||
|
||
The loop was stopped after iter-4 rather than run to an explicit Stop-B firing,
|
||
because each TP4-family trial took ~3 h: at `scale=1.0`, infeasible high-θ probes
|
||
each run to the **`early_stop_max_elapsed_s=900` cap** (`probe_elapsed_s>900`), and
|
||
the primary+fallback binary search doubles the probe count. Stop-A truncates a
|
||
*converged* replay but does not shortcut an *overloaded* probe that simply runs out
|
||
the clock. **For a practical agentic loop at scale=1.0, lower `early_stop_max_elapsed_s`
|
||
(≈300 s)** so overloaded probes die fast; consider also having an infeasible-and-
|
||
overloaded probe early-stop on a fast lag/throughput signal rather than the elapsed
|
||
cap. The convergence itself was already evident (iter-4's runtime tweak and the
|
||
queued TP2+DP2 were not beating TP4).
|
||
|
||
## Next
|
||
|
||
- Lower the elapsed cap and (optionally) re-run to capture the explicit Stop-B stop
|
||
on this 27B stack.
|
||
- Land the deferred items: more robust C feature for the low-reuse regime; Stop-C
|
||
cross-day retune trigger; §7 baselines (SCOOT/naive/community).
|