161 lines
8.1 KiB
Markdown
161 lines
8.1 KiB
Markdown
# Phase-aware telemetry intervention-response v3 results
|
|
|
|
Date: 2026-07-14 (Asia/Singapore).
|
|
|
|
Decision: **`STOP_NO_INCREMENTAL_TUNING_SIGNAL`**.
|
|
|
|
## Claim tested
|
|
|
|
After the replay has developed queue, batch, and KV state, does increasing MNS
|
|
from 16 to 64 create telemetry responses that exceed workload-repeat noise, and
|
|
does any such response identify whether the action repairs the full-run SLO
|
|
better than external prefix outcomes alone?
|
|
|
|
The first clause passed. The second clause failed. Long-window telemetry is
|
|
mechanistically informative, but this pilot does not support its necessity for
|
|
tuning this action.
|
|
|
|
## Setup
|
|
|
|
- dash0 GPU 0-3: four NVIDIA H20 GPUs; Qwen3-30B-A3B; patched vLLM
|
|
`0.24.1.dev3+opprof`; TP=4.
|
|
- Action: MNS `16 -> 64` with exact request, arrival, and input-length hashes.
|
|
- Trace: `chat_w20260312_1000`, replay-time scale 0.5, 300 seconds.
|
|
- Loads: 1.5 and 2.125 requests/s/GPU; three disjoint bands; endpoint order
|
|
A/B, B/A, A/B; load order low/mid, mid/low, low/mid.
|
|
- Five cumulative checkpoints: 30, 75, 150, 225, and 300 seconds; four
|
|
non-overlapping quarter blocks.
|
|
- Six fresh-server sessions, 12 measured runs, six action pairs, and eight
|
|
same-config repeat pairs.
|
|
|
|
The prior three-load attempt is not part of the result. Its MNS=16 workload at
|
|
3.125 requests/s/GPU could not drain by the 450-second client timeout and
|
|
produced no high-load result. V3 reran every retained point from scratch.
|
|
|
|
## End-to-end outcome
|
|
|
|
All three low-load pairs remained feasible (`true->true`, label 0). All three
|
|
pressure-load pairs changed from infeasible to feasible (`false->true`, label
|
|
1). MNS=16 pressure pass rates were 0.5604, 0.3145, and 0.2635; all three
|
|
MNS=64 pressure runs reached 1.0. This yielded a balanced 3/3 action label set.
|
|
|
|
## Mechanism result
|
|
|
|
No telemetry feature passed the action-versus-repeat gate at 10% or 25%. At
|
|
50%, graph padding first passed. At both 75% and 100%, graph padding and queue
|
|
waiting passed, satisfying the requirement for two features at the same pair
|
|
of adjacent checkpoints and with consistent directions in both load regimes.
|
|
|
|
| Feature | Direction for MNS 16->64 | 75% effect/repeat median | 75% above repeat p95 | 100% effect/repeat median | 100% above repeat p95 |
|
|
|---|---:|---:|---:|---:|---:|
|
|
| `queue_waiting_mean` | lower | 1346.22 | 3/6 | 898.70 | 3/6 |
|
|
| `graph_padding_fraction` | higher | 5.00 | 4/6 | 5.74 | 5/6 |
|
|
|
|
The very large queue effect/median-repeat ratios should not be read alone: its
|
|
repeat p95 was much larger than its repeat median, so the independent p95
|
|
coverage criterion remained binding. Full-window queue-waiting deltas were
|
|
-0.19 to -0.32 at low load and -20.99 to -32.16 at pressure load. Graph
|
|
padding increased in every pair, by 0.00133-0.00217 at low load and
|
|
0.00705-0.00873 at pressure load.
|
|
|
|
The mechanism is therefore a real tradeoff: larger MNS reduces queueing,
|
|
especially under pressure, while increasing CUDA-graph padding.
|
|
|
|
## Tuning-signal result
|
|
|
|
Leave-one-repetition-out balanced accuracy was evaluated against the best
|
|
external prefix-outcome feature at every predeclared checkpoint.
|
|
|
|
| Replay phase | Best external BA | Best telemetry BA | Incremental telemetry gate |
|
|
|---:|---:|---:|---|
|
|
| 10% | 0.833 | 0.833 | fail |
|
|
| 25% | 1.000 | 1.000 | fail |
|
|
| 50% | 0.833 | 1.000 | pass at this checkpoint only |
|
|
| 75% | 1.000 | 1.000 | fail |
|
|
| 100% | 1.000 | 1.000 | fail |
|
|
|
|
At 50%, several telemetry features exceeded the external baseline by at least
|
|
0.15, including the phase-stable mechanism feature
|
|
`graph_padding_fraction`. The advantage did not hold at either adjacent
|
|
checkpoint. Consequently no feature passed the frozen two-adjacent-phase
|
|
requirement.
|
|
|
|
The important ordering is that external TTFT already classified the action
|
|
perfectly at 25%, whereas the robust two-feature mechanism response did not
|
|
emerge until 75%-100%. In this setup telemetry explains *why* MNS helps, but it
|
|
does not provide earlier or more reliable action selection than direct prefix
|
|
outcomes.
|
|
|
|
## Research conclusion
|
|
|
|
The 5/10-second negative result was indeed too narrow. It only ruled out an
|
|
ultra-early telemetry verifier; it did not rule out engine-state information.
|
|
The 300-second pilot finds a clear and reproducible queueing-versus-padding
|
|
response.
|
|
|
|
However, this does not rescue the direct telemetry-guided tuning claim. For
|
|
this action and workload, the external signal is already as good or better
|
|
before the telemetry mechanism becomes stable. The project should therefore
|
|
not claim that engine instrumentation is necessary for tuning on this evidence,
|
|
and should not open an E2E policy test from this pilot.
|
|
|
|
The narrower simulator-residual route remains logically open: telemetry may
|
|
explain why a simulator misranks real configurations even when direct online
|
|
outcomes can guide a tuner. That is a different hypothesis and was not tested
|
|
here.
|
|
|
|
## Change and verification
|
|
|
|
Change: absolute 5/10-second prefixes were replaced by phase-aware 30/75/150/
|
|
225/300-second analysis; SLO early stop was disabled; full Layer-1 coverage,
|
|
hash, request-accounting, controller, and stream/footer gates were added. The
|
|
mechanism gate was corrected to require two features at the same adjacent
|
|
phase pair.
|
|
|
|
Expected effect: distinguish “telemetry has not developed yet” from “telemetry
|
|
does not identify or improve the action decision.”
|
|
|
|
Verification: five local analysis/controller test suites passed; remote
|
|
manifest preflight and command dry-run passed; six serialized sessions passed
|
|
all stream invariants; the analyzer was rerun and produced byte-identical
|
|
output.
|
|
|
|
Result: mechanism evidence passed; incremental tuning evidence failed. Audit
|
|
SHA256: `45f6f248712f9cbd3ed72036837ff6dc5b5c14c0f2eb6ba5cd5daceb1aa4ddb7`.
|
|
|
|
Remaining risk: this is a development pilot with one model, one TP, one action,
|
|
two retained loads, three request bands, and six action labels. It is adequate
|
|
to reject opening the next direct-policy stage, not to establish a universal
|
|
negative claim about telemetry.
|
|
|
|
## Research-validity audit
|
|
|
|
| Check | Verdict | Evidence / boundary |
|
|
|---|---|---|
|
|
| Real system and E2E outcome | PASS | Real H20/vLLM replay; full SLO outcome accompanies mechanism telemetry. |
|
|
| Matched action baseline | PASS | Exact request/arrival/length hashes for MNS 16 and 64; external prefix outcome is the decision baseline. |
|
|
| Repeats and order effects | PASS for pilot | Three disjoint bands; A/B, B/A, A/B endpoint order; counter-rotated load order. |
|
|
| Selective load removal | PASS with narrowed claim | The 3.125 load produced no result before any action comparison; the failure and cost are retained, and all kept points were freshly rerun. |
|
|
| Significance/generalization | NEEDS EVIDENCE for a paper claim | Only three bands, one model, one TP, one action, and six labels. This is explicitly a stage gate. |
|
|
| Calibration versus evaluation | NEEDS EVIDENCE for a positive policy claim | Frozen gates and leave-one-band-out folds reduce leakage, but a new workload/model hold-out is still required. |
|
|
| Platform/reproducibility | PASS | Commit, commands, manifest, controller state, platform fingerprint, raw remote paths, and audit hashes are recorded. |
|
|
|
|
## Data sanity
|
|
|
|
- Measured runs: n=12; elapsed 300.346-317.012 seconds; 12 distinct; pass rate
|
|
0.2635-1.0 with 4 distinct values; selected requests 1800-2550 with 2
|
|
distinct values.
|
|
- Sessions: n=6; 0.8413-0.8631 H20-hours; 6 distinct; Layer-1 records
|
|
58,465-64,776; 6 distinct. V3 cost was 5.0924 H20-hours; total including
|
|
the invalid attempt was 6.4505, below the 8.0 cap.
|
|
- Labels: n=6; min/max 0/1; 2 distinct. Action pairs were 6 and repeat pairs
|
|
were 8 at every checkpoint.
|
|
- Coverage-gap observations: n=60; start gaps 0.0427-0.1247 seconds; end gaps
|
|
0.00014-0.1705; maximum internal gaps 0.1695-0.6528, all below one second.
|
|
- Checked invariants: exact pair hashes and counts, all runs uncensored, full
|
|
request accounting, monotonic admitted/completed coverage, monotonic Layer-1
|
|
timestamps, nonnegative counters, bounded ratios, non-identical per-config
|
|
states, contiguous step indices, zero drops, footer/sidecar agreement, no
|
|
controller failures, all sessions complete, GPU idle after completion. No
|
|
red flags were found.
|