Record phase-aware telemetry pilot result

docs/intervention-response-v3-results-20260714.md

@@ -0,0 +1,160 @@
+# Phase-aware telemetry intervention-response v3 results
+
+Date: 2026-07-14 (Asia/Singapore).
+
+Decision: **`STOP_NO_INCREMENTAL_TUNING_SIGNAL`**.
+
+## Claim tested
+
+After the replay has developed queue, batch, and KV state, does increasing MNS
+from 16 to 64 create telemetry responses that exceed workload-repeat noise, and
+does any such response identify whether the action repairs the full-run SLO
+better than external prefix outcomes alone?
+
+The first clause passed.  The second clause failed.  Long-window telemetry is
+mechanistically informative, but this pilot does not support its necessity for
+tuning this action.
+
+## Setup
+
+- dash0 GPU 0-3: four NVIDIA H20 GPUs; Qwen3-30B-A3B; patched vLLM
+  `0.24.1.dev3+opprof`; TP=4.
+- Action: MNS `16 -> 64` with exact request, arrival, and input-length hashes.
+- Trace: `chat_w20260312_1000`, replay-time scale 0.5, 300 seconds.
+- Loads: 1.5 and 2.125 requests/s/GPU; three disjoint bands; endpoint order
+  A/B, B/A, A/B; load order low/mid, mid/low, low/mid.
+- Five cumulative checkpoints: 30, 75, 150, 225, and 300 seconds; four
+  non-overlapping quarter blocks.
+- Six fresh-server sessions, 12 measured runs, six action pairs, and eight
+  same-config repeat pairs.
+
+The prior three-load attempt is not part of the result.  Its MNS=16 workload at
+3.125 requests/s/GPU could not drain by the 450-second client timeout and
+produced no high-load result.  V3 reran every retained point from scratch.
+
+## End-to-end outcome
+
+All three low-load pairs remained feasible (`true->true`, label 0).  All three
+pressure-load pairs changed from infeasible to feasible (`false->true`, label
+1).  MNS=16 pressure pass rates were 0.5604, 0.3145, and 0.2635; all three
+MNS=64 pressure runs reached 1.0.  This yielded a balanced 3/3 action label set.
+
+## Mechanism result
+
+No telemetry feature passed the action-versus-repeat gate at 10% or 25%.  At
+50%, graph padding first passed.  At both 75% and 100%, graph padding and queue
+waiting passed, satisfying the requirement for two features at the same pair
+of adjacent checkpoints and with consistent directions in both load regimes.
+
+| Feature | Direction for MNS 16->64 | 75% effect/repeat median | 75% above repeat p95 | 100% effect/repeat median | 100% above repeat p95 |
+|---|---:|---:|---:|---:|---:|
+| `queue_waiting_mean` | lower | 1346.22 | 3/6 | 898.70 | 3/6 |
+| `graph_padding_fraction` | higher | 5.00 | 4/6 | 5.74 | 5/6 |
+
+The very large queue effect/median-repeat ratios should not be read alone: its
+repeat p95 was much larger than its repeat median, so the independent p95
+coverage criterion remained binding.  Full-window queue-waiting deltas were
+-0.19 to -0.32 at low load and -20.99 to -32.16 at pressure load.  Graph
+padding increased in every pair, by 0.00133-0.00217 at low load and
+0.00705-0.00873 at pressure load.
+
+The mechanism is therefore a real tradeoff: larger MNS reduces queueing,
+especially under pressure, while increasing CUDA-graph padding.
+
+## Tuning-signal result
+
+Leave-one-repetition-out balanced accuracy was evaluated against the best
+external prefix-outcome feature at every predeclared checkpoint.
+
+| Replay phase | Best external BA | Best telemetry BA | Incremental telemetry gate |
+|---:|---:|---:|---|
+| 10% | 0.833 | 0.833 | fail |
+| 25% | 1.000 | 1.000 | fail |
+| 50% | 0.833 | 1.000 | pass at this checkpoint only |
+| 75% | 1.000 | 1.000 | fail |
+| 100% | 1.000 | 1.000 | fail |
+
+At 50%, several telemetry features exceeded the external baseline by at least
+0.15, including the phase-stable mechanism feature
+`graph_padding_fraction`.  The advantage did not hold at either adjacent
+checkpoint.  Consequently no feature passed the frozen two-adjacent-phase
+requirement.
+
+The important ordering is that external TTFT already classified the action
+perfectly at 25%, whereas the robust two-feature mechanism response did not
+emerge until 75%-100%.  In this setup telemetry explains *why* MNS helps, but it
+does not provide earlier or more reliable action selection than direct prefix
+outcomes.
+
+## Research conclusion
+
+The 5/10-second negative result was indeed too narrow.  It only ruled out an
+ultra-early telemetry verifier; it did not rule out engine-state information.
+The 300-second pilot finds a clear and reproducible queueing-versus-padding
+response.
+
+However, this does not rescue the direct telemetry-guided tuning claim.  For
+this action and workload, the external signal is already as good or better
+before the telemetry mechanism becomes stable.  The project should therefore
+not claim that engine instrumentation is necessary for tuning on this evidence,
+and should not open an E2E policy test from this pilot.
+
+The narrower simulator-residual route remains logically open: telemetry may
+explain why a simulator misranks real configurations even when direct online
+outcomes can guide a tuner.  That is a different hypothesis and was not tested
+here.
+
+## Change and verification
+
+Change: absolute 5/10-second prefixes were replaced by phase-aware 30/75/150/
+225/300-second analysis; SLO early stop was disabled; full Layer-1 coverage,
+hash, request-accounting, controller, and stream/footer gates were added.  The
+mechanism gate was corrected to require two features at the same adjacent
+phase pair.
+
+Expected effect: distinguish “telemetry has not developed yet” from “telemetry
+does not identify or improve the action decision.”
+
+Verification: five local analysis/controller test suites passed; remote
+manifest preflight and command dry-run passed; six serialized sessions passed
+all stream invariants; the analyzer was rerun and produced byte-identical
+output.
+
+Result: mechanism evidence passed; incremental tuning evidence failed.  Audit
+SHA256: `45f6f248712f9cbd3ed72036837ff6dc5b5c14c0f2eb6ba5cd5daceb1aa4ddb7`.
+
+Remaining risk: this is a development pilot with one model, one TP, one action,
+two retained loads, three request bands, and six action labels.  It is adequate
+to reject opening the next direct-policy stage, not to establish a universal
+negative claim about telemetry.
+
+## Research-validity audit
+
+| Check | Verdict | Evidence / boundary |
+|---|---|---|
+| Real system and E2E outcome | PASS | Real H20/vLLM replay; full SLO outcome accompanies mechanism telemetry. |
+| Matched action baseline | PASS | Exact request/arrival/length hashes for MNS 16 and 64; external prefix outcome is the decision baseline. |
+| Repeats and order effects | PASS for pilot | Three disjoint bands; A/B, B/A, A/B endpoint order; counter-rotated load order. |
+| Selective load removal | PASS with narrowed claim | The 3.125 load produced no result before any action comparison; the failure and cost are retained, and all kept points were freshly rerun. |
+| Significance/generalization | NEEDS EVIDENCE for a paper claim | Only three bands, one model, one TP, one action, and six labels.  This is explicitly a stage gate. |
+| Calibration versus evaluation | NEEDS EVIDENCE for a positive policy claim | Frozen gates and leave-one-band-out folds reduce leakage, but a new workload/model hold-out is still required. |
+| Platform/reproducibility | PASS | Commit, commands, manifest, controller state, platform fingerprint, raw remote paths, and audit hashes are recorded. |
+
+## Data sanity
+
+- Measured runs: n=12; elapsed 300.346-317.012 seconds; 12 distinct; pass rate
+  0.2635-1.0 with 4 distinct values; selected requests 1800-2550 with 2
+  distinct values.
+- Sessions: n=6; 0.8413-0.8631 H20-hours; 6 distinct; Layer-1 records
+  58,465-64,776; 6 distinct.  V3 cost was 5.0924 H20-hours; total including
+  the invalid attempt was 6.4505, below the 8.0 cap.
+- Labels: n=6; min/max 0/1; 2 distinct.  Action pairs were 6 and repeat pairs
+  were 8 at every checkpoint.
+- Coverage-gap observations: n=60; start gaps 0.0427-0.1247 seconds; end gaps
+  0.00014-0.1705; maximum internal gaps 0.1695-0.6528, all below one second.
+- Checked invariants: exact pair hashes and counts, all runs uncensored, full
+  request accounting, monotonic admitted/completed coverage, monotonic Layer-1
+  timestamps, nonnegative counters, bounded ratios, non-identical per-config
+  states, contiguous step indices, zero drops, footer/sidecar agreement, no
+  controller failures, all sessions complete, GPU idle after completion.  No
+  red flags were found.