Audit telemetry intervention response for tuning

docs/intervention-response-v0-protocol-20260714.md

@@ -0,0 +1,71 @@
+# Telemetry intervention-response v0 protocol
+
+Status: **FROZEN BEFORE V0 ANALYSIS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Claim boundary
+
+The closed residual route asked whether one absolute engine-state snapshot can
+predict unmeasured configurations.  V0 asks a different, narrower question:
+
+> Does an adjacent, controlled MNS intervention produce an early engine-state
+> response that is distinguishable from same-config repeat noise?
+
+Passing this gate only authorizes a matched real-GPU pilot.  It does not prove
+that telemetry improves tuning, that any metric is a causal mediator, or that
+the response transfers to a new workload, topology, or knob family.
+
+## Data and estimand
+
+- Source: Phase 6 solo-authoritative Qwen3-30B-A3B/vLLM 0.24 Layer-1 streams.
+- Action pairs: primary runs at identical study hash, TP, sampling anchor, and
+  request-order hash, with adjacent `MNS={8,16,32,64}` values.
+- Noise pairs: primary versus confirmation at the same complete config,
+  anchor, and request-order hash.  Only primary-to-confirmation pairs are used;
+  confirmations are not combined into pseudo-independent all-pairs.
+- Fixed early windows: 5 seconds and 10 seconds from the measured interval
+  start.  All runs exceed 10 seconds, so early-stop censoring cannot change the
+  telemetry window.
+- Full-run pass rate and feasibility are descriptive only because an early
+  stop can make full elapsed durations differ.
+
+The statistical unit is a run pair.  Scheduler steps are summarized within a
+run and are never counted as independent trials.
+
+## Frozen response gate
+
+The directly measured gate features are scheduler-step rate, decode-batch
+mean, prefill-token fraction, waiting/running queue mean, KV-usage mean, and
+CUDA-graph padding fraction.
+
+A feature qualifies at one horizon only if:
+
+1. at least 75% of nonzero action deltas have the same sign;
+2. median absolute action delta is at least 2x the median absolute repeat
+   delta; and
+3. at least 50% of action deltas exceed the repeat-noise absolute p95.
+
+V0 opens a GPU pilot only if:
+
+- there are exactly 17 frozen adjacent-MNS action pairs;
+- there are at least 20 primary/confirmation repeat pairs;
+- all identity, finite-value, counter, and ratio invariants pass; and
+- at least two gate features qualify at both 5 and 10 seconds.
+
+Any data red flag stops the analysis before interpreting the response.
+
+## If V0 passes
+
+Register a dash0 pilot around a known scaling knee.  The pilot must use the
+same request sequence and arrival times, one serving job at a time, one changed
+knob, randomized `A/B` versus `B/A` order, common non-censored measurement
+windows, and trial-level repetitions.  It must compare a response-aware next
+action against an outcome-only policy under complete startup, warm-up, and
+H20-hour accounting.
+
+## If V0 fails
+
+Do not add telemetry fields or train a larger model.  The current Layer-1 state
+does not identify even an MNS intervention above repeat noise on this task, so
+the telemetry-guided tuning route remains diagnostic only.

docs/intervention-response-v0-results-20260714.md

@@ -0,0 +1,157 @@
+# Telemetry intervention-response v0/v1 results
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Decision
+
+**STOP before a new H20 pilot.**  The current Layer-1 aggregate telemetry does
+not identify a sufficiently general early response to an MNS intervention,
+and it does not improve action-efficacy prediction over exact external prefix
+outcomes on the available development tasks.
+
+This is a negative result about the present state representation and
+experiment design.  It does not establish that engine telemetry is useless for
+tuning, and it is not held-out evidence.
+
+## Hypothesis and frozen test
+
+The tested hypothesis was:
+
+> With the workload and all non-MNS settings held fixed, increasing MNS causes
+> a 5--10 second engine-state response that is larger than same-config repeat
+> noise and that predicts whether the action makes the full run feasible.
+
+A response feature had to satisfy all three frozen conditions at both 5 and
+10 seconds: at least 0.75 sign consistency, median absolute action effect at
+least 2x the repeat median, and at least 0.50 of action deltas above the repeat
+absolute p95.  At least two features had to pass.  A telemetry feature was
+decision-relevant only if its leave-one-repeat-out balanced accuracy was at
+least 0.75 and at least 0.15 above the best exact external prefix outcome.
+
+## What was implemented
+
+- A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
+- Exact action pairing with request-order hash, offered load, TP, load role,
+  and repetition held fixed.
+- Same-config repeat-noise estimation without treating scheduler steps as
+  independent samples.
+- Exact 5/10-second request-prefix outcomes using monotonic completion times.
+- A one-feature leave-one-repeat-out efficacy audit; no multivariate model was
+  fitted to the 12 examples.
+- Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity
+  invariants in machine-readable audit artifacts.
+- Trial-by-trial validation against the P1 manifest, plus content hashes for
+  every result, request file, and Layer-1 stream.
+
+## Experiment A: Phase-6 retrospective audit
+
+Phase 6 supplied 17 adjacent-MNS actions and 29 same-config
+primary/confirmation pairs.  No feature passed at either horizon, producing
+`STOP_NO_IDENTIFIABLE_RESPONSE`.
+
+The confirmation sample is not a clean replication distribution: confirmations
+were selectively run after disputed primary outcomes.  Several same-config
+pairs consequently followed radically different trajectories.  This result
+therefore remains a valid failure of the frozen v0 gate, but it cannot by itself
+separate normal run variance from confirmation-selection bias.
+
+## Experiment B: prospective-repeat confirmation
+
+P1 supplied three pre-arranged, disjoint request bands for every cell/load.
+Exact matched actions exist for TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at
+low/high load and repetitions 1/2/3.  This yields 12 action pairs and 24
+same-config consecutive-repeat pairs.
+
+The 24 adjacent repeat differences share their middle repetition within each
+three-run group.  They define a conservative empirical noise reference; they
+are not used as 24 independent samples in an inferential test.
+
+The result is `STOP_NO_PROSPECTIVE_RESPONSE`: zero features passed the response
+gate at either horizon.
+
+The strongest response was mean waiting-queue occupancy:
+
+| Horizon | Sign consistency | Action/repeat median | Action above repeat p95 | Gate |
+|---|---:|---:|---:|---|
+| 5 s | 1.000 | 1.292x | 0.167 | fail |
+| 10 s | 1.000 | 2.611x | 0.250 | fail |
+
+The direction is real enough to merit diagnosis, but the effect is not broad
+enough to guide a general action.  It is large for TP4/high-load trials and
+small or absent in other regimes.
+
+Full-run transitions contain six beneficial actions (`false -> true`) and six
+non-beneficial actions (three `false -> false`, three `true -> true`).  The
+beneficial label is also perfectly confounded with TP4 in this small dataset,
+so it cannot support a topology-general claim.
+
+| Horizon | Best telemetry delta | Balanced accuracy | Best external prefix delta | Balanced accuracy | Telemetry advantage |
+|---|---|---:|---|---:|---:|
+| 5 s | waiting queue | 0.750 | max TPOT / SLO | 0.833 | -0.083 |
+| 10 s | waiting queue | 0.750 | outstanding / admitted | 0.750 | 0.000 |
+
+No telemetry feature reaches the preregistered `+0.15` incremental threshold.
+
+## What this rules out
+
+It rules out using the current vector of 5/10-second global means as a solid
+mechanism for choosing the next config.  In particular, adding these aggregates
+to an LLM prompt or fitting a larger predictor would currently hide, rather
+than solve, the identifiability problem.
+
+It does not rule out an instrumentation-aware tuner built around a deliberately
+excited local system.  The existing runs were designed for endpoint/fidelity
+evaluation, not system identification: the MNS action is large, efficacy is
+confounded with TP, repeat bands contain different requests, and global means
+erase when queue buildup or service-rate changes occur.
+
+## Required redesign before spending H20-hours
+
+The next admissible experiment is a randomized, local A/B system-identification
+pilot around one fixed TP and one load knee:
+
+1. Replay the exact same request sequence and arrival times for both endpoints.
+2. Use small adjacent actions and randomized `A/B` versus `B/A` order.
+3. Record event-aligned response curves, including queue growth/drain rate,
+   prefill/decode service rate, and per-step service time, rather than only one
+   global mean.
+4. Separate a mechanism gate (repeatable response) from the end-to-end gate:
+   fewer trials or H20-hours to select a feasible near-optimal config than an
+   outcome-only tuner.
+5. Hold out a second load/workload for the final policy comparison.
+
+Until that design is frozen, a wider sweep would only generate more correlated
+observations and is not justified by the evidence above.
+
+## Reproduction
+
+```bash
+python3 runs/intervention-response-v0/test_analysis.py
+python3 runs/intervention-response-v0/test_p1_analysis.py
+
+python3 runs/intervention-response-v0/analyze_phase6.py \
+  --metrics runs/opprof-phase6/phase6/metrics.json \
+  --raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
+  --output runs/intervention-response-v0/phase6-audit.json
+
+python3 runs/intervention-response-v0/analyze_p1.py \
+  --run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
+  --manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
+  --output runs/intervention-response-v0/p1-audit.json
+```
+
+## Data sanity
+
+- Phase 6: action pairs `n=17`, repeat pairs `n=29`, trials `n=66`; MNS
+  action size min/max `8/32`, `3` distinct; action-state vectors `n=17`, `17`
+  distinct; streams `n=12`, bytes min/max `12,745,297/52,957,710`, `12`
+  distinct.
+- P1: action pairs `n=12`, repeat pairs `n=24`, trials `n=36`; MNS action
+  size min/max `48/56`, `2` distinct; efficacy labels `n=12`, min/max `0/1`,
+  `2` distinct; streams `n=6`, bytes min/max `17,449,143/29,431,988`, `6`
+  distinct.
+- Checked invariants: exact action request hashes and offered loads match;
+  all `36/36` P1 trials match the manifest; expected pair counts hold; all
+  deltas are finite; non-negative counters and bounded ratios hold; per-config
+  state vectors are not all identical; both efficacy classes are present.  No
+  red flags were observed.

docs/intervention-response-v1-p1-protocol-20260714.md

@@ -0,0 +1,58 @@
+# Intervention-response v1 prospective-repeat confirmation
+
+Status: **FROZEN AFTER PHASE-6 V0 FAILURE AND BEFORE P1 RESPONSE ANALYSIS**.
+
+Date: 2026-07-14 (Asia/Singapore).
+
+## Why this is a new confirmation, not a relaxed V0
+
+Phase-6 V0 failed its frozen global response gate.  Its 29 same-config
+confirmations were triggered after disputed outcomes, and the resulting noise
+sample contains extreme trajectory divergence by construction.  V0 remains
+failed and its thresholds are unchanged.
+
+The already-completed P1 campaign supplies a distinct test: three
+prospectively scheduled, disjoint repeat bands for every cell/load.  TP1 and
+TP4 use identical offered loads and exact request-order hashes across their MNS
+endpoints.  V1 asks whether an MNS response is identifiable against this
+prospective workload-repeat noise, and whether that response predicts action
+efficacy beyond exact external prefix outcomes.
+
+P1 is now development data.  No result here is held-out or paper-facing.
+
+## Frozen pairs
+
+- Action pairs: TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at low/high load and
+  repeat 1/2/3.  Endpoints must have identical TP, offered rate, repeat role,
+  and request-order hash. Expected `n=12`.
+- Repeat-noise pairs: consecutive pre-arranged repeat bands within each of six
+  cells and low/high load: `rep1 -> rep2`, `rep2 -> rep3`. Expected `n=24`.
+  Repeat bands intentionally contain different requests and therefore include
+  workload-sampling noise rather than pretending to be identical trials.
+  Adjacent differences share the middle run; the gate uses their empirical
+  magnitude only and does not treat the 24 differences as independent samples
+  for a p-value or confidence interval.
+- Prefix horizons: 5 and 10 seconds. Exact monotonic request completion times
+  and the same Layer-1 intervals are used.
+
+## Frozen gates
+
+The response-identifiability thresholds are exactly the Phase-6 V0 thresholds:
+75% sign consistency, 2x median effect/repeat noise, and at least 50% of action
+deltas above repeat absolute p95. At least two response features must qualify
+at both horizons.
+
+Action efficacy is one only for an infeasible-to-feasible full-run transition.
+The 12 action pairs must contain at least four examples of each class.
+
+For decision relevance, each individual external-outcome response feature and
+each individual telemetry-response feature is evaluated by leave-one-repeat-
+band-out threshold fitting. This intentionally avoids a multivariate model on
+12 examples. At least one telemetry feature must, at both horizons:
+
+1. reach balanced accuracy at least 0.75; and
+2. exceed the best external-outcome response feature by at least 0.15.
+
+Only if data validity, response identifiability, and incremental decision
+relevance all pass does V1 open a newly registered matched GPU pilot. No
+threshold or feature is changed after observing V1.