Telemetry intervention-response v0/v1 results

Date: 2026-07-14 (Asia/Singapore).

Decision

STOP before a new H20 pilot. The current Layer-1 aggregate telemetry does not identify a sufficiently general early response to an MNS intervention, and it does not improve action-efficacy prediction over exact external prefix outcomes on the available development tasks.

This is a negative result about the present state representation and experiment design. It does not establish that engine telemetry is useless for tuning, and it is not held-out evidence.

Hypothesis and frozen test

The tested hypothesis was:

With the workload and all non-MNS settings held fixed, increasing MNS causes a 5--10 second engine-state response that is larger than same-config repeat noise and that predicts whether the action makes the full run feasible.

A response feature had to satisfy all three frozen conditions at both 5 and 10 seconds: at least 0.75 sign consistency, median absolute action effect at least 2x the repeat median, and at least 0.50 of action deltas above the repeat absolute p95. At least two features had to pass. A telemetry feature was decision-relevant only if its leave-one-repeat-out balanced accuracy was at least 0.75 and at least 0.15 above the best exact external prefix outcome.

What was implemented

A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
Exact action pairing with request-order hash, offered load, TP, load role, and repetition held fixed.
Same-config repeat-noise estimation without treating scheduler steps as independent samples.
Exact 5/10-second request-prefix outcomes using monotonic completion times.
A one-feature leave-one-repeat-out efficacy audit; no multivariate model was fitted to the 12 examples.
Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity invariants in machine-readable audit artifacts.
Trial-by-trial validation against the P1 manifest, plus content hashes for every result, request file, and Layer-1 stream.

Experiment A: Phase-6 retrospective audit

Phase 6 supplied 17 adjacent-MNS actions and 29 same-config primary/confirmation pairs. No feature passed at either horizon, producing STOP_NO_IDENTIFIABLE_RESPONSE.

The confirmation sample is not a clean replication distribution: confirmations were selectively run after disputed primary outcomes. Several same-config pairs consequently followed radically different trajectories. This result therefore remains a valid failure of the frozen v0 gate, but it cannot by itself separate normal run variance from confirmation-selection bias.

Experiment B: prospective-repeat confirmation

P1 supplied three pre-arranged, disjoint request bands for every cell/load. Exact matched actions exist for TP1 MNS 8 -> 64 and TP4 MNS 16 -> 64, at low/high load and repetitions 1/2/3. This yields 12 action pairs and 24 same-config consecutive-repeat pairs.

The 24 adjacent repeat differences share their middle repetition within each three-run group. They define a conservative empirical noise reference; they are not used as 24 independent samples in an inferential test.

The result is STOP_NO_PROSPECTIVE_RESPONSE: zero features passed the response gate at either horizon.

The strongest response was mean waiting-queue occupancy:

Horizon	Sign consistency	Action/repeat median	Action above repeat p95	Gate
5 s	1.000	1.292x	0.167	fail
10 s	1.000	2.611x	0.250	fail

The direction is real enough to merit diagnosis, but the effect is not broad enough to guide a general action. It is large for TP4/high-load trials and small or absent in other regimes.

Full-run transitions contain six beneficial actions (false -> true) and six non-beneficial actions (three false -> false, three true -> true). The beneficial label is also perfectly confounded with TP4 in this small dataset, so it cannot support a topology-general claim.

Horizon	Best telemetry delta	Balanced accuracy	Best external prefix delta	Balanced accuracy	Telemetry advantage
5 s	waiting queue	0.750	max TPOT / SLO	0.833	-0.083
10 s	waiting queue	0.750	outstanding / admitted	0.750	0.000

No telemetry feature reaches the preregistered +0.15 incremental threshold.

What this rules out

It rules out using the current vector of 5/10-second global means as a solid mechanism for choosing the next config. In particular, adding these aggregates to an LLM prompt or fitting a larger predictor would currently hide, rather than solve, the identifiability problem.

It does not rule out an instrumentation-aware tuner built around a deliberately excited local system. The existing runs were designed for endpoint/fidelity evaluation, not system identification: the MNS action is large, efficacy is confounded with TP, repeat bands contain different requests, and global means erase when queue buildup or service-rate changes occur.

Required redesign before spending H20-hours

The next admissible experiment is a randomized, local A/B system-identification pilot around one fixed TP and one load knee:

Replay the exact same request sequence and arrival times for both endpoints.
Use small adjacent actions and randomized A/B versus B/A order.
Record event-aligned response curves, including queue growth/drain rate, prefill/decode service rate, and per-step service time, rather than only one global mean.
Separate a mechanism gate (repeatable response) from the end-to-end gate: fewer trials or H20-hours to select a feasible near-optimal config than an outcome-only tuner.
Hold out a second load/workload for the final policy comparison.

Until that design is frozen, a wider sweep would only generate more correlated observations and is not justified by the evidence above.

Reproduction

python3 runs/intervention-response-v0/test_analysis.py
python3 runs/intervention-response-v0/test_p1_analysis.py

python3 runs/intervention-response-v0/analyze_phase6.py \
  --metrics runs/opprof-phase6/phase6/metrics.json \
  --raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
  --output runs/intervention-response-v0/phase6-audit.json

python3 runs/intervention-response-v0/analyze_p1.py \
  --run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
  --manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
  --output runs/intervention-response-v0/p1-audit.json

Data sanity

Phase 6: action pairs n=17, repeat pairs n=29, trials n=66; MNS action size min/max 8/32, 3 distinct; action-state vectors n=17, 17 distinct; streams n=12, bytes min/max 12,745,297/52,957,710, 12 distinct.
P1: action pairs n=12, repeat pairs n=24, trials n=36; MNS action size min/max 48/56, 2 distinct; efficacy labels n=12, min/max 0/1, 2 distinct; streams n=6, bytes min/max 17,449,143/29,431,988, 6 distinct.
Checked invariants: exact action request hashes and offered loads match; all 36/36 P1 trials match the manifest; expected pair counts hold; all deltas are finite; non-negative counters and bounded ratios hold; per-config state vectors are not all identical; both efficacy classes are present. No red flags were observed.

7.3 KiB Raw Blame History