# Telemetry intervention-response v0/v1 results

Date: 2026-07-14 (Asia/Singapore).

## Decision

**STOP before a new H20 pilot.**  The current Layer-1 aggregate telemetry does
not identify a sufficiently general early response to an MNS intervention,
and it does not improve action-efficacy prediction over exact external prefix
outcomes on the available development tasks.

This is a negative result about the present state representation and
experiment design.  It does not establish that engine telemetry is useless for
tuning, and it is not held-out evidence.

## Hypothesis and frozen test

The tested hypothesis was:

> With the workload and all non-MNS settings held fixed, increasing MNS causes
> a 5--10 second engine-state response that is larger than same-config repeat
> noise and that predicts whether the action makes the full run feasible.

A response feature had to satisfy all three frozen conditions at both 5 and
10 seconds: at least 0.75 sign consistency, median absolute action effect at
least 2x the repeat median, and at least 0.50 of action deltas above the repeat
absolute p95.  At least two features had to pass.  A telemetry feature was
decision-relevant only if its leave-one-repeat-out balanced accuracy was at
least 0.75 and at least 0.15 above the best exact external prefix outcome.

## What was implemented

- A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
- Exact action pairing with request-order hash, offered load, TP, load role,
  and repetition held fixed.
- Same-config repeat-noise estimation without treating scheduler steps as
  independent samples.
- Exact 5/10-second request-prefix outcomes using monotonic completion times.
- A one-feature leave-one-repeat-out efficacy audit; no multivariate model was
  fitted to the 12 examples.
- Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity
  invariants in machine-readable audit artifacts.
- Trial-by-trial validation against the P1 manifest, plus content hashes for
  every result, request file, and Layer-1 stream.

## Experiment A: Phase-6 retrospective audit

Phase 6 supplied 17 adjacent-MNS actions and 29 same-config
primary/confirmation pairs.  No feature passed at either horizon, producing
`STOP_NO_IDENTIFIABLE_RESPONSE`.

The confirmation sample is not a clean replication distribution: confirmations
were selectively run after disputed primary outcomes.  Several same-config
pairs consequently followed radically different trajectories.  This result
therefore remains a valid failure of the frozen v0 gate, but it cannot by itself
separate normal run variance from confirmation-selection bias.

## Experiment B: prospective-repeat confirmation

P1 supplied three pre-arranged, disjoint request bands for every cell/load.
Exact matched actions exist for TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at
low/high load and repetitions 1/2/3.  This yields 12 action pairs and 24
same-config consecutive-repeat pairs.

The 24 adjacent repeat differences share their middle repetition within each
three-run group.  They define a conservative empirical noise reference; they
are not used as 24 independent samples in an inferential test.

The result is `STOP_NO_PROSPECTIVE_RESPONSE`: zero features passed the response
gate at either horizon.

The strongest response was mean waiting-queue occupancy:

| Horizon | Sign consistency | Action/repeat median | Action above repeat p95 | Gate |
|---|---:|---:|---:|---|
| 5 s | 1.000 | 1.292x | 0.167 | fail |
| 10 s | 1.000 | 2.611x | 0.250 | fail |

The direction is real enough to merit diagnosis, but the effect is not broad
enough to guide a general action.  It is large for TP4/high-load trials and
small or absent in other regimes.

Full-run transitions contain six beneficial actions (`false -> true`) and six
non-beneficial actions (three `false -> false`, three `true -> true`).  The
beneficial label is also perfectly confounded with TP4 in this small dataset,
so it cannot support a topology-general claim.

| Horizon | Best telemetry delta | Balanced accuracy | Best external prefix delta | Balanced accuracy | Telemetry advantage |
|---|---|---:|---|---:|---:|
| 5 s | waiting queue | 0.750 | max TPOT / SLO | 0.833 | -0.083 |
| 10 s | waiting queue | 0.750 | outstanding / admitted | 0.750 | 0.000 |

No telemetry feature reaches the preregistered `+0.15` incremental threshold.

## What this rules out

It rules out using the current vector of 5/10-second global means as a solid
mechanism for choosing the next config.  In particular, adding these aggregates
to an LLM prompt or fitting a larger predictor would currently hide, rather
than solve, the identifiability problem.

It does not rule out an instrumentation-aware tuner built around a deliberately
excited local system.  The existing runs were designed for endpoint/fidelity
evaluation, not system identification: the MNS action is large, efficacy is
confounded with TP, repeat bands contain different requests, and global means
erase when queue buildup or service-rate changes occur.

## Required redesign before spending H20-hours

The next admissible experiment is a randomized, local A/B system-identification
pilot around one fixed TP and one load knee:

1. Replay the exact same request sequence and arrival times for both endpoints.
2. Use small adjacent actions and randomized `A/B` versus `B/A` order.
3. Record event-aligned response curves, including queue growth/drain rate,
   prefill/decode service rate, and per-step service time, rather than only one
   global mean.
4. Separate a mechanism gate (repeatable response) from the end-to-end gate:
   fewer trials or H20-hours to select a feasible near-optimal config than an
   outcome-only tuner.
5. Hold out a second load/workload for the final policy comparison.

Until that design is frozen, a wider sweep would only generate more correlated
observations and is not justified by the evidence above.

## Reproduction

```bash
python3 runs/intervention-response-v0/test_analysis.py
python3 runs/intervention-response-v0/test_p1_analysis.py

python3 runs/intervention-response-v0/analyze_phase6.py \
  --metrics runs/opprof-phase6/phase6/metrics.json \
  --raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
  --output runs/intervention-response-v0/phase6-audit.json

python3 runs/intervention-response-v0/analyze_p1.py \
  --run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
  --manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
  --output runs/intervention-response-v0/p1-audit.json
```

## Data sanity

- Phase 6: action pairs `n=17`, repeat pairs `n=29`, trials `n=66`; MNS
  action size min/max `8/32`, `3` distinct; action-state vectors `n=17`, `17`
  distinct; streams `n=12`, bytes min/max `12,745,297/52,957,710`, `12`
  distinct.
- P1: action pairs `n=12`, repeat pairs `n=24`, trials `n=36`; MNS action
  size min/max `48/56`, `2` distinct; efficacy labels `n=12`, min/max `0/1`,
  `2` distinct; streams `n=6`, bytes min/max `17,449,143/29,431,988`, `6`
  distinct.
- Checked invariants: exact action request hashes and offered loads match;
  all `36/36` P1 trials match the manifest; expected pair counts hold; all
  deltas are finite; non-negative counters and bounded ratios hold; per-config
  state vectors are not all identical; both efficacy classes are present.  No
  red flags were observed.
