158 lines
7.3 KiB
Markdown
158 lines
7.3 KiB
Markdown
# Telemetry intervention-response v0/v1 results
|
|
|
|
Date: 2026-07-14 (Asia/Singapore).
|
|
|
|
## Decision
|
|
|
|
**STOP before a new H20 pilot.** The current Layer-1 aggregate telemetry does
|
|
not identify a sufficiently general early response to an MNS intervention,
|
|
and it does not improve action-efficacy prediction over exact external prefix
|
|
outcomes on the available development tasks.
|
|
|
|
This is a negative result about the present state representation and
|
|
experiment design. It does not establish that engine telemetry is useless for
|
|
tuning, and it is not held-out evidence.
|
|
|
|
## Hypothesis and frozen test
|
|
|
|
The tested hypothesis was:
|
|
|
|
> With the workload and all non-MNS settings held fixed, increasing MNS causes
|
|
> a 5--10 second engine-state response that is larger than same-config repeat
|
|
> noise and that predicts whether the action makes the full run feasible.
|
|
|
|
A response feature had to satisfy all three frozen conditions at both 5 and
|
|
10 seconds: at least 0.75 sign consistency, median absolute action effect at
|
|
least 2x the repeat median, and at least 0.50 of action deltas above the repeat
|
|
absolute p95. At least two features had to pass. A telemetry feature was
|
|
decision-relevant only if its leave-one-repeat-out balanced accuracy was at
|
|
least 0.75 and at least 0.15 above the best exact external prefix outcome.
|
|
|
|
## What was implemented
|
|
|
|
- A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
|
|
- Exact action pairing with request-order hash, offered load, TP, load role,
|
|
and repetition held fixed.
|
|
- Same-config repeat-noise estimation without treating scheduler steps as
|
|
independent samples.
|
|
- Exact 5/10-second request-prefix outcomes using monotonic completion times.
|
|
- A one-feature leave-one-repeat-out efficacy audit; no multivariate model was
|
|
fitted to the 12 examples.
|
|
- Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity
|
|
invariants in machine-readable audit artifacts.
|
|
- Trial-by-trial validation against the P1 manifest, plus content hashes for
|
|
every result, request file, and Layer-1 stream.
|
|
|
|
## Experiment A: Phase-6 retrospective audit
|
|
|
|
Phase 6 supplied 17 adjacent-MNS actions and 29 same-config
|
|
primary/confirmation pairs. No feature passed at either horizon, producing
|
|
`STOP_NO_IDENTIFIABLE_RESPONSE`.
|
|
|
|
The confirmation sample is not a clean replication distribution: confirmations
|
|
were selectively run after disputed primary outcomes. Several same-config
|
|
pairs consequently followed radically different trajectories. This result
|
|
therefore remains a valid failure of the frozen v0 gate, but it cannot by itself
|
|
separate normal run variance from confirmation-selection bias.
|
|
|
|
## Experiment B: prospective-repeat confirmation
|
|
|
|
P1 supplied three pre-arranged, disjoint request bands for every cell/load.
|
|
Exact matched actions exist for TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at
|
|
low/high load and repetitions 1/2/3. This yields 12 action pairs and 24
|
|
same-config consecutive-repeat pairs.
|
|
|
|
The 24 adjacent repeat differences share their middle repetition within each
|
|
three-run group. They define a conservative empirical noise reference; they
|
|
are not used as 24 independent samples in an inferential test.
|
|
|
|
The result is `STOP_NO_PROSPECTIVE_RESPONSE`: zero features passed the response
|
|
gate at either horizon.
|
|
|
|
The strongest response was mean waiting-queue occupancy:
|
|
|
|
| Horizon | Sign consistency | Action/repeat median | Action above repeat p95 | Gate |
|
|
|---|---:|---:|---:|---|
|
|
| 5 s | 1.000 | 1.292x | 0.167 | fail |
|
|
| 10 s | 1.000 | 2.611x | 0.250 | fail |
|
|
|
|
The direction is real enough to merit diagnosis, but the effect is not broad
|
|
enough to guide a general action. It is large for TP4/high-load trials and
|
|
small or absent in other regimes.
|
|
|
|
Full-run transitions contain six beneficial actions (`false -> true`) and six
|
|
non-beneficial actions (three `false -> false`, three `true -> true`). The
|
|
beneficial label is also perfectly confounded with TP4 in this small dataset,
|
|
so it cannot support a topology-general claim.
|
|
|
|
| Horizon | Best telemetry delta | Balanced accuracy | Best external prefix delta | Balanced accuracy | Telemetry advantage |
|
|
|---|---|---:|---|---:|---:|
|
|
| 5 s | waiting queue | 0.750 | max TPOT / SLO | 0.833 | -0.083 |
|
|
| 10 s | waiting queue | 0.750 | outstanding / admitted | 0.750 | 0.000 |
|
|
|
|
No telemetry feature reaches the preregistered `+0.15` incremental threshold.
|
|
|
|
## What this rules out
|
|
|
|
It rules out using the current vector of 5/10-second global means as a solid
|
|
mechanism for choosing the next config. In particular, adding these aggregates
|
|
to an LLM prompt or fitting a larger predictor would currently hide, rather
|
|
than solve, the identifiability problem.
|
|
|
|
It does not rule out an instrumentation-aware tuner built around a deliberately
|
|
excited local system. The existing runs were designed for endpoint/fidelity
|
|
evaluation, not system identification: the MNS action is large, efficacy is
|
|
confounded with TP, repeat bands contain different requests, and global means
|
|
erase when queue buildup or service-rate changes occur.
|
|
|
|
## Required redesign before spending H20-hours
|
|
|
|
The next admissible experiment is a randomized, local A/B system-identification
|
|
pilot around one fixed TP and one load knee:
|
|
|
|
1. Replay the exact same request sequence and arrival times for both endpoints.
|
|
2. Use small adjacent actions and randomized `A/B` versus `B/A` order.
|
|
3. Record event-aligned response curves, including queue growth/drain rate,
|
|
prefill/decode service rate, and per-step service time, rather than only one
|
|
global mean.
|
|
4. Separate a mechanism gate (repeatable response) from the end-to-end gate:
|
|
fewer trials or H20-hours to select a feasible near-optimal config than an
|
|
outcome-only tuner.
|
|
5. Hold out a second load/workload for the final policy comparison.
|
|
|
|
Until that design is frozen, a wider sweep would only generate more correlated
|
|
observations and is not justified by the evidence above.
|
|
|
|
## Reproduction
|
|
|
|
```bash
|
|
python3 runs/intervention-response-v0/test_analysis.py
|
|
python3 runs/intervention-response-v0/test_p1_analysis.py
|
|
|
|
python3 runs/intervention-response-v0/analyze_phase6.py \
|
|
--metrics runs/opprof-phase6/phase6/metrics.json \
|
|
--raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
|
|
--output runs/intervention-response-v0/phase6-audit.json
|
|
|
|
python3 runs/intervention-response-v0/analyze_p1.py \
|
|
--run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
|
|
--manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
|
|
--output runs/intervention-response-v0/p1-audit.json
|
|
```
|
|
|
|
## Data sanity
|
|
|
|
- Phase 6: action pairs `n=17`, repeat pairs `n=29`, trials `n=66`; MNS
|
|
action size min/max `8/32`, `3` distinct; action-state vectors `n=17`, `17`
|
|
distinct; streams `n=12`, bytes min/max `12,745,297/52,957,710`, `12`
|
|
distinct.
|
|
- P1: action pairs `n=12`, repeat pairs `n=24`, trials `n=36`; MNS action
|
|
size min/max `48/56`, `2` distinct; efficacy labels `n=12`, min/max `0/1`,
|
|
`2` distinct; streams `n=6`, bytes min/max `17,449,143/29,431,988`, `6`
|
|
distinct.
|
|
- Checked invariants: exact action request hashes and offered loads match;
|
|
all `36/36` P1 trials match the manifest; expected pair counts hold; all
|
|
deltas are finite; non-negative counters and bounded ratios hold; per-config
|
|
state vectors are not all identical; both efficacy classes are present. No
|
|
red flags were observed.
|