Compare commits

...

3 Commits

27 changed files with 27675 additions and 1 deletions

View File

@@ -0,0 +1,71 @@
# Telemetry intervention-response v0 protocol
Status: **FROZEN BEFORE V0 ANALYSIS**.
Date: 2026-07-14 (Asia/Singapore).
## Claim boundary
The closed residual route asked whether one absolute engine-state snapshot can
predict unmeasured configurations. V0 asks a different, narrower question:
> Does an adjacent, controlled MNS intervention produce an early engine-state
> response that is distinguishable from same-config repeat noise?
Passing this gate only authorizes a matched real-GPU pilot. It does not prove
that telemetry improves tuning, that any metric is a causal mediator, or that
the response transfers to a new workload, topology, or knob family.
## Data and estimand
- Source: Phase 6 solo-authoritative Qwen3-30B-A3B/vLLM 0.24 Layer-1 streams.
- Action pairs: primary runs at identical study hash, TP, sampling anchor, and
request-order hash, with adjacent `MNS={8,16,32,64}` values.
- Noise pairs: primary versus confirmation at the same complete config,
anchor, and request-order hash. Only primary-to-confirmation pairs are used;
confirmations are not combined into pseudo-independent all-pairs.
- Fixed early windows: 5 seconds and 10 seconds from the measured interval
start. All runs exceed 10 seconds, so early-stop censoring cannot change the
telemetry window.
- Full-run pass rate and feasibility are descriptive only because an early
stop can make full elapsed durations differ.
The statistical unit is a run pair. Scheduler steps are summarized within a
run and are never counted as independent trials.
## Frozen response gate
The directly measured gate features are scheduler-step rate, decode-batch
mean, prefill-token fraction, waiting/running queue mean, KV-usage mean, and
CUDA-graph padding fraction.
A feature qualifies at one horizon only if:
1. at least 75% of nonzero action deltas have the same sign;
2. median absolute action delta is at least 2x the median absolute repeat
delta; and
3. at least 50% of action deltas exceed the repeat-noise absolute p95.
V0 opens a GPU pilot only if:
- there are exactly 17 frozen adjacent-MNS action pairs;
- there are at least 20 primary/confirmation repeat pairs;
- all identity, finite-value, counter, and ratio invariants pass; and
- at least two gate features qualify at both 5 and 10 seconds.
Any data red flag stops the analysis before interpreting the response.
## If V0 passes
Register a dash0 pilot around a known scaling knee. The pilot must use the
same request sequence and arrival times, one serving job at a time, one changed
knob, randomized `A/B` versus `B/A` order, common non-censored measurement
windows, and trial-level repetitions. It must compare a response-aware next
action against an outcome-only policy under complete startup, warm-up, and
H20-hour accounting.
## If V0 fails
Do not add telemetry fields or train a larger model. The current Layer-1 state
does not identify even an MNS intervention above repeat noise on this task, so
the telemetry-guided tuning route remains diagnostic only.

View File

@@ -0,0 +1,157 @@
# Telemetry intervention-response v0/v1 results
Date: 2026-07-14 (Asia/Singapore).
## Decision
**STOP before a new H20 pilot.** The current Layer-1 aggregate telemetry does
not identify a sufficiently general early response to an MNS intervention,
and it does not improve action-efficacy prediction over exact external prefix
outcomes on the available development tasks.
This is a negative result about the present state representation and
experiment design. It does not establish that engine telemetry is useless for
tuning, and it is not held-out evidence.
## Hypothesis and frozen test
The tested hypothesis was:
> With the workload and all non-MNS settings held fixed, increasing MNS causes
> a 5--10 second engine-state response that is larger than same-config repeat
> noise and that predicts whether the action makes the full run feasible.
A response feature had to satisfy all three frozen conditions at both 5 and
10 seconds: at least 0.75 sign consistency, median absolute action effect at
least 2x the repeat median, and at least 0.50 of action deltas above the repeat
absolute p95. At least two features had to pass. A telemetry feature was
decision-relevant only if its leave-one-repeat-out balanced accuracy was at
least 0.75 and at least 0.15 above the best exact external prefix outcome.
## What was implemented
- A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
- Exact action pairing with request-order hash, offered load, TP, load role,
and repetition held fixed.
- Same-config repeat-noise estimation without treating scheduler steps as
independent samples.
- Exact 5/10-second request-prefix outcomes using monotonic completion times.
- A one-feature leave-one-repeat-out efficacy audit; no multivariate model was
fitted to the 12 examples.
- Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity
invariants in machine-readable audit artifacts.
- Trial-by-trial validation against the P1 manifest, plus content hashes for
every result, request file, and Layer-1 stream.
## Experiment A: Phase-6 retrospective audit
Phase 6 supplied 17 adjacent-MNS actions and 29 same-config
primary/confirmation pairs. No feature passed at either horizon, producing
`STOP_NO_IDENTIFIABLE_RESPONSE`.
The confirmation sample is not a clean replication distribution: confirmations
were selectively run after disputed primary outcomes. Several same-config
pairs consequently followed radically different trajectories. This result
therefore remains a valid failure of the frozen v0 gate, but it cannot by itself
separate normal run variance from confirmation-selection bias.
## Experiment B: prospective-repeat confirmation
P1 supplied three pre-arranged, disjoint request bands for every cell/load.
Exact matched actions exist for TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at
low/high load and repetitions 1/2/3. This yields 12 action pairs and 24
same-config consecutive-repeat pairs.
The 24 adjacent repeat differences share their middle repetition within each
three-run group. They define a conservative empirical noise reference; they
are not used as 24 independent samples in an inferential test.
The result is `STOP_NO_PROSPECTIVE_RESPONSE`: zero features passed the response
gate at either horizon.
The strongest response was mean waiting-queue occupancy:
| Horizon | Sign consistency | Action/repeat median | Action above repeat p95 | Gate |
|---|---:|---:|---:|---|
| 5 s | 1.000 | 1.292x | 0.167 | fail |
| 10 s | 1.000 | 2.611x | 0.250 | fail |
The direction is real enough to merit diagnosis, but the effect is not broad
enough to guide a general action. It is large for TP4/high-load trials and
small or absent in other regimes.
Full-run transitions contain six beneficial actions (`false -> true`) and six
non-beneficial actions (three `false -> false`, three `true -> true`). The
beneficial label is also perfectly confounded with TP4 in this small dataset,
so it cannot support a topology-general claim.
| Horizon | Best telemetry delta | Balanced accuracy | Best external prefix delta | Balanced accuracy | Telemetry advantage |
|---|---|---:|---|---:|---:|
| 5 s | waiting queue | 0.750 | max TPOT / SLO | 0.833 | -0.083 |
| 10 s | waiting queue | 0.750 | outstanding / admitted | 0.750 | 0.000 |
No telemetry feature reaches the preregistered `+0.15` incremental threshold.
## What this rules out
It rules out using the current vector of 5/10-second global means as a solid
mechanism for choosing the next config. In particular, adding these aggregates
to an LLM prompt or fitting a larger predictor would currently hide, rather
than solve, the identifiability problem.
It does not rule out an instrumentation-aware tuner built around a deliberately
excited local system. The existing runs were designed for endpoint/fidelity
evaluation, not system identification: the MNS action is large, efficacy is
confounded with TP, repeat bands contain different requests, and global means
erase when queue buildup or service-rate changes occur.
## Required redesign before spending H20-hours
The next admissible experiment is a randomized, local A/B system-identification
pilot around one fixed TP and one load knee:
1. Replay the exact same request sequence and arrival times for both endpoints.
2. Use small adjacent actions and randomized `A/B` versus `B/A` order.
3. Record event-aligned response curves, including queue growth/drain rate,
prefill/decode service rate, and per-step service time, rather than only one
global mean.
4. Separate a mechanism gate (repeatable response) from the end-to-end gate:
fewer trials or H20-hours to select a feasible near-optimal config than an
outcome-only tuner.
5. Hold out a second load/workload for the final policy comparison.
Until that design is frozen, a wider sweep would only generate more correlated
observations and is not justified by the evidence above.
## Reproduction
```bash
python3 runs/intervention-response-v0/test_analysis.py
python3 runs/intervention-response-v0/test_p1_analysis.py
python3 runs/intervention-response-v0/analyze_phase6.py \
--metrics runs/opprof-phase6/phase6/metrics.json \
--raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
--output runs/intervention-response-v0/phase6-audit.json
python3 runs/intervention-response-v0/analyze_p1.py \
--run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
--manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
--output runs/intervention-response-v0/p1-audit.json
```
## Data sanity
- Phase 6: action pairs `n=17`, repeat pairs `n=29`, trials `n=66`; MNS
action size min/max `8/32`, `3` distinct; action-state vectors `n=17`, `17`
distinct; streams `n=12`, bytes min/max `12,745,297/52,957,710`, `12`
distinct.
- P1: action pairs `n=12`, repeat pairs `n=24`, trials `n=36`; MNS action
size min/max `48/56`, `2` distinct; efficacy labels `n=12`, min/max `0/1`,
`2` distinct; streams `n=6`, bytes min/max `17,449,143/29,431,988`, `6`
distinct.
- Checked invariants: exact action request hashes and offered loads match;
all `36/36` P1 trials match the manifest; expected pair counts hold; all
deltas are finite; non-negative counters and bounded ratios hold; per-config
state vectors are not all identical; both efficacy classes are present. No
red flags were observed.

View File

@@ -0,0 +1,58 @@
# Intervention-response v1 prospective-repeat confirmation
Status: **FROZEN AFTER PHASE-6 V0 FAILURE AND BEFORE P1 RESPONSE ANALYSIS**.
Date: 2026-07-14 (Asia/Singapore).
## Why this is a new confirmation, not a relaxed V0
Phase-6 V0 failed its frozen global response gate. Its 29 same-config
confirmations were triggered after disputed outcomes, and the resulting noise
sample contains extreme trajectory divergence by construction. V0 remains
failed and its thresholds are unchanged.
The already-completed P1 campaign supplies a distinct test: three
prospectively scheduled, disjoint repeat bands for every cell/load. TP1 and
TP4 use identical offered loads and exact request-order hashes across their MNS
endpoints. V1 asks whether an MNS response is identifiable against this
prospective workload-repeat noise, and whether that response predicts action
efficacy beyond exact external prefix outcomes.
P1 is now development data. No result here is held-out or paper-facing.
## Frozen pairs
- Action pairs: TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at low/high load and
repeat 1/2/3. Endpoints must have identical TP, offered rate, repeat role,
and request-order hash. Expected `n=12`.
- Repeat-noise pairs: consecutive pre-arranged repeat bands within each of six
cells and low/high load: `rep1 -> rep2`, `rep2 -> rep3`. Expected `n=24`.
Repeat bands intentionally contain different requests and therefore include
workload-sampling noise rather than pretending to be identical trials.
Adjacent differences share the middle run; the gate uses their empirical
magnitude only and does not treat the 24 differences as independent samples
for a p-value or confidence interval.
- Prefix horizons: 5 and 10 seconds. Exact monotonic request completion times
and the same Layer-1 intervals are used.
## Frozen gates
The response-identifiability thresholds are exactly the Phase-6 V0 thresholds:
75% sign consistency, 2x median effect/repeat noise, and at least 50% of action
deltas above repeat absolute p95. At least two response features must qualify
at both horizons.
Action efficacy is one only for an infeasible-to-feasible full-run transition.
The 12 action pairs must contain at least four examples of each class.
For decision relevance, each individual external-outcome response feature and
each individual telemetry-response feature is evaluated by leave-one-repeat-
band-out threshold fitting. This intentionally avoids a multivariate model on
12 examples. At least one telemetry feature must, at both horizons:
1. reach balanced accuracy at least 0.75; and
2. exceed the best external-outcome response feature by at least 0.15.
Only if data validity, response identifiability, and incremental decision
relevance all pass does V1 open a newly registered matched GPU pilot. No
threshold or feature is changed after observing V1.

View File

@@ -0,0 +1,84 @@
# Phase-aware telemetry intervention-response v2 protocol
Status: **FROZEN BEFORE THE SYSTEMATIC 10% DECILE AUDIT**.
Date: 2026-07-14 (Asia/Singapore).
## Correction to v0/v1
The 5/10-second analyses tested an ultra-early verifier. They did not test
whether telemetry observed after the engine has developed queue, batch, and KV
state can guide tuning. The P1 replay lasts 60 seconds after time scaling, and
the 5/10-second prefixes contain only a small fraction of its requests.
V2 therefore replaces absolute cutoffs with replay phase. The old audits and
their negative decisions remain immutable, but their claim is narrowed to the
first 5/10 seconds.
## Historical corrective audit
The historical audit is development-only and cannot become confirmatory after
the horizon concern was observed.
- Infer each trial's intended replay duration as selected requests divided by
offered requests per second. All trials must agree.
- Find every complete 10% replay decile supported by every trial. Analyze all
such deciles; selecting only the best horizon is forbidden.
- At each decile report both:
- cumulative state from replay start to the checkpoint; and
- the non-overlapping 10%-wide state block ending at the checkpoint.
- Report admitted/completed request coverage, response-versus-repeat statistics,
telemetry versus external-outcome efficacy, and per-feature trajectory drift.
- Reuse the frozen v1 action and repeat pairs and the frozen response and
incremental-efficacy thresholds. These thresholds are descriptive in V2;
passing one post-hoc horizon does not open a contribution claim.
If early stopping prevents complete observation of the replay phases, the
historical decision is `REQUIRES_UNCENSORED_PHASE_AWARE_PILOT`, independent of
which early decile looks best.
## Uncensored matched pilot
The pilot is a mechanism gate, not paper evidence.
- Hardware/engine/model: solo placement on dash0, 4 NVIDIA H20 GPUs, patched
vLLM `0.24.1.dev3+opprof`, Qwen3-30B-A3B, fixed `TP=4`.
- Action: `MNS 16 -> 64`; topology, model, engine build, workload, arrival
sequence, offered load, and all other settings remain fixed.
- Workload: `chat_w20260312_1000` at replay-time scale `0.5`, hence 300 seconds.
- Offered loads per GPU: `1.5`, `2.125`, and `3.125` requests/s. These supply a
low control and the two already-observed P1 pressure regimes.
- Repetitions: three disjoint session bands, exact request sequence matched
across action endpoints. Endpoint order alternates `A/B`, `B/A`, `A/B`;
load order is counter-rotated across repetitions.
- A fresh server receives the accepted long-request warm-up and a bounded
burn-in before each measured session.
- SLO-unrecoverable early stop is disabled. Every run must observe the full
300-second arrival window; a separate 360-second safety deadline may mark a
run invalid but cannot manufacture a full-run label.
- Cumulative checkpoints: 10%, 25%, 50%, 75%, and 100%, or 30/75/150/225/300
seconds. Quarter blocks are analyzed separately from cumulative means.
- Placement is serialized. Co-location remains forbidden because Phase 6
observed material co-location-induced outcome shifts.
- Hard cap: 8 H20-hours including startup, warm-up, burn-in, invalid attempts,
and cleanup.
## Gates
Data validity requires complete 300-second Layer-1 coverage, zero dropped
records, exact request/arrival/length hashes across action endpoints, monotonic
timestamps, full request accounting, idle GPUs before and after each session,
and no co-resident GPU process.
Mechanism evidence requires at least two telemetry features whose matched action
response exceeds same-config repeat noise at two consecutive checkpoints under
the unchanged v1 response thresholds. The same features must have a consistent
direction in at least two of the three load regimes.
Decision evidence additionally requires both action-efficacy classes and an
instrumentation feature that reaches leave-one-repetition-out balanced accuracy
at least 0.75 and exceeds the best external prefix outcome by at least 0.15.
Without label balance the pilot can adjudicate mechanism evidence only.
No H20 run is launched if the local analyzer/tests, manifest preflight, GPU
probe, command dry-run, projected cost, or cleanup plan fails.

View File

@@ -0,0 +1,286 @@
# Telemetry-conditioned residual tuning roadmap
Status: **R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL**.
Date: 2026-07-14 (Asia/Singapore).
## Research question and claim boundary
The question is whether a small number of real engine observations can correct
a simulator's task-specific error over **unmeasured configurations**, and
whether that correction reduces the real-GPU cost of finding a high
SLO-goodput serving configuration.
The intended headline claim, if the evidence supports it, is:
> An engine-state-conditioned residual model turns a simulator prediction into
> a task-specific posterior over unmeasured serving configurations, allowing a
> sequential tuner to reach near-oracle SLO-goodput with materially fewer
> H20-hours than simulator-only and outcome-only tuning.
Classification accuracy, simulator-error diagnosis, and telemetry overhead are
supporting evidence. None is an end-to-end tuning contribution by itself.
The following method is closed and will not be revived under another name:
per-candidate five-second accept/reject as the headline contribution. The P1
result showed only 1.426% cost reduction in the frozen `k=2` workflow.
## Two models, one evaluation
Both branches use the same legal candidate set, real measurements, task split,
cost accounting, and acquisition function.
### Simulator-residual branch (primary)
For measured anchor `c_t` and unmeasured candidate `c'`:
```text
y_hat(c') = y_real(c_t)
+ [y_sim(c') - y_sim(c_t)]
+ f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO)
```
The simulator delta is the prior. The learned model may correct it only with
training-supported state/config transitions; uncertainty or distribution shift
must shrink the correction back toward the simulator prior.
### Telemetry-only branch (mandatory)
```text
y_hat(c') = y_real(c_t)
+ g(state_real(c_t), c' - c_t, workload, SLO)
```
This branch tests whether the simulator is actually necessary. It does not
use a hand-authored bottleneck-to-knob rule.
### Search policy
Legal configurations are enumerated independently of telemetry. A generic
cost-aware acquisition rule ranks candidates from predicted improvement,
uncertainty, and measured H20 cost. The current production harness's
bottleneck scores, topology-first ordering, and hand-set relief constants are
not consumed by either branch. The validator may enforce legality,
full-config no-repeat, failure accounting, and resource caps only.
## Hypotheses
| ID | Hypothesis | Direct test | Failure meaning |
|---|---|---|---|
| H0 | Existing artifacts can express a common, direct-measurement state without heuristic labels. | Engine/simulator extractor coverage and invariants. | Route is not currently implementable. |
| H1 | Simulator errors are predictable from engine/simulator state discrepancy at measured anchors. | Task-held-out pairwise inversion correction and new-inversion rate. | Telemetry is diagnostic but cannot correct the surface. |
| H2 | Telemetry alone predicts useful config transitions beyond outcome-only history. | Telemetry-only versus real-outcome-only sequential replay. | Direct telemetry-guided tuning has no independent value. |
| H3 | Residual correction changes actual tuning decisions and cost. | H20-hours to 95% oracle and regret AUC against the strongest safe baseline. | No system contribution even if H1/H2 prediction metrics improve. |
## Common-state contract
Only directly observed or exactly reconstructed quantities are admitted.
| Quantity | vLLM Layer-1 | Frontier | R0 status |
|---|---|---|---|
| Scheduled requests / batch size | Per scheduler step | Existing per-batch metric, disabled in P1 output | Common after CPU replay |
| Scheduled prefill/decode tokens | Per scheduler step | Existing per-batch metrics | Common after CPU replay |
| Scheduler/batch rate | Monotonic step timestamps | Batch count / simulated duration | Common after CPU replay |
| Waiting queue area | Time-weighted queue gauge | Sum of request waiting times | Common aggregate |
| Running request area | Time-weighted running gauge | Sum of E2E minus waiting time | Common aggregate, semantics audited |
| Preemption count | Per step | Per request | Common |
| KV usage/headroom | Exact blocks and ratio | Not in committed output | Engine-only until exact reconstruction exists |
| CUDA graph mode/padding | Exact per step | Not modeled | Engine-only omitted-mechanism signal |
| Request TTFT/TPOT/pass rate | Exact real outcomes | Exact simulated request metrics | Common outcome, not state |
Unavailable fields remain null. They cannot be imputed from a human
`prefill/decode/queueing` label.
Frontier already contains the required detailed batch and timestamped
stage-batch ledger output. P1 disabled it for artifact size. R0 replays the
same immutable fixtures with the existing output flags enabled; it does not
change the simulator model or calibration.
## Data separation
- Phase 6 / `chat_w20260311_1000`: development only.
- P1 / `chat_w20260312_1000`: development only.
- R1 / `chat_w20260313_1000`: new development surface.
- R2: trace windows not used for feature, model, threshold, candidate-space,
cutoff, or acquisition decisions.
- Splits are by complete workload/SLO task. Anchor- or pair-level random
splits are prohibited.
- Sequential-policy seeds measure algorithmic variability; they are not
counted as independent system tasks.
The two existing development tasks have an important limitation: the now-
available SLO-gated simulator reading already retains the real oracle at its
top rank/tie. They therefore cannot establish a positive end-to-end ranking
claim. They are used for plumbing, known false-feasible cases, and negative
evidence. R1 must be run as an unbiased complete surface, not selected after
observing simulator success or failure.
## Step-by-step roadmap
### R0.1 — Inventory and roadmap
Deliverables:
- this roadmap;
- rolling untracked `ONGOING.md`;
- exact engine/simulator field and artifact inventory.
Gate: every claimed input has an authoritative file path and provenance.
### R0.2 — Common-state plumbing
Deliverables:
- `runs/telemetry-residual/common_state.py`;
- synthetic correctness tests;
- one exact P1 Frontier replay with individual batch metrics and the full
stage-batch ledger enabled;
- paired engine/simulator state summary for the same fixture.
Gate:
- replay request count and SLO scorer exactly agree with the committed replay;
- batch/ledger outputs are non-empty;
- all counters are non-negative, ratios bounded, times monotonic;
- no GPU is visible to Frontier;
- output volume is practical before expanding to twelve replays.
### R0.3 — Development residual/headroom audit
Use all frozen P1 primary fixtures and corresponding engine intervals. Produce:
- common-state residuals per anchor;
- simulator-error labels and continuous SLO/goodput residuals;
- ordered source/target diagnostic that removes both config identities from
both roles in every training fold;
- oracle upper bound for cross-candidate correction;
- explicit comparison with simulator+outcome and telemetry-only features.
R0 is a feasibility gate, not headline evidence. Proceed to R1 only if:
1. state features are collected with the measured source anchor, vary across
cells, and are available before any target config is evaluated;
2. at least one known simulator error has a state discrepancy not exposed by
the matched external prefix outcome;
3. a prior-preserving model can correct development errors without introducing
a larger number of new errors under regularization sensitivity;
4. an oracle cross-candidate correction has at least 15% sequential tuning-cost
headroom under full startup/warm-up accounting.
### R0 result and decision
R0 completed without a data-validity red flag, but failed condition 3. The
decision is **STOP_BEFORE_R1**; no H20 job was launched for this route.
- All 12 detailed Frontier CPU replays exactly reproduced their committed SLO
scorers. Runtime was 23.943--54.786 seconds per replay, detailed artifacts
were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures.
- The paired surface contains 12 real/sim anchors, two known simulator
false-feasible anchors, and 120 legal cross-config ordered transitions. A
fold removes both the source and target TP/MNS identity from source and
target roles; the two offered-load anchors remain part of the same task.
- Raw Frontier feasibility is 83.33% on the repeated transition view. The
structurally correct hybrid model uses
`r_target = r_source + delta_r`; the direct model uses
`y_target = y_source + delta_y` and never reads simulator fields.
- Direct telemetry is not robust relative to real-outcome-only: its accuracy
delta over L2 `{0.1,1,10,100}` is `{-0.83,+1.67,0,-4.17}` percentage points,
and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%.
- Hybrid telemetry raises classification accuracy over the corresponding
simulator+outcome transition regression by 1.67--4.17 percentage points,
but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125. Its full
correction reaches only 46.67--53.33% absolute accuracy.
- Across 24 nonzero `(L2, raw-simulator-prior weight)` combinations, no model
both corrects an existing simulator error without more new errors and avoids
worsening RMSE/MAE. Whenever a correction fixes at least one error, it
corrupts at least 11 previously correct transitions.
- A perfect correction could skip the frozen simulator rank-2 real final and
save 0.043469 H20-hours: 15.45% of the prospective online `k=2` cost, or
14.40% when the prior failed launch is charged. On this development task the
simulator top-1 already is the real oracle with zero regret, so headroom
versus the observed-safe top-1 baseline is 0%.
The result does not prove that engine telemetry is useless. It shows that the
current one-task anchor-transition evidence cannot support either a safe
simulator-residual tuner or a simulator-free telemetry tuner. A larger model
or an R1 run would add capacity/data after a failed gate and is therefore not
authorized under this roadmap.
### R1 — New development surface
Status: **NOT LAUNCHED; CLOSED BY R0**.
Frozen starting setup:
- host: dash0, eight NVIDIA H20 GPUs;
- cells run solo; no co-location for SLO verdicts;
- patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16;
- trace: `chat_w20260313_1000`;
- output tokens: exactly 128;
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95;
- config surface: TP `{1,2,4}` × MNS `{8,16,32,64}`;
- hard campaign cap: 4 H20-hours.
The load ladder, repetitions, randomized order, exact commands, expected wall
time, and artifact paths are frozen only after R0. A resolved echo is required
before launch.
R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour
headroom over the strongest safe baseline with final regret at most 5%. R1 is
development evidence and cannot be reported as the held-out result.
### R2 — Held-out sequential tuning
Status: **NOT LAUNCHED; CLOSED BY R0**.
Required baselines:
1. random search;
2. real-outcome-only Bayesian/sequential search;
3. Frontier ranking plus real top-k final;
4. simulator plus real-outcome residual;
5. telemetry-only transition tuner;
6. simulator plus telemetry residual tuner;
7. complete real surface as oracle, not as a cost competitor.
Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface
SLO-goodput oracle. Secondary metrics are cost-normalized regret AUC, final
regret at fixed budgets, oracle false-prune, wall time, and per-task regressions.
The route is successful only if the winning telemetry method reduces the
primary cost by at least 20% versus the strongest safe baseline and ends within
5% regret on every headline task. If hybrid beats telemetry-only by at least
10%, simulator residual correction is the primary method. If telemetry-only
is within 5% or better, the simulator dependency is removed. If neither clears
the contribution bar, the route is closed and telemetry remains a diagnostic
facility only.
## Cost discipline
- R0 simulator work is CPU-only and must set empty CUDA visibility.
- R1 cannot exceed 4 H20-hours.
- R2 receives no budget until R1 passes.
- Startup, warm-up, burn-in, failed launches, real probes, continuation, and
final validation are charged. Benchmark-only annotation repeats are
reported separately and cannot disappear from campaign accounting.
## Final R0 sanity block
| Data | n | Min | Max | Distinct | Checked invariant |
|---|---:|---:|---:|---:|---|
| Phase 6 cells | 12 | TP1/MNS8 | TP4/MNS64 | 12 | Surface not identical; solo SLO tier authoritative |
| Phase 6 Layer-1 primary steps | 37 streams | 343 | 12,103 | 37 | Contiguous; zero drops |
| P1 primary anchors | 12 | infeasible | feasible | 2 labels | 7 feasible / 5 infeasible |
| P1 Frontier runtime | 12 | 24.093 s | 54.575 s | 12 | CPU-only; zero failures |
| Detailed Frontier replay runtime | 12 | 23.943 s | 54.786 s | 12 | Exact committed scorers; CUDA hidden |
| Detailed artifact bytes | 12 | 4,123,724 | 13,527,776 | 12 | Non-negative; practical CPU replay size |
| Cross-config transitions | 120 | real pass 0.1067 | real pass 1.0 | 6 outcomes | Both endpoint config identities held out |
| State residual vectors | 12 | 16 fields | 16 fields | 12 vectors | Finite; no missing common field |
| R0 E2E cost values | 4 | 0.237914 | 0.301935 H20-h | 4 | Non-negative; `k=1/2`, online/conservative |
Checked invariants: non-negative counts and costs; pass rates in `[0,1]`;
simulator results not all identical; exact request count/hash agreement; Layer-1
step continuity and zero drops; no co-resident SLO measurements; no calibration
or evaluation split reuse for a future headline claim. No current red flag
invalidates R0 plumbing. The R0 tuning gate itself failed because safe
prior-preserving correction was absent.

View File

@@ -0,0 +1,691 @@
#!/usr/bin/env python3
"""Prospective-repeat confirmation of the intervention-response hypothesis.
P1 contains three pre-arranged, disjoint request bands per cell/load. TP1 and
TP4 use matched offered loads and request sequences across their MNS endpoints.
This script asks both whether the MNS response exceeds prospective repeat noise
and whether an early telemetry delta predicts full-run action efficacy beyond
the corresponding external-outcome delta.
"""
from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import math
import re
import sys
from collections import defaultdict
from pathlib import Path
from statistics import fmean
from typing import Any, Iterable, Mapping
HERE = Path(__file__).resolve().parent
COMMON_STATE_DIR = HERE.parent / "telemetry-residual"
sys.path.insert(0, str(COMMON_STATE_DIR))
from common_state import load_jsonl, summarize_engine # noqa: E402
def _load_v0():
spec = importlib.util.spec_from_file_location(
"intervention_response_phase6_v0", HERE / "analyze_phase6.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
V0 = _load_v0()
SCHEMA = "intervention-response-p1-confirmation-v1"
HORIZONS_S = V0.HORIZONS_S
EXPECTED_ACTION_PAIRS = 12
EXPECTED_REPEAT_PAIRS = 24
MIN_EFFICACY_CLASS = 4
MIN_EFFICACY_BALANCED_ACCURACY = 0.75
MIN_EFFICACY_DELTA_OVER_OUTCOME = 0.15
OUTCOME_FEATURES = (
"admitted_fraction",
"completed_over_admitted",
"completed_pass_rate",
"completed_fail_fraction_of_total",
"outstanding_over_admitted",
"ttft_max_over_slo_max",
"ttft_mean_over_slo_max",
"tpot_max_over_slo",
"tpot_mean_over_slo",
"admitted_input_tokens_mean_over_limit",
)
RUN_PATTERN = re.compile(r"^(low|high)-rep([123])$")
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def _prefix_outcome(
result: Mapping[str, Any],
requests: list[dict[str, Any]],
horizon_s: float,
) -> dict[str, float]:
admitted = [request for request in requests if float(request["arrival_s"]) <= horizon_s]
completed = [
request
for request in requests
if request.get("completed_elapsed_s") is not None
and float(request["completed_elapsed_s"]) <= horizon_s
]
if not admitted:
raise ValueError("prefix contains no admitted request")
admitted_ids = {str(request["request_id"]) for request in admitted}
if any(str(request["request_id"]) not in admitted_ids for request in completed):
raise ValueError("completed request was not admitted in the prefix")
passed = sum(bool(request["slo_pass"]) for request in completed)
ttft = [float(request["ttft_ms"]) for request in completed]
tpot = [float(request["tpot_ms"]) for request in completed]
total = int(result["selection"]["count"])
if total != len(requests):
raise ValueError("request JSONL count does not match the result")
return {
"admitted_fraction": len(admitted) / total,
"completed_over_admitted": len(completed) / len(admitted),
"completed_pass_rate": passed / max(1, len(completed)),
"completed_fail_fraction_of_total": (len(completed) - passed) / total,
"outstanding_over_admitted": (len(admitted) - len(completed)) / len(admitted),
"ttft_max_over_slo_max": max(ttft, default=0.0) / 6000.0,
"ttft_mean_over_slo_max": fmean(ttft) / 6000.0 if ttft else 0.0,
"tpot_max_over_slo": max(tpot, default=0.0) / 50.0,
"tpot_mean_over_slo": fmean(tpot) / 50.0 if tpot else 0.0,
"admitted_input_tokens_mean_over_limit": fmean(
float(request["raw_input_tokens"]) for request in admitted
)
/ 8192.0,
}
def load_trials(
run_root: Path,
*,
horizons_s: tuple[float, ...] = HORIZONS_S,
) -> tuple[dict[float, list[dict[str, Any]]], list[dict[str, Any]]]:
by_horizon = {horizon: [] for horizon in horizons_s}
streams = []
for cell_dir in sorted((run_root / "cells").iterdir()):
if not cell_dir.is_dir():
continue
stream_paths = sorted((cell_dir / "opprof").glob("*.jsonl"))
if len(stream_paths) != 1:
raise ValueError(f"{cell_dir}: expected one Layer-1 stream")
stream_path = stream_paths[0]
stream = load_jsonl(stream_path)
streams.append(
{
"path": str(stream_path.resolve()),
"sha256": sha256_file(stream_path),
"bytes": stream_path.stat().st_size,
}
)
for run_dir in sorted(cell_dir.iterdir()):
match = RUN_PATTERN.match(run_dir.name)
if match is None:
continue
level, replicate_text = match.groups()
replicate = int(replicate_text)
result_path = run_dir / "result.json"
requests_path = run_dir / "requests.jsonl"
result = json.loads(result_path.read_text(encoding="utf-8"))
requests = load_jsonl(requests_path)
elapsed_s = float(result["interval"]["elapsed_s"])
start_ns = int(result["interval"]["start_mono_ns"])
for horizon_s in horizons_s:
if elapsed_s < horizon_s:
raise ValueError(
f"{result_path}: elapsed {elapsed_s} shorter than {horizon_s}s"
)
state = V0.flatten_state(
summarize_engine(
stream,
start_ns=start_ns,
end_ns=start_ns + int(horizon_s * 1e9),
request_count=int(result["selection"]["count"]),
)
)
by_horizon[horizon_s].append(
{
"trial_id": str(result_path.relative_to(run_root)),
"cell": str(result["cell"]),
"tp": int(result["tp"]),
"mns": int(result["mns"]),
"level": level,
"replicate": replicate,
"offered_rate_per_gpu": float(
result["selection"]["offered_req_s_per_gpu"]
),
"request_hash": str(
result["selection"]["request_id_order_sha256"]
),
"request_count": int(result["selection"]["count"]),
"result_sha256": sha256_file(result_path),
"requests_sha256": sha256_file(requests_path),
"full_pass_rate": float(result["pass_rate"]),
"full_feasible": bool(result["feasible"]),
"early_stopped": bool(result["early_stopped"]),
"state": state,
"outcome": _prefix_outcome(result, requests, horizon_s),
}
)
return by_horizon, streams
def validate_manifest(
trials: list[dict[str, Any]], manifest_path: Path
) -> dict[str, Any]:
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
if manifest.get("schema") != "fidelity-prefix-pilot-manifest-v1":
raise ValueError("unexpected P1 manifest schema")
cells = manifest.get("cells")
if not isinstance(cells, dict):
raise ValueError("P1 manifest has no cell mapping")
seen = set()
for trial in trials:
key = (trial["cell"], trial["level"], trial["replicate"])
if key in seen:
raise ValueError(f"duplicate P1 trial identity: {key}")
seen.add(key)
try:
cell = cells[trial["cell"]]
selection = cell["targets"][trial["level"]]["selections"][
f"{trial['level']}{trial['replicate']}"
]
except (KeyError, TypeError) as error:
raise ValueError(f"trial is absent from P1 manifest: {key}") from error
if int(cell["tp"]) != trial["tp"] or int(cell["mns"]) != trial["mns"]:
raise ValueError(f"trial config disagrees with P1 manifest: {key}")
if str(selection["request_id_order_sha256"]) != trial["request_hash"]:
raise ValueError(f"trial request hash disagrees with P1 manifest: {key}")
if int(selection["selected_count"]) != trial["request_count"]:
raise ValueError(f"trial request count disagrees with P1 manifest: {key}")
if not math.isclose(
float(selection["offered_req_s_per_gpu"]),
trial["offered_rate_per_gpu"],
rel_tol=0.0,
abs_tol=1e-12,
):
raise ValueError(f"trial offered load disagrees with P1 manifest: {key}")
expected = {
(cell_name, level, replicate)
for cell_name in cells
for level in ("low", "high")
for replicate in (1, 2, 3)
}
if seen != expected:
missing = sorted(expected - seen)
unexpected = sorted(seen - expected)
raise ValueError(
f"P1 trial/manifest coverage mismatch: missing={missing}, "
f"unexpected={unexpected}"
)
return {
"schema": str(manifest["schema"]),
"expected_trials": len(expected),
"matched_trials": len(seen),
}
def _delta(
source: Mapping[str, Any],
target: Mapping[str, Any],
features: Iterable[str],
) -> dict[str, float]:
return {
feature: float(target[feature]) - float(source[feature])
for feature in features
}
def _action_pair(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, Any]:
if source["tp"] != target["tp"]:
raise ValueError("action endpoints changed TP")
if source["level"] != target["level"] or source["replicate"] != target["replicate"]:
raise ValueError("action endpoints changed load role or repeat")
if source["request_hash"] != target["request_hash"]:
raise ValueError("action endpoints changed request sequence")
if not math.isclose(
source["offered_rate_per_gpu"],
target["offered_rate_per_gpu"],
rel_tol=0.0,
abs_tol=1e-12,
):
raise ValueError("action endpoints changed offered load")
if source["mns"] >= target["mns"]:
raise ValueError("action must increase MNS")
beneficial = target["full_feasible"] and not source["full_feasible"]
return {
"kind": "matched_mns_increase",
"group": {
"tp": source["tp"],
"level": source["level"],
"replicate": source["replicate"],
"request_hash": source["request_hash"],
"offered_rate_per_gpu": source["offered_rate_per_gpu"],
},
"source": {
key: source[key]
for key in (
"trial_id",
"result_sha256",
"requests_sha256",
"cell",
"mns",
"full_pass_rate",
"full_feasible",
"early_stopped",
)
},
"target": {
key: target[key]
for key in (
"trial_id",
"result_sha256",
"requests_sha256",
"cell",
"mns",
"full_pass_rate",
"full_feasible",
"early_stopped",
)
},
"delta_state": _delta(source["state"], target["state"], V0.ALL_FEATURES),
"delta_outcome": _delta(source["outcome"], target["outcome"], OUTCOME_FEATURES),
"full_action_efficacy": int(beneficial),
"full_feasibility_transition": (
f"{str(source['full_feasible']).lower()}->"
f"{str(target['full_feasible']).lower()}"
),
}
def _repeat_pair(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, Any]:
if source["cell"] != target["cell"] or source["level"] != target["level"]:
raise ValueError("repeat endpoints changed config or load role")
if target["replicate"] != source["replicate"] + 1:
raise ValueError("repeat endpoints are not consecutive pre-arranged bands")
if not math.isclose(
source["offered_rate_per_gpu"],
target["offered_rate_per_gpu"],
rel_tol=0.0,
abs_tol=1e-12,
):
raise ValueError("repeat endpoints changed offered load")
return {
"kind": "same_config_workload_repeat",
"group": {
"cell": source["cell"],
"tp": source["tp"],
"mns": source["mns"],
"level": source["level"],
"source_replicate": source["replicate"],
"target_replicate": target["replicate"],
},
"source": {
key: source[key]
for key in ("trial_id", "result_sha256", "requests_sha256")
},
"target": {
key: target[key]
for key in ("trial_id", "result_sha256", "requests_sha256")
},
"delta_state": _delta(source["state"], target["state"], V0.ALL_FEATURES),
"delta_outcome": _delta(source["outcome"], target["outcome"], OUTCOME_FEATURES),
}
def build_pairs(
trials: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
action_groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
repeat_groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
for trial in trials:
action_groups[
(
trial["tp"],
trial["level"],
trial["replicate"],
trial["request_hash"],
trial["offered_rate_per_gpu"],
)
].append(trial)
repeat_groups[(trial["cell"], trial["level"])].append(trial)
actions = []
for group in action_groups.values():
if len(group) != 2:
continue
source, target = sorted(group, key=lambda trial: trial["mns"])
actions.append(_action_pair(source, target))
repeats = []
for group in repeat_groups.values():
ordered = sorted(group, key=lambda trial: trial["replicate"])
if len(ordered) != 3:
raise ValueError("each prospective repeat group must contain three runs")
repeats.extend(
_repeat_pair(source, target)
for source, target in zip(ordered, ordered[1:], strict=False)
)
return actions, repeats
def _balanced_accuracy(labels: list[int], predictions: list[int]) -> float:
positive = [prediction for label, prediction in zip(labels, predictions) if label == 1]
negative = [prediction for label, prediction in zip(labels, predictions) if label == 0]
if not positive or not negative:
raise ValueError("balanced accuracy requires both classes")
sensitivity = sum(prediction == 1 for prediction in positive) / len(positive)
specificity = sum(prediction == 0 for prediction in negative) / len(negative)
return (sensitivity + specificity) / 2.0
def _threshold_candidates(values: list[float]) -> list[float]:
unique = sorted(set(values))
if len(unique) == 1:
return [unique[0] - 1.0, unique[0], unique[0] + 1.0]
scale = max(1.0, max(abs(value) for value in unique))
candidates = [unique[0] - scale * 1e-6]
candidates.extend(
(left + right) / 2.0
for left, right in zip(unique, unique[1:], strict=False)
)
candidates.append(unique[-1] + scale * 1e-6)
return candidates
def _fit_threshold(values: list[float], labels: list[int]) -> tuple[float, int, float]:
best: tuple[float, int, float, float] | None = None
for threshold in _threshold_candidates(values):
for direction in (-1, 1):
predictions = [int(direction * (value - threshold) >= 0.0) for value in values]
balanced = _balanced_accuracy(labels, predictions)
accuracy = sum(
prediction == label
for prediction, label in zip(predictions, labels, strict=True)
) / len(labels)
candidate = (balanced, accuracy, -abs(threshold), float(direction))
if best is None or candidate > best:
best = candidate
selected_threshold = threshold
selected_direction = direction
assert best is not None
return selected_threshold, selected_direction, best[0]
def one_feature_leave_repeat_out(
actions: list[dict[str, Any]],
*,
delta_key: str,
features: tuple[str, ...],
) -> dict[str, Any]:
labels = [int(pair["full_action_efficacy"]) for pair in actions]
results = {}
for feature in features:
predictions = []
held_out_labels = []
folds = []
for held_out in (1, 2, 3):
train = [pair for pair in actions if pair["group"]["replicate"] != held_out]
test = [pair for pair in actions if pair["group"]["replicate"] == held_out]
train_values = [float(pair[delta_key][feature]) for pair in train]
train_labels = [int(pair["full_action_efficacy"]) for pair in train]
threshold, direction, train_balanced = _fit_threshold(
train_values, train_labels
)
test_values = [float(pair[delta_key][feature]) for pair in test]
test_predictions = [
int(direction * (value - threshold) >= 0.0) for value in test_values
]
test_labels = [int(pair["full_action_efficacy"]) for pair in test]
predictions.extend(test_predictions)
held_out_labels.extend(test_labels)
folds.append(
{
"held_out_replicate": held_out,
"threshold": threshold,
"direction": direction,
"train_balanced_accuracy": train_balanced,
"test_labels": test_labels,
"test_predictions": test_predictions,
}
)
balanced = _balanced_accuracy(held_out_labels, predictions)
accuracy = sum(
prediction == label
for prediction, label in zip(predictions, held_out_labels, strict=True)
) / len(held_out_labels)
results[feature] = {
"balanced_accuracy": balanced,
"accuracy": accuracy,
"folds": folds,
}
best_feature = max(
results,
key=lambda feature: (
results[feature]["balanced_accuracy"],
results[feature]["accuracy"],
feature,
),
)
return {
"labels": V0.numeric(labels),
"positive": sum(labels),
"negative": len(labels) - sum(labels),
"features": results,
"best_feature": best_feature,
"best_balanced_accuracy": results[best_feature]["balanced_accuracy"],
"best_accuracy": results[best_feature]["accuracy"],
}
def analyze_horizon(trials: list[dict[str, Any]], horizon_s: float) -> dict[str, Any]:
actions, repeats = build_pairs(trials)
response = V0.response_statistics(actions, repeats)
qualifying_response = sorted(
feature for feature, item in response.items() if item["qualifies"]
)
outcome_cv = one_feature_leave_repeat_out(
actions,
delta_key="delta_outcome",
features=OUTCOME_FEATURES,
)
telemetry_cv = one_feature_leave_repeat_out(
actions,
delta_key="delta_state",
features=V0.GATE_FEATURES,
)
outcome_best = float(outcome_cv["best_balanced_accuracy"])
efficacy_qualifying = sorted(
feature
for feature, item in telemetry_cv["features"].items()
if item["balanced_accuracy"] >= MIN_EFFICACY_BALANCED_ACCURACY
and item["balanced_accuracy"]
>= outcome_best + MIN_EFFICACY_DELTA_OVER_OUTCOME
)
action_hashes_match = all(
pair["group"]["request_hash"] for pair in actions
)
labels = [int(pair["full_action_efficacy"]) for pair in actions]
invariants = {
"expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
"expected_repeat_pair_count": len(repeats) == EXPECTED_REPEAT_PAIRS,
"matched_action_request_hashes": action_hashes_match,
"efficacy_label_balance": (
sum(labels) >= MIN_EFFICACY_CLASS
and len(labels) - sum(labels) >= MIN_EFFICACY_CLASS
),
"finite_deltas": all(
math.isfinite(value)
for pair in [*actions, *repeats]
for values in (pair["delta_state"], pair["delta_outcome"])
for value in values.values()
),
"probabilities_bounded": all(
0.0 <= trial["outcome"][feature] <= 1.0
for trial in trials
for feature in (
"admitted_fraction",
"completed_over_admitted",
"completed_pass_rate",
"completed_fail_fraction_of_total",
"outstanding_over_admitted",
"admitted_input_tokens_mean_over_limit",
)
),
}
red_flags = [name for name, passed in invariants.items() if not passed]
transitions = defaultdict(int)
for pair in actions:
transitions[pair["full_feasibility_transition"]] += 1
return {
"horizon_s": horizon_s,
"actions": actions,
"repeats": repeats,
"response_statistics": response,
"qualifying_response_features": qualifying_response,
"efficacy": {
"outcome_delta": outcome_cv,
"telemetry_delta": telemetry_cv,
"telemetry_qualifying_features": efficacy_qualifying,
"minimum_balanced_accuracy": MIN_EFFICACY_BALANCED_ACCURACY,
"minimum_delta_over_best_outcome": MIN_EFFICACY_DELTA_OVER_OUTCOME,
"feasibility_transitions": dict(sorted(transitions.items())),
},
"sanity": {
"trials": len(trials),
"action_pairs": len(actions),
"repeat_pairs": len(repeats),
"invariants": invariants,
"red_flags": red_flags,
},
}
def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
trials_by_horizon, streams = load_trials(run_root)
manifest_validation = validate_manifest(
trials_by_horizon[min(trials_by_horizon)], manifest_path
)
horizons = {
str(int(horizon)): analyze_horizon(trials, horizon)
for horizon, trials in sorted(trials_by_horizon.items())
}
red_flags = sorted(
{
flag
for horizon in horizons.values()
for flag in horizon["sanity"]["red_flags"]
}
)
stable_response = sorted(
set.intersection(
*(
set(horizon["qualifying_response_features"])
for horizon in horizons.values()
)
)
)
stable_efficacy = sorted(
set.intersection(
*(
set(horizon["efficacy"]["telemetry_qualifying_features"])
for horizon in horizons.values()
)
)
)
if red_flags:
decision = "STOP_DATA_INVALID"
elif len(stable_response) < V0.MIN_STABLE_FEATURES:
decision = "STOP_NO_PROSPECTIVE_RESPONSE"
elif not stable_efficacy:
decision = "STOP_NO_INCREMENTAL_TUNING_SIGNAL"
else:
decision = "OPEN_MATCHED_GPU_PILOT"
payload = {
"schema": SCHEMA,
"status": "COMPLETE",
"decision": decision,
"claim_boundary": (
"Development-only confirmation on an already-consumed P1 task. "
"Passing can open a newly registered matched pilot but cannot be "
"reported as held-out tuning evidence."
),
"frozen_gate": {
"response_thresholds_identical_to_phase6_v0": True,
"expected_action_pairs": EXPECTED_ACTION_PAIRS,
"expected_repeat_pairs": EXPECTED_REPEAT_PAIRS,
"minimum_stable_response_features": V0.MIN_STABLE_FEATURES,
"minimum_efficacy_class": MIN_EFFICACY_CLASS,
"minimum_efficacy_balanced_accuracy": MIN_EFFICACY_BALANCED_ACCURACY,
"minimum_efficacy_delta_over_best_outcome": (
MIN_EFFICACY_DELTA_OVER_OUTCOME
),
},
"stable_response_features": stable_response,
"stable_incremental_efficacy_features": stable_efficacy,
"horizons": horizons,
"provenance": {
"analysis_script": str(Path(__file__).resolve()),
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
"phase6_v0_script_sha256": sha256_file(HERE / "analyze_phase6.py"),
"run_root": str(run_root.resolve()),
"manifest": str(manifest_path.resolve()),
"manifest_sha256": sha256_file(manifest_path),
"manifest_validation": manifest_validation,
"streams": streams,
},
"sanity": {
"stream_count": len(streams),
"stream_bytes": V0.numeric(item["bytes"] for item in streams),
"red_flags": red_flags,
},
}
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
return payload
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--run-root", type=Path, required=True)
parser.add_argument("--manifest", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
payload = audit(
run_root=args.run_root,
manifest_path=args.manifest,
output_path=args.output,
)
print(
json.dumps(
{
"decision": payload["decision"],
"stable_response_features": payload["stable_response_features"],
"stable_incremental_efficacy_features": payload[
"stable_incremental_efficacy_features"
],
"sanity": payload["sanity"],
},
indent=2,
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,520 @@
#!/usr/bin/env python3
"""Audit whether a controlled knob change produces identifiable telemetry deltas.
This is a development-only feasibility audit. It compares adjacent MNS
interventions at an identical TP, offered-load anchor, and request sequence
against same-config primary/confirmation repeat noise. It does not claim that
the observed response is causal or that it improves an end-to-end tuner.
"""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import sys
from collections import defaultdict
from pathlib import Path
from statistics import median
from typing import Any, Iterable, Mapping
HERE = Path(__file__).resolve().parent
COMMON_STATE_DIR = HERE.parent / "telemetry-residual"
sys.path.insert(0, str(COMMON_STATE_DIR))
from common_state import load_jsonl, summarize_engine # noqa: E402
SCHEMA = "intervention-response-audit-v0"
HORIZONS_S = (5.0, 10.0)
GATE_FEATURES = (
"scheduler_steps_per_s",
"decode_batch_size.mean",
"prefill_token_fraction",
"queue_waiting_mean",
"queue_running_mean",
"kv_usage_mean",
"graph_padding_fraction",
)
ALL_FEATURES = (
"scheduler_steps_per_s",
"batch_size.mean",
"batch_tokens.mean",
"decode_batch_size.mean",
"prefill_token_fraction",
"queue_waiting_mean",
"queue_running_mean",
"preemptions",
"kv_usage_mean",
"kv_usage_max",
"kv_usage_end_minus_start",
"graph_none_share",
"graph_full_share",
"graph_padding_fraction",
)
EXPECTED_ACTION_PAIRS = 17
MIN_REPEAT_PAIRS = 20
MIN_STABLE_FEATURES = 2
MIN_SIGN_CONSISTENCY = 0.75
MIN_EFFECT_TO_NOISE = 2.0
MIN_ABOVE_NOISE_P95_FRACTION = 0.5
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def numeric(values: Iterable[float]) -> dict[str, Any]:
finite = [float(value) for value in values]
if not finite:
raise ValueError("numeric summary requires at least one value")
if any(not math.isfinite(value) for value in finite):
raise ValueError("numeric summary received a non-finite value")
return {
"n": len(finite),
"min": min(finite),
"max": max(finite),
"distinct_n": len(set(finite)),
}
def quantile(values: Iterable[float], probability: float) -> float:
ordered = sorted(float(value) for value in values)
if not ordered:
raise ValueError("quantile requires at least one value")
if not 0.0 <= probability <= 1.0:
raise ValueError("quantile probability must be in [0, 1]")
position = probability * (len(ordered) - 1)
lower = math.floor(position)
upper = math.ceil(position)
if lower == upper:
return ordered[lower]
weight = position - lower
return ordered[lower] * (1.0 - weight) + ordered[upper] * weight
def flatten_state(summary: Mapping[str, Any]) -> dict[str, float]:
common = summary["common"]
engine = summary["engine_only"]
state = {
"scheduler_steps_per_s": float(common["scheduler_steps_per_s"]),
"batch_size.mean": float(common["batch_size"]["mean"]),
"batch_tokens.mean": float(common["batch_tokens"]["mean"]),
"decode_batch_size.mean": float(common["decode_batch_size"]["mean"]),
"prefill_token_fraction": float(common["prefill_token_fraction"]),
"queue_waiting_mean": float(common["queue_waiting_mean"]),
"queue_running_mean": float(common["queue_running_mean"]),
"preemptions": float(common["preemptions"]),
"kv_usage_mean": float(engine["kv_usage_mean"]),
"kv_usage_max": float(engine["kv_usage_max"]),
"kv_usage_end_minus_start": float(engine["kv_usage_end_minus_start"]),
"graph_none_share": float(engine["graph_none_share"]),
"graph_full_share": float(engine["graph_full_share"]),
"graph_padding_fraction": float(engine["graph_padding_fraction"]),
}
if set(state) != set(ALL_FEATURES):
raise ValueError("flattened state does not match the frozen feature set")
if any(not math.isfinite(value) for value in state.values()):
raise ValueError("flattened state contains a non-finite value")
return state
def _trial_role(path: Path) -> str:
return "confirmation" if path.parent.name.startswith("confirm-") else "primary"
def load_trials(
raw_root: Path,
*,
horizons_s: tuple[float, ...] = HORIZONS_S,
) -> tuple[dict[float, list[dict[str, Any]]], list[dict[str, Any]]]:
by_horizon = {horizon: [] for horizon in horizons_s}
stream_provenance = []
for cell_dir in sorted(path for path in raw_root.iterdir() if path.is_dir()):
streams = sorted((cell_dir / "opprof").glob("*.jsonl"))
if len(streams) != 1:
raise ValueError(f"{cell_dir}: expected exactly one Layer-1 stream")
stream = streams[0]
records = load_jsonl(stream)
stream_provenance.append(
{
"path": str(stream),
"sha256": sha256_file(stream),
"bytes": stream.stat().st_size,
}
)
result_paths = sorted(cell_dir.glob("anchor-*/result.json"))
result_paths.extend(sorted(cell_dir.glob("confirm-*-anchor-*/result.json")))
for result_path in result_paths:
result = json.loads(result_path.read_text(encoding="utf-8"))
start_ns = int(result["interval"]["start_mono_ns"])
elapsed_s = float(result["interval"]["elapsed_s"])
for horizon_s in horizons_s:
if elapsed_s < horizon_s:
raise ValueError(
f"{result_path}: elapsed {elapsed_s} is shorter than {horizon_s}s"
)
state = flatten_state(
summarize_engine(
records,
start_ns=start_ns,
end_ns=start_ns + int(horizon_s * 1e9),
request_count=int(result["selection"]["count"]),
)
)
by_horizon[horizon_s].append(
{
"trial_id": str(result_path.relative_to(raw_root)),
"result_sha256": sha256_file(result_path),
"role": _trial_role(result_path),
"cell": str(result["cell"]),
"study_sha256": str(result["study_sha256"]),
"tp": int(result["tp"]),
"mns": int(result["mns"]),
"anchor": float(result["anchor"]),
"request_hash": str(
result["selection"]["request_id_order_sha256"]
),
"request_count": int(result["selection"]["count"]),
"early_stopped": bool(result["early_stopped"]),
"full_pass_rate": float(result["pass_rate"]),
"full_feasible": bool(result["feasible"]),
"state": state,
}
)
return by_horizon, stream_provenance
def _group_key(trial: Mapping[str, Any]) -> tuple[Any, ...]:
return (
trial["study_sha256"],
trial["tp"],
trial["anchor"],
trial["request_hash"],
)
def _delta(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, float]:
return {
feature: float(target["state"][feature]) - float(source["state"][feature])
for feature in ALL_FEATURES
}
def _pair(source: Mapping[str, Any], target: Mapping[str, Any], kind: str) -> dict[str, Any]:
if _group_key(source) != _group_key(target):
raise ValueError("pair endpoints do not share workload identity")
return {
"kind": kind,
"group": {
"study_sha256": source["study_sha256"],
"tp": source["tp"],
"anchor": source["anchor"],
"request_hash": source["request_hash"],
},
"source": {
"trial_id": source["trial_id"],
"cell": source["cell"],
"mns": source["mns"],
"early_stopped": source["early_stopped"],
"full_pass_rate": source["full_pass_rate"],
"full_feasible": source["full_feasible"],
},
"target": {
"trial_id": target["trial_id"],
"cell": target["cell"],
"mns": target["mns"],
"early_stopped": target["early_stopped"],
"full_pass_rate": target["full_pass_rate"],
"full_feasible": target["full_feasible"],
},
"delta_state": _delta(source, target),
"descriptive_full_outcome": {
"delta_pass_rate": target["full_pass_rate"] - source["full_pass_rate"],
"feasibility_transition": (
f"{str(source['full_feasible']).lower()}->"
f"{str(target['full_feasible']).lower()}"
),
},
}
def build_pairs(
trials: list[dict[str, Any]],
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
primary_groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
primary_by_cell_anchor: dict[tuple[Any, ...], dict[str, Any]] = {}
confirmations = []
for trial in trials:
if trial["role"] == "primary":
primary_groups[_group_key(trial)].append(trial)
primary_by_cell_anchor[
(trial["cell"], trial["anchor"], trial["request_hash"])
] = trial
else:
confirmations.append(trial)
actions = []
for group in primary_groups.values():
ordered = sorted(group, key=lambda item: item["mns"])
for source, target in zip(ordered, ordered[1:], strict=False):
if target["mns"] == source["mns"] * 2:
actions.append(_pair(source, target, "mns_increase"))
repeats = []
for confirmation in confirmations:
key = (
confirmation["cell"],
confirmation["anchor"],
confirmation["request_hash"],
)
primary = primary_by_cell_anchor.get(key)
if primary is None:
raise ValueError(f"{confirmation['trial_id']}: missing matched primary")
if primary["mns"] != confirmation["mns"]:
raise ValueError("repeat endpoints changed MNS")
repeats.append(_pair(primary, confirmation, "same_config_repeat"))
return actions, repeats
def response_statistics(
actions: list[dict[str, Any]],
repeats: list[dict[str, Any]],
) -> dict[str, Any]:
statistics = {}
for feature in ALL_FEATURES:
action = [float(pair["delta_state"][feature]) for pair in actions]
noise = [float(pair["delta_state"][feature]) for pair in repeats]
action_abs = [abs(value) for value in action]
noise_abs = [abs(value) for value in noise]
positive = sum(value > 1e-12 for value in action)
negative = sum(value < -1e-12 for value in action)
zero = len(action) - positive - negative
nonzero = positive + negative
sign_consistency = max(positive, negative) / nonzero if nonzero else 0.0
action_median = median(action_abs)
noise_median = median(noise_abs)
noise_p95 = quantile(noise_abs, 0.95)
effect_to_noise = (
action_median / noise_median
if noise_median > 0
else (math.inf if action_median > 0 else 0.0)
)
above_noise = sum(value > noise_p95 for value in action_abs) / len(action_abs)
qualifies = (
feature in GATE_FEATURES
and sign_consistency >= MIN_SIGN_CONSISTENCY
and effect_to_noise >= MIN_EFFECT_TO_NOISE
and above_noise >= MIN_ABOVE_NOISE_P95_FRACTION
)
statistics[feature] = {
"action_delta": numeric(action),
"repeat_delta": numeric(noise),
"action_abs_median": action_median,
"repeat_abs_median": noise_median,
"repeat_abs_p95": noise_p95,
"effect_to_repeat_median": (
effect_to_noise if math.isfinite(effect_to_noise) else None
),
"effect_to_repeat_median_is_infinite": math.isinf(effect_to_noise),
"action_signs": {
"positive": positive,
"negative": negative,
"zero": zero,
"consistency": sign_consistency,
},
"action_above_repeat_p95_fraction": above_noise,
"gate_feature": feature in GATE_FEATURES,
"qualifies": qualifies,
}
return statistics
def analyze_horizon(trials: list[dict[str, Any]], horizon_s: float) -> dict[str, Any]:
actions, repeats = build_pairs(trials)
feature_statistics = response_statistics(actions, repeats)
qualifying = sorted(
feature for feature, item in feature_statistics.items() if item["qualifies"]
)
all_values = [
value
for trial in trials
for value in trial["state"].values()
]
action_vectors = {
tuple(round(float(pair["delta_state"][feature]), 12) for feature in ALL_FEATURES)
for pair in actions
}
pair_invariants = {
"expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
"sufficient_repeat_pair_count": len(repeats) >= MIN_REPEAT_PAIRS,
"all_pair_hashes_match": all(
pair["group"]["request_hash"] for pair in [*actions, *repeats]
),
"all_values_finite": all(math.isfinite(value) for value in all_values),
"state_vectors_not_all_identical": len(action_vectors) > 1,
"ratios_bounded": all(
0.0 <= trial["state"][feature] <= 1.0
for trial in trials
for feature in (
"prefill_token_fraction",
"kv_usage_mean",
"kv_usage_max",
"graph_none_share",
"graph_full_share",
"graph_padding_fraction",
)
),
"nonnegative_counters": all(
trial["state"][feature] >= 0.0
for trial in trials
for feature in (
"scheduler_steps_per_s",
"batch_size.mean",
"batch_tokens.mean",
"decode_batch_size.mean",
"queue_waiting_mean",
"queue_running_mean",
"preemptions",
)
),
}
red_flags = [name for name, passed in pair_invariants.items() if not passed]
pass_deltas = [
pair["descriptive_full_outcome"]["delta_pass_rate"] for pair in actions
]
transitions = defaultdict(int)
for pair in actions:
transitions[pair["descriptive_full_outcome"]["feasibility_transition"]] += 1
return {
"horizon_s": horizon_s,
"actions": actions,
"repeats": repeats,
"feature_statistics": feature_statistics,
"qualifying_features": qualifying,
"descriptive_full_outcome": {
"delta_pass_rate": numeric(pass_deltas),
"positive": sum(value > 1e-12 for value in pass_deltas),
"negative": sum(value < -1e-12 for value in pass_deltas),
"zero": sum(abs(value) <= 1e-12 for value in pass_deltas),
"feasibility_transitions": dict(sorted(transitions.items())),
"limitation": (
"Full outcomes may use different elapsed durations when a trial "
"early-stopped; they are descriptive and are not a gate input."
),
},
"sanity": {
"trials": len(trials),
"action_pairs": len(actions),
"repeat_pairs": len(repeats),
"distinct_action_vectors": len(action_vectors),
"invariants": pair_invariants,
"red_flags": red_flags,
},
}
def audit(
*,
metrics_path: Path,
raw_root: Path,
output_path: Path,
) -> dict[str, Any]:
trials_by_horizon, streams = load_trials(raw_root)
horizons = {
str(int(horizon)): analyze_horizon(trials, horizon)
for horizon, trials in sorted(trials_by_horizon.items())
}
red_flags = sorted(
{
red_flag
for horizon in horizons.values()
for red_flag in horizon["sanity"]["red_flags"]
}
)
stable_features = sorted(
set.intersection(
*(set(horizon["qualifying_features"]) for horizon in horizons.values())
)
)
if red_flags:
decision = "STOP_DATA_INVALID"
elif len(stable_features) < MIN_STABLE_FEATURES:
decision = "STOP_NO_IDENTIFIABLE_RESPONSE"
else:
decision = "OPEN_MATCHED_PILOT"
payload = {
"schema": SCHEMA,
"status": "COMPLETE",
"decision": decision,
"claim_boundary": (
"Development-only identifiability gate. Passing opens a controlled "
"real-GPU pilot; it does not establish tuning benefit or causality."
),
"frozen_gate": {
"horizons_s": list(HORIZONS_S),
"expected_action_pairs": EXPECTED_ACTION_PAIRS,
"minimum_repeat_pairs": MIN_REPEAT_PAIRS,
"minimum_stable_features": MIN_STABLE_FEATURES,
"minimum_sign_consistency": MIN_SIGN_CONSISTENCY,
"minimum_effect_to_repeat_median": MIN_EFFECT_TO_NOISE,
"minimum_action_above_repeat_p95_fraction": (
MIN_ABOVE_NOISE_P95_FRACTION
),
"gate_features": list(GATE_FEATURES),
},
"stable_qualifying_features": stable_features,
"horizons": horizons,
"provenance": {
"analysis_script": str(Path(__file__).resolve()),
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
"phase6_metrics": str(metrics_path.resolve()),
"phase6_metrics_sha256": sha256_file(metrics_path),
"raw_root": str(raw_root.resolve()),
"streams": streams,
},
"sanity": {
"stream_count": len(streams),
"stream_bytes": numeric(item["bytes"] for item in streams),
"red_flags": red_flags,
},
}
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
return payload
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--metrics", type=Path, required=True)
parser.add_argument("--raw-root", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
payload = audit(
metrics_path=args.metrics,
raw_root=args.raw_root,
output_path=args.output,
)
print(
json.dumps(
{
"decision": payload["decision"],
"stable_qualifying_features": payload[
"stable_qualifying_features"
],
"sanity": payload["sanity"],
},
indent=2,
sort_keys=True,
)
)
if __name__ == "__main__":
main()

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,57 @@
#!/usr/bin/env python3
from __future__ import annotations
import importlib.util
import math
from pathlib import Path
HERE = Path(__file__).resolve().parent
def load_module():
spec = importlib.util.spec_from_file_location(
"intervention_response_v0", HERE / "analyze_phase6.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
def pair(module, delta: dict[str, float]) -> dict[str, object]:
state = {feature: 0.0 for feature in module.ALL_FEATURES}
state.update(delta)
return {"delta_state": state}
def main() -> None:
module = load_module()
assert module.numeric([0.0, 1.0, 1.0]) == {
"n": 3,
"min": 0.0,
"max": 1.0,
"distinct_n": 2,
}
assert math.isclose(module.quantile([0.0, 10.0], 0.95), 9.5)
actions = [
pair(module, {"queue_waiting_mean": -1.0 - 0.1 * index})
for index in range(8)
]
repeats = [
pair(module, {"queue_waiting_mean": 0.01 * ((index % 3) - 1)})
for index in range(20)
]
stats = module.response_statistics(actions, repeats)
waiting = stats["queue_waiting_mean"]
assert waiting["qualifies"]
assert waiting["action_signs"]["negative"] == 8
assert waiting["action_signs"]["consistency"] == 1.0
assert waiting["effect_to_repeat_median"] > 2.0
assert not stats["kv_usage_mean"]["qualifies"]
print("intervention response v0 analysis: PASS")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,35 @@
#!/usr/bin/env python3
from __future__ import annotations
import importlib.util
from pathlib import Path
HERE = Path(__file__).resolve().parent
def load_module():
spec = importlib.util.spec_from_file_location(
"intervention_response_p1", HERE / "analyze_p1.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
def main() -> None:
module = load_module()
values = [-2.0, -1.0, 1.0, 2.0]
labels = [0, 0, 1, 1]
threshold, direction, balanced = module._fit_threshold(values, labels)
assert direction == 1
assert -1.0 < threshold < 1.0
assert balanced == 1.0
assert module._balanced_accuracy(labels, labels) == 1.0
assert module._balanced_accuracy(labels, [1, 1, 0, 0]) == 0.0
print("intervention response P1 confirmation analysis: PASS")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,467 @@
#!/usr/bin/env python3
"""Audit telemetry responses over every uncensored replay decile.
This corrective analysis keeps the frozen P1 pairs and thresholds, but replaces
the absolute 5/10-second cutoff with cumulative and non-overlapping 10%-of-trace
windows. It deliberately reports every common decile instead of selecting the
best-looking horizon.
"""
from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import math
from pathlib import Path
from statistics import median
from typing import Any, Iterable, Mapping
HERE = Path(__file__).resolve().parent
P1_PATH = HERE.parent / "intervention-response-v0" / "analyze_p1.py"
SCHEMA = "intervention-response-phase-aware-existing-v2"
DECILE_FRACTION = 0.1
MAX_DECILES = 10
def _load_p1():
spec = importlib.util.spec_from_file_location("intervention_response_p1", P1_PATH)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
P1 = _load_p1()
def numeric(values: Iterable[float | int]) -> dict[str, Any]:
finite = [float(value) for value in values]
result = P1.V0.numeric(finite)
result["median"] = median(finite)
return result
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def trial_directories(run_root: Path) -> list[Path]:
result = []
for cell in sorted((run_root / "cells").iterdir()):
if not cell.is_dir():
continue
for candidate in sorted(cell.iterdir()):
if candidate.is_dir() and P1.RUN_PATTERN.match(candidate.name):
result.append(candidate)
if not result:
raise ValueError("P1 run root contains no measured trial directories")
return result
def load_metadata(run_root: Path) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
metadata = []
streams = []
for cell in sorted((run_root / "cells").iterdir()):
if not cell.is_dir():
continue
stream_paths = sorted((cell / "opprof").glob("*.jsonl"))
if len(stream_paths) != 1:
raise ValueError(f"{cell}: expected one Layer-1 stream")
stream_path = stream_paths[0]
streams.append(
{
"cell": cell.name,
"path": str(stream_path.resolve()),
"sha256": sha256_file(stream_path),
"bytes": stream_path.stat().st_size,
}
)
for run_dir in trial_directories(run_root):
match = P1.RUN_PATTERN.match(run_dir.name)
assert match is not None
level, replicate_text = match.groups()
result_path = run_dir / "result.json"
requests_path = run_dir / "requests.jsonl"
result = json.loads(result_path.read_text(encoding="utf-8"))
selected = int(result["selection"]["count"])
offered = float(result["selection"]["offered_req_s"])
if selected <= 0 or offered <= 0.0:
raise ValueError(f"{result_path}: invalid selected count or offered rate")
metadata.append(
{
"trial_id": str(result_path.relative_to(run_root)),
"cell": str(result["cell"]),
"tp": int(result["tp"]),
"mns": int(result["mns"]),
"level": level,
"replicate": int(replicate_text),
"elapsed_s": float(result["interval"]["elapsed_s"]),
"trace_duration_s": round(selected / offered, 9),
"early_stopped": bool(result["early_stopped"]),
"request_count": selected,
"result_sha256": sha256_file(result_path),
"requests_sha256": sha256_file(requests_path),
}
)
return metadata, streams
def common_decile_fractions(
*, trace_duration_s: float, minimum_elapsed_s: float
) -> tuple[float, ...]:
if trace_duration_s <= 0.0 or minimum_elapsed_s <= 0.0:
raise ValueError("trace duration and elapsed time must be positive")
supported = min(
MAX_DECILES,
int(math.floor((minimum_elapsed_s / trace_duration_s) * 10.0 + 1e-12)),
)
return tuple(
round(index * DECILE_FRACTION, 10) for index in range(1, supported + 1)
)
def _trial_record(
*,
run_root: Path,
run_dir: Path,
result: Mapping[str, Any],
state: dict[str, float],
outcome: dict[str, float],
) -> dict[str, Any]:
match = P1.RUN_PATTERN.match(run_dir.name)
assert match is not None
level, replicate_text = match.groups()
result_path = run_dir / "result.json"
requests_path = run_dir / "requests.jsonl"
return {
"trial_id": str(result_path.relative_to(run_root)),
"cell": str(result["cell"]),
"tp": int(result["tp"]),
"mns": int(result["mns"]),
"level": level,
"replicate": int(replicate_text),
"offered_rate_per_gpu": float(
result["selection"]["offered_req_s_per_gpu"]
),
"request_hash": str(result["selection"]["request_id_order_sha256"]),
"request_count": int(result["selection"]["count"]),
"result_sha256": sha256_file(result_path),
"requests_sha256": sha256_file(requests_path),
"full_pass_rate": float(result["pass_rate"]),
"full_feasible": bool(result["feasible"]),
"early_stopped": bool(result["early_stopped"]),
"state": state,
"outcome": outcome,
}
def load_interval_trials(
run_root: Path,
intervals_s: tuple[tuple[float, float], ...],
) -> tuple[dict[tuple[float, float], list[dict[str, Any]]], list[dict[str, Any]]]:
by_interval = {interval: [] for interval in intervals_s}
stream_provenance = []
for cell in sorted((run_root / "cells").iterdir()):
if not cell.is_dir():
continue
stream_paths = sorted((cell / "opprof").glob("*.jsonl"))
if len(stream_paths) != 1:
raise ValueError(f"{cell}: expected one Layer-1 stream")
stream_path = stream_paths[0]
stream = P1.load_jsonl(stream_path)
stream_provenance.append(
{
"cell": cell.name,
"path": str(stream_path.resolve()),
"sha256": sha256_file(stream_path),
"bytes": stream_path.stat().st_size,
}
)
for run_dir in sorted(cell.iterdir()):
if not run_dir.is_dir() or P1.RUN_PATTERN.match(run_dir.name) is None:
continue
result_path = run_dir / "result.json"
requests_path = run_dir / "requests.jsonl"
result = json.loads(result_path.read_text(encoding="utf-8"))
requests = P1.load_jsonl(requests_path)
start_ns = int(result["interval"]["start_mono_ns"])
elapsed_s = float(result["interval"]["elapsed_s"])
for interval in intervals_s:
start_s, end_s = interval
if start_s < 0.0 or end_s <= start_s:
raise ValueError(f"invalid analysis interval: {interval}")
if elapsed_s + 1e-9 < end_s:
raise ValueError(
f"{result_path}: elapsed {elapsed_s} shorter than {end_s}s"
)
state = P1.V0.flatten_state(
P1.summarize_engine(
stream,
start_ns=start_ns + int(start_s * 1e9),
end_ns=start_ns + int(end_s * 1e9),
request_count=int(result["selection"]["count"]),
)
)
outcome = P1._prefix_outcome(result, requests, end_s)
by_interval[interval].append(
_trial_record(
run_root=run_root,
run_dir=run_dir,
result=result,
state=state,
outcome=outcome,
)
)
return by_interval, stream_provenance
def coverage(trials: list[dict[str, Any]]) -> dict[str, Any]:
admitted = [float(trial["outcome"]["admitted_fraction"]) for trial in trials]
completed = [
float(trial["outcome"]["admitted_fraction"])
* float(trial["outcome"]["completed_over_admitted"])
for trial in trials
]
return {
"admitted_fraction_of_total": numeric(admitted),
"completed_fraction_of_total": numeric(completed),
}
def slim_window_analysis(
trials: list[dict[str, Any]], *, start_s: float, end_s: float, fraction: float
) -> dict[str, Any]:
analysis = P1.analyze_horizon(trials, end_s)
return {
"start_s": start_s,
"end_s": end_s,
"end_fraction": fraction,
"coverage_at_end": coverage(trials),
"action_pairs": len(analysis["actions"]),
"repeat_pairs": len(analysis["repeats"]),
"response_statistics": analysis["response_statistics"],
"qualifying_response_features": analysis["qualifying_response_features"],
"efficacy": analysis["efficacy"],
"sanity": analysis["sanity"],
}
def _pearson(left: list[float], right: list[float]) -> float | None:
if len(left) != len(right) or not left:
raise ValueError("Pearson inputs must be non-empty and have equal length")
left_mean = sum(left) / len(left)
right_mean = sum(right) / len(right)
numerator = sum(
(x - left_mean) * (y - right_mean)
for x, y in zip(left, right, strict=True)
)
left_ss = sum((x - left_mean) ** 2 for x in left)
right_ss = sum((y - right_mean) ** 2 for y in right)
if left_ss == 0.0 or right_ss == 0.0:
return None
return numerator / math.sqrt(left_ss * right_ss)
def trajectory_summary(
block_trials: list[tuple[tuple[float, float], list[dict[str, Any]]]]
) -> dict[str, Any]:
if not block_trials:
raise ValueError("trajectory requires at least one block")
identities = []
states_by_block = []
for interval, trials in block_trials:
ordered = sorted(
trials,
key=lambda trial: (trial["cell"], trial["level"], trial["replicate"]),
)
current_identities = [
(trial["cell"], trial["level"], trial["replicate"]) for trial in ordered
]
if identities and current_identities != identities:
raise ValueError("trajectory blocks do not contain identical trials")
identities = current_identities
states_by_block.append((interval, [trial["state"] for trial in ordered]))
features = {}
for feature in P1.V0.ALL_FEATURES:
block_values = [
[float(state[feature]) for state in states]
for _interval, states in states_by_block
]
first = block_values[0]
last = block_values[-1]
delta = [right - left for left, right in zip(first, last, strict=True)]
features[feature] = {
"block_medians": [median(values) for values in block_values],
"first_to_last_delta": numeric(delta),
"first_to_last_abs_delta": numeric(abs(value) for value in delta),
"first_to_last_pearson": _pearson(first, last),
"changed_trials": sum(abs(value) > 1e-12 for value in delta),
}
return {
"trial_count": len(identities),
"blocks": [
{"start_s": interval[0], "end_s": interval[1]}
for interval, _states in states_by_block
],
"features": features,
}
def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
metadata, metadata_streams = load_metadata(run_root)
durations = [float(item["trace_duration_s"]) for item in metadata]
elapsed = [float(item["elapsed_s"]) for item in metadata]
duration = median(durations)
deciles = common_decile_fractions(
trace_duration_s=duration, minimum_elapsed_s=min(elapsed)
)
if not deciles:
raise ValueError("no complete replay decile is shared by all trials")
cumulative_intervals = tuple(
(0.0, round(duration * fraction, 9)) for fraction in deciles
)
block_intervals = tuple(
(
round(duration * (fraction - DECILE_FRACTION), 9),
round(duration * fraction, 9),
)
for fraction in deciles
)
all_intervals = tuple(dict.fromkeys([*cumulative_intervals, *block_intervals]))
trials_by_interval, streams = load_interval_trials(run_root, all_intervals)
manifest_validation = P1.validate_manifest(
trials_by_interval[cumulative_intervals[0]], manifest_path
)
cumulative = []
blocks = []
for fraction, cumulative_interval, block_interval in zip(
deciles, cumulative_intervals, block_intervals, strict=True
):
cumulative.append(
slim_window_analysis(
trials_by_interval[cumulative_interval],
start_s=cumulative_interval[0],
end_s=cumulative_interval[1],
fraction=fraction,
)
)
blocks.append(
slim_window_analysis(
trials_by_interval[block_interval],
start_s=block_interval[0],
end_s=block_interval[1],
fraction=fraction,
)
)
invariants = {
"expected_trial_count": len(metadata) == 36,
"trace_duration_consistent": max(durations) - min(durations) <= 1e-9,
"all_intervals_uncensored": all(
item["elapsed_s"] + 1e-9 >= cumulative_intervals[-1][1]
for item in metadata
),
"stream_provenance_consistent": metadata_streams == streams,
"manifest_trials_match": (
manifest_validation["expected_trials"]
== manifest_validation["matched_trials"]
== len(metadata)
),
"all_window_sanity_pass": all(
not item["sanity"]["red_flags"] for item in [*cumulative, *blocks]
),
}
red_flags = [name for name, passed in invariants.items() if not passed]
complete_full_trajectory = min(elapsed) + 1e-9 >= duration
if red_flags:
decision = "STOP_DATA_INVALID"
elif not complete_full_trajectory:
decision = "REQUIRES_UNCENSORED_PHASE_AWARE_PILOT"
else:
decision = "FULL_TRAJECTORY_AVAILABLE"
payload = {
"schema": SCHEMA,
"status": "COMPLETE",
"decision": decision,
"claim_boundary": (
"Post-hoc corrective audit over every common replay decile. It can "
"diagnose horizon sensitivity but cannot establish a held-out tuning claim."
),
"design": {
"decile_fraction": DECILE_FRACTION,
"available_deciles": list(deciles),
"trace_duration_s": duration,
"maximum_common_end_s": cumulative_intervals[-1][1],
"maximum_common_fraction": deciles[-1],
"select_best_horizon": False,
"cumulative_and_nonoverlapping_blocks": True,
},
"cumulative": cumulative,
"blocks": blocks,
"trajectory": trajectory_summary(
[(interval, trials_by_interval[interval]) for interval in block_intervals]
),
"provenance": {
"analysis_script": str(Path(__file__).resolve()),
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
"p1_analysis_script": str(P1_PATH.resolve()),
"p1_analysis_script_sha256": sha256_file(P1_PATH),
"run_root": str(run_root.resolve()),
"manifest": str(manifest_path.resolve()),
"manifest_sha256": sha256_file(manifest_path),
"manifest_validation": manifest_validation,
"streams": streams,
"trial_inputs": metadata,
},
"sanity": {
"trials": len(metadata),
"elapsed_s": numeric(elapsed),
"trace_duration_s": numeric(durations),
"early_stopped": sum(bool(item["early_stopped"]) for item in metadata),
"request_count": numeric(item["request_count"] for item in metadata),
"stream_bytes": numeric(item["bytes"] for item in streams),
"invariants": invariants,
"red_flags": red_flags,
},
}
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
return payload
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--run-root", type=Path, required=True)
parser.add_argument("--manifest", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
payload = audit(
run_root=args.run_root,
manifest_path=args.manifest,
output_path=args.output,
)
print(
json.dumps(
{
"decision": payload["decision"],
"design": payload["design"],
"sanity": payload["sanity"],
},
indent=2,
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,449 @@
#!/usr/bin/env python3
"""Analyze the uncensored 300-second phase-aware matched pilot."""
from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import math
from collections import defaultdict
from pathlib import Path
from typing import Any, Iterable, Mapping
HERE = Path(__file__).resolve().parent
P1_PATH = HERE.parent / "intervention-response-v0" / "analyze_p1.py"
SCHEMA = "intervention-response-phase-aware-pilot-analysis-v2"
EXPECTED_ACTION_PAIRS = 9
EXPECTED_REPEAT_PAIRS = 12
MIN_EFFICACY_CLASS = 3
def _load_p1():
spec = importlib.util.spec_from_file_location("intervention_response_p1", P1_PATH)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
P1 = _load_p1()
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def numeric(values: Iterable[float | int]) -> dict[str, Any]:
return P1.V0.numeric(values)
def _trial_record(
*,
run_root: Path,
session: Mapping[str, Any],
level: str,
result: Mapping[str, Any],
result_path: Path,
requests_path: Path,
state: dict[str, float],
outcome: dict[str, float],
) -> dict[str, Any]:
return {
"trial_id": str(result_path.relative_to(run_root)),
"cell": str(result["cell"]),
"tp": int(result["tp"]),
"mns": int(result["mns"]),
"level": level,
"replicate": int(session["replicate"]),
"offered_rate_per_gpu": float(
result["selection"]["offered_req_s_per_gpu"]
),
"request_hash": str(result["selection"]["request_id_order_sha256"]),
"request_count": int(result["selection"]["count"]),
"result_sha256": sha256_file(result_path),
"requests_sha256": sha256_file(requests_path),
"full_pass_rate": float(result["pass_rate"]),
"full_feasible": bool(result["feasible"]),
"early_stopped": bool(result["early_stopped"]),
"state": state,
"outcome": outcome,
}
def validate_result_against_manifest(
*,
result: Mapping[str, Any],
selection: Mapping[str, Any],
session: Mapping[str, Any],
level: str,
expected_duration_s: float,
) -> None:
identity = f"{session['session']}:{level}"
if int(result["mns"]) != int(session["mns"]) or int(result["tp"]) != 4:
raise ValueError(f"config mismatch: {identity}")
if bool(result["early_stopped"]):
raise ValueError(f"early-stopped measured result: {identity}")
if result.get("slo_early_stop_disabled") is not True:
raise ValueError(f"SLO early stop was enabled: {identity}")
if float(result["interval"]["elapsed_s"]) + 1e-9 < expected_duration_s:
raise ValueError(f"result does not cover full arrival window: {identity}")
if int(result["selection"]["count"]) != int(selection["selected_count"]):
raise ValueError(f"selection count mismatch: {identity}")
for result_key, manifest_key in (
("request_id_order_sha256", "request_id_order_sha256"),
("arrival_order_sha256", "arrival_order_sha256"),
("raw_length_order_sha256", "input_length_order_sha256"),
):
if result["selection"][result_key] != selection[manifest_key]:
raise ValueError(f"selection hash mismatch {result_key}: {identity}")
if int(result["observed_count"]) != int(selection["selected_count"]):
raise ValueError(f"request accounting mismatch: {identity}")
def load_interval_trials(
*,
run_root: Path,
manifest: Mapping[str, Any],
intervals_s: tuple[tuple[float, float], ...],
) -> tuple[dict[tuple[float, float], list[dict[str, Any]]], list[dict[str, Any]]]:
by_interval = {interval: [] for interval in intervals_s}
streams = []
duration_s = float(manifest["engine"]["duration_s"])
for session in manifest["sessions"]:
session_root = run_root / "sessions" / str(session["session"])
stream_paths = sorted((session_root / "opprof").glob("*.jsonl"))
if len(stream_paths) != 1:
raise ValueError(f"{session_root}: expected one Layer-1 stream")
stream_path = stream_paths[0]
stream = P1.load_jsonl(stream_path)
streams.append(
{
"session": str(session["session"]),
"path": str(stream_path.resolve()),
"sha256": sha256_file(stream_path),
"bytes": stream_path.stat().st_size,
}
)
repetition = manifest["repetitions"][str(session["replicate"])]
for level, selection in repetition["selections"].items():
result_path = session_root / level / "result.json"
requests_path = session_root / level / "requests.jsonl"
result = json.loads(result_path.read_text(encoding="utf-8"))
requests = P1.load_jsonl(requests_path)
validate_result_against_manifest(
result=result,
selection=selection,
session=session,
level=level,
expected_duration_s=duration_s,
)
start_ns = int(result["interval"]["start_mono_ns"])
for interval in intervals_s:
start_s, end_s = interval
state = P1.V0.flatten_state(
P1.summarize_engine(
stream,
start_ns=start_ns + int(start_s * 1e9),
end_ns=start_ns + int(end_s * 1e9),
request_count=int(result["selection"]["count"]),
)
)
outcome = P1._prefix_outcome(result, requests, end_s)
by_interval[interval].append(
_trial_record(
run_root=run_root,
session=session,
level=level,
result=result,
result_path=result_path,
requests_path=requests_path,
state=state,
outcome=outcome,
)
)
return by_interval, streams
def analyze_window(
trials: list[dict[str, Any]], *, start_s: float, end_s: float, fraction: float
) -> dict[str, Any]:
actions, repeats = P1.build_pairs(trials)
response = P1.V0.response_statistics(actions, repeats)
response_qualifying = sorted(
feature for feature, item in response.items() if item["qualifies"]
)
labels = [int(pair["full_action_efficacy"]) for pair in actions]
cross_validation_possible = all(
set(
int(pair["full_action_efficacy"])
for pair in actions
if pair["group"]["replicate"] != held_out
)
== {0, 1}
for held_out in (1, 2, 3)
)
if cross_validation_possible:
outcome_cv = P1.one_feature_leave_repeat_out(
actions, delta_key="delta_outcome", features=P1.OUTCOME_FEATURES
)
telemetry_cv = P1.one_feature_leave_repeat_out(
actions, delta_key="delta_state", features=P1.V0.GATE_FEATURES
)
outcome_best = float(outcome_cv["best_balanced_accuracy"])
efficacy_qualifying = sorted(
feature
for feature, item in telemetry_cv["features"].items()
if item["balanced_accuracy"] >= P1.MIN_EFFICACY_BALANCED_ACCURACY
and item["balanced_accuracy"]
>= outcome_best + P1.MIN_EFFICACY_DELTA_OVER_OUTCOME
)
else:
unavailable = {
"status": "UNAVAILABLE",
"reason": "each leave-one-repetition-out train fold needs both classes",
}
outcome_cv = unavailable
telemetry_cv = unavailable
efficacy_qualifying = []
transitions = defaultdict(int)
for pair in actions:
transitions[pair["full_feasibility_transition"]] += 1
admitted = [float(trial["outcome"]["admitted_fraction"]) for trial in trials]
completed = [
float(trial["outcome"]["admitted_fraction"])
* float(trial["outcome"]["completed_over_admitted"])
for trial in trials
]
action_metadata = [
{
"group": pair["group"],
"source": pair["source"],
"target": pair["target"],
"full_action_efficacy": pair["full_action_efficacy"],
"full_feasibility_transition": pair["full_feasibility_transition"],
"delta_state": pair["delta_state"],
"delta_outcome": pair["delta_outcome"],
}
for pair in actions
]
invariants = {
"expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
"expected_repeat_pair_count": len(repeats) == EXPECTED_REPEAT_PAIRS,
"finite_deltas": all(
math.isfinite(value)
for pair in [*actions, *repeats]
for key in ("delta_state", "delta_outcome")
for value in pair[key].values()
),
"all_results_uncensored": all(not trial["early_stopped"] for trial in trials),
}
return {
"start_s": start_s,
"end_s": end_s,
"end_fraction": fraction,
"coverage_at_end": {
"admitted_fraction_of_total": numeric(admitted),
"completed_fraction_of_total": numeric(completed),
},
"action_pairs": len(actions),
"repeat_pairs": len(repeats),
"actions": action_metadata,
"response_statistics": response,
"qualifying_response_features": response_qualifying,
"efficacy": {
"labels": numeric(labels),
"positive": sum(labels),
"negative": len(labels) - sum(labels),
"label_balance_sufficient": (
sum(labels) >= MIN_EFFICACY_CLASS
and len(labels) - sum(labels) >= MIN_EFFICACY_CLASS
),
"cross_validation_possible": cross_validation_possible,
"transitions": dict(sorted(transitions.items())),
"outcome_delta": outcome_cv,
"telemetry_delta": telemetry_cv,
"telemetry_qualifying_features": efficacy_qualifying,
},
"sanity": {
"trials": len(trials),
"invariants": invariants,
"red_flags": [name for name, passed in invariants.items() if not passed],
},
}
def stable_adjacent_features(windows: list[dict[str, Any]]) -> dict[str, list[str]]:
result = {}
for left, right in zip(windows, windows[1:], strict=False):
key = f"{left['end_fraction']:.2f}->{right['end_fraction']:.2f}"
result[key] = sorted(
set(left["qualifying_response_features"])
& set(right["qualifying_response_features"])
)
return result
def consistent_load_regimes(
windows: list[dict[str, Any]], stable: dict[str, list[str]]
) -> dict[str, Any]:
by_end = {float(window["end_fraction"]): window for window in windows}
result = {}
for transition, features in stable.items():
_left_text, right_text = transition.split("->")
window = by_end[float(right_text)]
for feature in features:
deltas_by_level: dict[str, list[float]] = defaultdict(list)
all_deltas = []
for action in window["actions"]:
value = float(action["delta_state"][feature])
deltas_by_level[str(action["group"]["level"])].append(value)
all_deltas.append(value)
positive = sum(value > 1e-12 for value in all_deltas)
negative = sum(value < -1e-12 for value in all_deltas)
direction = 1 if positive >= negative else -1
consistent = []
for level, values in sorted(deltas_by_level.items()):
matching = sum(direction * value > 1e-12 for value in values)
nonzero = sum(abs(value) > 1e-12 for value in values)
if nonzero and matching / nonzero >= 2.0 / 3.0:
consistent.append(level)
result[f"{transition}:{feature}"] = {
"direction": direction,
"consistent_load_regimes": consistent,
"passes_two_regimes": len(consistent) >= 2,
}
return result
def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
if manifest.get("schema") != "intervention-response-phase-aware-pilot-manifest-v2":
raise ValueError("unexpected phase-aware pilot manifest schema")
fractions = [float(value) for value in manifest["checkpoints"]["fractions"]]
seconds = [float(value) for value in manifest["checkpoints"]["seconds"]]
cumulative_intervals = tuple((0.0, end_s) for end_s in seconds)
quarter_intervals = ((0.0, 75.0), (75.0, 150.0), (150.0, 225.0), (225.0, 300.0))
intervals = tuple(dict.fromkeys([*cumulative_intervals, *quarter_intervals]))
trials_by_interval, streams = load_interval_trials(
run_root=run_root, manifest=manifest, intervals_s=intervals
)
cumulative = [
analyze_window(
trials_by_interval[interval],
start_s=interval[0],
end_s=interval[1],
fraction=fraction,
)
for fraction, interval in zip(fractions, cumulative_intervals, strict=True)
]
quarter_blocks = [
analyze_window(
trials_by_interval[interval],
start_s=interval[0],
end_s=interval[1],
fraction=interval[1] / 300.0,
)
for interval in quarter_intervals
]
stable = stable_adjacent_features(cumulative)
load_consistency = consistent_load_regimes(cumulative, stable)
mechanism_features = sorted(
{
key.split(":", 1)[1]
for key, item in load_consistency.items()
if item["passes_two_regimes"]
}
)
full = cumulative[-1]
efficacy_features = sorted(
set.intersection(
*(
set(window["efficacy"]["telemetry_qualifying_features"])
for window in cumulative
if window["end_fraction"] >= 0.25
)
)
)
red_flags = sorted(
{
flag
for window in [*cumulative, *quarter_blocks]
for flag in window["sanity"]["red_flags"]
}
)
if red_flags:
decision = "STOP_DATA_INVALID"
elif not mechanism_features:
decision = "STOP_NO_PHASE_STABLE_RESPONSE"
elif not full["efficacy"]["label_balance_sufficient"]:
decision = "MECHANISM_ONLY_NO_LABEL_BALANCE"
elif not efficacy_features:
decision = "STOP_NO_INCREMENTAL_TUNING_SIGNAL"
else:
decision = "OPEN_E2E_POLICY_TEST"
payload = {
"schema": SCHEMA,
"status": "COMPLETE",
"decision": decision,
"claim_boundary": "Development mechanism pilot; not a held-out paper claim.",
"mechanism_features": mechanism_features,
"stable_adjacent_features": stable,
"load_consistency": load_consistency,
"stable_incremental_efficacy_features": efficacy_features,
"cumulative": cumulative,
"quarter_blocks": quarter_blocks,
"provenance": {
"analysis_script": str(Path(__file__).resolve()),
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
"manifest": str(manifest_path.resolve()),
"manifest_sha256": sha256_file(manifest_path),
"run_root": str(run_root.resolve()),
"streams": streams,
},
"sanity": {
"streams": len(streams),
"stream_bytes": numeric(item["bytes"] for item in streams),
"red_flags": red_flags,
},
}
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
return payload
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--run-root", type=Path, required=True)
parser.add_argument("--manifest", type=Path, required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
payload = audit(
run_root=args.run_root,
manifest_path=args.manifest,
output_path=args.output,
)
print(
json.dumps(
{
"decision": payload["decision"],
"mechanism_features": payload["mechanism_features"],
"stable_incremental_efficacy_features": payload[
"stable_incremental_efficacy_features"
],
"sanity": payload["sanity"],
},
indent=2,
sort_keys=True,
)
)
if __name__ == "__main__":
main()

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,471 @@
#!/usr/bin/env python3
"""Serialized, resumable controller for the 300-second phase-aware pilot."""
from __future__ import annotations
import argparse
import json
import os
import shlex
import signal
import subprocess
import sys
import time
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
PHASE6 = HERE.parent / "opprof-phase6"
sys.path.insert(0, str(PHASE6))
import opprof_phase6_controller as base # noqa: E402
SCHEMA = "intervention-response-phase-aware-pilot-state-v2"
SESSION_ESTIMATE_H20_HOURS = 1.25
SAFETY_H20_HOURS = 0.20
CLIENT_TIMEOUT_S = 450.0
def atomic_json(path: Path, payload: Any) -> None:
base.atomic_json(path, payload)
def wait_all_idle(timeout_s: float = 30.0) -> None:
deadline = time.monotonic() + timeout_s
last_error: Exception | None = None
while time.monotonic() < deadline:
try:
base.assert_all_idle()
return
except RuntimeError as error:
last_error = error
time.sleep(1.0)
raise last_error or RuntimeError("GPU idle timeout")
def configure(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
base.WORKDIR = args.run_root.parent
base.RUN_ROOT = args.run_root
base.STATE = args.run_root / "controller-state.json"
base.SOURCE = args.vllm_source
base.VENV = args.venv
base.AITUNER = args.aituner_root
base.MODEL = args.model
base.CLIENT = args.client
base.GPU_LIMIT = float(manifest["budget"]["hard_cap_h20_hours"])
base.MARKER = "intervention-response-phase-aware-v2"
base.CELLS = {
f"tp4_mns{mns}": {"tp": 4, "mns": int(mns)}
for mns in manifest["engine"]["mns_endpoints"]
}
def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
if path.exists():
return json.loads(path.read_text(encoding="utf-8"))
return {
"schema": SCHEMA,
"status": "initialized",
"hard_cap_h20_hours": hard_cap,
"gpu_hours_total": 0.0,
"completed_sessions": 0,
"sessions": {},
"failures": [],
"started_at": time.time(),
}
def save_state(path: Path, state: dict[str, Any]) -> None:
atomic_json(path, state)
def append_echo(run_root: Path, line: str) -> None:
run_root.mkdir(parents=True, exist_ok=True)
with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
target.write(line + "\n")
print(line, flush=True)
def remaining_projection(session_count: int, index: int) -> float:
return (session_count - index) * SESSION_ESTIMATE_H20_HOURS + SAFETY_H20_HOURS
def start_server(
*, session: dict[str, Any], index: int, run_root: Path
) -> dict[str, Any]:
cell = f"tp4_mns{int(session['mns'])}"
gpus = (0, 1, 2, 3)
session_root = run_root / "sessions" / str(session["session"])
session_root.mkdir(parents=True, exist_ok=True)
port = 8950 + index
command = base.server_command(cell, gpus, port)
with (session_root / "commands.log").open("a", encoding="utf-8") as log:
log.write(f"SERVER {shlex.join(command)}\n")
server_log = (session_root / "server.log").open("ab", buffering=0)
environment = os.environ.copy()
environment.update(
{
"CUDA_VISIBLE_DEVICES": "0,1,2,3",
"VLLM_OPPROF_DIR": str(session_root / "opprof"),
"OPPROF_PHASE6_MARKER": base.MARKER,
"AITUNER_ROOT": str(base.AITUNER),
"HF_HUB_OFFLINE": "1",
"TRANSFORMERS_OFFLINE": "1",
"PYTHONUNBUFFERED": "1",
}
)
server = subprocess.Popen(
command,
cwd=base.SOURCE,
env=environment,
stdout=server_log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
base.OWNED_PGIDS.add(server.pid)
return {
"cell": cell,
"gpus": gpus,
"port": port,
"dir": session_root,
"server": server,
"server_handle": server_log,
"spawned_at": time.time(),
"results": [],
}
def client_command(
entry: dict[str, Any],
*,
study: str,
anchor: float,
output: Path,
warmup: bool,
) -> list[str]:
config = base.CELLS[entry["cell"]]
command = [
"taskset",
"-c",
base.cpu_mask(entry["gpus"]),
str(base.VENV / "bin/python"),
str(base.CLIENT),
"warmup" if warmup else "run-anchor",
"--study",
study,
"--cell",
entry["cell"],
"--anchor",
str(anchor),
"--tp",
str(config["tp"]),
"--mns",
str(config["mns"]),
"--base-url",
f"http://127.0.0.1:{entry['port']}",
"--result-dir",
str(output),
"--disable-slo-early-stop",
]
return command
def run_client(
*,
entry: dict[str, Any],
role: str,
study: str,
selection: dict[str, Any],
output: Path,
state: dict[str, Any],
warmup: bool = False,
) -> dict[str, Any]:
command = client_command(
entry,
study=study,
anchor=float(selection["anchor"]),
output=output,
warmup=warmup,
)
with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
log.write(f"CLIENT role={role} {shlex.join(command)}\n")
handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
environment = os.environ.copy()
environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
process = subprocess.Popen(
command,
cwd=base.WORKDIR,
env=environment,
stdout=handle,
stderr=subprocess.STDOUT,
start_new_session=True,
)
deadline = time.monotonic() + (180.0 if warmup else CLIENT_TIMEOUT_S)
try:
while process.poll() is None:
if time.monotonic() > deadline:
raise TimeoutError(f"client timeout: {entry['cell']} {role}")
if entry["server"].poll() is not None:
raise RuntimeError(f"server exited during {entry['cell']} {role}")
base.assert_no_other_compute()
if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
raise RuntimeError("phase-aware pilot H20-hour hard cap reached")
time.sleep(1.0)
except Exception:
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
pass
try:
process.wait(timeout=10.0)
except subprocess.TimeoutExpired:
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
pass
process.wait(timeout=10.0)
raise
finally:
handle.close()
if process.returncode:
raise RuntimeError(
f"client failed: cell={entry['cell']} role={role} rc={process.returncode}"
)
result = json.loads((output / "result.json").read_text(encoding="utf-8"))
validate_result(
result=result,
selection=selection,
role=role,
warmup=warmup,
)
entry["results"].append(
{
"anchor": float(selection["anchor"]),
"dir": str(output),
"kind": result["kind"],
}
)
return result
def validate_result(
*, result: dict[str, Any], selection: dict[str, Any], role: str, warmup: bool
) -> None:
if result.get("slo_early_stop_disabled") is not True:
raise RuntimeError(f"SLO early stop was not disabled: {role}")
if warmup:
if result["kind"] != "warmup" or int(result["selection"]["count"]) != 16:
raise RuntimeError(f"invalid warmup result: {role}")
if not all(
result["invariants"].get(key, False)
for key in ("warmup_16", "warmup_exact_16", "warmup_long")
):
raise RuntimeError(f"warmup invariant failed: {role}")
return
if bool(result["early_stopped"]):
raise RuntimeError(f"uncensored run early-stopped: {role}")
if int(result["selection"]["count"]) != int(selection["selected_count"]):
raise RuntimeError(f"selection count mismatch: {role}")
for key, manifest_key in (
("request_id_order_sha256", "request_id_order_sha256"),
("arrival_order_sha256", "arrival_order_sha256"),
("raw_length_order_sha256", "input_length_order_sha256"),
):
if result["selection"][key] != selection[manifest_key]:
raise RuntimeError(f"selection hash mismatch {key}: {role}")
if int(result["observed_count"]) != int(selection["selected_count"]):
raise RuntimeError(f"request accounting mismatch: {role}")
def execute_session(
*,
index: int,
session: dict[str, Any],
manifest: dict[str, Any],
run_root: Path,
state_path: Path,
state: dict[str, Any],
) -> None:
name = str(session["session"])
if state["sessions"].get(name, {}).get("status") == "complete":
return
projection = remaining_projection(len(manifest["sessions"]), index)
if state["gpu_hours_total"] + projection > base.GPU_LIMIT:
state["status"] = "budget_projection_stop"
state["budget_stop"] = {
"before_session": name,
"spent_h20_hours": state["gpu_hours_total"],
"remaining_projection_h20_hours": projection,
"hard_cap_h20_hours": base.GPU_LIMIT,
}
save_state(state_path, state)
raise RuntimeError(f"projected pilot cost exceeds hard cap before {name}")
replicate = str(session["replicate"])
repetition = manifest["repetitions"][replicate]
echo = (
f"PHASE_PILOT_SESSION_ECHO session={name} tp=4 mns={session['mns']} "
f"gpus=0-3 workload={manifest['source']['window_id']} duration_s=300 "
f"loads={','.join(repetition['load_order'])} disable_slo_early_stop=true "
f"spent_h20h={state['gpu_hours_total']:.6f} "
f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f} "
f"manifest={run_root / 'pilot-manifest.json'}"
)
append_echo(run_root, echo)
wait_all_idle()
session_state = {
"status": "starting",
"replicate": int(replicate),
"mns": int(session["mns"]),
"started_at": time.time(),
"runs": [],
}
state["status"] = "running"
state["sessions"][name] = session_state
save_state(state_path, state)
entry = start_server(session=session, index=index, run_root=run_root)
failure: Exception | None = None
try:
base.wait_ready(entry)
high = repetition["selections"]["high"]
session_state["status"] = "warmup"
save_state(state_path, state)
run_client(
entry=entry,
role="warmup",
study=repetition["study"],
selection=high,
output=entry["dir"] / "warmup",
state=state,
warmup=True,
)
session_state["status"] = "burnin"
save_state(state_path, state)
burnin = manifest["burnin"]
burnin_result = run_client(
entry=entry,
role="burnin",
study=burnin["study"],
selection=burnin,
output=entry["dir"] / "burnin",
state=state,
)
session_state["burnin"] = {
"pass_rate": burnin_result["pass_rate"],
"feasible": burnin_result["feasible"],
"elapsed_s": burnin_result["interval"]["elapsed_s"],
}
session_state["status"] = "measured"
save_state(state_path, state)
for level in repetition["load_order"]:
selection = repetition["selections"][level]
result = run_client(
entry=entry,
role=level,
study=repetition["study"],
selection=selection,
output=entry["dir"] / level,
state=state,
)
session_state["runs"].append(
{
"level": level,
"selected_count": selection["selected_count"],
"offered_req_s_per_gpu": selection["offered_req_s_per_gpu"],
"pass_rate": result["pass_rate"],
"feasible": result["feasible"],
"elapsed_s": result["interval"]["elapsed_s"],
"early_stopped": result["early_stopped"],
}
)
save_state(state_path, state)
session_state["status"] = "stopping"
save_state(state_path, state)
except Exception as error: # noqa: BLE001
failure = error
finally:
try:
base.stop_entry(entry)
except Exception as error: # noqa: BLE001
failure = failure or error
time.sleep(2.0)
try:
wait_all_idle()
except Exception as error: # noqa: BLE001
failure = failure or error
session_hours = base.live_gpu_hours([entry])
state["gpu_hours_total"] += session_hours
session_state["gpu_hours"] = session_hours
if failure is not None:
session_state["status"] = "failed"
session_state["failure"] = repr(failure)
state["status"] = "failed"
state["failures"].append({"session": name, "failure": repr(failure)})
save_state(state_path, state)
raise failure
validation = base.validate_cell(entry)
session_state["validation"] = validation
session_state["status"] = "complete"
session_state["completed_at"] = time.time()
state["completed_sessions"] += 1
save_state(state_path, state)
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser()
result.add_argument("--manifest", type=Path, required=True)
result.add_argument("--run-root", type=Path, required=True)
result.add_argument("--aituner-root", type=Path, required=True)
result.add_argument("--vllm-source", type=Path, required=True)
result.add_argument("--venv", type=Path, required=True)
result.add_argument("--model", type=Path, required=True)
result.add_argument("--client", type=Path, required=True)
return result
def main() -> None:
args = parser().parse_args()
manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
if manifest.get("schema") != "intervention-response-phase-aware-pilot-manifest-v2":
raise RuntimeError("unexpected phase-aware pilot manifest schema")
if manifest["status"] != "PASS":
raise RuntimeError("phase-aware pilot manifest did not pass preflight")
args.run_root.mkdir(parents=True, exist_ok=True)
copied_manifest = args.run_root / "pilot-manifest.json"
if not copied_manifest.exists():
atomic_json(copied_manifest, manifest)
configure(args, manifest)
state_path = args.run_root / "controller-state.json"
state = load_state(state_path, base.GPU_LIMIT)
state["status"] = "running"
save_state(state_path, state)
for index, session in enumerate(manifest["sessions"]):
execute_session(
index=index,
session=session,
manifest=manifest,
run_root=args.run_root,
state_path=state_path,
state=state,
)
state["status"] = "complete"
state["completed_at"] = time.time()
save_state(state_path, state)
wait_all_idle()
print(
json.dumps(
{
"status": state["status"],
"completed_sessions": state["completed_sessions"],
"gpu_hours_total": state["gpu_hours_total"],
},
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,267 @@
#!/usr/bin/env python3
"""Prepare the uncensored 300-second TP4 matched pilot on dash0."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import os
import sys
from pathlib import Path
from typing import Any
AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
sys.path.insert(0, str(AITUNER_ROOT / "src"))
from aituner.spec import load_study_spec # noqa: E402
from aituner.trace import load_trace_requests, select_requests_for_threshold # noqa: E402
SCHEMA = "intervention-response-phase-aware-pilot-manifest-v2"
TP = 4
MNS_ENDPOINTS = (16, 64)
REPLAY_TIME_SCALE = 0.5
EXPECTED_DURATION_S = 300.0
SAFETY_DEADLINE_S = 360.0
LOADS_REQ_S_GPU = {"low": 1.5, "mid": 2.125, "high": 3.125}
REPLICATE_ROLES = ("high1", "high2", "high3")
LOAD_ORDERS = {
1: ("low", "mid", "high"),
2: ("high", "low", "mid"),
3: ("mid", "high", "low"),
}
SESSION_ORDER = (
(1, 16),
(1, 64),
(2, 64),
(2, 16),
(3, 16),
(3, 64),
)
def atomic_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
os.replace(temporary, path)
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def order_hash(values: list[str]) -> str:
return hashlib.sha256("\n".join(values).encode()).hexdigest()
def attainable_anchor(requests: list[Any], target_count: int) -> tuple[float, list[Any]]:
ordered = sorted(float(request.sampling_u) for request in requests)
if not ordered:
raise ValueError("no requests remain after study filtering")
if target_count <= 0 or target_count > len(ordered):
raise ValueError(
f"target count {target_count} is outside available range 1..{len(ordered)}"
)
indices = sorted(
{
max(0, min(len(ordered) - 1, target_count - 1)),
max(0, min(len(ordered) - 1, target_count)),
}
)
candidates = []
for index in indices:
anchor = ordered[index]
selected = select_requests_for_threshold(requests, threshold=anchor)
candidates.append((abs(len(selected) - target_count), len(selected), anchor, selected))
_error, _count, anchor, selected = min(
candidates, key=lambda item: (item[0], item[1], item[2])
)
return anchor, selected
def selection_record(selected: list[Any], *, duration_s: float) -> dict[str, Any]:
return {
"anchor": max(float(request.sampling_u) for request in selected),
"selected_count": len(selected),
"offered_req_s": len(selected) / duration_s,
"offered_req_s_per_gpu": len(selected) / duration_s / TP,
"request_id_order_sha256": order_hash([request.row_id for request in selected]),
"arrival_order_sha256": order_hash(
[f"{request.arrival_s:.12f}" for request in selected]
),
"input_length_order_sha256": order_hash(
[str(request.prompt_tokens_hint) for request in selected]
),
}
def materialize_study(source: Path, target: Path, *, replicate: int) -> Path:
payload = json.loads(source.read_text(encoding="utf-8"))
payload["study_id"] = f"phase-aware-telemetry-v2-rep{replicate}"
payload["hardware"]["host_candidates"] = ["dash0"]
payload["engine"]["engine_version"] = "0.24.1.dev3+opprof"
trace = payload["trace"]
trace["replay_time_scale"] = REPLAY_TIME_SCALE
trace["early_stop_max_lag_s"] = None
trace["early_stop_max_elapsed_s"] = SAFETY_DEADLINE_S
trace["restart_engine_after_early_stop"] = False
trace["adaptive_stop"] = {"enabled": False}
atomic_json(target, payload)
return target
def build_manifest(
*, base_manifest_path: Path, private_root: Path
) -> dict[str, Any]:
base = json.loads(base_manifest_path.read_text(encoding="utf-8"))
if base.get("schema") != "fidelity-prefix-pilot-manifest-v1":
raise ValueError("unexpected base P1 manifest schema")
repetitions = {}
selection_hashes = []
selected_ids_by_replicate: dict[int, set[str]] = {}
for replicate, role in enumerate(REPLICATE_ROLES, start=1):
source_study = Path(base["private"]["studies"][role]["tp4"])
target_study = private_root / "studies" / f"rep{replicate}-tp4.json"
materialize_study(source_study, target_study, replicate=replicate)
study = load_study_spec(target_study)
window, requests = load_trace_requests(study, study_spec_path=target_study)
duration_s = float(window.window_end - window.window_start)
if not math.isclose(
duration_s, EXPECTED_DURATION_S, rel_tol=0.0, abs_tol=1e-9
):
raise ValueError(
f"rep{replicate}: replay duration {duration_s} != {EXPECTED_DURATION_S}"
)
selections = {}
selected_ids_by_level: dict[str, set[str]] = {}
for level, target_rate in LOADS_REQ_S_GPU.items():
target_count = round(target_rate * duration_s * TP)
anchor, selected = attainable_anchor(requests, target_count)
record = selection_record(selected, duration_s=duration_s)
record.update(
{
"anchor": anchor,
"target_count": target_count,
"target_req_s_per_gpu": target_rate,
}
)
selections[level] = record
selected_ids_by_level[level] = {request.row_id for request in selected}
selection_hashes.append(record["request_id_order_sha256"])
selected_ids_by_replicate[replicate] = selected_ids_by_level["high"]
repetitions[str(replicate)] = {
"source_role": role,
"study": str(target_study),
"study_sha256": sha256_file(target_study),
"duration_s": duration_s,
"load_order": list(LOAD_ORDERS[replicate]),
"selections": selections,
}
burnin = base["cells"]["tp4_mns16"]["targets"]["low"]["selections"][
"burnin"
]
burnin = dict(burnin)
burnin["study_sha256"] = sha256_file(Path(burnin["study"]))
sessions = [
{
"session": f"rep{replicate}-mns{mns}",
"replicate": replicate,
"mns": mns,
}
for replicate, mns in SESSION_ORDER
]
invariants = {
"three_repetitions": len(repetitions) == 3,
"six_sessions": len(sessions) == 6,
"load_levels_three": all(
len(item["selections"]) == 3 for item in repetitions.values()
),
"selection_hashes_unique": len(selection_hashes) == len(set(selection_hashes)),
"selection_sets_disjoint_across_repetitions": all(
not selected_ids_by_replicate[left] & selected_ids_by_replicate[right]
for left in selected_ids_by_replicate
for right in selected_ids_by_replicate
if left < right
),
"all_counts_positive": all(
selection["selected_count"] > 0
for item in repetitions.values()
for selection in item["selections"].values()
),
}
red_flags = [name for name, passed in invariants.items() if not passed]
return {
"schema": SCHEMA,
"status": "PASS" if not red_flags else "STOP",
"source": {
"base_manifest": str(base_manifest_path),
"base_manifest_sha256": sha256_file(base_manifest_path),
"window_id": base["source"]["window_id"],
"source_trace": base["source"]["trace"],
"source_trace_sha256": base["source"]["trace_sha256"],
},
"engine": {
"tp": TP,
"mns_endpoints": list(MNS_ENDPOINTS),
"replay_time_scale": REPLAY_TIME_SCALE,
"duration_s": EXPECTED_DURATION_S,
"safety_deadline_s": SAFETY_DEADLINE_S,
"disable_slo_early_stop": True,
},
"burnin": burnin,
"repetitions": repetitions,
"sessions": sessions,
"checkpoints": {
"fractions": [0.1, 0.25, 0.5, 0.75, 1.0],
"seconds": [30.0, 75.0, 150.0, 225.0, 300.0],
},
"budget": {
"hard_cap_h20_hours": 8.0,
"expected_wall_minutes": [95, 120],
"expected_h20_hours": [6.3, 8.0],
},
"sanity": {
"red_flags": red_flags,
"invariants": invariants,
"selected_sets": len(selection_hashes),
"distinct_selected_sets": len(set(selection_hashes)),
},
}
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--base-manifest", type=Path, required=True)
parser.add_argument("--private-root", type=Path, required=True)
parser.add_argument("--public-manifest", type=Path, required=True)
args = parser.parse_args()
manifest = build_manifest(
base_manifest_path=args.base_manifest, private_root=args.private_root
)
atomic_json(args.public_manifest, manifest)
print(
json.dumps(
{
"status": manifest["status"],
"manifest": str(args.public_manifest),
"sanity": manifest["sanity"],
},
sort_keys=True,
)
)
if manifest["status"] != "PASS":
raise RuntimeError(f"phase-aware pilot preflight failed: {manifest['sanity']}")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,107 @@
#!/usr/bin/env python3
from __future__ import annotations
import importlib.util
import math
from pathlib import Path
from types import SimpleNamespace
HERE = Path(__file__).resolve().parent
def load_module():
spec = importlib.util.spec_from_file_location(
"intervention_response_phase_aware_v2", HERE / "analyze_existing.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
def load_prepare_module():
spec = importlib.util.spec_from_file_location(
"intervention_response_phase_aware_prepare", HERE / "prepare_pilot.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
def load_controller_module():
spec = importlib.util.spec_from_file_location(
"intervention_response_phase_aware_controller", HERE / "pilot_controller.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
def load_pilot_analysis_module():
spec = importlib.util.spec_from_file_location(
"intervention_response_phase_aware_pilot_analysis", HERE / "analyze_pilot.py"
)
module = importlib.util.module_from_spec(spec)
assert spec.loader is not None
spec.loader.exec_module(module)
return module
def main() -> None:
module = load_module()
assert module.common_decile_fractions(
trace_duration_s=60.0, minimum_elapsed_s=19.448
) == (0.1, 0.2, 0.3)
assert module.common_decile_fractions(
trace_duration_s=60.0, minimum_elapsed_s=60.0
)[-1] == 1.0
stats = module.numeric([0.0, 1.0, 2.0])
assert stats == {
"n": 3,
"min": 0.0,
"max": 2.0,
"distinct_n": 3,
"median": 1.0,
}
assert math.isclose(module._pearson([1.0, 2.0], [2.0, 4.0]), 1.0)
assert module._pearson([1.0, 1.0], [2.0, 3.0]) is None
prepare = load_prepare_module()
requests = [
SimpleNamespace(
sampling_u=index / 10.0,
row_id=f"r{index}",
arrival_s=float(index),
prompt_tokens_hint=100 + index,
)
for index in range(1, 6)
]
_anchor, selected = prepare.attainable_anchor(requests, 3)
assert len(selected) == 3
record = prepare.selection_record(selected, duration_s=3.0)
assert record["selected_count"] == 3
assert record["offered_req_s_per_gpu"] == 0.25
assert len(prepare.SESSION_ORDER) == 6
assert {mns for _replicate, mns in prepare.SESSION_ORDER} == {16, 64}
controller = load_controller_module()
assert math.isclose(controller.remaining_projection(6, 0), 7.7)
assert math.isclose(controller.remaining_projection(6, 5), 1.45)
pilot_analysis = load_pilot_analysis_module()
stable = pilot_analysis.stable_adjacent_features(
[
{"end_fraction": 0.1, "qualifying_response_features": ["queue"]},
{
"end_fraction": 0.25,
"qualifying_response_features": ["kv", "queue"],
},
{"end_fraction": 0.5, "qualifying_response_features": ["queue"]},
]
)
assert stable == {"0.10->0.25": ["queue"], "0.25->0.50": ["queue"]}
print("phase-aware intervention response v2 analysis: PASS")
if __name__ == "__main__":
main()

View File

@@ -95,7 +95,11 @@ def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]:
base_url=args.base_url,
timeout_s=study.engine.request_timeout_s,
max_concurrency=study.trace.max_concurrency,
target_pass_rate=(0.0 if warmup else study.slo.target_pass_rate),
target_pass_rate=(
0.0
if warmup or args.disable_slo_early_stop
else study.slo.target_pass_rate
),
max_lag_s=study.trace.early_stop_max_lag_s,
max_elapsed_s=(
120.0 if warmup else _probe_drain_deadline(
@@ -162,6 +166,7 @@ def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]:
"feasible": bool(slo_summary["feasible"]),
"early_stopped": early_stopped,
"early_stop_reason": early_stop_reason,
"slo_early_stop_disabled": bool(args.disable_slo_early_stop),
"ttft_ms": numeric([item.ttft_ms for item in outcomes]),
"tpot_ms": numeric([item.tpot_ms for item in outcomes]),
"invariants": {
@@ -240,6 +245,7 @@ def parser() -> argparse.ArgumentParser:
q.add_argument("--mns", type=int, required=True)
q.add_argument("--base-url", required=True)
q.add_argument("--result-dir", required=True)
q.add_argument("--disable-slo-early-stop", action="store_true")
return p

View File

@@ -20,6 +20,7 @@ def main() -> None:
controller = load("p6controller", HERE / "opprof_phase6_controller.py")
solo = load("p6solo", HERE / "opprof_phase6_solo_controller.py")
analysis = load("p6analysis", HERE / "analyze_phase6.py")
client = load("p6client", HERE / "opprof_phase6_client.py")
assert len(controller.CELLS) == 12
primary = sum(3 if item.get("trap") else 2 for item in controller.CELLS.values())
assert primary == 25
@@ -36,6 +37,18 @@ def main() -> None:
assert sum(solo.CELL_ESTIMATE.values()) + solo.SAFETY_HOURS == 3.44
assert solo.next_below([.1, .2, .3], {.2, .3}) == .1
assert solo.next_above([.1, .2, .3], {.1, .2}) == .3
parsed = client.parser().parse_args([
"run-anchor",
"--study", "study.json",
"--cell", "tp4_mns16",
"--anchor", "0.5",
"--tp", "4",
"--mns", "16",
"--base-url", "http://127.0.0.1:8000",
"--result-dir", "out",
"--disable-slo-early-stop",
])
assert parsed.disable_slo_early_stop is True
print("phase6 tools: PASS")

View File

@@ -0,0 +1,174 @@
#!/usr/bin/env python3
"""Pair P1 engine intervals with detailed Frontier state summaries."""
from __future__ import annotations
import argparse
import json
import math
import sys
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
from common_state import load_jsonl, numeric, residual, summarize_engine # noqa: E402
def atomic_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
temporary.replace(path)
def adjudicated_labels(path: Path) -> dict[tuple[str, str], bool]:
payload = json.loads(path.read_text(encoding="utf-8"))
return {
(row["cell"], row["level"]): bool(row["adjudicated_feasible"])
for row in payload["examples"]
}
def state_runs(root: Path) -> list[Path]:
candidates = list(root.glob("*/result.json"))
if (root / "result.json").is_file():
candidates.append(root / "result.json")
return sorted(
path
for path in candidates
if json.loads(path.read_text(encoding="utf-8")).get("schema")
== "telemetry-residual-frontier-state-result-v1"
)
def execute(args: argparse.Namespace) -> dict[str, Any]:
labels = adjudicated_labels(args.pilot_metrics)
examples = []
red_flags = []
for state_result_path in state_runs(args.sim_state_root):
run_root = state_result_path.parent
manifest = json.loads((run_root / "run_manifest.json").read_text(encoding="utf-8"))
cell = manifest["entry"]["cell"]
role = manifest["entry"]["role"]
level = "low" if role.startswith("low") else "high"
real_run = args.real_root / "cells" / cell / f"{level}-rep1"
real_result = json.loads((real_run / "result.json").read_text(encoding="utf-8"))
stream_paths = sorted((args.real_root / "cells" / cell / "opprof").glob("*.jsonl"))
if len(stream_paths) != 1:
raise ValueError(f"expected one engine stream for {cell}, found {len(stream_paths)}")
engine = summarize_engine(
load_jsonl(stream_paths[0]),
start_ns=int(real_result["interval"]["start_mono_ns"]),
end_ns=int(real_result["interval"]["end_mono_ns"]),
request_count=int(real_result["selection"]["count"]),
)
simulator = json.loads((run_root / "common-state.json").read_text(encoding="utf-8"))
scorer = json.loads((run_root / "scorer_output.json").read_text(encoding="utf-8"))
if engine["interval"]["request_count"] != simulator["interval"]["request_count"]:
red_flags.append(f"request_count_mismatch:{cell}:{role}")
difference = residual(engine, simulator)
if any(not math.isfinite(float(value)) for value in difference["values"].values()):
red_flags.append(f"nonfinite_residual:{cell}:{role}")
real_feasible = labels[(cell, level)]
sim_feasible = bool(scorer["slo"]["feasible"])
examples.append(
{
"cell": cell,
"role": role,
"level": level,
"tp": int(cell.split("_")[0][2:]),
"mns": int(cell.split("_")[1][3:]),
"request_count": engine["interval"]["request_count"],
"real_feasible": real_feasible,
"sim_feasible": sim_feasible,
"simulator_error": sim_feasible != real_feasible,
"simulator_false_feasible": sim_feasible and not real_feasible,
"real_pass_rate_rep1": float(real_result["pass_rate"]),
"offered_req_s": float(real_result["selection"]["offered_req_s"]),
"offered_req_s_per_gpu": float(
real_result["selection"]["offered_req_s_per_gpu"]
),
"sim_pass_rate": float(scorer["slo"]["pass_rate"]),
"pass_rate_residual": float(real_result["pass_rate"])
- float(scorer["slo"]["pass_rate"]),
"engine": engine,
"simulator": simulator,
"state_residual": difference,
"paths": {
"real_result": str((real_run / "result.json").resolve()),
"engine_stream": str(stream_paths[0].resolve()),
"simulator_result": str(state_result_path.resolve()),
},
}
)
if not examples:
red_flags.append("no_state_examples")
request_counts = [int(row["request_count"]) for row in examples]
pass_rate_residuals = [float(row["pass_rate_residual"]) for row in examples]
result = {
"schema": "telemetry-residual-p1-state-pairs-v1",
"status": "PASS" if not red_flags else "STOP",
"scope": "P1 development plumbing; not held-out contribution evidence",
"examples": examples,
"red_flags": red_flags,
"sanity": {
"n": len(examples),
"request_count": numeric(request_counts) if request_counts else None,
"pass_rate_residual": numeric(pass_rate_residuals)
if pass_rate_residuals
else None,
"simulator_errors": sum(bool(row["simulator_error"]) for row in examples),
"invariants": {
"request_counts_match": not any(
flag.startswith("request_count_mismatch") for flag in red_flags
),
"finite_residuals": not any(
flag.startswith("nonfinite_residual") for flag in red_flags
),
"ratios_bounded": all(
0.0 <= row["real_pass_rate_rep1"] <= 1.0
and 0.0 <= row["sim_pass_rate"] <= 1.0
for row in examples
),
"per_config_not_identical": len(set(pass_rate_residuals)) > 1
if len(pass_rate_residuals) > 1
else None,
},
},
}
atomic_json(args.output, result)
if result["status"] != "PASS":
raise RuntimeError(red_flags)
return result
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser()
result.add_argument("--real-root", type=Path, required=True)
result.add_argument("--sim-state-root", type=Path, required=True)
result.add_argument("--pilot-metrics", type=Path, required=True)
result.add_argument("--output", type=Path, required=True)
return result
def main() -> None:
result = execute(parser().parse_args())
print(
json.dumps(
{
"status": result["status"],
"examples": len(result["examples"]),
"simulator_errors": result["sanity"]["simulator_errors"],
"sanity": result["sanity"],
"red_flags": result["red_flags"],
},
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,292 @@
#!/usr/bin/env python3
"""Make the registered R0 go/no-go decision from development artifacts."""
from __future__ import annotations
import argparse
import json
import math
from pathlib import Path
from typing import Any
def atomic_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
temporary.replace(path)
def load_pass(path: Path, name: str) -> dict[str, Any]:
payload = json.loads(path.read_text(encoding="utf-8"))
if payload.get("status") != "PASS":
raise RuntimeError(f"{name} is not a valid PASS artifact: {path}")
return payload
def reduction(reference: float, candidate: float) -> float:
if reference <= 0.0 or candidate < 0.0 or not math.isfinite(candidate):
raise ValueError("costs must be finite and non-negative with positive reference")
return 1.0 - candidate / reference
def execute(args: argparse.Namespace) -> dict[str, Any]:
paired = load_pass(args.paired_state, "paired state")
transfer = load_pass(args.transfer, "transfer diagnostic")
e2e = load_pass(args.pilot_e2e, "P1 E2E replay")
red_flags = []
if len(paired.get("examples", [])) != 12:
red_flags.append("paired_state_not_12_anchors")
if transfer.get("sanity", {}).get("transitions") != 120:
red_flags.append("transfer_not_120_cross_config_transitions")
if transfer.get("red_flags"):
red_flags.append("transfer_has_red_flags")
examples = paired["examples"]
state_available = all(
row["state_residual"]["coverage"]["missing"] == 0
and row["state_residual"]["coverage"]["available"] > 0
for row in examples
)
state_vectors = {
tuple(sorted(row["state_residual"]["values"].items())) for row in examples
}
state_varies = len(state_vectors) > 1
error_examples = [row for row in examples if row["simulator_error"]]
simulator_errors = len(error_examples)
error_state_discrepancy = any(
any(
abs(float(value)) > 1e-12
for value in row["state_residual"]["values"].values()
)
for row in error_examples
)
simulator = transfer["simulator"]
prior_safe = []
direct_sensitivity = []
hybrid_incremental = []
for regularization, detail in transfer["regularization_sensitivity"].items():
for weight, models in detail["hybrid"]["prior_shrinkage"].items():
if float(weight) == 0.0:
continue
telemetry = models["raw_simulator_prior"][
"sim_plus_outcome_plus_telemetry"
]
decision_safe = (
telemetry["simulator_errors_corrected"] >= 1
and telemetry["simulator_errors_corrected"]
>= telemetry["simulator_correct_corrupted"]
)
continuous_safe = (
telemetry["rmse"] <= simulator["rmse"] + 1e-12
and telemetry["mae"] <= simulator["mae"] + 1e-12
)
if decision_safe and continuous_safe:
prior_safe.append(
{
"regularization": float(regularization),
"prior_weight": float(weight),
"simulator_errors_corrected": telemetry[
"simulator_errors_corrected"
],
"simulator_correct_corrupted": telemetry[
"simulator_correct_corrupted"
],
"rmse": telemetry["rmse"],
"mae": telemetry["mae"],
}
)
direct = detail["direct"]
direct_cmp = direct["comparison"]
direct_sensitivity.append(
{
"regularization": float(regularization),
"accuracy_delta": direct_cmp["delta_telemetry_minus_baseline"][
"feasibility_accuracy"
],
"rmse_delta": direct_cmp["delta_telemetry_minus_baseline"]["rmse"],
"mae_delta": direct_cmp["delta_telemetry_minus_baseline"]["mae"],
"errors_corrected": direct_cmp["baseline_errors_corrected"],
"correct_corrupted": direct_cmp["baseline_correct_corrupted"],
"telemetry_accuracy": direct["telemetry_only"][
"feasibility_accuracy"
],
}
)
hybrid_cmp = detail["hybrid"]["comparison"]
hybrid_incremental.append(
{
"regularization": float(regularization),
"accuracy_delta": hybrid_cmp["delta_telemetry_minus_baseline"][
"feasibility_accuracy"
],
"rmse_delta": hybrid_cmp["delta_telemetry_minus_baseline"]["rmse"],
"mae_delta": hybrid_cmp["delta_telemetry_minus_baseline"]["mae"],
"errors_corrected": hybrid_cmp["baseline_errors_corrected"],
"correct_corrupted": hybrid_cmp["baseline_correct_corrupted"],
}
)
k1 = e2e["by_k"]["1"]["sim_top_k_plus_real_final"]
k2 = e2e["by_k"]["2"]["sim_top_k_plus_real_final"]
headroom = {
"interpretation": (
"oracle correction stops after the simulator top-1 real final instead "
"of evaluating the frozen safety top-2"
),
"online": {
"reference_k2_h20_hours": k2["online_h20_hours"],
"oracle_k1_h20_hours": k1["online_h20_hours"],
"absolute_h20_hours": k2["online_h20_hours"] - k1["online_h20_hours"],
"fraction": reduction(k2["online_h20_hours"], k1["online_h20_hours"]),
},
"with_prior_failure": {
"reference_k2_h20_hours": k2["conservative_h20_hours_with_prior_failure"],
"oracle_k1_h20_hours": k1["conservative_h20_hours_with_prior_failure"],
"absolute_h20_hours": k2["conservative_h20_hours_with_prior_failure"]
- k1["conservative_h20_hours_with_prior_failure"],
"fraction": reduction(
k2["conservative_h20_hours_with_prior_failure"],
k1["conservative_h20_hours_with_prior_failure"],
),
},
"versus_observed_safe_k1_fraction": 0.0,
"k1_zero_regret": k1["real_regret"] == 0.0,
"k2_zero_regret": k2["real_regret"] == 0.0,
}
condition_1 = state_available and state_varies
condition_2 = simulator_errors >= 1 and error_state_discrepancy
condition_3 = bool(prior_safe)
condition_4 = headroom["online"]["fraction"] >= 0.15
conditions = {
"state_available_and_varies": condition_1,
"known_simulator_error_has_state_discrepancy": condition_2,
"prior_preserving_safe_correction_exists": condition_3,
"oracle_online_headroom_at_least_15pct": condition_4,
}
gate_pass = not red_flags and all(conditions.values())
direct_incremental = all(
row["accuracy_delta"] >= -1e-12
and row["errors_corrected"] >= row["correct_corrupted"]
for row in direct_sensitivity
)
result = {
"schema": "telemetry-residual-r0-gate-v1",
"status": "STOP" if red_flags else "PASS",
"scope": "P1 development premise/headroom audit; not headline evidence",
"decision": "PROCEED_TO_R1" if gate_pass else "STOP_BEFORE_R1",
"r0_gate_pass": gate_pass,
"conditions": conditions,
"route_findings": {
"hybrid_prior_safe_candidates": prior_safe,
"hybrid_incremental_regularization": hybrid_incremental,
"direct_incremental_regularization": direct_sensitivity,
"direct_incremental_decision_signal_all_lambdas": direct_incremental,
"direct_best_absolute_accuracy": max(
row["telemetry_accuracy"] for row in direct_sensitivity
),
"raw_simulator_accuracy": simulator["feasibility_accuracy"],
},
"headroom": headroom,
"red_flags": red_flags,
"sanity": {
"anchors": {
"n": len(examples),
"min": min(row["real_pass_rate_rep1"] for row in examples),
"max": max(row["real_pass_rate_rep1"] for row in examples),
"distinct_n": len(
{row["real_pass_rate_rep1"] for row in examples}
),
},
"state_vectors": {
"n": len(examples),
"min": min(len(row["state_residual"]["values"]) for row in examples),
"max": max(len(row["state_residual"]["values"]) for row in examples),
"distinct_n": len(state_vectors),
},
"costs_h20_hours": {
"n": 4,
"min": min(
k1["online_h20_hours"],
k2["online_h20_hours"],
k1["conservative_h20_hours_with_prior_failure"],
k2["conservative_h20_hours_with_prior_failure"],
),
"max": max(
k1["online_h20_hours"],
k2["online_h20_hours"],
k1["conservative_h20_hours_with_prior_failure"],
k2["conservative_h20_hours_with_prior_failure"],
),
"distinct_n": len(
{
k1["online_h20_hours"],
k2["online_h20_hours"],
k1["conservative_h20_hours_with_prior_failure"],
k2["conservative_h20_hours_with_prior_failure"],
}
),
},
"invariants": {
"no_data_red_flags": not red_flags,
"state_nonempty_and_varied": condition_1,
"pass_rates_bounded": all(
0.0 <= row["real_pass_rate_rep1"] <= 1.0
and 0.0 <= row["sim_pass_rate"] <= 1.0
for row in examples
),
"costs_nonnegative": all(
value >= 0.0
for value in (
k1["online_h20_hours"],
k2["online_h20_hours"],
k1["conservative_h20_hours_with_prior_failure"],
k2["conservative_h20_hours_with_prior_failure"],
)
),
"per_config_not_identical": len(
{row["real_pass_rate_rep1"] for row in examples}
)
> 1,
},
},
}
atomic_json(args.output, result)
if result["status"] != "PASS":
raise RuntimeError(red_flags)
return result
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser()
result.add_argument("--paired-state", type=Path, required=True)
result.add_argument("--transfer", type=Path, required=True)
result.add_argument("--pilot-e2e", type=Path, required=True)
result.add_argument("--output", type=Path, required=True)
return result
def main() -> None:
result = execute(parser().parse_args())
print(
json.dumps(
{
"status": result["status"],
"decision": result["decision"],
"r0_gate_pass": result["r0_gate_pass"],
"conditions": result["conditions"],
"headroom": result["headroom"],
"sanity": result["sanity"],
"red_flags": result["red_flags"],
},
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,480 @@
#!/usr/bin/env python3
"""Development-only cross-config telemetry transfer diagnostic for P1.
Each example asks whether state observed at one source anchor helps predict the
pass rate at a different target config. The hybrid branch predicts the
real-minus-simulator residual; the direct branch never reads simulator state or
outcomes. Folds exclude both the source and target config identities. The two
offered-load anchors belong to one trace/SLO task, so this is a premise check
rather than generalization evidence.
"""
from __future__ import annotations
import argparse
import json
import math
from pathlib import Path
from typing import Any, Sequence
import numpy as np
REGULARIZATION = (0.1, 1.0, 10.0, 100.0)
PRIOR_SHRINKAGE = (0.0, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0)
def atomic_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
temporary.replace(path)
def finite(value: Any, *, name: str) -> float:
result = float(value)
if not math.isfinite(result):
raise ValueError(f"non-finite feature {name}={value!r}")
return result
def flatten_residual_state(example: dict[str, Any]) -> tuple[float, ...]:
residual = example["state_residual"]["values"]
values = [finite(residual[name], name=name) for name in sorted(residual)]
engine_only = example["engine"]["engine_only"]
values.extend(
finite(engine_only[name], name=name)
for name in sorted(engine_only)
)
return tuple(values)
def flatten_engine_state(example: dict[str, Any]) -> tuple[float, ...]:
values = []
for name in sorted(example["engine"]["common"]):
value = example["engine"]["common"][name]
if isinstance(value, dict):
values.extend(
finite(value[statistic], name=f"{name}.{statistic}")
for statistic in ("mean", "max", "cv")
)
elif value is not None:
values.append(finite(value, name=name))
engine_only = example["engine"]["engine_only"]
values.extend(
finite(engine_only[name], name=name)
for name in sorted(engine_only)
)
return tuple(values)
def config_transition_features(
source: dict[str, Any], target: dict[str, Any]
) -> tuple[float, ...]:
source_rate = finite(
source["offered_req_s_per_gpu"], name="source_offered_req_s_per_gpu"
)
target_rate = finite(
target["offered_req_s_per_gpu"], name="target_offered_req_s_per_gpu"
)
return (
math.log2(float(source["tp"])),
math.log2(float(source["mns"])),
math.log2(float(target["tp"])),
math.log2(float(target["mns"])),
math.log2(float(target["tp"]) / float(source["tp"])),
math.log2(float(target["mns"]) / float(source["mns"])),
math.log2(source_rate),
math.log2(target_rate),
math.log2(target_rate / source_rate),
)
def hybrid_base_features(
source: dict[str, Any], target: dict[str, Any]
) -> tuple[float, ...]:
return (
finite(source["pass_rate_residual"], name="source_pass_rate_residual"),
finite(source["sim_pass_rate"], name="source_sim_pass_rate"),
finite(target["sim_pass_rate"], name="target_sim_pass_rate"),
) + config_transition_features(source, target)
def direct_base_features(
source: dict[str, Any], target: dict[str, Any]
) -> tuple[float, ...]:
return (
finite(source["real_pass_rate_rep1"], name="source_real_pass_rate"),
) + config_transition_features(source, target)
def transitions(examples: Sequence[dict[str, Any]]) -> list[dict[str, Any]]:
rows = []
for source in examples:
residual_state = flatten_residual_state(source)
engine_state = flatten_engine_state(source)
for target in examples:
# Low/high are offered-load anchors inside one workload/SLO task.
# Cross-load transitions are legal; same-cell transitions are
# excluded because R0 asks about transfer to a new configuration.
if source["cell"] == target["cell"]:
continue
hybrid_base = hybrid_base_features(source, target)
direct_base = direct_base_features(source, target)
rows.append(
{
"source_cell": source["cell"],
"source_level": source["level"],
"target_cell": target["cell"],
"target_level": target["level"],
"hybrid_base": hybrid_base,
"hybrid_telemetry": hybrid_base + residual_state,
"direct_base": direct_base,
"direct_telemetry": direct_base + engine_state,
"target_residual": finite(
target["pass_rate_residual"], name="target_residual"
),
"target_residual_delta": finite(
target["pass_rate_residual"], name="target_residual"
)
- finite(source["pass_rate_residual"], name="source_residual"),
"target_real_pass_rate": finite(
target["real_pass_rate_rep1"], name="target_real_pass_rate"
),
"target_real_pass_rate_delta": finite(
target["real_pass_rate_rep1"], name="target_real_pass_rate"
)
- finite(
source["real_pass_rate_rep1"], name="source_real_pass_rate"
),
"source_residual": finite(
source["pass_rate_residual"], name="source_residual"
),
"source_real_pass_rate": finite(
source["real_pass_rate_rep1"], name="source_real_pass_rate"
),
"target_sim_pass_rate": finite(
target["sim_pass_rate"], name="target_sim_pass_rate"
),
"target_real_feasible": bool(target["real_feasible"]),
"target_sim_feasible": bool(target["sim_feasible"]),
}
)
return rows
def fit_predict(
train_x: np.ndarray,
train_y: np.ndarray,
test_x: np.ndarray,
regularization: float,
) -> np.ndarray:
mean = train_x.mean(axis=0)
std = train_x.std(axis=0)
std[std == 0.0] = 1.0
train = (train_x - mean) / std
test = (test_x - mean) / std
train = np.column_stack((np.ones(len(train)), train))
test = np.column_stack((np.ones(len(test)), test))
penalty = np.eye(train.shape[1], dtype=np.float64)
penalty[0, 0] = 0.0
weights = np.linalg.lstsq(
train.T @ train + regularization * penalty,
train.T @ train_y,
rcond=None,
)[0]
return test @ weights
def grouped_predictions(
rows: Sequence[dict[str, Any]],
*,
feature_name: str,
target_name: str,
regularization: float,
) -> np.ndarray:
predictions = np.zeros(len(rows), dtype=np.float64)
groups = sorted({(row["source_cell"], row["target_cell"]) for row in rows})
for source_cell, target_cell in groups:
held_out_cells = {source_cell, target_cell}
test_indexes = [
index
for index, row in enumerate(rows)
if row["source_cell"] == source_cell and row["target_cell"] == target_cell
]
train_indexes = [
index
for index, row in enumerate(rows)
if row["source_cell"] not in held_out_cells
and row["target_cell"] not in held_out_cells
]
if not test_indexes or not train_indexes:
raise ValueError(f"empty grouped fold for {source_cell}->{target_cell}")
train_x = np.asarray([rows[index][feature_name] for index in train_indexes])
train_y = np.asarray([rows[index][target_name] for index in train_indexes])
test_x = np.asarray([rows[index][feature_name] for index in test_indexes])
predictions[test_indexes] = fit_predict(
train_x, train_y, test_x, regularization
)
return predictions
def metrics(rows: Sequence[dict[str, Any]], predicted_pass: np.ndarray) -> dict[str, Any]:
truth = np.asarray(
[row["target_real_pass_rate"] for row in rows], dtype=np.float64
)
real_feasible = np.asarray(
[row["target_real_feasible"] for row in rows], dtype=bool
)
sim_feasible = np.asarray(
[row["target_sim_feasible"] for row in rows], dtype=bool
)
clipped_pass = np.clip(predicted_pass, 0.0, 1.0)
predicted_feasible = clipped_pass >= 0.95
baseline_correct = sim_feasible == real_feasible
model_correct = predicted_feasible == real_feasible
return {
"rmse": float(np.sqrt(np.mean((predicted_pass - truth) ** 2))),
"mae": float(np.mean(np.abs(predicted_pass - truth))),
"feasibility_accuracy": float(np.mean(model_correct)),
"false_feasible": int(np.sum(predicted_feasible & ~real_feasible)),
"false_infeasible": int(np.sum(~predicted_feasible & real_feasible)),
"simulator_errors_corrected": int(np.sum(~baseline_correct & model_correct)),
"simulator_correct_corrupted": int(np.sum(baseline_correct & ~model_correct)),
"predicted_pass_rate": {
"n": len(clipped_pass),
"min": float(clipped_pass.min()),
"max": float(clipped_pass.max()),
"distinct_n": len(set(float(value) for value in clipped_pass)),
},
}
def compare(
rows: Sequence[dict[str, Any]],
baseline_prediction: np.ndarray,
telemetry_prediction: np.ndarray,
baseline: dict[str, Any],
telemetry: dict[str, Any],
) -> dict[str, Any]:
truth = np.asarray([row["target_real_feasible"] for row in rows], dtype=bool)
baseline_feasible = np.clip(baseline_prediction, 0.0, 1.0) >= 0.95
telemetry_feasible = np.clip(telemetry_prediction, 0.0, 1.0) >= 0.95
baseline_correct = baseline_feasible == truth
telemetry_correct = telemetry_feasible == truth
return {
"delta_telemetry_minus_baseline": {
"rmse": telemetry["rmse"] - baseline["rmse"],
"mae": telemetry["mae"] - baseline["mae"],
"feasibility_accuracy": telemetry["feasibility_accuracy"]
- baseline["feasibility_accuracy"],
},
"baseline_errors_corrected": int(
np.sum(~baseline_correct & telemetry_correct)
),
"baseline_correct_corrupted": int(
np.sum(baseline_correct & ~telemetry_correct)
),
}
def execute(args: argparse.Namespace) -> dict[str, Any]:
paired = json.loads(args.paired_state.read_text(encoding="utf-8"))
if paired.get("status") != "PASS" or len(paired["examples"]) != 12:
raise RuntimeError("paired P1 state evidence is incomplete")
rows = transitions(paired["examples"])
residual_truth = np.asarray(
[row["target_residual"] for row in rows], dtype=np.float64
)
pass_truth = np.asarray(
[row["target_real_pass_rate"] for row in rows], dtype=np.float64
)
sim_pass = np.asarray(
[row["target_sim_pass_rate"] for row in rows], dtype=np.float64
)
simulator = metrics(rows, sim_pass)
sensitivity = {}
for regularization in REGULARIZATION:
hybrid_base_delta = grouped_predictions(
rows,
feature_name="hybrid_base",
target_name="target_residual_delta",
regularization=regularization,
)
hybrid_telemetry_delta = grouped_predictions(
rows,
feature_name="hybrid_telemetry",
target_name="target_residual_delta",
regularization=regularization,
)
source_residual = np.asarray(
[row["source_residual"] for row in rows], dtype=np.float64
)
hybrid_base_correction = source_residual + hybrid_base_delta
hybrid_telemetry_correction = source_residual + hybrid_telemetry_delta
hybrid_base_prediction = sim_pass + hybrid_base_correction
hybrid_telemetry_prediction = sim_pass + hybrid_telemetry_correction
hybrid_base = metrics(rows, hybrid_base_prediction)
hybrid_telemetry = metrics(rows, hybrid_telemetry_prediction)
prior_shrinkage = {}
for weight in PRIOR_SHRINKAGE:
prior_shrinkage[str(weight)] = {
"raw_simulator_prior": {
"sim_plus_outcome": metrics(
rows, sim_pass + weight * hybrid_base_correction
),
"sim_plus_outcome_plus_telemetry": metrics(
rows, sim_pass + weight * hybrid_telemetry_correction
),
},
"anchor_offset_prior": {
"sim_plus_outcome": metrics(
rows,
sim_pass + source_residual + weight * hybrid_base_delta,
),
"sim_plus_outcome_plus_telemetry": metrics(
rows,
sim_pass + source_residual + weight * hybrid_telemetry_delta,
),
},
}
direct_base_prediction = grouped_predictions(
rows,
feature_name="direct_base",
target_name="target_real_pass_rate_delta",
regularization=regularization,
)
direct_telemetry_prediction = grouped_predictions(
rows,
feature_name="direct_telemetry",
target_name="target_real_pass_rate_delta",
regularization=regularization,
)
source_real_pass = np.asarray(
[row["source_real_pass_rate"] for row in rows], dtype=np.float64
)
direct_base_prediction = source_real_pass + direct_base_prediction
direct_telemetry_prediction = source_real_pass + direct_telemetry_prediction
direct_base = metrics(rows, direct_base_prediction)
direct_telemetry = metrics(rows, direct_telemetry_prediction)
sensitivity[str(regularization)] = {
"hybrid": {
"sim_plus_outcome": hybrid_base,
"sim_plus_outcome_plus_telemetry": hybrid_telemetry,
"comparison": compare(
rows,
hybrid_base_prediction,
hybrid_telemetry_prediction,
hybrid_base,
hybrid_telemetry,
),
"prior_shrinkage": prior_shrinkage,
},
"direct": {
"real_outcome_only": direct_base,
"telemetry_only": direct_telemetry,
"comparison": compare(
rows,
direct_base_prediction,
direct_telemetry_prediction,
direct_base,
direct_telemetry,
),
},
}
red_flags = []
if any(not math.isfinite(value) for value in residual_truth):
red_flags.append("nonfinite_residual_target")
if any(not math.isfinite(value) for value in pass_truth):
red_flags.append("nonfinite_pass_rate_target")
result = {
"schema": "telemetry-residual-cross-config-diagnostic-v1",
"status": "PASS" if not red_flags else "STOP",
"scope": (
"single P1 trace/SLO-task development diagnostic; ordered transitions "
"are not independent tasks and cannot support a generalization claim"
),
"split": (
"hold out both ordered source config and target config identities; "
"each fold contains both offered-load anchors"
),
"features": {
"sim_plus_outcome": len(rows[0]["hybrid_base"]),
"sim_plus_outcome_plus_telemetry": len(rows[0]["hybrid_telemetry"]),
"real_outcome_only": len(rows[0]["direct_base"]),
"telemetry_only": len(rows[0]["direct_telemetry"]),
},
"simulator": simulator,
"regularization_sensitivity": sensitivity,
"red_flags": red_flags,
"sanity": {
"transitions": len(rows),
"load_levels": len(
{row["source_level"] for row in rows}
| {row["target_level"] for row in rows}
),
"cross_load_transitions": sum(
row["source_level"] != row["target_level"] for row in rows
),
"ordered_cell_pairs": len(
{(row["source_cell"], row["target_cell"]) for row in rows}
),
"target_residual": {
"n": len(residual_truth),
"min": float(residual_truth.min()),
"max": float(residual_truth.max()),
"distinct_n": len(set(float(value) for value in residual_truth)),
},
"target_real_pass_rate": {
"n": len(pass_truth),
"min": float(pass_truth.min()),
"max": float(pass_truth.max()),
"distinct_n": len(set(float(value) for value in pass_truth)),
},
"invariants": {
"finite_targets": not red_flags,
"ratios_bounded": all(
0.0 <= row["target_sim_pass_rate"] <= 1.0 for row in rows
),
"source_differs_from_target": all(
row["source_cell"] != row["target_cell"] for row in rows
),
"per_config_not_identical": len(
set(float(value) for value in pass_truth)
)
> 1,
},
},
}
atomic_json(args.output, result)
if result["status"] != "PASS":
raise RuntimeError(red_flags)
return result
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser()
result.add_argument("--paired-state", type=Path, required=True)
result.add_argument("--output", type=Path, required=True)
return result
def main() -> None:
result = execute(parser().parse_args())
print(
json.dumps(
{
"status": result["status"],
"transitions": result["sanity"]["transitions"],
"sensitivity": result["regularization_sensitivity"],
"sanity": result["sanity"],
"red_flags": result["red_flags"],
},
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,400 @@
#!/usr/bin/env python3
"""Extract directly measurable engine/simulator state into one schema.
The schema deliberately keeps common, engine-only, and simulator-only fields
separate. Missing simulator mechanisms must remain missing; they must not be
filled by a heuristic bottleneck label.
"""
from __future__ import annotations
import csv
import json
import math
from pathlib import Path
from statistics import fmean
from typing import Any, Iterable, Mapping, Sequence
SCHEMA = "telemetry-common-state-v1"
def numeric(values: Iterable[float | int]) -> dict[str, Any]:
finite = [float(value) for value in values]
if not finite:
raise ValueError("numeric summary requires at least one value")
if any(not math.isfinite(value) for value in finite):
raise ValueError("numeric summary received a non-finite value")
mean = fmean(finite)
variance = fmean((value - mean) ** 2 for value in finite)
return {
"n": len(finite),
"min": min(finite),
"max": max(finite),
"mean": mean,
"cv": math.sqrt(variance) / abs(mean) if mean else 0.0,
"distinct_n": len(set(finite)),
}
def load_jsonl(path: Path) -> list[dict[str, Any]]:
rows = []
with path.open(encoding="utf-8") as source:
for line_number, line in enumerate(source, start=1):
if not line.strip():
continue
row = json.loads(line)
if not isinstance(row, dict):
raise ValueError(f"{path}:{line_number}: expected a JSON object")
rows.append(row)
if not rows:
raise ValueError(f"{path}: no JSONL records")
return rows
def _as_number(value: Any, *, name: str) -> float:
if isinstance(value, bool) or not isinstance(value, (int, float)):
raise ValueError(f"{name} must be numeric, got {value!r}")
result = float(value)
if not math.isfinite(result):
raise ValueError(f"{name} must be finite, got {value!r}")
return result
def _time_weighted_mean(
records: Sequence[Mapping[str, Any]],
*,
start_ns: int,
end_ns: int,
value,
) -> float:
if end_ns <= start_ns:
raise ValueError("time-weighted interval must be positive")
selected = [
record
for record in records
if start_ns <= int(record["submit_mono_ns"]) <= end_ns
]
if not selected:
raise ValueError("time-weighted interval contains no records")
selected.sort(key=lambda record: int(record["submit_mono_ns"]))
cursor = start_ns
total = 0.0
current = _as_number(value(selected[0]), name="time-weighted value")
for record in selected[1:]:
timestamp = int(record["submit_mono_ns"])
if timestamp < cursor:
raise ValueError("telemetry timestamps are not monotonic")
total += current * (timestamp - cursor)
cursor = timestamp
current = _as_number(value(record), name="time-weighted value")
total += current * (end_ns - cursor)
return total / (end_ns - start_ns)
def summarize_engine(
records: Sequence[Mapping[str, Any]],
*,
start_ns: int,
end_ns: int,
request_count: int,
) -> dict[str, Any]:
"""Summarize a measured engine interval from Layer-1 records."""
if request_count <= 0:
raise ValueError("request_count must be positive")
layer1 = [record for record in records if "step_index" in record]
if not layer1:
raise ValueError("engine stream has no Layer-1 records")
step_indexes = [int(record["step_index"]) for record in layer1]
if len(step_indexes) != len(set(step_indexes)):
raise ValueError("engine Layer-1 step indexes are not unique")
if step_indexes != sorted(step_indexes):
raise ValueError("engine Layer-1 step indexes are not ordered")
if any(int(record.get("dropped_records_before", 0)) != 0 for record in layer1):
raise ValueError("engine Layer-1 stream reports dropped records")
interval = [
record
for record in layer1
if start_ns <= int(record["submit_mono_ns"]) <= end_ns
]
if not interval:
raise ValueError("engine interval has no Layer-1 records")
executed = [record for record in interval if bool(record["model_executed"])]
if not executed:
raise ValueError("engine interval has no executed model steps")
duration_s = (end_ns - start_ns) / 1e9
if duration_s <= 0:
raise ValueError("engine interval duration must be positive")
batch_sizes = [int(record["scheduled_requests"]) for record in executed]
prefill_tokens = [int(record["prefill_tokens"]) for record in executed]
decode_tokens = [int(record["decode_tokens"]) for record in executed]
batch_tokens = [
prefill + decode
for prefill, decode in zip(prefill_tokens, decode_tokens, strict=True)
]
decode_batches = [int(record["decode_batch_size"]) for record in executed]
if any(value < 0 for value in batch_sizes + batch_tokens + decode_batches):
raise ValueError("engine batch counters must be non-negative")
if any(
int(record["prefill_tokens"]) + int(record["decode_tokens"])
<= 0
for record in executed
):
raise ValueError("executed engine step has no scheduled tokens")
waiting_mean = _time_weighted_mean(
interval,
start_ns=start_ns,
end_ns=end_ns,
value=lambda record: record["queues"]["waiting"],
)
running_mean = _time_weighted_mean(
interval,
start_ns=start_ns,
end_ns=end_ns,
value=lambda record: record["queues"]["running"],
)
kv_mean = _time_weighted_mean(
interval,
start_ns=start_ns,
end_ns=end_ns,
value=lambda record: record["kv"]["usage"],
)
kv_values = [float(record["kv"]["usage"]) for record in interval]
if any(not 0.0 <= value <= 1.0 for value in kv_values):
raise ValueError("engine KV usage must be in [0, 1]")
total_prefill = sum(prefill_tokens)
total_decode = sum(decode_tokens)
graph_modes = [str(record["cudagraph"]["runtime_mode"]) for record in executed]
bucket_tokens = sum(int(record["cudagraph"]["bucket_tokens"]) for record in executed)
padding_tokens = sum(int(record["cudagraph"]["padding_tokens"]) for record in executed)
common = {
"scheduler_steps_per_s": len(executed) / duration_s,
"batch_size": numeric(batch_sizes),
"batch_tokens": numeric(batch_tokens),
"decode_batch_size": numeric(decode_batches),
"prefill_token_fraction": total_prefill / (total_prefill + total_decode),
"queue_waiting_mean": waiting_mean,
"queue_running_mean": running_mean,
"queue_waiting_time_per_request_ms": waiting_mean * duration_s * 1000.0 / request_count,
"queue_running_time_per_request_ms": running_mean * duration_s * 1000.0 / request_count,
"preemptions": sum(int(record["preemptions"]) for record in executed),
}
result = {
"schema": SCHEMA,
"source": "engine_layer1",
"interval": {
"start_ns": start_ns,
"end_ns": end_ns,
"duration_s": duration_s,
"request_count": request_count,
},
"common": common,
"engine_only": {
"kv_usage_mean": kv_mean,
"kv_usage_max": max(kv_values),
"kv_usage_end_minus_start": kv_values[-1] - kv_values[0],
"graph_none_share": graph_modes.count("NONE") / len(graph_modes),
"graph_full_share": graph_modes.count("FULL") / len(graph_modes),
"graph_padding_fraction": padding_tokens / max(1, bucket_tokens),
},
"simulator_only": {},
"sanity": {
"records": len(interval),
"executed_steps": len(executed),
"step_index_min": min(int(record["step_index"]) for record in interval),
"step_index_max": max(int(record["step_index"]) for record in interval),
"invariants": {
"positive_duration": duration_s > 0,
"positive_request_count": request_count > 0,
"zero_drops": True,
"nonnegative_counters": True,
"kv_bounded": True,
"batch_values_not_all_identical": any(
summary["distinct_n"] > 1
for summary in (
common["batch_size"],
common["batch_tokens"],
common["decode_batch_size"],
)
),
},
},
}
return result
def _csv_rows(path: Path) -> list[dict[str, str]]:
with path.open(encoding="utf-8", newline="") as source:
rows = list(csv.DictReader(source))
if not rows:
raise ValueError(f"{path}: CSV contains no rows")
return rows
def _column(rows: Sequence[Mapping[str, str]], name: str) -> list[float]:
if name not in rows[0]:
raise ValueError(f"CSV is missing required column {name!r}")
values = []
for row in rows:
text = row.get(name, "")
if text == "":
raise ValueError(f"CSV column {name!r} contains an empty value")
values.append(_as_number(float(text), name=name))
return values
def summarize_frontier(
*,
system_metrics_path: Path,
request_metrics_path: Path,
batch_metrics_path: Path | None = None,
ledger_path: Path | None = None,
) -> dict[str, Any]:
"""Summarize a Frontier run, retaining unavailable state as null."""
system = json.loads(system_metrics_path.read_text(encoding="utf-8"))
throughput = system["throughput_metrics"]
duration_s = _as_number(
throughput["total_duration_seconds"], name="total_duration_seconds"
)
if duration_s <= 0:
raise ValueError("Frontier duration must be positive")
request_rows = _csv_rows(request_metrics_path)
waiting_ms = _column(request_rows, "request_waiting_time_total")
e2e_ms = _column(request_rows, "request_e2e_time")
running_ms = [max(0.0, e2e - waiting) for e2e, waiting in zip(e2e_ms, waiting_ms, strict=True)]
duration_ms = duration_s * 1000.0
request_count = len(request_rows)
common: dict[str, Any] = {
"scheduler_steps_per_s": None,
"batch_size": None,
"batch_tokens": None,
"decode_batch_size": None,
"prefill_token_fraction": None,
"queue_waiting_mean": sum(waiting_ms) / duration_ms,
"queue_running_mean": sum(running_ms) / duration_ms,
"queue_waiting_time_per_request_ms": fmean(waiting_ms),
"queue_running_time_per_request_ms": fmean(running_ms),
"preemptions": sum(
int(float(row.get("request_total_preemption_count") or 0))
for row in request_rows
),
}
batch_rows: list[dict[str, str]] = []
if batch_metrics_path is not None:
batch_rows = _csv_rows(batch_metrics_path)
batch_sizes = _column(batch_rows, "batch_size")
batch_tokens = _column(batch_rows, "batch_num_tokens")
prefill_tokens = _column(batch_rows, "batch_num_prefill_tokens")
decode_tokens = _column(batch_rows, "batch_num_decode_tokens")
if any(value < 0 for value in batch_sizes + batch_tokens + prefill_tokens + decode_tokens):
raise ValueError("Frontier batch counters must be non-negative")
common.update(
{
"scheduler_steps_per_s": len(batch_rows) / duration_s,
"batch_size": numeric(batch_sizes),
"batch_tokens": numeric(batch_tokens),
"decode_batch_size": numeric(decode_tokens),
"prefill_token_fraction": sum(prefill_tokens)
/ max(1.0, sum(prefill_tokens) + sum(decode_tokens)),
}
)
ledger_rows: list[dict[str, Any]] = []
if ledger_path is not None:
ledger_rows = load_jsonl(ledger_path)
for row in ledger_rows:
start = _as_number(row["stage_start_ts"], name="stage_start_ts")
end = _as_number(row["stage_end_ts"], name="stage_end_ts")
if end < start:
raise ValueError("Frontier ledger has a negative stage duration")
batch_distinct = (
max(
summary["distinct_n"]
for summary in (
common["batch_size"],
common["batch_tokens"],
common["decode_batch_size"],
)
)
if batch_rows
else None
)
return {
"schema": SCHEMA,
"source": "frontier",
"interval": {
"duration_s": duration_s,
"request_count": request_count,
},
"common": common,
"engine_only": {
"kv_usage_mean": None,
"kv_usage_max": None,
"kv_usage_end_minus_start": None,
"graph_none_share": None,
"graph_full_share": None,
"graph_padding_fraction": None,
},
"simulator_only": {
"request_waiting_time_ms": numeric(waiting_ms),
"request_running_time_ms": numeric(running_ms),
"ledger_rows": len(ledger_rows) if ledger_path is not None else None,
},
"sanity": {
"request_rows": request_count,
"batch_rows": len(batch_rows),
"ledger_rows": len(ledger_rows),
"invariants": {
"positive_duration": duration_s > 0,
"positive_request_count": request_count > 0,
"nonnegative_counters": True,
"request_values_not_all_identical": max(
numeric(waiting_ms)["distinct_n"],
numeric(running_ms)["distinct_n"],
)
> 1,
"batch_values_not_all_identical": (
batch_distinct > 1 if batch_distinct is not None else None
),
},
},
}
def residual(real: Mapping[str, Any], simulated: Mapping[str, Any]) -> dict[str, Any]:
if real.get("schema") != SCHEMA or simulated.get("schema") != SCHEMA:
raise ValueError("residual inputs must use the common-state schema")
values = {}
missing = []
for name, real_value in real["common"].items():
sim_value = simulated["common"].get(name)
if isinstance(real_value, dict):
if not isinstance(sim_value, dict):
missing.append(name)
continue
for statistic in ("mean", "max", "cv"):
key = f"{name}.{statistic}"
values[key] = float(real_value[statistic]) - float(sim_value[statistic])
continue
if real_value is None or sim_value is None:
missing.append(name)
continue
values[name] = float(real_value) - float(sim_value)
return {
"schema": "telemetry-state-residual-v1",
"values": values,
"missing_common_fields": sorted(missing),
"coverage": {
"available": len(values),
"missing": len(missing),
"common_field_count": len(real["common"]),
},
}

View File

@@ -0,0 +1,316 @@
#!/usr/bin/env python3
"""Replay frozen Frontier fixtures with state-detail outputs enabled.
This runner does not modify the frozen fixture/config inputs. It reuses the
audited SimFid command and only turns on existing Frontier batch/ledger output
flags in a separate result root.
"""
from __future__ import annotations
import argparse
import hashlib
import importlib.util
import json
import os
import subprocess
import sys
import time
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
AITUNER_ROOT = HERE.parents[1]
sys.path.insert(0, str(HERE))
from common_state import numeric, summarize_frontier # noqa: E402, F401
def load_module(name: str, path: Path):
module_root = str(path.parent.resolve())
if module_root not in sys.path:
sys.path.insert(0, module_root)
spec = importlib.util.spec_from_file_location(name, path)
if spec is None or spec.loader is None:
raise ImportError(path)
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def git_capture(root: Path, *arguments: str) -> str:
return subprocess.run(
["git", "-C", str(root), *arguments],
check=True,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
).stdout
def atomic_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_suffix(path.suffix + ".tmp")
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
os.replace(temporary, path)
def enable_state_outputs(command: list[str]) -> list[str]:
result = list(command)
disabled_ledger = "--no-metrics_config_store_frontier_stage_batch_ledger"
enabled_ledger = "--metrics_config_store_frontier_stage_batch_ledger"
enabled_batches = "--metrics_config_keep_individual_batch_metrics"
disabled_batches = "--no-metrics_config_keep_individual_batch_metrics"
if result.count(disabled_ledger) != 1 or enabled_ledger in result:
raise ValueError("base command must explicitly disable one full Frontier ledger")
result[result.index(disabled_ledger)] = enabled_ledger
if disabled_batches in result:
if result.count(disabled_batches) != 1:
raise ValueError("duplicate disabled individual-batch flag")
result[result.index(disabled_batches)] = enabled_batches
elif enabled_batches not in result:
result.append(enabled_batches)
if result.count(enabled_ledger) != 1 or result.count(enabled_batches) != 1:
raise ValueError("state-detail flags were not enabled exactly once")
return result
def find_state_metrics(run_root: Path) -> dict[str, Path]:
patterns = {
"system": "frontier_metrics/**/system_metrics.json",
"requests": "frontier_metrics/**/request_metrics.csv",
"batches": "frontier_metrics/**/monolithic_batch_metrics.csv",
"ledger": "frontier_metrics/**/frontier_stage_batch_ledger.jsonl",
}
result = {}
for name, pattern in patterns.items():
matches = sorted(run_root.glob(pattern))
if len(matches) != 1:
raise ValueError(
f"{run_root}: expected exactly one {name} artifact, found {len(matches)}"
)
result[name] = matches[0]
if len({path.parent for path in result.values()}) != 1:
raise ValueError("Frontier state artifacts are not colocated")
return result
def execute(args: argparse.Namespace) -> dict[str, Any]:
prepared = json.loads(args.prepared_manifest.read_text(encoding="utf-8"))
if prepared.get("status") != "PASS":
raise RuntimeError("prepared simulator manifest did not pass")
matches = [
entry
for entry in prepared["entries"]
if entry["cell"] == args.cell and entry["role"] == args.role
]
if len(matches) != 1:
raise ValueError(
f"expected one prepared entry for {args.cell}/{args.role}, found {len(matches)}"
)
entry = matches[0]
if args.output.exists() and any(args.output.iterdir()):
raise FileExistsError(f"non-empty output already exists: {args.output}")
args.output.mkdir(parents=True, exist_ok=True)
driver = load_module(
"telemetry_residual_execution_driver",
args.replayserve_root / "runs/simfid_s2rb/results/execution_driver.py",
)
config_path = Path(entry["config"])
config = json.loads(config_path.read_text(encoding="utf-8"))
fixture_manifest_path = Path(entry["fixture_manifest"])
fixture = json.loads(fixture_manifest_path.read_text(encoding="utf-8"))
trace_path = Path(entry["frontier_csv"])
sidecar_path = Path(entry["sidecar"])
metrics_root = args.output / "frontier_metrics"
run_id = f"telemetry_residual_r0_{args.cell}_{args.role}"
knobs = config["frontier"]["knobs"]
base_command = driver.build_command(
trace_path=trace_path,
metrics_root=metrics_root,
run_id=run_id,
knobs=knobs,
)
driver.audit_command(base_command, knobs)
command = enable_state_outputs(base_command)
row = {
"hook_path": config["calibration"]["hook_path"],
"applied_a_tp": config["calibration"]["a_tp"],
"sidecar_path": str(sidecar_path),
"request_count": int(fixture["request_count"]),
"tensor_parallel_size": int(fixture["tensor_parallel_size"]),
}
environment = driver.environment_for(row)
if environment.get("CUDA_VISIBLE_DEVICES") != "":
raise ValueError("Frontier state replay must hide CUDA devices")
manifest = {
"schema": "telemetry-residual-frontier-state-run-v1",
"entry": {
"cell": args.cell,
"role": args.role,
"anchor": entry["anchor"],
"request_count": entry["selected_count"],
},
"inputs": {
"prepared_manifest": str(args.prepared_manifest.resolve()),
"prepared_manifest_sha256": sha256_file(args.prepared_manifest),
"config": str(config_path.resolve()),
"config_sha256": sha256_file(config_path),
"fixture_manifest": str(fixture_manifest_path.resolve()),
"fixture_manifest_sha256": sha256_file(fixture_manifest_path),
"frontier_csv": str(trace_path.resolve()),
"frontier_csv_sha256": sha256_file(trace_path),
"sidecar": str(sidecar_path.resolve()),
"sidecar_sha256": sha256_file(sidecar_path),
},
"frontier": {
"root": str(args.frontier_root.resolve()),
"git_head": git_capture(args.frontier_root, "rev-parse", "HEAD").strip(),
"git_status_short": git_capture(args.frontier_root, "status", "--short"),
},
"runner": {
"script": str(Path(__file__).resolve()),
"script_sha256": sha256_file(Path(__file__).resolve()),
"aituner_git_head": git_capture(AITUNER_ROOT, "rev-parse", "HEAD").strip(),
"aituner_git_status_short": git_capture(AITUNER_ROOT, "status", "--short"),
},
"environment": {
key: environment[key]
for key in (
"PYTHONPATH",
"FRONTIER_EXECUTION_TIME_SCALE",
"CUDA_VISIBLE_DEVICES",
"NVIDIA_VISIBLE_DEVICES",
"FRONTIER_LOG_LEVEL",
)
},
"command": command,
"state_outputs": {
"individual_batch_metrics": True,
"full_stage_batch_ledger": True,
},
"contains_prompt_text": False,
}
atomic_json(args.output / "run_manifest.json", manifest)
start = time.monotonic()
with (args.output / "stdout.log").open("w", encoding="utf-8") as stdout, (
args.output / "stderr.log"
).open("w", encoding="utf-8") as stderr:
try:
process = subprocess.run(
command,
cwd=args.frontier_root,
env=environment,
stdout=stdout,
stderr=stderr,
timeout=args.timeout_s,
)
return_code = int(process.returncode)
except subprocess.TimeoutExpired:
return_code = 124
runtime_s = time.monotonic() - start
if return_code != 0:
failure = {
"status": "STOP",
"return_code": return_code,
"runtime_s": runtime_s,
}
atomic_json(args.output / "failure.json", failure)
raise RuntimeError(f"Frontier state replay failed: {failure}")
paths = find_state_metrics(args.output)
summary = summarize_frontier(
system_metrics_path=paths["system"],
request_metrics_path=paths["requests"],
batch_metrics_path=paths["batches"],
ledger_path=paths["ledger"],
)
atomic_json(args.output / "common-state.json", summary)
scorer = driver.score_trial(row, paths["system"], paths["requests"])
atomic_json(args.output / "scorer_output.json", scorer)
sizes = {name: path.stat().st_size for name, path in paths.items()}
red_flags = []
if summary["interval"]["request_count"] != int(entry["selected_count"]):
red_flags.append("request_count_mismatch")
if summary["sanity"]["batch_rows"] <= 0:
red_flags.append("no_batch_rows")
if summary["sanity"]["ledger_rows"] <= 0:
red_flags.append("no_ledger_rows")
result = {
"schema": "telemetry-residual-frontier-state-result-v1",
"status": "PASS" if not red_flags else "STOP",
"runtime_s": runtime_s,
"return_code": return_code,
"paths": {name: str(path.resolve()) for name, path in paths.items()},
"bytes": sizes,
"common_state": str((args.output / "common-state.json").resolve()),
"red_flags": red_flags,
"sanity": {
"request_rows": summary["sanity"]["request_rows"],
"batch_rows": summary["sanity"]["batch_rows"],
"ledger_rows": summary["sanity"]["ledger_rows"],
"artifact_bytes": {
"n": len(sizes),
"min": min(sizes.values()),
"max": max(sizes.values()),
"distinct_n": len(set(sizes.values())),
},
"invariants": {
"zero_failures": return_code == 0,
"gpu_visibility_disabled": True,
"request_count_match": not red_flags,
"nonnegative_file_sizes": all(size >= 0 for size in sizes.values()),
"state_values_not_all_identical": summary["sanity"]["invariants"][
"batch_values_not_all_identical"
],
},
},
}
atomic_json(args.output / "result.json", result)
if result["status"] != "PASS":
raise RuntimeError(red_flags)
return result
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser()
result.add_argument("--prepared-manifest", type=Path, required=True)
result.add_argument("--output", type=Path, required=True)
result.add_argument("--replayserve-root", type=Path, required=True)
result.add_argument("--frontier-root", type=Path, required=True)
result.add_argument("--cell", required=True)
result.add_argument("--role", choices=("low1", "high1"), required=True)
result.add_argument("--timeout-s", type=float, default=300.0)
return result
def main() -> None:
result = execute(parser().parse_args())
print(
json.dumps(
{
"status": result["status"],
"runtime_s": result["runtime_s"],
"request_rows": result["sanity"]["request_rows"],
"batch_rows": result["sanity"]["batch_rows"],
"ledger_rows": result["sanity"]["ledger_rows"],
"red_flags": result["red_flags"],
},
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,169 @@
#!/usr/bin/env python3
"""Run the frozen 12-fixture P1 Frontier state-detail campaign, CPU only."""
from __future__ import annotations
import argparse
import json
import sys
import time
from pathlib import Path
from types import SimpleNamespace
from typing import Any
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import run_frontier_state as state_runner # noqa: E402
def atomic_json(path: Path, payload: Any) -> None:
state_runner.atomic_json(path, payload)
def scorer_without_runtime(value: dict[str, Any]) -> dict[str, Any]:
result = dict(value)
result.pop("runtime_s", None)
return result
def execute(args: argparse.Namespace) -> dict[str, Any]:
prepared = json.loads(args.prepared_manifest.read_text(encoding="utf-8"))
committed = json.loads(args.committed_results.read_text(encoding="utf-8"))
if prepared.get("status") != "PASS" or committed.get("status") != "PASS":
raise RuntimeError("prepared or committed simulator evidence did not pass")
committed_by_key = {
(row["cell"], row["role"]): scorer_without_runtime(row["scorer"])
for row in committed["results"]
}
entries = prepared["entries"]
if len(entries) != 12 or len(committed_by_key) != 12:
raise ValueError("P1 state campaign requires exactly 12 fixtures")
args.output.mkdir(parents=True, exist_ok=True)
results = []
campaign_start = time.monotonic()
for index, entry in enumerate(entries, start=1):
key = (entry["cell"], entry["role"])
output = args.output / f"{entry['cell']}_{entry['role']}"
result_path = output / "result.json"
if args.resume and result_path.is_file():
result = json.loads(result_path.read_text(encoding="utf-8"))
if result.get("status") != "PASS":
raise RuntimeError(f"cannot resume failed state replay: {result_path}")
resumed = True
else:
if output.exists() and any(output.iterdir()):
raise FileExistsError(f"non-empty state replay output: {output}")
print(
f"RUN {index:02d}/12 {entry['cell']}/{entry['role']}",
flush=True,
)
result = state_runner.execute(
SimpleNamespace(
prepared_manifest=args.prepared_manifest,
output=output,
replayserve_root=args.replayserve_root,
frontier_root=args.frontier_root,
cell=entry["cell"],
role=entry["role"],
timeout_s=args.timeout_s,
)
)
resumed = False
observed_scorer = json.loads(
(output / "scorer_output.json").read_text(encoding="utf-8")
)
exact_scorer_match = scorer_without_runtime(observed_scorer) == committed_by_key[key]
if not exact_scorer_match:
raise ValueError(f"state-output replay changed the committed scorer: {key}")
results.append(
{
"cell": entry["cell"],
"role": entry["role"],
"runtime_s": result["runtime_s"],
"request_rows": result["sanity"]["request_rows"],
"batch_rows": result["sanity"]["batch_rows"],
"ledger_rows": result["sanity"]["ledger_rows"],
"artifact_bytes": sum(result["bytes"].values()),
"exact_committed_scorer_match": exact_scorer_match,
"resumed": resumed,
"result": str(result_path.resolve()),
}
)
print(
f"DONE {index:02d}/12 {entry['cell']}/{entry['role']} "
f"runtime={result['runtime_s']:.3f}s batches={result['sanity']['batch_rows']}",
flush=True,
)
runtimes = [float(row["runtime_s"]) for row in results]
batches = [int(row["batch_rows"]) for row in results]
bytes_values = [int(row["artifact_bytes"]) for row in results]
red_flags = []
if len(results) != 12:
red_flags.append("runs_not_12")
if not all(row["exact_committed_scorer_match"] for row in results):
red_flags.append("committed_scorer_mismatch")
if any(value <= 0 for value in batches):
red_flags.append("empty_batch_output")
result = {
"schema": "telemetry-residual-frontier-state-campaign-v1",
"status": "PASS" if not red_flags else "STOP",
"prepared_manifest": str(args.prepared_manifest.resolve()),
"committed_results": str(args.committed_results.resolve()),
"campaign_elapsed_s": time.monotonic() - campaign_start,
"results": results,
"red_flags": red_flags,
"sanity": {
"n": len(results),
"runtime_s": state_runner.numeric(runtimes),
"batch_rows": state_runner.numeric(batches),
"artifact_bytes": state_runner.numeric(bytes_values),
"invariants": {
"runs_12": len(results) == 12,
"zero_failures": not red_flags,
"exact_committed_scorers": all(
row["exact_committed_scorer_match"] for row in results
),
"nonnegative_counts": all(value > 0 for value in batches),
"per_config_not_identical": len(set(batches)) > 1,
"gpu_visibility_disabled": True,
},
},
}
atomic_json(args.output / "campaign-metrics.json", result)
if result["status"] != "PASS":
raise RuntimeError(red_flags)
return result
def parser() -> argparse.ArgumentParser:
result = argparse.ArgumentParser()
result.add_argument("--prepared-manifest", type=Path, required=True)
result.add_argument("--committed-results", type=Path, required=True)
result.add_argument("--output", type=Path, required=True)
result.add_argument("--replayserve-root", type=Path, required=True)
result.add_argument("--frontier-root", type=Path, required=True)
result.add_argument("--timeout-s", type=float, default=300.0)
result.add_argument("--resume", action="store_true")
return result
def main() -> None:
result = execute(parser().parse_args())
print(
json.dumps(
{
"status": result["status"],
"runs": len(result["results"]),
"elapsed_s": result["campaign_elapsed_s"],
"sanity": result["sanity"],
"red_flags": result["red_flags"],
},
sort_keys=True,
)
)
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,183 @@
#!/usr/bin/env python3
from __future__ import annotations
import csv
import json
import math
import tempfile
from pathlib import Path
from common_state import numeric, residual, summarize_engine, summarize_frontier
from run_frontier_state import enable_state_outputs
def engine_record(
step: int,
timestamp_ns: int,
*,
batch: int,
prefill: int,
decode: int,
waiting: int,
running: int,
kv: float,
) -> dict[str, object]:
return {
"step_index": step,
"submit_mono_ns": timestamp_ns,
"model_executed": True,
"scheduled_requests": batch,
"decode_batch_size": decode,
"prefill_tokens": prefill,
"decode_tokens": decode,
"preemptions": 0,
"queues": {"waiting": waiting, "running": running},
"kv": {"usage": kv},
"cudagraph": {
"runtime_mode": "NONE" if step == 0 else "FULL",
"bucket_tokens": prefill + decode + 1,
"padding_tokens": 1,
},
"dropped_records_before": 0,
}
def main() -> None:
assert numeric((-2.0, -1.0))["cv"] >= 0.0
command = [
"python3",
"-m",
"frontier.main",
"--no-metrics_config_store_frontier_stage_batch_ledger",
]
state_command = enable_state_outputs(command)
assert "--metrics_config_store_frontier_stage_batch_ledger" in state_command
assert "--metrics_config_keep_individual_batch_metrics" in state_command
assert "--no-metrics_config_store_frontier_stage_batch_ledger" not in state_command
engine = summarize_engine(
[
engine_record(
0,
0,
batch=2,
prefill=6,
decode=0,
waiting=1,
running=2,
kv=0.1,
),
engine_record(
1,
1_000_000_000,
batch=4,
prefill=0,
decode=4,
waiting=3,
running=4,
kv=0.3,
),
],
start_ns=0,
end_ns=2_000_000_000,
request_count=2,
)
assert math.isclose(engine["common"]["queue_waiting_mean"], 2.0)
assert math.isclose(engine["common"]["queue_running_mean"], 3.0)
assert math.isclose(
engine["common"]["queue_waiting_time_per_request_ms"], 2000.0
)
assert engine["common"]["batch_size"]["distinct_n"] == 2
assert math.isclose(engine["common"]["prefill_token_fraction"], 0.6)
with tempfile.TemporaryDirectory() as temporary:
root = Path(temporary)
system = root / "system.json"
requests = root / "requests.csv"
batches = root / "batches.csv"
ledger = root / "ledger.jsonl"
system.write_text(
json.dumps(
{
"throughput_metrics": {
"total_duration_seconds": 2.0,
}
}
),
encoding="utf-8",
)
with requests.open("w", encoding="utf-8", newline="") as output:
writer = csv.DictWriter(
output,
fieldnames=[
"request_e2e_time",
"request_waiting_time_total",
"request_total_preemption_count",
],
)
writer.writeheader()
writer.writerow(
{
"request_e2e_time": 2000,
"request_waiting_time_total": 1000,
"request_total_preemption_count": 0,
}
)
writer.writerow(
{
"request_e2e_time": 3000,
"request_waiting_time_total": 3000,
"request_total_preemption_count": 0,
}
)
with batches.open("w", encoding="utf-8", newline="") as output:
writer = csv.DictWriter(
output,
fieldnames=[
"batch_size",
"batch_num_tokens",
"batch_num_prefill_tokens",
"batch_num_decode_tokens",
],
)
writer.writeheader()
writer.writerow(
{
"batch_size": 2,
"batch_num_tokens": 6,
"batch_num_prefill_tokens": 6,
"batch_num_decode_tokens": 0,
}
)
writer.writerow(
{
"batch_size": 4,
"batch_num_tokens": 4,
"batch_num_prefill_tokens": 0,
"batch_num_decode_tokens": 4,
}
)
ledger.write_text(
json.dumps({"stage_start_ts": 0.0, "stage_end_ts": 1.0}) + "\n"
+ json.dumps({"stage_start_ts": 1.0, "stage_end_ts": 2.0})
+ "\n",
encoding="utf-8",
)
simulator = summarize_frontier(
system_metrics_path=system,
request_metrics_path=requests,
batch_metrics_path=batches,
ledger_path=ledger,
)
assert math.isclose(simulator["common"]["queue_waiting_mean"], 2.0)
assert math.isclose(simulator["common"]["queue_running_mean"], 0.5)
assert simulator["common"]["batch_size"]["distinct_n"] == 2
difference = residual(engine, simulator)
assert difference["coverage"]["missing"] == 0
assert difference["coverage"]["available"] == 16
assert math.isclose(difference["values"]["queue_waiting_mean"], 0.0)
print("telemetry common state: PASS")
if __name__ == "__main__":
main()

View File

@@ -0,0 +1,67 @@
#!/usr/bin/env python3
"""Small structural tests for the cross-config transfer diagnostic."""
from __future__ import annotations
import importlib.util
from pathlib import Path
HERE = Path(__file__).resolve().parent
SPEC = importlib.util.spec_from_file_location(
"analyze_residual_transfer", HERE / "analyze_residual_transfer.py"
)
assert SPEC is not None and SPEC.loader is not None
MODULE = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(MODULE)
def example(cell: str, level: str, value: float) -> dict:
tp_text, mns_text = cell.split("_")
return {
"cell": cell,
"level": level,
"tp": int(tp_text[2:]),
"mns": int(mns_text[3:]),
"pass_rate_residual": value - 0.5,
"real_pass_rate_rep1": value,
"sim_pass_rate": 0.5,
"offered_req_s_per_gpu": 1.0 if level == "low" else 2.0,
"real_feasible": value >= 0.95,
"sim_feasible": False,
"state_residual": {"values": {"batch.mean": value}},
"engine": {
"common": {"batch": {"mean": value, "max": value, "cv": 0.0}},
"engine_only": {"kv": value},
},
}
def main() -> None:
cells = ("tp1_mns8", "tp1_mns16", "tp2_mns8", "tp2_mns16")
examples = [
example(cell, level, 0.1 + index / 10.0)
for level in ("low", "high")
for index, cell in enumerate(cells)
]
rows = MODULE.transitions(examples)
assert len(rows) == 48
assert any(row["source_level"] != row["target_level"] for row in rows)
assert all(row["source_cell"] != row["target_cell"] for row in rows)
assert len(rows[0]["hybrid_telemetry"]) > len(rows[0]["hybrid_base"])
assert len(rows[0]["direct_telemetry"]) > len(rows[0]["direct_base"])
for feature, target in (
("hybrid_base", "target_residual_delta"),
("hybrid_telemetry", "target_residual_delta"),
("direct_base", "target_real_pass_rate_delta"),
("direct_telemetry", "target_real_pass_rate_delta"),
):
prediction = MODULE.grouped_predictions(
rows, feature_name=feature, target_name=target, regularization=1.0
)
assert len(prediction) == len(rows)
print("telemetry residual transfer: PASS")
if __name__ == "__main__":
main()