Compare commits
3 Commits
f01819680d
...
0515ad8ecc
| Author | SHA1 | Date | |
|---|---|---|---|
| 0515ad8ecc | |||
| 791f7a8889 | |||
| 7a3631b528 |
71
docs/intervention-response-v0-protocol-20260714.md
Normal file
71
docs/intervention-response-v0-protocol-20260714.md
Normal file
@@ -0,0 +1,71 @@
|
||||
# Telemetry intervention-response v0 protocol
|
||||
|
||||
Status: **FROZEN BEFORE V0 ANALYSIS**.
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Claim boundary
|
||||
|
||||
The closed residual route asked whether one absolute engine-state snapshot can
|
||||
predict unmeasured configurations. V0 asks a different, narrower question:
|
||||
|
||||
> Does an adjacent, controlled MNS intervention produce an early engine-state
|
||||
> response that is distinguishable from same-config repeat noise?
|
||||
|
||||
Passing this gate only authorizes a matched real-GPU pilot. It does not prove
|
||||
that telemetry improves tuning, that any metric is a causal mediator, or that
|
||||
the response transfers to a new workload, topology, or knob family.
|
||||
|
||||
## Data and estimand
|
||||
|
||||
- Source: Phase 6 solo-authoritative Qwen3-30B-A3B/vLLM 0.24 Layer-1 streams.
|
||||
- Action pairs: primary runs at identical study hash, TP, sampling anchor, and
|
||||
request-order hash, with adjacent `MNS={8,16,32,64}` values.
|
||||
- Noise pairs: primary versus confirmation at the same complete config,
|
||||
anchor, and request-order hash. Only primary-to-confirmation pairs are used;
|
||||
confirmations are not combined into pseudo-independent all-pairs.
|
||||
- Fixed early windows: 5 seconds and 10 seconds from the measured interval
|
||||
start. All runs exceed 10 seconds, so early-stop censoring cannot change the
|
||||
telemetry window.
|
||||
- Full-run pass rate and feasibility are descriptive only because an early
|
||||
stop can make full elapsed durations differ.
|
||||
|
||||
The statistical unit is a run pair. Scheduler steps are summarized within a
|
||||
run and are never counted as independent trials.
|
||||
|
||||
## Frozen response gate
|
||||
|
||||
The directly measured gate features are scheduler-step rate, decode-batch
|
||||
mean, prefill-token fraction, waiting/running queue mean, KV-usage mean, and
|
||||
CUDA-graph padding fraction.
|
||||
|
||||
A feature qualifies at one horizon only if:
|
||||
|
||||
1. at least 75% of nonzero action deltas have the same sign;
|
||||
2. median absolute action delta is at least 2x the median absolute repeat
|
||||
delta; and
|
||||
3. at least 50% of action deltas exceed the repeat-noise absolute p95.
|
||||
|
||||
V0 opens a GPU pilot only if:
|
||||
|
||||
- there are exactly 17 frozen adjacent-MNS action pairs;
|
||||
- there are at least 20 primary/confirmation repeat pairs;
|
||||
- all identity, finite-value, counter, and ratio invariants pass; and
|
||||
- at least two gate features qualify at both 5 and 10 seconds.
|
||||
|
||||
Any data red flag stops the analysis before interpreting the response.
|
||||
|
||||
## If V0 passes
|
||||
|
||||
Register a dash0 pilot around a known scaling knee. The pilot must use the
|
||||
same request sequence and arrival times, one serving job at a time, one changed
|
||||
knob, randomized `A/B` versus `B/A` order, common non-censored measurement
|
||||
windows, and trial-level repetitions. It must compare a response-aware next
|
||||
action against an outcome-only policy under complete startup, warm-up, and
|
||||
H20-hour accounting.
|
||||
|
||||
## If V0 fails
|
||||
|
||||
Do not add telemetry fields or train a larger model. The current Layer-1 state
|
||||
does not identify even an MNS intervention above repeat noise on this task, so
|
||||
the telemetry-guided tuning route remains diagnostic only.
|
||||
157
docs/intervention-response-v0-results-20260714.md
Normal file
157
docs/intervention-response-v0-results-20260714.md
Normal file
@@ -0,0 +1,157 @@
|
||||
# Telemetry intervention-response v0/v1 results
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Decision
|
||||
|
||||
**STOP before a new H20 pilot.** The current Layer-1 aggregate telemetry does
|
||||
not identify a sufficiently general early response to an MNS intervention,
|
||||
and it does not improve action-efficacy prediction over exact external prefix
|
||||
outcomes on the available development tasks.
|
||||
|
||||
This is a negative result about the present state representation and
|
||||
experiment design. It does not establish that engine telemetry is useless for
|
||||
tuning, and it is not held-out evidence.
|
||||
|
||||
## Hypothesis and frozen test
|
||||
|
||||
The tested hypothesis was:
|
||||
|
||||
> With the workload and all non-MNS settings held fixed, increasing MNS causes
|
||||
> a 5--10 second engine-state response that is larger than same-config repeat
|
||||
> noise and that predicts whether the action makes the full run feasible.
|
||||
|
||||
A response feature had to satisfy all three frozen conditions at both 5 and
|
||||
10 seconds: at least 0.75 sign consistency, median absolute action effect at
|
||||
least 2x the repeat median, and at least 0.50 of action deltas above the repeat
|
||||
absolute p95. At least two features had to pass. A telemetry feature was
|
||||
decision-relevant only if its leave-one-repeat-out balanced accuracy was at
|
||||
least 0.75 and at least 0.15 above the best exact external prefix outcome.
|
||||
|
||||
## What was implemented
|
||||
|
||||
- A common-window analyzer over the existing per-scheduler-step Layer-1 stream.
|
||||
- Exact action pairing with request-order hash, offered load, TP, load role,
|
||||
and repetition held fixed.
|
||||
- Same-config repeat-noise estimation without treating scheduler steps as
|
||||
independent samples.
|
||||
- Exact 5/10-second request-prefix outcomes using monotonic completion times.
|
||||
- A one-feature leave-one-repeat-out efficacy audit; no multivariate model was
|
||||
fitted to the 12 examples.
|
||||
- Input hashes, stream hashes, frozen thresholds, pair-level deltas, and sanity
|
||||
invariants in machine-readable audit artifacts.
|
||||
- Trial-by-trial validation against the P1 manifest, plus content hashes for
|
||||
every result, request file, and Layer-1 stream.
|
||||
|
||||
## Experiment A: Phase-6 retrospective audit
|
||||
|
||||
Phase 6 supplied 17 adjacent-MNS actions and 29 same-config
|
||||
primary/confirmation pairs. No feature passed at either horizon, producing
|
||||
`STOP_NO_IDENTIFIABLE_RESPONSE`.
|
||||
|
||||
The confirmation sample is not a clean replication distribution: confirmations
|
||||
were selectively run after disputed primary outcomes. Several same-config
|
||||
pairs consequently followed radically different trajectories. This result
|
||||
therefore remains a valid failure of the frozen v0 gate, but it cannot by itself
|
||||
separate normal run variance from confirmation-selection bias.
|
||||
|
||||
## Experiment B: prospective-repeat confirmation
|
||||
|
||||
P1 supplied three pre-arranged, disjoint request bands for every cell/load.
|
||||
Exact matched actions exist for TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at
|
||||
low/high load and repetitions 1/2/3. This yields 12 action pairs and 24
|
||||
same-config consecutive-repeat pairs.
|
||||
|
||||
The 24 adjacent repeat differences share their middle repetition within each
|
||||
three-run group. They define a conservative empirical noise reference; they
|
||||
are not used as 24 independent samples in an inferential test.
|
||||
|
||||
The result is `STOP_NO_PROSPECTIVE_RESPONSE`: zero features passed the response
|
||||
gate at either horizon.
|
||||
|
||||
The strongest response was mean waiting-queue occupancy:
|
||||
|
||||
| Horizon | Sign consistency | Action/repeat median | Action above repeat p95 | Gate |
|
||||
|---|---:|---:|---:|---|
|
||||
| 5 s | 1.000 | 1.292x | 0.167 | fail |
|
||||
| 10 s | 1.000 | 2.611x | 0.250 | fail |
|
||||
|
||||
The direction is real enough to merit diagnosis, but the effect is not broad
|
||||
enough to guide a general action. It is large for TP4/high-load trials and
|
||||
small or absent in other regimes.
|
||||
|
||||
Full-run transitions contain six beneficial actions (`false -> true`) and six
|
||||
non-beneficial actions (three `false -> false`, three `true -> true`). The
|
||||
beneficial label is also perfectly confounded with TP4 in this small dataset,
|
||||
so it cannot support a topology-general claim.
|
||||
|
||||
| Horizon | Best telemetry delta | Balanced accuracy | Best external prefix delta | Balanced accuracy | Telemetry advantage |
|
||||
|---|---|---:|---|---:|---:|
|
||||
| 5 s | waiting queue | 0.750 | max TPOT / SLO | 0.833 | -0.083 |
|
||||
| 10 s | waiting queue | 0.750 | outstanding / admitted | 0.750 | 0.000 |
|
||||
|
||||
No telemetry feature reaches the preregistered `+0.15` incremental threshold.
|
||||
|
||||
## What this rules out
|
||||
|
||||
It rules out using the current vector of 5/10-second global means as a solid
|
||||
mechanism for choosing the next config. In particular, adding these aggregates
|
||||
to an LLM prompt or fitting a larger predictor would currently hide, rather
|
||||
than solve, the identifiability problem.
|
||||
|
||||
It does not rule out an instrumentation-aware tuner built around a deliberately
|
||||
excited local system. The existing runs were designed for endpoint/fidelity
|
||||
evaluation, not system identification: the MNS action is large, efficacy is
|
||||
confounded with TP, repeat bands contain different requests, and global means
|
||||
erase when queue buildup or service-rate changes occur.
|
||||
|
||||
## Required redesign before spending H20-hours
|
||||
|
||||
The next admissible experiment is a randomized, local A/B system-identification
|
||||
pilot around one fixed TP and one load knee:
|
||||
|
||||
1. Replay the exact same request sequence and arrival times for both endpoints.
|
||||
2. Use small adjacent actions and randomized `A/B` versus `B/A` order.
|
||||
3. Record event-aligned response curves, including queue growth/drain rate,
|
||||
prefill/decode service rate, and per-step service time, rather than only one
|
||||
global mean.
|
||||
4. Separate a mechanism gate (repeatable response) from the end-to-end gate:
|
||||
fewer trials or H20-hours to select a feasible near-optimal config than an
|
||||
outcome-only tuner.
|
||||
5. Hold out a second load/workload for the final policy comparison.
|
||||
|
||||
Until that design is frozen, a wider sweep would only generate more correlated
|
||||
observations and is not justified by the evidence above.
|
||||
|
||||
## Reproduction
|
||||
|
||||
```bash
|
||||
python3 runs/intervention-response-v0/test_analysis.py
|
||||
python3 runs/intervention-response-v0/test_p1_analysis.py
|
||||
|
||||
python3 runs/intervention-response-v0/analyze_phase6.py \
|
||||
--metrics runs/opprof-phase6/phase6/metrics.json \
|
||||
--raw-root runs/opprof-phase6/phase6/solo-authoritative/cells \
|
||||
--output runs/intervention-response-v0/phase6-audit.json
|
||||
|
||||
python3 runs/intervention-response-v0/analyze_p1.py \
|
||||
--run-root /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b \
|
||||
--manifest /home/gahow/phd/replayserve/runs/fidelity_p1_frontier_committed_20260714/real/p1b/pilot-manifest.json \
|
||||
--output runs/intervention-response-v0/p1-audit.json
|
||||
```
|
||||
|
||||
## Data sanity
|
||||
|
||||
- Phase 6: action pairs `n=17`, repeat pairs `n=29`, trials `n=66`; MNS
|
||||
action size min/max `8/32`, `3` distinct; action-state vectors `n=17`, `17`
|
||||
distinct; streams `n=12`, bytes min/max `12,745,297/52,957,710`, `12`
|
||||
distinct.
|
||||
- P1: action pairs `n=12`, repeat pairs `n=24`, trials `n=36`; MNS action
|
||||
size min/max `48/56`, `2` distinct; efficacy labels `n=12`, min/max `0/1`,
|
||||
`2` distinct; streams `n=6`, bytes min/max `17,449,143/29,431,988`, `6`
|
||||
distinct.
|
||||
- Checked invariants: exact action request hashes and offered loads match;
|
||||
all `36/36` P1 trials match the manifest; expected pair counts hold; all
|
||||
deltas are finite; non-negative counters and bounded ratios hold; per-config
|
||||
state vectors are not all identical; both efficacy classes are present. No
|
||||
red flags were observed.
|
||||
58
docs/intervention-response-v1-p1-protocol-20260714.md
Normal file
58
docs/intervention-response-v1-p1-protocol-20260714.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Intervention-response v1 prospective-repeat confirmation
|
||||
|
||||
Status: **FROZEN AFTER PHASE-6 V0 FAILURE AND BEFORE P1 RESPONSE ANALYSIS**.
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Why this is a new confirmation, not a relaxed V0
|
||||
|
||||
Phase-6 V0 failed its frozen global response gate. Its 29 same-config
|
||||
confirmations were triggered after disputed outcomes, and the resulting noise
|
||||
sample contains extreme trajectory divergence by construction. V0 remains
|
||||
failed and its thresholds are unchanged.
|
||||
|
||||
The already-completed P1 campaign supplies a distinct test: three
|
||||
prospectively scheduled, disjoint repeat bands for every cell/load. TP1 and
|
||||
TP4 use identical offered loads and exact request-order hashes across their MNS
|
||||
endpoints. V1 asks whether an MNS response is identifiable against this
|
||||
prospective workload-repeat noise, and whether that response predicts action
|
||||
efficacy beyond exact external prefix outcomes.
|
||||
|
||||
P1 is now development data. No result here is held-out or paper-facing.
|
||||
|
||||
## Frozen pairs
|
||||
|
||||
- Action pairs: TP1 `MNS 8 -> 64` and TP4 `MNS 16 -> 64`, at low/high load and
|
||||
repeat 1/2/3. Endpoints must have identical TP, offered rate, repeat role,
|
||||
and request-order hash. Expected `n=12`.
|
||||
- Repeat-noise pairs: consecutive pre-arranged repeat bands within each of six
|
||||
cells and low/high load: `rep1 -> rep2`, `rep2 -> rep3`. Expected `n=24`.
|
||||
Repeat bands intentionally contain different requests and therefore include
|
||||
workload-sampling noise rather than pretending to be identical trials.
|
||||
Adjacent differences share the middle run; the gate uses their empirical
|
||||
magnitude only and does not treat the 24 differences as independent samples
|
||||
for a p-value or confidence interval.
|
||||
- Prefix horizons: 5 and 10 seconds. Exact monotonic request completion times
|
||||
and the same Layer-1 intervals are used.
|
||||
|
||||
## Frozen gates
|
||||
|
||||
The response-identifiability thresholds are exactly the Phase-6 V0 thresholds:
|
||||
75% sign consistency, 2x median effect/repeat noise, and at least 50% of action
|
||||
deltas above repeat absolute p95. At least two response features must qualify
|
||||
at both horizons.
|
||||
|
||||
Action efficacy is one only for an infeasible-to-feasible full-run transition.
|
||||
The 12 action pairs must contain at least four examples of each class.
|
||||
|
||||
For decision relevance, each individual external-outcome response feature and
|
||||
each individual telemetry-response feature is evaluated by leave-one-repeat-
|
||||
band-out threshold fitting. This intentionally avoids a multivariate model on
|
||||
12 examples. At least one telemetry feature must, at both horizons:
|
||||
|
||||
1. reach balanced accuracy at least 0.75; and
|
||||
2. exceed the best external-outcome response feature by at least 0.15.
|
||||
|
||||
Only if data validity, response identifiability, and incremental decision
|
||||
relevance all pass does V1 open a newly registered matched GPU pilot. No
|
||||
threshold or feature is changed after observing V1.
|
||||
@@ -0,0 +1,84 @@
|
||||
# Phase-aware telemetry intervention-response v2 protocol
|
||||
|
||||
Status: **FROZEN BEFORE THE SYSTEMATIC 10% DECILE AUDIT**.
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Correction to v0/v1
|
||||
|
||||
The 5/10-second analyses tested an ultra-early verifier. They did not test
|
||||
whether telemetry observed after the engine has developed queue, batch, and KV
|
||||
state can guide tuning. The P1 replay lasts 60 seconds after time scaling, and
|
||||
the 5/10-second prefixes contain only a small fraction of its requests.
|
||||
|
||||
V2 therefore replaces absolute cutoffs with replay phase. The old audits and
|
||||
their negative decisions remain immutable, but their claim is narrowed to the
|
||||
first 5/10 seconds.
|
||||
|
||||
## Historical corrective audit
|
||||
|
||||
The historical audit is development-only and cannot become confirmatory after
|
||||
the horizon concern was observed.
|
||||
|
||||
- Infer each trial's intended replay duration as selected requests divided by
|
||||
offered requests per second. All trials must agree.
|
||||
- Find every complete 10% replay decile supported by every trial. Analyze all
|
||||
such deciles; selecting only the best horizon is forbidden.
|
||||
- At each decile report both:
|
||||
- cumulative state from replay start to the checkpoint; and
|
||||
- the non-overlapping 10%-wide state block ending at the checkpoint.
|
||||
- Report admitted/completed request coverage, response-versus-repeat statistics,
|
||||
telemetry versus external-outcome efficacy, and per-feature trajectory drift.
|
||||
- Reuse the frozen v1 action and repeat pairs and the frozen response and
|
||||
incremental-efficacy thresholds. These thresholds are descriptive in V2;
|
||||
passing one post-hoc horizon does not open a contribution claim.
|
||||
|
||||
If early stopping prevents complete observation of the replay phases, the
|
||||
historical decision is `REQUIRES_UNCENSORED_PHASE_AWARE_PILOT`, independent of
|
||||
which early decile looks best.
|
||||
|
||||
## Uncensored matched pilot
|
||||
|
||||
The pilot is a mechanism gate, not paper evidence.
|
||||
|
||||
- Hardware/engine/model: solo placement on dash0, 4 NVIDIA H20 GPUs, patched
|
||||
vLLM `0.24.1.dev3+opprof`, Qwen3-30B-A3B, fixed `TP=4`.
|
||||
- Action: `MNS 16 -> 64`; topology, model, engine build, workload, arrival
|
||||
sequence, offered load, and all other settings remain fixed.
|
||||
- Workload: `chat_w20260312_1000` at replay-time scale `0.5`, hence 300 seconds.
|
||||
- Offered loads per GPU: `1.5`, `2.125`, and `3.125` requests/s. These supply a
|
||||
low control and the two already-observed P1 pressure regimes.
|
||||
- Repetitions: three disjoint session bands, exact request sequence matched
|
||||
across action endpoints. Endpoint order alternates `A/B`, `B/A`, `A/B`;
|
||||
load order is counter-rotated across repetitions.
|
||||
- A fresh server receives the accepted long-request warm-up and a bounded
|
||||
burn-in before each measured session.
|
||||
- SLO-unrecoverable early stop is disabled. Every run must observe the full
|
||||
300-second arrival window; a separate 360-second safety deadline may mark a
|
||||
run invalid but cannot manufacture a full-run label.
|
||||
- Cumulative checkpoints: 10%, 25%, 50%, 75%, and 100%, or 30/75/150/225/300
|
||||
seconds. Quarter blocks are analyzed separately from cumulative means.
|
||||
- Placement is serialized. Co-location remains forbidden because Phase 6
|
||||
observed material co-location-induced outcome shifts.
|
||||
- Hard cap: 8 H20-hours including startup, warm-up, burn-in, invalid attempts,
|
||||
and cleanup.
|
||||
|
||||
## Gates
|
||||
|
||||
Data validity requires complete 300-second Layer-1 coverage, zero dropped
|
||||
records, exact request/arrival/length hashes across action endpoints, monotonic
|
||||
timestamps, full request accounting, idle GPUs before and after each session,
|
||||
and no co-resident GPU process.
|
||||
|
||||
Mechanism evidence requires at least two telemetry features whose matched action
|
||||
response exceeds same-config repeat noise at two consecutive checkpoints under
|
||||
the unchanged v1 response thresholds. The same features must have a consistent
|
||||
direction in at least two of the three load regimes.
|
||||
|
||||
Decision evidence additionally requires both action-efficacy classes and an
|
||||
instrumentation feature that reaches leave-one-repetition-out balanced accuracy
|
||||
at least 0.75 and exceeds the best external prefix outcome by at least 0.15.
|
||||
Without label balance the pilot can adjudicate mechanism evidence only.
|
||||
|
||||
No H20 run is launched if the local analyzer/tests, manifest preflight, GPU
|
||||
probe, command dry-run, projected cost, or cleanup plan fails.
|
||||
286
docs/telemetry-residual-tuning-roadmap-20260714.md
Normal file
286
docs/telemetry-residual-tuning-roadmap-20260714.md
Normal file
@@ -0,0 +1,286 @@
|
||||
# Telemetry-conditioned residual tuning roadmap
|
||||
|
||||
Status: **R0 COMPLETE / FAILED; R1 AND R2 CLOSED FOR THIS MODEL**.
|
||||
|
||||
Date: 2026-07-14 (Asia/Singapore).
|
||||
|
||||
## Research question and claim boundary
|
||||
|
||||
The question is whether a small number of real engine observations can correct
|
||||
a simulator's task-specific error over **unmeasured configurations**, and
|
||||
whether that correction reduces the real-GPU cost of finding a high
|
||||
SLO-goodput serving configuration.
|
||||
|
||||
The intended headline claim, if the evidence supports it, is:
|
||||
|
||||
> An engine-state-conditioned residual model turns a simulator prediction into
|
||||
> a task-specific posterior over unmeasured serving configurations, allowing a
|
||||
> sequential tuner to reach near-oracle SLO-goodput with materially fewer
|
||||
> H20-hours than simulator-only and outcome-only tuning.
|
||||
|
||||
Classification accuracy, simulator-error diagnosis, and telemetry overhead are
|
||||
supporting evidence. None is an end-to-end tuning contribution by itself.
|
||||
|
||||
The following method is closed and will not be revived under another name:
|
||||
per-candidate five-second accept/reject as the headline contribution. The P1
|
||||
result showed only 1.426% cost reduction in the frozen `k=2` workflow.
|
||||
|
||||
## Two models, one evaluation
|
||||
|
||||
Both branches use the same legal candidate set, real measurements, task split,
|
||||
cost accounting, and acquisition function.
|
||||
|
||||
### Simulator-residual branch (primary)
|
||||
|
||||
For measured anchor `c_t` and unmeasured candidate `c'`:
|
||||
|
||||
```text
|
||||
y_hat(c') = y_real(c_t)
|
||||
+ [y_sim(c') - y_sim(c_t)]
|
||||
+ f(state_real(c_t) - state_sim(c_t), c' - c_t, workload, SLO)
|
||||
```
|
||||
|
||||
The simulator delta is the prior. The learned model may correct it only with
|
||||
training-supported state/config transitions; uncertainty or distribution shift
|
||||
must shrink the correction back toward the simulator prior.
|
||||
|
||||
### Telemetry-only branch (mandatory)
|
||||
|
||||
```text
|
||||
y_hat(c') = y_real(c_t)
|
||||
+ g(state_real(c_t), c' - c_t, workload, SLO)
|
||||
```
|
||||
|
||||
This branch tests whether the simulator is actually necessary. It does not
|
||||
use a hand-authored bottleneck-to-knob rule.
|
||||
|
||||
### Search policy
|
||||
|
||||
Legal configurations are enumerated independently of telemetry. A generic
|
||||
cost-aware acquisition rule ranks candidates from predicted improvement,
|
||||
uncertainty, and measured H20 cost. The current production harness's
|
||||
bottleneck scores, topology-first ordering, and hand-set relief constants are
|
||||
not consumed by either branch. The validator may enforce legality,
|
||||
full-config no-repeat, failure accounting, and resource caps only.
|
||||
|
||||
## Hypotheses
|
||||
|
||||
| ID | Hypothesis | Direct test | Failure meaning |
|
||||
|---|---|---|---|
|
||||
| H0 | Existing artifacts can express a common, direct-measurement state without heuristic labels. | Engine/simulator extractor coverage and invariants. | Route is not currently implementable. |
|
||||
| H1 | Simulator errors are predictable from engine/simulator state discrepancy at measured anchors. | Task-held-out pairwise inversion correction and new-inversion rate. | Telemetry is diagnostic but cannot correct the surface. |
|
||||
| H2 | Telemetry alone predicts useful config transitions beyond outcome-only history. | Telemetry-only versus real-outcome-only sequential replay. | Direct telemetry-guided tuning has no independent value. |
|
||||
| H3 | Residual correction changes actual tuning decisions and cost. | H20-hours to 95% oracle and regret AUC against the strongest safe baseline. | No system contribution even if H1/H2 prediction metrics improve. |
|
||||
|
||||
## Common-state contract
|
||||
|
||||
Only directly observed or exactly reconstructed quantities are admitted.
|
||||
|
||||
| Quantity | vLLM Layer-1 | Frontier | R0 status |
|
||||
|---|---|---|---|
|
||||
| Scheduled requests / batch size | Per scheduler step | Existing per-batch metric, disabled in P1 output | Common after CPU replay |
|
||||
| Scheduled prefill/decode tokens | Per scheduler step | Existing per-batch metrics | Common after CPU replay |
|
||||
| Scheduler/batch rate | Monotonic step timestamps | Batch count / simulated duration | Common after CPU replay |
|
||||
| Waiting queue area | Time-weighted queue gauge | Sum of request waiting times | Common aggregate |
|
||||
| Running request area | Time-weighted running gauge | Sum of E2E minus waiting time | Common aggregate, semantics audited |
|
||||
| Preemption count | Per step | Per request | Common |
|
||||
| KV usage/headroom | Exact blocks and ratio | Not in committed output | Engine-only until exact reconstruction exists |
|
||||
| CUDA graph mode/padding | Exact per step | Not modeled | Engine-only omitted-mechanism signal |
|
||||
| Request TTFT/TPOT/pass rate | Exact real outcomes | Exact simulated request metrics | Common outcome, not state |
|
||||
|
||||
Unavailable fields remain null. They cannot be imputed from a human
|
||||
`prefill/decode/queueing` label.
|
||||
|
||||
Frontier already contains the required detailed batch and timestamped
|
||||
stage-batch ledger output. P1 disabled it for artifact size. R0 replays the
|
||||
same immutable fixtures with the existing output flags enabled; it does not
|
||||
change the simulator model or calibration.
|
||||
|
||||
## Data separation
|
||||
|
||||
- Phase 6 / `chat_w20260311_1000`: development only.
|
||||
- P1 / `chat_w20260312_1000`: development only.
|
||||
- R1 / `chat_w20260313_1000`: new development surface.
|
||||
- R2: trace windows not used for feature, model, threshold, candidate-space,
|
||||
cutoff, or acquisition decisions.
|
||||
- Splits are by complete workload/SLO task. Anchor- or pair-level random
|
||||
splits are prohibited.
|
||||
- Sequential-policy seeds measure algorithmic variability; they are not
|
||||
counted as independent system tasks.
|
||||
|
||||
The two existing development tasks have an important limitation: the now-
|
||||
available SLO-gated simulator reading already retains the real oracle at its
|
||||
top rank/tie. They therefore cannot establish a positive end-to-end ranking
|
||||
claim. They are used for plumbing, known false-feasible cases, and negative
|
||||
evidence. R1 must be run as an unbiased complete surface, not selected after
|
||||
observing simulator success or failure.
|
||||
|
||||
## Step-by-step roadmap
|
||||
|
||||
### R0.1 — Inventory and roadmap
|
||||
|
||||
Deliverables:
|
||||
|
||||
- this roadmap;
|
||||
- rolling untracked `ONGOING.md`;
|
||||
- exact engine/simulator field and artifact inventory.
|
||||
|
||||
Gate: every claimed input has an authoritative file path and provenance.
|
||||
|
||||
### R0.2 — Common-state plumbing
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `runs/telemetry-residual/common_state.py`;
|
||||
- synthetic correctness tests;
|
||||
- one exact P1 Frontier replay with individual batch metrics and the full
|
||||
stage-batch ledger enabled;
|
||||
- paired engine/simulator state summary for the same fixture.
|
||||
|
||||
Gate:
|
||||
|
||||
- replay request count and SLO scorer exactly agree with the committed replay;
|
||||
- batch/ledger outputs are non-empty;
|
||||
- all counters are non-negative, ratios bounded, times monotonic;
|
||||
- no GPU is visible to Frontier;
|
||||
- output volume is practical before expanding to twelve replays.
|
||||
|
||||
### R0.3 — Development residual/headroom audit
|
||||
|
||||
Use all frozen P1 primary fixtures and corresponding engine intervals. Produce:
|
||||
|
||||
- common-state residuals per anchor;
|
||||
- simulator-error labels and continuous SLO/goodput residuals;
|
||||
- ordered source/target diagnostic that removes both config identities from
|
||||
both roles in every training fold;
|
||||
- oracle upper bound for cross-candidate correction;
|
||||
- explicit comparison with simulator+outcome and telemetry-only features.
|
||||
|
||||
R0 is a feasibility gate, not headline evidence. Proceed to R1 only if:
|
||||
|
||||
1. state features are collected with the measured source anchor, vary across
|
||||
cells, and are available before any target config is evaluated;
|
||||
2. at least one known simulator error has a state discrepancy not exposed by
|
||||
the matched external prefix outcome;
|
||||
3. a prior-preserving model can correct development errors without introducing
|
||||
a larger number of new errors under regularization sensitivity;
|
||||
4. an oracle cross-candidate correction has at least 15% sequential tuning-cost
|
||||
headroom under full startup/warm-up accounting.
|
||||
|
||||
### R0 result and decision
|
||||
|
||||
R0 completed without a data-validity red flag, but failed condition 3. The
|
||||
decision is **STOP_BEFORE_R1**; no H20 job was launched for this route.
|
||||
|
||||
- All 12 detailed Frontier CPU replays exactly reproduced their committed SLO
|
||||
scorers. Runtime was 23.943--54.786 seconds per replay, detailed artifacts
|
||||
were 4.12--13.53 MB, CUDA visibility was empty, and there were zero failures.
|
||||
- The paired surface contains 12 real/sim anchors, two known simulator
|
||||
false-feasible anchors, and 120 legal cross-config ordered transitions. A
|
||||
fold removes both the source and target TP/MNS identity from source and
|
||||
target roles; the two offered-load anchors remain part of the same task.
|
||||
- Raw Frontier feasibility is 83.33% on the repeated transition view. The
|
||||
structurally correct hybrid model uses
|
||||
`r_target = r_source + delta_r`; the direct model uses
|
||||
`y_target = y_source + delta_y` and never reads simulator fields.
|
||||
- Direct telemetry is not robust relative to real-outcome-only: its accuracy
|
||||
delta over L2 `{0.1,1,10,100}` is `{-0.83,+1.67,0,-4.17}` percentage points,
|
||||
and its best absolute accuracy is 54.17%, below the raw simulator's 83.33%.
|
||||
- Hybrid telemetry raises classification accuracy over the corresponding
|
||||
simulator+outcome transition regression by 1.67--4.17 percentage points,
|
||||
but worsens pass-rate RMSE by 0.141--0.201 and MAE by 0.084--0.125. Its full
|
||||
correction reaches only 46.67--53.33% absolute accuracy.
|
||||
- Across 24 nonzero `(L2, raw-simulator-prior weight)` combinations, no model
|
||||
both corrects an existing simulator error without more new errors and avoids
|
||||
worsening RMSE/MAE. Whenever a correction fixes at least one error, it
|
||||
corrupts at least 11 previously correct transitions.
|
||||
- A perfect correction could skip the frozen simulator rank-2 real final and
|
||||
save 0.043469 H20-hours: 15.45% of the prospective online `k=2` cost, or
|
||||
14.40% when the prior failed launch is charged. On this development task the
|
||||
simulator top-1 already is the real oracle with zero regret, so headroom
|
||||
versus the observed-safe top-1 baseline is 0%.
|
||||
|
||||
The result does not prove that engine telemetry is useless. It shows that the
|
||||
current one-task anchor-transition evidence cannot support either a safe
|
||||
simulator-residual tuner or a simulator-free telemetry tuner. A larger model
|
||||
or an R1 run would add capacity/data after a failed gate and is therefore not
|
||||
authorized under this roadmap.
|
||||
|
||||
### R1 — New development surface
|
||||
|
||||
Status: **NOT LAUNCHED; CLOSED BY R0**.
|
||||
|
||||
Frozen starting setup:
|
||||
|
||||
- host: dash0, eight NVIDIA H20 GPUs;
|
||||
- cells run solo; no co-location for SLO verdicts;
|
||||
- patched vLLM 0.24.1.dev3, Qwen3-30B-A3B BF16;
|
||||
- trace: `chat_w20260313_1000`;
|
||||
- output tokens: exactly 128;
|
||||
- SLO: stepped TTFT 2/4/6 seconds, TPOT 50 ms, pass rate at least 0.95;
|
||||
- config surface: TP `{1,2,4}` × MNS `{8,16,32,64}`;
|
||||
- hard campaign cap: 4 H20-hours.
|
||||
|
||||
The load ladder, repetitions, randomized order, exact commands, expected wall
|
||||
time, and artifact paths are frozen only after R0. A resolved echo is required
|
||||
before launch.
|
||||
|
||||
R1 passes only if a frozen sequential replay shows at least 15% E2E H20-hour
|
||||
headroom over the strongest safe baseline with final regret at most 5%. R1 is
|
||||
development evidence and cannot be reported as the held-out result.
|
||||
|
||||
### R2 — Held-out sequential tuning
|
||||
|
||||
Status: **NOT LAUNCHED; CLOSED BY R0**.
|
||||
|
||||
Required baselines:
|
||||
|
||||
1. random search;
|
||||
2. real-outcome-only Bayesian/sequential search;
|
||||
3. Frontier ranking plus real top-k final;
|
||||
4. simulator plus real-outcome residual;
|
||||
5. telemetry-only transition tuner;
|
||||
6. simulator plus telemetry residual tuner;
|
||||
7. complete real surface as oracle, not as a cost competitor.
|
||||
|
||||
Primary metric: end-to-end H20-hours to first reach 95% of the real full-surface
|
||||
SLO-goodput oracle. Secondary metrics are cost-normalized regret AUC, final
|
||||
regret at fixed budgets, oracle false-prune, wall time, and per-task regressions.
|
||||
|
||||
The route is successful only if the winning telemetry method reduces the
|
||||
primary cost by at least 20% versus the strongest safe baseline and ends within
|
||||
5% regret on every headline task. If hybrid beats telemetry-only by at least
|
||||
10%, simulator residual correction is the primary method. If telemetry-only
|
||||
is within 5% or better, the simulator dependency is removed. If neither clears
|
||||
the contribution bar, the route is closed and telemetry remains a diagnostic
|
||||
facility only.
|
||||
|
||||
## Cost discipline
|
||||
|
||||
- R0 simulator work is CPU-only and must set empty CUDA visibility.
|
||||
- R1 cannot exceed 4 H20-hours.
|
||||
- R2 receives no budget until R1 passes.
|
||||
- Startup, warm-up, burn-in, failed launches, real probes, continuation, and
|
||||
final validation are charged. Benchmark-only annotation repeats are
|
||||
reported separately and cannot disappear from campaign accounting.
|
||||
|
||||
## Final R0 sanity block
|
||||
|
||||
| Data | n | Min | Max | Distinct | Checked invariant |
|
||||
|---|---:|---:|---:|---:|---|
|
||||
| Phase 6 cells | 12 | TP1/MNS8 | TP4/MNS64 | 12 | Surface not identical; solo SLO tier authoritative |
|
||||
| Phase 6 Layer-1 primary steps | 37 streams | 343 | 12,103 | 37 | Contiguous; zero drops |
|
||||
| P1 primary anchors | 12 | infeasible | feasible | 2 labels | 7 feasible / 5 infeasible |
|
||||
| P1 Frontier runtime | 12 | 24.093 s | 54.575 s | 12 | CPU-only; zero failures |
|
||||
| Detailed Frontier replay runtime | 12 | 23.943 s | 54.786 s | 12 | Exact committed scorers; CUDA hidden |
|
||||
| Detailed artifact bytes | 12 | 4,123,724 | 13,527,776 | 12 | Non-negative; practical CPU replay size |
|
||||
| Cross-config transitions | 120 | real pass 0.1067 | real pass 1.0 | 6 outcomes | Both endpoint config identities held out |
|
||||
| State residual vectors | 12 | 16 fields | 16 fields | 12 vectors | Finite; no missing common field |
|
||||
| R0 E2E cost values | 4 | 0.237914 | 0.301935 H20-h | 4 | Non-negative; `k=1/2`, online/conservative |
|
||||
|
||||
Checked invariants: non-negative counts and costs; pass rates in `[0,1]`;
|
||||
simulator results not all identical; exact request count/hash agreement; Layer-1
|
||||
step continuity and zero drops; no co-resident SLO measurements; no calibration
|
||||
or evaluation split reuse for a future headline claim. No current red flag
|
||||
invalidates R0 plumbing. The R0 tuning gate itself failed because safe
|
||||
prior-preserving correction was absent.
|
||||
691
runs/intervention-response-v0/analyze_p1.py
Normal file
691
runs/intervention-response-v0/analyze_p1.py
Normal file
@@ -0,0 +1,691 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prospective-repeat confirmation of the intervention-response hypothesis.
|
||||
|
||||
P1 contains three pre-arranged, disjoint request bands per cell/load. TP1 and
|
||||
TP4 use matched offered loads and request sequences across their MNS endpoints.
|
||||
This script asks both whether the MNS response exceeds prospective repeat noise
|
||||
and whether an early telemetry delta predicts full-run action efficacy beyond
|
||||
the corresponding external-outcome delta.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from statistics import fmean
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
COMMON_STATE_DIR = HERE.parent / "telemetry-residual"
|
||||
sys.path.insert(0, str(COMMON_STATE_DIR))
|
||||
|
||||
from common_state import load_jsonl, summarize_engine # noqa: E402
|
||||
|
||||
|
||||
def _load_v0():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_phase6_v0", HERE / "analyze_phase6.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
V0 = _load_v0()
|
||||
SCHEMA = "intervention-response-p1-confirmation-v1"
|
||||
HORIZONS_S = V0.HORIZONS_S
|
||||
EXPECTED_ACTION_PAIRS = 12
|
||||
EXPECTED_REPEAT_PAIRS = 24
|
||||
MIN_EFFICACY_CLASS = 4
|
||||
MIN_EFFICACY_BALANCED_ACCURACY = 0.75
|
||||
MIN_EFFICACY_DELTA_OVER_OUTCOME = 0.15
|
||||
OUTCOME_FEATURES = (
|
||||
"admitted_fraction",
|
||||
"completed_over_admitted",
|
||||
"completed_pass_rate",
|
||||
"completed_fail_fraction_of_total",
|
||||
"outstanding_over_admitted",
|
||||
"ttft_max_over_slo_max",
|
||||
"ttft_mean_over_slo_max",
|
||||
"tpot_max_over_slo",
|
||||
"tpot_mean_over_slo",
|
||||
"admitted_input_tokens_mean_over_limit",
|
||||
)
|
||||
RUN_PATTERN = re.compile(r"^(low|high)-rep([123])$")
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _prefix_outcome(
|
||||
result: Mapping[str, Any],
|
||||
requests: list[dict[str, Any]],
|
||||
horizon_s: float,
|
||||
) -> dict[str, float]:
|
||||
admitted = [request for request in requests if float(request["arrival_s"]) <= horizon_s]
|
||||
completed = [
|
||||
request
|
||||
for request in requests
|
||||
if request.get("completed_elapsed_s") is not None
|
||||
and float(request["completed_elapsed_s"]) <= horizon_s
|
||||
]
|
||||
if not admitted:
|
||||
raise ValueError("prefix contains no admitted request")
|
||||
admitted_ids = {str(request["request_id"]) for request in admitted}
|
||||
if any(str(request["request_id"]) not in admitted_ids for request in completed):
|
||||
raise ValueError("completed request was not admitted in the prefix")
|
||||
passed = sum(bool(request["slo_pass"]) for request in completed)
|
||||
ttft = [float(request["ttft_ms"]) for request in completed]
|
||||
tpot = [float(request["tpot_ms"]) for request in completed]
|
||||
total = int(result["selection"]["count"])
|
||||
if total != len(requests):
|
||||
raise ValueError("request JSONL count does not match the result")
|
||||
return {
|
||||
"admitted_fraction": len(admitted) / total,
|
||||
"completed_over_admitted": len(completed) / len(admitted),
|
||||
"completed_pass_rate": passed / max(1, len(completed)),
|
||||
"completed_fail_fraction_of_total": (len(completed) - passed) / total,
|
||||
"outstanding_over_admitted": (len(admitted) - len(completed)) / len(admitted),
|
||||
"ttft_max_over_slo_max": max(ttft, default=0.0) / 6000.0,
|
||||
"ttft_mean_over_slo_max": fmean(ttft) / 6000.0 if ttft else 0.0,
|
||||
"tpot_max_over_slo": max(tpot, default=0.0) / 50.0,
|
||||
"tpot_mean_over_slo": fmean(tpot) / 50.0 if tpot else 0.0,
|
||||
"admitted_input_tokens_mean_over_limit": fmean(
|
||||
float(request["raw_input_tokens"]) for request in admitted
|
||||
)
|
||||
/ 8192.0,
|
||||
}
|
||||
|
||||
|
||||
def load_trials(
|
||||
run_root: Path,
|
||||
*,
|
||||
horizons_s: tuple[float, ...] = HORIZONS_S,
|
||||
) -> tuple[dict[float, list[dict[str, Any]]], list[dict[str, Any]]]:
|
||||
by_horizon = {horizon: [] for horizon in horizons_s}
|
||||
streams = []
|
||||
for cell_dir in sorted((run_root / "cells").iterdir()):
|
||||
if not cell_dir.is_dir():
|
||||
continue
|
||||
stream_paths = sorted((cell_dir / "opprof").glob("*.jsonl"))
|
||||
if len(stream_paths) != 1:
|
||||
raise ValueError(f"{cell_dir}: expected one Layer-1 stream")
|
||||
stream_path = stream_paths[0]
|
||||
stream = load_jsonl(stream_path)
|
||||
streams.append(
|
||||
{
|
||||
"path": str(stream_path.resolve()),
|
||||
"sha256": sha256_file(stream_path),
|
||||
"bytes": stream_path.stat().st_size,
|
||||
}
|
||||
)
|
||||
for run_dir in sorted(cell_dir.iterdir()):
|
||||
match = RUN_PATTERN.match(run_dir.name)
|
||||
if match is None:
|
||||
continue
|
||||
level, replicate_text = match.groups()
|
||||
replicate = int(replicate_text)
|
||||
result_path = run_dir / "result.json"
|
||||
requests_path = run_dir / "requests.jsonl"
|
||||
result = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
requests = load_jsonl(requests_path)
|
||||
elapsed_s = float(result["interval"]["elapsed_s"])
|
||||
start_ns = int(result["interval"]["start_mono_ns"])
|
||||
for horizon_s in horizons_s:
|
||||
if elapsed_s < horizon_s:
|
||||
raise ValueError(
|
||||
f"{result_path}: elapsed {elapsed_s} shorter than {horizon_s}s"
|
||||
)
|
||||
state = V0.flatten_state(
|
||||
summarize_engine(
|
||||
stream,
|
||||
start_ns=start_ns,
|
||||
end_ns=start_ns + int(horizon_s * 1e9),
|
||||
request_count=int(result["selection"]["count"]),
|
||||
)
|
||||
)
|
||||
by_horizon[horizon_s].append(
|
||||
{
|
||||
"trial_id": str(result_path.relative_to(run_root)),
|
||||
"cell": str(result["cell"]),
|
||||
"tp": int(result["tp"]),
|
||||
"mns": int(result["mns"]),
|
||||
"level": level,
|
||||
"replicate": replicate,
|
||||
"offered_rate_per_gpu": float(
|
||||
result["selection"]["offered_req_s_per_gpu"]
|
||||
),
|
||||
"request_hash": str(
|
||||
result["selection"]["request_id_order_sha256"]
|
||||
),
|
||||
"request_count": int(result["selection"]["count"]),
|
||||
"result_sha256": sha256_file(result_path),
|
||||
"requests_sha256": sha256_file(requests_path),
|
||||
"full_pass_rate": float(result["pass_rate"]),
|
||||
"full_feasible": bool(result["feasible"]),
|
||||
"early_stopped": bool(result["early_stopped"]),
|
||||
"state": state,
|
||||
"outcome": _prefix_outcome(result, requests, horizon_s),
|
||||
}
|
||||
)
|
||||
return by_horizon, streams
|
||||
|
||||
|
||||
def validate_manifest(
|
||||
trials: list[dict[str, Any]], manifest_path: Path
|
||||
) -> dict[str, Any]:
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
if manifest.get("schema") != "fidelity-prefix-pilot-manifest-v1":
|
||||
raise ValueError("unexpected P1 manifest schema")
|
||||
cells = manifest.get("cells")
|
||||
if not isinstance(cells, dict):
|
||||
raise ValueError("P1 manifest has no cell mapping")
|
||||
seen = set()
|
||||
for trial in trials:
|
||||
key = (trial["cell"], trial["level"], trial["replicate"])
|
||||
if key in seen:
|
||||
raise ValueError(f"duplicate P1 trial identity: {key}")
|
||||
seen.add(key)
|
||||
try:
|
||||
cell = cells[trial["cell"]]
|
||||
selection = cell["targets"][trial["level"]]["selections"][
|
||||
f"{trial['level']}{trial['replicate']}"
|
||||
]
|
||||
except (KeyError, TypeError) as error:
|
||||
raise ValueError(f"trial is absent from P1 manifest: {key}") from error
|
||||
if int(cell["tp"]) != trial["tp"] or int(cell["mns"]) != trial["mns"]:
|
||||
raise ValueError(f"trial config disagrees with P1 manifest: {key}")
|
||||
if str(selection["request_id_order_sha256"]) != trial["request_hash"]:
|
||||
raise ValueError(f"trial request hash disagrees with P1 manifest: {key}")
|
||||
if int(selection["selected_count"]) != trial["request_count"]:
|
||||
raise ValueError(f"trial request count disagrees with P1 manifest: {key}")
|
||||
if not math.isclose(
|
||||
float(selection["offered_req_s_per_gpu"]),
|
||||
trial["offered_rate_per_gpu"],
|
||||
rel_tol=0.0,
|
||||
abs_tol=1e-12,
|
||||
):
|
||||
raise ValueError(f"trial offered load disagrees with P1 manifest: {key}")
|
||||
expected = {
|
||||
(cell_name, level, replicate)
|
||||
for cell_name in cells
|
||||
for level in ("low", "high")
|
||||
for replicate in (1, 2, 3)
|
||||
}
|
||||
if seen != expected:
|
||||
missing = sorted(expected - seen)
|
||||
unexpected = sorted(seen - expected)
|
||||
raise ValueError(
|
||||
f"P1 trial/manifest coverage mismatch: missing={missing}, "
|
||||
f"unexpected={unexpected}"
|
||||
)
|
||||
return {
|
||||
"schema": str(manifest["schema"]),
|
||||
"expected_trials": len(expected),
|
||||
"matched_trials": len(seen),
|
||||
}
|
||||
|
||||
|
||||
def _delta(
|
||||
source: Mapping[str, Any],
|
||||
target: Mapping[str, Any],
|
||||
features: Iterable[str],
|
||||
) -> dict[str, float]:
|
||||
return {
|
||||
feature: float(target[feature]) - float(source[feature])
|
||||
for feature in features
|
||||
}
|
||||
|
||||
|
||||
def _action_pair(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, Any]:
|
||||
if source["tp"] != target["tp"]:
|
||||
raise ValueError("action endpoints changed TP")
|
||||
if source["level"] != target["level"] or source["replicate"] != target["replicate"]:
|
||||
raise ValueError("action endpoints changed load role or repeat")
|
||||
if source["request_hash"] != target["request_hash"]:
|
||||
raise ValueError("action endpoints changed request sequence")
|
||||
if not math.isclose(
|
||||
source["offered_rate_per_gpu"],
|
||||
target["offered_rate_per_gpu"],
|
||||
rel_tol=0.0,
|
||||
abs_tol=1e-12,
|
||||
):
|
||||
raise ValueError("action endpoints changed offered load")
|
||||
if source["mns"] >= target["mns"]:
|
||||
raise ValueError("action must increase MNS")
|
||||
beneficial = target["full_feasible"] and not source["full_feasible"]
|
||||
return {
|
||||
"kind": "matched_mns_increase",
|
||||
"group": {
|
||||
"tp": source["tp"],
|
||||
"level": source["level"],
|
||||
"replicate": source["replicate"],
|
||||
"request_hash": source["request_hash"],
|
||||
"offered_rate_per_gpu": source["offered_rate_per_gpu"],
|
||||
},
|
||||
"source": {
|
||||
key: source[key]
|
||||
for key in (
|
||||
"trial_id",
|
||||
"result_sha256",
|
||||
"requests_sha256",
|
||||
"cell",
|
||||
"mns",
|
||||
"full_pass_rate",
|
||||
"full_feasible",
|
||||
"early_stopped",
|
||||
)
|
||||
},
|
||||
"target": {
|
||||
key: target[key]
|
||||
for key in (
|
||||
"trial_id",
|
||||
"result_sha256",
|
||||
"requests_sha256",
|
||||
"cell",
|
||||
"mns",
|
||||
"full_pass_rate",
|
||||
"full_feasible",
|
||||
"early_stopped",
|
||||
)
|
||||
},
|
||||
"delta_state": _delta(source["state"], target["state"], V0.ALL_FEATURES),
|
||||
"delta_outcome": _delta(source["outcome"], target["outcome"], OUTCOME_FEATURES),
|
||||
"full_action_efficacy": int(beneficial),
|
||||
"full_feasibility_transition": (
|
||||
f"{str(source['full_feasible']).lower()}->"
|
||||
f"{str(target['full_feasible']).lower()}"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def _repeat_pair(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, Any]:
|
||||
if source["cell"] != target["cell"] or source["level"] != target["level"]:
|
||||
raise ValueError("repeat endpoints changed config or load role")
|
||||
if target["replicate"] != source["replicate"] + 1:
|
||||
raise ValueError("repeat endpoints are not consecutive pre-arranged bands")
|
||||
if not math.isclose(
|
||||
source["offered_rate_per_gpu"],
|
||||
target["offered_rate_per_gpu"],
|
||||
rel_tol=0.0,
|
||||
abs_tol=1e-12,
|
||||
):
|
||||
raise ValueError("repeat endpoints changed offered load")
|
||||
return {
|
||||
"kind": "same_config_workload_repeat",
|
||||
"group": {
|
||||
"cell": source["cell"],
|
||||
"tp": source["tp"],
|
||||
"mns": source["mns"],
|
||||
"level": source["level"],
|
||||
"source_replicate": source["replicate"],
|
||||
"target_replicate": target["replicate"],
|
||||
},
|
||||
"source": {
|
||||
key: source[key]
|
||||
for key in ("trial_id", "result_sha256", "requests_sha256")
|
||||
},
|
||||
"target": {
|
||||
key: target[key]
|
||||
for key in ("trial_id", "result_sha256", "requests_sha256")
|
||||
},
|
||||
"delta_state": _delta(source["state"], target["state"], V0.ALL_FEATURES),
|
||||
"delta_outcome": _delta(source["outcome"], target["outcome"], OUTCOME_FEATURES),
|
||||
}
|
||||
|
||||
|
||||
def build_pairs(
|
||||
trials: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
action_groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
|
||||
repeat_groups: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
|
||||
for trial in trials:
|
||||
action_groups[
|
||||
(
|
||||
trial["tp"],
|
||||
trial["level"],
|
||||
trial["replicate"],
|
||||
trial["request_hash"],
|
||||
trial["offered_rate_per_gpu"],
|
||||
)
|
||||
].append(trial)
|
||||
repeat_groups[(trial["cell"], trial["level"])].append(trial)
|
||||
|
||||
actions = []
|
||||
for group in action_groups.values():
|
||||
if len(group) != 2:
|
||||
continue
|
||||
source, target = sorted(group, key=lambda trial: trial["mns"])
|
||||
actions.append(_action_pair(source, target))
|
||||
|
||||
repeats = []
|
||||
for group in repeat_groups.values():
|
||||
ordered = sorted(group, key=lambda trial: trial["replicate"])
|
||||
if len(ordered) != 3:
|
||||
raise ValueError("each prospective repeat group must contain three runs")
|
||||
repeats.extend(
|
||||
_repeat_pair(source, target)
|
||||
for source, target in zip(ordered, ordered[1:], strict=False)
|
||||
)
|
||||
return actions, repeats
|
||||
|
||||
|
||||
def _balanced_accuracy(labels: list[int], predictions: list[int]) -> float:
|
||||
positive = [prediction for label, prediction in zip(labels, predictions) if label == 1]
|
||||
negative = [prediction for label, prediction in zip(labels, predictions) if label == 0]
|
||||
if not positive or not negative:
|
||||
raise ValueError("balanced accuracy requires both classes")
|
||||
sensitivity = sum(prediction == 1 for prediction in positive) / len(positive)
|
||||
specificity = sum(prediction == 0 for prediction in negative) / len(negative)
|
||||
return (sensitivity + specificity) / 2.0
|
||||
|
||||
|
||||
def _threshold_candidates(values: list[float]) -> list[float]:
|
||||
unique = sorted(set(values))
|
||||
if len(unique) == 1:
|
||||
return [unique[0] - 1.0, unique[0], unique[0] + 1.0]
|
||||
scale = max(1.0, max(abs(value) for value in unique))
|
||||
candidates = [unique[0] - scale * 1e-6]
|
||||
candidates.extend(
|
||||
(left + right) / 2.0
|
||||
for left, right in zip(unique, unique[1:], strict=False)
|
||||
)
|
||||
candidates.append(unique[-1] + scale * 1e-6)
|
||||
return candidates
|
||||
|
||||
|
||||
def _fit_threshold(values: list[float], labels: list[int]) -> tuple[float, int, float]:
|
||||
best: tuple[float, int, float, float] | None = None
|
||||
for threshold in _threshold_candidates(values):
|
||||
for direction in (-1, 1):
|
||||
predictions = [int(direction * (value - threshold) >= 0.0) for value in values]
|
||||
balanced = _balanced_accuracy(labels, predictions)
|
||||
accuracy = sum(
|
||||
prediction == label
|
||||
for prediction, label in zip(predictions, labels, strict=True)
|
||||
) / len(labels)
|
||||
candidate = (balanced, accuracy, -abs(threshold), float(direction))
|
||||
if best is None or candidate > best:
|
||||
best = candidate
|
||||
selected_threshold = threshold
|
||||
selected_direction = direction
|
||||
assert best is not None
|
||||
return selected_threshold, selected_direction, best[0]
|
||||
|
||||
|
||||
def one_feature_leave_repeat_out(
|
||||
actions: list[dict[str, Any]],
|
||||
*,
|
||||
delta_key: str,
|
||||
features: tuple[str, ...],
|
||||
) -> dict[str, Any]:
|
||||
labels = [int(pair["full_action_efficacy"]) for pair in actions]
|
||||
results = {}
|
||||
for feature in features:
|
||||
predictions = []
|
||||
held_out_labels = []
|
||||
folds = []
|
||||
for held_out in (1, 2, 3):
|
||||
train = [pair for pair in actions if pair["group"]["replicate"] != held_out]
|
||||
test = [pair for pair in actions if pair["group"]["replicate"] == held_out]
|
||||
train_values = [float(pair[delta_key][feature]) for pair in train]
|
||||
train_labels = [int(pair["full_action_efficacy"]) for pair in train]
|
||||
threshold, direction, train_balanced = _fit_threshold(
|
||||
train_values, train_labels
|
||||
)
|
||||
test_values = [float(pair[delta_key][feature]) for pair in test]
|
||||
test_predictions = [
|
||||
int(direction * (value - threshold) >= 0.0) for value in test_values
|
||||
]
|
||||
test_labels = [int(pair["full_action_efficacy"]) for pair in test]
|
||||
predictions.extend(test_predictions)
|
||||
held_out_labels.extend(test_labels)
|
||||
folds.append(
|
||||
{
|
||||
"held_out_replicate": held_out,
|
||||
"threshold": threshold,
|
||||
"direction": direction,
|
||||
"train_balanced_accuracy": train_balanced,
|
||||
"test_labels": test_labels,
|
||||
"test_predictions": test_predictions,
|
||||
}
|
||||
)
|
||||
balanced = _balanced_accuracy(held_out_labels, predictions)
|
||||
accuracy = sum(
|
||||
prediction == label
|
||||
for prediction, label in zip(predictions, held_out_labels, strict=True)
|
||||
) / len(held_out_labels)
|
||||
results[feature] = {
|
||||
"balanced_accuracy": balanced,
|
||||
"accuracy": accuracy,
|
||||
"folds": folds,
|
||||
}
|
||||
best_feature = max(
|
||||
results,
|
||||
key=lambda feature: (
|
||||
results[feature]["balanced_accuracy"],
|
||||
results[feature]["accuracy"],
|
||||
feature,
|
||||
),
|
||||
)
|
||||
return {
|
||||
"labels": V0.numeric(labels),
|
||||
"positive": sum(labels),
|
||||
"negative": len(labels) - sum(labels),
|
||||
"features": results,
|
||||
"best_feature": best_feature,
|
||||
"best_balanced_accuracy": results[best_feature]["balanced_accuracy"],
|
||||
"best_accuracy": results[best_feature]["accuracy"],
|
||||
}
|
||||
|
||||
|
||||
def analyze_horizon(trials: list[dict[str, Any]], horizon_s: float) -> dict[str, Any]:
|
||||
actions, repeats = build_pairs(trials)
|
||||
response = V0.response_statistics(actions, repeats)
|
||||
qualifying_response = sorted(
|
||||
feature for feature, item in response.items() if item["qualifies"]
|
||||
)
|
||||
outcome_cv = one_feature_leave_repeat_out(
|
||||
actions,
|
||||
delta_key="delta_outcome",
|
||||
features=OUTCOME_FEATURES,
|
||||
)
|
||||
telemetry_cv = one_feature_leave_repeat_out(
|
||||
actions,
|
||||
delta_key="delta_state",
|
||||
features=V0.GATE_FEATURES,
|
||||
)
|
||||
outcome_best = float(outcome_cv["best_balanced_accuracy"])
|
||||
efficacy_qualifying = sorted(
|
||||
feature
|
||||
for feature, item in telemetry_cv["features"].items()
|
||||
if item["balanced_accuracy"] >= MIN_EFFICACY_BALANCED_ACCURACY
|
||||
and item["balanced_accuracy"]
|
||||
>= outcome_best + MIN_EFFICACY_DELTA_OVER_OUTCOME
|
||||
)
|
||||
action_hashes_match = all(
|
||||
pair["group"]["request_hash"] for pair in actions
|
||||
)
|
||||
labels = [int(pair["full_action_efficacy"]) for pair in actions]
|
||||
invariants = {
|
||||
"expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
|
||||
"expected_repeat_pair_count": len(repeats) == EXPECTED_REPEAT_PAIRS,
|
||||
"matched_action_request_hashes": action_hashes_match,
|
||||
"efficacy_label_balance": (
|
||||
sum(labels) >= MIN_EFFICACY_CLASS
|
||||
and len(labels) - sum(labels) >= MIN_EFFICACY_CLASS
|
||||
),
|
||||
"finite_deltas": all(
|
||||
math.isfinite(value)
|
||||
for pair in [*actions, *repeats]
|
||||
for values in (pair["delta_state"], pair["delta_outcome"])
|
||||
for value in values.values()
|
||||
),
|
||||
"probabilities_bounded": all(
|
||||
0.0 <= trial["outcome"][feature] <= 1.0
|
||||
for trial in trials
|
||||
for feature in (
|
||||
"admitted_fraction",
|
||||
"completed_over_admitted",
|
||||
"completed_pass_rate",
|
||||
"completed_fail_fraction_of_total",
|
||||
"outstanding_over_admitted",
|
||||
"admitted_input_tokens_mean_over_limit",
|
||||
)
|
||||
),
|
||||
}
|
||||
red_flags = [name for name, passed in invariants.items() if not passed]
|
||||
transitions = defaultdict(int)
|
||||
for pair in actions:
|
||||
transitions[pair["full_feasibility_transition"]] += 1
|
||||
return {
|
||||
"horizon_s": horizon_s,
|
||||
"actions": actions,
|
||||
"repeats": repeats,
|
||||
"response_statistics": response,
|
||||
"qualifying_response_features": qualifying_response,
|
||||
"efficacy": {
|
||||
"outcome_delta": outcome_cv,
|
||||
"telemetry_delta": telemetry_cv,
|
||||
"telemetry_qualifying_features": efficacy_qualifying,
|
||||
"minimum_balanced_accuracy": MIN_EFFICACY_BALANCED_ACCURACY,
|
||||
"minimum_delta_over_best_outcome": MIN_EFFICACY_DELTA_OVER_OUTCOME,
|
||||
"feasibility_transitions": dict(sorted(transitions.items())),
|
||||
},
|
||||
"sanity": {
|
||||
"trials": len(trials),
|
||||
"action_pairs": len(actions),
|
||||
"repeat_pairs": len(repeats),
|
||||
"invariants": invariants,
|
||||
"red_flags": red_flags,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
|
||||
trials_by_horizon, streams = load_trials(run_root)
|
||||
manifest_validation = validate_manifest(
|
||||
trials_by_horizon[min(trials_by_horizon)], manifest_path
|
||||
)
|
||||
horizons = {
|
||||
str(int(horizon)): analyze_horizon(trials, horizon)
|
||||
for horizon, trials in sorted(trials_by_horizon.items())
|
||||
}
|
||||
red_flags = sorted(
|
||||
{
|
||||
flag
|
||||
for horizon in horizons.values()
|
||||
for flag in horizon["sanity"]["red_flags"]
|
||||
}
|
||||
)
|
||||
stable_response = sorted(
|
||||
set.intersection(
|
||||
*(
|
||||
set(horizon["qualifying_response_features"])
|
||||
for horizon in horizons.values()
|
||||
)
|
||||
)
|
||||
)
|
||||
stable_efficacy = sorted(
|
||||
set.intersection(
|
||||
*(
|
||||
set(horizon["efficacy"]["telemetry_qualifying_features"])
|
||||
for horizon in horizons.values()
|
||||
)
|
||||
)
|
||||
)
|
||||
if red_flags:
|
||||
decision = "STOP_DATA_INVALID"
|
||||
elif len(stable_response) < V0.MIN_STABLE_FEATURES:
|
||||
decision = "STOP_NO_PROSPECTIVE_RESPONSE"
|
||||
elif not stable_efficacy:
|
||||
decision = "STOP_NO_INCREMENTAL_TUNING_SIGNAL"
|
||||
else:
|
||||
decision = "OPEN_MATCHED_GPU_PILOT"
|
||||
payload = {
|
||||
"schema": SCHEMA,
|
||||
"status": "COMPLETE",
|
||||
"decision": decision,
|
||||
"claim_boundary": (
|
||||
"Development-only confirmation on an already-consumed P1 task. "
|
||||
"Passing can open a newly registered matched pilot but cannot be "
|
||||
"reported as held-out tuning evidence."
|
||||
),
|
||||
"frozen_gate": {
|
||||
"response_thresholds_identical_to_phase6_v0": True,
|
||||
"expected_action_pairs": EXPECTED_ACTION_PAIRS,
|
||||
"expected_repeat_pairs": EXPECTED_REPEAT_PAIRS,
|
||||
"minimum_stable_response_features": V0.MIN_STABLE_FEATURES,
|
||||
"minimum_efficacy_class": MIN_EFFICACY_CLASS,
|
||||
"minimum_efficacy_balanced_accuracy": MIN_EFFICACY_BALANCED_ACCURACY,
|
||||
"minimum_efficacy_delta_over_best_outcome": (
|
||||
MIN_EFFICACY_DELTA_OVER_OUTCOME
|
||||
),
|
||||
},
|
||||
"stable_response_features": stable_response,
|
||||
"stable_incremental_efficacy_features": stable_efficacy,
|
||||
"horizons": horizons,
|
||||
"provenance": {
|
||||
"analysis_script": str(Path(__file__).resolve()),
|
||||
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
|
||||
"phase6_v0_script_sha256": sha256_file(HERE / "analyze_phase6.py"),
|
||||
"run_root": str(run_root.resolve()),
|
||||
"manifest": str(manifest_path.resolve()),
|
||||
"manifest_sha256": sha256_file(manifest_path),
|
||||
"manifest_validation": manifest_validation,
|
||||
"streams": streams,
|
||||
},
|
||||
"sanity": {
|
||||
"stream_count": len(streams),
|
||||
"stream_bytes": V0.numeric(item["bytes"] for item in streams),
|
||||
"red_flags": red_flags,
|
||||
},
|
||||
}
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
return payload
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--run-root", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
payload = audit(
|
||||
run_root=args.run_root,
|
||||
manifest_path=args.manifest,
|
||||
output_path=args.output,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"decision": payload["decision"],
|
||||
"stable_response_features": payload["stable_response_features"],
|
||||
"stable_incremental_efficacy_features": payload[
|
||||
"stable_incremental_efficacy_features"
|
||||
],
|
||||
"sanity": payload["sanity"],
|
||||
},
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
520
runs/intervention-response-v0/analyze_phase6.py
Normal file
520
runs/intervention-response-v0/analyze_phase6.py
Normal file
@@ -0,0 +1,520 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Audit whether a controlled knob change produces identifiable telemetry deltas.
|
||||
|
||||
This is a development-only feasibility audit. It compares adjacent MNS
|
||||
interventions at an identical TP, offered-load anchor, and request sequence
|
||||
against same-config primary/confirmation repeat noise. It does not claim that
|
||||
the observed response is causal or that it improves an end-to-end tuner.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from statistics import median
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
COMMON_STATE_DIR = HERE.parent / "telemetry-residual"
|
||||
sys.path.insert(0, str(COMMON_STATE_DIR))
|
||||
|
||||
from common_state import load_jsonl, summarize_engine # noqa: E402
|
||||
|
||||
|
||||
SCHEMA = "intervention-response-audit-v0"
|
||||
HORIZONS_S = (5.0, 10.0)
|
||||
GATE_FEATURES = (
|
||||
"scheduler_steps_per_s",
|
||||
"decode_batch_size.mean",
|
||||
"prefill_token_fraction",
|
||||
"queue_waiting_mean",
|
||||
"queue_running_mean",
|
||||
"kv_usage_mean",
|
||||
"graph_padding_fraction",
|
||||
)
|
||||
ALL_FEATURES = (
|
||||
"scheduler_steps_per_s",
|
||||
"batch_size.mean",
|
||||
"batch_tokens.mean",
|
||||
"decode_batch_size.mean",
|
||||
"prefill_token_fraction",
|
||||
"queue_waiting_mean",
|
||||
"queue_running_mean",
|
||||
"preemptions",
|
||||
"kv_usage_mean",
|
||||
"kv_usage_max",
|
||||
"kv_usage_end_minus_start",
|
||||
"graph_none_share",
|
||||
"graph_full_share",
|
||||
"graph_padding_fraction",
|
||||
)
|
||||
EXPECTED_ACTION_PAIRS = 17
|
||||
MIN_REPEAT_PAIRS = 20
|
||||
MIN_STABLE_FEATURES = 2
|
||||
MIN_SIGN_CONSISTENCY = 0.75
|
||||
MIN_EFFECT_TO_NOISE = 2.0
|
||||
MIN_ABOVE_NOISE_P95_FRACTION = 0.5
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def numeric(values: Iterable[float]) -> dict[str, Any]:
|
||||
finite = [float(value) for value in values]
|
||||
if not finite:
|
||||
raise ValueError("numeric summary requires at least one value")
|
||||
if any(not math.isfinite(value) for value in finite):
|
||||
raise ValueError("numeric summary received a non-finite value")
|
||||
return {
|
||||
"n": len(finite),
|
||||
"min": min(finite),
|
||||
"max": max(finite),
|
||||
"distinct_n": len(set(finite)),
|
||||
}
|
||||
|
||||
|
||||
def quantile(values: Iterable[float], probability: float) -> float:
|
||||
ordered = sorted(float(value) for value in values)
|
||||
if not ordered:
|
||||
raise ValueError("quantile requires at least one value")
|
||||
if not 0.0 <= probability <= 1.0:
|
||||
raise ValueError("quantile probability must be in [0, 1]")
|
||||
position = probability * (len(ordered) - 1)
|
||||
lower = math.floor(position)
|
||||
upper = math.ceil(position)
|
||||
if lower == upper:
|
||||
return ordered[lower]
|
||||
weight = position - lower
|
||||
return ordered[lower] * (1.0 - weight) + ordered[upper] * weight
|
||||
|
||||
|
||||
def flatten_state(summary: Mapping[str, Any]) -> dict[str, float]:
|
||||
common = summary["common"]
|
||||
engine = summary["engine_only"]
|
||||
state = {
|
||||
"scheduler_steps_per_s": float(common["scheduler_steps_per_s"]),
|
||||
"batch_size.mean": float(common["batch_size"]["mean"]),
|
||||
"batch_tokens.mean": float(common["batch_tokens"]["mean"]),
|
||||
"decode_batch_size.mean": float(common["decode_batch_size"]["mean"]),
|
||||
"prefill_token_fraction": float(common["prefill_token_fraction"]),
|
||||
"queue_waiting_mean": float(common["queue_waiting_mean"]),
|
||||
"queue_running_mean": float(common["queue_running_mean"]),
|
||||
"preemptions": float(common["preemptions"]),
|
||||
"kv_usage_mean": float(engine["kv_usage_mean"]),
|
||||
"kv_usage_max": float(engine["kv_usage_max"]),
|
||||
"kv_usage_end_minus_start": float(engine["kv_usage_end_minus_start"]),
|
||||
"graph_none_share": float(engine["graph_none_share"]),
|
||||
"graph_full_share": float(engine["graph_full_share"]),
|
||||
"graph_padding_fraction": float(engine["graph_padding_fraction"]),
|
||||
}
|
||||
if set(state) != set(ALL_FEATURES):
|
||||
raise ValueError("flattened state does not match the frozen feature set")
|
||||
if any(not math.isfinite(value) for value in state.values()):
|
||||
raise ValueError("flattened state contains a non-finite value")
|
||||
return state
|
||||
|
||||
|
||||
def _trial_role(path: Path) -> str:
|
||||
return "confirmation" if path.parent.name.startswith("confirm-") else "primary"
|
||||
|
||||
|
||||
def load_trials(
|
||||
raw_root: Path,
|
||||
*,
|
||||
horizons_s: tuple[float, ...] = HORIZONS_S,
|
||||
) -> tuple[dict[float, list[dict[str, Any]]], list[dict[str, Any]]]:
|
||||
by_horizon = {horizon: [] for horizon in horizons_s}
|
||||
stream_provenance = []
|
||||
for cell_dir in sorted(path for path in raw_root.iterdir() if path.is_dir()):
|
||||
streams = sorted((cell_dir / "opprof").glob("*.jsonl"))
|
||||
if len(streams) != 1:
|
||||
raise ValueError(f"{cell_dir}: expected exactly one Layer-1 stream")
|
||||
stream = streams[0]
|
||||
records = load_jsonl(stream)
|
||||
stream_provenance.append(
|
||||
{
|
||||
"path": str(stream),
|
||||
"sha256": sha256_file(stream),
|
||||
"bytes": stream.stat().st_size,
|
||||
}
|
||||
)
|
||||
result_paths = sorted(cell_dir.glob("anchor-*/result.json"))
|
||||
result_paths.extend(sorted(cell_dir.glob("confirm-*-anchor-*/result.json")))
|
||||
for result_path in result_paths:
|
||||
result = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
start_ns = int(result["interval"]["start_mono_ns"])
|
||||
elapsed_s = float(result["interval"]["elapsed_s"])
|
||||
for horizon_s in horizons_s:
|
||||
if elapsed_s < horizon_s:
|
||||
raise ValueError(
|
||||
f"{result_path}: elapsed {elapsed_s} is shorter than {horizon_s}s"
|
||||
)
|
||||
state = flatten_state(
|
||||
summarize_engine(
|
||||
records,
|
||||
start_ns=start_ns,
|
||||
end_ns=start_ns + int(horizon_s * 1e9),
|
||||
request_count=int(result["selection"]["count"]),
|
||||
)
|
||||
)
|
||||
by_horizon[horizon_s].append(
|
||||
{
|
||||
"trial_id": str(result_path.relative_to(raw_root)),
|
||||
"result_sha256": sha256_file(result_path),
|
||||
"role": _trial_role(result_path),
|
||||
"cell": str(result["cell"]),
|
||||
"study_sha256": str(result["study_sha256"]),
|
||||
"tp": int(result["tp"]),
|
||||
"mns": int(result["mns"]),
|
||||
"anchor": float(result["anchor"]),
|
||||
"request_hash": str(
|
||||
result["selection"]["request_id_order_sha256"]
|
||||
),
|
||||
"request_count": int(result["selection"]["count"]),
|
||||
"early_stopped": bool(result["early_stopped"]),
|
||||
"full_pass_rate": float(result["pass_rate"]),
|
||||
"full_feasible": bool(result["feasible"]),
|
||||
"state": state,
|
||||
}
|
||||
)
|
||||
return by_horizon, stream_provenance
|
||||
|
||||
|
||||
def _group_key(trial: Mapping[str, Any]) -> tuple[Any, ...]:
|
||||
return (
|
||||
trial["study_sha256"],
|
||||
trial["tp"],
|
||||
trial["anchor"],
|
||||
trial["request_hash"],
|
||||
)
|
||||
|
||||
|
||||
def _delta(source: Mapping[str, Any], target: Mapping[str, Any]) -> dict[str, float]:
|
||||
return {
|
||||
feature: float(target["state"][feature]) - float(source["state"][feature])
|
||||
for feature in ALL_FEATURES
|
||||
}
|
||||
|
||||
|
||||
def _pair(source: Mapping[str, Any], target: Mapping[str, Any], kind: str) -> dict[str, Any]:
|
||||
if _group_key(source) != _group_key(target):
|
||||
raise ValueError("pair endpoints do not share workload identity")
|
||||
return {
|
||||
"kind": kind,
|
||||
"group": {
|
||||
"study_sha256": source["study_sha256"],
|
||||
"tp": source["tp"],
|
||||
"anchor": source["anchor"],
|
||||
"request_hash": source["request_hash"],
|
||||
},
|
||||
"source": {
|
||||
"trial_id": source["trial_id"],
|
||||
"cell": source["cell"],
|
||||
"mns": source["mns"],
|
||||
"early_stopped": source["early_stopped"],
|
||||
"full_pass_rate": source["full_pass_rate"],
|
||||
"full_feasible": source["full_feasible"],
|
||||
},
|
||||
"target": {
|
||||
"trial_id": target["trial_id"],
|
||||
"cell": target["cell"],
|
||||
"mns": target["mns"],
|
||||
"early_stopped": target["early_stopped"],
|
||||
"full_pass_rate": target["full_pass_rate"],
|
||||
"full_feasible": target["full_feasible"],
|
||||
},
|
||||
"delta_state": _delta(source, target),
|
||||
"descriptive_full_outcome": {
|
||||
"delta_pass_rate": target["full_pass_rate"] - source["full_pass_rate"],
|
||||
"feasibility_transition": (
|
||||
f"{str(source['full_feasible']).lower()}->"
|
||||
f"{str(target['full_feasible']).lower()}"
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def build_pairs(
|
||||
trials: list[dict[str, Any]],
|
||||
) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
primary_groups: dict[tuple[Any, ...], list[dict[str, Any]]] = defaultdict(list)
|
||||
primary_by_cell_anchor: dict[tuple[Any, ...], dict[str, Any]] = {}
|
||||
confirmations = []
|
||||
for trial in trials:
|
||||
if trial["role"] == "primary":
|
||||
primary_groups[_group_key(trial)].append(trial)
|
||||
primary_by_cell_anchor[
|
||||
(trial["cell"], trial["anchor"], trial["request_hash"])
|
||||
] = trial
|
||||
else:
|
||||
confirmations.append(trial)
|
||||
|
||||
actions = []
|
||||
for group in primary_groups.values():
|
||||
ordered = sorted(group, key=lambda item: item["mns"])
|
||||
for source, target in zip(ordered, ordered[1:], strict=False):
|
||||
if target["mns"] == source["mns"] * 2:
|
||||
actions.append(_pair(source, target, "mns_increase"))
|
||||
|
||||
repeats = []
|
||||
for confirmation in confirmations:
|
||||
key = (
|
||||
confirmation["cell"],
|
||||
confirmation["anchor"],
|
||||
confirmation["request_hash"],
|
||||
)
|
||||
primary = primary_by_cell_anchor.get(key)
|
||||
if primary is None:
|
||||
raise ValueError(f"{confirmation['trial_id']}: missing matched primary")
|
||||
if primary["mns"] != confirmation["mns"]:
|
||||
raise ValueError("repeat endpoints changed MNS")
|
||||
repeats.append(_pair(primary, confirmation, "same_config_repeat"))
|
||||
return actions, repeats
|
||||
|
||||
|
||||
def response_statistics(
|
||||
actions: list[dict[str, Any]],
|
||||
repeats: list[dict[str, Any]],
|
||||
) -> dict[str, Any]:
|
||||
statistics = {}
|
||||
for feature in ALL_FEATURES:
|
||||
action = [float(pair["delta_state"][feature]) for pair in actions]
|
||||
noise = [float(pair["delta_state"][feature]) for pair in repeats]
|
||||
action_abs = [abs(value) for value in action]
|
||||
noise_abs = [abs(value) for value in noise]
|
||||
positive = sum(value > 1e-12 for value in action)
|
||||
negative = sum(value < -1e-12 for value in action)
|
||||
zero = len(action) - positive - negative
|
||||
nonzero = positive + negative
|
||||
sign_consistency = max(positive, negative) / nonzero if nonzero else 0.0
|
||||
action_median = median(action_abs)
|
||||
noise_median = median(noise_abs)
|
||||
noise_p95 = quantile(noise_abs, 0.95)
|
||||
effect_to_noise = (
|
||||
action_median / noise_median
|
||||
if noise_median > 0
|
||||
else (math.inf if action_median > 0 else 0.0)
|
||||
)
|
||||
above_noise = sum(value > noise_p95 for value in action_abs) / len(action_abs)
|
||||
qualifies = (
|
||||
feature in GATE_FEATURES
|
||||
and sign_consistency >= MIN_SIGN_CONSISTENCY
|
||||
and effect_to_noise >= MIN_EFFECT_TO_NOISE
|
||||
and above_noise >= MIN_ABOVE_NOISE_P95_FRACTION
|
||||
)
|
||||
statistics[feature] = {
|
||||
"action_delta": numeric(action),
|
||||
"repeat_delta": numeric(noise),
|
||||
"action_abs_median": action_median,
|
||||
"repeat_abs_median": noise_median,
|
||||
"repeat_abs_p95": noise_p95,
|
||||
"effect_to_repeat_median": (
|
||||
effect_to_noise if math.isfinite(effect_to_noise) else None
|
||||
),
|
||||
"effect_to_repeat_median_is_infinite": math.isinf(effect_to_noise),
|
||||
"action_signs": {
|
||||
"positive": positive,
|
||||
"negative": negative,
|
||||
"zero": zero,
|
||||
"consistency": sign_consistency,
|
||||
},
|
||||
"action_above_repeat_p95_fraction": above_noise,
|
||||
"gate_feature": feature in GATE_FEATURES,
|
||||
"qualifies": qualifies,
|
||||
}
|
||||
return statistics
|
||||
|
||||
|
||||
def analyze_horizon(trials: list[dict[str, Any]], horizon_s: float) -> dict[str, Any]:
|
||||
actions, repeats = build_pairs(trials)
|
||||
feature_statistics = response_statistics(actions, repeats)
|
||||
qualifying = sorted(
|
||||
feature for feature, item in feature_statistics.items() if item["qualifies"]
|
||||
)
|
||||
all_values = [
|
||||
value
|
||||
for trial in trials
|
||||
for value in trial["state"].values()
|
||||
]
|
||||
action_vectors = {
|
||||
tuple(round(float(pair["delta_state"][feature]), 12) for feature in ALL_FEATURES)
|
||||
for pair in actions
|
||||
}
|
||||
pair_invariants = {
|
||||
"expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
|
||||
"sufficient_repeat_pair_count": len(repeats) >= MIN_REPEAT_PAIRS,
|
||||
"all_pair_hashes_match": all(
|
||||
pair["group"]["request_hash"] for pair in [*actions, *repeats]
|
||||
),
|
||||
"all_values_finite": all(math.isfinite(value) for value in all_values),
|
||||
"state_vectors_not_all_identical": len(action_vectors) > 1,
|
||||
"ratios_bounded": all(
|
||||
0.0 <= trial["state"][feature] <= 1.0
|
||||
for trial in trials
|
||||
for feature in (
|
||||
"prefill_token_fraction",
|
||||
"kv_usage_mean",
|
||||
"kv_usage_max",
|
||||
"graph_none_share",
|
||||
"graph_full_share",
|
||||
"graph_padding_fraction",
|
||||
)
|
||||
),
|
||||
"nonnegative_counters": all(
|
||||
trial["state"][feature] >= 0.0
|
||||
for trial in trials
|
||||
for feature in (
|
||||
"scheduler_steps_per_s",
|
||||
"batch_size.mean",
|
||||
"batch_tokens.mean",
|
||||
"decode_batch_size.mean",
|
||||
"queue_waiting_mean",
|
||||
"queue_running_mean",
|
||||
"preemptions",
|
||||
)
|
||||
),
|
||||
}
|
||||
red_flags = [name for name, passed in pair_invariants.items() if not passed]
|
||||
pass_deltas = [
|
||||
pair["descriptive_full_outcome"]["delta_pass_rate"] for pair in actions
|
||||
]
|
||||
transitions = defaultdict(int)
|
||||
for pair in actions:
|
||||
transitions[pair["descriptive_full_outcome"]["feasibility_transition"]] += 1
|
||||
return {
|
||||
"horizon_s": horizon_s,
|
||||
"actions": actions,
|
||||
"repeats": repeats,
|
||||
"feature_statistics": feature_statistics,
|
||||
"qualifying_features": qualifying,
|
||||
"descriptive_full_outcome": {
|
||||
"delta_pass_rate": numeric(pass_deltas),
|
||||
"positive": sum(value > 1e-12 for value in pass_deltas),
|
||||
"negative": sum(value < -1e-12 for value in pass_deltas),
|
||||
"zero": sum(abs(value) <= 1e-12 for value in pass_deltas),
|
||||
"feasibility_transitions": dict(sorted(transitions.items())),
|
||||
"limitation": (
|
||||
"Full outcomes may use different elapsed durations when a trial "
|
||||
"early-stopped; they are descriptive and are not a gate input."
|
||||
),
|
||||
},
|
||||
"sanity": {
|
||||
"trials": len(trials),
|
||||
"action_pairs": len(actions),
|
||||
"repeat_pairs": len(repeats),
|
||||
"distinct_action_vectors": len(action_vectors),
|
||||
"invariants": pair_invariants,
|
||||
"red_flags": red_flags,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def audit(
|
||||
*,
|
||||
metrics_path: Path,
|
||||
raw_root: Path,
|
||||
output_path: Path,
|
||||
) -> dict[str, Any]:
|
||||
trials_by_horizon, streams = load_trials(raw_root)
|
||||
horizons = {
|
||||
str(int(horizon)): analyze_horizon(trials, horizon)
|
||||
for horizon, trials in sorted(trials_by_horizon.items())
|
||||
}
|
||||
red_flags = sorted(
|
||||
{
|
||||
red_flag
|
||||
for horizon in horizons.values()
|
||||
for red_flag in horizon["sanity"]["red_flags"]
|
||||
}
|
||||
)
|
||||
stable_features = sorted(
|
||||
set.intersection(
|
||||
*(set(horizon["qualifying_features"]) for horizon in horizons.values())
|
||||
)
|
||||
)
|
||||
if red_flags:
|
||||
decision = "STOP_DATA_INVALID"
|
||||
elif len(stable_features) < MIN_STABLE_FEATURES:
|
||||
decision = "STOP_NO_IDENTIFIABLE_RESPONSE"
|
||||
else:
|
||||
decision = "OPEN_MATCHED_PILOT"
|
||||
payload = {
|
||||
"schema": SCHEMA,
|
||||
"status": "COMPLETE",
|
||||
"decision": decision,
|
||||
"claim_boundary": (
|
||||
"Development-only identifiability gate. Passing opens a controlled "
|
||||
"real-GPU pilot; it does not establish tuning benefit or causality."
|
||||
),
|
||||
"frozen_gate": {
|
||||
"horizons_s": list(HORIZONS_S),
|
||||
"expected_action_pairs": EXPECTED_ACTION_PAIRS,
|
||||
"minimum_repeat_pairs": MIN_REPEAT_PAIRS,
|
||||
"minimum_stable_features": MIN_STABLE_FEATURES,
|
||||
"minimum_sign_consistency": MIN_SIGN_CONSISTENCY,
|
||||
"minimum_effect_to_repeat_median": MIN_EFFECT_TO_NOISE,
|
||||
"minimum_action_above_repeat_p95_fraction": (
|
||||
MIN_ABOVE_NOISE_P95_FRACTION
|
||||
),
|
||||
"gate_features": list(GATE_FEATURES),
|
||||
},
|
||||
"stable_qualifying_features": stable_features,
|
||||
"horizons": horizons,
|
||||
"provenance": {
|
||||
"analysis_script": str(Path(__file__).resolve()),
|
||||
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
|
||||
"phase6_metrics": str(metrics_path.resolve()),
|
||||
"phase6_metrics_sha256": sha256_file(metrics_path),
|
||||
"raw_root": str(raw_root.resolve()),
|
||||
"streams": streams,
|
||||
},
|
||||
"sanity": {
|
||||
"stream_count": len(streams),
|
||||
"stream_bytes": numeric(item["bytes"] for item in streams),
|
||||
"red_flags": red_flags,
|
||||
},
|
||||
}
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
return payload
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--metrics", type=Path, required=True)
|
||||
parser.add_argument("--raw-root", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
payload = audit(
|
||||
metrics_path=args.metrics,
|
||||
raw_root=args.raw_root,
|
||||
output_path=args.output,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"decision": payload["decision"],
|
||||
"stable_qualifying_features": payload[
|
||||
"stable_qualifying_features"
|
||||
],
|
||||
"sanity": payload["sanity"],
|
||||
},
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
6818
runs/intervention-response-v0/p1-audit.json
Normal file
6818
runs/intervention-response-v0/p1-audit.json
Normal file
File diff suppressed because it is too large
Load Diff
5121
runs/intervention-response-v0/phase6-audit.json
Normal file
5121
runs/intervention-response-v0/phase6-audit.json
Normal file
File diff suppressed because it is too large
Load Diff
57
runs/intervention-response-v0/test_analysis.py
Normal file
57
runs/intervention-response-v0/test_analysis.py
Normal file
@@ -0,0 +1,57 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import math
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def load_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_v0", HERE / "analyze_phase6.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def pair(module, delta: dict[str, float]) -> dict[str, object]:
|
||||
state = {feature: 0.0 for feature in module.ALL_FEATURES}
|
||||
state.update(delta)
|
||||
return {"delta_state": state}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
module = load_module()
|
||||
assert module.numeric([0.0, 1.0, 1.0]) == {
|
||||
"n": 3,
|
||||
"min": 0.0,
|
||||
"max": 1.0,
|
||||
"distinct_n": 2,
|
||||
}
|
||||
assert math.isclose(module.quantile([0.0, 10.0], 0.95), 9.5)
|
||||
|
||||
actions = [
|
||||
pair(module, {"queue_waiting_mean": -1.0 - 0.1 * index})
|
||||
for index in range(8)
|
||||
]
|
||||
repeats = [
|
||||
pair(module, {"queue_waiting_mean": 0.01 * ((index % 3) - 1)})
|
||||
for index in range(20)
|
||||
]
|
||||
stats = module.response_statistics(actions, repeats)
|
||||
waiting = stats["queue_waiting_mean"]
|
||||
assert waiting["qualifies"]
|
||||
assert waiting["action_signs"]["negative"] == 8
|
||||
assert waiting["action_signs"]["consistency"] == 1.0
|
||||
assert waiting["effect_to_repeat_median"] > 2.0
|
||||
assert not stats["kv_usage_mean"]["qualifies"]
|
||||
print("intervention response v0 analysis: PASS")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
35
runs/intervention-response-v0/test_p1_analysis.py
Normal file
35
runs/intervention-response-v0/test_p1_analysis.py
Normal file
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def load_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_p1", HERE / "analyze_p1.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def main() -> None:
|
||||
module = load_module()
|
||||
values = [-2.0, -1.0, 1.0, 2.0]
|
||||
labels = [0, 0, 1, 1]
|
||||
threshold, direction, balanced = module._fit_threshold(values, labels)
|
||||
assert direction == 1
|
||||
assert -1.0 < threshold < 1.0
|
||||
assert balanced == 1.0
|
||||
assert module._balanced_accuracy(labels, labels) == 1.0
|
||||
assert module._balanced_accuracy(labels, [1, 1, 0, 0]) == 0.0
|
||||
print("intervention response P1 confirmation analysis: PASS")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
467
runs/intervention-response-v2/analyze_existing.py
Normal file
467
runs/intervention-response-v2/analyze_existing.py
Normal file
@@ -0,0 +1,467 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Audit telemetry responses over every uncensored replay decile.
|
||||
|
||||
This corrective analysis keeps the frozen P1 pairs and thresholds, but replaces
|
||||
the absolute 5/10-second cutoff with cumulative and non-overlapping 10%-of-trace
|
||||
windows. It deliberately reports every common decile instead of selecting the
|
||||
best-looking horizon.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
from statistics import median
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
P1_PATH = HERE.parent / "intervention-response-v0" / "analyze_p1.py"
|
||||
SCHEMA = "intervention-response-phase-aware-existing-v2"
|
||||
DECILE_FRACTION = 0.1
|
||||
MAX_DECILES = 10
|
||||
|
||||
|
||||
def _load_p1():
|
||||
spec = importlib.util.spec_from_file_location("intervention_response_p1", P1_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
P1 = _load_p1()
|
||||
|
||||
|
||||
def numeric(values: Iterable[float | int]) -> dict[str, Any]:
|
||||
finite = [float(value) for value in values]
|
||||
result = P1.V0.numeric(finite)
|
||||
result["median"] = median(finite)
|
||||
return result
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def trial_directories(run_root: Path) -> list[Path]:
|
||||
result = []
|
||||
for cell in sorted((run_root / "cells").iterdir()):
|
||||
if not cell.is_dir():
|
||||
continue
|
||||
for candidate in sorted(cell.iterdir()):
|
||||
if candidate.is_dir() and P1.RUN_PATTERN.match(candidate.name):
|
||||
result.append(candidate)
|
||||
if not result:
|
||||
raise ValueError("P1 run root contains no measured trial directories")
|
||||
return result
|
||||
|
||||
|
||||
def load_metadata(run_root: Path) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]:
|
||||
metadata = []
|
||||
streams = []
|
||||
for cell in sorted((run_root / "cells").iterdir()):
|
||||
if not cell.is_dir():
|
||||
continue
|
||||
stream_paths = sorted((cell / "opprof").glob("*.jsonl"))
|
||||
if len(stream_paths) != 1:
|
||||
raise ValueError(f"{cell}: expected one Layer-1 stream")
|
||||
stream_path = stream_paths[0]
|
||||
streams.append(
|
||||
{
|
||||
"cell": cell.name,
|
||||
"path": str(stream_path.resolve()),
|
||||
"sha256": sha256_file(stream_path),
|
||||
"bytes": stream_path.stat().st_size,
|
||||
}
|
||||
)
|
||||
for run_dir in trial_directories(run_root):
|
||||
match = P1.RUN_PATTERN.match(run_dir.name)
|
||||
assert match is not None
|
||||
level, replicate_text = match.groups()
|
||||
result_path = run_dir / "result.json"
|
||||
requests_path = run_dir / "requests.jsonl"
|
||||
result = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
selected = int(result["selection"]["count"])
|
||||
offered = float(result["selection"]["offered_req_s"])
|
||||
if selected <= 0 or offered <= 0.0:
|
||||
raise ValueError(f"{result_path}: invalid selected count or offered rate")
|
||||
metadata.append(
|
||||
{
|
||||
"trial_id": str(result_path.relative_to(run_root)),
|
||||
"cell": str(result["cell"]),
|
||||
"tp": int(result["tp"]),
|
||||
"mns": int(result["mns"]),
|
||||
"level": level,
|
||||
"replicate": int(replicate_text),
|
||||
"elapsed_s": float(result["interval"]["elapsed_s"]),
|
||||
"trace_duration_s": round(selected / offered, 9),
|
||||
"early_stopped": bool(result["early_stopped"]),
|
||||
"request_count": selected,
|
||||
"result_sha256": sha256_file(result_path),
|
||||
"requests_sha256": sha256_file(requests_path),
|
||||
}
|
||||
)
|
||||
return metadata, streams
|
||||
|
||||
|
||||
def common_decile_fractions(
|
||||
*, trace_duration_s: float, minimum_elapsed_s: float
|
||||
) -> tuple[float, ...]:
|
||||
if trace_duration_s <= 0.0 or minimum_elapsed_s <= 0.0:
|
||||
raise ValueError("trace duration and elapsed time must be positive")
|
||||
supported = min(
|
||||
MAX_DECILES,
|
||||
int(math.floor((minimum_elapsed_s / trace_duration_s) * 10.0 + 1e-12)),
|
||||
)
|
||||
return tuple(
|
||||
round(index * DECILE_FRACTION, 10) for index in range(1, supported + 1)
|
||||
)
|
||||
|
||||
|
||||
def _trial_record(
|
||||
*,
|
||||
run_root: Path,
|
||||
run_dir: Path,
|
||||
result: Mapping[str, Any],
|
||||
state: dict[str, float],
|
||||
outcome: dict[str, float],
|
||||
) -> dict[str, Any]:
|
||||
match = P1.RUN_PATTERN.match(run_dir.name)
|
||||
assert match is not None
|
||||
level, replicate_text = match.groups()
|
||||
result_path = run_dir / "result.json"
|
||||
requests_path = run_dir / "requests.jsonl"
|
||||
return {
|
||||
"trial_id": str(result_path.relative_to(run_root)),
|
||||
"cell": str(result["cell"]),
|
||||
"tp": int(result["tp"]),
|
||||
"mns": int(result["mns"]),
|
||||
"level": level,
|
||||
"replicate": int(replicate_text),
|
||||
"offered_rate_per_gpu": float(
|
||||
result["selection"]["offered_req_s_per_gpu"]
|
||||
),
|
||||
"request_hash": str(result["selection"]["request_id_order_sha256"]),
|
||||
"request_count": int(result["selection"]["count"]),
|
||||
"result_sha256": sha256_file(result_path),
|
||||
"requests_sha256": sha256_file(requests_path),
|
||||
"full_pass_rate": float(result["pass_rate"]),
|
||||
"full_feasible": bool(result["feasible"]),
|
||||
"early_stopped": bool(result["early_stopped"]),
|
||||
"state": state,
|
||||
"outcome": outcome,
|
||||
}
|
||||
|
||||
|
||||
def load_interval_trials(
|
||||
run_root: Path,
|
||||
intervals_s: tuple[tuple[float, float], ...],
|
||||
) -> tuple[dict[tuple[float, float], list[dict[str, Any]]], list[dict[str, Any]]]:
|
||||
by_interval = {interval: [] for interval in intervals_s}
|
||||
stream_provenance = []
|
||||
for cell in sorted((run_root / "cells").iterdir()):
|
||||
if not cell.is_dir():
|
||||
continue
|
||||
stream_paths = sorted((cell / "opprof").glob("*.jsonl"))
|
||||
if len(stream_paths) != 1:
|
||||
raise ValueError(f"{cell}: expected one Layer-1 stream")
|
||||
stream_path = stream_paths[0]
|
||||
stream = P1.load_jsonl(stream_path)
|
||||
stream_provenance.append(
|
||||
{
|
||||
"cell": cell.name,
|
||||
"path": str(stream_path.resolve()),
|
||||
"sha256": sha256_file(stream_path),
|
||||
"bytes": stream_path.stat().st_size,
|
||||
}
|
||||
)
|
||||
for run_dir in sorted(cell.iterdir()):
|
||||
if not run_dir.is_dir() or P1.RUN_PATTERN.match(run_dir.name) is None:
|
||||
continue
|
||||
result_path = run_dir / "result.json"
|
||||
requests_path = run_dir / "requests.jsonl"
|
||||
result = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
requests = P1.load_jsonl(requests_path)
|
||||
start_ns = int(result["interval"]["start_mono_ns"])
|
||||
elapsed_s = float(result["interval"]["elapsed_s"])
|
||||
for interval in intervals_s:
|
||||
start_s, end_s = interval
|
||||
if start_s < 0.0 or end_s <= start_s:
|
||||
raise ValueError(f"invalid analysis interval: {interval}")
|
||||
if elapsed_s + 1e-9 < end_s:
|
||||
raise ValueError(
|
||||
f"{result_path}: elapsed {elapsed_s} shorter than {end_s}s"
|
||||
)
|
||||
state = P1.V0.flatten_state(
|
||||
P1.summarize_engine(
|
||||
stream,
|
||||
start_ns=start_ns + int(start_s * 1e9),
|
||||
end_ns=start_ns + int(end_s * 1e9),
|
||||
request_count=int(result["selection"]["count"]),
|
||||
)
|
||||
)
|
||||
outcome = P1._prefix_outcome(result, requests, end_s)
|
||||
by_interval[interval].append(
|
||||
_trial_record(
|
||||
run_root=run_root,
|
||||
run_dir=run_dir,
|
||||
result=result,
|
||||
state=state,
|
||||
outcome=outcome,
|
||||
)
|
||||
)
|
||||
return by_interval, stream_provenance
|
||||
|
||||
|
||||
def coverage(trials: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
admitted = [float(trial["outcome"]["admitted_fraction"]) for trial in trials]
|
||||
completed = [
|
||||
float(trial["outcome"]["admitted_fraction"])
|
||||
* float(trial["outcome"]["completed_over_admitted"])
|
||||
for trial in trials
|
||||
]
|
||||
return {
|
||||
"admitted_fraction_of_total": numeric(admitted),
|
||||
"completed_fraction_of_total": numeric(completed),
|
||||
}
|
||||
|
||||
|
||||
def slim_window_analysis(
|
||||
trials: list[dict[str, Any]], *, start_s: float, end_s: float, fraction: float
|
||||
) -> dict[str, Any]:
|
||||
analysis = P1.analyze_horizon(trials, end_s)
|
||||
return {
|
||||
"start_s": start_s,
|
||||
"end_s": end_s,
|
||||
"end_fraction": fraction,
|
||||
"coverage_at_end": coverage(trials),
|
||||
"action_pairs": len(analysis["actions"]),
|
||||
"repeat_pairs": len(analysis["repeats"]),
|
||||
"response_statistics": analysis["response_statistics"],
|
||||
"qualifying_response_features": analysis["qualifying_response_features"],
|
||||
"efficacy": analysis["efficacy"],
|
||||
"sanity": analysis["sanity"],
|
||||
}
|
||||
|
||||
|
||||
def _pearson(left: list[float], right: list[float]) -> float | None:
|
||||
if len(left) != len(right) or not left:
|
||||
raise ValueError("Pearson inputs must be non-empty and have equal length")
|
||||
left_mean = sum(left) / len(left)
|
||||
right_mean = sum(right) / len(right)
|
||||
numerator = sum(
|
||||
(x - left_mean) * (y - right_mean)
|
||||
for x, y in zip(left, right, strict=True)
|
||||
)
|
||||
left_ss = sum((x - left_mean) ** 2 for x in left)
|
||||
right_ss = sum((y - right_mean) ** 2 for y in right)
|
||||
if left_ss == 0.0 or right_ss == 0.0:
|
||||
return None
|
||||
return numerator / math.sqrt(left_ss * right_ss)
|
||||
|
||||
|
||||
def trajectory_summary(
|
||||
block_trials: list[tuple[tuple[float, float], list[dict[str, Any]]]]
|
||||
) -> dict[str, Any]:
|
||||
if not block_trials:
|
||||
raise ValueError("trajectory requires at least one block")
|
||||
identities = []
|
||||
states_by_block = []
|
||||
for interval, trials in block_trials:
|
||||
ordered = sorted(
|
||||
trials,
|
||||
key=lambda trial: (trial["cell"], trial["level"], trial["replicate"]),
|
||||
)
|
||||
current_identities = [
|
||||
(trial["cell"], trial["level"], trial["replicate"]) for trial in ordered
|
||||
]
|
||||
if identities and current_identities != identities:
|
||||
raise ValueError("trajectory blocks do not contain identical trials")
|
||||
identities = current_identities
|
||||
states_by_block.append((interval, [trial["state"] for trial in ordered]))
|
||||
|
||||
features = {}
|
||||
for feature in P1.V0.ALL_FEATURES:
|
||||
block_values = [
|
||||
[float(state[feature]) for state in states]
|
||||
for _interval, states in states_by_block
|
||||
]
|
||||
first = block_values[0]
|
||||
last = block_values[-1]
|
||||
delta = [right - left for left, right in zip(first, last, strict=True)]
|
||||
features[feature] = {
|
||||
"block_medians": [median(values) for values in block_values],
|
||||
"first_to_last_delta": numeric(delta),
|
||||
"first_to_last_abs_delta": numeric(abs(value) for value in delta),
|
||||
"first_to_last_pearson": _pearson(first, last),
|
||||
"changed_trials": sum(abs(value) > 1e-12 for value in delta),
|
||||
}
|
||||
return {
|
||||
"trial_count": len(identities),
|
||||
"blocks": [
|
||||
{"start_s": interval[0], "end_s": interval[1]}
|
||||
for interval, _states in states_by_block
|
||||
],
|
||||
"features": features,
|
||||
}
|
||||
|
||||
|
||||
def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
|
||||
metadata, metadata_streams = load_metadata(run_root)
|
||||
durations = [float(item["trace_duration_s"]) for item in metadata]
|
||||
elapsed = [float(item["elapsed_s"]) for item in metadata]
|
||||
duration = median(durations)
|
||||
deciles = common_decile_fractions(
|
||||
trace_duration_s=duration, minimum_elapsed_s=min(elapsed)
|
||||
)
|
||||
if not deciles:
|
||||
raise ValueError("no complete replay decile is shared by all trials")
|
||||
cumulative_intervals = tuple(
|
||||
(0.0, round(duration * fraction, 9)) for fraction in deciles
|
||||
)
|
||||
block_intervals = tuple(
|
||||
(
|
||||
round(duration * (fraction - DECILE_FRACTION), 9),
|
||||
round(duration * fraction, 9),
|
||||
)
|
||||
for fraction in deciles
|
||||
)
|
||||
all_intervals = tuple(dict.fromkeys([*cumulative_intervals, *block_intervals]))
|
||||
trials_by_interval, streams = load_interval_trials(run_root, all_intervals)
|
||||
manifest_validation = P1.validate_manifest(
|
||||
trials_by_interval[cumulative_intervals[0]], manifest_path
|
||||
)
|
||||
|
||||
cumulative = []
|
||||
blocks = []
|
||||
for fraction, cumulative_interval, block_interval in zip(
|
||||
deciles, cumulative_intervals, block_intervals, strict=True
|
||||
):
|
||||
cumulative.append(
|
||||
slim_window_analysis(
|
||||
trials_by_interval[cumulative_interval],
|
||||
start_s=cumulative_interval[0],
|
||||
end_s=cumulative_interval[1],
|
||||
fraction=fraction,
|
||||
)
|
||||
)
|
||||
blocks.append(
|
||||
slim_window_analysis(
|
||||
trials_by_interval[block_interval],
|
||||
start_s=block_interval[0],
|
||||
end_s=block_interval[1],
|
||||
fraction=fraction,
|
||||
)
|
||||
)
|
||||
|
||||
invariants = {
|
||||
"expected_trial_count": len(metadata) == 36,
|
||||
"trace_duration_consistent": max(durations) - min(durations) <= 1e-9,
|
||||
"all_intervals_uncensored": all(
|
||||
item["elapsed_s"] + 1e-9 >= cumulative_intervals[-1][1]
|
||||
for item in metadata
|
||||
),
|
||||
"stream_provenance_consistent": metadata_streams == streams,
|
||||
"manifest_trials_match": (
|
||||
manifest_validation["expected_trials"]
|
||||
== manifest_validation["matched_trials"]
|
||||
== len(metadata)
|
||||
),
|
||||
"all_window_sanity_pass": all(
|
||||
not item["sanity"]["red_flags"] for item in [*cumulative, *blocks]
|
||||
),
|
||||
}
|
||||
red_flags = [name for name, passed in invariants.items() if not passed]
|
||||
complete_full_trajectory = min(elapsed) + 1e-9 >= duration
|
||||
if red_flags:
|
||||
decision = "STOP_DATA_INVALID"
|
||||
elif not complete_full_trajectory:
|
||||
decision = "REQUIRES_UNCENSORED_PHASE_AWARE_PILOT"
|
||||
else:
|
||||
decision = "FULL_TRAJECTORY_AVAILABLE"
|
||||
|
||||
payload = {
|
||||
"schema": SCHEMA,
|
||||
"status": "COMPLETE",
|
||||
"decision": decision,
|
||||
"claim_boundary": (
|
||||
"Post-hoc corrective audit over every common replay decile. It can "
|
||||
"diagnose horizon sensitivity but cannot establish a held-out tuning claim."
|
||||
),
|
||||
"design": {
|
||||
"decile_fraction": DECILE_FRACTION,
|
||||
"available_deciles": list(deciles),
|
||||
"trace_duration_s": duration,
|
||||
"maximum_common_end_s": cumulative_intervals[-1][1],
|
||||
"maximum_common_fraction": deciles[-1],
|
||||
"select_best_horizon": False,
|
||||
"cumulative_and_nonoverlapping_blocks": True,
|
||||
},
|
||||
"cumulative": cumulative,
|
||||
"blocks": blocks,
|
||||
"trajectory": trajectory_summary(
|
||||
[(interval, trials_by_interval[interval]) for interval in block_intervals]
|
||||
),
|
||||
"provenance": {
|
||||
"analysis_script": str(Path(__file__).resolve()),
|
||||
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
|
||||
"p1_analysis_script": str(P1_PATH.resolve()),
|
||||
"p1_analysis_script_sha256": sha256_file(P1_PATH),
|
||||
"run_root": str(run_root.resolve()),
|
||||
"manifest": str(manifest_path.resolve()),
|
||||
"manifest_sha256": sha256_file(manifest_path),
|
||||
"manifest_validation": manifest_validation,
|
||||
"streams": streams,
|
||||
"trial_inputs": metadata,
|
||||
},
|
||||
"sanity": {
|
||||
"trials": len(metadata),
|
||||
"elapsed_s": numeric(elapsed),
|
||||
"trace_duration_s": numeric(durations),
|
||||
"early_stopped": sum(bool(item["early_stopped"]) for item in metadata),
|
||||
"request_count": numeric(item["request_count"] for item in metadata),
|
||||
"stream_bytes": numeric(item["bytes"] for item in streams),
|
||||
"invariants": invariants,
|
||||
"red_flags": red_flags,
|
||||
},
|
||||
}
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
return payload
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--run-root", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
payload = audit(
|
||||
run_root=args.run_root,
|
||||
manifest_path=args.manifest,
|
||||
output_path=args.output,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"decision": payload["decision"],
|
||||
"design": payload["design"],
|
||||
"sanity": payload["sanity"],
|
||||
},
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
449
runs/intervention-response-v2/analyze_pilot.py
Normal file
449
runs/intervention-response-v2/analyze_pilot.py
Normal file
@@ -0,0 +1,449 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyze the uncensored 300-second phase-aware matched pilot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import math
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable, Mapping
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
P1_PATH = HERE.parent / "intervention-response-v0" / "analyze_p1.py"
|
||||
SCHEMA = "intervention-response-phase-aware-pilot-analysis-v2"
|
||||
EXPECTED_ACTION_PAIRS = 9
|
||||
EXPECTED_REPEAT_PAIRS = 12
|
||||
MIN_EFFICACY_CLASS = 3
|
||||
|
||||
|
||||
def _load_p1():
|
||||
spec = importlib.util.spec_from_file_location("intervention_response_p1", P1_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
P1 = _load_p1()
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def numeric(values: Iterable[float | int]) -> dict[str, Any]:
|
||||
return P1.V0.numeric(values)
|
||||
|
||||
|
||||
def _trial_record(
|
||||
*,
|
||||
run_root: Path,
|
||||
session: Mapping[str, Any],
|
||||
level: str,
|
||||
result: Mapping[str, Any],
|
||||
result_path: Path,
|
||||
requests_path: Path,
|
||||
state: dict[str, float],
|
||||
outcome: dict[str, float],
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"trial_id": str(result_path.relative_to(run_root)),
|
||||
"cell": str(result["cell"]),
|
||||
"tp": int(result["tp"]),
|
||||
"mns": int(result["mns"]),
|
||||
"level": level,
|
||||
"replicate": int(session["replicate"]),
|
||||
"offered_rate_per_gpu": float(
|
||||
result["selection"]["offered_req_s_per_gpu"]
|
||||
),
|
||||
"request_hash": str(result["selection"]["request_id_order_sha256"]),
|
||||
"request_count": int(result["selection"]["count"]),
|
||||
"result_sha256": sha256_file(result_path),
|
||||
"requests_sha256": sha256_file(requests_path),
|
||||
"full_pass_rate": float(result["pass_rate"]),
|
||||
"full_feasible": bool(result["feasible"]),
|
||||
"early_stopped": bool(result["early_stopped"]),
|
||||
"state": state,
|
||||
"outcome": outcome,
|
||||
}
|
||||
|
||||
|
||||
def validate_result_against_manifest(
|
||||
*,
|
||||
result: Mapping[str, Any],
|
||||
selection: Mapping[str, Any],
|
||||
session: Mapping[str, Any],
|
||||
level: str,
|
||||
expected_duration_s: float,
|
||||
) -> None:
|
||||
identity = f"{session['session']}:{level}"
|
||||
if int(result["mns"]) != int(session["mns"]) or int(result["tp"]) != 4:
|
||||
raise ValueError(f"config mismatch: {identity}")
|
||||
if bool(result["early_stopped"]):
|
||||
raise ValueError(f"early-stopped measured result: {identity}")
|
||||
if result.get("slo_early_stop_disabled") is not True:
|
||||
raise ValueError(f"SLO early stop was enabled: {identity}")
|
||||
if float(result["interval"]["elapsed_s"]) + 1e-9 < expected_duration_s:
|
||||
raise ValueError(f"result does not cover full arrival window: {identity}")
|
||||
if int(result["selection"]["count"]) != int(selection["selected_count"]):
|
||||
raise ValueError(f"selection count mismatch: {identity}")
|
||||
for result_key, manifest_key in (
|
||||
("request_id_order_sha256", "request_id_order_sha256"),
|
||||
("arrival_order_sha256", "arrival_order_sha256"),
|
||||
("raw_length_order_sha256", "input_length_order_sha256"),
|
||||
):
|
||||
if result["selection"][result_key] != selection[manifest_key]:
|
||||
raise ValueError(f"selection hash mismatch {result_key}: {identity}")
|
||||
if int(result["observed_count"]) != int(selection["selected_count"]):
|
||||
raise ValueError(f"request accounting mismatch: {identity}")
|
||||
|
||||
|
||||
def load_interval_trials(
|
||||
*,
|
||||
run_root: Path,
|
||||
manifest: Mapping[str, Any],
|
||||
intervals_s: tuple[tuple[float, float], ...],
|
||||
) -> tuple[dict[tuple[float, float], list[dict[str, Any]]], list[dict[str, Any]]]:
|
||||
by_interval = {interval: [] for interval in intervals_s}
|
||||
streams = []
|
||||
duration_s = float(manifest["engine"]["duration_s"])
|
||||
for session in manifest["sessions"]:
|
||||
session_root = run_root / "sessions" / str(session["session"])
|
||||
stream_paths = sorted((session_root / "opprof").glob("*.jsonl"))
|
||||
if len(stream_paths) != 1:
|
||||
raise ValueError(f"{session_root}: expected one Layer-1 stream")
|
||||
stream_path = stream_paths[0]
|
||||
stream = P1.load_jsonl(stream_path)
|
||||
streams.append(
|
||||
{
|
||||
"session": str(session["session"]),
|
||||
"path": str(stream_path.resolve()),
|
||||
"sha256": sha256_file(stream_path),
|
||||
"bytes": stream_path.stat().st_size,
|
||||
}
|
||||
)
|
||||
repetition = manifest["repetitions"][str(session["replicate"])]
|
||||
for level, selection in repetition["selections"].items():
|
||||
result_path = session_root / level / "result.json"
|
||||
requests_path = session_root / level / "requests.jsonl"
|
||||
result = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
requests = P1.load_jsonl(requests_path)
|
||||
validate_result_against_manifest(
|
||||
result=result,
|
||||
selection=selection,
|
||||
session=session,
|
||||
level=level,
|
||||
expected_duration_s=duration_s,
|
||||
)
|
||||
start_ns = int(result["interval"]["start_mono_ns"])
|
||||
for interval in intervals_s:
|
||||
start_s, end_s = interval
|
||||
state = P1.V0.flatten_state(
|
||||
P1.summarize_engine(
|
||||
stream,
|
||||
start_ns=start_ns + int(start_s * 1e9),
|
||||
end_ns=start_ns + int(end_s * 1e9),
|
||||
request_count=int(result["selection"]["count"]),
|
||||
)
|
||||
)
|
||||
outcome = P1._prefix_outcome(result, requests, end_s)
|
||||
by_interval[interval].append(
|
||||
_trial_record(
|
||||
run_root=run_root,
|
||||
session=session,
|
||||
level=level,
|
||||
result=result,
|
||||
result_path=result_path,
|
||||
requests_path=requests_path,
|
||||
state=state,
|
||||
outcome=outcome,
|
||||
)
|
||||
)
|
||||
return by_interval, streams
|
||||
|
||||
|
||||
def analyze_window(
|
||||
trials: list[dict[str, Any]], *, start_s: float, end_s: float, fraction: float
|
||||
) -> dict[str, Any]:
|
||||
actions, repeats = P1.build_pairs(trials)
|
||||
response = P1.V0.response_statistics(actions, repeats)
|
||||
response_qualifying = sorted(
|
||||
feature for feature, item in response.items() if item["qualifies"]
|
||||
)
|
||||
labels = [int(pair["full_action_efficacy"]) for pair in actions]
|
||||
cross_validation_possible = all(
|
||||
set(
|
||||
int(pair["full_action_efficacy"])
|
||||
for pair in actions
|
||||
if pair["group"]["replicate"] != held_out
|
||||
)
|
||||
== {0, 1}
|
||||
for held_out in (1, 2, 3)
|
||||
)
|
||||
if cross_validation_possible:
|
||||
outcome_cv = P1.one_feature_leave_repeat_out(
|
||||
actions, delta_key="delta_outcome", features=P1.OUTCOME_FEATURES
|
||||
)
|
||||
telemetry_cv = P1.one_feature_leave_repeat_out(
|
||||
actions, delta_key="delta_state", features=P1.V0.GATE_FEATURES
|
||||
)
|
||||
outcome_best = float(outcome_cv["best_balanced_accuracy"])
|
||||
efficacy_qualifying = sorted(
|
||||
feature
|
||||
for feature, item in telemetry_cv["features"].items()
|
||||
if item["balanced_accuracy"] >= P1.MIN_EFFICACY_BALANCED_ACCURACY
|
||||
and item["balanced_accuracy"]
|
||||
>= outcome_best + P1.MIN_EFFICACY_DELTA_OVER_OUTCOME
|
||||
)
|
||||
else:
|
||||
unavailable = {
|
||||
"status": "UNAVAILABLE",
|
||||
"reason": "each leave-one-repetition-out train fold needs both classes",
|
||||
}
|
||||
outcome_cv = unavailable
|
||||
telemetry_cv = unavailable
|
||||
efficacy_qualifying = []
|
||||
transitions = defaultdict(int)
|
||||
for pair in actions:
|
||||
transitions[pair["full_feasibility_transition"]] += 1
|
||||
admitted = [float(trial["outcome"]["admitted_fraction"]) for trial in trials]
|
||||
completed = [
|
||||
float(trial["outcome"]["admitted_fraction"])
|
||||
* float(trial["outcome"]["completed_over_admitted"])
|
||||
for trial in trials
|
||||
]
|
||||
action_metadata = [
|
||||
{
|
||||
"group": pair["group"],
|
||||
"source": pair["source"],
|
||||
"target": pair["target"],
|
||||
"full_action_efficacy": pair["full_action_efficacy"],
|
||||
"full_feasibility_transition": pair["full_feasibility_transition"],
|
||||
"delta_state": pair["delta_state"],
|
||||
"delta_outcome": pair["delta_outcome"],
|
||||
}
|
||||
for pair in actions
|
||||
]
|
||||
invariants = {
|
||||
"expected_action_pair_count": len(actions) == EXPECTED_ACTION_PAIRS,
|
||||
"expected_repeat_pair_count": len(repeats) == EXPECTED_REPEAT_PAIRS,
|
||||
"finite_deltas": all(
|
||||
math.isfinite(value)
|
||||
for pair in [*actions, *repeats]
|
||||
for key in ("delta_state", "delta_outcome")
|
||||
for value in pair[key].values()
|
||||
),
|
||||
"all_results_uncensored": all(not trial["early_stopped"] for trial in trials),
|
||||
}
|
||||
return {
|
||||
"start_s": start_s,
|
||||
"end_s": end_s,
|
||||
"end_fraction": fraction,
|
||||
"coverage_at_end": {
|
||||
"admitted_fraction_of_total": numeric(admitted),
|
||||
"completed_fraction_of_total": numeric(completed),
|
||||
},
|
||||
"action_pairs": len(actions),
|
||||
"repeat_pairs": len(repeats),
|
||||
"actions": action_metadata,
|
||||
"response_statistics": response,
|
||||
"qualifying_response_features": response_qualifying,
|
||||
"efficacy": {
|
||||
"labels": numeric(labels),
|
||||
"positive": sum(labels),
|
||||
"negative": len(labels) - sum(labels),
|
||||
"label_balance_sufficient": (
|
||||
sum(labels) >= MIN_EFFICACY_CLASS
|
||||
and len(labels) - sum(labels) >= MIN_EFFICACY_CLASS
|
||||
),
|
||||
"cross_validation_possible": cross_validation_possible,
|
||||
"transitions": dict(sorted(transitions.items())),
|
||||
"outcome_delta": outcome_cv,
|
||||
"telemetry_delta": telemetry_cv,
|
||||
"telemetry_qualifying_features": efficacy_qualifying,
|
||||
},
|
||||
"sanity": {
|
||||
"trials": len(trials),
|
||||
"invariants": invariants,
|
||||
"red_flags": [name for name, passed in invariants.items() if not passed],
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def stable_adjacent_features(windows: list[dict[str, Any]]) -> dict[str, list[str]]:
|
||||
result = {}
|
||||
for left, right in zip(windows, windows[1:], strict=False):
|
||||
key = f"{left['end_fraction']:.2f}->{right['end_fraction']:.2f}"
|
||||
result[key] = sorted(
|
||||
set(left["qualifying_response_features"])
|
||||
& set(right["qualifying_response_features"])
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def consistent_load_regimes(
|
||||
windows: list[dict[str, Any]], stable: dict[str, list[str]]
|
||||
) -> dict[str, Any]:
|
||||
by_end = {float(window["end_fraction"]): window for window in windows}
|
||||
result = {}
|
||||
for transition, features in stable.items():
|
||||
_left_text, right_text = transition.split("->")
|
||||
window = by_end[float(right_text)]
|
||||
for feature in features:
|
||||
deltas_by_level: dict[str, list[float]] = defaultdict(list)
|
||||
all_deltas = []
|
||||
for action in window["actions"]:
|
||||
value = float(action["delta_state"][feature])
|
||||
deltas_by_level[str(action["group"]["level"])].append(value)
|
||||
all_deltas.append(value)
|
||||
positive = sum(value > 1e-12 for value in all_deltas)
|
||||
negative = sum(value < -1e-12 for value in all_deltas)
|
||||
direction = 1 if positive >= negative else -1
|
||||
consistent = []
|
||||
for level, values in sorted(deltas_by_level.items()):
|
||||
matching = sum(direction * value > 1e-12 for value in values)
|
||||
nonzero = sum(abs(value) > 1e-12 for value in values)
|
||||
if nonzero and matching / nonzero >= 2.0 / 3.0:
|
||||
consistent.append(level)
|
||||
result[f"{transition}:{feature}"] = {
|
||||
"direction": direction,
|
||||
"consistent_load_regimes": consistent,
|
||||
"passes_two_regimes": len(consistent) >= 2,
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def audit(*, run_root: Path, manifest_path: Path, output_path: Path) -> dict[str, Any]:
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
if manifest.get("schema") != "intervention-response-phase-aware-pilot-manifest-v2":
|
||||
raise ValueError("unexpected phase-aware pilot manifest schema")
|
||||
fractions = [float(value) for value in manifest["checkpoints"]["fractions"]]
|
||||
seconds = [float(value) for value in manifest["checkpoints"]["seconds"]]
|
||||
cumulative_intervals = tuple((0.0, end_s) for end_s in seconds)
|
||||
quarter_intervals = ((0.0, 75.0), (75.0, 150.0), (150.0, 225.0), (225.0, 300.0))
|
||||
intervals = tuple(dict.fromkeys([*cumulative_intervals, *quarter_intervals]))
|
||||
trials_by_interval, streams = load_interval_trials(
|
||||
run_root=run_root, manifest=manifest, intervals_s=intervals
|
||||
)
|
||||
cumulative = [
|
||||
analyze_window(
|
||||
trials_by_interval[interval],
|
||||
start_s=interval[0],
|
||||
end_s=interval[1],
|
||||
fraction=fraction,
|
||||
)
|
||||
for fraction, interval in zip(fractions, cumulative_intervals, strict=True)
|
||||
]
|
||||
quarter_blocks = [
|
||||
analyze_window(
|
||||
trials_by_interval[interval],
|
||||
start_s=interval[0],
|
||||
end_s=interval[1],
|
||||
fraction=interval[1] / 300.0,
|
||||
)
|
||||
for interval in quarter_intervals
|
||||
]
|
||||
stable = stable_adjacent_features(cumulative)
|
||||
load_consistency = consistent_load_regimes(cumulative, stable)
|
||||
mechanism_features = sorted(
|
||||
{
|
||||
key.split(":", 1)[1]
|
||||
for key, item in load_consistency.items()
|
||||
if item["passes_two_regimes"]
|
||||
}
|
||||
)
|
||||
full = cumulative[-1]
|
||||
efficacy_features = sorted(
|
||||
set.intersection(
|
||||
*(
|
||||
set(window["efficacy"]["telemetry_qualifying_features"])
|
||||
for window in cumulative
|
||||
if window["end_fraction"] >= 0.25
|
||||
)
|
||||
)
|
||||
)
|
||||
red_flags = sorted(
|
||||
{
|
||||
flag
|
||||
for window in [*cumulative, *quarter_blocks]
|
||||
for flag in window["sanity"]["red_flags"]
|
||||
}
|
||||
)
|
||||
if red_flags:
|
||||
decision = "STOP_DATA_INVALID"
|
||||
elif not mechanism_features:
|
||||
decision = "STOP_NO_PHASE_STABLE_RESPONSE"
|
||||
elif not full["efficacy"]["label_balance_sufficient"]:
|
||||
decision = "MECHANISM_ONLY_NO_LABEL_BALANCE"
|
||||
elif not efficacy_features:
|
||||
decision = "STOP_NO_INCREMENTAL_TUNING_SIGNAL"
|
||||
else:
|
||||
decision = "OPEN_E2E_POLICY_TEST"
|
||||
payload = {
|
||||
"schema": SCHEMA,
|
||||
"status": "COMPLETE",
|
||||
"decision": decision,
|
||||
"claim_boundary": "Development mechanism pilot; not a held-out paper claim.",
|
||||
"mechanism_features": mechanism_features,
|
||||
"stable_adjacent_features": stable,
|
||||
"load_consistency": load_consistency,
|
||||
"stable_incremental_efficacy_features": efficacy_features,
|
||||
"cumulative": cumulative,
|
||||
"quarter_blocks": quarter_blocks,
|
||||
"provenance": {
|
||||
"analysis_script": str(Path(__file__).resolve()),
|
||||
"analysis_script_sha256": sha256_file(Path(__file__).resolve()),
|
||||
"manifest": str(manifest_path.resolve()),
|
||||
"manifest_sha256": sha256_file(manifest_path),
|
||||
"run_root": str(run_root.resolve()),
|
||||
"streams": streams,
|
||||
},
|
||||
"sanity": {
|
||||
"streams": len(streams),
|
||||
"stream_bytes": numeric(item["bytes"] for item in streams),
|
||||
"red_flags": red_flags,
|
||||
},
|
||||
}
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
return payload
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--run-root", type=Path, required=True)
|
||||
parser.add_argument("--manifest", type=Path, required=True)
|
||||
parser.add_argument("--output", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
payload = audit(
|
||||
run_root=args.run_root,
|
||||
manifest_path=args.manifest,
|
||||
output_path=args.output,
|
||||
)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"decision": payload["decision"],
|
||||
"mechanism_features": payload["mechanism_features"],
|
||||
"stable_incremental_efficacy_features": payload[
|
||||
"stable_incremental_efficacy_features"
|
||||
],
|
||||
"sanity": payload["sanity"],
|
||||
},
|
||||
indent=2,
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
9915
runs/intervention-response-v2/existing-phase-audit.json
Normal file
9915
runs/intervention-response-v2/existing-phase-audit.json
Normal file
File diff suppressed because it is too large
Load Diff
471
runs/intervention-response-v2/pilot_controller.py
Normal file
471
runs/intervention-response-v2/pilot_controller.py
Normal file
@@ -0,0 +1,471 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Serialized, resumable controller for the 300-second phase-aware pilot."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import shlex
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
PHASE6 = HERE.parent / "opprof-phase6"
|
||||
sys.path.insert(0, str(PHASE6))
|
||||
|
||||
import opprof_phase6_controller as base # noqa: E402
|
||||
|
||||
|
||||
SCHEMA = "intervention-response-phase-aware-pilot-state-v2"
|
||||
SESSION_ESTIMATE_H20_HOURS = 1.25
|
||||
SAFETY_H20_HOURS = 0.20
|
||||
CLIENT_TIMEOUT_S = 450.0
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
base.atomic_json(path, payload)
|
||||
|
||||
|
||||
def wait_all_idle(timeout_s: float = 30.0) -> None:
|
||||
deadline = time.monotonic() + timeout_s
|
||||
last_error: Exception | None = None
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
base.assert_all_idle()
|
||||
return
|
||||
except RuntimeError as error:
|
||||
last_error = error
|
||||
time.sleep(1.0)
|
||||
raise last_error or RuntimeError("GPU idle timeout")
|
||||
|
||||
|
||||
def configure(args: argparse.Namespace, manifest: dict[str, Any]) -> None:
|
||||
base.WORKDIR = args.run_root.parent
|
||||
base.RUN_ROOT = args.run_root
|
||||
base.STATE = args.run_root / "controller-state.json"
|
||||
base.SOURCE = args.vllm_source
|
||||
base.VENV = args.venv
|
||||
base.AITUNER = args.aituner_root
|
||||
base.MODEL = args.model
|
||||
base.CLIENT = args.client
|
||||
base.GPU_LIMIT = float(manifest["budget"]["hard_cap_h20_hours"])
|
||||
base.MARKER = "intervention-response-phase-aware-v2"
|
||||
base.CELLS = {
|
||||
f"tp4_mns{mns}": {"tp": 4, "mns": int(mns)}
|
||||
for mns in manifest["engine"]["mns_endpoints"]
|
||||
}
|
||||
|
||||
|
||||
def load_state(path: Path, hard_cap: float) -> dict[str, Any]:
|
||||
if path.exists():
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"status": "initialized",
|
||||
"hard_cap_h20_hours": hard_cap,
|
||||
"gpu_hours_total": 0.0,
|
||||
"completed_sessions": 0,
|
||||
"sessions": {},
|
||||
"failures": [],
|
||||
"started_at": time.time(),
|
||||
}
|
||||
|
||||
|
||||
def save_state(path: Path, state: dict[str, Any]) -> None:
|
||||
atomic_json(path, state)
|
||||
|
||||
|
||||
def append_echo(run_root: Path, line: str) -> None:
|
||||
run_root.mkdir(parents=True, exist_ok=True)
|
||||
with (run_root / "launch-echo.log").open("a", encoding="utf-8") as target:
|
||||
target.write(line + "\n")
|
||||
print(line, flush=True)
|
||||
|
||||
|
||||
def remaining_projection(session_count: int, index: int) -> float:
|
||||
return (session_count - index) * SESSION_ESTIMATE_H20_HOURS + SAFETY_H20_HOURS
|
||||
|
||||
|
||||
def start_server(
|
||||
*, session: dict[str, Any], index: int, run_root: Path
|
||||
) -> dict[str, Any]:
|
||||
cell = f"tp4_mns{int(session['mns'])}"
|
||||
gpus = (0, 1, 2, 3)
|
||||
session_root = run_root / "sessions" / str(session["session"])
|
||||
session_root.mkdir(parents=True, exist_ok=True)
|
||||
port = 8950 + index
|
||||
command = base.server_command(cell, gpus, port)
|
||||
with (session_root / "commands.log").open("a", encoding="utf-8") as log:
|
||||
log.write(f"SERVER {shlex.join(command)}\n")
|
||||
server_log = (session_root / "server.log").open("ab", buffering=0)
|
||||
environment = os.environ.copy()
|
||||
environment.update(
|
||||
{
|
||||
"CUDA_VISIBLE_DEVICES": "0,1,2,3",
|
||||
"VLLM_OPPROF_DIR": str(session_root / "opprof"),
|
||||
"OPPROF_PHASE6_MARKER": base.MARKER,
|
||||
"AITUNER_ROOT": str(base.AITUNER),
|
||||
"HF_HUB_OFFLINE": "1",
|
||||
"TRANSFORMERS_OFFLINE": "1",
|
||||
"PYTHONUNBUFFERED": "1",
|
||||
}
|
||||
)
|
||||
server = subprocess.Popen(
|
||||
command,
|
||||
cwd=base.SOURCE,
|
||||
env=environment,
|
||||
stdout=server_log,
|
||||
stderr=subprocess.STDOUT,
|
||||
start_new_session=True,
|
||||
)
|
||||
base.OWNED_PGIDS.add(server.pid)
|
||||
return {
|
||||
"cell": cell,
|
||||
"gpus": gpus,
|
||||
"port": port,
|
||||
"dir": session_root,
|
||||
"server": server,
|
||||
"server_handle": server_log,
|
||||
"spawned_at": time.time(),
|
||||
"results": [],
|
||||
}
|
||||
|
||||
|
||||
def client_command(
|
||||
entry: dict[str, Any],
|
||||
*,
|
||||
study: str,
|
||||
anchor: float,
|
||||
output: Path,
|
||||
warmup: bool,
|
||||
) -> list[str]:
|
||||
config = base.CELLS[entry["cell"]]
|
||||
command = [
|
||||
"taskset",
|
||||
"-c",
|
||||
base.cpu_mask(entry["gpus"]),
|
||||
str(base.VENV / "bin/python"),
|
||||
str(base.CLIENT),
|
||||
"warmup" if warmup else "run-anchor",
|
||||
"--study",
|
||||
study,
|
||||
"--cell",
|
||||
entry["cell"],
|
||||
"--anchor",
|
||||
str(anchor),
|
||||
"--tp",
|
||||
str(config["tp"]),
|
||||
"--mns",
|
||||
str(config["mns"]),
|
||||
"--base-url",
|
||||
f"http://127.0.0.1:{entry['port']}",
|
||||
"--result-dir",
|
||||
str(output),
|
||||
"--disable-slo-early-stop",
|
||||
]
|
||||
return command
|
||||
|
||||
|
||||
def run_client(
|
||||
*,
|
||||
entry: dict[str, Any],
|
||||
role: str,
|
||||
study: str,
|
||||
selection: dict[str, Any],
|
||||
output: Path,
|
||||
state: dict[str, Any],
|
||||
warmup: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
command = client_command(
|
||||
entry,
|
||||
study=study,
|
||||
anchor=float(selection["anchor"]),
|
||||
output=output,
|
||||
warmup=warmup,
|
||||
)
|
||||
with (entry["dir"] / "commands.log").open("a", encoding="utf-8") as log:
|
||||
log.write(f"CLIENT role={role} {shlex.join(command)}\n")
|
||||
handle = (output.parent / f"{output.name}.log").open("ab", buffering=0)
|
||||
environment = os.environ.copy()
|
||||
environment.update({"AITUNER_ROOT": str(base.AITUNER), "PYTHONUNBUFFERED": "1"})
|
||||
process = subprocess.Popen(
|
||||
command,
|
||||
cwd=base.WORKDIR,
|
||||
env=environment,
|
||||
stdout=handle,
|
||||
stderr=subprocess.STDOUT,
|
||||
start_new_session=True,
|
||||
)
|
||||
deadline = time.monotonic() + (180.0 if warmup else CLIENT_TIMEOUT_S)
|
||||
try:
|
||||
while process.poll() is None:
|
||||
if time.monotonic() > deadline:
|
||||
raise TimeoutError(f"client timeout: {entry['cell']} {role}")
|
||||
if entry["server"].poll() is not None:
|
||||
raise RuntimeError(f"server exited during {entry['cell']} {role}")
|
||||
base.assert_no_other_compute()
|
||||
if state["gpu_hours_total"] + base.live_gpu_hours([entry]) >= base.GPU_LIMIT:
|
||||
raise RuntimeError("phase-aware pilot H20-hour hard cap reached")
|
||||
time.sleep(1.0)
|
||||
except Exception:
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGTERM)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
try:
|
||||
process.wait(timeout=10.0)
|
||||
except subprocess.TimeoutExpired:
|
||||
try:
|
||||
os.killpg(process.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
process.wait(timeout=10.0)
|
||||
raise
|
||||
finally:
|
||||
handle.close()
|
||||
if process.returncode:
|
||||
raise RuntimeError(
|
||||
f"client failed: cell={entry['cell']} role={role} rc={process.returncode}"
|
||||
)
|
||||
result = json.loads((output / "result.json").read_text(encoding="utf-8"))
|
||||
validate_result(
|
||||
result=result,
|
||||
selection=selection,
|
||||
role=role,
|
||||
warmup=warmup,
|
||||
)
|
||||
entry["results"].append(
|
||||
{
|
||||
"anchor": float(selection["anchor"]),
|
||||
"dir": str(output),
|
||||
"kind": result["kind"],
|
||||
}
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def validate_result(
|
||||
*, result: dict[str, Any], selection: dict[str, Any], role: str, warmup: bool
|
||||
) -> None:
|
||||
if result.get("slo_early_stop_disabled") is not True:
|
||||
raise RuntimeError(f"SLO early stop was not disabled: {role}")
|
||||
if warmup:
|
||||
if result["kind"] != "warmup" or int(result["selection"]["count"]) != 16:
|
||||
raise RuntimeError(f"invalid warmup result: {role}")
|
||||
if not all(
|
||||
result["invariants"].get(key, False)
|
||||
for key in ("warmup_16", "warmup_exact_16", "warmup_long")
|
||||
):
|
||||
raise RuntimeError(f"warmup invariant failed: {role}")
|
||||
return
|
||||
if bool(result["early_stopped"]):
|
||||
raise RuntimeError(f"uncensored run early-stopped: {role}")
|
||||
if int(result["selection"]["count"]) != int(selection["selected_count"]):
|
||||
raise RuntimeError(f"selection count mismatch: {role}")
|
||||
for key, manifest_key in (
|
||||
("request_id_order_sha256", "request_id_order_sha256"),
|
||||
("arrival_order_sha256", "arrival_order_sha256"),
|
||||
("raw_length_order_sha256", "input_length_order_sha256"),
|
||||
):
|
||||
if result["selection"][key] != selection[manifest_key]:
|
||||
raise RuntimeError(f"selection hash mismatch {key}: {role}")
|
||||
if int(result["observed_count"]) != int(selection["selected_count"]):
|
||||
raise RuntimeError(f"request accounting mismatch: {role}")
|
||||
|
||||
|
||||
def execute_session(
|
||||
*,
|
||||
index: int,
|
||||
session: dict[str, Any],
|
||||
manifest: dict[str, Any],
|
||||
run_root: Path,
|
||||
state_path: Path,
|
||||
state: dict[str, Any],
|
||||
) -> None:
|
||||
name = str(session["session"])
|
||||
if state["sessions"].get(name, {}).get("status") == "complete":
|
||||
return
|
||||
projection = remaining_projection(len(manifest["sessions"]), index)
|
||||
if state["gpu_hours_total"] + projection > base.GPU_LIMIT:
|
||||
state["status"] = "budget_projection_stop"
|
||||
state["budget_stop"] = {
|
||||
"before_session": name,
|
||||
"spent_h20_hours": state["gpu_hours_total"],
|
||||
"remaining_projection_h20_hours": projection,
|
||||
"hard_cap_h20_hours": base.GPU_LIMIT,
|
||||
}
|
||||
save_state(state_path, state)
|
||||
raise RuntimeError(f"projected pilot cost exceeds hard cap before {name}")
|
||||
|
||||
replicate = str(session["replicate"])
|
||||
repetition = manifest["repetitions"][replicate]
|
||||
echo = (
|
||||
f"PHASE_PILOT_SESSION_ECHO session={name} tp=4 mns={session['mns']} "
|
||||
f"gpus=0-3 workload={manifest['source']['window_id']} duration_s=300 "
|
||||
f"loads={','.join(repetition['load_order'])} disable_slo_early_stop=true "
|
||||
f"spent_h20h={state['gpu_hours_total']:.6f} "
|
||||
f"remaining_projection_h20h={projection:.3f} cap_h20h={base.GPU_LIMIT:.1f} "
|
||||
f"manifest={run_root / 'pilot-manifest.json'}"
|
||||
)
|
||||
append_echo(run_root, echo)
|
||||
wait_all_idle()
|
||||
session_state = {
|
||||
"status": "starting",
|
||||
"replicate": int(replicate),
|
||||
"mns": int(session["mns"]),
|
||||
"started_at": time.time(),
|
||||
"runs": [],
|
||||
}
|
||||
state["status"] = "running"
|
||||
state["sessions"][name] = session_state
|
||||
save_state(state_path, state)
|
||||
entry = start_server(session=session, index=index, run_root=run_root)
|
||||
failure: Exception | None = None
|
||||
try:
|
||||
base.wait_ready(entry)
|
||||
high = repetition["selections"]["high"]
|
||||
session_state["status"] = "warmup"
|
||||
save_state(state_path, state)
|
||||
run_client(
|
||||
entry=entry,
|
||||
role="warmup",
|
||||
study=repetition["study"],
|
||||
selection=high,
|
||||
output=entry["dir"] / "warmup",
|
||||
state=state,
|
||||
warmup=True,
|
||||
)
|
||||
session_state["status"] = "burnin"
|
||||
save_state(state_path, state)
|
||||
burnin = manifest["burnin"]
|
||||
burnin_result = run_client(
|
||||
entry=entry,
|
||||
role="burnin",
|
||||
study=burnin["study"],
|
||||
selection=burnin,
|
||||
output=entry["dir"] / "burnin",
|
||||
state=state,
|
||||
)
|
||||
session_state["burnin"] = {
|
||||
"pass_rate": burnin_result["pass_rate"],
|
||||
"feasible": burnin_result["feasible"],
|
||||
"elapsed_s": burnin_result["interval"]["elapsed_s"],
|
||||
}
|
||||
session_state["status"] = "measured"
|
||||
save_state(state_path, state)
|
||||
for level in repetition["load_order"]:
|
||||
selection = repetition["selections"][level]
|
||||
result = run_client(
|
||||
entry=entry,
|
||||
role=level,
|
||||
study=repetition["study"],
|
||||
selection=selection,
|
||||
output=entry["dir"] / level,
|
||||
state=state,
|
||||
)
|
||||
session_state["runs"].append(
|
||||
{
|
||||
"level": level,
|
||||
"selected_count": selection["selected_count"],
|
||||
"offered_req_s_per_gpu": selection["offered_req_s_per_gpu"],
|
||||
"pass_rate": result["pass_rate"],
|
||||
"feasible": result["feasible"],
|
||||
"elapsed_s": result["interval"]["elapsed_s"],
|
||||
"early_stopped": result["early_stopped"],
|
||||
}
|
||||
)
|
||||
save_state(state_path, state)
|
||||
session_state["status"] = "stopping"
|
||||
save_state(state_path, state)
|
||||
except Exception as error: # noqa: BLE001
|
||||
failure = error
|
||||
finally:
|
||||
try:
|
||||
base.stop_entry(entry)
|
||||
except Exception as error: # noqa: BLE001
|
||||
failure = failure or error
|
||||
time.sleep(2.0)
|
||||
try:
|
||||
wait_all_idle()
|
||||
except Exception as error: # noqa: BLE001
|
||||
failure = failure or error
|
||||
|
||||
session_hours = base.live_gpu_hours([entry])
|
||||
state["gpu_hours_total"] += session_hours
|
||||
session_state["gpu_hours"] = session_hours
|
||||
if failure is not None:
|
||||
session_state["status"] = "failed"
|
||||
session_state["failure"] = repr(failure)
|
||||
state["status"] = "failed"
|
||||
state["failures"].append({"session": name, "failure": repr(failure)})
|
||||
save_state(state_path, state)
|
||||
raise failure
|
||||
validation = base.validate_cell(entry)
|
||||
session_state["validation"] = validation
|
||||
session_state["status"] = "complete"
|
||||
session_state["completed_at"] = time.time()
|
||||
state["completed_sessions"] += 1
|
||||
save_state(state_path, state)
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
result = argparse.ArgumentParser()
|
||||
result.add_argument("--manifest", type=Path, required=True)
|
||||
result.add_argument("--run-root", type=Path, required=True)
|
||||
result.add_argument("--aituner-root", type=Path, required=True)
|
||||
result.add_argument("--vllm-source", type=Path, required=True)
|
||||
result.add_argument("--venv", type=Path, required=True)
|
||||
result.add_argument("--model", type=Path, required=True)
|
||||
result.add_argument("--client", type=Path, required=True)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parser().parse_args()
|
||||
manifest = json.loads(args.manifest.read_text(encoding="utf-8"))
|
||||
if manifest.get("schema") != "intervention-response-phase-aware-pilot-manifest-v2":
|
||||
raise RuntimeError("unexpected phase-aware pilot manifest schema")
|
||||
if manifest["status"] != "PASS":
|
||||
raise RuntimeError("phase-aware pilot manifest did not pass preflight")
|
||||
args.run_root.mkdir(parents=True, exist_ok=True)
|
||||
copied_manifest = args.run_root / "pilot-manifest.json"
|
||||
if not copied_manifest.exists():
|
||||
atomic_json(copied_manifest, manifest)
|
||||
configure(args, manifest)
|
||||
state_path = args.run_root / "controller-state.json"
|
||||
state = load_state(state_path, base.GPU_LIMIT)
|
||||
state["status"] = "running"
|
||||
save_state(state_path, state)
|
||||
for index, session in enumerate(manifest["sessions"]):
|
||||
execute_session(
|
||||
index=index,
|
||||
session=session,
|
||||
manifest=manifest,
|
||||
run_root=args.run_root,
|
||||
state_path=state_path,
|
||||
state=state,
|
||||
)
|
||||
state["status"] = "complete"
|
||||
state["completed_at"] = time.time()
|
||||
save_state(state_path, state)
|
||||
wait_all_idle()
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": state["status"],
|
||||
"completed_sessions": state["completed_sessions"],
|
||||
"gpu_hours_total": state["gpu_hours_total"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
267
runs/intervention-response-v2/prepare_pilot.py
Normal file
267
runs/intervention-response-v2/prepare_pilot.py
Normal file
@@ -0,0 +1,267 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prepare the uncensored 300-second TP4 matched pilot on dash0."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
|
||||
sys.path.insert(0, str(AITUNER_ROOT / "src"))
|
||||
|
||||
from aituner.spec import load_study_spec # noqa: E402
|
||||
from aituner.trace import load_trace_requests, select_requests_for_threshold # noqa: E402
|
||||
|
||||
|
||||
SCHEMA = "intervention-response-phase-aware-pilot-manifest-v2"
|
||||
TP = 4
|
||||
MNS_ENDPOINTS = (16, 64)
|
||||
REPLAY_TIME_SCALE = 0.5
|
||||
EXPECTED_DURATION_S = 300.0
|
||||
SAFETY_DEADLINE_S = 360.0
|
||||
LOADS_REQ_S_GPU = {"low": 1.5, "mid": 2.125, "high": 3.125}
|
||||
REPLICATE_ROLES = ("high1", "high2", "high3")
|
||||
LOAD_ORDERS = {
|
||||
1: ("low", "mid", "high"),
|
||||
2: ("high", "low", "mid"),
|
||||
3: ("mid", "high", "low"),
|
||||
}
|
||||
SESSION_ORDER = (
|
||||
(1, 16),
|
||||
(1, 64),
|
||||
(2, 64),
|
||||
(2, 16),
|
||||
(3, 16),
|
||||
(3, 64),
|
||||
)
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def order_hash(values: list[str]) -> str:
|
||||
return hashlib.sha256("\n".join(values).encode()).hexdigest()
|
||||
|
||||
|
||||
def attainable_anchor(requests: list[Any], target_count: int) -> tuple[float, list[Any]]:
|
||||
ordered = sorted(float(request.sampling_u) for request in requests)
|
||||
if not ordered:
|
||||
raise ValueError("no requests remain after study filtering")
|
||||
if target_count <= 0 or target_count > len(ordered):
|
||||
raise ValueError(
|
||||
f"target count {target_count} is outside available range 1..{len(ordered)}"
|
||||
)
|
||||
indices = sorted(
|
||||
{
|
||||
max(0, min(len(ordered) - 1, target_count - 1)),
|
||||
max(0, min(len(ordered) - 1, target_count)),
|
||||
}
|
||||
)
|
||||
candidates = []
|
||||
for index in indices:
|
||||
anchor = ordered[index]
|
||||
selected = select_requests_for_threshold(requests, threshold=anchor)
|
||||
candidates.append((abs(len(selected) - target_count), len(selected), anchor, selected))
|
||||
_error, _count, anchor, selected = min(
|
||||
candidates, key=lambda item: (item[0], item[1], item[2])
|
||||
)
|
||||
return anchor, selected
|
||||
|
||||
|
||||
def selection_record(selected: list[Any], *, duration_s: float) -> dict[str, Any]:
|
||||
return {
|
||||
"anchor": max(float(request.sampling_u) for request in selected),
|
||||
"selected_count": len(selected),
|
||||
"offered_req_s": len(selected) / duration_s,
|
||||
"offered_req_s_per_gpu": len(selected) / duration_s / TP,
|
||||
"request_id_order_sha256": order_hash([request.row_id for request in selected]),
|
||||
"arrival_order_sha256": order_hash(
|
||||
[f"{request.arrival_s:.12f}" for request in selected]
|
||||
),
|
||||
"input_length_order_sha256": order_hash(
|
||||
[str(request.prompt_tokens_hint) for request in selected]
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def materialize_study(source: Path, target: Path, *, replicate: int) -> Path:
|
||||
payload = json.loads(source.read_text(encoding="utf-8"))
|
||||
payload["study_id"] = f"phase-aware-telemetry-v2-rep{replicate}"
|
||||
payload["hardware"]["host_candidates"] = ["dash0"]
|
||||
payload["engine"]["engine_version"] = "0.24.1.dev3+opprof"
|
||||
trace = payload["trace"]
|
||||
trace["replay_time_scale"] = REPLAY_TIME_SCALE
|
||||
trace["early_stop_max_lag_s"] = None
|
||||
trace["early_stop_max_elapsed_s"] = SAFETY_DEADLINE_S
|
||||
trace["restart_engine_after_early_stop"] = False
|
||||
trace["adaptive_stop"] = {"enabled": False}
|
||||
atomic_json(target, payload)
|
||||
return target
|
||||
|
||||
|
||||
def build_manifest(
|
||||
*, base_manifest_path: Path, private_root: Path
|
||||
) -> dict[str, Any]:
|
||||
base = json.loads(base_manifest_path.read_text(encoding="utf-8"))
|
||||
if base.get("schema") != "fidelity-prefix-pilot-manifest-v1":
|
||||
raise ValueError("unexpected base P1 manifest schema")
|
||||
|
||||
repetitions = {}
|
||||
selection_hashes = []
|
||||
selected_ids_by_replicate: dict[int, set[str]] = {}
|
||||
for replicate, role in enumerate(REPLICATE_ROLES, start=1):
|
||||
source_study = Path(base["private"]["studies"][role]["tp4"])
|
||||
target_study = private_root / "studies" / f"rep{replicate}-tp4.json"
|
||||
materialize_study(source_study, target_study, replicate=replicate)
|
||||
study = load_study_spec(target_study)
|
||||
window, requests = load_trace_requests(study, study_spec_path=target_study)
|
||||
duration_s = float(window.window_end - window.window_start)
|
||||
if not math.isclose(
|
||||
duration_s, EXPECTED_DURATION_S, rel_tol=0.0, abs_tol=1e-9
|
||||
):
|
||||
raise ValueError(
|
||||
f"rep{replicate}: replay duration {duration_s} != {EXPECTED_DURATION_S}"
|
||||
)
|
||||
selections = {}
|
||||
selected_ids_by_level: dict[str, set[str]] = {}
|
||||
for level, target_rate in LOADS_REQ_S_GPU.items():
|
||||
target_count = round(target_rate * duration_s * TP)
|
||||
anchor, selected = attainable_anchor(requests, target_count)
|
||||
record = selection_record(selected, duration_s=duration_s)
|
||||
record.update(
|
||||
{
|
||||
"anchor": anchor,
|
||||
"target_count": target_count,
|
||||
"target_req_s_per_gpu": target_rate,
|
||||
}
|
||||
)
|
||||
selections[level] = record
|
||||
selected_ids_by_level[level] = {request.row_id for request in selected}
|
||||
selection_hashes.append(record["request_id_order_sha256"])
|
||||
selected_ids_by_replicate[replicate] = selected_ids_by_level["high"]
|
||||
repetitions[str(replicate)] = {
|
||||
"source_role": role,
|
||||
"study": str(target_study),
|
||||
"study_sha256": sha256_file(target_study),
|
||||
"duration_s": duration_s,
|
||||
"load_order": list(LOAD_ORDERS[replicate]),
|
||||
"selections": selections,
|
||||
}
|
||||
|
||||
burnin = base["cells"]["tp4_mns16"]["targets"]["low"]["selections"][
|
||||
"burnin"
|
||||
]
|
||||
burnin = dict(burnin)
|
||||
burnin["study_sha256"] = sha256_file(Path(burnin["study"]))
|
||||
sessions = [
|
||||
{
|
||||
"session": f"rep{replicate}-mns{mns}",
|
||||
"replicate": replicate,
|
||||
"mns": mns,
|
||||
}
|
||||
for replicate, mns in SESSION_ORDER
|
||||
]
|
||||
invariants = {
|
||||
"three_repetitions": len(repetitions) == 3,
|
||||
"six_sessions": len(sessions) == 6,
|
||||
"load_levels_three": all(
|
||||
len(item["selections"]) == 3 for item in repetitions.values()
|
||||
),
|
||||
"selection_hashes_unique": len(selection_hashes) == len(set(selection_hashes)),
|
||||
"selection_sets_disjoint_across_repetitions": all(
|
||||
not selected_ids_by_replicate[left] & selected_ids_by_replicate[right]
|
||||
for left in selected_ids_by_replicate
|
||||
for right in selected_ids_by_replicate
|
||||
if left < right
|
||||
),
|
||||
"all_counts_positive": all(
|
||||
selection["selected_count"] > 0
|
||||
for item in repetitions.values()
|
||||
for selection in item["selections"].values()
|
||||
),
|
||||
}
|
||||
red_flags = [name for name, passed in invariants.items() if not passed]
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"status": "PASS" if not red_flags else "STOP",
|
||||
"source": {
|
||||
"base_manifest": str(base_manifest_path),
|
||||
"base_manifest_sha256": sha256_file(base_manifest_path),
|
||||
"window_id": base["source"]["window_id"],
|
||||
"source_trace": base["source"]["trace"],
|
||||
"source_trace_sha256": base["source"]["trace_sha256"],
|
||||
},
|
||||
"engine": {
|
||||
"tp": TP,
|
||||
"mns_endpoints": list(MNS_ENDPOINTS),
|
||||
"replay_time_scale": REPLAY_TIME_SCALE,
|
||||
"duration_s": EXPECTED_DURATION_S,
|
||||
"safety_deadline_s": SAFETY_DEADLINE_S,
|
||||
"disable_slo_early_stop": True,
|
||||
},
|
||||
"burnin": burnin,
|
||||
"repetitions": repetitions,
|
||||
"sessions": sessions,
|
||||
"checkpoints": {
|
||||
"fractions": [0.1, 0.25, 0.5, 0.75, 1.0],
|
||||
"seconds": [30.0, 75.0, 150.0, 225.0, 300.0],
|
||||
},
|
||||
"budget": {
|
||||
"hard_cap_h20_hours": 8.0,
|
||||
"expected_wall_minutes": [95, 120],
|
||||
"expected_h20_hours": [6.3, 8.0],
|
||||
},
|
||||
"sanity": {
|
||||
"red_flags": red_flags,
|
||||
"invariants": invariants,
|
||||
"selected_sets": len(selection_hashes),
|
||||
"distinct_selected_sets": len(set(selection_hashes)),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--base-manifest", type=Path, required=True)
|
||||
parser.add_argument("--private-root", type=Path, required=True)
|
||||
parser.add_argument("--public-manifest", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
manifest = build_manifest(
|
||||
base_manifest_path=args.base_manifest, private_root=args.private_root
|
||||
)
|
||||
atomic_json(args.public_manifest, manifest)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": manifest["status"],
|
||||
"manifest": str(args.public_manifest),
|
||||
"sanity": manifest["sanity"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
if manifest["status"] != "PASS":
|
||||
raise RuntimeError(f"phase-aware pilot preflight failed: {manifest['sanity']}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
107
runs/intervention-response-v2/test_analysis.py
Normal file
107
runs/intervention-response-v2/test_analysis.py
Normal file
@@ -0,0 +1,107 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
import math
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def load_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_phase_aware_v2", HERE / "analyze_existing.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def load_prepare_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_phase_aware_prepare", HERE / "prepare_pilot.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def load_controller_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_phase_aware_controller", HERE / "pilot_controller.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def load_pilot_analysis_module():
|
||||
spec = importlib.util.spec_from_file_location(
|
||||
"intervention_response_phase_aware_pilot_analysis", HERE / "analyze_pilot.py"
|
||||
)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def main() -> None:
|
||||
module = load_module()
|
||||
assert module.common_decile_fractions(
|
||||
trace_duration_s=60.0, minimum_elapsed_s=19.448
|
||||
) == (0.1, 0.2, 0.3)
|
||||
assert module.common_decile_fractions(
|
||||
trace_duration_s=60.0, minimum_elapsed_s=60.0
|
||||
)[-1] == 1.0
|
||||
stats = module.numeric([0.0, 1.0, 2.0])
|
||||
assert stats == {
|
||||
"n": 3,
|
||||
"min": 0.0,
|
||||
"max": 2.0,
|
||||
"distinct_n": 3,
|
||||
"median": 1.0,
|
||||
}
|
||||
assert math.isclose(module._pearson([1.0, 2.0], [2.0, 4.0]), 1.0)
|
||||
assert module._pearson([1.0, 1.0], [2.0, 3.0]) is None
|
||||
prepare = load_prepare_module()
|
||||
requests = [
|
||||
SimpleNamespace(
|
||||
sampling_u=index / 10.0,
|
||||
row_id=f"r{index}",
|
||||
arrival_s=float(index),
|
||||
prompt_tokens_hint=100 + index,
|
||||
)
|
||||
for index in range(1, 6)
|
||||
]
|
||||
_anchor, selected = prepare.attainable_anchor(requests, 3)
|
||||
assert len(selected) == 3
|
||||
record = prepare.selection_record(selected, duration_s=3.0)
|
||||
assert record["selected_count"] == 3
|
||||
assert record["offered_req_s_per_gpu"] == 0.25
|
||||
assert len(prepare.SESSION_ORDER) == 6
|
||||
assert {mns for _replicate, mns in prepare.SESSION_ORDER} == {16, 64}
|
||||
controller = load_controller_module()
|
||||
assert math.isclose(controller.remaining_projection(6, 0), 7.7)
|
||||
assert math.isclose(controller.remaining_projection(6, 5), 1.45)
|
||||
pilot_analysis = load_pilot_analysis_module()
|
||||
stable = pilot_analysis.stable_adjacent_features(
|
||||
[
|
||||
{"end_fraction": 0.1, "qualifying_response_features": ["queue"]},
|
||||
{
|
||||
"end_fraction": 0.25,
|
||||
"qualifying_response_features": ["kv", "queue"],
|
||||
},
|
||||
{"end_fraction": 0.5, "qualifying_response_features": ["queue"]},
|
||||
]
|
||||
)
|
||||
assert stable == {"0.10->0.25": ["queue"], "0.25->0.50": ["queue"]}
|
||||
print("phase-aware intervention response v2 analysis: PASS")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -95,7 +95,11 @@ def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]:
|
||||
base_url=args.base_url,
|
||||
timeout_s=study.engine.request_timeout_s,
|
||||
max_concurrency=study.trace.max_concurrency,
|
||||
target_pass_rate=(0.0 if warmup else study.slo.target_pass_rate),
|
||||
target_pass_rate=(
|
||||
0.0
|
||||
if warmup or args.disable_slo_early_stop
|
||||
else study.slo.target_pass_rate
|
||||
),
|
||||
max_lag_s=study.trace.early_stop_max_lag_s,
|
||||
max_elapsed_s=(
|
||||
120.0 if warmup else _probe_drain_deadline(
|
||||
@@ -162,6 +166,7 @@ def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]:
|
||||
"feasible": bool(slo_summary["feasible"]),
|
||||
"early_stopped": early_stopped,
|
||||
"early_stop_reason": early_stop_reason,
|
||||
"slo_early_stop_disabled": bool(args.disable_slo_early_stop),
|
||||
"ttft_ms": numeric([item.ttft_ms for item in outcomes]),
|
||||
"tpot_ms": numeric([item.tpot_ms for item in outcomes]),
|
||||
"invariants": {
|
||||
@@ -240,6 +245,7 @@ def parser() -> argparse.ArgumentParser:
|
||||
q.add_argument("--mns", type=int, required=True)
|
||||
q.add_argument("--base-url", required=True)
|
||||
q.add_argument("--result-dir", required=True)
|
||||
q.add_argument("--disable-slo-early-stop", action="store_true")
|
||||
return p
|
||||
|
||||
|
||||
|
||||
@@ -20,6 +20,7 @@ def main() -> None:
|
||||
controller = load("p6controller", HERE / "opprof_phase6_controller.py")
|
||||
solo = load("p6solo", HERE / "opprof_phase6_solo_controller.py")
|
||||
analysis = load("p6analysis", HERE / "analyze_phase6.py")
|
||||
client = load("p6client", HERE / "opprof_phase6_client.py")
|
||||
assert len(controller.CELLS) == 12
|
||||
primary = sum(3 if item.get("trap") else 2 for item in controller.CELLS.values())
|
||||
assert primary == 25
|
||||
@@ -36,6 +37,18 @@ def main() -> None:
|
||||
assert sum(solo.CELL_ESTIMATE.values()) + solo.SAFETY_HOURS == 3.44
|
||||
assert solo.next_below([.1, .2, .3], {.2, .3}) == .1
|
||||
assert solo.next_above([.1, .2, .3], {.1, .2}) == .3
|
||||
parsed = client.parser().parse_args([
|
||||
"run-anchor",
|
||||
"--study", "study.json",
|
||||
"--cell", "tp4_mns16",
|
||||
"--anchor", "0.5",
|
||||
"--tp", "4",
|
||||
"--mns", "16",
|
||||
"--base-url", "http://127.0.0.1:8000",
|
||||
"--result-dir", "out",
|
||||
"--disable-slo-early-stop",
|
||||
])
|
||||
assert parsed.disable_slo_early_stop is True
|
||||
print("phase6 tools: PASS")
|
||||
|
||||
|
||||
|
||||
174
runs/telemetry-residual/analyze_p1_state.py
Normal file
174
runs/telemetry-residual/analyze_p1_state.py
Normal file
@@ -0,0 +1,174 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Pair P1 engine intervals with detailed Frontier state summaries."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
from common_state import load_jsonl, numeric, residual, summarize_engine # noqa: E402
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def adjudicated_labels(path: Path) -> dict[tuple[str, str], bool]:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
return {
|
||||
(row["cell"], row["level"]): bool(row["adjudicated_feasible"])
|
||||
for row in payload["examples"]
|
||||
}
|
||||
|
||||
|
||||
def state_runs(root: Path) -> list[Path]:
|
||||
candidates = list(root.glob("*/result.json"))
|
||||
if (root / "result.json").is_file():
|
||||
candidates.append(root / "result.json")
|
||||
return sorted(
|
||||
path
|
||||
for path in candidates
|
||||
if json.loads(path.read_text(encoding="utf-8")).get("schema")
|
||||
== "telemetry-residual-frontier-state-result-v1"
|
||||
)
|
||||
|
||||
|
||||
def execute(args: argparse.Namespace) -> dict[str, Any]:
|
||||
labels = adjudicated_labels(args.pilot_metrics)
|
||||
examples = []
|
||||
red_flags = []
|
||||
for state_result_path in state_runs(args.sim_state_root):
|
||||
run_root = state_result_path.parent
|
||||
manifest = json.loads((run_root / "run_manifest.json").read_text(encoding="utf-8"))
|
||||
cell = manifest["entry"]["cell"]
|
||||
role = manifest["entry"]["role"]
|
||||
level = "low" if role.startswith("low") else "high"
|
||||
real_run = args.real_root / "cells" / cell / f"{level}-rep1"
|
||||
real_result = json.loads((real_run / "result.json").read_text(encoding="utf-8"))
|
||||
stream_paths = sorted((args.real_root / "cells" / cell / "opprof").glob("*.jsonl"))
|
||||
if len(stream_paths) != 1:
|
||||
raise ValueError(f"expected one engine stream for {cell}, found {len(stream_paths)}")
|
||||
engine = summarize_engine(
|
||||
load_jsonl(stream_paths[0]),
|
||||
start_ns=int(real_result["interval"]["start_mono_ns"]),
|
||||
end_ns=int(real_result["interval"]["end_mono_ns"]),
|
||||
request_count=int(real_result["selection"]["count"]),
|
||||
)
|
||||
simulator = json.loads((run_root / "common-state.json").read_text(encoding="utf-8"))
|
||||
scorer = json.loads((run_root / "scorer_output.json").read_text(encoding="utf-8"))
|
||||
if engine["interval"]["request_count"] != simulator["interval"]["request_count"]:
|
||||
red_flags.append(f"request_count_mismatch:{cell}:{role}")
|
||||
difference = residual(engine, simulator)
|
||||
if any(not math.isfinite(float(value)) for value in difference["values"].values()):
|
||||
red_flags.append(f"nonfinite_residual:{cell}:{role}")
|
||||
real_feasible = labels[(cell, level)]
|
||||
sim_feasible = bool(scorer["slo"]["feasible"])
|
||||
examples.append(
|
||||
{
|
||||
"cell": cell,
|
||||
"role": role,
|
||||
"level": level,
|
||||
"tp": int(cell.split("_")[0][2:]),
|
||||
"mns": int(cell.split("_")[1][3:]),
|
||||
"request_count": engine["interval"]["request_count"],
|
||||
"real_feasible": real_feasible,
|
||||
"sim_feasible": sim_feasible,
|
||||
"simulator_error": sim_feasible != real_feasible,
|
||||
"simulator_false_feasible": sim_feasible and not real_feasible,
|
||||
"real_pass_rate_rep1": float(real_result["pass_rate"]),
|
||||
"offered_req_s": float(real_result["selection"]["offered_req_s"]),
|
||||
"offered_req_s_per_gpu": float(
|
||||
real_result["selection"]["offered_req_s_per_gpu"]
|
||||
),
|
||||
"sim_pass_rate": float(scorer["slo"]["pass_rate"]),
|
||||
"pass_rate_residual": float(real_result["pass_rate"])
|
||||
- float(scorer["slo"]["pass_rate"]),
|
||||
"engine": engine,
|
||||
"simulator": simulator,
|
||||
"state_residual": difference,
|
||||
"paths": {
|
||||
"real_result": str((real_run / "result.json").resolve()),
|
||||
"engine_stream": str(stream_paths[0].resolve()),
|
||||
"simulator_result": str(state_result_path.resolve()),
|
||||
},
|
||||
}
|
||||
)
|
||||
if not examples:
|
||||
red_flags.append("no_state_examples")
|
||||
request_counts = [int(row["request_count"]) for row in examples]
|
||||
pass_rate_residuals = [float(row["pass_rate_residual"]) for row in examples]
|
||||
result = {
|
||||
"schema": "telemetry-residual-p1-state-pairs-v1",
|
||||
"status": "PASS" if not red_flags else "STOP",
|
||||
"scope": "P1 development plumbing; not held-out contribution evidence",
|
||||
"examples": examples,
|
||||
"red_flags": red_flags,
|
||||
"sanity": {
|
||||
"n": len(examples),
|
||||
"request_count": numeric(request_counts) if request_counts else None,
|
||||
"pass_rate_residual": numeric(pass_rate_residuals)
|
||||
if pass_rate_residuals
|
||||
else None,
|
||||
"simulator_errors": sum(bool(row["simulator_error"]) for row in examples),
|
||||
"invariants": {
|
||||
"request_counts_match": not any(
|
||||
flag.startswith("request_count_mismatch") for flag in red_flags
|
||||
),
|
||||
"finite_residuals": not any(
|
||||
flag.startswith("nonfinite_residual") for flag in red_flags
|
||||
),
|
||||
"ratios_bounded": all(
|
||||
0.0 <= row["real_pass_rate_rep1"] <= 1.0
|
||||
and 0.0 <= row["sim_pass_rate"] <= 1.0
|
||||
for row in examples
|
||||
),
|
||||
"per_config_not_identical": len(set(pass_rate_residuals)) > 1
|
||||
if len(pass_rate_residuals) > 1
|
||||
else None,
|
||||
},
|
||||
},
|
||||
}
|
||||
atomic_json(args.output, result)
|
||||
if result["status"] != "PASS":
|
||||
raise RuntimeError(red_flags)
|
||||
return result
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
result = argparse.ArgumentParser()
|
||||
result.add_argument("--real-root", type=Path, required=True)
|
||||
result.add_argument("--sim-state-root", type=Path, required=True)
|
||||
result.add_argument("--pilot-metrics", type=Path, required=True)
|
||||
result.add_argument("--output", type=Path, required=True)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result = execute(parser().parse_args())
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": result["status"],
|
||||
"examples": len(result["examples"]),
|
||||
"simulator_errors": result["sanity"]["simulator_errors"],
|
||||
"sanity": result["sanity"],
|
||||
"red_flags": result["red_flags"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
292
runs/telemetry-residual/analyze_r0_gate.py
Normal file
292
runs/telemetry-residual/analyze_r0_gate.py
Normal file
@@ -0,0 +1,292 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Make the registered R0 go/no-go decision from development artifacts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def load_pass(path: Path, name: str) -> dict[str, Any]:
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
if payload.get("status") != "PASS":
|
||||
raise RuntimeError(f"{name} is not a valid PASS artifact: {path}")
|
||||
return payload
|
||||
|
||||
|
||||
def reduction(reference: float, candidate: float) -> float:
|
||||
if reference <= 0.0 or candidate < 0.0 or not math.isfinite(candidate):
|
||||
raise ValueError("costs must be finite and non-negative with positive reference")
|
||||
return 1.0 - candidate / reference
|
||||
|
||||
|
||||
def execute(args: argparse.Namespace) -> dict[str, Any]:
|
||||
paired = load_pass(args.paired_state, "paired state")
|
||||
transfer = load_pass(args.transfer, "transfer diagnostic")
|
||||
e2e = load_pass(args.pilot_e2e, "P1 E2E replay")
|
||||
red_flags = []
|
||||
|
||||
if len(paired.get("examples", [])) != 12:
|
||||
red_flags.append("paired_state_not_12_anchors")
|
||||
if transfer.get("sanity", {}).get("transitions") != 120:
|
||||
red_flags.append("transfer_not_120_cross_config_transitions")
|
||||
if transfer.get("red_flags"):
|
||||
red_flags.append("transfer_has_red_flags")
|
||||
|
||||
examples = paired["examples"]
|
||||
state_available = all(
|
||||
row["state_residual"]["coverage"]["missing"] == 0
|
||||
and row["state_residual"]["coverage"]["available"] > 0
|
||||
for row in examples
|
||||
)
|
||||
state_vectors = {
|
||||
tuple(sorted(row["state_residual"]["values"].items())) for row in examples
|
||||
}
|
||||
state_varies = len(state_vectors) > 1
|
||||
error_examples = [row for row in examples if row["simulator_error"]]
|
||||
simulator_errors = len(error_examples)
|
||||
error_state_discrepancy = any(
|
||||
any(
|
||||
abs(float(value)) > 1e-12
|
||||
for value in row["state_residual"]["values"].values()
|
||||
)
|
||||
for row in error_examples
|
||||
)
|
||||
|
||||
simulator = transfer["simulator"]
|
||||
prior_safe = []
|
||||
direct_sensitivity = []
|
||||
hybrid_incremental = []
|
||||
for regularization, detail in transfer["regularization_sensitivity"].items():
|
||||
for weight, models in detail["hybrid"]["prior_shrinkage"].items():
|
||||
if float(weight) == 0.0:
|
||||
continue
|
||||
telemetry = models["raw_simulator_prior"][
|
||||
"sim_plus_outcome_plus_telemetry"
|
||||
]
|
||||
decision_safe = (
|
||||
telemetry["simulator_errors_corrected"] >= 1
|
||||
and telemetry["simulator_errors_corrected"]
|
||||
>= telemetry["simulator_correct_corrupted"]
|
||||
)
|
||||
continuous_safe = (
|
||||
telemetry["rmse"] <= simulator["rmse"] + 1e-12
|
||||
and telemetry["mae"] <= simulator["mae"] + 1e-12
|
||||
)
|
||||
if decision_safe and continuous_safe:
|
||||
prior_safe.append(
|
||||
{
|
||||
"regularization": float(regularization),
|
||||
"prior_weight": float(weight),
|
||||
"simulator_errors_corrected": telemetry[
|
||||
"simulator_errors_corrected"
|
||||
],
|
||||
"simulator_correct_corrupted": telemetry[
|
||||
"simulator_correct_corrupted"
|
||||
],
|
||||
"rmse": telemetry["rmse"],
|
||||
"mae": telemetry["mae"],
|
||||
}
|
||||
)
|
||||
|
||||
direct = detail["direct"]
|
||||
direct_cmp = direct["comparison"]
|
||||
direct_sensitivity.append(
|
||||
{
|
||||
"regularization": float(regularization),
|
||||
"accuracy_delta": direct_cmp["delta_telemetry_minus_baseline"][
|
||||
"feasibility_accuracy"
|
||||
],
|
||||
"rmse_delta": direct_cmp["delta_telemetry_minus_baseline"]["rmse"],
|
||||
"mae_delta": direct_cmp["delta_telemetry_minus_baseline"]["mae"],
|
||||
"errors_corrected": direct_cmp["baseline_errors_corrected"],
|
||||
"correct_corrupted": direct_cmp["baseline_correct_corrupted"],
|
||||
"telemetry_accuracy": direct["telemetry_only"][
|
||||
"feasibility_accuracy"
|
||||
],
|
||||
}
|
||||
)
|
||||
hybrid_cmp = detail["hybrid"]["comparison"]
|
||||
hybrid_incremental.append(
|
||||
{
|
||||
"regularization": float(regularization),
|
||||
"accuracy_delta": hybrid_cmp["delta_telemetry_minus_baseline"][
|
||||
"feasibility_accuracy"
|
||||
],
|
||||
"rmse_delta": hybrid_cmp["delta_telemetry_minus_baseline"]["rmse"],
|
||||
"mae_delta": hybrid_cmp["delta_telemetry_minus_baseline"]["mae"],
|
||||
"errors_corrected": hybrid_cmp["baseline_errors_corrected"],
|
||||
"correct_corrupted": hybrid_cmp["baseline_correct_corrupted"],
|
||||
}
|
||||
)
|
||||
|
||||
k1 = e2e["by_k"]["1"]["sim_top_k_plus_real_final"]
|
||||
k2 = e2e["by_k"]["2"]["sim_top_k_plus_real_final"]
|
||||
headroom = {
|
||||
"interpretation": (
|
||||
"oracle correction stops after the simulator top-1 real final instead "
|
||||
"of evaluating the frozen safety top-2"
|
||||
),
|
||||
"online": {
|
||||
"reference_k2_h20_hours": k2["online_h20_hours"],
|
||||
"oracle_k1_h20_hours": k1["online_h20_hours"],
|
||||
"absolute_h20_hours": k2["online_h20_hours"] - k1["online_h20_hours"],
|
||||
"fraction": reduction(k2["online_h20_hours"], k1["online_h20_hours"]),
|
||||
},
|
||||
"with_prior_failure": {
|
||||
"reference_k2_h20_hours": k2["conservative_h20_hours_with_prior_failure"],
|
||||
"oracle_k1_h20_hours": k1["conservative_h20_hours_with_prior_failure"],
|
||||
"absolute_h20_hours": k2["conservative_h20_hours_with_prior_failure"]
|
||||
- k1["conservative_h20_hours_with_prior_failure"],
|
||||
"fraction": reduction(
|
||||
k2["conservative_h20_hours_with_prior_failure"],
|
||||
k1["conservative_h20_hours_with_prior_failure"],
|
||||
),
|
||||
},
|
||||
"versus_observed_safe_k1_fraction": 0.0,
|
||||
"k1_zero_regret": k1["real_regret"] == 0.0,
|
||||
"k2_zero_regret": k2["real_regret"] == 0.0,
|
||||
}
|
||||
|
||||
condition_1 = state_available and state_varies
|
||||
condition_2 = simulator_errors >= 1 and error_state_discrepancy
|
||||
condition_3 = bool(prior_safe)
|
||||
condition_4 = headroom["online"]["fraction"] >= 0.15
|
||||
conditions = {
|
||||
"state_available_and_varies": condition_1,
|
||||
"known_simulator_error_has_state_discrepancy": condition_2,
|
||||
"prior_preserving_safe_correction_exists": condition_3,
|
||||
"oracle_online_headroom_at_least_15pct": condition_4,
|
||||
}
|
||||
gate_pass = not red_flags and all(conditions.values())
|
||||
direct_incremental = all(
|
||||
row["accuracy_delta"] >= -1e-12
|
||||
and row["errors_corrected"] >= row["correct_corrupted"]
|
||||
for row in direct_sensitivity
|
||||
)
|
||||
result = {
|
||||
"schema": "telemetry-residual-r0-gate-v1",
|
||||
"status": "STOP" if red_flags else "PASS",
|
||||
"scope": "P1 development premise/headroom audit; not headline evidence",
|
||||
"decision": "PROCEED_TO_R1" if gate_pass else "STOP_BEFORE_R1",
|
||||
"r0_gate_pass": gate_pass,
|
||||
"conditions": conditions,
|
||||
"route_findings": {
|
||||
"hybrid_prior_safe_candidates": prior_safe,
|
||||
"hybrid_incremental_regularization": hybrid_incremental,
|
||||
"direct_incremental_regularization": direct_sensitivity,
|
||||
"direct_incremental_decision_signal_all_lambdas": direct_incremental,
|
||||
"direct_best_absolute_accuracy": max(
|
||||
row["telemetry_accuracy"] for row in direct_sensitivity
|
||||
),
|
||||
"raw_simulator_accuracy": simulator["feasibility_accuracy"],
|
||||
},
|
||||
"headroom": headroom,
|
||||
"red_flags": red_flags,
|
||||
"sanity": {
|
||||
"anchors": {
|
||||
"n": len(examples),
|
||||
"min": min(row["real_pass_rate_rep1"] for row in examples),
|
||||
"max": max(row["real_pass_rate_rep1"] for row in examples),
|
||||
"distinct_n": len(
|
||||
{row["real_pass_rate_rep1"] for row in examples}
|
||||
),
|
||||
},
|
||||
"state_vectors": {
|
||||
"n": len(examples),
|
||||
"min": min(len(row["state_residual"]["values"]) for row in examples),
|
||||
"max": max(len(row["state_residual"]["values"]) for row in examples),
|
||||
"distinct_n": len(state_vectors),
|
||||
},
|
||||
"costs_h20_hours": {
|
||||
"n": 4,
|
||||
"min": min(
|
||||
k1["online_h20_hours"],
|
||||
k2["online_h20_hours"],
|
||||
k1["conservative_h20_hours_with_prior_failure"],
|
||||
k2["conservative_h20_hours_with_prior_failure"],
|
||||
),
|
||||
"max": max(
|
||||
k1["online_h20_hours"],
|
||||
k2["online_h20_hours"],
|
||||
k1["conservative_h20_hours_with_prior_failure"],
|
||||
k2["conservative_h20_hours_with_prior_failure"],
|
||||
),
|
||||
"distinct_n": len(
|
||||
{
|
||||
k1["online_h20_hours"],
|
||||
k2["online_h20_hours"],
|
||||
k1["conservative_h20_hours_with_prior_failure"],
|
||||
k2["conservative_h20_hours_with_prior_failure"],
|
||||
}
|
||||
),
|
||||
},
|
||||
"invariants": {
|
||||
"no_data_red_flags": not red_flags,
|
||||
"state_nonempty_and_varied": condition_1,
|
||||
"pass_rates_bounded": all(
|
||||
0.0 <= row["real_pass_rate_rep1"] <= 1.0
|
||||
and 0.0 <= row["sim_pass_rate"] <= 1.0
|
||||
for row in examples
|
||||
),
|
||||
"costs_nonnegative": all(
|
||||
value >= 0.0
|
||||
for value in (
|
||||
k1["online_h20_hours"],
|
||||
k2["online_h20_hours"],
|
||||
k1["conservative_h20_hours_with_prior_failure"],
|
||||
k2["conservative_h20_hours_with_prior_failure"],
|
||||
)
|
||||
),
|
||||
"per_config_not_identical": len(
|
||||
{row["real_pass_rate_rep1"] for row in examples}
|
||||
)
|
||||
> 1,
|
||||
},
|
||||
},
|
||||
}
|
||||
atomic_json(args.output, result)
|
||||
if result["status"] != "PASS":
|
||||
raise RuntimeError(red_flags)
|
||||
return result
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
result = argparse.ArgumentParser()
|
||||
result.add_argument("--paired-state", type=Path, required=True)
|
||||
result.add_argument("--transfer", type=Path, required=True)
|
||||
result.add_argument("--pilot-e2e", type=Path, required=True)
|
||||
result.add_argument("--output", type=Path, required=True)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result = execute(parser().parse_args())
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": result["status"],
|
||||
"decision": result["decision"],
|
||||
"r0_gate_pass": result["r0_gate_pass"],
|
||||
"conditions": result["conditions"],
|
||||
"headroom": result["headroom"],
|
||||
"sanity": result["sanity"],
|
||||
"red_flags": result["red_flags"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
480
runs/telemetry-residual/analyze_residual_transfer.py
Normal file
480
runs/telemetry-residual/analyze_residual_transfer.py
Normal file
@@ -0,0 +1,480 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Development-only cross-config telemetry transfer diagnostic for P1.
|
||||
|
||||
Each example asks whether state observed at one source anchor helps predict the
|
||||
pass rate at a different target config. The hybrid branch predicts the
|
||||
real-minus-simulator residual; the direct branch never reads simulator state or
|
||||
outcomes. Folds exclude both the source and target config identities. The two
|
||||
offered-load anchors belong to one trace/SLO task, so this is a premise check
|
||||
rather than generalization evidence.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
from typing import Any, Sequence
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
REGULARIZATION = (0.1, 1.0, 10.0, 100.0)
|
||||
PRIOR_SHRINKAGE = (0.0, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0)
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def finite(value: Any, *, name: str) -> float:
|
||||
result = float(value)
|
||||
if not math.isfinite(result):
|
||||
raise ValueError(f"non-finite feature {name}={value!r}")
|
||||
return result
|
||||
|
||||
|
||||
def flatten_residual_state(example: dict[str, Any]) -> tuple[float, ...]:
|
||||
residual = example["state_residual"]["values"]
|
||||
values = [finite(residual[name], name=name) for name in sorted(residual)]
|
||||
engine_only = example["engine"]["engine_only"]
|
||||
values.extend(
|
||||
finite(engine_only[name], name=name)
|
||||
for name in sorted(engine_only)
|
||||
)
|
||||
return tuple(values)
|
||||
|
||||
|
||||
def flatten_engine_state(example: dict[str, Any]) -> tuple[float, ...]:
|
||||
values = []
|
||||
for name in sorted(example["engine"]["common"]):
|
||||
value = example["engine"]["common"][name]
|
||||
if isinstance(value, dict):
|
||||
values.extend(
|
||||
finite(value[statistic], name=f"{name}.{statistic}")
|
||||
for statistic in ("mean", "max", "cv")
|
||||
)
|
||||
elif value is not None:
|
||||
values.append(finite(value, name=name))
|
||||
engine_only = example["engine"]["engine_only"]
|
||||
values.extend(
|
||||
finite(engine_only[name], name=name)
|
||||
for name in sorted(engine_only)
|
||||
)
|
||||
return tuple(values)
|
||||
|
||||
|
||||
def config_transition_features(
|
||||
source: dict[str, Any], target: dict[str, Any]
|
||||
) -> tuple[float, ...]:
|
||||
source_rate = finite(
|
||||
source["offered_req_s_per_gpu"], name="source_offered_req_s_per_gpu"
|
||||
)
|
||||
target_rate = finite(
|
||||
target["offered_req_s_per_gpu"], name="target_offered_req_s_per_gpu"
|
||||
)
|
||||
return (
|
||||
math.log2(float(source["tp"])),
|
||||
math.log2(float(source["mns"])),
|
||||
math.log2(float(target["tp"])),
|
||||
math.log2(float(target["mns"])),
|
||||
math.log2(float(target["tp"]) / float(source["tp"])),
|
||||
math.log2(float(target["mns"]) / float(source["mns"])),
|
||||
math.log2(source_rate),
|
||||
math.log2(target_rate),
|
||||
math.log2(target_rate / source_rate),
|
||||
)
|
||||
|
||||
|
||||
def hybrid_base_features(
|
||||
source: dict[str, Any], target: dict[str, Any]
|
||||
) -> tuple[float, ...]:
|
||||
return (
|
||||
finite(source["pass_rate_residual"], name="source_pass_rate_residual"),
|
||||
finite(source["sim_pass_rate"], name="source_sim_pass_rate"),
|
||||
finite(target["sim_pass_rate"], name="target_sim_pass_rate"),
|
||||
) + config_transition_features(source, target)
|
||||
|
||||
|
||||
def direct_base_features(
|
||||
source: dict[str, Any], target: dict[str, Any]
|
||||
) -> tuple[float, ...]:
|
||||
return (
|
||||
finite(source["real_pass_rate_rep1"], name="source_real_pass_rate"),
|
||||
) + config_transition_features(source, target)
|
||||
|
||||
|
||||
def transitions(examples: Sequence[dict[str, Any]]) -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for source in examples:
|
||||
residual_state = flatten_residual_state(source)
|
||||
engine_state = flatten_engine_state(source)
|
||||
for target in examples:
|
||||
# Low/high are offered-load anchors inside one workload/SLO task.
|
||||
# Cross-load transitions are legal; same-cell transitions are
|
||||
# excluded because R0 asks about transfer to a new configuration.
|
||||
if source["cell"] == target["cell"]:
|
||||
continue
|
||||
hybrid_base = hybrid_base_features(source, target)
|
||||
direct_base = direct_base_features(source, target)
|
||||
rows.append(
|
||||
{
|
||||
"source_cell": source["cell"],
|
||||
"source_level": source["level"],
|
||||
"target_cell": target["cell"],
|
||||
"target_level": target["level"],
|
||||
"hybrid_base": hybrid_base,
|
||||
"hybrid_telemetry": hybrid_base + residual_state,
|
||||
"direct_base": direct_base,
|
||||
"direct_telemetry": direct_base + engine_state,
|
||||
"target_residual": finite(
|
||||
target["pass_rate_residual"], name="target_residual"
|
||||
),
|
||||
"target_residual_delta": finite(
|
||||
target["pass_rate_residual"], name="target_residual"
|
||||
)
|
||||
- finite(source["pass_rate_residual"], name="source_residual"),
|
||||
"target_real_pass_rate": finite(
|
||||
target["real_pass_rate_rep1"], name="target_real_pass_rate"
|
||||
),
|
||||
"target_real_pass_rate_delta": finite(
|
||||
target["real_pass_rate_rep1"], name="target_real_pass_rate"
|
||||
)
|
||||
- finite(
|
||||
source["real_pass_rate_rep1"], name="source_real_pass_rate"
|
||||
),
|
||||
"source_residual": finite(
|
||||
source["pass_rate_residual"], name="source_residual"
|
||||
),
|
||||
"source_real_pass_rate": finite(
|
||||
source["real_pass_rate_rep1"], name="source_real_pass_rate"
|
||||
),
|
||||
"target_sim_pass_rate": finite(
|
||||
target["sim_pass_rate"], name="target_sim_pass_rate"
|
||||
),
|
||||
"target_real_feasible": bool(target["real_feasible"]),
|
||||
"target_sim_feasible": bool(target["sim_feasible"]),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def fit_predict(
|
||||
train_x: np.ndarray,
|
||||
train_y: np.ndarray,
|
||||
test_x: np.ndarray,
|
||||
regularization: float,
|
||||
) -> np.ndarray:
|
||||
mean = train_x.mean(axis=0)
|
||||
std = train_x.std(axis=0)
|
||||
std[std == 0.0] = 1.0
|
||||
train = (train_x - mean) / std
|
||||
test = (test_x - mean) / std
|
||||
train = np.column_stack((np.ones(len(train)), train))
|
||||
test = np.column_stack((np.ones(len(test)), test))
|
||||
penalty = np.eye(train.shape[1], dtype=np.float64)
|
||||
penalty[0, 0] = 0.0
|
||||
weights = np.linalg.lstsq(
|
||||
train.T @ train + regularization * penalty,
|
||||
train.T @ train_y,
|
||||
rcond=None,
|
||||
)[0]
|
||||
return test @ weights
|
||||
|
||||
|
||||
def grouped_predictions(
|
||||
rows: Sequence[dict[str, Any]],
|
||||
*,
|
||||
feature_name: str,
|
||||
target_name: str,
|
||||
regularization: float,
|
||||
) -> np.ndarray:
|
||||
predictions = np.zeros(len(rows), dtype=np.float64)
|
||||
groups = sorted({(row["source_cell"], row["target_cell"]) for row in rows})
|
||||
for source_cell, target_cell in groups:
|
||||
held_out_cells = {source_cell, target_cell}
|
||||
test_indexes = [
|
||||
index
|
||||
for index, row in enumerate(rows)
|
||||
if row["source_cell"] == source_cell and row["target_cell"] == target_cell
|
||||
]
|
||||
train_indexes = [
|
||||
index
|
||||
for index, row in enumerate(rows)
|
||||
if row["source_cell"] not in held_out_cells
|
||||
and row["target_cell"] not in held_out_cells
|
||||
]
|
||||
if not test_indexes or not train_indexes:
|
||||
raise ValueError(f"empty grouped fold for {source_cell}->{target_cell}")
|
||||
train_x = np.asarray([rows[index][feature_name] for index in train_indexes])
|
||||
train_y = np.asarray([rows[index][target_name] for index in train_indexes])
|
||||
test_x = np.asarray([rows[index][feature_name] for index in test_indexes])
|
||||
predictions[test_indexes] = fit_predict(
|
||||
train_x, train_y, test_x, regularization
|
||||
)
|
||||
return predictions
|
||||
|
||||
|
||||
def metrics(rows: Sequence[dict[str, Any]], predicted_pass: np.ndarray) -> dict[str, Any]:
|
||||
truth = np.asarray(
|
||||
[row["target_real_pass_rate"] for row in rows], dtype=np.float64
|
||||
)
|
||||
real_feasible = np.asarray(
|
||||
[row["target_real_feasible"] for row in rows], dtype=bool
|
||||
)
|
||||
sim_feasible = np.asarray(
|
||||
[row["target_sim_feasible"] for row in rows], dtype=bool
|
||||
)
|
||||
clipped_pass = np.clip(predicted_pass, 0.0, 1.0)
|
||||
predicted_feasible = clipped_pass >= 0.95
|
||||
baseline_correct = sim_feasible == real_feasible
|
||||
model_correct = predicted_feasible == real_feasible
|
||||
return {
|
||||
"rmse": float(np.sqrt(np.mean((predicted_pass - truth) ** 2))),
|
||||
"mae": float(np.mean(np.abs(predicted_pass - truth))),
|
||||
"feasibility_accuracy": float(np.mean(model_correct)),
|
||||
"false_feasible": int(np.sum(predicted_feasible & ~real_feasible)),
|
||||
"false_infeasible": int(np.sum(~predicted_feasible & real_feasible)),
|
||||
"simulator_errors_corrected": int(np.sum(~baseline_correct & model_correct)),
|
||||
"simulator_correct_corrupted": int(np.sum(baseline_correct & ~model_correct)),
|
||||
"predicted_pass_rate": {
|
||||
"n": len(clipped_pass),
|
||||
"min": float(clipped_pass.min()),
|
||||
"max": float(clipped_pass.max()),
|
||||
"distinct_n": len(set(float(value) for value in clipped_pass)),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def compare(
|
||||
rows: Sequence[dict[str, Any]],
|
||||
baseline_prediction: np.ndarray,
|
||||
telemetry_prediction: np.ndarray,
|
||||
baseline: dict[str, Any],
|
||||
telemetry: dict[str, Any],
|
||||
) -> dict[str, Any]:
|
||||
truth = np.asarray([row["target_real_feasible"] for row in rows], dtype=bool)
|
||||
baseline_feasible = np.clip(baseline_prediction, 0.0, 1.0) >= 0.95
|
||||
telemetry_feasible = np.clip(telemetry_prediction, 0.0, 1.0) >= 0.95
|
||||
baseline_correct = baseline_feasible == truth
|
||||
telemetry_correct = telemetry_feasible == truth
|
||||
return {
|
||||
"delta_telemetry_minus_baseline": {
|
||||
"rmse": telemetry["rmse"] - baseline["rmse"],
|
||||
"mae": telemetry["mae"] - baseline["mae"],
|
||||
"feasibility_accuracy": telemetry["feasibility_accuracy"]
|
||||
- baseline["feasibility_accuracy"],
|
||||
},
|
||||
"baseline_errors_corrected": int(
|
||||
np.sum(~baseline_correct & telemetry_correct)
|
||||
),
|
||||
"baseline_correct_corrupted": int(
|
||||
np.sum(baseline_correct & ~telemetry_correct)
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
def execute(args: argparse.Namespace) -> dict[str, Any]:
|
||||
paired = json.loads(args.paired_state.read_text(encoding="utf-8"))
|
||||
if paired.get("status") != "PASS" or len(paired["examples"]) != 12:
|
||||
raise RuntimeError("paired P1 state evidence is incomplete")
|
||||
rows = transitions(paired["examples"])
|
||||
residual_truth = np.asarray(
|
||||
[row["target_residual"] for row in rows], dtype=np.float64
|
||||
)
|
||||
pass_truth = np.asarray(
|
||||
[row["target_real_pass_rate"] for row in rows], dtype=np.float64
|
||||
)
|
||||
sim_pass = np.asarray(
|
||||
[row["target_sim_pass_rate"] for row in rows], dtype=np.float64
|
||||
)
|
||||
simulator = metrics(rows, sim_pass)
|
||||
sensitivity = {}
|
||||
for regularization in REGULARIZATION:
|
||||
hybrid_base_delta = grouped_predictions(
|
||||
rows,
|
||||
feature_name="hybrid_base",
|
||||
target_name="target_residual_delta",
|
||||
regularization=regularization,
|
||||
)
|
||||
hybrid_telemetry_delta = grouped_predictions(
|
||||
rows,
|
||||
feature_name="hybrid_telemetry",
|
||||
target_name="target_residual_delta",
|
||||
regularization=regularization,
|
||||
)
|
||||
source_residual = np.asarray(
|
||||
[row["source_residual"] for row in rows], dtype=np.float64
|
||||
)
|
||||
hybrid_base_correction = source_residual + hybrid_base_delta
|
||||
hybrid_telemetry_correction = source_residual + hybrid_telemetry_delta
|
||||
hybrid_base_prediction = sim_pass + hybrid_base_correction
|
||||
hybrid_telemetry_prediction = sim_pass + hybrid_telemetry_correction
|
||||
hybrid_base = metrics(rows, hybrid_base_prediction)
|
||||
hybrid_telemetry = metrics(rows, hybrid_telemetry_prediction)
|
||||
prior_shrinkage = {}
|
||||
for weight in PRIOR_SHRINKAGE:
|
||||
prior_shrinkage[str(weight)] = {
|
||||
"raw_simulator_prior": {
|
||||
"sim_plus_outcome": metrics(
|
||||
rows, sim_pass + weight * hybrid_base_correction
|
||||
),
|
||||
"sim_plus_outcome_plus_telemetry": metrics(
|
||||
rows, sim_pass + weight * hybrid_telemetry_correction
|
||||
),
|
||||
},
|
||||
"anchor_offset_prior": {
|
||||
"sim_plus_outcome": metrics(
|
||||
rows,
|
||||
sim_pass + source_residual + weight * hybrid_base_delta,
|
||||
),
|
||||
"sim_plus_outcome_plus_telemetry": metrics(
|
||||
rows,
|
||||
sim_pass + source_residual + weight * hybrid_telemetry_delta,
|
||||
),
|
||||
},
|
||||
}
|
||||
|
||||
direct_base_prediction = grouped_predictions(
|
||||
rows,
|
||||
feature_name="direct_base",
|
||||
target_name="target_real_pass_rate_delta",
|
||||
regularization=regularization,
|
||||
)
|
||||
direct_telemetry_prediction = grouped_predictions(
|
||||
rows,
|
||||
feature_name="direct_telemetry",
|
||||
target_name="target_real_pass_rate_delta",
|
||||
regularization=regularization,
|
||||
)
|
||||
source_real_pass = np.asarray(
|
||||
[row["source_real_pass_rate"] for row in rows], dtype=np.float64
|
||||
)
|
||||
direct_base_prediction = source_real_pass + direct_base_prediction
|
||||
direct_telemetry_prediction = source_real_pass + direct_telemetry_prediction
|
||||
direct_base = metrics(rows, direct_base_prediction)
|
||||
direct_telemetry = metrics(rows, direct_telemetry_prediction)
|
||||
sensitivity[str(regularization)] = {
|
||||
"hybrid": {
|
||||
"sim_plus_outcome": hybrid_base,
|
||||
"sim_plus_outcome_plus_telemetry": hybrid_telemetry,
|
||||
"comparison": compare(
|
||||
rows,
|
||||
hybrid_base_prediction,
|
||||
hybrid_telemetry_prediction,
|
||||
hybrid_base,
|
||||
hybrid_telemetry,
|
||||
),
|
||||
"prior_shrinkage": prior_shrinkage,
|
||||
},
|
||||
"direct": {
|
||||
"real_outcome_only": direct_base,
|
||||
"telemetry_only": direct_telemetry,
|
||||
"comparison": compare(
|
||||
rows,
|
||||
direct_base_prediction,
|
||||
direct_telemetry_prediction,
|
||||
direct_base,
|
||||
direct_telemetry,
|
||||
),
|
||||
},
|
||||
}
|
||||
red_flags = []
|
||||
if any(not math.isfinite(value) for value in residual_truth):
|
||||
red_flags.append("nonfinite_residual_target")
|
||||
if any(not math.isfinite(value) for value in pass_truth):
|
||||
red_flags.append("nonfinite_pass_rate_target")
|
||||
result = {
|
||||
"schema": "telemetry-residual-cross-config-diagnostic-v1",
|
||||
"status": "PASS" if not red_flags else "STOP",
|
||||
"scope": (
|
||||
"single P1 trace/SLO-task development diagnostic; ordered transitions "
|
||||
"are not independent tasks and cannot support a generalization claim"
|
||||
),
|
||||
"split": (
|
||||
"hold out both ordered source config and target config identities; "
|
||||
"each fold contains both offered-load anchors"
|
||||
),
|
||||
"features": {
|
||||
"sim_plus_outcome": len(rows[0]["hybrid_base"]),
|
||||
"sim_plus_outcome_plus_telemetry": len(rows[0]["hybrid_telemetry"]),
|
||||
"real_outcome_only": len(rows[0]["direct_base"]),
|
||||
"telemetry_only": len(rows[0]["direct_telemetry"]),
|
||||
},
|
||||
"simulator": simulator,
|
||||
"regularization_sensitivity": sensitivity,
|
||||
"red_flags": red_flags,
|
||||
"sanity": {
|
||||
"transitions": len(rows),
|
||||
"load_levels": len(
|
||||
{row["source_level"] for row in rows}
|
||||
| {row["target_level"] for row in rows}
|
||||
),
|
||||
"cross_load_transitions": sum(
|
||||
row["source_level"] != row["target_level"] for row in rows
|
||||
),
|
||||
"ordered_cell_pairs": len(
|
||||
{(row["source_cell"], row["target_cell"]) for row in rows}
|
||||
),
|
||||
"target_residual": {
|
||||
"n": len(residual_truth),
|
||||
"min": float(residual_truth.min()),
|
||||
"max": float(residual_truth.max()),
|
||||
"distinct_n": len(set(float(value) for value in residual_truth)),
|
||||
},
|
||||
"target_real_pass_rate": {
|
||||
"n": len(pass_truth),
|
||||
"min": float(pass_truth.min()),
|
||||
"max": float(pass_truth.max()),
|
||||
"distinct_n": len(set(float(value) for value in pass_truth)),
|
||||
},
|
||||
"invariants": {
|
||||
"finite_targets": not red_flags,
|
||||
"ratios_bounded": all(
|
||||
0.0 <= row["target_sim_pass_rate"] <= 1.0 for row in rows
|
||||
),
|
||||
"source_differs_from_target": all(
|
||||
row["source_cell"] != row["target_cell"] for row in rows
|
||||
),
|
||||
"per_config_not_identical": len(
|
||||
set(float(value) for value in pass_truth)
|
||||
)
|
||||
> 1,
|
||||
},
|
||||
},
|
||||
}
|
||||
atomic_json(args.output, result)
|
||||
if result["status"] != "PASS":
|
||||
raise RuntimeError(red_flags)
|
||||
return result
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
result = argparse.ArgumentParser()
|
||||
result.add_argument("--paired-state", type=Path, required=True)
|
||||
result.add_argument("--output", type=Path, required=True)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result = execute(parser().parse_args())
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": result["status"],
|
||||
"transitions": result["sanity"]["transitions"],
|
||||
"sensitivity": result["regularization_sensitivity"],
|
||||
"sanity": result["sanity"],
|
||||
"red_flags": result["red_flags"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
400
runs/telemetry-residual/common_state.py
Normal file
400
runs/telemetry-residual/common_state.py
Normal file
@@ -0,0 +1,400 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Extract directly measurable engine/simulator state into one schema.
|
||||
|
||||
The schema deliberately keeps common, engine-only, and simulator-only fields
|
||||
separate. Missing simulator mechanisms must remain missing; they must not be
|
||||
filled by a heuristic bottleneck label.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
from statistics import fmean
|
||||
from typing import Any, Iterable, Mapping, Sequence
|
||||
|
||||
|
||||
SCHEMA = "telemetry-common-state-v1"
|
||||
|
||||
|
||||
def numeric(values: Iterable[float | int]) -> dict[str, Any]:
|
||||
finite = [float(value) for value in values]
|
||||
if not finite:
|
||||
raise ValueError("numeric summary requires at least one value")
|
||||
if any(not math.isfinite(value) for value in finite):
|
||||
raise ValueError("numeric summary received a non-finite value")
|
||||
mean = fmean(finite)
|
||||
variance = fmean((value - mean) ** 2 for value in finite)
|
||||
return {
|
||||
"n": len(finite),
|
||||
"min": min(finite),
|
||||
"max": max(finite),
|
||||
"mean": mean,
|
||||
"cv": math.sqrt(variance) / abs(mean) if mean else 0.0,
|
||||
"distinct_n": len(set(finite)),
|
||||
}
|
||||
|
||||
|
||||
def load_jsonl(path: Path) -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
with path.open(encoding="utf-8") as source:
|
||||
for line_number, line in enumerate(source, start=1):
|
||||
if not line.strip():
|
||||
continue
|
||||
row = json.loads(line)
|
||||
if not isinstance(row, dict):
|
||||
raise ValueError(f"{path}:{line_number}: expected a JSON object")
|
||||
rows.append(row)
|
||||
if not rows:
|
||||
raise ValueError(f"{path}: no JSONL records")
|
||||
return rows
|
||||
|
||||
|
||||
def _as_number(value: Any, *, name: str) -> float:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
raise ValueError(f"{name} must be numeric, got {value!r}")
|
||||
result = float(value)
|
||||
if not math.isfinite(result):
|
||||
raise ValueError(f"{name} must be finite, got {value!r}")
|
||||
return result
|
||||
|
||||
|
||||
def _time_weighted_mean(
|
||||
records: Sequence[Mapping[str, Any]],
|
||||
*,
|
||||
start_ns: int,
|
||||
end_ns: int,
|
||||
value,
|
||||
) -> float:
|
||||
if end_ns <= start_ns:
|
||||
raise ValueError("time-weighted interval must be positive")
|
||||
selected = [
|
||||
record
|
||||
for record in records
|
||||
if start_ns <= int(record["submit_mono_ns"]) <= end_ns
|
||||
]
|
||||
if not selected:
|
||||
raise ValueError("time-weighted interval contains no records")
|
||||
selected.sort(key=lambda record: int(record["submit_mono_ns"]))
|
||||
cursor = start_ns
|
||||
total = 0.0
|
||||
current = _as_number(value(selected[0]), name="time-weighted value")
|
||||
for record in selected[1:]:
|
||||
timestamp = int(record["submit_mono_ns"])
|
||||
if timestamp < cursor:
|
||||
raise ValueError("telemetry timestamps are not monotonic")
|
||||
total += current * (timestamp - cursor)
|
||||
cursor = timestamp
|
||||
current = _as_number(value(record), name="time-weighted value")
|
||||
total += current * (end_ns - cursor)
|
||||
return total / (end_ns - start_ns)
|
||||
|
||||
|
||||
def summarize_engine(
|
||||
records: Sequence[Mapping[str, Any]],
|
||||
*,
|
||||
start_ns: int,
|
||||
end_ns: int,
|
||||
request_count: int,
|
||||
) -> dict[str, Any]:
|
||||
"""Summarize a measured engine interval from Layer-1 records."""
|
||||
|
||||
if request_count <= 0:
|
||||
raise ValueError("request_count must be positive")
|
||||
layer1 = [record for record in records if "step_index" in record]
|
||||
if not layer1:
|
||||
raise ValueError("engine stream has no Layer-1 records")
|
||||
step_indexes = [int(record["step_index"]) for record in layer1]
|
||||
if len(step_indexes) != len(set(step_indexes)):
|
||||
raise ValueError("engine Layer-1 step indexes are not unique")
|
||||
if step_indexes != sorted(step_indexes):
|
||||
raise ValueError("engine Layer-1 step indexes are not ordered")
|
||||
if any(int(record.get("dropped_records_before", 0)) != 0 for record in layer1):
|
||||
raise ValueError("engine Layer-1 stream reports dropped records")
|
||||
|
||||
interval = [
|
||||
record
|
||||
for record in layer1
|
||||
if start_ns <= int(record["submit_mono_ns"]) <= end_ns
|
||||
]
|
||||
if not interval:
|
||||
raise ValueError("engine interval has no Layer-1 records")
|
||||
executed = [record for record in interval if bool(record["model_executed"])]
|
||||
if not executed:
|
||||
raise ValueError("engine interval has no executed model steps")
|
||||
duration_s = (end_ns - start_ns) / 1e9
|
||||
if duration_s <= 0:
|
||||
raise ValueError("engine interval duration must be positive")
|
||||
|
||||
batch_sizes = [int(record["scheduled_requests"]) for record in executed]
|
||||
prefill_tokens = [int(record["prefill_tokens"]) for record in executed]
|
||||
decode_tokens = [int(record["decode_tokens"]) for record in executed]
|
||||
batch_tokens = [
|
||||
prefill + decode
|
||||
for prefill, decode in zip(prefill_tokens, decode_tokens, strict=True)
|
||||
]
|
||||
decode_batches = [int(record["decode_batch_size"]) for record in executed]
|
||||
if any(value < 0 for value in batch_sizes + batch_tokens + decode_batches):
|
||||
raise ValueError("engine batch counters must be non-negative")
|
||||
if any(
|
||||
int(record["prefill_tokens"]) + int(record["decode_tokens"])
|
||||
<= 0
|
||||
for record in executed
|
||||
):
|
||||
raise ValueError("executed engine step has no scheduled tokens")
|
||||
|
||||
waiting_mean = _time_weighted_mean(
|
||||
interval,
|
||||
start_ns=start_ns,
|
||||
end_ns=end_ns,
|
||||
value=lambda record: record["queues"]["waiting"],
|
||||
)
|
||||
running_mean = _time_weighted_mean(
|
||||
interval,
|
||||
start_ns=start_ns,
|
||||
end_ns=end_ns,
|
||||
value=lambda record: record["queues"]["running"],
|
||||
)
|
||||
kv_mean = _time_weighted_mean(
|
||||
interval,
|
||||
start_ns=start_ns,
|
||||
end_ns=end_ns,
|
||||
value=lambda record: record["kv"]["usage"],
|
||||
)
|
||||
kv_values = [float(record["kv"]["usage"]) for record in interval]
|
||||
if any(not 0.0 <= value <= 1.0 for value in kv_values):
|
||||
raise ValueError("engine KV usage must be in [0, 1]")
|
||||
total_prefill = sum(prefill_tokens)
|
||||
total_decode = sum(decode_tokens)
|
||||
graph_modes = [str(record["cudagraph"]["runtime_mode"]) for record in executed]
|
||||
bucket_tokens = sum(int(record["cudagraph"]["bucket_tokens"]) for record in executed)
|
||||
padding_tokens = sum(int(record["cudagraph"]["padding_tokens"]) for record in executed)
|
||||
|
||||
common = {
|
||||
"scheduler_steps_per_s": len(executed) / duration_s,
|
||||
"batch_size": numeric(batch_sizes),
|
||||
"batch_tokens": numeric(batch_tokens),
|
||||
"decode_batch_size": numeric(decode_batches),
|
||||
"prefill_token_fraction": total_prefill / (total_prefill + total_decode),
|
||||
"queue_waiting_mean": waiting_mean,
|
||||
"queue_running_mean": running_mean,
|
||||
"queue_waiting_time_per_request_ms": waiting_mean * duration_s * 1000.0 / request_count,
|
||||
"queue_running_time_per_request_ms": running_mean * duration_s * 1000.0 / request_count,
|
||||
"preemptions": sum(int(record["preemptions"]) for record in executed),
|
||||
}
|
||||
result = {
|
||||
"schema": SCHEMA,
|
||||
"source": "engine_layer1",
|
||||
"interval": {
|
||||
"start_ns": start_ns,
|
||||
"end_ns": end_ns,
|
||||
"duration_s": duration_s,
|
||||
"request_count": request_count,
|
||||
},
|
||||
"common": common,
|
||||
"engine_only": {
|
||||
"kv_usage_mean": kv_mean,
|
||||
"kv_usage_max": max(kv_values),
|
||||
"kv_usage_end_minus_start": kv_values[-1] - kv_values[0],
|
||||
"graph_none_share": graph_modes.count("NONE") / len(graph_modes),
|
||||
"graph_full_share": graph_modes.count("FULL") / len(graph_modes),
|
||||
"graph_padding_fraction": padding_tokens / max(1, bucket_tokens),
|
||||
},
|
||||
"simulator_only": {},
|
||||
"sanity": {
|
||||
"records": len(interval),
|
||||
"executed_steps": len(executed),
|
||||
"step_index_min": min(int(record["step_index"]) for record in interval),
|
||||
"step_index_max": max(int(record["step_index"]) for record in interval),
|
||||
"invariants": {
|
||||
"positive_duration": duration_s > 0,
|
||||
"positive_request_count": request_count > 0,
|
||||
"zero_drops": True,
|
||||
"nonnegative_counters": True,
|
||||
"kv_bounded": True,
|
||||
"batch_values_not_all_identical": any(
|
||||
summary["distinct_n"] > 1
|
||||
for summary in (
|
||||
common["batch_size"],
|
||||
common["batch_tokens"],
|
||||
common["decode_batch_size"],
|
||||
)
|
||||
),
|
||||
},
|
||||
},
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def _csv_rows(path: Path) -> list[dict[str, str]]:
|
||||
with path.open(encoding="utf-8", newline="") as source:
|
||||
rows = list(csv.DictReader(source))
|
||||
if not rows:
|
||||
raise ValueError(f"{path}: CSV contains no rows")
|
||||
return rows
|
||||
|
||||
|
||||
def _column(rows: Sequence[Mapping[str, str]], name: str) -> list[float]:
|
||||
if name not in rows[0]:
|
||||
raise ValueError(f"CSV is missing required column {name!r}")
|
||||
values = []
|
||||
for row in rows:
|
||||
text = row.get(name, "")
|
||||
if text == "":
|
||||
raise ValueError(f"CSV column {name!r} contains an empty value")
|
||||
values.append(_as_number(float(text), name=name))
|
||||
return values
|
||||
|
||||
|
||||
def summarize_frontier(
|
||||
*,
|
||||
system_metrics_path: Path,
|
||||
request_metrics_path: Path,
|
||||
batch_metrics_path: Path | None = None,
|
||||
ledger_path: Path | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Summarize a Frontier run, retaining unavailable state as null."""
|
||||
|
||||
system = json.loads(system_metrics_path.read_text(encoding="utf-8"))
|
||||
throughput = system["throughput_metrics"]
|
||||
duration_s = _as_number(
|
||||
throughput["total_duration_seconds"], name="total_duration_seconds"
|
||||
)
|
||||
if duration_s <= 0:
|
||||
raise ValueError("Frontier duration must be positive")
|
||||
request_rows = _csv_rows(request_metrics_path)
|
||||
waiting_ms = _column(request_rows, "request_waiting_time_total")
|
||||
e2e_ms = _column(request_rows, "request_e2e_time")
|
||||
running_ms = [max(0.0, e2e - waiting) for e2e, waiting in zip(e2e_ms, waiting_ms, strict=True)]
|
||||
duration_ms = duration_s * 1000.0
|
||||
request_count = len(request_rows)
|
||||
|
||||
common: dict[str, Any] = {
|
||||
"scheduler_steps_per_s": None,
|
||||
"batch_size": None,
|
||||
"batch_tokens": None,
|
||||
"decode_batch_size": None,
|
||||
"prefill_token_fraction": None,
|
||||
"queue_waiting_mean": sum(waiting_ms) / duration_ms,
|
||||
"queue_running_mean": sum(running_ms) / duration_ms,
|
||||
"queue_waiting_time_per_request_ms": fmean(waiting_ms),
|
||||
"queue_running_time_per_request_ms": fmean(running_ms),
|
||||
"preemptions": sum(
|
||||
int(float(row.get("request_total_preemption_count") or 0))
|
||||
for row in request_rows
|
||||
),
|
||||
}
|
||||
batch_rows: list[dict[str, str]] = []
|
||||
if batch_metrics_path is not None:
|
||||
batch_rows = _csv_rows(batch_metrics_path)
|
||||
batch_sizes = _column(batch_rows, "batch_size")
|
||||
batch_tokens = _column(batch_rows, "batch_num_tokens")
|
||||
prefill_tokens = _column(batch_rows, "batch_num_prefill_tokens")
|
||||
decode_tokens = _column(batch_rows, "batch_num_decode_tokens")
|
||||
if any(value < 0 for value in batch_sizes + batch_tokens + prefill_tokens + decode_tokens):
|
||||
raise ValueError("Frontier batch counters must be non-negative")
|
||||
common.update(
|
||||
{
|
||||
"scheduler_steps_per_s": len(batch_rows) / duration_s,
|
||||
"batch_size": numeric(batch_sizes),
|
||||
"batch_tokens": numeric(batch_tokens),
|
||||
"decode_batch_size": numeric(decode_tokens),
|
||||
"prefill_token_fraction": sum(prefill_tokens)
|
||||
/ max(1.0, sum(prefill_tokens) + sum(decode_tokens)),
|
||||
}
|
||||
)
|
||||
|
||||
ledger_rows: list[dict[str, Any]] = []
|
||||
if ledger_path is not None:
|
||||
ledger_rows = load_jsonl(ledger_path)
|
||||
for row in ledger_rows:
|
||||
start = _as_number(row["stage_start_ts"], name="stage_start_ts")
|
||||
end = _as_number(row["stage_end_ts"], name="stage_end_ts")
|
||||
if end < start:
|
||||
raise ValueError("Frontier ledger has a negative stage duration")
|
||||
|
||||
batch_distinct = (
|
||||
max(
|
||||
summary["distinct_n"]
|
||||
for summary in (
|
||||
common["batch_size"],
|
||||
common["batch_tokens"],
|
||||
common["decode_batch_size"],
|
||||
)
|
||||
)
|
||||
if batch_rows
|
||||
else None
|
||||
)
|
||||
return {
|
||||
"schema": SCHEMA,
|
||||
"source": "frontier",
|
||||
"interval": {
|
||||
"duration_s": duration_s,
|
||||
"request_count": request_count,
|
||||
},
|
||||
"common": common,
|
||||
"engine_only": {
|
||||
"kv_usage_mean": None,
|
||||
"kv_usage_max": None,
|
||||
"kv_usage_end_minus_start": None,
|
||||
"graph_none_share": None,
|
||||
"graph_full_share": None,
|
||||
"graph_padding_fraction": None,
|
||||
},
|
||||
"simulator_only": {
|
||||
"request_waiting_time_ms": numeric(waiting_ms),
|
||||
"request_running_time_ms": numeric(running_ms),
|
||||
"ledger_rows": len(ledger_rows) if ledger_path is not None else None,
|
||||
},
|
||||
"sanity": {
|
||||
"request_rows": request_count,
|
||||
"batch_rows": len(batch_rows),
|
||||
"ledger_rows": len(ledger_rows),
|
||||
"invariants": {
|
||||
"positive_duration": duration_s > 0,
|
||||
"positive_request_count": request_count > 0,
|
||||
"nonnegative_counters": True,
|
||||
"request_values_not_all_identical": max(
|
||||
numeric(waiting_ms)["distinct_n"],
|
||||
numeric(running_ms)["distinct_n"],
|
||||
)
|
||||
> 1,
|
||||
"batch_values_not_all_identical": (
|
||||
batch_distinct > 1 if batch_distinct is not None else None
|
||||
),
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def residual(real: Mapping[str, Any], simulated: Mapping[str, Any]) -> dict[str, Any]:
|
||||
if real.get("schema") != SCHEMA or simulated.get("schema") != SCHEMA:
|
||||
raise ValueError("residual inputs must use the common-state schema")
|
||||
values = {}
|
||||
missing = []
|
||||
for name, real_value in real["common"].items():
|
||||
sim_value = simulated["common"].get(name)
|
||||
if isinstance(real_value, dict):
|
||||
if not isinstance(sim_value, dict):
|
||||
missing.append(name)
|
||||
continue
|
||||
for statistic in ("mean", "max", "cv"):
|
||||
key = f"{name}.{statistic}"
|
||||
values[key] = float(real_value[statistic]) - float(sim_value[statistic])
|
||||
continue
|
||||
if real_value is None or sim_value is None:
|
||||
missing.append(name)
|
||||
continue
|
||||
values[name] = float(real_value) - float(sim_value)
|
||||
return {
|
||||
"schema": "telemetry-state-residual-v1",
|
||||
"values": values,
|
||||
"missing_common_fields": sorted(missing),
|
||||
"coverage": {
|
||||
"available": len(values),
|
||||
"missing": len(missing),
|
||||
"common_field_count": len(real["common"]),
|
||||
},
|
||||
}
|
||||
316
runs/telemetry-residual/run_frontier_state.py
Normal file
316
runs/telemetry-residual/run_frontier_state.py
Normal file
@@ -0,0 +1,316 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Replay frozen Frontier fixtures with state-detail outputs enabled.
|
||||
|
||||
This runner does not modify the frozen fixture/config inputs. It reuses the
|
||||
audited SimFid command and only turns on existing Frontier batch/ledger output
|
||||
flags in a separate result root.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
AITUNER_ROOT = HERE.parents[1]
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
from common_state import numeric, summarize_frontier # noqa: E402, F401
|
||||
|
||||
|
||||
def load_module(name: str, path: Path):
|
||||
module_root = str(path.parent.resolve())
|
||||
if module_root not in sys.path:
|
||||
sys.path.insert(0, module_root)
|
||||
spec = importlib.util.spec_from_file_location(name, path)
|
||||
if spec is None or spec.loader is None:
|
||||
raise ImportError(path)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
sys.modules[spec.name] = module
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
def sha256_file(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def git_capture(root: Path, *arguments: str) -> str:
|
||||
return subprocess.run(
|
||||
["git", "-C", str(root), *arguments],
|
||||
check=True,
|
||||
text=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
).stdout
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n")
|
||||
os.replace(temporary, path)
|
||||
|
||||
|
||||
def enable_state_outputs(command: list[str]) -> list[str]:
|
||||
result = list(command)
|
||||
disabled_ledger = "--no-metrics_config_store_frontier_stage_batch_ledger"
|
||||
enabled_ledger = "--metrics_config_store_frontier_stage_batch_ledger"
|
||||
enabled_batches = "--metrics_config_keep_individual_batch_metrics"
|
||||
disabled_batches = "--no-metrics_config_keep_individual_batch_metrics"
|
||||
if result.count(disabled_ledger) != 1 or enabled_ledger in result:
|
||||
raise ValueError("base command must explicitly disable one full Frontier ledger")
|
||||
result[result.index(disabled_ledger)] = enabled_ledger
|
||||
if disabled_batches in result:
|
||||
if result.count(disabled_batches) != 1:
|
||||
raise ValueError("duplicate disabled individual-batch flag")
|
||||
result[result.index(disabled_batches)] = enabled_batches
|
||||
elif enabled_batches not in result:
|
||||
result.append(enabled_batches)
|
||||
if result.count(enabled_ledger) != 1 or result.count(enabled_batches) != 1:
|
||||
raise ValueError("state-detail flags were not enabled exactly once")
|
||||
return result
|
||||
|
||||
|
||||
def find_state_metrics(run_root: Path) -> dict[str, Path]:
|
||||
patterns = {
|
||||
"system": "frontier_metrics/**/system_metrics.json",
|
||||
"requests": "frontier_metrics/**/request_metrics.csv",
|
||||
"batches": "frontier_metrics/**/monolithic_batch_metrics.csv",
|
||||
"ledger": "frontier_metrics/**/frontier_stage_batch_ledger.jsonl",
|
||||
}
|
||||
result = {}
|
||||
for name, pattern in patterns.items():
|
||||
matches = sorted(run_root.glob(pattern))
|
||||
if len(matches) != 1:
|
||||
raise ValueError(
|
||||
f"{run_root}: expected exactly one {name} artifact, found {len(matches)}"
|
||||
)
|
||||
result[name] = matches[0]
|
||||
if len({path.parent for path in result.values()}) != 1:
|
||||
raise ValueError("Frontier state artifacts are not colocated")
|
||||
return result
|
||||
|
||||
|
||||
def execute(args: argparse.Namespace) -> dict[str, Any]:
|
||||
prepared = json.loads(args.prepared_manifest.read_text(encoding="utf-8"))
|
||||
if prepared.get("status") != "PASS":
|
||||
raise RuntimeError("prepared simulator manifest did not pass")
|
||||
matches = [
|
||||
entry
|
||||
for entry in prepared["entries"]
|
||||
if entry["cell"] == args.cell and entry["role"] == args.role
|
||||
]
|
||||
if len(matches) != 1:
|
||||
raise ValueError(
|
||||
f"expected one prepared entry for {args.cell}/{args.role}, found {len(matches)}"
|
||||
)
|
||||
entry = matches[0]
|
||||
if args.output.exists() and any(args.output.iterdir()):
|
||||
raise FileExistsError(f"non-empty output already exists: {args.output}")
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
driver = load_module(
|
||||
"telemetry_residual_execution_driver",
|
||||
args.replayserve_root / "runs/simfid_s2rb/results/execution_driver.py",
|
||||
)
|
||||
config_path = Path(entry["config"])
|
||||
config = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
fixture_manifest_path = Path(entry["fixture_manifest"])
|
||||
fixture = json.loads(fixture_manifest_path.read_text(encoding="utf-8"))
|
||||
trace_path = Path(entry["frontier_csv"])
|
||||
sidecar_path = Path(entry["sidecar"])
|
||||
metrics_root = args.output / "frontier_metrics"
|
||||
run_id = f"telemetry_residual_r0_{args.cell}_{args.role}"
|
||||
knobs = config["frontier"]["knobs"]
|
||||
base_command = driver.build_command(
|
||||
trace_path=trace_path,
|
||||
metrics_root=metrics_root,
|
||||
run_id=run_id,
|
||||
knobs=knobs,
|
||||
)
|
||||
driver.audit_command(base_command, knobs)
|
||||
command = enable_state_outputs(base_command)
|
||||
row = {
|
||||
"hook_path": config["calibration"]["hook_path"],
|
||||
"applied_a_tp": config["calibration"]["a_tp"],
|
||||
"sidecar_path": str(sidecar_path),
|
||||
"request_count": int(fixture["request_count"]),
|
||||
"tensor_parallel_size": int(fixture["tensor_parallel_size"]),
|
||||
}
|
||||
environment = driver.environment_for(row)
|
||||
if environment.get("CUDA_VISIBLE_DEVICES") != "":
|
||||
raise ValueError("Frontier state replay must hide CUDA devices")
|
||||
manifest = {
|
||||
"schema": "telemetry-residual-frontier-state-run-v1",
|
||||
"entry": {
|
||||
"cell": args.cell,
|
||||
"role": args.role,
|
||||
"anchor": entry["anchor"],
|
||||
"request_count": entry["selected_count"],
|
||||
},
|
||||
"inputs": {
|
||||
"prepared_manifest": str(args.prepared_manifest.resolve()),
|
||||
"prepared_manifest_sha256": sha256_file(args.prepared_manifest),
|
||||
"config": str(config_path.resolve()),
|
||||
"config_sha256": sha256_file(config_path),
|
||||
"fixture_manifest": str(fixture_manifest_path.resolve()),
|
||||
"fixture_manifest_sha256": sha256_file(fixture_manifest_path),
|
||||
"frontier_csv": str(trace_path.resolve()),
|
||||
"frontier_csv_sha256": sha256_file(trace_path),
|
||||
"sidecar": str(sidecar_path.resolve()),
|
||||
"sidecar_sha256": sha256_file(sidecar_path),
|
||||
},
|
||||
"frontier": {
|
||||
"root": str(args.frontier_root.resolve()),
|
||||
"git_head": git_capture(args.frontier_root, "rev-parse", "HEAD").strip(),
|
||||
"git_status_short": git_capture(args.frontier_root, "status", "--short"),
|
||||
},
|
||||
"runner": {
|
||||
"script": str(Path(__file__).resolve()),
|
||||
"script_sha256": sha256_file(Path(__file__).resolve()),
|
||||
"aituner_git_head": git_capture(AITUNER_ROOT, "rev-parse", "HEAD").strip(),
|
||||
"aituner_git_status_short": git_capture(AITUNER_ROOT, "status", "--short"),
|
||||
},
|
||||
"environment": {
|
||||
key: environment[key]
|
||||
for key in (
|
||||
"PYTHONPATH",
|
||||
"FRONTIER_EXECUTION_TIME_SCALE",
|
||||
"CUDA_VISIBLE_DEVICES",
|
||||
"NVIDIA_VISIBLE_DEVICES",
|
||||
"FRONTIER_LOG_LEVEL",
|
||||
)
|
||||
},
|
||||
"command": command,
|
||||
"state_outputs": {
|
||||
"individual_batch_metrics": True,
|
||||
"full_stage_batch_ledger": True,
|
||||
},
|
||||
"contains_prompt_text": False,
|
||||
}
|
||||
atomic_json(args.output / "run_manifest.json", manifest)
|
||||
start = time.monotonic()
|
||||
with (args.output / "stdout.log").open("w", encoding="utf-8") as stdout, (
|
||||
args.output / "stderr.log"
|
||||
).open("w", encoding="utf-8") as stderr:
|
||||
try:
|
||||
process = subprocess.run(
|
||||
command,
|
||||
cwd=args.frontier_root,
|
||||
env=environment,
|
||||
stdout=stdout,
|
||||
stderr=stderr,
|
||||
timeout=args.timeout_s,
|
||||
)
|
||||
return_code = int(process.returncode)
|
||||
except subprocess.TimeoutExpired:
|
||||
return_code = 124
|
||||
runtime_s = time.monotonic() - start
|
||||
if return_code != 0:
|
||||
failure = {
|
||||
"status": "STOP",
|
||||
"return_code": return_code,
|
||||
"runtime_s": runtime_s,
|
||||
}
|
||||
atomic_json(args.output / "failure.json", failure)
|
||||
raise RuntimeError(f"Frontier state replay failed: {failure}")
|
||||
|
||||
paths = find_state_metrics(args.output)
|
||||
summary = summarize_frontier(
|
||||
system_metrics_path=paths["system"],
|
||||
request_metrics_path=paths["requests"],
|
||||
batch_metrics_path=paths["batches"],
|
||||
ledger_path=paths["ledger"],
|
||||
)
|
||||
atomic_json(args.output / "common-state.json", summary)
|
||||
scorer = driver.score_trial(row, paths["system"], paths["requests"])
|
||||
atomic_json(args.output / "scorer_output.json", scorer)
|
||||
sizes = {name: path.stat().st_size for name, path in paths.items()}
|
||||
red_flags = []
|
||||
if summary["interval"]["request_count"] != int(entry["selected_count"]):
|
||||
red_flags.append("request_count_mismatch")
|
||||
if summary["sanity"]["batch_rows"] <= 0:
|
||||
red_flags.append("no_batch_rows")
|
||||
if summary["sanity"]["ledger_rows"] <= 0:
|
||||
red_flags.append("no_ledger_rows")
|
||||
result = {
|
||||
"schema": "telemetry-residual-frontier-state-result-v1",
|
||||
"status": "PASS" if not red_flags else "STOP",
|
||||
"runtime_s": runtime_s,
|
||||
"return_code": return_code,
|
||||
"paths": {name: str(path.resolve()) for name, path in paths.items()},
|
||||
"bytes": sizes,
|
||||
"common_state": str((args.output / "common-state.json").resolve()),
|
||||
"red_flags": red_flags,
|
||||
"sanity": {
|
||||
"request_rows": summary["sanity"]["request_rows"],
|
||||
"batch_rows": summary["sanity"]["batch_rows"],
|
||||
"ledger_rows": summary["sanity"]["ledger_rows"],
|
||||
"artifact_bytes": {
|
||||
"n": len(sizes),
|
||||
"min": min(sizes.values()),
|
||||
"max": max(sizes.values()),
|
||||
"distinct_n": len(set(sizes.values())),
|
||||
},
|
||||
"invariants": {
|
||||
"zero_failures": return_code == 0,
|
||||
"gpu_visibility_disabled": True,
|
||||
"request_count_match": not red_flags,
|
||||
"nonnegative_file_sizes": all(size >= 0 for size in sizes.values()),
|
||||
"state_values_not_all_identical": summary["sanity"]["invariants"][
|
||||
"batch_values_not_all_identical"
|
||||
],
|
||||
},
|
||||
},
|
||||
}
|
||||
atomic_json(args.output / "result.json", result)
|
||||
if result["status"] != "PASS":
|
||||
raise RuntimeError(red_flags)
|
||||
return result
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
result = argparse.ArgumentParser()
|
||||
result.add_argument("--prepared-manifest", type=Path, required=True)
|
||||
result.add_argument("--output", type=Path, required=True)
|
||||
result.add_argument("--replayserve-root", type=Path, required=True)
|
||||
result.add_argument("--frontier-root", type=Path, required=True)
|
||||
result.add_argument("--cell", required=True)
|
||||
result.add_argument("--role", choices=("low1", "high1"), required=True)
|
||||
result.add_argument("--timeout-s", type=float, default=300.0)
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result = execute(parser().parse_args())
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": result["status"],
|
||||
"runtime_s": result["runtime_s"],
|
||||
"request_rows": result["sanity"]["request_rows"],
|
||||
"batch_rows": result["sanity"]["batch_rows"],
|
||||
"ledger_rows": result["sanity"]["ledger_rows"],
|
||||
"red_flags": result["red_flags"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
169
runs/telemetry-residual/run_frontier_state_campaign.py
Normal file
169
runs/telemetry-residual/run_frontier_state_campaign.py
Normal file
@@ -0,0 +1,169 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the frozen 12-fixture P1 Frontier state-detail campaign, CPU only."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import Any
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
sys.path.insert(0, str(HERE))
|
||||
|
||||
import run_frontier_state as state_runner # noqa: E402
|
||||
|
||||
|
||||
def atomic_json(path: Path, payload: Any) -> None:
|
||||
state_runner.atomic_json(path, payload)
|
||||
|
||||
|
||||
def scorer_without_runtime(value: dict[str, Any]) -> dict[str, Any]:
|
||||
result = dict(value)
|
||||
result.pop("runtime_s", None)
|
||||
return result
|
||||
|
||||
|
||||
def execute(args: argparse.Namespace) -> dict[str, Any]:
|
||||
prepared = json.loads(args.prepared_manifest.read_text(encoding="utf-8"))
|
||||
committed = json.loads(args.committed_results.read_text(encoding="utf-8"))
|
||||
if prepared.get("status") != "PASS" or committed.get("status") != "PASS":
|
||||
raise RuntimeError("prepared or committed simulator evidence did not pass")
|
||||
committed_by_key = {
|
||||
(row["cell"], row["role"]): scorer_without_runtime(row["scorer"])
|
||||
for row in committed["results"]
|
||||
}
|
||||
entries = prepared["entries"]
|
||||
if len(entries) != 12 or len(committed_by_key) != 12:
|
||||
raise ValueError("P1 state campaign requires exactly 12 fixtures")
|
||||
args.output.mkdir(parents=True, exist_ok=True)
|
||||
results = []
|
||||
campaign_start = time.monotonic()
|
||||
for index, entry in enumerate(entries, start=1):
|
||||
key = (entry["cell"], entry["role"])
|
||||
output = args.output / f"{entry['cell']}_{entry['role']}"
|
||||
result_path = output / "result.json"
|
||||
if args.resume and result_path.is_file():
|
||||
result = json.loads(result_path.read_text(encoding="utf-8"))
|
||||
if result.get("status") != "PASS":
|
||||
raise RuntimeError(f"cannot resume failed state replay: {result_path}")
|
||||
resumed = True
|
||||
else:
|
||||
if output.exists() and any(output.iterdir()):
|
||||
raise FileExistsError(f"non-empty state replay output: {output}")
|
||||
print(
|
||||
f"RUN {index:02d}/12 {entry['cell']}/{entry['role']}",
|
||||
flush=True,
|
||||
)
|
||||
result = state_runner.execute(
|
||||
SimpleNamespace(
|
||||
prepared_manifest=args.prepared_manifest,
|
||||
output=output,
|
||||
replayserve_root=args.replayserve_root,
|
||||
frontier_root=args.frontier_root,
|
||||
cell=entry["cell"],
|
||||
role=entry["role"],
|
||||
timeout_s=args.timeout_s,
|
||||
)
|
||||
)
|
||||
resumed = False
|
||||
observed_scorer = json.loads(
|
||||
(output / "scorer_output.json").read_text(encoding="utf-8")
|
||||
)
|
||||
exact_scorer_match = scorer_without_runtime(observed_scorer) == committed_by_key[key]
|
||||
if not exact_scorer_match:
|
||||
raise ValueError(f"state-output replay changed the committed scorer: {key}")
|
||||
results.append(
|
||||
{
|
||||
"cell": entry["cell"],
|
||||
"role": entry["role"],
|
||||
"runtime_s": result["runtime_s"],
|
||||
"request_rows": result["sanity"]["request_rows"],
|
||||
"batch_rows": result["sanity"]["batch_rows"],
|
||||
"ledger_rows": result["sanity"]["ledger_rows"],
|
||||
"artifact_bytes": sum(result["bytes"].values()),
|
||||
"exact_committed_scorer_match": exact_scorer_match,
|
||||
"resumed": resumed,
|
||||
"result": str(result_path.resolve()),
|
||||
}
|
||||
)
|
||||
print(
|
||||
f"DONE {index:02d}/12 {entry['cell']}/{entry['role']} "
|
||||
f"runtime={result['runtime_s']:.3f}s batches={result['sanity']['batch_rows']}",
|
||||
flush=True,
|
||||
)
|
||||
runtimes = [float(row["runtime_s"]) for row in results]
|
||||
batches = [int(row["batch_rows"]) for row in results]
|
||||
bytes_values = [int(row["artifact_bytes"]) for row in results]
|
||||
red_flags = []
|
||||
if len(results) != 12:
|
||||
red_flags.append("runs_not_12")
|
||||
if not all(row["exact_committed_scorer_match"] for row in results):
|
||||
red_flags.append("committed_scorer_mismatch")
|
||||
if any(value <= 0 for value in batches):
|
||||
red_flags.append("empty_batch_output")
|
||||
result = {
|
||||
"schema": "telemetry-residual-frontier-state-campaign-v1",
|
||||
"status": "PASS" if not red_flags else "STOP",
|
||||
"prepared_manifest": str(args.prepared_manifest.resolve()),
|
||||
"committed_results": str(args.committed_results.resolve()),
|
||||
"campaign_elapsed_s": time.monotonic() - campaign_start,
|
||||
"results": results,
|
||||
"red_flags": red_flags,
|
||||
"sanity": {
|
||||
"n": len(results),
|
||||
"runtime_s": state_runner.numeric(runtimes),
|
||||
"batch_rows": state_runner.numeric(batches),
|
||||
"artifact_bytes": state_runner.numeric(bytes_values),
|
||||
"invariants": {
|
||||
"runs_12": len(results) == 12,
|
||||
"zero_failures": not red_flags,
|
||||
"exact_committed_scorers": all(
|
||||
row["exact_committed_scorer_match"] for row in results
|
||||
),
|
||||
"nonnegative_counts": all(value > 0 for value in batches),
|
||||
"per_config_not_identical": len(set(batches)) > 1,
|
||||
"gpu_visibility_disabled": True,
|
||||
},
|
||||
},
|
||||
}
|
||||
atomic_json(args.output / "campaign-metrics.json", result)
|
||||
if result["status"] != "PASS":
|
||||
raise RuntimeError(red_flags)
|
||||
return result
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
result = argparse.ArgumentParser()
|
||||
result.add_argument("--prepared-manifest", type=Path, required=True)
|
||||
result.add_argument("--committed-results", type=Path, required=True)
|
||||
result.add_argument("--output", type=Path, required=True)
|
||||
result.add_argument("--replayserve-root", type=Path, required=True)
|
||||
result.add_argument("--frontier-root", type=Path, required=True)
|
||||
result.add_argument("--timeout-s", type=float, default=300.0)
|
||||
result.add_argument("--resume", action="store_true")
|
||||
return result
|
||||
|
||||
|
||||
def main() -> None:
|
||||
result = execute(parser().parse_args())
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"status": result["status"],
|
||||
"runs": len(result["results"]),
|
||||
"elapsed_s": result["campaign_elapsed_s"],
|
||||
"sanity": result["sanity"],
|
||||
"red_flags": result["red_flags"],
|
||||
},
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
183
runs/telemetry-residual/test_common_state.py
Normal file
183
runs/telemetry-residual/test_common_state.py
Normal file
@@ -0,0 +1,183 @@
|
||||
#!/usr/bin/env python3
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from common_state import numeric, residual, summarize_engine, summarize_frontier
|
||||
from run_frontier_state import enable_state_outputs
|
||||
|
||||
|
||||
def engine_record(
|
||||
step: int,
|
||||
timestamp_ns: int,
|
||||
*,
|
||||
batch: int,
|
||||
prefill: int,
|
||||
decode: int,
|
||||
waiting: int,
|
||||
running: int,
|
||||
kv: float,
|
||||
) -> dict[str, object]:
|
||||
return {
|
||||
"step_index": step,
|
||||
"submit_mono_ns": timestamp_ns,
|
||||
"model_executed": True,
|
||||
"scheduled_requests": batch,
|
||||
"decode_batch_size": decode,
|
||||
"prefill_tokens": prefill,
|
||||
"decode_tokens": decode,
|
||||
"preemptions": 0,
|
||||
"queues": {"waiting": waiting, "running": running},
|
||||
"kv": {"usage": kv},
|
||||
"cudagraph": {
|
||||
"runtime_mode": "NONE" if step == 0 else "FULL",
|
||||
"bucket_tokens": prefill + decode + 1,
|
||||
"padding_tokens": 1,
|
||||
},
|
||||
"dropped_records_before": 0,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
assert numeric((-2.0, -1.0))["cv"] >= 0.0
|
||||
command = [
|
||||
"python3",
|
||||
"-m",
|
||||
"frontier.main",
|
||||
"--no-metrics_config_store_frontier_stage_batch_ledger",
|
||||
]
|
||||
state_command = enable_state_outputs(command)
|
||||
assert "--metrics_config_store_frontier_stage_batch_ledger" in state_command
|
||||
assert "--metrics_config_keep_individual_batch_metrics" in state_command
|
||||
assert "--no-metrics_config_store_frontier_stage_batch_ledger" not in state_command
|
||||
|
||||
engine = summarize_engine(
|
||||
[
|
||||
engine_record(
|
||||
0,
|
||||
0,
|
||||
batch=2,
|
||||
prefill=6,
|
||||
decode=0,
|
||||
waiting=1,
|
||||
running=2,
|
||||
kv=0.1,
|
||||
),
|
||||
engine_record(
|
||||
1,
|
||||
1_000_000_000,
|
||||
batch=4,
|
||||
prefill=0,
|
||||
decode=4,
|
||||
waiting=3,
|
||||
running=4,
|
||||
kv=0.3,
|
||||
),
|
||||
],
|
||||
start_ns=0,
|
||||
end_ns=2_000_000_000,
|
||||
request_count=2,
|
||||
)
|
||||
assert math.isclose(engine["common"]["queue_waiting_mean"], 2.0)
|
||||
assert math.isclose(engine["common"]["queue_running_mean"], 3.0)
|
||||
assert math.isclose(
|
||||
engine["common"]["queue_waiting_time_per_request_ms"], 2000.0
|
||||
)
|
||||
assert engine["common"]["batch_size"]["distinct_n"] == 2
|
||||
assert math.isclose(engine["common"]["prefill_token_fraction"], 0.6)
|
||||
|
||||
with tempfile.TemporaryDirectory() as temporary:
|
||||
root = Path(temporary)
|
||||
system = root / "system.json"
|
||||
requests = root / "requests.csv"
|
||||
batches = root / "batches.csv"
|
||||
ledger = root / "ledger.jsonl"
|
||||
system.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"throughput_metrics": {
|
||||
"total_duration_seconds": 2.0,
|
||||
}
|
||||
}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
with requests.open("w", encoding="utf-8", newline="") as output:
|
||||
writer = csv.DictWriter(
|
||||
output,
|
||||
fieldnames=[
|
||||
"request_e2e_time",
|
||||
"request_waiting_time_total",
|
||||
"request_total_preemption_count",
|
||||
],
|
||||
)
|
||||
writer.writeheader()
|
||||
writer.writerow(
|
||||
{
|
||||
"request_e2e_time": 2000,
|
||||
"request_waiting_time_total": 1000,
|
||||
"request_total_preemption_count": 0,
|
||||
}
|
||||
)
|
||||
writer.writerow(
|
||||
{
|
||||
"request_e2e_time": 3000,
|
||||
"request_waiting_time_total": 3000,
|
||||
"request_total_preemption_count": 0,
|
||||
}
|
||||
)
|
||||
with batches.open("w", encoding="utf-8", newline="") as output:
|
||||
writer = csv.DictWriter(
|
||||
output,
|
||||
fieldnames=[
|
||||
"batch_size",
|
||||
"batch_num_tokens",
|
||||
"batch_num_prefill_tokens",
|
||||
"batch_num_decode_tokens",
|
||||
],
|
||||
)
|
||||
writer.writeheader()
|
||||
writer.writerow(
|
||||
{
|
||||
"batch_size": 2,
|
||||
"batch_num_tokens": 6,
|
||||
"batch_num_prefill_tokens": 6,
|
||||
"batch_num_decode_tokens": 0,
|
||||
}
|
||||
)
|
||||
writer.writerow(
|
||||
{
|
||||
"batch_size": 4,
|
||||
"batch_num_tokens": 4,
|
||||
"batch_num_prefill_tokens": 0,
|
||||
"batch_num_decode_tokens": 4,
|
||||
}
|
||||
)
|
||||
ledger.write_text(
|
||||
json.dumps({"stage_start_ts": 0.0, "stage_end_ts": 1.0}) + "\n"
|
||||
+ json.dumps({"stage_start_ts": 1.0, "stage_end_ts": 2.0})
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
simulator = summarize_frontier(
|
||||
system_metrics_path=system,
|
||||
request_metrics_path=requests,
|
||||
batch_metrics_path=batches,
|
||||
ledger_path=ledger,
|
||||
)
|
||||
assert math.isclose(simulator["common"]["queue_waiting_mean"], 2.0)
|
||||
assert math.isclose(simulator["common"]["queue_running_mean"], 0.5)
|
||||
assert simulator["common"]["batch_size"]["distinct_n"] == 2
|
||||
difference = residual(engine, simulator)
|
||||
assert difference["coverage"]["missing"] == 0
|
||||
assert difference["coverage"]["available"] == 16
|
||||
assert math.isclose(difference["values"]["queue_waiting_mean"], 0.0)
|
||||
print("telemetry common state: PASS")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
67
runs/telemetry-residual/test_residual_transfer.py
Normal file
67
runs/telemetry-residual/test_residual_transfer.py
Normal file
@@ -0,0 +1,67 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Small structural tests for the cross-config transfer diagnostic."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
SPEC = importlib.util.spec_from_file_location(
|
||||
"analyze_residual_transfer", HERE / "analyze_residual_transfer.py"
|
||||
)
|
||||
assert SPEC is not None and SPEC.loader is not None
|
||||
MODULE = importlib.util.module_from_spec(SPEC)
|
||||
SPEC.loader.exec_module(MODULE)
|
||||
|
||||
|
||||
def example(cell: str, level: str, value: float) -> dict:
|
||||
tp_text, mns_text = cell.split("_")
|
||||
return {
|
||||
"cell": cell,
|
||||
"level": level,
|
||||
"tp": int(tp_text[2:]),
|
||||
"mns": int(mns_text[3:]),
|
||||
"pass_rate_residual": value - 0.5,
|
||||
"real_pass_rate_rep1": value,
|
||||
"sim_pass_rate": 0.5,
|
||||
"offered_req_s_per_gpu": 1.0 if level == "low" else 2.0,
|
||||
"real_feasible": value >= 0.95,
|
||||
"sim_feasible": False,
|
||||
"state_residual": {"values": {"batch.mean": value}},
|
||||
"engine": {
|
||||
"common": {"batch": {"mean": value, "max": value, "cv": 0.0}},
|
||||
"engine_only": {"kv": value},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cells = ("tp1_mns8", "tp1_mns16", "tp2_mns8", "tp2_mns16")
|
||||
examples = [
|
||||
example(cell, level, 0.1 + index / 10.0)
|
||||
for level in ("low", "high")
|
||||
for index, cell in enumerate(cells)
|
||||
]
|
||||
rows = MODULE.transitions(examples)
|
||||
assert len(rows) == 48
|
||||
assert any(row["source_level"] != row["target_level"] for row in rows)
|
||||
assert all(row["source_cell"] != row["target_cell"] for row in rows)
|
||||
assert len(rows[0]["hybrid_telemetry"]) > len(rows[0]["hybrid_base"])
|
||||
assert len(rows[0]["direct_telemetry"]) > len(rows[0]["direct_base"])
|
||||
for feature, target in (
|
||||
("hybrid_base", "target_residual_delta"),
|
||||
("hybrid_telemetry", "target_residual_delta"),
|
||||
("direct_base", "target_real_pass_rate_delta"),
|
||||
("direct_telemetry", "target_real_pass_rate_delta"),
|
||||
):
|
||||
prediction = MODULE.grouped_predictions(
|
||||
rows, feature_name=feature, target_name=target, regularization=1.0
|
||||
)
|
||||
assert len(prediction) == len(rows)
|
||||
print("telemetry residual transfer: PASS")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user