gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record phase-aware telemetry pilot result

Browse Source
This commit is contained in:
Gahow Wang
2026-07-14 19:18:57 +08:00
parent 7fd9563550
commit 26c2cdab2b
6 changed files with 17372 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

160
docs/intervention-response-v3-results-20260714.md Normal file
View File

@@ -0,0 +1,160 @@
# Phase-aware telemetry intervention-response v3 results
Date: 2026-07-14 (Asia/Singapore).
Decision: **`STOP_NO_INCREMENTAL_TUNING_SIGNAL`**.
## Claim tested
After the replay has developed queue, batch, and KV state, does increasing MNS
from 16 to 64 create telemetry responses that exceed workload-repeat noise, and
does any such response identify whether the action repairs the full-run SLO
better than external prefix outcomes alone?
The first clause passed. The second clause failed. Long-window telemetry is
mechanistically informative, but this pilot does not support its necessity for
tuning this action.
## Setup
- dash0 GPU 0-3: four NVIDIA H20 GPUs; Qwen3-30B-A3B; patched vLLM
`0.24.1.dev3+opprof`; TP=4.
- Action: MNS `16 -> 64` with exact request, arrival, and input-length hashes.
- Trace: `chat_w20260312_1000`, replay-time scale 0.5, 300 seconds.
- Loads: 1.5 and 2.125 requests/s/GPU; three disjoint bands; endpoint order
A/B, B/A, A/B; load order low/mid, mid/low, low/mid.
- Five cumulative checkpoints: 30, 75, 150, 225, and 300 seconds; four
non-overlapping quarter blocks.
- Six fresh-server sessions, 12 measured runs, six action pairs, and eight
same-config repeat pairs.
The prior three-load attempt is not part of the result. Its MNS=16 workload at
3.125 requests/s/GPU could not drain by the 450-second client timeout and
produced no high-load result. V3 reran every retained point from scratch.
## End-to-end outcome
All three low-load pairs remained feasible (`true->true`, label 0). All three
pressure-load pairs changed from infeasible to feasible (`false->true`, label
1). MNS=16 pressure pass rates were 0.5604, 0.3145, and 0.2635; all three
MNS=64 pressure runs reached 1.0. This yielded a balanced 3/3 action label set.
## Mechanism result
No telemetry feature passed the action-versus-repeat gate at 10% or 25%. At
50%, graph padding first passed. At both 75% and 100%, graph padding and queue
waiting passed, satisfying the requirement for two features at the same pair
of adjacent checkpoints and with consistent directions in both load regimes.
| Feature | Direction for MNS 16->64 | 75% effect/repeat median | 75% above repeat p95 | 100% effect/repeat median | 100% above repeat p95 |
|---|---:|---:|---:|---:|---:|
| `queue_waiting_mean` | lower | 1346.22 | 3/6 | 898.70 | 3/6 |
| `graph_padding_fraction` | higher | 5.00 | 4/6 | 5.74 | 5/6 |
The very large queue effect/median-repeat ratios should not be read alone: its
repeat p95 was much larger than its repeat median, so the independent p95
coverage criterion remained binding. Full-window queue-waiting deltas were
-0.19 to -0.32 at low load and -20.99 to -32.16 at pressure load. Graph
padding increased in every pair, by 0.00133-0.00217 at low load and
0.00705-0.00873 at pressure load.
The mechanism is therefore a real tradeoff: larger MNS reduces queueing,
especially under pressure, while increasing CUDA-graph padding.
## Tuning-signal result
Leave-one-repetition-out balanced accuracy was evaluated against the best
external prefix-outcome feature at every predeclared checkpoint.
| Replay phase | Best external BA | Best telemetry BA | Incremental telemetry gate |
|---:|---:|---:|---|
| 10% | 0.833 | 0.833 | fail |
| 25% | 1.000 | 1.000 | fail |
| 50% | 0.833 | 1.000 | pass at this checkpoint only |
| 75% | 1.000 | 1.000 | fail |
| 100% | 1.000 | 1.000 | fail |
At 50%, several telemetry features exceeded the external baseline by at least
0.15, including the phase-stable mechanism feature
`graph_padding_fraction`. The advantage did not hold at either adjacent
checkpoint. Consequently no feature passed the frozen two-adjacent-phase
requirement.
The important ordering is that external TTFT already classified the action
perfectly at 25%, whereas the robust two-feature mechanism response did not
emerge until 75%-100%. In this setup telemetry explains *why* MNS helps, but it
does not provide earlier or more reliable action selection than direct prefix
outcomes.
## Research conclusion
The 5/10-second negative result was indeed too narrow. It only ruled out an
ultra-early telemetry verifier; it did not rule out engine-state information.
The 300-second pilot finds a clear and reproducible queueing-versus-padding
response.
However, this does not rescue the direct telemetry-guided tuning claim. For
this action and workload, the external signal is already as good or better
before the telemetry mechanism becomes stable. The project should therefore
not claim that engine instrumentation is necessary for tuning on this evidence,
and should not open an E2E policy test from this pilot.
The narrower simulator-residual route remains logically open: telemetry may
explain why a simulator misranks real configurations even when direct online
outcomes can guide a tuner. That is a different hypothesis and was not tested
here.
## Change and verification
Change: absolute 5/10-second prefixes were replaced by phase-aware 30/75/150/
225/300-second analysis; SLO early stop was disabled; full Layer-1 coverage,
hash, request-accounting, controller, and stream/footer gates were added. The
mechanism gate was corrected to require two features at the same adjacent
phase pair.
Expected effect: distinguish “telemetry has not developed yet” from “telemetry
does not identify or improve the action decision.”
Verification: five local analysis/controller test suites passed; remote
manifest preflight and command dry-run passed; six serialized sessions passed
all stream invariants; the analyzer was rerun and produced byte-identical
output.
Result: mechanism evidence passed; incremental tuning evidence failed. Audit
SHA256: `45f6f248712f9cbd3ed72036837ff6dc5b5c14c0f2eb6ba5cd5daceb1aa4ddb7`.
Remaining risk: this is a development pilot with one model, one TP, one action,
two retained loads, three request bands, and six action labels. It is adequate
to reject opening the next direct-policy stage, not to establish a universal
negative claim about telemetry.
## Research-validity audit
| Check | Verdict | Evidence / boundary |
|---|---|---|
| Real system and E2E outcome | PASS | Real H20/vLLM replay; full SLO outcome accompanies mechanism telemetry. |
| Matched action baseline | PASS | Exact request/arrival/length hashes for MNS 16 and 64; external prefix outcome is the decision baseline. |
| Repeats and order effects | PASS for pilot | Three disjoint bands; A/B, B/A, A/B endpoint order; counter-rotated load order. |
| Selective load removal | PASS with narrowed claim | The 3.125 load produced no result before any action comparison; the failure and cost are retained, and all kept points were freshly rerun. |
| Significance/generalization | NEEDS EVIDENCE for a paper claim | Only three bands, one model, one TP, one action, and six labels. This is explicitly a stage gate. |
| Calibration versus evaluation | NEEDS EVIDENCE for a positive policy claim | Frozen gates and leave-one-band-out folds reduce leakage, but a new workload/model hold-out is still required. |
| Platform/reproducibility | PASS | Commit, commands, manifest, controller state, platform fingerprint, raw remote paths, and audit hashes are recorded. |
## Data sanity
- Measured runs: n=12; elapsed 300.346-317.012 seconds; 12 distinct; pass rate
0.2635-1.0 with 4 distinct values; selected requests 1800-2550 with 2
distinct values.
- Sessions: n=6; 0.8413-0.8631 H20-hours; 6 distinct; Layer-1 records
58,465-64,776; 6 distinct. V3 cost was 5.0924 H20-hours; total including
the invalid attempt was 6.4505, below the 8.0 cap.
- Labels: n=6; min/max 0/1; 2 distinct. Action pairs were 6 and repeat pairs
were 8 at every checkpoint.
- Coverage-gap observations: n=60; start gaps 0.0427-0.1247 seconds; end gaps
0.00014-0.1705; maximum internal gaps 0.1695-0.6528, all below one second.
- Checked invariants: exact pair hashes and counts, all runs uncensored, full
request accounting, monotonic admitted/completed coverage, monotonic Layer-1
timestamps, nonnegative counters, bounded ratios, non-identical per-config
states, contiguous step indices, zero drops, footer/sidecar agreement, no
controller failures, all sessions complete, GPU idle after completion. No
red flags were found.

49
runs/intervention-response-v2/failed-three-load-controller-state-v2.json Normal file
View File

@@ -0,0 +1,49 @@
{
"completed_sessions": 0,
"failures": [
{
"failure": "TimeoutError('client timeout: tp4_mns16 high')",
"session": "rep1-mns16"
}
],
"gpu_hours_total": 1.3580546813540988,
"hard_cap_h20_hours": 8.0,
"schema": "intervention-response-phase-aware-pilot-state-v2",
"sessions": {
"rep1-mns16": {
"burnin": {
"elapsed_s": 64.469448031,
"feasible": false,
"pass_rate": 0.5137254901960784
},
"failure": "TimeoutError('client timeout: tp4_mns16 high')",
"gpu_hours": 1.3580546813540988,
"mns": 16,
"replicate": 1,
"runs": [
{
"early_stopped": false,
"elapsed_s": 301.114789132,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
},
{
"early_stopped": false,
"elapsed_s": 313.040655133,
"feasible": false,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 0.5631372549019608,
"selected_count": 2550
}
],
"started_at": 1784021277.1191542,
"status": "failed"
}
},
"started_at": 1784021276.8224819,
"status": "failed"
}

16451
runs/intervention-response-v2/pilot-audit-v3.json Normal file
View File

File diff suppressed because it is too large Load Diff

342
runs/intervention-response-v2/pilot-controller-state-v3.json Normal file
View File

@@ -0,0 +1,342 @@
{
"completed_at": 1784027453.4005148,
"completed_sessions": 6,
"failures": [],
"gpu_hours_total": 5.092428478929731,
"hard_cap_h20_hours": 6.641945318645901,
"schema": "intervention-response-phase-aware-pilot-state-v3",
"sessions": {
"rep1-mns16": {
"burnin": {
"elapsed_s": 64.194648239,
"feasible": false,
"pass_rate": 0.5235294117647059
},
"completed_at": 1784023620.6911042,
"gpu_hours": 0.856446467505561,
"mns": 16,
"replicate": 1,
"runs": [
{
"early_stopped": false,
"elapsed_s": 301.068878512,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
},
{
"early_stopped": false,
"elapsed_s": 313.268231477,
"feasible": false,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 0.5603921568627451,
"selected_count": 2550
}
],
"started_at": 1784022846.257515,
"status": "complete",
"validation": {
"accounting_mode": "graceful-footer",
"cell": "tp4_mns16",
"invariants": {
"anchor_intervals_present": true,
"compile_capture_pre_ready": true,
"encoded_balanced": true,
"footer_sidecar_agrees": true,
"last_step_matches": true,
"layer1_contiguous": true,
"layer1_zero_drops": true,
"one_footer_last": true,
"one_ready_marker": true,
"sidecar_final": true,
"warmup_exact_16": true,
"warmup_long": true,
"written_matches_records": true
},
"layer1_records": 60017,
"post_ready_capture_events": [],
"stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep1-mns16/opprof/opprof-v1-dp0-pid458427-1784022917661752826.jsonl"
}
},
"rep1-mns64": {
"burnin": {
"elapsed_s": 60.79926316,
"feasible": true,
"pass_rate": 1.0
},
"completed_at": 1784024382.2164423,
"gpu_hours": 0.8415956518385146,
"mns": 64,
"replicate": 1,
"runs": [
{
"early_stopped": false,
"elapsed_s": 301.022700013,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
},
{
"early_stopped": false,
"elapsed_s": 301.100993458,
"feasible": true,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 1.0,
"selected_count": 2550
}
],
"started_at": 1784023620.9865165,
"status": "complete",
"validation": {
"accounting_mode": "graceful-footer",
"cell": "tp4_mns64",
"invariants": {
"anchor_intervals_present": true,
"compile_capture_pre_ready": true,
"encoded_balanced": true,
"footer_sidecar_agrees": true,
"last_step_matches": true,
"layer1_contiguous": true,
"layer1_zero_drops": true,
"one_footer_last": true,
"one_ready_marker": true,
"sidecar_final": true,
"warmup_exact_16": true,
"warmup_long": true,
"written_matches_records": true
},
"layer1_records": 62456,
"post_ready_capture_events": [],
"stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep1-mns64/opprof/opprof-v1-dp0-pid471212-1784023695315485142.jsonl"
}
},
"rep2-mns16": {
"burnin": {
"elapsed_s": 63.518994307,
"feasible": false,
"pass_rate": 0.6647058823529411
},
"completed_at": 1784025910.710227,
"gpu_hours": 0.8481734045346578,
"mns": 16,
"replicate": 2,
"runs": [
{
"early_stopped": false,
"elapsed_s": 315.780805484,
"feasible": false,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 0.31450980392156863,
"selected_count": 2550
},
{
"early_stopped": false,
"elapsed_s": 300.346368314,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
}
],
"started_at": 1784025143.731658,
"status": "complete",
"validation": {
"accounting_mode": "graceful-footer",
"cell": "tp4_mns16",
"invariants": {
"anchor_intervals_present": true,
"compile_capture_pre_ready": true,
"encoded_balanced": true,
"footer_sidecar_agrees": true,
"last_step_matches": true,
"layer1_contiguous": true,
"layer1_zero_drops": true,
"one_footer_last": true,
"one_ready_marker": true,
"sidecar_final": true,
"warmup_exact_16": true,
"warmup_long": true,
"written_matches_records": true
},
"layer1_records": 58465,
"post_ready_capture_events": [],
"stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep2-mns16/opprof/opprof-v1-dp0-pid496100-1784025206790752505.jsonl"
}
},
"rep2-mns64": {
"burnin": {
"elapsed_s": 60.803771161,
"feasible": true,
"pass_rate": 1.0
},
"completed_at": 1784025143.4362338,
"gpu_hours": 0.8412917502721151,
"mns": 64,
"replicate": 2,
"runs": [
{
"early_stopped": false,
"elapsed_s": 300.605515019,
"feasible": true,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 1.0,
"selected_count": 2550
},
{
"early_stopped": false,
"elapsed_s": 300.354084826,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
}
],
"started_at": 1784024382.5080929,
"status": "complete",
"validation": {
"accounting_mode": "graceful-footer",
"cell": "tp4_mns64",
"invariants": {
"anchor_intervals_present": true,
"compile_capture_pre_ready": true,
"encoded_balanced": true,
"footer_sidecar_agrees": true,
"last_step_matches": true,
"layer1_contiguous": true,
"layer1_zero_drops": true,
"one_footer_last": true,
"one_ready_marker": true,
"sidecar_final": true,
"warmup_exact_16": true,
"warmup_long": true,
"written_matches_records": true
},
"layer1_records": 61696,
"post_ready_capture_events": [],
"stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep2-mns64/opprof/opprof-v1-dp0-pid483758-1784024456752184540.jsonl"
}
},
"rep3-mns16": {
"burnin": {
"elapsed_s": 64.324514503,
"feasible": false,
"pass_rate": 0.5588235294117647
},
"completed_at": 1784026691.6293187,
"gpu_hours": 0.8631463612450494,
"mns": 16,
"replicate": 3,
"runs": [
{
"early_stopped": false,
"elapsed_s": 300.711109876,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
},
{
"early_stopped": false,
"elapsed_s": 317.011840655,
"feasible": false,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 0.2635294117647059,
"selected_count": 2550
}
],
"started_at": 1784025911.00421,
"status": "complete",
"validation": {
"accounting_mode": "graceful-footer",
"cell": "tp4_mns16",
"invariants": {
"anchor_intervals_present": true,
"compile_capture_pre_ready": true,
"encoded_balanced": true,
"footer_sidecar_agrees": true,
"last_step_matches": true,
"layer1_contiguous": true,
"layer1_zero_drops": true,
"one_footer_last": true,
"one_ready_marker": true,
"sidecar_final": true,
"warmup_exact_16": true,
"warmup_long": true,
"written_matches_records": true
},
"layer1_records": 61438,
"post_ready_capture_events": [],
"stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep3-mns16/opprof/opprof-v1-dp0-pid508913-1784025983046041167.jsonl"
}
},
"rep3-mns64": {
"burnin": {
"elapsed_s": 60.850091109,
"feasible": true,
"pass_rate": 1.0
},
"completed_at": 1784027453.3790896,
"gpu_hours": 0.8417748435338338,
"mns": 64,
"replicate": 3,
"runs": [
{
"early_stopped": false,
"elapsed_s": 300.710505069,
"feasible": true,
"level": "low",
"offered_req_s_per_gpu": 1.5,
"pass_rate": 1.0,
"selected_count": 1800
},
{
"early_stopped": false,
"elapsed_s": 300.869326418,
"feasible": true,
"level": "mid",
"offered_req_s_per_gpu": 2.125,
"pass_rate": 1.0,
"selected_count": 2550
}
],
"started_at": 1784026691.96421,
"status": "complete",
"validation": {
"accounting_mode": "graceful-footer",
"cell": "tp4_mns64",
"invariants": {
"anchor_intervals_present": true,
"compile_capture_pre_ready": true,
"encoded_balanced": true,
"footer_sidecar_agrees": true,
"last_step_matches": true,
"layer1_contiguous": true,
"layer1_zero_drops": true,
"one_footer_last": true,
"one_ready_marker": true,
"sidecar_final": true,
"warmup_exact_16": true,
"warmup_long": true,
"written_matches_records": true
},
"layer1_records": 64776,
"post_ready_capture_events": [],
"stream": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/runs/pilot/sessions/rep3-mns64/opprof/opprof-v1-dp0-pid521797-1784026767359409666.jsonl"
}
}
},
"started_at": 1784022845.964628,
"status": "complete"
}

324
runs/intervention-response-v2/pilot-manifest-v3.json Normal file
View File

@@ -0,0 +1,324 @@
{
"budget": {
"expected_h20_hours": [
4.8,
6.641945318645901
],
"expected_wall_minutes": [
75,
95
],
"global_hard_cap_h20_hours": 8.0,
"hard_cap_h20_hours": 6.641945318645901,
"prior_attempt_h20_hours": 1.3580546813540988,
"session_estimate_h20_hours": 1.0
},
"burnin": {
"anchor": 0.18919793755240089,
"arrival_order_sha256": "6c0ac4cb9a30ef501eeeacc8e6cc631c345e976db5ccf530ea5a1ec706d62a24",
"input_length_order_sha256": "7939cc20e1a00d1031d27d71508789f38decbbbb6ea59a1df18b2ec342fd2ef8",
"offered_req_s": 8.5,
"offered_req_s_per_gpu": 2.125,
"request_id_order_sha256": "84f4809acbc8acd3b1d14dfa357134a1dc0b9287341624b33f598dafeef54dc7",
"selected_count": 510,
"study": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/studies/burnin-tp4.json",
"study_sha256": "5d6c2098042909a863efd3112818fbee9bafe96f22898ac98b66846dbe1fef0f"
},
"checkpoints": {
"fractions": [
0.1,
0.25,
0.5,
0.75,
1.0
],
"seconds": [
30.0,
75.0,
150.0,
225.0,
300.0
]
},
"engine": {
"disable_slo_early_stop": true,
"duration_s": 300.0,
"mns_endpoints": [
16,
64
],
"replay_time_scale": 0.5,
"safety_deadline_s": 360.0,
"tp": 4
},
"private": {
"merged_traces": {
"1": {
"bytes": 337429767,
"path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
"request_id_scheme": "sha256(source_sha256:line_number:original_id)",
"rows": 9420,
"sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
"source_sha256": [
"b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
"d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
],
"sources": [
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
]
},
"2": {
"bytes": 337509330,
"path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
"request_id_scheme": "sha256(source_sha256:line_number:original_id)",
"rows": 9457,
"sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
"source_sha256": [
"4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
"bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
],
"sources": [
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
]
},
"3": {
"bytes": 337450256,
"path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
"request_id_scheme": "sha256(source_sha256:line_number:original_id)",
"rows": 9431,
"sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
"source_sha256": [
"1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
"6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
],
"sources": [
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
]
}
}
},
"repetitions": {
"1": {
"duration_s": 300.0,
"load_order": [
"low",
"mid"
],
"merged_trace": {
"bytes": 337429767,
"path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep1.jsonl",
"request_id_scheme": "sha256(source_sha256:line_number:original_id)",
"rows": 9420,
"sha256": "68983266aa0e66aa589562f7c08edbd966f9ba4405e20c105adb43777d2dfbf5",
"source_sha256": [
"b242d1d9086df3accab57b4c92445d5edd581e12f47e12cea227aa63964c6930",
"d23b549f7b69af3647308677bbf76f818a3c226a1c98f9a9f93f09ceee46be87"
],
"sources": [
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low1.jsonl",
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high1.jsonl"
]
},
"selections": {
"low": {
"anchor": 0.34815061842962747,
"arrival_order_sha256": "0b9473052a28b2eb98bb4bda59c726d6c74cd42656c00046f0559cefd38e0c42",
"input_length_order_sha256": "0172aa1fd7d994e9d95857c32293e25f6c537616b40e01034308a0d57cd94139",
"offered_req_s": 6.0,
"offered_req_s_per_gpu": 1.5,
"request_id_order_sha256": "af50f0c3edfc6f9fb74734b2501378fc8ba6fdd3dd3fe84f2bb1d68540eee69b",
"selected_count": 1800,
"target_count": 1800,
"target_req_s_per_gpu": 1.5
},
"mid": {
"anchor": 0.48686986110831465,
"arrival_order_sha256": "c2ad99986ce558da5901a9c5ec0a00bd69f198c981d8779235f2773a5c87f1c0",
"input_length_order_sha256": "9442bfebdc3fab5062dc1f4d688dc28c02afe3fd806c56dd8159f0ac7e6d0b94",
"offered_req_s": 8.5,
"offered_req_s_per_gpu": 2.125,
"request_id_order_sha256": "0bb61dbc9c26875e991d0d4f984134910d37463e5063f86ee960cf4f8aafb771",
"selected_count": 2550,
"target_count": 2550,
"target_req_s_per_gpu": 2.125
}
},
"source_roles": [
"low1",
"high1"
],
"study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep1-tp4.json",
"study_sha256": "ecfff96e33d458eb1e3b9a6d24386f00cc6f1b19ff926e2ec6320b3f671a7ae3"
},
"2": {
"duration_s": 300.0,
"load_order": [
"mid",
"low"
],
"merged_trace": {
"bytes": 337509330,
"path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep2.jsonl",
"request_id_scheme": "sha256(source_sha256:line_number:original_id)",
"rows": 9457,
"sha256": "f38e8938f6a481fc6725b71b21aa04ff7eaf79783cdfd6e41aa2f074156f00c2",
"source_sha256": [
"4cbb0baac082bd54af562ce2f39104c5c23b4671672da365a67b1e8c146adf9f",
"bb0bcd2564a88000f435f12feb21c7c902eafc9ea5fe916adfe9d1eae47f3f9a"
],
"sources": [
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low2.jsonl",
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high2.jsonl"
]
},
"selections": {
"low": {
"anchor": 0.3411190868318572,
"arrival_order_sha256": "f4eafbdf7420e8a97e38d46fff1aec5fdbc2a049133862a793b4635b51061b7f",
"input_length_order_sha256": "cdf82cfc546651f1859c53348eace72348572f09e6e1879ad77d51b489a998fe",
"offered_req_s": 6.0,
"offered_req_s_per_gpu": 1.5,
"request_id_order_sha256": "adc728f189a1ce0301609cdbae7d1dc2318609b13eafb94bcb90903bb7998eb1",
"selected_count": 1800,
"target_count": 1800,
"target_req_s_per_gpu": 1.5
},
"mid": {
"anchor": 0.4825698948735577,
"arrival_order_sha256": "b9fc12cf3f86bc8a79bee65296e65aa2b8bf2aeca46b2887094c669adcbb9a00",
"input_length_order_sha256": "d8d4bd6fc8ba852a45605b673b6b3e4f33b58f459e69f2a032d226ee175b074e",
"offered_req_s": 8.5,
"offered_req_s_per_gpu": 2.125,
"request_id_order_sha256": "56a0616b6b54abafd37875c7cb25f8639afef2706ccc55dfbe568f45859ea382",
"selected_count": 2550,
"target_count": 2550,
"target_req_s_per_gpu": 2.125
}
},
"source_roles": [
"low2",
"high2"
],
"study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep2-tp4.json",
"study_sha256": "d92a576db031db24bb58f354ea725d7f7567cb76699d387117ac5a6c9317bbb9"
},
"3": {
"duration_s": 300.0,
"load_order": [
"low",
"mid"
],
"merged_trace": {
"bytes": 337450256,
"path": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/traces/rep3.jsonl",
"request_id_scheme": "sha256(source_sha256:line_number:original_id)",
"rows": 9431,
"sha256": "3094084b0bb20cc02eecf465091a5c919b4e5b112f704cdc36a563d1efdcee46",
"source_sha256": [
"1f7ececb142f9a363d2d1ca25eb7b8488b2cc319a51b55faa384f2a3d51f2142",
"6f326234791e1cff4ff866bface0d097d0d6e3844eebb1c97653d8e9c35e9397"
],
"sources": [
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/low3.jsonl",
"/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/private/traces/high3.jsonl"
]
},
"selections": {
"low": {
"anchor": 0.3459572828521402,
"arrival_order_sha256": "68de977e2ea7396ffe9406675a3d2c5ebecd1b9ea5afc8116c836cc53cfca188",
"input_length_order_sha256": "dffa79545e738b899bec2e9d282764a3388fe38b162982fb2d26928e17ef6016",
"offered_req_s": 6.0,
"offered_req_s_per_gpu": 1.5,
"request_id_order_sha256": "4993c26442fc6b5a4c09b6f576fd8b3cb1be7ce83b7a771e4a57ee7600de688d",
"selected_count": 1800,
"target_count": 1800,
"target_req_s_per_gpu": 1.5
},
"mid": {
"anchor": 0.48664343020532463,
"arrival_order_sha256": "efce7339e22d3618cb4d55e6b55bfddb2c563c18faba2a992d5829c13e3f55e9",
"input_length_order_sha256": "0792b05fff6729fbd92ab2bb4cb6d31bea7799e232ad42772936bc06efbafb54",
"offered_req_s": 8.5,
"offered_req_s_per_gpu": 2.125,
"request_id_order_sha256": "2a2fabe2c4cf176aeb7e0d32fb8e7dbb1f27429a2e7a0cd18d7d186f23096f19",
"selected_count": 2550,
"target_count": 2550,
"target_req_s_per_gpu": 2.125
}
},
"source_roles": [
"low3",
"high3"
],
"study": "/home/admin/cpfs/wjh/intervention-response-v3-20260714/private/studies/rep3-tp4.json",
"study_sha256": "fb8ffe256dace32f4ca8a8d49b662d98c3b69b94ecc8fa826e43068b238884ab"
}
},
"sanity": {
"distinct_selected_sets": 6,
"invariants": {
"all_counts_positive": true,
"load_levels_two": true,
"selection_hashes_unique": true,
"selection_sets_disjoint_across_repetitions": true,
"six_sessions": true,
"three_repetitions": true
},
"red_flags": [],
"selected_sets": 6
},
"schema": "intervention-response-phase-aware-pilot-manifest-v3",
"sessions": [
{
"mns": 16,
"replicate": 1,
"session": "rep1-mns16"
},
{
"mns": 64,
"replicate": 1,
"session": "rep1-mns64"
},
{
"mns": 64,
"replicate": 2,
"session": "rep2-mns64"
},
{
"mns": 16,
"replicate": 2,
"session": "rep2-mns16"
},
{
"mns": 16,
"replicate": 3,
"session": "rep3-mns16"
},
{
"mns": 64,
"replicate": 3,
"session": "rep3-mns64"
}
],
"source": {
"base_manifest": "/home/admin/cpfs/wjh/fidelity-prefix-pilot-20260714/runs/p1b/pilot-manifest.json",
"base_manifest_sha256": "86b249a087ab1bcd51cd8de3d1da6bf34afd41af977eb58c93229b941dbf5c8a",
"prior_attempt_failure": [
{
"failure": "TimeoutError('client timeout: tp4_mns16 high')",
"session": "rep1-mns16"
}
],
"prior_attempt_h20_hours": 1.3580546813540988,
"prior_attempt_state": "/home/admin/cpfs/wjh/intervention-response-v2-20260714/runs/pilot/controller-state.json",
"prior_attempt_state_sha256": "c88030ae36862abfc643a855eb3630fb18cd74a58b388514aa31ec1ab485a431",
"source_trace": "/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260312_1000.jsonl",
"source_trace_sha256": "875ba869775deb78086477919f03b322da14e2673c7d070e26528c4190912757",
"window_id": "chat_w20260312_1000"
},
"status": "PASS"
}

46
runs/intervention-response-v2/platform-v3.json Normal file
View File

@@ -0,0 +1,46 @@
{
"schema": "intervention-response-platform-v3",
"host_alias": "dash0",
"hostname": "ds-07429c65-1-6c5fd97778-9vhkr",
"os": "Ubuntu 24.04.2 LTS",
"kernel": "Linux 5.10.134-013.8.2.kangaroo.al8.x86_64 x86_64 GNU/Linux",
"cpu": {
"logical_cpus": 160,
"model": "Intel(R) Xeon(R) Processor"
},
"gpu": {
"count": 8,
"experiment_ids": [
0,
1,
2,
3
],
"memory_mib_each": 97871,
"model": "NVIDIA H20"
},
"runtime": {
"driver": "580.95.05",
"python": "3.12.3",
"torch": "2.11.0+cu130",
"torch_cuda": "13.0",
"vllm_editable_source": "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0",
"vllm_git_head": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
"vllm_git_status_clean": true,
"vllm_import_path": "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/__init__.py",
"vllm_reported_version": "0.24.1.dev3+g668cfb7e2"
},
"model": {
"bytes": 61084188040,
"path": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B"
},
"aituner_commit": "7fd956355072399b1b56946ecd8007da8f9d2fb0",
"experiment": {
"completed_at": "2026-07-14T19:10:53+08:00",
"incremental_h20_hours": 5.092428478929731,
"prior_invalid_attempt_h20_hours": 1.3580546813540988,
"started_at": "2026-07-14T17:54:05+08:00",
"total_h20_hours": 6.45048316028383,
"wall_minutes": 76.79059811433156
}
}