gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add OpProf campaign: protocols, results, patches, run evidence (P0-P6)

Browse Source 
Workload-conditioned operator profiling on patched vLLM 0.24.0 +
Qwen3-30B-A3B/H20. H1b PASS (irregular patterns carry +23-45pp R64
raggedness, 8-45% token-efficiency loss vs rectangular controls);
mechanism decomposition kills the padding narrative and finds the
arrival-uniformization artifact (-12.9%); cross-version churn surface
shows TP2/MNS64 -29.4% across vLLM 0.20->0.24 while the argmax held.
Raw Layer-1 JSONL streams (507 MB) stay on disk, git-ignored; footer
sidecars and metrics are tracked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-07-13 11:06:10 +08:00
parent 607e88da3c
commit d5b276180d
412 changed files with 125056 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

434
docs/opprof/patch-design.md Normal file
View File

@@ -0,0 +1,434 @@
# OpProf dual-layer instrumentation patch design
Status: **design for review; no vLLM patch has been implemented**.
Target source: vLLM `v0.24.0`, commit `ee0da84ab9e04ac7610e28580af62c365e898389`.
Target campaign: Qwen3-30B-A3B serving on NVIDIA H20 (SM90), using the V1 engine.
All source paths and line numbers below are relative to the pinned clone.
## Approved dispositions (2026-07-11)
- Use JSONL with `msgspec`, the proposed context/chunk histogram edges, and an
8192-record bounded queue.
- Keep exact expert loads Layer-2-only.
- Use the community BF16 Qwen3-30B-A3B checkpoint. TP1 is primary, with TP2 and
TP4 counterpoints; record the selected MoE backend logs in every run.
- Use two profiler warm-up iterations followed by eight active iterations.
- Apply the 3% Layer-1 overhead gate to the **upper bound of the 95% confidence
interval** for every primary serving metric.
- Reject `--disable-log-stats` when OpProf is enabled; retain the five-file
fail-fast design.
## Goal and success criteria
The patch should make every serving iteration conditionable on its request and
execution pattern while keeping the always-on path below a 3% serving overhead
budget. Heavy kernel tracing and exact MoE routes are sampled separately.
The proposed split is:
1. **Layer 1:** one compact composition record per scheduler step, emitted by
the scheduler process rather than every tensor-parallel worker.
2. **Layer 2:** short, sampled windows using vLLM's existing torch-profiler
configuration and `/start_profile` and `/stop_profile` endpoints.
The design deliberately does not add hooks to GPU kernels, attention layers,
the Qwen model, or the OpenAI serving API. Those surfaces are not needed to
answer the Phase 0 profiling questions.
## Assumptions and tradeoffs
- The campaign uses the V1 engine and the default vLLM scheduler. Both the
synchronous and `AsyncScheduler` paths inherit the base scheduler hooks.
- The H20 backend statement assumes the usual unquantized BF16/FP16
Qwen3-30B-A3B checkpoint, `moe_backend=auto`, and no LoRA. Quantization,
batched expert format, or an explicit backend can change kernel selection.
- Layer 1 requires normal stats collection, which is enabled by default. If
OpProf is enabled together with `--disable-log-stats`, initialization should
fail with an actionable error. Supporting that unusual combination would
require an extra stats path and a larger patch.
- The context-length histogram records the sequence length at the end of the
scheduled model input (`num_computed_tokens` before scheduling plus tokens
scheduled in this step). This matches the worker-side sequence-length
construction at `vllm/v1/worker/gpu_model_runner.py:2010-2019` and avoids
retaining raw request lengths.
- “Decode tokens” means tokens scheduled for requests classified by vLLM as
generation-phase requests, including scheduled speculative tokens. It is not
the number of subsequently accepted output tokens. The existing classifier
and its chunked-prefill semantics are at `vllm/v1/utils.py:780-813`.
## Existing facilities to reuse
These are zero-patch wins:
| Need | Existing v0.24.0 facility | Design consequence |
|---|---|---|
| Scheduled request and token map | `SchedulerOutput.num_scheduled_tokens`, total tokens, preempted IDs, and new/cached request data at `vllm/v1/core/sched/output.py:180-219` | Derive composition in the scheduler; do not add fields to `SchedulerOutput`. |
| Prefill/decode classification | `compute_iteration_details()` at `vllm/v1/utils.py:780-813` | Reuse the exact definition already used by the iteration log. |
| Queue/KV/prefix stats | `SchedulerStats` at `vllm/v1/metrics/stats.py:170-198` and construction at `vllm/v1/core/sched/scheduler.py:2228-2264` | Reuse field semantics, but snapshot at schedule time to avoid async misattribution. |
| Per-step CUDA-graph descriptor | `CUDAGraphStat` at `vllm/compilation/cuda_graph.py:32-37`, populated at `vllm/v1/worker/gpu_model_runner.py:3919-3934` | Reuse the object; only broaden the condition under which it is returned. |
| Sampled CPU/CUDA tracing | profiler config and schedule at `vllm/config/profiler.py:33-105`, endpoints at `vllm/entrypoints/serve/profile/api_router.py:21-45` | Layer 2 needs configuration and orchestration, not a new profiler implementation or endpoint. |
| Exact routed expert IDs | routed-expert capture callback and buffers at `vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84` and `vllm/v1/worker/gpu_model_runner.py:7382-7437` | Use only in a separate sampled Layer-2 run; it is too expensive for Layer 1. |
## Layer 1: always-on composition records
### Data flow and hook placement
```text
Scheduler.schedule()
-> snapshot request composition, queues, KV and prefix deltas
-> SchedulerOutput -> TP workers -> ModelRunnerOutput.cudagraph_stats
-> Scheduler.update_from_output()
-> finalize the same step record -> bounded queue -> background JSONL writer
```
The scheduler is the ownership boundary for Layer 1. It sees the full logical
batch once, so recording in each TP worker would duplicate data and require an
aggregation protocol.
Exact proposed hooks against the pinned clone:
1. **Initialize one recorder in the base scheduler constructor.** The scheduler
already owns stats state and passes the normal `log_stats` setting into the
KV-cache manager (`vllm/v1/core/sched/scheduler.py:78-87` and
`vllm/v1/core/sched/scheduler.py:250-260`). If `VLLM_OPPROF_DIR` is set,
construct the recorder and reject `log_stats=False`.
2. **Begin a record inside `Scheduler.schedule()`.** Read prefix counters at
method entry, then finish the snapshot immediately after `SchedulerOutput`
and connector metadata are constructed and before
`_update_after_schedule()` mutates request state
(`vllm/v1/core/sched/scheduler.py:1012-1100`). Prefix lookup counters are
updated synchronously during this schedule call
(`vllm/v1/core/kv_cache_manager.py:202-242` and
`vllm/v1/core/sched/scheduler.py:675-712,897-915`), so before/after deltas
remain attributable even when several batches are in flight.
3. **Finalize in `Scheduler.update_from_output()`.** Pair the returned
`ModelRunnerOutput` with the exact `SchedulerOutput` at
`vllm/v1/core/sched/scheduler.py:1464-1477`, and enqueue after normal output
processing near `vllm/v1/core/sched/scheduler.py:1791-1803`. Store pending
drafts by `id(scheduler_output)` and remove them on finalize; the identifier
never leaves the process. This is robust to the batch queue, which retains
and later returns the matching output object
(`vllm/v1/engine/core.py:519-632`).
4. **Return the existing CUDA-graph stat when OpProf is enabled.** Change the
condition at `vllm/v1/worker/gpu_model_runner.py:3919-3926` from only
`observability_config.cudagraph_metrics` to that flag **or**
`VLLM_OPPROF_DIR`. The file already imports `vllm.envs` at line 23. No CUDA
synchronization or tensor transfer is introduced.
5. **Close through the existing scheduler shutdown.** Drain and join the writer
before the scheduler reports shutdown complete at
`vllm/v1/core/sched/scheduler.py:2285-2295`; retain an `atexit` fallback for
abnormal embedding/test lifecycles.
The scheduler assigns a monotonically increasing local `step_index` at begin.
Every real scheduler call is represented, including zero-token steps. A DP-only
dummy execution has no `SchedulerOutput` and is outside this composition
stream; if DP is later in campaign scope, add a distinct `dummy_step` marker
rather than pretending it has a request composition.
### Record schema
Use schema-versioned JSON Lines. An illustrative record is:
```json
{"schema":1,"engine_id":"dp0-pid1234","step_index":42,"submit_wall_ns":1783761000000000000,"submit_mono_ns":8920000000,"complete_mono_ns":8922371000,"model_executed":true,"scheduled_requests":37,"decode_batch_size":32,"prefill_requests":5,"prefill_tokens":896,"decode_tokens":32,"chunked_prefill":{"first":2,"middle":1,"final":1,"unsplit":1,"tokens":896,"chunk_size_hist":[0,0,1,1,1,1,1,0,0]},"context_length_hist":[0,0,3,6,12,10,5,1,0,0,0,0],"preemptions":0,"queues":{"running":37,"waiting":9,"deferred":0},"kv":{"total_blocks":120000,"free_blocks":42000,"used_blocks":78000,"usage":0.65},"prefix":{"local":{"requests":2,"queries":4096,"hits":3072,"preempted_requests":0,"preempted_queries":0,"preempted_hits":0},"external":null},"cudagraph":{"hit":true,"runtime_mode":"PIECEWISE","unpadded_tokens":928,"bucket_tokens":1024,"padding_tokens":96},"moe_expert_load":null,"dropped_records_before":0}
```
Required field semantics:
- `submit_wall_ns` permits joining to external workload logs;
`submit_mono_ns` and `complete_mono_ns` provide stable within-process
ordering and elapsed time without wall-clock jumps.
- `scheduled_requests` is the number of entries in
`num_scheduled_tokens`. `decode_batch_size`, `prefill_requests`,
`prefill_tokens`, and `decode_tokens` reuse vLLM's classifier.
- `chunked_prefill` contains aggregate split information, never request IDs.
A context request is `first` when the scheduled range ends before its prompt
ends, `middle` when it was already a prefill chunk and remains incomplete,
`final` when a prior chunk completes, and `unsplit` when its prefill completes
in one step. `chunk_size_hist` uses token-count buckets
`(16, 32, 64, 128, 256, 512, 1024, 2048, +inf)`.
- `context_length_hist` includes every scheduled request, with fixed upper
edges `(128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536,
131072, +inf)`. The final array therefore has 12 bins. The example above is
illustrative and will be checked against the final chosen edges in tests.
- `preemptions` is the size of the step's `preempted_req_ids`, created at
`vllm/v1/core/sched/scheduler.py:1059-1076`. It is not an interval-derived
Prometheus value.
- Queue lengths and KV blocks are captured immediately after scheduling. KV
usage follows the existing null-block-adjusted calculation at
`vllm/v1/core/block_pool.py:692-711`. `deferred` means skipped/deferred
waiting requests, matching `num_skipped_waiting_reqs` in
`vllm/v1/metrics/stats.py:174-183`.
- Local and external prefix fields are per-schedule deltas copied from the
existing mutable counters. The existing stats drain resets those objects at
`vllm/v1/core/kv_cache_manager.py:190-200` and
`vllm/v1/core/sched/scheduler.py:2235-2242`; OpProf must read, not drain or
replace, them. These retain vLLM's lookup semantics: a local lookup can be
counted before allocation later rejects that waiting request, so prefix
`requests` is not required to be less than or equal to scheduled prefill
requests.
- `cudagraph.hit` is the operational serving definition `runtime_mode != NONE`.
`FULL` selects the full graph path and `PIECEWISE` selects graph-wrapped
compiled regions; the latter must not be interpreted as full-step graph
coverage. Startup normally captures all dispatcher descriptors at
`vllm/v1/worker/gpu_model_runner.py:6595-6643`, but the stat itself has no
capture-versus-replay bit, so a rare lazy first capture cannot be
distinguished from replay. `bucket_tokens` is `num_padded_tokens` in the
existing stat.
- `moe_expert_load` is explicitly null in Layer 1. This avoids accidentally
presenting unavailable data as zero.
### Encoding, buffering, and flush
JSONL is recommended over a compact binary ring buffer for Phase 1 because the
schema will evolve during campaign bring-up, records need to be directly
inspectable beside Kineto traces, and `msgspec` is already a vLLM dependency
(`requirements/common.txt:35`).
Encoding one small dictionary with a reused `msgspec.json.Encoder` avoids the
standard `json` module's larger CPU cost. Once the schema stabilizes, the same
record can be moved to a binary format only if Phase 2 measurements show JSONL
is the bottleneck.
The foreground path encodes once and performs non-blocking `put_nowait()` into
a bounded queue of 8192 records. A single daemon writer thread drains into one
file per EngineCore/DP rank and process. It flushes userspace buffers every
1 MiB or one second, whichever comes first, and on clean process exit; it never
calls `fsync()` per record. When the queue is full, the serving thread drops the
new record, increments a counter, and the next successful record reports the
gap in `dropped_records_before`. Shutdown emits a footer with encoded, written,
and dropped counts. The file name includes schema, DP rank, PID, and start time,
but not TP rank because there is one scheduler record stream.
The sole on/off switch is:
```text
VLLM_OPPROF_DIR=/absolute/output/directory
```
Unset or empty means a true no-op: no recorder, queue, thread, record
construction, or graph-stat broadening. Directory validation and a clear
startup log happen before serving. Runtime toggling is intentionally omitted;
it adds synchronization and ambiguous partial files without helping the
campaign.
### Expected cost and Phase 2 overhead gate
These are estimates, not measurements:
- composition and two fixed histograms: about 20-80 microseconds per step,
linear in scheduled requests;
- `msgspec` encoding plus a non-blocking queue insertion: about 5-20
microseconds per step;
- CUDA-graph stat construction: below 2 microseconds and no GPU sync;
- disk I/O: off the serving critical path, subject to bounded-queue drops.
The expected foreground total is roughly 25-100 microseconds per step. At a
2 ms decode step, the high end would exceed 3%, so the budget is a measurement
gate, not a claim.
Phase 2 should use the same Qwen3 checkpoint, request trace, random seed,
hardware, vLLM commit, TP/EP topology, CUDA-graph configuration, and cache
state for off/on comparisons. Alternate off/on ordering, warm up before each
measurement, and run at least five paired repeats. Report throughput and
p50/p95 TTFT and TPOT, plus CPU utilization, log bytes per step, queue high
watermark, and dropped-record count. The acceptance rule is less than 3%
regression for every declared primary serving metric, with bootstrap confidence
intervals reported. Whether the point estimate or upper 95% bound is the hard
gate is an open decision.
Correctness checks for every run are: contiguous step indices except explicitly
reported drops, scheduled token sums matching the prefill/decode split,
histogram counts matching scheduled request/chunk counts, KV ratio in `[0,1]`,
non-negative counters, and identical generated outputs for the deterministic
test trace.
## Layer 2: sampled kernel windows
### Trigger and window
Use the existing profiler API with a run-specific output directory and
`ignore_frontend=true`. The worker start/stop RPC already fans out to all
workers (`vllm/v1/executor/abstract.py:256-257` and
`vllm/v1/executor/multiproc_executor.py:340-402`), and each GPU worker creates a
CPU+CUDA profiler with a rank-qualified trace name
(`vllm/v1/worker/gpu_worker.py:929-980`). No Layer-2 vLLM code patch is needed.
Recommended initial configuration:
```text
profiler=torch
torch_profiler_dir=<run>/kineto
ignore_frontend=true
delay_iterations=0
max_iterations=8
wait_iterations=0
warmup_iterations=2
active_iterations=8
torch_profiler_record_shapes=false
torch_profiler_with_memory=false
torch_profiler_with_stack=false
torch_profiler_with_flops=false
torch_profiler_use_gzip=true
torch_profiler_dump_cuda_time_total=true
```
The primary trigger should be step-count based: an external campaign controller
POSTs `/start_profile` immediately before a desired composition regime. The
built-in schedule records two warm-up plus eight active model iterations and
`max_iterations=8` then stops the underlying profiler on the following worker
step. The “exceeds max” control semantics are explicit at
`vllm/profiler/wrapper.py:83-114` and in
`tests/v1/worker/test_gpu_profiler.py:77-98`; the controller should still POST
`/stop_profile` afterward to clear the active control state.
Worker iteration boundaries already call `profiler.step()` and annotate context
and generation token counts at `vllm/v1/worker/gpu_worker.py:803-827`. On-demand
manual POST remains useful for debugging. A time-based trigger is a fallback
for live traffic but is less reproducible because it yields a variable number
of steps. Phase 1 should confirm that the trace contains exactly eight active
iteration annotations; profiler scheduling and trace callbacks live at
`vllm/profiler/wrapper.py:159-226,290-307`.
The profile directory should be unique per campaign server run. The HTTP start
endpoint does not accept a `profile_prefix`, and a worker that is restarted for
another window retains the trace name chosen on first initialization
(`vllm/v1/worker/gpu_worker.py:939-975`). Multiple windows in one server run
therefore share a directory and need a controller manifest containing start/
stop wall times and produced filenames. Use a new directory only when the
server is restarted; adding a new endpoint parameter is not justified for
Phase 1. Every TP worker emits its own trace, and the rank suffix contains DP,
PP, TP, DCP, EP, and global rank information
(`vllm/distributed/utils.py:664-695`). Keep these traces separate and aggregate
only offline.
### CUDA graphs and Kineto interpretation
Keep CUDA graphs enabled in the primary sampled windows so kernel time reflects
the serving configuration. The pinned code profiles CUDA activity
(`vllm/v1/worker/gpu_worker.py:953-960`) while a captured path invokes
`CUDAGraph.replay()` instead of rerunning the Python callable
(`vllm/compilation/cuda_graph.py:233-360`). Therefore Kineto/CUPTI can observe
CUDA activity launched during replay, but the Python/PyTorch operator scopes
that created the graph are not re-executed and cannot be assumed to retain
per-layer attribution. vLLM explicitly documents the related limitation that
layerwise NVTX tracing does not work with CUDA graphs
(`vllm/config/observability.py:60-63`).
The clone does not contain a stronger guarantee about whether a particular
PyTorch/CUPTI build will present every graph node as an individually named
kernel versus a coarser graph-launch view. Treat that as an empirical Phase 2
check. Run one matched eager (`cudagraph_mode=NONE`) taxonomy window to identify
kernel families, but do not use its timings as the production baseline.
### Calibration and MoE routing
Join each profiler iteration annotation to Layer-1 records by rank-independent
step order, the profiler window marker, timestamps, and the prefill/decode token
counts. Aggregate kernels into attention, router/top-k, expert GEMMs, dense
GEMMs, normalization/activation, collectives, and sampling. For TP, report both
sum-of-rank GPU work and the maximum per-rank critical-path time. Fit or tabulate
kernel-time attribution conditioned on composition, context histogram, graph
runtime mode, and capture bucket, then validate on held-out windows rather than
the same samples used for calibration.
For the Phase 2 “operator time approximately equals iteration time” gate, do
not naively add overlapping kernels. Compute the union of CUDA kernel intervals
per rank, use the maximum rank as the distributed GPU critical path, and retain
an explicit CPU/queue/unattributed residual against Layer 1's submit-to-complete
span. Operator-category interval unions plus the residual must reconstruct the
span; summed GPU work is reported separately and may legitimately exceed wall
time.
Exact per-layer expert loads are Layer-2-only. A separate server start with
`--enable-return-routed-experts` exposes per-token, per-layer top-k IDs through
the existing router callback (`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).
Aggregate those arrays offline into per-layer expert counts, entropy, max/mean
load, coefficient of variation, and top-k imbalance. Do not enable it in the
normal Layer-1 or baseline Kineto run: its worker transit buffer costs a few MB,
and the scheduler-side slot buffer can reach multiple GB with a CPU fancy-index
copy per step (`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`).
## Patch surface estimate
Proposed implementation and tests:
| File | Approximate changed/new lines | Purpose |
|---|---:|---|
| `vllm/envs.py` | 4 | Declare and parse `VLLM_OPPROF_DIR`. |
| `vllm/v1/opprof.py` (new) | 220 | Schema, schedule snapshot/deltas, histograms, pending-step pairing, encoder, bounded writer, drop/footer handling. |
| `vllm/v1/core/sched/scheduler.py` | 32 | Initialize recorder; begin before state mutation; finalize with the matching model output. |
| `vllm/v1/worker/gpu_model_runner.py` | 4 | Return existing `CUDAGraphStat` when either built-in metrics or OpProf needs it. |
| `tests/v1/core/test_opprof.py` (new) | 190 | Schema/invariants, chunk classification, async pairing, disabled no-op, bounded-queue drops, shutdown flush. |
| **Total** | **5 files / about 450 lines** | About 260 production lines and 190 test lines. |
No patch is proposed for `ProfilerConfig`, profile endpoints, profiler wrapper,
`SchedulerOutput`, `SchedulerStats`, Prometheus/logging, CUDA kernels, fused MoE
kernels, the Qwen model, or frontend APIs. If support for
`--disable-log-stats` is required, add about 12 lines in
`vllm/v1/core/kv_cache_manager.py`, making the estimate six files and about 462
lines; the smaller fail-fast design is recommended for this campaign.
## Risks and mitigations
- **Foreground CPU overhead:** request scanning and JSON encoding are the likely
hot spots. Use fixed-size arrays, a reused encoder, no raw lists, no blocking
I/O, and enforce the Phase 2 gate.
- **Async-scheduling races:** queue/KV/prefix state observed during
`update_from_output()` may include later in-flight schedules. Snapshot it
inside the synchronous `schedule()` call; only the immutable returned
CUDA-graph stat is added at completion. Pair by the exact `SchedulerOutput`
object and assert one begin/one finalize.
- **TP and EP aggregation:** Layer 1 is scheduler-owned and emitted once. Layer
2 remains per rank. Interpret TP critical path as the slowest rank and, for an
expert-parallel routed-expert sample, aggregate expert ownership offline.
- **Log volume:** at 500 steps/s, a 1.0 KiB record is about 44 GB/day per
EngineCore. Rotate by campaign run, compress completed files, and use the
bounded queue/drop counters. Production-long captures need a later sampling
or rotation policy; Phase 1 files should be bounded by run duration.
- **Graph semantics:** `PIECEWISE` is partial graph coverage, not a binary full
hit. Preserve `runtime_mode`, unpadded tokens, bucket, and padding rather than
reducing the record to one boolean.
- **Profiler perturbation:** Kineto has medium-to-high overhead and trace flushes
can stall. Use short windows, unique directories, disabled stacks/shapes and
memory by default, and never use Layer-2 latency as an unprofiled performance
result.
- **Backend drift:** the unquantized SM90 oracle prioritizes Triton, but
shape-specific fallbacks remain possible. Record startup backend logs and the
full checkpoint/parallel/quantization configuration with every run.
## Open decisions for review
1. Approve JSONL with `msgspec`, the two proposed histogram edge sets, and a
bounded 8192-record writer queue.
2. Approve exact expert-load collection as Layer-2-only, rather than adding a
new always-on GPU histogram kernel.
3. Confirm the Qwen3 checkpoint precision/quantization and intended TP/EP/DP
topology; the Triton backend conclusion depends on these inputs.
4. Choose the first profiler window: recommended two warm-up plus eight active
iterations, or a longer active window.
5. Decide whether the 3% gate applies to metric point estimates or to the upper
bound of their 95% confidence intervals.
6. Confirm that campaign runs may reject `--disable-log-stats`; supporting it
adds one file and about 12 lines.
## Data sanity block
- **Patch estimate:** n=5 files; per-file line-estimate min=4, max=220,
distinct values=4; total about 450 lines, of which about 260 are production
and 190 are tests.
- **Histogram definitions:** n=2 arrays; bin-count min=9, max=12,
distinct=2. In the illustrative record, context-bin sum=37 equals scheduled
requests, chunk-bin sum=5 equals prefill requests, KV used=120000-42000, and
graph padding=1024-928.
- **Estimated foreground components:** n=3 timed components; min is below 2
microseconds for graph-stat construction, max is 80 microseconds for
composition; distinct ranges=3. Their conservative combined foreground
estimate is 25-100 microseconds per step.
- **Invariants checked:** counters and histogram bins are non-negative; KV
usage and prefix hit ratios are bounded in `[0,1]`; graph bucket is at least
unpadded tokens; `NONE` is the only graph miss mode; no raw request IDs or
context-length lists are serialized; one scheduler stream avoids TP
duplicates.
- **Measurement status:** n=0 benchmark runs; min/max and cross-configuration
distinctness are not applicable. The cost figures are estimates and cannot
support an overhead conclusion until Phase 2.

446
docs/opprof/phase0-recon-vllm-0.24.0.md Normal file
View File

@@ -0,0 +1,446 @@
# Phase 0 recon: vLLM 0.24.0 observability
Status: source recon and patch design complete; **no instrumentation patch was
implemented**.
Campaign target: operator-level, pattern-conditioned profiling of
Qwen3-30B-A3B serving on NVIDIA H20. This report is based on the pinned source
clone only. No GPU was used, no package or model was installed, and no dash
host was contacted.
Unless explicitly prefixed with `v0.20.0:`, every source path and line number
below is relative to `/home/gahow/phd/vllm-v0.24.0` at the pinned commit.
## Outcome
vLLM 0.24.0 already contains most of the raw ingredients, but they are split
across scheduler internals, aggregate metrics, worker traces, and an optional
routed-expert return path. The smallest useful patch is therefore not a new
profiler. It is one scheduler-owned per-step record that joins existing batch
composition to the existing `CUDAGraphStat`; short torch-profiler windows and
exact expert routes remain sampled Layer 2 facilities.
The proposed patch is detailed in
[`patch-design.md`](patch-design.md). Estimated surface: five files and about
450 lines including tests, with about 260 production lines.
## 1. Source pin and compatibility record
The requested tag existed and the exact requested clone command completed
successfully:
```text
git clone --depth 1 --branch v0.24.0 https://github.com/vllm-project/vllm /home/gahow/phd/vllm-v0.24.0
```
| Field | Pinned value |
|---|---|
| Repository | `https://github.com/vllm-project/vllm` |
| Local clone | `/home/gahow/phd/vllm-v0.24.0` |
| Tag | `v0.24.0` (`git describe --tags --exact-match`) |
| Commit | `ee0da84ab9e04ac7610e28580af62c365e898389` |
| Clone completion | `2026-07-11T08:26:16Z` (`2026-07-11 16:26:16 +08:00`) |
| Checkout | Detached HEAD, clean |
| Top-level entries | 45, including hidden entries and `.git` |
| Listing SHA-256 | `75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58` |
The listing digest is reproducible from entry names only, not file contents:
```text
LC_ALL=C find . -mindepth 1 -maxdepth 1 -printf '%f\n' |
LC_ALL=C sort | sha256sum
```
This definition includes `.git`, sorts bytewise under the C locale, and hashes
the 45 LF-terminated root names. It is recorded to make the otherwise ambiguous
phrase “sha256 of the top-level directory listing” precise.
### Torch and CUDA requirements
- Build and NVIDIA runtime requirements pin `torch==2.11.0` in
`pyproject.toml:1-13`, `requirements/build/cuda.txt:1-10`, and
`requirements/cuda.txt:1-10`; CMake repeats 2.11.0 as the supported CUDA and
ROCm torch version at `CMakeLists.txt:62-72`.
- The NVIDIA requirements also pin `torchvision==0.26.0`,
`flashinfer-python==0.6.12`, `flashinfer-cubin==0.6.12`, and CUDA-13 extras
for CUTLASS DSL and Humming kernels (`requirements/cuda.txt:6-29`). Setup
strips/substitutes those extras for CUDA 12 builds
(`setup.py:1066-1083`).
- The source default is CUDA 13.0 (`vllm/envs.py:85-87,555-563`) and the Docker
default is CUDA 13.0.2 (`docker/Dockerfile:14-42`). Wheel detection maps CUDA
major 12 to `cu129` and major 13 to `cu130` (`setup.py:528-568`).
- The pinned installation document still describes the default precompiled
binaries as CUDA 12.9 and lists CUDA 12.8 and 13.0 variants
(`docs/getting_started/installation/gpu.cuda.inc.md:4-19,24-54`). This is a
packaging/default distinction, not evidence that 13.0 is unsupported.
- NVIDIA compute capability 7.5+ is documented, and SM90 is in every relevant
CMake CUDA architecture set (`docs/getting_started/installation/gpu.cuda.inc.md:9-19`;
`CMakeLists.txt:105-118`). Optional FA3 and DeepGEMM builds need CUDA 12.3+,
FlashMLA needs 12.9+, and the Hopper CUTLASS MoE build needs 12.3+
(`setup.py:1111-1139`; `CMakeLists.txt:844-868`).
Implication for dash0 later: H20/SM90 is a supported architecture, but the
actual driver and installed CUDA were intentionally not queried in Phase 0.
Phase 2 must choose a torch 2.11.0-compatible `cu129` or `cu130` artifact after
checking the host driver. A wheel built against a different torch/CUDA build is
not interchangeable; the source documentation explicitly warns about binary
incompatibility (`docs/getting_started/installation/gpu.cuda.inc.md:14-19`).
## 2. Built-in observability inventory
### 2.1 Torch profiler integration
**Configuration and control.** v0.24.0 does not contain the legacy
`VLLM_TORCH_PROFILER_DIR` symbol: a repository-wide exact-name search at the
pinned commit returned zero matches. Profiling is now configured through
`ProfilerConfig`: `profiler` is `torch` or `cuda`, and
`torch_profiler_dir` holds the absolute/URI output location
(`vllm/config/profiler.py:33-46,125-145`). Optional controls include stacks,
FLOPs, gzip, CUDA-time tables, shapes, memory, frontend exclusion, delayed
start, maximum iterations, and wait/warmup/active iteration scheduling
(`vllm/config/profiler.py:48-105`).
The documented server form is a `--profiler-config` JSON object, for example
`{"profiler":"torch","torch_profiler_dir":"/abs/run/kineto"}`, followed by
POSTs to the two endpoints (`docs/contributing/profiling.md:46-85`).
When profiler config is present, the serving router exposes POST
`/start_profile` and `/stop_profile`; without profiler config the router is not
attached (`vllm/entrypoints/serve/profile/api_router.py:21-45`). The endpoints
are therefore not an unauthenticated always-on feature independent of server
configuration; they exist within the configured serving application and should
be protected like the rest of the control plane.
**Captured activity.** Each GPU worker creates a torch profiler with both CPU
and CUDA activities, using the configured shapes/memory/stack/FLOPs switches
(`vllm/v1/worker/gpu_worker.py:929-980` and
`vllm/profiler/wrapper.py:159-226`). TensorBoard-compatible traces are written
with an optional gzip handler. The wrapper also writes CPU/CUDA aggregate
tables; only rank 0 prints the table to stdout
(`vllm/profiler/wrapper.py:251-287`). The source documentation calls the
profiler medium-overhead and warns that trace size and final flushing can be
large (`docs/contributing/profiling.md:1-38`).
The V1 worker calls `profiler.step()` once per execution and wraps model
execution in a record-function name containing context/generation request and
token counts (`vllm/v1/worker/gpu_worker.py:803-827,895-898`). If frontend
profiling is not ignored, `AsyncLLM` also creates a separate CPU-only trace
(`vllm/v1/engine/async_llm.py:178-200`).
**TP behavior.** `EngineCore.profile()` delegates to the executor
(`vllm/v1/engine/core.py:662-663`), whose profile RPC is a collective worker
call (`vllm/v1/executor/abstract.py:256-257`). Multiprocessing broadcasts calls
that do not declare a unique output rank, so the profile call reaches every TP
worker (`vllm/v1/executor/multiproc_executor.py:340-402`). Each worker emits its
own rank-qualified trace; the suffix includes DP/PP/TP/DCP/EP/global rank
components (`vllm/distributed/utils.py:664-695`). For TP, kernel analysis must
therefore preserve per-rank files, use the slowest rank for critical-path time,
and optionally sum ranks to quantify total GPU work.
### 2.2 Iteration-level scheduler data and exported metrics
The V1 scheduler interface states that each `schedule()` output corresponds to
one model forward and maps request IDs to scheduled tokens
(`vllm/v1/core/sched/interface.py:51-80`). `SchedulerOutput` already carries
new/cached request data, per-request and total scheduled tokens, common-prefix
blocks, and the request IDs preempted in that step
(`vllm/v1/core/sched/output.py:180-219`). The base scheduler constructs this
object at `vllm/v1/core/sched/scheduler.py:1012-1076`, then advances request
computed-token and chunk state at `vllm/v1/core/sched/scheduler.py:1130-1155`.
What is available today:
| Requested datum | Internal per-step source | Externally exposed today |
|---|---|---|
| Batch size | `len(SchedulerOutput.num_scheduled_tokens)`; the map is exact | No raw per-step batch-size Prometheus series. Optional iteration log reports context and generation request counts. |
| Scheduled prefill/decode tokens | `compute_iteration_details()` classifies new/cached context requests and sums scheduled tokens (`vllm/v1/utils.py:766-813`) | Optional log reports exact scheduled context/generation counts and elapsed time around the execution wait (`vllm/v1/engine/core.py:433-472`). Prometheus prompt/generation counters are cumulative output-side stats, not a raw scheduled-step stream. |
| Chunked-prefill splits | `Request.is_prefill_chunk`, `num_computed_tokens`, and prompt/output state exist (`vllm/v1/request.py:130-175`); extended chunks are classified as context | No chunk ordinal, first/middle/final split, or chunk-size series. These must be derived before `_update_after_schedule()` mutates state. |
| Preemptions | Exact step set in `SchedulerOutput.preempted_req_ids` | Stdout aggregates over its logging interval; `vllm:num_preemptions` is cumulative (`vllm/v1/metrics/loggers.py:219-281,624-631`). |
| Running/waiting/deferred queues | `SchedulerStats.num_running_reqs`, `num_waiting_reqs`, and `num_skipped_waiting_reqs` (`vllm/v1/metrics/stats.py:170-183`) | Latest-value gauges `vllm:num_requests_running`, `vllm:num_requests_waiting`, and reason-labeled capacity/deferred gauges (`vllm/v1/metrics/loggers.py:453-496`). Running queue size is not the same as scheduled batch size. |
| KV-cache usage | Scheduler reads `kv_cache_manager.usage`; the block pool computes a null-block-adjusted ratio in `[0,1]` (`vllm/v1/core/kv_cache_manager.py:181-188`; `vllm/v1/core/block_pool.py:692-711`) | Latest-value `vllm:kv_cache_usage_perc`; stdout prints the latest percent (`vllm/v1/metrics/loggers.py:250-260,524-532`). Raw total/free blocks are not exported per step. |
| Prefix-cache counters | `PrefixCacheStats` contains request/query/hit and separately preempted request/query/hit counters (`vllm/v1/metrics/stats.py:114-143`) for local and optional connector caches | Prometheus exposes cumulative token queries/hits for local and external caches (`vllm/v1/metrics/loggers.py:547-565,1063-1101`); stdout reports interval-derived hit rates. It does not expose the preempted sub-counters separately. |
| Total step tokens | Exact scheduled total is in `SchedulerOutput` | `vllm:iteration_tokens_total` is a histogram observed from output-side computed prompt plus generation tokens (`vllm/v1/metrics/loggers.py:693-724,1148-1171`), not a joinable raw step record. |
`IterationStats` also has a wall timestamp, output-side prompt/generation data,
preemptions, and finished-request latency samples
(`vllm/v1/metrics/stats.py:325-347`). `Scheduler.make_stats()` produces latest
queue/KV state, drains prefix stats, and passes through the step's CUDA-graph
stat when an output is processed
(`vllm/v1/core/sched/scheduler.py:2228-2264`). These are valuable existing
plumbing, but async scheduling can schedule newer batches before an older
output is processed. Consequently, those latest/drained values are not a safe
per-step composition join without a schedule-time snapshot.
Bottom line: no new scheduler instrumentation is needed to discover batch and
token composition, preemptions, queues, KV use, or prefix events. A patch is
needed only to serialize their **same-step association** and the missing
histograms/chunk split.
### 2.3 CUDA-graph observability
Yes, v0.24.0 can determine the runtime graph mode and capture bucket for each
model step internally:
- `CUDAGraphStat` records unpadded tokens, padded tokens, padding, and runtime
mode (`vllm/compilation/cuda_graph.py:32-37`).
- The GPU model runner asks the dispatcher for a concrete runtime mode and
`BatchDescriptor`, then conditionally attaches the stat to its output
(`vllm/v1/worker/gpu_model_runner.py:3822-3934`). It reaches the scheduler in
`ModelRunnerOutput` and `update_from_output()`
(`vllm/v1/outputs.py:231-281`;
`vllm/v1/core/sched/scheduler.py:1464-1477`).
- The dispatcher is the source of truth: it pads to a captured descriptor,
tries `FULL`, then `PIECEWISE`, and returns `NONE` when no graph key matches
(`vllm/v1/cudagraph_dispatcher.py:235-324`; the design explanation is at
`docs/design/cuda_graphs.md:81-120`). At serving time, `FULL` and `PIECEWISE`
dispatch to a captured wrapper; `NONE` calls the runnable directly
(`vllm/compilation/cuda_graph.py:233-360`).
Thus `runtime_mode != NONE` is a graph-path dispatch, and `num_padded_tokens` is
the selected token bucket. `PIECEWISE` means graph-wrapped compiled pieces, not
a full-step graph hit. Startup normally captures all dispatcher descriptors
(`vllm/v1/worker/gpu_model_runner.py:6595-6643`), but `CUDAGraphStat` has no
capture-versus-replay bit; a lazy first capture and a replay have the same stat.
With built-in `cudagraph_metrics`, the stat is aggregated
into a frequency table and printed at the logging interval
(`vllm/config/observability.py:56-58`;
`vllm/compilation/cuda_graph.py:40-124`); it is not exposed as a raw per-step
Prometheus event. OpProf only needs to preserve the already-computed object on
every enabled step.
Capture modes and sizes live in `CompilationConfig`:
`cudagraph_mode`, `cudagraph_capture_sizes`, and
`max_cudagraph_capture_size` are defined at
`vllm/config/compilation.py:587-688`. The V1 default is
`FULL_AND_PIECEWISE`. Final sizes are generated or validated in
`VllmConfig._set_cudagraph_sizes()`; the default is 1/2/4, then steps of 8 below
256 and 16 above it, bounded by token/batch configuration, with user overrides
written back to the final list (`vllm/config/vllm.py:1635-1800`).
### 2.4 MoE routing and Qwen3-30B-A3B backend
Qwen3 MoE creates a replicated gate and `FusedMoE` with the model's number of
experts and top-k, and normally runs its router internally
(`vllm/model_executor/models/qwen3_moe.py:137-249`). The causal-LM wrapper
retains the list of MoE layers and their expert topology
(`vllm/model_executor/models/qwen3_moe.py:662-729`). Router top-k IDs are
available immediately after routing and before EPLB remapping
(`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`).
There is an exact, built-in access path: `--enable-return-routed-experts`
(`vllm/config/model.py:209-215`; `vllm/engine/arg_utils.py:826-830`). It installs
per-layer callbacks, writes top-k IDs to preallocated GPU buffers, copies the
step to pinned CPU memory, and returns it through `ModelRunnerOutput`
(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84`;
`vllm/v1/worker/gpu_model_runner.py:3652-3666,4637-4683,7382-7437`). This is
reachable without modifying the fused kernels or saving router logits.
Raw router-logit tensors are not plumbed to engine outputs; exposing those
would be deeper and much higher-volume surgery than using the existing top-k
ID callback.
It is not cheap enough for Layer 1. The per-worker transit buffer is a few MB;
the scheduler's whole-slot buffer can reach multiple GB and performs a CPU
fancy-index write each step
(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`;
the scheduler store is at `vllm/v1/core/sched/scheduler.py:1501-1520`). It is
also incompatible with pipeline parallelism greater than one and with KV
connectors (`vllm/config/vllm.py:871-893`). Exact per-layer expert load should
therefore be collected in a separate sampled Layer-2 run and reduced offline.
EPLB has load state, but enabling it adds communication and its logs are
aggregate balancing summaries; it is not a free observability tap
(`vllm/distributed/eplb/eplb_state.py:62-171,478-575`).
For the normal unquantized BF16/FP16 Qwen3-30B-A3B configuration on H20/SM90,
the automatic unquantized oracle explicitly moves both FlashInfer backends
behind Triton on Hopper because they are expected to be slower
(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:43-86`). It selects
the first supported backend and documents possible shape-specific fallbacks
(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:152-250`). The
primary backend is therefore **Triton** under those assumptions. A quantized
checkpoint, LoRA, expert activation format, or explicit `--moe-backend` can
change that conclusion and must be recorded with the campaign run.
### 2.5 V1 execution loop and natural hooks
The control path is:
```text
EngineCore busy loop
-> Scheduler.schedule()
-> executor.execute_model(SchedulerOutput)
-> GPUWorker -> GPUModelRunner.execute_model()
-> Scheduler.update_from_output(SchedulerOutput, ModelRunnerOutput)
```
Evidence and hook implications:
- `EngineCore.run_busy_loop()` and `_process_engine_step()` drive steps at
`vllm/v1/engine/core.py:1259-1317`.
- The normal step retains the exact scheduler output across the future and
passes it back to `update_from_output()`
(`vllm/v1/engine/core.py:479-508`). The batch-queue path does the same for
multiple in-flight batches (`vllm/v1/engine/core.py:519-632`).
- The scheduler's stable interface is schedule/update at
`vllm/v1/core/sched/interface.py:51-107`. The base implementation builds the
output at `vllm/v1/core/sched/scheduler.py:1012-1100`, mutates request state
immediately afterward at lines 1130-1155, and consumes model results at
lines 1464-1803.
- Worker execution wraps `gpu_model_runner.execute_model()` at
`vllm/v1/worker/gpu_worker.py:895-898`; the model runner's entry is
`vllm/v1/worker/gpu_model_runner.py:4055-4069`.
The natural Layer-1 hooks are therefore in the base scheduler: snapshot after
the `SchedulerOutput` is complete but before request state advances, then
finalize with the matching model output. This covers the async subclass and
avoids frontend/output aggregation. The only worker change is broadening the
existing CUDA-graph-stat condition. Layer 2 uses the existing profile control
path unchanged.
### 2.6 Relevant changes from 0.20.0
For comparison only, tag `v0.20.0` was fetched into the allowed clone without
changing the pinned working tree. It resolves to commit
`88d34c6409e9fb3c7b8ca0c04756f061d2099eb1`.
- Async scheduling and the batch-queue architecture already existed in 0.20.0;
`SchedulerConfig` selected `AsyncScheduler` at
`v0.20.0:vllm/config/scheduler.py:146-176`, and the core schedule/future/update
paths were at `v0.20.0:vllm/v1/engine/core.py:402-431,443-533`. It is
incorrect to treat async scheduling as new in 0.24.0.
- 0.24.0 adds DP prefill cadence/throttling and passes a prefill-throttle flag
into `schedule()` (`vllm/config/scheduler.py:140-161` and
`vllm/v1/engine/core.py:474-490`). It also hardens zero-token/dummy-iteration
handling (`vllm/v1/engine/core.py:433-472`) and changes when the batch queue
fills.
- The 0.24 scheduler contains explicit scheduled/processed sequence fences and
snapshots state that must survive later async schedules, including routed
experts (`vllm/v1/core/sched/scheduler.py:293-322,1093-1165`). That is direct
evidence that an OpProf hook must not read mutable request/queue state only
when an older output returns.
- `vllm/config/profiler.py`, `vllm/profiler/wrapper.py`, and
`vllm/entrypoints/serve/profile/api_router.py` are byte-identical between the
two tags (`git diff --quiet v0.20.0..v0.24.0` returned success). Profiler
endpoint semantics did not move; the scheduler/core line numbers and safe
composition hook did.
Practical conclusion: do not port a 0.20 line-number patch into EngineCore.
Hook the 0.24 base scheduler's schedule/update pair so sync, async, and batch
queue paths share one implementation.
## 3. Proposed dual-layer design
Layer 1 emits one schema-versioned JSONL record per scheduler step from the
EngineCore/scheduler process. It contains step/timestamps; scheduled and decode
batch counts; scheduled prefill/decode tokens; aggregate first/middle/final
chunk information; fixed-bucket context and chunk-size histograms; exact step
preemptions; schedule-time running/waiting/deferred queues; total/free/used KV
blocks and ratio; local/external prefix deltas; and CUDA-graph mode/bucket. No
request IDs or raw context-length lists are written. MoE expert load is null and
explicitly marked Layer-2-only.
Records are encoded with the existing `msgspec` dependency
(`requirements/common.txt:35`), placed into a
bounded non-blocking queue, and drained by one background JSONL writer. It
flushes every 1 MiB or one second and at clean shutdown, with explicit drop/gap
counters and no per-step `fsync`. `VLLM_OPPROF_DIR` is the startup-only switch;
unset is a true no-op. The unmeasured foreground estimate is 25-100
microseconds per step, so Phase 2 must enforce the less-than-3% overhead gate
with paired off/on serving runs.
Layer 2 configures the already-built torch profiler and samples short windows,
initially two warm-up plus eight active worker iterations. Set
`max_iterations=8`; the wrapper stops the underlying profiler on the subsequent
step after the counter exceeds the limit, as its existing test documents
(`vllm/profiler/wrapper.py:83-114`;
`tests/v1/worker/test_gpu_profiler.py:77-98`). The controller then calls
`/stop_profile` to clear control state. Step-count/on-demand endpoint control is
preferred to wall time. Layer 2 writes one trace per TP worker and joins to
Layer 1 through the existing iteration annotation, order, token counts, and
timestamps.
Under CUDA-graph replay, the profiler is configured for CUDA activity, but the
Python callable is not re-executed: vLLM calls `CUDAGraph.replay()`
(`vllm/compilation/cuda_graph.py:233-360`). CUDA activity may therefore be
visible through Kineto/CUPTI, while Python/PyTorch scopes cannot be assumed to
provide the original per-layer attribution. Layerwise NVTX is explicitly
unsupported with graphs (`vllm/config/observability.py:60-63`). Phase 2 must
check whether the pinned torch/CUPTI combination exposes individual replayed
kernels or a coarser graph launch. A matched eager window can identify kernel
families but must not replace graph-enabled production timing.
Layer-2 kernel time is grouped into attention, router/top-k, expert/dense GEMMs,
normalization/activation, collectives, and sampling. Per-rank critical-path
times calibrate a composition/graph-bucket-conditioned Layer-1 attribution;
held-out profiler windows validate that mapping. Exact routed experts are
enabled only in a separate sampled server run.
### Patch surface
| File | Approximate lines |
|---|---:|
| `vllm/envs.py` | 4 |
| `vllm/v1/opprof.py` (new) | 220 |
| `vllm/v1/core/sched/scheduler.py` | 32 |
| `vllm/v1/worker/gpu_model_runner.py` | 4 |
| `tests/v1/core/test_opprof.py` (new) | 190 |
| **Total** | **5 files / about 450 lines** |
Zero-patch wins: profiler config/endpoints/wrapper, `SchedulerOutput`, existing
metrics plumbing, CUDA kernels, fused MoE/router code, Qwen3 model code, and
frontend APIs. Supporting OpProf together with `--disable-log-stats` would add
about 12 lines in `vllm/v1/core/kv_cache_manager.py`; the design recommends
failing fast instead.
### Main risks
- Histogram construction and JSON encoding are the likely foreground overhead;
the 3% budget must be measured, not inferred.
- Async scheduling can misassociate latest queue/cache state with an older
output; snapshot inside `schedule()` and pair by the exact output object.
- Layer 1 must be scheduler-owned to avoid TP duplicates; Layer 2 remains
per-rank and needs offline aggregation.
- At 500 steps/s and about 1 KiB/record, an uncompressed stream is roughly
44 GB/day per EngineCore. Runs must be duration-bounded, with queue/drop
counters and post-run compression.
- Kineto perturbs execution, and graph replay loses Python-scope attribution;
it is a sampled calibration instrument, not an always-on latency source.
## 4. Open design decisions
1. Approve JSONL/`msgspec`, an 8192-record queue, context edges
`(128, 256, 512, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, +inf)`, and chunk-size
edges `(16, 32, 64, 128, 256, 512, 1K, 2K, +inf)`.
2. Approve exact MoE expert load as Layer-2-only. The alternative is a new GPU
histogram path, with a larger patch and an overhead/graph-capture risk.
3. Confirm checkpoint precision/quantization and intended TP/EP/DP topology.
The H20 Triton conclusion assumes unquantized BF16/FP16, auto backend, and no
LoRA.
4. Choose the initial profiler window: recommended two warm-up plus eight
active iterations, or a longer active window.
5. Decide whether the less-than-3% gate applies to point estimates or the upper
95% confidence bound for every primary metric.
6. Confirm that OpProf campaign runs may reject `--disable-log-stats`; support
for it increases the patch to six files and about 462 lines.
## Data sanity block
- **Pinned root listing:** n=45 names; lexicographic min=`.buildkite`,
max=`vllm`; distinct=45; SHA-256=`75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58`.
- **Source versions compared:** n=2 tags; min=`v0.20.0`, max=`v0.24.0`;
distinct=2; pinned working-tree version remains exactly `v0.24.0`.
- **Torch pin observations:** n=4 declarations
(`pyproject`, build requirements, CUDA runtime requirements, CMake);
min=max=`2.11.0`; distinct=1.
- **Evidence-bearing source files inspected and cited:** n=44
distinct v0.24.0 paths; min/max not applicable to categorical paths;
v0.20.0 comparison touched n=5 distinct paths; min/max not applicable.
- **Invariants checked:** exact tag equals `v0.24.0`; HEAD equals the recorded
commit; clone is clean; top-level names are unique; requested path was used;
torch pins agree; CUDA-graph modes are in `{NONE, PIECEWISE, FULL}`; KV usage
definition is bounded in `[0,1]`; scheduled-token total is defined as the sum
of per-request scheduled tokens; graph bucket is not smaller than unpadded
tokens; no remote/GPU/install/model-download action occurred.
- **Benchmark/statistical conclusion check:** no performance benchmark was run,
so there are no per-configuration curves or measured overhead values on which
to perform identical-value, monotonicity, or continuity checks. All overhead
numbers in the design are labeled estimates.

1233
docs/opprof/phase2-smoke-dash0.md Normal file
View File

File diff suppressed because it is too large Load Diff

1092
docs/opprof/phase3-protocol.md Normal file
View File

File diff suppressed because it is too large Load Diff

262
docs/opprof/phase3-results.md Normal file
View File

@@ -0,0 +1,262 @@
# OpProf Phase 3 dash0 results
Status: **FINAL A-P3-7 PARTIAL-MATRIX ANALYSIS — H1a INCONCLUSIVE; H1b PASS**.
Date: 2026-07-12. The matrix is permanently frozen at 40/52 accepted measured
runs and 20/24 complete pattern/config cells. A-P3-7 permits existential
confirmation on complete comparisons but forbids refutation from incomplete
coverage. The result is therefore **PARTIAL**: H1b is confirmed by five
evaluable contrasts, while H1a cannot be evaluated on a valid pattern pair.
The final machine result is
`runs/opprof-phase3/phase3/metrics.json` (SHA-256
`dcc44941bfe1eb56c7a069148e5565eddd1ccf966e6914aa16b64bd67d02f0f0`).
The final analyzer SHA-256 is
`483c170838a1e81a170d8425b4be22cc1be02ea4bc9492b474daf7ae86186d67`;
7/7 no-GPU analysis tests passed, including ResourceWarning-as-error.
## Data-sanity result first
All **23/23** machine-checked invariants pass before inference: 40 unique
accepted primary markers selected only through completed-stage records; exact
240-second clean windows; zero clean failures; finite moderate rates within
5%; balanced Layer-1 footer/sidecar accounting; ratios in range; 72 accepted
trace files with the registered eight-file first-wave deviation; zero other
GPU processes; exact 20-cell coverage; and the exact A-P3-7 missing-cell and
missing-contrast sets.
The accepted data contain 77,109 clean completions, 352,907 clean Layer-1
steps, and 542,350 total Layer-1 records. Device-time classifiability is
97.05–99.64%. No malformed trace, accounting mismatch, counter underflow,
private-text leak, or unexpected identical-result family was found.
Declared limitations are not hidden as sanity passes: eight accepted
saturation trace files were lost during the registered first-wave controller
cleanup; all four confirmation runs are absent; MoE per-layer CV is unavailable
because layer scopes cover less than 80% of MoE GEMM time; and only one of nine
completed C00-moderate patterns passes the Layer-2 inference gates.
## Frozen coverage and A-P3-7 logic
The four incomplete cells, each missing saturation and moderate, are exactly:
- P03/C11;
- P05/C00;
- P10/C00-TP2; and
- P11/C00.
The four missing confirmations are P10, P06, P03, and P01 C00-moderate.
Formal H1a coverage is therefore the nine completed C00 patterns P01, P02,
P03, P04, P06, P07, P08, P09, and P10. Formal H1b has six evaluable and two
non-evaluable frozen contrasts. A-P3-7's asymmetry is mandatory: one valid hit
can confirm an existential hypothesis, but absence of a hit cannot support the
original complete-matrix null.
## H1a — operator composition
**Verdict: INCONCLUSIVE; refutation is not allowed.**
All moderate traces were parseable and classifiable, but only P04's two
windows jointly passed classifiability, `abs(SMD)<=0.25` representativeness,
and fixed recovery. P01/P02/P03/P09 failed representativeness in both windows;
P06/P07/P08/P10 additionally failed at least one recovery. Thus only one of
nine completed patterns is inferentially evaluable, leaving zero pattern pairs,
zero ranking tests with data, and no Kendall tau-b pairs. No qualifying
inversion can be accepted, and this is not evidence for a shared ranking.
Descriptively, P04 is attention-led at both loads. P06 and P10 change from
attention-led at saturation to MoE-GEMM-led at moderate load; the other
available patterns remain MoE-GEMM-led. These are ranking changes in sampled
windows, not H1a evidence because their window gates fail.
### Per-pattern operator shares, both load points
Values are the mean percentage across the two Layer-2 windows. `E` means the
row is inferentially evaluable; `D` means data are available but descriptive
only because a registered window/recovery gate failed; `NE-cell` means the cell
is missing; `NE-trace` means the accepted run has no retained trace. `Other` is
unclassified device activity. Collective, sampler, and dense-GEMM shares round
to 0.00% in every available C00 row.
| Pattern | Load | Use | Attention | MoE GEMM | Router | Collective | Sampler | Dense GEMM | Norm/elt | KV | Other | Top family |
|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| P01 | saturation | D | 16.32 | 77.81 | 0.85 | 0.00 | 0.00 | 0.00 | 3.90 | 0.10 | 1.02 | MoE GEMM |
| P01 | moderate | D | 12.09 | 82.43 | 1.03 | 0.00 | 0.00 | 0.00 | 3.38 | 0.11 | 0.96 | MoE GEMM |
| P02 | saturation | D | 32.67 | 62.39 | 1.05 | 0.00 | 0.00 | 0.00 | 2.98 | 0.16 | 0.74 | MoE GEMM |
| P02 | moderate | D | 20.91 | 72.53 | 1.64 | 0.00 | 0.00 | 0.00 | 3.68 | 0.25 | 1.00 | MoE GEMM |
| P03 | saturation | D | 39.11 | 56.60 | 0.44 | 0.00 | 0.00 | 0.00 | 2.98 | 0.03 | 0.84 | MoE GEMM |
| P03 | moderate | D | 23.32 | 57.45 | 3.54 | 0.00 | 0.00 | 0.00 | 12.29 | 0.83 | 2.57 | MoE GEMM |
| P04 | saturation | D | 65.88 | 29.77 | 1.16 | 0.00 | 0.00 | 0.00 | 2.30 | 0.17 | 0.72 | Attention |
| P04 | moderate | **E** | 47.88 | 40.64 | 3.28 | 0.00 | 0.00 | 0.00 | 5.85 | 0.45 | 1.90 | Attention |
| P05 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P05 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P06 | saturation | D | 56.59 | 39.35 | 1.04 | 0.00 | 0.00 | 0.00 | 2.25 | 0.15 | 0.62 | Attention |
| P06 | moderate | D | 31.60 | 57.52 | 2.97 | 0.00 | 0.00 | 0.00 | 5.69 | 0.43 | 1.80 | MoE GEMM |
| P07 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
| P07 | moderate | D | 28.67 | 62.18 | 2.42 | 0.00 | 0.00 | 0.00 | 4.86 | 0.37 | 1.49 | MoE GEMM |
| P08 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — |
| P08 | moderate | D | 34.59 | 57.06 | 2.22 | 0.00 | 0.00 | 0.00 | 4.44 | 0.34 | 1.34 | MoE GEMM |
| P09 | saturation | D | 26.23 | 69.03 | 0.36 | 0.00 | 0.00 | 0.00 | 3.43 | 0.00 | 0.95 | MoE GEMM |
| P09 | moderate | D | 16.19 | 75.00 | 2.14 | 0.00 | 0.00 | 0.00 | 4.83 | 0.35 | 1.49 | MoE GEMM |
| P10 | saturation | D | 67.80 | 28.90 | 0.76 | 0.00 | 0.00 | 0.00 | 1.87 | 0.10 | 0.57 | Attention |
| P10 | moderate | D | 30.61 | 54.04 | 2.81 | 0.00 | 0.00 | 0.00 | 9.80 | 0.68 | 2.06 | MoE GEMM |
| P11 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — |
| P11 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — |
### Clean CUDA-graph modes at moderate load
| Pattern | FULL | PIECEWISE | NONE/eager |
|---|---:|---:|---:|
| P01 | 1.98% | 73.19% | 24.83% |
| P02 | 83.16% | 13.00% | 3.84% |
| P03 | 96.37% | 0.00% | 3.63% |
| P04 | 98.32% | 0.02% | 1.66% |
| P05 | — | — | — |
| P06 | 99.10% | 0.05% | 0.85% |
| P07 | 98.76% | 0.00% | 1.24% |
| P08 | 98.64% | 0.01% | 1.36% |
| P09 | 91.79% | 3.69% | 4.52% |
| P10 | 98.33% | 0.12% | 1.55% |
| P11 | — | — | — |
P01 is the clear mode outlier: PIECEWISE covers 73.19% of clean moderate
steps. Full per-mode operator segments and active-step counts remain in the
machine result; no mode with fewer than eight sampled steps is summarized.
## H1b — rectangular-benchmark miss
**Verdict: PASS.** Five of six evaluable frozen contrasts pass at least one
waste threshold with a simultaneous positive bound and at least 5% useful-token
efficiency loss. The two P05 contrasts are `NOT EVALUABLE`; no value is
imputed. Holm correction retains the original eight planned contrasts per
metric. Reported simultaneous intervals below are percentage-point effects;
passing bootstrap p-values are zero at 100,000-resample resolution.
| Frozen contrast | Status | Passing evidence | Useful-token efficiency loss |
|---|---|---|---:|
| P05 vs P01 | **NOT EVALUABLE** | P05/C00 missing | — |
| P05 vs P03 | **NOT EVALUABLE** | P05/C00 missing | — |
| P06 vs P02 | **PASS** | R64 +23.015 pp, simultaneous 95% [22.345, 23.683] | 11.609% |
| P06 vs P04 | **PASS** | R64 +35.440 pp [34.826, 36.059] | 22.846% |
| P09 vs P01 | **PASS** | padding +6.673 pp [5.887, 7.492]; R64 +39.616 pp [39.065, 40.166] | 8.325% |
| P09 vs P03 | EVALUABLE, NO HIT | padding +8.542 pp and R64 +52.041 pp are material, but association gate fails | **3.840%**, below 5% |
| P10 vs P03 | **PASS** | padding +5.565 pp [2.010, 10.341]; R64 +44.787 pp [43.635, 45.971] | 44.693% |
| P10 vs P04 | **PASS** | padding +5.399 pp [1.868, 10.213]; R64 +44.787 pp [43.638, 45.949] | 14.260% |
This confirms the existential H1b claim on completed contrasts. It cannot
support a null claim about P05 or the missing matrix.
## Waste accounting
The preregistered contrast thresholds are 5 percentage points for padding,
10 points for graph miss/overflow, 0.15 absolute for R64, 0.10 for mixed-batch
interference, and 0.15 for MoE layer CV, plus the 5% efficiency/residual
association gate. Values below are clean C00-moderate results; efficiency is
scheduled useful tokens per model-step millisecond.
| Pattern | Padding | Graph miss | Overflow | R64 | Mixed interference | Efficiency | Supported/total mixed steps |
|---|---:|---:|---:|---:|---|---:|---:|
| P01 | 1.869% | 24.826% | 24.826% | 0.3687 | N/A | 4.9673 | 0/5,646 |
| P02 | 1.998% | 3.837% | 3.837% | 0.3687 | N/A | 2.6664 | 0/1,536 |
| P03 | 0.000% | 1.736% | 1.736% | 0.2444 | N/A | 4.7356 | 0/138 |
| P04 | 0.167% | 1.301% | 1.301% | 0.2444 | N/A | 3.0547 | 0/141 |
| P05 | — | — | — | — | — | — | — |
| P06 | 0.276% | 0.830% | 0.830% | 0.5988 | N/A | 2.3568 | 0/152 |
| P07 | 0.140% | 1.240% | 1.240% | 0.0000 | N/A | 2.7513 | 0/186 |
| P08 | 0.005% | 1.357% | 1.357% | 0.0000 | N/A | 2.1478 | 0/178 |
| P09 | **8.542%** | 4.518% | 4.518% | **0.7648** | N/A | 4.5537 | 0/1,054 |
| P10 | **5.565%** | 0.921% | 0.921% | **0.6923** | N/A | 2.6191 | 0/92 |
| P11 | — | — | — | — | — | — | — |
No irregular-versus-control graph-miss or overflow contrast reaches the
positive 10-point H1b threshold. P01 has the largest absolute miss rate but is
a rectangular control; P09 actually reduces miss versus P01 by 20.308 points.
The registered leave-one-pattern-out robust fits were applied for mixed-batch
interference, but no pattern retained 30 supported mixed steps inside both
pure-fit convex supports; every result is N/A rather than extrapolated. No
separate LOAO operator-ranking procedure was preregistered, so none was added
post hoc. The required two-window rule is used unchanged, and confirmation-run
robustness is `NOT EVALUABLE` because all four confirmations are missing.
### Top three waste findings
1. **P10 real trace:** R64 is 44.787 points above both long rectangular
controls; padding is 5.565/5.399 points higher; efficiency is 44.693% worse
than P03 and 14.260% worse than P04.
2. **P09 production-shaped mix:** versus P01, R64 rises 39.616 points and
padding 6.673 points with 8.325% lower efficiency. Its still-larger P03
contrast does not pass because efficiency loss is only 3.840%.
3. **P06 bimodal long-output burst:** R64 rises 23.015 points versus P02 and
35.440 points versus P04, paired with 11.609% and 22.846% efficiency loss.
## Pattern-conditioned operational findings
- **P10/TP2 non-stabilization:** the fresh run completed 17 warm-up requests,
but trailing scheduled-token throughput fell 18,022.4 → 16,384.0 → 14,062.6
tokens/s. Its 36.76% fitted drift exceeds A-P3-6's 10% limit; same-wave
synthetic P11 and P03 completed 512 and 102 warm-up requests and passed their
applicable registered gates. The orchestrator accepts this as a
pattern-conditioned finding, not an accepted throughput measurement.
- **Long-context drain:** P10/C01 saturation naturally drained for 288.619
seconds. The clean window and accounting were valid; the original 120-second
short-pattern watchdog was miscalibrated, while the amended 600-second P10
budget correctly retained the run.
- **Failure boundary:** P01/C01 moderate had 5,828 clean successes and zero
clean failures. Two `ServerDisconnectedError` records occurred only in
excluded post-profile time; separating clean and auxiliary windows prevented
valid data from being discarded.
- **Layer-2 perturbation:** despite 97%+ kernel classifiability, eight of nine
completed moderate patterns failed representativeness or recovery. Together
with the Phase-2 51.3% active-window perturbation, this shows that sparse
Kineto windows are operationally fragile for these serving patterns.
These incidents consistently show long-context and mixed serving patterns
breaking assumptions calibrated on shorter rectangular loads: fixed drain
watchdogs, completion-count warm-up, global failure scope, and profile-window
representativeness.
## Matrix and GPU accounting
| Item | Final value |
|---|---:|
| Planned measured runs | 52 |
| Accepted measured runs | **40** |
| Complete cells | **20/24** |
| Accepted confirmations | 0/4 |
| Accepted clean failures | 0 |
| Drain-quarantined accepted runs | 0 |
| Accepted Layer-2 trace files | 72 |
| Registered missing trace files | 8 |
| Other-user GPU processes | 0 |
| Cumulative GPU use | **14.025875 H20-hours** |
| Reserved/unused headroom | **1.974125 H20-hours** |
No GPU work was run for A-P3-7 analysis. The reserved headroom remains unused,
and all eight GPUs were at 0 MiB/0% before and after the CPU-only analysis.
## Data sanity block
| Numeric family | n | finite | missing | min | max | distinct | Check |
|---|---:|---:|---:|---:|---:|---:|---|
| Accepted throughput (req/s) | 40 | 40 | 0 | 0.445833 | 43.300000 | 37 | finite, positive, not identical |
| Token efficiency (tokens/ms) | 40 | 40 | 0 | 2.147827 | 8.295849 | 40 | finite, positive, distinct |
| Drain duration (s) | 40 | 40 | 0 | 0.305419 | 288.619108 | 40 | non-negative; no quarantine |
| Clean Layer-1 steps/run | 40 | 40 | 0 | 718 | 20,171 | 40 | sum 352,907; continuous per run |
| Operator classifiable fraction | 72 | 72 | 0 | 0.970458 | 0.996378 | 72 | all >=0.70 and within [0,1] |
| Operator-family shares | 576 | 576 | 0 | 0.000000 | 0.873417 | 348 | within [0,1] |
| Waste ratios | 240 | 240 | 0 | 0.000000 | 1.000000 | 168 | within [0,1] |
| Waste-contrast effects | 36 | 24 | 12 expected | -0.203082 | 0.520413 | 17 | signed; N/A preserved |
| Kendall tau-b | 0 | 0 | 0 | — | — | 0 | expected: only one valid pattern |
| Accepted clean failures | 40 | 40 | 0 | 0 | 0 | 1 expected | sum zero |
| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | cleanup passed |
Checked invariants: accepted IDs unique; exact 40-run/20-cell coverage; exact
four missing cells and two missing contrasts; 240-second clean windows; zero
clean failures; moderate rate within 5%; balanced Layer-1 accounting and zero
drops; trace parsing and classifiability; ratios in range; exact 72+8 trace
accounting; clock/load snapshots present; no other-user process; drain
quarantine below 20%; non-identical pattern results; and exact reproduction of
the 36.76% A-P3-6 operational result. **No data-sanity red flag remains.** H1a
is inconclusive because of declared Layer-2 validity limits; H1b passes on five
complete contrasts; the compound hypothesis is partial and cannot be refuted.

306
docs/opprof/phase4-optimization-plan.md Normal file
View File

@@ -0,0 +1,306 @@
# OpProf Phase 4 measured optimization plan
Status: **PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS**.
Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run,
20/24-cell dataset and the single optional Phase-4 capture-size validation.
Phase-3 protocol and results are frozen. Bounds below are ceilings in the
units actually measured; padding or raggedness percentages are not relabeled
as end-to-end throughput gains.
## Pinned implementation context
- Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary.
- vLLM source: accepted OpProf tip
`23450fb21ac255b0cf710f4ee965ee694921975d` on v0.24.0.
- vLLM 0.24.0 exposes `--cudagraph-capture-sizes`,
`--max-cudagraph-capture-size`, `--max-num-seqs`, and
`--max-num-batched-tokens` (`vllm/engine/arg_utils.py:1390-1467`).
- An explicit capture-size list replaces the inferred list. The default is
`[1,2,4]`, multiples of eight below 256, then multiples of sixteen through
the maximum, normally 512 (`vllm/config/vllm.py:1669-1792`).
- Chunked prefill remains enabled. vLLM schedules decode first, then fills the
remaining MBT budget with prefill and chunks an over-budget prefill
(`docs/configuration/optimization.md:45-59`).
The ranking is by measured opportunity, readiness, and downside together—not
by a normalized composite score. Items that share the same raggedness bound
are explicitly non-additive.
## Ranked optimization list
| Rank | Tier | Target and affected regime | Measured bound | Owner | Decision |
|---:|---|---|---|---|---|
| 1 | Config now / scheduler backlog | Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic | P08 vs matched P07: **+82.14% saturation req/s**, 80% prefix-hit ratio, **62.29% fewer prefill tokens** | Serving/config owner now; cache-aware dispatch upstream | Deploy behind a workload classifier; do not apply to no-sharing traffic |
| 2 | Scheduler | Length-aware cohorting/admission for ragged P10/P09/P06 | At most **44.79 pp** R64 contrast and **44.69%** measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06 | Upstream vLLM scheduler | Highest structural backlog item; preserve fairness and arrival semantics |
| 3 | Config now | Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches | Distributional bound **4.98 pp P09 / 5.26 pp P10** padding; P09 validation achieved **4.980 pp** | Serving/config owner | Mechanism confirmed; canary only because p95 was +3.01% in one pair |
| 4 | Config now | Route P06/P10-like pools to MNS=64 | Saturation throughput **+3.37% P06 / +3.70% P10** versus C00 | Serving/config owner | Pattern-specific trial only; same setting was −24.27% on P01 |
| 5 | Kernel backlog | Ragged-aware MoE GEMM and attention over the measured shape stream | Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate | Engine/kernel colleagues | Optimize exact weighted shapes below; require serving confirmation |
| 6 | Config guardrail | Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048 | Avoided saturation regressions up to **11.64% P03**, 6.53% P01, 5.82% P10 | Serving/config owner | Encode as a policy guardrail, not a positive optimization claim |
## Tier A — configuration-level actions deployable today
### A1. Prefix-affine routing with prefix caching
**Measured regime.** P07 and P08 have the same 1,280-token prompt length,
512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses
eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix
caching already enabled:
| Metric, saturation | P07 no sharing | P08 high sharing | Delta |
|---|---:|---:|---:|
| Prefix-query hit ratio | 0.00 | 0.80 | +0.80 |
| Clean prefill tokens | 1,528,968 | 576,512 | **−62.29%** |
| Completed throughput | 5.1083 req/s | 9.3042 req/s | **+82.14%** |
**Root mechanism.** The matched cell changes only controlled prefix sharing;
the observed hit and prefill-token changes directly identify cache reuse. The
82.14% throughput delta is a point observation and the maximum evidence-backed
gain for this exact regime, not a fleet-wide forecast.
**Action now.** Keep `--enable-prefix-caching`; hash an application-known
stable prefix or conversation identity to a replica so related requests do
not destroy affinity through round-robin load balancing. Apply only when the
online prefix-query hit ratio resembles P08, not P07.
**Verification.** Interleave affinity ON/OFF on the same replicas and fixed
request stream. Primary gates are prefix-hit ratio, prefill-token reduction,
completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A
throughput gain with queue/fairness or KV-capacity regression does not pass.
### A2. Measured CUDAGraph capture sizes
**Measured regime.** P09 moderate has decode-batch p50/p95/max 4/16/25. P10
moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those
batches pad upward. Replaying the Phase-3 hit distribution predicts that
adding exactly `{3,5,6,7,9}` to the complete default list can remove:
- **4.9776 percentage points** of P09's 8.5421% graph-hit padding; and
- **5.2579 points** of P10's 5.5652% padding.
The list must be `default ∪ {3,5,6,7,9}`; passing only five sizes would replace
and discard the rest of the default list.
**Closed-loop result.** One P09 moderate ON/OFF pair used fresh servers, the
same fixed seed and accepted saturation-rate source, 60 excluded warm-up
seconds, and 240 clean seconds per arm. No Layer-2 profiler ran.
| Metric | ON: exact sizes | OFF: default | ON relative to OFF |
|---|---:|---:|---:|
| Graph-hit padding | **3.5659%** | 8.5456% | **−4.9798 pp; −58.27%** |
| Useful tokens/model-step ms | 4.56222 | 4.55405 | +0.179% |
| Completed throughput | 4.9583 req/s | 4.9333 req/s | +0.507% |
| Mean E2E latency | 1.6123 s | 1.6812 s | −4.10% |
| p95 E2E latency | 3.9930 s | 3.8763 s | **+3.01%** |
| Clean failures | 0 | 0 | equal |
The observed padding reduction differs from the Phase-3 bound by only 0.0022
percentage points, confirming the mechanism. It does **not** establish a broad
performance win: token efficiency and throughput moved less than 1%, p95 moved
the wrong way, and there is one ordered pair with no CI.
Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes
versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server
ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary;
require interleaved replication with a p95 non-regression gate before rollout.
### A3. Pattern-specific MNS pools
The complete saturation comparisons show that `--max-num-seqs 64` is an
interaction, not a globally better default:
| Pattern | C10 MNS=64 vs C00 | Interpretation |
|---|---:|---|
| P01 short/short | **−24.27% req/s** | Reject for dense short traffic |
| P03 long/short | −1.41% | No measured benefit |
| P06 bimodal/long burst | **+3.37%** | Candidate pattern pool |
| P10 real long-context | **+3.70%** | Candidate pattern pool |
The evidence identifies the config×pattern interaction but not a lower-level
cause. Do not attribute it to a particular kernel or queue effect without a
new bisection. Verification is five interleaved saturation pairs per intended
class plus TTFT, queue depth, preemption, KV usage, and exact-work checks.
### A4. MBT policy guardrail
`--max-num-batched-tokens 2048` versus the default 8192 changed saturation
throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined
MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining
MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is
an avoided regression, not new speedup.
## Tier B — upstream scheduler changes
### B1. Length-aware cohorting without starvation
**Measured mechanism.** R64 is the rectangular padding fraction of the exact
arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988
for P06. Their passing control contrasts are:
| Irregular pattern | Control | R64 excess | Useful-token efficiency loss |
|---|---|---:|---:|
| P10 | P03 | **44.79 pp** | **44.69%** |
| P10 | P04 | **44.79 pp** | **14.26%** |
| P09 | P01 | **39.62 pp** | **8.32%** |
| P06 | P02 | **23.01 pp** | **11.61%** |
| P06 | P04 | **35.44 pp** | **22.85%** |
These are upper bounds on waste a length-aware path could avoid. R64 is not
observed GPU time, and efficiency association is not causal. The scheduler
change should maintain several ready queues by remaining prompt/context band,
select a less-ragged cohort subject to the existing decode-first token budget,
and impose a finite age/fairness bound. It must not rewrite request arrivals or
drop long requests.
**Verification.** Add a runtime per-step raggedness counter rather than using
manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful
tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption,
KV occupancy, and the full length histogram. The gain cannot exceed the
corresponding R64/efficiency bounds above, and it is non-additive with a
ragged-aware kernel.
### B2. Automatic cache-aware dispatch
The upstream form of A1 is a scheduler/replica dispatcher that chooses a live
prefix-cache owner while respecting load. Its collaboration contract is the
measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix
IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing
penalty and lost cache hits must be reported together; a synthetic cache hit
increase without end-to-end balance is insufficient.
### B3. Histogram-driven capture-list generation
Static A2 proves that Layer-1 can choose useful sizes. An upstream controller
could select a bounded number of exact sizes from padding contribution
`count(size) * (next_bucket-size)`, while retaining the default list and a
memory/startup budget. P09's top five `{3,5,6,7,9}` recovered 4.98 points at a
0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The
selector must freeze its list before measurement and never continually tune on
the scored window.
## Tier C — kernel-level backlog and collaboration interface
H1a is inconclusive, so Phase 3 does not prove a universal top operator. The
available moderate windows are still useful shape inputs: descriptive MoE-GEMM
shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%,
16.19%, and 30.61%. Only P04's operator windows pass inference gates, where
attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim
shape-local improvement, not a resolved global bottleneck.
### Exact shape stream for kernel engineers
`P`, `D`, and `N` below are per-step prefill tokens, decode tokens, and
scheduled requests. Counts are from clean C00-moderate Layer-1 records.
Context mix is the fraction of scheduled request-context observations in
`<=1024 / 1025–8192 / 8193–32768 / >32768` bins.
| Pattern | Model steps: decode / mixed / prefill | N p50 / p95 / max | Dominant exact `(P,D,N): count` | Context mix | Chunk signal |
|---|---|---|---|---|---|
| P01 | 5,760: 114 / 5,646 / 0 | 69 / 74 / 77 | `(0,66,66):18`, `(0,67,67):18`; mixed P p50=334, D p50=68 | 100.00 / 0 / 0 / 0% | 6,235 unsplit; sizes 129–512 dominate |
| P04 | 12,455: 12,291 / 141 / 23 | 8 / 8 / 16 | `(0,8,8):12,129`, `(8191,1,3):23` | 0.03 / 94.62 / 5.35 / 0% | 238/319 chunks >2,048; first/final 122/123 |
| P06 | 17,464: 17,310 / 152 / 2 | 8 / 16 / 16 | `(0,8,8):13,103`, `(0,16,16):4,098` | 55.11 / 42.93 / 1.96 / 0% | 174/422 >2,048; 136 in 257–512 |
| P09 | 12,837: 11,783 / 1,054 / 0 | 4 / 16 / 25 | `(0,3,3):3,397`, `(0,2,2):1,549`, `(0,4,4):1,526`, `(0,5,5):1,050` | 51.70 / 48.26 / 0.04 / 0% | 228/1,193 >2,048; 345 in 1,025–2,048 |
| P10 | 18,130: 17,941 / 92 / 97 | 1 / 4 / 7 | `(0,1,1):13,572`, `(0,2,2):2,675`, `(0,3,3):792`, `(0,4,4):524`, `(8192,0,1):38` | 12.25 / 39.94 / 47.76 / 0.05% | 138/191 >2,048; first/middle/final/unsplit 49/29/49/64 |
The exported kernel-benchmark interface should be a prompt-free weighted table
with `(P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count)` plus
step duration and useful tokens. Use the frozen histogram edges already emitted
by Layer 1: context `128..131072` powers of two and chunk `16..2048` powers of
two. Preserve the joint tuples; independent marginal sampling would erase the
mixed-batch structure.
### Kernel targets and acceptance
1. **Ragged MoE GEMM:** accept variable token counts without padding every
expert/layer tile to the largest sequence. Weight microbenchmarks by the P06,
P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity
as B1, not an additional gain.
2. **Attention:** retain P04 `(D,N)=(8,8)` as the valid long rectangular control
and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report
useful-token time, workspace, and graph compatibility.
3. **Serving confirmation:** kernel time must improve on the exact weighted
stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95,
memory, and correctness. A rectangular-only kernel win does not close the
Phase-3 finding.
`moe_expert_load` was unavailable in Phase 3. No expert-imbalance mechanism or
gain is claimed; expert-specific packing requires a new low-overhead route
histogram before implementation.
## Honest limits and Phase-5 measurement requirement
- H1a remains inconclusive: only P04 had two representative/recovered operator
windows. Eight other completed moderate patterns failed window validity even
though kernel classifiability was 97.05–99.64%.
- A Phase 5 operator study needs longer, time-stratified samples that reproduce
clean scheduled-token, prefill-fraction, decode-batch, and graph-mode
distributions, plus a lower-perturbation per-op timer. It must demonstrate
overhead before using shares for optimization; the Phase-2 Kineto active
window perturbed throughput by 51.3%.
- Confirmation runs are absent. The MNS and MBT config effects are single-run
point estimates and require replication before production decisions.
- R64 is an offline rectangular-padding upper bound, not measured GPU idle
time. H1b's efficiency association does not establish causality.
- Mixed-batch interference was N/A because no cell retained 30 supported mixed
steps inside both leave-one-pattern-out pure-fit supports.
- Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11,
P05/C00, P10/C00-TP2, and P11/C00 are absent.
- The capture validation is one ordered ON/OFF pair. Its padding endpoint is
mechanism-valid, but performance deltas have no CI and p95 regressed.
- Layer 1 did not collect expert-route identities; kernel engineers cannot infer
routed-expert imbalance from these artifacts.
## Verification and stop rules for Phase 4 work
Every proposed experiment keeps the original fixed manifest/seed/work, excludes
warm-up, records Layer-1 accounting, and changes one mechanism. A candidate
stops on clean failure, footer imbalance/drop, output mismatch, GPU
contamination, memory regression beyond its declared budget, or violation of
the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are
reported together; no metric shopping or silent pattern substitution is
allowed.
## GPU accounting
The optional capture pair consumed **0.296389 H20-hours**, taking cumulative
campaign use from 14.025875 to **14.322265 H20-hours**. Remaining headroom is
**1.677735 H20-hours**. Both arms returned GPU0 to zero, all eight GPUs were
0 MiB/0% at final inspection, and no other-user process appeared.
Artifacts are under
`runs/opprof-phase3/phase4/capture-p09/`; `result.json` SHA-256 is
`5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4`.
## Final ranked one-liners
1. **Prefix affinity (config now):** P08's measured ceiling is **+82.14% req/s** with 62.29% fewer prefill tokens versus matched P07.
2. **Length-aware scheduler:** raggedness ceiling is **44.79 pp R64 / 44.69% efficiency gap** on P10; smaller confirmed bounds apply to P09/P06.
3. **Exact capture sizes (config now):** ceiling **5.26 pp P10 / 4.98 pp P09 padding**; P09 validation removed 4.980 pp but did not prove p95 gain.
4. **MNS64 pattern pools (config now):** measured ceiling **+3.70% req/s P10 / +3.37% P06**, with a −24.27% P01 counterexample.
5. **Ragged kernels (kernel backlog):** share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive.
6. **MBT8192 guardrail (config now):** avoids measured regressions up to **11.64%**; MBT2048 has no general positive case.
## Data sanity block
| Numeric family | n | finite | missing | min | max | distinct | Invariant/result |
|---|---:|---:|---:|---:|---:|---:|---|
| Ranked items | 6 | 6 | 0 | rank 1 | rank 6 | 6 | Three tiers represented; bounds not summed |
| Sentinel saturation config deltas | 11 | 11 | 0 | −30.216% | +3.704% | 11 | Both gains and regressions retained |
| Passing R64 contrast effects | 5 | 5 | 0 | 0.230148 | 0.447872 | 4 | Ratios in [0,1]; duplicate P10 controls expected |
| Capture-arm padding fraction | 2 | 2 | 0 | 0.035659 | 0.085456 | 2 | Non-negative; ON < OFF |
| Capture-arm token efficiency | 2 | 2 | 0 | 4.554051 | 4.562222 | 2 | Positive; +0.179% ON |
| Capture-arm throughput (req/s) | 2 | 2 | 0 | 4.933333 | 4.958333 | 2 | Same offered rate 4.920833 req/s |
| Capture-arm clean failures | 2 | 2 | 0 | 0 | 0 | 1 expected | Exact 240 s and zero failures |
| Capture-arm Layer-1 records | 2 | 2 | 0 | 16,491 | 17,579 | 2 | Every footer/sidecar invariant true; zero drops |
| Optional validation GPU-hours | 1 | 1 | 0 | 0.296389 | 0.296389 | 1 | Positive; cumulative 14.322265 < 16 |
| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | Cleanup passed |
Checked invariants: Phase-3 metrics remain frozen; every cited number resolves
to accepted metrics or the checksum-recorded validation; config comparisons
use saturation rather than normalized moderate throughput; padding/raggedness
bounds are not presented as throughput; duplicate/non-independent bounds are
not added; both validation arms use identical work and offered rate; clean
failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar
balance, and zero drops pass; ratios lie in their declared domains; all GPU
memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No
data-sanity red flag remains.

640
docs/opprof/phase5-protocol.md Normal file
View File

@@ -0,0 +1,640 @@
# OpProf Phase 5 pre-registered mechanism-decomposition protocol
Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.
Date frozen: 2026-07-12 (Asia/Singapore). This document specifies Phase 5 only.
It does not authorize a GPU launch, helper implementation, trace transfer, or
change to the accepted vLLM patch series. Any change to an estimand, request
set, service order, arrival transform, capture-size set, load, validity gate,
or decision threshold requires a dated amendment before the affected run.
## Approved dispositions (orchestrator; 2026-07-12)
All five open decisions below are approved before execution. These dispositions
are normative and close the protocol review gate without changing any frozen
estimand, command delta, validity threshold, or budget:
1. The recorded-arrival **bridge ledger** is approved. Every machine and human
output must state that it decomposes the recorded-arrival P5 gap anchored to
P3 controls, not literally P3's already-uniform P10 gap; A3 supplies the
explicit bridge back to P3.
2. The dual P03/P04 control ledgers are approved. Both are always reported and
a dominant-mechanism call must pass the frozen rule under both denominators.
3. A1 is approved exactly as specified: 142 requests, 32-request reorder
blocks, 16-request analysis cohorts, frozen bins, and a 64-second fairness
cap.
4. Three P10 replicates per arm, P3 control reuse behind the 3% bridge gate,
the conditional control reruns, and optional-tier ordering under the 6.0
H20-hour hard cap are approved.
5. Layer-1-only primary measurement is approved. Routed-expert telemetry is
analysis-only, private, optional, and never receives a causal ledger share.
This approval authorizes the later execution turn only when its preflight,
echo-before-launch, detached-controller, long-context, privacy, accounting,
cleanup, and budget gates all pass.
## Amendment A-P5-1 — rate-following cold-start gate (orchestrator; 2026-07-12)
The first Phase-5 wave correctly hard-stopped because all four offered-load
arms failed the inherited A-P3-6 throughput-drift gate: recorded base/A4 drift
was approximately 216.5%, A1 was 190.6%, and uniform A3 was 13.23%, versus the
frozen 10% limit. Their 240-second clean windows, output work, offered rates,
Layer-1 accounting, and drains otherwise passed. The gate was semantically
wrong for these arms: a rate-following run's scheduled-token throughput follows
its arrival process by design, so recorded non-stationarity is treatment signal,
not cold-start contamination. The large recorded-versus-uniform drift gap is
retained as direct arrival-mechanism evidence. Throughput-drift stationarity
remains appropriate and unchanged for saturation arms.
For every **rate-following/offered-load** arm, A-P3-6 is replaced by all three
of the following cold-start-artifact gates:
1. Every logged torch.compile or CUDA-graph capture first-occurrence event must
precede the clean boundary. Match server-log event messages containing
`torch.compile took`, `Directly load AOT compilation`, `Compiling`, or
`Capturing CUDA graphs` (case-insensitive). Timestamped events are compared
against `t0_wall_ns + 60 s`. vLLM's capture progress lines have no timestamp;
they count as pre-client only when their log-line order precedes the server
ready/startup-complete marker, which itself precedes client `t0`. Any matching
event after readiness must have a parseable timestamp and precede clean;
otherwise the run is invalid.
2. At least 16 requests must complete successfully in `[0,60 s)`, including at
least one request whose recorded `input_tokens >= 8192`.
3. No first-occurrence capture event may appear inside `[60,300 s)`. Parse the
configured startup capture-size set and the completed FULL/PIECEWISE startup
capture passes from the server log. From Layer 1, define a captured replay
descriptor as `(runtime_mode,bucket_tokens)` for every model-executed
`cudagraph.hit=true` step. Every clean descriptor must be covered by the
startup-captured mode and bucket set, and no server-log compile/capture event
may occur inside clean. Warm and clean descriptor sets are both reported;
a descriptor's first **replay** in clean is not mislabeled as a first capture
when startup logs prove it was already captured. An uncovered descriptor or
clean-window capture/compile event invalidates that run only.
The report records matched server-log events, warm-up completion/long-request
counts, warm-up and clean descriptor sets, and clean-only descriptors per run.
Absence of any required log timestamp, Layer-1 interval, or request record is a
gate failure. The original 10% A-P3-6 drift criterion remains mandatory for
closed-loop saturation arms. This amendment changes no request set, arrival
transform, service order, server configuration, clean interval, metric,
bootstrap, share estimator, dominance rule, control-reuse gate, or GPU budget.
## Goal, system boundary, and success criterion
Phase 3 measured a total useful-token-efficiency gap between irregular patterns
and rectangular controls but did not causally allocate it. Phase 5 asks how much
of the P10 gap is recovered when one treatment at a time removes:
1. intra-cohort input-length raggedness;
2. CUDA-graph decode-batch capture-bucket mismatch;
3. recorded arrival burstiness; or
4. usable natural-prefix structure.
The primary system remains Qwen3-30B-A3B BF16, patched vLLM 0.24.0, C00, TP1,
one H20 per server on dash0. The primary load is the P3 P10 rate
`lambda = 0.60 * 0.7875 = 0.4725 request/s`; it is held fixed across arms so an
ablation changes one treatment, not offered demand. The 240-second clean window
and Layer-1 definition of
```text
E_token = sum(prefill_tokens + decode_tokens) / sum(model-step duration_ms)
```
are inherited unchanged. Layer 2 is not needed for the causal ledger and is not
enabled in primary throughput runs.
Success is a mechanism ledger with an absolute `E_token` for every arm, an
un-normalized share and bootstrap confidence interval for every mechanism, and
an explicit residual/interaction line. A null or negative share is a valid
result. Merely recovering the expected sign is not success.
## Pinned Phase-3 evidence and the arrival-estimand discrepancy
The frozen P3 C00-TP1 moderate values are:
| Cell | `E_token` (tokens/ms) | Role |
|---|---:|---|
| P10 | 2.6191132083 | irregular P3 base |
| P03 | 4.7355997154 | long-input/short-output rectangular control |
| P04 | 3.0547035940 | long-input/long-output rectangular control |
P10 therefore lost 44.693% versus P03 and 14.260% versus P04. P3 used both
controls, so Phase 5 reports two parallel ledgers, one per frozen control. It
never selects the denominator that makes a mechanism look largest. A mechanism
is called control-robust only when its decision agrees under both ledgers.
There is one blocking provenance discrepancy. The private source contains
`timestamp` and `source_timestamp`, but P3's materializer discarded them and
set every P10 row to `arrival=steady`; the P3 client then admitted requests at
exactly `1/lambda`. Thus P3 has no recorded-arrival burstiness to remove.
The recommended resolution, pending orchestrator approval, is a **P5 bridge
ledger**: the P5 base replays the same P10 requests at rate-normalized recorded
timestamps, A3 uniformizes those timestamps, and P03/P04 remain the frozen P3
controls. A3 also acts as a bridge back to the P3 steady workload. This meets
the requested arrival ablation but decomposes a recorded-arrival P5 gap, not
literally the already-uniform P3 gap. The report must show both
`E_A3 - E_P3_P10` and its CI before relating the P5 ledger to P3.
If the orchestrator rejects this rebase, the P3-exact alternative is mandatory:
A3's share is `0 / N/A by construction`, and recorded arrival is reported only
as a stress sensitivity, not as a mechanism share. It is scientifically invalid
to call the reverse, burstiness-injecting treatment an ablation that removes
arrival dynamics. No GPU work may begin before this decision is recorded.
## Common request set and exact arrival transforms
The primary request set is the first **142** rows of the frozen 4,011-row P10
selection in original source order. This is exactly the number of admissions at
`lambda=0.4725` over `[0,300 s)`: scheduled times are
`0, 1/lambda, ..., 141/lambda`. All five arms contain the same 142 request IDs,
prompts, per-request input/output lengths, and aggregate input/output token
totals. There is no wrap, replacement, cancellation, or resampling.
The Phase-5 materializer must preserve `timestamp` as private metadata. Let
`z_i` be its stable source-order timestamp for request `i`, and let `N=142`.
The two arrival vectors are frozen as:
```text
recorded-scaled: a_i = (z_i-z_0) * ((N-1) / (lambda*(z_(N-1)-z_0)))
uniformized: a_i = i / lambda
```
`z_(N-1)` must exceed `z_0`; timestamps must be finite and nondecreasing.
Ties remain ties. Both vectors start at zero, end at `141/lambda`, have the same
mean rate, and use the same request order except in A1. The client schedules
against `a_i` directly and does not add jitter. This preserves the recorded
inter-arrival shape while preventing mean-rate differences from masquerading as
an arrival mechanism.
The protocol requires a small `scripts/opprof_phase5_client.py` extension in a
later, no-GPU implementation turn. Before execution, CPU-only tests must prove:
- exact 142-row identity and token sums across all manifests;
- timestamp normalization endpoints and nonnegative gaps;
- uniform gaps equal `1/0.4725` within 1 microsecond;
- the A1 fairness bound and deterministic ordering;
- fixed 60+240-second timing, no wrap, exact output work, and text redaction.
Its reviewed SHA-256 and all manifest SHA-256 values are frozen in the detached
controller before GPU use.
## Falsifiable mechanism estimands
For arm `m` and rectangular control `c in {P03,P04}`:
```text
gap_c = E_control,c - E_base
delta_m = E_ablated,m - E_base
share_m,c = delta_m / gap_c
```
The same base and same control are used for all four mechanisms within a
ledger. No share is clipped to `[0,1]`. Shares may be negative, exceed one, or
sum above/below one because single-factor interventions can overlap or interact.
The arithmetic residual/interaction line is
```text
share_residual+interaction,c = 1 - sum_m share_m,c
```
with a joint bootstrap CI. This is bookkeeping, not proof that the remainder is
one separable mechanism. It may contain unmeasured mechanisms, non-additivity,
double-counting, MoE routing, chunked-prefill interference, and measurement
error. Individual shares are never renormalized to total 100%; the residual is
never clipped to make the table visually close.
### A1 — length-binned service order
**Hypothesis.** Length-homogeneous local cohorts reduce ragged-attention/SM
imbalance, so `E_A1 > E_base`. The hypothesis is falsified if the registered
manipulation check fails or the Holm-corrected efficiency contrast is not
positive.
Starting from consecutive **32-request reorder blocks** in original P10 order,
assign each request to the fixed input-length bins
```text
[0,512], [513,1024], [1025,2048], [2049,4096],
[4097,8192], [8193,16384], [16385,32768]
```
and stable-sort each block by `(bin_id, input_tokens, original_index)`. This
creates two more homogeneous 16-request analysis cohorts per complete reorder
block. Assign the sorted requests to the block's unchanged recorded-scaled
arrival slots. If that assignment would delay any request by more than **64
seconds** relative
to its original slot, choose the earliest-deadline request first until all
deadlines are feasible, then resume the length order. Early movement is allowed;
late movement is capped. Ties are stable and no sorting crosses a 32-request
block.
On consecutive complete 16-request cohorts of evaluation-slice service order,
define `R16 = 1 - sum(L_i) / sum(16*max_cohort(L_i))`; the incomplete final
cohort is excluded and reported. Frozen pre-run values are base `R16=0.641744`
and sorted `R16=0.473409`, a 0.168334 absolute (26.23% relative) reduction;
plain sorting's maximum added delay is 62.744 seconds. The manipulation passes
only if regenerated values match these within `1e-6`, `R16` falls by at least
20% relative and 0.15 absolute, and no request violates the 64-second delay
bound.
Arrival-slot timestamps, request/content multiset, per-request output lengths,
total tokens, server config, and prefix-caching setting are identical to base.
A1 estimates the total effect of changing cohort composition. It does not
claim to isolate a particular attention kernel: service order can mediate
decode-batch composition, chunked-prefill mixing, cache locality, and
content-bound MoE routing. If the prefix-query hit ratio changes by more than
one percentage point, or the normalized inter-arrival vector changes at all,
the arm is labeled confounded and has no publishable raggedness share.
### A2 — measured decode-B capture sizes (config-tier deliverable)
**Hypothesis.** Exact capture sizes for P10's observed pure-decode batch support
remove decode-bucket slack, so pure-decode padding falls and `E_A2 > E_base`.
No recovery falsifies the efficiency hypothesis; failure to remove the targeted
padding invalidates the ablation rather than supporting a null mechanism.
The P3 P10/C00/rho=0.60 clean Layer-1 stream has SHA-256
`51ad4be12178da91d2af484d0946a2274afd3bcbbee33f37940cfe0ff2ea7fa7`.
Its 17,941 pure-decode steps have the exact decode-B histogram:
| B | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|---:|---:|---:|---:|---:|---:|---:|---:|
| Steps | 13,572 | 2,675 | 792 | 524 | 161 | 202 | 15 |
Defaults already contain 1, 2, 4, and 8. A2 therefore adds exactly
`{3,5,6,7}` and freezes the complete server list as:
```text
1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128
136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256
272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512
```
This covers 100% of P3's observed P10 pure-decode B support. The A2
manipulation requires at least 99% support coverage in the new clean runs and a
90% reduction in pure-decode padding tokens versus base. It targets decode
capture-bucket mismatch; it does not remove prefill-token padding, eager
overflow, graph launch overhead, or length raggedness itself. Capture startup
time and memory are reported as config cost, not treated as free.
### A3 — recorded arrival to uniform arrival
**Hypothesis.** Uniformizing the same rate and request order reduces burst-driven
decode-batch/queue variance, so `E_A3 > E_base`. The mechanism is falsified if
the clean 5-second decode-B CV and waiting-queue CV do not fall, or if the
Holm-corrected efficiency contrast is not positive.
A3 changes only `recorded-scaled` arrival slots to `i/lambda`. Content, order,
input/output lengths, aggregate tokens, server config, prefix caching, and
capture sizes are unchanged. It estimates the total service effect of arrival
shape, including its legitimate downstream changes to batching and queueing.
It does not isolate a scheduler instruction cost. Under the P3-exact fallback,
this arm is the base-equivalent bridge and its arrival share is N/A as described
above.
### A4 — natural prefix caching disabled
**Hypothesis.** Direction is deliberately two-sided. Natural P10 reuse may
increase `E_token` through KV reuse, in which case disabling it gives a negative
share; alternatively, low-value/fragmented cache structure may impose overhead
or alter batching, giving a positive share. Either direction is publishable.
A4 omits only `--enable-prefix-caching`. Prompts, natural repeated-prefix
structure, recorded arrival slots, request order, token totals, scheduler limits,
and capture sizes remain identical to base. Required manipulation checks are
zero local prefix cache hits/queries in the disabled arm and unchanged prompt
hashes. This estimates the contribution of exploiting C structure, not the
intrinsic content similarity of the prompts.
### Mechanisms without a clean ablation
MoE routing skew cannot be removed while preserving P10 content and model
semantics: changing tokens, router weights, top-k, or expert placement changes
more than routing skew. It receives no causal share.
If budget remains, one **analysis-only**, separately started P10 sample may add
`--enable-return-routed-experts`. It is excluded from every `E_token` clean
window and ledger numerator. Offline analysis reports per-layer expert-count
entropy, Gini coefficient, coefficient of variation, max/mean load, and their
association with same-step token-normalized duration. The arrays remain private.
This telemetry is sampled and perturbing, can consume a large scheduler-side
buffer, and provides correlation rather than causal attribution; it can only
help interpret the residual/interaction line. Phase 3's MoE layer-duration CV
was N/A, so it cannot substitute for these routed-expert counts.
## Exact commands and config deltas
The following interface is normative for the later implementation. Variables:
```bash
P5C='python scripts/opprof_phase5_client.py'
PRIVATE=/home/admin/cpfs/wjh/opprof-phase5-private/manifests
P3PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl
P3SOURCE=/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl
MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
RATE=0.4725
```
Materialize the five private manifests without printing prompt text:
```bash
$P5C transform --in "$P3PRIVATE" --take-first 142 \
--timestamp-source "$P3SOURCE" --join-key source_index \
--timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
--service-order original --out "$PRIVATE/P10-base.jsonl"
$P5C transform --in "$P3PRIVATE" --take-first 142 \
--timestamp-source "$P3SOURCE" --join-key source_index \
--timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \
--service-order length-binned --reorder-block-size 32 \
--analysis-cohort-size 16 \
--length-bin-edges 512,1024,2048,4096,8192,16384,32768 \
--max-added-delay-seconds 64 --out "$PRIVATE/P10-A1.jsonl"
$P5C transform --in "$P3PRIVATE" --take-first 142 \
--timestamp-source "$P3SOURCE" --join-key source_index \
--timestamp-field timestamp --arrival uniform --target-rate "$RATE" \
--service-order original --out "$PRIVATE/P10-A3.jsonl"
ln -s P10-base.jsonl "$PRIVATE/P10-A2.jsonl"
ln -s P10-base.jsonl "$PRIVATE/P10-A4.jsonl"
```
The symlinks make unchanged request bytes explicit; the controller hashes the
resolved content and requires A2/A4 hashes to equal base. A1/A3 require equal
sorted request-ID sets and equal input/output token sums.
Common server command (`ARM` is `base`, `A1`, `A2`, `A3`, or `A4`):
```bash
taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU" \
VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
vllm serve "$MODEL" --host 127.0.0.1 --port "$PORT" \
--tensor-parallel-size 1 --enable-chunked-prefill \
--enable-prefix-caching --shutdown-timeout 600
```
- A2 adds
`--cudagraph-capture-sizes 1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512`.
- A4 removes `--enable-prefix-caching`.
- Base, A1, and A3 have no server delta.
Every moderate client command is:
```bash
taskset -c "$CPUSET" $P5C run \
--manifest "$PRIVATE/P10-$ARM.jsonl" \
--base-url "http://127.0.0.1:$PORT" --model "$MODEL" \
--load-point moderate --fixed-request-rate "$RATE" \
--max-concurrency 256 --ignore-eos --temperature 0 \
--warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 \
--post-clean-seconds 0 --drain-timeout-seconds 600 \
--workload-seed 20260712 --server-seed 20260712 \
--result-dir "$RUN_DIR/client"
```
There are no profiler endpoint calls. Exact commands, effective config, startup
capture list, compile-cache key, hashes, clocks, host load, and GPU process list
are recorded per run.
## Execution and validity discipline
### Placement, order, and detached ownership
A-P3-1 already rejected eight-way and authorized four-way placement. Phase 5
therefore uses at most four simultaneous TP1 servers, GPU0--GPU3, with the
frozen disjoint CPU masks `0-19`, `20-39`, `40-59`, and `60-79`. Topology must
still match P3. A changed host topology, source/patch/runtime hash, or clock
policy invalidates reuse of the placement gate and stops for review.
Three independent replicates are run for each of the five primary arms (15
measured runs). Sort all `(replicate,arm)` assignments by SHA-256 of
`"20260715:<replicate>:<arm>"`, pack them into waves of 4, 4, 4, and 3, and
rotate GPU assignments. The final three-target wave uses the frozen P06/C00
saturation background in the fourth slot so measurements retain the validated
four-way host regime. No wave contains two replicates of the same arm; if the
hash order would do so, stable-swap the later item with the first legal item and
record the resolved order before launch. Background data are not analyzed.
As required by A-P3-2, a dash0-resident `setsid`/`nohup` controller owns every
process, has `--resume`, atomically records state, and skips only fully validated
runs with matching hashes. Interactive SSH never owns a server. Before each
wave it logs one echo line with resolved arms, GPUs/CPUs, manifests, rate,
paths, reserved H20-hours, expected duration, and disk headroom.
Use three unmeasured 60-second burn-ins before measured arms: common C00,
A2 capture-list C00, and prefix-cache-off C00. They warm compile/AOT artifacts
but are not measurements. Per-run-unique OpProf directories must remain ignored
by vLLM compile factors, preserving the accepted Phase-2 fix.
### Long-context-safe warm-up and drain gates
These gates are active from the first run; no short-context default is tried
first.
- Warm-up is exactly 60 seconds and excluded. P10 passes with at least 32
successful warm-up completions, **or** at least 16 completions plus the exact
A-P3-6 stabilization test: model-executed steps in `[45,50)`, `[50,55)`, and
`[55,60)`; at least 16 steps/bin; positive scheduled-token rates `R_j`; and
OLS drift `abs(slope)*15/mean(R_j) <= 0.10`. Missing bins, discontinuity, or
accounting failure invalidates the run.
- Drain timeout is 600 seconds. A timeout marks the run drain-quarantined but
does not invalidate an otherwise valid clean window. More than 20% of primary
runs quarantined stops Phase 5 for review.
- The clean window is exactly three contiguous 80-second segments. Admission
and completion accounting follows P3. Offered rate must be within 5% of
0.4725 request/s, with zero clean failures and exact output tokens.
- Layer-1 JSONL/footer/sidecar accounting, zero drops, contiguous steps, no
profile leakage, clock/load capture, other-user-process absence, and final
zero GPU memory are hard gates inherited from P3.
No semantic failure is retried with altered parameters. One exact retry is
allowed for an infrastructure artifact failure, and both attempts remain in
the operational findings.
## Control reuse, secondary scope, and budget
P03/P04 P3 controls may be reused because they already use the same model,
patch, C00-TP1 config, normalized load, 240-second clean metric, placement
regime, and Layer-1 schema. Reuse avoids spending GPU time without changing the
denominator.
Reuse is valid only if source, patch, model, runtime, clocks, and placement
hashes match and fresh A3-uniformized P10 differs from frozen P3 P10 by at most
3% in `E_token` with a bootstrap CI containing zero difference. If this bridge
gate fails, temporal/runtime drift is plausible: rerun P03 and P04 under their
exact P3 manifests and commands, three replicates each, before computing any
share. A failed bridge never licenses rescaling old controls.
After the complete P10 ledger, optional work is ordered as follows and starts
only if the controller's conservative reservation remains below the 6.0
H20-hour hard cap:
1. one saturation run per P10 arm, using the full 4,011-row manifest, for
descriptive mechanism persistence only;
2. one five-arm moderate ledger each for P09 and P06, reported as exploratory
within-run-bootstrap evidence, not equal replication to P10; and
3. one routed-expert analysis-only sample.
For secondary A2, the precomputed P3 pure-decode support additions are P09
`{3,5,6,7,9,10,11,12,13,14,15,17,18,19,20,21,22,23,25}` and P06
`{3,9,10,11,12,13,14,15}`; default sizes are retained. P09 A3 is a no-change
steady negative control, while P06 A3 changes its registered burst-8 arrivals
to uniform spacing at the same mean rate. P09/P06 conclusions are always
labeled secondary.
Expected accounting, including server-owned startup/shutdown time:
| Tier | New measured runs | Expected H20-hours | Expected 4-way wall |
|---|---:|---:|---:|
| P10: 5 arms x 3 replicates | 15 | 1.6-1.9 | 35-55 min |
| Three compile burn-ins | 0 | 0.1-0.2 | 3-6 min |
| Conditional P03/P04 reruns | 6 | 0.6-0.8 | 15-25 min |
| Optional P10 saturation | 5 | 0.5-0.8 | 15-25 min |
| Optional P09/P06, 5 arms each | 10 | 0.9-1.2 | 20-35 min |
| Optional routed-expert sample | 1 analysis-only | 0.1-0.2 | 5-10 min |
| **Maximum planned** | **36 ledger + 1 analysis-only** | **3.8-5.1 expected** | **about 1.5-2.5 h** |
The hard cap is **6.0 H20-hours new Phase-5 spend**, not a target. Actual time
while a server owns GPU memory is charged. Optional tiers are skipped rather
than overrunning the cap. The 600-second drain allowance is a watchdog, not an
assumption in the expected estimate; repeated long drains consume the optional
budget first.
With no Kineto traces, primary P10 artifacts are estimated at 0.4-0.6 GB and
all planned public artifacts below 1.5 GB. Stop if public Phase-5 artifacts
exceed 3 GB or CPFS free space falls below 100 GB. Prompt-bearing manifests and
routed-expert arrays remain in mode-0700 private storage and are not counted as
public deliverables.
## Statistical analysis and decision rules
Use 5-second moving-block bootstrap over clean time, 100,000 resamples, seed
20260716. For P10, first resample the three run IDs, then resample 5-second
blocks within each selected run; arms and reused P3 controls are resampled
independently. Every `E`, `delta`, share, share sum, and residual/interaction
gets a percentile 95% CI. Absolute `E` and delta accompany every ratio.
Bootstrap ratio draws are never deleted because their denominator is
inconvenient. If either control gap has a point estimate `<=0`, its CI includes
zero, or more than 5% of bootstrap denominator draws are `<=0`, that control's
ledger is **INCONCLUSIVE (unstable denominator)**.
The confirmatory family is the four two-sided tests of `E_ablated-E_base=0` on
P10. Apply Holm correction at family-wise alpha 0.05 across A1--A4. The same
corrected delta test serves both control ledgers; duplicating denominators does
not create eight tests. A1--A3 only support the expected recovery claim when
their corrected contrast is positive. A4 may be significant in either
direction. Manipulation-check failure makes the corresponding share N/A even
if efficiency changes.
A mechanism is **dominant** only if, under both P03 and P04 ledgers:
- point `share >= 0.30`;
- the 95% share CI excludes 0.15 on the high side (`CI_low > 0.15`); and
- its Holm-corrected efficiency contrast is significant in the expected
direction (two-sided for A4).
Meeting the rule under only one control is reported as **control-sensitive**,
not dominant.
The primary ledger is **publishable** when all five P10 arms have three valid
replicates, both control denominators are stable, all four manipulation checks
are evaluable, every share/residual has a finite CI, the fresh A3/P3 bridge is
reported, and no privacy/data-sanity red flag exists. It may be publishable
with no dominant mechanism. It is **inconclusive** if any primary arm is
missing, a denominator is unstable, the arrival-rebase decision is unresolved,
or two or more share CIs have width greater than 0.50. One failed mechanism
manipulation yields a publishable partial ledger only if that line is explicitly
N/A and the headline claim excludes it.
No additive causal claim is made from `sum(share_m)`. Pairwise and higher-order
interactions are not identified by this one-factor-at-a-time matrix; a combined
all-off arm would be a new experiment requiring amendment, not an improvised
way to close the ledger.
## Deliverable and artifact contract
Execution, if later approved, must produce:
- `docs/opprof/phase5-results.md`;
- `runs/opprof-phase5/phase5/metrics.json` with schema, arm/run values,
bootstrap draws' seed and summary, Holm results, dual-control ledgers,
residual/interaction, gates, and GPU accounting;
- per-run exact commands, environment/provenance, client records, Layer-1
stream/footer/sidecar, monitor data, and machine-readable sanity JSON; and
- an **Operational findings** section covering warm-up stabilization, drains,
compile/capture startup cost, contamination, retries, and any mismatch between
expected and realized mechanism removal.
Every raw-run summary, aggregate table, and final metrics file ends with the P3
sanity schema: `n`, finite/missing, min, max, distinct count, and applicable
invariants. Red flags are reported first and stop inferential analysis. Public
artifacts may contain request IDs, hashes, lengths, counts, and timing only;
prompt, messages, content, generated text, source substrings, and routed-expert
arrays are forbidden.
## Final ablation table
| Mechanism | What changes | What is preserved |
|---|---|---|
| Base | Recorded P10 timestamps are rate-normalized and replayed in source order | Frozen 142 requests, content/tokens, C00-TP1, prefix cache on, default capture list, `lambda=0.4725` |
| A1: length raggedness | Stable length-bin sort within 32-request reorder blocks into 16-request cohorts, 64-second late cap | Requests/content/token totals, arrival-slot vector, output lengths, server config, prefix setting |
| A2: capture mismatch | Add exact P10 decode-B sizes `{3,5,6,7}` to the full default capture list | Manifest/order/arrivals/content/tokens, scheduler limits, prefix setting |
| A3: arrival dynamics | Recorded-scaled slots become uniform `i/0.4725` slots | Request order/content/tokens, mean rate, server/capture/prefix config |
| A4: prefix structure | Remove `--enable-prefix-caching` | Natural prompt structure, request order/arrivals/content/tokens, capture/scheduler config |
| Residual + interactions | No extra run; joint arithmetic remainder `1-sum(shares)` | Raw unnormalized mechanism shares; no clipping or forced 100% allocation |
## Final run count and GPU estimate
Primary commitment: **15 new measured P10 runs plus three unmeasured burn-ins,
1.7-2.1 expected H20-hours, 35-60 minutes four-way wall, and 0.4-0.6 GB public
disk**. Conditional control reruns add 6 runs; all optional tiers bring the
maximum to **36 ledger runs plus one analysis-only run, 3.8-5.1 expected
H20-hours, about 1.5-2.5 hours wall, and less than 1.5 GB public disk**. New
Phase-5 GPU use stops at **6.0 H20-hours** under all circumstances.
## Resolved decisions from the orchestrator
1. **Approved:** use the recommended recorded-arrival P5 bridge ledger, with
its explicit limitation that it is not a literal decomposition of P3's
already-uniform P10 gap; otherwise select the P3-exact A3=N/A fallback.
2. **Approved:** dual P03/P04 control ledgers and the requirement that “dominant” hold
under both, rather than selecting one P3 denominator.
3. **Approved:** the 142-request slice, 32-request reorder blocks, 16-request analysis
cohorts, fixed bins, and 64-second fairness cap as A1's isolation/latency
tradeoff.
4. **Approved:** three P10 replicates, reuse of P3 controls behind the 3% bridge gate,
and the optional-tier order within the 6.0-H20-hour cap.
5. **Approved:** Layer-1-only primary runs and treating routed-expert telemetry as
private analysis-only evidence with no causal share.
## Protocol sanity block
| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| P3 control/base `E_token` | 3 | 2.619113 | 4.735600 | 3 | Finite, positive, not identical |
| P3 P10 control gaps | 2 | 0.435590 | 2.116487 | 2 | Positive; dual denominators retained |
| P10 request rows/arm | 5 arms | 142 | 142 | 1 expected | Same IDs and input/output token sums required |
| Selected source timestamps (s) | 142 | 0.014 | 21.445 | 142 | Finite, nondecreasing; gap min/max 0.002/0.851 s, 127 distinct gaps |
| Primary offered rate (req/s) | 5 arms | 0.4725 | 0.4725 | 1 expected | Positive; achieved rate must be within 5% |
| Warm-up / clean / drain gates (s) | 3 values | 60 | 600 | 3 | Clean is exactly `3*80=240`; long-context drain is 600 |
| A1 input-length bins | 7 | 0 | 32768 | 7 intervals | Ordered, contiguous, cover frozen P10 range |
| A1 frozen `R16` values | 2 | 0.473409 | 0.641744 | 2 | Sorted is lower by 0.168334 absolute / 26.23% relative; not identical |
| A1 added-delay values (s) | 142 | 0 | 62.744 | >1 expected | Non-negative and all below 64-second cap |
| P3 P10 pure-decode steps | 17,941 | B=1 | B=7 | 7 B values | Counts sum to 17,941; non-negative; stream SHA pinned |
| P10 A2 added capture sizes | 4 | 3 | 7 | 4 | Exactly missing observed support `{3,5,6,7}`; 100% P3 support covered |
| Primary measured runs | 15 | 3/arm | 3/arm | 1 expected | `5 arms * 3 replicates`; no arm omitted |
| Maximum GPU analysis/ledger runs | 1 plan | 37 | 37 | 1 expected | `15+6+5+10+1=37`; burn-ins excluded |
| Expected H20-hours | 1 plan | 3.8 | 5.1 | 2 bounds | Finite, non-negative, below hard cap 6.0 |
| Share domain | 5 ledger lines/control | Unbounded | Unbounded | N/A | No clipping; ratios may be negative or >1 |
| Bootstrap resamples | 1 setting | 100,000 | 100,000 | 1 expected | Seed 20260716; no denominator-draw deletion |
| Phase-5 GPU runs in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
Checked invariants: `0.60*0.7875=0.4725`; P3 control gaps are positive;
the seven decode-B counts sum to 17,941; default plus `{3,5,6,7}` covers all
observed P10 pure-decode B values; `5*3=15` primary runs; maximum planned count
is `15+6+5+10+1=37`; expected GPU use remains below the 6.0-hour hard cap;
ratios are not constrained to `[0,1]`; expected constants are labeled; and no
GPU command, helper change, manifest transform, or experiment was executed in
this protocol-only turn. The unresolved P3-arrival/P5-arrival estimand mismatch
is reported first as a blocking decision rather than hidden in the ledger.

223
docs/opprof/phase5-results.md Normal file
View File

@@ -0,0 +1,223 @@
# OpProf Phase 5 dash0 results
Status: **FINAL — INCONCLUSIVE MECHANISM LEDGER; NO DOMINANCE CALL**.
Date: 2026-07-12. All 15 registered P10 primary runs completed with three
replicates per arm and passed A-P5-1. The fresh A3 bridge passed, so the frozen
P3 P03/P04 controls were reused. The ledger is nevertheless inconclusive:
P04's reused-control denominator is unstable, and A2, A3, and A4 fail their
predeclared manipulation checks. The protocol therefore permits one official
share (A1 under P03), requires the other lines to be N/A, and forbids a
dual-control dominance conclusion.
The machine result is `runs/opprof-phase5/phase5/metrics.json` (SHA-256
`fe16934866806ac542f7974cac061beeb1f1c58f5bfb36698efd8aeaa99b8cb7`).
The final analyzer SHA-256 is
`8b0ff8658614227ee00cf8d10c82c4b5edf6f6210db172fd6efc9cc149e51e10`;
its no-GPU tools and analysis tests pass.
This remains explicitly a **recorded-arrival P5 bridge ledger anchored to P3
controls**, not a literal decomposition of P3's already-uniform P10 gap.
## Data-sanity result first
**RED FLAG: the P04 control denominator is unstable.** P5 recorded-arrival
base has `E_token=3.013752`, while reused P3 P04 has `E_token=3.054704`, leaving
only a `0.040952` gap with bootstrap 95% CI `[-0.649096, 0.752594]`; 45.353% of
denominator draws are nonpositive. This violates every registered stability
form: the CI contains zero and the nonpositive fraction exceeds 5%.
P03 remains stable: its gap is `1.721848`, CI `[1.459092, 1.988772]`, with zero
nonpositive draws. All execution-validity checks pass: 15/15 primary runs,
three replicates per arm, exact 240-second clean windows, zero clean failures,
offered rates within 5%, continuous/balanced Layer-1 accounting, identical
request IDs and token sums, all A-P5-1 cold-start gates, no drain quarantine,
and GPU spend below the cap. The red flag is reported before the ledger and no
conclusion is built on P04's raw ratio draws.
## Bridge and absolute efficiency
A3 reproduces the P3 uniform-arrival P10 base: `delta_E=+0.007041`, 95% CI
`[-0.329799, 0.354859]`, or 0.269% absolute relative difference. The CI contains
zero and the point difference is below 3%, so the bridge passes and P3 control
reuse is valid under the frozen rule. Conditional control reruns were not
launched.
| Arm | `E_token` | Bootstrap 95% CI | Delta from P5 base |
|---|---:|---:|---:|
| Base, recorded | 3.013752 | [2.754462, 3.267964] | — |
| A1 length-binned | 3.078999 | [2.801659, 3.349307] | +0.065247 |
| A2 capture sizes | 3.062128 | [2.802812, 3.314276] | +0.048376 |
| A3 uniform arrival | 2.626154 | [2.451852, 2.792684] | -0.387598 |
| A4 prefix cache off | 3.020959 | [2.760494, 3.274105] | +0.007207 |
## Manipulation checks and Holm family
| Arm | Manipulation result | Efficiency delta 95% CI | Raw p | Holm p |
|---|---|---:|---:|---:|
| A1 | **PASS:** R16 0.641744 -> 0.473409; max delay 62.743 s; prefix-hit-ratio delta -0.296 pp | [-0.309461, 0.442565] | 0.73108 | 1.00000 |
| A2 | **FAIL:** support coverage 98.879% <99%; padding reduction 82.541% <90% | [-0.315361, 0.411446] | 0.79022 | 1.00000 |
| A3 | **FAIL:** decode-B CV falls 0.8706 -> 0.6551, but waiting CV rises 6.0145 -> 6.8557 | [-0.693444, -0.078141] | 0.01384 | 0.05536 |
| A4 | **FAIL:** disabled arm still reports 3,093,324 local prefix queries and 52,896 hits, not zero | [-0.357637, 0.369309] | 0.97230 | 1.00000 |
Per protocol, a failed manipulation makes that mechanism's official share N/A
even if its raw efficiency contrast is nonzero. A3's uncorrected contrast is
negative, and it also misses family-wise significance after Holm correction.
## A-P5-1 arrival evidence
The four originally invalidated arms supplied the evidence for A-P5-1. Under
the superseded throughput-drift calculation, recorded base/A1/A4 span
190.55%-216.50% normalized drift, while uniform A3 is 13.23%. This gap is
retained as direct evidence that the old gate measured arrival shape in a
rate-following experiment; it is not converted into an efficiency share.
All 15 replacement/continued primary runs pass the amended cold-start gate:
25-28 warm-up completions, 12-16 completions with input at least 8192 tokens,
all compile/capture first occurrences before clean, zero clean-window
first-capture events, and zero uncovered Layer-1 graph descriptors. The 10%
drift gate was retained unchanged for the P06 saturation background arm.
## Operational findings
- The detached controller completed three cache burn-ins, four four-way waves,
all drains, validation, budget accounting, and cleanup. The four invalidated
first-wave arms were rerun exactly; none of their superseded measurements
enters the ledger.
- The first exact rerun produced complete clean artifacts but hit a post-hoc
validator `NameError` from a missing `math` import. Immutable artifacts were
CPU-re-adjudicated under A-P5-1, the import was repaired, and no third GPU run
was charged.
- CPU analysis repairs admitted legitimate idle fixed-time blocks as `[0,0]`,
supported the older P3 control schema, converted one NumPy boolean at the
JSON boundary, and explicitly separated diagnostic from reportable shares.
These changes alter no GPU data, estimator, seed, threshold, or resample
count.
- P10 drains range from 0.676 to 6.004 seconds. The separate P06 saturation
background run drained naturally in 87.903 seconds, below the 600-second
gate, with 564/564 clean completions and zero failures.
- Public Phase-5 artifacts occupy 286,205,365 bytes (304 MiB), below the 3 GB
stop threshold; CPFS retained 1,286 GB free.
- MoE routing skew remains content-bound and receives no causal share. The
optional routed-expert analysis-only run was not required for this primary
ledger and was not launched.
## Launch echo
```text
LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails
```
Resume after A-P5-1 reused the completed burn-ins and compile/cudagraph caches.
The first launch through final controller completion took approximately 57.9
minutes wall time, including the adjudication/resume interval.
## Run completion stats
| Item | Count/result |
|---|---:|
| Valid primary P10 runs | **15/15** |
| Replicates | **3 per arm** |
| Valid P06 background runs | 1 |
| Completed burn-ins | 3 |
| Superseded pre-amendment arms | 4 |
| Conditional fresh controls | 0; bridge passed, P3 reused |
| Optional secondary/routed-expert runs | 0 |
| A-P5-1 primary passes | 15/15 |
| Clean failures | 0 |
| Drain quarantines | 0 |
## Mechanism ledger
Official shares follow the frozen N/A rules. Values in parentheses are raw
diagnostic ratios retained for audit, not causal/reportable shares.
| Mechanism | P03 share (95% CI) | P04 share (95% CI) | Disposition |
|---|---|---|---|
| A1 length raggedness | **0.0379 [-0.2011, 0.2344]** | N/A; unstable denominator (diagnostic 1.5933 [-6.8925, 7.0636]) | Manipulation passes; no significant recovery |
| A2 capture mismatch | N/A; manipulation failed (diagnostic 0.0281 [-0.2049, 0.2170]) | N/A; unstable denominator (diagnostic 1.1813 [-6.2871, 6.4924]) | Config did not meet 99%/90% removal gates |
| A3 arrival dynamics | N/A; manipulation failed (diagnostic -0.2251 [-0.4625, -0.0403]) | N/A; unstable denominator (diagnostic -9.4647 [-17.2450, 17.4102]) | Queue-variance gate failed; Holm p=0.05536 |
| A4 prefix structure | N/A; manipulation failed (diagnostic 0.0042 [-0.2340, 0.1948]) | N/A; unstable denominator (diagnostic 0.1760 [-6.2590, 6.4925]) | Zero-query/hit gate failed |
Shares are neither clipped nor renormalized. P04's extreme raw ratios are the
registered consequence of dividing by a near-zero, sign-unstable denominator,
not evidence of large mechanisms.
## Dominance verdicts
| Mechanism | P03 rule | P04 rule | Dual-control verdict |
|---|---|---|---|
| A1 | Fails share/CI/Holm thresholds | Not evaluable | **NOT EVALUABLE** |
| A2 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
| A3 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
| A4 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** |
No mechanism may be called dominant because the frozen rule requires both
control denominators. This is an inconclusive verdict, not evidence that every
mechanism is small.
## Residual + interaction line
The official residual is N/A under both controls because the official share
ledger is incomplete. The unchanged arithmetic over all raw diagnostic shares
is reported only for transparency:
| Control | Diagnostic share sum | Diagnostic residual + interaction (95% CI) | Status |
|---|---:|---:|---|
| P03 | -0.1549 | 1.1549 [0.5536, 1.9397] | Diagnostic only |
| P04 | -6.5142 | 7.5142 [-21.3965, 22.8767] | Diagnostic only; unstable denominator |
Nothing is forced to 100%. The residual can include interactions, unablated
MoE routing, chunked-prefill effects, double-counting, and measurement error.
## Config-tier A2 measured delta
A2 added the P3-derived sizes `{3,5,6,7}` to the full default capture list.
Its measured `delta_E` is **+0.048376**, 95% CI
`[-0.315361, 0.411446]`, or **+1.605%** relative to base. Pure-decode padding
falls from 13.7405% to 2.3989%, an 82.541% reduction. New recorded-arrival
runs expose decode-B support 1-13, so uncaptured 9-13 leave coverage at 98.879%;
both this and the sub-90% padding reduction invalidate the causal A2 share.
The config cost is reproducible: base graph capture is 11 seconds and 0.80 GiB
in all three runs; A2 is 12-14 seconds and 0.84 GiB, a 1-3 second and 0.04 GiB
increase. The measured config-tier result is therefore a small, statistically
uncertain efficiency gain with a failed preregistered removal gate.
## GPU total
New Phase-5 spend is **2.4388167443 H20-hours** of the 6.0-hour cap, including
the superseded first wave, exact reruns, continued primary matrix, burn-ins,
and background arm. **3.5611832557 H20-hours remain unspent.** Final state on
all eight H20s is 0 MiB and 0% utilization with zero compute processes.
## Sanity block
**Anomaly first:** P04's gap CI contains zero and 45.353% of its denominator
draws are nonpositive; the dual-control ledger is inconclusive and inferential
claims stop there.
| Numeric family | n | Finite | Missing | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---:|---:|---|
| Primary run `E_token` | 15 | 15 | 0 | 2.623534 | 3.083263 | 15 | Positive; per-arm results not all identical |
| Clean Layer-1 steps | 15 | 15 | 0 | 11,178 | 18,400 | 15 | Non-negative; continuous/balanced per run |
| Clean duration (s) | 15 | 15 | 0 | 240 | 240 | 1 expected | Exact `3*80`; zero clean failures |
| Offered rate (req/s) | 15 | 15 | 0 | 0.470833 | 0.487500 | 2 | Every run within 5% of 0.4725 |
| Warm-up completions | 15 | 15 | 0 | 25 | 28 | 2 | Every run >=16 |
| Warm-up long completions | 15 | 15 | 0 | 12 | 16 | 3 | Every run >=1 with input >=8192 |
| Clean first-capture events | 15 | 15 | 0 | 0 | 0 | 1 expected | Zero events and zero uncovered descriptors |
| P10 drain duration (s) | 15 | 15 | 0 | 0.676181 | 6.003959 | 15 | Non-negative; all below 600 |
| Control gap `E_token` | 2 | 2 | 0 | 0.040952 | 1.721848 | 2 | P03 stable; **P04 unstable red flag** |
| Raw diagnostic shares | 8 | 8 | 0 | -9.464746 | 1.593268 | 8 | Finite; intentionally not constrained to [0,1] |
| Diagnostic residuals | 2 | 2 | 0 | 1.154931 | 7.514211 | 2 | Finite; never forced to close |
| New H20-hours | 1 | 1 | 0 | 2.438817 | 2.438817 | 1 | Non-negative; below 6.0 |
Checked invariants: source/helper/protocol provenance; identical request IDs,
content/token totals, and prompt-preserving manifests; C00-TP1 four-way
placement; detached controller ownership; exact clean windows; zero failures;
offered-rate tolerance; A-P5-1 compile/capture ordering and coverage; Layer-1
footer/sidecar balance, continuous step indices, token composition, and zero
drops; non-negative counters; drain gates; bridge decision; no public prompt or
generated-text leakage; public-disk/free-space limits; GPU hard-cap compliance;
and zero-process/zero-memory cleanup. Manipulation failures are A2, A3, and A4;
the sole machine sanity red flag is P04 denominator instability.

638
docs/opprof/phase6-protocol.md Normal file
View File

@@ -0,0 +1,638 @@
# OpProf Phase 6 pre-registered cross-version churn protocol
Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**.
Date frozen: 2026-07-12 (Asia/Singapore). Phase 6 asks whether the vLLM 0.20.0
best configuration and ranking on the C1
`TP={1,2,4} x max-num-seqs={8,16,32,64}` surface survive an upgrade to patched
vLLM 0.24.0. This is a paired historical-anchor experiment, not a fresh tuner
search. It measures churn at the exact recorded C1 request sets and reports
where the recorded local frontier moved.
## Approved dispositions (orchestrator; 2026-07-12)
All five open decisions are approved exactly as proposed, without changing an
anchor, threshold, estimator, validity gate, or budget:
1. The upgrade-path estimand is approved. Resolved default changes are part of
observed churn; dash1->dash0 and co-location remain explicit limitations and
the result is not described as a pure engine-version causal effect.
2. The adaptive 25-anchor subset is approved: every cell receives its old peak
plus the direction-relevant adjacent anchor, and TP4/MNS16 receives both
neighbors.
3. The 5% material-frontier threshold, `tau-b>=0.8` ranking-survival threshold,
and zero-anchored floor-bucket argmax/trap rules are approved.
4. One primary observation per historical anchor plus the 0.35-H20-hour
confirmation reserve is approved.
5. The A-P5-1-class warm-up long tier is approved as raw input `>4096` tokens
for this 0-8192 workload.
This approval authorizes the later execution only after pinned-input, detached
ownership, cold-start, echo-before-launch, placement, privacy, cleanup, and
projected-budget gates pass. The 3.0-H20-hour cap remains absolute.
## Amendment A-P6-1 — checkpoint-sidecar accounting and conditional cap
The first Phase-6 W1 request replays were incorrectly rejected by requiring a
clean in-stream footer and `final=true` sidecar after a non-graceful shutdown.
That contradicts the already accepted A-P3-5/P5 accounting rule. Before any
resume, all four W1 cell streams and their eight measured anchor intervals must
be CPU-re-adjudicated against the latest atomic checkpoint sidecar. With no
in-stream footer, the sidecar is authoritative when every complete JSONL line
decodes, step indices are contiguous, on-disk data-record count equals
`written_records`, final data-line step equals `last_step_index`,
`encoded_records = written_records + dropped_records`, `dropped_records=0`, and
the checkpoint is within the configured one-second flush interval (plus the
accepted 100-ms scheduling tolerance) of the recorded shutdown boundary. At
most one post-checkpoint flush interval may be absent. A balanced stream accepts
all covered W1 replay intervals without a GPU rerun.
All subsequent waves use the graceful P5 path: start `vllm serve` with positive
`--shutdown-timeout`, signal each API parent with SIGINT, wait up to 150 seconds
for EngineCore drain/finalization, and use process-group SIGTERM/SIGKILL only as
fallback. The preferred result is one in-stream footer plus an agreeing
`final=true` sidecar; the atomic checkpoint rule remains the crash fallback.
Only if any W1 sidecar fails the CPU balance/coverage gate does A-P6-1c activate:
the hard cap becomes **3.5 H20-hours**, the activation is logged before launch,
and W1 is rerun under graceful shutdown. If all W1 sidecars balance, A-P6-1c is
not activated, W1 is accepted in place, its already charged 0.299393 H20-hours
is retained, and execution resumes at W2 under the original 3.0-hour cap.
## Amendment A-P6-2 — authoritative solo frontier tier and 6.0-hour cap
The user raises the cumulative Phase-6 hard cap from **3.0 to 6.0 H20-hours**.
This amendment responds to the W2/W3 confirmation evidence: when neighboring
wave clients became idle, pass rate changed from `0.412 -> 1.000` and `0.028 ->
0.957` at TP2/MNS32 and from `0.036 -> 1.000` at TP4/MNS16, while Layer-1
waiting means collapsed from `1.22 -> 0.00`, `8.50 -> 0.35`, and `28.33 ->
0.06`. Because the v0.20 C1 baseline ran one cell at a time on dash1,
co-located SLO feasibility is not a valid authoritative tier for the paired
frontier. Co-location remains an explicitly reported, metric-dependent
methodological ablation; its throughput-only observations are indicative.
Every SLO-frontier-decisive Phase-6 anchor is therefore run **solo**: exactly
one vLLM server and one replay client are active on dash0, no other Phase-6
server/client is resident, and only the cell's `TP` GPUs are allocated. Each
solo cell gets a fresh server process, the frozen exact-selection client, an
independent 16-request long-tier warm-up, A-P5-1 cold-start gates, Layer-1
telemetry, and graceful A-P6-1 shutdown. The solo result is authoritative for
feasibility, frontier, floor bucket, argmax, tau-b, and trap decisions. A
co-located-only result may not fill a missing solo frontier or support a paper
claim.
The mandatory solo set has no cell exclusions:
| Cell set | Mandatory solo anchors | Inclusion reason |
|---|---|---|
| TP1/MNS8 | existing L+P | co-located peak failed |
| TP1/MNS16,32,64 | existing P+U | co-located U passed and censored the frontier |
| TP2/MNS8,16 | existing L+P | co-located peak failed; L distinguishes bracket vs left censoring |
| TP2/MNS32 | existing L+P | old argmax; both anchors split primary/confirmation |
| TP2/MNS64 | existing L+P | the indicative `>29.4% down` bound is unquotable until solo-confirmed |
| TP4/MNS8 | existing L+P | co-located peak failed; trap-neighbor comparison |
| TP4/MNS16 | existing L+P+U | named trap; peak split and L failed co-located |
| TP4/MNS32,64 | P plus direction-relevant L/U | W4 completion; unmeasured under the 3.0-hour cap |
After these 25 anchors, the controller may crawl only the pinned 92-anchor
history: upward while the highest solo anchor passes or downward while the
lowest solo anchor fails, stopping at the first opposite-feasibility anchor.
There is no interpolation and no unrecorded request set. A pass rate in
`[0.93,0.97]`, or a solo/co-located disagreement that changes argmax/trap or a
material-frontier claim, triggers a same-solo-placement confirmation; a split
gets a third trial for the frozen 2-of-3 rule if projected budget permits.
Unresolved or history-edge frontiers remain explicitly censored.
Priority is W4, TP2/MNS32, TP2/MNS64, TP4/MNS16, then the remaining failed or
censored cells, so a budget stop preserves the most decision-relevant evidence.
All twelve cells are nevertheless included. The planning envelope is 3.44 new
H20-hours for serialized startup, warm-up, 25 mandatory anchors, bounded crawl,
graceful drain, and a 0.20-hour safety reserve. Added to the already charged
2.291173 hours, the launch projection is **5.731173/6.0 H20-hours**. Before
each one-cell wave, the controller echoes cell, anchors, GPU placement, input
paths, charged spend, remaining projection, and cap. Projected or actual spend
reaching 6.0 stops further GPU work.
Final analysis preserves the complete W1-W3 co-located attempt history and
reports a solo-versus-co-located delta table for every exact anchor pair.
ARGMAX, RANKING, and TRAP use solo values only. RANKING remains inconclusive
and tau-b non-evaluable unless all 12 solo frontiers are bounded under the
pinned history.
## Headline claim and falsifiable outcomes
The paper-facing question is:
> Does the C1 vLLM 0.20.0 per-GPU SLO-feasible ranking, especially global best
> TP2/MNS32 and local trap TP4/MNS16, remain valid on vLLM 0.24.0 when the exact
> historical request-selection and SLO semantics are replayed?
Phase 6 can produce four distinct outcomes:
1. **SURVIVES:** TP2/MNS32 remains in the top floor bucket, the full-surface
Kendall tau-b is at least 0.8, and the TP4/MNS16 trap persists.
2. **ARGMAX MOVED:** TP2/MNS32 is not in the top v0.24 measured-frontier bucket.
3. **RANKING MOVED WITHOUT ARGMAX MOVEMENT:** the old best survives but tau-b is
below 0.8 or the trap changes status.
4. **INCONCLUSIVE/PARTIAL:** a decision-critical frontier is unbracketed, an
anchor is not reproducible, repeated boundary verdicts disagree, the 12-cell
surface is incomplete, or the 3.0-H20-hour cap stops required work.
The experiment quantifies observed upgrade churn. Because engine version,
host, resolved defaults, and co-location differ together, it does not identify
a pure causal effect of one vLLM commit.
## Frozen vLLM 0.20.0 surface
The historical objective is the maximum measured SLO-feasible offered rate
divided by TP:
```text
f20(c) = max feasible selected_request_count / 60 seconds / TP
```
| TP | MNS8 | MNS16 | MNS32 | MNS64 |
|---:|---:|---:|---:|---:|
| 1 | 2.1000 | 2.3500 | 2.2833 | 2.2833 |
| 2 | 2.2750 | 2.2750 | **3.2833** | 3.2583 |
| 4 | 1.2833 | **2.4417** | 2.4417 | 2.4417 |
Values are request/s/GPU. The global best is TP2/MNS32. TP4/MNS16 is the
registered local trap: it is the first point on a TP4 plateau, has no improving
adjacent MNS move, but is below the global best.
The pinned ground-truth asset is
`/home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json`, SHA-256
`23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670`.
It contains 92 probe observations: eight anchors for each TP1/TP2 cell and seven
for each TP4 cell.
## Comparability contract
### Exact workload and selection semantics
The private materialized workload is
`trace_windows/traces/chat_w20260311_1000.jsonl`, 32,606 rows, SHA-256
`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`.
Its window record is `trace_windows/windows.json`, SHA-256
`23c432e7439508b07991cabfe1db9977ebb55f8ddc39ca214a627fc5f5ae4725`:
window `[0,600]` seconds becomes `[0,60]` under replay scale 0.1.
The later helper must call or byte-for-byte reproduce the following pinned
AITuner implementations, rather than use the Phase-3/5 client:
| Source | SHA-256 | Frozen role |
|---|---|---|
| `src/aituner/trace.py` | `6bb19f0e3eb42b1aecc6969a5f20d65eb505a91564dd117dd149155a6114eb35` | load, stable time sort, cap/downsample, threshold selection |
| `src/aituner/worker.py` | `18337fe273a48d1cbb2bc364f774d100f8ffa2be5f9cd6a7c58c8ed4717c5e48` | rate-following submit loop, early stop, denominator accounting |
| `src/aituner/slo.py` | `1d2e5b1ed9d0d6d6c07f8b5b6fa56d36ed9bb62710be54c125e9b2890f4f96d5` | stepped SLO evaluation |
| `src/aituner/http_client.py` | `213b80d5b37cc74333517f46ab4e8cdba1a274bdbfabdfc14207b314f61799ac` | urllib/SSE TTFT, TPOT, usage accounting |
The exact order is normative:
1. Read all JSONL rows in file order. Retain raw `input_length` in `[0,8192]`.
2. Build the same chat body and stable-sort by scaled arrival time only. Stable
ties retain source order; sorting by `(timestamp,sampling_u,...)` is forbidden.
3. For TP1/TP2, even-downsample the full filtered, time-sorted list to 512
entries **before** threshold selection, using index
`floor(i*N/512)` for `i=0..511`. TP4 is uncapped.
4. Select exactly the requests with `sampling_u <= anchor`. Preserve their
selected service order. Do not resample, wrap, shuffle, or interpolate.
5. Submit at each row's materialized `timestamp*0.1`; do not uniformize,
rebase individual gaps, or replace the recorded burst process.
6. Send the original messages through `/v1/chat/completions` with `stream=true`,
`stream_options.include_usage=true`, and the row's temperature when present.
Set both `min_tokens=128` and `max_tokens=128`.
7. Use maximum concurrency 64. Client request timeout remains 900 seconds.
8. Verify `usage.completion_tokens==128`; missing usage, a different token count,
HTTP failure, timeout, and every request not submitted because the SLO verdict
became unrecoverable are failures in the original selected-request denominator.
9. Preserve original early-stop logic: after more than
`N-ceil(0.95*N)` failed evaluations, synthesize failures for the remaining
selected requests. Disable adaptive Stop-A; it was absent in C1.
An offline, prompt-free preflight must regenerate all 92 historical request
counts exactly. The local reconstruction already yields 17,710 filtered rows,
512 capped rows, and **92/92 count matches**. Any future mismatch in count,
request-ID hash, arrival hash, raw-length hash, or requested-token sum stops the
GPU launch.
### Exact SLO and score
Request feasibility uses the original raw trace input length, not tokenizer
usage after chat templating:
```text
raw input <= 4096: TTFT <= 2000 ms
raw input <= 32768: TTFT <= 4000 ms
otherwise: TTFT <= 6000 ms
all requests: TPOT <= 50 ms
anchor feasible: SLO-passing selected requests / all selected requests >= 0.95
```
Although the C1 filter limits inputs to 8192, the complete three-step rule is
retained. TTFT is client wall time from request start to first nonempty content.
TPOT is `(last_content_time-first_content_time)/(usage_completion_tokens-1)`.
The paired surface score is always selected count divided by 60 seconds and TP;
completed throughput is a diagnostic and never replaces the historical score.
### Serving configuration
The v0.24 command repeats the explicit C1 launch surface:
```bash
MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
VENV=/tmp/wjh-opprof-phase2-dash0-20260711/.venv
RUN_ROOT=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6
taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU_IDS" \
VLLM_OPPROF_DIR="$RUN_DIR/opprof" \
"$VENV/bin/vllm" serve "$MODEL" \
--host 127.0.0.1 --port "$PORT" \
--served-model-name qwen3-30b-a3b-community \
--max-num-batched-tokens 8192 \
--max-num-seqs "$MNS" \
--tensor-parallel-size "$TP" \
--shutdown-timeout 120
```
`--shutdown-timeout` is operational and outside the measured interval. As in
C1, do **not** explicitly set block size, GPU-memory utilization, prefix
caching, chunked prefill, CUDA-graph capture sizes, DP, or EP. Their resolved
v0.24 values must be parsed from startup logs and reported. Allowing version
defaults to change is intentional: this experiment asks whether an operator's
old explicit configuration remains good after upgrading, not whether internal
defaults can be manually made identical.
Preflight records `vllm.__version__`, source commit, patch checksums, Python,
PyTorch, CUDA, NCCL, Triton, driver, model/tokenizer file hashes, MoE backend,
resolved scheduler/cache/graph config, compile-cache key, and exact command.
The expected source is the existing Phase-3/5 patched vLLM 0.24 environment;
any missing OpProf patch or unexpected model path is a stop condition.
### Client equivalence, not a new client covariate
The later `scripts/opprof_phase6_client.py` is permitted only as a thin anchor
driver around the four pinned AITuner modules. Its interface is normative:
```bash
P6C='python scripts/opprof_phase6_client.py'
$P6C preflight \
--trace trace_windows/traces/chat_w20260311_1000.jsonl \
--windows trace_windows/windows.json --window-id chat_w20260311_1000 \
--ground-truth /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json
$P6C run-anchor --cell "$CELL" --anchor "$U" \
--base-url "http://127.0.0.1:$PORT" \
--model qwen3-30b-a3b-community --completion-tokens 128 \
--replay-time-scale 0.1 --max-concurrency 64 \
--max-requests "$CAP_OR_NONE" --target-pass-rate 0.95 \
--ttft-step-ms '4096:2000,32768:4000,inf:6000' --tpot-ms 50 \
--request-timeout-seconds 900 --probe-deadline-ceiling-seconds 900 \
--result-dir "$ANCHOR_DIR"
```
Before GPU execution, a no-server golden test must prove identical selected IDs,
order, arrivals, raw input lengths, serialized bodies excluding model/URL, and
SLO decisions for synthetic outcomes. The wrapper may add interval markers and
sanity JSON, but may not change HTTP transport, scheduling, completion checks,
early stop, or latency formulas.
### Covariate ledger and estimand limit
| Covariate | C1 v0.20 | Phase 6 v0.24 | Treatment/handling |
|---|---|---|---|
| Engine | community vLLM 0.20.0 | patched vLLM 0.24.0 | Primary intended churn dimension; exact builds recorded |
| Host | dash1 | dash0 | Both 8x H20 96 GB class; CPU/NUMA/clocks/driver/kernel recorded, not assumed identical |
| GPU visibility | TP1 C1 env exposed GPUs `0,1`, TP1 consumed one | expose exactly TP GPUs | Declared operational covariate; H20-hour charge follows TP |
| Trial placement | recovered C1 harness trials | up to four co-located servers | CPU/NUMA pinned; reject outside GPU processes and report co-location limitation |
| HTTP client | pinned AITuner urllib/SSE | same code paths through thin wrapper | Eliminated as a semantic covariate by golden tests |
| Workload | materialized C1 chat window | byte-identical file/hash | Fixed |
| Selection/cap | cap-before-threshold; TP4 uncapped | exact same | Fixed and 92-count preflighted |
| SLO length | raw input length | raw input length | Fixed; server usage length is diagnostic only |
| Explicit flags | TP, MNS, MBT8192 | same | Fixed |
| Unspecified defaults | v0.20 defaults | v0.24 defaults | Intentionally allowed to churn; resolved values reported |
| Telemetry | no Layer 1 | zero-overhead OpProf Layer 1 | Added measurement; overhead evidence -0.04196%, CI [-0.17443%, 0.04550%] |
Therefore “version churn” means the observed paired upgrade path under these
documented platform covariates. It must not be rewritten as a same-host causal
microbenchmark.
## Budget-constrained anchor subset
### Adaptive adjacent-anchor design
The primary subset improves on measuring next-infeasible anchors only for a
chosen top four:
1. Measure the recorded v0.20 peak anchor for all 12 cells.
2. For every non-trap cell, select exactly one adjacent recorded anchor by a
result-independent rule fixed now:
- if the v0.20 peak remains feasible on v0.24, measure the nearest higher
recorded anchor, which was infeasible on v0.20;
- if the peak becomes infeasible, measure the nearest lower recorded anchor,
which was feasible on v0.20.
3. For TP4/MNS16, measure lower, peak, and higher anchors unconditionally. This
directly brackets the registered trap.
Thus exactly 25 primary anchors are measured: TP1 eight, TP2 eight, and TP4
nine. The design covers every cell and spends the second anchor in the direction
where its frontier could have moved. It also includes the suggested top-four
test. Under the floor-bucket rule, the nominal top-four cutoff expands to five
cells because TP4/MNS16/32/64 share one bucket; all five receive an upward test
whenever their old peak still passes.
If an adjacent anchor does not bracket the transition—higher also passes or
lower also fails—the cell is censored. Remaining budget may crawl one recorded
anchor farther outward, in this priority order: old global best TP2/MNS32,
TP2/MNS64, the TP4 plateau MNS16/32/64, then other cells by old score. Crawling
uses only that cell's recorded history and stops at the first transition. No
new `sampling_u` value may be invented. Failure to close a decision-critical
bracket before the budget stop makes the applicable argmax/ranking verdict
inconclusive.
### Boundary confirmations
The original surface has one observation per anchor. Phase 6 keeps that paired
unit but reserves at most 0.35 H20-hours for decision-triggered confirmation.
Repeat an anchor if its v0.24 pass rate lies in `[0.93,0.97]`, or if its single
verdict alone changes argmax or trap status. Priority is argmax, trap, material
frontier calls, then other cells. Agreement across two runs freezes the verdict;
disagreement requests a third run if budget permits and uses 2-of-3. Without a
third run, that anchor and every dependent headline verdict are inconclusive.
Request outcomes are arrival-correlated, so a binomial CI is descriptive only
and cannot replace these repeat rules.
## Execution plan
### Preflight and detached ownership
The Phase-6 controller follows A-P3-2 detached ownership: one controller PID,
per-stage atomic state, process-group/environment markers, resumable completed
anchors, exact command logs, and cleanup restricted to its own markers. Before
launch it must verify dash0 has eight H20s, no outside compute processes on
assigned GPUs, at least 100 GB CPFS free, pinned source/workload/helper hashes,
all 92 selection counts, legal TP placement, and a projected total below 3.0
H20-hours.
The autonomous launch log must echo one resolved line before starting:
```text
LAUNCH_ECHO host=dash0 engine=vllm-0.24.0-patched model=Qwen3-30B-A3B trace_sha=f539f38e... cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16 clean_horizon=60s drain=derived<=120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h
```
### Rate-following cold-start and drain gates
The throughput-drift stationarity gate is forbidden for these rate-following
anchors. A-P5-1-class cold-start gates are applied instead:
1. Every torch.compile/CUDA-graph first-occurrence event must precede the first
measured anchor, verified from server logs and Layer 1.
2. A deterministic unmeasured warm set must complete at least 16 exact-output
requests, including at least one request with raw input `>4096` tokens. The
peak-selected sets contain 43-184 such requests, so this gate is feasible.
3. No first-occurrence compile/capture event may occur in any measured anchor;
all graph-hit `(runtime_mode,bucket_tokens)` descriptors must be covered by
startup capture logs. A violation invalidates that anchor only.
4. Before the next anchor, all selected requests must be accounted for and three
consecutive controller samples must show running/waiting/deferred queues at
zero. Prefix cache is not explicitly enabled; nevertheless a full drain is
required.
The probe deadline reproduces `_probe_drain_deadline`: last selected arrival +
raw-length TTFT budget + `128*50 ms` + 30 seconds, capped by the historical
900-second ceiling. With raw inputs <=8192 this is about 100.4 seconds from
probe start; the controller has a 120-second class watchdog for cleanup. A
deadline miss is an SLO-infeasible anchor, not permission to omit denominator
requests. The server then shuts down through the official 120-second path.
### Four-wave packing
| Wave | Cells | Placement | Per-server measured anchors | Planned wall |
|---|---|---|---:|---:|
| W1 | TP1/MNS8,16,32,64 | four servers on GPUs 0,1,2,3; GPUs 4-7 idle | 2 each | 4-6 min |
| W2 | TP2/MNS8,16,32,64 | four servers on GPU pairs 0-1,2-3,4-5,6-7 | 2 each | 4-6 min |
| W3 | TP4/MNS8 and TP4/MNS16 trap | GPU quads 0-3 and 4-7 | 2 and 3 | 5-7 min |
| W4 | TP4/MNS32 and TP4/MNS64 | GPU quads 0-3 and 4-7 | 2 each | 4-6 min |
Each server starts once, passes warm-up once, runs its peak first, drains, then
runs the predeclared adjacent anchor. Trap lower/upper order after peak is
counterbalanced by execution seed 20260717. Anchor confirmation occurs before
that server is released when possible. The controller records GPU clocks,
power, utilization, memory, CPU load, NUMA placement, and outside processes
through every wave. Any contamination invalidates the whole co-located wave.
Actual H20-hours are charged from server launch through verified cleanup for
every allocated GPU, including idle time while a paired TP4 cell finishes.
Before each wave or confirmation, the controller recomputes
`spent + current + conservative_remaining`; it stops before launch if the
projection can reach 3.0. Optional crawling and confirmations are sacrificed
before any hard-cap violation.
## Measurement and mechanism notes
For every anchor, report selected/admitted/completed/exact-output/SLO-pass
counts; early-stop reason; offered and completed req/s; TTFT/TPOT mean and
p50/p90/p95/p99; failures by raw-length TTFT bucket; elapsed/drain time; and
peak GPU memory/utilization. The surface objective remains offered req/s/GPU at
the highest measured feasible recorded anchor.
Layer 1 is active over the complete anchor service interval, from the first
submission through the last selected-request accounting event. Aggregate:
- prefill, decode, and mixed-step counts and token shares;
- decode-batch distribution and 5-second CV;
- FULL/PIECEWISE/NONE graph-mode shares, bucket padding, and uncovered support;
- running/waiting/deferred queues and queue CV;
- preemption count/rate;
- KV blocks used/total, mean/max usage, and allocation pressure; and
- prefix queries/hits as a resolved-default diagnostic.
Mechanism notes compare v0.24 cells/anchors and may relate a frontier change to
preemptions, KV pressure, batching, or graph modes. C1 lacks Layer-1 telemetry,
so these notes cannot claim that a measured composition field itself changed
from v0.20 or caused the version drift. Historical v0.20 TTFT/TPOT failure
reasons may be paired directly; device-mechanism attribution remains
descriptive.
## Statistical and ranking analysis
### Floor buckets
For each unrounded score vector `s`, use the SimFid inventory rule:
```text
tol(s) = max(1e-9, 1e-6 * max_c(abs(s(c))))
bucket_s(c) = floor(s(c) / tol(s))
```
Two scores tie only when their integer buckets match. Bucketing occurs before
display rounding, pair signs, tau-b, top selection, or trap evaluation. Report
`tol`, bucket integers, and occupied intervals `[j*tol,(j+1)*tol)`. For v0.20,
`tol20=3.283333333333333e-6`; the TP4 MNS16/32/64 tie makes nominal top-4 an
effective top-5 set.
### Per-cell frontier
For each cell, define the v0.24 measured recorded-anchor frontier:
```text
f24(c) = max_{measured u with accepted feasible verdict} N_c(u) / 60 / TP
drift(c) = (f24(c) - f20(c)) / f20(c)
```
The predeclared material threshold is **X=5% relative**, large enough to avoid
calling one-request anchor quantization a paper-level churn effect. Report two
orthogonal labels:
- **BOUNDARY DOWN/UP:** old peak flips feasible->infeasible, or the old adjacent
infeasible anchor flips to feasible, respectively;
- **MATERIAL FRONTIER MOVED:** `abs(drift)>5%`; otherwise report the boundary
flip as sub-material.
If peak passes and higher fails, and `abs(drift)<=5%`, the local recorded
frontier is **STABLE**. If peak and lower both fail or peak and every affordable
higher anchor pass, report a directional bound and **UNBRACKETED**, never an
invented peak.
### Argmax, ranking, and trap
- **ARGMAX MOVED** when TP2/MNS32 is not in the top `bucket_f24` among all 12
decision-bounded cells. **ARGMAX SURVIVED** requires it to share the top
bucket and no censored competitor capable of overtaking it. Otherwise it is
inconclusive.
- Compute Kendall tau-b over all 12 floor-bucketed `f20/f24` scores and report
concordant, discordant, v0.20-tied, v0.24-tied, and comparable-pair counts.
**RANKING SURVIVES** requires an evaluable 12-cell surface, argmax survival,
`tau-b>=0.8`, and no >5% top-bucket pair reversal. `tau-b<0.8` is **RANKING
MOVED**. A partial surface receives no suite-level tau claim.
- **TRAP PERSISTS** when TP4/MNS16 is in a bucket at least as high as both
adjacent TP4 cells MNS8 and MNS32, yet below the global-best bucket. It
**ESCAPES** if an adjacent TP4 move is strictly better, and **CEASES TO BE A
TRAP** if it joins the global-best bucket. Missing/unbracketed inputs make trap
status inconclusive.
No arithmetic average of normalized scores is reported. Absolute rates,
relative drift, the paired table, and every excluded/censored anchor accompany
all ranking summaries.
## Later deliverables and artifact contract
Execution, after approval, must produce:
- `docs/opprof/phase6-results.md`;
- `runs/opprof-phase6/phase6/metrics.json` containing the complete paired
surface, anchor/run values, selection hashes, feasibility, floor buckets,
frontier/argmax/ranking/trap verdicts, confirmation history, Layer-1 notes,
covariates, and GPU accounting;
- exact commands, controller state, environment/machine fingerprints, server
logs, client outcomes, Layer-1 JSONL/footer/sidecar, monitor samples, and
per-anchor sanity JSON; and
- an operational-findings section covering startup/capture gates, drains,
early stops, co-location, default changes, retries, contamination, and budget.
The results begin with data sanity and stop inferential claims on any red flag.
Every numeric family ends with `n`, finite/missing, min, max, distinct count,
and applicable invariants. Prompt text, messages, generated content, source
substrings, and private request bodies stay in mode-0700 storage. Public files
contain only hashes, IDs, lengths, timestamps, outcomes, counters, and summaries.
## Anchor-subset table
Notation is `u / selected N / recorded req/s/GPU`. `P` is always measured. For
non-trap cells, measure `U` if P passes on v0.24, otherwise `L`. TP4/MNS16
measures all `L+P+U`. Planning cost per anchor includes the 60-second replay and
20 seconds expected drain/accounting; shared startup/warm-up is budgeted below.
| Cell | L: nearest recorded feasible below P | P: recorded peak | U: nearest recorded infeasible above P | Primary selection | H20-hour cost each |
|---|---|---|---|---|---:|
| TP1/MNS8 | 0.21875 / 121 / 2.0167 | 0.2265625 / 126 / 2.1000 | 0.23046875 / 130 / 2.1667 | P + (U if pass else L) | 0.0222 |
| TP1/MNS16 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | 0.25 / 143 / 2.3833 | P + (U if pass else L) | 0.0222 |
| TP1/MNS32 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
| TP1/MNS64 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 |
| TP2/MNS8 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
| TP2/MNS16 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 |
| TP2/MNS32 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / **3.2833** | 0.7578125 / 396 / 3.3000 | P + (U if pass else L) | 0.0444 |
| TP2/MNS64 | 0.5 / 276 / 2.3000 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / 3.2833 | P + (U if pass else L) | 0.0444 |
| TP4/MNS8 | 0.016055910008 / 301 / 1.2542 | 0.016591107009 / 308 / 1.2833 | 0.017126304009 / 317 / 1.3208 | P + (U if pass else L) | 0.0889 |
| TP4/MNS16 trap | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / **2.4417** | 0.034252608017 / 600 / 2.5000 | **L + P + U** | 0.0889 |
| TP4/MNS32 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
| TP4/MNS64 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 |
## Total GPU estimate versus 3.0 cap
The 25 primary anchors contain 8 TP1, 8 TP2, and 9 TP4 replays. At the
80-second per-anchor planning charge they consume `0.1778 + 0.3556 + 0.8000 =
1.3334` H20-hours. Four shared startup/warm-up/cleanup envelopes plus runtime
variance are budgeted at 1.3166 hours, for a **2.65-H20-hour primary plan**.
The remaining **0.35 H20-hours** is reserved for boundary confirmations or
decision-critical outward crawling. The absolute launch cap is **3.0
H20-hours**; projected or actual spend reaching the cap stops all further work.
Expected wall time is 25-35 minutes over four waves. Expected public disk is
0.3-0.6 GB and must remain below 3 GB.
## Decision rules
1. Anchor feasibility is exact-output request pass rate `>=0.95` using raw-length
stepped TTFT and 50-ms TPOT; absent/early-stopped requests remain failures.
2. Boundary movement is an old peak feasible->infeasible flip or an old adjacent
infeasible->feasible flip. Material frontier churn is `abs(drift)>5%`.
3. Apply zero-anchored floor buckets with set-wide `tol=max(1e-9,1e-6*max|s|)`
before every tie/rank decision.
4. ARGMAX MOVED iff TP2/MNS32 is outside the v0.24 top bucket on a bounded
12-cell subset; censored contenders make survival inconclusive.
5. RANKING SURVIVES only with all 12 cells, argmax survival, tau-b >=0.8, and no
>5% top-bucket pair reversal. Tau-b <0.8 is RANKING MOVED.
6. TP4/MNS16 trap persists only when it is no worse than adjacent MNS8/MNS32 and
remains below the global best; an improving neighbor means escape.
7. Boundary `[0.93,0.97]` or verdict-changing anchors require confirmation when
budget permits. Disagreement without a 2-of-3 resolution is inconclusive.
## Open decisions for orchestrator
1. Approve the **upgrade-path estimand**: resolved default changes are part of
churn, while dash1->dash0 and co-location remain explicit limitations rather
than pretending this is a pure engine-version causal effect.
2. Approve the **adaptive 25-anchor subset**, which gives every cell a
direction-relevant neighbor and the trap both neighbors, instead of spending
next-infeasible anchors only on a tie-ambiguous top four.
3. Approve **5% material frontier drift**, `tau-b>=0.8`, and the stated
argmax/trap floor-bucket rules.
4. Approve one primary observation per historical anchor with the 0.35-hour
reserve for `[0.93,0.97]`/verdict-changing confirmations, rather than reducing
surface coverage to replicate every anchor.
5. Approve the A-P5-1-class warm-up adaptation from `input>=8192` to
**raw input >4096**, the longest SLO-relevant tier available reliably in this
0-8192 C1 workload.
## Sanity block
| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| C1 cells | 12 | TP=1,MNS=8 | TP=4,MNS=64 | 12 | Exact 3x4 Cartesian surface |
| Historical probe observations | 92 | 7/cell | 8/cell | 2 per-cell counts | `8*8 + 4*7 = 92` |
| Materialized trace rows | 1 file | 32,606 | 32,606 | 1 expected | SHA pinned; prompt content not emitted |
| Raw-length-filtered rows | 1 reconstruction | 17,710 | 17,710 | 1 expected | Raw length 22-8192; non-negative |
| Capped TP1/TP2 source rows | 1 set | 512 | 512 | 1 expected | Stable time sort, then even downsample |
| Reconstructed historical counts | 92 | 66 | 600 | 34 | 92/92 match ground truth |
| C1 peak scores (req/s/GPU) | 12 | 1.283333 | 3.283333 | 8 | Finite, positive, not all identical |
| v0.20 floor tolerance | 1 vector | 3.283333e-6 | 3.283333e-6 | 1 | Applied before display rounding |
| Nominal top-4 effective size | 1 cutoff | 5 | 5 | 1 | TP4 three-way cutoff tie retained |
| Primary peak anchors | 12 | 1/cell | 1/cell | 1 expected | No cell omitted |
| Realized adjacent/trap anchors | 13 | 1/non-trap | 2/trap | 2 | Predeclared direction rule; total 25 |
| Primary anchors by TP | 25 | 8 at TP1/TP2 | 9 at TP4 | 2 counts | `8+8+9=25` |
| Selected requests in anchor candidates | 36 candidates | 121 | 600 | >1 expected | TP1/2 cap-before-threshold; TP4 uncapped |
| Per-anchor planning cost (H20-hour) | 3 TP classes | 0.0222 | 0.0889 | 3 | Proportional to TP for 80 seconds |
| Primary planned H20-hours | 1 plan | 2.65 | 2.65 | 1 | Non-negative; below 3.0 |
| Confirmation/crawl reserve | 1 plan | 0.35 | 0.35 | 1 | Primary plus reserve equals hard cap |
| GPU work in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied |
Checked invariants: `12=3*4`; `92=8*8+4*7`; all 92 local selection counts
match; cap precedes threshold; arrivals span the fixed 60-second replay; raw
input length, not retokenized usage, selects TTFT thresholds; completion is
exactly 128; SLO denominator includes all selected requests; surface scores use
60 seconds and divide by TP; floor buckets precede ranking; the top-four cutoff
expands to five; `25=12+11+2`; planned `2.65+0.35=3.0`; ratios and counters have
their declared domains; and no GPU run, helper, manifest, remote artifact, or
results file was created in this protocol-only turn.

226
docs/opprof/phase6-results.md Normal file
View File

@@ -0,0 +1,226 @@
# OpProf Phase 6 dash0 results
Status: **COMPLETE, DATA-VALID A-P6-2 SOLO TIER; ARGMAX/RANKING/TRAP INCONCLUSIVE UNDER THE FROZEN RULES**.
Phase 6 completed all 12 C1 cells on patched vLLM
`0.24.1.dev3+g668cfb7e2` (`4b253fd`). A-P6-2 makes the serialized solo tier
authoritative because the v0.20 baseline was solo and the original W2/W3
confirmations exposed large pass-rate and queue changes after neighboring wave
clients became idle. The authoritative campaign contains 37 primary anchors,
29 same-placement confirmations, and 12 valid Layer-1 streams.
The machine-readable result is
`runs/opprof-phase6/phase6/metrics.json`, SHA-256
`290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16`.
Co-located W1-W3 artifacts are preserved beside the solo tree rather than
overwritten. Public artifacts contain no prompt or generated text.
## Data sanity first
There are **no data-sanity red flags**. All 37 authoritative primary anchors
reproduce the pinned v0.20 request count exactly; all client invariants pass;
all 12 solo cells have one graceful footer and agreeing final sidecar; Layer-1
indices are contiguous with zero drops; every anchor interval is represented;
and all cold-start compile/capture events precede measurement.
The scientific decision gate is separate from data validity. Eight cells have
bounded monotonic solo frontiers. TP2/MNS16, TP4/MNS32, and TP4/MNS64 pass the
highest recorded request set and are right-censored at the pinned-history edge.
TP4/MNS16 is non-monotonic: L resolves feasible, P resolves infeasible, and U
resolves feasible. These four cells prohibit a full-surface floor-bucket rank,
tau-b, and trap decision; no value is imputed beyond the recorded history.
## Launch echo
```text
A-P6-2_LAUNCH_ECHO host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.291173 new_est_h20h=3.440 cumulative_est_h20h=5.731173 cap_h20h=6.0 est_wall=45-70min
SOLO_WAVE_ECHO order=TP4/MNS32,TP4/MNS64,TP2/MNS32,TP2/MNS64,TP4/MNS16,TP2/MNS8,TP2/MNS16,TP4/MNS8,TP1/MNS8,TP1/MNS16,TP1/MNS32,TP1/MNS64 placement=one_server_one_client
FINAL_ACCOUNTING status=complete cells=12 primary=37 confirmations=29 solo_h20h=3.353371 campaign_h20h=5.644544 cap_h20h=6.0
```
The exact per-cell echoes, GPU placement, trace and ground-truth paths, spend,
and projection are retained in
`runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log`.
## Run completion stats
| Item | Result |
|---|---:|
| Historical selections reconstructed | 92/92 exact |
| Authoritative solo cells | 12/12 |
| Solo primary anchors | 37 |
| Solo confirmations | 29 |
| Solo anchor trials | 66 |
| Solo warm-ups | 12; exactly 16 completions each |
| Long prompts per warm-up | 2-8 raw inputs `>4096` |
| Bounded solo frontiers | 8/12 |
| History-edge right-censored cells | 3 |
| Non-monotonic cells | 1 |
| Prior co-located primary/confirm trials | 21 + 3 |
| Total accepted anchor trials across attempts | 90 |
| Co-located spend before A-P6-2 | 2.291173 H20-hours |
| A-P6-2 solo spend | 3.353371 H20-hours |
| Total campaign spend | **5.644544 H20-hours** |
| Artifact footprint | 467 MiB, below 3 GiB |
## Final paired surface
`A*` marks the authoritative A-P6-2 solo tier. Exact values are bounded
frontiers. `>=` is a recorded-history lower bound. The TP4/MNS16 value is not a
frontier because its feasibility sequence is non-monotonic.
| Cell | Tier | v0.20 frontier | v0.24 authoritative result | Drift | Status |
|---|---|---:|---:|---:|---|
| TP1/MNS8 | A* solo | 2.1000 | **2.3833** | **+13.49%** | bounded up |
| TP1/MNS16 | A* solo | 2.3500 | 2.3833 | +1.42% | bounded up |
| TP1/MNS32 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
| TP1/MNS64 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up |
| TP2/MNS8 | A* solo | 2.2750 | 2.2417 | -1.47% | bounded down |
| TP2/MNS16 | A* solo | 2.2750 | >=2.3000 | >=+1.10% | right-censored history edge |
| TP2/MNS32 | A* solo | **3.2833** | **3.2583** | -0.76% | bounded down |
| TP2/MNS64 | A* solo | 3.2583 | **2.3000** | **-29.41%** | bounded down |
| TP4/MNS8 | A* solo | 1.2833 | 1.3208 | +2.92% | bounded up |
| TP4/MNS16 | A* solo | 2.4417 | N/E; observed max feasible 2.5000 | N/E | **non-monotonic: L pass, peak fail, U pass** |
| TP4/MNS32 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
| TP4/MNS64 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge |
The solo tier validates two material frontier changes under the predeclared 5%
threshold: TP1/MNS8 improves by 13.49%, while TP2/MNS64 declines by 29.41%.
The previous TP2/MNS64 decline is now quotable because both its passing 2.3000
anchor and failing 3.2583 anchor were repeated solo.
## Frozen decision-rule verdicts
- **ARGMAX: INCONCLUSIVE.** TP2/MNS32 is the highest bounded observed frontier
at 3.2583 req/s/GPU, but the frozen rule requires a bounded 12-cell subset.
TP2/MNS16 and TP4/MNS32/64 terminate at passing history-edge anchors, and
TP4/MNS16 has no coherent frontier. The top floor bucket is therefore not
defined for all contenders.
- **RANKING: INCONCLUSIVE.** Only 8/12 frontiers are bounded. Per protocol,
Kendall tau-b is **not evaluable** (`tau_b=null`) rather than computed from
censored lower bounds. The `tau-b>=0.8` survival rule cannot be applied.
- **TRAP: INCONCLUSIVE.** TP4/MNS16 is non-monotonic and its TP4/MNS32 neighbor
is right-censored. Neither trap persistence nor escape satisfies the frozen
floor-bucket rule.
## Solo versus co-located exact-anchor deltas
The co-located column is the simultaneous-wave primary. The solo column is the
accepted median after same-placement 2-of-3 adjudication. Delta is percentage
points. `P/F` are SLO feasible/infeasible at 0.95.
| Cell | Anchor | Co-located primary | Solo authoritative | Delta | State change |
|---|---:|---:|---:|---:|---|
| TP1/MNS8 | 0.21875000 | 0.992 | 0.992 | +0.00pp | P->P |
| TP1/MNS8 | 0.22656250 | 0.206 | 0.992 | **+78.57pp** | F->P |
| TP1/MNS16 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS16 | 0.25000000 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS32 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS32 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS64 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P |
| TP1/MNS64 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P |
| TP2/MNS8 | 0.49218750 | 0.691 | 1.000 | +30.86pp | F->P |
| TP2/MNS8 | 0.49609375 | 0.092 | 0.581 | +48.90pp | F->F |
| TP2/MNS16 | 0.49218750 | 1.000 | 1.000 | +0.00pp | P->P |
| TP2/MNS16 | 0.49609375 | 0.088 | 1.000 | **+91.21pp** | F->P |
| TP2/MNS32 | 0.75000000 | 0.412 | 1.000 | +58.82pp | F->P |
| TP2/MNS32 | 0.75390625 | 0.028 | 0.575 | +54.70pp | F->F |
| TP2/MNS64 | 0.50000000 | 0.460 | 1.000 | +53.99pp | F->P |
| TP2/MNS64 | 0.75000000 | 0.141 | 0.561 | +42.07pp | F->F |
| TP4/MNS8 | 0.016055910008 | 1.000 | 1.000 | +0.00pp | P->P |
| TP4/MNS8 | 0.016591107009 | 0.071 | 1.000 | **+92.86pp** | F->P |
| TP4/MNS16 | 0.033182214016 | 0.609 | 1.000 | +39.13pp | F->P |
| TP4/MNS16 | 0.033717411016 | 0.036 | 0.238 | +20.22pp | F->F |
| TP4/MNS16 | 0.034252608017 | 0.847 | 1.000 | +15.33pp | F->P |
The deltas range from 0 to +92.86pp across 21 exact pairs. Four TP1/MNS16-64
pairs are unchanged, while several TP2/TP4 and TP1/MNS8 anchors flip. Thus
co-location validity is **metric- and cell-dependent**, not a universal offset.
Throughput-only values may co-locate, but SLO-frontier feasibility cannot be
mixed across placement tiers.
## Layer-1 mechanism notes for material drift
Layer-1 is mechanism context, not a matched v0.20 causal decomposition. P3 P10
is TP1 at a different workload/output contract. All 37 solo primaries have
zero preemptions.
| Cell/anchor | State | Waiting mean/max | Decode-B mean | KV mean | `NONE` graph share | Padding |
|---|---|---:|---:|---:|---:|---:|
| TP1/MNS8, 0.25 | frontier pass | 0.74 / 8 | 4.25 | 0.0407 | 3.26% | 0.34% |
| TP1/MNS8, 0.50 | next fail | 11.66 / 27 | 7.19 | 0.0644 | 5.04% | 0.02% |
| TP2/MNS64, 0.50 | frontier pass | 0.00 / 0 | 3.28 | 0.0084 | 0.00% | 18.88% |
| TP2/MNS64, 0.75 | old-peak fail, primary | 0.00 / 0 | 18.05 | 0.0448 | 9.47% | 0.52% |
| TP2/MNS64, 0.75 | old-peak fail, confirm | 0.0002 / 1 | 11.52 | 0.0285 | 4.57% | 1.13% |
For TP1/MNS8, the new frontier remains feasible despite a larger decode batch
and modest queueing; failure at 0.5 coincides with a 16x waiting-mean increase,
higher KV usage, and more eager/non-full graph execution. For TP2/MNS64, the
passing lower anchor has small decode batches and full/piecewise graphs despite
high padding, whereas both old-peak trials fail with much larger decode batches
and graph fallback. Zero preemptions and near-zero queueing rule out preemption
and server waiting as the main TP2/MNS64 explanation, but the unmatched v0.20
telemetry prevents assigning causality uniquely to graph mode or batch shape.
## Operational findings and full attempt history
1. The initial ownership-monitor false start charged 0.132918 H20-hours without
measurement. The repaired W1 charged 0.299393; A-P6-1 later accepted its
eight anchors from balanced checkpoint sidecars without rerun.
2. Co-located W2/W3 completed 13 primary anchors and three confirmations but
exposed placement-sensitive SLO results. The 3.0-hour projection stopped W4
at 2.291173 total; those values remain indicative and preserved.
3. A-P6-2 completed W4 and remeasured the full surface solo. Every solo server
used `--shutdown-timeout 120`, emitted a graceful footer, and passed Layer-1
accounting.
4. Placement was not the only transient. TP4/MNS32 and TP4/MNS64 first full
replays failed at 0.031/0.114 but subsequent identical solo trials passed;
TP4/MNS16 produced an order-dependent non-monotonic sequence. The 16-request
A-P5-1 gate removes compile/capture cold start but not necessarily the first
full-trace transient. The predeclared 2-of-3 rule prevented single-trial
conclusions, but a future protocol should use a full-trace burn-in or
randomized/reversed anchor order.
5. Two cleanup checks briefly observed 1-4 MiB at 0% with zero compute PIDs.
Memory decayed to 0 MiB within seconds, graceful footers validated, and no
rerun was needed. Cleanup now polls for 30 seconds rather than treating
transient driver bookkeeping as a failed experiment.
## GPU total
The complete campaign charged **5.6445441643 H20-hours** against the
user-approved **6.0** cap:
- original co-located attempts: 2.2911728495;
- A-P6-2 authoritative solo tier: 3.3533713148;
- remaining headroom: 0.3554558357 H20-hours.
Final dash0 state is 0 MiB and 0% utilization on all eight H20s, with zero
controller, client, vLLM, EngineCore, or worker processes.
## Sanity block
| Numeric family | n | Min | Max | Distinct | Checked invariant/result |
|---|---:|---:|---:|---:|---|
| Historical selection checks | 92 | 66 requests | 600 requests | 34 | 92/92 exact |
| Solo primary pass rates | 37 | 0.0307 | 1.0000 | 17 | Ratios in `[0,1]`; not identical |
| Solo selected requests | 37 | 121 | 600 | 18 | Exact v0.20 count per anchor |
| Layer-1 steps per primary | 37 | 343 | 12,103 | 37 | Contiguous; non-negative; zero drops |
| Layer-1 records per cell | 12 | 14,174 | 58,725 | 12 | Footer/sidecar balanced |
| Solo preemptions | 37 | 0 | 0 | 1 expected | Zero throughout |
| Reported v0.24 frontier values/bounds | 11 | 1.3208 | 3.2583 | 6 | TP4/MNS16 omitted as non-monotonic |
| Frontier drift values/bounds | 12 | -29.41% | +13.49% | 9 | 11 finite; one non-monotonic missing |
| Bounded/censored cells | 12 | 8 bounded | 4 unbounded | 2 | Censoring never imputed |
| Exact solo/co-located deltas | 21 | +0.00pp | +92.86pp | 13 | Same request set and anchor |
| Solo confirmation pass rates | 29 | 0.4130 | 1.0000 | 10 | Frozen 2-of-3 applied |
| Solo cell H20-hours | 12 | 0.0845 | 0.5534 | 12 | Non-negative; serialized placement |
| Total campaign H20-hours | 1 | 5.644544 | 5.644544 | 1 | Below 6.0 hard cap |
| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes |
Checked invariants: exact cap-before-threshold selection; raw-length SLO
evaluation; exact 128-token completions or counted failures; nondecreasing
arrivals; all selected outcomes represented in the denominator; one solo
server/client at a time; compile/capture outside measured intervals; 12/12
graceful footer balances; non-negative counters; pass ratios in range; no prompt
text in public artifacts; complete attempt-history retention; authoritative
tier separation; explicit censoring/non-monotonicity; tau-b suppression when
the full surface is unbounded; hard-cap compliance; and complete GPU cleanup.

80
docs/opprof_campaign_state.md Normal file
View File

@@ -0,0 +1,80 @@
# OpProf campaign state — pattern-conditioned operator profiling (vLLM 0.24.0 / Qwen3-30B-A3B / H20)
Started 2026-07-11. Orchestrator: Claude (session a42d23fb). Workers: codex via companion.
Discipline inherited from SimFid campaign (replayserve/docs/simfid_campaign_state.md):
pre-registered acceptance gates per phase, codex implement + independent review,
orchestrator verifies, echo-before-expensive-action, GPU actions logged.
## User decisions (2026-07-11)
- vLLM pinned to **0.24.0** (latest community); record commit + sha256 at clone time.
- GPU: **dash0 8×H20, approved for the whole campaign** (Phases 2-4). Orchestrator
still echoes load/duration before each launch. Single H20 sufficient for TP1.
- Observation patch: **dual-layer** — always-on lightweight per-iteration composition
telemetry (<3% overhead budget) + sampled heavy torch.profiler/CUPTI windows.
- L-C-A is REFERENCE ONLY in this campaign — pattern axes defined operationally
(input-len dist / output len / arrival shape / prefix sharing), no L-C-A dependency.
## Phase plan (pre-registered)
- P0 recon (local): clone v0.24.0, inventory built-in observability, patch design doc.
Gate: USER reviews patch design before P1 implementation.
- P1 patch dev + no-GPU tests (local). Gate: strict review + orchestrator verify.
- P2 single-H20 smoke on dash0: artifacts complete; measured overhead (on/off same
load) within budget; op-time sums ≈ iteration time; composition matches request-level
ground truth. Hard gate before P3.
- P3 pattern matrix (~8-12 patterns × 2-3 configs, short replays, dash0): per-pattern
operator time breakdown + composition distributions + waste accounting
(padding% / graph-miss / ragged imbalance / mixed-batch interference).
Hypothesis go/no-go: does operator bottleneck ranking change across patterns?
- P4 optimization proposal (ranked, measured bounds) + one cheap config-level
closed-loop validation (e.g., cudagraph capture sizes vs measured B distribution).
- Non-goals: no new kernels; no L-C-A ablation (separate track); no production clusters.
## Artifact layout
- vLLM source: /home/gahow/phd/vllm-v0.24.0 (clone, pinned)
- Patches: aituner patches/vllm-0.24.0-opprof/ (patch files + apply script + no-GPU tests)
- Docs: aituner docs/opprof/ (phase0-recon.md, patch-design.md, later phase reports)
## Job log
- P0 recon dispatched: task-mrg3nm1f-spyl85 (2026-07-11T08:25:42Z). Scope: clone v0.24.0 to ~/phd/vllm-v0.24.0, observability inventory with file:line, dual-layer patch design doc. Gate: user reviews design before P1.
- P0 ACCEPTED (task-mrg3nm1f, 34m54s): pin ee0da84a=v0.24.0 verified; VLLM_TORCH_PROFILER_DIR-absent claim verified; profiler moved to ProfilerConfig + entrypoints/serve/profile/api_router.py (verified); design doc 421 lines. USER APPROVED design with 5 delegated decisions (JSONL+msgspec+8192 queue; expert loads L2-only; 2+8 profiler window; 3% gate on 95% CI upper bound; reject --disable-log-stats) + checkpoint decision: community BF16, TP1 primary, TP2/4 counterpoints.
- P1 dispatching: implement per approved design; branch in ~/phd/vllm-v0.24.0 + exported patches/vllm-0.24.0-opprof/ in aituner; no-GPU tests must pass locally (opprof core import-light).
- P1 completed (task-mrg588j2, 26m36s): branch opprof tip c60e7eeb, +560/-1 across 5 files (within ±30% of design). Orchestrator pre-checks: pytest 11/11 reproduced; patch series applies cleanly on pristine v0.24.0 worktree and applied tree is byte-identical to branch (diff=0); scheduler/model_runner integration diff reviewed — surgical, matches design hooks. Strict review dispatching.
- Strict review (task-mrg68puk, 14m59s): FAIL — 0 Blocking / 3 Major / 1 Minor. M1 writer-thread silent death + shutdown deadlock (opprof.py:106/145); M2 apply.sh already-applied bypasses base/dirty checks; M3 foreground raw lists violate no-raw-list hotspot contract (opprof.py:182). Minor: missing zero-token+None-stat test; true Scheduler integration test DEFERRED to P2 smoke gate (no local vllm install). Reviewer verified OK: fail-fast log_stats ValueError, pairing across sync/batch-queue/PP/spec-decode, schedule-time snapshots, DP-unique paths, patch-id equality 3/3. Orchestrator adjudication: ACCEPT all; fix worker dispatching (fresh task — resume-last would hit reviewer thread).
- P1 fix round CLOSED (task-mrg6tnum, 13m34s): branch tip 668cfb7e (3 clean commits on ee0da84a). Orchestrator closure: 14/14 tests reproduced; M3 raw-lists-gone verified (bisect accumulation); BOTH apply.sh refusal cases independently reproduced (dirty exit=1; wrong-base-with-identical-patches exit=1); M1 writer failure handling inspected (one-time log, liveness check, bounded close, drop visibility). P1 ACCEPTED.
- ECHO P2 (GPU, dash0): deploy patched vLLM 0.24.0 (python-only patch, VLLM_USE_PRECOMPILED path) on dash0, serve Qwen3-30B-A3B BF16 TP1 on ONE free H20; artifacts smoke + overhead gate (interleaved 5xON/5xOFF identical load, 3% gate on 95% CI upper bound) + one 2+8 Layer-2 profiler window + integration assertions (records==steps, no pending leak, footer accounting). Est 1.5-2.5h wall, single GPU. Deferred review item (real Scheduler integration) is covered by these assertions.
- P2 dispatched: task-mrg7erq5 (dash0 smoke; ssh dash0 ONLY authorized; hard gates: artifacts/schema/footer/no-pending-leak, overhead 95%CI-upper <=3%, Layer-2 window loadable). Est 1.5-2.5h.
- P2 first run (task-mrg7erq5, 1h37m): overhead gate FAIL honored (13.11%, CI [6.24,19.53]) — but measurement judged INVALID by orchestrator: OFF-arm spread 29% across identical runs, one pair NEGATIVE overhead, 10s window with cold-start per run, ON-first order confound (run 1 = coldest = ON). Physical bound: per-step cost is bisect+encode+enqueue at ~11 steps/s. Artifacts gate passed except PIECEWISE not exercised by smoke load. 14 remote tests pass; TRITON backend 12/12; cleanup verified. AMENDMENT A-P2-1 (pre-registered before rerun): warmup excluded, >=120s steady-state window, ignore_eos fixed output lengths, ABBA counterbalanced order + discard first pair, host load recorded, recorder microbenchmark as secondary evidence, one PIECEWISE-exercising artifact run. GATE UNCHANGED (3% on 95% CI upper).
- P2 A-P2-1 rerun (task-mrgdbfju + predecessor, valid measurement): overhead gate REAL FAIL — 4.1816%, CI [3.1364, 4.7117], gate 3%. ON tight (28.18-28.26), OFF (28.80-29.64), pairs 2.07-4.74%. Artifacts PASS incl. PIECEWISE addendum (129 records, accounting balanced). Layer-2 correctly not run. LOCALIZATION PUZZLE: producer microbenchmark 29.1µs/step x 9.7 steps/s ≈ 0.04s/run vs observed ON-OFF gap 5.9s — direct recording cost explains <1% of overhead; structural suspect (capture scan @200 concurrency, writer-thread GIL, queue locking, flush I/O locus). GPU spend so far 117 H20-min. Diagnostic+fix round dispatching (stage-bisection first, minimal fix, gate rerun).
- Bisection round (task-mrge67ke, 34m9s): recorder pipeline exonerated — all 4 stages ~0.5% vs stage-off; BUT all stages kept VLLM_OPPROF_DIR set → gpu_model_runner CUDAGraphStat construction/propagation path active in all; cross-attempt (attempt-2 true-OFF 29.45 vs bisection off 28.39) localizes ~3.6% to that path. Worker honored outside-OpProf stop boundary; temp diffs reversed; 14/14 tests both sides. Orchestrator hypothesis for next round: CUDAGraphStat in ModelRunnerOutput poisons msgspec IPC fast path (pickle fallback for whole output) OR upstream cudagraph_metrics feature carries inherent cost. Next: STATIC analysis first, then 3-trial confirm (true-baseline / upstream-metrics-only / env-set-recorder-off), then targeted fix.
- P2 CLOSED: ALL GATES PASS (task-mrgfgedp, 2h17m). ROOT CAUSE of the 4.18%: VLLM_OPPROF_DIR entered compile_factors() (envs.py:2011-2105) → hashed into torch.compile cache path (backends.py:1024-1065) → every unique ON dir = cold graph/AOT cache (compile 36.41s vs 6.07s) and ~4% slower artifacts. Recording itself was NEVER the cost (recorder 29.1µs/step; serializer-poisoning hypothesis WRONG — TP1 in-process executor, msgspec handles CUDAGraphStat natively). 3-trial confirmation: baseline 29.72 / upstream-metrics-only 29.64 (0.25%) / env-set 28.45 (4.25%). Fix: +1 production line (ignore-list) + tests; tip bbfa717; 5-patch series; 15/15 tests x3 environments. FINAL GATE: overhead -0.042%, CI [-0.174, +0.046] — PASS. Layer-2 PASS (16.4MB Kineto, 2+8 window, 790,843 events, 7,367 kernel events; active-window perturbation 51.3% — confirms sampled-only design). Artifacts PASS (PIECEWISE 129/139). Orchestrator verified: fix commits, ignore-list diff, 15/15 local rerun, 5 patches. Caveat noted: attempt-4 JSONLs footerless (simultaneous shutdown) — excluded from accounting. UPSTREAM FINDING worth reporting: any per-run-unique env var hashed into compile factors silently causes cold-compile + slower artifacts.
- P3 dispatching: protocol-first (worker drafts docs/opprof/phase3-protocol.md, STOPS for orchestrator review before execution).
- USER DIRECTIVE (2026-07-12): parallelize P3 across 8xH20, one experiment per GPU (~8x speedup). Orchestrator caveat: no GPU contention but shared host CPU/memory-bandwidth (API server/tokenizer/scheduler/bench are host-side; P2 showed host-side effects can fabricate %-level differences; saturation-point calibration depends on absolute throughput). P3 protocol must include: pre-registered co-location validity check (same cell solo vs alongside 7 neighbors; throughput + op-share deltas <2-3% to authorize 8-way; fallback 4-way + revalidate), CPU affinity pinning per server/client pair, and host load recording per run.
- P3 protocol (task-mrh6vfqp, 27m13s, 835 lines): ACCEPTED with orchestrator amendments. Dispositions on 6 open decisions: (1) fixed-duration client build+test APPROVED; (2) P10 private-trace transfer to dash0 CPFS wjh APPROVED (sampling_u<=0.125, input<=32768, output cap 256; no prompt text in artifacts; data returns to same org cluster it came from); (3) MNS{64,1024}xMBT{2048,8192} sparse factorial APPROVED; (4) P10/C00 sole TP2 counterpoint APPROVED; (5) 60% load / burst-8 / 240s / 4 windows APPROVED; (6) kernel mapping + 70% classifiability gate + thresholds APPROVED. AMENDMENT A-P3-1 (user directive): 8-way GPU parallelism, one cell per GPU; pre-registered co-location validity check (same cell solo vs with 7 neighbors, throughput AND op-share deltas <3% to authorize; fallback 4-way + revalidate); CPU affinity pinning (disjoint core sets per server+client); saturation calibration under the SAME co-location regime as measurement; host load + clocks per run; revised wall estimate ~1.5-2h. AMENDMENT A-P3-2 (E2 lesson): execution via detached resumable controller script on dash0 (setsid, --resume, state file) so runs survive codex turn deaths.
- P3 E-a ACCEPTED (task-mrh7wd5x, 1h27m, 3.96 H20-h): co-location gate → 8-way FAIL (throughput Δ=0.000% but op-share shifts up to 4.2pp > 3% threshold), 4-way PASS (shares ≤0.075pp) — AUTHORIZED 4-WAY. Client+controller 12/12 no-GPU tests; provenance hashed. P10 transfer verified (4,011 rows, tokenizer parity 4011/4011 exact, 0600/0700 perms, no prompt text public). Orchestrator decisions: (1) 4-way verdict accepted despite validation-stream footer absence (shares from Kineto, unaffected); (2) shutdown-fix (API-parent-first) GPU quick-verify REQUIRED before E-b matrix; (3) A-P3-3: per-class drain budget — 240s for output-512 burst cells (P04-class), 120s others; (4) A-P3-4: P3 hard stop raised 8→16 H20-hours (user blanket dash0 approval; E-a consumed 3.96).
- E-b step1 STOP (task-mrhb2dxw, 10m42s): shutdown-fix verification FAILED hard footer gate — vLLM 0.24.0 API shutdown selected mode=abort timeout=0s and SIGTERMed EngineCore before scheduler/opprof teardown; finally-footer never runs. Run itself healthy (5268 reqs, 43.9 req/s, 1701 records, 0 drops, step range contiguous). Matrix NOT launched (correct). Orchestrator decision: footer must not depend on graceful shutdown — (a) 10-min static check for an official graceful path in 0.24.0; if unreliable, (b) checkpoint-footer sidecar in opprof.py (atomic write each flush cadence; accounting gate accepts sidecar when stream footer absent; bounded loss = last flush interval), new commit + re-export + tests, GPU re-verify incl. abort-kill case, then matrix.
- E-b round 2 (task-mrhbhj9j, ~1h5m): sidecar fix DONE (commit f8b68f24, +195/-7, patch tip 23450fb2, 7 patches, 18/18+12/12 tests; graceful path found: vllm serve --shutdown-timeout N; dual GPU verification PASS incl. SIGKILL with 24ms checkpoint sidecar balancing). Matrix launched, STOPPED at pre-registered drain gate: P10/C01 saturation drain 288.6s > 120s budget (clean window + accounting VALID; drain is post-measurement hang-watchdog only; 0.74 req/s saturated 32k-context queue drains slowly by nature). 3/4 first-wave runs PASS with plausible ordering. GPU 5.47/16. AMENDMENT A-P3-5: P10-class drain budget 600s; drain violations = quarantine-run-and-continue (stop only if >20% of runs violate); P10/C01 existing data re-adjudicated under amended rule if accounting balanced.
- NETWORK OUTAGE (~19:2xZ 2026-07-12 onward): dash0 AND dash1 unreachable from local (pre-auth timeout) — ingress/network fault, not host failure. Detached matrix controller on dash0 unaffected by design (A-P3-2); worker holds 1/min read-only reachability probe with retained artifact hashes for post-recovery integrity check. On reconnect: verify controller state (expect matrix advanced or complete), resume worker thread if its turn died.
- E-b round 3 blocked cleanly on ssh outage (task-mrhdu37w): resumable handoff at runs/opprof-phase3/phase3/access-blocker-20260712.json; last durable 8/52 accepted, 0 quarantine, 0 clean-failures, 6.82 H20-h; controller PID 2237019 untouched, likely still advancing locally. Also delivered mid-flight: P03 profile-control starvation client fix (13/13 tests) + frozen analyzer analyze_phase3.py (4/4 tests). Reconnection watcher armed.
- NETWORK RESTORED (~21:1xZ per user). Controller advanced 8→12/52 during outage then FAILED ~2.2h ago (idle since). GPU 8.13/16. E-b worker resumed (task-mrhkbq8s): integrity check vs retained hashes → diagnose controller failure → minimal fix → --resume matrix (40 runs remain) → frozen analysis.
- E-b round 4 (task-mrhkbq8s, ~2h): 40/52 accepted (20/24 cells), 0 clean-window failures, 542,350 L1 records, 72 L2 traces, GPU 13.31/16. Two aux-gate fixes shipped (+58/-11, 15/15+4/4 tests): profiler-control connection, clean-window failure boundary (post-clean disconnects wrongly counted), 3-sample GPU preflight. FINAL BLOCKER: P10/TP2 warm-up gate demands 32/32 completions; long-context prompts cannot finish in budget (15/32, 21/32) though clean windows are valid. Same failure class as drain budget: auxiliary gate miscalibrated for long-context pattern. AMENDMENT A-P3-6: P10-class warm-up gate = 32 completions OR (>=16 completions AND Layer-1-verified throughput stabilization over trailing warm-up steps), with mandatory post-hoc stability evidence in the report. Finish remaining cells within 16h cap; if cap hits, stop and analyze with documented gaps.
- E-b round 5 (task-mrhnw1rm, 23m46s): P10/TP2 FAILED frozen A-P3-6 stabilization on fresh data (drift 36.76% vs 10%; bins 11/10/16 vs 16) — the cell genuinely does not stabilize; clean window itself valid. Worker correctly refused to relax. GPU 14.03/16. ORCHESTRATOR DECISION: freeze matrix at 40/52 runs / 20/24 cells; abandon remaining 12 runs. AMENDMENT A-P3-7: per-contrast evaluation — each frozen contrast evaluates iff both cells complete; missing → NOT EVALUABLE (never imputed); H1a/H1b use existential logic on completed cells (CONFIRM possible, REFUTE NOT possible at 20/24 — must be stated explicitly). P10/TP2 non-stabilization recorded as a pattern-conditioned operational FINDING.
- PHASE 3 FINAL (task-mrhos386, 19m37s): **H1a INCONCLUSIVE / H1b PASS (5 of 6 evaluable contrasts) / compound PARTIAL** under A-P3-7. H1b effects large + Holm-corrected p≈0: P10 real trace R64 +44.79pp vs both long rectangular controls (efficiency −14.3%/−44.7%); P09 production mix R64 +39.62pp, padding +6.67pp (−8.3%); P06 bimodal burst R64 +23.0/+35.4pp (−11.6%/−22.8%). H1a inconclusive because only 1/9 patterns Layer-2 windows passed representativeness gates (descriptive: P06/P10 flip attention-led↔MoE-GEMM-led across load points). Orchestrator verified metrics.json per-contrast records match report exactly. GPU final 14.03/16 (1.97 unused). P4 dispatching.
## Phase 5 — mechanism-decomposition ablations (USER-APPROVED 2026-07-12)
- Motivation (user challenge): the sign of H1b is obvious; the contribution is the mechanism LEDGER. vLLM does no request-level padding (continuous batching); measured bucket padding (+5.4-6.7pp) explains ~1/6 of the 14-45% E_token gap; the remaining ~30pp attribution (ragged-attention SM imbalance / cudagraph bucket mismatch / chunked-prefill mix interference / MoE routing skew / scheduler batch-variance) is unmeasured in the literature.
- Design skeleton (protocol to formalize): controlled ablations each isolating ONE mechanism on P10 (primary) + P09/P06 (secondary): (i) length-sorted/binned replay [kills intra-cohort raggedness, keeps content+totals]; (ii) capture sizes matched to measured decode-B distribution [kills bucket mismatch; doubles as the config-tier deliverable]; (iii) arrival-shape ablation [steady vs burst, same lengths]; (iv) prefix-caching on/off [C structure]. Accounting must include an explicit interaction/residual term — shares are NOT forced to sum to 100%.
- Metric: E_token under the P3 discipline (C00-TP1, rho=0.60, 240s clean windows). New GPU cap: +6 H20-hours for P5 (P3 cap closed at 14.03/16).
- Gates: protocol-first (orchestrator review before any GPU run); P4 acceptance re-scoped — speculative recovery claims in P4 must defer to P5 measured shares.
- P5 protocol (task-mrhq3ud3, 13m49s): ACCEPTED. Data-grounded specifics verified by worker from P3 raw Layer-1: P10 decode-B support is {1..7} (17,941 pure-decode steps) → A2 capture sizes {3,5,6,7}; A1 = 32-request reorder blocks (R16 0.6417→0.4734, −26.2% rel), 142-request slice @0.4725 req/s, 64s fairness cap. Orchestrator dispositions on 5 open decisions: (1) BLOCKING → approve recorded-arrival BRIDGE ledger (production-faithful estimand; explicit not-literal-P3-decomposition limitation); (2) dual P03/P04 control ledgers, dominant must hold under both — approved; (3) A1 parameters — approved; (4) 3 replicates/arm, P3-control reuse behind 3% bridge gate, optional tier within 6.0 H20-h — approved; (5) Layer-1-only primary, routed-expert telemetry analysis-only no causal share — approved. Execution authorized.
- P5 wave 1 (detached exec, 30min, 0.65 H20-h): HARD STOP correct — all 4 attempted arms failed frozen warm-up stabilization (drift 190.6-216.5% recorded arms, 13.2% uniform arm, gate 10%; completions 25-28/32). ROOT CAUSE: semantic mismatch — throughput-drift gate designed for saturation/steady loads is wrong for rate-following arms whose throughput follows a non-stationary recorded arrival BY DESIGN. Purpose of warm-up = no cold-start contamination, not stationarity. Bonus signal: 190% vs 13% drift gap is itself direct arrival-mechanism evidence. AMENDMENT A-P5-1: rate-following arms use cold-start-artifact gates (all compile/capture events before clean window per Layer-1 graph-mode series + >=16 warm-up completions incl >=1 long-context + zero first-occurrence capture events inside clean window); drift gate retained for saturation arms only. Budget 5.35 remaining.
- P5 round 2 (detached, ~70min, 2.44/6.0 H20-h): 15/15 primary valid under A-P5-1, bridge PASS (0.269%). OFFICIAL ledger INCONCLUSIVE per frozen rules (only A1-under-P03 evaluable: share 0.038, CI [-0.20,0.23]; A2/A3/A4 manipulation-failed; P04 denominator unstable 45% nonpositive draws). REAL FINDINGS: (1) P3 P10-vs-P04 gap was largely a MATERIALIZATION ARTIFACT — P5 recorded-arrival base E=3.014 ≈ P04 control 3.055 (gap 0.041 CI spans 0), while uniformizing arrival (as P3 did) drops E to 2.626 (−12.9%, raw p=0.014, Holm 0.055): burstiness HELPS at low rate via batch formation (decode-B CV 0.87→0.66 under uniform but waiting CV rises); P5 A3 reproduces P3 base within 0.27% — bridge design caught our own artifact. (2) All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-fix +1.6% n.s. (P4 independent validation +0.18%, padding bound matched within 0.002pp), prefix ~0. Remaining P10-vs-P03 gap (~36%) explained by NONE — workload physics, not recoverable waste at this regime. (3) P4 ranked list: #1 prefix affinity +82.14% ceiling (P08 vs P07), #4 pattern-specific MNS64 has a −24.27% P01 counterexample (sign flips by pattern). OpProf campaign phases P0-P5 all complete; ~16.5 H20-h total.
## Phase 6 — cross-version churn quantification (USER-APPROVED 2026-07-13 "请你测试一下")
- Question: does the vLLM 0.20.0 C1 12-cell TP×MNS optimum/ranking survive on 0.24.0? Paired-surface churn evidence for the paper motivation (matrix × real-eval cost × churn).
- Ground truth: C1 recovered stores (verified): TP1 [2.10,2.35,2.283,2.283], TP2 [2.275,2.275,3.283,3.258], TP4 [1.283,2.442,2.442,2.442] req/s/GPU; best TP2/MNS32=3.2833; trap TP4/MNS16; per-cell recorded anchors + request counts in replayserve docs/assets/simfid_s2r/ground_truth.json; workload = materialized chat_w20260311_1000 (local + dash0 CPFS), output override 128, scale 0.1, pass>=0.95, stepped TTFT 2/4/6s, TPOT 50ms; caps 512 TP1/TP2, uncapped TP4.
- Budget: 3.0 H20-hours hard cap → anchor-subset design required (TP4 cells cost 4 GPU-h per wall-hour). Covariate to note: C1 ran on dash1, P6 on dash0 (same H20 class).
- Gate: protocol-first, orchestrator review before GPU.
- P6 protocol ACCEPTED: 92/92 historical anchor counts reproduced locally pre-GPU (comparability verified); adaptive 25-anchor design (12 peak + 11 adjacent + 2 trap-bracket), 2.65 primary + 0.35 confirmation reserve = 3.0 cap exactly. Decision rules: feasibility flip / >5% drift / floor-bucket argmax / tau-b>=0.8 ranking survival / trap escape rule / 2-of-3 confirmation. All 5 open decisions APPROVED incl. upgrade-path estimand (resolved defaults = churn; dash1→dash0 + colocation = stated limitations) and warmup long-tier adaptation raw>4096. Execution dispatched.
- P6 W1 hard stop (0.43 H20-h spent): 8 replays REJECTED for missing in-stream footers (wrong shutdown path) + redo projection 3.03 > 3.0 cap. Orchestrator diagnosis: P6 validator failed to apply the P5-ACCEPTED sidecar-footer accounting rule (records == sidecar counters within one flush interval). AMENDMENT A-P6-1: (a) sidecar footers are valid accounting per P5 rule — re-adjudicate W1 artifacts WITHOUT rerun if sidecars balance; (b) controller must use the graceful path (vllm serve --shutdown-timeout, found in P5) for subsequent waves; (c) IF W1 salvage fails, cap raised 3.0→3.5 as fallback (worker recommendation, within user posture).
- P6 r2 partial (2.29/3.0 H20-h, hard stop before W4): 21/25 anchors, 10/12 cells. Signals: TP2/MNS64 >=29.4% DOWN (left-censored), TP1 cells right-censored UP, TP2 family down — surface shape moved opposite directions. CONFOUND (worker confirmation runs): co-location flips SLO-frontier pass rates (0.41→1.00, 0.03→0.96 solo; waiting 8.5→0.35) — co-location valid for throughput/op-shares (P3) but NOT for pass-rate cliffs; 0.20 baseline was solo. USER DECISION: cap raised to 6.0; A-P6-2 = frontier anchors SOLO placement; finish W4 + solo re-confirm key cells.
- P6 FINAL (A-P6-2 solo tier complete, 5.64/6.0 H20-h, 66 authoritative trials, 12/12 cells): formal verdicts INCONCLUSIVE per frozen letter (4 cells censored/non-monotonic prevent bounded 12-cell surface; tau_b=null, never manufactured). SUBSTANTIVE PAIRED SURFACE (all solo-authoritative): TP2/MNS64 −29.41% CONFIRMED (3.2583→2.3000 bounded; mechanism: decode-B 11.5-18 + 4.6-9.5% NONE-graph at old anchor); TP1/MNS8 +13.49% and TP1 plateau converges to 2.3833; old best TP2/MNS32 −0.76% (held, highest bounded); TP4 family right-censored UP >=2.50; TP4/MNS16 frontier became NON-MONOTONIC in load. CO-LOCATION DELTA TABLE: up to +92.86pp pass-rate flips (21 exact-request anchor pairs) — co-location validity is metric-dependent (standalone methodological finding). Orchestrator verified −0.2941 drift in metrics.json. Campaign total incl. P6: ~22.2 H20-h.

487
patches/vllm-0.24.0-opprof/0001-Add-lightweight-per-step-OpProf-telemetry.patch Normal file
View File

@@ -0,0 +1,487 @@
From f6f1cacbce0e39992d04843f652c1adda373ae43 Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sat, 11 Jul 2026 17:29:02 +0800
Subject: [PATCH 1/5] Add lightweight per-step OpProf telemetry
Assisted-by: OpenAI Codex
---
vllm/envs.py | 4 +
vllm/v1/core/sched/scheduler.py | 28 +++
vllm/v1/opprof.py | 337 +++++++++++++++++++++++++++++
vllm/v1/worker/gpu_model_runner.py | 6 +-
4 files changed, 374 insertions(+), 1 deletion(-)
create mode 100644 vllm/v1/opprof.py
diff --git a/vllm/envs.py b/vllm/envs.py
index 27a85bb..b3093e9 100755
--- a/vllm/envs.py
+++ b/vllm/envs.py
@@ -45,6 +45,7 @@ if TYPE_CHECKING:
VLLM_LOGGING_COLOR: str = "auto"
NO_COLOR: bool = False
VLLM_LOG_STATS_INTERVAL: float = 10.0
+ VLLM_OPPROF_DIR: str = ""
VLLM_TRACE_FUNCTION: int = 0
VLLM_USE_FLASHINFER_SAMPLER: bool = True
VLLM_PP_LAYER_PARTITION: str | None = None
@@ -786,6 +787,9 @@ environment_variables: dict[str, Callable[[], Any]] = {
if (val := float(os.getenv("VLLM_LOG_STATS_INTERVAL", "10."))) > 0.0
else 10.0
),
+ # Directory for per-step OpProf JSONL telemetry.
+ # Empty disables OpProf.
+ "VLLM_OPPROF_DIR": lambda: os.getenv("VLLM_OPPROF_DIR", ""),
# Trace function calls
# If set to 1, vllm will trace function calls
# Useful for debugging
diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
index 90d93a1..303c562 100644
--- a/vllm/v1/core/sched/scheduler.py
+++ b/vllm/v1/core/sched/scheduler.py
@@ -7,6 +7,7 @@ from collections.abc import Iterable
from dataclasses import replace
from typing import Any
+import vllm.envs as envs
from vllm.compilation.cuda_graph import CUDAGraphStat
from vllm.config import VllmConfig
from vllm.distributed.ec_transfer.ec_connector.base import (
@@ -55,6 +56,7 @@ from vllm.v1.engine import EngineCoreEventType, EngineCoreOutput, EngineCoreOutp
from vllm.v1.kv_cache_interface import KVCacheConfig
from vllm.v1.metrics.perf import ModelMetrics, PerfStats
from vllm.v1.metrics.stats import PrefixCacheStats, SchedulerStats
+from vllm.v1.opprof import OpProfRecorder
from vllm.v1.outputs import DraftTokenIds, KVConnectorOutput, ModelRunnerOutput
from vllm.v1.request import Request, RequestStatus, StreamingUpdate
from vllm.v1.spec_decode.dynamic.utils import build_dynamic_sd_schedule_lookup
@@ -271,6 +273,12 @@ class Scheduler(SchedulerInterface):
if self.connector is not None:
self.connector.bind_gpu_block_pool(self.kv_cache_manager.block_pool)
+ self.opprof = OpProfRecorder.create(
+ output_dir=envs.VLLM_OPPROF_DIR,
+ dp_rank=self.parallel_config.data_parallel_index,
+ log_stats=self.log_stats,
+ )
+
self.use_pp = self.parallel_config.pipeline_parallel_size > 1
self.use_v2_model_runner = vllm_config.use_v2_model_runner
# Scheduler iteration counter. Drives the V2+PP+async decode-throttle
@@ -386,6 +394,9 @@ class Scheduler(SchedulerInterface):
return num_new_tokens
def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput:
+ opprof_start = (
+ self.opprof.capture_start(self) if self.opprof is not None else None
+ )
self.current_step += 1
# NOTE(woosuk) on the scheduling algorithm:
# There's no "decoding phase" nor "prefill phase" in the scheduler.
@@ -1090,6 +1101,14 @@ class Scheduler(SchedulerInterface):
)
scheduler_output.ec_connector_metadata = ec_meta
+ if self.opprof is not None:
+ assert opprof_start is not None
+ self.opprof.begin(
+ scheduler=self,
+ output=scheduler_output,
+ start=opprof_start,
+ )
+
# Advance the fence only for non-empty steps (those that actually
# write KV and have their output processed later in update_from_output).
if self.defer_block_free and total_num_scheduled_tokens > 0:
@@ -1800,6 +1819,12 @@ class Scheduler(SchedulerInterface):
engine_core_outputs[0] = eco = EngineCoreOutputs()
eco.scheduler_stats = stats
+ if self.opprof is not None:
+ self.opprof.finalize(
+ output=scheduler_output,
+ cudagraph_stat=cudagraph_stats,
+ )
+
return engine_core_outputs
@staticmethod
@@ -2292,6 +2317,9 @@ class Scheduler(SchedulerInterface):
if self.ec_connector is not None:
self.ec_connector.shutdown()
+ if self.opprof is not None:
+ self.opprof.close()
+
logger.debug_once("[shutdown] Scheduler: complete")
########################################################################
diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
new file mode 100644
index 0000000..f0330d0
--- /dev/null
+++ b/vllm/v1/opprof.py
@@ -0,0 +1,337 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+import atexit
+import logging
+import os
+import queue
+import threading
+import time
+from bisect import bisect_left
+from pathlib import Path
+from typing import Any
+
+import msgspec
+
+logger = logging.getLogger(__name__)
+
+SCHEMA_VERSION = 1
+CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18))
+CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12))
+DEFAULT_QUEUE_CAPACITY = 8192
+_CLOSE_TIMEOUT_SECONDS = 1.0
+_STOP = object()
+_PREFIX_FIELDS = ( # noqa: SIM905
+ "requests queries hits preempted_requests preempted_queries preempted_hits"
+).split()
+
+
+ScheduleStart = tuple[int, int, dict[str, dict[str, int] | None]]
+
+
+def classify_chunk(was_chunk: bool, end: int, target: int) -> str:
+ assert 0 <= end <= target
+ if was_chunk:
+ return "middle" if end < target else "final"
+ return "first" if end < target else "unsplit"
+
+
+def _prefix_values(stats: Any | None) -> dict[str, int] | None:
+ if stats is None:
+ return None
+ return {name: int(getattr(stats, name, 0)) for name in _PREFIX_FIELDS}
+
+
+def _prefix_snapshot(scheduler: Any) -> dict[str, dict[str, int] | None]:
+ return {
+ "local": _prefix_values(scheduler.kv_cache_manager.prefix_cache_stats),
+ "external": _prefix_values(scheduler.connector_prefix_cache_stats),
+ }
+
+
+def _prefix_delta(
+ before: dict[str, int] | None, after: dict[str, int] | None
+) -> dict[str, int] | None:
+ if before is None and after is None:
+ return None
+ before = before or dict.fromkeys(_PREFIX_FIELDS, 0)
+ after = after or dict.fromkeys(_PREFIX_FIELDS, 0)
+ delta = {name: after[name] - before[name] for name in _PREFIX_FIELDS}
+ assert all(value >= 0 for value in delta.values())
+ return delta
+
+
+class JSONLWriter:
+ def __init__(
+ self,
+ path: Path,
+ capacity: int = DEFAULT_QUEUE_CAPACITY,
+ start: bool = True,
+ ) -> None:
+ self._queue: queue.Queue[Any] = queue.Queue(capacity)
+ self._encoder = msgspec.json.Encoder()
+ self._file = path.open("xb", buffering=1 << 20)
+ self._thread = threading.Thread(target=self._run, daemon=True)
+ self._failure_lock = threading.Lock()
+ self._started = self._closed = False
+ self.failed = False
+ self.failure: Exception | None = None
+ self.encoded_records = self.written_records = 0
+ self.dropped_records = self._unreported_drops = 0
+ if start:
+ self.start()
+
+ def start(self) -> None:
+ if not self._started:
+ self._started = True
+ self._thread.start()
+
+ def _record_failure(self, error: Exception) -> None:
+ with self._failure_lock:
+ if self.failed:
+ return
+ self.failed = True
+ self.failure = error
+ logger.error("OpProf writer failed: %s", error)
+
+ def _writer_unavailable(self) -> bool:
+ if self.failed:
+ return True
+ if self._started and not self._thread.is_alive():
+ self._record_failure(RuntimeError("writer thread stopped unexpectedly"))
+ return True
+ return False
+
+ def _drop(self, pending: int) -> bool:
+ self.dropped_records += 1
+ self._unreported_drops = pending + 1
+ return False
+
+ def submit(self, record: dict[str, Any]) -> bool:
+ if self._closed:
+ raise RuntimeError("OpProf writer is closed")
+ pending = self._unreported_drops
+ record["dropped_records_before"] = pending
+ if self._writer_unavailable():
+ return self._drop(pending)
+ payload = self._encoder.encode(record) + b"\n"
+ self.encoded_records += 1
+ if self._writer_unavailable():
+ return self._drop(pending)
+ try:
+ self._queue.put_nowait(payload)
+ except queue.Full:
+ return self._drop(pending)
+ self._unreported_drops = 0
+ return True
+
+ def _run(self) -> None:
+ buffered = 0
+ last_flush = time.monotonic()
+ try:
+ while True:
+ try:
+ item = self._queue.get(timeout=1.0)
+ except queue.Empty:
+ self._file.flush()
+ buffered = 0
+ last_flush = time.monotonic()
+ continue
+ try:
+ if item is _STOP:
+ break
+ self._file.write(item)
+ buffered += len(item)
+ self.written_records += 1
+ now = time.monotonic()
+ if buffered >= 1 << 20 or now - last_flush >= 1.0:
+ self._file.flush()
+ buffered = 0
+ last_flush = now
+ finally:
+ self._queue.task_done()
+ except Exception as error:
+ self._record_failure(error)
+ finally:
+ footer = dict(
+ schema=SCHEMA_VERSION,
+ record_type="footer",
+ encoded_records=self.encoded_records,
+ written_records=self.written_records,
+ dropped_records=self.dropped_records,
+ )
+ try:
+ self._file.write(self._encoder.encode(footer) + b"\n")
+ self._file.flush()
+ except Exception as error:
+ self._record_failure(error)
+ finally:
+ try:
+ self._file.close()
+ except Exception as error:
+ self._record_failure(error)
+
+ def close(self) -> None:
+ if self._closed:
+ return
+ self._closed = True
+ self.start()
+ if not self._thread.is_alive():
+ return
+ try:
+ self._queue.put(_STOP, timeout=_CLOSE_TIMEOUT_SECONDS)
+ except queue.Full:
+ if not self._thread.is_alive():
+ return
+ try:
+ self._queue.put_nowait(_STOP)
+ except queue.Full:
+ self._record_failure(
+ TimeoutError("timed out enqueueing writer stop sentinel")
+ )
+ return
+ self._thread.join(timeout=_CLOSE_TIMEOUT_SECONDS)
+ if self._thread.is_alive():
+ self._record_failure(TimeoutError("timed out joining writer thread"))
+
+
+class OpProfRecorder:
+ def __init__(self, engine_id: str, writer: JSONLWriter) -> None:
+ self.engine_id = engine_id
+ self.writer = writer
+ self._next_step = 0
+ self._pending: dict[int, dict[str, Any]] = {}
+ atexit.register(self.close)
+
+ @classmethod
+ def create(
+ cls, output_dir: str, dp_rank: int, log_stats: bool
+ ) -> "OpProfRecorder | None":
+ if not output_dir:
+ return None
+ if not log_stats:
+ raise ValueError("VLLM_OPPROF_DIR requires log stats to be enabled")
+ directory = Path(output_dir).expanduser()
+ if not directory.is_absolute():
+ raise ValueError("VLLM_OPPROF_DIR must be an absolute path")
+ directory.mkdir(parents=True, exist_ok=True)
+ engine_id = f"dp{dp_rank}-pid{os.getpid()}"
+ name = f"opprof-v{SCHEMA_VERSION}-{engine_id}-{time.time_ns()}.jsonl"
+ return cls(engine_id, JSONLWriter(directory / name))
+
+ @staticmethod
+ def capture_start(scheduler: Any) -> ScheduleStart:
+ return time.time_ns(), time.monotonic_ns(), _prefix_snapshot(scheduler)
+
+ def begin(self, scheduler: Any, output: Any, start: ScheduleStart) -> None:
+ key = id(output)
+ assert key not in self._pending, "duplicate OpProf begin"
+ new_ids = {request.req_id for request in output.scheduled_new_reqs}
+ prefill_requests = prefill_tokens = decode_requests = decode_tokens = 0
+ context_length_hist = [0] * (len(CONTEXT_LENGTH_EDGES) + 1)
+ chunk_size_hist = [0] * (len(CHUNK_SIZE_EDGES) + 1)
+ chunks: dict[str, Any] = dict.fromkeys(
+ ("first", "middle", "final", "unsplit"), 0
+ )
+ for req_id, num_tokens in output.num_scheduled_tokens.items():
+ request = scheduler.requests[req_id]
+ end = request.num_computed_tokens + num_tokens
+ assert end >= 0
+ context_length_hist[bisect_left(CONTEXT_LENGTH_EDGES, end)] += 1
+ is_prefill = (
+ req_id in new_ids
+ or output.scheduled_cached_reqs.is_context_phase(req_id)
+ )
+ if is_prefill:
+ prefill_requests += 1
+ prefill_tokens += num_tokens
+ assert num_tokens >= 0
+ chunk_size_hist[bisect_left(CHUNK_SIZE_EDGES, num_tokens)] += 1
+ target = request.num_tokens + request.num_output_placeholders
+ chunks[classify_chunk(request.is_prefill_chunk, end, target)] += 1
+ else:
+ decode_requests += 1
+ decode_tokens += num_tokens
+ prefix_after = _prefix_snapshot(scheduler)
+ block_pool = scheduler.kv_cache_manager.block_pool
+ total_blocks = block_pool.num_gpu_blocks - 1
+ free_blocks = block_pool.get_num_free_blocks()
+ assert 0 <= free_blocks <= total_blocks
+ chunks["tokens"] = prefill_tokens
+ chunks["chunk_size_hist"] = chunk_size_hist
+ values = dict(
+ schema=SCHEMA_VERSION,
+ engine_id=self.engine_id,
+ step_index=self._next_step,
+ submit_wall_ns=start[0],
+ submit_mono_ns=start[1],
+ model_executed=output.total_num_scheduled_tokens > 0,
+ scheduled_requests=len(output.num_scheduled_tokens),
+ decode_batch_size=decode_requests,
+ prefill_requests=prefill_requests,
+ prefill_tokens=prefill_tokens,
+ decode_tokens=decode_tokens,
+ chunked_prefill=chunks,
+ context_length_hist=context_length_hist,
+ preemptions=len(output.preempted_req_ids or ()),
+ queues=dict(
+ running=len(scheduler.running),
+ waiting=len(scheduler.waiting),
+ deferred=len(scheduler.skipped_waiting),
+ ),
+ kv=dict(
+ total_blocks=total_blocks,
+ free_blocks=free_blocks,
+ used_blocks=total_blocks - free_blocks,
+ usage=scheduler.kv_cache_manager.usage,
+ ),
+ prefix=dict(
+ local=_prefix_delta(start[2]["local"], prefix_after["local"]),
+ external=_prefix_delta(start[2]["external"], prefix_after["external"]),
+ ),
+ )
+ assert prefill_tokens + decode_tokens == output.total_num_scheduled_tokens
+ self._pending[key] = values
+ self._next_step += 1
+
+ def finalize(self, output: Any, cudagraph_stat: Any | None) -> bool:
+ try:
+ values = self._pending.pop(id(output))
+ except KeyError:
+ raise AssertionError("missing or already finalized OpProf step") from None
+ if cudagraph_stat is None:
+ assert not values["model_executed"]
+ cudagraph = dict(
+ hit=False,
+ runtime_mode="NONE",
+ unpadded_tokens=0,
+ bucket_tokens=0,
+ padding_tokens=0,
+ )
+ else:
+ mode = str(cudagraph_stat.runtime_mode).rsplit(".", 1)[-1]
+ cudagraph = dict(
+ hit=mode != "NONE",
+ runtime_mode=mode,
+ unpadded_tokens=cudagraph_stat.num_unpadded_tokens,
+ bucket_tokens=cudagraph_stat.num_padded_tokens,
+ padding_tokens=cudagraph_stat.num_paddings,
+ )
+ record = dict(
+ values,
+ complete_mono_ns=time.monotonic_ns(),
+ cudagraph=cudagraph,
+ moe_expert_load=None,
+ dropped_records_before=0,
+ )
+ return self.writer.submit(record)
+
+ def close(self) -> None:
+ self.writer.close()
+
+ @property
+ def failed(self) -> bool:
+ return self.writer.failed
+
+ @property
+ def failure(self) -> Exception | None:
+ return self.writer.failure
diff --git a/vllm/v1/worker/gpu_model_runner.py b/vllm/v1/worker/gpu_model_runner.py
index 74938a8..c11d773 100644
--- a/vllm/v1/worker/gpu_model_runner.py
+++ b/vllm/v1/worker/gpu_model_runner.py
@@ -437,6 +437,7 @@ class GPUModelRunner(
self.scheduler_config = vllm_config.scheduler_config
self.speculative_config = vllm_config.speculative_config
self.observability_config = vllm_config.observability_config
+ self.opprof_enabled = bool(envs.VLLM_OPPROF_DIR)
model_config = self.model_config
cache_config = self.cache_config
@@ -3917,7 +3918,10 @@ class GPUModelRunner(
assert batch_descriptor.num_tokens == num_tokens_padded
cudagraph_stats = None
- if self.vllm_config.observability_config.cudagraph_metrics:
+ if (
+ self.vllm_config.observability_config.cudagraph_metrics
+ or self.opprof_enabled
+ ):
cudagraph_stats = CUDAGraphStat(
num_unpadded_tokens=num_tokens,
num_padded_tokens=batch_descriptor.num_tokens,
--
2.43.0

417
patches/vllm-0.24.0-opprof/0002-Add-standalone-OpProf-telemetry-tests.patch Normal file
View File

@@ -0,0 +1,417 @@
From 4f4ee674f217698436b00c3ab6357f59a792477a Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sat, 11 Jul 2026 17:29:02 +0800
Subject: [PATCH 2/5] Add standalone OpProf telemetry tests
Assisted-by: OpenAI Codex
---
tests/v1/core/test_opprof.py | 397 +++++++++++++++++++++++++++++++++++
1 file changed, 397 insertions(+)
create mode 100644 tests/v1/core/test_opprof.py
diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
new file mode 100644
index 0000000..9bfbfcc
--- /dev/null
+++ b/tests/v1/core/test_opprof.py
@@ -0,0 +1,397 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
+"""Standalone tests: this file intentionally does not import the vllm package."""
+
+import errno
+import importlib.util
+import logging
+import sys
+import threading
+from pathlib import Path
+from types import SimpleNamespace
+
+import msgspec
+import pytest
+
+_ROOT = Path(__file__).parents[3]
+_SPEC = importlib.util.spec_from_file_location(
+ "opprof_standalone", _ROOT / "vllm" / "v1" / "opprof.py"
+)
+assert _SPEC is not None and _SPEC.loader is not None
+opprof = importlib.util.module_from_spec(_SPEC)
+sys.modules[_SPEC.name] = opprof
+_SPEC.loader.exec_module(opprof)
+
+
+class CachedRequests:
+ def __init__(self, context_ids=()):
+ self.context_ids = set(context_ids)
+
+ def is_context_phase(self, req_id):
+ return req_id in self.context_ids
+
+
+def prefix_stats(**overrides):
+ values = dict.fromkeys(opprof._PREFIX_FIELDS, 0)
+ values.update(overrides)
+ return SimpleNamespace(**values)
+
+
+def request(computed, total, was_chunk=False, placeholders=0):
+ return SimpleNamespace(
+ num_computed_tokens=computed,
+ num_tokens=total,
+ num_output_placeholders=placeholders,
+ is_prefill_chunk=was_chunk,
+ )
+
+
+def scheduler(requests, local=None, external=None):
+ block_pool = SimpleNamespace(
+ num_gpu_blocks=101,
+ get_num_free_blocks=lambda: 40,
+ )
+ kv_manager = SimpleNamespace(
+ prefix_cache_stats=local or prefix_stats(),
+ block_pool=block_pool,
+ usage=0.6,
+ )
+ return SimpleNamespace(
+ requests=requests,
+ kv_cache_manager=kv_manager,
+ connector_prefix_cache_stats=external,
+ running=list(range(len(requests))),
+ waiting=[0, 1],
+ skipped_waiting=[0],
+ )
+
+
+def schedule_output(tokens, context_ids=(), new_ids=(), preempted=()):
+ return SimpleNamespace(
+ scheduled_new_reqs=[SimpleNamespace(req_id=req_id) for req_id in new_ids],
+ scheduled_cached_reqs=CachedRequests(context_ids),
+ num_scheduled_tokens=tokens,
+ total_num_scheduled_tokens=sum(tokens.values()),
+ preempted_req_ids=set(preempted),
+ )
+
+
+def graph(mode="FULL", unpadded=1, padded=1):
+ return SimpleNamespace(
+ runtime_mode=mode,
+ num_unpadded_tokens=unpadded,
+ num_padded_tokens=padded,
+ num_paddings=padded - unpadded,
+ )
+
+
+def recorder(tmp_path, *, capacity=8192, start=True):
+ path = tmp_path / "opprof.jsonl"
+ writer = opprof.JSONLWriter(path, capacity=capacity, start=start)
+ return opprof.OpProfRecorder("dp0-pid1", writer), path
+
+
+def emit(rec, sched, output, cg=None):
+ start = rec.capture_start(sched)
+ rec.begin(sched, output, start)
+ return rec.finalize(output, cg or graph())
+
+
+def read_jsonl(path):
+ return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()]
+
+
+def test_import_light_and_approved_constants():
+ assert "torch" not in sys.modules
+ assert "vllm" not in sys.modules
+ assert opprof.DEFAULT_QUEUE_CAPACITY == 8192
+ assert tuple(1 << i for i in range(7, 18)) == opprof.CONTEXT_LENGTH_EDGES
+ assert tuple(1 << i for i in range(4, 12)) == opprof.CHUNK_SIZE_EDGES
+
+
+def test_schema_and_invariants(tmp_path):
+ sched = scheduler(
+ {
+ "first": request(0, 100),
+ "final": request(64, 100, was_chunk=True),
+ "decode": request(1024, 1025),
+ }
+ )
+ output = schedule_output(
+ {"first": 64, "final": 36, "decode": 1},
+ context_ids={"final"},
+ new_ids={"first"},
+ preempted={"old"},
+ )
+ rec, path = recorder(tmp_path)
+ start = rec.capture_start(sched)
+ sched.kv_cache_manager.prefix_cache_stats = prefix_stats(
+ requests=1, queries=100, hits=64
+ )
+ rec.begin(sched, output, start)
+ assert rec.finalize(output, graph("FULL", 101, 128))
+ rec.close()
+
+ record, footer = read_jsonl(path)
+ assert record["schema"] == 1
+ assert record["scheduled_requests"] == 3
+ assert record["prefill_requests"] == 2
+ assert record["decode_batch_size"] == 1
+ assert record["prefill_tokens"] + record["decode_tokens"] == 101
+ assert sum(record["context_length_hist"]) == 3
+ assert len(record["context_length_hist"]) == 12
+ assert sum(record["chunked_prefill"]["chunk_size_hist"]) == 2
+ assert len(record["chunked_prefill"]["chunk_size_hist"]) == 9
+ assert record["chunked_prefill"]["first"] == 1
+ assert record["chunked_prefill"]["final"] == 1
+ assert record["preemptions"] == 1
+ assert record["kv"] == {
+ "total_blocks": 100,
+ "free_blocks": 40,
+ "used_blocks": 60,
+ "usage": 0.6,
+ }
+ assert record["prefix"]["local"]["hits"] == 64
+ assert record["moe_expert_load"] is None
+ assert record["complete_mono_ns"] >= record["submit_mono_ns"]
+ assert footer["record_type"] == "footer"
+ assert footer["written_records"] == 1
+
+
+def test_capture_record_matches_pre_refactor_golden(tmp_path, monkeypatch):
+ sched = scheduler(
+ {
+ "edge": request(0, 128),
+ "after": request(65, 129, was_chunk=True),
+ "decode": request(256, 257),
+ }
+ )
+ output = schedule_output(
+ {"edge": 128, "after": 64, "decode": 1},
+ context_ids={"after"},
+ new_ids={"edge"},
+ preempted={"old"},
+ )
+ rec, path = recorder(tmp_path)
+ zero_prefix = dict.fromkeys(opprof._PREFIX_FIELDS, 0)
+ start = (100, 200, {"local": zero_prefix, "external": None})
+ monkeypatch.setattr(opprof.time, "monotonic_ns", lambda: 300)
+
+ rec.begin(sched, output, start)
+ assert rec.finalize(output, graph("FULL", 193, 256))
+ rec.close()
+
+ record = read_jsonl(path)[0]
+ assert record == {
+ "schema": 1,
+ "engine_id": "dp0-pid1",
+ "step_index": 0,
+ "submit_wall_ns": 100,
+ "submit_mono_ns": 200,
+ "model_executed": True,
+ "scheduled_requests": 3,
+ "decode_batch_size": 1,
+ "prefill_requests": 2,
+ "prefill_tokens": 192,
+ "decode_tokens": 1,
+ "chunked_prefill": {
+ "first": 0,
+ "middle": 0,
+ "final": 1,
+ "unsplit": 1,
+ "tokens": 192,
+ "chunk_size_hist": [0, 0, 1, 1, 0, 0, 0, 0, 0],
+ },
+ "context_length_hist": [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
+ "preemptions": 1,
+ "queues": {"running": 3, "waiting": 2, "deferred": 1},
+ "kv": {
+ "total_blocks": 100,
+ "free_blocks": 40,
+ "used_blocks": 60,
+ "usage": 0.6,
+ },
+ "prefix": {"local": zero_prefix, "external": None},
+ "complete_mono_ns": 300,
+ "cudagraph": {
+ "hit": True,
+ "runtime_mode": "FULL",
+ "unpadded_tokens": 193,
+ "bucket_tokens": 256,
+ "padding_tokens": 63,
+ },
+ "moe_expert_load": None,
+ "dropped_records_before": 0,
+ }
+
+
+@pytest.mark.parametrize(
+ ("was_chunk", "end", "target", "expected"),
+ [
+ (False, 64, 100, "first"),
+ (True, 80, 100, "middle"),
+ (True, 100, 100, "final"),
+ (False, 100, 100, "unsplit"),
+ ],
+)
+def test_chunk_classification(was_chunk, end, target, expected):
+ assert opprof.classify_chunk(was_chunk, end, target) == expected
+
+
+def test_async_pairing_out_of_order_and_double_finalize(tmp_path):
+ sched = scheduler({"a": request(10, 11), "b": request(200, 201)})
+ first = schedule_output({"a": 1})
+ second = schedule_output({"b": 1})
+ rec, path = recorder(tmp_path)
+ rec.begin(sched, first, rec.capture_start(sched))
+ rec.begin(sched, second, rec.capture_start(sched))
+ assert rec.finalize(second, graph())
+ assert rec.finalize(first, graph("NONE"))
+ with pytest.raises(AssertionError, match="already finalized"):
+ rec.finalize(second, graph())
+ assert not rec._pending
+ rec.close()
+ records = read_jsonl(path)[:-1]
+ assert [record["step_index"] for record in records] == [1, 0]
+ assert records[0]["context_length_hist"][1] == 1
+ assert records[1]["context_length_hist"][0] == 1
+
+
+def test_disabled_noop_and_log_stats_fail_fast(tmp_path):
+ assert opprof.OpProfRecorder.create("", dp_rank=0, log_stats=False) is None
+ with pytest.raises(ValueError, match="requires log stats"):
+ opprof.OpProfRecorder.create(str(tmp_path), dp_rank=0, log_stats=False)
+ assert not list(tmp_path.iterdir())
+
+
+def test_bounded_queue_drop_accounting(tmp_path):
+ sched = scheduler({str(i): request(i, i + 1) for i in range(3)})
+ rec, path = recorder(tmp_path, capacity=1, start=False)
+ assert emit(rec, sched, schedule_output({"0": 1}))
+ assert not emit(rec, sched, schedule_output({"1": 1}))
+ rec.writer.start()
+ rec.writer._queue.join()
+ assert emit(rec, sched, schedule_output({"2": 1}))
+ rec.close()
+
+ first, after_drop, footer = read_jsonl(path)
+ assert first["step_index"] == 0
+ assert after_drop["step_index"] == 2
+ assert after_drop["dropped_records_before"] == 1
+ assert footer["encoded_records"] == 3
+ assert footer["written_records"] == 2
+ assert footer["dropped_records"] == 1
+
+
+def test_writer_enospc_is_exposed_and_shutdown_is_bounded(
+ tmp_path, monkeypatch, caplog
+):
+ sched = scheduler(
+ {
+ "first": request(0, 1),
+ "after_failure": request(1, 2),
+ }
+ )
+ rec, _ = recorder(tmp_path, capacity=1, start=False)
+ assert emit(rec, sched, schedule_output({"first": 1}))
+
+ real_file = rec.writer._file
+
+ def fail_enospc(*_args, **_kwargs):
+ raise OSError(errno.ENOSPC, "No space left on device")
+
+ failing_file = SimpleNamespace(
+ write=fail_enospc,
+ flush=fail_enospc,
+ close=real_file.close,
+ )
+ monkeypatch.setattr(rec.writer, "_file", failing_file)
+ caplog.set_level(logging.ERROR, logger=opprof.__name__)
+
+ rec.writer.start()
+ rec.writer._thread.join(timeout=1.0)
+ assert not rec.writer._thread.is_alive()
+
+ producer_result = emit(
+ rec, sched, schedule_output({"after_failure": 1})
+ )
+
+ closer = threading.Thread(target=rec.close, daemon=True)
+ closer.start()
+ closer.join(timeout=1.0)
+ assert not closer.is_alive(), "OpProf close blocked after writer failure"
+ assert not producer_result
+ assert rec.writer.dropped_records == 1
+ assert rec.failed
+ assert isinstance(rec.failure, OSError)
+ assert rec.failure.errno == errno.ENOSPC
+ errors = [
+ record
+ for record in caplog.records
+ if "OpProf writer failed" in record.getMessage()
+ ]
+ assert len(errors) == 1
+
+
+def test_shutdown_flush_is_idempotent(tmp_path):
+ sched = scheduler({"decode": request(8, 9)})
+ rec, path = recorder(tmp_path)
+ assert emit(rec, sched, schedule_output({"decode": 1}))
+ rec.close()
+ rec.close()
+ record, footer = read_jsonl(path)
+ assert record["step_index"] == 0
+ assert footer["written_records"] == 1
+ assert path.stat().st_size > 0
+
+
+def test_piecewise_cudagraph_record_preserved(tmp_path):
+ sched = scheduler({"decode": request(4096, 4097)})
+ output = schedule_output({"decode": 1})
+ rec, path = recorder(tmp_path)
+ assert emit(rec, sched, output, graph("PIECEWISE", 513, 520))
+ rec.close()
+ record = read_jsonl(path)[0]
+ assert record["cudagraph"] == {
+ "hit": True,
+ "runtime_mode": "PIECEWISE",
+ "unpadded_tokens": 513,
+ "bucket_tokens": 520,
+ "padding_tokens": 7,
+ }
+
+
+def test_zero_scheduled_tokens_finalize_without_cudagraph(tmp_path):
+ sched = scheduler({})
+ output = schedule_output({})
+ rec, path = recorder(tmp_path)
+ rec.begin(sched, output, rec.capture_start(sched))
+ assert rec._pending
+
+ assert rec.finalize(output, cudagraph_stat=None)
+ assert not rec._pending
+ rec.close()
+
+ record = read_jsonl(path)[0]
+ assert record["model_executed"] is False
+ assert record["scheduled_requests"] == 0
+ assert record["prefill_requests"] == 0
+ assert record["prefill_tokens"] == 0
+ assert record["decode_batch_size"] == 0
+ assert record["decode_tokens"] == 0
+ assert record["context_length_hist"] == [0] * 12
+ assert record["chunked_prefill"] == {
+ "first": 0,
+ "middle": 0,
+ "final": 0,
+ "unsplit": 0,
+ "tokens": 0,
+ "chunk_size_hist": [0] * 9,
+ }
+ assert record["cudagraph"] == {
+ "hit": False,
+ "runtime_mode": "NONE",
+ "unpadded_tokens": 0,
+ "bucket_tokens": 0,
+ "padding_tokens": 0,
+ }
--
2.43.0

52
patches/vllm-0.24.0-opprof/0003-Log-the-OpProf-output-path-at-startup.patch Normal file
View File

@@ -0,0 +1,52 @@
From 668cfb7e27e488454dbf09a4927b8a60d6d49b40 Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sat, 11 Jul 2026 17:32:27 +0800
Subject: [PATCH 3/5] Log the OpProf output path at startup
Assisted-by: OpenAI Codex
---
tests/v1/core/test_opprof.py | 1 +
vllm/v1/core/sched/scheduler.py | 2 ++
vllm/v1/opprof.py | 1 +
3 files changed, 4 insertions(+)
diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
index 9bfbfcc..79c1fae 100644
--- a/tests/v1/core/test_opprof.py
+++ b/tests/v1/core/test_opprof.py
@@ -336,6 +336,7 @@ def test_writer_enospc_is_exposed_and_shutdown_is_bounded(
def test_shutdown_flush_is_idempotent(tmp_path):
sched = scheduler({"decode": request(8, 9)})
rec, path = recorder(tmp_path)
+ assert rec.writer.path == path
assert emit(rec, sched, schedule_output({"decode": 1}))
rec.close()
rec.close()
diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py
index 303c562..769a02a 100644
--- a/vllm/v1/core/sched/scheduler.py
+++ b/vllm/v1/core/sched/scheduler.py
@@ -278,6 +278,8 @@ class Scheduler(SchedulerInterface):
dp_rank=self.parallel_config.data_parallel_index,
log_stats=self.log_stats,
)
+ if self.opprof is not None:
+ logger.info("OpProf telemetry enabled: %s", self.opprof.writer.path)
self.use_pp = self.parallel_config.pipeline_parallel_size > 1
self.use_v2_model_runner = vllm_config.use_v2_model_runner
diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
index f0330d0..75f63de 100644
--- a/vllm/v1/opprof.py
+++ b/vllm/v1/opprof.py
@@ -68,6 +68,7 @@ class JSONLWriter:
start: bool = True,
) -> None:
self._queue: queue.Queue[Any] = queue.Queue(capacity)
+ self.path = path
self._encoder = msgspec.json.Encoder()
self._file = path.open("xb", buffering=1 << 20)
self._thread = threading.Thread(target=self._run, daemon=True)
--
2.43.0

64
patches/vllm-0.24.0-opprof/0004-Exclude-OpProf-output-path-from-compile-cache-key.patch Normal file
View File

@@ -0,0 +1,64 @@
From 335da4abe60e0177872e0b7751e86eeec6756a2b Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sat, 11 Jul 2026 22:32:30 +0800
Subject: [PATCH 4/5] Exclude OpProf output path from compile cache key
---
tests/v1/core/test_opprof.py | 21 ++++++++++++++++++++-
vllm/envs.py | 1 +
2 files changed, 21 insertions(+), 1 deletion(-)
diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
index 79c1fae..a820e7e 100644
--- a/tests/v1/core/test_opprof.py
+++ b/tests/v1/core/test_opprof.py
@@ -8,7 +8,7 @@ import logging
import sys
import threading
from pathlib import Path
-from types import SimpleNamespace
+from types import ModuleType, SimpleNamespace
import msgspec
import pytest
@@ -23,6 +23,25 @@ sys.modules[_SPEC.name] = opprof
_SPEC.loader.exec_module(opprof)
+def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch):
+ spec = importlib.util.spec_from_file_location(
+ "envs_standalone", _ROOT / "vllm" / "envs.py"
+ )
+ assert spec is not None and spec.loader is not None
+ envs = importlib.util.module_from_spec(spec)
+ monkeypatch.setitem(sys.modules, spec.name, envs)
+ spec.loader.exec_module(envs)
+
+ monkeypatch.setitem(sys.modules, "vllm", ModuleType("vllm"))
+ monkeypatch.setitem(sys.modules, "vllm.config", ModuleType("vllm.config"))
+ config_utils = ModuleType("vllm.config.utils")
+ config_utils.__dict__["normalize_value"] = lambda value: value
+ monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils)
+ monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof")
+
+ assert "VLLM_OPPROF_DIR" not in envs.compile_factors()
+
+
class CachedRequests:
def __init__(self, context_ids=()):
self.context_ids = set(context_ids)
diff --git a/vllm/envs.py b/vllm/envs.py
index b3093e9..5634708 100755
--- a/vllm/envs.py
+++ b/vllm/envs.py
@@ -2044,6 +2044,7 @@ def compile_factors() -> dict[str, object]:
"VLLM_LOGGING_CONFIG_PATH",
"VLLM_LOGGING_COLOR",
"VLLM_LOG_STATS_INTERVAL",
+ "VLLM_OPPROF_DIR",
"VLLM_DEBUG_LOG_API_SERVER_RESPONSE",
"VLLM_TUNED_CONFIG_FOLDER",
"VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR",
--
2.43.0

24
patches/vllm-0.24.0-opprof/0005-Keep-compile-factor-regression-import-light.patch Normal file
View File

@@ -0,0 +1,24 @@
From bbfa7176a6a3686a88ee66696f1ad8d754559d96 Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sat, 11 Jul 2026 22:38:08 +0800
Subject: [PATCH 5/5] Keep compile-factor regression import-light
---
tests/v1/core/test_opprof.py | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
index a820e7e..0b8a8a1 100644
--- a/tests/v1/core/test_opprof.py
+++ b/tests/v1/core/test_opprof.py
@@ -37,6 +37,7 @@ def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch):
config_utils = ModuleType("vllm.config.utils")
config_utils.__dict__["normalize_value"] = lambda value: value
monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils)
+ monkeypatch.setitem(sys.modules, "torch", ModuleType("torch"))
monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof")
assert "VLLM_OPPROF_DIR" not in envs.compile_factors()
--
2.43.0

306
patches/vllm-0.24.0-opprof/0006-Checkpoint-OpProf-accounting-across-hard-kills.patch Normal file
View File

@@ -0,0 +1,306 @@
From f8b68f2452c424d22de4a69527a427207dcfbca5 Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sun, 12 Jul 2026 12:56:39 +0800
Subject: [PATCH] Checkpoint OpProf accounting across hard kills
---
tests/v1/core/test_opprof.py | 118 +++++++++++++++++++++++++++++++++++
vllm/v1/opprof.py | 84 ++++++++++++++++++++++---
2 files changed, 195 insertions(+), 7 deletions(-)
diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py
index 0b8a8a1..007c5bb 100644
--- a/tests/v1/core/test_opprof.py
+++ b/tests/v1/core/test_opprof.py
@@ -5,6 +5,8 @@
import errno
import importlib.util
import logging
+import os
+import subprocess
import sys
import threading
from pathlib import Path
@@ -121,6 +123,12 @@ def read_jsonl(path):
return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()]
+def read_sidecar(path):
+ return msgspec.json.decode(
+ path.with_name(f"{path.name}.footer.json").read_bytes()
+ )
+
+
def test_import_light_and_approved_constants():
assert "torch" not in sys.modules
assert "vllm" not in sys.modules
@@ -366,6 +374,116 @@ def test_shutdown_flush_is_idempotent(tmp_path):
assert path.stat().st_size > 0
+def test_sidecar_updates_are_atomic(tmp_path, monkeypatch):
+ writer = opprof.JSONLWriter(tmp_path / "atomic.jsonl", start=False)
+ writer._write_sidecar(
+ encoded_records=1,
+ written_records=1,
+ dropped_records=0,
+ last_step_index=0,
+ final=False,
+ )
+ original = writer.sidecar_path.read_bytes()
+ real_replace = os.replace
+ replacements = []
+
+ def inspect_replace(source, destination):
+ assert Path(destination) == writer.sidecar_path
+ assert writer.sidecar_path.read_bytes() == original
+ candidate = msgspec.json.decode(Path(source).read_bytes())
+ assert candidate["written_records"] == 2
+ replacements.append(candidate)
+ real_replace(source, destination)
+
+ monkeypatch.setattr(opprof.os, "replace", inspect_replace)
+ writer._write_sidecar(
+ encoded_records=3,
+ written_records=2,
+ dropped_records=1,
+ last_step_index=2,
+ final=False,
+ )
+ monkeypatch.setattr(opprof.os, "replace", real_replace)
+
+ assert len(replacements) == 1
+ assert read_sidecar(writer.path)["encoded_records"] == 3
+ assert not list(tmp_path.glob(".*.tmp-*"))
+ writer.close()
+
+
+def test_hard_kill_sidecar_balances_last_flush(tmp_path):
+ path = tmp_path / "hard-kill.jsonl"
+ child = """
+import importlib.util
+import sys
+import time
+from pathlib import Path
+
+source, output = sys.argv[1:]
+spec = importlib.util.spec_from_file_location("opprof_hard_kill", source)
+module = importlib.util.module_from_spec(spec)
+sys.modules[spec.name] = module
+spec.loader.exec_module(module)
+writer = module.JSONLWriter(Path(output))
+for step in range(3):
+ assert writer.submit({"schema": 1, "step_index": step})
+writer._queue.join()
+while not writer.sidecar_path.exists():
+ time.sleep(0.01)
+print("READY", flush=True)
+time.sleep(60)
+"""
+ process = subprocess.Popen(
+ [
+ sys.executable,
+ "-c",
+ child,
+ str(_ROOT / "vllm/v1/opprof.py"),
+ str(path),
+ ],
+ stdout=subprocess.PIPE,
+ text=True,
+ )
+ try:
+ assert process.stdout is not None
+ assert process.stdout.readline().strip() == "READY"
+ process.kill()
+ assert process.wait(timeout=5) < 0
+ finally:
+ if process.poll() is None:
+ process.kill()
+ process.wait(timeout=5)
+
+ records = read_jsonl(path)
+ sidecar = read_sidecar(path)
+ assert len(records) == 3
+ assert all(record.get("record_type") != "footer" for record in records)
+ assert sidecar["final"] is False
+ assert sidecar["written_records"] == len(records)
+ assert sidecar["encoded_records"] == (
+ sidecar["written_records"] + sidecar["dropped_records"]
+ )
+ assert sidecar["last_step_index"] == records[-1]["step_index"] == 2
+
+
+def test_clean_footer_and_final_sidecar_agree(tmp_path):
+ sched = scheduler({"decode": request(8, 9)})
+ rec, path = recorder(tmp_path)
+ assert emit(rec, sched, schedule_output({"decode": 1}))
+ rec.close()
+
+ record, footer = read_jsonl(path)
+ sidecar = read_sidecar(path)
+ assert sidecar["record_type"] == "footer_checkpoint"
+ assert sidecar["stream"] == path.name
+ assert sidecar["final"] is True
+ assert sidecar["last_step_index"] == record["step_index"] == 0
+ assert sidecar["checkpoint_wall_ns"] > 0
+ assert sidecar["flush_interval_seconds"] == opprof.FLUSH_INTERVAL_SECONDS
+ for counter in ("encoded_records", "written_records", "dropped_records"):
+ assert sidecar[counter] == footer[counter]
+
+
def test_piecewise_cudagraph_record_preserved(tmp_path):
sched = scheduler({"decode": request(4096, 4097)})
output = schedule_output({"decode": 1})
diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py
index 75f63de..28d9635 100644
--- a/vllm/v1/opprof.py
+++ b/vllm/v1/opprof.py
@@ -7,6 +7,7 @@ import queue
import threading
import time
from bisect import bisect_left
+from contextlib import suppress
from pathlib import Path
from typing import Any
@@ -18,6 +19,7 @@ SCHEMA_VERSION = 1
CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18))
CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12))
DEFAULT_QUEUE_CAPACITY = 8192
+FLUSH_INTERVAL_SECONDS = 1.0
_CLOSE_TIMEOUT_SECONDS = 1.0
_STOP = object()
_PREFIX_FIELDS = ( # noqa: SIM905
@@ -69,6 +71,7 @@ class JSONLWriter:
) -> None:
self._queue: queue.Queue[Any] = queue.Queue(capacity)
self.path = path
+ self.sidecar_path = path.with_name(f"{path.name}.footer.json")
self._encoder = msgspec.json.Encoder()
self._file = path.open("xb", buffering=1 << 20)
self._thread = threading.Thread(target=self._run, daemon=True)
@@ -78,6 +81,9 @@ class JSONLWriter:
self.failure: Exception | None = None
self.encoded_records = self.written_records = 0
self.dropped_records = self._unreported_drops = 0
+ self._checkpoint_encoded_records = 0
+ self._checkpoint_dropped_records = 0
+ self._last_written_step_index: int | None = None
if start:
self.start()
@@ -119,33 +125,90 @@ class JSONLWriter:
if self._writer_unavailable():
return self._drop(pending)
try:
- self._queue.put_nowait(payload)
+ self._queue.put_nowait(
+ (
+ payload,
+ self.encoded_records,
+ self.dropped_records,
+ int(record["step_index"]),
+ )
+ )
except queue.Full:
return self._drop(pending)
self._unreported_drops = 0
return True
+ def _write_sidecar(
+ self,
+ *,
+ encoded_records: int,
+ written_records: int,
+ dropped_records: int,
+ last_step_index: int | None,
+ final: bool,
+ ) -> None:
+ sidecar = dict(
+ schema=SCHEMA_VERSION,
+ record_type="footer_checkpoint",
+ stream=self.path.name,
+ encoded_records=encoded_records,
+ written_records=written_records,
+ dropped_records=dropped_records,
+ last_step_index=last_step_index,
+ checkpoint_wall_ns=time.time_ns(),
+ flush_interval_seconds=FLUSH_INTERVAL_SECONDS,
+ final=final,
+ )
+ temporary_path = self.sidecar_path.with_name(
+ f".{self.sidecar_path.name}.tmp-{os.getpid()}-{threading.get_ident()}"
+ )
+ try:
+ with temporary_path.open("wb") as temporary_file:
+ temporary_file.write(self._encoder.encode(sidecar) + b"\n")
+ temporary_file.flush()
+ os.replace(temporary_path, self.sidecar_path)
+ finally:
+ with suppress(FileNotFoundError):
+ temporary_path.unlink()
+
+ def _flush_checkpoint(self) -> None:
+ self._file.flush()
+ self._write_sidecar(
+ encoded_records=self._checkpoint_encoded_records,
+ written_records=self.written_records,
+ dropped_records=self._checkpoint_dropped_records,
+ last_step_index=self._last_written_step_index,
+ final=False,
+ )
+
def _run(self) -> None:
buffered = 0
last_flush = time.monotonic()
try:
while True:
try:
- item = self._queue.get(timeout=1.0)
+ item = self._queue.get(timeout=FLUSH_INTERVAL_SECONDS)
except queue.Empty:
- self._file.flush()
+ self._flush_checkpoint()
buffered = 0
last_flush = time.monotonic()
continue
try:
if item is _STOP:
break
- self._file.write(item)
- buffered += len(item)
+ payload, encoded, dropped, step_index = item
+ self._file.write(payload)
+ buffered += len(payload)
self.written_records += 1
+ self._checkpoint_encoded_records = encoded
+ self._checkpoint_dropped_records = dropped
+ self._last_written_step_index = step_index
now = time.monotonic()
- if buffered >= 1 << 20 or now - last_flush >= 1.0:
- self._file.flush()
+ if (
+ buffered >= 1 << 20
+ or now - last_flush >= FLUSH_INTERVAL_SECONDS
+ ):
+ self._flush_checkpoint()
buffered = 0
last_flush = now
finally:
@@ -163,6 +226,13 @@ class JSONLWriter:
try:
self._file.write(self._encoder.encode(footer) + b"\n")
self._file.flush()
+ self._write_sidecar(
+ encoded_records=footer["encoded_records"],
+ written_records=footer["written_records"],
+ dropped_records=footer["dropped_records"],
+ last_step_index=self._last_written_step_index,
+ final=True,
+ )
except Exception as error:
self._record_failure(error)
finally:
--
2.43.0

27
patches/vllm-0.24.0-opprof/0007-Recreate-scheduled-torch-profiler-between-windows.patch Normal file
View File

@@ -0,0 +1,27 @@
From 23450fb21ac255b0cf710f4ee965ee694921975d Mon Sep 17 00:00:00 2001
From: Gahow Wang <gahow.wang@gmail.com>
Date: Sun, 12 Jul 2026 13:12:52 +0800
Subject: [PATCH] Recreate scheduled torch profiler between windows
---
vllm/v1/worker/gpu_worker.py | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py
index 5e266a3..0058f96 100644
--- a/vllm/v1/worker/gpu_worker.py
+++ b/vllm/v1/worker/gpu_worker.py
@@ -978,6 +978,10 @@ class Worker(WorkerBase):
logger.warning("Profiler was not started, nothing to stop.")
return
self.profiler.stop()
+ # A scheduled torch.profiler.profile does not reset its schedule
+ # after stop(). Recreate it for the next /start_profile window.
+ if isinstance(self.profiler, TorchProfilerWrapper):
+ self.profiler = None
def execute_dummy_batch(self) -> None:
num_tokens = getattr(self.model_runner, "uniform_decode_query_len", 1)
--
2.43.0

119
patches/vllm-0.24.0-opprof/README.md Normal file
View File

@@ -0,0 +1,119 @@
# vLLM 0.24.0 OpProf patch series
## Goal
Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM `v0.24.0`
at base commit `ee0da84ab9e04ac7610e28580af62c365e898389`. The series adds one
scheduler-owned composition record per step without installing new runtime
dependencies or changing GPU kernels.
## Contents
- `0001-Add-lightweight-per-step-OpProf-telemetry.patch`: adds the environment
switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the
existing CUDA-graph stat. Writer failures are exposed without blocking
producers or shutdown, and request histograms are accumulated in-place.
- `0002-Add-standalone-OpProf-telemetry-tests.patch`: adds CPU-only tests that
load the recorder directly without importing or installing vLLM or torch,
including ENOSPC, golden-record, and zero-token regressions.
- `0003-Log-the-OpProf-output-path-at-startup.patch`: logs the resolved JSONL
output path and covers it in the standalone shutdown test.
- `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch`: prevents the
per-run telemetry destination from invalidating vLLM's torch.compile/AOT
cache and adds an import-light regression test.
- `0005-Keep-compile-factor-regression-import-light.patch`: isolates the new
regression from torch in full vLLM test environments.
- `0006-Checkpoint-OpProf-accounting-across-hard-kills.patch`: atomically
checkpoints balanced writer counters beside each JSONL stream once per
flush interval, with clean-close and hard-kill regressions.
- `0007-Recreate-scheduled-torch-profiler-between-windows.patch`: discards a
stopped scheduled torch-profiler wrapper so each subsequent official profile
endpoint call receives a fresh 2+8 schedule and emits its own trace.
- `apply.sh`: verifies the exact base, refuses dirty/wrong revisions, applies
all numbered patches with `git am`, and exits successfully only when the
exact series is already applied directly on the required base.
- `pytest-evidence.txt`: exact isolated test command, dependency versions, and
all-pass summary.
The source branch tip used to generate the patches is
`23450fb21ac255b0cf710f4ee965ee694921975d` (`opprof`).
## Apply
Prerequisite: a clean checkout whose `HEAD` is the exact base commit.
```bash
./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0
```
Running the command again is a no-op only when the five matching patch commits
are rooted directly at the required base. A partially applied series, dirty
tree, unrelated commit, or any other `HEAD` is rejected instead of being
guessed around.
## Enable and output
Set an absolute output directory before starting vLLM:
```bash
export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof
```
Unset or empty disables the feature before recorder construction. Combining it
with `--disable-log-stats` fails fast, as approved.
Each EngineCore/DP scheduler writes one file named approximately
`opprof-v1-dp0-pid1234-<start_ns>.jsonl`. Records contain schema/engine/step and
timestamps; scheduled prefill/decode composition; first/middle/final/unsplit
prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions;
running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas;
CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap
accounting. A clean close writes a final writer-count footer in the stream.
Every JSONL flush also atomically replaces
`<stream>.footer.json` through a same-directory temporary file. The sidecar
contains the encoded, written, and dropped counts through that durable flush,
the last written step index, a wall-clock timestamp, the one-second flush
interval, and whether it is final. Queue entries carry their submission
ordinal and cumulative drops, so a periodic checkpoint always satisfies
`encoded = written + dropped` without decoding records in the writer thread.
On clean close the in-stream footer is authoritative and the final sidecar must
agree with all three counters. If a hard kill prevents the in-stream footer,
the latest sidecar is authoritative: the decoded data-line count must equal
its `written_records`, its final data-line step must equal
`last_step_index`, and its counters must balance. Data after that checkpoint
may be lost, bounded by at most the configured one-second flush interval.
The bounded queue holds 8192 encoded records. Producers never wait for disk;
full queues or a failed writer drop the new record and report the gap on the
next successful record. A writer I/O failure is exposed through recorder state,
logged once, and cannot make shutdown wait indefinitely. The writer flushes at
1 MiB, one second, or shutdown.
## Test
Only pytest and msgspec are required. `--confcutdir` prevents vLLM's global
test configuration from importing its full dependency stack.
```bash
cd /path/to/vllm-v0.24.0
uv run --no-project --with pytest --with msgspec \
pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
```
Expected summary:
```text
18 passed in 1.09s
```
## Caveats
- Layer 1 intentionally records no expert-load arrays. Exact routed experts
remain a separate Layer-2 run.
- `PIECEWISE` means graph-wrapped compiled regions, not full-step graph replay.
- Phase 2 must measure the always-on overhead; acceptance requires the upper
bound of the 95% confidence interval to remain below 3% for every primary
serving metric.
- Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2
and TP4 counterpoints. Record the selected MoE backend log every run.

58
patches/vllm-0.24.0-opprof/apply.sh Executable file
View File

@@ -0,0 +1,58 @@
#!/usr/bin/env bash
set -euo pipefail
base_commit=ee0da84ab9e04ac7610e28580af62c365e898389
script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)
repo=${1:-.}
patches=("$script_dir"/0*.patch)
git -C "$repo" rev-parse --git-dir >/dev/null
if [[ -n $(git -C "$repo" status --porcelain) ]]; then
echo "Refusing to apply to a dirty worktree." >&2
exit 1
fi
if ((${#patches[@]} == 0)) || [[ ! -f ${patches[0]} ]]; then
echo "No numbered patch files found in $script_dir" >&2
exit 1
fi
head_commit=$(git -C "$repo" rev-parse HEAD)
mapfile -t recent_commits < <(
git -C "$repo" rev-list --max-count="${#patches[@]}" --reverse HEAD
)
patches_match=true
((${#recent_commits[@]} == ${#patches[@]})) || patches_match=false
for i in "${!patches[@]}"; do
$patches_match || break
patch_id=$(git patch-id --stable <"${patches[$i]}" | awk '{print $1}')
commit_id=$(
git -C "$repo" show --pretty=format: "${recent_commits[$i]}" |
git patch-id --stable | awk '{print $1}'
)
[[ $patch_id == "$commit_id" ]] || patches_match=false
done
first_parent=""
if ((${#recent_commits[@]} > 0)); then
first_parent=$(
git -C "$repo" rev-parse --verify "${recent_commits[0]}^" 2>/dev/null || true
)
fi
if $patches_match && [[ $first_parent == "$base_commit" ]]; then
echo "OpProf patch series is already applied."
exit 0
fi
if [[ $head_commit == "$base_commit" ]]; then
git -C "$repo" am "${patches[@]}"
echo "Applied OpProf patch series to $repo"
exit 0
fi
if $patches_match; then
echo "Refusing to treat OpProf patches as already applied:" >&2
echo "first patch parent is $first_parent, expected $base_commit" >&2
exit 1
fi
echo "Refusing to apply: HEAD is $head_commit; expected $base_commit" >&2
echo "or the exact OpProf patch series rooted at that base." >&2
exit 1

16
patches/vllm-0.24.0-opprof/pytest-evidence.txt Normal file
View File

@@ -0,0 +1,16 @@
OpProf standalone pytest evidence
Date: 2026-07-12
Source branch: opprof
Source tip: 23450fb21ac255b0cf710f4ee965ee694921975d
Base: ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0)
Environment: Python 3.11.13, pytest 9.1.1, msgspec 0.21.1
vLLM installed: no
torch installed in isolated test environment: no
GPU/remote access: no
Command:
uv run --no-project --with pytest --with msgspec pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q
Output:
.................. [100%]
18 passed in 1.09s

784
runs/opprof-phase3-ea/provenance/opprof_phase3_client.py Normal file
View File

@@ -0,0 +1,784 @@
#!/usr/bin/env python3
"""Token-exact fixed-duration client for the OpProf Phase-3 protocol."""
from __future__ import annotations
import argparse
import asyncio
import gzip
import hashlib
import json
import math
import os
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import aiohttp
SCHEMA = 1
TOKEN_BASE = 1000
TOKEN_SPAN = 100000
class ManifestExhausted(RuntimeError):
pass
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def atomic_json(path: Path, value: Any, mode: int = 0o640) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
with os.fdopen(fd, "w", encoding="utf-8") as f:
json.dump(value, f, sort_keys=True, indent=2)
f.write("\n")
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def atomic_jsonl(path: Path, rows: list[dict[str, Any]], mode: int = 0o640) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
with os.fdopen(fd, "w", encoding="utf-8") as f:
for row in rows:
f.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n")
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def parse_range(value: str) -> tuple[int, int]:
lo_text, hi_text = value.split(":", 1)
lo, hi = int(lo_text), int(hi_text)
if lo <= 0 or hi < lo:
raise argparse.ArgumentTypeError(f"invalid positive range: {value}")
return lo, hi
def _integer_counts(weights: list[float], total: int) -> list[int]:
raw = [w * total for w in weights]
counts = [math.floor(x) for x in raw]
order = sorted(
range(len(raw)), key=lambda i: raw[i] - counts[i], reverse=True
)
for idx in order[: total - sum(counts)]:
counts[idx] += 1
return counts
def numeric_sanity(values: list[float | int]) -> dict[str, Any]:
finite = [float(x) for x in values if math.isfinite(float(x))]
return {
"n": len(values),
"finite_n": len(finite),
"missing_n": len(values) - len(finite),
"min": min(finite) if finite else None,
"max": max(finite) if finite else None,
"distinct_n": len(set(finite)),
}
def manifest_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
return {
"schema": SCHEMA,
"rows": len(rows),
"input_tokens": numeric_sanity([int(r["input_tokens"]) for r in rows]),
"output_tokens": numeric_sanity([int(r["output_tokens"]) for r in rows]),
"arrival_values": sorted({str(r["arrival"]) for r in rows}),
"pattern_values": sorted({str(r["pattern_id"]) for r in rows}),
}
def materialize(args: argparse.Namespace) -> dict[str, Any]:
import numpy as np
rng = np.random.default_rng(args.workload_seed)
n = args.num_requests
if args.kind == "prefix-pool":
if args.num_prefixes <= 0 or args.prefix_len <= 0 or args.suffix_fixed <= 0:
raise ValueError("prefix-pool requires positive pool/prefix/suffix")
lengths = np.full(n, args.prefix_len + args.suffix_fixed, dtype=np.int64)
prefix_ids = np.arange(n, dtype=np.int64) % args.num_prefixes
rng.shuffle(prefix_ids)
else:
prefix_ids = np.full(n, -1, dtype=np.int64)
if args.input_uniform:
lo, hi = parse_range(args.input_uniform)
lengths = rng.integers(lo, hi + 1, n, dtype=np.int64)
elif args.input_fixed:
lengths = np.full(n, args.input_fixed, dtype=np.int64)
elif args.input_mixture:
spec = json.loads(args.input_mixture)
if not isinstance(spec, dict) or not spec:
raise ValueError("input mixture must be a non-empty JSON object")
keys = list(spec)
weights = [float(spec[key]) for key in keys]
if any(w < 0 for w in weights) or not math.isclose(sum(weights), 1.0):
raise ValueError("mixture weights must be non-negative and sum to 1")
pieces = []
for key, count in zip(
keys, _integer_counts(weights, n), strict=True
):
kind, lo_text, hi_text = key.split(":")
if kind != "uniform":
raise ValueError(f"unsupported mixture component: {key}")
pieces.append(
rng.integers(
int(lo_text), int(hi_text) + 1, count, dtype=np.int64
)
)
lengths = np.concatenate(pieces)
rng.shuffle(lengths)
else:
raise ValueError("exactly one input distribution is required")
if args.output_fixed <= 0 or args.arrival not in {"steady", "burst:8"}:
raise ValueError("invalid output length or arrival class")
rows = []
for i in range(n):
row = {
"schema": SCHEMA,
"request_id": f"{args.id}-{i:05d}",
"pattern_id": args.id,
"kind": args.kind,
"input_tokens": int(lengths[i]),
"output_tokens": args.output_fixed,
"arrival": args.arrival,
"token_seed": int(args.workload_seed * 1000003 + i),
}
if args.kind == "prefix-pool":
row.update(
{
"prefix_id": int(prefix_ids[i]),
"num_prefixes": args.num_prefixes,
"prefix_tokens": args.prefix_len,
}
)
rows.append(row)
out = Path(args.out)
atomic_jsonl(out, rows, mode=0o600)
summary = manifest_summary(rows)
summary.update({"sha256": sha256_file(out), "path": str(out)})
atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
print(json.dumps(summary, sort_keys=True))
return summary
def materialize_private(args: argparse.Namespace) -> dict[str, Any]:
from transformers import AutoTokenizer
source = Path(args.source)
selected: list[dict[str, Any]] = []
with source.open(encoding="utf-8") as f:
for source_index, line in enumerate(f):
row = json.loads(line)
if (
float(row["sampling_u"]) <= args.sampling_u_max
and int(row["input_length"]) <= args.max_input_tokens
):
selected.append(
{
"schema": SCHEMA,
"request_id": f"{args.id}-{len(selected):05d}",
"pattern_id": args.id,
"kind": "private-trace",
"input_tokens": int(row["input_length"]),
"output_tokens": min(
int(row["output_length"]), args.output_cap
),
"arrival": args.arrival,
"source_index": source_index,
"prompt": row["prompt"],
}
)
tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
diffs = [
len(tokenizer.encode(row["prompt"], add_special_tokens=False))
- row["input_tokens"]
for row in selected
]
exact = sum(diff == 0 for diff in diffs)
exact_fraction = exact / len(diffs) if diffs else 0.0
max_abs = max((abs(diff) for diff in diffs), default=-1)
if exact_fraction < 0.99 or max_abs > 1:
raise RuntimeError(
"tokenizer parity gate failed: "
f"exact_fraction={exact_fraction:.6f} max_abs_error={max_abs}"
)
out = Path(args.out)
atomic_jsonl(out, selected, mode=0o600)
summary = manifest_summary(selected)
summary.update(
{
"sha256": sha256_file(out),
"source_sha256": sha256_file(source),
"tokenizer_exact_n": exact,
"tokenizer_exact_fraction": exact_fraction,
"tokenizer_max_abs_error": max_abs,
"path": str(out),
}
)
atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
print(json.dumps(summary, sort_keys=True))
return summary
def load_manifest(path: Path) -> list[dict[str, Any]]:
rows = [json.loads(line) for line in path.read_text().splitlines() if line]
required = {
"request_id",
"pattern_id",
"input_tokens",
"output_tokens",
"arrival",
}
if not rows:
raise ValueError("empty manifest")
for row in rows:
if not required.issubset(row):
raise ValueError(f"manifest row lacks {sorted(required - set(row))}")
if len({row["request_id"] for row in rows}) != len(rows):
raise ValueError("duplicate request_id")
return rows
def _token_stream(seed: int, count: int) -> list[int]:
state = seed & 0xFFFFFFFF
out = []
for _ in range(count):
state = (1664525 * state + 1013904223) & 0xFFFFFFFF
out.append(TOKEN_BASE + state % TOKEN_SPAN)
return out
def synthetic_prompt(row: dict[str, Any]) -> list[int]:
length = int(row["input_tokens"])
seed = int(row.get("token_seed", 0))
if row.get("kind") == "prefix-pool":
prefix_n = int(row["prefix_tokens"])
tokens = _token_stream(0xA5A50000 + int(row["prefix_id"]), prefix_n)
tokens += _token_stream(seed, length - prefix_n)
offset = prefix_n
else:
tokens = _token_stream(seed, length)
offset = 0
if length - offset >= 3:
index = int(row["request_id"].rsplit("-", 1)[1])
tokens[offset : offset + 3] = [
TOKEN_BASE + index % 100,
TOKEN_BASE + (index // 100) % 100,
TOKEN_BASE + (index // 10000) % 100,
]
return tokens
@dataclass
class RunContext:
args: argparse.Namespace
rows: list[dict[str, Any]]
t0: float
clean_end: float
stop_event: asyncio.Event
lock: asyncio.Lock
next_index: int = 0
in_flight: int = 0
max_in_flight: int = 0
exhausted: bool = False
admission_stop_s: float | None = None
async def next_row(self) -> dict[str, Any]:
async with self.lock:
if self.next_index >= len(self.rows):
self.exhausted = True
raise ManifestExhausted(
f"manifest exhausted after {self.next_index} admissions"
)
row = self.rows[self.next_index]
self.next_index += 1
return row
async def request_one(
ctx: RunContext,
session: aiohttp.ClientSession,
row: dict[str, Any],
scheduled: float,
) -> dict[str, Any]:
loop = asyncio.get_running_loop()
admitted = loop.time()
ctx.in_flight += 1
ctx.max_in_flight = max(ctx.max_in_flight, ctx.in_flight)
status = 0
actual_output: int | None = None
first_token: float | None = None
error_kind: str | None = None
try:
prompt: str | list[int] = (
row["prompt"]
if row.get("kind") == "private-trace"
else synthetic_prompt(row)
)
if not isinstance(prompt, str) and len(prompt) != int(row["input_tokens"]):
raise AssertionError("synthetic prompt length drift")
payload = {
"model": ctx.args.model,
"prompt": prompt,
"max_tokens": int(row["output_tokens"]),
"temperature": ctx.args.temperature,
"ignore_eos": ctx.args.ignore_eos,
"stream": True,
"stream_options": {"include_usage": True},
"add_special_tokens": False,
"seed": ctx.args.server_seed,
}
headers = {
"Content-Type": "application/json",
"x-request-id": str(row["request_id"]),
}
async with session.post(
ctx.args.base_url.rstrip("/") + "/v1/completions",
json=payload,
headers=headers,
) as response:
status = response.status
if status != 200:
error_kind = f"http_{status}"
else:
buf = b""
async for chunk in response.content.iter_any():
buf += chunk
while b"\n" in buf:
line, buf = buf.split(b"\n", 1)
line = line.strip()
if not line.startswith(b"data:"):
continue
data = line[5:].strip()
if data == b"[DONE]":
continue
event = json.loads(data)
if event.get("choices") and first_token is None:
first_token = loop.time()
if event.get("usage") is not None:
actual_output = int(event["usage"]["completion_tokens"])
if actual_output is None:
error_kind = "missing_usage"
except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
error_kind = type(exc).__name__
except Exception as exc:
error_kind = type(exc).__name__
finally:
completed = loop.time()
ctx.in_flight -= 1
success = (
status == 200
and error_kind is None
and actual_output == int(row["output_tokens"])
)
if status == 200 and actual_output is not None and not success:
error_kind = "output_token_mismatch"
return {
"schema": SCHEMA,
"request_id": row["request_id"],
"scheduled_s": scheduled - ctx.t0,
"admitted_s": admitted - ctx.t0,
"first_token_s": None if first_token is None else first_token - ctx.t0,
"completed_s": completed - ctx.t0,
"input_tokens": int(row["input_tokens"]),
"requested_output_tokens": int(row["output_tokens"]),
"actual_output_tokens": actual_output,
"http_status": status,
"success": success,
"error_kind": error_kind,
}
async def saturation_load(
ctx: RunContext, session: aiohttp.ClientSession
) -> list[dict[str, Any]]:
results: list[dict[str, Any]] = []
async def worker() -> None:
while not ctx.stop_event.is_set():
try:
row = await ctx.next_row()
except ManifestExhausted:
ctx.stop_event.set()
return
results.append(
await request_one(ctx, session, row, asyncio.get_running_loop().time())
)
tasks = [
asyncio.create_task(worker()) for _ in range(ctx.args.max_concurrency)
]
await asyncio.gather(*tasks)
return results
async def finite_load(
ctx: RunContext, session: aiohttp.ClientSession, rate: float
) -> list[dict[str, Any]]:
sem = asyncio.Semaphore(ctx.args.max_concurrency)
tasks: list[asyncio.Task[dict[str, Any]]] = []
batch = 8 if str(ctx.rows[0]["arrival"]) == "burst:8" else 1
period = batch / rate
event_index = 0
async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]:
async with sem:
return await request_one(ctx, session, row, scheduled)
while not ctx.stop_event.is_set():
scheduled = ctx.t0 + event_index * period
delay = scheduled - asyncio.get_running_loop().time()
if delay > 0:
try:
await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay)
break
except asyncio.TimeoutError:
pass
if ctx.stop_event.is_set():
break
try:
for _ in range(batch):
tasks.append(
asyncio.create_task(limited(await ctx.next_row(), scheduled))
)
except ManifestExhausted:
ctx.stop_event.set()
break
event_index += 1
return await asyncio.gather(*tasks) if tasks else []
async def post_profile(
session: aiohttp.ClientSession, base_url: str, endpoint: str
) -> tuple[float, float, int]:
loop = asyncio.get_running_loop()
before = loop.time()
async with session.post(base_url.rstrip("/") + endpoint) as response:
status = response.status
await response.read()
return before, loop.time(), status
def _trace_loadable(path: Path) -> bool:
try:
opener = gzip.open if path.suffix == ".gz" else open
with opener(path, "rt", encoding="utf-8") as f:
parsed = json.load(f)
return isinstance(parsed, dict) and isinstance(parsed.get("traceEvents"), list)
except (OSError, EOFError, json.JSONDecodeError):
return False
async def wait_new_trace(
trace_dir: Path, before: set[Path], timeout: float
) -> Path:
deadline = asyncio.get_running_loop().time() + timeout
while asyncio.get_running_loop().time() < deadline:
for path in sorted(set(trace_dir.glob("*.pt.trace.json*")) - before):
if _trace_loadable(path):
return path
await asyncio.sleep(0.25)
raise TimeoutError(f"no new loadable trace within {timeout}s")
async def timeline(
ctx: RunContext, session: aiohttp.ClientSession
) -> list[dict[str, Any]]:
args = ctx.args
profiles: list[dict[str, Any]] = []
await asyncio.sleep(max(0, ctx.clean_end - asyncio.get_running_loop().time()))
if args.profile_after_clean:
trace_dir = Path(args.profile_trace_dir)
for window in range(args.num_profile_windows):
prior = set(trace_dir.glob("*.pt.trace.json*"))
start_before, start_after, start_status = await post_profile(
session, args.base_url, "/start_profile"
)
trace = await wait_new_trace(
trace_dir, prior, args.profile_timeout_seconds
)
trace_ready = asyncio.get_running_loop().time()
stop_before, stop_after, stop_status = await post_profile(
session, args.base_url, "/stop_profile"
)
profiles.append(
{
"window": window + 1,
"start_call_s": start_before - ctx.t0,
"start_return_s": start_after - ctx.t0,
"trace_ready_s": trace_ready - ctx.t0,
"stop_call_s": stop_before - ctx.t0,
"stop_return_s": stop_after - ctx.t0,
"start_status": start_status,
"stop_status": stop_status,
"trace_file": trace.name,
"trace_sha256": sha256_file(trace),
}
)
if start_status != 200 or stop_status != 200:
raise RuntimeError("profile endpoint returned non-200")
await asyncio.sleep(args.recovery_seconds)
else:
await asyncio.sleep(args.post_clean_seconds)
ctx.admission_stop_s = asyncio.get_running_loop().time() - ctx.t0
ctx.stop_event.set()
return profiles
def segment_summary(
records: list[dict[str, Any]], start: float, end: float
) -> dict[str, Any]:
admitted = [r for r in records if start <= r["admitted_s"] < end]
completed = [r for r in records if start <= r["completed_s"] < end]
successes = [r for r in completed if r["success"]]
duration = end - start
return {
"start_s": start,
"end_s": end,
"duration_s": duration,
"admitted": len(admitted),
"completed": len(successes),
"failed": len(completed) - len(successes),
"offered_rps": len(admitted) / duration,
"completed_throughput_rps": len(successes) / duration,
"input_tokens": sum(r["input_tokens"] for r in successes),
"output_tokens": sum(r["actual_output_tokens"] or 0 for r in successes),
}
async def run_load(args: argparse.Namespace) -> dict[str, Any]:
manifest = Path(args.manifest)
rows = load_manifest(manifest)
arrivals = {row["arrival"] for row in rows}
if len(arrivals) != 1:
raise ValueError("a manifest must have one arrival class")
if args.load_point == "saturation":
if args.request_rate != "inf":
raise ValueError("saturation requires --request-rate inf")
rate = math.inf
else:
if not args.saturation_result:
raise ValueError("moderate requires --saturation-result")
sat = json.loads(Path(args.saturation_result).read_text())
rate = args.rate_fraction * float(sat["clean"]["completed_throughput_rps"])
if not math.isfinite(rate) or rate <= 0:
raise ValueError("derived moderate rate must be positive and finite")
loop = asyncio.get_running_loop()
t0 = loop.time()
clean_seconds = args.clean_segment_seconds * args.num_clean_segments
ctx = RunContext(
args=args,
rows=rows,
t0=t0,
clean_end=t0 + args.warmup_seconds + clean_seconds,
stop_event=asyncio.Event(),
lock=asyncio.Lock(),
)
timeout = aiohttp.ClientTimeout(total=None, connect=30, sock_read=600)
connector = aiohttp.TCPConnector(limit=args.max_concurrency)
async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session:
profile_task = asyncio.create_task(timeline(ctx, session))
load_task = asyncio.create_task(
saturation_load(ctx, session)
if math.isinf(rate)
else finite_load(ctx, session, rate)
)
try:
profiles = await profile_task
except Exception:
ctx.stop_event.set()
await load_task
raise
records = await load_task
clean_start = args.warmup_seconds
clean_end = clean_start + clean_seconds
clean = segment_summary(records, clean_start, clean_end)
segments = []
for i in range(args.num_clean_segments):
start = clean_start + i * args.clean_segment_seconds
segments.append(
{
"name": chr(ord("A") + i),
**segment_summary(
records, start, start + args.clean_segment_seconds
),
}
)
successful = [r for r in records if r["success"]]
elapsed_seconds = loop.time() - t0
if ctx.admission_stop_s is None:
raise RuntimeError("admission stop timestamp was not recorded")
drain_seconds = elapsed_seconds - ctx.admission_stop_s
result = {
"schema": SCHEMA,
"manifest_sha256": sha256_file(manifest),
"manifest_rows": len(rows),
"manifest_admitted": ctx.next_index,
"manifest_wrapped": False,
"manifest_exhausted": ctx.exhausted,
"load_point": args.load_point,
"request_rate": "inf" if math.isinf(rate) else rate,
"rate_fraction": None if math.isinf(rate) else args.rate_fraction,
"arrival": next(iter(arrivals)),
"warmup_seconds": args.warmup_seconds,
"clean_segment_seconds": args.clean_segment_seconds,
"num_clean_segments": args.num_clean_segments,
"elapsed_seconds": elapsed_seconds,
"admission_stop_s": ctx.admission_stop_s,
"drain_seconds": drain_seconds,
"max_in_flight": ctx.max_in_flight,
"records": len(records),
"successful_records": len(successful),
"failed_records": len(records) - len(successful),
"clean": clean,
"segments": segments,
"profiles": profiles,
}
sanity = {
"schema": SCHEMA,
"numeric": {
"input_tokens": numeric_sanity([r["input_tokens"] for r in records]),
"requested_output_tokens": numeric_sanity(
[r["requested_output_tokens"] for r in records]
),
"actual_output_tokens": numeric_sanity(
[
r["actual_output_tokens"]
for r in records
if r["actual_output_tokens"] is not None
]
),
"scheduled_s": numeric_sanity([r["scheduled_s"] for r in records]),
"admitted_s": numeric_sanity([r["admitted_s"] for r in records]),
"completed_s": numeric_sanity([r["completed_s"] for r in records]),
},
"invariants": {
"clean_duration_exact": math.isclose(clean["duration_s"], clean_seconds),
"segment_count_exact": len(segments) == args.num_clean_segments,
"manifest_no_wrap": ctx.next_index <= len(rows),
"manifest_not_exhausted": not ctx.exhausted,
"concurrency_bounded": ctx.max_in_flight <= args.max_concurrency,
"drain_within_timeout": drain_seconds <= args.drain_timeout_seconds,
"output_tokens_exact": all(
r["actual_output_tokens"] == r["requested_output_tokens"]
for r in successful
),
"clean_failures_zero": clean["failed"] == 0,
"profile_count_exact": len(profiles)
== (args.num_profile_windows if args.profile_after_clean else 0),
"profile_status_ok": all(
p["start_status"] == 200 and p["stop_status"] == 200
for p in profiles
),
},
}
if not math.isinf(rate):
sanity["invariants"]["moderate_offered_within_5pct"] = (
abs(clean["offered_rps"] / rate - 1) <= 0.05
)
out = Path(args.result_dir)
out.mkdir(parents=True, exist_ok=True)
atomic_jsonl(out / "requests.jsonl", sorted(records, key=lambda r: r["admitted_s"]))
atomic_jsonl(out / "segments.jsonl", segments)
atomic_json(out / "result.json", result)
atomic_json(out / "sanity.json", sanity)
if ctx.exhausted:
raise ManifestExhausted("manifest exhausted; result retained for diagnosis")
failed = [name for name, ok in sanity["invariants"].items() if not ok]
if failed:
raise RuntimeError(f"client sanity failure: {failed}")
return result
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser()
sub = parser.add_subparsers(dest="command", required=True)
mat = sub.add_parser("materialize")
mat.add_argument("--id", required=True)
mat.add_argument("--kind", choices=("synthetic", "prefix-pool"), required=True)
group = mat.add_mutually_exclusive_group()
group.add_argument("--input-uniform")
group.add_argument("--input-fixed", type=int)
group.add_argument("--input-mixture")
mat.add_argument("--output-fixed", type=int, required=True)
mat.add_argument("--prefix", default="none")
mat.add_argument("--arrival", required=True)
mat.add_argument("--num-requests", type=int, required=True)
mat.add_argument("--workload-seed", type=int, required=True)
mat.add_argument("--num-prefixes", type=int, default=0)
mat.add_argument("--prefix-len", type=int, default=0)
mat.add_argument("--suffix-fixed", type=int, default=0)
mat.add_argument("--out", required=True)
private = sub.add_parser("materialize-private")
private.add_argument("--id", required=True)
private.add_argument("--source", required=True)
private.add_argument("--sampling-u-max", type=float, required=True)
private.add_argument("--max-input-tokens", type=int, required=True)
private.add_argument("--output-cap", type=int, required=True)
private.add_argument("--preserve-prompts", action="store_true", required=True)
private.add_argument("--disable-shuffle", action="store_true", required=True)
private.add_argument("--arrival", required=True)
private.add_argument("--model", required=True)
private.add_argument("--out", required=True)
run = sub.add_parser("run")
run.add_argument("--manifest", required=True)
run.add_argument("--base-url", required=True)
run.add_argument("--model", required=True)
run.add_argument("--load-point", choices=("saturation", "moderate"), required=True)
run.add_argument("--request-rate")
run.add_argument("--saturation-result")
run.add_argument("--rate-fraction", type=float, default=0.60)
run.add_argument("--max-concurrency", type=int, default=256)
run.add_argument("--ignore-eos", action="store_true")
run.add_argument("--temperature", type=float, default=0.0)
run.add_argument("--warmup-seconds", type=float, default=60)
run.add_argument("--clean-segment-seconds", type=float, default=80)
run.add_argument("--num-clean-segments", type=int, default=3)
run.add_argument("--profile-after-clean", action="store_true")
run.add_argument("--num-profile-windows", type=int, default=0)
run.add_argument("--profile-warmup-iterations", type=int, default=2)
run.add_argument("--profile-active-iterations", type=int, default=8)
run.add_argument("--profile-trace-dir")
run.add_argument("--profile-timeout-seconds", type=float, default=120)
run.add_argument("--recovery-seconds", type=float, default=30)
run.add_argument("--post-clean-seconds", type=float, default=0)
run.add_argument("--drain-timeout-seconds", type=float, default=120)
run.add_argument("--workload-seed", type=int, default=20260712)
run.add_argument("--server-seed", type=int, default=20260712)
run.add_argument("--result-dir", required=True)
return parser
def main() -> None:
args = build_parser().parse_args()
if args.command == "materialize":
materialize(args)
elif args.command == "materialize-private":
materialize_private(args)
else:
if args.profile_after_clean and not args.profile_trace_dir:
raise ValueError("--profile-after-clean requires --profile-trace-dir")
print(json.dumps(asyncio.run(run_load(args)), sort_keys=True))
if __name__ == "__main__":
main()

1045
runs/opprof-phase3-ea/provenance/opprof_phase3_controller.py Normal file
View File

File diff suppressed because it is too large Load Diff

353
runs/opprof-phase3-ea/provenance/test_phase3_tools.py Normal file
View File

@@ -0,0 +1,353 @@
#!/usr/bin/env python3
from __future__ import annotations
import asyncio
import importlib.util
import json
import sys
import tempfile
import unittest
from unittest import mock
from pathlib import Path
from types import SimpleNamespace
from aiohttp import web
HERE = Path(__file__).parent
def load_module(name: str, filename: str):
spec = importlib.util.spec_from_file_location(name, HERE / filename)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
assert spec.loader is not None
spec.loader.exec_module(module)
return module
client = load_module("phase3_client", "opprof_phase3_client.py")
controller = load_module("phase3_controller", "opprof_phase3_controller.py")
def rows(n: int, arrival: str = "steady") -> list[dict]:
return [
{
"schema": 1,
"request_id": f"T-{i:05d}",
"pattern_id": "T",
"kind": "synthetic",
"input_tokens": 8,
"output_tokens": 2,
"arrival": arrival,
"token_seed": i + 1,
}
for i in range(n)
]
class MockServer:
def __init__(self) -> None:
self.active = 0
self.max_active = 0
self.payloads = []
self.runner = None
self.port = None
async def completion(self, request):
payload = await request.json()
self.payloads.append(payload)
self.active += 1
self.max_active = max(self.max_active, self.active)
await asyncio.sleep(0.01)
response = web.StreamResponse(
status=200, headers={"Content-Type": "text/event-stream"}
)
await response.prepare(request)
await response.write(
b'data: {"choices":[{"text":"x"}],"usage":null}\n\n'
)
usage = json.dumps(
{
"choices": [],
"usage": {
"prompt_tokens": len(payload["prompt"]),
"completion_tokens": payload["max_tokens"],
},
}
).encode()
await response.write(b"data: " + usage + b"\n\n")
await response.write(b"data: [DONE]\n\n")
await response.write_eof()
self.active -= 1
return response
async def start(self):
app = web.Application()
app.router.add_post("/v1/completions", self.completion)
self.runner = web.AppRunner(app)
await self.runner.setup()
site = web.TCPSite(self.runner, "127.0.0.1", 0)
await site.start()
self.port = site._server.sockets[0].getsockname()[1]
async def stop(self):
await self.runner.cleanup()
def run_args(
manifest: Path,
result_dir: Path,
port: int,
load_point: str = "saturation",
saturation_result: Path | None = None,
):
return SimpleNamespace(
manifest=str(manifest),
base_url=f"http://127.0.0.1:{port}",
model="mock",
load_point=load_point,
request_rate="inf" if load_point == "saturation" else None,
saturation_result=None if saturation_result is None else str(saturation_result),
rate_fraction=0.60,
max_concurrency=3,
ignore_eos=True,
temperature=0,
warmup_seconds=0.04,
clean_segment_seconds=0.04,
num_clean_segments=3,
profile_after_clean=False,
num_profile_windows=0,
profile_warmup_iterations=2,
profile_active_iterations=8,
profile_trace_dir=None,
profile_timeout_seconds=1,
recovery_seconds=0,
post_clean_seconds=0,
drain_timeout_seconds=1,
workload_seed=20260712,
server_seed=20260712,
result_dir=str(result_dir),
)
class Phase3ToolTests(unittest.TestCase):
def test_synthetic_prompt_exact_and_unique_early_prefix(self):
generated = [client.synthetic_prompt(row) for row in rows(200)]
self.assertTrue(all(len(value) == 8 for value in generated))
self.assertEqual(len({tuple(value[:3]) for value in generated}), 200)
def test_p05_manifest_has_exact_half_modes(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "P05.jsonl"
args = SimpleNamespace(
id="P05",
kind="synthetic",
input_uniform=None,
input_fixed=None,
input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}',
output_fixed=64,
prefix="none",
arrival="steady",
num_requests=100,
workload_seed=20260712,
num_prefixes=0,
prefix_len=0,
suffix_fixed=0,
out=str(out),
)
client.materialize(args)
manifest = client.load_manifest(out)
self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50)
self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50)
def test_prefix_pool_balanced(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "P08.jsonl"
args = SimpleNamespace(
id="P08",
kind="prefix-pool",
input_uniform=None,
input_fixed=None,
input_mixture=None,
output_fixed=512,
prefix="none",
arrival="burst:8",
num_requests=80,
workload_seed=20260712,
num_prefixes=8,
prefix_len=1024,
suffix_fixed=256,
out=str(out),
)
client.materialize(args)
manifest = client.load_manifest(out)
counts = {i: 0 for i in range(8)}
for row in manifest:
counts[row["prefix_id"]] += 1
self.assertEqual(row["input_tokens"], 1280)
self.assertEqual(set(counts.values()), {10})
def test_fixed_duration_saturation_and_redaction(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000))
mock = MockServer()
await mock.start()
try:
result = await client.run_load(
run_args(manifest, root / "result", mock.port)
)
finally:
await mock.stop()
self.assertEqual(result["max_in_flight"], 3)
self.assertEqual(mock.max_active, 3)
self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9)
self.assertEqual(len(result["segments"]), 3)
self.assertLessEqual(result["drain_seconds"], 1)
self.assertTrue(
all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads)
)
text = (root / "result/requests.jsonl").read_text()
self.assertNotIn('"prompt":', text)
self.assertNotIn('"text":', text)
asyncio.run(case())
def test_drain_timeout_is_a_hard_sanity_failure(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000))
mock = MockServer()
await mock.start()
try:
args = run_args(manifest, root / "result", mock.port)
args.drain_timeout_seconds = 0
with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"):
await client.run_load(args)
finally:
await mock.stop()
sanity = json.loads((root / "result/sanity.json").read_text())
self.assertFalse(sanity["invariants"]["drain_within_timeout"])
asyncio.run(case())
def test_burst_schedule_is_eight_at_once(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000, "burst:8"))
sat = root / "sat.json"
client.atomic_json(
sat, {"clean": {"completed_throughput_rps": 100.0}}
)
mock = MockServer()
await mock.start()
try:
args = run_args(
manifest, root / "result", mock.port, "moderate", sat
)
args.warmup_seconds = 0
args.clean_segment_seconds = 0.4 / 3
result = await client.run_load(args)
finally:
await mock.stop()
records = [
json.loads(line)
for line in (root / "result/requests.jsonl").read_text().splitlines()
]
groups = {}
for record in records:
groups.setdefault(round(record["scheduled_s"], 6), 0)
groups[round(record["scheduled_s"], 6)] += 1
self.assertTrue(all(value == 8 for value in groups.values()))
starts = sorted(groups)
if len(starts) > 1:
self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5)
self.assertAlmostEqual(result["request_rate"], 60.0)
asyncio.run(case())
def test_manifest_exhaustion_stops_instead_of_wrapping(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(2))
mock = MockServer()
await mock.start()
try:
with self.assertRaises(client.ManifestExhausted):
await client.run_load(
run_args(manifest, root / "result", mock.port)
)
finally:
await mock.stop()
asyncio.run(case())
def test_cpu_affinity_sets_are_disjoint_and_cover_host(self):
values = []
for spec in controller.CPU_MAP.values():
lo, hi = [int(x) for x in spec.split("-")]
values.extend(range(lo, hi + 1))
self.assertEqual(sorted(values), list(range(160)))
self.assertEqual(len(values), len(set(values)))
def test_kernel_mapping_priority(self):
cases = {
"void vllm::moe::topkGating": "moe_router",
"ncclDevKernel_AllReduce": "collective",
"flash_fwd_kernel": "attention",
"nvjet_sm90_tst": "moe_gemm",
"argmax_kernel": "sampler",
"cutlass_gemm": "dense_gemm",
"triton_red_fused_add_rms_norm": "norm_elementwise",
"cache_swap_kernel": "kv_memory",
"unknown": "other",
}
self.assertEqual(
{name: controller.classify_kernel(name) for name in cases}, cases
)
def test_atomic_state_replacement(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "state.json"
controller.atomic_json(path, {"schema": 1, "value": 1})
controller.atomic_json(path, {"schema": 1, "value": 2})
self.assertEqual(json.loads(path.read_text())["value"], 2)
self.assertFalse(list(path.parent.glob("*.tmp.*")))
def test_fingerprint_survives_json_roundtrip(self):
original_run_text = controller.run_text
original_hash = controller.sha256_file
try:
controller.run_text = lambda *args, **kwargs: "deadbeef\n"
controller.sha256_file = lambda path: "a" * 64
fingerprint = controller.make_fingerprint()
finally:
controller.run_text = original_run_text
controller.sha256_file = original_hash
self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint)
self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)})
def test_server_shutdown_signals_parent_before_group(self):
process = SimpleNamespace(pid=12345, poll=lambda: None)
with (
mock.patch.object(controller.os, "kill") as kill,
mock.patch.object(controller.os, "killpg") as killpg,
mock.patch.object(controller, "_process_group_alive", return_value=False),
):
controller.stop_servers([process])
kill.assert_called_once_with(12345, controller.signal.SIGINT)
killpg.assert_not_called()
if __name__ == "__main__":
unittest.main(verbosity=2)

47
runs/opprof-phase3/phase3/access-blocker-20260712.json Normal file
View File

@@ -0,0 +1,47 @@
{
"schema": 1,
"status": "external_access_blocked",
"host": "dash0",
"last_reachability_probe_local": "2026-07-12T16:16:39+08:00",
"failure": "SSH connection timed out during banner exchange before any remote command executed",
"last_durable_remote_state": {
"observed_before_outage": true,
"controller_pid": 2237019,
"controller_status": "running",
"active_stage": "primary-02-saturation",
"active_stage_phase": "starting_servers",
"completed_measured_runs": 8,
"drain_quarantined_runs": 0,
"clean_window_failures": 0,
"missing_trace_files": 8,
"gpu_hours_total": 6.8156048206885655
},
"resume": {
"state": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/controller-state.json",
"log": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/phase3-matrix-controller.log",
"command": "cd /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712 && /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python scripts/opprof_phase3_matrix.py run --resume",
"rule": "inspect state/PID/GPU ownership first; do not relaunch a live controller"
},
"profile_control_repair": {
"reason": "P03/C01 saturated all data-plane connector slots, so /start_profile timed out before reaching the server",
"change": "dedicated aiohttp TCPConnector(limit=2) and ClientSession for profile control calls",
"old_client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84",
"new_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
"remote_record": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/repairs/repair-004-profile-control-connector.json",
"tests": "13/13 unittest PASS; ruff PASS; py_compile PASS"
},
"analysis_ready": {
"script_sha256": "205d9012d9462d84403a0f1435c8a449389aa4bef448d8d6dbced707a11559b0",
"test_sha256": "eac6f55042865c9a24811eef9c1a13c23a26dd1504b54348513d8e6a7ba2b939",
"tests": "4/4 unittest PASS; ruff PASS; py_compile PASS",
"executed_on_matrix": false
},
"not_claimed": [
"matrix completion",
"final quarantine count",
"H1a verdict",
"H1b verdict",
"final GPU hours",
"final GPU cleanup"
]
}

1
runs/opprof-phase3/phase3/analysis/analyze-ap37-final.log Normal file
View File

File diff suppressed because one or more lines are too long

1
runs/opprof-phase3/phase3/analysis/analyze-ap37.log Normal file
View File

File diff suppressed because one or more lines are too long

1
runs/opprof-phase3/phase3/analysis/launch-ap37.log Normal file
View File

@@ -0,0 +1 @@
CPU_ANALYSIS A-P3-7: accepted_runs=40, run_root=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3, private_manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests, bootstrap=100000 seed=20260714, output=runs/phase3/analysis/metrics-ap37.json, expected=10-30m CPU-only, GPU_cost=0

737
runs/opprof-phase3/phase3/controller-state-stop-ap36.json Normal file
View File

@@ -0,0 +1,737 @@
{
"clean_window_failures": 0,
"completed_burnins": 5,
"completed_measured_runs": 40,
"controller_pid": 2438791,
"created_at": 1783833885.960389,
"drain_quarantined_runs": 0,
"fingerprint": {
"cells": [
"P08-C00",
"P03-C10",
"P07-C00",
"P10-C01",
"P01-C10",
"P01-C01",
"P10-C10",
"P03-C01",
"P09-C00",
"P06-C01",
"P10-C11",
"P01-C00",
"P01-C11",
"P10-C00",
"P04-C00",
"P03-C00",
"P06-C10",
"P02-C00",
"P06-C00",
"P06-C11",
"P11-C00",
"P10-C00-TP2",
"P03-C11",
"P05-C00"
],
"client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
"common_controller_sha256": "95f7169a1771e385aab40fcaecd967dc5cff0c21ea67bdd139382774ba43f01f",
"controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
"cpu_map": {
"0": "0-19",
"1": "20-39",
"2": "40-59",
"3": "60-79",
"4": "80-99",
"5": "100-119",
"6": "120-139",
"7": "140-159"
},
"manifests": {
"P01": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl",
"rows": 32768,
"sha256": "13ffb226c83373f54c4a7afea6c78cb7cd29720f1858d56728826fc1367b31a4"
},
"P02": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P02.jsonl",
"rows": 32768,
"sha256": "0138ada3fccc98298daee66c26bd1952c987cb42ed8d5341d66b698a597417f9"
},
"P03": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P03.jsonl",
"rows": 32768,
"sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1"
},
"P04": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P04.jsonl",
"rows": 32768,
"sha256": "caf8d0941b093956a81e1413adc4a4ea9d92460f1b2ed0f6a9b118d0119d6247"
},
"P05": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P05.jsonl",
"rows": 32768,
"sha256": "192e213109f8cb429b99d9eb0f227bb4390fc03f63b02d5456569369bff5a3d7"
},
"P06": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P06.jsonl",
"rows": 32768,
"sha256": "65954bc6e47de9e7be07b8975f97f0bc4639979ef6d33e8c664557ded34b9f96"
},
"P07": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P07.jsonl",
"rows": 32768,
"sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6"
},
"P08": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P08.jsonl",
"rows": 32768,
"sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9"
},
"P09": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
"rows": 32768,
"sha256": "7af92ee3c27dc7d2cf895d6ff3a6e737ec4b6da13d6841ca59e1166f28a0ae1e"
},
"P10": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl",
"rows": 4011,
"sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33"
},
"P11": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P11.jsonl",
"rows": 32768,
"sha256": "7d196df38963528ff181cf72ce39c8ad913c8f61d40b1425410d3c6c30b6be18"
}
},
"model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B",
"source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
"source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1"
},
"gpu_hours_this_stage": 0.7173924536837472,
"gpu_hours_total": 14.025875418755744,
"missing_trace_files": 8,
"repairs": [
{
"change": "dedicated aiohttp TCPConnector(limit=2) for profile endpoint session; request stream unchanged",
"evidence": "/start_profile connection acquisition timed out before server receipt while 256 data-plane connections were occupied",
"failed_run": "P03-C01-saturation",
"failed_stage": "primary-02-saturation",
"new_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
"old_client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84",
"repair": "profile-control-connector-isolation",
"retry": "one exact whole-wave retry; failed directories retained as .interrupted-*",
"schema": 1,
"tests": {
"py_compile": "PASS",
"ruff": "PASS",
"unittest": "13/13 PASS"
},
"timestamp": 1783837763.431071
},
{
"budget": {
"hard_cap": 16.0,
"projected_remaining_h20_hours": 6.3857560787267165,
"projected_total_h20_hours": 14.513826778670154,
"projected_total_with_15pct_remaining_contingency": 15.471690190479162,
"used_h20_hours": 8.128070699943438
},
"cause": "controller required failed_records==0 across excluded profile/recovery time although the registered hard gate is zero clean-window failures",
"change": "replace all_failures_zero with clean-window boundary validation plus failed-record accounting; excluded failures remain reported",
"evidence": {
"clean_completed": 5828,
"clean_failed": 0,
"error_kind": "ServerDisconnectedError",
"excluded_window_failures": 2,
"failure_completion_s": [
373.1505718010012,
373.9740898209857
],
"run": "P01-C01-moderate"
},
"failed_stage": "primary-02-moderate",
"gpu_rerun": false,
"new_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f",
"old_controller_sha256": "167e48f98f307e16ee44b321068a82a13813b3bf9a4d76882f774f15fe85e595",
"repair": "clean-window-failure-scope",
"schema": 1,
"tests": {
"py_compile": "PASS",
"ruff": "PASS",
"unittest": "14/14 PASS"
},
"timestamp": 1783847092.6920242
},
{
"cause": "single preflight NVML sample observed 4 MiB on GPU0-3 with 0% utilization and no compute apps immediately after a verified cleanup",
"change": "bounded 60s preflight requiring three consecutive samples with zero memory, zero utilization, and no compute apps; no nonzero-memory tolerance",
"failed_stage": "primary-06-saturation",
"gpu_hours_total_unchanged": 11.920448313262728,
"gpu_launch_before_failure": false,
"new_common_controller_sha256": "95f7169a1771e385aab40fcaecd967dc5cff0c21ea67bdd139382774ba43f01f",
"old_common_controller_sha256": "15ad254298a38c4a9318468db89ab32707b6196bb38d5e2a007f2267529397a5",
"repair": "stable-zero-preflight",
"schema": 1,
"tests": {
"py_compile": "PASS",
"ruff": "PASS",
"unittest": "15/15 PASS"
},
"timestamp": 1783850898.1496143
},
{
"code_change": false,
"gpu_hours_used": 12.606695837948058,
"hard_cap": 16.0,
"projected_remaining_h20_hours": 2.1455133807990285,
"projected_total_with_15pct_contingency": 15.07403622586694,
"protocol_action": "single exact whole-wave infrastructure retry under identical 4-way placement; gate unchanged",
"reason": "first TP2 P10 inference-shape cold start/autotune after readiness yielded 15 successful warmup completions against required 32; clean window and artifacts otherwise valid",
"repair": "tp2-warmup-exact-retry",
"run": "P10-C00-TP2-saturation",
"schema": 1,
"stage": "primary-06-saturation",
"timestamp": 1783851710.3100822,
"warmup_required": 32,
"warmup_success": 15
},
{
"accepted_runs_preserved": 40,
"amendment": "A-P3-6",
"changed_fingerprint_keys": [
"controller_sha256"
],
"created_at": 1783853258.3098204,
"failed_stage_preserved": "primary-06-saturation",
"new_controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
"old_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f",
"projected_remaining_h20_hours": 2.1,
"projected_total_h20_hours": 15.408482965071997,
"repair_id": "repair-008-ap36",
"retained_attempt_re_adjudicated": false,
"retained_attempt_reason": "21 completions but A-P3-6 normalized drift 2.3385345997286295 and bin step counts 13/9/44",
"schema": 1
}
],
"schema": 1,
"stages": {
"burnin-01": {
"assignments": [
{
"cell": "P06-C00",
"gpus": [
0
]
},
{
"cell": "P06-C10",
"gpus": [
1
]
},
{
"cell": "P06-C01",
"gpus": [
2
]
},
{
"cell": "P06-C11",
"gpus": [
3
]
}
],
"burnin": true,
"clients": {},
"completed_at": 1783834288.1238313,
"confirmation": false,
"gpu_hours": 0.30808498481909435,
"load_point": "saturation",
"profile": false,
"servers": {},
"started_at": 1783833886.4448946,
"status": "complete",
"validation_attempt1_failure": "RuntimeError(\"server config/backend failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/burnins/C01: {'triton_moe': True, 'chunked_mbt': False, 'tp_effective': True, 'drain_shutdown': True, 'mns_effective': True}\")"
},
"burnin-02": {
"assignments": [
{
"cell": "P06-C00-TP2",
"gpus": [
0,
1
]
}
],
"burnin": true,
"clients": {},
"completed_at": 1783834671.921921,
"confirmation": false,
"gpu_hours": 0.19903395679261948,
"load_point": "saturation",
"profile": false,
"servers": {},
"started_at": 1783834301.7813473,
"status": "complete"
},
"primary-01-moderate": {
"assignments": [
{
"cell": "P08-C00",
"gpus": [
0
]
},
{
"cell": "P03-C10",
"gpus": [
1
]
},
{
"cell": "P07-C00",
"gpus": [
2
]
},
{
"cell": "P10-C01",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783836723.3639715,
"confirmation": false,
"gpu_hours": 0.5195140059126748,
"load_point": "moderate",
"profile": true,
"servers": {},
"started_at": 1783836150.2169476,
"status": "complete",
"validation_attempt1_failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P10-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")",
"warmup_feasibility": "P10 target=min(32,floor(0.445*60))=26; observed=27"
},
"primary-01-saturation": {
"assignments": [
{
"cell": "P08-C00",
"gpus": [
0
]
},
{
"cell": "P03-C10",
"gpus": [
1
]
},
{
"cell": "P07-C00",
"gpus": [
2
]
},
{
"cell": "P10-C01",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783836130.8498049,
"confirmation": false,
"drain_readjudication": "P10/C01 288.619107924s <= 600s",
"gpu_hours": 0.8836704309119119,
"load_point": "saturation",
"missing_trace_files": 8,
"profile": true,
"servers": {},
"started_at": 1783834671.9563339,
"status": "complete",
"validation_attempt1_failure": "RuntimeError(\"client failures: {'P10-C01-saturation': 1}\")"
},
"primary-02-moderate": {
"assignments": [
{
"cell": "P01-C10",
"gpus": [
0
]
},
{
"cell": "P01-C01",
"gpus": [
1
]
},
{
"cell": "P10-C10",
"gpus": [
2
]
},
{
"cell": "P03-C01",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783847092.7229948,
"confirmation": false,
"failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P01-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': False, 'manifest_no_wrap': True, 'warmup_completions': True, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")",
"gpu_hours": 0.5151156750652525,
"load_point": "moderate",
"profile": true,
"readjudicated_without_gpu": true,
"servers": {},
"started_at": 1783838513.4472992,
"status": "complete",
"validation_attempt1_failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P01-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': False, 'manifest_no_wrap': True, 'warmup_completions': True, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")"
},
"primary-02-saturation": {
"assignments": [
{
"cell": "P01-C10",
"gpus": [
0
]
},
{
"cell": "P01-C01",
"gpus": [
1
]
},
{
"cell": "P10-C10",
"gpus": [
2
]
},
{
"cell": "P03-C01",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783838513.4027848,
"confirmation": false,
"gpu_hours": 0.7973502041896184,
"load_point": "saturation",
"profile": true,
"servers": {},
"started_at": 1783837773.6088765,
"status": "complete"
},
"primary-03-moderate": {
"assignments": [
{
"cell": "P09-C00",
"gpus": [
0
]
},
{
"cell": "P06-C01",
"gpus": [
1
]
},
{
"cell": "P10-C11",
"gpus": [
2
]
},
{
"cell": "P01-C00",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783848369.3507695,
"confirmation": false,
"gpu_hours": 0.5205460974905226,
"load_point": "moderate",
"profile": true,
"servers": {},
"started_at": 1783847881.06146,
"status": "complete"
},
"primary-03-saturation": {
"assignments": [
{
"cell": "P09-C00",
"gpus": [
0
]
},
{
"cell": "P06-C01",
"gpus": [
1
]
},
{
"cell": "P10-C11",
"gpus": [
2
]
},
{
"cell": "P01-C00",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783847881.0197804,
"confirmation": false,
"gpu_hours": 0.8273645816908942,
"load_point": "saturation",
"profile": true,
"servers": {},
"started_at": 1783847109.5683346,
"status": "complete"
},
"primary-04-moderate": {
"assignments": [
{
"cell": "P01-C11",
"gpus": [
0
]
},
{
"cell": "P10-C00",
"gpus": [
1
]
},
{
"cell": "P04-C00",
"gpus": [
2
]
},
{
"cell": "P03-C00",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783849583.5228572,
"confirmation": false,
"gpu_hours": 0.5205216948853598,
"load_point": "moderate",
"profile": true,
"servers": {},
"started_at": 1783849096.4363403,
"status": "complete"
},
"primary-04-saturation": {
"assignments": [
{
"cell": "P01-C11",
"gpus": [
0
]
},
{
"cell": "P10-C00",
"gpus": [
1
]
},
{
"cell": "P04-C00",
"gpus": [
2
]
},
{
"cell": "P03-C00",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783849096.3948114,
"confirmation": false,
"gpu_hours": 0.7872003297011058,
"load_point": "saturation",
"profile": true,
"servers": {},
"started_at": 1783848369.390857,
"status": "complete"
},
"primary-05-moderate": {
"assignments": [
{
"cell": "P06-C10",
"gpus": [
0
]
},
{
"cell": "P02-C00",
"gpus": [
1
]
},
{
"cell": "P06-C00",
"gpus": [
2
]
},
{
"cell": "P06-C11",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783850641.6781306,
"confirmation": false,
"gpu_hours": 0.5138198028008143,
"load_point": "moderate",
"profile": true,
"servers": {},
"started_at": 1783850163.1211865,
"status": "complete"
},
"primary-05-saturation": {
"assignments": [
{
"cell": "P06-C10",
"gpus": [
0
]
},
{
"cell": "P02-C00",
"gpus": [
1
]
},
{
"cell": "P06-C00",
"gpus": [
2
]
},
{
"cell": "P06-C11",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {},
"completed_at": 1783850163.0472832,
"confirmation": false,
"gpu_hours": 0.6229251067505942,
"load_point": "saturation",
"profile": true,
"servers": {},
"started_at": 1783849583.5652869,
"status": "complete"
},
"primary-06-saturation": {
"assignments": [
{
"cell": "P11-C00",
"gpus": [
0
]
},
{
"cell": "P10-C00-TP2",
"gpus": [
1,
2
]
},
{
"cell": "P03-C11",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {
"P03-C11-saturation": {
"pgid": 2440965,
"pid": 2440965
},
"P10-C00-TP2-saturation": {
"pgid": 2440964,
"pid": 2440964
},
"P11-C00-saturation": {
"pgid": 2440962,
"pid": 2440962
}
},
"confirmation": false,
"failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P10-C00-TP2/saturation: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'failed_records_accounted': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]; warmup_completions=17; warmup_gate_branch=failed; warmup_stability={'passed': False, 'reason': 'A-P3-6 stabilization criterion not met', 'window_seconds': [45.0, 60.0], 'bin_seconds': 5.0, 'step_counts': [11, 10, 16], 'scheduled_tokens': [90112, 81920, 70313], 'scheduled_token_throughput': [18022.4, 16384.0, 14062.6], 'mean_scheduled_token_throughput': 16156.333333333334, 'slope_tokens_per_second_squared': -395.98000000000013, 'normalized_drift': 0.367639109533929, 'normalized_drift_limit': 0.1, 'step_indices_continuous': True}\")",
"gpu_hours": 0.7173924536837472,
"load_point": "saturation",
"profile": true,
"servers": {
"P03-C11-saturation": {
"gpus": [
3
],
"pgid": 2438868,
"pid": 2438868
},
"P10-C00-TP2-saturation": {
"gpus": [
1,
2
],
"pgid": 2438867,
"pid": 2438867
},
"P11-C00-saturation": {
"gpus": [
0
],
"pgid": 2438866,
"pid": 2438866
}
},
"started_at": 1783853279.9695158,
"status": "failed"
}
},
"status": "failed",
"updated_at": 1783853946.066117
}

39015
runs/opprof-phase3/phase3/metrics.json Normal file
View File

File diff suppressed because it is too large Load Diff

103
runs/opprof-phase3/phase3/remote-evidence/P03-C10/result.json Normal file
View File

@@ -0,0 +1,103 @@
{
"admission_stop_s": 417.3262625900097,
"arrival": "steady",
"clean": {
"admitted": 559,
"completed": 559,
"completed_throughput_rps": 2.3291666666666666,
"duration_s": 240.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 3429894,
"offered_rps": 2.3291666666666666,
"output_tokens": 35776,
"start_s": 60.0
},
"clean_segment_seconds": 80.0,
"drain_seconds": 91.29178713698639,
"elapsed_seconds": 508.6180497269961,
"failed_records": 0,
"load_point": "saturation",
"manifest_admitted": 1094,
"manifest_exhausted": false,
"manifest_rows": 32768,
"manifest_sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1",
"manifest_wrapped": false,
"max_in_flight": 256,
"num_clean_segments": 3,
"profiles": [
{
"start_call_s": 300.0043962339987,
"start_return_s": 301.3819164079905,
"start_status": 200,
"stop_call_s": 321.8697677869932,
"stop_return_s": 326.31530767300865,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835059474707213.pt.trace.json.gz",
"trace_ready_s": 321.86976291501196,
"trace_sha256": "527ddb81b1d54a12a463508bc163d85a2c443d94c8a897959a39735e030b90a2",
"window": 1
},
{
"start_call_s": 356.32926020701416,
"start_return_s": 361.34978177401354,
"start_status": 200,
"stop_call_s": 381.577829301008,
"stop_return_s": 387.31277802900877,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835119198018909.pt.trace.json.gz",
"trace_ready_s": 381.57782580400817,
"trace_sha256": "0882d9705f1208580d5e693f94ba64d89ab12ef0aae2731a82d460a46a3e4cb1",
"window": 2
}
],
"rate_fraction": null,
"records": 1094,
"request_rate": "inf",
"schema": 1,
"segments": [
{
"admitted": 191,
"completed": 191,
"completed_throughput_rps": 2.3875,
"duration_s": 80.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 1154628,
"name": "A",
"offered_rps": 2.3875,
"output_tokens": 12224,
"start_s": 60.0
},
{
"admitted": 184,
"completed": 184,
"completed_throughput_rps": 2.3,
"duration_s": 80.0,
"end_s": 220.0,
"failed": 0,
"input_tokens": 1135217,
"name": "B",
"offered_rps": 2.3,
"output_tokens": 11776,
"start_s": 140.0
},
{
"admitted": 184,
"completed": 184,
"completed_throughput_rps": 2.3,
"duration_s": 80.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 1140049,
"name": "C",
"offered_rps": 2.3,
"output_tokens": 11776,
"start_s": 220.0
}
],
"successful_records": 1094,
"t0_mono_ns": 166323993366724,
"t0_wall_ns": 1783834751119796663,
"warmup_seconds": 60.0
}

65
runs/opprof-phase3/phase3/remote-evidence/P03-C10/sanity.json Normal file
View File

@@ -0,0 +1,65 @@
{
"invariants": {
"clean_duration_exact": true,
"clean_failures_zero": true,
"concurrency_bounded": true,
"drain_within_timeout": true,
"manifest_no_wrap": true,
"manifest_not_exhausted": true,
"output_tokens_exact": true,
"profile_count_exact": true,
"profile_status_ok": true,
"segment_count_exact": true
},
"numeric": {
"actual_output_tokens": {
"distinct_n": 1,
"finite_n": 1094,
"max": 64.0,
"min": 64.0,
"missing_n": 0,
"n": 1094
},
"admitted_s": {
"distinct_n": 1094,
"finite_n": 1094,
"max": 416.7779763180006,
"min": 0.0008771540015004575,
"missing_n": 0,
"n": 1094
},
"completed_s": {
"distinct_n": 1094,
"finite_n": 1094,
"max": 508.61568894199445,
"min": 22.18080371801625,
"missing_n": 0,
"n": 1094
},
"input_tokens": {
"distinct_n": 964,
"finite_n": 1094,
"max": 8190.0,
"min": 4097.0,
"missing_n": 0,
"n": 1094
},
"requested_output_tokens": {
"distinct_n": 1,
"finite_n": 1094,
"max": 64.0,
"min": 64.0,
"missing_n": 0,
"n": 1094
},
"scheduled_s": {
"distinct_n": 1094,
"finite_n": 1094,
"max": 416.7779758319957,
"min": 0.0008756940078455955,
"missing_n": 0,
"n": 1094
}
},
"schema": 1
}

103
runs/opprof-phase3/phase3/remote-evidence/P07-C00/result.json Normal file
View File

@@ -0,0 +1,103 @@
{
"admission_stop_s": 417.66287675101194,
"arrival": "burst:8",
"clean": {
"admitted": 1226,
"completed": 1226,
"completed_throughput_rps": 5.108333333333333,
"duration_s": 240.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 1569280,
"offered_rps": 5.108333333333333,
"output_tokens": 627712,
"start_s": 60.0
},
"clean_segment_seconds": 80.0,
"drain_seconds": 26.098238860984566,
"elapsed_seconds": 443.7611156119965,
"failed_records": 0,
"load_point": "saturation",
"manifest_admitted": 2078,
"manifest_exhausted": false,
"manifest_rows": 32768,
"manifest_sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6",
"manifest_wrapped": false,
"max_in_flight": 256,
"num_clean_segments": 3,
"profiles": [
{
"start_call_s": 300.0039349270228,
"start_return_s": 300.9924320260179,
"start_status": 200,
"stop_call_s": 323.90002606701455,
"stop_return_s": 328.078528626007,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835058771060551.pt.trace.json.gz",
"trace_ready_s": 323.9000201699964,
"trace_sha256": "2d12038b03c6f9c28d1a488abd880ff2763e1f3c0ab502cfdd2507b649de2af7",
"window": 1
},
{
"start_call_s": 358.0940292410087,
"start_return_s": 358.8905565890018,
"start_status": 200,
"stop_call_s": 382.04741394601297,
"stop_return_s": 387.64796055300394,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835116964699401.pt.trace.json.gz",
"trace_ready_s": 382.04741024502437,
"trace_sha256": "2184d4f83ea22c0e5455dd3914922548ce8d3d5eba34f89562df69326e3832b5",
"window": 2
}
],
"rate_fraction": null,
"records": 2078,
"request_rate": "inf",
"schema": 1,
"segments": [
{
"admitted": 394,
"completed": 394,
"completed_throughput_rps": 4.925,
"duration_s": 80.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 504320,
"name": "A",
"offered_rps": 4.925,
"output_tokens": 201728,
"start_s": 60.0
},
{
"admitted": 397,
"completed": 397,
"completed_throughput_rps": 4.9625,
"duration_s": 80.0,
"end_s": 220.0,
"failed": 0,
"input_tokens": 508160,
"name": "B",
"offered_rps": 4.9625,
"output_tokens": 203264,
"start_s": 140.0
},
{
"admitted": 435,
"completed": 435,
"completed_throughput_rps": 5.4375,
"duration_s": 80.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 556800,
"name": "C",
"offered_rps": 5.4375,
"output_tokens": 222720,
"start_s": 220.0
}
],
"successful_records": 2078,
"t0_mono_ns": 166323999932716,
"t0_wall_ns": 1783834751126361482,
"warmup_seconds": 60.0
}

65
runs/opprof-phase3/phase3/remote-evidence/P07-C00/sanity.json Normal file
View File

@@ -0,0 +1,65 @@
{
"invariants": {
"clean_duration_exact": true,
"clean_failures_zero": true,
"concurrency_bounded": true,
"drain_within_timeout": true,
"manifest_no_wrap": true,
"manifest_not_exhausted": true,
"output_tokens_exact": true,
"profile_count_exact": true,
"profile_status_ok": true,
"segment_count_exact": true
},
"numeric": {
"actual_output_tokens": {
"distinct_n": 1,
"finite_n": 2078,
"max": 512.0,
"min": 512.0,
"missing_n": 0,
"n": 2078
},
"admitted_s": {
"distinct_n": 2078,
"finite_n": 2078,
"max": 417.40899016702315,
"min": 0.0008179180149454623,
"missing_n": 0,
"n": 2078
},
"completed_s": {
"distinct_n": 2078,
"finite_n": 2078,
"max": 443.75592052700813,
"min": 35.15151289102505,
"missing_n": 0,
"n": 2078
},
"input_tokens": {
"distinct_n": 1,
"finite_n": 2078,
"max": 1280.0,
"min": 1280.0,
"missing_n": 0,
"n": 2078
},
"requested_output_tokens": {
"distinct_n": 1,
"finite_n": 2078,
"max": 512.0,
"min": 512.0,
"missing_n": 0,
"n": 2078
},
"scheduled_s": {
"distinct_n": 2078,
"finite_n": 2078,
"max": 417.4089898300008,
"min": 0.0008165129984263331,
"missing_n": 0,
"n": 2078
}
},
"schema": 1
}

103
runs/opprof-phase3/phase3/remote-evidence/P08-C00/result.json Normal file
View File

@@ -0,0 +1,103 @@
{
"admission_stop_s": 493.2576180040196,
"arrival": "burst:8",
"clean": {
"admitted": 2233,
"completed": 2233,
"completed_throughput_rps": 9.304166666666667,
"duration_s": 240.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 2858240,
"offered_rps": 9.304166666666667,
"output_tokens": 1143296,
"start_s": 60.0
},
"clean_segment_seconds": 80.0,
"drain_seconds": 14.923095890000695,
"elapsed_seconds": 508.1807138940203,
"failed_records": 0,
"load_point": "saturation",
"manifest_admitted": 3941,
"manifest_exhausted": false,
"manifest_rows": 32768,
"manifest_sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9",
"manifest_wrapped": false,
"max_in_flight": 256,
"num_clean_segments": 3,
"profiles": [
{
"start_call_s": 300.0054247789958,
"start_return_s": 323.45228312400286,
"start_status": 200,
"stop_call_s": 354.6066670610162,
"stop_return_s": 381.28106322701205,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835082061801278.pt.trace.json.gz",
"trace_ready_s": 354.6066622030048,
"trace_sha256": "53d36238e82e5d031d8ed7841094c56d715b7cd5009368b64e380bf89a0f0dec",
"window": 1
},
{
"start_call_s": 411.30330635100836,
"start_return_s": 411.4401313569979,
"start_status": 200,
"stop_call_s": 436.0742275560042,
"stop_return_s": 463.23921855099616,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835169024306574.pt.trace.json.gz",
"trace_ready_s": 436.0742245099973,
"trace_sha256": "e088bd800a4e43446effbbcd4fce7d9ba1defaceb378a6e0db04d64780e1d09a",
"window": 2
}
],
"rate_fraction": null,
"records": 3941,
"request_rate": "inf",
"schema": 1,
"segments": [
{
"admitted": 697,
"completed": 697,
"completed_throughput_rps": 8.7125,
"duration_s": 80.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 892160,
"name": "A",
"offered_rps": 8.7125,
"output_tokens": 356864,
"start_s": 60.0
},
{
"admitted": 768,
"completed": 768,
"completed_throughput_rps": 9.6,
"duration_s": 80.0,
"end_s": 220.0,
"failed": 0,
"input_tokens": 983040,
"name": "B",
"offered_rps": 9.6,
"output_tokens": 393216,
"start_s": 140.0
},
{
"admitted": 768,
"completed": 768,
"completed_throughput_rps": 9.6,
"duration_s": 80.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 983040,
"name": "C",
"offered_rps": 9.6,
"output_tokens": 393216,
"start_s": 220.0
}
],
"successful_records": 3941,
"t0_mono_ns": 166324009670273,
"t0_wall_ns": 1783834751136099481,
"warmup_seconds": 60.0
}

65
runs/opprof-phase3/phase3/remote-evidence/P08-C00/sanity.json Normal file
View File

@@ -0,0 +1,65 @@
{
"invariants": {
"clean_duration_exact": true,
"clean_failures_zero": true,
"concurrency_bounded": true,
"drain_within_timeout": true,
"manifest_no_wrap": true,
"manifest_not_exhausted": true,
"output_tokens_exact": true,
"profile_count_exact": true,
"profile_status_ok": true,
"segment_count_exact": true
},
"numeric": {
"actual_output_tokens": {
"distinct_n": 1,
"finite_n": 3941,
"max": 512.0,
"min": 512.0,
"missing_n": 0,
"n": 3941
},
"admitted_s": {
"distinct_n": 3941,
"finite_n": 3941,
"max": 493.25117621201207,
"min": 0.0009094980196096003,
"missing_n": 0,
"n": 3941
},
"completed_s": {
"distinct_n": 3941,
"finite_n": 3941,
"max": 508.1755032700021,
"min": 30.030000179016497,
"missing_n": 0,
"n": 3941
},
"input_tokens": {
"distinct_n": 1,
"finite_n": 3941,
"max": 1280.0,
"min": 1280.0,
"missing_n": 0,
"n": 3941
},
"requested_output_tokens": {
"distinct_n": 1,
"finite_n": 3941,
"max": 512.0,
"min": 512.0,
"missing_n": 0,
"n": 3941
},
"scheduled_s": {
"distinct_n": 3941,
"finite_n": 3941,
"max": 493.25117589501315,
"min": 0.000907983019715175,
"missing_n": 0,
"n": 3941
}
},
"schema": 1
}

103
runs/opprof-phase3/phase3/remote-evidence/P10-C01/result.json Normal file
View File

@@ -0,0 +1,103 @@
{
"admission_stop_s": 420.6437278209778,
"arrival": "steady",
"clean": {
"admitted": 178,
"completed": 178,
"completed_throughput_rps": 0.7416666666666667,
"duration_s": 240.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 1755158,
"offered_rps": 0.7416666666666667,
"output_tokens": 40658,
"start_s": 60.0
},
"clean_segment_seconds": 80.0,
"drain_seconds": 288.6191079240234,
"elapsed_seconds": 709.2628357450012,
"failed_records": 0,
"load_point": "saturation",
"manifest_admitted": 558,
"manifest_exhausted": false,
"manifest_rows": 4011,
"manifest_sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33",
"manifest_wrapped": false,
"max_in_flight": 256,
"num_clean_segments": 3,
"profiles": [
{
"start_call_s": 300.0035294489935,
"start_return_s": 310.3520467719936,
"start_status": 200,
"stop_call_s": 325.0655262139917,
"stop_return_s": 338.54296391498065,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835066783986388.pt.trace.json.gz",
"trace_ready_s": 325.0655210709956,
"trace_sha256": "d457af416fa91f44e617c7259167fca467bea8466a3e4c4b916f811fb5899c73",
"window": 1
},
{
"start_call_s": 368.55468072599615,
"start_return_s": 372.31818850699347,
"start_status": 200,
"stop_call_s": 386.0271748309897,
"stop_return_s": 390.6334540609969,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835128341345321.pt.trace.json.gz",
"trace_ready_s": 386.02717122397735,
"trace_sha256": "41e8b34a0464cfd92d16b21c6242113b23594081697f5d1e88cdc3647542a96a",
"window": 2
}
],
"rate_fraction": null,
"records": 558,
"request_rate": "inf",
"schema": 1,
"segments": [
{
"admitted": 48,
"completed": 48,
"completed_throughput_rps": 0.6,
"duration_s": 80.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 548329,
"name": "A",
"offered_rps": 0.6,
"output_tokens": 11027,
"start_s": 60.0
},
{
"admitted": 70,
"completed": 70,
"completed_throughput_rps": 0.875,
"duration_s": 80.0,
"end_s": 220.0,
"failed": 0,
"input_tokens": 574251,
"name": "B",
"offered_rps": 0.875,
"output_tokens": 16298,
"start_s": 140.0
},
{
"admitted": 60,
"completed": 60,
"completed_throughput_rps": 0.75,
"duration_s": 80.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 632578,
"name": "C",
"offered_rps": 0.75,
"output_tokens": 13333,
"start_s": 220.0
}
],
"successful_records": 558,
"t0_mono_ns": 166325149689000,
"t0_wall_ns": 1783834752276118250,
"warmup_seconds": 60.0
}

65
runs/opprof-phase3/phase3/remote-evidence/P10-C01/sanity.json Normal file
View File

@@ -0,0 +1,65 @@
{
"invariants": {
"clean_duration_exact": true,
"clean_failures_zero": true,
"concurrency_bounded": true,
"drain_within_timeout": false,
"manifest_no_wrap": true,
"manifest_not_exhausted": true,
"output_tokens_exact": true,
"profile_count_exact": true,
"profile_status_ok": true,
"segment_count_exact": true
},
"numeric": {
"actual_output_tokens": {
"distinct_n": 92,
"finite_n": 558,
"max": 256.0,
"min": 6.0,
"missing_n": 0,
"n": 558
},
"admitted_s": {
"distinct_n": 558,
"finite_n": 558,
"max": 420.08466253298684,
"min": 0.000850101001560688,
"missing_n": 0,
"n": 558
},
"completed_s": {
"distinct_n": 558,
"finite_n": 558,
"max": 709.2613093179825,
"min": 14.001143716974184,
"missing_n": 0,
"n": 558
},
"input_tokens": {
"distinct_n": 535,
"finite_n": 558,
"max": 32527.0,
"min": 70.0,
"missing_n": 0,
"n": 558
},
"requested_output_tokens": {
"distinct_n": 92,
"finite_n": 558,
"max": 256.0,
"min": 6.0,
"missing_n": 0,
"n": 558
},
"scheduled_s": {
"distinct_n": 558,
"finite_n": 558,
"max": 420.08466206499725,
"min": 0.0008485929865855724,
"missing_n": 0,
"n": 558
}
},
"schema": 1
}

258
runs/opprof-phase3/phase3/remote-evidence/controller-state.json Normal file
View File

@@ -0,0 +1,258 @@
{
"completed_burnins": 5,
"completed_measured_runs": 0,
"controller_pid": 2187729,
"created_at": 1783833885.960389,
"fingerprint": {
"cells": [
"P08-C00",
"P03-C10",
"P07-C00",
"P10-C01",
"P01-C10",
"P01-C01",
"P10-C10",
"P03-C01",
"P09-C00",
"P06-C01",
"P10-C11",
"P01-C00",
"P01-C11",
"P10-C00",
"P04-C00",
"P03-C00",
"P06-C10",
"P02-C00",
"P06-C00",
"P06-C11",
"P11-C00",
"P10-C00-TP2",
"P03-C11",
"P05-C00"
],
"client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84",
"common_controller_sha256": "15ad254298a38c4a9318468db89ab32707b6196bb38d5e2a007f2267529397a5",
"controller_sha256": "c2d3232fb99c66cea55e30e6cb1aa6a84d3a816434ed8155a309f41df2837f73",
"cpu_map": {
"0": "0-19",
"1": "20-39",
"2": "40-59",
"3": "60-79",
"4": "80-99",
"5": "100-119",
"6": "120-139",
"7": "140-159"
},
"manifests": {
"P01": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl",
"rows": 32768,
"sha256": "13ffb226c83373f54c4a7afea6c78cb7cd29720f1858d56728826fc1367b31a4"
},
"P02": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P02.jsonl",
"rows": 32768,
"sha256": "0138ada3fccc98298daee66c26bd1952c987cb42ed8d5341d66b698a597417f9"
},
"P03": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P03.jsonl",
"rows": 32768,
"sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1"
},
"P04": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P04.jsonl",
"rows": 32768,
"sha256": "caf8d0941b093956a81e1413adc4a4ea9d92460f1b2ed0f6a9b118d0119d6247"
},
"P05": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P05.jsonl",
"rows": 32768,
"sha256": "192e213109f8cb429b99d9eb0f227bb4390fc03f63b02d5456569369bff5a3d7"
},
"P06": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P06.jsonl",
"rows": 32768,
"sha256": "65954bc6e47de9e7be07b8975f97f0bc4639979ef6d33e8c664557ded34b9f96"
},
"P07": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P07.jsonl",
"rows": 32768,
"sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6"
},
"P08": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P08.jsonl",
"rows": 32768,
"sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9"
},
"P09": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
"rows": 32768,
"sha256": "7af92ee3c27dc7d2cf895d6ff3a6e737ec4b6da13d6841ca59e1166f28a0ae1e"
},
"P10": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl",
"rows": 4011,
"sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33"
},
"P11": {
"path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P11.jsonl",
"rows": 32768,
"sha256": "7d196df38963528ff181cf72ce39c8ad913c8f61d40b1425410d3c6c30b6be18"
}
},
"model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B",
"source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
"source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1"
},
"gpu_hours_this_stage": 0.8836704309119119,
"gpu_hours_total": 5.473521912336349,
"schema": 1,
"stages": {
"burnin-01": {
"assignments": [
{
"cell": "P06-C00",
"gpus": [
0
]
},
{
"cell": "P06-C10",
"gpus": [
1
]
},
{
"cell": "P06-C01",
"gpus": [
2
]
},
{
"cell": "P06-C11",
"gpus": [
3
]
}
],
"burnin": true,
"clients": {},
"completed_at": 1783834288.1238313,
"confirmation": false,
"gpu_hours": 0.30808498481909435,
"load_point": "saturation",
"profile": false,
"servers": {},
"started_at": 1783833886.4448946,
"status": "complete",
"validation_attempt1_failure": "RuntimeError(\"server config/backend failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/burnins/C01: {'triton_moe': True, 'chunked_mbt': False, 'tp_effective': True, 'drain_shutdown': True, 'mns_effective': True}\")"
},
"burnin-02": {
"assignments": [
{
"cell": "P06-C00-TP2",
"gpus": [
0,
1
]
}
],
"burnin": true,
"clients": {},
"completed_at": 1783834671.921921,
"confirmation": false,
"gpu_hours": 0.19903395679261948,
"load_point": "saturation",
"profile": false,
"servers": {},
"started_at": 1783834301.7813473,
"status": "complete"
},
"primary-01-saturation": {
"assignments": [
{
"cell": "P08-C00",
"gpus": [
0
]
},
{
"cell": "P03-C10",
"gpus": [
1
]
},
{
"cell": "P07-C00",
"gpus": [
2
]
},
{
"cell": "P10-C01",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {
"P03-C10-saturation": {
"pgid": 2195599,
"pid": 2195599
},
"P07-C00-saturation": {
"pgid": 2195600,
"pid": 2195600
},
"P08-C00-saturation": {
"pgid": 2195597,
"pid": 2195597
},
"P10-C01-saturation": {
"pgid": 2195601,
"pid": 2195601
}
},
"confirmation": false,
"failure": "RuntimeError(\"client failures: {'P10-C01-saturation': 1}\")",
"gpu_hours": 0.8836704309119119,
"load_point": "saturation",
"profile": true,
"servers": {
"P03-C10-saturation": {
"gpus": [
1
],
"pgid": 2193680,
"pid": 2193680
},
"P07-C00-saturation": {
"gpus": [
2
],
"pgid": 2193681,
"pid": 2193681
},
"P08-C00-saturation": {
"gpus": [
0
],
"pgid": 2193679,
"pid": 2193679
},
"P10-C01-saturation": {
"gpus": [
3
],
"pgid": 2193682,
"pid": 2193682
}
},
"started_at": 1783834671.9563339,
"status": "failed"
}
},
"status": "failed",
"updated_at": 1783835479.1651883
}

16
runs/opprof-phase3/phase3/remote-evidence/repair-001-explicit-mbt-log.json Normal file
View File

@@ -0,0 +1,16 @@
{
"at": 1783834288.13871,
"finding": "Explicit MBT=2048 is logged in non-default args; scheduler.py emits the separate Chunked-prefill line only for default MBT.",
"gpu_rerun": false,
"kind": "validator-only",
"new_controller_sha256": "c2d3232fb99c66cea55e30e6cb1aa6a84d3a816434ed8155a309f41df2837f73",
"old_controller_sha256": "74859614a8da3341bf827bb75b667db7d219f059d1d29c1182b9a77c2ee59e92",
"protocol_or_workload_changed": false,
"revalidated_runs": [
"P06-C00-burnin",
"P06-C10-burnin",
"P06-C01-burnin",
"P06-C11-burnin"
],
"schema": 1
}

355
runs/opprof-phase3/phase3/remote-evidence/sidecar-verification-state.json Normal file
View File

@@ -0,0 +1,355 @@
{
"gpu_seconds": 427.43714332580566,
"results": {
"graceful": {
"accounting": {
"bytes": 1462585,
"checkpoint_age_seconds": -0.382308841,
"counters": {
"dropped_records": 0,
"encoded_records": 1598,
"written_records": 1598
},
"footer_count": 1,
"invariants": {
"all_schema_1": true,
"encoded_balanced": true,
"final_sidecar": true,
"footer_sidecar_agree": true,
"last_step_matches": true,
"one_footer_last": true,
"steps_contiguous": true,
"written_matches_records": true,
"zero_drops": true
},
"last_step_index": 1597,
"records": 1598,
"sidecar": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/graceful/opprof/opprof-v1-dp0-pid2163417-1783832631507578426.jsonl.footer.json",
"stream": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/graceful/opprof/opprof-v1-dp0-pid2163417-1783832631507578426.jsonl"
},
"clean": {
"admitted": 5202,
"completed": 5202,
"completed_throughput_rps": 43.35,
"duration_s": 120.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 1662118,
"offered_rps": 43.35,
"output_tokens": 332928,
"start_s": 20.0
},
"failed_records": 0,
"gpu_seconds": 219.92383646965027,
"gpu_zero_samples": [
[
{
"index": 0,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 1,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 2,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 3,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 4,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 5,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 6,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 7,
"memory_mib": 0,
"utilization_pct": 0
}
],
[
{
"index": 0,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 1,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 2,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 3,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 4,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 5,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 6,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 7,
"memory_mib": 0,
"utilization_pct": 0
}
],
[
{
"index": 0,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 1,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 2,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 3,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 4,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 5,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 6,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 7,
"memory_mib": 0,
"utilization_pct": 0
}
]
],
"mode": "graceful",
"schema": 1,
"server_returncode": 0,
"status": "pass",
"termination_wall_ns": 1783832775735239240
},
"hard-kill": {
"accounting": {
"bytes": 1460860,
"checkpoint_age_seconds": 0.02436319,
"counters": {
"dropped_records": 0,
"encoded_records": 1596,
"written_records": 1596
},
"footer_count": 0,
"invariants": {
"all_schema_1": true,
"checkpoint_sidecar": true,
"checkpoint_within_bound": true,
"encoded_balanced": true,
"last_step_matches": true,
"no_in_stream_footer": true,
"steps_contiguous": true,
"written_matches_records": true,
"zero_drops": true
},
"last_step_index": 1595,
"records": 1596,
"sidecar": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/hard-kill/opprof/opprof-v1-dp0-pid2166360-1783832841962667333.jsonl.footer.json",
"stream": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/hard-kill/opprof/opprof-v1-dp0-pid2166360-1783832841962667333.jsonl"
},
"clean": {
"admitted": 5179,
"completed": 5179,
"completed_throughput_rps": 43.15833333333333,
"duration_s": 120.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 1655384,
"offered_rps": 43.15833333333333,
"output_tokens": 331456,
"start_s": 20.0
},
"failed_records": 0,
"gpu_seconds": 207.5133068561554,
"gpu_zero_samples": [
[
{
"index": 0,
"memory_mib": 0,
"utilization_pct": 1
},
{
"index": 1,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 2,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 3,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 4,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 5,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 6,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 7,
"memory_mib": 0,
"utilization_pct": 0
}
],
[
{
"index": 0,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 1,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 2,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 3,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 4,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 5,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 6,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 7,
"memory_mib": 0,
"utilization_pct": 0
}
],
[
{
"index": 0,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 1,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 2,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 3,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 4,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 5,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 6,
"memory_mib": 0,
"utilization_pct": 0
},
{
"index": 7,
"memory_mib": 0,
"utilization_pct": 0
}
]
],
"mode": "hard-kill",
"schema": 1,
"server_returncode": -9,
"status": "pass",
"termination_wall_ns": 1783832986410202706
}
},
"schema": 1,
"status": "complete"
}

17
runs/opprof-phase3/phase3/repair-008-ap36.json Normal file
View File

@@ -0,0 +1,17 @@
{
"accepted_runs_preserved": 40,
"amendment": "A-P3-6",
"changed_fingerprint_keys": [
"controller_sha256"
],
"created_at": 1783853258.3098204,
"failed_stage_preserved": "primary-06-saturation",
"new_controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
"old_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f",
"projected_remaining_h20_hours": 2.1,
"projected_total_h20_hours": 15.408482965071997,
"repair_id": "repair-008-ap36",
"retained_attempt_re_adjudicated": false,
"retained_attempt_reason": "21 completions but A-P3-6 normalized drift 2.3385345997286295 and bin step counts 13/9/44",
"schema": 1
}

469
runs/opprof-phase3/phase4/capture-p09/controller-state.json Normal file
View File

@@ -0,0 +1,469 @@
{
"arms": {
"OFF": {
"client_pid": 2487398,
"server_pid": 2486295,
"started_at": 1783856664.5207698,
"status": "complete",
"summary": {
"arm": "OFF",
"bucket_tokens": 237911,
"capture_sizes": "default",
"clean_completed": 1184,
"clean_completed_throughput_rps": 4.933333333333334,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7853007119847462,
"e2e_latency_mean_s": 1.681152764688729,
"e2e_latency_p95_s": 3.8763178953449815,
"gpu_hours": 0.13751606033907995,
"graph_hit_steps": 12252,
"graph_miss_rate": 0.045199501246882795,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 16491,
"model_step_duration_ms": 479367.749483,
"model_steps": 12832,
"padding_fraction": 0.08545632610514016,
"padding_tokens": 20331,
"schema": 1,
"token_efficiency_per_ms": 4.5540506267984115,
"useful_tokens": 2183065
}
},
"ON": {
"client_pid": 2479800,
"server_pid": 2477652,
"started_at": 1783856086.549906,
"status": "complete",
"summary": {
"arm": "ON",
"bucket_tokens": 227714,
"capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"clean_completed": 1190,
"clean_completed_throughput_rps": 4.958333333333333,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7876331160077825,
"e2e_latency_mean_s": 1.612280680370388,
"e2e_latency_p95_s": 3.9930475192406423,
"gpu_hours": 0.15887338949574364,
"graph_hit_steps": 14253,
"graph_miss_rate": 0.0389724226282786,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 17579,
"model_step_duration_ms": 478655.543996,
"model_steps": 14831,
"padding_fraction": 0.035658764941988635,
"padding_tokens": 8120,
"schema": 1,
"token_efficiency_per_ms": 4.562222306607711,
"useful_tokens": 2183733
}
}
},
"controller_pid": 2477456,
"created_at": 1783856081.3481774,
"gpu_hours_increment": 0.2963894498348236,
"plan": {
"added_capture_sizes": [
3,
5,
6,
7,
9
],
"clean_seconds": 240,
"gpu": 0,
"gpu_hour_limit": 16.0,
"load": "moderate",
"manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
"measured_padding_recovery_bound": 0.049776380388728225,
"on_capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"order": [
"ON",
"OFF"
],
"pattern": "P09",
"primary_metric": "clean graph-hit padding_fraction",
"prior_gpu_hours": 14.025875418755744,
"profile": false,
"projected_increment_gpu_hours": 0.5,
"projected_total_gpu_hours": 14.525875418755744,
"rate_fraction": 0.6,
"saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json",
"schema": 1,
"secondary_metrics": [
"Layer-1 useful scheduled tokens/model-step millisecond",
"clean completed request throughput",
"clean request latency"
],
"warmup_seconds": 60
},
"result": {
"delta": {
"completed_throughput_relative": 0.0050675675675675436,
"e2e_mean_latency_relative": -0.04096717785851722,
"e2e_p95_latency_relative": 0.03011353223527924,
"padding_fraction_points": -0.049797561163151524,
"padding_reduction_fraction": 0.5827252753866776,
"token_efficiency_relative": 0.0017943761453185214
},
"gpu_hours_increment": 0.2963894498348236,
"gpu_hours_total": 14.322264868590567,
"off": {
"arm": "OFF",
"bucket_tokens": 237911,
"capture_sizes": "default",
"clean_completed": 1184,
"clean_completed_throughput_rps": 4.933333333333334,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7853007119847462,
"e2e_latency_mean_s": 1.681152764688729,
"e2e_latency_p95_s": 3.8763178953449815,
"gpu_hours": 0.13751606033907995,
"graph_hit_steps": 12252,
"graph_miss_rate": 0.045199501246882795,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 16491,
"model_step_duration_ms": 479367.749483,
"model_steps": 12832,
"padding_fraction": 0.08545632610514016,
"padding_tokens": 20331,
"schema": 1,
"token_efficiency_per_ms": 4.5540506267984115,
"useful_tokens": 2183065
},
"on": {
"arm": "ON",
"bucket_tokens": 227714,
"capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"clean_completed": 1190,
"clean_completed_throughput_rps": 4.958333333333333,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7876331160077825,
"e2e_latency_mean_s": 1.612280680370388,
"e2e_latency_p95_s": 3.9930475192406423,
"gpu_hours": 0.15887338949574364,
"graph_hit_steps": 14253,
"graph_miss_rate": 0.0389724226282786,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 17579,
"model_step_duration_ms": 478655.543996,
"model_steps": 14831,
"padding_fraction": 0.035658764941988635,
"padding_tokens": 8120,
"schema": 1,
"token_efficiency_per_ms": 4.562222306607711,
"useful_tokens": 2183733
},
"plan": {
"added_capture_sizes": [
3,
5,
6,
7,
9
],
"clean_seconds": 240,
"gpu": 0,
"gpu_hour_limit": 16.0,
"load": "moderate",
"manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
"measured_padding_recovery_bound": 0.049776380388728225,
"on_capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"order": [
"ON",
"OFF"
],
"pattern": "P09",
"primary_metric": "clean graph-hit padding_fraction",
"prior_gpu_hours": 14.025875418755744,
"profile": false,
"projected_increment_gpu_hours": 0.5,
"projected_total_gpu_hours": 14.525875418755744,
"rate_fraction": 0.6,
"saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json",
"schema": 1,
"secondary_metrics": [
"Layer-1 useful scheduled tokens/model-step millisecond",
"clean completed request throughput",
"clean request latency"
],
"warmup_seconds": 60
},
"schema": 1
},
"schema": 1,
"status": "complete",
"updated_at": 1783857160.3890784
}

5
runs/opprof-phase3/phase4/capture-p09/controller.log Normal file
View File

@@ -0,0 +1,5 @@
GPU_COMMAND P09-capture-ON-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512; expected=6-9m
GPU_COMMAND P09-capture-ON-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/on/client; expected=5-7m
GPU_COMMAND P09-capture-OFF-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120; expected=6-9m
GPU_COMMAND P09-capture-OFF-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/off/client; expected=5-7m
{"delta": {"completed_throughput_relative": 0.0050675675675675436, "e2e_mean_latency_relative": -0.04096717785851722, "e2e_p95_latency_relative": 0.03011353223527924, "padding_fraction_points": -0.049797561163151524, "padding_reduction_fraction": 0.5827252753866776, "token_efficiency_relative": 0.0017943761453185214}, "gpu_hours_increment": 0.2963894498348236, "gpu_hours_total": 14.322264868590567, "off": {"arm": "OFF", "bucket_tokens": 237911, "capture_sizes": "default", "clean_completed": 1184, "clean_completed_throughput_rps": 4.933333333333334, "clean_failed": 0, "clean_offered_rps": 4.920833333333333, "drain_seconds": 0.7853007119847462, "e2e_latency_mean_s": 1.681152764688729, "e2e_latency_p95_s": 3.8763178953449815, "gpu_hours": 0.13751606033907995, "graph_hit_steps": 12252, "graph_miss_rate": 0.045199501246882795, "layer1_invariants": {"cudagraph_identity": true, "footer_balanced": true, "footer_written_matches": true, "schema_1": true, "sidecar_agrees": true, "sidecar_final": true, "steps_unique_contiguous": true, "token_composition": true, "zero_drops": true}, "layer1_records": 16491, "model_step_duration_ms": 479367.749483, "model_steps": 12832, "padding_fraction": 0.08545632610514016, "padding_tokens": 20331, "schema": 1, "token_efficiency_per_ms": 4.5540506267984115, "useful_tokens": 2183065}, "on": {"arm": "ON", "bucket_tokens": 227714, "capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], "clean_completed": 1190, "clean_completed_throughput_rps": 4.958333333333333, "clean_failed": 0, "clean_offered_rps": 4.920833333333333, "drain_seconds": 0.7876331160077825, "e2e_latency_mean_s": 1.612280680370388, "e2e_latency_p95_s": 3.9930475192406423, "gpu_hours": 0.15887338949574364, "graph_hit_steps": 14253, "graph_miss_rate": 0.0389724226282786, "layer1_invariants": {"cudagraph_identity": true, "footer_balanced": true, "footer_written_matches": true, "schema_1": true, "sidecar_agrees": true, "sidecar_final": true, "steps_unique_contiguous": true, "token_composition": true, "zero_drops": true}, "layer1_records": 17579, "model_step_duration_ms": 478655.543996, "model_steps": 14831, "padding_fraction": 0.035658764941988635, "padding_tokens": 8120, "schema": 1, "token_efficiency_per_ms": 4.562222306607711, "useful_tokens": 2183733}, "plan": {"added_capture_sizes": [3, 5, 6, 7, 9], "clean_seconds": 240, "gpu": 0, "gpu_hour_limit": 16.0, "load": "moderate", "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", "measured_padding_recovery_bound": 0.049776380388728225, "on_capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], "order": ["ON", "OFF"], "pattern": "P09", "primary_metric": "clean graph-hit padding_fraction", "prior_gpu_hours": 14.025875418755744, "profile": false, "projected_increment_gpu_hours": 0.5, "projected_total_gpu_hours": 14.525875418755744, "rate_fraction": 0.6, "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json", "schema": 1, "secondary_metrics": ["Layer-1 useful scheduled tokens/model-step millisecond", "clean completed request throughput", "clean request latency"], "warmup_seconds": 60}, "schema": 1}

1
runs/opprof-phase3/phase4/capture-p09/launch.log Normal file
View File

@@ -0,0 +1 @@
GPU_VALIDATION Phase4 P09 capture sizes: order=ON,OFF, GPU0 TP1, manifest=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl, saturation_source=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json, added_sizes=3,5,6,7,9, warmup=60s, clean=240s/arm, projected=0.50 H20-hours, cumulative_projection=14.525875/16, expected_wall=15-22m

66
runs/opprof-phase3/phase4/capture-p09/off/client-sanity.json Normal file
View File

@@ -0,0 +1,66 @@
{
"invariants": {
"clean_duration_exact": true,
"clean_failures_zero": true,
"concurrency_bounded": true,
"drain_within_timeout": true,
"manifest_no_wrap": true,
"manifest_not_exhausted": true,
"moderate_offered_within_5pct": true,
"output_tokens_exact": true,
"profile_count_exact": true,
"profile_status_ok": true,
"segment_count_exact": true
},
"numeric": {
"actual_output_tokens": {
"distinct_n": 1,
"finite_n": 1477,
"max": 64.0,
"min": 64.0,
"missing_n": 0,
"n": 1477
},
"admitted_s": {
"distinct_n": 1477,
"finite_n": 1477,
"max": 299.8483776419889,
"min": 0.00022695399820804596,
"missing_n": 0,
"n": 1477
},
"completed_s": {
"distinct_n": 1477,
"finite_n": 1477,
"max": 300.78334441498737,
"min": 0.6682249770092312,
"missing_n": 0,
"n": 1477
},
"input_tokens": {
"distinct_n": 944,
"finite_n": 1477,
"max": 8161.0,
"min": 128.0,
"missing_n": 0,
"n": 1477
},
"requested_output_tokens": {
"distinct_n": 1,
"finite_n": 1477,
"max": 64.0,
"min": 64.0,
"missing_n": 0,
"n": 1477
},
"scheduled_s": {
"distinct_n": 1477,
"finite_n": 1477,
"max": 299.84763839512016,
"min": 0.0,
"missing_n": 0,
"n": 1477
}
},
"schema": 1
}

2
runs/opprof-phase3/phase4/capture-p09/off/commands.log Normal file
View File

@@ -0,0 +1,2 @@
GPU_COMMAND P09-capture-OFF-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 ; expected=6-9m
GPU_COMMAND P09-capture-OFF-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/off/client ; expected=5-7m

34
runs/opprof-phase3/phase4/capture-p09/off/summary.json Normal file
View File

@@ -0,0 +1,34 @@
{
"arm": "OFF",
"bucket_tokens": 237911,
"capture_sizes": "default",
"clean_completed": 1184,
"clean_completed_throughput_rps": 4.933333333333334,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7853007119847462,
"e2e_latency_mean_s": 1.681152764688729,
"e2e_latency_p95_s": 3.8763178953449815,
"gpu_hours": 0.13751606033907995,
"graph_hit_steps": 12252,
"graph_miss_rate": 0.045199501246882795,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 16491,
"model_step_duration_ms": 479367.749483,
"model_steps": 12832,
"padding_fraction": 0.08545632610514016,
"padding_tokens": 20331,
"schema": 1,
"token_efficiency_per_ms": 4.5540506267984115,
"useful_tokens": 2183065
}

66
runs/opprof-phase3/phase4/capture-p09/on/client-sanity.json Normal file
View File

@@ -0,0 +1,66 @@
{
"invariants": {
"clean_duration_exact": true,
"clean_failures_zero": true,
"concurrency_bounded": true,
"drain_within_timeout": true,
"manifest_no_wrap": true,
"manifest_not_exhausted": true,
"moderate_offered_within_5pct": true,
"output_tokens_exact": true,
"profile_count_exact": true,
"profile_status_ok": true,
"segment_count_exact": true
},
"numeric": {
"actual_output_tokens": {
"distinct_n": 1,
"finite_n": 1477,
"max": 64.0,
"min": 64.0,
"missing_n": 0,
"n": 1477
},
"admitted_s": {
"distinct_n": 1477,
"finite_n": 1477,
"max": 299.8487557930057,
"min": 0.00024887899053283036,
"missing_n": 0,
"n": 1477
},
"completed_s": {
"distinct_n": 1477,
"finite_n": 1477,
"max": 300.78616064199014,
"min": 15.648636110010557,
"missing_n": 0,
"n": 1477
},
"input_tokens": {
"distinct_n": 944,
"finite_n": 1477,
"max": 8161.0,
"min": 128.0,
"missing_n": 0,
"n": 1477
},
"requested_output_tokens": {
"distinct_n": 1,
"finite_n": 1477,
"max": 64.0,
"min": 64.0,
"missing_n": 0,
"n": 1477
},
"scheduled_s": {
"distinct_n": 1477,
"finite_n": 1477,
"max": 299.84763839512016,
"min": 0.0,
"missing_n": 0,
"n": 1477
}
},
"schema": 1
}

2
runs/opprof-phase3/phase4/capture-p09/on/commands.log Normal file
View File

@@ -0,0 +1,2 @@
GPU_COMMAND P09-capture-ON-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 ; expected=6-9m
GPU_COMMAND P09-capture-ON-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/on/client ; expected=5-7m

91
runs/opprof-phase3/phase4/capture-p09/on/summary.json Normal file
View File

@@ -0,0 +1,91 @@
{
"arm": "ON",
"bucket_tokens": 227714,
"capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"clean_completed": 1190,
"clean_completed_throughput_rps": 4.958333333333333,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7876331160077825,
"e2e_latency_mean_s": 1.612280680370388,
"e2e_latency_p95_s": 3.9930475192406423,
"gpu_hours": 0.15887338949574364,
"graph_hit_steps": 14253,
"graph_miss_rate": 0.0389724226282786,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 17579,
"model_step_duration_ms": 478655.543996,
"model_steps": 14831,
"padding_fraction": 0.035658764941988635,
"padding_tokens": 8120,
"schema": 1,
"token_efficiency_per_ms": 4.562222306607711,
"useful_tokens": 2183733
}

230
runs/opprof-phase3/phase4/capture-p09/result.json Normal file
View File

@@ -0,0 +1,230 @@
{
"delta": {
"completed_throughput_relative": 0.0050675675675675436,
"e2e_mean_latency_relative": -0.04096717785851722,
"e2e_p95_latency_relative": 0.03011353223527924,
"padding_fraction_points": -0.049797561163151524,
"padding_reduction_fraction": 0.5827252753866776,
"token_efficiency_relative": 0.0017943761453185214
},
"gpu_hours_increment": 0.2963894498348236,
"gpu_hours_total": 14.322264868590567,
"off": {
"arm": "OFF",
"bucket_tokens": 237911,
"capture_sizes": "default",
"clean_completed": 1184,
"clean_completed_throughput_rps": 4.933333333333334,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7853007119847462,
"e2e_latency_mean_s": 1.681152764688729,
"e2e_latency_p95_s": 3.8763178953449815,
"gpu_hours": 0.13751606033907995,
"graph_hit_steps": 12252,
"graph_miss_rate": 0.045199501246882795,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 16491,
"model_step_duration_ms": 479367.749483,
"model_steps": 12832,
"padding_fraction": 0.08545632610514016,
"padding_tokens": 20331,
"schema": 1,
"token_efficiency_per_ms": 4.5540506267984115,
"useful_tokens": 2183065
},
"on": {
"arm": "ON",
"bucket_tokens": 227714,
"capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"clean_completed": 1190,
"clean_completed_throughput_rps": 4.958333333333333,
"clean_failed": 0,
"clean_offered_rps": 4.920833333333333,
"drain_seconds": 0.7876331160077825,
"e2e_latency_mean_s": 1.612280680370388,
"e2e_latency_p95_s": 3.9930475192406423,
"gpu_hours": 0.15887338949574364,
"graph_hit_steps": 14253,
"graph_miss_rate": 0.0389724226282786,
"layer1_invariants": {
"cudagraph_identity": true,
"footer_balanced": true,
"footer_written_matches": true,
"schema_1": true,
"sidecar_agrees": true,
"sidecar_final": true,
"steps_unique_contiguous": true,
"token_composition": true,
"zero_drops": true
},
"layer1_records": 17579,
"model_step_duration_ms": 478655.543996,
"model_steps": 14831,
"padding_fraction": 0.035658764941988635,
"padding_tokens": 8120,
"schema": 1,
"token_efficiency_per_ms": 4.562222306607711,
"useful_tokens": 2183733
},
"plan": {
"added_capture_sizes": [
3,
5,
6,
7,
9
],
"clean_seconds": 240,
"gpu": 0,
"gpu_hour_limit": 16.0,
"load": "moderate",
"manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl",
"measured_padding_recovery_bound": 0.049776380388728225,
"on_capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
9,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"order": [
"ON",
"OFF"
],
"pattern": "P09",
"primary_metric": "clean graph-hit padding_fraction",
"prior_gpu_hours": 14.025875418755744,
"profile": false,
"projected_increment_gpu_hours": 0.5,
"projected_total_gpu_hours": 14.525875418755744,
"rate_fraction": 0.6,
"saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json",
"schema": 1,
"secondary_metrics": [
"Layer-1 useful scheduled tokens/model-step millisecond",
"clean completed request throughput",
"clean request latency"
],
"warmup_seconds": 60
},
"schema": 1
}

373
runs/opprof-phase3/phase4/capture_validation.py Normal file
View File

@@ -0,0 +1,373 @@
#!/usr/bin/env python3
"""One-pair P09 CUDAGraph capture-size validation for OpProf Phase 4."""
from __future__ import annotations
import argparse
import json
import math
import os
import shlex
import signal
import subprocess
import sys
import time
import urllib.request
from pathlib import Path
from typing import Any
import numpy as np
import opprof_phase3_controller as common
import opprof_phase3_matrix as matrix
WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
ROOT = WORKDIR / "runs/phase4-capture-p09"
PHASE3 = WORKDIR / "runs/phase3"
PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests")
MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0")
VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
STATE = ROOT / "controller-state.json"
GPU = 0
PORT = 8200
PRIOR_GPU_HOURS = 14.025875418755744
GPU_HOUR_LIMIT = 16.0
EXPECTED_INCREMENT_HOURS = 0.5
ADDED_CAPTURE_SIZES = (3, 5, 6, 7, 9)
DEFAULT_CAPTURE_SIZES = (
1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104,
112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200,
208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336,
352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512,
)
ON_CAPTURE_SIZES = tuple(sorted(set(DEFAULT_CAPTURE_SIZES + ADDED_CAPTURE_SIZES)))
def plan() -> dict[str, Any]:
return {
"schema": 1,
"pattern": "P09",
"load": "moderate",
"order": ["ON", "OFF"],
"gpu": GPU,
"warmup_seconds": 60,
"clean_seconds": 240,
"profile": False,
"manifest": str(PRIVATE / "P09.jsonl"),
"saturation_result": str(
PHASE3 / "primary/P09-C00/saturation/client/result.json"
),
"rate_fraction": 0.60,
"added_capture_sizes": list(ADDED_CAPTURE_SIZES),
"on_capture_sizes": list(ON_CAPTURE_SIZES),
"measured_padding_recovery_bound": 0.049776380388728225,
"primary_metric": "clean graph-hit padding_fraction",
"secondary_metrics": [
"Layer-1 useful scheduled tokens/model-step millisecond",
"clean completed request throughput",
"clean request latency",
],
"prior_gpu_hours": PRIOR_GPU_HOURS,
"projected_increment_gpu_hours": EXPECTED_INCREMENT_HOURS,
"projected_total_gpu_hours": PRIOR_GPU_HOURS + EXPECTED_INCREMENT_HOURS,
"gpu_hour_limit": GPU_HOUR_LIMIT,
}
def save_state(state: dict[str, Any]) -> None:
state["updated_at"] = time.time()
state["controller_pid"] = os.getpid()
common.atomic_json(STATE, state)
def wait_ready(server: subprocess.Popen[Any]) -> None:
deadline = time.monotonic() + 300
while time.monotonic() < deadline:
if server.poll() is not None:
raise RuntimeError("server exited before readiness")
try:
with urllib.request.urlopen(
f"http://127.0.0.1:{PORT}/health", timeout=1
) as response:
if response.status == 200:
return
except Exception:
pass
time.sleep(1)
raise TimeoutError("server readiness timeout")
def server_command(arm: str) -> list[str]:
command = [
"taskset", "-c", "0-19", str(VENV / "bin/vllm"), "serve", str(MODEL),
"--host", "127.0.0.1", "--port", str(PORT),
"--tensor-parallel-size", "1", "--enable-chunked-prefill",
"--enable-prefix-caching", "--shutdown-timeout", "120",
]
if arm == "ON":
command.extend(("--cudagraph-capture-sizes", *map(str, ON_CAPTURE_SIZES)))
return command
def client_command(run_dir: Path) -> list[str]:
return [
"taskset", "-c", "0-19", str(VENV / "bin/python"), str(CLIENT), "run",
"--manifest", str(PRIVATE / "P09.jsonl"),
"--base-url", f"http://127.0.0.1:{PORT}", "--model", str(MODEL),
"--load-point", "moderate", "--saturation-result",
str(PHASE3 / "primary/P09-C00/saturation/client/result.json"),
"--rate-fraction", "0.60", "--max-concurrency", "256", "--ignore-eos",
"--temperature", "0", "--warmup-seconds", "60",
"--clean-segment-seconds", "80", "--num-clean-segments", "3",
"--recovery-seconds", "30", "--drain-timeout-seconds", "120",
"--workload-seed", "20260712", "--server-seed", "20260712",
"--result-dir", str(run_dir / "client"),
]
def summarize(run_dir: Path, arm: str) -> dict[str, Any]:
result = json.loads((run_dir / "client/result.json").read_text())
requests = [
json.loads(line)
for line in (run_dir / "client/requests.jsonl").read_text().splitlines()
]
t0 = int(result["t0_mono_ns"])
start, end = t0 + int(60e9), t0 + int(300e9)
stream = next((run_dir / "opprof").glob("*.jsonl"))
records = []
for line in stream.read_text().splitlines():
record = json.loads(line)
if (
"step_index" in record
and start <= int(record["submit_mono_ns"]) < end
):
records.append(record)
model = [record for record in records if record["model_executed"]]
hits = [
record
for record in model
if record["cudagraph"]["hit"]
and int(record["cudagraph"]["bucket_tokens"]) > 0
]
pad = sum(int(record["cudagraph"]["padding_tokens"]) for record in hits)
bucket = sum(int(record["cudagraph"]["bucket_tokens"]) for record in hits)
useful = sum(
int(record["prefill_tokens"]) + int(record["decode_tokens"])
for record in records
)
duration_ms = sum(
(int(record["complete_mono_ns"]) - int(record["submit_mono_ns"])) / 1e6
for record in records
)
completed = [
request
for request in requests
if request["success"] and 60 <= float(request["completed_s"]) < 300
]
e2e = np.asarray(
[float(request["completed_s"] - request["admitted_s"]) for request in completed]
)
layer1 = matrix.validate_layer1(run_dir)
return {
"schema": 1,
"arm": arm,
"capture_sizes": list(ON_CAPTURE_SIZES) if arm == "ON" else "default",
"clean_completed": len(completed),
"clean_failed": int(result["clean"]["failed"]),
"clean_completed_throughput_rps": float(
result["clean"]["completed_throughput_rps"]
),
"clean_offered_rps": float(result["clean"]["offered_rps"]),
"e2e_latency_mean_s": float(e2e.mean()),
"e2e_latency_p95_s": float(np.quantile(e2e, 0.95)),
"model_steps": len(model),
"graph_hit_steps": len(hits),
"padding_tokens": pad,
"bucket_tokens": bucket,
"padding_fraction": pad / bucket,
"graph_miss_rate": sum(not record["cudagraph"]["hit"] for record in model)
/ len(model),
"useful_tokens": useful,
"model_step_duration_ms": duration_ms,
"token_efficiency_per_ms": useful / duration_ms,
"layer1_records": layer1["records"],
"layer1_invariants": layer1["invariants"],
"drain_seconds": float(result["drain_seconds"]),
}
def run_arm(state: dict[str, Any], arm: str) -> None:
if state["arms"].get(arm, {}).get("status") == "complete":
return
run_dir = ROOT / arm.lower()
if run_dir.exists():
run_dir.rename(run_dir.with_name(f"{run_dir.name}.interrupted-{int(time.time())}"))
run_dir.mkdir(parents=True)
common.preflight([GPU], run_dir)
server_cmd = server_command(arm)
client_cmd = client_command(run_dir)
commands_path = run_dir / "commands.log"
common.command_log(commands_path, f"P09-capture-{arm}-server", server_cmd, "6-9m")
common.command_log(commands_path, f"P09-capture-{arm}-client", client_cmd, "5-7m")
print(
f"GPU_COMMAND P09-capture-{arm}-server: {shlex.join(server_cmd)}; "
"expected=6-9m",
flush=True,
)
print(
f"GPU_COMMAND P09-capture-{arm}-client: {shlex.join(client_cmd)}; "
"expected=5-7m",
flush=True,
)
environment = os.environ.copy()
environment.update(
{
"CUDA_VISIBLE_DEVICES": str(GPU),
"VLLM_OPPROF_DIR": str(run_dir / "opprof"),
"HF_HUB_OFFLINE": "1",
"TRANSFORMERS_OFFLINE": "1",
"PYTHONUNBUFFERED": "1",
}
)
server_log = (run_dir / "server.log").open("ab", buffering=0)
server_started = time.time()
server = subprocess.Popen(
server_cmd,
cwd=SOURCE,
env=environment,
stdout=server_log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
client = None
client_log = None
monitor = None
owned = {server.pid}
state["arms"][arm] = {
"status": "starting",
"server_pid": server.pid,
"started_at": server_started,
}
save_state(state)
failure = None
try:
wait_ready(server)
monitor = common.Monitor(run_dir / "monitor.jsonl", owned)
monitor.start()
client_log = (run_dir / "client.log").open("ab", buffering=0)
client = subprocess.Popen(
client_cmd,
cwd=WORKDIR,
stdout=client_log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
owned.add(client.pid)
state["arms"][arm].update(status="running", client_pid=client.pid)
save_state(state)
deadline = time.monotonic() + 900
while client.poll() is None and time.monotonic() < deadline:
if server.poll() is not None:
raise RuntimeError("server exited during client load")
if monitor.other_apps:
raise RuntimeError(f"other GPU process appeared: {monitor.other_apps}")
time.sleep(2)
if client.poll() is None:
raise TimeoutError("client exceeded 900 seconds")
if client.returncode:
raise RuntimeError(f"client failed with {client.returncode}")
except Exception as error:
failure = error
finally:
if client is not None and client.poll() is None:
try:
os.killpg(client.pid, signal.SIGKILL)
except ProcessLookupError:
pass
common.stop_servers([server])
server_log.close()
if client_log is not None:
client_log.close()
if monitor is not None:
monitor.stop()
common.verify_idle([GPU], run_dir)
gpu_hours = (time.time() - server_started) / 3600
state["gpu_hours_increment"] += gpu_hours
if PRIOR_GPU_HOURS + state["gpu_hours_increment"] >= GPU_HOUR_LIMIT:
failure = failure or RuntimeError("GPU-hour limit reached")
if failure is not None:
state["arms"][arm].update(status="failed", failure=repr(failure))
state["status"] = "failed"
save_state(state)
raise failure
summary = summarize(run_dir, arm)
summary["gpu_hours"] = gpu_hours
common.atomic_json(run_dir / "summary.json", summary)
state["arms"][arm].update(status="complete", summary=summary)
save_state(state)
def run() -> None:
ROOT.mkdir(parents=True, exist_ok=True)
if PRIOR_GPU_HOURS + EXPECTED_INCREMENT_HOURS >= GPU_HOUR_LIMIT:
raise RuntimeError("projected validation exceeds GPU-hour budget")
if STATE.exists():
state = json.loads(STATE.read_text())
else:
state = {
"schema": 1,
"status": "running",
"created_at": time.time(),
"plan": plan(),
"arms": {},
"gpu_hours_increment": 0.0,
}
for arm in ("ON", "OFF"):
run_arm(state, arm)
on, off = state["arms"]["ON"]["summary"], state["arms"]["OFF"]["summary"]
result = {
"schema": 1,
"plan": plan(),
"on": on,
"off": off,
"delta": {
"padding_fraction_points": on["padding_fraction"] - off["padding_fraction"],
"padding_reduction_fraction": 1 - on["padding_fraction"] / off["padding_fraction"],
"token_efficiency_relative": on["token_efficiency_per_ms"]
/ off["token_efficiency_per_ms"]
- 1,
"completed_throughput_relative": on["clean_completed_throughput_rps"]
/ off["clean_completed_throughput_rps"]
- 1,
"e2e_mean_latency_relative": on["e2e_latency_mean_s"]
/ off["e2e_latency_mean_s"]
- 1,
"e2e_p95_latency_relative": on["e2e_latency_p95_s"]
/ off["e2e_latency_p95_s"]
- 1,
},
"gpu_hours_increment": state["gpu_hours_increment"],
"gpu_hours_total": PRIOR_GPU_HOURS + state["gpu_hours_increment"],
}
common.atomic_json(ROOT / "result.json", result)
state["status"] = "complete"
state["result"] = result
save_state(state)
print(json.dumps(result, sort_keys=True))
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("command", choices=("plan", "run"))
args = parser.parse_args()
if args.command == "plan":
print(json.dumps(plan(), indent=2, sort_keys=True))
else:
run()
if __name__ == "__main__":
main()

1611
runs/opprof-phase3/provenance/analyze_phase3.py Normal file
View File

File diff suppressed because it is too large Load Diff

794
runs/opprof-phase3/provenance/opprof_phase3_client.py Normal file
View File

@@ -0,0 +1,794 @@
#!/usr/bin/env python3
"""Token-exact fixed-duration client for the OpProf Phase-3 protocol."""
from __future__ import annotations
import argparse
import asyncio
import gzip
import hashlib
import json
import math
import os
import time
from dataclasses import dataclass
from pathlib import Path
from typing import Any
import aiohttp
SCHEMA = 1
TOKEN_BASE = 1000
TOKEN_SPAN = 100000
class ManifestExhausted(RuntimeError):
pass
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def atomic_json(path: Path, value: Any, mode: int = 0o640) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
with os.fdopen(fd, "w", encoding="utf-8") as f:
json.dump(value, f, sort_keys=True, indent=2)
f.write("\n")
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def atomic_jsonl(path: Path, rows: list[dict[str, Any]], mode: int = 0o640) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode)
with os.fdopen(fd, "w", encoding="utf-8") as f:
for row in rows:
f.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n")
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def parse_range(value: str) -> tuple[int, int]:
lo_text, hi_text = value.split(":", 1)
lo, hi = int(lo_text), int(hi_text)
if lo <= 0 or hi < lo:
raise argparse.ArgumentTypeError(f"invalid positive range: {value}")
return lo, hi
def _integer_counts(weights: list[float], total: int) -> list[int]:
raw = [w * total for w in weights]
counts = [math.floor(x) for x in raw]
order = sorted(
range(len(raw)), key=lambda i: raw[i] - counts[i], reverse=True
)
for idx in order[: total - sum(counts)]:
counts[idx] += 1
return counts
def numeric_sanity(values: list[float | int]) -> dict[str, Any]:
finite = [float(x) for x in values if math.isfinite(float(x))]
return {
"n": len(values),
"finite_n": len(finite),
"missing_n": len(values) - len(finite),
"min": min(finite) if finite else None,
"max": max(finite) if finite else None,
"distinct_n": len(set(finite)),
}
def manifest_summary(rows: list[dict[str, Any]]) -> dict[str, Any]:
return {
"schema": SCHEMA,
"rows": len(rows),
"input_tokens": numeric_sanity([int(r["input_tokens"]) for r in rows]),
"output_tokens": numeric_sanity([int(r["output_tokens"]) for r in rows]),
"arrival_values": sorted({str(r["arrival"]) for r in rows}),
"pattern_values": sorted({str(r["pattern_id"]) for r in rows}),
}
def materialize(args: argparse.Namespace) -> dict[str, Any]:
import numpy as np
rng = np.random.default_rng(args.workload_seed)
n = args.num_requests
if args.kind == "prefix-pool":
if args.num_prefixes <= 0 or args.prefix_len <= 0 or args.suffix_fixed <= 0:
raise ValueError("prefix-pool requires positive pool/prefix/suffix")
lengths = np.full(n, args.prefix_len + args.suffix_fixed, dtype=np.int64)
prefix_ids = np.arange(n, dtype=np.int64) % args.num_prefixes
rng.shuffle(prefix_ids)
else:
prefix_ids = np.full(n, -1, dtype=np.int64)
if args.input_uniform:
lo, hi = parse_range(args.input_uniform)
lengths = rng.integers(lo, hi + 1, n, dtype=np.int64)
elif args.input_fixed:
lengths = np.full(n, args.input_fixed, dtype=np.int64)
elif args.input_mixture:
spec = json.loads(args.input_mixture)
if not isinstance(spec, dict) or not spec:
raise ValueError("input mixture must be a non-empty JSON object")
keys = list(spec)
weights = [float(spec[key]) for key in keys]
if any(w < 0 for w in weights) or not math.isclose(sum(weights), 1.0):
raise ValueError("mixture weights must be non-negative and sum to 1")
pieces = []
for key, count in zip(
keys, _integer_counts(weights, n), strict=True
):
kind, lo_text, hi_text = key.split(":")
if kind != "uniform":
raise ValueError(f"unsupported mixture component: {key}")
pieces.append(
rng.integers(
int(lo_text), int(hi_text) + 1, count, dtype=np.int64
)
)
lengths = np.concatenate(pieces)
rng.shuffle(lengths)
else:
raise ValueError("exactly one input distribution is required")
if args.output_fixed <= 0 or args.arrival not in {"steady", "burst:8"}:
raise ValueError("invalid output length or arrival class")
rows = []
for i in range(n):
row = {
"schema": SCHEMA,
"request_id": f"{args.id}-{i:05d}",
"pattern_id": args.id,
"kind": args.kind,
"input_tokens": int(lengths[i]),
"output_tokens": args.output_fixed,
"arrival": args.arrival,
"token_seed": int(args.workload_seed * 1000003 + i),
}
if args.kind == "prefix-pool":
row.update(
{
"prefix_id": int(prefix_ids[i]),
"num_prefixes": args.num_prefixes,
"prefix_tokens": args.prefix_len,
}
)
rows.append(row)
out = Path(args.out)
atomic_jsonl(out, rows, mode=0o600)
summary = manifest_summary(rows)
summary.update({"sha256": sha256_file(out), "path": str(out)})
atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
print(json.dumps(summary, sort_keys=True))
return summary
def materialize_private(args: argparse.Namespace) -> dict[str, Any]:
from transformers import AutoTokenizer
source = Path(args.source)
selected: list[dict[str, Any]] = []
with source.open(encoding="utf-8") as f:
for source_index, line in enumerate(f):
row = json.loads(line)
if (
float(row["sampling_u"]) <= args.sampling_u_max
and int(row["input_length"]) <= args.max_input_tokens
):
selected.append(
{
"schema": SCHEMA,
"request_id": f"{args.id}-{len(selected):05d}",
"pattern_id": args.id,
"kind": "private-trace",
"input_tokens": int(row["input_length"]),
"output_tokens": min(
int(row["output_length"]), args.output_cap
),
"arrival": args.arrival,
"source_index": source_index,
"prompt": row["prompt"],
}
)
tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True)
diffs = [
len(tokenizer.encode(row["prompt"], add_special_tokens=False))
- row["input_tokens"]
for row in selected
]
exact = sum(diff == 0 for diff in diffs)
exact_fraction = exact / len(diffs) if diffs else 0.0
max_abs = max((abs(diff) for diff in diffs), default=-1)
if exact_fraction < 0.99 or max_abs > 1:
raise RuntimeError(
"tokenizer parity gate failed: "
f"exact_fraction={exact_fraction:.6f} max_abs_error={max_abs}"
)
out = Path(args.out)
atomic_jsonl(out, selected, mode=0o600)
summary = manifest_summary(selected)
summary.update(
{
"sha256": sha256_file(out),
"source_sha256": sha256_file(source),
"tokenizer_exact_n": exact,
"tokenizer_exact_fraction": exact_fraction,
"tokenizer_max_abs_error": max_abs,
"path": str(out),
}
)
atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600)
print(json.dumps(summary, sort_keys=True))
return summary
def load_manifest(path: Path) -> list[dict[str, Any]]:
rows = [json.loads(line) for line in path.read_text().splitlines() if line]
required = {
"request_id",
"pattern_id",
"input_tokens",
"output_tokens",
"arrival",
}
if not rows:
raise ValueError("empty manifest")
for row in rows:
if not required.issubset(row):
raise ValueError(f"manifest row lacks {sorted(required - set(row))}")
if len({row["request_id"] for row in rows}) != len(rows):
raise ValueError("duplicate request_id")
return rows
def _token_stream(seed: int, count: int) -> list[int]:
state = seed & 0xFFFFFFFF
out = []
for _ in range(count):
state = (1664525 * state + 1013904223) & 0xFFFFFFFF
out.append(TOKEN_BASE + state % TOKEN_SPAN)
return out
def synthetic_prompt(row: dict[str, Any]) -> list[int]:
length = int(row["input_tokens"])
seed = int(row.get("token_seed", 0))
if row.get("kind") == "prefix-pool":
prefix_n = int(row["prefix_tokens"])
tokens = _token_stream(0xA5A50000 + int(row["prefix_id"]), prefix_n)
tokens += _token_stream(seed, length - prefix_n)
offset = prefix_n
else:
tokens = _token_stream(seed, length)
offset = 0
if length - offset >= 3:
index = int(row["request_id"].rsplit("-", 1)[1])
tokens[offset : offset + 3] = [
TOKEN_BASE + index % 100,
TOKEN_BASE + (index // 100) % 100,
TOKEN_BASE + (index // 10000) % 100,
]
return tokens
@dataclass
class RunContext:
args: argparse.Namespace
rows: list[dict[str, Any]]
t0: float
clean_end: float
stop_event: asyncio.Event
lock: asyncio.Lock
next_index: int = 0
in_flight: int = 0
max_in_flight: int = 0
exhausted: bool = False
admission_stop_s: float | None = None
async def next_row(self) -> dict[str, Any]:
async with self.lock:
if self.next_index >= len(self.rows):
self.exhausted = True
raise ManifestExhausted(
f"manifest exhausted after {self.next_index} admissions"
)
row = self.rows[self.next_index]
self.next_index += 1
return row
async def request_one(
ctx: RunContext,
session: aiohttp.ClientSession,
row: dict[str, Any],
scheduled: float,
) -> dict[str, Any]:
loop = asyncio.get_running_loop()
admitted = loop.time()
ctx.in_flight += 1
ctx.max_in_flight = max(ctx.max_in_flight, ctx.in_flight)
status = 0
actual_output: int | None = None
first_token: float | None = None
error_kind: str | None = None
try:
prompt: str | list[int] = (
row["prompt"]
if row.get("kind") == "private-trace"
else synthetic_prompt(row)
)
if not isinstance(prompt, str) and len(prompt) != int(row["input_tokens"]):
raise AssertionError("synthetic prompt length drift")
payload = {
"model": ctx.args.model,
"prompt": prompt,
"max_tokens": int(row["output_tokens"]),
"temperature": ctx.args.temperature,
"ignore_eos": ctx.args.ignore_eos,
"stream": True,
"stream_options": {"include_usage": True},
"add_special_tokens": False,
"seed": ctx.args.server_seed,
}
headers = {
"Content-Type": "application/json",
"x-request-id": str(row["request_id"]),
}
async with session.post(
ctx.args.base_url.rstrip("/") + "/v1/completions",
json=payload,
headers=headers,
) as response:
status = response.status
if status != 200:
error_kind = f"http_{status}"
else:
buf = b""
async for chunk in response.content.iter_any():
buf += chunk
while b"\n" in buf:
line, buf = buf.split(b"\n", 1)
line = line.strip()
if not line.startswith(b"data:"):
continue
data = line[5:].strip()
if data == b"[DONE]":
continue
event = json.loads(data)
if event.get("choices") and first_token is None:
first_token = loop.time()
if event.get("usage") is not None:
actual_output = int(event["usage"]["completion_tokens"])
if actual_output is None:
error_kind = "missing_usage"
except (aiohttp.ClientError, asyncio.TimeoutError) as exc:
error_kind = type(exc).__name__
except Exception as exc:
error_kind = type(exc).__name__
finally:
completed = loop.time()
ctx.in_flight -= 1
success = (
status == 200
and error_kind is None
and actual_output == int(row["output_tokens"])
)
if status == 200 and actual_output is not None and not success:
error_kind = "output_token_mismatch"
return {
"schema": SCHEMA,
"request_id": row["request_id"],
"scheduled_s": scheduled - ctx.t0,
"admitted_s": admitted - ctx.t0,
"first_token_s": None if first_token is None else first_token - ctx.t0,
"completed_s": completed - ctx.t0,
"input_tokens": int(row["input_tokens"]),
"requested_output_tokens": int(row["output_tokens"]),
"actual_output_tokens": actual_output,
"http_status": status,
"success": success,
"error_kind": error_kind,
}
async def saturation_load(
ctx: RunContext, session: aiohttp.ClientSession
) -> list[dict[str, Any]]:
results: list[dict[str, Any]] = []
async def worker() -> None:
while not ctx.stop_event.is_set():
try:
row = await ctx.next_row()
except ManifestExhausted:
ctx.stop_event.set()
return
results.append(
await request_one(ctx, session, row, asyncio.get_running_loop().time())
)
tasks = [
asyncio.create_task(worker()) for _ in range(ctx.args.max_concurrency)
]
await asyncio.gather(*tasks)
return results
async def finite_load(
ctx: RunContext, session: aiohttp.ClientSession, rate: float
) -> list[dict[str, Any]]:
sem = asyncio.Semaphore(ctx.args.max_concurrency)
tasks: list[asyncio.Task[dict[str, Any]]] = []
batch = 8 if str(ctx.rows[0]["arrival"]) == "burst:8" else 1
period = batch / rate
event_index = 0
async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]:
async with sem:
return await request_one(ctx, session, row, scheduled)
while not ctx.stop_event.is_set():
scheduled = ctx.t0 + event_index * period
delay = scheduled - asyncio.get_running_loop().time()
if delay > 0:
try:
await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay)
break
except asyncio.TimeoutError:
pass
if ctx.stop_event.is_set():
break
try:
for _ in range(batch):
tasks.append(
asyncio.create_task(limited(await ctx.next_row(), scheduled))
)
except ManifestExhausted:
ctx.stop_event.set()
break
event_index += 1
return await asyncio.gather(*tasks) if tasks else []
async def post_profile(
session: aiohttp.ClientSession, base_url: str, endpoint: str
) -> tuple[float, float, int]:
loop = asyncio.get_running_loop()
before = loop.time()
async with session.post(base_url.rstrip("/") + endpoint) as response:
status = response.status
await response.read()
return before, loop.time(), status
def _trace_loadable(path: Path) -> bool:
try:
opener = gzip.open if path.suffix == ".gz" else open
with opener(path, "rt", encoding="utf-8") as f:
parsed = json.load(f)
return isinstance(parsed, dict) and isinstance(parsed.get("traceEvents"), list)
except (OSError, EOFError, json.JSONDecodeError):
return False
async def wait_new_trace(
trace_dir: Path, before: set[Path], timeout: float
) -> Path:
deadline = asyncio.get_running_loop().time() + timeout
while asyncio.get_running_loop().time() < deadline:
for path in sorted(set(trace_dir.glob("*.pt.trace.json*")) - before):
if _trace_loadable(path):
return path
await asyncio.sleep(0.25)
raise TimeoutError(f"no new loadable trace within {timeout}s")
async def timeline(
ctx: RunContext, session: aiohttp.ClientSession
) -> list[dict[str, Any]]:
args = ctx.args
profiles: list[dict[str, Any]] = []
await asyncio.sleep(max(0, ctx.clean_end - asyncio.get_running_loop().time()))
if args.profile_after_clean:
trace_dir = Path(args.profile_trace_dir)
for window in range(args.num_profile_windows):
prior = set(trace_dir.glob("*.pt.trace.json*"))
start_before, start_after, start_status = await post_profile(
session, args.base_url, "/start_profile"
)
trace = await wait_new_trace(
trace_dir, prior, args.profile_timeout_seconds
)
trace_ready = asyncio.get_running_loop().time()
stop_before, stop_after, stop_status = await post_profile(
session, args.base_url, "/stop_profile"
)
profiles.append(
{
"window": window + 1,
"start_call_s": start_before - ctx.t0,
"start_return_s": start_after - ctx.t0,
"trace_ready_s": trace_ready - ctx.t0,
"stop_call_s": stop_before - ctx.t0,
"stop_return_s": stop_after - ctx.t0,
"start_status": start_status,
"stop_status": stop_status,
"trace_file": trace.name,
"trace_sha256": sha256_file(trace),
}
)
if start_status != 200 or stop_status != 200:
raise RuntimeError("profile endpoint returned non-200")
await asyncio.sleep(args.recovery_seconds)
else:
await asyncio.sleep(args.post_clean_seconds)
ctx.admission_stop_s = asyncio.get_running_loop().time() - ctx.t0
ctx.stop_event.set()
return profiles
def segment_summary(
records: list[dict[str, Any]], start: float, end: float
) -> dict[str, Any]:
admitted = [r for r in records if start <= r["admitted_s"] < end]
completed = [r for r in records if start <= r["completed_s"] < end]
successes = [r for r in completed if r["success"]]
duration = end - start
return {
"start_s": start,
"end_s": end,
"duration_s": duration,
"admitted": len(admitted),
"completed": len(successes),
"failed": len(completed) - len(successes),
"offered_rps": len(admitted) / duration,
"completed_throughput_rps": len(successes) / duration,
"input_tokens": sum(r["input_tokens"] for r in successes),
"output_tokens": sum(r["actual_output_tokens"] or 0 for r in successes),
}
async def run_load(args: argparse.Namespace) -> dict[str, Any]:
manifest = Path(args.manifest)
rows = load_manifest(manifest)
arrivals = {row["arrival"] for row in rows}
if len(arrivals) != 1:
raise ValueError("a manifest must have one arrival class")
if args.load_point == "saturation":
if args.request_rate != "inf":
raise ValueError("saturation requires --request-rate inf")
rate = math.inf
else:
if not args.saturation_result:
raise ValueError("moderate requires --saturation-result")
sat = json.loads(Path(args.saturation_result).read_text())
rate = args.rate_fraction * float(sat["clean"]["completed_throughput_rps"])
if not math.isfinite(rate) or rate <= 0:
raise ValueError("derived moderate rate must be positive and finite")
loop = asyncio.get_running_loop()
t0 = loop.time()
t0_mono_ns = int(t0 * 1e9)
t0_wall_ns = time.time_ns()
clean_seconds = args.clean_segment_seconds * args.num_clean_segments
ctx = RunContext(
args=args,
rows=rows,
t0=t0,
clean_end=t0 + args.warmup_seconds + clean_seconds,
stop_event=asyncio.Event(),
lock=asyncio.Lock(),
)
timeout = aiohttp.ClientTimeout(total=None, connect=30, sock_read=600)
connector = aiohttp.TCPConnector(limit=args.max_concurrency)
control_connector = aiohttp.TCPConnector(limit=2)
async with (
aiohttp.ClientSession(timeout=timeout, connector=connector) as session,
aiohttp.ClientSession(
timeout=timeout, connector=control_connector
) as control_session,
):
profile_task = asyncio.create_task(timeline(ctx, control_session))
load_task = asyncio.create_task(
saturation_load(ctx, session)
if math.isinf(rate)
else finite_load(ctx, session, rate)
)
try:
profiles = await profile_task
except Exception:
ctx.stop_event.set()
await load_task
raise
records = await load_task
clean_start = args.warmup_seconds
clean_end = clean_start + clean_seconds
clean = segment_summary(records, clean_start, clean_end)
segments = []
for i in range(args.num_clean_segments):
start = clean_start + i * args.clean_segment_seconds
segments.append(
{
"name": chr(ord("A") + i),
**segment_summary(
records, start, start + args.clean_segment_seconds
),
}
)
successful = [r for r in records if r["success"]]
elapsed_seconds = loop.time() - t0
if ctx.admission_stop_s is None:
raise RuntimeError("admission stop timestamp was not recorded")
drain_seconds = elapsed_seconds - ctx.admission_stop_s
result = {
"schema": SCHEMA,
"manifest_sha256": sha256_file(manifest),
"manifest_rows": len(rows),
"manifest_admitted": ctx.next_index,
"manifest_wrapped": False,
"manifest_exhausted": ctx.exhausted,
"load_point": args.load_point,
"t0_mono_ns": t0_mono_ns,
"t0_wall_ns": t0_wall_ns,
"request_rate": "inf" if math.isinf(rate) else rate,
"rate_fraction": None if math.isinf(rate) else args.rate_fraction,
"arrival": next(iter(arrivals)),
"warmup_seconds": args.warmup_seconds,
"clean_segment_seconds": args.clean_segment_seconds,
"num_clean_segments": args.num_clean_segments,
"elapsed_seconds": elapsed_seconds,
"admission_stop_s": ctx.admission_stop_s,
"drain_seconds": drain_seconds,
"max_in_flight": ctx.max_in_flight,
"records": len(records),
"successful_records": len(successful),
"failed_records": len(records) - len(successful),
"clean": clean,
"segments": segments,
"profiles": profiles,
}
sanity = {
"schema": SCHEMA,
"numeric": {
"input_tokens": numeric_sanity([r["input_tokens"] for r in records]),
"requested_output_tokens": numeric_sanity(
[r["requested_output_tokens"] for r in records]
),
"actual_output_tokens": numeric_sanity(
[
r["actual_output_tokens"]
for r in records
if r["actual_output_tokens"] is not None
]
),
"scheduled_s": numeric_sanity([r["scheduled_s"] for r in records]),
"admitted_s": numeric_sanity([r["admitted_s"] for r in records]),
"completed_s": numeric_sanity([r["completed_s"] for r in records]),
},
"invariants": {
"clean_duration_exact": math.isclose(clean["duration_s"], clean_seconds),
"segment_count_exact": len(segments) == args.num_clean_segments,
"manifest_no_wrap": ctx.next_index <= len(rows),
"manifest_not_exhausted": not ctx.exhausted,
"concurrency_bounded": ctx.max_in_flight <= args.max_concurrency,
"drain_within_timeout": drain_seconds <= args.drain_timeout_seconds,
"output_tokens_exact": all(
r["actual_output_tokens"] == r["requested_output_tokens"]
for r in successful
),
"clean_failures_zero": clean["failed"] == 0,
"profile_count_exact": len(profiles)
== (args.num_profile_windows if args.profile_after_clean else 0),
"profile_status_ok": all(
p["start_status"] == 200 and p["stop_status"] == 200
for p in profiles
),
},
}
if not math.isinf(rate):
sanity["invariants"]["moderate_offered_within_5pct"] = (
abs(clean["offered_rps"] / rate - 1) <= 0.05
)
out = Path(args.result_dir)
out.mkdir(parents=True, exist_ok=True)
atomic_jsonl(out / "requests.jsonl", sorted(records, key=lambda r: r["admitted_s"]))
atomic_jsonl(out / "segments.jsonl", segments)
atomic_json(out / "result.json", result)
atomic_json(out / "sanity.json", sanity)
if ctx.exhausted:
raise ManifestExhausted("manifest exhausted; result retained for diagnosis")
failed = [name for name, ok in sanity["invariants"].items() if not ok]
if failed:
raise RuntimeError(f"client sanity failure: {failed}")
return result
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser()
sub = parser.add_subparsers(dest="command", required=True)
mat = sub.add_parser("materialize")
mat.add_argument("--id", required=True)
mat.add_argument("--kind", choices=("synthetic", "prefix-pool"), required=True)
group = mat.add_mutually_exclusive_group()
group.add_argument("--input-uniform")
group.add_argument("--input-fixed", type=int)
group.add_argument("--input-mixture")
mat.add_argument("--output-fixed", type=int, required=True)
mat.add_argument("--prefix", default="none")
mat.add_argument("--arrival", required=True)
mat.add_argument("--num-requests", type=int, required=True)
mat.add_argument("--workload-seed", type=int, required=True)
mat.add_argument("--num-prefixes", type=int, default=0)
mat.add_argument("--prefix-len", type=int, default=0)
mat.add_argument("--suffix-fixed", type=int, default=0)
mat.add_argument("--out", required=True)
private = sub.add_parser("materialize-private")
private.add_argument("--id", required=True)
private.add_argument("--source", required=True)
private.add_argument("--sampling-u-max", type=float, required=True)
private.add_argument("--max-input-tokens", type=int, required=True)
private.add_argument("--output-cap", type=int, required=True)
private.add_argument("--preserve-prompts", action="store_true", required=True)
private.add_argument("--disable-shuffle", action="store_true", required=True)
private.add_argument("--arrival", required=True)
private.add_argument("--model", required=True)
private.add_argument("--out", required=True)
run = sub.add_parser("run")
run.add_argument("--manifest", required=True)
run.add_argument("--base-url", required=True)
run.add_argument("--model", required=True)
run.add_argument("--load-point", choices=("saturation", "moderate"), required=True)
run.add_argument("--request-rate")
run.add_argument("--saturation-result")
run.add_argument("--rate-fraction", type=float, default=0.60)
run.add_argument("--max-concurrency", type=int, default=256)
run.add_argument("--ignore-eos", action="store_true")
run.add_argument("--temperature", type=float, default=0.0)
run.add_argument("--warmup-seconds", type=float, default=60)
run.add_argument("--clean-segment-seconds", type=float, default=80)
run.add_argument("--num-clean-segments", type=int, default=3)
run.add_argument("--profile-after-clean", action="store_true")
run.add_argument("--num-profile-windows", type=int, default=0)
run.add_argument("--profile-warmup-iterations", type=int, default=2)
run.add_argument("--profile-active-iterations", type=int, default=8)
run.add_argument("--profile-trace-dir")
run.add_argument("--profile-timeout-seconds", type=float, default=120)
run.add_argument("--recovery-seconds", type=float, default=30)
run.add_argument("--post-clean-seconds", type=float, default=0)
run.add_argument("--drain-timeout-seconds", type=float, default=120)
run.add_argument("--workload-seed", type=int, default=20260712)
run.add_argument("--server-seed", type=int, default=20260712)
run.add_argument("--result-dir", required=True)
return parser
def main() -> None:
args = build_parser().parse_args()
if args.command == "materialize":
materialize(args)
elif args.command == "materialize-private":
materialize_private(args)
else:
if args.profile_after_clean and not args.profile_trace_dir:
raise ValueError("--profile-after-clean requires --profile-trace-dir")
print(json.dumps(asyncio.run(run_load(args)), sort_keys=True))
if __name__ == "__main__":
main()

1056
runs/opprof-phase3/provenance/opprof_phase3_controller.py Normal file
View File

File diff suppressed because it is too large Load Diff

1045
runs/opprof-phase3/provenance/opprof_phase3_controller_ea.py Normal file
View File

File diff suppressed because it is too large Load Diff

1252
runs/opprof-phase3/provenance/opprof_phase3_matrix.py Normal file
View File

File diff suppressed because it is too large Load Diff

109
runs/opprof-phase3/provenance/test_phase3_analysis.py Normal file
View File

@@ -0,0 +1,109 @@
from __future__ import annotations
import json
import tempfile
import unittest
from pathlib import Path
import analyze_phase3 as analysis
class Phase3AnalysisTests(unittest.TestCase):
def test_ap36_stability_formula(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
(root / "client").mkdir()
(root / "opprof").mkdir()
(root / "client/result.json").write_text(
json.dumps({"t0_mono_ns": 0, "warmup_seconds": 60})
)
(root / "client/requests.jsonl").write_text(
"".join(
json.dumps({"success": True, "completed_s": index + 1}) + "\n"
for index in range(16)
)
)
records = []
for bin_index in range(3):
for step in range(16):
records.append(
{
"step_index": len(records),
"model_executed": True,
"submit_mono_ns": int(
(45 + 5 * bin_index + (step + 0.5) / 16 * 5)
* 1e9
),
"prefill_tokens": 100,
"decode_tokens": 0,
}
)
(root / "opprof/test.jsonl").write_text(
"".join(json.dumps(item) + "\n" for item in records)
)
result = analysis.ap36_warmup_stability(root)
self.assertTrue(result["passes"])
self.assertEqual(result["normalized_drift"], 0)
def test_ap37_partial_verdict_can_confirm_but_not_refute(self):
self.assertEqual(analysis.partial_verdict(True), "PASS")
self.assertEqual(analysis.partial_verdict(False), "INCONCLUSIVE")
def test_accepted_markers_come_only_from_complete_stages(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
complete = root / "stages/primary-01-saturation"
complete.mkdir(parents=True)
(complete / "stage-complete.json").write_text(
json.dumps({"runs": ["P01-C00-saturation"]})
)
accepted = root / "primary/P01-C00/saturation"
accepted.mkdir(parents=True)
(accepted / "run-complete.json").write_text(
json.dumps({"run_id": "P01-C00-saturation"})
)
unaccepted = root / "primary/P05-C00/saturation"
unaccepted.mkdir(parents=True)
(unaccepted / "run-complete.json").write_text(
json.dumps({"run_id": "P05-C00-saturation"})
)
primary, confirmations, stages, excluded = analysis.accepted_marker_paths(
root
)
self.assertEqual(primary, [accepted / "run-complete.json"])
self.assertEqual(confirmations, [])
self.assertEqual(stages, [complete])
self.assertEqual(excluded, ["P05-C00-saturation"])
def test_r64_is_ratio_of_cohort_sums(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "manifest.jsonl"
rows = [{"input_tokens": value} for value in (1, 3, 2, 2)]
path.write_text("".join(json.dumps(row) + "\n" for row in rows))
value, pieces = analysis.manifest_raggedness(path, 2)
self.assertEqual(pieces, [(2.0, 6.0), (0.0, 4.0)])
self.assertAlmostEqual(value, 0.2)
def test_one_percentage_point_ranking_ties(self):
shares = dict.fromkeys(analysis.FAMILIES, 0.0)
shares.update(attention=0.40, moe_gemm=0.395, moe_router=0.20)
ranked = {item["family"]: item["rank"] for item in analysis.ranked_families(shares)}
self.assertEqual(ranked["attention"], ranked["moe_gemm"])
self.assertGreater(ranked["moe_router"], ranked["attention"])
def test_holm_uses_declared_total_test_family(self):
values = [{"p": 0.001}, {"p": 0.01}]
analysis.holm(values, total_tests=10)
self.assertAlmostEqual(values[0]["p_holm"], 0.01)
self.assertAlmostEqual(values[1]["p_holm"], 0.09)
def test_robust_spline_prediction_is_nonnegative(self):
rows = [(float(x), float(n), float(2 * x + n)) for x in range(1, 20) for n in (1, 4)]
predict, hull = analysis.fit_nonnegative_robust(rows)
self.assertGreaterEqual(predict(3, 2), 0)
self.assertTrue(analysis.inside_convex(hull, (3, 2)))
self.assertFalse(analysis.inside_convex(hull, (100, 2)))
if __name__ == "__main__":
unittest.main()

486
runs/opprof-phase3/provenance/test_phase3_tools.py Normal file
View File

@@ -0,0 +1,486 @@
#!/usr/bin/env python3
from __future__ import annotations
import asyncio
import importlib.util
import json
import sys
import tempfile
import unittest
from unittest import mock
from pathlib import Path
from types import SimpleNamespace
from aiohttp import web
HERE = Path(__file__).parent
def load_module(name: str, filename: str):
spec = importlib.util.spec_from_file_location(name, HERE / filename)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
assert spec.loader is not None
spec.loader.exec_module(module)
return module
client = load_module("phase3_client", "opprof_phase3_client.py")
controller = load_module("phase3_controller", "opprof_phase3_controller.py")
sys.modules["opprof_phase3_controller"] = controller
matrix = load_module("phase3_matrix", "opprof_phase3_matrix.py")
def rows(n: int, arrival: str = "steady") -> list[dict]:
return [
{
"schema": 1,
"request_id": f"T-{i:05d}",
"pattern_id": "T",
"kind": "synthetic",
"input_tokens": 8,
"output_tokens": 2,
"arrival": arrival,
"token_seed": i + 1,
}
for i in range(n)
]
class MockServer:
def __init__(self, delay: float = 0.01, trace_dir: Path | None = None) -> None:
self.active = 0
self.max_active = 0
self.payloads = []
self.runner = None
self.port = None
self.delay = delay
self.trace_dir = trace_dir
self.profile_count = 0
async def completion(self, request):
payload = await request.json()
self.payloads.append(payload)
self.active += 1
self.max_active = max(self.max_active, self.active)
await asyncio.sleep(self.delay)
response = web.StreamResponse(
status=200, headers={"Content-Type": "text/event-stream"}
)
await response.prepare(request)
await response.write(
b'data: {"choices":[{"text":"x"}],"usage":null}\n\n'
)
usage = json.dumps(
{
"choices": [],
"usage": {
"prompt_tokens": len(payload["prompt"]),
"completion_tokens": payload["max_tokens"],
},
}
).encode()
await response.write(b"data: " + usage + b"\n\n")
await response.write(b"data: [DONE]\n\n")
await response.write_eof()
self.active -= 1
return response
async def start_profile(self, request):
self.profile_count += 1
path = self.trace_dir / f"window-{self.profile_count}.pt.trace.json"
path.write_text('{"traceEvents": []}')
return web.Response(status=200)
async def stop_profile(self, request):
return web.Response(status=200)
async def start(self):
app = web.Application()
app.router.add_post("/v1/completions", self.completion)
if self.trace_dir is not None:
app.router.add_post("/start_profile", self.start_profile)
app.router.add_post("/stop_profile", self.stop_profile)
self.runner = web.AppRunner(app)
await self.runner.setup()
site = web.TCPSite(self.runner, "127.0.0.1", 0)
await site.start()
self.port = site._server.sockets[0].getsockname()[1]
async def stop(self):
await self.runner.cleanup()
def run_args(
manifest: Path,
result_dir: Path,
port: int,
load_point: str = "saturation",
saturation_result: Path | None = None,
):
return SimpleNamespace(
manifest=str(manifest),
base_url=f"http://127.0.0.1:{port}",
model="mock",
load_point=load_point,
request_rate="inf" if load_point == "saturation" else None,
saturation_result=None if saturation_result is None else str(saturation_result),
rate_fraction=0.60,
max_concurrency=3,
ignore_eos=True,
temperature=0,
warmup_seconds=0.04,
clean_segment_seconds=0.04,
num_clean_segments=3,
profile_after_clean=False,
num_profile_windows=0,
profile_warmup_iterations=2,
profile_active_iterations=8,
profile_trace_dir=None,
profile_timeout_seconds=1,
recovery_seconds=0,
post_clean_seconds=0,
drain_timeout_seconds=1,
workload_seed=20260712,
server_seed=20260712,
result_dir=str(result_dir),
)
class Phase3ToolTests(unittest.TestCase):
def write_warmup_stream(self, root: Path, tokens: list[int]) -> None:
stream_dir = root / "opprof"
stream_dir.mkdir()
records = []
step = 0
for bin_index, token_count in enumerate(tokens):
per_step, remainder = divmod(token_count, 16)
for in_bin in range(16):
records.append(
{
"step_index": step,
"model_executed": True,
"submit_mono_ns": int(
1e9
+ (45 + bin_index * 5 + (in_bin + 0.5) / 16 * 5)
* 1e9
),
"prefill_tokens": per_step + (in_bin < remainder),
"decode_tokens": 0,
}
)
step += 1
(stream_dir / "test.jsonl").write_text(
"".join(json.dumps(record) + "\n" for record in records)
)
def test_p10_warmup_stability_accepts_flat_trailing_quartile(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self.write_warmup_stream(root, [1600, 1600, 1600])
result = matrix.p10_warmup_stability(root, int(1e9))
self.assertTrue(result["passed"])
self.assertEqual(result["step_counts"], [16, 16, 16])
self.assertEqual(result["scheduled_token_throughput"], [320, 320, 320])
self.assertEqual(result["normalized_drift"], 0)
def test_p10_warmup_stability_rejects_large_drift(self):
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
self.write_warmup_stream(root, [1600, 3200, 4800])
result = matrix.p10_warmup_stability(root, int(1e9))
self.assertFalse(result["passed"])
self.assertGreater(result["normalized_drift"], 0.10)
def test_only_clean_window_request_failures_are_hard_failures(self):
requests = [
{"success": False, "completed_s": 59.9, "error_kind": "warmup"},
{"success": False, "completed_s": 60.0, "error_kind": "clean"},
{"success": False, "completed_s": 299.9, "error_kind": "clean"},
{"success": False, "completed_s": 300.0, "error_kind": "recovery"},
{"success": True, "completed_s": 100.0, "error_kind": None},
]
summary = matrix.summarize_request_failures(requests, 60.0, 300.0)
self.assertEqual(summary["failed"], 4)
self.assertEqual(summary["clean_failed"], 2)
self.assertEqual(summary["excluded"], 2)
self.assertEqual(summary["excluded_kinds"], {"warmup": 1, "recovery": 1})
def test_synthetic_prompt_exact_and_unique_early_prefix(self):
generated = [client.synthetic_prompt(row) for row in rows(200)]
self.assertTrue(all(len(value) == 8 for value in generated))
self.assertEqual(len({tuple(value[:3]) for value in generated}), 200)
def test_p05_manifest_has_exact_half_modes(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "P05.jsonl"
args = SimpleNamespace(
id="P05",
kind="synthetic",
input_uniform=None,
input_fixed=None,
input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}',
output_fixed=64,
prefix="none",
arrival="steady",
num_requests=100,
workload_seed=20260712,
num_prefixes=0,
prefix_len=0,
suffix_fixed=0,
out=str(out),
)
client.materialize(args)
manifest = client.load_manifest(out)
self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50)
self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50)
def test_prefix_pool_balanced(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "P08.jsonl"
args = SimpleNamespace(
id="P08",
kind="prefix-pool",
input_uniform=None,
input_fixed=None,
input_mixture=None,
output_fixed=512,
prefix="none",
arrival="burst:8",
num_requests=80,
workload_seed=20260712,
num_prefixes=8,
prefix_len=1024,
suffix_fixed=256,
out=str(out),
)
client.materialize(args)
manifest = client.load_manifest(out)
counts = {i: 0 for i in range(8)}
for row in manifest:
counts[row["prefix_id"]] += 1
self.assertEqual(row["input_tokens"], 1280)
self.assertEqual(set(counts.values()), {10})
def test_fixed_duration_saturation_and_redaction(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000))
mock = MockServer()
await mock.start()
try:
result = await client.run_load(
run_args(manifest, root / "result", mock.port)
)
finally:
await mock.stop()
self.assertEqual(result["max_in_flight"], 3)
self.assertEqual(mock.max_active, 3)
self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9)
self.assertEqual(len(result["segments"]), 3)
self.assertLessEqual(result["drain_seconds"], 1)
self.assertTrue(
all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads)
)
text = (root / "result/requests.jsonl").read_text()
self.assertNotIn('"prompt":', text)
self.assertNotIn('"text":', text)
asyncio.run(case())
def test_profile_control_plane_is_not_starved_by_data_connector(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
trace_dir = root / "traces"
trace_dir.mkdir()
client.atomic_jsonl(manifest, rows(1000))
server = MockServer(delay=0.5, trace_dir=trace_dir)
await server.start()
args = run_args(manifest, root / "result", server.port)
args.max_concurrency = 1
args.warmup_seconds = 0.01
args.clean_segment_seconds = 0.01
args.num_clean_segments = 1
args.profile_after_clean = True
args.num_profile_windows = 1
args.profile_trace_dir = str(trace_dir)
args.recovery_seconds = 0
try:
result = await client.run_load(args)
finally:
await server.stop()
self.assertEqual(server.max_active, 1)
self.assertEqual(server.profile_count, 1)
self.assertLess(result["profiles"][0]["start_return_s"], 0.25)
asyncio.run(case())
def test_drain_timeout_is_a_hard_sanity_failure(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000))
mock = MockServer()
await mock.start()
try:
args = run_args(manifest, root / "result", mock.port)
args.drain_timeout_seconds = 0
with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"):
await client.run_load(args)
finally:
await mock.stop()
sanity = json.loads((root / "result/sanity.json").read_text())
self.assertFalse(sanity["invariants"]["drain_within_timeout"])
asyncio.run(case())
def test_burst_schedule_is_eight_at_once(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000, "burst:8"))
sat = root / "sat.json"
client.atomic_json(
sat, {"clean": {"completed_throughput_rps": 100.0}}
)
mock = MockServer()
await mock.start()
try:
args = run_args(
manifest, root / "result", mock.port, "moderate", sat
)
args.warmup_seconds = 0
args.clean_segment_seconds = 0.4 / 3
result = await client.run_load(args)
finally:
await mock.stop()
records = [
json.loads(line)
for line in (root / "result/requests.jsonl").read_text().splitlines()
]
groups = {}
for record in records:
groups.setdefault(round(record["scheduled_s"], 6), 0)
groups[round(record["scheduled_s"], 6)] += 1
self.assertTrue(all(value == 8 for value in groups.values()))
starts = sorted(groups)
if len(starts) > 1:
self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5)
self.assertAlmostEqual(result["request_rate"], 60.0)
asyncio.run(case())
def test_manifest_exhaustion_stops_instead_of_wrapping(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(2))
mock = MockServer()
await mock.start()
try:
with self.assertRaises(client.ManifestExhausted):
await client.run_load(
run_args(manifest, root / "result", mock.port)
)
finally:
await mock.stop()
asyncio.run(case())
def test_cpu_affinity_sets_are_disjoint_and_cover_host(self):
values = []
for spec in controller.CPU_MAP.values():
lo, hi = [int(x) for x in spec.split("-")]
values.extend(range(lo, hi + 1))
self.assertEqual(sorted(values), list(range(160)))
self.assertEqual(len(values), len(set(values)))
def test_preflight_waits_for_three_stable_zero_samples(self):
def sample(memory):
return [
{
"index": 0,
"uuid": "GPU-test",
"memory_used_mib": memory,
"utilization_pct": 0,
}
]
with tempfile.TemporaryDirectory() as tmp:
with (
mock.patch.object(
controller,
"gpu_query",
side_effect=[sample(4), sample(0), sample(0), sample(0)],
),
mock.patch.object(controller, "compute_apps", return_value=[]),
mock.patch.object(controller, "run_text", return_value=""),
mock.patch.object(controller.time, "sleep"),
):
root = Path(tmp)
controller.preflight([0], root)
samples = json.loads((root / "gpu-before-samples.json").read_text())
self.assertEqual(len(samples), 4)
self.assertEqual(samples[0]["gpus"][0]["memory_used_mib"], 4)
self.assertEqual(samples[-1]["gpus"][0]["memory_used_mib"], 0)
def test_kernel_mapping_priority(self):
cases = {
"void vllm::moe::topkGating": "moe_router",
"ncclDevKernel_AllReduce": "collective",
"flash_fwd_kernel": "attention",
"nvjet_sm90_tst": "moe_gemm",
"argmax_kernel": "sampler",
"cutlass_gemm": "dense_gemm",
"triton_red_fused_add_rms_norm": "norm_elementwise",
"cache_swap_kernel": "kv_memory",
"unknown": "other",
}
self.assertEqual(
{name: controller.classify_kernel(name) for name in cases}, cases
)
def test_atomic_state_replacement(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "state.json"
controller.atomic_json(path, {"schema": 1, "value": 1})
controller.atomic_json(path, {"schema": 1, "value": 2})
self.assertEqual(json.loads(path.read_text())["value"], 2)
self.assertFalse(list(path.parent.glob("*.tmp.*")))
def test_fingerprint_survives_json_roundtrip(self):
original_run_text = controller.run_text
original_hash = controller.sha256_file
try:
controller.run_text = lambda *args, **kwargs: "deadbeef\n"
controller.sha256_file = lambda path: "a" * 64
fingerprint = controller.make_fingerprint()
finally:
controller.run_text = original_run_text
controller.sha256_file = original_hash
self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint)
self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)})
def test_server_shutdown_signals_parent_before_group(self):
process = SimpleNamespace(pid=12345, poll=lambda: None)
with (
mock.patch.object(controller.os, "kill") as kill,
mock.patch.object(controller.os, "killpg") as killpg,
mock.patch.object(controller, "_process_group_alive", return_value=False),
):
controller.stop_servers([process])
kill.assert_called_once_with(12345, controller.signal.SIGINT)
killpg.assert_not_called()
if __name__ == "__main__":
unittest.main(verbosity=2)

353
runs/opprof-phase3/provenance/test_phase3_tools_ea.py Normal file
View File

@@ -0,0 +1,353 @@
#!/usr/bin/env python3
from __future__ import annotations
import asyncio
import importlib.util
import json
import sys
import tempfile
import unittest
from unittest import mock
from pathlib import Path
from types import SimpleNamespace
from aiohttp import web
HERE = Path(__file__).parent
def load_module(name: str, filename: str):
spec = importlib.util.spec_from_file_location(name, HERE / filename)
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
assert spec.loader is not None
spec.loader.exec_module(module)
return module
client = load_module("phase3_client", "opprof_phase3_client.py")
controller = load_module("phase3_controller", "opprof_phase3_controller.py")
def rows(n: int, arrival: str = "steady") -> list[dict]:
return [
{
"schema": 1,
"request_id": f"T-{i:05d}",
"pattern_id": "T",
"kind": "synthetic",
"input_tokens": 8,
"output_tokens": 2,
"arrival": arrival,
"token_seed": i + 1,
}
for i in range(n)
]
class MockServer:
def __init__(self) -> None:
self.active = 0
self.max_active = 0
self.payloads = []
self.runner = None
self.port = None
async def completion(self, request):
payload = await request.json()
self.payloads.append(payload)
self.active += 1
self.max_active = max(self.max_active, self.active)
await asyncio.sleep(0.01)
response = web.StreamResponse(
status=200, headers={"Content-Type": "text/event-stream"}
)
await response.prepare(request)
await response.write(
b'data: {"choices":[{"text":"x"}],"usage":null}\n\n'
)
usage = json.dumps(
{
"choices": [],
"usage": {
"prompt_tokens": len(payload["prompt"]),
"completion_tokens": payload["max_tokens"],
},
}
).encode()
await response.write(b"data: " + usage + b"\n\n")
await response.write(b"data: [DONE]\n\n")
await response.write_eof()
self.active -= 1
return response
async def start(self):
app = web.Application()
app.router.add_post("/v1/completions", self.completion)
self.runner = web.AppRunner(app)
await self.runner.setup()
site = web.TCPSite(self.runner, "127.0.0.1", 0)
await site.start()
self.port = site._server.sockets[0].getsockname()[1]
async def stop(self):
await self.runner.cleanup()
def run_args(
manifest: Path,
result_dir: Path,
port: int,
load_point: str = "saturation",
saturation_result: Path | None = None,
):
return SimpleNamespace(
manifest=str(manifest),
base_url=f"http://127.0.0.1:{port}",
model="mock",
load_point=load_point,
request_rate="inf" if load_point == "saturation" else None,
saturation_result=None if saturation_result is None else str(saturation_result),
rate_fraction=0.60,
max_concurrency=3,
ignore_eos=True,
temperature=0,
warmup_seconds=0.04,
clean_segment_seconds=0.04,
num_clean_segments=3,
profile_after_clean=False,
num_profile_windows=0,
profile_warmup_iterations=2,
profile_active_iterations=8,
profile_trace_dir=None,
profile_timeout_seconds=1,
recovery_seconds=0,
post_clean_seconds=0,
drain_timeout_seconds=1,
workload_seed=20260712,
server_seed=20260712,
result_dir=str(result_dir),
)
class Phase3ToolTests(unittest.TestCase):
def test_synthetic_prompt_exact_and_unique_early_prefix(self):
generated = [client.synthetic_prompt(row) for row in rows(200)]
self.assertTrue(all(len(value) == 8 for value in generated))
self.assertEqual(len({tuple(value[:3]) for value in generated}), 200)
def test_p05_manifest_has_exact_half_modes(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "P05.jsonl"
args = SimpleNamespace(
id="P05",
kind="synthetic",
input_uniform=None,
input_fixed=None,
input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}',
output_fixed=64,
prefix="none",
arrival="steady",
num_requests=100,
workload_seed=20260712,
num_prefixes=0,
prefix_len=0,
suffix_fixed=0,
out=str(out),
)
client.materialize(args)
manifest = client.load_manifest(out)
self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50)
self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50)
def test_prefix_pool_balanced(self):
with tempfile.TemporaryDirectory() as tmp:
out = Path(tmp) / "P08.jsonl"
args = SimpleNamespace(
id="P08",
kind="prefix-pool",
input_uniform=None,
input_fixed=None,
input_mixture=None,
output_fixed=512,
prefix="none",
arrival="burst:8",
num_requests=80,
workload_seed=20260712,
num_prefixes=8,
prefix_len=1024,
suffix_fixed=256,
out=str(out),
)
client.materialize(args)
manifest = client.load_manifest(out)
counts = {i: 0 for i in range(8)}
for row in manifest:
counts[row["prefix_id"]] += 1
self.assertEqual(row["input_tokens"], 1280)
self.assertEqual(set(counts.values()), {10})
def test_fixed_duration_saturation_and_redaction(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000))
mock = MockServer()
await mock.start()
try:
result = await client.run_load(
run_args(manifest, root / "result", mock.port)
)
finally:
await mock.stop()
self.assertEqual(result["max_in_flight"], 3)
self.assertEqual(mock.max_active, 3)
self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9)
self.assertEqual(len(result["segments"]), 3)
self.assertLessEqual(result["drain_seconds"], 1)
self.assertTrue(
all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads)
)
text = (root / "result/requests.jsonl").read_text()
self.assertNotIn('"prompt":', text)
self.assertNotIn('"text":', text)
asyncio.run(case())
def test_drain_timeout_is_a_hard_sanity_failure(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000))
mock = MockServer()
await mock.start()
try:
args = run_args(manifest, root / "result", mock.port)
args.drain_timeout_seconds = 0
with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"):
await client.run_load(args)
finally:
await mock.stop()
sanity = json.loads((root / "result/sanity.json").read_text())
self.assertFalse(sanity["invariants"]["drain_within_timeout"])
asyncio.run(case())
def test_burst_schedule_is_eight_at_once(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(1000, "burst:8"))
sat = root / "sat.json"
client.atomic_json(
sat, {"clean": {"completed_throughput_rps": 100.0}}
)
mock = MockServer()
await mock.start()
try:
args = run_args(
manifest, root / "result", mock.port, "moderate", sat
)
args.warmup_seconds = 0
args.clean_segment_seconds = 0.4 / 3
result = await client.run_load(args)
finally:
await mock.stop()
records = [
json.loads(line)
for line in (root / "result/requests.jsonl").read_text().splitlines()
]
groups = {}
for record in records:
groups.setdefault(round(record["scheduled_s"], 6), 0)
groups[round(record["scheduled_s"], 6)] += 1
self.assertTrue(all(value == 8 for value in groups.values()))
starts = sorted(groups)
if len(starts) > 1:
self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5)
self.assertAlmostEqual(result["request_rate"], 60.0)
asyncio.run(case())
def test_manifest_exhaustion_stops_instead_of_wrapping(self):
async def case():
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
manifest = root / "m.jsonl"
client.atomic_jsonl(manifest, rows(2))
mock = MockServer()
await mock.start()
try:
with self.assertRaises(client.ManifestExhausted):
await client.run_load(
run_args(manifest, root / "result", mock.port)
)
finally:
await mock.stop()
asyncio.run(case())
def test_cpu_affinity_sets_are_disjoint_and_cover_host(self):
values = []
for spec in controller.CPU_MAP.values():
lo, hi = [int(x) for x in spec.split("-")]
values.extend(range(lo, hi + 1))
self.assertEqual(sorted(values), list(range(160)))
self.assertEqual(len(values), len(set(values)))
def test_kernel_mapping_priority(self):
cases = {
"void vllm::moe::topkGating": "moe_router",
"ncclDevKernel_AllReduce": "collective",
"flash_fwd_kernel": "attention",
"nvjet_sm90_tst": "moe_gemm",
"argmax_kernel": "sampler",
"cutlass_gemm": "dense_gemm",
"triton_red_fused_add_rms_norm": "norm_elementwise",
"cache_swap_kernel": "kv_memory",
"unknown": "other",
}
self.assertEqual(
{name: controller.classify_kernel(name) for name in cases}, cases
)
def test_atomic_state_replacement(self):
with tempfile.TemporaryDirectory() as tmp:
path = Path(tmp) / "state.json"
controller.atomic_json(path, {"schema": 1, "value": 1})
controller.atomic_json(path, {"schema": 1, "value": 2})
self.assertEqual(json.loads(path.read_text())["value"], 2)
self.assertFalse(list(path.parent.glob("*.tmp.*")))
def test_fingerprint_survives_json_roundtrip(self):
original_run_text = controller.run_text
original_hash = controller.sha256_file
try:
controller.run_text = lambda *args, **kwargs: "deadbeef\n"
controller.sha256_file = lambda path: "a" * 64
fingerprint = controller.make_fingerprint()
finally:
controller.run_text = original_run_text
controller.sha256_file = original_hash
self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint)
self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)})
def test_server_shutdown_signals_parent_before_group(self):
process = SimpleNamespace(pid=12345, poll=lambda: None)
with (
mock.patch.object(controller.os, "kill") as kill,
mock.patch.object(controller.os, "killpg") as killpg,
mock.patch.object(controller, "_process_group_alive", return_value=False),
):
controller.stop_servers([process])
kill.assert_called_once_with(12345, controller.signal.SIGINT)
killpg.assert_not_called()
if __name__ == "__main__":
unittest.main(verbosity=2)

353
runs/opprof-phase3/provenance/verify_shutdown_footer.py Normal file
View File

@@ -0,0 +1,353 @@
#!/usr/bin/env python3
"""One-GPU verification of API-parent-first OpProf shutdown."""
from __future__ import annotations
import json
import os
import shlex
import shutil
import signal
import subprocess
import time
import urllib.request
from pathlib import Path
from typing import Any
WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
RUN_DIR = WORKDIR / "runs/e-b-shutdown-verification"
SOURCE = Path(
"/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0"
)
VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
MANIFEST = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl")
PORT = 8010
def atomic_json(path: Path, value: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_name(path.name + f".tmp.{os.getpid()}")
with tmp.open("w", encoding="utf-8") as f:
json.dump(value, f, sort_keys=True, indent=2)
f.write("\n")
f.flush()
os.fsync(f.fileno())
os.replace(tmp, path)
def run_text(command: list[str], check: bool = True) -> str:
result = subprocess.run(
command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT
)
if check and result.returncode:
raise RuntimeError(
f"command failed ({result.returncode}): {shlex.join(command)}\n"
f"{result.stdout}"
)
return result.stdout
def gpu_rows() -> list[dict[str, int]]:
output = run_text(
[
"nvidia-smi",
"--query-gpu=index,memory.used,utilization.gpu",
"--format=csv,noheader,nounits",
]
)
rows = []
for line in output.strip().splitlines():
index, memory, utilization = [int(part.strip()) for part in line.split(",")]
rows.append(
{"index": index, "memory_mib": memory, "utilization_pct": utilization}
)
return rows
def compute_apps() -> str:
return run_text(
[
"nvidia-smi",
"--query-compute-apps=gpu_uuid,pid,process_name,used_memory",
"--format=csv,noheader,nounits",
],
check=False,
).strip()
def group_alive(pgid: int, process: subprocess.Popen[Any] | None = None) -> bool:
if process is not None:
process.poll() # reap an exited API parent before probing its group
try:
os.killpg(pgid, 0)
return True
except ProcessLookupError:
return False
def wait_ready(process: subprocess.Popen[Any], timeout: float = 300) -> None:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if process.poll() is not None:
raise RuntimeError("server exited before readiness")
try:
with urllib.request.urlopen(
f"http://127.0.0.1:{PORT}/health", timeout=1
) as response:
if response.status == 200:
return
except Exception:
pass
time.sleep(1)
raise TimeoutError("server readiness timeout")
def shutdown_parent_first(process: subprocess.Popen[Any]) -> dict[str, Any]:
called_at = time.time()
os.kill(process.pid, signal.SIGINT)
deadline = time.monotonic() + 120
while time.monotonic() < deadline and group_alive(process.pid, process):
time.sleep(1)
graceful = not group_alive(process.pid, process)
if not graceful:
for sig in (signal.SIGTERM, signal.SIGKILL):
try:
os.killpg(process.pid, sig)
except ProcessLookupError:
break
time.sleep(5)
return {
"api_parent_pid": process.pid,
"sigint_wall_time": called_at,
"group_gone_wall_time": time.time(),
"graceful": graceful,
"returncode": process.poll(),
}
def wait_stable_zero() -> list[list[dict[str, int]]]:
samples = []
consecutive = 0
deadline = time.monotonic() + 120
while time.monotonic() < deadline and consecutive < 3:
rows = gpu_rows()
samples.append(rows)
zero = rows[0]["memory_mib"] == 0 and not compute_apps()
consecutive = consecutive + 1 if zero else 0
if consecutive < 3:
time.sleep(5)
if consecutive < 3:
raise RuntimeError("GPU0 did not reach three stable zero samples")
return samples
def validate_footer() -> dict[str, Any]:
files = sorted((RUN_DIR / "opprof").glob("*.jsonl"))
if len(files) != 1:
raise RuntimeError(f"expected one OpProf JSONL, found {len(files)}")
lines = files[0].read_text().splitlines()
decoded = [json.loads(line) for line in lines]
if not decoded or decoded[-1].get("record_type") != "footer":
raise RuntimeError("Layer-1 footer missing")
records, footer = decoded[:-1], decoded[-1]
indices = [record["step_index"] for record in records]
invariants = {
"all_schema_1": all(item.get("schema") == 1 for item in decoded),
"one_footer_last": sum(
item.get("record_type") == "footer" for item in decoded
)
== 1,
"steps_contiguous": indices == list(range(len(indices))),
"written_matches_records": footer["written_records"] == len(records),
"encoded_balanced": footer["encoded_records"]
== footer["written_records"] + footer["dropped_records"],
"zero_drops": footer["dropped_records"] == 0
and all(record["dropped_records_before"] == 0 for record in records),
}
if not all(invariants.values()):
raise RuntimeError(f"footer accounting invalid: {invariants}, {footer}")
return {
"path": str(files[0]),
"bytes": files[0].stat().st_size,
"records": len(records),
"footer": footer,
"invariants": invariants,
}
def main() -> None:
if RUN_DIR.exists():
raise RuntimeError(f"refusing to overwrite {RUN_DIR}")
RUN_DIR.mkdir(parents=True)
state: dict[str, Any] = {
"schema": 1,
"status": "preflight",
"started_at": time.time(),
"gpu": 0,
}
atomic_json(RUN_DIR / "state.json", state)
before = gpu_rows()
if before[0]["memory_mib"] != 0 or before[0]["utilization_pct"] != 0:
raise RuntimeError(f"GPU0 is not idle: {before[0]}")
if compute_apps():
raise RuntimeError("a compute process exists before verification")
(RUN_DIR / "gpu-before.json").write_text(json.dumps(before, indent=2) + "\n")
(RUN_DIR / "clocks-before.txt").write_text(
run_text(["nvidia-smi", "-q", "-d", "CLOCK"])
)
profile_config = {
"profiler": "torch",
"torch_profiler_dir": "/tmp/wjh-opprof-p3-footer-verify",
"ignore_frontend": True,
"wait_iterations": 0,
"warmup_iterations": 2,
"active_iterations": 8,
}
trace_dir = Path(profile_config["torch_profiler_dir"])
if trace_dir.exists():
shutil.rmtree(trace_dir)
trace_dir.mkdir()
server_command = [
"taskset",
"-c",
"0-19",
str(VENV / "bin/vllm"),
"serve",
str(MODEL),
"--host",
"127.0.0.1",
"--port",
str(PORT),
"--tensor-parallel-size",
"1",
"--enable-chunked-prefill",
"--enable-prefix-caching",
"--profiler-config",
json.dumps(profile_config, separators=(",", ":")),
]
client_command = [
"taskset",
"-c",
"0-19",
str(VENV / "bin/python"),
str(CLIENT),
"run",
"--manifest",
str(MANIFEST),
"--base-url",
f"http://127.0.0.1:{PORT}",
"--model",
str(MODEL),
"--load-point",
"saturation",
"--request-rate",
"inf",
"--max-concurrency",
"256",
"--ignore-eos",
"--temperature",
"0",
"--warmup-seconds",
"30",
"--clean-segment-seconds",
"40",
"--num-clean-segments",
"3",
"--drain-timeout-seconds",
"120",
"--workload-seed",
"20260712",
"--result-dir",
str(RUN_DIR / "client"),
]
with (RUN_DIR / "commands.log").open("w") as f:
f.write(
"GPU_COMMAND shutdown footer server: "
+ shlex.join(server_command)
+ " ; expected=60-180s startup + 150-180s load\n"
)
f.write(
"GPU_COMMAND shutdown footer client: "
+ shlex.join(client_command)
+ " ; expected=30s warmup + 120s clean + drain\n"
)
server_log = (RUN_DIR / "server.log").open("ab", buffering=0)
env = os.environ.copy()
env.update(
{
"CUDA_VISIBLE_DEVICES": "0",
"VLLM_OPPROF_DIR": str(RUN_DIR / "opprof"),
"HF_HUB_OFFLINE": "1",
"TRANSFORMERS_OFFLINE": "1",
"PYTHONUNBUFFERED": "1",
}
)
server: subprocess.Popen[Any] | None = None
try:
print(
"GPU_COMMAND shutdown-fix verification: P01/C00 GPU0, "
"30s warmup + 120s clean, expected 4-6 wall-min",
flush=True,
)
server = subprocess.Popen(
server_command,
cwd=SOURCE,
env=env,
stdout=server_log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
state.update({"status": "server_starting", "server_pid": server.pid})
atomic_json(RUN_DIR / "state.json", state)
wait_ready(server)
state.update({"status": "client_running", "server_ready_at": time.time()})
atomic_json(RUN_DIR / "state.json", state)
with (RUN_DIR / "client.log").open("ab", buffering=0) as client_log:
result = subprocess.run(
client_command,
cwd=WORKDIR,
stdout=client_log,
stderr=subprocess.STDOUT,
)
if result.returncode:
raise RuntimeError(f"client exited {result.returncode}")
state["status"] = "parent_first_shutdown"
atomic_json(RUN_DIR / "state.json", state)
shutdown = shutdown_parent_first(server)
if not shutdown["graceful"]:
raise RuntimeError(f"server required escalation: {shutdown}")
footer = validate_footer()
zero_samples = wait_stable_zero()
client_result = json.loads((RUN_DIR / "client/result.json").read_text())
result_json = {
"schema": 1,
"status": "pass",
"gpu": 0,
"shutdown": shutdown,
"footer": footer,
"clean": client_result["clean"],
"failed_records": client_result["failed_records"],
"gpu_zero_samples": zero_samples,
"gpu_seconds": zero_samples and time.time() - state["started_at"],
}
atomic_json(RUN_DIR / "result.json", result_json)
state.update({"status": "complete", "completed_at": time.time()})
atomic_json(RUN_DIR / "state.json", state)
print(json.dumps(result_json, sort_keys=True), flush=True)
except Exception as error:
state.update({"status": "failed", "failure": repr(error)})
atomic_json(RUN_DIR / "state.json", state)
if server is not None and group_alive(server.pid, server):
try:
os.killpg(server.pid, signal.SIGKILL)
except ProcessLookupError:
pass
raise
finally:
server_log.close()
if __name__ == "__main__":
main()

392
runs/opprof-phase3/provenance/verify_sidecar_shutdown.py Normal file
View File

@@ -0,0 +1,392 @@
#!/usr/bin/env python3
"""GPU verification of graceful and hard-kill OpProf accounting."""
from __future__ import annotations
import json
import os
import shlex
import signal
import subprocess
import time
import urllib.request
from pathlib import Path
from typing import Any
WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
RUN_ROOT = WORKDIR / "runs/e-b-sidecar-verification"
SOURCE = Path(
"/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0"
)
VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
MANIFEST = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl")
FLUSH_INTERVAL_SECONDS = 1.0
CHECKPOINT_TOLERANCE_SECONDS = 0.1
def atomic_json(path: Path, value: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_name(f"{path.name}.tmp-{os.getpid()}")
with temporary.open("w", encoding="utf-8") as output:
json.dump(value, output, indent=2, sort_keys=True)
output.write("\n")
output.flush()
os.fsync(output.fileno())
os.replace(temporary, path)
def run_text(command: list[str], check: bool = True) -> str:
result = subprocess.run(
command,
text=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
)
if check and result.returncode:
raise RuntimeError(
f"command failed ({result.returncode}): {shlex.join(command)}\n"
f"{result.stdout}"
)
return result.stdout
def gpu_rows() -> list[dict[str, int]]:
output = run_text(
[
"nvidia-smi",
"--query-gpu=index,memory.used,utilization.gpu",
"--format=csv,noheader,nounits",
]
)
rows = []
for line in output.strip().splitlines():
index, memory, utilization = (int(value.strip()) for value in line.split(","))
rows.append(
{
"index": index,
"memory_mib": memory,
"utilization_pct": utilization,
}
)
return rows
def compute_apps() -> str:
return run_text(
[
"nvidia-smi",
"--query-compute-apps=gpu_uuid,pid,process_name,used_memory",
"--format=csv,noheader,nounits",
],
check=False,
).strip()
def assert_idle() -> list[dict[str, int]]:
rows = gpu_rows()
if any(row["memory_mib"] or row["utilization_pct"] for row in rows):
raise RuntimeError(f"dash0 is not GPU-idle: {rows}")
applications = compute_apps()
if applications:
raise RuntimeError(f"compute applications present: {applications}")
return rows
def wait_ready(process: subprocess.Popen[Any], port: int) -> None:
deadline = time.monotonic() + 300
while time.monotonic() < deadline:
if process.poll() is not None:
raise RuntimeError("server exited before readiness")
try:
with urllib.request.urlopen(
f"http://127.0.0.1:{port}/health", timeout=1
) as response:
if response.status == 200:
return
except Exception:
pass
time.sleep(1)
raise TimeoutError("server readiness timeout")
def wait_zero() -> tuple[list[list[dict[str, int]]], float]:
samples: list[list[dict[str, int]]] = []
consecutive = 0
deadline = time.monotonic() + 180
while time.monotonic() < deadline and consecutive < 3:
rows = gpu_rows()
samples.append(rows)
zero = all(row["memory_mib"] == 0 for row in rows) and not compute_apps()
consecutive = consecutive + 1 if zero else 0
if consecutive < 3:
time.sleep(2)
if consecutive < 3:
raise RuntimeError("GPUs did not reach three stable zero samples")
return samples, time.time()
def wait_fresh_sidecar(run_dir: Path) -> dict[str, Any]:
deadline = time.monotonic() + 10
while time.monotonic() < deadline:
files = sorted((run_dir / "opprof").glob("*.jsonl.footer.json"))
if len(files) == 1:
sidecar = json.loads(files[0].read_text())
age = time.time_ns() - sidecar["checkpoint_wall_ns"]
if 0 <= age <= 250_000_000:
return sidecar
time.sleep(0.02)
raise TimeoutError("could not observe a fresh OpProf sidecar")
def validate_accounting(
run_dir: Path, mode: str, termination_wall_ns: int
) -> dict[str, Any]:
streams = sorted((run_dir / "opprof").glob("*.jsonl"))
sidecars = sorted((run_dir / "opprof").glob("*.jsonl.footer.json"))
if len(streams) != 1 or len(sidecars) != 1:
raise RuntimeError(
f"expected one stream/sidecar, got {len(streams)}/{len(sidecars)}"
)
raw = streams[0].read_bytes()
if not raw.endswith(b"\n"):
raise RuntimeError("partial final JSONL line")
decoded = [json.loads(line) for line in raw.splitlines()]
footers = [row for row in decoded if row.get("record_type") == "footer"]
records = [row for row in decoded if row.get("record_type") != "footer"]
sidecar = json.loads(sidecars[0].read_text())
indices = [row["step_index"] for row in records]
checkpoint_age = (
termination_wall_ns - sidecar["checkpoint_wall_ns"]
) / 1e9
common = {
"all_schema_1": all(row.get("schema") == 1 for row in decoded)
and sidecar.get("schema") == 1,
"steps_contiguous": indices == list(range(len(indices))),
"written_matches_records": sidecar["written_records"] == len(records),
"encoded_balanced": sidecar["encoded_records"]
== sidecar["written_records"] + sidecar["dropped_records"],
"last_step_matches": bool(records)
and sidecar["last_step_index"] == records[-1]["step_index"],
"zero_drops": sidecar["dropped_records"] == 0
and all(row["dropped_records_before"] == 0 for row in records),
}
if mode == "graceful":
footer_ok = len(footers) == 1 and decoded[-1] is footers[0]
agreement = footer_ok and all(
footers[0][counter] == sidecar[counter]
for counter in ("encoded_records", "written_records", "dropped_records")
)
specific = {
"one_footer_last": footer_ok,
"final_sidecar": sidecar["final"] is True,
"footer_sidecar_agree": agreement,
}
else:
specific = {
"no_in_stream_footer": not footers,
"checkpoint_sidecar": sidecar["final"] is False,
"checkpoint_within_bound": checkpoint_age
<= FLUSH_INTERVAL_SECONDS + CHECKPOINT_TOLERANCE_SECONDS,
}
invariants = {**common, **specific}
if not all(invariants.values()):
raise RuntimeError(
f"{mode} accounting invalid: {invariants}; sidecar={sidecar}"
)
return {
"stream": str(streams[0]),
"sidecar": str(sidecars[0]),
"bytes": len(raw),
"records": len(records),
"footer_count": len(footers),
"checkpoint_age_seconds": checkpoint_age,
"counters": {
key: sidecar[key]
for key in ("encoded_records", "written_records", "dropped_records")
},
"last_step_index": sidecar["last_step_index"],
"invariants": invariants,
}
def run_trial(mode: str, port: int) -> dict[str, Any]:
run_dir = RUN_ROOT / mode
if run_dir.exists():
raise RuntimeError(f"refusing to overwrite {run_dir}")
run_dir.mkdir(parents=True)
before = assert_idle()
atomic_json(run_dir / "gpu-before.json", before)
(run_dir / "clocks-before.txt").write_text(
run_text(["nvidia-smi", "-q", "-d", "CLOCK"])
)
server_command = [
"taskset",
"-c",
"0-19",
str(VENV / "bin/vllm"),
"serve",
str(MODEL),
"--host",
"127.0.0.1",
"--port",
str(port),
"--tensor-parallel-size",
"1",
"--enable-chunked-prefill",
"--enable-prefix-caching",
"--shutdown-timeout",
"120",
]
client_command = [
"taskset",
"-c",
"0-19",
str(VENV / "bin/python"),
str(CLIENT),
"run",
"--manifest",
str(MANIFEST),
"--base-url",
f"http://127.0.0.1:{port}",
"--model",
str(MODEL),
"--load-point",
"saturation",
"--request-rate",
"inf",
"--max-concurrency",
"256",
"--ignore-eos",
"--temperature",
"0",
"--warmup-seconds",
"20",
"--clean-segment-seconds",
"40",
"--num-clean-segments",
"3",
"--drain-timeout-seconds",
"120",
"--workload-seed",
"20260712",
"--result-dir",
str(run_dir / "client"),
]
with (run_dir / "commands.log").open("w", encoding="utf-8") as output:
output.write(
f"GPU_COMMAND {mode} server: {shlex.join(server_command)}; "
"expected=60-180s startup + 140-180s load\n"
)
output.write(
f"GPU_COMMAND {mode} client: {shlex.join(client_command)}; "
"expected=20s warmup + 120s clean + drain\n"
)
environment = os.environ.copy()
environment.update(
{
"CUDA_VISIBLE_DEVICES": "0",
"VLLM_OPPROF_DIR": str(run_dir / "opprof"),
"HF_HUB_OFFLINE": "1",
"TRANSFORMERS_OFFLINE": "1",
"PYTHONUNBUFFERED": "1",
}
)
started = time.time()
server_log = (run_dir / "server.log").open("ab", buffering=0)
server: subprocess.Popen[Any] | None = None
try:
print(
f"GPU_COMMAND sidecar-{mode}: P01/C00 GPU0, 20s warmup + "
"120s clean, expected 4-6 wall-min",
flush=True,
)
server = subprocess.Popen(
server_command,
cwd=SOURCE,
env=environment,
stdout=server_log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
atomic_json(
run_dir / "state.json",
{"status": "server_starting", "server_pid": server.pid},
)
wait_ready(server, port)
with (run_dir / "client.log").open("ab", buffering=0) as client_log:
client = subprocess.run(
client_command,
cwd=WORKDIR,
stdout=client_log,
stderr=subprocess.STDOUT,
)
if client.returncode:
raise RuntimeError(f"client exited {client.returncode}")
if mode == "graceful":
termination_wall_ns = time.time_ns()
os.kill(server.pid, signal.SIGINT)
server.wait(timeout=150)
log_text = (run_dir / "server.log").read_text(errors="replace")
if "mode=drain timeout=120s" not in log_text:
raise RuntimeError("official drain-mode log not observed")
else:
wait_fresh_sidecar(run_dir)
termination_wall_ns = time.time_ns()
os.killpg(server.pid, signal.SIGKILL)
server.wait(timeout=30)
zero_samples, zero_at = wait_zero()
accounting = validate_accounting(run_dir, mode, termination_wall_ns)
client_result = json.loads((run_dir / "client/result.json").read_text())
result = {
"schema": 1,
"status": "pass",
"mode": mode,
"server_returncode": server.returncode,
"termination_wall_ns": termination_wall_ns,
"accounting": accounting,
"clean": client_result["clean"],
"failed_records": client_result["failed_records"],
"gpu_zero_samples": zero_samples,
"gpu_seconds": zero_at - started,
}
atomic_json(run_dir / "result.json", result)
atomic_json(run_dir / "state.json", {"status": "complete"})
return result
except Exception as error:
atomic_json(
run_dir / "state.json",
{"status": "failed", "failure": repr(error)},
)
if server is not None and server.poll() is None:
try:
os.killpg(server.pid, signal.SIGKILL)
except ProcessLookupError:
pass
wait_zero()
raise
finally:
server_log.close()
def main() -> None:
if RUN_ROOT.exists():
raise RuntimeError(f"refusing to overwrite {RUN_ROOT}")
RUN_ROOT.mkdir(parents=True)
state: dict[str, Any] = {"schema": 1, "status": "running", "results": {}}
atomic_json(RUN_ROOT / "state.json", state)
for offset, mode in enumerate(("graceful", "hard-kill")):
result = run_trial(mode, 8010 + offset)
state["results"][mode] = result
atomic_json(RUN_ROOT / "state.json", state)
state["status"] = "complete"
state["gpu_seconds"] = sum(
result["gpu_seconds"] for result in state["results"].values()
)
atomic_json(RUN_ROOT / "state.json", state)
print(json.dumps(state, sort_keys=True), flush=True)
if __name__ == "__main__":
main()

537
runs/opprof-phase5/analyze_phase5.py Normal file
View File

@@ -0,0 +1,537 @@
#!/usr/bin/env python3
"""Frozen Phase-5 bridge/control ledger analysis."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
from pathlib import Path
from typing import Any
import numpy as np
SEED = 20260716
RESAMPLES = 100_000
RATE = 0.4725
ARMS = ("base", "A1", "A2", "A3", "A4")
MECHANISMS = ("A1", "A2", "A3", "A4")
CAPTURE_SIZES = {
1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88,
96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192,
200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336,
352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512,
}
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def numeric(values: list[float | int | None]) -> dict[str, Any]:
finite = [float(value) for value in values if value is not None and math.isfinite(float(value))]
return {
"n": len(values),
"finite_n": len(finite),
"missing_n": len(values) - len(finite),
"min": min(finite) if finite else None,
"max": max(finite) if finite else None,
"distinct_n": len(set(finite)),
}
def ci(draws: np.ndarray) -> list[float]:
low, high = np.quantile(draws, [0.025, 0.975])
return [float(low), float(high)]
def cv(values: list[float]) -> float:
array = np.asarray(values, dtype=np.float64)
mean = float(array.mean())
if mean == 0:
return 0.0 if np.all(array == 0) else math.inf
return float(array.std(ddof=0) / mean)
def parse_run(run_dir: Path) -> dict[str, Any]:
result = json.loads((run_dir / "client/result.json").read_text())
complete = json.loads((run_dir / "run-complete.json").read_text())
t0 = int(result["t0_mono_ns"])
clean_start = float(result["clean"]["start_s"])
clean_end = float(result["clean"]["end_s"])
stream = next((run_dir / "opprof").glob("*.jsonl"))
records = []
for line in stream.read_text().splitlines():
item = json.loads(line)
if "step_index" not in item:
continue
relative = (int(item["submit_mono_ns"]) - t0) / 1e9
if clean_start <= relative < clean_end:
item["relative_s"] = relative
records.append(item)
blocks = []
decode_means = []
waiting_means = []
for block_index in range(48):
start = clean_start + 5 * block_index
selected = [item for item in records if start <= item["relative_s"] < start + 5]
if not selected:
# Fixed-time blocks are the resampling unit. A rate-following trace
# can legitimately have an idle block, which contributes no tokens
# and no model-step time but must remain in the arrival-variance
# distribution.
blocks.append([0, 0.0])
decode_means.append(0.0)
waiting_means.append(0.0)
continue
tokens = sum(int(item["prefill_tokens"]) + int(item["decode_tokens"]) for item in selected)
duration = sum(
(int(item["complete_mono_ns"]) - int(item["submit_mono_ns"])) / 1e6
for item in selected
)
blocks.append([tokens, duration])
decode_means.append(float(np.mean([int(item["decode_batch_size"]) for item in selected])))
waiting_means.append(float(np.mean([int(item["queues"]["waiting"]) for item in selected])))
pure_decode = [
item for item in records
if int(item["prefill_tokens"]) == 0 and int(item["decode_batch_size"]) > 0
]
pure_bucket = sum(int(item["cudagraph"]["bucket_tokens"]) for item in pure_decode)
pure_padding = sum(int(item["cudagraph"]["padding_tokens"]) for item in pure_decode)
support = sorted({int(item["decode_batch_size"]) for item in pure_decode})
covered = sum(int(item["decode_batch_size"]) in CAPTURE_SIZES for item in pure_decode)
prefix_queries = 0
prefix_hits = 0
prefix_present = 0
for item in records:
local = item.get("prefix", {}).get("local")
if local is None:
continue
prefix_present += 1
prefix_queries += int(local.get("queries", 0))
prefix_hits += int(local.get("hits", 0))
block_array = np.asarray(blocks, dtype=np.float64)
return {
"run_id": complete["run_id"],
"run_dir": str(run_dir),
"stream_sha256": sha256_file(stream),
"blocks": block_array,
"token_efficiency_per_ms": float(block_array[:, 0].sum() / block_array[:, 1].sum()),
"clean_steps": len(records),
"clean_duration_s": clean_end - clean_start,
"clean_failed": int(result["clean"]["failed"]),
"offered_rps": float(result["clean"]["offered_rps"]),
"drain_seconds": float(result["drain_seconds"]),
"warmup_completions": int(complete["client"]["warmup_completions"]),
"warmup_gate_branch": complete["client"].get("warmup_gate_branch", "P3-pre-A-P5-1"),
"warmup_stability": complete["client"].get("warmup_stability"),
"cold_start_gate": complete["client"].get("cold_start_gate"),
"decode_B_block_cv": cv(decode_means),
"waiting_block_cv": cv(waiting_means),
"pure_decode_steps": len(pure_decode),
"pure_decode_support": support,
"pure_decode_support_coverage": covered / len(pure_decode),
"pure_decode_padding_tokens": pure_padding,
"pure_decode_bucket_tokens": pure_bucket,
"pure_decode_padding_fraction": pure_padding / pure_bucket,
"prefix_present_steps": prefix_present,
"prefix_queries": prefix_queries,
"prefix_hits": prefix_hits,
"prefix_hit_ratio": prefix_hits / prefix_queries if prefix_queries else 0.0,
"layer1_invariants": complete["layer1"]["invariants"],
"client_invariants": complete["client"]["invariants"],
"server_invariants": complete["server_invariants"],
"drain_quarantined": bool(complete["drain_quarantined"]),
}
def hierarchical_draws(runs: list[dict[str, Any]], rng: np.random.Generator) -> np.ndarray:
count = len(runs)
output = np.empty(RESAMPLES, dtype=np.float64)
for start in range(0, RESAMPLES, 5000):
size = min(5000, RESAMPLES - start)
token_sum = np.zeros(size, dtype=np.float64)
duration_sum = np.zeros(size, dtype=np.float64)
selected_runs = rng.integers(0, count, size=(size, count))
for position in range(count):
block_indices = rng.integers(0, 48, size=(size, 48))
for run_index, run in enumerate(runs):
mask = selected_runs[:, position] == run_index
if not np.any(mask):
continue
blocks = run["blocks"]
choices = block_indices[mask]
token_sum[mask] += blocks[choices, 0].sum(axis=1)
duration_sum[mask] += blocks[choices, 1].sum(axis=1)
output[start : start + size] = token_sum / duration_sum
return output
def point_efficiency(runs: list[dict[str, Any]]) -> float:
tokens = sum(float(run["blocks"][:, 0].sum()) for run in runs)
duration = sum(float(run["blocks"][:, 1].sum()) for run in runs)
return tokens / duration
def two_sided_p(draws: np.ndarray) -> float:
return float(min(1.0, 2 * min(np.mean(draws <= 0), np.mean(draws >= 0))))
def holm(pvalues: dict[str, float]) -> dict[str, float]:
ordered = sorted(pvalues, key=pvalues.get)
adjusted: dict[str, float] = {}
running = 0.0
total = len(ordered)
for rank, key in enumerate(ordered):
running = max(running, (total - rank) * pvalues[key])
adjusted[key] = min(1.0, running)
return adjusted
def discover_primary(root: Path) -> dict[str, list[dict[str, Any]]]:
result = {arm: [] for arm in ARMS}
for marker in sorted((root / "primary").glob("*/moderate/run-complete.json")):
if "background" in str(marker):
continue
parsed = parse_run(marker.parent)
arm = parsed["run_id"].split("-r", 1)[0]
if arm in result:
result[arm].append(parsed)
if any(len(result[arm]) != 3 for arm in ARMS):
raise RuntimeError(f"primary replication mismatch: { {key:len(value) for key,value in result.items()} }")
return result
def control_runs(root: Path, p3root: Path, pattern: str) -> tuple[list[dict[str, Any]], str]:
fresh = sorted((root / "primary").glob(f"control-{pattern}-r*-C00/moderate/run-complete.json"))
if fresh:
if len(fresh) != 3:
raise RuntimeError(f"partial fresh control set: {pattern}: {len(fresh)}")
return [parse_run(path.parent) for path in fresh], "P5-rerun"
return [parse_run(p3root / f"primary/{pattern}-C00/moderate")], "P3-reused"
def aggregate_aux(runs: list[dict[str, Any]], key: str) -> float:
return float(np.mean([float(run[key]) for run in runs]))
def analyze(root: Path, p3root: Path, private: Path) -> dict[str, Any]:
primary = discover_primary(root)
p3_base = [parse_run(p3root / "primary/P10-C00/moderate")]
controls = {}
control_source = {}
for pattern in ("P03", "P04"):
controls[pattern], control_source[pattern] = control_runs(root, p3root, pattern)
rng = np.random.default_rng(SEED)
draws = {arm: hierarchical_draws(primary[arm], rng) for arm in ARMS}
p3_base_draws = hierarchical_draws(p3_base, rng)
control_draws = {pattern: hierarchical_draws(controls[pattern], rng) for pattern in controls}
points = {arm: point_efficiency(primary[arm]) for arm in ARMS}
p3_base_point = point_efficiency(p3_base)
control_points = {pattern: point_efficiency(controls[pattern]) for pattern in controls}
bridge_draws = draws["A3"] - p3_base_draws
bridge_point = points["A3"] - p3_base_point
bridge_ci = ci(bridge_draws)
bridge = {
"point_delta": bridge_point,
"relative_abs_delta": abs(bridge_point) / p3_base_point,
"ci95": bridge_ci,
"within_3pct": abs(bridge_point) / p3_base_point <= 0.03,
"ci_contains_zero": bridge_ci[0] <= 0 <= bridge_ci[1],
}
bridge["reuse_passed"] = bridge["within_3pct"] and bridge["ci_contains_zero"]
deltas = {arm: draws[arm] - draws["base"] for arm in MECHANISMS}
raw_p = {arm: two_sided_p(deltas[arm]) for arm in MECHANISMS}
holm_p = holm(raw_p)
manifests = {
name: json.loads((private / f"P10-{name}.jsonl.summary.json").read_text())
for name in ("base", "A1", "A3")
}
base_prefix = aggregate_aux(primary["base"], "prefix_hit_ratio")
a1_prefix = aggregate_aux(primary["A1"], "prefix_hit_ratio")
base_padding = aggregate_aux(primary["base"], "pure_decode_padding_fraction")
a2_padding = aggregate_aux(primary["A2"], "pure_decode_padding_fraction")
padding_reduction = (base_padding - a2_padding) / base_padding if base_padding > 0 else 0.0
manipulations = {
"A1": {
"passed": (
manifests["base"]["r16"] - manifests["A1"]["r16"] >= 0.15
and (manifests["base"]["r16"] - manifests["A1"]["r16"]) / manifests["base"]["r16"] >= 0.20
and manifests["A1"]["max_added_delay_seconds"] <= 64
and abs(a1_prefix - base_prefix) <= 0.01
),
"base_R16": manifests["base"]["r16"],
"ablated_R16": manifests["A1"]["r16"],
"prefix_hit_ratio_delta": a1_prefix - base_prefix,
"max_added_delay_seconds": manifests["A1"]["max_added_delay_seconds"],
},
"A2": {
"passed": aggregate_aux(primary["A2"], "pure_decode_support_coverage") >= 0.99 and padding_reduction >= 0.90,
"support_coverage": aggregate_aux(primary["A2"], "pure_decode_support_coverage"),
"base_padding_fraction": base_padding,
"ablated_padding_fraction": a2_padding,
"padding_reduction": padding_reduction,
"observed_support": sorted({value for run in primary["A2"] for value in run["pure_decode_support"]}),
},
"A3": {
"passed": (
aggregate_aux(primary["A3"], "decode_B_block_cv") < aggregate_aux(primary["base"], "decode_B_block_cv")
and aggregate_aux(primary["A3"], "waiting_block_cv") < aggregate_aux(primary["base"], "waiting_block_cv")
),
"base_decode_B_cv": aggregate_aux(primary["base"], "decode_B_block_cv"),
"ablated_decode_B_cv": aggregate_aux(primary["A3"], "decode_B_block_cv"),
"base_waiting_cv": aggregate_aux(primary["base"], "waiting_block_cv"),
"ablated_waiting_cv": aggregate_aux(primary["A3"], "waiting_block_cv"),
},
"A4": {
"passed": sum(run["prefix_queries"] for run in primary["A4"]) == 0 and sum(run["prefix_hits"] for run in primary["A4"]) == 0,
"prefix_queries": sum(run["prefix_queries"] for run in primary["A4"]),
"prefix_hits": sum(run["prefix_hits"] for run in primary["A4"]),
},
}
ledgers = {}
for control in ("P03", "P04"):
denominator = control_draws[control] - draws["base"]
point_denominator = control_points[control] - points["base"]
denominator_ci = ci(denominator)
stable_denominator = bool(
point_denominator > 0
and denominator_ci[0] > 0
and np.mean(denominator <= 0) <= 0.05
)
rows = {}
share_draws = {}
for arm in MECHANISMS:
share_draws[arm] = deltas[arm] / denominator
point = (points[arm] - points["base"]) / point_denominator
interval = ci(share_draws[arm])
reportable = stable_denominator and manipulations[arm]["passed"]
rows[arm] = {
"delta_E": points[arm] - points["base"],
"delta_E_ci95": ci(deltas[arm]),
"share": point if reportable else None,
"share_ci95": interval if reportable else None,
"diagnostic_share": point,
"diagnostic_share_ci95": interval,
"share_status": (
"REPORTABLE"
if reportable
else (
"N/A—unstable control denominator"
if not stable_denominator
else "N/A—manipulation failed"
)
),
"p_two_sided": raw_p[arm],
"p_holm": holm_p[arm],
"manipulation_passed": manipulations[arm]["passed"],
}
residual_draws = 1.0 - sum(share_draws.values())
residual_point = 1.0 - sum(row["diagnostic_share"] for row in rows.values())
ledgers[control] = {
"status": "EVALUABLE" if stable_denominator else "INCONCLUSIVE—unstable denominator",
"control_source": control_source[control],
"control_E": control_points[control],
"base_E": points["base"],
"gap_E": point_denominator,
"gap_ci95": denominator_ci,
"denominator_nonpositive_fraction": float(np.mean(denominator <= 0)),
"stable_denominator": stable_denominator,
"mechanisms": rows,
"diagnostic_share_sum": sum(row["diagnostic_share"] for row in rows.values()),
"residual_interaction": residual_point,
"residual_interaction_ci95": ci(residual_draws),
"residual_status": (
"REPORTABLE"
if stable_denominator and all(item["passed"] for item in manipulations.values())
else "DIAGNOSTIC_ONLY—incomplete official share ledger"
),
}
dominance = {}
for arm in MECHANISMS:
per_control = {}
for control in ("P03", "P04"):
row = ledgers[control]["mechanisms"][arm]
if not ledgers[control]["stable_denominator"] or not row["manipulation_passed"]:
per_control[control] = "NOT_EVALUABLE"
continue
direction_ok = row["delta_E"] > 0 if arm != "A4" else True
per_control[control] = (
row["share"] >= 0.30
and row["share_ci95"][0] > 0.15
and row["p_holm"] < 0.05
and direction_ok
and row["manipulation_passed"]
)
dominance[arm] = {
"per_control": per_control,
"verdict": (
"NOT EVALUABLE"
if any(value == "NOT_EVALUABLE" for value in per_control.values())
else (
"DOMINANT"
if all(per_control.values())
else ("CONTROL-SENSITIVE" if any(per_control.values()) else "NOT DOMINANT")
)
),
}
all_runs = [run for arm in ARMS for run in primary[arm]]
share_widths = [
ledgers[control]["mechanisms"][arm]["diagnostic_share_ci95"][1]
- ledgers[control]["mechanisms"][arm]["diagnostic_share_ci95"][0]
for control in ("P03", "P04") for arm in MECHANISMS
]
state = json.loads((root / "controller-state.json").read_text())
amendment_evidence_path = root / "a-p5-1-retained-audit.jsonl"
amendment_evidence = []
if amendment_evidence_path.exists():
amendment_evidence = [
json.loads(line) for line in amendment_evidence_path.read_text().splitlines()
]
invariants = {
"primary_runs_15": len(all_runs) == 15,
"three_replicates_per_arm": all(len(primary[arm]) == 3 for arm in ARMS),
"clean_duration_240": all(math.isclose(run["clean_duration_s"], 240.0) for run in all_runs),
"clean_failures_zero": all(run["clean_failed"] == 0 for run in all_runs),
"offered_rate_within_5pct": all(abs(run["offered_rps"] / RATE - 1) <= 0.05 for run in all_runs),
"layer1_accounting": all(all(run["layer1_invariants"].values()) for run in all_runs),
"client_invariants": all(all(value for key, value in run["client_invariants"].items() if key != "drain_re_adjudicated") for run in all_runs),
"server_invariants": all(all(run["server_invariants"].values()) for run in all_runs),
"a_p5_1_cold_start_gates": all(
run["cold_start_gate"] is not None
and run["cold_start_gate"]["passed"]
for run in all_runs
),
"drain_quarantine_under_20pct": sum(run["drain_quarantined"] for run in all_runs) / 15 <= 0.20,
"gpu_budget_below_6": float(state["gpu_hours_total"]) < 6.0,
"manifests_same_ids": len({manifests[name]["request_id_set_sha256"] for name in manifests}) == 1,
"manifests_same_token_sums": len({manifests[name]["input_tokens"]["sum"] for name in manifests}) == 1 and len({manifests[name]["output_tokens"]["sum"] for name in manifests}) == 1,
"control_denominators_stable": all(ledgers[control]["stable_denominator"] for control in ledgers),
"bridge_decision_resolved": bridge["reuse_passed"] or all(source == "P5-rerun" for source in control_source.values()),
"ratios_finite": all(math.isfinite(ledgers[c]["mechanisms"][a]["diagnostic_share"]) for c in ledgers for a in MECHANISMS),
"per_arm_results_not_all_identical": len({round(points[arm], 12) for arm in ARMS}) > 1,
}
red_flags = [key for key, value in invariants.items() if not value]
publishable = (
not red_flags
and all(item["passed"] for item in manipulations.values())
and sum(width > 0.50 for width in share_widths) < 2
)
return {
"schema": 1,
"status": "PUBLISHABLE" if publishable else "INCONCLUSIVE_OR_PARTIAL",
"limitation": "Recorded-arrival P5 bridge ledger anchored to P3 controls; not a literal decomposition of P3's already-uniform P10 gap.",
"analysis_seed": SEED,
"bootstrap_resamples": RESAMPLES,
"efficiency": {
arm: {
"point": points[arm],
"ci95": ci(draws[arm]),
"runs": [
{key: value for key, value in run.items() if key not in {"blocks", "warmup_stability"}}
for run in primary[arm]
],
}
for arm in ARMS
},
"p3_base_E": p3_base_point,
"bridge": bridge,
"control_sources": control_source,
"manipulations": manipulations,
"holm": {"family": list(MECHANISMS), "raw_p": raw_p, "adjusted_p": holm_p},
"ledgers": ledgers,
"dominance": dominance,
"config_tier_A2": {
"delta_E": points["A2"] - points["base"],
"relative_E_delta": points["A2"] / points["base"] - 1,
"delta_E_ci95": ci(deltas["A2"]),
"base_padding_fraction": base_padding,
"A2_padding_fraction": a2_padding,
"padding_reduction": padding_reduction,
},
"amendment_A_P5_1": {
"reason": "Rate-following throughput drift tracks arrival shape and is not a cold-start stationarity test.",
"retained_failed_run_evidence": amendment_evidence,
"recorded_drift_range": (
[
min(
item["superseded_drift_evidence"]["normalized_drift"]
for item in amendment_evidence
if item["run"] != "A3-r1-C00"
),
max(
item["superseded_drift_evidence"]["normalized_drift"]
for item in amendment_evidence
if item["run"] != "A3-r1-C00"
),
]
if amendment_evidence else None
),
"uniform_A3_drift": (
next(
item["superseded_drift_evidence"]["normalized_drift"]
for item in amendment_evidence
if item["run"] == "A3-r1-C00"
)
if amendment_evidence else None
),
},
"gpu": {
"new_h20_hours": float(state["gpu_hours_total"]),
"hard_cap": 6.0,
"controller_status": state["status"],
"completed_measured_runs_including_background": state["completed_measured_runs"],
"completed_burnins": state["completed_burnins"],
},
"sanity": {
"red_flags": red_flags,
"invariants": invariants,
"numeric": {
"primary_E": numeric([run["token_efficiency_per_ms"] for run in all_runs]),
"clean_steps": numeric([run["clean_steps"] for run in all_runs]),
"offered_rps": numeric([run["offered_rps"] for run in all_runs]),
"drain_seconds": numeric([run["drain_seconds"] for run in all_runs]),
"diagnostic_share": numeric([ledgers[c]["mechanisms"][a]["diagnostic_share"] for c in ledgers for a in MECHANISMS]),
"residual_interaction": numeric([ledgers[c]["residual_interaction"] for c in ledgers]),
"share_ci_width": numeric(share_widths),
},
"declared": {
"manipulation_failures": [arm for arm, item in manipulations.items() if not item["passed"]],
"control_sources": control_source,
"bridge_reuse_passed": bridge["reuse_passed"],
},
},
}
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--root", type=Path, required=True)
parser.add_argument("--p3-root", type=Path, required=True)
parser.add_argument("--private", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
result = analyze(args.root, args.p3_root, args.private)
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
print(json.dumps({
"status": result["status"],
"bridge": result["bridge"],
"red_flags": result["sanity"]["red_flags"],
"gpu": result["gpu"],
}, sort_keys=True))
if __name__ == "__main__":
main()

242
runs/opprof-phase5/opprof_phase5_client.py Normal file
View File

@@ -0,0 +1,242 @@
#!/usr/bin/env python3
"""Phase-5 private-manifest transforms and timestamp-scheduled P3 client wrapper."""
from __future__ import annotations
import argparse
import asyncio
import json
import math
import sys
from pathlib import Path
from typing import Any
import aiohttp
import opprof_phase3_client as p3
def _numeric(values: list[float | int]) -> dict[str, Any]:
finite = [float(value) for value in values if math.isfinite(float(value))]
return {
"n": len(values),
"finite_n": len(finite),
"missing_n": len(values) - len(finite),
"min": min(finite) if finite else None,
"max": max(finite) if finite else None,
"distinct_n": len(set(finite)),
"sum": sum(finite),
}
def _r16(rows: list[dict[str, Any]]) -> float:
groups = [rows[index : index + 16] for index in range(0, len(rows) - 15, 16)]
useful = sum(sum(int(row["input_tokens"]) for row in group) for group in groups)
rectangular = sum(16 * max(int(row["input_tokens"]) for row in group) for group in groups)
return 1.0 - useful / rectangular
def _source_timestamps(path: Path, indices: set[int], field: str) -> dict[int, float]:
result: dict[int, float] = {}
maximum = max(indices)
with path.open(encoding="utf-8") as source:
for index, line in enumerate(source):
if index in indices:
value = float(json.loads(line)[field])
if not math.isfinite(value):
raise ValueError(f"non-finite source timestamp at {index}")
result[index] = value
if index >= maximum:
break
if set(result) != indices:
raise ValueError("timestamp source did not cover all source_index values")
return result
def transform(args: argparse.Namespace) -> dict[str, Any]:
rows = p3.load_manifest(Path(args.input))[: args.take_first]
if len(rows) != args.take_first:
raise ValueError(f"requested {args.take_first} rows, found {len(rows)}")
indices = [int(row[args.join_key]) for row in rows]
timestamps = _source_timestamps(
Path(args.timestamp_source), set(indices), args.timestamp_field
)
source_times = [timestamps[index] for index in indices]
if any(right < left for left, right in zip(source_times, source_times[1:])):
raise ValueError("selected timestamps are not nondecreasing")
if source_times[-1] <= source_times[0]:
raise ValueError("selected timestamp span is not positive")
end_s = (len(rows) - 1) / args.target_rate
scale = end_s / (source_times[-1] - source_times[0])
recorded_slots = [(value - source_times[0]) * scale for value in source_times]
uniform_slots = [index / args.target_rate for index in range(len(rows))]
slots = recorded_slots if args.arrival == "recorded-scaled" else uniform_slots
for index, row in enumerate(rows):
row["arrival"] = args.arrival
row["arrival_s"] = slots[index]
row["original_index"] = index
row["source_timestamp"] = source_times[index]
original = list(rows)
max_added_delay = 0.0
if args.service_order == "length-binned":
edges = [int(item) for item in args.length_bin_edges.split(",")]
def bin_id(row: dict[str, Any]) -> int:
length = int(row["input_tokens"])
for index, edge in enumerate(edges):
if length <= edge:
return index
raise ValueError(f"input length {length} exceeds final edge")
reordered: list[dict[str, Any]] = []
for offset in range(0, len(rows), args.reorder_block_size):
block = rows[offset : offset + args.reorder_block_size]
ordered = sorted(
block,
key=lambda row: (
bin_id(row),
int(row["input_tokens"]),
int(row["original_index"]),
),
)
block_slots = slots[offset : offset + len(block)]
for position, row in enumerate(ordered):
added = max(0.0, block_slots[position] - float(row["arrival_s"]))
max_added_delay = max(max_added_delay, added)
row["arrival_s"] = block_slots[position]
reordered.extend(ordered)
rows = reordered
if max_added_delay > args.max_added_delay_seconds + 1e-9:
raise ValueError(
f"fairness cap exceeded: {max_added_delay} > {args.max_added_delay_seconds}"
)
elif args.service_order != "original":
raise ValueError(f"unsupported service order: {args.service_order}")
if sorted(row["request_id"] for row in rows) != sorted(
row["request_id"] for row in original
):
raise AssertionError("request identity changed")
for key in ("input_tokens", "output_tokens"):
if sum(int(row[key]) for row in rows) != sum(int(row[key]) for row in original):
raise AssertionError(f"{key} total changed")
arrival_values = [float(row["arrival_s"]) for row in rows]
if any(right < left for left, right in zip(arrival_values, arrival_values[1:])):
raise AssertionError("assigned arrival slots are not nondecreasing")
output = Path(args.out)
p3.atomic_jsonl(output, rows, mode=0o600)
summary = {
"schema": 1,
"path": str(output),
"sha256": p3.sha256_file(output),
"rows": len(rows),
"arrival": args.arrival,
"service_order": args.service_order,
"target_rate": args.target_rate,
"input_tokens": _numeric([int(row["input_tokens"]) for row in rows]),
"output_tokens": _numeric([int(row["output_tokens"]) for row in rows]),
"arrival_s": _numeric(arrival_values),
"source_timestamp": _numeric(source_times),
"r16": _r16(rows),
"max_added_delay_seconds": max_added_delay,
"request_id_set_sha256": p3.hashlib.sha256(
"\n".join(sorted(str(row["request_id"]) for row in rows)).encode()
).hexdigest(),
"invariants": {
"same_request_ids": True,
"same_input_tokens": True,
"same_output_tokens": True,
"arrival_nondecreasing": True,
"fairness_cap": max_added_delay <= args.max_added_delay_seconds + 1e-9,
"no_prompt_in_summary": True,
},
}
p3.atomic_json(output.with_suffix(output.suffix + ".summary.json"), summary, mode=0o600)
print(json.dumps(summary, sort_keys=True))
return summary
async def finite_timestamp_load(
ctx: p3.RunContext, session: aiohttp.ClientSession, rate: float
) -> list[dict[str, Any]]:
if "arrival_s" not in ctx.rows[0]:
return await _ORIGINAL_FINITE_LOAD(ctx, session, rate)
sem = asyncio.Semaphore(ctx.args.max_concurrency)
tasks: list[asyncio.Task[dict[str, Any]]] = []
async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]:
async with sem:
return await p3.request_one(ctx, session, row, scheduled)
for expected in ctx.rows:
scheduled = ctx.t0 + float(expected["arrival_s"])
delay = scheduled - asyncio.get_running_loop().time()
if delay > 0:
try:
await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay)
break
except asyncio.TimeoutError:
pass
if ctx.stop_event.is_set():
break
row = await ctx.next_row()
if row["request_id"] != expected["request_id"]:
raise AssertionError("timestamp scheduler row drift")
tasks.append(asyncio.create_task(limited(row, scheduled)))
return await asyncio.gather(*tasks) if tasks else []
_ORIGINAL_FINITE_LOAD = p3.finite_load
p3.finite_load = finite_timestamp_load
def build_transform_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser()
parser.add_argument("--in", dest="input", required=True)
parser.add_argument("--take-first", type=int, required=True)
parser.add_argument("--timestamp-source", required=True)
parser.add_argument("--join-key", default="source_index")
parser.add_argument("--timestamp-field", default="timestamp")
parser.add_argument("--arrival", choices=("recorded-scaled", "uniform"), required=True)
parser.add_argument("--target-rate", type=float, required=True)
parser.add_argument("--service-order", choices=("original", "length-binned"), required=True)
parser.add_argument("--reorder-block-size", type=int, default=32)
parser.add_argument("--analysis-cohort-size", type=int, default=16)
parser.add_argument("--length-bin-edges", default="512,1024,2048,4096,8192,16384,32768")
parser.add_argument("--max-added-delay-seconds", type=float, default=64)
parser.add_argument("--out", required=True)
return parser
def main() -> None:
if len(sys.argv) > 1 and sys.argv[1] == "transform":
transform(build_transform_parser().parse_args(sys.argv[2:]))
return
fixed_rate = None
if "--fixed-request-rate" in sys.argv:
index = sys.argv.index("--fixed-request-rate")
fixed_rate = float(sys.argv[index + 1])
del sys.argv[index : index + 2]
args = p3.build_parser().parse_args()
if args.command != "run":
p3.main()
return
if fixed_rate is not None:
if args.load_point != "moderate" or fixed_rate <= 0 or not math.isfinite(fixed_rate):
raise ValueError("--fixed-request-rate requires positive finite moderate rate")
result_dir = Path(args.result_dir)
result_dir.mkdir(parents=True, exist_ok=True)
source = result_dir / "fixed-rate-source.json"
p3.atomic_json(source, {"clean": {"completed_throughput_rps": fixed_rate}})
args.saturation_result = str(source)
args.rate_fraction = 1.0
if args.profile_after_clean and not args.profile_trace_dir:
raise ValueError("--profile-after-clean requires --profile-trace-dir")
print(json.dumps(asyncio.run(p3.run_load(args)), sort_keys=True))
if __name__ == "__main__":
main()

670
runs/opprof-phase5/opprof_phase5_controller.py Normal file
View File

@@ -0,0 +1,670 @@
#!/usr/bin/env python3
"""Detached, resumable Phase-5 four-way primary/control controller."""
from __future__ import annotations
import argparse
import datetime as dt
import hashlib
import json
import math
import os
import re
import shlex
import shutil
import subprocess
import time
from pathlib import Path
from typing import Any
import opprof_phase3_matrix as m
WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712")
RUN_ROOT = WORKDIR / "runs/phase5"
PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase5-private/manifests")
P3_PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests")
MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0")
VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
CLIENT = WORKDIR / "scripts/opprof_phase5_client.py"
P3_CLIENT = WORKDIR / "scripts/opprof_phase3_client.py"
STATE = RUN_ROOT / "controller-state.json"
RATE = 0.4725
CAPTURE_SIZES = (
1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88,
96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192,
200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336,
352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512,
)
m.WORKDIR = WORKDIR
m.RUN_ROOT = RUN_ROOT
m.PRIVATE = PRIVATE
m.MODEL = MODEL
m.SOURCE = SOURCE
m.VENV = VENV
m.CLIENT = CLIENT
m.STATE = STATE
m.GPU_HOUR_LIMIT = 6.0
m.PRIOR_GPU_HOURS = 0.0
m.CONFIGS = {
"C00": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []},
"A2": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []},
"A4": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []},
}
def sha256_file(path: Path) -> str:
return m.sha256_file(path)
def arm_name(pattern: str) -> str:
return pattern.split("-r", 1)[0]
def manifest_for(pattern: str, burnin: bool) -> Path:
if burnin or pattern.startswith("background"):
return P3_PRIVATE / "P06.jsonl"
if pattern.startswith("control-P03"):
return P3_PRIVATE / "P03.jsonl"
if pattern.startswith("control-P04"):
return P3_PRIVATE / "P04.jsonl"
arm = arm_name(pattern)
return PRIVATE / ("P10-base.jsonl" if arm in {"base", "A2", "A4"} else f"P10-{arm}.jsonl")
def saturation_result_for(pattern: str) -> Path:
if pattern.startswith("control-P03"):
cell = "P03-C00"
elif pattern.startswith("control-P04"):
cell = "P04-C00"
else:
cell = "P10-C00"
return WORKDIR / f"runs/phase3/primary/{cell}/saturation/client/result.json"
def drain_budget(_pattern: str) -> int:
return 600
m.drain_budget = drain_budget
def server_command(assignment: m.Assignment, port: int, _trace_dir: Path) -> list[str]:
arm = arm_name(assignment.cell.pattern)
command = [
"taskset", "-c", m.cpu_mask(assignment.gpus), str(VENV / "bin/vllm"),
"serve", str(MODEL), "--host", "127.0.0.1", "--port", str(port),
"--tensor-parallel-size", "1", "--enable-chunked-prefill",
]
if arm != "A4" and assignment.cell.config != "A4":
command.append("--enable-prefix-caching")
command.extend(("--shutdown-timeout", "600"))
if arm == "A2" or assignment.cell.config == "A2":
command.extend(("--cudagraph-capture-sizes", *map(str, CAPTURE_SIZES)))
return command
def client_command(
assignment: m.Assignment,
port: int,
run_dir: Path,
_load_point: str,
_profile: bool,
burnin: bool,
_saturation_result: Path | None,
) -> list[str]:
pattern = assignment.cell.pattern
common = [
"taskset", "-c", m.cpu_mask(assignment.gpus), str(VENV / "bin/python"),
str(CLIENT), "run", "--manifest", str(manifest_for(pattern, burnin)),
"--base-url", f"http://127.0.0.1:{port}", "--model", str(MODEL),
"--max-concurrency", "256", "--ignore-eos", "--temperature", "0",
"--workload-seed", "20260712", "--server-seed", "20260712",
"--result-dir", str(run_dir / "client"),
]
if burnin:
return common + [
"--load-point", "saturation", "--request-rate", "inf",
"--warmup-seconds", "0", "--clean-segment-seconds", "20",
"--num-clean-segments", "3", "--post-clean-seconds", "0",
"--drain-timeout-seconds", "600",
]
if pattern.startswith("background"):
return common + [
"--load-point", "saturation", "--request-rate", "inf",
"--warmup-seconds", "60", "--clean-segment-seconds", "80",
"--num-clean-segments", "3", "--post-clean-seconds", "0",
"--drain-timeout-seconds", "600",
]
if pattern.startswith("control-"):
return common + [
"--load-point", "moderate", "--saturation-result",
str(saturation_result_for(pattern)), "--rate-fraction", "0.60",
"--warmup-seconds", "60", "--clean-segment-seconds", "80",
"--num-clean-segments", "3", "--post-clean-seconds", "0",
"--drain-timeout-seconds", "600",
]
return common + [
"--load-point", "moderate", "--fixed-request-rate", str(RATE),
"--warmup-seconds", "60", "--clean-segment-seconds", "80",
"--num-clean-segments", "3", "--post-clean-seconds", "0",
"--drain-timeout-seconds", "600",
]
m.server_command = server_command
m.client_command = client_command
_ORIGINAL_VALIDATE_CLIENT = m.validate_client
def _log_event_time(line: str, t0_wall_ns: int) -> float | None:
match = re.search(r"INFO\s+(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2})", line)
if match is None:
return None
month, day, hour, minute, second = map(int, match.groups())
year = dt.datetime.fromtimestamp(t0_wall_ns / 1e9, tz=dt.timezone.utc).year
stamp = dt.datetime(year, month, day, hour, minute, second, tzinfo=dt.timezone.utc)
return stamp.timestamp()
def cold_start_gate(run_dir: Path) -> dict[str, Any]:
result = json.loads((run_dir / "client/result.json").read_text())
requests = [
json.loads(line)
for line in (run_dir / "client/requests.jsonl").read_text().splitlines()
]
warm = [
request for request in requests
if request["success"] and 0 <= float(request["completed_s"]) < 60
]
warm_long = [request for request in warm if int(request["input_tokens"]) >= 8192]
t0_mono_ns = int(result["t0_mono_ns"])
stream = next((run_dir / "opprof").glob("*.jsonl"))
warm_descriptors: set[tuple[str, int]] = set()
clean_descriptors: set[tuple[str, int]] = set()
for line in stream.read_text().splitlines():
item = json.loads(line)
if "step_index" not in item or not item.get("model_executed"):
continue
graph = item["cudagraph"]
if not graph.get("hit"):
continue
relative_s = (int(item["submit_mono_ns"]) - t0_mono_ns) / 1e9
descriptor = (str(graph["runtime_mode"]), int(graph["bucket_tokens"]))
if 0 <= relative_s < 60:
warm_descriptors.add(descriptor)
elif 60 <= relative_s < 300:
clean_descriptors.add(descriptor)
log_lines = (run_dir / "server.log").read_text(errors="replace").splitlines()
ready_indices = [
index for index, line in enumerate(log_lines)
if "Application startup complete" in line
]
if len(ready_indices) != 1:
raise RuntimeError(f"expected one server ready marker: {run_dir}: {ready_indices}")
ready_index = ready_indices[0]
event_pattern = re.compile(
r"torch\.compile took|Directly load AOT compilation|\bCompiling\b|Capturing CUDA graphs",
re.IGNORECASE,
)
events = []
clean_boundary_wall_s = int(result["t0_wall_ns"]) / 1e9 + 60
clean_end_wall_s = int(result["t0_wall_ns"]) / 1e9 + 300
event_gate = True
clean_events = 0
for index, line in enumerate(log_lines):
if not event_pattern.search(line):
continue
timestamp = _log_event_time(line, int(result["t0_wall_ns"]))
phase = "pre-ready" if index < ready_index else "post-ready"
if index >= ready_index:
if timestamp is None:
event_gate = False
phase = "post-ready-unparseable"
elif timestamp >= clean_boundary_wall_s:
event_gate = False
phase = "clean" if timestamp < clean_end_wall_s else "post-clean"
if timestamp < clean_end_wall_s:
clean_events += 1
else:
phase = "warmup"
events.append(
{
"line": index + 1,
"phase": phase,
"timestamp_s": timestamp,
"message_prefix": line[:200],
}
)
config_match = re.search(
r"cudagraph_capture_sizes['\"]?:\s*\[([^\]]+)\]",
"\n".join(log_lines[:ready_index]),
)
startup_sizes = (
{int(value.strip()) for value in config_match.group(1).split(",")}
if config_match else set()
)
startup_modes = set()
for line in log_lines[:ready_index]:
if "Capturing CUDA graphs" not in line:
continue
if "PIECEWISE" in line:
startup_modes.add("PIECEWISE")
if "FULL" in line:
startup_modes.add("FULL")
uncovered = sorted(
descriptor for descriptor in clean_descriptors
if descriptor[0] not in startup_modes or descriptor[1] not in startup_sizes
)
passed = (
event_gate
and len(warm) >= 16
and len(warm_long) >= 1
and startup_modes == {"FULL", "PIECEWISE"}
and bool(startup_sizes)
and not uncovered
and clean_events == 0
)
return {
"amendment": "A-P5-1",
"passed": passed,
"warmup_completions": len(warm),
"warmup_long_completions": len(warm_long),
"warmup_long_min_tokens": min(
(int(request["input_tokens"]) for request in warm_long), default=None
),
"server_ready_line": ready_index + 1,
"compile_capture_events": events,
"event_gate_passed": event_gate,
"clean_capture_events": clean_events,
"startup_capture_sizes": sorted(startup_sizes),
"startup_capture_modes": sorted(startup_modes),
"warmup_descriptors": [list(item) for item in sorted(warm_descriptors)],
"clean_descriptors": [list(item) for item in sorted(clean_descriptors)],
"uncovered_clean_descriptors": [list(item) for item in uncovered],
"invariants": {
"events_preclean": event_gate,
"warmup_completions_ge_16": len(warm) >= 16,
"warmup_long_completion": len(warm_long) >= 1,
"startup_modes_complete": startup_modes == {"FULL", "PIECEWISE"},
"startup_sizes_present": bool(startup_sizes),
"clean_descriptors_covered": not uncovered,
"zero_clean_capture_events": clean_events == 0,
},
}
def validate_rate_client(run_dir: Path) -> dict[str, Any]:
result = json.loads((run_dir / "client/result.json").read_text())
sanity = json.loads((run_dir / "client/sanity.json").read_text())
requests = [
json.loads(line)
for line in (run_dir / "client/requests.jsonl").read_text().splitlines()
]
failed_sanity = [
key for key, value in sanity["invariants"].items()
if not value and key != "drain_within_timeout"
]
failure_summary = m.summarize_request_failures(
requests, float(result["clean"]["start_s"]), float(result["clean"]["end_s"])
)
cold = cold_start_gate(run_dir)
quarantined = float(result["drain_seconds"]) > 600
invariants = {
"client_sanity": not failed_sanity,
"clean_duration": math.isclose(float(result["clean"]["duration_s"]), 240.0),
"clean_failures_zero": result["clean"]["failed"] == 0
and failure_summary["clean_failed"] == 0,
"failed_records_accounted": result["failed_records"] == failure_summary["failed"],
"manifest_no_wrap": not result["manifest_wrapped"]
and not result["manifest_exhausted"],
"warmup_cold_start_gate": cold["passed"],
"profile_count": len(result["profiles"]) == 0,
"drain_re_adjudicated": not quarantined,
}
non_drain = {key: value for key, value in invariants.items() if key != "drain_re_adjudicated"}
if not all(non_drain.values()):
raise RuntimeError(
f"A-P5-1 rate-client invariant failure: {run_dir}: "
f"invariants={invariants}; failed_sanity={failed_sanity}; cold={cold}"
)
return {
"result": result,
"sanity": sanity,
"request_count": len(requests),
"warmup_completions": cold["warmup_completions"],
"warmup_required": 16,
"warmup_gate_branch": "A-P5-1-cold-start",
"warmup_stability": None,
"cold_start_gate": cold,
"drain_budget_seconds": 600,
"drain_quarantined": quarantined,
"excluded_window_failures": failure_summary["excluded"],
"excluded_window_failure_kinds": failure_summary["excluded_kinds"],
"invariants": invariants,
}
def validate_run(
entry: dict[str, Any], profile: bool, burnin: bool, allow_missing_traces: bool = False
) -> dict[str, Any]:
del profile, allow_missing_traces
pattern = entry["assignment"].cell.pattern
if burnin or pattern.startswith("background"):
validation_pattern = "P06"
elif pattern.startswith("control-P03"):
validation_pattern = "P03"
elif pattern.startswith("control-P04"):
validation_pattern = "P04"
else:
validation_pattern = "P10"
if burnin or pattern.startswith("background"):
client = _ORIGINAL_VALIDATE_CLIENT(
entry["run_dir"], validation_pattern, False, burnin
)
else:
client = validate_rate_client(entry["run_dir"])
layer1 = m.validate_layer1(entry["run_dir"])
log = (entry["run_dir"] / "server.log").read_text(errors="replace")
invariants = {
"triton_moe": "Using TRITON Unquantized MoE backend" in log,
"chunked_mbt": "Chunked prefill is enabled with max_num_batched_tokens=8192" in log,
"tp1": "tensor_parallel_size=1" in log,
"drain_shutdown": "mode=drain timeout=600s" in log,
"a2_sizes": entry["assignment"].cell.config != "A2"
or all(str(size) in log for size in (3, 5, 6, 7)),
}
if not all(invariants.values()):
raise RuntimeError(f"server invariant failure: {entry['run_id']}: {invariants}")
forbidden = re.compile(r'"(?:prompt|messages|content|text)"\s*:')
for path in (
entry["run_dir"] / "client/requests.jsonl",
entry["run_dir"] / "client/result.json",
Path(layer1["stream"]),
):
if forbidden.search(path.read_text(errors="replace")):
raise RuntimeError(f"private text leaked: {path}")
summary = {
"schema": 1,
"run_id": entry["run_id"],
"pattern": pattern,
"config": entry["assignment"].cell.config,
"gpus": entry["assignment"].gpus,
"client": client,
"layer1": layer1,
"traces": [],
"missing_trace_files": 0,
"layer2_missing_after_controller_cleanup": False,
"drain_quarantined": client["drain_quarantined"],
"server_invariants": invariants,
}
m.atomic_json(entry["run_dir"] / "run-complete.json", summary)
return summary
m.validate_run = validate_run
def manifests() -> dict[str, Any]:
result = {}
for name in ("base", "A1", "A3"):
path = PRIVATE / f"P10-{name}.jsonl"
summary = json.loads(path.with_suffix(path.suffix + ".summary.json").read_text())
if summary["rows"] != 142 or summary["sha256"] != sha256_file(path):
raise RuntimeError(f"manifest verification failed: {name}")
result[name] = {"path": str(path), "sha256": summary["sha256"]}
return result
def fingerprint() -> dict[str, Any]:
return {
"source_commit": subprocess.check_output(
["git", "-C", str(SOURCE), "rev-parse", "HEAD"], text=True
).strip(),
"source_tree": subprocess.check_output(
["git", "-C", str(SOURCE), "rev-parse", "HEAD^{tree}"], text=True
).strip(),
"client_sha256": sha256_file(CLIENT),
"controller_sha256": sha256_file(Path(__file__)),
"p3_client_sha256": sha256_file(P3_CLIENT),
"p3_matrix_sha256": sha256_file(Path(m.__file__).resolve()),
"manifests": manifests(),
"capture_sizes": list(CAPTURE_SIZES),
"rate": RATE,
}
def load_state(resume: bool) -> dict[str, Any]:
if STATE.exists():
if not resume:
raise RuntimeError("controller state exists; use --resume")
return json.loads(STATE.read_text())
return {
"schema": 1,
"status": "created",
"created_at": time.time(),
"controller_pid": os.getpid(),
"gpu_hours_total": 0.0,
"gpu_hours_this_stage": 0.0,
"completed_measured_runs": 0,
"completed_burnins": 0,
"drain_quarantined_runs": 0,
"clean_window_failures": 0,
"missing_trace_files": 0,
"stages": {},
"fingerprint": {},
}
def save_state(state: dict[str, Any]) -> None:
state["controller_pid"] = os.getpid()
state["updated_at"] = time.time()
m.atomic_json(STATE, state)
m.save_state = save_state
def ensure_provenance() -> None:
destination = RUN_ROOT / "provenance"
destination.mkdir(parents=True, exist_ok=True)
sources = [CLIENT, Path(__file__).resolve(), Path(m.__file__).resolve(), Path(m.common.__file__).resolve()]
hashes = {}
for source in sources:
target = destination / source.name
digest = sha256_file(source)
if target.exists() and sha256_file(target) != digest:
target = destination / f"{source.stem}.{digest[:12]}{source.suffix}"
if target.exists() and sha256_file(target) != digest:
raise RuntimeError(f"content-addressed provenance mismatch: {target}")
if not target.exists():
shutil.copy2(source, target)
hashes[target.name] = digest
m.atomic_json(destination / "sha256.json", hashes)
def primary_cells() -> list[m.Cell]:
config = {"base": "C00", "A1": "C00", "A2": "A2", "A3": "C00", "A4": "A4"}
items = [m.Cell(f"{arm}-r{replicate}", config[arm]) for arm in config for replicate in range(1, 4)]
return sorted(
items,
key=lambda cell: hashlib.sha256(
f"20260715:{cell.pattern.rsplit('-r',1)[1]}:{arm_name(cell.pattern)}".encode()
).hexdigest(),
)
def pack_unique(items: list[m.Cell], prefix: str) -> list[list[m.Assignment]]:
remaining = list(items)
waves: list[list[m.Assignment]] = []
wave_index = 0
while remaining:
selected: list[m.Cell] = []
used: set[str] = set()
for cell in list(remaining):
key = arm_name(cell.pattern)
if key in used:
continue
selected.append(cell)
used.add(key)
remaining.remove(cell)
if len(selected) == 4:
break
while len(selected) < 4:
selected.append(m.Cell(f"background-{prefix}-{wave_index}-{len(selected)}", "C00"))
assignments = []
for slot, cell in enumerate(selected):
gpu = (slot + wave_index) % 4
assignments.append(m.Assignment(cell, (gpu,)))
waves.append(assignments)
wave_index += 1
return waves
def pack_primary(items: list[m.Cell]) -> list[list[m.Assignment]]:
"""Pack SHA-ordered cells as 4/4/4/3 without duplicate arms per wave."""
capacities = (4, 4, 4, 3)
waves: list[list[m.Cell]] = [[] for _ in capacities]
def place(index: int) -> bool:
if index == len(items):
return all(len(wave) == capacity for wave, capacity in zip(waves, capacities, strict=True))
cell = items[index]
arm = arm_name(cell.pattern)
for wave_index, capacity in enumerate(capacities):
if len(waves[wave_index]) >= capacity:
continue
if any(arm_name(existing.pattern) == arm for existing in waves[wave_index]):
continue
waves[wave_index].append(cell)
if place(index + 1):
return True
waves[wave_index].pop()
return False
if not place(0):
raise RuntimeError("cannot pack frozen primary assignments into 4/4/4/3")
result: list[list[m.Assignment]] = []
for wave_index, cells in enumerate(waves):
if len(cells) == 3:
cells.append(m.Cell("background-primary-final", "C00"))
result.append(
[
m.Assignment(cell, ((slot + wave_index) % 4,))
for slot, cell in enumerate(cells)
]
)
return result
def execute_primary(resume: bool, amendment_a_p5_1: bool = False) -> None:
RUN_ROOT.mkdir(parents=True, exist_ok=True)
state = load_state(resume)
if resume:
m.cleanup_recorded(state)
current = fingerprint()
if state["fingerprint"] and state["fingerprint"] != current:
failure = str(state.get("stages", {}).get("primary-01", {}).get("failure", ""))
if not (
amendment_a_p5_1
and state.get("status") == "failed"
and "warmup" in failure.lower()
):
raise RuntimeError("resume fingerprint differs from frozen Phase-5 plan")
state.setdefault("amendments", {})["A-P5-1"] = {
"approved": True,
"applied_at": time.time(),
"reason": "replace rate-following drift gate with cold-start gates",
"prior_fingerprint": state["fingerprint"],
"replacement_fingerprint": current,
"retained_gpu_hours": state["gpu_hours_total"],
"burnins_reused": state["completed_burnins"],
}
state["fingerprint"] = current
state["status"] = "amended_resume_A-P5-1"
save_state(state)
state["fingerprint"] = current
state["status"] = "running_primary"
save_state(state)
ensure_provenance()
burnins = [
m.Assignment(m.Cell("burnin-C00", "C00"), (0,)),
m.Assignment(m.Cell("burnin-A2", "A2"), (1,)),
m.Assignment(m.Cell("burnin-A4", "A4"), (2,)),
]
m.run_stage(state, "burnins", burnins, "saturation", profile=False, burnin=True)
waves = pack_primary(primary_cells())
for index, wave in enumerate(waves, 1):
m.run_stage(state, f"primary-{index:02d}", wave, "moderate", profile=False)
primary = [
path for path in (RUN_ROOT / "primary").glob("*-r*-*/moderate/run-complete.json")
if "background" not in str(path)
]
if len(primary) != 15:
raise RuntimeError(f"primary completion mismatch: {len(primary)} != 15")
state["primary_runs"] = 15
state["background_runs"] = state["completed_measured_runs"] - 15
state["status"] = "primary_complete"
state["completed_at"] = time.time()
save_state(state)
def execute_controls(resume: bool) -> None:
state = load_state(resume)
m.cleanup_recorded(state)
if state.get("fingerprint") != fingerprint():
raise RuntimeError("control resume fingerprint mismatch")
cells = [
m.Cell(f"control-{pattern}-r{replicate}", "C00")
for pattern in ("P03", "P04") for replicate in range(1, 4)
]
for index, wave in enumerate(pack_unique(cells, "controls"), 1):
m.run_stage(state, f"controls-{index:02d}", wave, "moderate", profile=False)
state["status"] = "controls_complete"
state["controls_completed_at"] = time.time()
save_state(state)
def plan() -> dict[str, Any]:
return {
"schema": 1,
"primary_runs": 15,
"burnins": 3,
"waves": [[{"cell": a.cell.cell_id, "gpus": a.gpus} for a in wave] for wave in pack_primary(primary_cells())],
"rate": RATE,
"clean_seconds": 240,
"drain_seconds": 600,
"gpu_hour_limit": 6.0,
}
def main() -> None:
parser = argparse.ArgumentParser()
sub = parser.add_subparsers(dest="command", required=True)
for name in ("primary", "controls"):
item = sub.add_parser(name)
item.add_argument("--resume", action="store_true")
if name == "primary":
item.add_argument("--amendment-a-p5-1", action="store_true")
sub.add_parser("plan")
sub.add_parser("status")
args = parser.parse_args()
if args.command == "primary":
execute_primary(args.resume, args.amendment_a_p5_1)
elif args.command == "controls":
execute_controls(args.resume)
elif args.command == "plan":
print(json.dumps(plan(), sort_keys=True, indent=2))
else:
print(STATE.read_text() if STATE.exists() else '{"status":"absent"}')
if __name__ == "__main__":
main()

212
runs/opprof-phase5/phase5/controller-state.json Normal file
View File

@@ -0,0 +1,212 @@
{
"clean_window_failures": 0,
"completed_burnins": 3,
"completed_measured_runs": 0,
"controller_pid": 2505724,
"created_at": 1783858074.2830184,
"drain_quarantined_runs": 0,
"fingerprint": {
"capture_sizes": [
1,
2,
3,
4,
5,
6,
7,
8,
16,
24,
32,
40,
48,
56,
64,
72,
80,
88,
96,
104,
112,
120,
128,
136,
144,
152,
160,
168,
176,
184,
192,
200,
208,
216,
224,
232,
240,
248,
256,
272,
288,
304,
320,
336,
352,
368,
384,
400,
416,
432,
448,
464,
480,
496,
512
],
"client_sha256": "f935486ab2588c1fca514be29b59361f53118eb000ea037863de48ce4fc76b16",
"controller_sha256": "ea2e2066a8b6d9427c59fe80db44cf87a86776570ca9066bfdff0c0b0e7f4a46",
"manifests": {
"A1": {
"path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-A1.jsonl",
"sha256": "cab8468983fb7397ec88eb5e88a44c8b1b53d8021b7867e7bec6f58d3d903806"
},
"A3": {
"path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-A3.jsonl",
"sha256": "ec5eaa29fcd3ec421e4f68a902c7e7fea9c0bc6b16f1013cef30ccc1f97bae24"
},
"base": {
"path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-base.jsonl",
"sha256": "d3b4f540ddd629dd0e34fff55c3b3837f359bdd6e5947309a1ea2a358f811ffc"
}
},
"p3_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0",
"p3_matrix_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f",
"rate": 0.4725,
"source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05",
"source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1"
},
"gpu_hours_this_stage": 0.4224752351972792,
"gpu_hours_total": 0.6476566231913037,
"missing_trace_files": 0,
"schema": 1,
"stages": {
"burnins": {
"assignments": [
{
"cell": "burnin-C00-C00",
"gpus": [
0
]
},
{
"cell": "burnin-A2-A2",
"gpus": [
1
]
},
{
"cell": "burnin-A4-A4",
"gpus": [
2
]
}
],
"burnin": true,
"clients": {},
"completed_at": 1783858361.2828288,
"confirmation": false,
"gpu_hours": 0.22518138799402448,
"load_point": "saturation",
"profile": false,
"servers": {},
"started_at": 1783858074.4911764,
"status": "complete"
},
"primary-01": {
"assignments": [
{
"cell": "base-r2-C00",
"gpus": [
0
]
},
{
"cell": "A3-r1-C00",
"gpus": [
1
]
},
{
"cell": "A1-r2-C00",
"gpus": [
2
]
},
{
"cell": "A4-r1-A4",
"gpus": [
3
]
}
],
"burnin": false,
"clients": {
"A1-r2-C00-moderate": {
"pgid": 2512122,
"pid": 2512122
},
"A3-r1-C00-moderate": {
"pgid": 2512121,
"pid": 2512121
},
"A4-r1-A4-moderate": {
"pgid": 2512123,
"pid": 2512123
},
"base-r2-C00-moderate": {
"pgid": 2512119,
"pid": 2512119
}
},
"confirmation": false,
"failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5/primary/base-r2-C00/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'failed_records_accounted': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]; warmup_completions=25; warmup_gate_branch=failed; warmup_stability={'passed': False, 'reason': 'A-P3-6 stabilization criterion not met', 'window_seconds': [45.0, 60.0], 'bin_seconds': 5.0, 'step_counts': [380, 201, 187], 'scheduled_tokens': [26927, 27616, 463], 'scheduled_token_throughput': [5385.4, 5523.2, 92.6], 'mean_scheduled_token_throughput': 3667.066666666666, 'slope_tokens_per_second_squared': -529.28, 'normalized_drift': 2.1650001817983497, 'normalized_drift_limit': 0.1, 'step_indices_continuous': True}\")",
"gpu_hours": 0.4224752351972792,
"load_point": "moderate",
"profile": false,
"servers": {
"A1-r2-C00-moderate": {
"gpus": [
2
],
"pgid": 2510424,
"pid": 2510424
},
"A3-r1-C00-moderate": {
"gpus": [
1
],
"pgid": 2510423,
"pid": 2510423
},
"A4-r1-A4-moderate": {
"gpus": [
3
],
"pgid": 2510425,
"pid": 2510425
},
"base-r2-C00-moderate": {
"gpus": [
0
],
"pgid": 2510422,
"pid": 2510422
}
},
"started_at": 1783858361.3184204,
"status": "failed"
}
},
"status": "failed",
"updated_at": 1783858758.067235
}

1
runs/opprof-phase5/phase5/launch-echo.log Normal file
View File

@@ -0,0 +1 @@
LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails

14178
runs/opprof-phase5/phase5/metrics.json Normal file
View File

File diff suppressed because it is too large Load Diff

115
runs/opprof-phase5/phase5/plan.json Normal file
View File

@@ -0,0 +1,115 @@
{
"burnins": 3,
"clean_seconds": 240,
"drain_seconds": 600,
"gpu_hour_limit": 6.0,
"primary_runs": 15,
"rate": 0.4725,
"schema": 1,
"waves": [
[
{
"cell": "base-r2-C00",
"gpus": [
0
]
},
{
"cell": "A3-r1-C00",
"gpus": [
1
]
},
{
"cell": "A1-r2-C00",
"gpus": [
2
]
},
{
"cell": "A4-r1-A4",
"gpus": [
3
]
}
],
[
{
"cell": "A1-r3-C00",
"gpus": [
1
]
},
{
"cell": "base-r1-C00",
"gpus": [
2
]
},
{
"cell": "A3-r3-C00",
"gpus": [
3
]
},
{
"cell": "A2-r1-A2",
"gpus": [
0
]
}
],
[
{
"cell": "base-r3-C00",
"gpus": [
2
]
},
{
"cell": "A3-r2-C00",
"gpus": [
3
]
},
{
"cell": "A4-r2-A4",
"gpus": [
0
]
},
{
"cell": "A2-r3-A2",
"gpus": [
1
]
}
],
[
{
"cell": "A4-r3-A4",
"gpus": [
3
]
},
{
"cell": "A1-r1-C00",
"gpus": [
0
]
},
{
"cell": "A2-r2-A2",
"gpus": [
1
]
},
{
"cell": "background-primary-final-C00",
"gpus": [
2
]
}
]
]
}

34
runs/opprof-phase5/test_phase5_analysis.py Normal file
View File

@@ -0,0 +1,34 @@
#!/usr/bin/env python3
from __future__ import annotations
import numpy as np
import analyze_phase5 as a
def main() -> None:
adjusted = a.holm({"A1": 0.001, "A2": 0.02, "A3": 0.04, "A4": 0.5})
assert adjusted == {"A1": 0.004, "A2": 0.06, "A3": 0.08, "A4": 0.5}
assert a.ci(np.arange(100, dtype=np.float64)) == [2.475, 96.52499999999999]
runs = [
{"blocks": np.asarray([[10.0, 2.0]] * 48)},
{"blocks": np.asarray([[20.0, 4.0]] * 48)},
{"blocks": np.asarray([[30.0, 6.0]] * 48)},
]
draws = a.hierarchical_draws(runs, np.random.default_rng(a.SEED))
assert draws.shape == (a.RESAMPLES,)
assert np.allclose(draws, 5.0)
assert a.point_efficiency(runs) == 5.0
idle_blocks = np.asarray([[0.0, 0.0]] + [[10.0, 2.0]] * 47)
idle_draws = a.hierarchical_draws(
[{"blocks": idle_blocks}], np.random.default_rng(a.SEED)
)
assert np.all(np.isfinite(idle_draws))
assert np.allclose(idle_draws, 5.0)
assert a.point_efficiency([{"blocks": idle_blocks}]) == 5.0
assert a.two_sided_p(np.asarray([-1.0, 1.0])) == 1.0
print("phase5 analysis: PASS")
if __name__ == "__main__":
main()

68
runs/opprof-phase5/test_phase5_tools.py Normal file
View File

@@ -0,0 +1,68 @@
#!/usr/bin/env python3
from __future__ import annotations
import json
import subprocess
import sys
import tempfile
from pathlib import Path
HERE = Path(__file__).resolve().parent
CLIENT = HERE / "opprof_phase5_client.py"
def write_jsonl(path: Path, rows: list[dict]) -> None:
path.write_text("".join(json.dumps(row) + "\n" for row in rows))
def run(command: list[str]) -> None:
completed = subprocess.run(command, text=True, capture_output=True)
if completed.returncode:
raise RuntimeError(f"command failed: {completed.stderr}")
def main() -> None:
with tempfile.TemporaryDirectory() as tmp_text:
tmp = Path(tmp_text)
manifest = tmp / "p3.jsonl"
source = tmp / "source.jsonl"
base = tmp / "base.jsonl"
a1 = tmp / "a1.jsonl"
rows = []
source_rows = []
lengths = [128, 8192, 256, 4096] * 8
for index, length in enumerate(lengths):
rows.append({
"request_id": f"P10-{index}", "pattern_id": "P10",
"input_tokens": length, "output_tokens": 8,
"arrival": "steady", "kind": "private-trace",
"source_index": index, "prompt": f"private-{index}",
})
source_rows.append({"timestamp": index * 0.1, "prompt": f"private-{index}"})
write_jsonl(manifest, rows)
write_jsonl(source, source_rows)
common = [
sys.executable, str(CLIENT), "transform", "--in", str(manifest),
"--take-first", "32", "--timestamp-source", str(source),
"--join-key", "source_index", "--timestamp-field", "timestamp",
"--arrival", "recorded-scaled", "--target-rate", "0.5",
]
run(common + ["--service-order", "original", "--out", str(base)])
run(common + [
"--service-order", "length-binned", "--reorder-block-size", "32",
"--analysis-cohort-size", "16", "--max-added-delay-seconds", "64",
"--out", str(a1),
])
base_summary = json.loads((base.with_suffix(".jsonl.summary.json")).read_text())
a1_summary = json.loads((a1.with_suffix(".jsonl.summary.json")).read_text())
assert base_summary["rows"] == a1_summary["rows"] == 32
assert base_summary["request_id_set_sha256"] == a1_summary["request_id_set_sha256"]
assert a1_summary["r16"] < base_summary["r16"]
assert a1_summary["max_added_delay_seconds"] <= 64
assert "private-" not in json.dumps(a1_summary)
print("phase5 tools: PASS")
if __name__ == "__main__":
main()

466
runs/opprof-phase6/analyze_phase6.py Normal file
View File

@@ -0,0 +1,466 @@
#!/usr/bin/env python3
"""Frozen Phase-6 paired-anchor, frontier, rank, and Layer-1 analysis."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
from collections import Counter
from pathlib import Path
from typing import Any
import numpy as np
OLD_BEST = "tp2_mns32"
TOP20 = {"tp2_mns32", "tp2_mns64", "tp4_mns16", "tp4_mns32", "tp4_mns64"}
def sha256_file(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
def numeric(values: list[float | int | None]) -> dict[str, Any]:
finite = [float(x) for x in values if x is not None and math.isfinite(float(x))]
return {
"n": len(values), "finite_n": len(finite), "missing_n": len(values) - len(finite),
"min": min(finite) if finite else None, "max": max(finite) if finite else None,
"distinct_n": len(set(finite)),
}
def cv(values: list[float]) -> float:
if not values:
return 0.0
array = np.asarray(values, dtype=np.float64)
mean = float(array.mean())
return float(array.std(ddof=0) / mean) if mean else 0.0
def floor_buckets(scores: dict[str, float]) -> tuple[float, dict[str, int]]:
tol = max(1e-9, 1e-6 * max(abs(x) for x in scores.values()))
return tol, {key: math.floor(value / tol) for key, value in scores.items()}
def kendall_tau_b(x: dict[str, int], y: dict[str, int]) -> dict[str, Any]:
keys = sorted(x)
c = d = tx = ty = joint = 0
for i, left in enumerate(keys):
for right in keys[i + 1:]:
dx = (x[left] > x[right]) - (x[left] < x[right])
dy = (y[left] > y[right]) - (y[left] < y[right])
if dx == 0 and dy == 0:
joint += 1
elif dx == 0:
tx += 1
elif dy == 0:
ty += 1
elif dx == dy:
c += 1
else:
d += 1
denom = math.sqrt((c + d + tx) * (c + d + ty))
return {
"tau_b": (c - d) / denom if denom else None, "concordant": c,
"discordant": d, "v20_only_ties": tx, "v24_only_ties": ty,
"joint_ties": joint, "pairs": len(keys) * (len(keys) - 1) // 2,
}
def layer_metrics(records: list[dict[str, Any]]) -> dict[str, Any]:
model = [x for x in records if x.get("model_executed")]
mode = Counter(str(x["cudagraph"]["runtime_mode"]) for x in model)
total_bucket = sum(int(x["cudagraph"]["bucket_tokens"]) for x in model)
total_padding = sum(int(x["cudagraph"]["padding_tokens"]) for x in model)
kv = [float(x["kv"]["usage"]) for x in model]
waiting = [float(x["queues"]["waiting"]) for x in model]
decode_b = [float(x["decode_batch_size"]) for x in model]
return {
"steps": len(records), "model_steps": len(model),
"prefill_only_steps": sum(int(x["prefill_tokens"]) > 0 and int(x["decode_tokens"]) == 0 for x in model),
"decode_only_steps": sum(int(x["prefill_tokens"]) == 0 and int(x["decode_tokens"]) > 0 for x in model),
"mixed_steps": sum(int(x["prefill_tokens"]) > 0 and int(x["decode_tokens"]) > 0 for x in model),
"prefill_tokens": sum(int(x["prefill_tokens"]) for x in model),
"decode_tokens": sum(int(x["decode_tokens"]) for x in model),
"preemptions": sum(int(x["preemptions"]) for x in model),
"kv_usage_mean": float(np.mean(kv)) if kv else None,
"kv_usage_max": max(kv) if kv else None,
"waiting_mean": float(np.mean(waiting)) if waiting else None,
"waiting_max": max(waiting) if waiting else None,
"waiting_cv": cv(waiting), "decode_B_mean": float(np.mean(decode_b)) if decode_b else None,
"decode_B_cv": cv(decode_b), "graph_mode_counts": dict(sorted(mode.items())),
"graph_mode_shares": {key: value / len(model) for key, value in sorted(mode.items())} if model else {},
"padding_fraction": total_padding / total_bucket if total_bucket else 0.0,
}
def load_stream(path: Path) -> list[dict[str, Any]]:
return [json.loads(line) for line in path.read_text().splitlines() if "step_index" in json.loads(line)]
def p3_reference(p3_root: Path) -> dict[str, Any]:
run = p3_root / "primary/P10-C00/moderate"
result = json.loads((run / "client/result.json").read_text())
t0 = int(result["t0_mono_ns"])
lo = t0 + int(float(result["clean"]["start_s"]) * 1e9)
hi = t0 + int(float(result["clean"]["end_s"]) * 1e9)
stream = next((run / "opprof").glob("*.jsonl"))
records = [x for x in load_stream(stream) if lo <= int(x["submit_mono_ns"]) < hi]
return {
"limitation": "P3 P10 reference is TP1/MNS-default, steady arrival, sampling_u<=0.125, output<=256; it is contextual rather than a matched v0.20 composition baseline.",
"stream_sha256": sha256_file(stream), "metrics": layer_metrics(records),
}
def accepted_anchor(anchor_dir: Path) -> dict[str, Any]:
primary = json.loads((anchor_dir / "result.json").read_text())
cell_dir = anchor_dir.parent
anchor = float(primary["anchor"])
confirms = []
for path in sorted(cell_dir.glob("confirm-*-anchor-*/result.json")):
item = json.loads(path.read_text())
if math.isclose(float(item["anchor"]), anchor, rel_tol=0, abs_tol=1e-15):
confirms.append(item)
trials = [primary, *confirms]
votes = [bool(x["feasible"]) for x in trials]
if len(votes) == 1 or len(set(votes)) == 1:
verdict = votes[0]
resolved = True
elif len(votes) >= 3:
verdict = sum(votes) >= 2
resolved = True
else:
verdict = None
resolved = False
return {
"anchor": anchor, "primary": primary, "confirmations": confirms,
"trial_count": len(trials), "pass_rates": [x["pass_rate"] for x in trials],
"feasible_votes": votes, "accepted_feasible": verdict, "resolved": resolved,
"accepted_pass_rate": float(np.median([x["pass_rate"] for x in trials])),
}
def matching_anchor_dir(cell_dir: Path, anchor: float) -> Path | None:
for path in cell_dir.glob("anchor-*/result.json"):
item = json.loads(path.read_text())
if math.isclose(float(item["anchor"]), anchor, rel_tol=0, abs_tol=1e-15):
return path.parent
return None
def analyze(root: Path, ground_path: Path, p3_root: Path) -> dict[str, Any]:
ground = json.loads(ground_path.read_text())
old_cells = {x["cell_id"]: x for x in ground["cells"]}
old_probe = {
(x["cell_id"], float(p["sampling_u"])): p
for x in ground["cells"] for p in x["probe_history"]
}
cells: dict[str, Any] = {}
all_primary = []
all_client_invariants = []
selection_matches = []
for cell, old in old_cells.items():
cell_dir = root / "solo-authoritative/cells" / cell
colocated_dir = root / "cells" / cell
f20 = float(old["measured_objective"]["slo_feasible_req_s_per_gpu"])
if not (cell_dir / "cell-valid.json").exists():
cells[cell] = {
"tp": old["tensor_parallel_size"], "mns": old["max_num_seqs"],
"measurement_status": "UNMEASURED_SOLO",
"f20": f20, "f24": None, "drift": None,
"observed_max_feasible_rate": None,
"material_frontier_moved": None, "boundary": "UNMEASURED",
"bounded": False, "censor": "UNMEASURED_SOLO",
"anchors": [], "cell_valid": None,
}
continue
valid = json.loads((cell_dir / "cell-valid.json").read_text())
stream = next((cell_dir / "opprof").glob("*.jsonl"))
stream_records = load_stream(stream)
anchors = []
for path in sorted(cell_dir.glob("anchor-*/result.json")):
accepted = accepted_anchor(path.parent)
primary = accepted["primary"]
key = (cell, float(primary["anchor"]))
historical = old_probe[key]
lo = int(primary["interval"]["start_mono_ns"])
hi = int(primary["interval"]["end_mono_ns"])
records = [x for x in stream_records if lo <= int(x["submit_mono_ns"]) <= hi]
accepted["layer1"] = layer_metrics(records)
for confirmation in accepted["confirmations"]:
confirm_lo = int(confirmation["interval"]["start_mono_ns"])
confirm_hi = int(confirmation["interval"]["end_mono_ns"])
confirmation["layer1"] = layer_metrics([
x for x in stream_records
if confirm_lo <= int(x["submit_mono_ns"]) <= confirm_hi
])
accepted["v20"] = {
"pass_rate": historical["pass_rate"], "feasible": historical["feasible"],
"request_count": historical["request_count"],
"rate_per_gpu": historical["request_rate_per_gpu_req_s_gpu"],
}
accepted["v24"] = {
"pass_rate": accepted["accepted_pass_rate"],
"feasible": accepted["accepted_feasible"],
"request_count": primary["selection"]["count"],
"rate_per_gpu": primary["selection"]["offered_req_s_per_gpu"],
}
colocated_anchor_dir = matching_anchor_dir(colocated_dir, float(primary["anchor"]))
if colocated_anchor_dir is not None:
colocated = accepted_anchor(colocated_anchor_dir)
accepted["colocated"] = {
"primary_pass_rate": colocated["primary"]["pass_rate"],
"primary_feasible": colocated["primary"]["feasible"],
"trial_pass_rates": colocated["pass_rates"],
"accepted_feasible": colocated["accepted_feasible"],
"solo_minus_colocated_primary_pass_rate": (
accepted["accepted_pass_rate"] - colocated["primary"]["pass_rate"]
),
}
else:
accepted["colocated"] = None
accepted["feasibility_flip"] = (
accepted["accepted_feasible"] is not None
and accepted["accepted_feasible"] != historical["feasible"]
)
anchors.append(accepted)
all_primary.append(primary)
all_client_invariants.extend(primary["invariants"].values())
selection_matches.append(primary["selection"]["count"] == historical["request_count"])
peak_u = float(old["measured_objective"]["best_sampling_u"])
peak = next(x for x in anchors if math.isclose(x["anchor"], peak_u, abs_tol=1e-15))
feasible = [x for x in anchors if x["accepted_feasible"] is True]
infeasible = [x for x in anchors if x["accepted_feasible"] is False]
unresolved = [x for x in anchors if x["accepted_feasible"] is None]
observed_frontier = max((x["v24"]["rate_per_gpu"] for x in feasible), default=None)
lower = [x for x in anchors if x["anchor"] < peak_u]
upper = [x for x in anchors if x["anchor"] > peak_u]
if unresolved:
bounded, censor = False, "UNRESOLVED_SOLO_ANCHOR"
elif feasible and infeasible:
monotonic = max(x["anchor"] for x in feasible) < min(x["anchor"] for x in infeasible)
bounded = monotonic
censor = None if bounded else "NONMONOTONIC_SOLO_ANCHORS"
elif feasible:
bounded, censor = False, "RIGHT_CENSORED_HISTORY_EDGE"
elif infeasible:
bounded, censor = False, "LEFT_CENSORED_HISTORY_EDGE"
else:
bounded, censor = False, "NO_RESOLVED_SOLO_ANCHOR"
frontier = (
None if censor in {
"UNRESOLVED_SOLO_ANCHOR", "NONMONOTONIC_SOLO_ANCHORS",
"NO_RESOLVED_SOLO_ANCHOR",
} else observed_frontier
)
drift = (frontier / f20 - 1) if frontier is not None else None
if censor == "NONMONOTONIC_SOLO_ANCHORS":
boundary = "NONMONOTONIC"
elif peak["accepted_feasible"] is False:
boundary = "DOWN"
elif peak["accepted_feasible"] is None:
boundary = "UNRESOLVED"
elif any(x["accepted_feasible"] is True for x in upper):
boundary = "UP"
else:
boundary = "STABLE"
cells[cell] = {
"tp": old["tensor_parallel_size"], "mns": old["max_num_seqs"],
"measurement_status": "SOLO_AUTHORITATIVE",
"f20": f20, "f24": frontier, "drift": drift,
"observed_max_feasible_rate": observed_frontier,
"material_frontier_moved": bounded and drift is not None and abs(drift) > .05,
"boundary": boundary, "bounded": bounded, "censor": censor,
"anchors": anchors, "cell_valid": valid,
}
scores20 = {key: value["f20"] for key, value in cells.items()}
scores24 = {key: value["f24"] for key, value in cells.items() if value["f24"] is not None}
tol20, buckets20 = floor_buckets(scores20)
tol24, buckets24 = floor_buckets(scores24)
full_bounded = len(scores24) == 12 and all(x["bounded"] for x in cells.values())
max24 = max(buckets24.values())
if not full_bounded:
argmax = "INCONCLUSIVE"
else:
argmax = "SURVIVED" if buckets24[OLD_BEST] == max24 else "MOVED"
tau = kendall_tau_b(buckets20, buckets24) if full_bounded else None
reversals = []
for left in sorted(TOP20):
for right in sorted(TOP20):
if left >= right:
continue
if not cells[left]["bounded"] or not cells[right]["bounded"]:
continue
if left not in scores24 or right not in scores24:
continue
old_delta = scores20[left] - scores20[right]
if abs(old_delta) / max(scores20[left], scores20[right]) <= .05:
continue
new_delta = scores24[left] - scores24[right]
if old_delta * new_delta < 0:
reversals.append([left, right])
if not full_bounded or argmax == "INCONCLUSIVE" or tau is None:
ranking = "INCONCLUSIVE"
elif argmax == "MOVED" or tau["tau_b"] < .8 or reversals:
ranking = "MOVED"
else:
ranking = "SURVIVED"
trap_inputs = ["tp4_mns8", "tp4_mns16", "tp4_mns32"]
if not all(cells[x]["bounded"] for x in trap_inputs):
trap = "INCONCLUSIVE"
elif buckets24["tp4_mns16"] == max24:
trap = "CEASES_TO_BE_A_TRAP"
elif buckets24["tp4_mns16"] >= max(buckets24["tp4_mns8"], buckets24["tp4_mns32"]):
trap = "PERSISTS"
else:
trap = "ESCAPES"
colocated_state = json.loads((root / "controller-state.json").read_text())
solo_state_path = root / "solo-authoritative/controller-state.json"
solo_state = json.loads(solo_state_path.read_text()) if solo_state_path.exists() else None
state = solo_state or colocated_state
measured_cells = [
x for x in cells.values() if x["measurement_status"] == "SOLO_AUTHORITATIVE"
]
coverage = {
"solo_primary_anchors_at_least_25": len(all_primary) >= 25,
"solo_measured_cells_12": len(measured_cells) == 12,
}
invariants = {
"surface_rows_12": len(cells) == 12,
"selection_counts_match": all(selection_matches),
"client_invariants": all(all_client_invariants),
"cell_validity": all(
all(x["cell_valid"]["invariants"].values()) for x in measured_cells
),
"gpu_below_6": float(state["gpu_hours_total"]) < 6.0,
"rates_nonnegative": all(x["f24"] is None or x["f24"] >= 0 for x in cells.values()),
"surface_not_identical": len(set(scores24.values())) > 1,
}
red_flags = [key for key, value in {**coverage, **invariants}.items() if not value]
drifted = [key for key, value in cells.items() if value["material_frontier_moved"] is True]
flip_cells = [
key for key, value in cells.items()
if any(anchor["feasibility_flip"] for anchor in value["anchors"])
]
partial = bool([key for key, value in coverage.items() if not value])
w1_audit_path = root / "w1-readjudication-A-P6-1.json"
w1_audit = json.loads(w1_audit_path.read_text()) if w1_audit_path.exists() else None
censored = {key: value["censor"] for key, value in cells.items() if not value["bounded"]}
colocated_deltas = [
{
"cell": cell, "anchor": anchor["anchor"],
"solo_pass_rate": anchor["accepted_pass_rate"],
"solo_feasible": anchor["accepted_feasible"],
**anchor["colocated"],
}
for cell, value in cells.items() for anchor in value["anchors"]
if anchor.get("colocated") is not None
]
return {
"schema": 1,
"status": "BUDGET_STOP_PARTIAL" if partial else ("VALID" if not red_flags else "INVALID"),
"authoritative_tier": "A-P6-2 solo host placement",
"limitation": "Upgrade-path churn includes dash1->dash0 and resolved-default changes. Co-located W1-W3 values are indicative only; P3 composition is contextual, not a matched v0.20 Layer-1 baseline.",
"ground_truth_sha256": sha256_file(ground_path), "cells": cells,
"floor_buckets": {"v20_tol": tol20, "v20": buckets20, "v24_tol": tol24, "v24": buckets24},
"verdicts": {
"argmax": argmax, "ranking": ranking, "trap": trap,
"full_surface_bounded": full_bounded, "tau_b": tau,
"top_pair_reversals_gt5pct": reversals,
},
"materially_drifted_cells": drifted,
"feasibility_flip_cells": flip_cells,
"decision_blockers": {
"coverage": [key for key, value in coverage.items() if not value],
"unbounded_or_unresolved_cells": censored,
},
"solo_vs_colocated": colocated_deltas,
"w1_readjudication": w1_audit,
"run_stats": {
"measured_cells": len(measured_cells),
"surface_cells": len(cells),
"primary_anchor_runs": len(all_primary),
"confirmation_runs": sum(
len(anchor["confirmations"])
for cell in cells.values() for anchor in cell["anchors"]
),
"accepted_anchor_trials": sum(
anchor["trial_count"]
for cell in cells.values() for anchor in cell["anchors"]
),
"warmup_runs": len(measured_cells),
"solo_cell_gpu_hours": {
key: value.get("gpu_hours") for key, value in (solo_state or {}).get("cells", {}).items()
},
"colocated_wave_gpu_hours": {
key: value.get("gpu_hours") for key, value in colocated_state["waves"].items()
},
"launch_echo": (
(root / "launch-echo.log").read_text().splitlines()
+ ((root / "solo-authoritative/launch-echo.log").read_text().splitlines()
if (root / "solo-authoritative/launch-echo.log").exists() else [])
),
},
"attempt_history": {
"colocated_status": colocated_state["status"],
"colocated_h20_hours": colocated_state["gpu_hours_total"],
"colocated_primary_anchors": colocated_state["completed_primary_anchors"],
"colocated_confirmations": colocated_state["completed_confirmations"],
"colocated_budget_stop": colocated_state.get("budget_stop"),
"solo_status": (solo_state or {}).get("status"),
"solo_repairs": (solo_state or {}).get("repairs", []),
"solo_failures": (solo_state or {}).get("failures", []),
"raw_roots": {
"colocated": str(root / "cells"),
"solo_authoritative": str(root / "solo-authoritative/cells"),
},
},
"p3_composition_reference": p3_reference(p3_root),
"gpu": {
"new_h20_hours": state["gpu_hours_total"], "hard_cap": 6.0,
"prior_colocated_h20_hours": colocated_state["gpu_hours_total"],
"solo_h20_hours": (solo_state or {}).get("solo_gpu_hours", 0.0),
"completed_primary_anchors": (solo_state or {}).get("primary_anchors", 0),
"confirmations": (solo_state or {}).get("confirmations", 0),
"controller_status": state["status"],
"budget_stop": state.get("budget_stop"),
},
"sanity": {
"red_flags": red_flags, "coverage": coverage, "invariants": invariants,
"numeric": {
"v20_score": numeric(list(scores20.values())),
"v24_score": numeric(list(scores24.values())),
"drift": numeric([x["drift"] for x in cells.values()]),
"primary_pass_rate": numeric([x["pass_rate"] for x in all_primary]),
"selected_count": numeric([x["selection"]["count"] for x in all_primary]),
"layer1_steps": numeric([
anchor["layer1"]["steps"] for cell in cells.values() for anchor in cell["anchors"]
]),
},
},
}
def main() -> None:
p = argparse.ArgumentParser()
p.add_argument("--root", type=Path, required=True)
p.add_argument("--ground-truth", type=Path, required=True)
p.add_argument("--p3-root", type=Path, required=True)
p.add_argument("--out", type=Path, required=True)
args = p.parse_args()
result = analyze(args.root, args.ground_truth, args.p3_root)
args.out.parent.mkdir(parents=True, exist_ok=True)
args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n")
print(json.dumps({"status": result["status"], "verdicts": result["verdicts"], "red_flags": result["sanity"]["red_flags"], "gpu": result["gpu"]}, sort_keys=True))
if __name__ == "__main__":
main()

250
runs/opprof-phase6/opprof_phase6_client.py Normal file
View File

@@ -0,0 +1,250 @@
#!/usr/bin/env python3
"""Exact C1 anchor replay using the pinned AITuner trace/worker/SLO paths."""
from __future__ import annotations
import argparse
import dataclasses
import hashlib
import json
import math
import os
import sys
import time
from pathlib import Path
from typing import Any
AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2]))
sys.path.insert(0, str(AITUNER_ROOT / "src"))
os.environ.setdefault("AITUNER_CODEX_BASE_URL", "http://127.0.0.1:1")
from aituner.slo import evaluate_request, summarize_evaluations # noqa: E402
from aituner.spec import load_study_spec # noqa: E402
from aituner.trace import load_trace_requests, select_requests_for_threshold # noqa: E402
from aituner.worker import _probe_drain_deadline, _replay_requests # noqa: E402
def atomic_json(path: Path, payload: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".tmp")
tmp.write_text(json.dumps(payload, sort_keys=True, indent=2) + "\n")
os.replace(tmp, path)
def sha256_file(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(1 << 20), b""):
h.update(chunk)
return h.hexdigest()
def numeric(values: list[float | int | None]) -> dict[str, Any]:
finite = [float(x) for x in values if x is not None and math.isfinite(float(x))]
return {
"n": len(values), "finite_n": len(finite), "missing_n": len(values) - len(finite),
"min": min(finite) if finite else None, "max": max(finite) if finite else None,
"distinct_n": len(set(finite)),
}
def load_selected(study_path: Path, anchor: float):
study = load_study_spec(study_path)
window, requests = load_trace_requests(study, study_spec_path=study_path)
selected = select_requests_for_threshold(requests, threshold=anchor)
return study, window, requests, selected
def selected_summary(selected, duration_s: float, tp: int) -> dict[str, Any]:
ids = "\n".join(item.row_id for item in selected)
arrival = "\n".join(f"{item.arrival_s:.12f}" for item in selected)
lengths = "\n".join(str(item.prompt_tokens_hint) for item in selected)
return {
"count": len(selected),
"offered_req_s": len(selected) / duration_s,
"offered_req_s_per_gpu": len(selected) / duration_s / tp,
"request_id_order_sha256": hashlib.sha256(ids.encode()).hexdigest(),
"arrival_order_sha256": hashlib.sha256(arrival.encode()).hexdigest(),
"raw_length_order_sha256": hashlib.sha256(lengths.encode()).hexdigest(),
"arrival_s": numeric([item.arrival_s for item in selected]),
"raw_input_tokens": numeric([item.prompt_tokens_hint for item in selected]),
"long_gt4096": sum(int(item.prompt_tokens_hint or 0) > 4096 for item in selected),
}
def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]:
study_path = Path(args.study)
study, window, _requests, selected = load_selected(study_path, args.anchor)
if warmup:
first = selected[:16]
if not any(int(item.prompt_tokens_hint or 0) > 4096 for item in first):
long_item = next(item for item in selected if int(item.prompt_tokens_hint or 0) > 4096)
first = [*selected[:15], long_item]
first = sorted({item.row_id: item for item in first}.values(), key=lambda item: item.arrival_s)
if len(first) < 16:
raise RuntimeError("warmup set has fewer than 16 unique requests")
start = first[0].arrival_s
selected = [dataclasses.replace(item, arrival_s=item.arrival_s - start) for item in first]
duration_s = float(window.window_end - window.window_start)
interval_start_mono_ns = time.monotonic_ns()
interval_start_wall_ns = time.time_ns()
outcomes, early_stopped, early_stop_reason = _replay_requests(
selected,
base_url=args.base_url,
timeout_s=study.engine.request_timeout_s,
max_concurrency=study.trace.max_concurrency,
target_pass_rate=(0.0 if warmup else study.slo.target_pass_rate),
max_lag_s=study.trace.early_stop_max_lag_s,
max_elapsed_s=(
120.0 if warmup else _probe_drain_deadline(
selected, study.slo, ceiling=study.trace.early_stop_max_elapsed_s
)
),
evaluate_outcome=lambda outcome: evaluate_request(outcome, study.slo),
drain_inflight_on_early_stop=True,
)
interval_end_mono_ns = time.monotonic_ns()
interval_end_wall_ns = time.time_ns()
evaluations, slo_summary = summarize_evaluations(outcomes, study.slo)
by_id = {item.row_id: item for item in selected}
details = []
for outcome, evaluation in zip(outcomes, evaluations):
request = by_id[outcome.request_id]
details.append({
"request_id": outcome.request_id,
"sampling_u": request.sampling_u,
"arrival_s": request.arrival_s,
"raw_input_tokens": request.prompt_tokens_hint,
"success": outcome.success,
"ttft_ms": outcome.ttft_ms,
"tpot_ms": outcome.tpot_ms,
"completion_tokens": outcome.completion_tokens,
"completion_tokens_source": outcome.completion_tokens_source,
"slo_pass": evaluation.passed,
"reasons": evaluation.reasons,
"error": outcome.error,
})
out = Path(args.result_dir)
out.mkdir(parents=True, exist_ok=True)
with (out / "requests.jsonl").open("w") as f:
for item in details:
f.write(json.dumps(item, sort_keys=True) + "\n")
summary = selected_summary(selected, duration_s, args.tp)
exact = sum(
item.success and item.completion_tokens_source == "usage" and item.completion_tokens == 128
for item in outcomes
)
result = {
"schema": 1,
"kind": "warmup" if warmup else "anchor",
"cell": args.cell,
"anchor": args.anchor,
"tp": args.tp,
"mns": args.mns,
"study_sha256": sha256_file(study_path),
"interval": {
"start_mono_ns": interval_start_mono_ns, "end_mono_ns": interval_end_mono_ns,
"start_wall_ns": interval_start_wall_ns, "end_wall_ns": interval_end_wall_ns,
"elapsed_s": (interval_end_mono_ns - interval_start_mono_ns) / 1e9,
},
"selection": summary,
"observed_count": len(outcomes),
"exact_output_count": exact,
"slo_pass_count": slo_summary["slo_pass_count"],
"pass_rate": slo_summary["slo_pass_rate"],
"feasible": bool(slo_summary["feasible"]),
"early_stopped": early_stopped,
"early_stop_reason": early_stop_reason,
"ttft_ms": numeric([item.ttft_ms for item in outcomes]),
"tpot_ms": numeric([item.tpot_ms for item in outcomes]),
"invariants": {
"selected_nonempty": bool(selected),
"outcomes_cover_selected": len(outcomes) == len(selected),
"exact_output_or_failed": all(
(not item.success) or (
item.completion_tokens_source == "usage" and item.completion_tokens == 128
) for item in outcomes
),
"raw_lengths_present": all(item.prompt_tokens_hint is not None for item in selected),
"arrival_nondecreasing": all(
b.arrival_s >= a.arrival_s for a, b in zip(selected, selected[1:])
),
"warmup_16": (len(outcomes) >= 16 if warmup else True),
"warmup_exact_16": (exact >= 16 if warmup else True),
"warmup_long": (
any(int(item.prompt_tokens_hint or 0) > 4096 for item in selected)
if warmup else True
),
},
}
atomic_json(out / "result.json", result)
print(json.dumps({k: result[k] for k in ("cell", "anchor", "kind", "pass_rate", "feasible")}))
if not all(result["invariants"].values()):
raise RuntimeError(f"client invariants failed: {result['invariants']}")
return result
def preflight(args: argparse.Namespace) -> None:
ground = json.loads(Path(args.ground_truth).read_text())
studies = {1: Path(args.primary_study), 2: Path(args.primary_study), 4: Path(args.tp4_study)}
loaded = {}
mismatches = []
values = []
for cell in ground["cells"]:
tp = int(cell["tensor_parallel_size"])
if tp not in loaded:
_study, _window, requests, _selected = load_selected(studies[tp], 0.0)
loaded[tp] = requests
for historical in cell["probe_history"]:
selected = select_requests_for_threshold(
loaded[tp], threshold=float(historical["sampling_u"])
)
values.append(len(selected))
if len(selected) != int(historical["request_count"]):
mismatches.append({
"cell": cell["cell_id"], "anchor": historical["sampling_u"],
"expected": historical["request_count"], "actual": len(selected),
})
result = {
"schema": 1, "observations": len(values), "mismatches": mismatches,
"request_counts": numeric(values),
"invariants": {"observations_92": len(values) == 92, "counts_match": not mismatches},
}
atomic_json(Path(args.out), result)
print(json.dumps(result, sort_keys=True))
if not all(result["invariants"].values()):
raise RuntimeError("preflight count reconstruction failed")
def parser() -> argparse.ArgumentParser:
p = argparse.ArgumentParser()
sub = p.add_subparsers(dest="command", required=True)
pf = sub.add_parser("preflight")
pf.add_argument("--ground-truth", required=True)
pf.add_argument("--primary-study", required=True)
pf.add_argument("--tp4-study", required=True)
pf.add_argument("--out", required=True)
for name in ("warmup", "run-anchor"):
q = sub.add_parser(name)
q.add_argument("--study", required=True)
q.add_argument("--cell", required=True)
q.add_argument("--anchor", type=float, required=True)
q.add_argument("--tp", type=int, required=True)
q.add_argument("--mns", type=int, required=True)
q.add_argument("--base-url", required=True)
q.add_argument("--result-dir", required=True)
return p
def main() -> None:
args = parser().parse_args()
if args.command == "preflight":
preflight(args)
else:
run_replay(args, warmup=args.command == "warmup")
if __name__ == "__main__":
main()

492
runs/opprof-phase6/opprof_phase6_controller.py Normal file
View File

@@ -0,0 +1,492 @@
#!/usr/bin/env python3
"""Detached adaptive Phase-6 controller for the frozen 25-anchor surface."""
from __future__ import annotations
import argparse
import json
import os
import re
import shlex
import signal
import subprocess
import time
import urllib.request
from pathlib import Path
from typing import Any
WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712")
RUN_ROOT = WORKDIR / "runs/phase6"
STATE = RUN_ROOT / "controller-state.json"
SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0")
VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv")
AITUNER = Path("/home/admin/cpfs/wjh/aituner/aituner")
MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B")
CLIENT = WORKDIR / "scripts/opprof_phase6_client.py"
PRIMARY_STUDY = WORKDIR / "provenance/study-primary.json"
TP4_STUDY = WORKDIR / "provenance/study-tp4.json"
GROUND = WORKDIR / "provenance/ground_truth.json"
GPU_LIMIT = 3.0
CPU_MAP = {i: f"{20*i}-{20*i+19}" for i in range(8)}
MARKER = "opprof-phase6-20260712"
OWNED_PGIDS: set[int] = set()
CELLS = {
"tp1_mns8": {"tp": 1, "mns": 8, "lower": .21875, "peak": .2265625, "upper": .23046875},
"tp1_mns16": {"tp": 1, "mns": 16, "lower": .2421875, "peak": .24609375, "upper": .25},
"tp1_mns32": {"tp": 1, "mns": 32, "lower": .234375, "peak": .2421875, "upper": .24609375},
"tp1_mns64": {"tp": 1, "mns": 64, "lower": .234375, "peak": .2421875, "upper": .24609375},
"tp2_mns8": {"tp": 2, "mns": 8, "lower": .4921875, "peak": .49609375, "upper": .5},
"tp2_mns16": {"tp": 2, "mns": 16, "lower": .4921875, "peak": .49609375, "upper": .5},
"tp2_mns32": {"tp": 2, "mns": 32, "lower": .75, "peak": .75390625, "upper": .7578125},
"tp2_mns64": {"tp": 2, "mns": 64, "lower": .5, "peak": .75, "upper": .75390625},
"tp4_mns8": {"tp": 4, "mns": 8, "lower": .016055910008, "peak": .016591107009, "upper": .017126304009},
"tp4_mns16": {"tp": 4, "mns": 16, "lower": .033182214016, "peak": .033717411016, "upper": .034252608017, "trap": True},
"tp4_mns32": {"tp": 4, "mns": 32, "lower": .033182214016, "peak": .033717411016, "upper": .034252608017},
"tp4_mns64": {"tp": 4, "mns": 64, "lower": .033182214016, "peak": .033717411016, "upper": .034252608017},
}
WAVES = [
("W1-tp1", [("tp1_mns8", (0,)), ("tp1_mns16", (1,)), ("tp1_mns32", (2,)), ("tp1_mns64", (3,))], .35),
("W2-tp2", [("tp2_mns8", (0,1)), ("tp2_mns16", (2,3)), ("tp2_mns32", (4,5)), ("tp2_mns64", (6,7))], .65),
("W3-tp4-trap", [("tp4_mns8", (0,1,2,3)), ("tp4_mns16", (4,5,6,7))], .85),
("W4-tp4", [("tp4_mns32", (0,1,2,3)), ("tp4_mns64", (4,5,6,7))], .65),
]
def atomic_json(path: Path, value: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".tmp")
tmp.write_text(json.dumps(value, sort_keys=True, indent=2) + "\n")
os.replace(tmp, path)
def load_state() -> dict[str, Any]:
if STATE.exists():
return json.loads(STATE.read_text())
return {
"schema": 1, "status": "initialized", "gpu_hours_total": 0.0,
"completed_primary_anchors": 0, "completed_confirmations": 0,
"waves": {}, "failures": [], "started_at": time.time(),
}
def save_state(state: dict[str, Any]) -> None:
atomic_json(STATE, state)
def cpu_mask(gpus: tuple[int, ...]) -> str:
return ",".join(CPU_MAP[g] for g in gpus)
def study_for(tp: int) -> Path:
return TP4_STUDY if tp == 4 else PRIMARY_STUDY
def run_text(command: list[str], check: bool = True) -> str:
result = subprocess.run(command, text=True, capture_output=True)
if check and result.returncode:
raise RuntimeError(f"command failed {command}: {result.stderr}")
return result.stdout
def compute_pids() -> list[int]:
text = run_text([
"nvidia-smi", "--query-compute-apps=pid", "--format=csv,noheader,nounits"
], check=False)
return sorted({int(x.strip()) for x in text.splitlines() if x.strip().isdigit()})
def pid_owned(pid: int) -> bool:
try:
if os.getpgid(pid) in OWNED_PGIDS:
return True
except ProcessLookupError:
return True
try:
env = Path(f"/proc/{pid}/environ").read_bytes().split(b"\0")
except (FileNotFoundError, PermissionError):
return False
return f"OPPROF_PHASE6_MARKER={MARKER}".encode() in env
def assert_no_other_compute() -> None:
other = [pid for pid in compute_pids() if not pid_owned(pid)]
if other:
raise RuntimeError(f"outside GPU processes detected: {other}")
def assert_all_idle() -> None:
if compute_pids():
raise RuntimeError(f"GPU compute processes remain: {compute_pids()}")
rows = run_text([
"nvidia-smi", "--query-gpu=index,memory.used,utilization.gpu",
"--format=csv,noheader,nounits",
])
bad = []
for line in rows.splitlines():
index, memory, util = [int(x.strip()) for x in line.split(",")]
if memory != 0 or util != 0:
bad.append((index, memory, util))
if bad:
raise RuntimeError(f"GPU cleanup failure: {bad}")
def wait_ready(entry: dict[str, Any], timeout: float = 300.0) -> None:
deadline = time.monotonic() + timeout
url = f"http://127.0.0.1:{entry['port']}/v1/models"
while time.monotonic() < deadline:
if entry["server"].poll() is not None:
raise RuntimeError(f"server exited before ready: {entry['cell']}")
try:
with urllib.request.urlopen(url, timeout=2) as response:
if response.status < 500:
return
except Exception:
pass
assert_no_other_compute()
time.sleep(1)
raise TimeoutError(f"server ready timeout: {entry['cell']}")
def server_command(cell: str, gpus: tuple[int, ...], port: int) -> list[str]:
cfg = CELLS[cell]
return [
"taskset", "-c", cpu_mask(gpus), str(VENV / "bin/vllm"), "serve", str(MODEL),
"--host", "127.0.0.1", "--port", str(port),
"--served-model-name", "qwen3-30b-a3b-community",
"--max-num-batched-tokens", "8192", "--max-num-seqs", str(cfg["mns"]),
"--tensor-parallel-size", str(cfg["tp"]), "--shutdown-timeout", "120",
]
def client_command(entry: dict[str, Any], anchor: float, out: Path, warmup: bool) -> list[str]:
cfg = CELLS[entry["cell"]]
return [
"taskset", "-c", cpu_mask(entry["gpus"]), str(VENV / "bin/python"), str(CLIENT),
"warmup" if warmup else "run-anchor", "--study", str(study_for(cfg["tp"])),
"--cell", entry["cell"], "--anchor", str(anchor), "--tp", str(cfg["tp"]),
"--mns", str(cfg["mns"]), "--base-url", f"http://127.0.0.1:{entry['port']}",
"--result-dir", str(out),
]
def live_gpu_hours(entries: list[dict[str, Any]]) -> float:
now = time.time()
return sum(
len(e["gpus"]) * ((e.get("stopped_at") or now) - e["spawned_at"])
for e in entries
) / 3600
def run_clients(
entries: list[dict[str, Any]], assignments: list[tuple[dict[str, Any], float, Path]],
state: dict[str, Any], wave_name: str, warmup: bool = False,
) -> list[dict[str, Any]]:
processes = []
handles = []
for entry, anchor, out in assignments:
command = client_command(entry, anchor, out, warmup)
with (entry["dir"] / "commands.log").open("a") as f:
f.write(f"CLIENT {shlex.join(command)}\n")
handle = (out.parent / f"{out.name}.log").open("ab", buffering=0)
handles.append(handle)
client_env = os.environ.copy()
client_env.update({"AITUNER_ROOT": str(AITUNER), "PYTHONUNBUFFERED": "1"})
p = subprocess.Popen(
command, cwd=WORKDIR, env=client_env, stdout=handle,
stderr=subprocess.STDOUT, start_new_session=True,
)
processes.append((entry, anchor, out, p))
deadline = time.monotonic() + 180
while any(p.poll() is None for *_rest, p in processes):
if time.monotonic() > deadline:
raise TimeoutError(f"client batch timeout: {wave_name}")
for entry in entries:
if entry.get("stopped_at") is None and entry["server"].poll() is not None:
raise RuntimeError(f"server exited during client: {entry['cell']}")
assert_no_other_compute()
if state["gpu_hours_total"] + live_gpu_hours(entries) >= GPU_LIMIT:
raise RuntimeError("3.0 H20-hour hard stop reached")
time.sleep(1)
for handle in handles:
handle.close()
bad = [(e["cell"], a, p.returncode) for e, a, _o, p in processes if p.returncode]
if bad:
raise RuntimeError(f"client failures: {bad}")
results = []
for entry, anchor, out, _p in processes:
result = json.loads((out / "result.json").read_text())
results.append(result)
entry.setdefault("results", []).append({"anchor": anchor, "dir": str(out), "kind": result["kind"]})
return results
def stop_entry(entry: dict[str, Any]) -> None:
if entry.get("stopped_at") is not None:
return
process = entry["server"]
try:
# Official vLLM shutdown: signal the API parent so EngineCore drains and
# emits the in-stream footer/final sidecar. Process-group signals are
# fallback only.
os.kill(process.pid, signal.SIGINT)
except ProcessLookupError:
pass
try:
process.wait(timeout=150)
except subprocess.TimeoutExpired:
try:
os.killpg(process.pid, signal.SIGTERM)
except ProcessLookupError:
pass
try:
process.wait(timeout=10)
except subprocess.TimeoutExpired:
try:
os.killpg(process.pid, signal.SIGKILL)
except ProcessLookupError:
pass
process.wait(timeout=30)
entry["stopped_at"] = time.time()
entry["server_handle"].close()
def validate_cell(entry: dict[str, Any]) -> dict[str, Any]:
log = (entry["dir"] / "server.log").read_text(errors="replace").splitlines()
ready = [i for i, line in enumerate(log) if "Application startup complete" in line]
event = re.compile(r"torch\.compile took|Directly load AOT compilation|\bCompiling\b|Capturing CUDA graphs", re.I)
post_ready_events = [i + 1 for i, line in enumerate(log) if event.search(line) and ready and i > ready[0]]
streams = sorted((entry["dir"] / "opprof").glob("*.jsonl"))
sidecars = sorted((entry["dir"] / "opprof").glob("*.jsonl.footer.json"))
if len(streams) != 1 or len(sidecars) != 1:
raise RuntimeError(f"Layer1 stream/sidecar mismatch: {entry['cell']}")
decoded = [json.loads(line) for line in streams[0].read_text().splitlines()]
footers = [item for item in decoded if item.get("record_type") == "footer"]
records = [item for item in decoded if "step_index" in item]
sidecar = json.loads(sidecars[0].read_text())
indices = [int(item["step_index"]) for item in records]
warm = json.loads((entry["dir"] / "warmup/result.json").read_text())
intervals_ok = True
for item in entry.get("results", []):
if item["kind"] != "anchor":
continue
result = json.loads((Path(item["dir"]) / "result.json").read_text())
lo, hi = result["interval"]["start_mono_ns"], result["interval"]["end_mono_ns"]
intervals_ok &= any(lo <= int(r["submit_mono_ns"]) <= hi for r in records)
common = {
"one_ready_marker": len(ready) == 1,
"compile_capture_pre_ready": not post_ready_events,
"warmup_exact_16": warm["exact_output_count"] >= 16,
"warmup_long": warm["selection"]["long_gt4096"] >= 1,
"layer1_contiguous": indices == list(range(len(indices))),
"written_matches_records": sidecar.get("written_records") == len(records),
"encoded_balanced": sidecar.get("encoded_records") == sidecar.get("written_records") + sidecar.get("dropped_records"),
"last_step_matches": bool(records) and sidecar.get("last_step_index") == records[-1]["step_index"],
"layer1_zero_drops": sidecar.get("dropped_records") == 0,
"anchor_intervals_present": intervals_ok,
}
if footers:
accounting = {
"one_footer_last": len(footers) == 1 and decoded[-1] is footers[0],
"sidecar_final": sidecar.get("final") is True,
"footer_sidecar_agrees": all(
footers[0].get(key) == sidecar.get(key)
for key in ("encoded_records", "written_records", "dropped_records")
),
}
accounting_mode = "graceful-footer"
else:
delta = abs(streams[0].stat().st_mtime_ns - int(sidecar["checkpoint_wall_ns"])) / 1e9
accounting = {
"checkpoint_sidecar": sidecar.get("final") is False,
"checkpoint_within_flush_of_stream": delta <= float(sidecar["flush_interval_seconds"]) + .1,
}
accounting_mode = "checkpoint-sidecar-fallback"
invariants = {**common, **accounting}
if not all(invariants.values()):
raise RuntimeError(f"cell validity failure {entry['cell']}: {invariants}")
result = {
"cell": entry["cell"], "invariants": invariants, "layer1_records": len(records),
"stream": str(streams[0]), "post_ready_capture_events": post_ready_events,
"accounting_mode": accounting_mode,
}
atomic_json(entry["dir"] / "cell-valid.json", result)
return result
def historical_expected() -> dict[tuple[str, float], dict[str, Any]]:
ground = json.loads(GROUND.read_text())
result = {}
for cell in ground["cells"]:
for probe in cell["probe_history"]:
result[(cell["cell_id"], float(probe["sampling_u"]))] = probe
return result
def execute_wave(index: int, state: dict[str, Any], expected: dict[tuple[str, float], dict[str, Any]]) -> None:
wave_name, assignments, estimate = WAVES[index]
if state["waves"].get(wave_name, {}).get("status") == "complete":
return
future = sum(w[2] for w in WAVES[index:]) + .10
if state["gpu_hours_total"] + future >= GPU_LIMIT:
raise RuntimeError(f"projected budget exceeds cap before {wave_name}: {state['gpu_hours_total']+future}")
echo = (
f"WAVE_ECHO wave={wave_name} assignments="
+ ",".join(f"{cell}:gpu{'+'.join(map(str, gpus))}" for cell, gpus in assignments)
+ f" spent_h20h={state['gpu_hours_total']:.6f} wave_est_h20h={estimate:.3f} "
+ f"remaining_projection_h20h={future:.3f} cap_h20h={GPU_LIMIT:.1f} "
+ f"ground_truth={GROUND} workload=chat_w20260311_1000.jsonl"
)
with (RUN_ROOT / "launch-echo.log").open("a") as handle:
handle.write(echo + "\n")
print(echo, flush=True)
assert_all_idle()
wave_dir = RUN_ROOT / "waves" / wave_name
wave_dir.mkdir(parents=True, exist_ok=True)
entries = []
state["status"] = "running"
state["waves"][wave_name] = {"status": "starting", "estimate_h20_hours": estimate, "started_at": time.time()}
save_state(state)
failure = None
try:
for offset, (cell, gpus) in enumerate(assignments):
cell_dir = RUN_ROOT / "cells" / cell
cell_dir.mkdir(parents=True, exist_ok=True)
port = 8500 + index * 10 + offset
command = server_command(cell, gpus, port)
with (cell_dir / "commands.log").open("a") as f:
f.write(f"SERVER {shlex.join(command)}\n")
handle = (cell_dir / "server.log").open("ab", buffering=0)
env = os.environ.copy()
env.update({
"CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)),
"VLLM_OPPROF_DIR": str(cell_dir / "opprof"),
"OPPROF_PHASE6_MARKER": MARKER, "AITUNER_ROOT": str(AITUNER),
"HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1", "PYTHONUNBUFFERED": "1",
})
server = subprocess.Popen(command, cwd=SOURCE, env=env, stdout=handle, stderr=subprocess.STDOUT, start_new_session=True)
OWNED_PGIDS.add(server.pid)
entries.append({
"cell": cell, "gpus": gpus, "port": port, "dir": cell_dir,
"server": server, "server_handle": handle, "spawned_at": time.time(),
})
for entry in entries:
wait_ready(entry)
state["waves"][wave_name]["status"] = "warmup"
save_state(state)
run_clients(entries, [
(e, CELLS[e["cell"]]["peak"], e["dir"] / "warmup") for e in entries
], state, wave_name, warmup=True)
state["waves"][wave_name]["status"] = "peaks"
save_state(state)
peaks = run_clients(entries, [
(e, CELLS[e["cell"]]["peak"], e["dir"] / f"anchor-{CELLS[e['cell']]['peak']}")
for e in entries
], state, wave_name)
for result in peaks:
old = expected[(result["cell"], float(result["anchor"]))]
if result["selection"]["count"] != old["request_count"]:
raise RuntimeError(f"selection mismatch {result['cell']} peak")
neighbor_jobs = []
for entry, result in zip(entries, peaks, strict=True):
key = "upper" if result["feasible"] else "lower"
anchor = CELLS[entry["cell"]][key]
neighbor_jobs.append((entry, anchor, entry["dir"] / f"anchor-{anchor}"))
neighbors = run_clients(entries, neighbor_jobs, state, wave_name)
for result in neighbors:
old = expected[(result["cell"], float(result["anchor"]))]
if result["selection"]["count"] != old["request_count"]:
raise RuntimeError(f"selection mismatch {result['cell']} neighbor")
trap_entry = next((e for e in entries if CELLS[e["cell"]].get("trap")), None)
if trap_entry is not None:
used = {float(item["anchor"]) for item in trap_entry["results"] if item["kind"] == "anchor"}
extra = next(CELLS[trap_entry["cell"]][k] for k in ("lower", "upper") if CELLS[trap_entry["cell"]][k] not in used)
extra_result = run_clients(entries, [(trap_entry, extra, trap_entry["dir"] / f"anchor-{extra}")], state, wave_name)[0]
if extra_result["selection"]["count"] != expected[(extra_result["cell"], float(extra))]["request_count"]:
raise RuntimeError("trap extra selection mismatch")
# Confirm only protocol-triggered anchors while the relevant server is hot.
triggers = []
for entry in entries:
for item in entry.get("results", []):
if item["kind"] != "anchor":
continue
result = json.loads((Path(item["dir"]) / "result.json").read_text())
old = expected[(entry["cell"], float(result["anchor"]))]
flip = bool(result["feasible"]) != bool(old["feasible"])
if .93 <= float(result["pass_rate"]) <= .97 or (entry["cell"] in {"tp2_mns32", "tp4_mns16"} and flip):
priority = 0 if entry["cell"] == "tp2_mns32" else (1 if entry["cell"] == "tp4_mns16" else 2)
triggers.append((priority, entry, float(result["anchor"])))
triggers.sort(key=lambda x: x[0])
for _priority, entry, anchor in triggers:
projected_extra = len(entry["gpus"]) * 80 / 3600
future_primary = sum(w[2] for w in WAVES[index + 1:])
if state["gpu_hours_total"] + live_gpu_hours(entries) + future_primary + projected_extra + .03 >= GPU_LIMIT:
state.setdefault("unconfirmed_triggers", []).append({"cell": entry["cell"], "anchor": anchor})
continue
confirm_index = 1 + sum(1 for item in entry.get("results", []) if item["kind"] == "anchor" and Path(item["dir"]).name.startswith("confirm"))
out = entry["dir"] / f"confirm-{confirm_index}-anchor-{anchor}"
run_clients(entries, [(entry, anchor, out)], state, wave_name)
state["completed_confirmations"] += 1
state["waves"][wave_name]["status"] = "stopping"
save_state(state)
except Exception as error:
failure = error
finally:
for entry in entries:
try:
stop_entry(entry)
except Exception as error:
failure = failure or error
time.sleep(2)
try:
assert_all_idle()
except Exception as error:
failure = failure or error
wave_hours = live_gpu_hours(entries)
state["gpu_hours_total"] += wave_hours
state["waves"][wave_name]["gpu_hours"] = wave_hours
if failure is not None:
state["waves"][wave_name]["status"] = "failed"
state["waves"][wave_name]["failure"] = repr(failure)
state["status"] = "failed"
state["failures"].append({"wave": wave_name, "failure": repr(failure)})
save_state(state)
raise failure
validations = [validate_cell(entry) for entry in entries]
primary_count = sum(
1 for entry in entries for item in entry.get("results", [])
if item["kind"] == "anchor" and not Path(item["dir"]).name.startswith("confirm")
)
state["completed_primary_anchors"] += primary_count
state["waves"][wave_name].update({
"status": "complete", "completed_at": time.time(), "primary_anchors": primary_count,
"validations": validations,
})
save_state(state)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--resume", action="store_true")
args = parser.parse_args()
RUN_ROOT.mkdir(parents=True, exist_ok=True)
state = load_state()
expected = historical_expected()
state["status"] = "running"
save_state(state)
for index in range(len(WAVES)):
execute_wave(index, state, expected)
state["status"] = "primary_complete"
state["completed_at"] = time.time()
save_state(state)
print(json.dumps({
"status": state["status"], "primary_anchors": state["completed_primary_anchors"],
"confirmations": state["completed_confirmations"], "gpu_hours": state["gpu_hours_total"],
}, sort_keys=True))
if __name__ == "__main__":
main()

419
runs/opprof-phase6/opprof_phase6_solo_controller.py Normal file
View File

@@ -0,0 +1,419 @@
#!/usr/bin/env python3
"""A-P6-2 serialized solo controller for authoritative Phase-6 frontiers."""
from __future__ import annotations
import json
import math
import os
import shlex
import subprocess
import time
from pathlib import Path
from typing import Any
import opprof_phase6_controller as base
SOLO_ROOT = base.RUN_ROOT / "solo-authoritative"
STATE = SOLO_ROOT / "controller-state.json"
CAMPAIGN_STATE = base.RUN_ROOT / "controller-state.json"
GPU_LIMIT = 6.0
SAFETY_HOURS = 0.20
MARKER = "opprof-phase6-solo-A-P6-2"
TRACE = base.AITUNER / "trace_windows/traces/chat_w20260311_1000.jsonl"
ORDER = [
"tp4_mns32", "tp4_mns64", "tp2_mns32", "tp2_mns64",
"tp4_mns16", "tp2_mns8", "tp2_mns16", "tp4_mns8",
"tp1_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64",
]
# Exact co-located primaries are remeasured before adaptive crawl. W4 had no
# prior primary, so it starts at P and selects L/U from the solo result.
CORE = {
"tp1_mns8": [base.CELLS["tp1_mns8"]["peak"], base.CELLS["tp1_mns8"]["lower"]],
"tp1_mns16": [base.CELLS["tp1_mns16"]["peak"], base.CELLS["tp1_mns16"]["upper"]],
"tp1_mns32": [base.CELLS["tp1_mns32"]["peak"], base.CELLS["tp1_mns32"]["upper"]],
"tp1_mns64": [base.CELLS["tp1_mns64"]["peak"], base.CELLS["tp1_mns64"]["upper"]],
"tp2_mns8": [base.CELLS["tp2_mns8"]["peak"], base.CELLS["tp2_mns8"]["lower"]],
"tp2_mns16": [base.CELLS["tp2_mns16"]["peak"], base.CELLS["tp2_mns16"]["lower"]],
"tp2_mns32": [base.CELLS["tp2_mns32"]["peak"], base.CELLS["tp2_mns32"]["lower"]],
"tp2_mns64": [base.CELLS["tp2_mns64"]["peak"], base.CELLS["tp2_mns64"]["lower"]],
"tp4_mns8": [base.CELLS["tp4_mns8"]["peak"], base.CELLS["tp4_mns8"]["lower"]],
"tp4_mns16": [
base.CELLS["tp4_mns16"]["peak"], base.CELLS["tp4_mns16"]["lower"],
base.CELLS["tp4_mns16"]["upper"],
],
"tp4_mns32": [base.CELLS["tp4_mns32"]["peak"]],
"tp4_mns64": [base.CELLS["tp4_mns64"]["peak"]],
}
CELL_ESTIMATE = {cell: {1: .11, 2: .22, 4: .48}[cfg["tp"]] for cell, cfg in base.CELLS.items()}
def atomic_json(path: Path, value: Any) -> None:
base.atomic_json(path, value)
def load_state() -> dict[str, Any]:
if STATE.exists():
return json.loads(STATE.read_text())
campaign = json.loads(CAMPAIGN_STATE.read_text())
return {
"schema": 1, "amendment": "A-P6-2", "status": "initialized",
"hard_cap_h20_hours": GPU_LIMIT,
"prior_h20_hours": float(campaign["gpu_hours_total"]),
"gpu_hours_total": float(campaign["gpu_hours_total"]),
"solo_gpu_hours": 0.0, "completed_cells": 0,
"primary_anchors": 0, "confirmations": 0,
"cells": {}, "failures": [], "started_at": time.time(),
}
def save_state(state: dict[str, Any]) -> None:
atomic_json(STATE, state)
def historical() -> tuple[dict[tuple[str, float], dict[str, Any]], dict[str, list[float]]]:
ground = json.loads(base.GROUND.read_text())
expected = {}
histories = {}
for cell in ground["cells"]:
anchors = []
for probe in cell["probe_history"]:
anchor = float(probe["sampling_u"])
expected[(cell["cell_id"], anchor)] = probe
anchors.append(anchor)
histories[cell["cell_id"]] = sorted(anchors)
return expected, histories
def same_anchor(left: float, right: float) -> bool:
return math.isclose(left, right, rel_tol=0, abs_tol=1e-15)
def colocated_primary(cell: str, anchor: float) -> dict[str, Any] | None:
cell_dir = base.RUN_ROOT / "cells" / cell
for path in cell_dir.glob("anchor-*/result.json"):
item = json.loads(path.read_text())
if same_anchor(float(item["anchor"]), anchor):
return item
return None
def append_echo(line: str) -> None:
SOLO_ROOT.mkdir(parents=True, exist_ok=True)
with (SOLO_ROOT / "launch-echo.log").open("a") as handle:
handle.write(line + "\n")
print(line, flush=True)
def wait_all_idle(timeout: float = 30.0) -> None:
deadline = time.monotonic() + timeout
error = None
while time.monotonic() < deadline:
try:
base.assert_all_idle()
return
except RuntimeError as current:
error = current
time.sleep(1)
raise error or RuntimeError("GPU cleanup did not reach idle")
def remaining_projection(index: int) -> float:
return sum(CELL_ESTIMATE[cell] for cell in ORDER[index:]) + SAFETY_HOURS
def start_entry(cell: str, index: int) -> dict[str, Any]:
cfg = base.CELLS[cell]
gpus = tuple(range(int(cfg["tp"])))
cell_dir = SOLO_ROOT / "cells" / cell
cell_dir.mkdir(parents=True, exist_ok=True)
port = 8700 + index
command = base.server_command(cell, gpus, port)
with (cell_dir / "commands.log").open("a") as handle:
handle.write(f"SERVER {shlex.join(command)}\n")
server_handle = (cell_dir / "server.log").open("ab", buffering=0)
env = os.environ.copy()
env.update({
"CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)),
"VLLM_OPPROF_DIR": str(cell_dir / "opprof"),
"OPPROF_PHASE6_MARKER": MARKER, "AITUNER_ROOT": str(base.AITUNER),
"HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1", "PYTHONUNBUFFERED": "1",
})
server = subprocess.Popen(
command, cwd=base.SOURCE, env=env, stdout=server_handle,
stderr=subprocess.STDOUT, start_new_session=True,
)
base.OWNED_PGIDS.add(server.pid)
return {
"cell": cell, "gpus": gpus, "port": port, "dir": cell_dir,
"server": server, "server_handle": server_handle,
"spawned_at": time.time(), "results": [],
}
def run_one(
entry: dict[str, Any], anchor: float, out: Path, state: dict[str, Any],
cell_state: dict[str, Any], role: str,
) -> dict[str, Any]:
result = base.run_clients(
[entry], [(entry, anchor, out)], state, f"solo-{entry['cell']}"
)[0]
expected_count = cell_state["expected_counts"][str(anchor)]
if int(result["selection"]["count"]) != int(expected_count):
raise RuntimeError(
f"selection mismatch {entry['cell']} {anchor}: "
f"{result['selection']['count']} != {expected_count}"
)
cell_state.setdefault("runs", []).append({
"anchor": anchor, "role": role, "dir": str(out),
"pass_rate": result["pass_rate"], "feasible": result["feasible"],
})
save_state(state)
return result
def anchor_trials(cell_state: dict[str, Any], anchor: float) -> list[dict[str, Any]]:
return [
item for item in cell_state.get("runs", [])
if same_anchor(float(item["anchor"]), anchor)
]
def accepted_feasible(cell_state: dict[str, Any], anchor: float) -> bool | None:
trials = anchor_trials(cell_state, anchor)
votes = [bool(item["feasible"]) for item in trials]
if not votes:
return None
if len(votes) == 1 or len(set(votes)) == 1:
return votes[0]
if len(votes) >= 3:
return sum(votes) >= 2
return None
def optional_fits(
state: dict[str, Any], entry: dict[str, Any], future_after: float,
) -> bool:
replay = len(entry["gpus"]) * 80 / 3600
projected = (
float(state["gpu_hours_total"]) + base.live_gpu_hours([entry])
+ future_after + replay + SAFETY_HOURS
)
return projected < GPU_LIMIT
def maybe_confirm(
entry: dict[str, Any], anchor: float, primary: dict[str, Any],
state: dict[str, Any], cell_state: dict[str, Any], expected: dict[tuple[str, float], dict[str, Any]],
future_after: float,
) -> None:
old = expected[(entry["cell"], anchor)]
coloc = colocated_primary(entry["cell"], anchor)
disagreement = (
bool(primary["feasible"]) != bool(old["feasible"])
or (coloc is not None and bool(primary["feasible"]) != bool(coloc["feasible"]))
)
boundary = .93 <= float(primary["pass_rate"]) <= .97
if not (disagreement or boundary):
return
while len(anchor_trials(cell_state, anchor)) < 3:
trials = anchor_trials(cell_state, anchor)
if len(trials) >= 2 and len({bool(item["feasible"]) for item in trials}) == 1:
return
if not optional_fits(state, entry, future_after):
cell_state.setdefault("deferred_confirmations", []).append(anchor)
return
trial = len(trials) + 1
out = entry["dir"] / f"confirm-{trial - 1}-anchor-{anchor}"
run_one(entry, anchor, out, state, cell_state, f"confirmation-{trial}")
state["confirmations"] += 1
def run_primary(
entry: dict[str, Any], anchor: float, state: dict[str, Any],
cell_state: dict[str, Any], expected: dict[tuple[str, float], dict[str, Any]],
future_after: float, role: str,
) -> dict[str, Any]:
existing = [item for item in cell_state.get("runs", []) if item["role"].startswith("primary")]
if any(same_anchor(float(item["anchor"]), anchor) for item in existing):
path = next(
Path(item["dir"]) for item in existing
if same_anchor(float(item["anchor"]), anchor)
)
return json.loads((path / "result.json").read_text())
out = entry["dir"] / f"anchor-{anchor}"
result = run_one(entry, anchor, out, state, cell_state, role)
state["primary_anchors"] += 1
maybe_confirm(entry, anchor, result, state, cell_state, expected, future_after)
return result
def next_below(history: list[float], tested: set[float]) -> float | None:
if not tested:
return None
candidates = [x for x in history if x < min(tested) and x not in tested]
return max(candidates) if candidates else None
def next_above(history: list[float], tested: set[float]) -> float | None:
if not tested:
return None
candidates = [x for x in history if x > max(tested) and x not in tested]
return min(candidates) if candidates else None
def execute_cell(
index: int, cell: str, state: dict[str, Any],
expected: dict[tuple[str, float], dict[str, Any]], histories: dict[str, list[float]],
) -> None:
if state["cells"].get(cell, {}).get("status") == "complete":
return
future = remaining_projection(index)
if float(state["gpu_hours_total"]) + future >= GPU_LIMIT:
state["status"] = "budget_projection_stop"
state["budget_stop"] = {
"before_cell": cell, "spent_h20_hours": state["gpu_hours_total"],
"remaining_projection_h20_hours": future,
"projected_total_h20_hours": state["gpu_hours_total"] + future,
"hard_cap_h20_hours": GPU_LIMIT,
}
save_state(state)
raise RuntimeError(f"projected budget exceeds cap before {cell}")
cfg = base.CELLS[cell]
echo = (
f"SOLO_WAVE_ECHO cell={cell} tp={cfg['tp']} mns={cfg['mns']} "
f"gpus=0-{int(cfg['tp'])-1} mandatory={','.join(map(str, CORE[cell]))} "
f"spent_h20h={state['gpu_hours_total']:.6f} cell_est_h20h={CELL_ESTIMATE[cell]:.3f} "
f"remaining_projection_h20h={future:.3f} cap_h20h={GPU_LIMIT:.1f} "
f"ground_truth={base.GROUND} trace={TRACE}"
)
append_echo(echo)
wait_all_idle()
cell_state = {
"status": "starting", "started_at": time.time(), "tp": cfg["tp"], "mns": cfg["mns"],
"mandatory": CORE[cell],
"expected_counts": {
str(anchor): expected[(cell, anchor)]["request_count"] for anchor in histories[cell]
},
"runs": [],
}
state["status"] = "running"
state["cells"][cell] = cell_state
save_state(state)
entry = start_entry(cell, index)
failure = None
future_after = sum(CELL_ESTIMATE[item] for item in ORDER[index + 1:])
try:
base.wait_ready(entry)
cell_state["status"] = "warmup"
save_state(state)
warm = base.run_clients(
[entry], [(entry, cfg["peak"], entry["dir"] / "warmup")],
state, f"solo-{cell}", warmup=True,
)[0]
cell_state["warmup"] = {
"exact_output_count": warm["exact_output_count"],
"long_gt4096": warm["selection"]["long_gt4096"],
}
cell_state["status"] = "mandatory"
save_state(state)
for anchor in CORE[cell]:
run_primary(entry, anchor, state, cell_state, expected, future_after, "primary-mandatory")
peak = float(cfg["peak"])
peak_vote = accepted_feasible(cell_state, peak)
if cell in {"tp4_mns32", "tp4_mns64"} and peak_vote is not None:
direction = float(cfg["upper"] if peak_vote else cfg["lower"])
run_primary(entry, direction, state, cell_state, expected, future_after, "primary-direction")
cell_state["status"] = "crawl"
save_state(state)
while True:
primary_anchors = {
float(item["anchor"]) for item in cell_state["runs"]
if item["role"].startswith("primary")
}
votes = {anchor: accepted_feasible(cell_state, anchor) for anchor in primary_anchors}
pass_anchors = [anchor for anchor, vote in votes.items() if vote is True]
fail_anchors = [anchor for anchor, vote in votes.items() if vote is False]
if pass_anchors and fail_anchors and max(pass_anchors) < min(fail_anchors):
break
if any(vote is None for vote in votes.values()):
cell_state["censor"] = "UNRESOLVED_SOLO_ANCHOR"
break
if pass_anchors and not fail_anchors:
candidate = next_above(histories[cell], primary_anchors)
elif fail_anchors and not pass_anchors:
candidate = next_below(histories[cell], primary_anchors)
else:
# Non-monotonic anchors already contain both states but no valid bracket.
cell_state["censor"] = "NONMONOTONIC_SOLO_ANCHORS"
break
if candidate is None:
cell_state["censor"] = "HISTORY_EDGE"
break
if not optional_fits(state, entry, future_after):
cell_state["censor"] = "BUDGET_CENSORED"
break
run_primary(entry, candidate, state, cell_state, expected, future_after, "primary-crawl")
cell_state["status"] = "stopping"
save_state(state)
except Exception as error:
failure = error
finally:
try:
base.stop_entry(entry)
except Exception as error:
failure = failure or error
time.sleep(2)
try:
wait_all_idle()
except Exception as error:
failure = failure or error
hours = base.live_gpu_hours([entry])
state["gpu_hours_total"] += hours
state["solo_gpu_hours"] += hours
cell_state["gpu_hours"] = hours
if failure is not None:
cell_state["status"] = "failed"
cell_state["failure"] = repr(failure)
state["status"] = "failed"
state["failures"].append({"cell": cell, "failure": repr(failure)})
save_state(state)
raise failure
validation = base.validate_cell(entry)
cell_state["validation"] = validation
cell_state["status"] = "complete"
cell_state["completed_at"] = time.time()
state["completed_cells"] += 1
save_state(state)
def main() -> None:
SOLO_ROOT.mkdir(parents=True, exist_ok=True)
base.GPU_LIMIT = GPU_LIMIT
base.MARKER = MARKER
expected, histories = historical()
state = load_state()
state["status"] = "running"
save_state(state)
for index, cell in enumerate(ORDER):
execute_cell(index, cell, state, expected, histories)
state["status"] = "complete"
state["completed_at"] = time.time()
save_state(state)
print(json.dumps({
"status": state["status"], "cells": state["completed_cells"],
"primary_anchors": state["primary_anchors"],
"confirmations": state["confirmations"],
"solo_gpu_hours": state["solo_gpu_hours"],
"campaign_gpu_hours": state["gpu_hours_total"],
}, sort_keys=True))
if __name__ == "__main__":
main()

103
runs/opprof-phase6/p3-reference/primary/P10-C00/moderate/client/result.json Normal file
View File

@@ -0,0 +1,103 @@
{
"admission_stop_s": 363.15318555399426,
"arrival": "steady",
"clean": {
"admitted": 113,
"completed": 111,
"completed_throughput_rps": 0.4625,
"duration_s": 240.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 968714,
"offered_rps": 0.4708333333333333,
"output_tokens": 25409,
"start_s": 60.0
},
"clean_segment_seconds": 80.0,
"drain_seconds": 1.3658369119802956,
"elapsed_seconds": 364.51902246597456,
"failed_records": 0,
"load_point": "moderate",
"manifest_admitted": 172,
"manifest_exhausted": false,
"manifest_rows": 4011,
"manifest_sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33",
"manifest_wrapped": false,
"max_in_flight": 7,
"num_clean_segments": 3,
"profiles": [
{
"start_call_s": 300.00394913199125,
"start_return_s": 300.03973603399936,
"start_status": 200,
"stop_call_s": 300.89519165997626,
"stop_return_s": 301.6286224719952,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849466906401944.pt.trace.json.gz",
"trace_ready_s": 300.8951868820004,
"trace_sha256": "a15670f86d843d40e7c4b6f354dd0193ea1259824a96440392d923f5270ba095",
"window": 1
},
{
"start_call_s": 331.62960390897933,
"start_return_s": 331.63382844498847,
"start_status": 200,
"stop_call_s": 332.4971265429922,
"stop_return_s": 333.15040952397976,
"stop_status": 200,
"trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849498465727706.pt.trace.json.gz",
"trace_ready_s": 332.4971234249824,
"trace_sha256": "0f5962e193e09a9b2f1df6cea23f92419dcf2b322ee4ef2135a4fc3e1fe0617c",
"window": 2
}
],
"rate_fraction": 0.6,
"records": 172,
"request_rate": 0.4725,
"schema": 1,
"segments": [
{
"admitted": 38,
"completed": 37,
"completed_throughput_rps": 0.4625,
"duration_s": 80.0,
"end_s": 140.0,
"failed": 0,
"input_tokens": 404192,
"name": "A",
"offered_rps": 0.475,
"output_tokens": 8525,
"start_s": 60.0
},
{
"admitted": 37,
"completed": 38,
"completed_throughput_rps": 0.475,
"duration_s": 80.0,
"end_s": 220.0,
"failed": 0,
"input_tokens": 327301,
"name": "B",
"offered_rps": 0.4625,
"output_tokens": 8617,
"start_s": 140.0
},
{
"admitted": 38,
"completed": 36,
"completed_throughput_rps": 0.45,
"duration_s": 80.0,
"end_s": 300.0,
"failed": 0,
"input_tokens": 237221,
"name": "C",
"offered_rps": 0.475,
"output_tokens": 8267,
"start_s": 220.0
}
],
"successful_records": 172,
"t0_mono_ns": 180739547167834,
"t0_wall_ns": 1783849166673595809,
"warmup_seconds": 60.0
}

1
runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns16", "anchor": 0.24609375, "kind": "anchor", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.24609375,
"cell": "tp1_mns16",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 141,
"feasible": true,
"interval": {
"elapsed_s": 61.328174978,
"end_mono_ns": 199601637648726,
"end_wall_ns": 1783868028764076133,
"start_mono_ns": 199540309473748,
"start_wall_ns": 1783867967435901551
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "anchor",
"mns": 16,
"observed_count": 141,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4",
"arrival_s": {
"distinct_n": 141,
"finite_n": 141,
"max": 59.78950000000005,
"min": 0.008300000000008368,
"missing_n": 0,
"n": 141
},
"count": 141,
"long_gt4096": 50,
"offered_req_s": 2.35,
"offered_req_s_per_gpu": 2.35,
"raw_input_tokens": {
"distinct_n": 133,
"finite_n": 141,
"max": 8149.0,
"min": 72.0,
"missing_n": 0,
"n": 141
},
"raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba",
"request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3"
},
"slo_pass_count": 141,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 141,
"finite_n": 141,
"max": 39.29021051171873,
"min": 4.445230456776771,
"missing_n": 0,
"n": 141
},
"ttft_ms": {
"distinct_n": 141,
"finite_n": 141,
"max": 1787.3226249939762,
"min": 37.47074800776318,
"missing_n": 0,
"n": 141
}
}

1
runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns16", "anchor": 0.25, "kind": "anchor", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.25,
"cell": "tp1_mns16",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 143,
"feasible": true,
"interval": {
"elapsed_s": 61.471211524,
"end_mono_ns": 199668842271892,
"end_wall_ns": 1783868095968699176,
"start_mono_ns": 199607371060368,
"start_wall_ns": 1783868034497487431
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "anchor",
"mns": 16,
"observed_count": 143,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366",
"arrival_s": {
"distinct_n": 143,
"finite_n": 143,
"max": 59.78950000000005,
"min": 0.008300000000008368,
"missing_n": 0,
"n": 143
},
"count": 143,
"long_gt4096": 51,
"offered_req_s": 2.3833333333333333,
"offered_req_s_per_gpu": 2.3833333333333333,
"raw_input_tokens": {
"distinct_n": 135,
"finite_n": 143,
"max": 8149.0,
"min": 72.0,
"missing_n": 0,
"n": 143
},
"raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0",
"request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda"
},
"slo_pass_count": 143,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 143,
"finite_n": 143,
"max": 44.43295162989699,
"min": 4.492281684945301,
"missing_n": 0,
"n": 143
},
"ttft_ms": {
"distinct_n": 143,
"finite_n": 143,
"max": 1784.063341008732,
"min": 57.29520702152513,
"missing_n": 0,
"n": 143
}
}

21
runs/opprof-phase6/phase6/cells/tp1_mns16/cell-valid.json Normal file
View File

@@ -0,0 +1,21 @@
{
"accounting_mode": "A-P6-1-checkpoint-sidecar",
"cell": "tp1_mns16",
"invariants": {
"all_anchor_intervals_covered": true,
"all_schema_1": true,
"checkpoint_after_all_anchor_intervals": true,
"checkpoint_sidecar": true,
"checkpoint_within_flush_of_stream": true,
"complete_final_newline": true,
"encoded_balanced": true,
"last_step_matches": true,
"no_in_stream_footer": true,
"steps_contiguous": true,
"two_anchor_intervals": true,
"written_matches_records": true,
"zero_drops": true
},
"layer1_records": 9212,
"stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl"
}

4
runs/opprof-phase6/phase6/cells/tp1_mns16/commands.log Normal file
View File

@@ -0,0 +1,4 @@
SERVER taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8501 --served-model-name qwen3-30b-a3b-community --max-num-batched-tokens 8192 --max-num-seqs 16 --tensor-parallel-size 1 --shutdown-timeout 120
CLIENT taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py warmup --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.24609375 --tp 1 --mns 16 --base-url http://127.0.0.1:8501 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/warmup
CLIENT taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.24609375 --tp 1 --mns 16 --base-url http://127.0.0.1:8501 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.24609375
CLIENT taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.25 --tp 1 --mns 16 --base-url http://127.0.0.1:8501 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.25

1
runs/opprof-phase6/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl.footer.json Normal file
View File

@@ -0,0 +1 @@
{"schema":1,"record_type":"footer_checkpoint","stream":"opprof-v1-dp0-pid2638886-1783867898065648658.jsonl","encoded_records":9212,"written_records":9212,"dropped_records":0,"last_step_index":9211,"checkpoint_wall_ns":1783868096969027153,"flush_interval_seconds":1.0,"final":false}

442
runs/opprof-phase6/phase6/cells/tp1_mns16/server.log Normal file
View File

@@ -0,0 +1,442 @@
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339]
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339]
(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8501, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120}
(APIServer pid=2638187) INFO 07-12 14:50:52 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2638187) INFO 07-12 14:50:52 [model.py:1725] Using max model len 40960
(APIServer pid=2638187) INFO 07-12 14:50:52 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2638187) INFO 07-12 14:50:52 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2638187) INFO 07-12 14:50:52 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2638886) INFO 07-12 14:51:03 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2638886) INFO 07-12 14:51:05 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:37021 backend=nccl
(EngineCore pid=2638886) INFO 07-12 14:51:05 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=2638886) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=2638886) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(EngineCore pid=2638886) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=2638886) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3
(EngineCore pid=2638886) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(EngineCore pid=2638886) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB.
(EngineCore pid=2638886) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:01<00:15, 1.06s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:02<00:14, 1.07s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:03<00:13, 1.06s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:04<00:13, 1.10s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:05<00:11, 1.09s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:06<00:10, 1.09s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:07<00:09, 1.10s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:08<00:08, 1.09s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:09<00:07, 1.08s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:10<00:06, 1.06s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:11<00:05, 1.06s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:12<00:04, 1.04s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:13<00:03, 1.06s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:15<00:02, 1.09s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:16<00:01, 1.09s/it]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:16<00:00, 1.17it/s]
(EngineCore pid=2638886)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:16<00:00, 1.03s/it]
(EngineCore pid=2638886)
(EngineCore pid=2638886) INFO 07-12 14:51:25 [default_loader.py:430] Loading weights took 16.52 seconds
(EngineCore pid=2638886) INFO 07-12 14:51:25 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=2638886) INFO 07-12 14:51:25 [gpu_model_runner.py:5259] Model loading took 56.88 GiB memory and 17.268242 seconds
(EngineCore pid=2638886) INFO 07-12 14:51:29 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/f4a50989f8/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=2638886) INFO 07-12 14:51:29 [backends.py:1148] Dynamo bytecode transform time: 3.31 s
(EngineCore pid=2638886) INFO 07-12 14:51:31 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.252 s
(EngineCore pid=2638886) INFO 07-12 14:51:31 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/fe5a82dbe929018b7aea7dee05b5cd31a21fe2682aca78ee4cbf2b37c8a086d6/rank_0_0/model
(EngineCore pid=2638886) INFO 07-12 14:51:31 [monitor.py:53] torch.compile took 5.96 s in total
(EngineCore pid=2638886) INFO 07-12 14:51:32 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2638886) INFO 07-12 14:51:32 [monitor.py:81] Initial profiling/warmup run took 0.20 s
(EngineCore pid=2638886) INFO 07-12 14:51:32 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=7 (largest=32), FULL=5 (largest=16)
(EngineCore pid=2638886) INFO 07-12 14:51:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.08 GiB total
(EngineCore pid=2638886) INFO 07-12 14:51:34 [gpu_worker.py:508] Available KV cache memory: 29.43 GiB
(EngineCore pid=2638886) INFO 07-12 14:51:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9191 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9209. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2638886) INFO 07-12 14:51:34 [kv_cache_utils.py:2146] GPU KV cache size: 321,456 tokens
(EngineCore pid=2638886) INFO 07-12 14:51:34 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 7.85x
(EngineCore pid=2638886) INFO 07-12 14:51:35 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=2638886) INFO 07-12 14:51:35 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=2638886) 2026-07-12 14:51:35,155 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore pid=2638886) 2026-07-12 14:51:35,199 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=2638886)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 14%|█▍ | 1/7 [00:00<00:00, 9.98it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 43%|████▎ | 3/7 [00:00<00:00, 10.33it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 71%|███████▏ | 5/7 [00:00<00:00, 9.18it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 86%|████████▌ | 6/7 [00:00<00:00, 8.44it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:00<00:00, 7.90it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 7/7 [00:00<00:00, 8.49it/s]
(EngineCore pid=2638886)
Capturing CUDA graphs (decode, FULL): 0%| | 0/5 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 40%|████ | 2/5 [00:00<00:00, 10.74it/s]
Capturing CUDA graphs (decode, FULL): 80%|████████ | 4/5 [00:00<00:00, 11.19it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 5/5 [00:00<00:00, 11.25it/s]
(EngineCore pid=2638886) INFO 07-12 14:51:37 [gpu_model_runner.py:6660] Graph capturing finished in 2 secs, took 0.10 GiB
(EngineCore pid=2638886) INFO 07-12 14:51:37 [gpu_worker.py:667] CUDA graph pool memory: 0.1 GiB (actual), 0.08 GiB (estimated), difference: 0.02 GiB (15.7%).
(EngineCore pid=2638886) INFO 07-12 14:51:37 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2638886) INFO 07-12 14:51:37 [core.py:337] init engine (profile, create kv cache, warmup model) took 11.83 s (compilation: 5.96 s)
(EngineCore pid=2638886) INFO 07-12 14:51:38 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl
(EngineCore pid=2638886) INFO 07-12 14:51:38 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2638886) INFO 07-12 14:51:38 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2638187) INFO 07-12 14:51:38 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2638187) WARNING 07-12 14:51:38 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2638187) INFO 07-12 14:51:38 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2638187) INFO 07-12 14:51:38 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8501
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:37] Available routes are:
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /openapi.json, Methods: GET, HEAD
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /docs, Methods: GET, HEAD
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: GET, HEAD
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /redoc, Methods: GET, HEAD
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2638187) INFO 07-12 14:51:38 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2638187) INFO: Started server process [2638187]
(APIServer pid=2638187) INFO: Waiting for application startup.
(APIServer pid=2638187) INFO: Application startup complete.
(APIServer pid=2638187) INFO: 127.0.0.1:52764 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:52778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638886) WARNING 07-12 14:52:29 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2638187) INFO: 127.0.0.1:52788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:52802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638886) WARNING 07-12 14:52:29 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2638187) INFO: 127.0.0.1:52808 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35810 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35842 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35884 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:52:39 [loggers.py:273] Engine 000: Avg prompt throughput: 5299.4 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 4.2%
(APIServer pid=2638187) INFO: 127.0.0.1:44164 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44180 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44184 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44196 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44212 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:52:49 [loggers.py:273] Engine 000: Avg prompt throughput: 8.5 tokens/s, Avg generation throughput: 67.6 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 4.9%, Prefix cache hit rate: 35.1%
(APIServer pid=2638187) INFO: 127.0.0.1:44244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:44254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38566 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38586 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38608 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38638 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38658 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38674 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38698 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38742 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:52:59 [loggers.py:273] Engine 000: Avg prompt throughput: 2139.7 tokens/s, Avg generation throughput: 267.4 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 43.4%
(APIServer pid=2638187) INFO: 127.0.0.1:38776 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:38806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45670 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45680 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45688 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45712 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45728 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45730 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45738 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45756 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45772 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45786 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:45828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:53:09 [loggers.py:273] Engine 000: Avg prompt throughput: 5435.1 tokens/s, Avg generation throughput: 320.0 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 33.3%
(APIServer pid=2638187) INFO: 127.0.0.1:48040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48094 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48110 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48136 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:53:19 [loggers.py:273] Engine 000: Avg prompt throughput: 4881.2 tokens/s, Avg generation throughput: 164.9 tokens/s, Running: 7 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.6%, Prefix cache hit rate: 29.1%
(APIServer pid=2638187) INFO: 127.0.0.1:48158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48194 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:48204 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39700 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39714 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39750 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39864 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39912 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:53:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6255.1 tokens/s, Avg generation throughput: 265.4 tokens/s, Running: 10 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.7%, Prefix cache hit rate: 24.2%
(APIServer pid=2638187) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39978 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39918 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:39992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40006 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:40072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:53:39 [loggers.py:273] Engine 000: Avg prompt throughput: 9198.7 tokens/s, Avg generation throughput: 388.3 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.2%, Prefix cache hit rate: 20.8%
(APIServer pid=2638187) INFO: 127.0.0.1:40078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42934 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42944 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42958 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:42996 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:43012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:43016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:43018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:43030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:53:49 [loggers.py:273] Engine 000: Avg prompt throughput: 6382.3 tokens/s, Avg generation throughput: 331.2 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 19.1%
(APIServer pid=2638187) INFO: 127.0.0.1:53576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53596 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53606 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53618 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53630 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53654 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53662 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53690 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53702 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:53:59 [loggers.py:273] Engine 000: Avg prompt throughput: 3970.7 tokens/s, Avg generation throughput: 126.8 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.3%, Prefix cache hit rate: 17.8%
(APIServer pid=2638187) INFO: 127.0.0.1:53718 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53880 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53926 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53968 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:53988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54004 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54012 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54026 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54066 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54088 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:54:09 [loggers.py:273] Engine 000: Avg prompt throughput: 6063.3 tokens/s, Avg generation throughput: 290.7 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 10.9%, Prefix cache hit rate: 17.1%
(APIServer pid=2638187) INFO: 127.0.0.1:54092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54104 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:54120 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56386 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56388 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56404 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56416 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56420 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56432 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56434 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56442 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56456 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56466 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56478 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:54:19 [loggers.py:273] Engine 000: Avg prompt throughput: 4203.4 tokens/s, Avg generation throughput: 273.5 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 16.7%
(APIServer pid=2638187) INFO: 127.0.0.1:56492 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:56502 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34858 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34920 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34938 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34954 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34988 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:34994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:35024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:54:29 [loggers.py:273] Engine 000: Avg prompt throughput: 4720.8 tokens/s, Avg generation throughput: 256.7 tokens/s, Running: 4 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 16.3%
(APIServer pid=2638187) INFO: 127.0.0.1:35028 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57030 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57050 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57062 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57076 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57106 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57122 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57146 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57160 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57170 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57186 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57198 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57210 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57232 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:54:39 [loggers.py:273] Engine 000: Avg prompt throughput: 8295.5 tokens/s, Avg generation throughput: 280.7 tokens/s, Running: 16 reqs, Waiting: 1 reqs, GPU KV cache usage: 15.5%, Prefix cache hit rate: 15.5%
(APIServer pid=2638187) INFO: 127.0.0.1:57270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:57276 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36788 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36812 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36824 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36838 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36850 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36856 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36870 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36890 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36908 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36922 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36930 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36942 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36946 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO 07-12 14:54:49 [loggers.py:273] Engine 000: Avg prompt throughput: 8540.1 tokens/s, Avg generation throughput: 333.9 tokens/s, Running: 14 reqs, Waiting: 0 reqs, GPU KV cache usage: 17.1%, Prefix cache hit rate: 14.6%
(APIServer pid=2638187) INFO: 127.0.0.1:36952 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638187) INFO: 127.0.0.1:36962 - "POST /v1/chat/completions HTTP/1.1" 200 OK

1
runs/opprof-phase6/phase6/cells/tp1_mns16/warmup.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns16", "anchor": 0.24609375, "kind": "warmup", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns16/warmup/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.24609375,
"cell": "tp1_mns16",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 16,
"feasible": true,
"interval": {
"elapsed_s": 8.031388113,
"end_mono_ns": 199530259142500,
"end_wall_ns": 1783867957385569525,
"start_mono_ns": 199522227754387,
"start_wall_ns": 1783867949354181463
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "warmup",
"mns": 16,
"observed_count": 16,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "1bfb7b673b63b8aaea00c075792180307e1a32e354711a8d3c4978f9a013bc02",
"arrival_s": {
"distinct_n": 16,
"finite_n": 16,
"max": 6.901699999999983,
"min": 0.0,
"missing_n": 0,
"n": 16
},
"count": 16,
"long_gt4096": 8,
"offered_req_s": 0.26666666666666666,
"offered_req_s_per_gpu": 0.26666666666666666,
"raw_input_tokens": {
"distinct_n": 16,
"finite_n": 16,
"max": 7270.0,
"min": 72.0,
"missing_n": 0,
"n": 16
},
"raw_length_order_sha256": "e858719eb07cdeb674aa17d9299d05dec852db11c263bc9391c53cd0f177db1c",
"request_id_order_sha256": "0fa4b7f4dc274280eb173a0ee0974643ab283b887418ce8f10e958e7e8d90161"
},
"slo_pass_count": 16,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 16,
"finite_n": 16,
"max": 28.370322086701652,
"min": 4.984751267873821,
"missing_n": 0,
"n": 16
},
"ttft_ms": {
"distinct_n": 16,
"finite_n": 16,
"max": 696.6323160158936,
"min": 74.85444800113328,
"missing_n": 0,
"n": 16
}
}

1
runs/opprof-phase6/phase6/cells/tp1_mns32/anchor-0.2421875.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns32", "anchor": 0.2421875, "kind": "anchor", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns32/anchor-0.2421875/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.2421875,
"cell": "tp1_mns32",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 137,
"feasible": true,
"interval": {
"elapsed_s": 61.294849452,
"end_mono_ns": 199601604247251,
"end_wall_ns": 1783868028730674502,
"start_mono_ns": 199540309397799,
"start_wall_ns": 1783867967435824912
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "anchor",
"mns": 32,
"observed_count": 137,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "e8e91fd47d9152811ed7bd79e20c0f45ba6677560a419542d9c17abd82bebb4b",
"arrival_s": {
"distinct_n": 137,
"finite_n": 137,
"max": 59.78950000000005,
"min": 0.008300000000008368,
"missing_n": 0,
"n": 137
},
"count": 137,
"long_gt4096": 47,
"offered_req_s": 2.283333333333333,
"offered_req_s_per_gpu": 2.283333333333333,
"raw_input_tokens": {
"distinct_n": 129,
"finite_n": 137,
"max": 8149.0,
"min": 72.0,
"missing_n": 0,
"n": 137
},
"raw_length_order_sha256": "93176915562ff9a118f3550157ddd1025d73a6b70cf4d03be9765de9a1b5d744",
"request_id_order_sha256": "adb62bea7f7a12c1e33fa1572ec1d0e274100013ad90e14c6ef5c549b0e0d017"
},
"slo_pass_count": 137,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 137,
"finite_n": 137,
"max": 35.17025839386134,
"min": 4.435019960625127,
"missing_n": 0,
"n": 137
},
"ttft_ms": {
"distinct_n": 137,
"finite_n": 137,
"max": 1485.6346309825312,
"min": 34.36101900297217,
"missing_n": 0,
"n": 137
}
}

1
runs/opprof-phase6/phase6/cells/tp1_mns32/anchor-0.24609375.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns32", "anchor": 0.24609375, "kind": "anchor", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns32/anchor-0.24609375/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.24609375,
"cell": "tp1_mns32",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 141,
"feasible": true,
"interval": {
"elapsed_s": 61.288241408,
"end_mono_ns": 199668653603418,
"end_wall_ns": 1783868095780030817,
"start_mono_ns": 199607365362010,
"start_wall_ns": 1783868034491789246
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "anchor",
"mns": 32,
"observed_count": 141,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4",
"arrival_s": {
"distinct_n": 141,
"finite_n": 141,
"max": 59.78950000000005,
"min": 0.008300000000008368,
"missing_n": 0,
"n": 141
},
"count": 141,
"long_gt4096": 50,
"offered_req_s": 2.35,
"offered_req_s_per_gpu": 2.35,
"raw_input_tokens": {
"distinct_n": 133,
"finite_n": 141,
"max": 8149.0,
"min": 72.0,
"missing_n": 0,
"n": 141
},
"raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba",
"request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3"
},
"slo_pass_count": 141,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 141,
"finite_n": 141,
"max": 45.32469625197036,
"min": 4.495332385807511,
"missing_n": 0,
"n": 141
},
"ttft_ms": {
"distinct_n": 141,
"finite_n": 141,
"max": 1799.771893012803,
"min": 49.012041999958456,
"missing_n": 0,
"n": 141
}
}

21
runs/opprof-phase6/phase6/cells/tp1_mns32/cell-valid.json Normal file
View File

@@ -0,0 +1,21 @@
{
"accounting_mode": "A-P6-1-checkpoint-sidecar",
"cell": "tp1_mns32",
"invariants": {
"all_anchor_intervals_covered": true,
"all_schema_1": true,
"checkpoint_after_all_anchor_intervals": true,
"checkpoint_sidecar": true,
"checkpoint_within_flush_of_stream": true,
"complete_final_newline": true,
"encoded_balanced": true,
"last_step_matches": true,
"no_in_stream_footer": true,
"steps_contiguous": true,
"two_anchor_intervals": true,
"written_matches_records": true,
"zero_drops": true
},
"layer1_records": 9633,
"stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl"
}

4
runs/opprof-phase6/phase6/cells/tp1_mns32/commands.log Normal file
View File

@@ -0,0 +1,4 @@
SERVER taskset -c 40-59 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8502 --served-model-name qwen3-30b-a3b-community --max-num-batched-tokens 8192 --max-num-seqs 32 --tensor-parallel-size 1 --shutdown-timeout 120
CLIENT taskset -c 40-59 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py warmup --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns32 --anchor 0.2421875 --tp 1 --mns 32 --base-url http://127.0.0.1:8502 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/warmup
CLIENT taskset -c 40-59 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns32 --anchor 0.2421875 --tp 1 --mns 32 --base-url http://127.0.0.1:8502 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/anchor-0.2421875
CLIENT taskset -c 40-59 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns32 --anchor 0.24609375 --tp 1 --mns 32 --base-url http://127.0.0.1:8502 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/anchor-0.24609375

1
runs/opprof-phase6/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl.footer.json Normal file
View File

@@ -0,0 +1 @@
{"schema":1,"record_type":"footer_checkpoint","stream":"opprof-v1-dp0-pid2638900-1783867898685556703.jsonl","encoded_records":9633,"written_records":9633,"dropped_records":0,"last_step_index":9632,"checkpoint_wall_ns":1783868096780742186,"flush_interval_seconds":1.0,"final":false}

436
runs/opprof-phase6/phase6/cells/tp1_mns32/server.log Normal file
View File

@@ -0,0 +1,436 @@
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:339]
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:339] █ █ █▄ ▄█
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:339]
(APIServer pid=2638188) INFO 07-12 14:50:41 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8502, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 32, 'shutdown_timeout': 120}
(APIServer pid=2638188) INFO 07-12 14:50:52 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=2638188) INFO 07-12 14:50:52 [model.py:1725] Using max model len 40960
(APIServer pid=2638188) INFO 07-12 14:50:52 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=2638188) INFO 07-12 14:50:52 [vllm.py:1006] Asynchronous scheduling is enabled.
(APIServer pid=2638188) INFO 07-12 14:50:52 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(EngineCore pid=2638900) INFO 07-12 14:51:03 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=2638900) INFO 07-12 14:51:05 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:50247 backend=nccl
(EngineCore pid=2638900) INFO 07-12 14:51:05 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(EngineCore pid=2638900) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=2638900) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B...
(EngineCore pid=2638900) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
(EngineCore pid=2638900) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3
(EngineCore pid=2638900) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS'].
(EngineCore pid=2638900) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB.
(EngineCore pid=2638900) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00<?, ?it/s]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:01<00:16, 1.07s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 12% Completed | 2/16 [00:02<00:14, 1.07s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 19% Completed | 3/16 [00:03<00:13, 1.06s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 25% Completed | 4/16 [00:04<00:13, 1.10s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 31% Completed | 5/16 [00:05<00:11, 1.09s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 38% Completed | 6/16 [00:06<00:10, 1.09s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 44% Completed | 7/16 [00:07<00:09, 1.10s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 50% Completed | 8/16 [00:08<00:08, 1.09s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 56% Completed | 9/16 [00:09<00:07, 1.08s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 62% Completed | 10/16 [00:10<00:06, 1.06s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 69% Completed | 11/16 [00:11<00:05, 1.06s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 75% Completed | 12/16 [00:12<00:04, 1.04s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 81% Completed | 13/16 [00:13<00:03, 1.06s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 88% Completed | 14/16 [00:15<00:02, 1.09s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 94% Completed | 15/16 [00:16<00:01, 1.09s/it]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:16<00:00, 1.18it/s]
(EngineCore pid=2638900)
Loading safetensors checkpoint shards: 100% Completed | 16/16 [00:16<00:00, 1.03s/it]
(EngineCore pid=2638900)
(EngineCore pid=2638900) INFO 07-12 14:51:25 [default_loader.py:430] Loading weights took 16.49 seconds
(EngineCore pid=2638900) INFO 07-12 14:51:25 [unquantized.py:312] Using MoEPrepareAndFinalizeNoDPEPModular
(EngineCore pid=2638900) INFO 07-12 14:51:25 [gpu_model_runner.py:5259] Model loading took 56.88 GiB memory and 17.231743 seconds
(EngineCore pid=2638900) INFO 07-12 14:51:29 [backends.py:1089] Using cache directory: /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/30ef41b5f5/rank_0_0/backbone for vLLM's torch.compile
(EngineCore pid=2638900) INFO 07-12 14:51:29 [backends.py:1148] Dynamo bytecode transform time: 3.33 s
(EngineCore pid=2638900) INFO 07-12 14:51:31 [backends.py:292] Directly load the compiled graph(s) for compile range (1, 8192) from the cache, took 2.266 s
(EngineCore pid=2638900) INFO 07-12 14:51:32 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/738c624149a63d13eaf115eec4d2189ece948ac500e524d3e06e801d9915352d/rank_0_0/model
(EngineCore pid=2638900) INFO 07-12 14:51:32 [monitor.py:53] torch.compile took 6.02 s in total
(EngineCore pid=2638900) INFO 07-12 14:51:32 [fused_moe.py:1058] Using configuration from /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/model_executor/layers/fused_moe/configs/E=128,N=768,device_name=NVIDIA_H20.json for MoE layer.
(EngineCore pid=2638900) INFO 07-12 14:51:32 [monitor.py:81] Initial profiling/warmup run took 0.20 s
(EngineCore pid=2638900) INFO 07-12 14:51:33 [gpu_model_runner.py:6487] Profiling CUDA graph memory: PIECEWISE=11 (largest=64), FULL=7 (largest=32)
(EngineCore pid=2638900) INFO 07-12 14:51:34 [gpu_model_runner.py:6592] Estimated CUDA graph memory: 0.11 GiB total
(EngineCore pid=2638900) INFO 07-12 14:51:34 [gpu_worker.py:508] Available KV cache memory: 29.4 GiB
(EngineCore pid=2638900) INFO 07-12 14:51:34 [gpu_worker.py:523] CUDA graph memory profiling is enabled (default since v0.21.0). The current --gpu-memory-utilization=0.9200 is equivalent to --gpu-memory-utilization=0.9188 without CUDA graph memory profiling. To maintain the same effective KV cache size as before, increase --gpu-memory-utilization to 0.9212. To disable, set VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0.
(EngineCore pid=2638900) INFO 07-12 14:51:34 [kv_cache_utils.py:2146] GPU KV cache size: 321,136 tokens
(EngineCore pid=2638900) INFO 07-12 14:51:34 [kv_cache_utils.py:2147] Maximum concurrency for 40,960 tokens per request: 7.84x
(EngineCore pid=2638900) INFO 07-12 14:51:35 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=2638900) INFO 07-12 14:51:35 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=2638900) 2026-07-12 14:51:35,118 - INFO - autotuner.py:622 - flashinfer.jit: [Autotuner]: Autotuning process starts ...
(EngineCore pid=2638900) 2026-07-12 14:51:35,162 - INFO - autotuner.py:641 - flashinfer.jit: [Autotuner]: Autotuning process ends
(EngineCore pid=2638900)
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/11 [00:00<?, ?it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 9%|▉ | 1/11 [00:00<00:01, 9.36it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 18%|█▊ | 2/11 [00:00<00:00, 9.41it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 27%|██▋ | 3/11 [00:00<00:00, 9.30it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 36%|███▋ | 4/11 [00:00<00:00, 9.23it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 45%|████▌ | 5/11 [00:00<00:00, 9.16it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 55%|█████▍ | 6/11 [00:00<00:00, 9.24it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 64%|██████▎ | 7/11 [00:00<00:00, 8.56it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 73%|███████▎ | 8/11 [00:00<00:00, 8.19it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 82%|████████▏ | 9/11 [00:01<00:00, 7.88it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 91%|█████████ | 10/11 [00:01<00:00, 7.84it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:01<00:00, 7.60it/s]
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 11/11 [00:01<00:00, 8.31it/s]
(EngineCore pid=2638900)
Capturing CUDA graphs (decode, FULL): 0%| | 0/7 [00:00<?, ?it/s]
Capturing CUDA graphs (decode, FULL): 29%|██▊ | 2/7 [00:00<00:00, 10.42it/s]
Capturing CUDA graphs (decode, FULL): 57%|█████▋ | 4/7 [00:00<00:00, 10.82it/s]
Capturing CUDA graphs (decode, FULL): 86%|████████▌ | 6/7 [00:00<00:00, 10.80it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 7/7 [00:00<00:00, 10.91it/s]
(EngineCore pid=2638900) INFO 07-12 14:51:37 [gpu_model_runner.py:6660] Graph capturing finished in 3 secs, took 0.13 GiB
(EngineCore pid=2638900) INFO 07-12 14:51:37 [gpu_worker.py:667] CUDA graph pool memory: 0.13 GiB (actual), 0.11 GiB (estimated), difference: 0.02 GiB (12.1%).
(EngineCore pid=2638900) INFO 07-12 14:51:37 [jit_monitor.py:60] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=2638900) INFO 07-12 14:51:38 [core.py:337] init engine (profile, create kv cache, warmup model) took 12.52 s (compilation: 6.02 s)
(EngineCore pid=2638900) INFO 07-12 14:51:38 [scheduler.py:282] OpProf telemetry enabled: /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl
(EngineCore pid=2638900) INFO 07-12 14:51:38 [vllm.py:1006] Asynchronous scheduling is enabled.
(EngineCore pid=2638900) INFO 07-12 14:51:38 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=2638188) INFO 07-12 14:51:38 [api_server.py:577] Supported tasks: ['generate']
(APIServer pid=2638188) WARNING 07-12 14:51:38 [model.py:1477] Default vLLM sampling parameters have been overridden by the model's `generation_config.json`: `{'temperature': 0.6, 'top_k': 20, 'top_p': 0.95}`. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=2638188) INFO 07-12 14:51:39 [hf.py:548] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
(APIServer pid=2638188) INFO 07-12 14:51:39 [api_server.py:581] Starting vLLM server on http://127.0.0.1:8502
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:37] Available routes are:
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=2638188) INFO 07-12 14:51:39 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=2638188) INFO: Started server process [2638188]
(APIServer pid=2638188) INFO: Waiting for application startup.
(APIServer pid=2638188) INFO: Application startup complete.
(APIServer pid=2638188) INFO: 127.0.0.1:43588 - "GET /v1/models HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:43598 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638900) WARNING 07-12 14:52:29 [jit_monitor.py:106] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2638188) INFO: 127.0.0.1:43614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:43624 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(EngineCore pid=2638900) WARNING 07-12 14:52:29 [jit_monitor.py:106] Triton kernel JIT compilation during inference: fused_moe_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=2638188) INFO: 127.0.0.1:43626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59174 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59190 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59200 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59224 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59236 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59238 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59244 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59254 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59262 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:52:39 [loggers.py:273] Engine 000: Avg prompt throughput: 5299.4 tokens/s, Avg generation throughput: 204.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 4.2%
(APIServer pid=2638188) INFO: 127.0.0.1:33708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33720 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33734 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33740 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33746 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33760 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33768 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33784 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:52:49 [loggers.py:273] Engine 000: Avg prompt throughput: 8.5 tokens/s, Avg generation throughput: 95.3 tokens/s, Running: 3 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.1%, Prefix cache hit rate: 35.1%
(APIServer pid=2638188) INFO: 127.0.0.1:33798 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33814 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42158 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42166 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42182 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42208 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42230 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42252 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42260 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42268 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42270 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42280 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42294 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42306 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42320 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42324 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:52:59 [loggers.py:273] Engine 000: Avg prompt throughput: 3058.0 tokens/s, Avg generation throughput: 240.8 tokens/s, Running: 6 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 40.3%
(APIServer pid=2638188) INFO: 127.0.0.1:42332 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:42340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40640 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40650 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40664 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40668 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40672 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40678 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40682 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40692 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40704 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40708 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40724 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40732 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40744 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40752 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40762 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40778 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40794 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40802 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:53:09 [loggers.py:273] Engine 000: Avg prompt throughput: 5130.2 tokens/s, Avg generation throughput: 333.9 tokens/s, Running: 2 reqs, Waiting: 0 reqs, GPU KV cache usage: 2.3%, Prefix cache hit rate: 33.3%
(APIServer pid=2638188) INFO: 127.0.0.1:38132 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38142 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38144 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38162 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38176 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38192 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38202 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38218 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38222 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38226 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38234 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:53:19 [loggers.py:273] Engine 000: Avg prompt throughput: 3991.7 tokens/s, Avg generation throughput: 168.0 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 9.7%, Prefix cache hit rate: 29.4%
(APIServer pid=2638188) INFO: 127.0.0.1:38242 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38246 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38248 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:38250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55940 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55956 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55960 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55964 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55986 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:55992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56008 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56022 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56064 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56078 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56084 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:53:29 [loggers.py:273] Engine 000: Avg prompt throughput: 6239.1 tokens/s, Avg generation throughput: 226.6 tokens/s, Running: 11 reqs, Waiting: 0 reqs, GPU KV cache usage: 13.3%, Prefix cache hit rate: 24.2%
(APIServer pid=2638188) INFO: 127.0.0.1:56116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56126 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:56140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33352 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33366 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33398 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33412 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33428 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33430 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33440 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33454 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33468 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33474 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33490 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33500 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33506 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33508 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33536 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33552 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33570 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:53:39 [loggers.py:273] Engine 000: Avg prompt throughput: 7873.3 tokens/s, Avg generation throughput: 414.1 tokens/s, Running: 8 reqs, Waiting: 0 reqs, GPU KV cache usage: 8.4%, Prefix cache hit rate: 21.7%
(APIServer pid=2638188) INFO: 127.0.0.1:59514 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59526 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59542 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59576 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59592 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59600 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59614 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59620 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59626 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59628 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59644 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59660 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:59666 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:53:49 [loggers.py:273] Engine 000: Avg prompt throughput: 5848.7 tokens/s, Avg generation throughput: 274.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 19.8%
(APIServer pid=2638188) INFO: 127.0.0.1:45016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45032 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45038 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45048 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45052 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45072 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45098 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:45124 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:53:59 [loggers.py:273] Engine 000: Avg prompt throughput: 4474.5 tokens/s, Avg generation throughput: 142.1 tokens/s, Running: 5 reqs, Waiting: 0 reqs, GPU KV cache usage: 6.5%, Prefix cache hit rate: 18.4%
(APIServer pid=2638188) INFO: 127.0.0.1:45128 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36338 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36364 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36374 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36376 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36390 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36406 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36414 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36424 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36436 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36446 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36458 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36460 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36472 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36486 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36498 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36510 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36524 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36530 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36544 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36550 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36564 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36568 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36578 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36590 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:36594 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:54:09 [loggers.py:273] Engine 000: Avg prompt throughput: 5897.7 tokens/s, Avg generation throughput: 295.1 tokens/s, Running: 12 reqs, Waiting: 0 reqs, GPU KV cache usage: 11.9%, Prefix cache hit rate: 17.5%
(APIServer pid=2638188) INFO: 127.0.0.1:45994 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46010 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46014 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46018 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46034 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46044 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46058 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46068 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46082 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46096 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46102 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46116 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46130 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:46140 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:54:19 [loggers.py:273] Engine 000: Avg prompt throughput: 3893.6 tokens/s, Avg generation throughput: 269.8 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.4%, Prefix cache hit rate: 17.2%
(APIServer pid=2638188) INFO: 127.0.0.1:46150 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48758 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48770 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48782 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48790 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48800 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48804 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48806 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48820 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48828 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48836 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48848 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48862 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48872 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48878 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48888 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48900 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:48914 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:54:29 [loggers.py:273] Engine 000: Avg prompt throughput: 4692.9 tokens/s, Avg generation throughput: 265.8 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 16.7%
(APIServer pid=2638188) INFO: 127.0.0.1:48924 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39882 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39894 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39898 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39902 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39910 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39916 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39928 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39932 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39948 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39962 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39966 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39970 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39974 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39976 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39982 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:39992 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40000 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40002 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40016 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40024 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40040 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40054 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40070 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40080 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40092 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40100 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:40112 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO 07-12 14:54:39 [loggers.py:273] Engine 000: Avg prompt throughput: 8338.1 tokens/s, Avg generation throughput: 294.5 tokens/s, Running: 17 reqs, Waiting: 0 reqs, GPU KV cache usage: 15.7%, Prefix cache hit rate: 16.0%
(APIServer pid=2638188) INFO: 127.0.0.1:40114 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33240 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33250 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33258 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33272 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33278 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33284 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33298 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33300 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33304 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33308 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33322 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33328 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33340 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33354 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33368 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33378 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33382 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=2638188) INFO: 127.0.0.1:33384 - "POST /v1/chat/completions HTTP/1.1" 200 OK

1
runs/opprof-phase6/phase6/cells/tp1_mns32/warmup.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns32", "anchor": 0.2421875, "kind": "warmup", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns32/warmup/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.2421875,
"cell": "tp1_mns32",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 16,
"feasible": true,
"interval": {
"elapsed_s": 8.025285552,
"end_mono_ns": 199530264041528,
"end_wall_ns": 1783867957390468387,
"start_mono_ns": 199522238755976,
"start_wall_ns": 1783867949365182961
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "warmup",
"mns": 32,
"observed_count": 16,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "1bfb7b673b63b8aaea00c075792180307e1a32e354711a8d3c4978f9a013bc02",
"arrival_s": {
"distinct_n": 16,
"finite_n": 16,
"max": 6.901699999999983,
"min": 0.0,
"missing_n": 0,
"n": 16
},
"count": 16,
"long_gt4096": 8,
"offered_req_s": 0.26666666666666666,
"offered_req_s_per_gpu": 0.26666666666666666,
"raw_input_tokens": {
"distinct_n": 16,
"finite_n": 16,
"max": 7270.0,
"min": 72.0,
"missing_n": 0,
"n": 16
},
"raw_length_order_sha256": "e858719eb07cdeb674aa17d9299d05dec852db11c263bc9391c53cd0f177db1c",
"request_id_order_sha256": "0fa4b7f4dc274280eb173a0ee0974643ab283b887418ce8f10e958e7e8d90161"
},
"slo_pass_count": 16,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 16,
"finite_n": 16,
"max": 28.123532992047,
"min": 4.953902173286861,
"missing_n": 0,
"n": 16
},
"ttft_ms": {
"distinct_n": 16,
"finite_n": 16,
"max": 682.8775229805615,
"min": 74.39436702406965,
"missing_n": 0,
"n": 16
}
}

1
runs/opprof-phase6/phase6/cells/tp1_mns64/anchor-0.2421875.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns64", "anchor": 0.2421875, "kind": "anchor", "pass_rate": 1.0, "feasible": true}

74
runs/opprof-phase6/phase6/cells/tp1_mns64/anchor-0.2421875/result.json Normal file
View File

@@ -0,0 +1,74 @@
{
"anchor": 0.2421875,
"cell": "tp1_mns64",
"early_stop_reason": "",
"early_stopped": false,
"exact_output_count": 137,
"feasible": true,
"interval": {
"elapsed_s": 61.27871102,
"end_mono_ns": 199601593594258,
"end_wall_ns": 1783868028720022637,
"start_mono_ns": 199540314883238,
"start_wall_ns": 1783867967441310423
},
"invariants": {
"arrival_nondecreasing": true,
"exact_output_or_failed": true,
"outcomes_cover_selected": true,
"raw_lengths_present": true,
"selected_nonempty": true,
"warmup_16": true,
"warmup_exact_16": true,
"warmup_long": true
},
"kind": "anchor",
"mns": 64,
"observed_count": 137,
"pass_rate": 1.0,
"schema": 1,
"selection": {
"arrival_order_sha256": "e8e91fd47d9152811ed7bd79e20c0f45ba6677560a419542d9c17abd82bebb4b",
"arrival_s": {
"distinct_n": 137,
"finite_n": 137,
"max": 59.78950000000005,
"min": 0.008300000000008368,
"missing_n": 0,
"n": 137
},
"count": 137,
"long_gt4096": 47,
"offered_req_s": 2.283333333333333,
"offered_req_s_per_gpu": 2.283333333333333,
"raw_input_tokens": {
"distinct_n": 129,
"finite_n": 137,
"max": 8149.0,
"min": 72.0,
"missing_n": 0,
"n": 137
},
"raw_length_order_sha256": "93176915562ff9a118f3550157ddd1025d73a6b70cf4d03be9765de9a1b5d744",
"request_id_order_sha256": "adb62bea7f7a12c1e33fa1572ec1d0e274100013ad90e14c6ef5c549b0e0d017"
},
"slo_pass_count": 137,
"study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece",
"tp": 1,
"tpot_ms": {
"distinct_n": 137,
"finite_n": 137,
"max": 34.5918034883051,
"min": 4.451991834606257,
"missing_n": 0,
"n": 137
},
"ttft_ms": {
"distinct_n": 137,
"finite_n": 137,
"max": 1472.6779730117414,
"min": 33.561496005859226,
"missing_n": 0,
"n": 137
}
}

1
runs/opprof-phase6/phase6/cells/tp1_mns64/anchor-0.24609375.log Normal file
View File

@@ -0,0 +1 @@
{"cell": "tp1_mns64", "anchor": 0.24609375, "kind": "anchor", "pass_rate": 1.0, "feasible": true}

Some files were not shown because too many files have changed in this diff Show More