diff --git a/docs/opprof/patch-design.md b/docs/opprof/patch-design.md new file mode 100644 index 0000000..764016d --- /dev/null +++ b/docs/opprof/patch-design.md @@ -0,0 +1,434 @@ +# OpProf dual-layer instrumentation patch design + +Status: **design for review; no vLLM patch has been implemented**. + +Target source: vLLM `v0.24.0`, commit `ee0da84ab9e04ac7610e28580af62c365e898389`. +Target campaign: Qwen3-30B-A3B serving on NVIDIA H20 (SM90), using the V1 engine. +All source paths and line numbers below are relative to the pinned clone. + +## Approved dispositions (2026-07-11) + +- Use JSONL with `msgspec`, the proposed context/chunk histogram edges, and an + 8192-record bounded queue. +- Keep exact expert loads Layer-2-only. +- Use the community BF16 Qwen3-30B-A3B checkpoint. TP1 is primary, with TP2 and + TP4 counterpoints; record the selected MoE backend logs in every run. +- Use two profiler warm-up iterations followed by eight active iterations. +- Apply the 3% Layer-1 overhead gate to the **upper bound of the 95% confidence + interval** for every primary serving metric. +- Reject `--disable-log-stats` when OpProf is enabled; retain the five-file + fail-fast design. + +## Goal and success criteria + +The patch should make every serving iteration conditionable on its request and +execution pattern while keeping the always-on path below a 3% serving overhead +budget. Heavy kernel tracing and exact MoE routes are sampled separately. + +The proposed split is: + +1. **Layer 1:** one compact composition record per scheduler step, emitted by + the scheduler process rather than every tensor-parallel worker. +2. **Layer 2:** short, sampled windows using vLLM's existing torch-profiler + configuration and `/start_profile` and `/stop_profile` endpoints. + +The design deliberately does not add hooks to GPU kernels, attention layers, +the Qwen model, or the OpenAI serving API. Those surfaces are not needed to +answer the Phase 0 profiling questions. + +## Assumptions and tradeoffs + +- The campaign uses the V1 engine and the default vLLM scheduler. Both the + synchronous and `AsyncScheduler` paths inherit the base scheduler hooks. +- The H20 backend statement assumes the usual unquantized BF16/FP16 + Qwen3-30B-A3B checkpoint, `moe_backend=auto`, and no LoRA. Quantization, + batched expert format, or an explicit backend can change kernel selection. +- Layer 1 requires normal stats collection, which is enabled by default. If + OpProf is enabled together with `--disable-log-stats`, initialization should + fail with an actionable error. Supporting that unusual combination would + require an extra stats path and a larger patch. +- The context-length histogram records the sequence length at the end of the + scheduled model input (`num_computed_tokens` before scheduling plus tokens + scheduled in this step). This matches the worker-side sequence-length + construction at `vllm/v1/worker/gpu_model_runner.py:2010-2019` and avoids + retaining raw request lengths. +- “Decode tokens” means tokens scheduled for requests classified by vLLM as + generation-phase requests, including scheduled speculative tokens. It is not + the number of subsequently accepted output tokens. The existing classifier + and its chunked-prefill semantics are at `vllm/v1/utils.py:780-813`. + +## Existing facilities to reuse + +These are zero-patch wins: + +| Need | Existing v0.24.0 facility | Design consequence | +|---|---|---| +| Scheduled request and token map | `SchedulerOutput.num_scheduled_tokens`, total tokens, preempted IDs, and new/cached request data at `vllm/v1/core/sched/output.py:180-219` | Derive composition in the scheduler; do not add fields to `SchedulerOutput`. | +| Prefill/decode classification | `compute_iteration_details()` at `vllm/v1/utils.py:780-813` | Reuse the exact definition already used by the iteration log. | +| Queue/KV/prefix stats | `SchedulerStats` at `vllm/v1/metrics/stats.py:170-198` and construction at `vllm/v1/core/sched/scheduler.py:2228-2264` | Reuse field semantics, but snapshot at schedule time to avoid async misattribution. | +| Per-step CUDA-graph descriptor | `CUDAGraphStat` at `vllm/compilation/cuda_graph.py:32-37`, populated at `vllm/v1/worker/gpu_model_runner.py:3919-3934` | Reuse the object; only broaden the condition under which it is returned. | +| Sampled CPU/CUDA tracing | profiler config and schedule at `vllm/config/profiler.py:33-105`, endpoints at `vllm/entrypoints/serve/profile/api_router.py:21-45` | Layer 2 needs configuration and orchestration, not a new profiler implementation or endpoint. | +| Exact routed expert IDs | routed-expert capture callback and buffers at `vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84` and `vllm/v1/worker/gpu_model_runner.py:7382-7437` | Use only in a separate sampled Layer-2 run; it is too expensive for Layer 1. | + +## Layer 1: always-on composition records + +### Data flow and hook placement + +```text +Scheduler.schedule() + -> snapshot request composition, queues, KV and prefix deltas + -> SchedulerOutput -> TP workers -> ModelRunnerOutput.cudagraph_stats + -> Scheduler.update_from_output() + -> finalize the same step record -> bounded queue -> background JSONL writer +``` + +The scheduler is the ownership boundary for Layer 1. It sees the full logical +batch once, so recording in each TP worker would duplicate data and require an +aggregation protocol. + +Exact proposed hooks against the pinned clone: + +1. **Initialize one recorder in the base scheduler constructor.** The scheduler + already owns stats state and passes the normal `log_stats` setting into the + KV-cache manager (`vllm/v1/core/sched/scheduler.py:78-87` and + `vllm/v1/core/sched/scheduler.py:250-260`). If `VLLM_OPPROF_DIR` is set, + construct the recorder and reject `log_stats=False`. +2. **Begin a record inside `Scheduler.schedule()`.** Read prefix counters at + method entry, then finish the snapshot immediately after `SchedulerOutput` + and connector metadata are constructed and before + `_update_after_schedule()` mutates request state + (`vllm/v1/core/sched/scheduler.py:1012-1100`). Prefix lookup counters are + updated synchronously during this schedule call + (`vllm/v1/core/kv_cache_manager.py:202-242` and + `vllm/v1/core/sched/scheduler.py:675-712,897-915`), so before/after deltas + remain attributable even when several batches are in flight. +3. **Finalize in `Scheduler.update_from_output()`.** Pair the returned + `ModelRunnerOutput` with the exact `SchedulerOutput` at + `vllm/v1/core/sched/scheduler.py:1464-1477`, and enqueue after normal output + processing near `vllm/v1/core/sched/scheduler.py:1791-1803`. Store pending + drafts by `id(scheduler_output)` and remove them on finalize; the identifier + never leaves the process. This is robust to the batch queue, which retains + and later returns the matching output object + (`vllm/v1/engine/core.py:519-632`). +4. **Return the existing CUDA-graph stat when OpProf is enabled.** Change the + condition at `vllm/v1/worker/gpu_model_runner.py:3919-3926` from only + `observability_config.cudagraph_metrics` to that flag **or** + `VLLM_OPPROF_DIR`. The file already imports `vllm.envs` at line 23. No CUDA + synchronization or tensor transfer is introduced. +5. **Close through the existing scheduler shutdown.** Drain and join the writer + before the scheduler reports shutdown complete at + `vllm/v1/core/sched/scheduler.py:2285-2295`; retain an `atexit` fallback for + abnormal embedding/test lifecycles. + +The scheduler assigns a monotonically increasing local `step_index` at begin. +Every real scheduler call is represented, including zero-token steps. A DP-only +dummy execution has no `SchedulerOutput` and is outside this composition +stream; if DP is later in campaign scope, add a distinct `dummy_step` marker +rather than pretending it has a request composition. + +### Record schema + +Use schema-versioned JSON Lines. An illustrative record is: + +```json +{"schema":1,"engine_id":"dp0-pid1234","step_index":42,"submit_wall_ns":1783761000000000000,"submit_mono_ns":8920000000,"complete_mono_ns":8922371000,"model_executed":true,"scheduled_requests":37,"decode_batch_size":32,"prefill_requests":5,"prefill_tokens":896,"decode_tokens":32,"chunked_prefill":{"first":2,"middle":1,"final":1,"unsplit":1,"tokens":896,"chunk_size_hist":[0,0,1,1,1,1,1,0,0]},"context_length_hist":[0,0,3,6,12,10,5,1,0,0,0,0],"preemptions":0,"queues":{"running":37,"waiting":9,"deferred":0},"kv":{"total_blocks":120000,"free_blocks":42000,"used_blocks":78000,"usage":0.65},"prefix":{"local":{"requests":2,"queries":4096,"hits":3072,"preempted_requests":0,"preempted_queries":0,"preempted_hits":0},"external":null},"cudagraph":{"hit":true,"runtime_mode":"PIECEWISE","unpadded_tokens":928,"bucket_tokens":1024,"padding_tokens":96},"moe_expert_load":null,"dropped_records_before":0} +``` + +Required field semantics: + +- `submit_wall_ns` permits joining to external workload logs; + `submit_mono_ns` and `complete_mono_ns` provide stable within-process + ordering and elapsed time without wall-clock jumps. +- `scheduled_requests` is the number of entries in + `num_scheduled_tokens`. `decode_batch_size`, `prefill_requests`, + `prefill_tokens`, and `decode_tokens` reuse vLLM's classifier. +- `chunked_prefill` contains aggregate split information, never request IDs. + A context request is `first` when the scheduled range ends before its prompt + ends, `middle` when it was already a prefill chunk and remains incomplete, + `final` when a prior chunk completes, and `unsplit` when its prefill completes + in one step. `chunk_size_hist` uses token-count buckets + `(16, 32, 64, 128, 256, 512, 1024, 2048, +inf)`. +- `context_length_hist` includes every scheduled request, with fixed upper + edges `(128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, + 131072, +inf)`. The final array therefore has 12 bins. The example above is + illustrative and will be checked against the final chosen edges in tests. +- `preemptions` is the size of the step's `preempted_req_ids`, created at + `vllm/v1/core/sched/scheduler.py:1059-1076`. It is not an interval-derived + Prometheus value. +- Queue lengths and KV blocks are captured immediately after scheduling. KV + usage follows the existing null-block-adjusted calculation at + `vllm/v1/core/block_pool.py:692-711`. `deferred` means skipped/deferred + waiting requests, matching `num_skipped_waiting_reqs` in + `vllm/v1/metrics/stats.py:174-183`. +- Local and external prefix fields are per-schedule deltas copied from the + existing mutable counters. The existing stats drain resets those objects at + `vllm/v1/core/kv_cache_manager.py:190-200` and + `vllm/v1/core/sched/scheduler.py:2235-2242`; OpProf must read, not drain or + replace, them. These retain vLLM's lookup semantics: a local lookup can be + counted before allocation later rejects that waiting request, so prefix + `requests` is not required to be less than or equal to scheduled prefill + requests. +- `cudagraph.hit` is the operational serving definition `runtime_mode != NONE`. + `FULL` selects the full graph path and `PIECEWISE` selects graph-wrapped + compiled regions; the latter must not be interpreted as full-step graph + coverage. Startup normally captures all dispatcher descriptors at + `vllm/v1/worker/gpu_model_runner.py:6595-6643`, but the stat itself has no + capture-versus-replay bit, so a rare lazy first capture cannot be + distinguished from replay. `bucket_tokens` is `num_padded_tokens` in the + existing stat. +- `moe_expert_load` is explicitly null in Layer 1. This avoids accidentally + presenting unavailable data as zero. + +### Encoding, buffering, and flush + +JSONL is recommended over a compact binary ring buffer for Phase 1 because the +schema will evolve during campaign bring-up, records need to be directly +inspectable beside Kineto traces, and `msgspec` is already a vLLM dependency +(`requirements/common.txt:35`). +Encoding one small dictionary with a reused `msgspec.json.Encoder` avoids the +standard `json` module's larger CPU cost. Once the schema stabilizes, the same +record can be moved to a binary format only if Phase 2 measurements show JSONL +is the bottleneck. + +The foreground path encodes once and performs non-blocking `put_nowait()` into +a bounded queue of 8192 records. A single daemon writer thread drains into one +file per EngineCore/DP rank and process. It flushes userspace buffers every +1 MiB or one second, whichever comes first, and on clean process exit; it never +calls `fsync()` per record. When the queue is full, the serving thread drops the +new record, increments a counter, and the next successful record reports the +gap in `dropped_records_before`. Shutdown emits a footer with encoded, written, +and dropped counts. The file name includes schema, DP rank, PID, and start time, +but not TP rank because there is one scheduler record stream. + +The sole on/off switch is: + +```text +VLLM_OPPROF_DIR=/absolute/output/directory +``` + +Unset or empty means a true no-op: no recorder, queue, thread, record +construction, or graph-stat broadening. Directory validation and a clear +startup log happen before serving. Runtime toggling is intentionally omitted; +it adds synchronization and ambiguous partial files without helping the +campaign. + +### Expected cost and Phase 2 overhead gate + +These are estimates, not measurements: + +- composition and two fixed histograms: about 20-80 microseconds per step, + linear in scheduled requests; +- `msgspec` encoding plus a non-blocking queue insertion: about 5-20 + microseconds per step; +- CUDA-graph stat construction: below 2 microseconds and no GPU sync; +- disk I/O: off the serving critical path, subject to bounded-queue drops. + +The expected foreground total is roughly 25-100 microseconds per step. At a +2 ms decode step, the high end would exceed 3%, so the budget is a measurement +gate, not a claim. + +Phase 2 should use the same Qwen3 checkpoint, request trace, random seed, +hardware, vLLM commit, TP/EP topology, CUDA-graph configuration, and cache +state for off/on comparisons. Alternate off/on ordering, warm up before each +measurement, and run at least five paired repeats. Report throughput and +p50/p95 TTFT and TPOT, plus CPU utilization, log bytes per step, queue high +watermark, and dropped-record count. The acceptance rule is less than 3% +regression for every declared primary serving metric, with bootstrap confidence +intervals reported. Whether the point estimate or upper 95% bound is the hard +gate is an open decision. + +Correctness checks for every run are: contiguous step indices except explicitly +reported drops, scheduled token sums matching the prefill/decode split, +histogram counts matching scheduled request/chunk counts, KV ratio in `[0,1]`, +non-negative counters, and identical generated outputs for the deterministic +test trace. + +## Layer 2: sampled kernel windows + +### Trigger and window + +Use the existing profiler API with a run-specific output directory and +`ignore_frontend=true`. The worker start/stop RPC already fans out to all +workers (`vllm/v1/executor/abstract.py:256-257` and +`vllm/v1/executor/multiproc_executor.py:340-402`), and each GPU worker creates a +CPU+CUDA profiler with a rank-qualified trace name +(`vllm/v1/worker/gpu_worker.py:929-980`). No Layer-2 vLLM code patch is needed. + +Recommended initial configuration: + +```text +profiler=torch +torch_profiler_dir=/kineto +ignore_frontend=true +delay_iterations=0 +max_iterations=8 +wait_iterations=0 +warmup_iterations=2 +active_iterations=8 +torch_profiler_record_shapes=false +torch_profiler_with_memory=false +torch_profiler_with_stack=false +torch_profiler_with_flops=false +torch_profiler_use_gzip=true +torch_profiler_dump_cuda_time_total=true +``` + +The primary trigger should be step-count based: an external campaign controller +POSTs `/start_profile` immediately before a desired composition regime. The +built-in schedule records two warm-up plus eight active model iterations and +`max_iterations=8` then stops the underlying profiler on the following worker +step. The “exceeds max” control semantics are explicit at +`vllm/profiler/wrapper.py:83-114` and in +`tests/v1/worker/test_gpu_profiler.py:77-98`; the controller should still POST +`/stop_profile` afterward to clear the active control state. +Worker iteration boundaries already call `profiler.step()` and annotate context +and generation token counts at `vllm/v1/worker/gpu_worker.py:803-827`. On-demand +manual POST remains useful for debugging. A time-based trigger is a fallback +for live traffic but is less reproducible because it yields a variable number +of steps. Phase 1 should confirm that the trace contains exactly eight active +iteration annotations; profiler scheduling and trace callbacks live at +`vllm/profiler/wrapper.py:159-226,290-307`. + +The profile directory should be unique per campaign server run. The HTTP start +endpoint does not accept a `profile_prefix`, and a worker that is restarted for +another window retains the trace name chosen on first initialization +(`vllm/v1/worker/gpu_worker.py:939-975`). Multiple windows in one server run +therefore share a directory and need a controller manifest containing start/ +stop wall times and produced filenames. Use a new directory only when the +server is restarted; adding a new endpoint parameter is not justified for +Phase 1. Every TP worker emits its own trace, and the rank suffix contains DP, +PP, TP, DCP, EP, and global rank information +(`vllm/distributed/utils.py:664-695`). Keep these traces separate and aggregate +only offline. + +### CUDA graphs and Kineto interpretation + +Keep CUDA graphs enabled in the primary sampled windows so kernel time reflects +the serving configuration. The pinned code profiles CUDA activity +(`vllm/v1/worker/gpu_worker.py:953-960`) while a captured path invokes +`CUDAGraph.replay()` instead of rerunning the Python callable +(`vllm/compilation/cuda_graph.py:233-360`). Therefore Kineto/CUPTI can observe +CUDA activity launched during replay, but the Python/PyTorch operator scopes +that created the graph are not re-executed and cannot be assumed to retain +per-layer attribution. vLLM explicitly documents the related limitation that +layerwise NVTX tracing does not work with CUDA graphs +(`vllm/config/observability.py:60-63`). + +The clone does not contain a stronger guarantee about whether a particular +PyTorch/CUPTI build will present every graph node as an individually named +kernel versus a coarser graph-launch view. Treat that as an empirical Phase 2 +check. Run one matched eager (`cudagraph_mode=NONE`) taxonomy window to identify +kernel families, but do not use its timings as the production baseline. + +### Calibration and MoE routing + +Join each profiler iteration annotation to Layer-1 records by rank-independent +step order, the profiler window marker, timestamps, and the prefill/decode token +counts. Aggregate kernels into attention, router/top-k, expert GEMMs, dense +GEMMs, normalization/activation, collectives, and sampling. For TP, report both +sum-of-rank GPU work and the maximum per-rank critical-path time. Fit or tabulate +kernel-time attribution conditioned on composition, context histogram, graph +runtime mode, and capture bucket, then validate on held-out windows rather than +the same samples used for calibration. + +For the Phase 2 “operator time approximately equals iteration time” gate, do +not naively add overlapping kernels. Compute the union of CUDA kernel intervals +per rank, use the maximum rank as the distributed GPU critical path, and retain +an explicit CPU/queue/unattributed residual against Layer 1's submit-to-complete +span. Operator-category interval unions plus the residual must reconstruct the +span; summed GPU work is reported separately and may legitimately exceed wall +time. + +Exact per-layer expert loads are Layer-2-only. A separate server start with +`--enable-return-routed-experts` exposes per-token, per-layer top-k IDs through +the existing router callback (`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`). +Aggregate those arrays offline into per-layer expert counts, entropy, max/mean +load, coefficient of variation, and top-k imbalance. Do not enable it in the +normal Layer-1 or baseline Kineto run: its worker transit buffer costs a few MB, +and the scheduler-side slot buffer can reach multiple GB with a CPU fancy-index +copy per step (`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`). + +## Patch surface estimate + +Proposed implementation and tests: + +| File | Approximate changed/new lines | Purpose | +|---|---:|---| +| `vllm/envs.py` | 4 | Declare and parse `VLLM_OPPROF_DIR`. | +| `vllm/v1/opprof.py` (new) | 220 | Schema, schedule snapshot/deltas, histograms, pending-step pairing, encoder, bounded writer, drop/footer handling. | +| `vllm/v1/core/sched/scheduler.py` | 32 | Initialize recorder; begin before state mutation; finalize with the matching model output. | +| `vllm/v1/worker/gpu_model_runner.py` | 4 | Return existing `CUDAGraphStat` when either built-in metrics or OpProf needs it. | +| `tests/v1/core/test_opprof.py` (new) | 190 | Schema/invariants, chunk classification, async pairing, disabled no-op, bounded-queue drops, shutdown flush. | +| **Total** | **5 files / about 450 lines** | About 260 production lines and 190 test lines. | + +No patch is proposed for `ProfilerConfig`, profile endpoints, profiler wrapper, +`SchedulerOutput`, `SchedulerStats`, Prometheus/logging, CUDA kernels, fused MoE +kernels, the Qwen model, or frontend APIs. If support for +`--disable-log-stats` is required, add about 12 lines in +`vllm/v1/core/kv_cache_manager.py`, making the estimate six files and about 462 +lines; the smaller fail-fast design is recommended for this campaign. + +## Risks and mitigations + +- **Foreground CPU overhead:** request scanning and JSON encoding are the likely + hot spots. Use fixed-size arrays, a reused encoder, no raw lists, no blocking + I/O, and enforce the Phase 2 gate. +- **Async-scheduling races:** queue/KV/prefix state observed during + `update_from_output()` may include later in-flight schedules. Snapshot it + inside the synchronous `schedule()` call; only the immutable returned + CUDA-graph stat is added at completion. Pair by the exact `SchedulerOutput` + object and assert one begin/one finalize. +- **TP and EP aggregation:** Layer 1 is scheduler-owned and emitted once. Layer + 2 remains per rank. Interpret TP critical path as the slowest rank and, for an + expert-parallel routed-expert sample, aggregate expert ownership offline. +- **Log volume:** at 500 steps/s, a 1.0 KiB record is about 44 GB/day per + EngineCore. Rotate by campaign run, compress completed files, and use the + bounded queue/drop counters. Production-long captures need a later sampling + or rotation policy; Phase 1 files should be bounded by run duration. +- **Graph semantics:** `PIECEWISE` is partial graph coverage, not a binary full + hit. Preserve `runtime_mode`, unpadded tokens, bucket, and padding rather than + reducing the record to one boolean. +- **Profiler perturbation:** Kineto has medium-to-high overhead and trace flushes + can stall. Use short windows, unique directories, disabled stacks/shapes and + memory by default, and never use Layer-2 latency as an unprofiled performance + result. +- **Backend drift:** the unquantized SM90 oracle prioritizes Triton, but + shape-specific fallbacks remain possible. Record startup backend logs and the + full checkpoint/parallel/quantization configuration with every run. + +## Open decisions for review + +1. Approve JSONL with `msgspec`, the two proposed histogram edge sets, and a + bounded 8192-record writer queue. +2. Approve exact expert-load collection as Layer-2-only, rather than adding a + new always-on GPU histogram kernel. +3. Confirm the Qwen3 checkpoint precision/quantization and intended TP/EP/DP + topology; the Triton backend conclusion depends on these inputs. +4. Choose the first profiler window: recommended two warm-up plus eight active + iterations, or a longer active window. +5. Decide whether the 3% gate applies to metric point estimates or to the upper + bound of their 95% confidence intervals. +6. Confirm that campaign runs may reject `--disable-log-stats`; supporting it + adds one file and about 12 lines. + +## Data sanity block + +- **Patch estimate:** n=5 files; per-file line-estimate min=4, max=220, + distinct values=4; total about 450 lines, of which about 260 are production + and 190 are tests. +- **Histogram definitions:** n=2 arrays; bin-count min=9, max=12, + distinct=2. In the illustrative record, context-bin sum=37 equals scheduled + requests, chunk-bin sum=5 equals prefill requests, KV used=120000-42000, and + graph padding=1024-928. +- **Estimated foreground components:** n=3 timed components; min is below 2 + microseconds for graph-stat construction, max is 80 microseconds for + composition; distinct ranges=3. Their conservative combined foreground + estimate is 25-100 microseconds per step. +- **Invariants checked:** counters and histogram bins are non-negative; KV + usage and prefix hit ratios are bounded in `[0,1]`; graph bucket is at least + unpadded tokens; `NONE` is the only graph miss mode; no raw request IDs or + context-length lists are serialized; one scheduler stream avoids TP + duplicates. +- **Measurement status:** n=0 benchmark runs; min/max and cross-configuration + distinctness are not applicable. The cost figures are estimates and cannot + support an overhead conclusion until Phase 2. diff --git a/docs/opprof/phase0-recon-vllm-0.24.0.md b/docs/opprof/phase0-recon-vllm-0.24.0.md new file mode 100644 index 0000000..bb4ceb0 --- /dev/null +++ b/docs/opprof/phase0-recon-vllm-0.24.0.md @@ -0,0 +1,446 @@ +# Phase 0 recon: vLLM 0.24.0 observability + +Status: source recon and patch design complete; **no instrumentation patch was +implemented**. + +Campaign target: operator-level, pattern-conditioned profiling of +Qwen3-30B-A3B serving on NVIDIA H20. This report is based on the pinned source +clone only. No GPU was used, no package or model was installed, and no dash +host was contacted. + +Unless explicitly prefixed with `v0.20.0:`, every source path and line number +below is relative to `/home/gahow/phd/vllm-v0.24.0` at the pinned commit. + +## Outcome + +vLLM 0.24.0 already contains most of the raw ingredients, but they are split +across scheduler internals, aggregate metrics, worker traces, and an optional +routed-expert return path. The smallest useful patch is therefore not a new +profiler. It is one scheduler-owned per-step record that joins existing batch +composition to the existing `CUDAGraphStat`; short torch-profiler windows and +exact expert routes remain sampled Layer 2 facilities. + +The proposed patch is detailed in +[`patch-design.md`](patch-design.md). Estimated surface: five files and about +450 lines including tests, with about 260 production lines. + +## 1. Source pin and compatibility record + +The requested tag existed and the exact requested clone command completed +successfully: + +```text +git clone --depth 1 --branch v0.24.0 https://github.com/vllm-project/vllm /home/gahow/phd/vllm-v0.24.0 +``` + +| Field | Pinned value | +|---|---| +| Repository | `https://github.com/vllm-project/vllm` | +| Local clone | `/home/gahow/phd/vllm-v0.24.0` | +| Tag | `v0.24.0` (`git describe --tags --exact-match`) | +| Commit | `ee0da84ab9e04ac7610e28580af62c365e898389` | +| Clone completion | `2026-07-11T08:26:16Z` (`2026-07-11 16:26:16 +08:00`) | +| Checkout | Detached HEAD, clean | +| Top-level entries | 45, including hidden entries and `.git` | +| Listing SHA-256 | `75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58` | + +The listing digest is reproducible from entry names only, not file contents: + +```text +LC_ALL=C find . -mindepth 1 -maxdepth 1 -printf '%f\n' | + LC_ALL=C sort | sha256sum +``` + +This definition includes `.git`, sorts bytewise under the C locale, and hashes +the 45 LF-terminated root names. It is recorded to make the otherwise ambiguous +phrase “sha256 of the top-level directory listing” precise. + +### Torch and CUDA requirements + +- Build and NVIDIA runtime requirements pin `torch==2.11.0` in + `pyproject.toml:1-13`, `requirements/build/cuda.txt:1-10`, and + `requirements/cuda.txt:1-10`; CMake repeats 2.11.0 as the supported CUDA and + ROCm torch version at `CMakeLists.txt:62-72`. +- The NVIDIA requirements also pin `torchvision==0.26.0`, + `flashinfer-python==0.6.12`, `flashinfer-cubin==0.6.12`, and CUDA-13 extras + for CUTLASS DSL and Humming kernels (`requirements/cuda.txt:6-29`). Setup + strips/substitutes those extras for CUDA 12 builds + (`setup.py:1066-1083`). +- The source default is CUDA 13.0 (`vllm/envs.py:85-87,555-563`) and the Docker + default is CUDA 13.0.2 (`docker/Dockerfile:14-42`). Wheel detection maps CUDA + major 12 to `cu129` and major 13 to `cu130` (`setup.py:528-568`). +- The pinned installation document still describes the default precompiled + binaries as CUDA 12.9 and lists CUDA 12.8 and 13.0 variants + (`docs/getting_started/installation/gpu.cuda.inc.md:4-19,24-54`). This is a + packaging/default distinction, not evidence that 13.0 is unsupported. +- NVIDIA compute capability 7.5+ is documented, and SM90 is in every relevant + CMake CUDA architecture set (`docs/getting_started/installation/gpu.cuda.inc.md:9-19`; + `CMakeLists.txt:105-118`). Optional FA3 and DeepGEMM builds need CUDA 12.3+, + FlashMLA needs 12.9+, and the Hopper CUTLASS MoE build needs 12.3+ + (`setup.py:1111-1139`; `CMakeLists.txt:844-868`). + +Implication for dash0 later: H20/SM90 is a supported architecture, but the +actual driver and installed CUDA were intentionally not queried in Phase 0. +Phase 2 must choose a torch 2.11.0-compatible `cu129` or `cu130` artifact after +checking the host driver. A wheel built against a different torch/CUDA build is +not interchangeable; the source documentation explicitly warns about binary +incompatibility (`docs/getting_started/installation/gpu.cuda.inc.md:14-19`). + +## 2. Built-in observability inventory + +### 2.1 Torch profiler integration + +**Configuration and control.** v0.24.0 does not contain the legacy +`VLLM_TORCH_PROFILER_DIR` symbol: a repository-wide exact-name search at the +pinned commit returned zero matches. Profiling is now configured through +`ProfilerConfig`: `profiler` is `torch` or `cuda`, and +`torch_profiler_dir` holds the absolute/URI output location +(`vllm/config/profiler.py:33-46,125-145`). Optional controls include stacks, +FLOPs, gzip, CUDA-time tables, shapes, memory, frontend exclusion, delayed +start, maximum iterations, and wait/warmup/active iteration scheduling +(`vllm/config/profiler.py:48-105`). + +The documented server form is a `--profiler-config` JSON object, for example +`{"profiler":"torch","torch_profiler_dir":"/abs/run/kineto"}`, followed by +POSTs to the two endpoints (`docs/contributing/profiling.md:46-85`). + +When profiler config is present, the serving router exposes POST +`/start_profile` and `/stop_profile`; without profiler config the router is not +attached (`vllm/entrypoints/serve/profile/api_router.py:21-45`). The endpoints +are therefore not an unauthenticated always-on feature independent of server +configuration; they exist within the configured serving application and should +be protected like the rest of the control plane. + +**Captured activity.** Each GPU worker creates a torch profiler with both CPU +and CUDA activities, using the configured shapes/memory/stack/FLOPs switches +(`vllm/v1/worker/gpu_worker.py:929-980` and +`vllm/profiler/wrapper.py:159-226`). TensorBoard-compatible traces are written +with an optional gzip handler. The wrapper also writes CPU/CUDA aggregate +tables; only rank 0 prints the table to stdout +(`vllm/profiler/wrapper.py:251-287`). The source documentation calls the +profiler medium-overhead and warns that trace size and final flushing can be +large (`docs/contributing/profiling.md:1-38`). + +The V1 worker calls `profiler.step()` once per execution and wraps model +execution in a record-function name containing context/generation request and +token counts (`vllm/v1/worker/gpu_worker.py:803-827,895-898`). If frontend +profiling is not ignored, `AsyncLLM` also creates a separate CPU-only trace +(`vllm/v1/engine/async_llm.py:178-200`). + +**TP behavior.** `EngineCore.profile()` delegates to the executor +(`vllm/v1/engine/core.py:662-663`), whose profile RPC is a collective worker +call (`vllm/v1/executor/abstract.py:256-257`). Multiprocessing broadcasts calls +that do not declare a unique output rank, so the profile call reaches every TP +worker (`vllm/v1/executor/multiproc_executor.py:340-402`). Each worker emits its +own rank-qualified trace; the suffix includes DP/PP/TP/DCP/EP/global rank +components (`vllm/distributed/utils.py:664-695`). For TP, kernel analysis must +therefore preserve per-rank files, use the slowest rank for critical-path time, +and optionally sum ranks to quantify total GPU work. + +### 2.2 Iteration-level scheduler data and exported metrics + +The V1 scheduler interface states that each `schedule()` output corresponds to +one model forward and maps request IDs to scheduled tokens +(`vllm/v1/core/sched/interface.py:51-80`). `SchedulerOutput` already carries +new/cached request data, per-request and total scheduled tokens, common-prefix +blocks, and the request IDs preempted in that step +(`vllm/v1/core/sched/output.py:180-219`). The base scheduler constructs this +object at `vllm/v1/core/sched/scheduler.py:1012-1076`, then advances request +computed-token and chunk state at `vllm/v1/core/sched/scheduler.py:1130-1155`. + +What is available today: + +| Requested datum | Internal per-step source | Externally exposed today | +|---|---|---| +| Batch size | `len(SchedulerOutput.num_scheduled_tokens)`; the map is exact | No raw per-step batch-size Prometheus series. Optional iteration log reports context and generation request counts. | +| Scheduled prefill/decode tokens | `compute_iteration_details()` classifies new/cached context requests and sums scheduled tokens (`vllm/v1/utils.py:766-813`) | Optional log reports exact scheduled context/generation counts and elapsed time around the execution wait (`vllm/v1/engine/core.py:433-472`). Prometheus prompt/generation counters are cumulative output-side stats, not a raw scheduled-step stream. | +| Chunked-prefill splits | `Request.is_prefill_chunk`, `num_computed_tokens`, and prompt/output state exist (`vllm/v1/request.py:130-175`); extended chunks are classified as context | No chunk ordinal, first/middle/final split, or chunk-size series. These must be derived before `_update_after_schedule()` mutates state. | +| Preemptions | Exact step set in `SchedulerOutput.preempted_req_ids` | Stdout aggregates over its logging interval; `vllm:num_preemptions` is cumulative (`vllm/v1/metrics/loggers.py:219-281,624-631`). | +| Running/waiting/deferred queues | `SchedulerStats.num_running_reqs`, `num_waiting_reqs`, and `num_skipped_waiting_reqs` (`vllm/v1/metrics/stats.py:170-183`) | Latest-value gauges `vllm:num_requests_running`, `vllm:num_requests_waiting`, and reason-labeled capacity/deferred gauges (`vllm/v1/metrics/loggers.py:453-496`). Running queue size is not the same as scheduled batch size. | +| KV-cache usage | Scheduler reads `kv_cache_manager.usage`; the block pool computes a null-block-adjusted ratio in `[0,1]` (`vllm/v1/core/kv_cache_manager.py:181-188`; `vllm/v1/core/block_pool.py:692-711`) | Latest-value `vllm:kv_cache_usage_perc`; stdout prints the latest percent (`vllm/v1/metrics/loggers.py:250-260,524-532`). Raw total/free blocks are not exported per step. | +| Prefix-cache counters | `PrefixCacheStats` contains request/query/hit and separately preempted request/query/hit counters (`vllm/v1/metrics/stats.py:114-143`) for local and optional connector caches | Prometheus exposes cumulative token queries/hits for local and external caches (`vllm/v1/metrics/loggers.py:547-565,1063-1101`); stdout reports interval-derived hit rates. It does not expose the preempted sub-counters separately. | +| Total step tokens | Exact scheduled total is in `SchedulerOutput` | `vllm:iteration_tokens_total` is a histogram observed from output-side computed prompt plus generation tokens (`vllm/v1/metrics/loggers.py:693-724,1148-1171`), not a joinable raw step record. | + +`IterationStats` also has a wall timestamp, output-side prompt/generation data, +preemptions, and finished-request latency samples +(`vllm/v1/metrics/stats.py:325-347`). `Scheduler.make_stats()` produces latest +queue/KV state, drains prefix stats, and passes through the step's CUDA-graph +stat when an output is processed +(`vllm/v1/core/sched/scheduler.py:2228-2264`). These are valuable existing +plumbing, but async scheduling can schedule newer batches before an older +output is processed. Consequently, those latest/drained values are not a safe +per-step composition join without a schedule-time snapshot. + +Bottom line: no new scheduler instrumentation is needed to discover batch and +token composition, preemptions, queues, KV use, or prefix events. A patch is +needed only to serialize their **same-step association** and the missing +histograms/chunk split. + +### 2.3 CUDA-graph observability + +Yes, v0.24.0 can determine the runtime graph mode and capture bucket for each +model step internally: + +- `CUDAGraphStat` records unpadded tokens, padded tokens, padding, and runtime + mode (`vllm/compilation/cuda_graph.py:32-37`). +- The GPU model runner asks the dispatcher for a concrete runtime mode and + `BatchDescriptor`, then conditionally attaches the stat to its output + (`vllm/v1/worker/gpu_model_runner.py:3822-3934`). It reaches the scheduler in + `ModelRunnerOutput` and `update_from_output()` + (`vllm/v1/outputs.py:231-281`; + `vllm/v1/core/sched/scheduler.py:1464-1477`). +- The dispatcher is the source of truth: it pads to a captured descriptor, + tries `FULL`, then `PIECEWISE`, and returns `NONE` when no graph key matches + (`vllm/v1/cudagraph_dispatcher.py:235-324`; the design explanation is at + `docs/design/cuda_graphs.md:81-120`). At serving time, `FULL` and `PIECEWISE` + dispatch to a captured wrapper; `NONE` calls the runnable directly + (`vllm/compilation/cuda_graph.py:233-360`). + +Thus `runtime_mode != NONE` is a graph-path dispatch, and `num_padded_tokens` is +the selected token bucket. `PIECEWISE` means graph-wrapped compiled pieces, not +a full-step graph hit. Startup normally captures all dispatcher descriptors +(`vllm/v1/worker/gpu_model_runner.py:6595-6643`), but `CUDAGraphStat` has no +capture-versus-replay bit; a lazy first capture and a replay have the same stat. +With built-in `cudagraph_metrics`, the stat is aggregated +into a frequency table and printed at the logging interval +(`vllm/config/observability.py:56-58`; +`vllm/compilation/cuda_graph.py:40-124`); it is not exposed as a raw per-step +Prometheus event. OpProf only needs to preserve the already-computed object on +every enabled step. + +Capture modes and sizes live in `CompilationConfig`: +`cudagraph_mode`, `cudagraph_capture_sizes`, and +`max_cudagraph_capture_size` are defined at +`vllm/config/compilation.py:587-688`. The V1 default is +`FULL_AND_PIECEWISE`. Final sizes are generated or validated in +`VllmConfig._set_cudagraph_sizes()`; the default is 1/2/4, then steps of 8 below +256 and 16 above it, bounded by token/batch configuration, with user overrides +written back to the final list (`vllm/config/vllm.py:1635-1800`). + +### 2.4 MoE routing and Qwen3-30B-A3B backend + +Qwen3 MoE creates a replicated gate and `FusedMoE` with the model's number of +experts and top-k, and normally runs its router internally +(`vllm/model_executor/models/qwen3_moe.py:137-249`). The causal-LM wrapper +retains the list of MoE layers and their expert topology +(`vllm/model_executor/models/qwen3_moe.py:662-729`). Router top-k IDs are +available immediately after routing and before EPLB remapping +(`vllm/model_executor/layers/fused_moe/router/base_router.py:233-278`). + +There is an exact, built-in access path: `--enable-return-routed-experts` +(`vllm/config/model.py:209-215`; `vllm/engine/arg_utils.py:826-830`). It installs +per-layer callbacks, writes top-k IDs to preallocated GPU buffers, copies the +step to pinned CPU memory, and returns it through `ModelRunnerOutput` +(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:58-84`; +`vllm/v1/worker/gpu_model_runner.py:3652-3666,4637-4683,7382-7437`). This is +reachable without modifying the fused kernels or saving router logits. +Raw router-logit tensors are not plumbed to engine outputs; exposing those +would be deeper and much higher-volume surgery than using the existing top-k +ID callback. + +It is not cheap enough for Layer 1. The per-worker transit buffer is a few MB; +the scheduler's whole-slot buffer can reach multiple GB and performs a CPU +fancy-index write each step +(`vllm/model_executor/layers/fused_moe/routed_experts_capturer.py:223-305`; +the scheduler store is at `vllm/v1/core/sched/scheduler.py:1501-1520`). It is +also incompatible with pipeline parallelism greater than one and with KV +connectors (`vllm/config/vllm.py:871-893`). Exact per-layer expert load should +therefore be collected in a separate sampled Layer-2 run and reduced offline. +EPLB has load state, but enabling it adds communication and its logs are +aggregate balancing summaries; it is not a free observability tap +(`vllm/distributed/eplb/eplb_state.py:62-171,478-575`). + +For the normal unquantized BF16/FP16 Qwen3-30B-A3B configuration on H20/SM90, +the automatic unquantized oracle explicitly moves both FlashInfer backends +behind Triton on Hopper because they are expected to be slower +(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:43-86`). It selects +the first supported backend and documents possible shape-specific fallbacks +(`vllm/model_executor/layers/fused_moe/oracle/unquantized.py:152-250`). The +primary backend is therefore **Triton** under those assumptions. A quantized +checkpoint, LoRA, expert activation format, or explicit `--moe-backend` can +change that conclusion and must be recorded with the campaign run. + +### 2.5 V1 execution loop and natural hooks + +The control path is: + +```text +EngineCore busy loop + -> Scheduler.schedule() + -> executor.execute_model(SchedulerOutput) + -> GPUWorker -> GPUModelRunner.execute_model() + -> Scheduler.update_from_output(SchedulerOutput, ModelRunnerOutput) +``` + +Evidence and hook implications: + +- `EngineCore.run_busy_loop()` and `_process_engine_step()` drive steps at + `vllm/v1/engine/core.py:1259-1317`. +- The normal step retains the exact scheduler output across the future and + passes it back to `update_from_output()` + (`vllm/v1/engine/core.py:479-508`). The batch-queue path does the same for + multiple in-flight batches (`vllm/v1/engine/core.py:519-632`). +- The scheduler's stable interface is schedule/update at + `vllm/v1/core/sched/interface.py:51-107`. The base implementation builds the + output at `vllm/v1/core/sched/scheduler.py:1012-1100`, mutates request state + immediately afterward at lines 1130-1155, and consumes model results at + lines 1464-1803. +- Worker execution wraps `gpu_model_runner.execute_model()` at + `vllm/v1/worker/gpu_worker.py:895-898`; the model runner's entry is + `vllm/v1/worker/gpu_model_runner.py:4055-4069`. + +The natural Layer-1 hooks are therefore in the base scheduler: snapshot after +the `SchedulerOutput` is complete but before request state advances, then +finalize with the matching model output. This covers the async subclass and +avoids frontend/output aggregation. The only worker change is broadening the +existing CUDA-graph-stat condition. Layer 2 uses the existing profile control +path unchanged. + +### 2.6 Relevant changes from 0.20.0 + +For comparison only, tag `v0.20.0` was fetched into the allowed clone without +changing the pinned working tree. It resolves to commit +`88d34c6409e9fb3c7b8ca0c04756f061d2099eb1`. + +- Async scheduling and the batch-queue architecture already existed in 0.20.0; + `SchedulerConfig` selected `AsyncScheduler` at + `v0.20.0:vllm/config/scheduler.py:146-176`, and the core schedule/future/update + paths were at `v0.20.0:vllm/v1/engine/core.py:402-431,443-533`. It is + incorrect to treat async scheduling as new in 0.24.0. +- 0.24.0 adds DP prefill cadence/throttling and passes a prefill-throttle flag + into `schedule()` (`vllm/config/scheduler.py:140-161` and + `vllm/v1/engine/core.py:474-490`). It also hardens zero-token/dummy-iteration + handling (`vllm/v1/engine/core.py:433-472`) and changes when the batch queue + fills. +- The 0.24 scheduler contains explicit scheduled/processed sequence fences and + snapshots state that must survive later async schedules, including routed + experts (`vllm/v1/core/sched/scheduler.py:293-322,1093-1165`). That is direct + evidence that an OpProf hook must not read mutable request/queue state only + when an older output returns. +- `vllm/config/profiler.py`, `vllm/profiler/wrapper.py`, and + `vllm/entrypoints/serve/profile/api_router.py` are byte-identical between the + two tags (`git diff --quiet v0.20.0..v0.24.0` returned success). Profiler + endpoint semantics did not move; the scheduler/core line numbers and safe + composition hook did. + +Practical conclusion: do not port a 0.20 line-number patch into EngineCore. +Hook the 0.24 base scheduler's schedule/update pair so sync, async, and batch +queue paths share one implementation. + +## 3. Proposed dual-layer design + +Layer 1 emits one schema-versioned JSONL record per scheduler step from the +EngineCore/scheduler process. It contains step/timestamps; scheduled and decode +batch counts; scheduled prefill/decode tokens; aggregate first/middle/final +chunk information; fixed-bucket context and chunk-size histograms; exact step +preemptions; schedule-time running/waiting/deferred queues; total/free/used KV +blocks and ratio; local/external prefix deltas; and CUDA-graph mode/bucket. No +request IDs or raw context-length lists are written. MoE expert load is null and +explicitly marked Layer-2-only. + +Records are encoded with the existing `msgspec` dependency +(`requirements/common.txt:35`), placed into a +bounded non-blocking queue, and drained by one background JSONL writer. It +flushes every 1 MiB or one second and at clean shutdown, with explicit drop/gap +counters and no per-step `fsync`. `VLLM_OPPROF_DIR` is the startup-only switch; +unset is a true no-op. The unmeasured foreground estimate is 25-100 +microseconds per step, so Phase 2 must enforce the less-than-3% overhead gate +with paired off/on serving runs. + +Layer 2 configures the already-built torch profiler and samples short windows, +initially two warm-up plus eight active worker iterations. Set +`max_iterations=8`; the wrapper stops the underlying profiler on the subsequent +step after the counter exceeds the limit, as its existing test documents +(`vllm/profiler/wrapper.py:83-114`; +`tests/v1/worker/test_gpu_profiler.py:77-98`). The controller then calls +`/stop_profile` to clear control state. Step-count/on-demand endpoint control is +preferred to wall time. Layer 2 writes one trace per TP worker and joins to +Layer 1 through the existing iteration annotation, order, token counts, and +timestamps. + +Under CUDA-graph replay, the profiler is configured for CUDA activity, but the +Python callable is not re-executed: vLLM calls `CUDAGraph.replay()` +(`vllm/compilation/cuda_graph.py:233-360`). CUDA activity may therefore be +visible through Kineto/CUPTI, while Python/PyTorch scopes cannot be assumed to +provide the original per-layer attribution. Layerwise NVTX is explicitly +unsupported with graphs (`vllm/config/observability.py:60-63`). Phase 2 must +check whether the pinned torch/CUPTI combination exposes individual replayed +kernels or a coarser graph launch. A matched eager window can identify kernel +families but must not replace graph-enabled production timing. + +Layer-2 kernel time is grouped into attention, router/top-k, expert/dense GEMMs, +normalization/activation, collectives, and sampling. Per-rank critical-path +times calibrate a composition/graph-bucket-conditioned Layer-1 attribution; +held-out profiler windows validate that mapping. Exact routed experts are +enabled only in a separate sampled server run. + +### Patch surface + +| File | Approximate lines | +|---|---:| +| `vllm/envs.py` | 4 | +| `vllm/v1/opprof.py` (new) | 220 | +| `vllm/v1/core/sched/scheduler.py` | 32 | +| `vllm/v1/worker/gpu_model_runner.py` | 4 | +| `tests/v1/core/test_opprof.py` (new) | 190 | +| **Total** | **5 files / about 450 lines** | + +Zero-patch wins: profiler config/endpoints/wrapper, `SchedulerOutput`, existing +metrics plumbing, CUDA kernels, fused MoE/router code, Qwen3 model code, and +frontend APIs. Supporting OpProf together with `--disable-log-stats` would add +about 12 lines in `vllm/v1/core/kv_cache_manager.py`; the design recommends +failing fast instead. + +### Main risks + +- Histogram construction and JSON encoding are the likely foreground overhead; + the 3% budget must be measured, not inferred. +- Async scheduling can misassociate latest queue/cache state with an older + output; snapshot inside `schedule()` and pair by the exact output object. +- Layer 1 must be scheduler-owned to avoid TP duplicates; Layer 2 remains + per-rank and needs offline aggregation. +- At 500 steps/s and about 1 KiB/record, an uncompressed stream is roughly + 44 GB/day per EngineCore. Runs must be duration-bounded, with queue/drop + counters and post-run compression. +- Kineto perturbs execution, and graph replay loses Python-scope attribution; + it is a sampled calibration instrument, not an always-on latency source. + +## 4. Open design decisions + +1. Approve JSONL/`msgspec`, an 8192-record queue, context edges + `(128, 256, 512, 1K, 2K, 4K, 8K, 16K, 32K, 64K, 128K, +inf)`, and chunk-size + edges `(16, 32, 64, 128, 256, 512, 1K, 2K, +inf)`. +2. Approve exact MoE expert load as Layer-2-only. The alternative is a new GPU + histogram path, with a larger patch and an overhead/graph-capture risk. +3. Confirm checkpoint precision/quantization and intended TP/EP/DP topology. + The H20 Triton conclusion assumes unquantized BF16/FP16, auto backend, and no + LoRA. +4. Choose the initial profiler window: recommended two warm-up plus eight + active iterations, or a longer active window. +5. Decide whether the less-than-3% gate applies to point estimates or the upper + 95% confidence bound for every primary metric. +6. Confirm that OpProf campaign runs may reject `--disable-log-stats`; support + for it increases the patch to six files and about 462 lines. + +## Data sanity block + +- **Pinned root listing:** n=45 names; lexicographic min=`.buildkite`, + max=`vllm`; distinct=45; SHA-256=`75222e5d3bacdd043e421c417496aaf0fa5a9429b7244698b6d3ca2218b85b58`. +- **Source versions compared:** n=2 tags; min=`v0.20.0`, max=`v0.24.0`; + distinct=2; pinned working-tree version remains exactly `v0.24.0`. +- **Torch pin observations:** n=4 declarations + (`pyproject`, build requirements, CUDA runtime requirements, CMake); + min=max=`2.11.0`; distinct=1. +- **Evidence-bearing source files inspected and cited:** n=44 + distinct v0.24.0 paths; min/max not applicable to categorical paths; + v0.20.0 comparison touched n=5 distinct paths; min/max not applicable. +- **Invariants checked:** exact tag equals `v0.24.0`; HEAD equals the recorded + commit; clone is clean; top-level names are unique; requested path was used; + torch pins agree; CUDA-graph modes are in `{NONE, PIECEWISE, FULL}`; KV usage + definition is bounded in `[0,1]`; scheduled-token total is defined as the sum + of per-request scheduled tokens; graph bucket is not smaller than unpadded + tokens; no remote/GPU/install/model-download action occurred. +- **Benchmark/statistical conclusion check:** no performance benchmark was run, + so there are no per-configuration curves or measured overhead values on which + to perform identical-value, monotonicity, or continuity checks. All overhead + numbers in the design are labeled estimates. diff --git a/docs/opprof/phase2-smoke-dash0.md b/docs/opprof/phase2-smoke-dash0.md new file mode 100644 index 0000000..1270a2b --- /dev/null +++ b/docs/opprof/phase2-smoke-dash0.md @@ -0,0 +1,1233 @@ +# OpProf Phase 2 single-H20 smoke on dash0 + +Final status: **PASS**. After preserving the earlier measurements and tracing +the entire CUDA-graph-stat path, attempt 4 found that `VLLM_OPPROF_DIR` was +accidentally included in vLLM's torch.compile cache key. Each unique ON output +directory therefore forced a distinct graph/AOT cache and confounded the +attempt-2 comparison; CUDA-graph-stat construction and msgspec serialization +were not the material cost. The minimal production fix is one ignored-factor +entry in `vllm/envs.py`. + +The unchanged A-P2-1 rerun produced mean ON throughput 29.666501 req/s, mean +OFF throughput 29.654057 req/s, and overhead -0.04196% with paired-bootstrap +95% CI [-0.17443%, 0.04550%]. Its upper bound is below the unchanged 3% gate. +The accepted Layer-2 run then produced a loadable 16,416,452-byte Kineto trace +with exactly two discarded warm-up iterations and eight active profiler steps. + +Attempt 2 remains preserved below as a valid measurement of the pre-fix patch: +4.1816% overhead with 95% CI [3.1364%, 4.7117%]. Attempt 3 then exonerated the +recorder stages and localized the boundary outside `opprof.py`, as required, +before attempt 4 identified the exact cache-key operation and made the +evidence-driven fix. + +Attempt 1 remains preserved below as an invalidated measurement. Its raw +calculation was 13.1133% overhead with 95% CI [6.2434%, 19.5259%], but the +orchestrator rejected that approximately 10-second, cold-start-confounded +protocol rather than adjudicating the patch from it. + +Attempt 1 also had a reproducibility anomaly: its input/output length arrays +and token totals were identical in all ten trials, but the generated-text +digest differed in all ten. Between 185 and 190 of 200 responses in each later +trial matched trial 1 exactly. The bundled client also generated ten distinct +request-ID prefixes even though `--seed 20260711` and `--temperature 0` were +fixed. A-P2-1 removed the work-volume consequence with fixed completion length. + +## Dated amendment A-P2-1 (2026-07-11) + +This amendment was written before launching any rerun. Attempt 1 remains below +as an invalidated measurement and will not be erased or pooled with the amended +data. The 3% CI-upper-bound gate is unchanged. + +1. Each server will receive 128 fixed-seed warm-up requests after readiness and + before timing. These requests are excluded from the measurement. One + additional unmeasured server start and representative warm-up will run before + measured trial 1 so that no measured arm is the globally coldest start. +2. Every measured trial will use the same 4,000-request stream. A trial is + acceptable only if its measured window is at least 120 seconds; 4,000 also + exceeds the 2,000-request minimum. If any trial is shorter than 120 seconds, + the amended series is invalid and stops without a gate conclusion. +3. The random input distribution remains centered at 512 tokens with input + range ratio 0.5. Output range ratio is fixed at 0, `--ignore-eos` is + explicit, `--random-output-len 64` fixes `max_tokens=64`, temperature is 0, + seed is 20260711, and request-ID prefix is fixed as `opprof-p2a-`. Thus every + successful request must produce exactly 64 completion tokens in every run. +4. Ten fresh-server trials use the counterbalanced sequence + ON,OFF,OFF,ON,ON,OFF,OFF,ON,ON,OFF. Adjacent trials form five pairs; within + each pair, the ON/OFF orientation alternates. The only intended arm + difference remains whether `VLLM_OPPROF_DIR` is set. +5. Before and after each trial, the record will include `/proc/loadavg`, a + `nvidia-smi -q -d CLOCK` snapshot for physical GPU 0, GPU memory/utilization, + and a host-process snapshot. A periodic process/GPU sampler will record any + newly appearing process owned by another user. Server lifetime is measured + from launch to completed teardown. +6. Before serving, an offline dash0 microbenchmark will time the complete + `capture_start` + `begin` + `finalize` + non-blocking writer `submit` path + for a synthetic 64-request scheduler state over 10,000 measured iterations. + It will report total and per-step time; setup, writer drain, and close are + outside the timed region. +7. An artifact addendum will launch one extra short server forced to vLLM + 0.24.0 `cudagraph_mode=PIECEWISE`. Its accepted footer-valid JSONL must + contain at least one PIECEWISE record with `bucket_tokens >= + unpadded_tokens` and `padding_tokens = bucket_tokens - unpadded_tokens`. +8. The primary statistic remains `1 - mean(ON) / mean(OFF)`. Its 95% CI will + use a paired percentile bootstrap over the five adjacent counterbalanced + pairs, 100,000 resamples, seed 20260711, NumPy's default linear percentile, + and no outlier removal. If the upper bound exceeds 3%, execution stops again. + Layer-2 runs only after this unchanged gate passes. + +### A-P2-1 execution clarification (2026-07-11) + +This clarification was recorded before restarting the accepted ten-run series. +The first 4,000-request invocation used unbounded client concurrency and is an +excluded admission pilot, not an overhead trial: the client hit its 1,024-file +descriptor ceiling, so only 984 requests succeeded and 3,016 failed with +`OSError: [Errno 24] Too many open files`. Its 42.799-second partial duration +and 22.991182 req/s successful-request rate are not gate inputs. + +The accepted series retains 4,000 measured requests and every setting above, +but fixes `--max-concurrency 200` identically in both arms. This bounds client +file descriptors while keeping the server saturated: the earlier valid +200-request burst sustained approximately 21 req/s, implying an approximately +190-second 4,000-request window. All ten accepted trials restart from pair 1; +the excluded pilot is not pooled, substituted, or counted. The >=120-second, +4,000-success, zero-failure, and fixed-64-output acceptance checks remain hard. + +## Goal and hard-gate outcome + +The goal was to deploy the accepted Layer-1 patch at base `ee0da84a` and +tip-equivalent `668cfb7e`, validate one scheduler-owned JSONL stream on one H20, +measure its serving overhead with the pre-registered ten-run protocol, and then +sample one two-warm-up/eight-active torch-profiler window only if the earlier +gates passed. + +| Hard gate | Result | Evidence | +|---|---|---| +| Artifacts | **PASS** | The original footer-valid artifact passed schema, footer, step, drop, histogram, KV, graph-field, and composition checks. The A-P2-1 addendum forced PIECEWISE and added a footer-valid stream with 129 PIECEWISE records preserving bucket/padding identities. | +| Overhead CI | **PASS** | Attempt-4 fixed point overhead -0.04196%; paired-bootstrap 95% CI [-0.17443%, 0.04550%]; required upper bound <=3%. Attempt 2's valid pre-fix FAIL is retained separately. | +| Layer-2 | **PASS** | Exactly 2 warm-up + 8 active profiler iterations; gzip/JSON-loadable Kineto trace, 790,843 events and 7,367 kernel events. The active steps ran uncaptured (`NONE`) because their approximately 8,192-token batches exceeded the captured bucket limit; mode coverage is reported, not hidden. | + +## Environment and deployment + +| Field | Value | +|---|---| +| Host | `dash0` (`ds-07429c65-1-6c5fd97778-9vhkr`) | +| OS / kernel | Ubuntu 24.04 / Linux `5.10.134-013.8.2.kangaroo.al8.x86_64` | +| CPU / RAM | 2 sockets, 80 cores/socket, 160 logical CPUs, 1,554,500,000 KiB RAM; CPU string `Intel(R) Xeon(R) Processor` | +| GPU used | Physical GPU 0, NVIDIA H20, UUID `GPU-ad3e049a-5bf0-44b7-e7f1-9af297b172af` | +| Driver / reported CUDA | NVIDIA 580.95.05 / CUDA 13.0 | +| Wheel path | Exact-base precompiled `cu130` wheel; metadata for commit `ee0da84ab9e04ac7610e28580af62c365e898389` returned HTTP 200 and contained vLLM 0.24.0 x86-64 | +| Python / torch | Python 3.12.3; torch `2.11.0+cu130`; `torch.version.cuda=13.0` | +| vLLM | Editable installation still reports build metadata `0.24.1.dev3+g668cfb7e2`; runtime source contains the three accepted commits plus two fix/test commits. Compiled extensions remain from the exact v0.24.0 base wheel. | +| Compiled extension check | `vllm._C_stable_libtorch` and `vllm._moe_C_stable_libtorch` imported successfully | +| msgspec | 0.21.1 | +| Source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` | +| Source state | dash0 HEAD `9ad32ff2eeb147c074beeb54418551923a57857b` and local HEAD `bbfa7176a6a3686a88ee66696f1ad8d754559d96`; commit hashes differ because `git am` dates differ, but all five patch IDs match. `HEAD~5=ee0da84ab9e04ac7610e28580af62c365e898389`; both trees clean. | +| Active venv | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, 8.6 GiB | +| Persistent audit workdir | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711` | +| Weights | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B` | +| Weight validation | `Qwen3MoeForCausalLM`, `torch_dtype=bfloat16`, no `quantization_config`, 128 experts, top-8; 16/16 shards present, 61,066,575,648 shard bytes | +| MoE backend | `TRITON Unquantized MoE backend`, observed in every campaign server-start log, including all 19 attempt-4 launches (48/48 total) | + +The driver reports CUDA 13.0, and vLLM 0.24.0 maps CUDA major 13 to `cu130`. +The `cu129` path was therefore not selected. The exact `cu130` metadata probe +was completed before installation. + +The installation command was: + +```bash +VLLM_USE_PRECOMPILED=1 \ +VLLM_PRECOMPILED_WHEEL_COMMIT=ee0da84ab9e04ac7610e28580af62c365e898389 \ +VLLM_PRECOMPILED_WHEEL_VARIANT=cu130 \ +/tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python -m pip install -e \ + /home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0 +``` + +The successful local-rootfs install took 377 seconds. `pip check` reported no +broken requirements. + +The final exported patch files are checksum-identical to the dash0-applied +series: + +| File | SHA-256 | +|---|---| +| `0001-Add-lightweight-per-step-OpProf-telemetry.patch` | `3b6843906dde103ac47fe6862a092cc507de2f6e721cf39c147d89b6b70ce27d` | +| `0002-Add-standalone-OpProf-telemetry-tests.patch` | `897c770d5f137bd49a852530d5e8dc54666a867f754ef8fd7825c7fd64bbbb99` | +| `0003-Log-the-OpProf-output-path-at-startup.patch` | `c60b44f23d02e753b46e06d9a8d845203095802ce32538c11411b1b7069e69e7` | +| `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch` | `c76bb2cf20dcdfd3b9d0ef40d4e235f8fa9e6e703cd9af9cd084790f65caf0f0` | +| `0005-Keep-compile-factor-regression-import-light.patch` | `16ab2f1101067d9df2831fba312d3e82dfa9d24404954ac817e3fd9681be5537` | +| `apply.sh` | `2e2a4392b3b5714ccf38d0cf2174d829c557af23597f91fa6162390b684dc92c` | + +`apply.sh` returned `OpProf patch series is already applied.` only after +checking that the clone contained the exact five patch IDs rooted at the +required base. A second clean-base worktree applied all five patches and its +idempotent reapplication returned the same message. + +Remote unit-test command and result: + +```bash +/tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python -m pytest \ + --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q +``` + +```text +............... [100%] +15 passed in 0.18s +``` + +The final import-light local run passed 15/15 in 0.02 seconds, Ruff passed, and +the remote clean-base application passed 15/15 in 0.07 seconds. The original +deployed 14/14 result is retained in the archived pre-fix evidence. + +## Serving protocol + +### Attempt 1 protocol (invalidated) + +All servers used one physical H20. GPU 0 had 0 MiB, 0% utilization, and no +compute process immediately before every launch. The server command was: + +```bash +CUDA_VISIBLE_DEVICES=0 \ +VLLM_OPPROF_DIR=/opprof \ +/tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve \ + /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \ + --tensor-parallel-size 1 +``` + +OFF trials used the same command with `VLLM_OPPROF_DIR` explicitly unset. +No serving knobs beyond the model path and explicit TP1 were changed. Startup +logs selected the default FULL+PIECEWISE CUDA-graph configuration and reported +the Triton unquantized MoE backend. + +The artifact smoke used this fixed-seed, mixed-length load with no client +warm-up: + +```bash +vllm bench serve --backend openai --host 127.0.0.1 --port 8000 \ + --endpoint /v1/completions \ + --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \ + --dataset-name random --num-prompts 200 \ + --random-input-len 512 --random-output-len 64 \ + --random-range-ratio '{"input":0.5,"output":0.5}' \ + --request-rate inf --seed 20260711 --temperature 0 \ + --save-result --save-detailed --disable-tqdm \ + --percentile-metrics ttft,tpot,e2el --metric-percentiles 50,95,99 +``` + +The overhead trials added `--num-warmups 16`; the bundled benchmark executes +those warm-up requests before starting its measured timer. Every trial used a +fresh server. The exact order was ON,OFF repeated five times. The only intended +ON/OFF difference was the OpProf environment variable. + +After a measured load had become idle, the EngineCore process was signalled +first and allowed to execute `scheduler.shutdown()` and drain the writer. The +API parent was then stopped. This ordering was used identically for ON and OFF +overhead trials because the default parent-first shutdown force-killed the +EngineCore before the first smoke footer could be written. + +### A-P2-1 accepted protocol + +The amended order was +ON,OFF,OFF,ON,ON,OFF,OFF,ON,ON,OFF. One default-configuration burn-in server +ran before measured pair 1. Every measured server received 128 excluded warm-up +requests followed by this fixed-work command: + +```bash +vllm bench serve --backend openai --host 127.0.0.1 --port 8000 \ + --endpoint /v1/completions \ + --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \ + --dataset-name random --num-prompts 4000 --num-warmups 128 \ + --max-concurrency 200 \ + --random-input-len 512 --random-output-len 64 \ + --random-range-ratio '{"input":0.5,"output":0.0}' \ + --ignore-eos --request-rate inf --seed 20260711 --temperature 0 \ + --request-id-prefix opprof-p2a- \ + --save-result --save-detailed --disable-tqdm \ + --percentile-metrics ttft,tpot,e2el --metric-percentiles 50,95,99 +``` + +All ten accepted runs completed 4,000 requests with zero failures. Every +completion was exactly 64 tokens; input/output length digests each had one +distinct value across runs. Each internal measured duration exceeded 120 +seconds. The client generated seven distinct text-content digests, but this no +longer changed generation work because `ignore_eos` and `max_tokens=64` fixed +every completion length. + +Before and after every run, the controller captured `/proc/loadavg`, full +`nvidia-smi -q -d CLOCK` output, selected-GPU state, and a host process +snapshot. A five-second sampler recorded clocks and newly appearing processes. +The server was restarted and torn down identically for every arm. A transient +4 MiB, 0%-utilization driver-accounting sample after trial 1 stopped the first +controller before trial 2 launched; after five consecutive 0 MiB/0% samples, +the unchanged sequence resumed at trial 2. + +## Artifact gate + +### Invalid first attempt + +The first smoke completed 200/200 requests in 9.634701 seconds at 20.758299 +req/s. It produced 110 step records, including one runtime PIECEWISE record: + +```text +runtime_mode=PIECEWISE, unpadded_tokens=478, bucket_tokens=480, +padding_tokens=2 +``` + +The parent-first default shutdown then force-killed the EngineCore and the file +had no footer. This file was retained as failure evidence but excluded from the +accepted artifact checks. + +### Footer-valid rerun + +The unchanged load rerun completed 200/200 requests in 9.391693 seconds at +21.295415 req/s. Its JSONL contained 109 records followed by one footer. +Every line decoded against a strict, unknown-field-forbidding `msgspec.Struct` +schema. + +| Check | Result | +|---|---| +| JSONL / schema | 110/110 lines schema-valid; 109 records plus one terminal footer | +| Footer | `encoded=109`, `written=109`, `dropped=0`; `encoded=written+dropped` | +| Steps / pending leak | Step indices exactly 0 through 108; Prometheus counted 107 executed iterations and the stream contained 2 zero-token steps, totaling 109 | +| Queue drops | Footer drops 0 and sum of `dropped_records_before` 0 | +| Per-record invariants | All counters non-negative; context/chunk histograms sum correctly; KV block accounting exact; KV usage in [0,1]; completion timestamp not before submission | +| CUDA-graph fields | Hit/mode identity valid; bucket >= unpadded; padding = bucket - unpadded for every record | +| Layer-1 MoE field | `moe_expert_load` null in every record | +| Last state | Zero-token record, queues all zero, KV used blocks zero | + +Observed graph modes in the footer-valid file were FULL=93 and NONE=16. The +bucket family had n=109, min=0, max=8192, distinct=32; padding had n=109, +min=0, max=7, distinct=8. No valid-file record exercised PIECEWISE. A planned +single-request PIECEWISE probe was not run after the overhead stop rule fired, +so the strict artifact gate remained unresolved at the end of attempt 1 despite +the other artifact checks passing. + +### Composition cross-check + +Random inputs covered 261 to 761 tokens; requested/actual outputs covered 32 +to 96 tokens because the random OpenAI-compatible benchmark ignores EOS for +this dataset. + +| Quantity | OpProf | Client ground truth | Delta / tolerance | +|---|---:|---:|---:| +| Prefill tokens | 102,848 | 102,848 input tokens | 0; exact | +| Decode tokens | 11,987 | 12,187 completion tokens | -200, or 1.6411%; allowed structural tolerance was one first token per completed request | +| Decode plus first-token correction | 12,187 | 12,187 completion tokens | 0; exact | + +The one-token/request correction is expected because the first completion +token is sampled from the prefill logits; only later completion tokens require +decode inputs. + +### A-P2-1 PIECEWISE addendum + +The addendum forced the exact vLLM 0.24.0 configuration +`--compilation-config '{"cudagraph_mode":"PIECEWISE"}'` and ran 16 excluded +warm-ups plus 64 requests with fixed 64-token completions. EngineCore-first +shutdown produced one footer-valid file. Every line decoded with the same +strict, unknown-field-forbidding msgspec schema. + +| Check | Result | +|---|---| +| File / footer | 139 records plus one footer; `encoded=139`, `written=139`, `dropped=0` | +| Step accounting | Indices 0 through 138, contiguous and unique; no pending leak | +| Runtime modes | PIECEWISE=129, NONE=10 | +| PIECEWISE unpadded tokens | n=129, min=1, max=241, distinct=9 | +| PIECEWISE buckets | n=129, min=1, max=248, distinct=6 | +| PIECEWISE padding | n=129, min=0, max=7, distinct=5 | +| Identity | Every record had bucket >= unpadded and padding = bucket - unpadded | + +This closes the attempt-1 coverage red flag. The final artifact gate is +**PASS**: the original artifact supplies the composition and scheduler-step +cross-checks, while the addendum supplies footer-valid runtime PIECEWISE +coverage without weakening any earlier invariant. + +## Attempt 1 overhead (invalidated measurement) + +Metric: bundled-client `request_throughput`, in req/s. The registered statistic +was `1 - mean(ON) / mean(OFF)`. The 95% interval used paired percentile +bootstrap resampling of the five adjacent ON/OFF pairs, 100,000 resamples, +analysis seed 20260711, NumPy's default linear percentile, and no outlier +removal. + +| Trial | Mode | Throughput (req/s) | Measured duration (s) | Server GPU lifetime (s) | +|---:|:---:|---:|---:|---:| +| 1 | ON | 20.487487 | 9.762056 | 155 | +| 2 | OFF | 22.864574 | 8.747156 | 139 | +| 3 | ON | 22.685507 | 8.816201 | 144 | +| 4 | OFF | 26.350950 | 7.589859 | 107 | +| 5 | ON | 19.877357 | 10.061700 | 144 | +| 6 | OFF | 23.302197 | 8.582882 | 92 | +| 7 | ON | 20.419460 | 9.794578 | 150 | +| 8 | OFF | 20.378458 | 9.814285 | 93 | +| 9 | ON | 20.068985 | 9.965626 | 145 | +| 10 | OFF | 26.269114 | 7.613504 | 91 | + +Adjacent-pair overheads were 10.3964%, 13.9101%, 14.6975%, -0.2012%, and +23.6024%. + +| Statistic | Value | +|---|---:| +| Mean ON | 20.707759 req/s | +| Mean OFF | 23.833059 req/s | +| Point overhead | 13.1133% | +| 95% CI | [6.2434%, 19.5259%] | +| Raw calculation | 19.5259% > 3%; later invalidated as a measurement | + +All five ON artifacts closed with valid footer accounting and zero drops. +Their record counts were 183, 186, 183, 184, and 184. + +The shape-level load checks passed: all trials completed 200 requests with no +reported failures; every result had 102,848 input and 12,187 output tokens; and +the input-length and output-length digests each had one distinct value. The +token-content check did not pass. Generated-text digests had ten distinct +values, and exact response matches against trial 1 were +`[200, 189, 186, 187, 190, 190, 186, 188, 188, 185]`. In addition, the bundled +client's automatically selected `request_id_prefix` had ten distinct values. +The performance numbers are therefore reported for the gate as registered but +should not be used to claim that instrumentation caused the entire measured +difference. + +The orchestrator subsequently invalidated this attempt because the OFF-arm +spread was 29%, the approximately 10-second windows included cold compile and +autotune effects, and strict ON,OFF alternation assigned the globally coldest +run to ON. These ten rows are not pooled with A-P2-1 and do not adjudicate the +patch. + +## A-P2-1 amended overhead gate + +The first unbounded 4,000-request invocation was an excluded admission pilot: +the client exceeded its 1,024-file-descriptor ceiling, completing 984 requests +and failing 3,016 with `Too many open files`. The accepted series restarted +from pair 1 with maximum concurrency 200, as pre-recorded in the execution +clarification above. + +| Trial | Mode | Throughput (req/s) | Measured duration (s) | Server allocation (s) | +|---:|:---:|---:|---:|---:| +| 1 | ON | 28.202897 | 141.829399 | 297 | +| 2 | OFF | 28.800396 | 138.886980 | 267 | +| 3 | OFF | 29.585085 | 135.203261 | 260 | +| 4 | ON | 28.183682 | 141.926096 | 307 | +| 5 | ON | 28.256715 | 141.559272 | 311 | +| 6 | OFF | 29.640096 | 134.952330 | 272 | +| 7 | OFF | 29.632434 | 134.987225 | 266 | +| 8 | ON | 28.249738 | 141.594235 | 312 | +| 9 | ON | 28.187008 | 141.909351 | 308 | +| 10 | OFF | 29.578830 | 135.231855 | 258 | + +Adjacent-pair overheads were 2.0746%, 4.7369%, 4.6673%, 4.6662%, and +4.7055%. The frozen paired-bootstrap analysis used 100,000 resamples, seed +20260711, NumPy's default linear percentile, and no filtering. + +| Statistic | Value | +|---|---:| +| Mean ON | 28.216008 req/s | +| Mean OFF | 29.447368 req/s | +| Point overhead | 4.1816% | +| 95% CI | [3.1364%, 4.7117%] | +| Gate | **FAIL: 4.7117% > 3%** | + +This measurement passed its protocol-validity checks: ten distinct positive +throughputs, ten durations from 134.952330 to 141.926096 seconds, 4,000 +successes and zero failures per run, exactly 2,031,054 input tokens and 256,000 +output tokens per run, one input-length digest, one output-length digest, and +exactly 64 completion tokens per request. Generated text content still had +seven distinct digests, but it could not change token work. + +All five ON JSONLs strict-decoded and closed cleanly. Their record counts were +1,405, 1,406, 1,405, 1,403, and 1,403; each footer had +`encoded=written=records`, `dropped=0`, contiguous step indices, and valid +bucket/padding identities. + +### Host and clock observations + +| Trial | Mode | 1-minute load before / after | Startup (s) | New other-user processes | Other-user GPU samples | +|---:|:---:|---:|---:|---:|---:| +| 1 | ON | 0.15 / 1.63 | 111 | 3 | 0 | +| 2 | OFF | 0.23 / 1.22 | 73 | 11 | 0 | +| 3 | OFF | 1.03 / 1.24 | 65 | 2 | 0 | +| 4 | ON | 1.05 / 1.07 | 113 | 2 | 0 | +| 5 | ON | 0.99 / 1.22 | 115 | 4 | 0 | +| 6 | OFF | 1.11 / 0.93 | 64 | 3 | 0 | +| 7 | OFF | 0.79 / 1.20 | 70 | 2 | 0 | +| 8 | ON | 1.01 / 1.57 | 116 | 2 | 0 | +| 9 | ON | 1.27 / 0.97 | 113 | 3 | 0 | +| 10 | OFF | 0.82 / 1.51 | 64 | 7 | 0 | + +The sampler observed 39 newly appearing root-owned host processes: 13 +`python3`, 6 `bash`, 3 each of `sed`, `runtime-bridge`, `nvidia-smi`, `grep`, +and `config-sync`, 2 `sshd`, and one each of `metrics-monitor`, `envd`, and +`easworker-proxy`. Some were orchestrator state checks. None appeared as an +other-user GPU compute process, and selected GPU 0 had no contamination. + +All 545 periodic GPU samples were P0 with one 1,980 MHz maximum SM clock. +Among 318 active samples, observed SM clocks spanned 1,740 to 1,980 MHz; means +were 1,970.0 MHz ON and 1,973.9 MHz OFF. One-minute host load stayed between +0.15 and 1.63 on 160 logical CPUs. These observations do not support GPU clock +throttling or host CPU saturation as the 4.1816% explanation. Full per-run +before/after clock snapshots and process lists are retained in the archive. + +### Recorder microbenchmark and localization + +The offline benchmark pinned one CPU, used a synthetic state with 64 requests +(8 prefill, 56 decode, 312 scheduled tokens), ran 1,000 excluded warm-ups, and +timed 10,000 complete `capture_start + begin + finalize + submit` calls. The +writer drain and close were outside the timing, as pre-registered. + +| Producer-path metric | Value | +|---|---:| +| Whole-loop cost | **29.1467 us/step** | +| Per-iteration mean / median | 28.9279 / 27.7620 us | +| p95 / p99 | 36.6643 / 40.6231 us | +| Min / max | 26.293 / 70.111 us | +| Distinct nanosecond timings | 4,144 / 10,000 | +| Writer accounting | 11,000 encoded, 11,000 written, 0 dropped, including warm-ups | + +Production ON files averaged 1,404.4 records over active spans of 144.727 to +144.969 seconds: 9.696 steps/s, 914.7 bytes/record, and 8.87 KB/s. Applying +29.1467 us to 1,404.4 steps predicts only about 0.041 seconds per run, whereas +mean ON duration exceeded mean OFF duration by 5.911 seconds, or approximately +4.209 ms per production record. Raw msgspec encode volume and non-blocking +enqueue cost in isolation therefore cannot explain the measured gap. + +The remaining localization hypothesis is an interaction absent from the +offline benchmark: the real `begin` loop can inspect up to 200 requests rather +than 64, and the production writer thread runs concurrently, introducing +possible GIL handoff, queue-lock, flush/I/O, or engine-loop cadence effects. +The systematic startup split (ON 111-116 seconds; OFF 64-73 seconds), although +excluded from throughput, is additional evidence that the enabled path has a +broader runtime interaction. This turn did not profile, tune, or fix any of +these hypotheses after the hard-gate failure. + +## Attempt 3 stage bisection and stop + +### Temporary diagnostic only + +The installed editable dash0 source received a temporary 30-addition, +3-deletion diff, never committed or exported. It introduced +`VLLM_OPPROF_STAGE` with these exact boundaries: + +- `off`: `OpProfRecorder.create()` returned `None`; +- `capture`: `capture_start` and `begin` ran; `finalize` paired/popped the + pending step, then skipped record construction, encode, and submit; +- `encode`: the full producer and queue path ran while the writer thread wrote + to a Python null sink; and +- `full`: unchanged JSONL behavior. + +The exact diff is archived as +`runs/stage-bisection-attempt3/temporary-stage-toggle.diff`, SHA-256 +`6c9186089fd61bc88b92ba38692f299b772cf3bf3b437fd9f9508eddf9ec3f2c`. +Before GPU work, py_compile, semantic stage checks, and all 14 remote unit tests +passed. After the bisection, the diff was reversed and `opprof.py` returned to +SHA-256 `679c9ae3026cd9eb3cdb48a681a91a9f7693c1dc52cc5ca7ade46cb5b268adbe`. + +### Bisection table reported before any fix + +Each fresh server received 128 excluded warm-ups and the same 4,200-request +stream at concurrency 200. All stages completed 4,200/4,200 requests, zero +failures, 2,132,262 input tokens, and 268,800 output tokens. Every completion +was exactly 64 tokens, and input/output length digests were identical. + +| Stage | Throughput (req/s) | Duration (s) | Overhead vs `off` | Startup / allocation (s) | JSONL files / bytes | +|---|---:|---:|---:|---:|---:| +| off | 28.389494 | 147.942050 | 0.000% | 113 / 302 | 0 / 0 | +| capture | 28.257144 | 148.634981 | 0.466% | 114 / 304 | 1 / 96 | +| encode/null sink | 28.221609 | 148.822131 | 0.591% | 126 / 316 | 0 / 0 | +| full/CPFS | 28.269199 | 148.571596 | 0.424% | 111 / 301 | 1 / 1,346,466 | + +The full output directory was on CPFS (`fuse.aliyun-alinas-efc` / `fuseblk`); +`/tmp` was rootfs overlay. Full CPFS output was faster than the null-sink stage +and only 0.424% below `off`, so filesystem I/O was not implicated and the +conditional full-on-`/tmp` trial was not run. + +### Culprit boundary and mandated stop + +No `opprof.py` stage reproduced the valid 4.1816% gate gap. Source inspection +then found the bisection-boundary violation: + +- `vllm/v1/worker/gpu_model_runner.py` sets + `self.opprof_enabled = bool(envs.VLLM_OPPROF_DIR)` during initialization; and +- its cudagraph dispatch path constructs and propagates `CUDAGraphStat` when + either cudagraph metrics or `self.opprof_enabled` is true. + +All four diagnostic servers had non-empty `VLLM_OPPROF_DIR`, so this +worker/model-runner path stayed enabled even when stage `off` returned no +recorder. The stage named `off` was therefore equivalent only at the scheduler +recorder boundary, not to the accepted gate's environment-unset OFF arm. The +evidence localizes the missing overhead outside `opprof.py`, specifically to +the `CUDAGraphStat` construction/propagation path or its downstream interaction; +it does not isolate which operation inside that path is expensive. + +This matches the user's explicit outside-OpProf stop example. Consequently no +minimal fix was made: real branch line delta **0**, no commit, and no patch +re-export. The local and remote branches remain at `668cfb7e`; all three patch +files and `apply.sh` retain their accepted checksums. The unchanged local +standalone suite passed 14/14 in 0.02 seconds. No quick pair, full gate rerun, +or Layer-2 run was authorized after this stop condition. + +## Attempt 4 static CUDA-graph-stat path analysis (2026-07-11; before GPU) + +This analysis was completed and recorded before launching any attempt-4 GPU +trial. The authoritative tree is the accepted vLLM v0.24.0 patch tip +`668cfb7e27e488454dbf09a4927b8a60d6d49b40` in +`/home/gahow/phd/vllm-v0.24.0`. + +### Worker to scheduler: no serializer in this TP1 configuration + +- With TP1/PP1/DP1, `world_size == 1` selects backend `uni` + (`vllm/config/parallel.py:915-916`); `Executor.get_class()` maps that backend + to `UniProcExecutor` (`vllm/v1/executor/abstract.py:73-76`). +- `UniProcExecutor.collective_rpc()` invokes `run_method()` on its in-process + `driver_worker` and returns the object directly + (`vllm/v1/executor/uniproc_executor.py:79-106`); `execute_model()` is only a + thin call through that method (`:108-121`). The EngineCore receives the + resulting `ModelRunnerOutput` from the completed future and passes it to the + scheduler in the same process (`vllm/v1/engine/core.py:479-508`). +- Therefore the comment that `ModelRunnerOutput` can be serialized + (`vllm/v1/outputs.py:231-234`) does not describe this TP1 execution path. + For comparison only, the multiprocessing executor uses its worker response + `MessageQueue` (`vllm/v1/executor/multiproc_executor.py:376-396`), whose + implementation explicitly pickles the complete response + (`vllm/distributed/device_communicators/shm_broadcast.py:727-745`) and + unpickles it at `:772-805`. It does not use a msgspec unknown-type hook. + +Conclusion: adding a `CUDAGraphStat` field cannot poison a worker-to-scheduler +serializer fast path in the measured TP1 server because that serializer does +not run. + +### Exact producer work when the stat is enabled + +The accepted OpProf patch adds only two CUDA-graph-stat enablement lines to +the upstream path: it caches `bool(VLLM_OPPROF_DIR)` at runner construction +(`vllm/v1/worker/gpu_model_runner.py:429-440`) and ORs that flag with upstream +`observability_config.cudagraph_metrics` (`:3920-3925`). When the condition is +true, each executed step performs: + +1. three existing CPU integer reads/arithmetic operations, +2. `str(cudagraph_mode)`, and +3. construction of one frozen four-field `CUDAGraphStat` dataclass + (`vllm/compilation/cuda_graph.py:32-37` and + `vllm/v1/worker/gpu_model_runner.py:3925-3930`). + +The dispatch decision and `BatchDescriptor` already exist before this branch +(`gpu_model_runner.py:3867-3918`); the metrics flag is not passed to the +dispatcher. The stat is carried through `ExecuteModelState` (`:405-418`, +`:4402-4413`, `:4452-4464`) and assigned to `ModelRunnerOutput` (`:4625-4639`). +There are no CUDA events, event reads, tensor `.item()` calls, stream +synchronizations, dispatcher re-queries, or feature-specific logs in this +branch. The only nearby `.item()` is the pre-existing DP>1 coordination at +`:3907-3918`, which is not entered in this DP1 smoke and is independent of the +metrics condition. + +### Scheduler and EngineCore-to-frontend handling + +The scheduler reads the field once +(`vllm/v1/core/sched/scheduler.py:1485-1497`), supplies it to the existing +`make_stats()` path (`:1812-1822`), and stores it as the +`SchedulerStats.cudagraph_stats` field (`:2255-2291`; field definition at +`vllm/v1/metrics/stats.py:170-198`). OpProf consumes the same object locally +after that assignment (`scheduler.py:1824-1828`). + +`EngineCoreOutputs` is a typed, array-like `msgspec.Struct` with an optional +`SchedulerStats` (`vllm/v1/engine/__init__.py:220-234`). The EngineCore output +thread calls `MsgpackEncoder.encode_into()` and sends the resulting ZMQ frames +(`vllm/v1/engine/core.py:1589-1654`); the async frontend constructs +`MsgpackDecoder(EngineCoreOutputs)` (`vllm/v1/engine/core_client.py:585-595`) +and decodes those frames at `:1005-1011`. + +`MsgpackEncoder` is a msgspec MessagePack encoder +(`vllm/v1/serial_utils.py:136-171`). Its custom hook handles tensors, +ndarrays, slices, multimodal objects, and utility results; an unknown object +raises unless insecure serialization is explicitly enabled, in which case +only that object is placed in a pickle/cloudpickle extension +(`:191-235`). Insecure fallback is off by default +(`vllm/envs.py:197`, `:1508-1509`). `CUDAGraphStat` is a standard dataclass, +which msgspec handles natively: a local typed encode/decode witness saw zero +hook calls with either `None` or a populated stat. Thus the stat neither uses +an extension hook nor forces pickle for the whole `EngineCoreOutputs`. + +As a scale check only (not a GPU gate result), a 500,000-iteration synthetic +typed encode+decode loop measured 1.5340 us/message without the nested stat +and 1.7215 us/message with it, a 0.1875 us/message delta; encoded size grew +from 358 to 451 bytes. At the observed roughly 9.7 steps/s this is far below +the valid attempt-2 throughput gap. This witness checks serializer mechanism, +not end-to-end performance. + +On the frontend, `AsyncLLM` hands `scheduler_stats` to the output processor +and logger manager (`vllm/v1/engine/async_llm.py:656-702`). Beyond OpProf, +the CUDA-graph-specific consumer exists only when the upstream CLI/config +flag is true: `LoggingStatLogger` creates `CUDAGraphLogging` +(`vllm/v1/metrics/loggers.py:99-123`), appends one dataclass per step +(`:163-190`), and at each normal logging interval builds a `Counter`, sorts +rows, formats a table, logs it, and resets the list +(`vllm/compilation/cuda_graph.py:65-124` and +`vllm/v1/metrics/loggers.py:219-283`). With only `VLLM_OPPROF_DIR` set and +`cudagraph_metrics=False`, this logger is absent; the stat is still encoded to +the frontend but has no feature-specific frontend consumer. + +Static mechanism conclusion: serializer fast-path poisoning is ruled out for +the measured TP1 worker path and was not observed on the remaining typed +msgspec leg. The common hot-path delta to test is the upstream four-field +stat production and propagation; the upstream-only versus env-set/recorder-off +trials distinguish upstream feature cost from OpProf-specific enablement. + +Static-witness sanity: encode/decode configurations n=2, min/max +1.5340/1.7215 us, distinct=2; message sizes n=2, min/max 358/451 bytes, +distinct=2. All timings and sizes were positive, the typed round trip +preserved values, and hook-call count was exactly zero in both configurations. + +### Three-arm confirmation reported before fixing + +These three fresh-server trials reused the frozen A-P2-1 stream: 128 excluded +warm-up requests, then 4,000 measured requests at concurrency 200, random +input lengths 256-768 from seed 20260711, greedy temperature 0, `ignore_eos`, +and exactly 64 output tokens. All arms completed 4,000/4,000 requests with +zero failures, 2,031,054 input tokens, and 256,000 output tokens. + +| Trial | Configuration | Throughput (req/s) | Duration (s) | Startup (s) | Server lifetime (s) | Difference vs baseline | +|---:|---|---:|---:|---:|---:|---:| +| 1 | True baseline: env unset, CUDA-graph metrics off | 29.716821 | 134.603900 | 74 | 258 | 0 | +| 2 | Upstream only: env unset, `--cudagraph-metrics` | 29.641918 | 134.944034 | 64 | 248 | 0.2521% | +| 3 | `VLLM_OPPROF_DIR` set, temporary recorder `off` | 28.452675 | 140.584322 | 122 | 307 | 4.2540% | + +The env-set arm was also 4.0120% below upstream-only. Input-length and +output-length digests were identical across all three arms. Generated-text +digests differed, as already seen in the longer A-P2-1 series; exact token +volume remained fixed. GPU0 was stably 0 MiB/0% before and after each arm, +and all three selected the TRITON unquantized MoE backend. + +The table rules out the proposed upstream-stat/serializer mechanism: upstream +metrics were only 0.2521% below baseline, while the OpProf environment alone +reproduced the material gap with no recorder. + +### Exact operation: compile-cache key poisoning by the output path + +`envs.compile_factors()` starts from every registered vLLM environment +variable and removes only names in its explicit `ignored_factors` set +(`vllm/envs.py:2011-2017`, `:2089-2104`). The accepted patch registered +`VLLM_OPPROF_DIR` but did not add it to that exclusion set. `VllmBackend` +hashes those environment factors and combines the environment, vLLM config, +traced-code, and compiler hashes into the torch.compile cache-directory key +(`vllm/compilation/backends.py:1024-1065`). + +The persisted `cache_key_factors.json` files make the mechanism exact: + +- baseline/upstream used cache directory `2f59bf1436`; env-set/recorder-off + used `c0e8930879`; +- config hash `05ca9c77d6`, code hash + `7d1415ae9c2a984358bee50a7ae3079b0e9f0afd27ecdff56e25f9ef57e89c15`, + and compiler hash `1efad2be6d` were identical; +- the sorted environment maps differed in exactly one entry: + `VLLM_OPPROF_DIR=""` versus the per-run output directory. + +Consequently every ON trial's unique artifact directory generated a unique +compile cache key. The baseline/upstream server loaded the existing graph +and AOT artifact (`torch.compile` 6.07 seconds); env-set/recorder-off compiled +and saved a new graph/AOT artifact (`torch.compile` 36.41 seconds). Its total +engine initialization was 70.22 seconds versus 21.85 seconds for baseline, +matching the startup signature seen throughout attempts 2 and 3. This cache +churn also placed each ON arm on a newly compiled/autotuned artifact instead +of the common warmed artifact, which is the systematic measurement confound. + +Fix hypothesis, frozen before implementation: `VLLM_OPPROF_DIR` controls only +the host-side telemetry destination and cannot affect the model computation +graph, so excluding it from `compile_factors()` will make ON and OFF share the +same compilation/AOT cache. Verification requires the same cache directory +in a quick ON/OFF pair and quick-pair overhead below 2% before the full gate. + +### Minimal fix, export, and verification + +The production change adds exactly one string, `"VLLM_OPPROF_DIR"`, to +`compile_factors()`'s `ignored_factors` set in `vllm/envs.py:2011-2093` +(new entry at `:2047`). It +does not touch CUDA-graph dispatch, scheduler plumbing, serialization, the +recorder, or the import surface. The regression test loads `envs.py` directly +with light module stubs, sets the output path, and asserts that the name is not +returned as a compile factor. + +The two fix commits are local `335da4a`/`bbfa717` and patch-ID-equivalent +dash0 commits `fbbc886`/`9ad32ff`. Relative to accepted tip `668cfb7e`, the +fix delta is **+1/-0 production line** and **+21/-1 test lines**, or +22/-1 +overall. The fifth commit exists only to keep the new regression import-light; +it does not alter production code. + +Verification summary: + +- local standalone suite: 15/15 passed in 0.02 seconds; +- local Ruff check: passed; +- dash0 real-tip suite: 15/15 passed in 0.18 seconds; +- a fresh worktree at exact base `ee0da84a`: all five exported patches applied, + 15/15 passed in 0.07 seconds, and a second `apply.sh` invocation was a no-op; +- the final real branch and clean-base worktree both retained the import-light + contract (no installed vLLM or torch required by the local test run). + +An earlier four-patch clean-base validation exposed that the first version of +the regression test imported torch before the existing import-light assertion. +That test-isolation bug did not affect production code or GPU measurements; it +was corrected by the fifth commit before the final clean-base result above. + +### Quick signal pair after the fix + +One unmeasured 128-request burn-in preceded a fixed-work ON/OFF pair. Both +measured servers used compile cache `f66a11246c`. + +| Arm | Throughput (req/s) | Duration (s) | Startup / lifetime (s) | +|---|---:|---:|---:| +| ON | 29.700657 | 134.677153 | 72 / 256 | +| OFF | 29.677979 | 134.780065 | 63 / 247 | + +Quick-pair overhead was -0.07641%, below the pre-declared 2% continuation +threshold, so the full unchanged protocol was run. + +### Attempt 4 full unchanged A-P2-1 gate + +The burn-in server lived for 94 seconds. The ten measured fresh servers then +used the frozen ON,OFF,OFF,ON,ON,OFF,OFF,ON,ON,OFF order, 128 excluded warm-up +requests, and the same 4,000-request fixed stream. Every run completed 4,000 +requests with zero failures, 2,031,054 input tokens, and 256,000 output tokens. +Every completion was exactly 64 tokens. All eleven servers, including burn-in, +used compile cache `f66a11246c`. + +| Trial | Pair | Arm | Throughput (req/s) | Duration (s) | Startup / lifetime (s) | Host load 1m before / after | +|---:|---:|---|---:|---:|---:|---:| +| 1 | 1 | ON | 29.712835 | 134.621958 | 60 / 244 | 0.49 / 1.48 | +| 2 | 1 | OFF | 29.624180 | 135.024835 | 64 / 248 | 1.26 / 1.78 | +| 3 | 2 | OFF | 29.658364 | 134.869208 | 64 / 249 | 1.51 / 1.55 | +| 4 | 2 | ON | 29.647687 | 134.917776 | 63 / 247 | 1.43 / 1.91 | +| 5 | 3 | ON | 29.637994 | 134.961900 | 76 / 262 | 1.62 / 2.36 | +| 6 | 3 | OFF | 29.634841 | 134.976258 | 62 / 247 | 1.90 / 1.75 | +| 7 | 4 | OFF | 29.677443 | 134.782500 | 63 / 247 | 1.56 / 1.42 | +| 8 | 4 | ON | 29.657455 | 134.873342 | 62 / 246 | 1.20 / 1.63 | +| 9 | 5 | ON | 29.676533 | 134.786632 | 61 / 246 | 1.38 / 1.20 | +| 10 | 5 | OFF | 29.675455 | 134.791530 | 64 / 249 | 1.09 / 1.49 | + +| Pair | ON (req/s) | OFF (req/s) | `1 - ON/OFF` | +|---:|---:|---:|---:| +| 1 | 29.712835 | 29.624180 | -0.29927% | +| 2 | 29.647687 | 29.658364 | 0.03600% | +| 3 | 29.637994 | 29.634841 | -0.01064% | +| 4 | 29.657455 | 29.677443 | 0.06735% | +| 5 | 29.676533 | 29.675455 | -0.00363% | + +Mean ON was **29.666501 req/s**, mean OFF was **29.654057 req/s**, and the +pre-registered aggregate overhead `1 - mean(ON)/mean(OFF)` was **-0.04196%**. +The paired percentile bootstrap (100,000 resamples, seed 20260711) gave a 95% +CI of **[-0.17443%, 0.04550%]**. Its 0.04550% upper bound is below 3%: +**overhead hard gate PASS**. + +All pre-launch clock snapshots reported current SM 345 MHz; all immediate +post-run snapshots and active samples reported 1,980 MHz. Maximum SM remained +1,980 MHz and memory 2,619 MHz throughout. Before/after compute-process +snapshots were empty, periodic GPU +samples contained only the campaign EngineCore, and no other user's GPU process +appeared. One-minute host load stayed between 0.23 and 2.36 on 160 logical CPUs. +Input- and output-length digests each had one value across all ten runs; +generated-text digests had six values, but exact work volume was invariant. + +## Layer-2 window + +The accepted retry used vLLM's profile endpoint during the same fixed 4,000 +request load. `VLLM_TORCH_PROFILER_DIR` pointed to dash0-local `/tmp` during +capture, with `VLLM_TORCH_PROFILER_WITH_STACK=1`, +`VLLM_TORCH_PROFILER_RECORD_SHAPES=1`, and the v0.24.0 schedule configured for +wait 0, warm-up 2, active 8, repeat 1. The resulting files were copied into the +persistent run directory before teardown. + +The trace +`dp0_pp0_tp0_dcp0_ep0_rank0.1783785319893394827.pt.trace.json.gz` is +16,416,452 bytes with SHA-256 +`4d6bb869ff03ddfd421d3111414ec9532330a9cbefdfa9e9fed1982070f6d72e`. +It decompressed and parsed as JSON schema version 1, containing 790,843 trace +events, 7,367 kernel events, and 35 distinct kernel names. The trace contains +exactly eight active `ProfilerStep` annotations numbered 2 through 9 and eight +matching execute annotations; the two preceding iterations were warm-up and +were correctly discarded. Visible kernels include the Triton fused-MoE kernel, +FlashAttention, reshape-and-cache, top-k gating, and NVJet GEMMs. + +All eight active execute annotations totaled approximately 8,192 scheduled +tokens (context 8,025-8,048 plus generation 144-167). They exceeded the +largest captured CUDA-graph bucket, so OpProf classified all ten records in +the scheduled collection interval as `NONE`. This explains why the trace +shows 7,367 individual kernel launches and zero `cudaGraphLaunch` runtime +events. The immediately preceding equal-length steady window contained 75 +`FULL` and 20 `NONE` records; the broader pre-endpoint interval contained 215 +`FULL`, 54 `NONE`, and two `PIECEWISE` records. The trace therefore proves +kernel visibility for the uncaptured mode actually active in the profiler +window; it does **not** claim individual-kernel visibility inside a FULL or +PIECEWISE graph replay. + +Perturbation is indicative only. During the 8.174811-second scheduled +collection interval, 131 requests completed (16.0248 req/s) versus 269 in the +equal preceding interval (32.9060 req/s), a 51.30% reduction. The complete +24.102657-second start/stop endpoint interval sustained 5.4351 req/s versus +19.4584 immediately before and 29.8722 immediately after, reductions of +72.07% and 81.81%; trace finalization inside `stop_profile` dominates that +larger interval. Whole-load throughput was 25.869730 req/s, 12.80% below the +fixed-gate ON mean. These values are profiler perturbation, not Layer-1 gate +inputs. + +The first Layer-2 controller attempt is excluded: its benchmark-progress +marker remained block-buffered, so profiling started after the load had +finished and produced no trace. No inference is drawn from that run. The +accepted retry used an unbuffered progress source and triggered during load. + +**Layer-2 gate PASS:** the configured 2+8 window executed, its Kineto file is +loadable and kernel-bearing, observed CUDA-graph modes are documented, and +perturbation was measured. + +## Deviations and failure notes + +1. Two direct remote GitHub clones were interrupted by checkout/fetch transport + behavior. `rsync` was unavailable on dash0. The accepted local clean clone + was transferred as a gzip tar stream; `apply.sh` then verified the exact + patch IDs and required base parent. +2. A CPFS-hosted venv spent 21 minutes in slow package metadata/page writes and + was stopped before completion. It was preserved as failure evidence. The + successful fresh venv was installed on dash0 local rootfs to avoid CPFS I/O + variance in repeated startups and OpProf writes. +3. `/usr/bin/time` was absent, so install duration was recorded with epoch + timestamps. +4. The initial binary-import check incorrectly tried `vllm._C`. vLLM 0.24.0's + CUDA path uses `_C_stable_libtorch` and `_moe_C_stable_libtorch`; both actual + modules imported successfully before testing or serving. +5. Parent-first shutdown produced no footer. EngineCore-first teardown produced + a footer but causes the API parent to log an expected teardown-only + `EngineDeadError` after all requests and metrics are complete. +6. After overhead trial 6, the first cleanup sample showed 4 MiB and 0% + utilization with no compute process. The controller stopped before trial 7. + GPU memory was 0 MiB on two later checks; trials 7-10 resumed unchanged, and + cleanup polling was extended to wait for driver accounting to return to zero. +7. The footer-valid smoke did not contain PIECEWISE. The invalid first file did, + but it was not promoted into accepted gate evidence. +8. The fixed-seed commands did not yield bit-identical generated outputs, and + the bundled client generated a random request-ID prefix for each invocation. + A-P2-1 fixed the request-ID prefix and exact output length; content digests + still varied, but work volume did not. +9. A-P2-1's first unbounded 4,000-request admission pilot exceeded the client's + file-descriptor limit. It was excluded before any CI calculation. Maximum + concurrency 200 was documented before the accepted ten-run series began. +10. After accepted trial 1, a 4 MiB/0% transient appeared after an earlier 0 MiB + cleanup sample. Trial 2 had not launched. Five subsequent 0 MiB/0% samples + were required before resuming the frozen sequence at trial 2. +11. The process sampler recorded 39 transient root-owned processes, including + orchestrator checks and cluster control processes. None used a GPU, host + load remained low, and no process was touched. +12. A-P2-1 fixed completion work but still produced seven generated-text + digests. Because all 40,000 accepted completions were exactly 64 tokens, + this did not recreate attempt 1's work-volume confound. +13. Two malformed temporary stage-diff attempts failed before serving. The + first failed `git apply --check`; the second was immediately reversed after + py_compile exposed incorrect zero-context placement. The final logged diff + passed py_compile, stage semantic checks, and 14 tests before GPU use. +14. Attempt 3 showed that stage `off` could not disable the model-runner's + environment-controlled `CUDAGraphStat` path. The outside-OpProf stop rule + correctly prevented a speculative change in that turn; attempt 4 resumed + only after explicit authorization to trace that path. +15. The first attempt-4 confirmation controller incorrectly applied an ON-file + acceptance check to the true-baseline arm and stopped after that valid + trial. A first resume wrapper then failed a shell declaration before any GPU + launch. The remaining two arms resumed in order with unchanged commands; + neither control-plane error changed a measured result. +16. The first clean-base four-patch test exposed a test-only import-light + regression: the new test loaded real torch before the existing assertion. + Production code was unaffected. The fifth patch stubbed torch, and the + final five-patch clean-base validation passed 15/15. +17. **Footer anomaly:** the attempt-4 quick/full-gate and accepted Layer-2 + controllers terminated the API and EngineCore process group simultaneously + after metrics were complete. Their record lines are contiguous and + decodable, but the five full-gate ON files and accepted Layer-2 file have no + footer. They are therefore not promoted as artifact-gate evidence. The + artifact hard gate remains based on the earlier footer-valid streams, and + the one-line cache-factor fix cannot alter recorder shutdown behavior. +18. The first Layer-2 controller used a block-buffered benchmark marker, so the + endpoint fired after load completion and produced no trace. That run is + explicitly excluded; the unbuffered retry produced the accepted trace. + +## Artifacts and reproducibility record + +- Persistent campaign workdir: + `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711` +- Runtime artifact archive: + `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/phase2-runtime-artifacts.tar.gz` +- Archive SHA-256: + `040eb234818564f880489fca4a1a8b21278210dcd04b9f613e6000a28fbb3beb` +- Archive inventory: 203 files, 7 JSONL files, 12 serving result JSON files; + compressed size 1.6 MiB, uncompressed run tree 6.4 MiB. +- Remote analysis summaries are included at + `runs/artifact-smoke-rerun/artifact-gate.txt` and + `runs/overhead/overhead-analysis.json` inside the archive. + +The amended evidence is separately preserved so attempt 1 remains immutable: + +- A-P2-1 archive: + `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/amendment-a-p2-1-runtime-artifacts.tar.gz` +- A-P2-1 archive SHA-256: + `acbf5eb2f495e322ecb087f8238540806bfa1c412cad3be0d8265f17c2e938cb` +- Inventory: 344 files; 24 MiB compressed and 120 MiB uncompressed. +- Primary analysis: + `runs/amendment-a-p2-1/overhead-attempt2-accepted/overhead-analysis.json` +- Microbenchmark: + `runs/amendment-a-p2-1/microbenchmark/result.json` +- PIECEWISE validation: + `runs/amendment-a-p2-1/piecewise-addendum/artifact-validation.json` + +Attempt 3 is separately archived: + +- Stage-bisection archive: + `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/attempt3-stage-bisection-artifacts.tar.gz` +- Archive SHA-256: + `e2ee764b905a19162f41fbe153283de3b133024c5548813749837c89779dbe58` +- Inventory: 82 files; 12 MiB compressed and 50 MiB uncompressed. +- Bisection analysis: + `runs/stage-bisection-attempt3/bisection-analysis.json` + +Attempt 4, including the accepted Kineto trace, is separately archived: + +- Fix/confirmation/gate/Layer-2 archive: + `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/attempt4-fix-and-gate-artifacts.tar.gz` +- Archive SHA-256: + `55d227d644d3d42c2a68edc19fcb2d47231f6971aa74dca9d2a9ef44de2cefa2` +- Inventory: 536 entries; 56,705,396 compressed bytes. +- Three-arm raw table: + `runs/attempt4-stat-path-confirmation/results.tsv` +- Full gate analysis: + `runs/attempt4-fix-full-gate/overhead-analysis.json` +- Layer-2 analysis and trace: + `runs/attempt4-layer2-window-accepted/layer2-analysis.json` and + `runs/attempt4-layer2-window-accepted/traces/`. + +## Final required summary + +- **Artifacts hard gate: PASS.** Original composition/accounting checks passed; + the addendum supplied 129 footer-valid PIECEWISE records with exact + mode/bucket/padding semantics. +- **Overhead CI hard gate: PASS.** Fixed ON 29.666501 req/s, OFF 29.654057 + req/s, overhead -0.04196%, paired-bootstrap 95% CI + [-0.17443%, 0.04550%]. The unchanged limit was 3%. +- **Attempt history:** attempt 1's 13.1133% / [6.2434%, 19.5259%] is retained + as invalidated; attempt 2's 4.1816% / [3.1364%, 4.7117%] is retained as the + valid pre-fix FAIL; attempt 3 was diagnostic; attempt 4 is the final PASS. +- **Attempt-3 bisection:** off 28.389494, capture 28.257144, encode/null-sink + 28.221609, and full/CPFS 28.269199 req/s; incremental overheads 0%, 0.466%, + 0.591%, and 0.424%. +- **Exact mechanism:** no TP1 worker serializer exists; the remaining msgspec + leg natively encodes `CUDAGraphStat` with zero hook/pickle calls. Instead, + `envs.compile_factors()` (`vllm/envs.py:2011-2017,2089-2104` in the pinned + pre-fix tree) included `VLLM_OPPROF_DIR`, and `VllmBackend` folded that map + into the graph/AOT cache key (`vllm/compilation/backends.py:1024-1065`). A + unique output directory therefore meant a unique, cold cache. +- **Fix / line delta:** ignore `VLLM_OPPROF_DIR` in compile factors; +1/-0 + production line, +21/-1 test lines. Final local tip `bbfa717`, dash0 + patch-ID-equivalent tip `9ad32ff`; five exported patches. +- **Unit tests:** 15/15 local (0.02 s), 15/15 dash0 real tip (0.18 s), and 15/15 + dash0 clean-base exported-series application (0.07 s); Ruff passed and the + import-light contract is preserved. +- **Layer-2 hard gate: PASS.** Loadable 16,416,452-byte Kineto trace; exactly + 2 warm-up + 8 active steps, 790,843 events, 7,367 kernels, and 35 distinct + kernel names. Active mode was `NONE` because each profiled batch was about + 8,192 tokens; the coverage caveat is explicit. +- **Recorder microbenchmark:** 29.1467 us/step by whole-loop timing; 27.7620 us + median and 36.6643 us p95 over 10,000 iterations. +- **PIECEWISE verification:** PASS; 129/139 records PIECEWISE, buckets 1-248, + padding 0-7, footer encoded=written=139 and dropped=0. +- **MoE backend observed:** TRITON unquantized in 48/48 campaign server logs. +- **Weights path:** `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`. +- **Total GPU time consumed:** attempt 4 added 4,629 launch-to-clean seconds + (77.15 H20-minutes): confirmation 813, quick signal 648, full gate 2,579, + excluded Layer-2 attempt 329, accepted Layer-2 260. Whole campaign total is + conservatively **11,018.58 seconds = 183.64 H20-minutes = 3.061 H20-hours**. +- **Cleanup:** all eight H20s ended at 0 MiB/0%, with no campaign process. + +## Data sanity block + +Attempt 4 is the final gate conclusion. Attempts 1-3 remain listed for +auditability and are not pooled into its estimate. + +### Attempt 1 (invalidated measurement) + +| Numeric family | n | Min | Max | Distinct | Sanity note | +|---|---:|---:|---:|---:|---| +| Initial dash0 GPU memory used (MiB) | 8 | 0 | 0 | 1 | All eight H20s initially idle | +| Initial dash0 GPU utilization (%) | 8 | 0 | 0 | 1 | All eight H20s initially idle | +| Selected-GPU overhead precheck memory (MiB) | 10 | 0 | 0 | 1 | Required before every trial | +| Selected-GPU overhead precheck utilization (%) | 10 | 0 | 0 | 1 | Required before every trial | +| Selected-GPU immediate post-cleanup memory (MiB) | 10 | 0 | 4 | 2 | Trial 6 anomaly stopped the controller; next two checks were 0 | +| Model shard bytes | 16 | 1,087,928,584 | 3,999,975,472 | 11 | Sum 61,066,575,648; no shard missing | +| Unit-test cases | 14 | N/A | N/A | 14 names | 14 passed, 0 failed | +| Footer-valid artifact step index | 109 | 0 | 108 | 109 | File order contiguous and monotonic | +| Artifact scheduled tokens/step | 109 | 0 | 8,192 | 73 | Non-negative | +| Artifact prefill tokens/step | 109 | 0 | 8,188 | 15 | Non-negative | +| Artifact decode tokens/step | 109 | 0 | 200 | 80 | Non-negative | +| Artifact scheduled requests/step | 109 | 0 | 200 | 77 | Non-negative | +| Artifact KV usage ratio | 109 | 0 | 0.370965 | 106 | Every ratio in [0,1] | +| Artifact CUDA-graph bucket | 109 | 0 | 8,192 | 32 | Bucket >= unpadded for every record | +| Artifact CUDA-graph padding | 109 | 0 | 7 | 8 | Exact bucket-minus-unpadded identity | +| Smoke input lengths | 200 | 261 | 761 | 172 | Sum 102,848 | +| Smoke output lengths | 200 | 32 | 96 | 59 | Sum 12,187 | +| ON throughput (req/s) | 5 | 19.877357 | 22.685507 | 5 | Not all identical | +| OFF throughput (req/s) | 5 | 20.378458 | 26.350950 | 5 | Not all identical | +| All throughput (req/s) | 10 | 19.877357 | 26.350950 | 10 | Positive; high run-level spread | +| Measured benchmark duration (s) | 10 | 7.589859 | 10.061700 | 10 | Positive | +| Adjacent-pair overhead fraction | 5 | -0.002012 | 0.236024 | 5 | One negative pair, four positive pairs | +| Bootstrap overhead fraction | 100,000 | -0.002012 | 0.236024 | 232 | Fixed paired resampling; no filtering | +| ON JSONL record count | 5 | 183 | 186 | 3 | Footer-written count matched records; drops all zero | +| ON JSONL bytes | 5 | 165,982 | 168,696 | 5 | Non-negative, non-identical | +| Overhead server GPU lifetime (s) | 10 | 91 | 155 | 9 | Sum 1,260 seconds | +| All server GPU lifetime (s) | 12 | 91 | 414 | 11 | Sum 1,863 seconds | +| Overhead startup time (s) | 10 | 66 | 126 | 7 | Excluded from throughput timing | +| Client wall time including warm-up (s) | 10 | 21 | 25 | 5 | Main metric used internal measured duration | +| Completed requests | 10 | 200 | 200 | 1 | Expected identical fixed workload | +| Failed requests | 10 | 0 | 0 | 1 | Expected invariant | +| Input tokens | 10 | 102,848 | 102,848 | 1 | Expected identical fixed workload | +| Output tokens | 10 | 12,187 | 12,187 | 1 | Expected identical fixed shapes | +| Exact generated responses vs trial 1 | 10 | 185 | 200 | 7 | **Anomaly:** generated-text digests distinct in all ten trials | + +### Attempt 2 A-P2-1 valid pre-fix measurement + +| Numeric family | n | Min | Max | Distinct | Sanity note | +|---|---:|---:|---:|---:|---| +| Accepted pre-launch GPU memory (MiB) | 10 | 0 | 0 | 1 | Every accepted server launched only after stable zero | +| Accepted pre-launch GPU utilization (%) | 10 | 0 | 0 | 1 | No selected-GPU process present | +| PIECEWISE addendum step index | 139 | 0 | 138 | 139 | Contiguous and monotonic | +| PIECEWISE runtime records | 129 | 1 unpadded token | 241 unpadded tokens | 9 token counts | Every runtime mode was PIECEWISE | +| PIECEWISE bucket tokens | 129 | 1 | 248 | 6 | Bucket >= unpadded | +| PIECEWISE padding tokens | 129 | 0 | 7 | 5 | Exact bucket-minus-unpadded identity | +| Microbenchmark per-iteration cost (us) | 10,000 | 26.293 | 70.111 | 4,144 ns values | Mean 28.9279, median 27.7620, p95 36.6643 | +| Microbenchmark empty-timer cost (us) | 10,000 | 0.082 | 12.012 | 84 ns values | Harness cost small relative to producer path | +| Accepted ON throughput (req/s) | 5 | 28.183682 | 28.256715 | 5 | Range 0.259% of minimum; not identical | +| Accepted OFF throughput (req/s) | 5 | 28.800396 | 29.640096 | 5 | First OFF lower than later OFF runs | +| All accepted throughput (req/s) | 10 | 28.183682 | 29.640096 | 10 | Positive and non-identical | +| Accepted measured duration (s) | 10 | 134.952330 | 141.926096 | 10 | Every run >=120 seconds | +| Accepted pair overhead fraction | 5 | 0.020746 | 0.047369 | 5 | All five pairs positive | +| Accepted bootstrap overhead fraction | 100,000 | 0.020746 | 0.047369 | 263 | CI [0.031364, 0.047117] | +| Completed requests/run | 10 | 4,000 | 4,000 | 1 | Expected fixed workload | +| Failed requests/run | 10 | 0 | 0 | 1 | Required invariant | +| Input tokens/run | 10 | 2,031,054 | 2,031,054 | 1 | Input-length digest distinct=1 | +| Output tokens/run | 10 | 256,000 | 256,000 | 1 | Output-length digest distinct=1 | +| Input lengths in one fixed stream | 4,000 | 256 | 768 | 512 | Same array in all ten runs | +| Output lengths in one fixed stream | 4,000 | 64 | 64 | 1 | `ignore_eos` fixed all work | +| Generated-text digests | 10 | N/A | N/A | 7 | Content varied; token volume did not | +| Accepted ON JSONL records | 5 | 1,403 | 1,406 | 3 | Footers matched; all drops zero | +| Accepted ON JSONL bytes | 5 | 1,283,109 | 1,286,129 | 5 | Mean 914.7 bytes/record | +| Accepted ON active stream span (s) | 5 | 144.726692 | 144.968812 | 5 | Continuous submit-to-complete spans | +| Accepted ON step rate (steps/s) | 5 | 9.681545 | 9.707954 | 5 | Mean 9.696167 | +| Scheduled requests/telemetry record | 7,022 | 0 | 200 | 127 | Non-negative; configured cap 200 | +| Periodic GPU samples | 545 | P0 | P0 | 1 pstate | No pstate transition | +| Active GPU SM clock (MHz) | 318 | 1,740 | 1,980 | 12 | Max-clock family distinct=1 at 1,980 MHz | +| Active GPU power (W) | 318 | 118.46 | 501.12 | 311 | Non-negative | +| One-minute load average | 20 | 0.15 | 1.63 | 19 | Low relative to 160 logical CPUs | +| Newly observed other-user processes | 39 | N/A | N/A | 11 command names | All root-owned; explicitly reported | +| Other-user GPU samples/run | 10 | 0 | 0 | 1 | No GPU contamination | +| Accepted startup time (s) | 10 | 64 | 116 | 8 | ON 111-116; OFF 64-73; excluded from metric | +| Accepted server allocation (s) | 10 | 258 | 312 | 10 | Sum 2,858 seconds | +| Amended server allocation (s) | 13 | 102.54 | 312 | 13 | Burn-in, PIECEWISE, excluded pilot, accepted series; sum 3,303.58 seconds | +| Campaign allocation through attempt 2 (s) | 25 | 91 | 414 | 24 | Sum 5,166.58 seconds = 86.11 H20-minutes | + +### Attempt 3 diagnostic bisection + +| Numeric family | n | Min | Max | Distinct | Sanity note | +|---|---:|---:|---:|---:|---| +| Stage pre-launch GPU memory (MiB) | 4 | 0 | 0 | 1 | Stable zero before each server | +| Stage pre-launch GPU utilization (%) | 4 | 0 | 0 | 1 | No selected-GPU process | +| Stage throughput (req/s) | 4 | 28.221609 | 28.389494 | 4 | Total spread 0.595% of minimum | +| Stage measured duration (s) | 4 | 147.942050 | 148.822131 | 4 | Every stage approximately 140 seconds or longer | +| Stage overhead vs `off` fraction | 4 | 0 | 0.005914 | 4 | No stage approached 0.041816 | +| Completed requests/stage | 4 | 4,200 | 4,200 | 1 | Fixed work | +| Failed requests/stage | 4 | 0 | 0 | 1 | Required invariant | +| Input tokens/stage | 4 | 2,132,262 | 2,132,262 | 1 | Input digest distinct=1 | +| Output tokens/stage | 4 | 268,800 | 268,800 | 1 | Output digest distinct=1; every output 64 tokens | +| Stage startup time (s) | 4 | 111 | 126 | 4 | Excluded from measurement | +| Stage server allocation (s) | 4 | 301 | 316 | 4 | Sum 1,223 seconds | +| Stage JSONL file count | 4 | 0 | 1 | 2 | Off/encode 0; capture/full 1 | +| Stage JSONL bytes | 4 | 0 | 1,346,466 | 3 | Capture footer 96 bytes; full CPFS 1,346,466 | +| Temporary-diff unit tests | 14 | pass | pass | 14 names | 14/14 remote before trials | +| Restored-branch unit tests | 14 | pass | pass | 14 names | 14/14 local after reversal | +| Whole-campaign server allocation (s) | 29 | 91 | 414 | 28 | Sum 6,389.58 seconds = 106.49 H20-minutes | + +### Attempt 4 confirmation, fix, and final gate + +| Numeric family | n | Min | Max | Distinct | Sanity note | +|---|---:|---:|---:|---:|---| +| Static typed msgspec round-trip time (us) | 2 | 1.5340 | 1.7215 | 2 | Populated-stat delta 0.1875 us; hook calls 0/0 | +| Static typed MessagePack bytes | 2 | 358 | 451 | 2 | Positive; populated stat is larger as expected | +| Confirmation throughput (req/s) | 3 | 28.452675 | 29.716821 | 3 | Baseline/upstream close; env-set arm isolated | +| Confirmation measured duration (s) | 3 | 134.603900 | 140.584322 | 3 | Every arm >=120 seconds | +| Confirmation difference vs baseline | 3 | 0 | 0.042540 | 3 | Upstream 0.002521; env-set 0.042540 | +| Confirmation compile-cache key | 3 | N/A | N/A | 2 | Baseline/upstream same; env-set alone differed | +| Compared torch.compile time (s) | 2 | 6.07 | 36.41 | 2 | Cache load versus compile/save | +| Compared engine initialization (s) | 2 | 21.85 | 70.22 | 2 | Same cache-key split as compile time | +| Confirmation server allocation (s) | 3 | 248 | 307 | 3 | Sum 813 seconds | +| Quick-pair throughput (req/s) | 2 | 29.677979 | 29.700657 | 2 | Positive, not identical | +| Quick-pair overhead fraction | 1 | -0.000764 | -0.000764 | 1 | Negative means ON was 0.0764% faster; <2% continuation threshold | +| Fixed-gate pre-launch GPU memory (MiB) | 11 | 0 | 0 | 1 | Burn-in plus ten trials all launched from zero | +| Fixed-gate pre-launch GPU utilization (%) | 11 | 0 | 0 | 1 | No selected-GPU process present | +| Fixed-gate ON throughput (req/s) | 5 | 29.637994 | 29.712835 | 5 | Positive and not all identical | +| Fixed-gate OFF throughput (req/s) | 5 | 29.624180 | 29.677443 | 5 | Positive and not all identical | +| All fixed-gate throughput (req/s) | 10 | 29.624180 | 29.712835 | 10 | Total spread 0.299% of minimum | +| Fixed-gate measured duration (s) | 10 | 134.621958 | 135.024835 | 10 | Every run >=120 seconds | +| Fixed-gate pair overhead fraction | 5 | -0.002993 | 0.000674 | 5 | Three ON-faster and two OFF-faster pairs | +| Fixed-gate bootstrap overhead fraction | 100,000 | -0.002993 | 0.000674 | 224 | CI [-0.001744, 0.000455], no filtering | +| Completed requests/run | 10 | 4,000 | 4,000 | 1 | Fixed work invariant | +| Failed requests/run | 10 | 0 | 0 | 1 | Required invariant | +| Input tokens/run | 10 | 2,031,054 | 2,031,054 | 1 | Input-length digest distinct=1 | +| Output tokens/run | 10 | 256,000 | 256,000 | 1 | Output-length digest distinct=1; every completion 64 | +| Generated-text digests | 10 | N/A | N/A | 6 | Content varied; exact work did not | +| Fixed-gate startup time (s) | 10 | 60 | 76 | 6 | Excluded from throughput metric | +| Fixed-gate measured server lifetime (s) | 10 | 244 | 262 | 6 | Sum 2,485 seconds | +| Full-protocol server allocation (s) | 11 | 94 | 262 | 7 | Includes burn-in; sum 2,579 seconds | +| One-minute host-load snapshots | 22 | 0.23 | 2.36 | 20 | Low relative to 160 logical CPUs | +| Current-SM clock snapshots (MHz) | 22 | 345 | 1,980 | 2 | Eleven pre-launch 345; eleven immediate post-run 1,980; max 1,980 throughout | +| Idle memory-clock snapshots (MHz) | 22 | 2,619 | 2,619 | 1 | Stable before/after | +| Other-user GPU process count/run | 11 | 0 | 0 | 1 | Periodic samples contained only campaign processes | +| Fixed-gate ON JSONL records | 5 | 1,406 | 1,412 | 4 | Contiguous step indices and msgspec-decodable | +| Fixed-gate ON JSONL bytes | 5 | 1,285,722 | 1,291,357 | 5 | Positive and not identical | +| Fixed-gate ON footer count | 5 | 0 | 0 | 1 | **Anomaly:** simultaneous teardown; excluded from artifact accounting | +| Final unit-test cases | 15 | pass | pass | 15 names | Local, real-tip remote, and clean-base remote all passed | + +### Layer-2 accepted window + +| Numeric family | n | Min | Max | Distinct | Sanity note | +|---|---:|---:|---:|---:|---| +| Kineto trace files | 1 | 16,416,452 bytes | 16,416,452 bytes | 1 | gzip and JSON loadable; SHA-256 recorded | +| Trace category counts | 13 | 1 | 713,524 | 13 | Sum 790,843 total events | +| Kernel event count | 1 | 7,367 | 7,367 | 1 | 35 distinct kernel names | +| Active profiler-step number | 8 | 2 | 9 | 8 | Contiguous; configured active count exactly met | +| Active execute context tokens | 8 | 8,025 | 8,048 | 5 | Non-negative | +| Active execute generation tokens | 8 | 144 | 167 | 5 | Context + generation exactly 8,192 each | +| Scheduled collection window duration (s) | 1 | 8.174811 | 8.174811 | 1 | Covers the active trace collection | +| Equal-window completed requests | 2 | 131 | 269 | 2 | Profile versus immediately preceding steady window | +| Equal-window request rate (req/s) | 2 | 16.024836 | 32.905961 | 2 | Indicative 51.30% profiler perturbation | +| Endpoint-window request rate (req/s) | 3 | 5.435085 | 29.872225 | 3 | Profile, equal pre, equal post; stop/finalization included | +| Layer-2 whole-load throughput (req/s) | 1 | 25.869730 | 25.869730 | 1 | 4,000 completed, zero failed | +| OpProf collection-window per-mode record counts | 1 | 10 | 10 | 1 | Only observed mode was `NONE` | +| OpProf endpoint-pre per-mode record counts | 3 | 2 | 215 | 3 | FULL 215, NONE 54, PIECEWISE 2; sum 271 | +| Layer-2 OpProf record count | 1 | 1,407 | 1,407 | 1 | Contiguous; used only for mode/timing evidence | +| Layer-2 OpProf footer count | 1 | 0 | 0 | 1 | **Anomaly:** same simultaneous-teardown issue; not artifact-gate evidence | +| Attempt-4 server allocation (s) | 19 | 94 | 329 | 13 | Sum 4,629 seconds = 77.15 H20-minutes | +| Whole-campaign aggregate GPU time (s) | 1 | 11,018.58 | 11,018.58 | 1 | 183.64 H20-minutes; one GPU only | + +Checked invariants across accepted evidence: non-negative counters; KV ratios +in [0,1]; exact KV block accounting and histogram sums; contiguous unique step +indices; footer `encoded=written+dropped` and zero drops in the footer-valid +artifact evidence; graph hit/mode/bucket/padding consistency; footer-valid +PIECEWISE runtime coverage; fixed ABBA order; five adjacent pairs; positive +throughput; 4,000 successes and zero failures per fixed-gate run; durations +>=120 seconds; output lengths exactly 64; input/output arrays identical across +runs; shared post-fix compile-cache key; and per-arm results not all identical. +Negative overhead values are permitted because they denote an ON-faster sample, +not a non-negative counter or probability. Monotonicity/curve continuity is not +applicable to this interleaved repeated-measures experiment. + +Attempt 3 additionally checked identical fixed work, zero failures, durations +>=140 seconds, stable-zero GPU prechecks, clean teardown, and four distinct +positive throughputs. Its key invariant failure was conceptual rather than +numeric: stage `off` returned no recorder but could not disable the external +model-runner flag derived directly from `VLLM_OPPROF_DIR`. + +Red flags and anomalies were reported before conclusions: attempt 1's invalid +noise/cold-start protocol; the excluded file-descriptor pilot; transient 4 MiB +driver accounting; 39 root-process arrivals; varying content digests; the +attempt-2 mode-correlated startup split and valid CI exceeding 3%; attempt 3's +non-equivalent `off` boundary; the initial clean-base test-isolation failure; +the excluded buffered-marker Layer-2 run; and missing footers caused by the +attempt-4 simultaneous process-group teardown. The PIECEWISE coverage red flag +is closed, not suppressed. Attempt-4 files without footers are explicitly +excluded from artifact accounting rather than treated as footer-valid. diff --git a/docs/opprof/phase3-protocol.md b/docs/opprof/phase3-protocol.md new file mode 100644 index 0000000..21eb118 --- /dev/null +++ b/docs/opprof/phase3-protocol.md @@ -0,0 +1,1092 @@ +# OpProf Phase 3 pre-registered pattern-matrix protocol + +Status: **DRAFT FOR ORCHESTRATOR REVIEW — DO NOT EXECUTE**. + +Date frozen: 2026-07-12 (Asia/Singapore). This document defines the Phase 3 +experiment before any Phase 3 GPU launch. It does not authorize a run, change +the accepted five-patch series, or expose the prompt-bearing trace. Any change +to a pattern, load fraction, configuration, profiler placement, metric, +decision threshold, or exclusion rule requires a dated amendment reviewed +before the affected GPU run. + +## Amendments (pre-execution; 2026-07-12) + +All six open decisions in this document are approved as proposed. The two +amendments below are normative and take precedence wherever the original +serial-execution wording or estimate conflicts with them. Neither amendment +changes the pattern matrix, load points, profiler placement, metrics, or hard +validity gates. + +### A-P3-1 — eight-way parallel execution and co-location gate + +The matrix is executed on dash0 with one independent pattern/config cell per +physical GPU, up to eight cells concurrently on GPU0--GPU7. Each GPU has its +own server, client, port, run directory, and `CUDA_VISIBLE_DEVICES` namespace. +No TP1 server spans GPUs. The P10/TP2 counterpoint is scheduled as one cell +using two GPUs and leaves the corresponding second slot empty. + +Eight-way execution is authorized only after this pre-registered co-location +validity check: + +1. Use P05/C00 on physical GPU0 as the target. First calibrate its saturation + throughput and run its moderate load (`0.60*T_sat`) with no other Phase 3 + GPU work on the host. +2. Recalibrate P05/C00 saturation and run the same moderate target on GPU0 + while P01, P02, P03, P04, P06, P07, and P08 under C00 run closed-loop + saturation on GPU1--GPU7, respectively. The seven background loads remain + active through the target's warm-up, clean interval, profile windows, and + recovery intervals. Only the target is profiled. +3. Each saturation calibration uses the same placement regime as the + moderate run it supplies: solo calibration for solo moderate and eight-way + calibration with all seven backgrounds for eight-way moderate. The target + manifest, fixed seed, server configuration, fresh-server timeline, 60-second + warm-up, and 240-second clean interval are otherwise identical. +4. Let `T_solo` and `T_coloc` be target completed-request throughput over the + clean interval. Let `F3` be the three operator families with greatest + aggregate device-duration share in the solo target's accepted Layer-2 + validation window. This one-time placement check samples exactly one 2+8 + window per moderate target; it is not a matrix cell and does not change the + two-window-per-load matrix rule. The check passes only if + `abs(T_coloc/T_solo - 1) < 0.03` and, for every `f` in `F3`, + `abs(share_coloc[f] - share_solo[f]) < 0.03`. The latter thresholds are + absolute share fractions (three percentage points), not relative percent + changes. Both target validation windows (one solo and one co-located) must + satisfy the existing trace and + classifiability gates before this comparison is made. +5. If eight-way fails, repeat the regime-specific saturation and moderate + comparison at four-way placement: P05/C00 on GPU0 plus P01, P03, and P06 + C00 saturation backgrounds on GPU1--GPU3. Four-way is authorized only if + the same throughput and `F3` share criteria pass. Failure at four-way stops + execution for orchestrator review; it does not authorize a smaller regime. + +The CPU map is fixed from dash0's observed topology on 2026-07-12. Server and +client descendants inherit the same `taskset` mask, and masks are disjoint: + +| Physical GPU | NUMA node | Server/client logical CPUs | +|---:|---:|---| +| 0 | 0 | 0-19 | +| 1 | 0 | 20-39 | +| 2 | 0 | 40-59 | +| 3 | 0 | 60-79 | +| 4 | 1 | 80-99 | +| 5 | 1 | 100-119 | +| 6 | 1 | 120-139 | +| 7 | 1 | 140-159 | + +`nvidia-smi topo -m` must still show GPU0--3 local to NUMA0 and GPU4--7 local +to NUMA1 before execution. Every run records host load average and a full GPU +clock snapshot before and after its measured interval, in addition to the +original process-contamination and environment records. Saturation calibration +runs use the same co-location regime as their measurement runs even when an +isolated calibration would be operationally simpler. + +The aggregate H20-hour estimate remains approximately 7.5 hours because the +same cells and durations execute; the expected matrix wall time becomes +approximately 1.5--2 hours after burn-ins, dependent pairs, TP2 placement, and +confirmation runs are scheduled. The one-time co-location validation is +reported separately from matrix cost. + +### A-P3-2 — detached, resumable execution + +All GPU work is launched by a controller resident in the private dash0 work +directory under `setsid`/`nohup`; an interactive Codex or SSH lifetime is never +the owner of a measurement. The controller has a `--resume` mode and an atomic +per-run state file. A run reaches `complete` only after client, Layer-1, +Layer-2, sanity, command, clock/load, and zero-GPU-memory cleanup checks have +all succeeded. Resumption skips only such complete runs, rejects a changed +source/helper/manifest/config hash, and otherwise cleans up controller-owned +PIDs before restarting the incomplete run from its fresh-server boundary. +Completed output directories are idempotent and are never overwritten. + +The controller records its PID/session, child process groups, phase, +timestamps, exit status, exact commands, and failure reason; state updates use +write-then-rename. It handles termination by stopping only its recorded child +process groups and preserving resumable state. Immutable copies and SHA-256 +hashes of the client, controller, operator mapping, and launch configuration +live in each campaign's `runs/.../provenance/` directory. The orchestrator +polls the state and logs; it does not keep a foreground SSH process alive. + +### A-P3-3 — per-class drain budget + +The post-admission drain hard stop is class-specific. Output-512 burst cells +receive **240 seconds**; all other cells receive **120 seconds**. Under the +frozen pattern table, the 240-second class is exactly P04, P06, P07, P08, and +P11, including their configuration variants and confirmation runs. P02 has a +512-token output but steady arrivals, so it remains in the 120-second class. +The timer starts when final admission stops after the load point's last +profile/recovery interval. Exceeding the applicable budget invalidates the run +and stops the campaign; it does not authorize cancellation, truncation, or a +larger post-hoc allowance. This amendment supersedes the uniform 120-second +drain wording below but changes no clean interval or throughput denominator. + +### A-P3-4 — cumulative Phase-3 GPU budget + +The Phase-3 hard stop is **16.0 H20-hours cumulative**, including E-a +co-location validation, the shutdown-fix GPU verification, compile burn-ins, +the 52-run matrix, retries, and cleanup time while a server still owns GPU +memory. E-a consumed 3.964 H20-hours. Before every launch wave, the detached +controller records consumed cost and refuses to launch if the wave's +conservative reservation would exceed 16.0 H20-hours. This amendment +supersedes the 8.0 H20-hour hard stop below; the original 7.5 H20-hour matrix +estimate remains an estimate, not an additional allowance. + +### A-P3-5 — crash-resilient Layer-1 accounting + +The accepted OpProf series now ends at +`23450fb21ac255b0cf710f4ee965ee694921975d` and contains seven patches. Every +one-second JSONL flush atomically replaces `.footer.json` through a +same-directory temporary file. This checkpoint carries balanced encoded, +written, and dropped counters through the flush, the last written step index, +its wall-clock timestamp, the configured flush interval, and a `final` flag. + +The controller's preferred shutdown remains the official vLLM 0.24.0 path: +start `vllm serve` with a positive `--shutdown-timeout` and signal the API +parent, which selects EngineCore `drain` mode. A clean run must contain an +in-stream footer, and the `final=true` sidecar must agree with its encoded, +written, and dropped counters. Graceful shutdown is not an accounting +dependency, however. If a deliberate or external hard kill leaves no +in-stream footer, the latest atomic sidecar is authoritative. All complete +JSONL data lines must decode; their count must equal sidecar +`written_records`; the final data-line step must equal sidecar +`last_step_index`; `encoded = written + dropped`; and `dropped=0`. The +controller records the signal timestamp and requires the checkpoint age to be +no more than the configured one-second flush interval, with at most 100 ms of +separately reported timestamp/scheduling tolerance. At most one flush interval +after the authoritative checkpoint may be lost. A missing/invalid sidecar, +counter mismatch, older checkpoint, partial JSON line, or nonzero drop count +invalidates the run. + +The seventh patch also makes the preregistered two Layer-2 windows executable: +after `/stop_profile`, a stopped scheduled `TorchProfilerWrapper` is discarded +so the next `/start_profile` constructs a fresh 2+8 schedule. A CPU reproducer +on the pinned torch 2.11 showed that reusing the stopped object returned +without error but emitted no second trace. This repair changes neither the +profile schedule nor its placement. + +### A-P3-5 — drain quarantine (orchestrator amendment; 2026-07-12) + +The orchestrator assigned this amendment the same identifier as the preceding +crash-accounting amendment. This subsection is therefore distinguished by its +title; it supersedes only the drain-budget and drain-stop clauses of A-P3-3 and +leaves crash-resilient accounting unchanged. + +Drain is post-measurement and serves as a hang watchdog. P10-class cells using +the long-context real trace receive **600 seconds**. P04/P06-class cells retain +the **240-second** allowance from A-P3-3; this class remains exactly P04, P06, +P07, P08, and P11. All other cells retain **120 seconds**. + +Exceeding the applicable allowance marks a run `drain-quarantined` but no +longer stops the matrix. If its 240-second clean window, request/output +correctness, Layer-1 accounting, and Layer-2 artifacts are otherwise valid, +the run remains usable in analysis and is explicitly flagged in every table. +The matrix stops for drain behavior only when more than 20% of measured runs +are drain-quarantined. The existing greater-than-5% stop for clean-window +failures is unchanged. + +The preserved P10/C01 saturation run from the first primary wave is +re-adjudicated against 600 seconds. Its observed 288.619107924-second drain is +within that allowance; no GPU rerun is authorized for the clean measurement. + +### A-P3-6 — P10 warm-up stabilization gate (orchestrator amendment; 2026-07-12) + +The warm-up gate measures whether service has reached steady state; completed +requests are only a proxy for that condition. For P10-class runs only, warm-up +passes if either at least 32 requests complete successfully during the +60-second warm-up, or at least 16 complete successfully and trailing Layer-1 +telemetry meets the stabilization test below. All non-P10 warm-up rules remain +unchanged. This amendment changes neither the 240-second clean window nor any +throughput denominator. + +The stabilization test is frozen before re-adjudicating retained telemetry: + +1. Select `model_executed=true` Layer-1 records by `submit_mono_ns` in the + trailing warm-up quartile `[t0+45 s,t0+60 s)` and split them into the fixed + wall-time bins `[45,50)`, `[50,55)`, and `[55,60)` seconds. +2. Each bin must contain at least 16 model-executed steps. For bin `j`, define + scheduled-token throughput + `R_j=sum(prefill_tokens+decode_tokens)/5 s`. Non-finite or non-positive + values, missing bins, discontinuous step indices, or a failed Layer-1 + accounting invariant fail the branch. +3. Fit ordinary least squares to `(47.5,R_1)`, `(52.5,R_2)`, and + `(57.5,R_3)`. Let `s` be the fitted slope and `R_bar` the mean of the three + rates. Stabilization passes only if + `abs(s)*15/R_bar <= 0.10`, i.e. the fitted drift across the complete + trailing quartile is no more than 10% of its mean. + +Every P10 result admitted through the OR branch records the warm-up completion +count, three step counts, three token-throughput values, mean, fitted slope, +normalized drift, and pass/fail result. The final report must render this +post-hoc evidence explicitly; no missing value is imputed. Re-adjudication may +accept an already measured clean window only when this exact test passes. + +### A-P3-7 — frozen partial-matrix inference (orchestrator amendment; 2026-07-12) + +The Phase-3 matrix is final and frozen at 40/52 accepted measured runs and +20/24 complete pattern/config cells. No missing cell is rerun, replaced, or +imputed. The four incomplete cells, each missing both saturation and moderate +load points, are exactly **P03/C11**, **P05/C00**, **P10/C00-TP2**, and +**P11/C00**. The four Layer-1-only C00-moderate confirmations for P10, P06, +P03, and P01 are also absent; confirmation-run repeatability is therefore not +evaluable and cannot be substituted with within-run resampling. + +H1a is evaluated over the completed primary C00 operator-share matrix. Its +formal moderate-load comparison contains exactly P01, P02, P03, P04, P06, +P07, P08, P09, and P10; P05 and P11 are missing. Saturation results for the +same completed patterns remain supporting evidence. A qualifying inversion +among completed patterns confirms H1a because H1a is existential. If none is +found, H1a is **INCONCLUSIVE**, not refuted: the original refutation rule +requires the complete eleven-pattern matrix and simultaneous bounds for every +registered comparison. + +Each frozen H1b contrast evaluates if and only if both of its C00-moderate +cells are complete. Contrast status is fixed as follows: + +| Frozen H1b contrast | A-P3-7 status | +|---|---| +| P05 vs P01 | NOT EVALUABLE — P05/C00 missing | +| P05 vs P03 | NOT EVALUABLE — P05/C00 missing | +| P06 vs P02 | EVALUABLE | +| P06 vs P04 | EVALUABLE | +| P09 vs P01 | EVALUABLE | +| P09 vs P03 | EVALUABLE | +| P10 vs P03 | EVALUABLE | +| P10 vs P04 | EVALUABLE | + +The six evaluable contrasts retain their original waste thresholds, 5% useful +token-efficiency/step-residual association, correction families, seeds, and +decision logic. One qualifying evaluable contrast confirms H1b because H1b is +existential. If none qualifies, H1b is **INCONCLUSIVE**, never refuted, because +two frozen contrasts and four cells are missing. Thus 20/24 coverage permits +confirmation of H1a, H1b, or the compound hypothesis, but it cannot support +the original complete-matrix null/refutation claim. Missing results are +reported as `NOT EVALUABLE` and never assigned zero effect, copied controls, +or synthetic uncertainty. + +## Goal and falsifiable hypothesis + +The system boundary is Qwen3-30B-A3B BF16 served by patched vLLM 0.24.0 on +dash0 H20 GPUs. Phase 3 asks whether the device-level bottleneck observed under +one serving pattern generalizes to other operationally defined patterns. + +The headline hypothesis has two jointly required parts: + +- **H1a — composition:** at the same normalized offered load, at least one pair + of serving patterns has a material inversion in its top GPU operator-family + ranking. “Material” is defined below as a replicated rank inversion with at + least a five-percentage-point share gap, not a visual change in a bar chart. +- **H1b — rectangular-benchmark miss:** at least one non-rectangular serving + pattern exhibits material padding, CUDA-graph miss/bucket mismatch, sequence + raggedness, or mixed-prefill/decode interference that a rectangular offline + batch omits, and that waste is associated with at least a 5% loss in useful + token efficiency or step-time residual relative to its rectangular control. + +The **null outcome** is that the same operator family remains top-ranked across +all primary patterns at matched load, all simultaneous 95% bounds on ranking +gaps are below five percentage points, and every pre-declared waste contrast is +below its materiality threshold. If uncertainty or profiler opacity prevents +either conclusion, the result is **INCONCLUSIVE**, not evidence for the null. + +The full round hypothesis is confirmed only if both H1a and H1b pass. Passing +only one is reported as a partial finding and does not justify the compound +claim. + +## Pinned context and non-goals + +| Item | Frozen value | +|---|---| +| vLLM base | v0.24.0, `ee0da84ab9e04ac7610e28580af62c365e898389` | +| Accepted local patch tip | `23450fb21ac255b0cf710f4ee965ee694921975d` | +| Patch series | Seven checksum-recorded patches in `patches/vllm-0.24.0-opprof/` | +| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, community BF16 | +| Hardware | dash0 H20; TP1 primary, one TP2 pattern/config counterpoint | +| Primary backend expectation | TRITON unquantized MoE; every run must record the observed backend | +| Layer-1 overhead evidence | -0.04196%, 95% CI [-0.17443%, 0.04550%] | +| Layer-2 perturbation evidence | 51.30% request-rate reduction during the active P2 collection window | +| Seeds | workload `20260712`; execution order `20260713`; bootstrap/permutation `20260714` | + +This protocol does not depend on L-C-A labels, does not tune kernels, does not +change graph capture sizes, and does not claim production-cluster generality. +L-C-A may be computed later as a descriptive annotation but cannot define, +filter, or reorder the Phase 3 cells. + +Pinned vLLM source facts used here: + +- on an H20-class device (at least 70 GiB, not A100), the OpenAI server defaults + to `max_num_seqs=1024` and `max_num_batched_tokens=8192` + (`vllm/engine/arg_utils.py:2375-2456`); +- chunked prefill is supported and defaults on for this model + (`vllm/engine/arg_utils.py:2458-2498`); +- the random dataset uses integer-uniform length ranges and generates one fixed + prefix per manifest (`vllm/benchmarks/datasets/datasets.py:557-771`); and +- the bundled finite-rate generator uses Gamma inter-arrivals, but it has no + fixed-duration mode (`vllm/benchmarks/serve.py:391-499`). + +The last point requires a small fixed-duration client before execution. Its +normative interface and behavior are frozen below; the helper is not +implemented in this protocol-only turn. + +## Operational pattern axes + +### Input-length distribution + +All lengths are post-tokenization token counts under the model tokenizer. + +- **short-uniform:** discrete integer uniform `U[128,512]`. +- **long-uniform:** discrete integer uniform `U[4096,8192]`. +- **bimodal:** independent 50/50 draw from `U[128,512]` and + `U[4096,8192]`; exactly half of a 32,768-request manifest comes from each + mode, followed by a seed-fixed permutation. +- **production-shaped mixed:** exactly 50% `U[128,512]`, 30% + `U[1024,2048]`, and 20% `U[4096,8192]`, followed by a seed-fixed + permutation. This is synthetic and is not called a production trace. +- **real-trace anchor:** the approved private subset of + `chat_w20260311_1000` described below. It preserves prompt text and observed + input lengths, filters inputs above 32,768 tokens, and caps requested output + at 256 tokens for bounded execution. + +### Output length + +- **short:** exactly 64 tokens. +- **long:** exactly 512 tokens. +- **trace-capped:** `min(observed_output_length, 256)`, used only by the real + anchor. + +Every request uses greedy temperature 0 and `ignore_eos`; output work is +therefore exact even if generated content is not bit-identical. + +### Arrival shape + +Arrival shape applies at the moderate load point. The saturation point uses +`request_rate=inf`, so all patterns intentionally collapse to a continuously +backlogged client; saturation is used to isolate service composition, not to +test inter-arrival shape. + +- **steady:** deterministic spacing `1/lambda` seconds, first request at + `t=0`. +- **bursty:** deterministic bursts of exactly eight simultaneous requests, + first burst at `t=0`, with burst period `8/lambda` seconds and no within-burst + delay or jitter. + +Here `lambda` is the pattern/config-specific moderate rate defined as 60% of +that pair's measured saturation request throughput. These definitions avoid +conflating the arrival experiment with random arrival noise. + +P02 and P11 are the matched steady/bursty arrival pair: both use short-uniform +input, 512-token output, and no shared prefix. + +### Prefix sharing + +- **none:** no intentionally shared prefix. Synthetic token streams are unique + per request. The manifest must have no common prefix longer than 16 tokens, + and the clean Layer-1 prefix-query hit ratio must remain below 1%; otherwise + the cell is invalid. +- **high:** a pool of eight independently generated 1,024-token prefixes. Each + request appends a unique 256-token suffix, so 80% of each 1,280-token prompt + is shared within one-eighth of requests. Requests are seed-shuffled after + balanced assignment to prefixes. +- **natural:** no prefix is added or removed from the private real prompts. The + observed prefix-hit ratio is reported rather than constrained. + +P07 and P08 are a matched no-prefix/high-prefix pair: both have total input +length 1,280, output length 512, and bursty arrival. + +## Required fixed-duration client + +Before any GPU use, an execution turn must implement and test +`scripts/opprof_phase3_client.py`. The orchestrator must review its diff and +freeze its SHA-256. The helper may reuse vLLM's tokenizer and endpoint request +function, but must not modify vLLM. + +The client contract is: + +1. `materialize` creates a token-exact JSONL manifest using the pinned model + tokenizer and `numpy.random.default_rng(20260712)`. Synthetic manifests + contain 32,768 requests and no manifest may wrap during a run. +2. `run` supports `--request-rate inf` as a closed-loop saturation stream with + at most 256 outstanding requests. A finite rate uses the exact steady or + eight-request periodic-burst schedule above. +3. Admission continues during profiler and recovery intervals. The client + stops admitting after the final clean segment and drains outstanding + requests for at most 120 seconds; it never cancels and counts a partial + response as a failure. +4. The client records request ID, scheduled/admitted/first-token/completion + monotonic timestamps, prompt tokens, requested and actual output tokens, + HTTP status, and no prompt or generated text. It emits one result file per + segment and one manifest SHA-256. +5. `/start_profile` and `/stop_profile` calls are synchronous and timestamped. + Both profile windows occur after the uninterrupted clean interval. Profile + and recovery intervals are tagged and excluded from throughput/latency + summaries. +6. A dry, CPU-only test must prove fixed segment duration, saturation + concurrency, exact burst timing, no manifest wrap, exact output-token + requests, and prompt-field redaction. + +The exact saturation command interface is: + +```bash +python scripts/opprof_phase3_client.py run \ + --manifest "$MANIFEST" --base-url http://127.0.0.1:8000 \ + --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \ + --load-point saturation --request-rate inf \ + --max-concurrency 256 --ignore-eos --temperature 0 \ + --warmup-seconds 60 --clean-segment-seconds 80 \ + --num-clean-segments 3 --profile-after-clean --num-profile-windows 2 \ + --profile-warmup-iterations 2 --profile-active-iterations 8 \ + --recovery-seconds 30 --drain-timeout-seconds 120 \ + --workload-seed 20260712 --result-dir "$PAIR_DIR/saturation/client" +``` + +The exact moderate command is: + +```bash +python scripts/opprof_phase3_client.py run \ + --manifest "$MANIFEST" --base-url http://127.0.0.1:8000 \ + --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \ + --load-point moderate \ + --saturation-result "$PAIR_DIR/saturation/client/result.json" \ + --rate-fraction 0.60 \ + --max-concurrency 256 --ignore-eos --temperature 0 \ + --warmup-seconds 60 --clean-segment-seconds 80 \ + --num-clean-segments 3 --profile-after-clean --num-profile-windows 2 \ + --profile-warmup-iterations 2 --profile-active-iterations 8 \ + --recovery-seconds 30 --drain-timeout-seconds 120 \ + --workload-seed 20260712 --result-dir "$PAIR_DIR/moderate/client" +``` + +The client derives `lambda=0.60*T_sat` from the accepted saturation result and +refuses a manually supplied finite rate. Arrival class comes from the immutable +manifest. Total throughput-accounted duration is exactly `3*80=240` seconds +per measured run. + +## Private real-trace anchor + +The local prompt-bearing source is: + +```text +/home/gahow/phd/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +``` + +It is 1,183,031,556 bytes with SHA-256 +`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`. +Metadata records 32,606 requests over 600 seconds. The frozen selection +`sampling_u <= 0.125 && input_length <= 32768` yields 4,011 requests locally; +after the 256-token output cap, input min/median/p95/max is +69/7,009/25,629/32,704 and output min/median/p95/max is 2/256/256/256. +The anchor preserves prompt text, prompt/output-length correlation, and source +order; it deliberately replaces the original 600-second timestamps with the +same controlled saturation/moderate load protocol as every other cell. It is a +real-content anchor, not a native-traffic replay. + +If and only if the orchestrator approves this anchor, execution may copy the +source to +`/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/` on dash0. The +directory must be mode 0700 and files mode 0600. The source and derived +manifest must never enter a run directory, tar archive, Git, client result, or +Kineto/OpProf artifact. Transfer is accepted only after the local and remote +SHA-256 match. + +The private materialization command is specified in the final pattern table. +It retains only `prompt`, capped `output_tokens`, and an expected input-token +count. A tokenizer parity precheck must find exact input length for at least +99% of rows and absolute error at most one token for every row; failure stops +the anchor rather than rewriting prompts. + +## Server configuration axes + +Chunked prefill remains explicitly enabled in every configuration. Disabling +it on this model makes vLLM raise MBT to the model length when MBT is omitted, +or reject MBT below model length when supplied +(`vllm/config/scheduler.py:272-284`). That would conflate chunking with a much +larger token budget. Phase 3 therefore uses an MBT factor instead. + +| Config | TP | MNS | MBT | Chunked prefill | Purpose | +|---|---:|---:|---:|---|---| +| C00 | 1 | default = 1024 | default = 8192 | on | Primary vLLM/H20 baseline | +| C10 | 1 | small = 64 | default = 8192 | on | Bound sequence concurrency / decode batch | +| C01 | 1 | default = 1024 | small = 2048 | on | Force smaller prefill chunks and token batches | +| C11 | 1 | small = 64 | small = 2048 | on | Test MNS×MBT interaction | +| C00-TP2 | 2 | default = 1024 | default = 8192 | on | One communication/parallelism counterpoint | + +C00 runs for all eleven patterns. The complete 2×2 TP1 factorial C00/C10/C01/C11 +runs only for sentinel patterns P01, P03, P06, and P10. C00-TP2 runs only for +P10. This sparse design identifies each scheduler factor on short-decode, +long-prefill, irregular mixed, and real-trace regimes without spending a full +factorial on every pattern. + +The common server command is: + +```bash +CUDA_VISIBLE_DEVICES="$GPU_IDS" \ +VLLM_OPPROF_DIR="$RUN_DIR/opprof" \ +vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B \ + --tensor-parallel-size "$TP" --enable-chunked-prefill \ + --enable-prefix-caching \ + --profiler-config "{\"profiler\":\"torch\",\"torch_profiler_dir\":\"$TRACE_TMP\",\"torch_profiler_with_stack\":true,\"torch_profiler_record_shapes\":true,\"torch_profiler_use_gzip\":true,\"ignore_frontend\":true,\"wait_iterations\":0,\"warmup_iterations\":2,\"active_iterations\":8}" +``` + +C00/C00-TP2 omit MNS and MBT flags. C10 adds `--max-num-seqs 64`; C01 adds +`--max-num-batched-tokens 2048`; C11 adds both. Startup logs must show the +frozen effective values. Defaults changing from 1024/8192 is a stop condition, +not permission to edit the protocol. + +## Load and execution protocol + +### Unit of analysis and load points + +A **pattern/config cell** is one pattern row under one server configuration. +An **execution run** is one load point for a pattern/config cell. Each primary +cell has exactly two fresh-server runs: + +1. **saturation:** closed-loop `request_rate=inf`, maximum 256 outstanding; +2. **moderate:** the same manifest at `lambda=0.60*T_sat`, where `T_sat` is + clean-segment completed-request throughput from that cell's accepted + saturation run. + +Thus patterns are matched at normalized load `rho=0.60`, not at equal absolute +requests/s or tokens/s. Saturation is analyzed separately. Moderate admission +is valid only if achieved offered rate is within 5% of `lambda` and no sustained +client lag exceeds one second. + +### Warm-up, clean measurement, and profiler placement + +Every measured run uses a fresh server and this timeline: + +```text +60 s warm-up (excluded; require >=32 completions) +80 s clean segment A +80 s clean segment B +80 s clean segment C +Layer-2 window 1: 2 warm-up + 8 active engine iterations (excluded) +30 s recovery (excluded) +Layer-2 window 2: 2 warm-up + 8 active engine iterations (excluded) +30 s recovery / trace flush (excluded) +stop admission; drain <=120 s (excluded) +``` + +The clean A/B/C segments are contiguous. The first profile window occurs only +after exactly 240 accumulated clean seconds; the second follows the first +fixed 30-second recovery, never an observed favorable batch. There are exactly +**four Layer-2 windows per pattern/config cell**: two in saturation and two in +moderate. The 240 clean seconds, not server lifetime or profiler interval, form +the throughput/latency denominator. This ordering prevents the 51.30% +active-window perturbation measured in P2 from contaminating throughput. + +Offered rate runs continuously from warm-up through the clean interval. +Completed throughput counts completions timestamped inside the 240-second clean +interval; latency summarizes those same completions, including a request +admitted during warm-up if it completes cleanly after the boundary. Admission +rate is counted by admission timestamp. This is a stationary-window rule and +is applied identically to every pattern; no profiler interval precedes clean +measurement. + +After each profile window, the 10-second rolling completion rate and +waiting-queue depth must return within 10% of clean-segment-C medians by the end +of the fixed 30-second recovery. This gate protects the second window and +validates post-profile recovery; no clean throughput is measured afterward. +Failure invalidates the run, and recovery is not silently extended. + +### Compile warm-up and execution order + +Before measured runs, launch one unmeasured burn-in server for each of the five +server configurations C00, C10, C01, C11, and C00-TP2. Each burn-in performs +60 seconds of P06 saturation and exits gracefully. It warms compile/AOT and +Triton artifacts but is never pooled with a pattern result. + +After all burn-ins, sort pattern/config IDs by +`SHA256("20260713:" + pattern_id + ":" + config_id)` ascending. Run each +saturation/moderate pair contiguously because moderate depends on saturation. +Four Layer-1-only confirmation runs repeat moderate C00 for P10, P06, P03, and +P01, in that fixed reverse order, after the matrix. They use 60-second warm-up +and 240 clean seconds but no profiler endpoint; they quantify fresh-server +repeatability without changing the pre-declared four Layer-2 windows per cell. + +Every GPU launch must first record `nvidia-smi`, selected UUID(s), clock state, +host load, other-user processes, exact command, expected duration, source and +manifest hashes, compile-cache key, and effective vLLM config. One H20 is used +except for the two C00-TP2 runs and its TP2 burn-in. No other user's process may +be touched. The preferred controller path starts the server with a positive +`--shutdown-timeout` and signals the API parent so EngineCore drains before the +process manager exits. Accounting is validated by the A-P3-5 footer/sidecar +rule even when graceful teardown does not occur, and GPU memory must finally +return to zero. + +## Measurement plan + +### Layer 1: always on + +Every measured and confirmation run sets `VLLM_OPPROF_DIR`. Layer 1 is the +source of clean-segment composition and waste counters: + +- scheduled requests, prefill/decode batch and token counts; +- chunked-prefill class and chunk-size histogram; +- context-length histogram; +- waiting/running/deferred queues, preemptions, KV usage, and prefix counters; +- step submit/complete timestamps; and +- CUDA-graph hit, runtime mode, unpadded tokens, bucket tokens, and padding. + +Each run must produce exactly one schema-v1 JSONL for TP1 or one scheduler-owned +stream for the TP2 engine. Every line must msgspec-decode; step indices must be +unique and contiguous. On clean close, the in-stream footer must satisfy +`encoded = written + dropped`, `dropped=0`, and its step count must equal the +record count, while the final sidecar agrees. Without an in-stream footer, the +atomic sidecar and on-disk records must satisfy the A-P3-5 hard-kill rule. +Profile/recovery records remain in the raw stream but are tagged by monotonic +time and excluded from clean summaries. + +Composition is summarized separately for clean A/B/C, each profiler window, +each recovery interval, and the complete clean union. No profile-window record +may leak into throughput, latency, or unperturbed composition statistics. + +### Layer 2: sampled Kineto windows + +Each profile endpoint invocation uses the server's frozen wait=0, warm-up=2, +active=8 schedule. A logical window is accepted only if: + +- `/start_profile` and `/stop_profile` return HTTP 200; +- the trace gzip decompresses and parses as JSON; +- active `ProfilerStep` numbers are exactly 2 through 9; +- exactly eight matching execute annotations are present per TP rank; +- kernel event count is positive; and +- every active execute annotation joins unambiguously to its Layer-1 step. + +TP1 yields one worker trace per logical window. TP2 is expected to yield two +rank traces; both are required and analyzed per rank. Rank times are never +summed and called latency. + +The client writes a monotonic timestamp immediately before/after each endpoint +call. The analysis joins execute annotations to Layer-1 records by timestamp, +order, and the annotation's context/generation token counts. A join is invalid +if more than one Layer-1 record is compatible or if the annotation and Layer-1 +token totals disagree. + +Profiler representativeness is checked against the adjacent clean segment for +scheduled tokens, prefill fraction, decode batch size, and graph-mode shares. +For each feature, use standardized mean difference +`SMD=(mean_profile-mean_clean)/pooled_sd`; a constant feature uses exact +equality. If `abs(SMD)>0.25` for any feature in either window, that load point's +operator breakdown is invalid. One complete run may be retried with the same +command; a second failure is reported as Layer-2-inconclusive, not resampled +until it looks representative. + +### Kernel-to-operator-family mapping + +The primary operator share is summed GPU kernel duration in a family divided +by total GPU kernel/replay activity duration in the eight active steps, +including unmatched/opaque activity in the denominator. It is computed per +window and rank. CPU launch time, CUDA runtime time, memcpy/memset time, and +unioned device-busy time are reported separately. Because streams may overlap, +kernel-duration shares are composition shares, not fractions of wall time. + +Mapping is priority-ordered; first match wins. The execution helper must emit a +versioned regex table and SHA-256 before GPU work. The rules below are the +normative v1 mapping: + +| Priority | Family | Name/scope rule | +|---:|---|---| +| 1 | `moe_router` | `topkGating`, `moe_align`, `select_expert`, or MoE-scoped routing/top-k kernels; this precedes sampler matching | +| 2 | `collective` | `nccl`, `all_reduce`, `allreduce`, `reduce_scatter`, `all_gather`, or vLLM custom-all-reduce names | +| 3 | `attention` | `flash`, `fmha`, `paged_attention`, `unified_attention`, attention-scoped CUTLASS kernels, or attention KV/cache reshape kernels | +| 4 | `moe_gemm` | `fused_moe`, `moe.*gemm`, `nvjet`, or CUTLASS/Triton GEMM whose parent scope is FusedMoE/MoE | +| 5 | `sampler` | non-MoE `sampling`, `topk`, `topp`, `argmax`, `multinomial`, logits-processor, or penalty kernels | +| 6 | `dense_gemm` | GEMM/matmul/CUTLASS kernels outside attention and MoE scopes | +| 7 | `norm_elementwise` | RMS/layer norm, activation, residual/add, reduction, fill, and vectorized elementwise kernels | +| 8 | `kv_memory` | cache copy/swap kernels and KV movement not already attributed to attention; CUDA memcpy/memset activities remain separately reported | +| 9 | `other` | every unmatched kernel; no post-hoc cell-specific reassignment | + +Attention is subdivided by the joined Layer-1 step: + +- `attention_prefill`: prefill tokens >0 and decode tokens =0; +- `attention_decode`: decode tokens >0 and prefill tokens =0; and +- `attention_mixed`: both are nonzero. + +Mixed attention is not force-split in proportion to tokens. The headline +bottleneck ranking uses `attention_total`; the three subfamilies explain which +serving phase supplied it. + +Before inference, report the top 20 unmatched kernel names by duration. If +`other` plus opaque graph replay exceeds 30% of GPU kernel time in any primary +window, that window is not classifiable. No mapping amendment may be derived +from one favored cell: a reviewed global amendment must be applied to every +trace, with old and new results both retained. + +### CUDA-graph mode segmentation + +Every active step is segmented by its Layer-1 `runtime_mode`: + +- `FULL`; +- `PIECEWISE`; +- `NONE`, reported as eager/ungraphed; or +- `ambiguous`, which invalidates the step. + +Operator shares and waste are reported both overall and by mode when a mode has +at least eight active steps across the two windows of a load point. If CUPTI +shows only a graph-launch event with no child kernel activity for FULL or +PIECEWISE, its device time is labeled `graph_replay_opaque`; it is never +distributed among operator families. A cell cannot support H1a unless at least +70% of its device kernel time is classifiable. + +## Metrics and pre-declared analysis + +### End-to-end and composition metrics + +For each clean segment and their 240-second union, report: + +- completed and failed requests; achieved offered and completed requests/s; +- input, output, and total useful tokens/s; +- TTFT, TPOT, and end-to-end latency: mean, p50, p95, and p99; +- scheduled tokens/step, requests/step, prefill/decode tokens/step, decode batch, + queue depths, KV usage, preemptions, and prefix hit/query ratios; +- FULL/PIECEWISE/NONE shares and step duration by mode; and +- absolute values plus ratios to C00, never ratios alone. + +Primary useful-token efficiency is +`E_token = sum(clean prefill_tokens + clean decode_tokens) / +sum(clean step_duration_ms)` in tokens/ms. End-to-end input/output tokens/s is +reported beside it. H1b's “5% worse efficiency” means +`1-E_irregular/E_control >= 0.05` using `E_token`. + +The primary cross-pattern comparison is C00-TP1 at moderate `rho=0.60`. +C00 saturation is secondary. C10/C01/C11 estimate scheduler-factor and +interaction sensitivity on sentinels. C00-TP2 is descriptive parallelism +evidence; a single TP2 pattern does not support a scaling claim. + +### Operator shares and bottleneck ranking + +For operator family `f` in window `w` and rank `r`: + +```text +share[f,w,r] = sum GPU kernel duration assigned to f + / sum duration of all GPU kernel/replay activity +``` + +Each load point has two independent wall-separated windows. Rank families by +share within each window; ties within one percentage point receive the same +rank. TP2 reports each rank and their mean/range, not their sum. + +The cross-pattern ranking-change test considers every pair of primary C00 +patterns and every pair of operator families. An inversion is accepted only +when: + +1. family A exceeds B by at least five percentage points in **both** windows of + pattern p; +2. B exceeds A by at least five percentage points in **both** windows of + pattern q; +3. a window-stratified permutation test over the 16 active steps per pattern, + seed 20260714 and 100,000 permutations, rejects equal ordering after Holm + correction across all tested pattern/family pairs at family-wise alpha 0.05; +4. at least 70% of kernel time is classifiable in all four windows. + +Reproduction at saturation or in another server configuration is reported as +strengthening evidence but is not required: saturation deliberately removes +the arrival treatment, and only four patterns receive config variants. The two +fixed, wall-separated windows are the pre-declared replication unit for H1a. + +Also report Kendall's tau-b between every pair of family-rank vectors. Tau is +descriptive and cannot replace the inversion rule. + +### Waste accounting + +All ratios are computed as ratios of sums, not means of per-step percentages. + +**CUDA-graph padding.** For graph-hit steps with positive bucket `b_s` and +unpadded tokens `u_s`: + +```text +padding_fraction = sum_s (b_s - u_s) / sum_s b_s +``` + +Also report padded tokens per useful scheduled token and the distribution of +`b_s-u_s`. NONE steps are excluded from padding and counted under graph miss. + +**Graph miss and bucket mismatch.** Let `B` be the capture bucket set parsed +from the startup config and `b*(u)=min{b in B: b>=u}`. + +```text +graph_miss_rate = count(model_executed and not hit) / count(model_executed) +eligible_miss_rate = count(model_executed and not hit and b*(u) exists) + / count(model_executed) +overflow_rate = count(model_executed and not hit and b*(u) absent) + / count(model_executed) +bucket_slack = sum(b*(u)-u) / sum(b*(u)) over eligible u +``` + +Report all four by pattern, config, load, and runtime mode. + +**Sequence raggedness.** Partition the immutable arrival-order manifest into +consecutive complete cohorts of 64 requests. For input lengths `L_gi`: + +```text +R64 = 1 - sum_g sum_i L_gi / sum_g (64 * max_i L_gi) +``` + +This is the exact padding fraction an offline rectangular batch-of-64 input +benchmark would incur on the same requests. The incomplete final cohort is +excluded. Report sensitivity at cohort sizes 32 and 128, but R64 is primary. + +**Mixed-batch interference.** On clean Layer-1 steps, fit nonnegative robust +splines `F_p(P,N)` on pure-prefill steps and `F_d(D,N)` on pure-decode steps, +where `P`/`D` are prefill/decode tokens and `N` is scheduled requests. Let +`alpha` be median zero-token scheduler-step time. For a mixed step: + +```text +t_add = F_p(P,N) + F_d(D,N) - alpha +I_mix = sum_mixed (t_observed - t_add) / sum_mixed t_add +``` + +Fits are config/load-specific and use all patterns, with leave-one-pattern-out +prediction for the reported cell. A mixed step outside the convex support of +both pure-step fits is excluded and counted. If fewer than 30 supported mixed +steps remain, `I_mix` is N/A rather than extrapolated. This is an interference +association, not a causal kernel ablation. + +**MoE/ragged proxy.** Report coefficient of variation of MoE GEMM kernel +durations across layers within each active step: +`CV_moe = sd(duration_l)/mean(duration_l)`, only when layer scopes are +available for at least 80% of MoE GEMM time. Otherwise it is N/A. Exact expert +routes are not enabled because the server-wide routed-expert return path would +contaminate clean throughput. + +### Statistical summaries + +- Layer-1 and end-to-end intervals use a 5-second moving-block bootstrap over + clean time blocks, 100,000 resamples, seed 20260714. +- Layer-2 ranking uses the window-stratified permutation rule above. Active + steps are not presented as independent server replicates. +- The four Layer-1-only sentinel confirmations report run-level absolute and + relative deltas; no outlier is removed. +- Multiple ranking tests use Holm correction. Waste contrasts use Holm + correction within each waste family. +- No arithmetic mean of normalized scores is used. Cross-cell normalized + aggregates, if requested later, use a geometric mean and retain all cells. + +### Decision rule + +H1a passes if at least one inversion meets all four ranking criteria. H1b +passes if an irregular cell (P05, P06, P09, or P10) versus its declared +rectangular control has a corrected 95% contrast excluding zero and exceeds at +least one threshold: + +- padding fraction: 5 percentage points; +- graph-miss or overflow rate: 10 percentage points; +- R64: 0.15 absolute; +- `I_mix`: 0.10; or +- `CV_moe`: 0.15; + +and that contrast coincides with at least 5% worse useful-token efficiency or +positive step-time residual. The frozen control contrasts are P05 versus both +P01 and P03, P06 versus both P02 and P04, P09 versus both P01 and P03, and P10 +versus both P03 and P04. Every listed contrast enters the same correction +family and is reported; none is selected afterward by the best p-value. + +The compound hypothesis is **CONFIRMED** only when H1a and H1b pass. It is +**REFUTED / NULL SUPPORTED** only when the same top family holds in both +windows of every primary pattern, simultaneous 95% upper bounds on all family +share gaps are below five percentage points, and all H1b contrasts are below +threshold. Every other outcome is **INCONCLUSIVE** or **PARTIAL**. + +## Artifact, privacy, and reproducibility contract + +Each run directory contains only: + +```text +manifest.sha256 # hash and non-sensitive length summary +commands.log # exact server/client/profile commands +environment.json # source, patch, wheel, driver, GPU, clocks +server.log +client/{segments.jsonl,result.json,sanity.json} +opprof/*.jsonl +traces/*.pt.trace.json.gz +analysis/{layer1.json,kernels.json,waste.json,sanity.json} +``` + +Synthetic manifests may be archived. P10's manifest, prompts, generated text, +and any response body are private inputs and are forbidden from this tree. +Before packaging, a scanner must reject any artifact containing a `prompt`, +`messages`, `content`, or generated-text field, or any source prompt substring. +The private path and manifest hash may appear in the exact command log; file +contents may not. Only counts, length distributions, hashes, and timings may +leave the private directory. + +Record source/patched commit, patch checksums, Python/torch/CUDA/vLLM/msgspec +versions, driver, model path and shard hashes, GPU UUID(s), server effective +config, MoE backend, compile cache key, client/helper SHA, workload hash, clock +snapshots, host load, other-user process samples, and all raw analysis seeds. + +## Budget and hard stop conditions + +The matrix contains 24 pattern/config cells: 23 TP1 and one TP2. Each has two +load runs, for 48 primary measured runs. Four Layer-1-only confirmation runs +bring the measured total to **52**. Five unmeasured compile burn-ins are not +counted as pattern runs but are included in cost. + +Using the P2 fixed-cache startup and profile endpoint timings, budget each +primary run as approximately: + +```text +70 s startup + 60 s warm-up + 240 s clean ++ 2 * (24 s profiler + 30 s recovery) + 10 s shutdown = 488 s +``` + +Expected serialized wall time is about 7.1 hours. Accounting for two GPUs in +the two TP2 runs and its burn-in gives **about 7.5 H20-hours**, with a hard +campaign stop at 8.0 H20-hours. The four confirmation runs are budgeted at +380 seconds each. Queueing for a free GPU is not GPU time. + +There are 96 logical Layer-2 windows. TP1 contributes 92 trace files; TP2 may +contribute eight rank files, for 100 expected trace files. At the P2 trace size +of 16.4 MB, Kineto uses about 1.6 GB. Layer 1, logs, client data, and analysis +should keep public artifacts below 4 GB. The private source plus derived P10 +manifest is separately budgeted below 2 GB. Stop if public artifacts exceed +8 GB or CPFS free space falls below 100 GB. + +Stop the matrix immediately and request review if any of these occurs: + +1. source/patch/helper/manifest hash mismatch, effective config mismatch, or a + per-run output directory changes the compile-cache key; +2. selected GPU not idle, another user's GPU process appears, GPU OOM, server + error, or final GPU cleanup does not return to zero; +3. any clean-segment request fails, an output-token count differs from the + manifest, the moderate offered rate misses its target by more than 5%, or + drain exceeds 120 seconds; +4. Layer-1 schema/footer/step accounting fails, any record drops, or any + profile/recovery step enters a clean summary; +5. either trace is missing/unloadable, has other than 2+8 scheduled iterations, + lacks kernels, cannot join to Layer 1, or profiler composition has + `abs(SMD)>0.25` twice; +6. recovery does not meet its fixed 30-second criterion; +7. more than 30% of device kernel time is `other`/opaque in a primary window, + or fewer than eight active steps are available for a required mode segment; +8. a prompt, message, content, generated-text field, or source prompt substring + appears outside the private directory; +9. public disk or the 8.0 H20-hour cap is reached; or +10. a data-sanity red flag defined below appears. + +No failed cell is silently replaced, retuned, filtered, or omitted. One exact +rerun is allowed only for an infrastructure/profile-artifact failure; both +attempts remain in the report. A semantic workload/config failure requires a +reviewed protocol amendment. + +## Data sanity block contract + +Every raw-run summary, per-cell analysis, aggregate table, figure source, and +final report must end with a machine-readable and rendered sanity block. For +**every numeric family** it reports `n`, finite count, missing count, minimum, +maximum, and distinct-value count. It also records expected constants rather +than treating them as suspicious. + +At minimum, the block checks: + +- non-negative request/token/queue/KV/kernel/duration counters; +- ratios in [0,1], except signed residual/interference metrics explicitly + identified as signed; +- exactly 240 clean seconds, three 80-second segments, two profiler windows per + load, 2+8 profiler iterations per window, and four logical windows per cell; +- continuous unique Layer-1 steps and balanced footer accounting with zero + drops; +- exact manifest hash, no wrap, exact requested/actual output work, zero clean + failures, and moderate rate within 5%; +- bucket >= unpadded, padding identity, graph-mode consistency, and waste + numerator <= denominator; +- trace loadability, positive kernel count, rank completeness, unambiguous + Layer-1 join, and at least 70% classifiable device time; +- input/output and arrival distributions match the frozen manifest; +- per-pattern and per-config outputs are not all identical unless the field is + a declared constant; +- clocks/load/GPU-process contamination and graceful zero-memory cleanup; and +- no private prompt/generated text in public artifacts; private paths are + allowed only in exact command logs. + +Negative non-negative counters, ratios outside bounds, missing footers, +unexpectedly identical pattern/config results, discontinuous step/time series, +manifest drift, profiler leakage, or private-data leakage are red flags. Report +the anomaly first and stop analysis; do not build a conclusion on it. + +## Final preregistration summary + +The table below is normative. `P3C='python scripts/opprof_phase3_client.py'`, +`PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests`, and every +synthetic command uses the pinned model tokenizer. Each command is one logical +line even where Markdown wraps it. + +### Pattern-cell table + +| ID | Operational cell | Exact manifest-generator command | +|---|---|---| +| P01 | short-uniform input; short output; steady; no prefix | `$P3C materialize --id P01 --kind synthetic --input-uniform 128:512 --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P01.jsonl"` | +| P02 | short-uniform input; long output; steady; no prefix | `$P3C materialize --id P02 --kind synthetic --input-uniform 128:512 --output-fixed 512 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P02.jsonl"` | +| P03 | long-uniform input; short output; steady; no prefix | `$P3C materialize --id P03 --kind synthetic --input-uniform 4096:8192 --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P03.jsonl"` | +| P04 | long-uniform input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P04 --kind synthetic --input-uniform 4096:8192 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P04.jsonl"` | +| P05 | 50/50 short/long bimodal input; short output; steady; no prefix | `$P3C materialize --id P05 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}' --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P05.jsonl"` | +| P06 | 50/50 short/long bimodal input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P06 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}' --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P06.jsonl"` | +| P07 | fixed 1,280-token input; long output; bursty 8-at-once; no prefix control | `$P3C materialize --id P07 --kind synthetic --input-fixed 1280 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P07.jsonl"` | +| P08 | fixed 1,280-token input; long output; bursty 8-at-once; high prefix sharing | `$P3C materialize --id P08 --kind prefix-pool --num-prefixes 8 --prefix-len 1024 --suffix-fixed 256 --output-fixed 512 --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P08.jsonl"` | +| P09 | production-shaped 50/30/20 mixed input; short output; steady; no prefix | `$P3C materialize --id P09 --kind synthetic --input-mixture '{"uniform:128:512":0.5,"uniform:1024:2048":0.3,"uniform:4096:8192":0.2}' --output-fixed 64 --prefix none --arrival steady --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P09.jsonl"` | +| P10 | private real chat prompts; observed input <=32,768; output capped 256; steady; natural prefix | `$P3C materialize-private --id P10 --source /home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl --sampling-u-max 0.125 --max-input-tokens 32768 --output-cap 256 --preserve-prompts --disable-shuffle --arrival steady --out "$PRIVATE/P10.jsonl"` | +| P11 | short-uniform input; long output; bursty 8-at-once; no prefix | `$P3C materialize --id P11 --kind synthetic --input-uniform 128:512 --output-fixed 512 --prefix none --arrival burst:8 --num-requests 32768 --workload-seed 20260712 --out "$PRIVATE/P11.jsonl"` | + +### Total run count and GPU-hour estimate + +- Pattern cells: **11**. +- TP1 pattern/config cells: **23** — C00 for all eleven, plus C10/C01/C11 + for P01/P03/P06/P10. +- TP2 counterpoint cells: **1** — P10/C00-TP2. +- Primary runs: **48** — saturation and moderate for each of 24 cells. +- Layer-1-only confirmation runs: **4**. +- Total measured runs: **52**, plus five unmeasured compile burn-ins. +- Layer-2 windows: **96 logical windows**, 2+8 iterations each; 100 expected + rank trace files. +- Fixed throughput-accounted time: **240 seconds per measured run**. +- Expected cost: **about 7.1 hours serialized wall time and 7.5 H20-hours**; + hard stop at 8.0 H20-hours. +- Expected public artifact volume: approximately 2.5–4 GB; hard stop at 8 GB. + +### Open decisions for the orchestrator + +1. **Blocking:** approve the fixed-duration helper interface/algorithm and a + no-GPU implementation-and-test turn before execution. The existing bundled + client cannot enforce this protocol's fixed clean duration and profile masks. +2. **Blocking:** approve copying the sensitive P10 source to the mode-0700 + dash0 private directory, its `sampling_u<=0.125`, input<=32,768 selection, + and 256-token output cap. Rejection requires a protocol amendment; P10 will + not be silently replaced. +3. Approve the sparse scheduler design: MNS 64/default 1024 and MBT + 2048/default 8192, full factorial only on P01/P03/P06/P10, chunked prefill + always on. +4. Approve P10/C00 as the single TP2 counterpoint rather than adding TP4 or a + second pattern. +5. Approve normalized moderate load `rho=0.60`, deterministic eight-request + bursts, 240 clean seconds, and exactly four Layer-2 windows per cell. +6. Approve the mapping priority, 70% classifiability gate, materiality + thresholds, and confirm/refute/inconclusive decision rule. + +### Protocol sanity block + +| Numeric family | n | Min | Max | Distinct | Invariant/result | +|---|---:|---:|---:|---:|---| +| Pattern rows | 11 | P01 | P11 | 11 | Within required 8–12 range | +| Server configs | 5 | TP1/MNS64/MBT2048 | TP2/MNS1024/MBT8192 | 5 | Four TP1 factorial configs plus one TP2 counterpoint | +| TP size | 5 configs | 1 | 2 | 2 | TP1 primary; exactly one TP2 cell | +| MNS level | 5 configs | 64 | 1024 | 2 | Small/default only | +| MBT level | 5 configs | 2048 | 8192 | 2 | Small/default only; chunking always on | +| Pattern/config cells | 24 | 1 config/pattern | 5 configs for P10 incl. TP2 | 3 counts | Per-pattern config counts are 1, 4, or 5; 23 TP1 + 1 TP2 | +| Primary load runs | 48 | 2/cell | 2/cell | 1 | Saturation plus moderate for every cell | +| Confirmation runs | 4 | 1/sentinel | 1/sentinel | 1 | P10/P06/P03/P01 moderate C00, Layer-1 only | +| Total measured runs | 1 aggregate | 52 | 52 | 1 | `48+4=52` | +| Clean duration/run | 52 | 240 s | 240 s | 1 expected | Three exact 80-second segments | +| Logical Layer-2 windows/cell | 24 | 4 | 4 | 1 expected | Two per load point | +| Logical Layer-2 windows | 1 aggregate | 96 | 96 | 1 | `24*4=96` | +| Active profiler iterations | 1 aggregate | 768 | 768 | 1 | `96*8=768`; every window also has two warm-ups | +| Expected rank trace files | 1 aggregate | 100 | 100 | 1 | 92 TP1 plus 8 TP2 rank files | +| Local real-trace rows | 1 source | 32,606 | 32,606 | 1 | Prompt contents never printed or copied in this turn | +| Frozen P10 selected rows | 1 subset | 4,011 | 4,011 | 1 | Selection reproduced locally without prompt output | +| P10 selected input tokens | 4,011 | 69 | 32,704 | 3,424 | Sum 36,499,001; median 7,009; p95 25,629 | +| P10 capped output tokens | 4,011 | 2 | 256 | 236 | Sum 878,291; median/p95 both 256 | +| Estimated H20-hours | 1 budget | 7.5 | 7.5 | 1 | Positive and below hard cap 8.0 | +| Phase-3 GPU/SSH actions this turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied | + +Checked arithmetic invariants: `11+4*3=23` TP1 cells; adding one TP2 cell gives +24; two load points give 48 primary runs; four confirmations give 52 measured +runs; `24*4=96` logical profiler windows; TP2 adds four extra rank files, giving +100. Ratios and thresholds are within their declared domains. Expected constants +are labeled. No Phase 3 GPU run, SSH action, trace transfer, helper +implementation, patch change, or prompt disclosure occurred in this turn. diff --git a/docs/opprof/phase3-results.md b/docs/opprof/phase3-results.md new file mode 100644 index 0000000..52fd9b0 --- /dev/null +++ b/docs/opprof/phase3-results.md @@ -0,0 +1,262 @@ +# OpProf Phase 3 dash0 results + +Status: **FINAL A-P3-7 PARTIAL-MATRIX ANALYSIS — H1a INCONCLUSIVE; H1b PASS**. + +Date: 2026-07-12. The matrix is permanently frozen at 40/52 accepted measured +runs and 20/24 complete pattern/config cells. A-P3-7 permits existential +confirmation on complete comparisons but forbids refutation from incomplete +coverage. The result is therefore **PARTIAL**: H1b is confirmed by five +evaluable contrasts, while H1a cannot be evaluated on a valid pattern pair. + +The final machine result is +`runs/opprof-phase3/phase3/metrics.json` (SHA-256 +`dcc44941bfe1eb56c7a069148e5565eddd1ccf966e6914aa16b64bd67d02f0f0`). +The final analyzer SHA-256 is +`483c170838a1e81a170d8425b4be22cc1be02ea4bc9492b474daf7ae86186d67`; +7/7 no-GPU analysis tests passed, including ResourceWarning-as-error. + +## Data-sanity result first + +All **23/23** machine-checked invariants pass before inference: 40 unique +accepted primary markers selected only through completed-stage records; exact +240-second clean windows; zero clean failures; finite moderate rates within +5%; balanced Layer-1 footer/sidecar accounting; ratios in range; 72 accepted +trace files with the registered eight-file first-wave deviation; zero other +GPU processes; exact 20-cell coverage; and the exact A-P3-7 missing-cell and +missing-contrast sets. + +The accepted data contain 77,109 clean completions, 352,907 clean Layer-1 +steps, and 542,350 total Layer-1 records. Device-time classifiability is +97.05–99.64%. No malformed trace, accounting mismatch, counter underflow, +private-text leak, or unexpected identical-result family was found. + +Declared limitations are not hidden as sanity passes: eight accepted +saturation trace files were lost during the registered first-wave controller +cleanup; all four confirmation runs are absent; MoE per-layer CV is unavailable +because layer scopes cover less than 80% of MoE GEMM time; and only one of nine +completed C00-moderate patterns passes the Layer-2 inference gates. + +## Frozen coverage and A-P3-7 logic + +The four incomplete cells, each missing saturation and moderate, are exactly: + +- P03/C11; +- P05/C00; +- P10/C00-TP2; and +- P11/C00. + +The four missing confirmations are P10, P06, P03, and P01 C00-moderate. +Formal H1a coverage is therefore the nine completed C00 patterns P01, P02, +P03, P04, P06, P07, P08, P09, and P10. Formal H1b has six evaluable and two +non-evaluable frozen contrasts. A-P3-7's asymmetry is mandatory: one valid hit +can confirm an existential hypothesis, but absence of a hit cannot support the +original complete-matrix null. + +## H1a — operator composition + +**Verdict: INCONCLUSIVE; refutation is not allowed.** + +All moderate traces were parseable and classifiable, but only P04's two +windows jointly passed classifiability, `abs(SMD)<=0.25` representativeness, +and fixed recovery. P01/P02/P03/P09 failed representativeness in both windows; +P06/P07/P08/P10 additionally failed at least one recovery. Thus only one of +nine completed patterns is inferentially evaluable, leaving zero pattern pairs, +zero ranking tests with data, and no Kendall tau-b pairs. No qualifying +inversion can be accepted, and this is not evidence for a shared ranking. + +Descriptively, P04 is attention-led at both loads. P06 and P10 change from +attention-led at saturation to MoE-GEMM-led at moderate load; the other +available patterns remain MoE-GEMM-led. These are ranking changes in sampled +windows, not H1a evidence because their window gates fail. + +### Per-pattern operator shares, both load points + +Values are the mean percentage across the two Layer-2 windows. `E` means the +row is inferentially evaluable; `D` means data are available but descriptive +only because a registered window/recovery gate failed; `NE-cell` means the cell +is missing; `NE-trace` means the accepted run has no retained trace. `Other` is +unclassified device activity. Collective, sampler, and dense-GEMM shares round +to 0.00% in every available C00 row. + +| Pattern | Load | Use | Attention | MoE GEMM | Router | Collective | Sampler | Dense GEMM | Norm/elt | KV | Other | Top family | +|---|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---| +| P01 | saturation | D | 16.32 | 77.81 | 0.85 | 0.00 | 0.00 | 0.00 | 3.90 | 0.10 | 1.02 | MoE GEMM | +| P01 | moderate | D | 12.09 | 82.43 | 1.03 | 0.00 | 0.00 | 0.00 | 3.38 | 0.11 | 0.96 | MoE GEMM | +| P02 | saturation | D | 32.67 | 62.39 | 1.05 | 0.00 | 0.00 | 0.00 | 2.98 | 0.16 | 0.74 | MoE GEMM | +| P02 | moderate | D | 20.91 | 72.53 | 1.64 | 0.00 | 0.00 | 0.00 | 3.68 | 0.25 | 1.00 | MoE GEMM | +| P03 | saturation | D | 39.11 | 56.60 | 0.44 | 0.00 | 0.00 | 0.00 | 2.98 | 0.03 | 0.84 | MoE GEMM | +| P03 | moderate | D | 23.32 | 57.45 | 3.54 | 0.00 | 0.00 | 0.00 | 12.29 | 0.83 | 2.57 | MoE GEMM | +| P04 | saturation | D | 65.88 | 29.77 | 1.16 | 0.00 | 0.00 | 0.00 | 2.30 | 0.17 | 0.72 | Attention | +| P04 | moderate | **E** | 47.88 | 40.64 | 3.28 | 0.00 | 0.00 | 0.00 | 5.85 | 0.45 | 1.90 | Attention | +| P05 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — | +| P05 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — | +| P06 | saturation | D | 56.59 | 39.35 | 1.04 | 0.00 | 0.00 | 0.00 | 2.25 | 0.15 | 0.62 | Attention | +| P06 | moderate | D | 31.60 | 57.52 | 2.97 | 0.00 | 0.00 | 0.00 | 5.69 | 0.43 | 1.80 | MoE GEMM | +| P07 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — | +| P07 | moderate | D | 28.67 | 62.18 | 2.42 | 0.00 | 0.00 | 0.00 | 4.86 | 0.37 | 1.49 | MoE GEMM | +| P08 | saturation | NE-trace | — | — | — | — | — | — | — | — | — | — | +| P08 | moderate | D | 34.59 | 57.06 | 2.22 | 0.00 | 0.00 | 0.00 | 4.44 | 0.34 | 1.34 | MoE GEMM | +| P09 | saturation | D | 26.23 | 69.03 | 0.36 | 0.00 | 0.00 | 0.00 | 3.43 | 0.00 | 0.95 | MoE GEMM | +| P09 | moderate | D | 16.19 | 75.00 | 2.14 | 0.00 | 0.00 | 0.00 | 4.83 | 0.35 | 1.49 | MoE GEMM | +| P10 | saturation | D | 67.80 | 28.90 | 0.76 | 0.00 | 0.00 | 0.00 | 1.87 | 0.10 | 0.57 | Attention | +| P10 | moderate | D | 30.61 | 54.04 | 2.81 | 0.00 | 0.00 | 0.00 | 9.80 | 0.68 | 2.06 | MoE GEMM | +| P11 | saturation | NE-cell | — | — | — | — | — | — | — | — | — | — | +| P11 | moderate | NE-cell | — | — | — | — | — | — | — | — | — | — | + +### Clean CUDA-graph modes at moderate load + +| Pattern | FULL | PIECEWISE | NONE/eager | +|---|---:|---:|---:| +| P01 | 1.98% | 73.19% | 24.83% | +| P02 | 83.16% | 13.00% | 3.84% | +| P03 | 96.37% | 0.00% | 3.63% | +| P04 | 98.32% | 0.02% | 1.66% | +| P05 | — | — | — | +| P06 | 99.10% | 0.05% | 0.85% | +| P07 | 98.76% | 0.00% | 1.24% | +| P08 | 98.64% | 0.01% | 1.36% | +| P09 | 91.79% | 3.69% | 4.52% | +| P10 | 98.33% | 0.12% | 1.55% | +| P11 | — | — | — | + +P01 is the clear mode outlier: PIECEWISE covers 73.19% of clean moderate +steps. Full per-mode operator segments and active-step counts remain in the +machine result; no mode with fewer than eight sampled steps is summarized. + +## H1b — rectangular-benchmark miss + +**Verdict: PASS.** Five of six evaluable frozen contrasts pass at least one +waste threshold with a simultaneous positive bound and at least 5% useful-token +efficiency loss. The two P05 contrasts are `NOT EVALUABLE`; no value is +imputed. Holm correction retains the original eight planned contrasts per +metric. Reported simultaneous intervals below are percentage-point effects; +passing bootstrap p-values are zero at 100,000-resample resolution. + +| Frozen contrast | Status | Passing evidence | Useful-token efficiency loss | +|---|---|---|---:| +| P05 vs P01 | **NOT EVALUABLE** | P05/C00 missing | — | +| P05 vs P03 | **NOT EVALUABLE** | P05/C00 missing | — | +| P06 vs P02 | **PASS** | R64 +23.015 pp, simultaneous 95% [22.345, 23.683] | 11.609% | +| P06 vs P04 | **PASS** | R64 +35.440 pp [34.826, 36.059] | 22.846% | +| P09 vs P01 | **PASS** | padding +6.673 pp [5.887, 7.492]; R64 +39.616 pp [39.065, 40.166] | 8.325% | +| P09 vs P03 | EVALUABLE, NO HIT | padding +8.542 pp and R64 +52.041 pp are material, but association gate fails | **3.840%**, below 5% | +| P10 vs P03 | **PASS** | padding +5.565 pp [2.010, 10.341]; R64 +44.787 pp [43.635, 45.971] | 44.693% | +| P10 vs P04 | **PASS** | padding +5.399 pp [1.868, 10.213]; R64 +44.787 pp [43.638, 45.949] | 14.260% | + +This confirms the existential H1b claim on completed contrasts. It cannot +support a null claim about P05 or the missing matrix. + +## Waste accounting + +The preregistered contrast thresholds are 5 percentage points for padding, +10 points for graph miss/overflow, 0.15 absolute for R64, 0.10 for mixed-batch +interference, and 0.15 for MoE layer CV, plus the 5% efficiency/residual +association gate. Values below are clean C00-moderate results; efficiency is +scheduled useful tokens per model-step millisecond. + +| Pattern | Padding | Graph miss | Overflow | R64 | Mixed interference | Efficiency | Supported/total mixed steps | +|---|---:|---:|---:|---:|---|---:|---:| +| P01 | 1.869% | 24.826% | 24.826% | 0.3687 | N/A | 4.9673 | 0/5,646 | +| P02 | 1.998% | 3.837% | 3.837% | 0.3687 | N/A | 2.6664 | 0/1,536 | +| P03 | 0.000% | 1.736% | 1.736% | 0.2444 | N/A | 4.7356 | 0/138 | +| P04 | 0.167% | 1.301% | 1.301% | 0.2444 | N/A | 3.0547 | 0/141 | +| P05 | — | — | — | — | — | — | — | +| P06 | 0.276% | 0.830% | 0.830% | 0.5988 | N/A | 2.3568 | 0/152 | +| P07 | 0.140% | 1.240% | 1.240% | 0.0000 | N/A | 2.7513 | 0/186 | +| P08 | 0.005% | 1.357% | 1.357% | 0.0000 | N/A | 2.1478 | 0/178 | +| P09 | **8.542%** | 4.518% | 4.518% | **0.7648** | N/A | 4.5537 | 0/1,054 | +| P10 | **5.565%** | 0.921% | 0.921% | **0.6923** | N/A | 2.6191 | 0/92 | +| P11 | — | — | — | — | — | — | — | + +No irregular-versus-control graph-miss or overflow contrast reaches the +positive 10-point H1b threshold. P01 has the largest absolute miss rate but is +a rectangular control; P09 actually reduces miss versus P01 by 20.308 points. + +The registered leave-one-pattern-out robust fits were applied for mixed-batch +interference, but no pattern retained 30 supported mixed steps inside both +pure-fit convex supports; every result is N/A rather than extrapolated. No +separate LOAO operator-ranking procedure was preregistered, so none was added +post hoc. The required two-window rule is used unchanged, and confirmation-run +robustness is `NOT EVALUABLE` because all four confirmations are missing. + +### Top three waste findings + +1. **P10 real trace:** R64 is 44.787 points above both long rectangular + controls; padding is 5.565/5.399 points higher; efficiency is 44.693% worse + than P03 and 14.260% worse than P04. +2. **P09 production-shaped mix:** versus P01, R64 rises 39.616 points and + padding 6.673 points with 8.325% lower efficiency. Its still-larger P03 + contrast does not pass because efficiency loss is only 3.840%. +3. **P06 bimodal long-output burst:** R64 rises 23.015 points versus P02 and + 35.440 points versus P04, paired with 11.609% and 22.846% efficiency loss. + +## Pattern-conditioned operational findings + +- **P10/TP2 non-stabilization:** the fresh run completed 17 warm-up requests, + but trailing scheduled-token throughput fell 18,022.4 → 16,384.0 → 14,062.6 + tokens/s. Its 36.76% fitted drift exceeds A-P3-6's 10% limit; same-wave + synthetic P11 and P03 completed 512 and 102 warm-up requests and passed their + applicable registered gates. The orchestrator accepts this as a + pattern-conditioned finding, not an accepted throughput measurement. +- **Long-context drain:** P10/C01 saturation naturally drained for 288.619 + seconds. The clean window and accounting were valid; the original 120-second + short-pattern watchdog was miscalibrated, while the amended 600-second P10 + budget correctly retained the run. +- **Failure boundary:** P01/C01 moderate had 5,828 clean successes and zero + clean failures. Two `ServerDisconnectedError` records occurred only in + excluded post-profile time; separating clean and auxiliary windows prevented + valid data from being discarded. +- **Layer-2 perturbation:** despite 97%+ kernel classifiability, eight of nine + completed moderate patterns failed representativeness or recovery. Together + with the Phase-2 51.3% active-window perturbation, this shows that sparse + Kineto windows are operationally fragile for these serving patterns. + +These incidents consistently show long-context and mixed serving patterns +breaking assumptions calibrated on shorter rectangular loads: fixed drain +watchdogs, completion-count warm-up, global failure scope, and profile-window +representativeness. + +## Matrix and GPU accounting + +| Item | Final value | +|---|---:| +| Planned measured runs | 52 | +| Accepted measured runs | **40** | +| Complete cells | **20/24** | +| Accepted confirmations | 0/4 | +| Accepted clean failures | 0 | +| Drain-quarantined accepted runs | 0 | +| Accepted Layer-2 trace files | 72 | +| Registered missing trace files | 8 | +| Other-user GPU processes | 0 | +| Cumulative GPU use | **14.025875 H20-hours** | +| Reserved/unused headroom | **1.974125 H20-hours** | + +No GPU work was run for A-P3-7 analysis. The reserved headroom remains unused, +and all eight GPUs were at 0 MiB/0% before and after the CPU-only analysis. + +## Data sanity block + +| Numeric family | n | finite | missing | min | max | distinct | Check | +|---|---:|---:|---:|---:|---:|---:|---| +| Accepted throughput (req/s) | 40 | 40 | 0 | 0.445833 | 43.300000 | 37 | finite, positive, not identical | +| Token efficiency (tokens/ms) | 40 | 40 | 0 | 2.147827 | 8.295849 | 40 | finite, positive, distinct | +| Drain duration (s) | 40 | 40 | 0 | 0.305419 | 288.619108 | 40 | non-negative; no quarantine | +| Clean Layer-1 steps/run | 40 | 40 | 0 | 718 | 20,171 | 40 | sum 352,907; continuous per run | +| Operator classifiable fraction | 72 | 72 | 0 | 0.970458 | 0.996378 | 72 | all >=0.70 and within [0,1] | +| Operator-family shares | 576 | 576 | 0 | 0.000000 | 0.873417 | 348 | within [0,1] | +| Waste ratios | 240 | 240 | 0 | 0.000000 | 1.000000 | 168 | within [0,1] | +| Waste-contrast effects | 36 | 24 | 12 expected | -0.203082 | 0.520413 | 17 | signed; N/A preserved | +| Kendall tau-b | 0 | 0 | 0 | — | — | 0 | expected: only one valid pattern | +| Accepted clean failures | 40 | 40 | 0 | 0 | 0 | 1 expected | sum zero | +| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | cleanup passed | + +Checked invariants: accepted IDs unique; exact 40-run/20-cell coverage; exact +four missing cells and two missing contrasts; 240-second clean windows; zero +clean failures; moderate rate within 5%; balanced Layer-1 accounting and zero +drops; trace parsing and classifiability; ratios in range; exact 72+8 trace +accounting; clock/load snapshots present; no other-user process; drain +quarantine below 20%; non-identical pattern results; and exact reproduction of +the 36.76% A-P3-6 operational result. **No data-sanity red flag remains.** H1a +is inconclusive because of declared Layer-2 validity limits; H1b passes on five +complete contrasts; the compound hypothesis is partial and cannot be refuted. diff --git a/docs/opprof/phase4-optimization-plan.md b/docs/opprof/phase4-optimization-plan.md new file mode 100644 index 0000000..610b7f0 --- /dev/null +++ b/docs/opprof/phase4-optimization-plan.md @@ -0,0 +1,306 @@ +# OpProf Phase 4 measured optimization plan + +Status: **PROPOSAL FROM ACCEPTED PHASE-3 DATA; NO UNMEASURED GAIN CLAIMS**. + +Date: 2026-07-12. This plan uses only the accepted Phase-3 40/52-run, +20/24-cell dataset and the single optional Phase-4 capture-size validation. +Phase-3 protocol and results are frozen. Bounds below are ceilings in the +units actually measured; padding or raggedness percentages are not relabeled +as end-to-end throughput gains. + +## Pinned implementation context + +- Model/hardware: Qwen3-30B-A3B BF16, one H20, TP1 primary. +- vLLM source: accepted OpProf tip + `23450fb21ac255b0cf710f4ee965ee694921975d` on v0.24.0. +- vLLM 0.24.0 exposes `--cudagraph-capture-sizes`, + `--max-cudagraph-capture-size`, `--max-num-seqs`, and + `--max-num-batched-tokens` (`vllm/engine/arg_utils.py:1390-1467`). +- An explicit capture-size list replaces the inferred list. The default is + `[1,2,4]`, multiples of eight below 256, then multiples of sixteen through + the maximum, normally 512 (`vllm/config/vllm.py:1669-1792`). +- Chunked prefill remains enabled. vLLM schedules decode first, then fills the + remaining MBT budget with prefill and chunks an over-budget prefill + (`docs/configuration/optimization.md:45-59`). + +The ranking is by measured opportunity, readiness, and downside together—not +by a normalized composite score. Items that share the same raggedness bound +are explicitly non-additive. + +## Ranked optimization list + +| Rank | Tier | Target and affected regime | Measured bound | Owner | Decision | +|---:|---|---|---|---|---| +| 1 | Config now / scheduler backlog | Preserve prefix affinity and prefix caching for P08-like shared-prefix traffic | P08 vs matched P07: **+82.14% saturation req/s**, 80% prefix-hit ratio, **62.29% fewer prefill tokens** | Serving/config owner now; cache-aware dispatch upstream | Deploy behind a workload classifier; do not apply to no-sharing traffic | +| 2 | Scheduler | Length-aware cohorting/admission for ragged P10/P09/P06 | At most **44.79 pp** R64 contrast and **44.69%** measured efficiency gap on P10; 39.62 pp/8.32% on P09; 35.44 pp/22.85% on P06 | Upstream vLLM scheduler | Highest structural backlog item; preserve fairness and arrival semantics | +| 3 | Config now | Add exact small CUDAGraph sizes for P09/P10-like moderate decode batches | Distributional bound **4.98 pp P09 / 5.26 pp P10** padding; P09 validation achieved **4.980 pp** | Serving/config owner | Mechanism confirmed; canary only because p95 was +3.01% in one pair | +| 4 | Config now | Route P06/P10-like pools to MNS=64 | Saturation throughput **+3.37% P06 / +3.70% P10** versus C00 | Serving/config owner | Pattern-specific trial only; same setting was −24.27% on P01 | +| 5 | Kernel backlog | Ragged-aware MoE GEMM and attention over the measured shape stream | Shares the rank-2 ceiling; no independent additive gain. Descriptive MoE share is 54.04–75.00% on P10/P09/P06 moderate | Engine/kernel colleagues | Optimize exact weighted shapes below; require serving confirmation | +| 6 | Config guardrail | Keep MBT=8192 for short/high-throughput and long-prefill classes; do not globally set 2048 | Avoided saturation regressions up to **11.64% P03**, 6.53% P01, 5.82% P10 | Serving/config owner | Encode as a policy guardrail, not a positive optimization claim | + +## Tier A — configuration-level actions deployable today + +### A1. Prefix-affine routing with prefix caching + +**Measured regime.** P07 and P08 have the same 1,280-token prompt length, +512-token output, burst-of-eight arrival, C00 config, and seed. P08 alone uses +eight 1,024-token shared prefixes plus a unique 256-token suffix. With prefix +caching already enabled: + +| Metric, saturation | P07 no sharing | P08 high sharing | Delta | +|---|---:|---:|---:| +| Prefix-query hit ratio | 0.00 | 0.80 | +0.80 | +| Clean prefill tokens | 1,528,968 | 576,512 | **−62.29%** | +| Completed throughput | 5.1083 req/s | 9.3042 req/s | **+82.14%** | + +**Root mechanism.** The matched cell changes only controlled prefix sharing; +the observed hit and prefill-token changes directly identify cache reuse. The +82.14% throughput delta is a point observation and the maximum evidence-backed +gain for this exact regime, not a fleet-wide forecast. + +**Action now.** Keep `--enable-prefix-caching`; hash an application-known +stable prefix or conversation identity to a replica so related requests do +not destroy affinity through round-robin load balancing. Apply only when the +online prefix-query hit ratio resembles P08, not P07. + +**Verification.** Interleave affinity ON/OFF on the same replicas and fixed +request stream. Primary gates are prefix-hit ratio, prefill-token reduction, +completed req/s, TTFT p95, per-replica queue imbalance, and KV occupancy. A +throughput gain with queue/fairness or KV-capacity regression does not pass. + +### A2. Measured CUDAGraph capture sizes + +**Measured regime.** P09 moderate has decode-batch p50/p95/max 4/16/25. P10 +moderate has 1/4/7. Default captures skip sizes 3, 5, 6, 7, and 9, so those +batches pad upward. Replaying the Phase-3 hit distribution predicts that +adding exactly `{3,5,6,7,9}` to the complete default list can remove: + +- **4.9776 percentage points** of P09's 8.5421% graph-hit padding; and +- **5.2579 points** of P10's 5.5652% padding. + +The list must be `default ∪ {3,5,6,7,9}`; passing only five sizes would replace +and discard the rest of the default list. + +**Closed-loop result.** One P09 moderate ON/OFF pair used fresh servers, the +same fixed seed and accepted saturation-rate source, 60 excluded warm-up +seconds, and 240 clean seconds per arm. No Layer-2 profiler ran. + +| Metric | ON: exact sizes | OFF: default | ON relative to OFF | +|---|---:|---:|---:| +| Graph-hit padding | **3.5659%** | 8.5456% | **−4.9798 pp; −58.27%** | +| Useful tokens/model-step ms | 4.56222 | 4.55405 | +0.179% | +| Completed throughput | 4.9583 req/s | 4.9333 req/s | +0.507% | +| Mean E2E latency | 1.6123 s | 1.6812 s | −4.10% | +| p95 E2E latency | 3.9930 s | 3.8763 s | **+3.01%** | +| Clean failures | 0 | 0 | equal | + +The observed padding reduction differs from the Phase-3 bound by only 0.0022 +percentage points, confirming the mechanism. It does **not** establish a broad +performance win: token efficiency and throughput moved less than 1%, p95 moved +the wrong way, and there is one ordered pair with no CI. + +Operational cost also matters: ON captured 56 FULL and 56 PIECEWISE sizes +versus 51/51 OFF, estimated graph memory increased 0.64→0.68 GiB, and server +ownership was 76.9 seconds longer. Deploy only as a pattern-specific canary; +require interleaved replication with a p95 non-regression gate before rollout. + +### A3. Pattern-specific MNS pools + +The complete saturation comparisons show that `--max-num-seqs 64` is an +interaction, not a globally better default: + +| Pattern | C10 MNS=64 vs C00 | Interpretation | +|---|---:|---| +| P01 short/short | **−24.27% req/s** | Reject for dense short traffic | +| P03 long/short | −1.41% | No measured benefit | +| P06 bimodal/long burst | **+3.37%** | Candidate pattern pool | +| P10 real long-context | **+3.70%** | Candidate pattern pool | + +The evidence identifies the config×pattern interaction but not a lower-level +cause. Do not attribute it to a particular kernel or queue effect without a +new bisection. Verification is five interleaved saturation pairs per intended +class plus TTFT, queue depth, preemption, KV usage, and exact-work checks. + +### A4. MBT policy guardrail + +`--max-num-batched-tokens 2048` versus the default 8192 changed saturation +throughput by −6.53% P01, −11.64% P03, +0.35% P06, and −5.82% P10. The combined +MNS64/MBT2048 setting was −30.22% on P01. Phase 3 therefore supports retaining +MBT8192 for these classes and rejects a global MBT2048 rollout. The bound is +an avoided regression, not new speedup. + +## Tier B — upstream scheduler changes + +### B1. Length-aware cohorting without starvation + +**Measured mechanism.** R64 is the rectangular padding fraction of the exact +arrival-order prompt stream. It is 0.6923 for P10, 0.7648 for P09, and 0.5988 +for P06. Their passing control contrasts are: + +| Irregular pattern | Control | R64 excess | Useful-token efficiency loss | +|---|---|---:|---:| +| P10 | P03 | **44.79 pp** | **44.69%** | +| P10 | P04 | **44.79 pp** | **14.26%** | +| P09 | P01 | **39.62 pp** | **8.32%** | +| P06 | P02 | **23.01 pp** | **11.61%** | +| P06 | P04 | **35.44 pp** | **22.85%** | + +These are upper bounds on waste a length-aware path could avoid. R64 is not +observed GPU time, and efficiency association is not causal. The scheduler +change should maintain several ready queues by remaining prompt/context band, +select a less-ragged cohort subject to the existing decode-first token budget, +and impose a finite age/fairness bound. It must not rewrite request arrivals or +drop long requests. + +**Verification.** Add a runtime per-step raggedness counter rather than using +manifest R64 as a surrogate. Compare fixed-arrival ON/OFF runs for useful +tokens/model-step ms, TTFT/E2E p95, queue age, starvation count, preemption, +KV occupancy, and the full length histogram. The gain cannot exceed the +corresponding R64/efficiency bounds above, and it is non-additive with a +ragged-aware kernel. + +### B2. Automatic cache-aware dispatch + +The upstream form of A1 is a scheduler/replica dispatcher that chooses a live +prefix-cache owner while respecting load. Its collaboration contract is the +measured P07/P08 tuple: 1,024 shared + 256 unique prompt tokens, eight prefix +IDs, burst size eight, output 512, target hit ratio 0.80. The load-balancing +penalty and lost cache hits must be reported together; a synthetic cache hit +increase without end-to-end balance is insufficient. + +### B3. Histogram-driven capture-list generation + +Static A2 proves that Layer-1 can choose useful sizes. An upstream controller +could select a bounded number of exact sizes from padding contribution +`count(size) * (next_bucket-size)`, while retaining the default list and a +memory/startup budget. P09's top five `{3,5,6,7,9}` recovered 4.98 points at a +0.04-GiB graph-memory and 76.9-second server-lifetime cost in this run. The +selector must freeze its list before measurement and never continually tune on +the scored window. + +## Tier C — kernel-level backlog and collaboration interface + +H1a is inconclusive, so Phase 3 does not prove a universal top operator. The +available moderate windows are still useful shape inputs: descriptive MoE-GEMM +shares are 57.52% P06, 75.00% P09, and 54.04% P10; attention shares are 31.60%, +16.19%, and 30.61%. Only P04's operator windows pass inference gates, where +attention is 47.88% and MoE GEMM 40.64%. Kernel work must therefore claim +shape-local improvement, not a resolved global bottleneck. + +### Exact shape stream for kernel engineers + +`P`, `D`, and `N` below are per-step prefill tokens, decode tokens, and +scheduled requests. Counts are from clean C00-moderate Layer-1 records. +Context mix is the fraction of scheduled request-context observations in +`<=1024 / 1025–8192 / 8193–32768 / >32768` bins. + +| Pattern | Model steps: decode / mixed / prefill | N p50 / p95 / max | Dominant exact `(P,D,N): count` | Context mix | Chunk signal | +|---|---|---|---|---|---| +| P01 | 5,760: 114 / 5,646 / 0 | 69 / 74 / 77 | `(0,66,66):18`, `(0,67,67):18`; mixed P p50=334, D p50=68 | 100.00 / 0 / 0 / 0% | 6,235 unsplit; sizes 129–512 dominate | +| P04 | 12,455: 12,291 / 141 / 23 | 8 / 8 / 16 | `(0,8,8):12,129`, `(8191,1,3):23` | 0.03 / 94.62 / 5.35 / 0% | 238/319 chunks >2,048; first/final 122/123 | +| P06 | 17,464: 17,310 / 152 / 2 | 8 / 16 / 16 | `(0,8,8):13,103`, `(0,16,16):4,098` | 55.11 / 42.93 / 1.96 / 0% | 174/422 >2,048; 136 in 257–512 | +| P09 | 12,837: 11,783 / 1,054 / 0 | 4 / 16 / 25 | `(0,3,3):3,397`, `(0,2,2):1,549`, `(0,4,4):1,526`, `(0,5,5):1,050` | 51.70 / 48.26 / 0.04 / 0% | 228/1,193 >2,048; 345 in 1,025–2,048 | +| P10 | 18,130: 17,941 / 92 / 97 | 1 / 4 / 7 | `(0,1,1):13,572`, `(0,2,2):2,675`, `(0,3,3):792`, `(0,4,4):524`, `(8192,0,1):38` | 12.25 / 39.94 / 47.76 / 0.05% | 138/191 >2,048; first/middle/final/unsplit 49/29/49/64 | + +The exported kernel-benchmark interface should be a prompt-free weighted table +with `(P,D,N,context_bin,chunk_class,chunk_size_bin,runtime_mode,count)` plus +step duration and useful tokens. Use the frozen histogram edges already emitted +by Layer 1: context `128..131072` powers of two and chunk `16..2048` powers of +two. Preserve the joint tuples; independent marginal sampling would erase the +mixed-batch structure. + +### Kernel targets and acceptance + +1. **Ragged MoE GEMM:** accept variable token counts without padding every + expert/layer tile to the largest sequence. Weight microbenchmarks by the P06, + P09, and P10 tuples above. The ceiling is the same R64/efficiency opportunity + as B1, not an additional gain. +2. **Attention:** retain P04 `(D,N)=(8,8)` as the valid long rectangular control + and test P10's mostly 1–4 decode batches plus 8,192-token chunks. Report + useful-token time, workspace, and graph compatibility. +3. **Serving confirmation:** kernel time must improve on the exact weighted + stream, then pass a fixed-arrival serving A/B for throughput, TTFT/p95, + memory, and correctness. A rectangular-only kernel win does not close the + Phase-3 finding. + +`moe_expert_load` was unavailable in Phase 3. No expert-imbalance mechanism or +gain is claimed; expert-specific packing requires a new low-overhead route +histogram before implementation. + +## Honest limits and Phase-5 measurement requirement + +- H1a remains inconclusive: only P04 had two representative/recovered operator + windows. Eight other completed moderate patterns failed window validity even + though kernel classifiability was 97.05–99.64%. +- A Phase 5 operator study needs longer, time-stratified samples that reproduce + clean scheduled-token, prefill-fraction, decode-batch, and graph-mode + distributions, plus a lower-perturbation per-op timer. It must demonstrate + overhead before using shares for optimization; the Phase-2 Kineto active + window perturbed throughput by 51.3%. +- Confirmation runs are absent. The MNS and MBT config effects are single-run + point estimates and require replication before production decisions. +- R64 is an offline rectangular-padding upper bound, not measured GPU idle + time. H1b's efficiency association does not establish causality. +- Mixed-batch interference was N/A because no cell retained 30 supported mixed + steps inside both leave-one-pattern-out pure-fit supports. +- Results cover one model, BF16, H20, mostly TP1, and 20/24 cells. P03/C11, + P05/C00, P10/C00-TP2, and P11/C00 are absent. +- The capture validation is one ordered ON/OFF pair. Its padding endpoint is + mechanism-valid, but performance deltas have no CI and p95 regressed. +- Layer 1 did not collect expert-route identities; kernel engineers cannot infer + routed-expert imbalance from these artifacts. + +## Verification and stop rules for Phase 4 work + +Every proposed experiment keeps the original fixed manifest/seed/work, excludes +warm-up, records Layer-1 accounting, and changes one mechanism. A candidate +stops on clean failure, footer imbalance/drop, output mismatch, GPU +contamination, memory regression beyond its declared budget, or violation of +the 16-H20-hour campaign cap. Throughput, latency, memory, and correctness are +reported together; no metric shopping or silent pattern substitution is +allowed. + +## GPU accounting + +The optional capture pair consumed **0.296389 H20-hours**, taking cumulative +campaign use from 14.025875 to **14.322265 H20-hours**. Remaining headroom is +**1.677735 H20-hours**. Both arms returned GPU0 to zero, all eight GPUs were +0 MiB/0% at final inspection, and no other-user process appeared. + +Artifacts are under +`runs/opprof-phase3/phase4/capture-p09/`; `result.json` SHA-256 is +`5bb91df28790f6f3c34e4e9ed8e35a1cb8100f93086a4286689d587fd732f2a4`. + +## Final ranked one-liners + +1. **Prefix affinity (config now):** P08's measured ceiling is **+82.14% req/s** with 62.29% fewer prefill tokens versus matched P07. +2. **Length-aware scheduler:** raggedness ceiling is **44.79 pp R64 / 44.69% efficiency gap** on P10; smaller confirmed bounds apply to P09/P06. +3. **Exact capture sizes (config now):** ceiling **5.26 pp P10 / 4.98 pp P09 padding**; P09 validation removed 4.980 pp but did not prove p95 gain. +4. **MNS64 pattern pools (config now):** measured ceiling **+3.70% req/s P10 / +3.37% P06**, with a −24.27% P01 counterexample. +5. **Ragged kernels (kernel backlog):** share rank 2's bound; no additive E2E bound is supported while H1a is inconclusive. +6. **MBT8192 guardrail (config now):** avoids measured regressions up to **11.64%**; MBT2048 has no general positive case. + +## Data sanity block + +| Numeric family | n | finite | missing | min | max | distinct | Invariant/result | +|---|---:|---:|---:|---:|---:|---:|---| +| Ranked items | 6 | 6 | 0 | rank 1 | rank 6 | 6 | Three tiers represented; bounds not summed | +| Sentinel saturation config deltas | 11 | 11 | 0 | −30.216% | +3.704% | 11 | Both gains and regressions retained | +| Passing R64 contrast effects | 5 | 5 | 0 | 0.230148 | 0.447872 | 4 | Ratios in [0,1]; duplicate P10 controls expected | +| Capture-arm padding fraction | 2 | 2 | 0 | 0.035659 | 0.085456 | 2 | Non-negative; ON < OFF | +| Capture-arm token efficiency | 2 | 2 | 0 | 4.554051 | 4.562222 | 2 | Positive; +0.179% ON | +| Capture-arm throughput (req/s) | 2 | 2 | 0 | 4.933333 | 4.958333 | 2 | Same offered rate 4.920833 req/s | +| Capture-arm clean failures | 2 | 2 | 0 | 0 | 0 | 1 expected | Exact 240 s and zero failures | +| Capture-arm Layer-1 records | 2 | 2 | 0 | 16,491 | 17,579 | 2 | Every footer/sidecar invariant true; zero drops | +| Optional validation GPU-hours | 1 | 1 | 0 | 0.296389 | 0.296389 | 1 | Positive; cumulative 14.322265 < 16 | +| Final GPU memory (MiB) | 8 | 8 | 0 | 0 | 0 | 1 expected | Cleanup passed | + +Checked invariants: Phase-3 metrics remain frozen; every cited number resolves +to accepted metrics or the checksum-recorded validation; config comparisons +use saturation rather than normalized moderate throughput; padding/raggedness +bounds are not presented as throughput; duplicate/non-independent bounds are +not added; both validation arms use identical work and offered rate; clean +failures are zero; output work, Layer-1 schema, step continuity, footer/sidecar +balance, and zero drops pass; ratios lie in their declared domains; all GPU +memory returned to zero; and cumulative GPU use stays below 16 H20-hours. No +data-sanity red flag remains. diff --git a/docs/opprof/phase5-protocol.md b/docs/opprof/phase5-protocol.md new file mode 100644 index 0000000..db76a3b --- /dev/null +++ b/docs/opprof/phase5-protocol.md @@ -0,0 +1,640 @@ +# OpProf Phase 5 pre-registered mechanism-decomposition protocol + +Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**. + +Date frozen: 2026-07-12 (Asia/Singapore). This document specifies Phase 5 only. +It does not authorize a GPU launch, helper implementation, trace transfer, or +change to the accepted vLLM patch series. Any change to an estimand, request +set, service order, arrival transform, capture-size set, load, validity gate, +or decision threshold requires a dated amendment before the affected run. + +## Approved dispositions (orchestrator; 2026-07-12) + +All five open decisions below are approved before execution. These dispositions +are normative and close the protocol review gate without changing any frozen +estimand, command delta, validity threshold, or budget: + +1. The recorded-arrival **bridge ledger** is approved. Every machine and human + output must state that it decomposes the recorded-arrival P5 gap anchored to + P3 controls, not literally P3's already-uniform P10 gap; A3 supplies the + explicit bridge back to P3. +2. The dual P03/P04 control ledgers are approved. Both are always reported and + a dominant-mechanism call must pass the frozen rule under both denominators. +3. A1 is approved exactly as specified: 142 requests, 32-request reorder + blocks, 16-request analysis cohorts, frozen bins, and a 64-second fairness + cap. +4. Three P10 replicates per arm, P3 control reuse behind the 3% bridge gate, + the conditional control reruns, and optional-tier ordering under the 6.0 + H20-hour hard cap are approved. +5. Layer-1-only primary measurement is approved. Routed-expert telemetry is + analysis-only, private, optional, and never receives a causal ledger share. + +This approval authorizes the later execution turn only when its preflight, +echo-before-launch, detached-controller, long-context, privacy, accounting, +cleanup, and budget gates all pass. + +## Amendment A-P5-1 — rate-following cold-start gate (orchestrator; 2026-07-12) + +The first Phase-5 wave correctly hard-stopped because all four offered-load +arms failed the inherited A-P3-6 throughput-drift gate: recorded base/A4 drift +was approximately 216.5%, A1 was 190.6%, and uniform A3 was 13.23%, versus the +frozen 10% limit. Their 240-second clean windows, output work, offered rates, +Layer-1 accounting, and drains otherwise passed. The gate was semantically +wrong for these arms: a rate-following run's scheduled-token throughput follows +its arrival process by design, so recorded non-stationarity is treatment signal, +not cold-start contamination. The large recorded-versus-uniform drift gap is +retained as direct arrival-mechanism evidence. Throughput-drift stationarity +remains appropriate and unchanged for saturation arms. + +For every **rate-following/offered-load** arm, A-P3-6 is replaced by all three +of the following cold-start-artifact gates: + +1. Every logged torch.compile or CUDA-graph capture first-occurrence event must + precede the clean boundary. Match server-log event messages containing + `torch.compile took`, `Directly load AOT compilation`, `Compiling`, or + `Capturing CUDA graphs` (case-insensitive). Timestamped events are compared + against `t0_wall_ns + 60 s`. vLLM's capture progress lines have no timestamp; + they count as pre-client only when their log-line order precedes the server + ready/startup-complete marker, which itself precedes client `t0`. Any matching + event after readiness must have a parseable timestamp and precede clean; + otherwise the run is invalid. +2. At least 16 requests must complete successfully in `[0,60 s)`, including at + least one request whose recorded `input_tokens >= 8192`. +3. No first-occurrence capture event may appear inside `[60,300 s)`. Parse the + configured startup capture-size set and the completed FULL/PIECEWISE startup + capture passes from the server log. From Layer 1, define a captured replay + descriptor as `(runtime_mode,bucket_tokens)` for every model-executed + `cudagraph.hit=true` step. Every clean descriptor must be covered by the + startup-captured mode and bucket set, and no server-log compile/capture event + may occur inside clean. Warm and clean descriptor sets are both reported; + a descriptor's first **replay** in clean is not mislabeled as a first capture + when startup logs prove it was already captured. An uncovered descriptor or + clean-window capture/compile event invalidates that run only. + +The report records matched server-log events, warm-up completion/long-request +counts, warm-up and clean descriptor sets, and clean-only descriptors per run. +Absence of any required log timestamp, Layer-1 interval, or request record is a +gate failure. The original 10% A-P3-6 drift criterion remains mandatory for +closed-loop saturation arms. This amendment changes no request set, arrival +transform, service order, server configuration, clean interval, metric, +bootstrap, share estimator, dominance rule, control-reuse gate, or GPU budget. + +## Goal, system boundary, and success criterion + +Phase 3 measured a total useful-token-efficiency gap between irregular patterns +and rectangular controls but did not causally allocate it. Phase 5 asks how much +of the P10 gap is recovered when one treatment at a time removes: + +1. intra-cohort input-length raggedness; +2. CUDA-graph decode-batch capture-bucket mismatch; +3. recorded arrival burstiness; or +4. usable natural-prefix structure. + +The primary system remains Qwen3-30B-A3B BF16, patched vLLM 0.24.0, C00, TP1, +one H20 per server on dash0. The primary load is the P3 P10 rate +`lambda = 0.60 * 0.7875 = 0.4725 request/s`; it is held fixed across arms so an +ablation changes one treatment, not offered demand. The 240-second clean window +and Layer-1 definition of + +```text +E_token = sum(prefill_tokens + decode_tokens) / sum(model-step duration_ms) +``` + +are inherited unchanged. Layer 2 is not needed for the causal ledger and is not +enabled in primary throughput runs. + +Success is a mechanism ledger with an absolute `E_token` for every arm, an +un-normalized share and bootstrap confidence interval for every mechanism, and +an explicit residual/interaction line. A null or negative share is a valid +result. Merely recovering the expected sign is not success. + +## Pinned Phase-3 evidence and the arrival-estimand discrepancy + +The frozen P3 C00-TP1 moderate values are: + +| Cell | `E_token` (tokens/ms) | Role | +|---|---:|---| +| P10 | 2.6191132083 | irregular P3 base | +| P03 | 4.7355997154 | long-input/short-output rectangular control | +| P04 | 3.0547035940 | long-input/long-output rectangular control | + +P10 therefore lost 44.693% versus P03 and 14.260% versus P04. P3 used both +controls, so Phase 5 reports two parallel ledgers, one per frozen control. It +never selects the denominator that makes a mechanism look largest. A mechanism +is called control-robust only when its decision agrees under both ledgers. + +There is one blocking provenance discrepancy. The private source contains +`timestamp` and `source_timestamp`, but P3's materializer discarded them and +set every P10 row to `arrival=steady`; the P3 client then admitted requests at +exactly `1/lambda`. Thus P3 has no recorded-arrival burstiness to remove. + +The recommended resolution, pending orchestrator approval, is a **P5 bridge +ledger**: the P5 base replays the same P10 requests at rate-normalized recorded +timestamps, A3 uniformizes those timestamps, and P03/P04 remain the frozen P3 +controls. A3 also acts as a bridge back to the P3 steady workload. This meets +the requested arrival ablation but decomposes a recorded-arrival P5 gap, not +literally the already-uniform P3 gap. The report must show both +`E_A3 - E_P3_P10` and its CI before relating the P5 ledger to P3. + +If the orchestrator rejects this rebase, the P3-exact alternative is mandatory: +A3's share is `0 / N/A by construction`, and recorded arrival is reported only +as a stress sensitivity, not as a mechanism share. It is scientifically invalid +to call the reverse, burstiness-injecting treatment an ablation that removes +arrival dynamics. No GPU work may begin before this decision is recorded. + +## Common request set and exact arrival transforms + +The primary request set is the first **142** rows of the frozen 4,011-row P10 +selection in original source order. This is exactly the number of admissions at +`lambda=0.4725` over `[0,300 s)`: scheduled times are +`0, 1/lambda, ..., 141/lambda`. All five arms contain the same 142 request IDs, +prompts, per-request input/output lengths, and aggregate input/output token +totals. There is no wrap, replacement, cancellation, or resampling. + +The Phase-5 materializer must preserve `timestamp` as private metadata. Let +`z_i` be its stable source-order timestamp for request `i`, and let `N=142`. +The two arrival vectors are frozen as: + +```text +recorded-scaled: a_i = (z_i-z_0) * ((N-1) / (lambda*(z_(N-1)-z_0))) +uniformized: a_i = i / lambda +``` + +`z_(N-1)` must exceed `z_0`; timestamps must be finite and nondecreasing. +Ties remain ties. Both vectors start at zero, end at `141/lambda`, have the same +mean rate, and use the same request order except in A1. The client schedules +against `a_i` directly and does not add jitter. This preserves the recorded +inter-arrival shape while preventing mean-rate differences from masquerading as +an arrival mechanism. + +The protocol requires a small `scripts/opprof_phase5_client.py` extension in a +later, no-GPU implementation turn. Before execution, CPU-only tests must prove: + +- exact 142-row identity and token sums across all manifests; +- timestamp normalization endpoints and nonnegative gaps; +- uniform gaps equal `1/0.4725` within 1 microsecond; +- the A1 fairness bound and deterministic ordering; +- fixed 60+240-second timing, no wrap, exact output work, and text redaction. + +Its reviewed SHA-256 and all manifest SHA-256 values are frozen in the detached +controller before GPU use. + +## Falsifiable mechanism estimands + +For arm `m` and rectangular control `c in {P03,P04}`: + +```text +gap_c = E_control,c - E_base +delta_m = E_ablated,m - E_base +share_m,c = delta_m / gap_c +``` + +The same base and same control are used for all four mechanisms within a +ledger. No share is clipped to `[0,1]`. Shares may be negative, exceed one, or +sum above/below one because single-factor interventions can overlap or interact. + +The arithmetic residual/interaction line is + +```text +share_residual+interaction,c = 1 - sum_m share_m,c +``` + +with a joint bootstrap CI. This is bookkeeping, not proof that the remainder is +one separable mechanism. It may contain unmeasured mechanisms, non-additivity, +double-counting, MoE routing, chunked-prefill interference, and measurement +error. Individual shares are never renormalized to total 100%; the residual is +never clipped to make the table visually close. + +### A1 — length-binned service order + +**Hypothesis.** Length-homogeneous local cohorts reduce ragged-attention/SM +imbalance, so `E_A1 > E_base`. The hypothesis is falsified if the registered +manipulation check fails or the Holm-corrected efficiency contrast is not +positive. + +Starting from consecutive **32-request reorder blocks** in original P10 order, +assign each request to the fixed input-length bins + +```text +[0,512], [513,1024], [1025,2048], [2049,4096], +[4097,8192], [8193,16384], [16385,32768] +``` + +and stable-sort each block by `(bin_id, input_tokens, original_index)`. This +creates two more homogeneous 16-request analysis cohorts per complete reorder +block. Assign the sorted requests to the block's unchanged recorded-scaled +arrival slots. If that assignment would delay any request by more than **64 +seconds** relative +to its original slot, choose the earliest-deadline request first until all +deadlines are feasible, then resume the length order. Early movement is allowed; +late movement is capped. Ties are stable and no sorting crosses a 32-request +block. + +On consecutive complete 16-request cohorts of evaluation-slice service order, +define `R16 = 1 - sum(L_i) / sum(16*max_cohort(L_i))`; the incomplete final +cohort is excluded and reported. Frozen pre-run values are base `R16=0.641744` +and sorted `R16=0.473409`, a 0.168334 absolute (26.23% relative) reduction; +plain sorting's maximum added delay is 62.744 seconds. The manipulation passes +only if regenerated values match these within `1e-6`, `R16` falls by at least +20% relative and 0.15 absolute, and no request violates the 64-second delay +bound. +Arrival-slot timestamps, request/content multiset, per-request output lengths, +total tokens, server config, and prefix-caching setting are identical to base. + +A1 estimates the total effect of changing cohort composition. It does not +claim to isolate a particular attention kernel: service order can mediate +decode-batch composition, chunked-prefill mixing, cache locality, and +content-bound MoE routing. If the prefix-query hit ratio changes by more than +one percentage point, or the normalized inter-arrival vector changes at all, +the arm is labeled confounded and has no publishable raggedness share. + +### A2 — measured decode-B capture sizes (config-tier deliverable) + +**Hypothesis.** Exact capture sizes for P10's observed pure-decode batch support +remove decode-bucket slack, so pure-decode padding falls and `E_A2 > E_base`. +No recovery falsifies the efficiency hypothesis; failure to remove the targeted +padding invalidates the ablation rather than supporting a null mechanism. + +The P3 P10/C00/rho=0.60 clean Layer-1 stream has SHA-256 +`51ad4be12178da91d2af484d0946a2274afd3bcbbee33f37940cfe0ff2ea7fa7`. +Its 17,941 pure-decode steps have the exact decode-B histogram: + +| B | 1 | 2 | 3 | 4 | 5 | 6 | 7 | +|---:|---:|---:|---:|---:|---:|---:|---:| +| Steps | 13,572 | 2,675 | 792 | 524 | 161 | 202 | 15 | + +Defaults already contain 1, 2, 4, and 8. A2 therefore adds exactly +`{3,5,6,7}` and freezes the complete server list as: + +```text +1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 +136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 +272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 +``` + +This covers 100% of P3's observed P10 pure-decode B support. The A2 +manipulation requires at least 99% support coverage in the new clean runs and a +90% reduction in pure-decode padding tokens versus base. It targets decode +capture-bucket mismatch; it does not remove prefill-token padding, eager +overflow, graph launch overhead, or length raggedness itself. Capture startup +time and memory are reported as config cost, not treated as free. + +### A3 — recorded arrival to uniform arrival + +**Hypothesis.** Uniformizing the same rate and request order reduces burst-driven +decode-batch/queue variance, so `E_A3 > E_base`. The mechanism is falsified if +the clean 5-second decode-B CV and waiting-queue CV do not fall, or if the +Holm-corrected efficiency contrast is not positive. + +A3 changes only `recorded-scaled` arrival slots to `i/lambda`. Content, order, +input/output lengths, aggregate tokens, server config, prefix caching, and +capture sizes are unchanged. It estimates the total service effect of arrival +shape, including its legitimate downstream changes to batching and queueing. +It does not isolate a scheduler instruction cost. Under the P3-exact fallback, +this arm is the base-equivalent bridge and its arrival share is N/A as described +above. + +### A4 — natural prefix caching disabled + +**Hypothesis.** Direction is deliberately two-sided. Natural P10 reuse may +increase `E_token` through KV reuse, in which case disabling it gives a negative +share; alternatively, low-value/fragmented cache structure may impose overhead +or alter batching, giving a positive share. Either direction is publishable. + +A4 omits only `--enable-prefix-caching`. Prompts, natural repeated-prefix +structure, recorded arrival slots, request order, token totals, scheduler limits, +and capture sizes remain identical to base. Required manipulation checks are +zero local prefix cache hits/queries in the disabled arm and unchanged prompt +hashes. This estimates the contribution of exploiting C structure, not the +intrinsic content similarity of the prompts. + +### Mechanisms without a clean ablation + +MoE routing skew cannot be removed while preserving P10 content and model +semantics: changing tokens, router weights, top-k, or expert placement changes +more than routing skew. It receives no causal share. + +If budget remains, one **analysis-only**, separately started P10 sample may add +`--enable-return-routed-experts`. It is excluded from every `E_token` clean +window and ledger numerator. Offline analysis reports per-layer expert-count +entropy, Gini coefficient, coefficient of variation, max/mean load, and their +association with same-step token-normalized duration. The arrays remain private. +This telemetry is sampled and perturbing, can consume a large scheduler-side +buffer, and provides correlation rather than causal attribution; it can only +help interpret the residual/interaction line. Phase 3's MoE layer-duration CV +was N/A, so it cannot substitute for these routed-expert counts. + +## Exact commands and config deltas + +The following interface is normative for the later implementation. Variables: + +```bash +P5C='python scripts/opprof_phase5_client.py' +PRIVATE=/home/admin/cpfs/wjh/opprof-phase5-private/manifests +P3PRIVATE=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl +P3SOURCE=/home/admin/cpfs/wjh/opprof-phase3-private/trace_windows/chat_w20260311_1000.jsonl +MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B +RATE=0.4725 +``` + +Materialize the five private manifests without printing prompt text: + +```bash +$P5C transform --in "$P3PRIVATE" --take-first 142 \ + --timestamp-source "$P3SOURCE" --join-key source_index \ + --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \ + --service-order original --out "$PRIVATE/P10-base.jsonl" +$P5C transform --in "$P3PRIVATE" --take-first 142 \ + --timestamp-source "$P3SOURCE" --join-key source_index \ + --timestamp-field timestamp --arrival recorded-scaled --target-rate "$RATE" \ + --service-order length-binned --reorder-block-size 32 \ + --analysis-cohort-size 16 \ + --length-bin-edges 512,1024,2048,4096,8192,16384,32768 \ + --max-added-delay-seconds 64 --out "$PRIVATE/P10-A1.jsonl" +$P5C transform --in "$P3PRIVATE" --take-first 142 \ + --timestamp-source "$P3SOURCE" --join-key source_index \ + --timestamp-field timestamp --arrival uniform --target-rate "$RATE" \ + --service-order original --out "$PRIVATE/P10-A3.jsonl" +ln -s P10-base.jsonl "$PRIVATE/P10-A2.jsonl" +ln -s P10-base.jsonl "$PRIVATE/P10-A4.jsonl" +``` + +The symlinks make unchanged request bytes explicit; the controller hashes the +resolved content and requires A2/A4 hashes to equal base. A1/A3 require equal +sorted request-ID sets and equal input/output token sums. + +Common server command (`ARM` is `base`, `A1`, `A2`, `A3`, or `A4`): + +```bash +taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU" \ + VLLM_OPPROF_DIR="$RUN_DIR/opprof" \ + vllm serve "$MODEL" --host 127.0.0.1 --port "$PORT" \ + --tensor-parallel-size 1 --enable-chunked-prefill \ + --enable-prefix-caching --shutdown-timeout 600 +``` + +- A2 adds + `--cudagraph-capture-sizes 1 2 3 4 5 6 7 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512`. +- A4 removes `--enable-prefix-caching`. +- Base, A1, and A3 have no server delta. + +Every moderate client command is: + +```bash +taskset -c "$CPUSET" $P5C run \ + --manifest "$PRIVATE/P10-$ARM.jsonl" \ + --base-url "http://127.0.0.1:$PORT" --model "$MODEL" \ + --load-point moderate --fixed-request-rate "$RATE" \ + --max-concurrency 256 --ignore-eos --temperature 0 \ + --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 \ + --post-clean-seconds 0 --drain-timeout-seconds 600 \ + --workload-seed 20260712 --server-seed 20260712 \ + --result-dir "$RUN_DIR/client" +``` + +There are no profiler endpoint calls. Exact commands, effective config, startup +capture list, compile-cache key, hashes, clocks, host load, and GPU process list +are recorded per run. + +## Execution and validity discipline + +### Placement, order, and detached ownership + +A-P3-1 already rejected eight-way and authorized four-way placement. Phase 5 +therefore uses at most four simultaneous TP1 servers, GPU0--GPU3, with the +frozen disjoint CPU masks `0-19`, `20-39`, `40-59`, and `60-79`. Topology must +still match P3. A changed host topology, source/patch/runtime hash, or clock +policy invalidates reuse of the placement gate and stops for review. + +Three independent replicates are run for each of the five primary arms (15 +measured runs). Sort all `(replicate,arm)` assignments by SHA-256 of +`"20260715::"`, pack them into waves of 4, 4, 4, and 3, and +rotate GPU assignments. The final three-target wave uses the frozen P06/C00 +saturation background in the fourth slot so measurements retain the validated +four-way host regime. No wave contains two replicates of the same arm; if the +hash order would do so, stable-swap the later item with the first legal item and +record the resolved order before launch. Background data are not analyzed. + +As required by A-P3-2, a dash0-resident `setsid`/`nohup` controller owns every +process, has `--resume`, atomically records state, and skips only fully validated +runs with matching hashes. Interactive SSH never owns a server. Before each +wave it logs one echo line with resolved arms, GPUs/CPUs, manifests, rate, +paths, reserved H20-hours, expected duration, and disk headroom. + +Use three unmeasured 60-second burn-ins before measured arms: common C00, +A2 capture-list C00, and prefix-cache-off C00. They warm compile/AOT artifacts +but are not measurements. Per-run-unique OpProf directories must remain ignored +by vLLM compile factors, preserving the accepted Phase-2 fix. + +### Long-context-safe warm-up and drain gates + +These gates are active from the first run; no short-context default is tried +first. + +- Warm-up is exactly 60 seconds and excluded. P10 passes with at least 32 + successful warm-up completions, **or** at least 16 completions plus the exact + A-P3-6 stabilization test: model-executed steps in `[45,50)`, `[50,55)`, and + `[55,60)`; at least 16 steps/bin; positive scheduled-token rates `R_j`; and + OLS drift `abs(slope)*15/mean(R_j) <= 0.10`. Missing bins, discontinuity, or + accounting failure invalidates the run. +- Drain timeout is 600 seconds. A timeout marks the run drain-quarantined but + does not invalidate an otherwise valid clean window. More than 20% of primary + runs quarantined stops Phase 5 for review. +- The clean window is exactly three contiguous 80-second segments. Admission + and completion accounting follows P3. Offered rate must be within 5% of + 0.4725 request/s, with zero clean failures and exact output tokens. +- Layer-1 JSONL/footer/sidecar accounting, zero drops, contiguous steps, no + profile leakage, clock/load capture, other-user-process absence, and final + zero GPU memory are hard gates inherited from P3. + +No semantic failure is retried with altered parameters. One exact retry is +allowed for an infrastructure artifact failure, and both attempts remain in +the operational findings. + +## Control reuse, secondary scope, and budget + +P03/P04 P3 controls may be reused because they already use the same model, +patch, C00-TP1 config, normalized load, 240-second clean metric, placement +regime, and Layer-1 schema. Reuse avoids spending GPU time without changing the +denominator. + +Reuse is valid only if source, patch, model, runtime, clocks, and placement +hashes match and fresh A3-uniformized P10 differs from frozen P3 P10 by at most +3% in `E_token` with a bootstrap CI containing zero difference. If this bridge +gate fails, temporal/runtime drift is plausible: rerun P03 and P04 under their +exact P3 manifests and commands, three replicates each, before computing any +share. A failed bridge never licenses rescaling old controls. + +After the complete P10 ledger, optional work is ordered as follows and starts +only if the controller's conservative reservation remains below the 6.0 +H20-hour hard cap: + +1. one saturation run per P10 arm, using the full 4,011-row manifest, for + descriptive mechanism persistence only; +2. one five-arm moderate ledger each for P09 and P06, reported as exploratory + within-run-bootstrap evidence, not equal replication to P10; and +3. one routed-expert analysis-only sample. + +For secondary A2, the precomputed P3 pure-decode support additions are P09 +`{3,5,6,7,9,10,11,12,13,14,15,17,18,19,20,21,22,23,25}` and P06 +`{3,9,10,11,12,13,14,15}`; default sizes are retained. P09 A3 is a no-change +steady negative control, while P06 A3 changes its registered burst-8 arrivals +to uniform spacing at the same mean rate. P09/P06 conclusions are always +labeled secondary. + +Expected accounting, including server-owned startup/shutdown time: + +| Tier | New measured runs | Expected H20-hours | Expected 4-way wall | +|---|---:|---:|---:| +| P10: 5 arms x 3 replicates | 15 | 1.6-1.9 | 35-55 min | +| Three compile burn-ins | 0 | 0.1-0.2 | 3-6 min | +| Conditional P03/P04 reruns | 6 | 0.6-0.8 | 15-25 min | +| Optional P10 saturation | 5 | 0.5-0.8 | 15-25 min | +| Optional P09/P06, 5 arms each | 10 | 0.9-1.2 | 20-35 min | +| Optional routed-expert sample | 1 analysis-only | 0.1-0.2 | 5-10 min | +| **Maximum planned** | **36 ledger + 1 analysis-only** | **3.8-5.1 expected** | **about 1.5-2.5 h** | + +The hard cap is **6.0 H20-hours new Phase-5 spend**, not a target. Actual time +while a server owns GPU memory is charged. Optional tiers are skipped rather +than overrunning the cap. The 600-second drain allowance is a watchdog, not an +assumption in the expected estimate; repeated long drains consume the optional +budget first. + +With no Kineto traces, primary P10 artifacts are estimated at 0.4-0.6 GB and +all planned public artifacts below 1.5 GB. Stop if public Phase-5 artifacts +exceed 3 GB or CPFS free space falls below 100 GB. Prompt-bearing manifests and +routed-expert arrays remain in mode-0700 private storage and are not counted as +public deliverables. + +## Statistical analysis and decision rules + +Use 5-second moving-block bootstrap over clean time, 100,000 resamples, seed +20260716. For P10, first resample the three run IDs, then resample 5-second +blocks within each selected run; arms and reused P3 controls are resampled +independently. Every `E`, `delta`, share, share sum, and residual/interaction +gets a percentile 95% CI. Absolute `E` and delta accompany every ratio. + +Bootstrap ratio draws are never deleted because their denominator is +inconvenient. If either control gap has a point estimate `<=0`, its CI includes +zero, or more than 5% of bootstrap denominator draws are `<=0`, that control's +ledger is **INCONCLUSIVE (unstable denominator)**. + +The confirmatory family is the four two-sided tests of `E_ablated-E_base=0` on +P10. Apply Holm correction at family-wise alpha 0.05 across A1--A4. The same +corrected delta test serves both control ledgers; duplicating denominators does +not create eight tests. A1--A3 only support the expected recovery claim when +their corrected contrast is positive. A4 may be significant in either +direction. Manipulation-check failure makes the corresponding share N/A even +if efficiency changes. + +A mechanism is **dominant** only if, under both P03 and P04 ledgers: + +- point `share >= 0.30`; +- the 95% share CI excludes 0.15 on the high side (`CI_low > 0.15`); and +- its Holm-corrected efficiency contrast is significant in the expected + direction (two-sided for A4). + +Meeting the rule under only one control is reported as **control-sensitive**, +not dominant. + +The primary ledger is **publishable** when all five P10 arms have three valid +replicates, both control denominators are stable, all four manipulation checks +are evaluable, every share/residual has a finite CI, the fresh A3/P3 bridge is +reported, and no privacy/data-sanity red flag exists. It may be publishable +with no dominant mechanism. It is **inconclusive** if any primary arm is +missing, a denominator is unstable, the arrival-rebase decision is unresolved, +or two or more share CIs have width greater than 0.50. One failed mechanism +manipulation yields a publishable partial ledger only if that line is explicitly +N/A and the headline claim excludes it. + +No additive causal claim is made from `sum(share_m)`. Pairwise and higher-order +interactions are not identified by this one-factor-at-a-time matrix; a combined +all-off arm would be a new experiment requiring amendment, not an improvised +way to close the ledger. + +## Deliverable and artifact contract + +Execution, if later approved, must produce: + +- `docs/opprof/phase5-results.md`; +- `runs/opprof-phase5/phase5/metrics.json` with schema, arm/run values, + bootstrap draws' seed and summary, Holm results, dual-control ledgers, + residual/interaction, gates, and GPU accounting; +- per-run exact commands, environment/provenance, client records, Layer-1 + stream/footer/sidecar, monitor data, and machine-readable sanity JSON; and +- an **Operational findings** section covering warm-up stabilization, drains, + compile/capture startup cost, contamination, retries, and any mismatch between + expected and realized mechanism removal. + +Every raw-run summary, aggregate table, and final metrics file ends with the P3 +sanity schema: `n`, finite/missing, min, max, distinct count, and applicable +invariants. Red flags are reported first and stop inferential analysis. Public +artifacts may contain request IDs, hashes, lengths, counts, and timing only; +prompt, messages, content, generated text, source substrings, and routed-expert +arrays are forbidden. + +## Final ablation table + +| Mechanism | What changes | What is preserved | +|---|---|---| +| Base | Recorded P10 timestamps are rate-normalized and replayed in source order | Frozen 142 requests, content/tokens, C00-TP1, prefix cache on, default capture list, `lambda=0.4725` | +| A1: length raggedness | Stable length-bin sort within 32-request reorder blocks into 16-request cohorts, 64-second late cap | Requests/content/token totals, arrival-slot vector, output lengths, server config, prefix setting | +| A2: capture mismatch | Add exact P10 decode-B sizes `{3,5,6,7}` to the full default capture list | Manifest/order/arrivals/content/tokens, scheduler limits, prefix setting | +| A3: arrival dynamics | Recorded-scaled slots become uniform `i/0.4725` slots | Request order/content/tokens, mean rate, server/capture/prefix config | +| A4: prefix structure | Remove `--enable-prefix-caching` | Natural prompt structure, request order/arrivals/content/tokens, capture/scheduler config | +| Residual + interactions | No extra run; joint arithmetic remainder `1-sum(shares)` | Raw unnormalized mechanism shares; no clipping or forced 100% allocation | + +## Final run count and GPU estimate + +Primary commitment: **15 new measured P10 runs plus three unmeasured burn-ins, +1.7-2.1 expected H20-hours, 35-60 minutes four-way wall, and 0.4-0.6 GB public +disk**. Conditional control reruns add 6 runs; all optional tiers bring the +maximum to **36 ledger runs plus one analysis-only run, 3.8-5.1 expected +H20-hours, about 1.5-2.5 hours wall, and less than 1.5 GB public disk**. New +Phase-5 GPU use stops at **6.0 H20-hours** under all circumstances. + +## Resolved decisions from the orchestrator + +1. **Approved:** use the recommended recorded-arrival P5 bridge ledger, with + its explicit limitation that it is not a literal decomposition of P3's + already-uniform P10 gap; otherwise select the P3-exact A3=N/A fallback. +2. **Approved:** dual P03/P04 control ledgers and the requirement that “dominant” hold + under both, rather than selecting one P3 denominator. +3. **Approved:** the 142-request slice, 32-request reorder blocks, 16-request analysis + cohorts, fixed bins, and 64-second fairness cap as A1's isolation/latency + tradeoff. +4. **Approved:** three P10 replicates, reuse of P3 controls behind the 3% bridge gate, + and the optional-tier order within the 6.0-H20-hour cap. +5. **Approved:** Layer-1-only primary runs and treating routed-expert telemetry as + private analysis-only evidence with no causal share. + +## Protocol sanity block + +| Numeric family | n | Min | Max | Distinct | Checked invariant/result | +|---|---:|---:|---:|---:|---| +| P3 control/base `E_token` | 3 | 2.619113 | 4.735600 | 3 | Finite, positive, not identical | +| P3 P10 control gaps | 2 | 0.435590 | 2.116487 | 2 | Positive; dual denominators retained | +| P10 request rows/arm | 5 arms | 142 | 142 | 1 expected | Same IDs and input/output token sums required | +| Selected source timestamps (s) | 142 | 0.014 | 21.445 | 142 | Finite, nondecreasing; gap min/max 0.002/0.851 s, 127 distinct gaps | +| Primary offered rate (req/s) | 5 arms | 0.4725 | 0.4725 | 1 expected | Positive; achieved rate must be within 5% | +| Warm-up / clean / drain gates (s) | 3 values | 60 | 600 | 3 | Clean is exactly `3*80=240`; long-context drain is 600 | +| A1 input-length bins | 7 | 0 | 32768 | 7 intervals | Ordered, contiguous, cover frozen P10 range | +| A1 frozen `R16` values | 2 | 0.473409 | 0.641744 | 2 | Sorted is lower by 0.168334 absolute / 26.23% relative; not identical | +| A1 added-delay values (s) | 142 | 0 | 62.744 | >1 expected | Non-negative and all below 64-second cap | +| P3 P10 pure-decode steps | 17,941 | B=1 | B=7 | 7 B values | Counts sum to 17,941; non-negative; stream SHA pinned | +| P10 A2 added capture sizes | 4 | 3 | 7 | 4 | Exactly missing observed support `{3,5,6,7}`; 100% P3 support covered | +| Primary measured runs | 15 | 3/arm | 3/arm | 1 expected | `5 arms * 3 replicates`; no arm omitted | +| Maximum GPU analysis/ledger runs | 1 plan | 37 | 37 | 1 expected | `15+6+5+10+1=37`; burn-ins excluded | +| Expected H20-hours | 1 plan | 3.8 | 5.1 | 2 bounds | Finite, non-negative, below hard cap 6.0 | +| Share domain | 5 ledger lines/control | Unbounded | Unbounded | N/A | No clipping; ratios may be negative or >1 | +| Bootstrap resamples | 1 setting | 100,000 | 100,000 | 1 expected | Seed 20260716; no denominator-draw deletion | +| Phase-5 GPU runs in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied | + +Checked invariants: `0.60*0.7875=0.4725`; P3 control gaps are positive; +the seven decode-B counts sum to 17,941; default plus `{3,5,6,7}` covers all +observed P10 pure-decode B values; `5*3=15` primary runs; maximum planned count +is `15+6+5+10+1=37`; expected GPU use remains below the 6.0-hour hard cap; +ratios are not constrained to `[0,1]`; expected constants are labeled; and no +GPU command, helper change, manifest transform, or experiment was executed in +this protocol-only turn. The unresolved P3-arrival/P5-arrival estimand mismatch +is reported first as a blocking decision rather than hidden in the ledger. diff --git a/docs/opprof/phase5-results.md b/docs/opprof/phase5-results.md new file mode 100644 index 0000000..b3e020f --- /dev/null +++ b/docs/opprof/phase5-results.md @@ -0,0 +1,223 @@ +# OpProf Phase 5 dash0 results + +Status: **FINAL — INCONCLUSIVE MECHANISM LEDGER; NO DOMINANCE CALL**. + +Date: 2026-07-12. All 15 registered P10 primary runs completed with three +replicates per arm and passed A-P5-1. The fresh A3 bridge passed, so the frozen +P3 P03/P04 controls were reused. The ledger is nevertheless inconclusive: +P04's reused-control denominator is unstable, and A2, A3, and A4 fail their +predeclared manipulation checks. The protocol therefore permits one official +share (A1 under P03), requires the other lines to be N/A, and forbids a +dual-control dominance conclusion. + +The machine result is `runs/opprof-phase5/phase5/metrics.json` (SHA-256 +`fe16934866806ac542f7974cac061beeb1f1c58f5bfb36698efd8aeaa99b8cb7`). +The final analyzer SHA-256 is +`8b0ff8658614227ee00cf8d10c82c4b5edf6f6210db172fd6efc9cc149e51e10`; +its no-GPU tools and analysis tests pass. + +This remains explicitly a **recorded-arrival P5 bridge ledger anchored to P3 +controls**, not a literal decomposition of P3's already-uniform P10 gap. + +## Data-sanity result first + +**RED FLAG: the P04 control denominator is unstable.** P5 recorded-arrival +base has `E_token=3.013752`, while reused P3 P04 has `E_token=3.054704`, leaving +only a `0.040952` gap with bootstrap 95% CI `[-0.649096, 0.752594]`; 45.353% of +denominator draws are nonpositive. This violates every registered stability +form: the CI contains zero and the nonpositive fraction exceeds 5%. + +P03 remains stable: its gap is `1.721848`, CI `[1.459092, 1.988772]`, with zero +nonpositive draws. All execution-validity checks pass: 15/15 primary runs, +three replicates per arm, exact 240-second clean windows, zero clean failures, +offered rates within 5%, continuous/balanced Layer-1 accounting, identical +request IDs and token sums, all A-P5-1 cold-start gates, no drain quarantine, +and GPU spend below the cap. The red flag is reported before the ledger and no +conclusion is built on P04's raw ratio draws. + +## Bridge and absolute efficiency + +A3 reproduces the P3 uniform-arrival P10 base: `delta_E=+0.007041`, 95% CI +`[-0.329799, 0.354859]`, or 0.269% absolute relative difference. The CI contains +zero and the point difference is below 3%, so the bridge passes and P3 control +reuse is valid under the frozen rule. Conditional control reruns were not +launched. + +| Arm | `E_token` | Bootstrap 95% CI | Delta from P5 base | +|---|---:|---:|---:| +| Base, recorded | 3.013752 | [2.754462, 3.267964] | — | +| A1 length-binned | 3.078999 | [2.801659, 3.349307] | +0.065247 | +| A2 capture sizes | 3.062128 | [2.802812, 3.314276] | +0.048376 | +| A3 uniform arrival | 2.626154 | [2.451852, 2.792684] | -0.387598 | +| A4 prefix cache off | 3.020959 | [2.760494, 3.274105] | +0.007207 | + +## Manipulation checks and Holm family + +| Arm | Manipulation result | Efficiency delta 95% CI | Raw p | Holm p | +|---|---|---:|---:|---:| +| A1 | **PASS:** R16 0.641744 -> 0.473409; max delay 62.743 s; prefix-hit-ratio delta -0.296 pp | [-0.309461, 0.442565] | 0.73108 | 1.00000 | +| A2 | **FAIL:** support coverage 98.879% <99%; padding reduction 82.541% <90% | [-0.315361, 0.411446] | 0.79022 | 1.00000 | +| A3 | **FAIL:** decode-B CV falls 0.8706 -> 0.6551, but waiting CV rises 6.0145 -> 6.8557 | [-0.693444, -0.078141] | 0.01384 | 0.05536 | +| A4 | **FAIL:** disabled arm still reports 3,093,324 local prefix queries and 52,896 hits, not zero | [-0.357637, 0.369309] | 0.97230 | 1.00000 | + +Per protocol, a failed manipulation makes that mechanism's official share N/A +even if its raw efficiency contrast is nonzero. A3's uncorrected contrast is +negative, and it also misses family-wise significance after Holm correction. + +## A-P5-1 arrival evidence + +The four originally invalidated arms supplied the evidence for A-P5-1. Under +the superseded throughput-drift calculation, recorded base/A1/A4 span +190.55%-216.50% normalized drift, while uniform A3 is 13.23%. This gap is +retained as direct evidence that the old gate measured arrival shape in a +rate-following experiment; it is not converted into an efficiency share. + +All 15 replacement/continued primary runs pass the amended cold-start gate: +25-28 warm-up completions, 12-16 completions with input at least 8192 tokens, +all compile/capture first occurrences before clean, zero clean-window +first-capture events, and zero uncovered Layer-1 graph descriptors. The 10% +drift gate was retained unchanged for the P06 saturation background arm. + +## Operational findings + +- The detached controller completed three cache burn-ins, four four-way waves, + all drains, validation, budget accounting, and cleanup. The four invalidated + first-wave arms were rerun exactly; none of their superseded measurements + enters the ledger. +- The first exact rerun produced complete clean artifacts but hit a post-hoc + validator `NameError` from a missing `math` import. Immutable artifacts were + CPU-re-adjudicated under A-P5-1, the import was repaired, and no third GPU run + was charged. +- CPU analysis repairs admitted legitimate idle fixed-time blocks as `[0,0]`, + supported the older P3 control schema, converted one NumPy boolean at the + JSON boundary, and explicitly separated diagnostic from reportable shares. + These changes alter no GPU data, estimator, seed, threshold, or resample + count. +- P10 drains range from 0.676 to 6.004 seconds. The separate P06 saturation + background run drained naturally in 87.903 seconds, below the 600-second + gate, with 564/564 clean completions and zero failures. +- Public Phase-5 artifacts occupy 286,205,365 bytes (304 MiB), below the 3 GB + stop threshold; CPFS retained 1,286 GB free. +- MoE routing skew remains content-bound and receives no causal share. The + optional routed-expert analysis-only run was not required for this primary + ledger and was not launched. + +## Launch echo + +```text +LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails +``` + +Resume after A-P5-1 reused the completed burn-ins and compile/cudagraph caches. +The first launch through final controller completion took approximately 57.9 +minutes wall time, including the adjudication/resume interval. + +## Run completion stats + +| Item | Count/result | +|---|---:| +| Valid primary P10 runs | **15/15** | +| Replicates | **3 per arm** | +| Valid P06 background runs | 1 | +| Completed burn-ins | 3 | +| Superseded pre-amendment arms | 4 | +| Conditional fresh controls | 0; bridge passed, P3 reused | +| Optional secondary/routed-expert runs | 0 | +| A-P5-1 primary passes | 15/15 | +| Clean failures | 0 | +| Drain quarantines | 0 | + +## Mechanism ledger + +Official shares follow the frozen N/A rules. Values in parentheses are raw +diagnostic ratios retained for audit, not causal/reportable shares. + +| Mechanism | P03 share (95% CI) | P04 share (95% CI) | Disposition | +|---|---|---|---| +| A1 length raggedness | **0.0379 [-0.2011, 0.2344]** | N/A; unstable denominator (diagnostic 1.5933 [-6.8925, 7.0636]) | Manipulation passes; no significant recovery | +| A2 capture mismatch | N/A; manipulation failed (diagnostic 0.0281 [-0.2049, 0.2170]) | N/A; unstable denominator (diagnostic 1.1813 [-6.2871, 6.4924]) | Config did not meet 99%/90% removal gates | +| A3 arrival dynamics | N/A; manipulation failed (diagnostic -0.2251 [-0.4625, -0.0403]) | N/A; unstable denominator (diagnostic -9.4647 [-17.2450, 17.4102]) | Queue-variance gate failed; Holm p=0.05536 | +| A4 prefix structure | N/A; manipulation failed (diagnostic 0.0042 [-0.2340, 0.1948]) | N/A; unstable denominator (diagnostic 0.1760 [-6.2590, 6.4925]) | Zero-query/hit gate failed | + +Shares are neither clipped nor renormalized. P04's extreme raw ratios are the +registered consequence of dividing by a near-zero, sign-unstable denominator, +not evidence of large mechanisms. + +## Dominance verdicts + +| Mechanism | P03 rule | P04 rule | Dual-control verdict | +|---|---|---|---| +| A1 | Fails share/CI/Holm thresholds | Not evaluable | **NOT EVALUABLE** | +| A2 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** | +| A3 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** | +| A4 | Not evaluable: manipulation failed | Not evaluable | **NOT EVALUABLE** | + +No mechanism may be called dominant because the frozen rule requires both +control denominators. This is an inconclusive verdict, not evidence that every +mechanism is small. + +## Residual + interaction line + +The official residual is N/A under both controls because the official share +ledger is incomplete. The unchanged arithmetic over all raw diagnostic shares +is reported only for transparency: + +| Control | Diagnostic share sum | Diagnostic residual + interaction (95% CI) | Status | +|---|---:|---:|---| +| P03 | -0.1549 | 1.1549 [0.5536, 1.9397] | Diagnostic only | +| P04 | -6.5142 | 7.5142 [-21.3965, 22.8767] | Diagnostic only; unstable denominator | + +Nothing is forced to 100%. The residual can include interactions, unablated +MoE routing, chunked-prefill effects, double-counting, and measurement error. + +## Config-tier A2 measured delta + +A2 added the P3-derived sizes `{3,5,6,7}` to the full default capture list. +Its measured `delta_E` is **+0.048376**, 95% CI +`[-0.315361, 0.411446]`, or **+1.605%** relative to base. Pure-decode padding +falls from 13.7405% to 2.3989%, an 82.541% reduction. New recorded-arrival +runs expose decode-B support 1-13, so uncaptured 9-13 leave coverage at 98.879%; +both this and the sub-90% padding reduction invalidate the causal A2 share. + +The config cost is reproducible: base graph capture is 11 seconds and 0.80 GiB +in all three runs; A2 is 12-14 seconds and 0.84 GiB, a 1-3 second and 0.04 GiB +increase. The measured config-tier result is therefore a small, statistically +uncertain efficiency gain with a failed preregistered removal gate. + +## GPU total + +New Phase-5 spend is **2.4388167443 H20-hours** of the 6.0-hour cap, including +the superseded first wave, exact reruns, continued primary matrix, burn-ins, +and background arm. **3.5611832557 H20-hours remain unspent.** Final state on +all eight H20s is 0 MiB and 0% utilization with zero compute processes. + +## Sanity block + +**Anomaly first:** P04's gap CI contains zero and 45.353% of its denominator +draws are nonpositive; the dual-control ledger is inconclusive and inferential +claims stop there. + +| Numeric family | n | Finite | Missing | Min | Max | Distinct | Checked invariant/result | +|---|---:|---:|---:|---:|---:|---:|---| +| Primary run `E_token` | 15 | 15 | 0 | 2.623534 | 3.083263 | 15 | Positive; per-arm results not all identical | +| Clean Layer-1 steps | 15 | 15 | 0 | 11,178 | 18,400 | 15 | Non-negative; continuous/balanced per run | +| Clean duration (s) | 15 | 15 | 0 | 240 | 240 | 1 expected | Exact `3*80`; zero clean failures | +| Offered rate (req/s) | 15 | 15 | 0 | 0.470833 | 0.487500 | 2 | Every run within 5% of 0.4725 | +| Warm-up completions | 15 | 15 | 0 | 25 | 28 | 2 | Every run >=16 | +| Warm-up long completions | 15 | 15 | 0 | 12 | 16 | 3 | Every run >=1 with input >=8192 | +| Clean first-capture events | 15 | 15 | 0 | 0 | 0 | 1 expected | Zero events and zero uncovered descriptors | +| P10 drain duration (s) | 15 | 15 | 0 | 0.676181 | 6.003959 | 15 | Non-negative; all below 600 | +| Control gap `E_token` | 2 | 2 | 0 | 0.040952 | 1.721848 | 2 | P03 stable; **P04 unstable red flag** | +| Raw diagnostic shares | 8 | 8 | 0 | -9.464746 | 1.593268 | 8 | Finite; intentionally not constrained to [0,1] | +| Diagnostic residuals | 2 | 2 | 0 | 1.154931 | 7.514211 | 2 | Finite; never forced to close | +| New H20-hours | 1 | 1 | 0 | 2.438817 | 2.438817 | 1 | Non-negative; below 6.0 | + +Checked invariants: source/helper/protocol provenance; identical request IDs, +content/token totals, and prompt-preserving manifests; C00-TP1 four-way +placement; detached controller ownership; exact clean windows; zero failures; +offered-rate tolerance; A-P5-1 compile/capture ordering and coverage; Layer-1 +footer/sidecar balance, continuous step indices, token composition, and zero +drops; non-negative counters; drain gates; bridge decision; no public prompt or +generated-text leakage; public-disk/free-space limits; GPU hard-cap compliance; +and zero-process/zero-memory cleanup. Manipulation failures are A2, A3, and A4; +the sole machine sanity red flag is P04 denominator instability. diff --git a/docs/opprof/phase6-protocol.md b/docs/opprof/phase6-protocol.md new file mode 100644 index 0000000..c24da30 --- /dev/null +++ b/docs/opprof/phase6-protocol.md @@ -0,0 +1,638 @@ +# OpProf Phase 6 pre-registered cross-version churn protocol + +Status: **ACCEPTED FOR EXECUTION — ALL FIVE ORCHESTRATOR DECISIONS RESOLVED**. + +Date frozen: 2026-07-12 (Asia/Singapore). Phase 6 asks whether the vLLM 0.20.0 +best configuration and ranking on the C1 +`TP={1,2,4} x max-num-seqs={8,16,32,64}` surface survive an upgrade to patched +vLLM 0.24.0. This is a paired historical-anchor experiment, not a fresh tuner +search. It measures churn at the exact recorded C1 request sets and reports +where the recorded local frontier moved. + +## Approved dispositions (orchestrator; 2026-07-12) + +All five open decisions are approved exactly as proposed, without changing an +anchor, threshold, estimator, validity gate, or budget: + +1. The upgrade-path estimand is approved. Resolved default changes are part of + observed churn; dash1->dash0 and co-location remain explicit limitations and + the result is not described as a pure engine-version causal effect. +2. The adaptive 25-anchor subset is approved: every cell receives its old peak + plus the direction-relevant adjacent anchor, and TP4/MNS16 receives both + neighbors. +3. The 5% material-frontier threshold, `tau-b>=0.8` ranking-survival threshold, + and zero-anchored floor-bucket argmax/trap rules are approved. +4. One primary observation per historical anchor plus the 0.35-H20-hour + confirmation reserve is approved. +5. The A-P5-1-class warm-up long tier is approved as raw input `>4096` tokens + for this 0-8192 workload. + +This approval authorizes the later execution only after pinned-input, detached +ownership, cold-start, echo-before-launch, placement, privacy, cleanup, and +projected-budget gates pass. The 3.0-H20-hour cap remains absolute. + +## Amendment A-P6-1 — checkpoint-sidecar accounting and conditional cap + +The first Phase-6 W1 request replays were incorrectly rejected by requiring a +clean in-stream footer and `final=true` sidecar after a non-graceful shutdown. +That contradicts the already accepted A-P3-5/P5 accounting rule. Before any +resume, all four W1 cell streams and their eight measured anchor intervals must +be CPU-re-adjudicated against the latest atomic checkpoint sidecar. With no +in-stream footer, the sidecar is authoritative when every complete JSONL line +decodes, step indices are contiguous, on-disk data-record count equals +`written_records`, final data-line step equals `last_step_index`, +`encoded_records = written_records + dropped_records`, `dropped_records=0`, and +the checkpoint is within the configured one-second flush interval (plus the +accepted 100-ms scheduling tolerance) of the recorded shutdown boundary. At +most one post-checkpoint flush interval may be absent. A balanced stream accepts +all covered W1 replay intervals without a GPU rerun. + +All subsequent waves use the graceful P5 path: start `vllm serve` with positive +`--shutdown-timeout`, signal each API parent with SIGINT, wait up to 150 seconds +for EngineCore drain/finalization, and use process-group SIGTERM/SIGKILL only as +fallback. The preferred result is one in-stream footer plus an agreeing +`final=true` sidecar; the atomic checkpoint rule remains the crash fallback. + +Only if any W1 sidecar fails the CPU balance/coverage gate does A-P6-1c activate: +the hard cap becomes **3.5 H20-hours**, the activation is logged before launch, +and W1 is rerun under graceful shutdown. If all W1 sidecars balance, A-P6-1c is +not activated, W1 is accepted in place, its already charged 0.299393 H20-hours +is retained, and execution resumes at W2 under the original 3.0-hour cap. + +## Amendment A-P6-2 — authoritative solo frontier tier and 6.0-hour cap + +The user raises the cumulative Phase-6 hard cap from **3.0 to 6.0 H20-hours**. +This amendment responds to the W2/W3 confirmation evidence: when neighboring +wave clients became idle, pass rate changed from `0.412 -> 1.000` and `0.028 -> +0.957` at TP2/MNS32 and from `0.036 -> 1.000` at TP4/MNS16, while Layer-1 +waiting means collapsed from `1.22 -> 0.00`, `8.50 -> 0.35`, and `28.33 -> +0.06`. Because the v0.20 C1 baseline ran one cell at a time on dash1, +co-located SLO feasibility is not a valid authoritative tier for the paired +frontier. Co-location remains an explicitly reported, metric-dependent +methodological ablation; its throughput-only observations are indicative. + +Every SLO-frontier-decisive Phase-6 anchor is therefore run **solo**: exactly +one vLLM server and one replay client are active on dash0, no other Phase-6 +server/client is resident, and only the cell's `TP` GPUs are allocated. Each +solo cell gets a fresh server process, the frozen exact-selection client, an +independent 16-request long-tier warm-up, A-P5-1 cold-start gates, Layer-1 +telemetry, and graceful A-P6-1 shutdown. The solo result is authoritative for +feasibility, frontier, floor bucket, argmax, tau-b, and trap decisions. A +co-located-only result may not fill a missing solo frontier or support a paper +claim. + +The mandatory solo set has no cell exclusions: + +| Cell set | Mandatory solo anchors | Inclusion reason | +|---|---|---| +| TP1/MNS8 | existing L+P | co-located peak failed | +| TP1/MNS16,32,64 | existing P+U | co-located U passed and censored the frontier | +| TP2/MNS8,16 | existing L+P | co-located peak failed; L distinguishes bracket vs left censoring | +| TP2/MNS32 | existing L+P | old argmax; both anchors split primary/confirmation | +| TP2/MNS64 | existing L+P | the indicative `>29.4% down` bound is unquotable until solo-confirmed | +| TP4/MNS8 | existing L+P | co-located peak failed; trap-neighbor comparison | +| TP4/MNS16 | existing L+P+U | named trap; peak split and L failed co-located | +| TP4/MNS32,64 | P plus direction-relevant L/U | W4 completion; unmeasured under the 3.0-hour cap | + +After these 25 anchors, the controller may crawl only the pinned 92-anchor +history: upward while the highest solo anchor passes or downward while the +lowest solo anchor fails, stopping at the first opposite-feasibility anchor. +There is no interpolation and no unrecorded request set. A pass rate in +`[0.93,0.97]`, or a solo/co-located disagreement that changes argmax/trap or a +material-frontier claim, triggers a same-solo-placement confirmation; a split +gets a third trial for the frozen 2-of-3 rule if projected budget permits. +Unresolved or history-edge frontiers remain explicitly censored. + +Priority is W4, TP2/MNS32, TP2/MNS64, TP4/MNS16, then the remaining failed or +censored cells, so a budget stop preserves the most decision-relevant evidence. +All twelve cells are nevertheless included. The planning envelope is 3.44 new +H20-hours for serialized startup, warm-up, 25 mandatory anchors, bounded crawl, +graceful drain, and a 0.20-hour safety reserve. Added to the already charged +2.291173 hours, the launch projection is **5.731173/6.0 H20-hours**. Before +each one-cell wave, the controller echoes cell, anchors, GPU placement, input +paths, charged spend, remaining projection, and cap. Projected or actual spend +reaching 6.0 stops further GPU work. + +Final analysis preserves the complete W1-W3 co-located attempt history and +reports a solo-versus-co-located delta table for every exact anchor pair. +ARGMAX, RANKING, and TRAP use solo values only. RANKING remains inconclusive +and tau-b non-evaluable unless all 12 solo frontiers are bounded under the +pinned history. + +## Headline claim and falsifiable outcomes + +The paper-facing question is: + +> Does the C1 vLLM 0.20.0 per-GPU SLO-feasible ranking, especially global best +> TP2/MNS32 and local trap TP4/MNS16, remain valid on vLLM 0.24.0 when the exact +> historical request-selection and SLO semantics are replayed? + +Phase 6 can produce four distinct outcomes: + +1. **SURVIVES:** TP2/MNS32 remains in the top floor bucket, the full-surface + Kendall tau-b is at least 0.8, and the TP4/MNS16 trap persists. +2. **ARGMAX MOVED:** TP2/MNS32 is not in the top v0.24 measured-frontier bucket. +3. **RANKING MOVED WITHOUT ARGMAX MOVEMENT:** the old best survives but tau-b is + below 0.8 or the trap changes status. +4. **INCONCLUSIVE/PARTIAL:** a decision-critical frontier is unbracketed, an + anchor is not reproducible, repeated boundary verdicts disagree, the 12-cell + surface is incomplete, or the 3.0-H20-hour cap stops required work. + +The experiment quantifies observed upgrade churn. Because engine version, +host, resolved defaults, and co-location differ together, it does not identify +a pure causal effect of one vLLM commit. + +## Frozen vLLM 0.20.0 surface + +The historical objective is the maximum measured SLO-feasible offered rate +divided by TP: + +```text +f20(c) = max feasible selected_request_count / 60 seconds / TP +``` + +| TP | MNS8 | MNS16 | MNS32 | MNS64 | +|---:|---:|---:|---:|---:| +| 1 | 2.1000 | 2.3500 | 2.2833 | 2.2833 | +| 2 | 2.2750 | 2.2750 | **3.2833** | 3.2583 | +| 4 | 1.2833 | **2.4417** | 2.4417 | 2.4417 | + +Values are request/s/GPU. The global best is TP2/MNS32. TP4/MNS16 is the +registered local trap: it is the first point on a TP4 plateau, has no improving +adjacent MNS move, but is below the global best. + +The pinned ground-truth asset is +`/home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json`, SHA-256 +`23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670`. +It contains 92 probe observations: eight anchors for each TP1/TP2 cell and seven +for each TP4 cell. + +## Comparability contract + +### Exact workload and selection semantics + +The private materialized workload is +`trace_windows/traces/chat_w20260311_1000.jsonl`, 32,606 rows, SHA-256 +`f539f38eb0ee0f750e3c23ff47df6eed3faf723a25f1444d55665a85871750b9`. +Its window record is `trace_windows/windows.json`, SHA-256 +`23c432e7439508b07991cabfe1db9977ebb55f8ddc39ca214a627fc5f5ae4725`: +window `[0,600]` seconds becomes `[0,60]` under replay scale 0.1. + +The later helper must call or byte-for-byte reproduce the following pinned +AITuner implementations, rather than use the Phase-3/5 client: + +| Source | SHA-256 | Frozen role | +|---|---|---| +| `src/aituner/trace.py` | `6bb19f0e3eb42b1aecc6969a5f20d65eb505a91564dd117dd149155a6114eb35` | load, stable time sort, cap/downsample, threshold selection | +| `src/aituner/worker.py` | `18337fe273a48d1cbb2bc364f774d100f8ffa2be5f9cd6a7c58c8ed4717c5e48` | rate-following submit loop, early stop, denominator accounting | +| `src/aituner/slo.py` | `1d2e5b1ed9d0d6d6c07f8b5b6fa56d36ed9bb62710be54c125e9b2890f4f96d5` | stepped SLO evaluation | +| `src/aituner/http_client.py` | `213b80d5b37cc74333517f46ab4e8cdba1a274bdbfabdfc14207b314f61799ac` | urllib/SSE TTFT, TPOT, usage accounting | + +The exact order is normative: + +1. Read all JSONL rows in file order. Retain raw `input_length` in `[0,8192]`. +2. Build the same chat body and stable-sort by scaled arrival time only. Stable + ties retain source order; sorting by `(timestamp,sampling_u,...)` is forbidden. +3. For TP1/TP2, even-downsample the full filtered, time-sorted list to 512 + entries **before** threshold selection, using index + `floor(i*N/512)` for `i=0..511`. TP4 is uncapped. +4. Select exactly the requests with `sampling_u <= anchor`. Preserve their + selected service order. Do not resample, wrap, shuffle, or interpolate. +5. Submit at each row's materialized `timestamp*0.1`; do not uniformize, + rebase individual gaps, or replace the recorded burst process. +6. Send the original messages through `/v1/chat/completions` with `stream=true`, + `stream_options.include_usage=true`, and the row's temperature when present. + Set both `min_tokens=128` and `max_tokens=128`. +7. Use maximum concurrency 64. Client request timeout remains 900 seconds. +8. Verify `usage.completion_tokens==128`; missing usage, a different token count, + HTTP failure, timeout, and every request not submitted because the SLO verdict + became unrecoverable are failures in the original selected-request denominator. +9. Preserve original early-stop logic: after more than + `N-ceil(0.95*N)` failed evaluations, synthesize failures for the remaining + selected requests. Disable adaptive Stop-A; it was absent in C1. + +An offline, prompt-free preflight must regenerate all 92 historical request +counts exactly. The local reconstruction already yields 17,710 filtered rows, +512 capped rows, and **92/92 count matches**. Any future mismatch in count, +request-ID hash, arrival hash, raw-length hash, or requested-token sum stops the +GPU launch. + +### Exact SLO and score + +Request feasibility uses the original raw trace input length, not tokenizer +usage after chat templating: + +```text +raw input <= 4096: TTFT <= 2000 ms +raw input <= 32768: TTFT <= 4000 ms +otherwise: TTFT <= 6000 ms +all requests: TPOT <= 50 ms +anchor feasible: SLO-passing selected requests / all selected requests >= 0.95 +``` + +Although the C1 filter limits inputs to 8192, the complete three-step rule is +retained. TTFT is client wall time from request start to first nonempty content. +TPOT is `(last_content_time-first_content_time)/(usage_completion_tokens-1)`. +The paired surface score is always selected count divided by 60 seconds and TP; +completed throughput is a diagnostic and never replaces the historical score. + +### Serving configuration + +The v0.24 command repeats the explicit C1 launch surface: + +```bash +MODEL=/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B +VENV=/tmp/wjh-opprof-phase2-dash0-20260711/.venv +RUN_ROOT=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6 + +taskset -c "$CPUSET" env CUDA_VISIBLE_DEVICES="$GPU_IDS" \ + VLLM_OPPROF_DIR="$RUN_DIR/opprof" \ + "$VENV/bin/vllm" serve "$MODEL" \ + --host 127.0.0.1 --port "$PORT" \ + --served-model-name qwen3-30b-a3b-community \ + --max-num-batched-tokens 8192 \ + --max-num-seqs "$MNS" \ + --tensor-parallel-size "$TP" \ + --shutdown-timeout 120 +``` + +`--shutdown-timeout` is operational and outside the measured interval. As in +C1, do **not** explicitly set block size, GPU-memory utilization, prefix +caching, chunked prefill, CUDA-graph capture sizes, DP, or EP. Their resolved +v0.24 values must be parsed from startup logs and reported. Allowing version +defaults to change is intentional: this experiment asks whether an operator's +old explicit configuration remains good after upgrading, not whether internal +defaults can be manually made identical. + +Preflight records `vllm.__version__`, source commit, patch checksums, Python, +PyTorch, CUDA, NCCL, Triton, driver, model/tokenizer file hashes, MoE backend, +resolved scheduler/cache/graph config, compile-cache key, and exact command. +The expected source is the existing Phase-3/5 patched vLLM 0.24 environment; +any missing OpProf patch or unexpected model path is a stop condition. + +### Client equivalence, not a new client covariate + +The later `scripts/opprof_phase6_client.py` is permitted only as a thin anchor +driver around the four pinned AITuner modules. Its interface is normative: + +```bash +P6C='python scripts/opprof_phase6_client.py' + +$P6C preflight \ + --trace trace_windows/traces/chat_w20260311_1000.jsonl \ + --windows trace_windows/windows.json --window-id chat_w20260311_1000 \ + --ground-truth /home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json + +$P6C run-anchor --cell "$CELL" --anchor "$U" \ + --base-url "http://127.0.0.1:$PORT" \ + --model qwen3-30b-a3b-community --completion-tokens 128 \ + --replay-time-scale 0.1 --max-concurrency 64 \ + --max-requests "$CAP_OR_NONE" --target-pass-rate 0.95 \ + --ttft-step-ms '4096:2000,32768:4000,inf:6000' --tpot-ms 50 \ + --request-timeout-seconds 900 --probe-deadline-ceiling-seconds 900 \ + --result-dir "$ANCHOR_DIR" +``` + +Before GPU execution, a no-server golden test must prove identical selected IDs, +order, arrivals, raw input lengths, serialized bodies excluding model/URL, and +SLO decisions for synthetic outcomes. The wrapper may add interval markers and +sanity JSON, but may not change HTTP transport, scheduling, completion checks, +early stop, or latency formulas. + +### Covariate ledger and estimand limit + +| Covariate | C1 v0.20 | Phase 6 v0.24 | Treatment/handling | +|---|---|---|---| +| Engine | community vLLM 0.20.0 | patched vLLM 0.24.0 | Primary intended churn dimension; exact builds recorded | +| Host | dash1 | dash0 | Both 8x H20 96 GB class; CPU/NUMA/clocks/driver/kernel recorded, not assumed identical | +| GPU visibility | TP1 C1 env exposed GPUs `0,1`, TP1 consumed one | expose exactly TP GPUs | Declared operational covariate; H20-hour charge follows TP | +| Trial placement | recovered C1 harness trials | up to four co-located servers | CPU/NUMA pinned; reject outside GPU processes and report co-location limitation | +| HTTP client | pinned AITuner urllib/SSE | same code paths through thin wrapper | Eliminated as a semantic covariate by golden tests | +| Workload | materialized C1 chat window | byte-identical file/hash | Fixed | +| Selection/cap | cap-before-threshold; TP4 uncapped | exact same | Fixed and 92-count preflighted | +| SLO length | raw input length | raw input length | Fixed; server usage length is diagnostic only | +| Explicit flags | TP, MNS, MBT8192 | same | Fixed | +| Unspecified defaults | v0.20 defaults | v0.24 defaults | Intentionally allowed to churn; resolved values reported | +| Telemetry | no Layer 1 | zero-overhead OpProf Layer 1 | Added measurement; overhead evidence -0.04196%, CI [-0.17443%, 0.04550%] | + +Therefore “version churn” means the observed paired upgrade path under these +documented platform covariates. It must not be rewritten as a same-host causal +microbenchmark. + +## Budget-constrained anchor subset + +### Adaptive adjacent-anchor design + +The primary subset improves on measuring next-infeasible anchors only for a +chosen top four: + +1. Measure the recorded v0.20 peak anchor for all 12 cells. +2. For every non-trap cell, select exactly one adjacent recorded anchor by a + result-independent rule fixed now: + - if the v0.20 peak remains feasible on v0.24, measure the nearest higher + recorded anchor, which was infeasible on v0.20; + - if the peak becomes infeasible, measure the nearest lower recorded anchor, + which was feasible on v0.20. +3. For TP4/MNS16, measure lower, peak, and higher anchors unconditionally. This + directly brackets the registered trap. + +Thus exactly 25 primary anchors are measured: TP1 eight, TP2 eight, and TP4 +nine. The design covers every cell and spends the second anchor in the direction +where its frontier could have moved. It also includes the suggested top-four +test. Under the floor-bucket rule, the nominal top-four cutoff expands to five +cells because TP4/MNS16/32/64 share one bucket; all five receive an upward test +whenever their old peak still passes. + +If an adjacent anchor does not bracket the transition—higher also passes or +lower also fails—the cell is censored. Remaining budget may crawl one recorded +anchor farther outward, in this priority order: old global best TP2/MNS32, +TP2/MNS64, the TP4 plateau MNS16/32/64, then other cells by old score. Crawling +uses only that cell's recorded history and stops at the first transition. No +new `sampling_u` value may be invented. Failure to close a decision-critical +bracket before the budget stop makes the applicable argmax/ranking verdict +inconclusive. + +### Boundary confirmations + +The original surface has one observation per anchor. Phase 6 keeps that paired +unit but reserves at most 0.35 H20-hours for decision-triggered confirmation. +Repeat an anchor if its v0.24 pass rate lies in `[0.93,0.97]`, or if its single +verdict alone changes argmax or trap status. Priority is argmax, trap, material +frontier calls, then other cells. Agreement across two runs freezes the verdict; +disagreement requests a third run if budget permits and uses 2-of-3. Without a +third run, that anchor and every dependent headline verdict are inconclusive. +Request outcomes are arrival-correlated, so a binomial CI is descriptive only +and cannot replace these repeat rules. + +## Execution plan + +### Preflight and detached ownership + +The Phase-6 controller follows A-P3-2 detached ownership: one controller PID, +per-stage atomic state, process-group/environment markers, resumable completed +anchors, exact command logs, and cleanup restricted to its own markers. Before +launch it must verify dash0 has eight H20s, no outside compute processes on +assigned GPUs, at least 100 GB CPFS free, pinned source/workload/helper hashes, +all 92 selection counts, legal TP placement, and a projected total below 3.0 +H20-hours. + +The autonomous launch log must echo one resolved line before starting: + +```text +LAUNCH_ECHO host=dash0 engine=vllm-0.24.0-patched model=Qwen3-30B-A3B trace_sha=f539f38e... cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16 clean_horizon=60s drain=derived<=120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h +``` + +### Rate-following cold-start and drain gates + +The throughput-drift stationarity gate is forbidden for these rate-following +anchors. A-P5-1-class cold-start gates are applied instead: + +1. Every torch.compile/CUDA-graph first-occurrence event must precede the first + measured anchor, verified from server logs and Layer 1. +2. A deterministic unmeasured warm set must complete at least 16 exact-output + requests, including at least one request with raw input `>4096` tokens. The + peak-selected sets contain 43-184 such requests, so this gate is feasible. +3. No first-occurrence compile/capture event may occur in any measured anchor; + all graph-hit `(runtime_mode,bucket_tokens)` descriptors must be covered by + startup capture logs. A violation invalidates that anchor only. +4. Before the next anchor, all selected requests must be accounted for and three + consecutive controller samples must show running/waiting/deferred queues at + zero. Prefix cache is not explicitly enabled; nevertheless a full drain is + required. + +The probe deadline reproduces `_probe_drain_deadline`: last selected arrival + +raw-length TTFT budget + `128*50 ms` + 30 seconds, capped by the historical +900-second ceiling. With raw inputs <=8192 this is about 100.4 seconds from +probe start; the controller has a 120-second class watchdog for cleanup. A +deadline miss is an SLO-infeasible anchor, not permission to omit denominator +requests. The server then shuts down through the official 120-second path. + +### Four-wave packing + +| Wave | Cells | Placement | Per-server measured anchors | Planned wall | +|---|---|---|---:|---:| +| W1 | TP1/MNS8,16,32,64 | four servers on GPUs 0,1,2,3; GPUs 4-7 idle | 2 each | 4-6 min | +| W2 | TP2/MNS8,16,32,64 | four servers on GPU pairs 0-1,2-3,4-5,6-7 | 2 each | 4-6 min | +| W3 | TP4/MNS8 and TP4/MNS16 trap | GPU quads 0-3 and 4-7 | 2 and 3 | 5-7 min | +| W4 | TP4/MNS32 and TP4/MNS64 | GPU quads 0-3 and 4-7 | 2 each | 4-6 min | + +Each server starts once, passes warm-up once, runs its peak first, drains, then +runs the predeclared adjacent anchor. Trap lower/upper order after peak is +counterbalanced by execution seed 20260717. Anchor confirmation occurs before +that server is released when possible. The controller records GPU clocks, +power, utilization, memory, CPU load, NUMA placement, and outside processes +through every wave. Any contamination invalidates the whole co-located wave. + +Actual H20-hours are charged from server launch through verified cleanup for +every allocated GPU, including idle time while a paired TP4 cell finishes. +Before each wave or confirmation, the controller recomputes +`spent + current + conservative_remaining`; it stops before launch if the +projection can reach 3.0. Optional crawling and confirmations are sacrificed +before any hard-cap violation. + +## Measurement and mechanism notes + +For every anchor, report selected/admitted/completed/exact-output/SLO-pass +counts; early-stop reason; offered and completed req/s; TTFT/TPOT mean and +p50/p90/p95/p99; failures by raw-length TTFT bucket; elapsed/drain time; and +peak GPU memory/utilization. The surface objective remains offered req/s/GPU at +the highest measured feasible recorded anchor. + +Layer 1 is active over the complete anchor service interval, from the first +submission through the last selected-request accounting event. Aggregate: + +- prefill, decode, and mixed-step counts and token shares; +- decode-batch distribution and 5-second CV; +- FULL/PIECEWISE/NONE graph-mode shares, bucket padding, and uncovered support; +- running/waiting/deferred queues and queue CV; +- preemption count/rate; +- KV blocks used/total, mean/max usage, and allocation pressure; and +- prefix queries/hits as a resolved-default diagnostic. + +Mechanism notes compare v0.24 cells/anchors and may relate a frontier change to +preemptions, KV pressure, batching, or graph modes. C1 lacks Layer-1 telemetry, +so these notes cannot claim that a measured composition field itself changed +from v0.20 or caused the version drift. Historical v0.20 TTFT/TPOT failure +reasons may be paired directly; device-mechanism attribution remains +descriptive. + +## Statistical and ranking analysis + +### Floor buckets + +For each unrounded score vector `s`, use the SimFid inventory rule: + +```text +tol(s) = max(1e-9, 1e-6 * max_c(abs(s(c)))) +bucket_s(c) = floor(s(c) / tol(s)) +``` + +Two scores tie only when their integer buckets match. Bucketing occurs before +display rounding, pair signs, tau-b, top selection, or trap evaluation. Report +`tol`, bucket integers, and occupied intervals `[j*tol,(j+1)*tol)`. For v0.20, +`tol20=3.283333333333333e-6`; the TP4 MNS16/32/64 tie makes nominal top-4 an +effective top-5 set. + +### Per-cell frontier + +For each cell, define the v0.24 measured recorded-anchor frontier: + +```text +f24(c) = max_{measured u with accepted feasible verdict} N_c(u) / 60 / TP +drift(c) = (f24(c) - f20(c)) / f20(c) +``` + +The predeclared material threshold is **X=5% relative**, large enough to avoid +calling one-request anchor quantization a paper-level churn effect. Report two +orthogonal labels: + +- **BOUNDARY DOWN/UP:** old peak flips feasible->infeasible, or the old adjacent + infeasible anchor flips to feasible, respectively; +- **MATERIAL FRONTIER MOVED:** `abs(drift)>5%`; otherwise report the boundary + flip as sub-material. + +If peak passes and higher fails, and `abs(drift)<=5%`, the local recorded +frontier is **STABLE**. If peak and lower both fail or peak and every affordable +higher anchor pass, report a directional bound and **UNBRACKETED**, never an +invented peak. + +### Argmax, ranking, and trap + +- **ARGMAX MOVED** when TP2/MNS32 is not in the top `bucket_f24` among all 12 + decision-bounded cells. **ARGMAX SURVIVED** requires it to share the top + bucket and no censored competitor capable of overtaking it. Otherwise it is + inconclusive. +- Compute Kendall tau-b over all 12 floor-bucketed `f20/f24` scores and report + concordant, discordant, v0.20-tied, v0.24-tied, and comparable-pair counts. + **RANKING SURVIVES** requires an evaluable 12-cell surface, argmax survival, + `tau-b>=0.8`, and no >5% top-bucket pair reversal. `tau-b<0.8` is **RANKING + MOVED**. A partial surface receives no suite-level tau claim. +- **TRAP PERSISTS** when TP4/MNS16 is in a bucket at least as high as both + adjacent TP4 cells MNS8 and MNS32, yet below the global-best bucket. It + **ESCAPES** if an adjacent TP4 move is strictly better, and **CEASES TO BE A + TRAP** if it joins the global-best bucket. Missing/unbracketed inputs make trap + status inconclusive. + +No arithmetic average of normalized scores is reported. Absolute rates, +relative drift, the paired table, and every excluded/censored anchor accompany +all ranking summaries. + +## Later deliverables and artifact contract + +Execution, after approval, must produce: + +- `docs/opprof/phase6-results.md`; +- `runs/opprof-phase6/phase6/metrics.json` containing the complete paired + surface, anchor/run values, selection hashes, feasibility, floor buckets, + frontier/argmax/ranking/trap verdicts, confirmation history, Layer-1 notes, + covariates, and GPU accounting; +- exact commands, controller state, environment/machine fingerprints, server + logs, client outcomes, Layer-1 JSONL/footer/sidecar, monitor samples, and + per-anchor sanity JSON; and +- an operational-findings section covering startup/capture gates, drains, + early stops, co-location, default changes, retries, contamination, and budget. + +The results begin with data sanity and stop inferential claims on any red flag. +Every numeric family ends with `n`, finite/missing, min, max, distinct count, +and applicable invariants. Prompt text, messages, generated content, source +substrings, and private request bodies stay in mode-0700 storage. Public files +contain only hashes, IDs, lengths, timestamps, outcomes, counters, and summaries. + +## Anchor-subset table + +Notation is `u / selected N / recorded req/s/GPU`. `P` is always measured. For +non-trap cells, measure `U` if P passes on v0.24, otherwise `L`. TP4/MNS16 +measures all `L+P+U`. Planning cost per anchor includes the 60-second replay and +20 seconds expected drain/accounting; shared startup/warm-up is budgeted below. + +| Cell | L: nearest recorded feasible below P | P: recorded peak | U: nearest recorded infeasible above P | Primary selection | H20-hour cost each | +|---|---|---|---|---|---:| +| TP1/MNS8 | 0.21875 / 121 / 2.0167 | 0.2265625 / 126 / 2.1000 | 0.23046875 / 130 / 2.1667 | P + (U if pass else L) | 0.0222 | +| TP1/MNS16 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | 0.25 / 143 / 2.3833 | P + (U if pass else L) | 0.0222 | +| TP1/MNS32 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 | +| TP1/MNS64 | 0.234375 / 132 / 2.2000 | 0.2421875 / 137 / 2.2833 | 0.24609375 / 141 / 2.3500 | P + (U if pass else L) | 0.0222 | +| TP2/MNS8 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 | +| TP2/MNS16 | 0.4921875 / 269 / 2.2417 | 0.49609375 / 273 / 2.2750 | 0.5 / 276 / 2.3000 | P + (U if pass else L) | 0.0444 | +| TP2/MNS32 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / **3.2833** | 0.7578125 / 396 / 3.3000 | P + (U if pass else L) | 0.0444 | +| TP2/MNS64 | 0.5 / 276 / 2.3000 | 0.75 / 391 / 3.2583 | 0.75390625 / 394 / 3.2833 | P + (U if pass else L) | 0.0444 | +| TP4/MNS8 | 0.016055910008 / 301 / 1.2542 | 0.016591107009 / 308 / 1.2833 | 0.017126304009 / 317 / 1.3208 | P + (U if pass else L) | 0.0889 | +| TP4/MNS16 trap | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / **2.4417** | 0.034252608017 / 600 / 2.5000 | **L + P + U** | 0.0889 | +| TP4/MNS32 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 | +| TP4/MNS64 | 0.033182214016 / 575 / 2.3958 | 0.033717411016 / 586 / 2.4417 | 0.034252608017 / 600 / 2.5000 | P + (U if pass else L) | 0.0889 | + +## Total GPU estimate versus 3.0 cap + +The 25 primary anchors contain 8 TP1, 8 TP2, and 9 TP4 replays. At the +80-second per-anchor planning charge they consume `0.1778 + 0.3556 + 0.8000 = +1.3334` H20-hours. Four shared startup/warm-up/cleanup envelopes plus runtime +variance are budgeted at 1.3166 hours, for a **2.65-H20-hour primary plan**. +The remaining **0.35 H20-hours** is reserved for boundary confirmations or +decision-critical outward crawling. The absolute launch cap is **3.0 +H20-hours**; projected or actual spend reaching the cap stops all further work. +Expected wall time is 25-35 minutes over four waves. Expected public disk is +0.3-0.6 GB and must remain below 3 GB. + +## Decision rules + +1. Anchor feasibility is exact-output request pass rate `>=0.95` using raw-length + stepped TTFT and 50-ms TPOT; absent/early-stopped requests remain failures. +2. Boundary movement is an old peak feasible->infeasible flip or an old adjacent + infeasible->feasible flip. Material frontier churn is `abs(drift)>5%`. +3. Apply zero-anchored floor buckets with set-wide `tol=max(1e-9,1e-6*max|s|)` + before every tie/rank decision. +4. ARGMAX MOVED iff TP2/MNS32 is outside the v0.24 top bucket on a bounded + 12-cell subset; censored contenders make survival inconclusive. +5. RANKING SURVIVES only with all 12 cells, argmax survival, tau-b >=0.8, and no + >5% top-bucket pair reversal. Tau-b <0.8 is RANKING MOVED. +6. TP4/MNS16 trap persists only when it is no worse than adjacent MNS8/MNS32 and + remains below the global best; an improving neighbor means escape. +7. Boundary `[0.93,0.97]` or verdict-changing anchors require confirmation when + budget permits. Disagreement without a 2-of-3 resolution is inconclusive. + +## Open decisions for orchestrator + +1. Approve the **upgrade-path estimand**: resolved default changes are part of + churn, while dash1->dash0 and co-location remain explicit limitations rather + than pretending this is a pure engine-version causal effect. +2. Approve the **adaptive 25-anchor subset**, which gives every cell a + direction-relevant neighbor and the trap both neighbors, instead of spending + next-infeasible anchors only on a tie-ambiguous top four. +3. Approve **5% material frontier drift**, `tau-b>=0.8`, and the stated + argmax/trap floor-bucket rules. +4. Approve one primary observation per historical anchor with the 0.35-hour + reserve for `[0.93,0.97]`/verdict-changing confirmations, rather than reducing + surface coverage to replicate every anchor. +5. Approve the A-P5-1-class warm-up adaptation from `input>=8192` to + **raw input >4096**, the longest SLO-relevant tier available reliably in this + 0-8192 C1 workload. + +## Sanity block + +| Numeric family | n | Min | Max | Distinct | Checked invariant/result | +|---|---:|---:|---:|---:|---| +| C1 cells | 12 | TP=1,MNS=8 | TP=4,MNS=64 | 12 | Exact 3x4 Cartesian surface | +| Historical probe observations | 92 | 7/cell | 8/cell | 2 per-cell counts | `8*8 + 4*7 = 92` | +| Materialized trace rows | 1 file | 32,606 | 32,606 | 1 expected | SHA pinned; prompt content not emitted | +| Raw-length-filtered rows | 1 reconstruction | 17,710 | 17,710 | 1 expected | Raw length 22-8192; non-negative | +| Capped TP1/TP2 source rows | 1 set | 512 | 512 | 1 expected | Stable time sort, then even downsample | +| Reconstructed historical counts | 92 | 66 | 600 | 34 | 92/92 match ground truth | +| C1 peak scores (req/s/GPU) | 12 | 1.283333 | 3.283333 | 8 | Finite, positive, not all identical | +| v0.20 floor tolerance | 1 vector | 3.283333e-6 | 3.283333e-6 | 1 | Applied before display rounding | +| Nominal top-4 effective size | 1 cutoff | 5 | 5 | 1 | TP4 three-way cutoff tie retained | +| Primary peak anchors | 12 | 1/cell | 1/cell | 1 expected | No cell omitted | +| Realized adjacent/trap anchors | 13 | 1/non-trap | 2/trap | 2 | Predeclared direction rule; total 25 | +| Primary anchors by TP | 25 | 8 at TP1/TP2 | 9 at TP4 | 2 counts | `8+8+9=25` | +| Selected requests in anchor candidates | 36 candidates | 121 | 600 | >1 expected | TP1/2 cap-before-threshold; TP4 uncapped | +| Per-anchor planning cost (H20-hour) | 3 TP classes | 0.0222 | 0.0889 | 3 | Proportional to TP for 80 seconds | +| Primary planned H20-hours | 1 plan | 2.65 | 2.65 | 1 | Non-negative; below 3.0 | +| Confirmation/crawl reserve | 1 plan | 0.35 | 0.35 | 1 | Primary plus reserve equals hard cap | +| GPU work in this protocol turn | 1 turn | 0 | 0 | 1 expected | Protocol-only requirement satisfied | + +Checked invariants: `12=3*4`; `92=8*8+4*7`; all 92 local selection counts +match; cap precedes threshold; arrivals span the fixed 60-second replay; raw +input length, not retokenized usage, selects TTFT thresholds; completion is +exactly 128; SLO denominator includes all selected requests; surface scores use +60 seconds and divide by TP; floor buckets precede ranking; the top-four cutoff +expands to five; `25=12+11+2`; planned `2.65+0.35=3.0`; ratios and counters have +their declared domains; and no GPU run, helper, manifest, remote artifact, or +results file was created in this protocol-only turn. diff --git a/docs/opprof/phase6-results.md b/docs/opprof/phase6-results.md new file mode 100644 index 0000000..2d77fde --- /dev/null +++ b/docs/opprof/phase6-results.md @@ -0,0 +1,226 @@ +# OpProf Phase 6 dash0 results + +Status: **COMPLETE, DATA-VALID A-P6-2 SOLO TIER; ARGMAX/RANKING/TRAP INCONCLUSIVE UNDER THE FROZEN RULES**. + +Phase 6 completed all 12 C1 cells on patched vLLM +`0.24.1.dev3+g668cfb7e2` (`4b253fd`). A-P6-2 makes the serialized solo tier +authoritative because the v0.20 baseline was solo and the original W2/W3 +confirmations exposed large pass-rate and queue changes after neighboring wave +clients became idle. The authoritative campaign contains 37 primary anchors, +29 same-placement confirmations, and 12 valid Layer-1 streams. + +The machine-readable result is +`runs/opprof-phase6/phase6/metrics.json`, SHA-256 +`290ba7fcb8727291166de7e4d47afdc84e230052495c81dd087db0ace9f93a16`. +Co-located W1-W3 artifacts are preserved beside the solo tree rather than +overwritten. Public artifacts contain no prompt or generated text. + +## Data sanity first + +There are **no data-sanity red flags**. All 37 authoritative primary anchors +reproduce the pinned v0.20 request count exactly; all client invariants pass; +all 12 solo cells have one graceful footer and agreeing final sidecar; Layer-1 +indices are contiguous with zero drops; every anchor interval is represented; +and all cold-start compile/capture events precede measurement. + +The scientific decision gate is separate from data validity. Eight cells have +bounded monotonic solo frontiers. TP2/MNS16, TP4/MNS32, and TP4/MNS64 pass the +highest recorded request set and are right-censored at the pinned-history edge. +TP4/MNS16 is non-monotonic: L resolves feasible, P resolves infeasible, and U +resolves feasible. These four cells prohibit a full-surface floor-bucket rank, +tau-b, and trap decision; no value is imputed beyond the recorded history. + +## Launch echo + +```text +A-P6-2_LAUNCH_ECHO host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.291173 new_est_h20h=3.440 cumulative_est_h20h=5.731173 cap_h20h=6.0 est_wall=45-70min +SOLO_WAVE_ECHO order=TP4/MNS32,TP4/MNS64,TP2/MNS32,TP2/MNS64,TP4/MNS16,TP2/MNS8,TP2/MNS16,TP4/MNS8,TP1/MNS8,TP1/MNS16,TP1/MNS32,TP1/MNS64 placement=one_server_one_client +FINAL_ACCOUNTING status=complete cells=12 primary=37 confirmations=29 solo_h20h=3.353371 campaign_h20h=5.644544 cap_h20h=6.0 +``` + +The exact per-cell echoes, GPU placement, trace and ground-truth paths, spend, +and projection are retained in +`runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log`. + +## Run completion stats + +| Item | Result | +|---|---:| +| Historical selections reconstructed | 92/92 exact | +| Authoritative solo cells | 12/12 | +| Solo primary anchors | 37 | +| Solo confirmations | 29 | +| Solo anchor trials | 66 | +| Solo warm-ups | 12; exactly 16 completions each | +| Long prompts per warm-up | 2-8 raw inputs `>4096` | +| Bounded solo frontiers | 8/12 | +| History-edge right-censored cells | 3 | +| Non-monotonic cells | 1 | +| Prior co-located primary/confirm trials | 21 + 3 | +| Total accepted anchor trials across attempts | 90 | +| Co-located spend before A-P6-2 | 2.291173 H20-hours | +| A-P6-2 solo spend | 3.353371 H20-hours | +| Total campaign spend | **5.644544 H20-hours** | +| Artifact footprint | 467 MiB, below 3 GiB | + +## Final paired surface + +`A*` marks the authoritative A-P6-2 solo tier. Exact values are bounded +frontiers. `>=` is a recorded-history lower bound. The TP4/MNS16 value is not a +frontier because its feasibility sequence is non-monotonic. + +| Cell | Tier | v0.20 frontier | v0.24 authoritative result | Drift | Status | +|---|---|---:|---:|---:|---| +| TP1/MNS8 | A* solo | 2.1000 | **2.3833** | **+13.49%** | bounded up | +| TP1/MNS16 | A* solo | 2.3500 | 2.3833 | +1.42% | bounded up | +| TP1/MNS32 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up | +| TP1/MNS64 | A* solo | 2.2833 | 2.3833 | +4.38% | bounded up | +| TP2/MNS8 | A* solo | 2.2750 | 2.2417 | -1.47% | bounded down | +| TP2/MNS16 | A* solo | 2.2750 | >=2.3000 | >=+1.10% | right-censored history edge | +| TP2/MNS32 | A* solo | **3.2833** | **3.2583** | -0.76% | bounded down | +| TP2/MNS64 | A* solo | 3.2583 | **2.3000** | **-29.41%** | bounded down | +| TP4/MNS8 | A* solo | 1.2833 | 1.3208 | +2.92% | bounded up | +| TP4/MNS16 | A* solo | 2.4417 | N/E; observed max feasible 2.5000 | N/E | **non-monotonic: L pass, peak fail, U pass** | +| TP4/MNS32 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge | +| TP4/MNS64 | A* solo | 2.4417 | >=2.5000 | >=+2.39% | right-censored history edge | + +The solo tier validates two material frontier changes under the predeclared 5% +threshold: TP1/MNS8 improves by 13.49%, while TP2/MNS64 declines by 29.41%. +The previous TP2/MNS64 decline is now quotable because both its passing 2.3000 +anchor and failing 3.2583 anchor were repeated solo. + +## Frozen decision-rule verdicts + +- **ARGMAX: INCONCLUSIVE.** TP2/MNS32 is the highest bounded observed frontier + at 3.2583 req/s/GPU, but the frozen rule requires a bounded 12-cell subset. + TP2/MNS16 and TP4/MNS32/64 terminate at passing history-edge anchors, and + TP4/MNS16 has no coherent frontier. The top floor bucket is therefore not + defined for all contenders. +- **RANKING: INCONCLUSIVE.** Only 8/12 frontiers are bounded. Per protocol, + Kendall tau-b is **not evaluable** (`tau_b=null`) rather than computed from + censored lower bounds. The `tau-b>=0.8` survival rule cannot be applied. +- **TRAP: INCONCLUSIVE.** TP4/MNS16 is non-monotonic and its TP4/MNS32 neighbor + is right-censored. Neither trap persistence nor escape satisfies the frozen + floor-bucket rule. + +## Solo versus co-located exact-anchor deltas + +The co-located column is the simultaneous-wave primary. The solo column is the +accepted median after same-placement 2-of-3 adjudication. Delta is percentage +points. `P/F` are SLO feasible/infeasible at 0.95. + +| Cell | Anchor | Co-located primary | Solo authoritative | Delta | State change | +|---|---:|---:|---:|---:|---| +| TP1/MNS8 | 0.21875000 | 0.992 | 0.992 | +0.00pp | P->P | +| TP1/MNS8 | 0.22656250 | 0.206 | 0.992 | **+78.57pp** | F->P | +| TP1/MNS16 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P | +| TP1/MNS16 | 0.25000000 | 1.000 | 1.000 | +0.00pp | P->P | +| TP1/MNS32 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P | +| TP1/MNS32 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P | +| TP1/MNS64 | 0.24218750 | 1.000 | 1.000 | +0.00pp | P->P | +| TP1/MNS64 | 0.24609375 | 1.000 | 1.000 | +0.00pp | P->P | +| TP2/MNS8 | 0.49218750 | 0.691 | 1.000 | +30.86pp | F->P | +| TP2/MNS8 | 0.49609375 | 0.092 | 0.581 | +48.90pp | F->F | +| TP2/MNS16 | 0.49218750 | 1.000 | 1.000 | +0.00pp | P->P | +| TP2/MNS16 | 0.49609375 | 0.088 | 1.000 | **+91.21pp** | F->P | +| TP2/MNS32 | 0.75000000 | 0.412 | 1.000 | +58.82pp | F->P | +| TP2/MNS32 | 0.75390625 | 0.028 | 0.575 | +54.70pp | F->F | +| TP2/MNS64 | 0.50000000 | 0.460 | 1.000 | +53.99pp | F->P | +| TP2/MNS64 | 0.75000000 | 0.141 | 0.561 | +42.07pp | F->F | +| TP4/MNS8 | 0.016055910008 | 1.000 | 1.000 | +0.00pp | P->P | +| TP4/MNS8 | 0.016591107009 | 0.071 | 1.000 | **+92.86pp** | F->P | +| TP4/MNS16 | 0.033182214016 | 0.609 | 1.000 | +39.13pp | F->P | +| TP4/MNS16 | 0.033717411016 | 0.036 | 0.238 | +20.22pp | F->F | +| TP4/MNS16 | 0.034252608017 | 0.847 | 1.000 | +15.33pp | F->P | + +The deltas range from 0 to +92.86pp across 21 exact pairs. Four TP1/MNS16-64 +pairs are unchanged, while several TP2/TP4 and TP1/MNS8 anchors flip. Thus +co-location validity is **metric- and cell-dependent**, not a universal offset. +Throughput-only values may co-locate, but SLO-frontier feasibility cannot be +mixed across placement tiers. + +## Layer-1 mechanism notes for material drift + +Layer-1 is mechanism context, not a matched v0.20 causal decomposition. P3 P10 +is TP1 at a different workload/output contract. All 37 solo primaries have +zero preemptions. + +| Cell/anchor | State | Waiting mean/max | Decode-B mean | KV mean | `NONE` graph share | Padding | +|---|---|---:|---:|---:|---:|---:| +| TP1/MNS8, 0.25 | frontier pass | 0.74 / 8 | 4.25 | 0.0407 | 3.26% | 0.34% | +| TP1/MNS8, 0.50 | next fail | 11.66 / 27 | 7.19 | 0.0644 | 5.04% | 0.02% | +| TP2/MNS64, 0.50 | frontier pass | 0.00 / 0 | 3.28 | 0.0084 | 0.00% | 18.88% | +| TP2/MNS64, 0.75 | old-peak fail, primary | 0.00 / 0 | 18.05 | 0.0448 | 9.47% | 0.52% | +| TP2/MNS64, 0.75 | old-peak fail, confirm | 0.0002 / 1 | 11.52 | 0.0285 | 4.57% | 1.13% | + +For TP1/MNS8, the new frontier remains feasible despite a larger decode batch +and modest queueing; failure at 0.5 coincides with a 16x waiting-mean increase, +higher KV usage, and more eager/non-full graph execution. For TP2/MNS64, the +passing lower anchor has small decode batches and full/piecewise graphs despite +high padding, whereas both old-peak trials fail with much larger decode batches +and graph fallback. Zero preemptions and near-zero queueing rule out preemption +and server waiting as the main TP2/MNS64 explanation, but the unmatched v0.20 +telemetry prevents assigning causality uniquely to graph mode or batch shape. + +## Operational findings and full attempt history + +1. The initial ownership-monitor false start charged 0.132918 H20-hours without + measurement. The repaired W1 charged 0.299393; A-P6-1 later accepted its + eight anchors from balanced checkpoint sidecars without rerun. +2. Co-located W2/W3 completed 13 primary anchors and three confirmations but + exposed placement-sensitive SLO results. The 3.0-hour projection stopped W4 + at 2.291173 total; those values remain indicative and preserved. +3. A-P6-2 completed W4 and remeasured the full surface solo. Every solo server + used `--shutdown-timeout 120`, emitted a graceful footer, and passed Layer-1 + accounting. +4. Placement was not the only transient. TP4/MNS32 and TP4/MNS64 first full + replays failed at 0.031/0.114 but subsequent identical solo trials passed; + TP4/MNS16 produced an order-dependent non-monotonic sequence. The 16-request + A-P5-1 gate removes compile/capture cold start but not necessarily the first + full-trace transient. The predeclared 2-of-3 rule prevented single-trial + conclusions, but a future protocol should use a full-trace burn-in or + randomized/reversed anchor order. +5. Two cleanup checks briefly observed 1-4 MiB at 0% with zero compute PIDs. + Memory decayed to 0 MiB within seconds, graceful footers validated, and no + rerun was needed. Cleanup now polls for 30 seconds rather than treating + transient driver bookkeeping as a failed experiment. + +## GPU total + +The complete campaign charged **5.6445441643 H20-hours** against the +user-approved **6.0** cap: + +- original co-located attempts: 2.2911728495; +- A-P6-2 authoritative solo tier: 3.3533713148; +- remaining headroom: 0.3554558357 H20-hours. + +Final dash0 state is 0 MiB and 0% utilization on all eight H20s, with zero +controller, client, vLLM, EngineCore, or worker processes. + +## Sanity block + +| Numeric family | n | Min | Max | Distinct | Checked invariant/result | +|---|---:|---:|---:|---:|---| +| Historical selection checks | 92 | 66 requests | 600 requests | 34 | 92/92 exact | +| Solo primary pass rates | 37 | 0.0307 | 1.0000 | 17 | Ratios in `[0,1]`; not identical | +| Solo selected requests | 37 | 121 | 600 | 18 | Exact v0.20 count per anchor | +| Layer-1 steps per primary | 37 | 343 | 12,103 | 37 | Contiguous; non-negative; zero drops | +| Layer-1 records per cell | 12 | 14,174 | 58,725 | 12 | Footer/sidecar balanced | +| Solo preemptions | 37 | 0 | 0 | 1 expected | Zero throughout | +| Reported v0.24 frontier values/bounds | 11 | 1.3208 | 3.2583 | 6 | TP4/MNS16 omitted as non-monotonic | +| Frontier drift values/bounds | 12 | -29.41% | +13.49% | 9 | 11 finite; one non-monotonic missing | +| Bounded/censored cells | 12 | 8 bounded | 4 unbounded | 2 | Censoring never imputed | +| Exact solo/co-located deltas | 21 | +0.00pp | +92.86pp | 13 | Same request set and anchor | +| Solo confirmation pass rates | 29 | 0.4130 | 1.0000 | 10 | Frozen 2-of-3 applied | +| Solo cell H20-hours | 12 | 0.0845 | 0.5534 | 12 | Non-negative; serialized placement | +| Total campaign H20-hours | 1 | 5.644544 | 5.644544 | 1 | Below 6.0 hard cap | +| Final GPU memory/utilization | 8 | 0 MiB / 0% | 0 MiB / 0% | 1 expected | Zero compute processes | + +Checked invariants: exact cap-before-threshold selection; raw-length SLO +evaluation; exact 128-token completions or counted failures; nondecreasing +arrivals; all selected outcomes represented in the denominator; one solo +server/client at a time; compile/capture outside measured intervals; 12/12 +graceful footer balances; non-negative counters; pass ratios in range; no prompt +text in public artifacts; complete attempt-history retention; authoritative +tier separation; explicit censoring/non-monotonicity; tau-b suppression when +the full surface is unbounded; hard-cap compliance; and complete GPU cleanup. diff --git a/docs/opprof_campaign_state.md b/docs/opprof_campaign_state.md new file mode 100644 index 0000000..4501483 --- /dev/null +++ b/docs/opprof_campaign_state.md @@ -0,0 +1,80 @@ +# OpProf campaign state — pattern-conditioned operator profiling (vLLM 0.24.0 / Qwen3-30B-A3B / H20) + +Started 2026-07-11. Orchestrator: Claude (session a42d23fb). Workers: codex via companion. +Discipline inherited from SimFid campaign (replayserve/docs/simfid_campaign_state.md): +pre-registered acceptance gates per phase, codex implement + independent review, +orchestrator verifies, echo-before-expensive-action, GPU actions logged. + +## User decisions (2026-07-11) +- vLLM pinned to **0.24.0** (latest community); record commit + sha256 at clone time. +- GPU: **dash0 8×H20, approved for the whole campaign** (Phases 2-4). Orchestrator + still echoes load/duration before each launch. Single H20 sufficient for TP1. +- Observation patch: **dual-layer** — always-on lightweight per-iteration composition + telemetry (<3% overhead budget) + sampled heavy torch.profiler/CUPTI windows. +- L-C-A is REFERENCE ONLY in this campaign — pattern axes defined operationally + (input-len dist / output len / arrival shape / prefix sharing), no L-C-A dependency. + +## Phase plan (pre-registered) +- P0 recon (local): clone v0.24.0, inventory built-in observability, patch design doc. + Gate: USER reviews patch design before P1 implementation. +- P1 patch dev + no-GPU tests (local). Gate: strict review + orchestrator verify. +- P2 single-H20 smoke on dash0: artifacts complete; measured overhead (on/off same + load) within budget; op-time sums ≈ iteration time; composition matches request-level + ground truth. Hard gate before P3. +- P3 pattern matrix (~8-12 patterns × 2-3 configs, short replays, dash0): per-pattern + operator time breakdown + composition distributions + waste accounting + (padding% / graph-miss / ragged imbalance / mixed-batch interference). + Hypothesis go/no-go: does operator bottleneck ranking change across patterns? +- P4 optimization proposal (ranked, measured bounds) + one cheap config-level + closed-loop validation (e.g., cudagraph capture sizes vs measured B distribution). +- Non-goals: no new kernels; no L-C-A ablation (separate track); no production clusters. + +## Artifact layout +- vLLM source: /home/gahow/phd/vllm-v0.24.0 (clone, pinned) +- Patches: aituner patches/vllm-0.24.0-opprof/ (patch files + apply script + no-GPU tests) +- Docs: aituner docs/opprof/ (phase0-recon.md, patch-design.md, later phase reports) + +## Job log +- P0 recon dispatched: task-mrg3nm1f-spyl85 (2026-07-11T08:25:42Z). Scope: clone v0.24.0 to ~/phd/vllm-v0.24.0, observability inventory with file:line, dual-layer patch design doc. Gate: user reviews design before P1. +- P0 ACCEPTED (task-mrg3nm1f, 34m54s): pin ee0da84a=v0.24.0 verified; VLLM_TORCH_PROFILER_DIR-absent claim verified; profiler moved to ProfilerConfig + entrypoints/serve/profile/api_router.py (verified); design doc 421 lines. USER APPROVED design with 5 delegated decisions (JSONL+msgspec+8192 queue; expert loads L2-only; 2+8 profiler window; 3% gate on 95% CI upper bound; reject --disable-log-stats) + checkpoint decision: community BF16, TP1 primary, TP2/4 counterpoints. +- P1 dispatching: implement per approved design; branch in ~/phd/vllm-v0.24.0 + exported patches/vllm-0.24.0-opprof/ in aituner; no-GPU tests must pass locally (opprof core import-light). +- P1 completed (task-mrg588j2, 26m36s): branch opprof tip c60e7eeb, +560/-1 across 5 files (within ±30% of design). Orchestrator pre-checks: pytest 11/11 reproduced; patch series applies cleanly on pristine v0.24.0 worktree and applied tree is byte-identical to branch (diff=0); scheduler/model_runner integration diff reviewed — surgical, matches design hooks. Strict review dispatching. +- Strict review (task-mrg68puk, 14m59s): FAIL — 0 Blocking / 3 Major / 1 Minor. M1 writer-thread silent death + shutdown deadlock (opprof.py:106/145); M2 apply.sh already-applied bypasses base/dirty checks; M3 foreground raw lists violate no-raw-list hotspot contract (opprof.py:182). Minor: missing zero-token+None-stat test; true Scheduler integration test DEFERRED to P2 smoke gate (no local vllm install). Reviewer verified OK: fail-fast log_stats ValueError, pairing across sync/batch-queue/PP/spec-decode, schedule-time snapshots, DP-unique paths, patch-id equality 3/3. Orchestrator adjudication: ACCEPT all; fix worker dispatching (fresh task — resume-last would hit reviewer thread). +- P1 fix round CLOSED (task-mrg6tnum, 13m34s): branch tip 668cfb7e (3 clean commits on ee0da84a). Orchestrator closure: 14/14 tests reproduced; M3 raw-lists-gone verified (bisect accumulation); BOTH apply.sh refusal cases independently reproduced (dirty exit=1; wrong-base-with-identical-patches exit=1); M1 writer failure handling inspected (one-time log, liveness check, bounded close, drop visibility). P1 ACCEPTED. +- ECHO P2 (GPU, dash0): deploy patched vLLM 0.24.0 (python-only patch, VLLM_USE_PRECOMPILED path) on dash0, serve Qwen3-30B-A3B BF16 TP1 on ONE free H20; artifacts smoke + overhead gate (interleaved 5xON/5xOFF identical load, 3% gate on 95% CI upper bound) + one 2+8 Layer-2 profiler window + integration assertions (records==steps, no pending leak, footer accounting). Est 1.5-2.5h wall, single GPU. Deferred review item (real Scheduler integration) is covered by these assertions. +- P2 dispatched: task-mrg7erq5 (dash0 smoke; ssh dash0 ONLY authorized; hard gates: artifacts/schema/footer/no-pending-leak, overhead 95%CI-upper <=3%, Layer-2 window loadable). Est 1.5-2.5h. +- P2 first run (task-mrg7erq5, 1h37m): overhead gate FAIL honored (13.11%, CI [6.24,19.53]) — but measurement judged INVALID by orchestrator: OFF-arm spread 29% across identical runs, one pair NEGATIVE overhead, 10s window with cold-start per run, ON-first order confound (run 1 = coldest = ON). Physical bound: per-step cost is bisect+encode+enqueue at ~11 steps/s. Artifacts gate passed except PIECEWISE not exercised by smoke load. 14 remote tests pass; TRITON backend 12/12; cleanup verified. AMENDMENT A-P2-1 (pre-registered before rerun): warmup excluded, >=120s steady-state window, ignore_eos fixed output lengths, ABBA counterbalanced order + discard first pair, host load recorded, recorder microbenchmark as secondary evidence, one PIECEWISE-exercising artifact run. GATE UNCHANGED (3% on 95% CI upper). +- P2 A-P2-1 rerun (task-mrgdbfju + predecessor, valid measurement): overhead gate REAL FAIL — 4.1816%, CI [3.1364, 4.7117], gate 3%. ON tight (28.18-28.26), OFF (28.80-29.64), pairs 2.07-4.74%. Artifacts PASS incl. PIECEWISE addendum (129 records, accounting balanced). Layer-2 correctly not run. LOCALIZATION PUZZLE: producer microbenchmark 29.1µs/step x 9.7 steps/s ≈ 0.04s/run vs observed ON-OFF gap 5.9s — direct recording cost explains <1% of overhead; structural suspect (capture scan @200 concurrency, writer-thread GIL, queue locking, flush I/O locus). GPU spend so far 117 H20-min. Diagnostic+fix round dispatching (stage-bisection first, minimal fix, gate rerun). +- Bisection round (task-mrge67ke, 34m9s): recorder pipeline exonerated — all 4 stages ~0.5% vs stage-off; BUT all stages kept VLLM_OPPROF_DIR set → gpu_model_runner CUDAGraphStat construction/propagation path active in all; cross-attempt (attempt-2 true-OFF 29.45 vs bisection off 28.39) localizes ~3.6% to that path. Worker honored outside-OpProf stop boundary; temp diffs reversed; 14/14 tests both sides. Orchestrator hypothesis for next round: CUDAGraphStat in ModelRunnerOutput poisons msgspec IPC fast path (pickle fallback for whole output) OR upstream cudagraph_metrics feature carries inherent cost. Next: STATIC analysis first, then 3-trial confirm (true-baseline / upstream-metrics-only / env-set-recorder-off), then targeted fix. +- P2 CLOSED: ALL GATES PASS (task-mrgfgedp, 2h17m). ROOT CAUSE of the 4.18%: VLLM_OPPROF_DIR entered compile_factors() (envs.py:2011-2105) → hashed into torch.compile cache path (backends.py:1024-1065) → every unique ON dir = cold graph/AOT cache (compile 36.41s vs 6.07s) and ~4% slower artifacts. Recording itself was NEVER the cost (recorder 29.1µs/step; serializer-poisoning hypothesis WRONG — TP1 in-process executor, msgspec handles CUDAGraphStat natively). 3-trial confirmation: baseline 29.72 / upstream-metrics-only 29.64 (0.25%) / env-set 28.45 (4.25%). Fix: +1 production line (ignore-list) + tests; tip bbfa717; 5-patch series; 15/15 tests x3 environments. FINAL GATE: overhead -0.042%, CI [-0.174, +0.046] — PASS. Layer-2 PASS (16.4MB Kineto, 2+8 window, 790,843 events, 7,367 kernel events; active-window perturbation 51.3% — confirms sampled-only design). Artifacts PASS (PIECEWISE 129/139). Orchestrator verified: fix commits, ignore-list diff, 15/15 local rerun, 5 patches. Caveat noted: attempt-4 JSONLs footerless (simultaneous shutdown) — excluded from accounting. UPSTREAM FINDING worth reporting: any per-run-unique env var hashed into compile factors silently causes cold-compile + slower artifacts. +- P3 dispatching: protocol-first (worker drafts docs/opprof/phase3-protocol.md, STOPS for orchestrator review before execution). +- USER DIRECTIVE (2026-07-12): parallelize P3 across 8xH20, one experiment per GPU (~8x speedup). Orchestrator caveat: no GPU contention but shared host CPU/memory-bandwidth (API server/tokenizer/scheduler/bench are host-side; P2 showed host-side effects can fabricate %-level differences; saturation-point calibration depends on absolute throughput). P3 protocol must include: pre-registered co-location validity check (same cell solo vs alongside 7 neighbors; throughput + op-share deltas <2-3% to authorize 8-way; fallback 4-way + revalidate), CPU affinity pinning per server/client pair, and host load recording per run. +- P3 protocol (task-mrh6vfqp, 27m13s, 835 lines): ACCEPTED with orchestrator amendments. Dispositions on 6 open decisions: (1) fixed-duration client build+test APPROVED; (2) P10 private-trace transfer to dash0 CPFS wjh APPROVED (sampling_u<=0.125, input<=32768, output cap 256; no prompt text in artifacts; data returns to same org cluster it came from); (3) MNS{64,1024}xMBT{2048,8192} sparse factorial APPROVED; (4) P10/C00 sole TP2 counterpoint APPROVED; (5) 60% load / burst-8 / 240s / 4 windows APPROVED; (6) kernel mapping + 70% classifiability gate + thresholds APPROVED. AMENDMENT A-P3-1 (user directive): 8-way GPU parallelism, one cell per GPU; pre-registered co-location validity check (same cell solo vs with 7 neighbors, throughput AND op-share deltas <3% to authorize; fallback 4-way + revalidate); CPU affinity pinning (disjoint core sets per server+client); saturation calibration under the SAME co-location regime as measurement; host load + clocks per run; revised wall estimate ~1.5-2h. AMENDMENT A-P3-2 (E2 lesson): execution via detached resumable controller script on dash0 (setsid, --resume, state file) so runs survive codex turn deaths. +- P3 E-a ACCEPTED (task-mrh7wd5x, 1h27m, 3.96 H20-h): co-location gate → 8-way FAIL (throughput Δ=0.000% but op-share shifts up to 4.2pp > 3% threshold), 4-way PASS (shares ≤0.075pp) — AUTHORIZED 4-WAY. Client+controller 12/12 no-GPU tests; provenance hashed. P10 transfer verified (4,011 rows, tokenizer parity 4011/4011 exact, 0600/0700 perms, no prompt text public). Orchestrator decisions: (1) 4-way verdict accepted despite validation-stream footer absence (shares from Kineto, unaffected); (2) shutdown-fix (API-parent-first) GPU quick-verify REQUIRED before E-b matrix; (3) A-P3-3: per-class drain budget — 240s for output-512 burst cells (P04-class), 120s others; (4) A-P3-4: P3 hard stop raised 8→16 H20-hours (user blanket dash0 approval; E-a consumed 3.96). +- E-b step1 STOP (task-mrhb2dxw, 10m42s): shutdown-fix verification FAILED hard footer gate — vLLM 0.24.0 API shutdown selected mode=abort timeout=0s and SIGTERMed EngineCore before scheduler/opprof teardown; finally-footer never runs. Run itself healthy (5268 reqs, 43.9 req/s, 1701 records, 0 drops, step range contiguous). Matrix NOT launched (correct). Orchestrator decision: footer must not depend on graceful shutdown — (a) 10-min static check for an official graceful path in 0.24.0; if unreliable, (b) checkpoint-footer sidecar in opprof.py (atomic write each flush cadence; accounting gate accepts sidecar when stream footer absent; bounded loss = last flush interval), new commit + re-export + tests, GPU re-verify incl. abort-kill case, then matrix. +- E-b round 2 (task-mrhbhj9j, ~1h5m): sidecar fix DONE (commit f8b68f24, +195/-7, patch tip 23450fb2, 7 patches, 18/18+12/12 tests; graceful path found: vllm serve --shutdown-timeout N; dual GPU verification PASS incl. SIGKILL with 24ms checkpoint sidecar balancing). Matrix launched, STOPPED at pre-registered drain gate: P10/C01 saturation drain 288.6s > 120s budget (clean window + accounting VALID; drain is post-measurement hang-watchdog only; 0.74 req/s saturated 32k-context queue drains slowly by nature). 3/4 first-wave runs PASS with plausible ordering. GPU 5.47/16. AMENDMENT A-P3-5: P10-class drain budget 600s; drain violations = quarantine-run-and-continue (stop only if >20% of runs violate); P10/C01 existing data re-adjudicated under amended rule if accounting balanced. +- NETWORK OUTAGE (~19:2xZ 2026-07-12 onward): dash0 AND dash1 unreachable from local (pre-auth timeout) — ingress/network fault, not host failure. Detached matrix controller on dash0 unaffected by design (A-P3-2); worker holds 1/min read-only reachability probe with retained artifact hashes for post-recovery integrity check. On reconnect: verify controller state (expect matrix advanced or complete), resume worker thread if its turn died. +- E-b round 3 blocked cleanly on ssh outage (task-mrhdu37w): resumable handoff at runs/opprof-phase3/phase3/access-blocker-20260712.json; last durable 8/52 accepted, 0 quarantine, 0 clean-failures, 6.82 H20-h; controller PID 2237019 untouched, likely still advancing locally. Also delivered mid-flight: P03 profile-control starvation client fix (13/13 tests) + frozen analyzer analyze_phase3.py (4/4 tests). Reconnection watcher armed. +- NETWORK RESTORED (~21:1xZ per user). Controller advanced 8→12/52 during outage then FAILED ~2.2h ago (idle since). GPU 8.13/16. E-b worker resumed (task-mrhkbq8s): integrity check vs retained hashes → diagnose controller failure → minimal fix → --resume matrix (40 runs remain) → frozen analysis. +- E-b round 4 (task-mrhkbq8s, ~2h): 40/52 accepted (20/24 cells), 0 clean-window failures, 542,350 L1 records, 72 L2 traces, GPU 13.31/16. Two aux-gate fixes shipped (+58/-11, 15/15+4/4 tests): profiler-control connection, clean-window failure boundary (post-clean disconnects wrongly counted), 3-sample GPU preflight. FINAL BLOCKER: P10/TP2 warm-up gate demands 32/32 completions; long-context prompts cannot finish in budget (15/32, 21/32) though clean windows are valid. Same failure class as drain budget: auxiliary gate miscalibrated for long-context pattern. AMENDMENT A-P3-6: P10-class warm-up gate = 32 completions OR (>=16 completions AND Layer-1-verified throughput stabilization over trailing warm-up steps), with mandatory post-hoc stability evidence in the report. Finish remaining cells within 16h cap; if cap hits, stop and analyze with documented gaps. +- E-b round 5 (task-mrhnw1rm, 23m46s): P10/TP2 FAILED frozen A-P3-6 stabilization on fresh data (drift 36.76% vs 10%; bins 11/10/16 vs 16) — the cell genuinely does not stabilize; clean window itself valid. Worker correctly refused to relax. GPU 14.03/16. ORCHESTRATOR DECISION: freeze matrix at 40/52 runs / 20/24 cells; abandon remaining 12 runs. AMENDMENT A-P3-7: per-contrast evaluation — each frozen contrast evaluates iff both cells complete; missing → NOT EVALUABLE (never imputed); H1a/H1b use existential logic on completed cells (CONFIRM possible, REFUTE NOT possible at 20/24 — must be stated explicitly). P10/TP2 non-stabilization recorded as a pattern-conditioned operational FINDING. +- PHASE 3 FINAL (task-mrhos386, 19m37s): **H1a INCONCLUSIVE / H1b PASS (5 of 6 evaluable contrasts) / compound PARTIAL** under A-P3-7. H1b effects large + Holm-corrected p≈0: P10 real trace R64 +44.79pp vs both long rectangular controls (efficiency −14.3%/−44.7%); P09 production mix R64 +39.62pp, padding +6.67pp (−8.3%); P06 bimodal burst R64 +23.0/+35.4pp (−11.6%/−22.8%). H1a inconclusive because only 1/9 patterns Layer-2 windows passed representativeness gates (descriptive: P06/P10 flip attention-led↔MoE-GEMM-led across load points). Orchestrator verified metrics.json per-contrast records match report exactly. GPU final 14.03/16 (1.97 unused). P4 dispatching. + +## Phase 5 — mechanism-decomposition ablations (USER-APPROVED 2026-07-12) +- Motivation (user challenge): the sign of H1b is obvious; the contribution is the mechanism LEDGER. vLLM does no request-level padding (continuous batching); measured bucket padding (+5.4-6.7pp) explains ~1/6 of the 14-45% E_token gap; the remaining ~30pp attribution (ragged-attention SM imbalance / cudagraph bucket mismatch / chunked-prefill mix interference / MoE routing skew / scheduler batch-variance) is unmeasured in the literature. +- Design skeleton (protocol to formalize): controlled ablations each isolating ONE mechanism on P10 (primary) + P09/P06 (secondary): (i) length-sorted/binned replay [kills intra-cohort raggedness, keeps content+totals]; (ii) capture sizes matched to measured decode-B distribution [kills bucket mismatch; doubles as the config-tier deliverable]; (iii) arrival-shape ablation [steady vs burst, same lengths]; (iv) prefix-caching on/off [C structure]. Accounting must include an explicit interaction/residual term — shares are NOT forced to sum to 100%. +- Metric: E_token under the P3 discipline (C00-TP1, rho=0.60, 240s clean windows). New GPU cap: +6 H20-hours for P5 (P3 cap closed at 14.03/16). +- Gates: protocol-first (orchestrator review before any GPU run); P4 acceptance re-scoped — speculative recovery claims in P4 must defer to P5 measured shares. +- P5 protocol (task-mrhq3ud3, 13m49s): ACCEPTED. Data-grounded specifics verified by worker from P3 raw Layer-1: P10 decode-B support is {1..7} (17,941 pure-decode steps) → A2 capture sizes {3,5,6,7}; A1 = 32-request reorder blocks (R16 0.6417→0.4734, −26.2% rel), 142-request slice @0.4725 req/s, 64s fairness cap. Orchestrator dispositions on 5 open decisions: (1) BLOCKING → approve recorded-arrival BRIDGE ledger (production-faithful estimand; explicit not-literal-P3-decomposition limitation); (2) dual P03/P04 control ledgers, dominant must hold under both — approved; (3) A1 parameters — approved; (4) 3 replicates/arm, P3-control reuse behind 3% bridge gate, optional tier within 6.0 H20-h — approved; (5) Layer-1-only primary, routed-expert telemetry analysis-only no causal share — approved. Execution authorized. +- P5 wave 1 (detached exec, 30min, 0.65 H20-h): HARD STOP correct — all 4 attempted arms failed frozen warm-up stabilization (drift 190.6-216.5% recorded arms, 13.2% uniform arm, gate 10%; completions 25-28/32). ROOT CAUSE: semantic mismatch — throughput-drift gate designed for saturation/steady loads is wrong for rate-following arms whose throughput follows a non-stationary recorded arrival BY DESIGN. Purpose of warm-up = no cold-start contamination, not stationarity. Bonus signal: 190% vs 13% drift gap is itself direct arrival-mechanism evidence. AMENDMENT A-P5-1: rate-following arms use cold-start-artifact gates (all compile/capture events before clean window per Layer-1 graph-mode series + >=16 warm-up completions incl >=1 long-context + zero first-occurrence capture events inside clean window); drift gate retained for saturation arms only. Budget 5.35 remaining. +- P5 round 2 (detached, ~70min, 2.44/6.0 H20-h): 15/15 primary valid under A-P5-1, bridge PASS (0.269%). OFFICIAL ledger INCONCLUSIVE per frozen rules (only A1-under-P03 evaluable: share 0.038, CI [-0.20,0.23]; A2/A3/A4 manipulation-failed; P04 denominator unstable 45% nonpositive draws). REAL FINDINGS: (1) P3 P10-vs-P04 gap was largely a MATERIALIZATION ARTIFACT — P5 recorded-arrival base E=3.014 ≈ P04 control 3.055 (gap 0.041 CI spans 0), while uniformizing arrival (as P3 did) drops E to 2.626 (−12.9%, raw p=0.014, Holm 0.055): burstiness HELPS at low rate via batch formation (decode-B CV 0.87→0.66 under uniform but waiting CV rises); P5 A3 reproduces P3 base within 0.27% — bridge design caught our own artifact. (2) All four intuitive mechanisms ≈0 at rho=0.60: raggedness 3.8% n.s., capture-fix +1.6% n.s. (P4 independent validation +0.18%, padding bound matched within 0.002pp), prefix ~0. Remaining P10-vs-P03 gap (~36%) explained by NONE — workload physics, not recoverable waste at this regime. (3) P4 ranked list: #1 prefix affinity +82.14% ceiling (P08 vs P07), #4 pattern-specific MNS64 has a −24.27% P01 counterexample (sign flips by pattern). OpProf campaign phases P0-P5 all complete; ~16.5 H20-h total. + +## Phase 6 — cross-version churn quantification (USER-APPROVED 2026-07-13 "请你测试一下") +- Question: does the vLLM 0.20.0 C1 12-cell TP×MNS optimum/ranking survive on 0.24.0? Paired-surface churn evidence for the paper motivation (matrix × real-eval cost × churn). +- Ground truth: C1 recovered stores (verified): TP1 [2.10,2.35,2.283,2.283], TP2 [2.275,2.275,3.283,3.258], TP4 [1.283,2.442,2.442,2.442] req/s/GPU; best TP2/MNS32=3.2833; trap TP4/MNS16; per-cell recorded anchors + request counts in replayserve docs/assets/simfid_s2r/ground_truth.json; workload = materialized chat_w20260311_1000 (local + dash0 CPFS), output override 128, scale 0.1, pass>=0.95, stepped TTFT 2/4/6s, TPOT 50ms; caps 512 TP1/TP2, uncapped TP4. +- Budget: 3.0 H20-hours hard cap → anchor-subset design required (TP4 cells cost 4 GPU-h per wall-hour). Covariate to note: C1 ran on dash1, P6 on dash0 (same H20 class). +- Gate: protocol-first, orchestrator review before GPU. +- P6 protocol ACCEPTED: 92/92 historical anchor counts reproduced locally pre-GPU (comparability verified); adaptive 25-anchor design (12 peak + 11 adjacent + 2 trap-bracket), 2.65 primary + 0.35 confirmation reserve = 3.0 cap exactly. Decision rules: feasibility flip / >5% drift / floor-bucket argmax / tau-b>=0.8 ranking survival / trap escape rule / 2-of-3 confirmation. All 5 open decisions APPROVED incl. upgrade-path estimand (resolved defaults = churn; dash1→dash0 + colocation = stated limitations) and warmup long-tier adaptation raw>4096. Execution dispatched. +- P6 W1 hard stop (0.43 H20-h spent): 8 replays REJECTED for missing in-stream footers (wrong shutdown path) + redo projection 3.03 > 3.0 cap. Orchestrator diagnosis: P6 validator failed to apply the P5-ACCEPTED sidecar-footer accounting rule (records == sidecar counters within one flush interval). AMENDMENT A-P6-1: (a) sidecar footers are valid accounting per P5 rule — re-adjudicate W1 artifacts WITHOUT rerun if sidecars balance; (b) controller must use the graceful path (vllm serve --shutdown-timeout, found in P5) for subsequent waves; (c) IF W1 salvage fails, cap raised 3.0→3.5 as fallback (worker recommendation, within user posture). +- P6 r2 partial (2.29/3.0 H20-h, hard stop before W4): 21/25 anchors, 10/12 cells. Signals: TP2/MNS64 >=29.4% DOWN (left-censored), TP1 cells right-censored UP, TP2 family down — surface shape moved opposite directions. CONFOUND (worker confirmation runs): co-location flips SLO-frontier pass rates (0.41→1.00, 0.03→0.96 solo; waiting 8.5→0.35) — co-location valid for throughput/op-shares (P3) but NOT for pass-rate cliffs; 0.20 baseline was solo. USER DECISION: cap raised to 6.0; A-P6-2 = frontier anchors SOLO placement; finish W4 + solo re-confirm key cells. +- P6 FINAL (A-P6-2 solo tier complete, 5.64/6.0 H20-h, 66 authoritative trials, 12/12 cells): formal verdicts INCONCLUSIVE per frozen letter (4 cells censored/non-monotonic prevent bounded 12-cell surface; tau_b=null, never manufactured). SUBSTANTIVE PAIRED SURFACE (all solo-authoritative): TP2/MNS64 −29.41% CONFIRMED (3.2583→2.3000 bounded; mechanism: decode-B 11.5-18 + 4.6-9.5% NONE-graph at old anchor); TP1/MNS8 +13.49% and TP1 plateau converges to 2.3833; old best TP2/MNS32 −0.76% (held, highest bounded); TP4 family right-censored UP >=2.50; TP4/MNS16 frontier became NON-MONOTONIC in load. CO-LOCATION DELTA TABLE: up to +92.86pp pass-rate flips (21 exact-request anchor pairs) — co-location validity is metric-dependent (standalone methodological finding). Orchestrator verified −0.2941 drift in metrics.json. Campaign total incl. P6: ~22.2 H20-h. diff --git a/patches/vllm-0.24.0-opprof/0001-Add-lightweight-per-step-OpProf-telemetry.patch b/patches/vllm-0.24.0-opprof/0001-Add-lightweight-per-step-OpProf-telemetry.patch new file mode 100644 index 0000000..ee6f2f8 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0001-Add-lightweight-per-step-OpProf-telemetry.patch @@ -0,0 +1,487 @@ +From f6f1cacbce0e39992d04843f652c1adda373ae43 Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sat, 11 Jul 2026 17:29:02 +0800 +Subject: [PATCH 1/5] Add lightweight per-step OpProf telemetry + +Assisted-by: OpenAI Codex +--- + vllm/envs.py | 4 + + vllm/v1/core/sched/scheduler.py | 28 +++ + vllm/v1/opprof.py | 337 +++++++++++++++++++++++++++++ + vllm/v1/worker/gpu_model_runner.py | 6 +- + 4 files changed, 374 insertions(+), 1 deletion(-) + create mode 100644 vllm/v1/opprof.py + +diff --git a/vllm/envs.py b/vllm/envs.py +index 27a85bb..b3093e9 100755 +--- a/vllm/envs.py ++++ b/vllm/envs.py +@@ -45,6 +45,7 @@ if TYPE_CHECKING: + VLLM_LOGGING_COLOR: str = "auto" + NO_COLOR: bool = False + VLLM_LOG_STATS_INTERVAL: float = 10.0 ++ VLLM_OPPROF_DIR: str = "" + VLLM_TRACE_FUNCTION: int = 0 + VLLM_USE_FLASHINFER_SAMPLER: bool = True + VLLM_PP_LAYER_PARTITION: str | None = None +@@ -786,6 +787,9 @@ environment_variables: dict[str, Callable[[], Any]] = { + if (val := float(os.getenv("VLLM_LOG_STATS_INTERVAL", "10."))) > 0.0 + else 10.0 + ), ++ # Directory for per-step OpProf JSONL telemetry. ++ # Empty disables OpProf. ++ "VLLM_OPPROF_DIR": lambda: os.getenv("VLLM_OPPROF_DIR", ""), + # Trace function calls + # If set to 1, vllm will trace function calls + # Useful for debugging +diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py +index 90d93a1..303c562 100644 +--- a/vllm/v1/core/sched/scheduler.py ++++ b/vllm/v1/core/sched/scheduler.py +@@ -7,6 +7,7 @@ from collections.abc import Iterable + from dataclasses import replace + from typing import Any + ++import vllm.envs as envs + from vllm.compilation.cuda_graph import CUDAGraphStat + from vllm.config import VllmConfig + from vllm.distributed.ec_transfer.ec_connector.base import ( +@@ -55,6 +56,7 @@ from vllm.v1.engine import EngineCoreEventType, EngineCoreOutput, EngineCoreOutp + from vllm.v1.kv_cache_interface import KVCacheConfig + from vllm.v1.metrics.perf import ModelMetrics, PerfStats + from vllm.v1.metrics.stats import PrefixCacheStats, SchedulerStats ++from vllm.v1.opprof import OpProfRecorder + from vllm.v1.outputs import DraftTokenIds, KVConnectorOutput, ModelRunnerOutput + from vllm.v1.request import Request, RequestStatus, StreamingUpdate + from vllm.v1.spec_decode.dynamic.utils import build_dynamic_sd_schedule_lookup +@@ -271,6 +273,12 @@ class Scheduler(SchedulerInterface): + if self.connector is not None: + self.connector.bind_gpu_block_pool(self.kv_cache_manager.block_pool) + ++ self.opprof = OpProfRecorder.create( ++ output_dir=envs.VLLM_OPPROF_DIR, ++ dp_rank=self.parallel_config.data_parallel_index, ++ log_stats=self.log_stats, ++ ) ++ + self.use_pp = self.parallel_config.pipeline_parallel_size > 1 + self.use_v2_model_runner = vllm_config.use_v2_model_runner + # Scheduler iteration counter. Drives the V2+PP+async decode-throttle +@@ -386,6 +394,9 @@ class Scheduler(SchedulerInterface): + return num_new_tokens + + def schedule(self, throttle_prefills: bool = False) -> SchedulerOutput: ++ opprof_start = ( ++ self.opprof.capture_start(self) if self.opprof is not None else None ++ ) + self.current_step += 1 + # NOTE(woosuk) on the scheduling algorithm: + # There's no "decoding phase" nor "prefill phase" in the scheduler. +@@ -1090,6 +1101,14 @@ class Scheduler(SchedulerInterface): + ) + scheduler_output.ec_connector_metadata = ec_meta + ++ if self.opprof is not None: ++ assert opprof_start is not None ++ self.opprof.begin( ++ scheduler=self, ++ output=scheduler_output, ++ start=opprof_start, ++ ) ++ + # Advance the fence only for non-empty steps (those that actually + # write KV and have their output processed later in update_from_output). + if self.defer_block_free and total_num_scheduled_tokens > 0: +@@ -1800,6 +1819,12 @@ class Scheduler(SchedulerInterface): + engine_core_outputs[0] = eco = EngineCoreOutputs() + eco.scheduler_stats = stats + ++ if self.opprof is not None: ++ self.opprof.finalize( ++ output=scheduler_output, ++ cudagraph_stat=cudagraph_stats, ++ ) ++ + return engine_core_outputs + + @staticmethod +@@ -2292,6 +2317,9 @@ class Scheduler(SchedulerInterface): + if self.ec_connector is not None: + self.ec_connector.shutdown() + ++ if self.opprof is not None: ++ self.opprof.close() ++ + logger.debug_once("[shutdown] Scheduler: complete") + + ######################################################################## +diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py +new file mode 100644 +index 0000000..f0330d0 +--- /dev/null ++++ b/vllm/v1/opprof.py +@@ -0,0 +1,337 @@ ++# SPDX-License-Identifier: Apache-2.0 ++# SPDX-FileCopyrightText: Copyright contributors to the vLLM project ++import atexit ++import logging ++import os ++import queue ++import threading ++import time ++from bisect import bisect_left ++from pathlib import Path ++from typing import Any ++ ++import msgspec ++ ++logger = logging.getLogger(__name__) ++ ++SCHEMA_VERSION = 1 ++CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18)) ++CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12)) ++DEFAULT_QUEUE_CAPACITY = 8192 ++_CLOSE_TIMEOUT_SECONDS = 1.0 ++_STOP = object() ++_PREFIX_FIELDS = ( # noqa: SIM905 ++ "requests queries hits preempted_requests preempted_queries preempted_hits" ++).split() ++ ++ ++ScheduleStart = tuple[int, int, dict[str, dict[str, int] | None]] ++ ++ ++def classify_chunk(was_chunk: bool, end: int, target: int) -> str: ++ assert 0 <= end <= target ++ if was_chunk: ++ return "middle" if end < target else "final" ++ return "first" if end < target else "unsplit" ++ ++ ++def _prefix_values(stats: Any | None) -> dict[str, int] | None: ++ if stats is None: ++ return None ++ return {name: int(getattr(stats, name, 0)) for name in _PREFIX_FIELDS} ++ ++ ++def _prefix_snapshot(scheduler: Any) -> dict[str, dict[str, int] | None]: ++ return { ++ "local": _prefix_values(scheduler.kv_cache_manager.prefix_cache_stats), ++ "external": _prefix_values(scheduler.connector_prefix_cache_stats), ++ } ++ ++ ++def _prefix_delta( ++ before: dict[str, int] | None, after: dict[str, int] | None ++) -> dict[str, int] | None: ++ if before is None and after is None: ++ return None ++ before = before or dict.fromkeys(_PREFIX_FIELDS, 0) ++ after = after or dict.fromkeys(_PREFIX_FIELDS, 0) ++ delta = {name: after[name] - before[name] for name in _PREFIX_FIELDS} ++ assert all(value >= 0 for value in delta.values()) ++ return delta ++ ++ ++class JSONLWriter: ++ def __init__( ++ self, ++ path: Path, ++ capacity: int = DEFAULT_QUEUE_CAPACITY, ++ start: bool = True, ++ ) -> None: ++ self._queue: queue.Queue[Any] = queue.Queue(capacity) ++ self._encoder = msgspec.json.Encoder() ++ self._file = path.open("xb", buffering=1 << 20) ++ self._thread = threading.Thread(target=self._run, daemon=True) ++ self._failure_lock = threading.Lock() ++ self._started = self._closed = False ++ self.failed = False ++ self.failure: Exception | None = None ++ self.encoded_records = self.written_records = 0 ++ self.dropped_records = self._unreported_drops = 0 ++ if start: ++ self.start() ++ ++ def start(self) -> None: ++ if not self._started: ++ self._started = True ++ self._thread.start() ++ ++ def _record_failure(self, error: Exception) -> None: ++ with self._failure_lock: ++ if self.failed: ++ return ++ self.failed = True ++ self.failure = error ++ logger.error("OpProf writer failed: %s", error) ++ ++ def _writer_unavailable(self) -> bool: ++ if self.failed: ++ return True ++ if self._started and not self._thread.is_alive(): ++ self._record_failure(RuntimeError("writer thread stopped unexpectedly")) ++ return True ++ return False ++ ++ def _drop(self, pending: int) -> bool: ++ self.dropped_records += 1 ++ self._unreported_drops = pending + 1 ++ return False ++ ++ def submit(self, record: dict[str, Any]) -> bool: ++ if self._closed: ++ raise RuntimeError("OpProf writer is closed") ++ pending = self._unreported_drops ++ record["dropped_records_before"] = pending ++ if self._writer_unavailable(): ++ return self._drop(pending) ++ payload = self._encoder.encode(record) + b"\n" ++ self.encoded_records += 1 ++ if self._writer_unavailable(): ++ return self._drop(pending) ++ try: ++ self._queue.put_nowait(payload) ++ except queue.Full: ++ return self._drop(pending) ++ self._unreported_drops = 0 ++ return True ++ ++ def _run(self) -> None: ++ buffered = 0 ++ last_flush = time.monotonic() ++ try: ++ while True: ++ try: ++ item = self._queue.get(timeout=1.0) ++ except queue.Empty: ++ self._file.flush() ++ buffered = 0 ++ last_flush = time.monotonic() ++ continue ++ try: ++ if item is _STOP: ++ break ++ self._file.write(item) ++ buffered += len(item) ++ self.written_records += 1 ++ now = time.monotonic() ++ if buffered >= 1 << 20 or now - last_flush >= 1.0: ++ self._file.flush() ++ buffered = 0 ++ last_flush = now ++ finally: ++ self._queue.task_done() ++ except Exception as error: ++ self._record_failure(error) ++ finally: ++ footer = dict( ++ schema=SCHEMA_VERSION, ++ record_type="footer", ++ encoded_records=self.encoded_records, ++ written_records=self.written_records, ++ dropped_records=self.dropped_records, ++ ) ++ try: ++ self._file.write(self._encoder.encode(footer) + b"\n") ++ self._file.flush() ++ except Exception as error: ++ self._record_failure(error) ++ finally: ++ try: ++ self._file.close() ++ except Exception as error: ++ self._record_failure(error) ++ ++ def close(self) -> None: ++ if self._closed: ++ return ++ self._closed = True ++ self.start() ++ if not self._thread.is_alive(): ++ return ++ try: ++ self._queue.put(_STOP, timeout=_CLOSE_TIMEOUT_SECONDS) ++ except queue.Full: ++ if not self._thread.is_alive(): ++ return ++ try: ++ self._queue.put_nowait(_STOP) ++ except queue.Full: ++ self._record_failure( ++ TimeoutError("timed out enqueueing writer stop sentinel") ++ ) ++ return ++ self._thread.join(timeout=_CLOSE_TIMEOUT_SECONDS) ++ if self._thread.is_alive(): ++ self._record_failure(TimeoutError("timed out joining writer thread")) ++ ++ ++class OpProfRecorder: ++ def __init__(self, engine_id: str, writer: JSONLWriter) -> None: ++ self.engine_id = engine_id ++ self.writer = writer ++ self._next_step = 0 ++ self._pending: dict[int, dict[str, Any]] = {} ++ atexit.register(self.close) ++ ++ @classmethod ++ def create( ++ cls, output_dir: str, dp_rank: int, log_stats: bool ++ ) -> "OpProfRecorder | None": ++ if not output_dir: ++ return None ++ if not log_stats: ++ raise ValueError("VLLM_OPPROF_DIR requires log stats to be enabled") ++ directory = Path(output_dir).expanduser() ++ if not directory.is_absolute(): ++ raise ValueError("VLLM_OPPROF_DIR must be an absolute path") ++ directory.mkdir(parents=True, exist_ok=True) ++ engine_id = f"dp{dp_rank}-pid{os.getpid()}" ++ name = f"opprof-v{SCHEMA_VERSION}-{engine_id}-{time.time_ns()}.jsonl" ++ return cls(engine_id, JSONLWriter(directory / name)) ++ ++ @staticmethod ++ def capture_start(scheduler: Any) -> ScheduleStart: ++ return time.time_ns(), time.monotonic_ns(), _prefix_snapshot(scheduler) ++ ++ def begin(self, scheduler: Any, output: Any, start: ScheduleStart) -> None: ++ key = id(output) ++ assert key not in self._pending, "duplicate OpProf begin" ++ new_ids = {request.req_id for request in output.scheduled_new_reqs} ++ prefill_requests = prefill_tokens = decode_requests = decode_tokens = 0 ++ context_length_hist = [0] * (len(CONTEXT_LENGTH_EDGES) + 1) ++ chunk_size_hist = [0] * (len(CHUNK_SIZE_EDGES) + 1) ++ chunks: dict[str, Any] = dict.fromkeys( ++ ("first", "middle", "final", "unsplit"), 0 ++ ) ++ for req_id, num_tokens in output.num_scheduled_tokens.items(): ++ request = scheduler.requests[req_id] ++ end = request.num_computed_tokens + num_tokens ++ assert end >= 0 ++ context_length_hist[bisect_left(CONTEXT_LENGTH_EDGES, end)] += 1 ++ is_prefill = ( ++ req_id in new_ids ++ or output.scheduled_cached_reqs.is_context_phase(req_id) ++ ) ++ if is_prefill: ++ prefill_requests += 1 ++ prefill_tokens += num_tokens ++ assert num_tokens >= 0 ++ chunk_size_hist[bisect_left(CHUNK_SIZE_EDGES, num_tokens)] += 1 ++ target = request.num_tokens + request.num_output_placeholders ++ chunks[classify_chunk(request.is_prefill_chunk, end, target)] += 1 ++ else: ++ decode_requests += 1 ++ decode_tokens += num_tokens ++ prefix_after = _prefix_snapshot(scheduler) ++ block_pool = scheduler.kv_cache_manager.block_pool ++ total_blocks = block_pool.num_gpu_blocks - 1 ++ free_blocks = block_pool.get_num_free_blocks() ++ assert 0 <= free_blocks <= total_blocks ++ chunks["tokens"] = prefill_tokens ++ chunks["chunk_size_hist"] = chunk_size_hist ++ values = dict( ++ schema=SCHEMA_VERSION, ++ engine_id=self.engine_id, ++ step_index=self._next_step, ++ submit_wall_ns=start[0], ++ submit_mono_ns=start[1], ++ model_executed=output.total_num_scheduled_tokens > 0, ++ scheduled_requests=len(output.num_scheduled_tokens), ++ decode_batch_size=decode_requests, ++ prefill_requests=prefill_requests, ++ prefill_tokens=prefill_tokens, ++ decode_tokens=decode_tokens, ++ chunked_prefill=chunks, ++ context_length_hist=context_length_hist, ++ preemptions=len(output.preempted_req_ids or ()), ++ queues=dict( ++ running=len(scheduler.running), ++ waiting=len(scheduler.waiting), ++ deferred=len(scheduler.skipped_waiting), ++ ), ++ kv=dict( ++ total_blocks=total_blocks, ++ free_blocks=free_blocks, ++ used_blocks=total_blocks - free_blocks, ++ usage=scheduler.kv_cache_manager.usage, ++ ), ++ prefix=dict( ++ local=_prefix_delta(start[2]["local"], prefix_after["local"]), ++ external=_prefix_delta(start[2]["external"], prefix_after["external"]), ++ ), ++ ) ++ assert prefill_tokens + decode_tokens == output.total_num_scheduled_tokens ++ self._pending[key] = values ++ self._next_step += 1 ++ ++ def finalize(self, output: Any, cudagraph_stat: Any | None) -> bool: ++ try: ++ values = self._pending.pop(id(output)) ++ except KeyError: ++ raise AssertionError("missing or already finalized OpProf step") from None ++ if cudagraph_stat is None: ++ assert not values["model_executed"] ++ cudagraph = dict( ++ hit=False, ++ runtime_mode="NONE", ++ unpadded_tokens=0, ++ bucket_tokens=0, ++ padding_tokens=0, ++ ) ++ else: ++ mode = str(cudagraph_stat.runtime_mode).rsplit(".", 1)[-1] ++ cudagraph = dict( ++ hit=mode != "NONE", ++ runtime_mode=mode, ++ unpadded_tokens=cudagraph_stat.num_unpadded_tokens, ++ bucket_tokens=cudagraph_stat.num_padded_tokens, ++ padding_tokens=cudagraph_stat.num_paddings, ++ ) ++ record = dict( ++ values, ++ complete_mono_ns=time.monotonic_ns(), ++ cudagraph=cudagraph, ++ moe_expert_load=None, ++ dropped_records_before=0, ++ ) ++ return self.writer.submit(record) ++ ++ def close(self) -> None: ++ self.writer.close() ++ ++ @property ++ def failed(self) -> bool: ++ return self.writer.failed ++ ++ @property ++ def failure(self) -> Exception | None: ++ return self.writer.failure +diff --git a/vllm/v1/worker/gpu_model_runner.py b/vllm/v1/worker/gpu_model_runner.py +index 74938a8..c11d773 100644 +--- a/vllm/v1/worker/gpu_model_runner.py ++++ b/vllm/v1/worker/gpu_model_runner.py +@@ -437,6 +437,7 @@ class GPUModelRunner( + self.scheduler_config = vllm_config.scheduler_config + self.speculative_config = vllm_config.speculative_config + self.observability_config = vllm_config.observability_config ++ self.opprof_enabled = bool(envs.VLLM_OPPROF_DIR) + + model_config = self.model_config + cache_config = self.cache_config +@@ -3917,7 +3918,10 @@ class GPUModelRunner( + assert batch_descriptor.num_tokens == num_tokens_padded + + cudagraph_stats = None +- if self.vllm_config.observability_config.cudagraph_metrics: ++ if ( ++ self.vllm_config.observability_config.cudagraph_metrics ++ or self.opprof_enabled ++ ): + cudagraph_stats = CUDAGraphStat( + num_unpadded_tokens=num_tokens, + num_padded_tokens=batch_descriptor.num_tokens, +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/0002-Add-standalone-OpProf-telemetry-tests.patch b/patches/vllm-0.24.0-opprof/0002-Add-standalone-OpProf-telemetry-tests.patch new file mode 100644 index 0000000..086d266 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0002-Add-standalone-OpProf-telemetry-tests.patch @@ -0,0 +1,417 @@ +From 4f4ee674f217698436b00c3ab6357f59a792477a Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sat, 11 Jul 2026 17:29:02 +0800 +Subject: [PATCH 2/5] Add standalone OpProf telemetry tests + +Assisted-by: OpenAI Codex +--- + tests/v1/core/test_opprof.py | 397 +++++++++++++++++++++++++++++++++++ + 1 file changed, 397 insertions(+) + create mode 100644 tests/v1/core/test_opprof.py + +diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py +new file mode 100644 +index 0000000..9bfbfcc +--- /dev/null ++++ b/tests/v1/core/test_opprof.py +@@ -0,0 +1,397 @@ ++# SPDX-License-Identifier: Apache-2.0 ++# SPDX-FileCopyrightText: Copyright contributors to the vLLM project ++"""Standalone tests: this file intentionally does not import the vllm package.""" ++ ++import errno ++import importlib.util ++import logging ++import sys ++import threading ++from pathlib import Path ++from types import SimpleNamespace ++ ++import msgspec ++import pytest ++ ++_ROOT = Path(__file__).parents[3] ++_SPEC = importlib.util.spec_from_file_location( ++ "opprof_standalone", _ROOT / "vllm" / "v1" / "opprof.py" ++) ++assert _SPEC is not None and _SPEC.loader is not None ++opprof = importlib.util.module_from_spec(_SPEC) ++sys.modules[_SPEC.name] = opprof ++_SPEC.loader.exec_module(opprof) ++ ++ ++class CachedRequests: ++ def __init__(self, context_ids=()): ++ self.context_ids = set(context_ids) ++ ++ def is_context_phase(self, req_id): ++ return req_id in self.context_ids ++ ++ ++def prefix_stats(**overrides): ++ values = dict.fromkeys(opprof._PREFIX_FIELDS, 0) ++ values.update(overrides) ++ return SimpleNamespace(**values) ++ ++ ++def request(computed, total, was_chunk=False, placeholders=0): ++ return SimpleNamespace( ++ num_computed_tokens=computed, ++ num_tokens=total, ++ num_output_placeholders=placeholders, ++ is_prefill_chunk=was_chunk, ++ ) ++ ++ ++def scheduler(requests, local=None, external=None): ++ block_pool = SimpleNamespace( ++ num_gpu_blocks=101, ++ get_num_free_blocks=lambda: 40, ++ ) ++ kv_manager = SimpleNamespace( ++ prefix_cache_stats=local or prefix_stats(), ++ block_pool=block_pool, ++ usage=0.6, ++ ) ++ return SimpleNamespace( ++ requests=requests, ++ kv_cache_manager=kv_manager, ++ connector_prefix_cache_stats=external, ++ running=list(range(len(requests))), ++ waiting=[0, 1], ++ skipped_waiting=[0], ++ ) ++ ++ ++def schedule_output(tokens, context_ids=(), new_ids=(), preempted=()): ++ return SimpleNamespace( ++ scheduled_new_reqs=[SimpleNamespace(req_id=req_id) for req_id in new_ids], ++ scheduled_cached_reqs=CachedRequests(context_ids), ++ num_scheduled_tokens=tokens, ++ total_num_scheduled_tokens=sum(tokens.values()), ++ preempted_req_ids=set(preempted), ++ ) ++ ++ ++def graph(mode="FULL", unpadded=1, padded=1): ++ return SimpleNamespace( ++ runtime_mode=mode, ++ num_unpadded_tokens=unpadded, ++ num_padded_tokens=padded, ++ num_paddings=padded - unpadded, ++ ) ++ ++ ++def recorder(tmp_path, *, capacity=8192, start=True): ++ path = tmp_path / "opprof.jsonl" ++ writer = opprof.JSONLWriter(path, capacity=capacity, start=start) ++ return opprof.OpProfRecorder("dp0-pid1", writer), path ++ ++ ++def emit(rec, sched, output, cg=None): ++ start = rec.capture_start(sched) ++ rec.begin(sched, output, start) ++ return rec.finalize(output, cg or graph()) ++ ++ ++def read_jsonl(path): ++ return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()] ++ ++ ++def test_import_light_and_approved_constants(): ++ assert "torch" not in sys.modules ++ assert "vllm" not in sys.modules ++ assert opprof.DEFAULT_QUEUE_CAPACITY == 8192 ++ assert tuple(1 << i for i in range(7, 18)) == opprof.CONTEXT_LENGTH_EDGES ++ assert tuple(1 << i for i in range(4, 12)) == opprof.CHUNK_SIZE_EDGES ++ ++ ++def test_schema_and_invariants(tmp_path): ++ sched = scheduler( ++ { ++ "first": request(0, 100), ++ "final": request(64, 100, was_chunk=True), ++ "decode": request(1024, 1025), ++ } ++ ) ++ output = schedule_output( ++ {"first": 64, "final": 36, "decode": 1}, ++ context_ids={"final"}, ++ new_ids={"first"}, ++ preempted={"old"}, ++ ) ++ rec, path = recorder(tmp_path) ++ start = rec.capture_start(sched) ++ sched.kv_cache_manager.prefix_cache_stats = prefix_stats( ++ requests=1, queries=100, hits=64 ++ ) ++ rec.begin(sched, output, start) ++ assert rec.finalize(output, graph("FULL", 101, 128)) ++ rec.close() ++ ++ record, footer = read_jsonl(path) ++ assert record["schema"] == 1 ++ assert record["scheduled_requests"] == 3 ++ assert record["prefill_requests"] == 2 ++ assert record["decode_batch_size"] == 1 ++ assert record["prefill_tokens"] + record["decode_tokens"] == 101 ++ assert sum(record["context_length_hist"]) == 3 ++ assert len(record["context_length_hist"]) == 12 ++ assert sum(record["chunked_prefill"]["chunk_size_hist"]) == 2 ++ assert len(record["chunked_prefill"]["chunk_size_hist"]) == 9 ++ assert record["chunked_prefill"]["first"] == 1 ++ assert record["chunked_prefill"]["final"] == 1 ++ assert record["preemptions"] == 1 ++ assert record["kv"] == { ++ "total_blocks": 100, ++ "free_blocks": 40, ++ "used_blocks": 60, ++ "usage": 0.6, ++ } ++ assert record["prefix"]["local"]["hits"] == 64 ++ assert record["moe_expert_load"] is None ++ assert record["complete_mono_ns"] >= record["submit_mono_ns"] ++ assert footer["record_type"] == "footer" ++ assert footer["written_records"] == 1 ++ ++ ++def test_capture_record_matches_pre_refactor_golden(tmp_path, monkeypatch): ++ sched = scheduler( ++ { ++ "edge": request(0, 128), ++ "after": request(65, 129, was_chunk=True), ++ "decode": request(256, 257), ++ } ++ ) ++ output = schedule_output( ++ {"edge": 128, "after": 64, "decode": 1}, ++ context_ids={"after"}, ++ new_ids={"edge"}, ++ preempted={"old"}, ++ ) ++ rec, path = recorder(tmp_path) ++ zero_prefix = dict.fromkeys(opprof._PREFIX_FIELDS, 0) ++ start = (100, 200, {"local": zero_prefix, "external": None}) ++ monkeypatch.setattr(opprof.time, "monotonic_ns", lambda: 300) ++ ++ rec.begin(sched, output, start) ++ assert rec.finalize(output, graph("FULL", 193, 256)) ++ rec.close() ++ ++ record = read_jsonl(path)[0] ++ assert record == { ++ "schema": 1, ++ "engine_id": "dp0-pid1", ++ "step_index": 0, ++ "submit_wall_ns": 100, ++ "submit_mono_ns": 200, ++ "model_executed": True, ++ "scheduled_requests": 3, ++ "decode_batch_size": 1, ++ "prefill_requests": 2, ++ "prefill_tokens": 192, ++ "decode_tokens": 1, ++ "chunked_prefill": { ++ "first": 0, ++ "middle": 0, ++ "final": 1, ++ "unsplit": 1, ++ "tokens": 192, ++ "chunk_size_hist": [0, 0, 1, 1, 0, 0, 0, 0, 0], ++ }, ++ "context_length_hist": [1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0], ++ "preemptions": 1, ++ "queues": {"running": 3, "waiting": 2, "deferred": 1}, ++ "kv": { ++ "total_blocks": 100, ++ "free_blocks": 40, ++ "used_blocks": 60, ++ "usage": 0.6, ++ }, ++ "prefix": {"local": zero_prefix, "external": None}, ++ "complete_mono_ns": 300, ++ "cudagraph": { ++ "hit": True, ++ "runtime_mode": "FULL", ++ "unpadded_tokens": 193, ++ "bucket_tokens": 256, ++ "padding_tokens": 63, ++ }, ++ "moe_expert_load": None, ++ "dropped_records_before": 0, ++ } ++ ++ ++@pytest.mark.parametrize( ++ ("was_chunk", "end", "target", "expected"), ++ [ ++ (False, 64, 100, "first"), ++ (True, 80, 100, "middle"), ++ (True, 100, 100, "final"), ++ (False, 100, 100, "unsplit"), ++ ], ++) ++def test_chunk_classification(was_chunk, end, target, expected): ++ assert opprof.classify_chunk(was_chunk, end, target) == expected ++ ++ ++def test_async_pairing_out_of_order_and_double_finalize(tmp_path): ++ sched = scheduler({"a": request(10, 11), "b": request(200, 201)}) ++ first = schedule_output({"a": 1}) ++ second = schedule_output({"b": 1}) ++ rec, path = recorder(tmp_path) ++ rec.begin(sched, first, rec.capture_start(sched)) ++ rec.begin(sched, second, rec.capture_start(sched)) ++ assert rec.finalize(second, graph()) ++ assert rec.finalize(first, graph("NONE")) ++ with pytest.raises(AssertionError, match="already finalized"): ++ rec.finalize(second, graph()) ++ assert not rec._pending ++ rec.close() ++ records = read_jsonl(path)[:-1] ++ assert [record["step_index"] for record in records] == [1, 0] ++ assert records[0]["context_length_hist"][1] == 1 ++ assert records[1]["context_length_hist"][0] == 1 ++ ++ ++def test_disabled_noop_and_log_stats_fail_fast(tmp_path): ++ assert opprof.OpProfRecorder.create("", dp_rank=0, log_stats=False) is None ++ with pytest.raises(ValueError, match="requires log stats"): ++ opprof.OpProfRecorder.create(str(tmp_path), dp_rank=0, log_stats=False) ++ assert not list(tmp_path.iterdir()) ++ ++ ++def test_bounded_queue_drop_accounting(tmp_path): ++ sched = scheduler({str(i): request(i, i + 1) for i in range(3)}) ++ rec, path = recorder(tmp_path, capacity=1, start=False) ++ assert emit(rec, sched, schedule_output({"0": 1})) ++ assert not emit(rec, sched, schedule_output({"1": 1})) ++ rec.writer.start() ++ rec.writer._queue.join() ++ assert emit(rec, sched, schedule_output({"2": 1})) ++ rec.close() ++ ++ first, after_drop, footer = read_jsonl(path) ++ assert first["step_index"] == 0 ++ assert after_drop["step_index"] == 2 ++ assert after_drop["dropped_records_before"] == 1 ++ assert footer["encoded_records"] == 3 ++ assert footer["written_records"] == 2 ++ assert footer["dropped_records"] == 1 ++ ++ ++def test_writer_enospc_is_exposed_and_shutdown_is_bounded( ++ tmp_path, monkeypatch, caplog ++): ++ sched = scheduler( ++ { ++ "first": request(0, 1), ++ "after_failure": request(1, 2), ++ } ++ ) ++ rec, _ = recorder(tmp_path, capacity=1, start=False) ++ assert emit(rec, sched, schedule_output({"first": 1})) ++ ++ real_file = rec.writer._file ++ ++ def fail_enospc(*_args, **_kwargs): ++ raise OSError(errno.ENOSPC, "No space left on device") ++ ++ failing_file = SimpleNamespace( ++ write=fail_enospc, ++ flush=fail_enospc, ++ close=real_file.close, ++ ) ++ monkeypatch.setattr(rec.writer, "_file", failing_file) ++ caplog.set_level(logging.ERROR, logger=opprof.__name__) ++ ++ rec.writer.start() ++ rec.writer._thread.join(timeout=1.0) ++ assert not rec.writer._thread.is_alive() ++ ++ producer_result = emit( ++ rec, sched, schedule_output({"after_failure": 1}) ++ ) ++ ++ closer = threading.Thread(target=rec.close, daemon=True) ++ closer.start() ++ closer.join(timeout=1.0) ++ assert not closer.is_alive(), "OpProf close blocked after writer failure" ++ assert not producer_result ++ assert rec.writer.dropped_records == 1 ++ assert rec.failed ++ assert isinstance(rec.failure, OSError) ++ assert rec.failure.errno == errno.ENOSPC ++ errors = [ ++ record ++ for record in caplog.records ++ if "OpProf writer failed" in record.getMessage() ++ ] ++ assert len(errors) == 1 ++ ++ ++def test_shutdown_flush_is_idempotent(tmp_path): ++ sched = scheduler({"decode": request(8, 9)}) ++ rec, path = recorder(tmp_path) ++ assert emit(rec, sched, schedule_output({"decode": 1})) ++ rec.close() ++ rec.close() ++ record, footer = read_jsonl(path) ++ assert record["step_index"] == 0 ++ assert footer["written_records"] == 1 ++ assert path.stat().st_size > 0 ++ ++ ++def test_piecewise_cudagraph_record_preserved(tmp_path): ++ sched = scheduler({"decode": request(4096, 4097)}) ++ output = schedule_output({"decode": 1}) ++ rec, path = recorder(tmp_path) ++ assert emit(rec, sched, output, graph("PIECEWISE", 513, 520)) ++ rec.close() ++ record = read_jsonl(path)[0] ++ assert record["cudagraph"] == { ++ "hit": True, ++ "runtime_mode": "PIECEWISE", ++ "unpadded_tokens": 513, ++ "bucket_tokens": 520, ++ "padding_tokens": 7, ++ } ++ ++ ++def test_zero_scheduled_tokens_finalize_without_cudagraph(tmp_path): ++ sched = scheduler({}) ++ output = schedule_output({}) ++ rec, path = recorder(tmp_path) ++ rec.begin(sched, output, rec.capture_start(sched)) ++ assert rec._pending ++ ++ assert rec.finalize(output, cudagraph_stat=None) ++ assert not rec._pending ++ rec.close() ++ ++ record = read_jsonl(path)[0] ++ assert record["model_executed"] is False ++ assert record["scheduled_requests"] == 0 ++ assert record["prefill_requests"] == 0 ++ assert record["prefill_tokens"] == 0 ++ assert record["decode_batch_size"] == 0 ++ assert record["decode_tokens"] == 0 ++ assert record["context_length_hist"] == [0] * 12 ++ assert record["chunked_prefill"] == { ++ "first": 0, ++ "middle": 0, ++ "final": 0, ++ "unsplit": 0, ++ "tokens": 0, ++ "chunk_size_hist": [0] * 9, ++ } ++ assert record["cudagraph"] == { ++ "hit": False, ++ "runtime_mode": "NONE", ++ "unpadded_tokens": 0, ++ "bucket_tokens": 0, ++ "padding_tokens": 0, ++ } +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/0003-Log-the-OpProf-output-path-at-startup.patch b/patches/vllm-0.24.0-opprof/0003-Log-the-OpProf-output-path-at-startup.patch new file mode 100644 index 0000000..5a0e8aa --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0003-Log-the-OpProf-output-path-at-startup.patch @@ -0,0 +1,52 @@ +From 668cfb7e27e488454dbf09a4927b8a60d6d49b40 Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sat, 11 Jul 2026 17:32:27 +0800 +Subject: [PATCH 3/5] Log the OpProf output path at startup + +Assisted-by: OpenAI Codex +--- + tests/v1/core/test_opprof.py | 1 + + vllm/v1/core/sched/scheduler.py | 2 ++ + vllm/v1/opprof.py | 1 + + 3 files changed, 4 insertions(+) + +diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py +index 9bfbfcc..79c1fae 100644 +--- a/tests/v1/core/test_opprof.py ++++ b/tests/v1/core/test_opprof.py +@@ -336,6 +336,7 @@ def test_writer_enospc_is_exposed_and_shutdown_is_bounded( + def test_shutdown_flush_is_idempotent(tmp_path): + sched = scheduler({"decode": request(8, 9)}) + rec, path = recorder(tmp_path) ++ assert rec.writer.path == path + assert emit(rec, sched, schedule_output({"decode": 1})) + rec.close() + rec.close() +diff --git a/vllm/v1/core/sched/scheduler.py b/vllm/v1/core/sched/scheduler.py +index 303c562..769a02a 100644 +--- a/vllm/v1/core/sched/scheduler.py ++++ b/vllm/v1/core/sched/scheduler.py +@@ -278,6 +278,8 @@ class Scheduler(SchedulerInterface): + dp_rank=self.parallel_config.data_parallel_index, + log_stats=self.log_stats, + ) ++ if self.opprof is not None: ++ logger.info("OpProf telemetry enabled: %s", self.opprof.writer.path) + + self.use_pp = self.parallel_config.pipeline_parallel_size > 1 + self.use_v2_model_runner = vllm_config.use_v2_model_runner +diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py +index f0330d0..75f63de 100644 +--- a/vllm/v1/opprof.py ++++ b/vllm/v1/opprof.py +@@ -68,6 +68,7 @@ class JSONLWriter: + start: bool = True, + ) -> None: + self._queue: queue.Queue[Any] = queue.Queue(capacity) ++ self.path = path + self._encoder = msgspec.json.Encoder() + self._file = path.open("xb", buffering=1 << 20) + self._thread = threading.Thread(target=self._run, daemon=True) +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/0004-Exclude-OpProf-output-path-from-compile-cache-key.patch b/patches/vllm-0.24.0-opprof/0004-Exclude-OpProf-output-path-from-compile-cache-key.patch new file mode 100644 index 0000000..a2dec50 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0004-Exclude-OpProf-output-path-from-compile-cache-key.patch @@ -0,0 +1,64 @@ +From 335da4abe60e0177872e0b7751e86eeec6756a2b Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sat, 11 Jul 2026 22:32:30 +0800 +Subject: [PATCH 4/5] Exclude OpProf output path from compile cache key + +--- + tests/v1/core/test_opprof.py | 21 ++++++++++++++++++++- + vllm/envs.py | 1 + + 2 files changed, 21 insertions(+), 1 deletion(-) + +diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py +index 79c1fae..a820e7e 100644 +--- a/tests/v1/core/test_opprof.py ++++ b/tests/v1/core/test_opprof.py +@@ -8,7 +8,7 @@ import logging + import sys + import threading + from pathlib import Path +-from types import SimpleNamespace ++from types import ModuleType, SimpleNamespace + + import msgspec + import pytest +@@ -23,6 +23,25 @@ sys.modules[_SPEC.name] = opprof + _SPEC.loader.exec_module(opprof) + + ++def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch): ++ spec = importlib.util.spec_from_file_location( ++ "envs_standalone", _ROOT / "vllm" / "envs.py" ++ ) ++ assert spec is not None and spec.loader is not None ++ envs = importlib.util.module_from_spec(spec) ++ monkeypatch.setitem(sys.modules, spec.name, envs) ++ spec.loader.exec_module(envs) ++ ++ monkeypatch.setitem(sys.modules, "vllm", ModuleType("vllm")) ++ monkeypatch.setitem(sys.modules, "vllm.config", ModuleType("vllm.config")) ++ config_utils = ModuleType("vllm.config.utils") ++ config_utils.__dict__["normalize_value"] = lambda value: value ++ monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils) ++ monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof") ++ ++ assert "VLLM_OPPROF_DIR" not in envs.compile_factors() ++ ++ + class CachedRequests: + def __init__(self, context_ids=()): + self.context_ids = set(context_ids) +diff --git a/vllm/envs.py b/vllm/envs.py +index b3093e9..5634708 100755 +--- a/vllm/envs.py ++++ b/vllm/envs.py +@@ -2044,6 +2044,7 @@ def compile_factors() -> dict[str, object]: + "VLLM_LOGGING_CONFIG_PATH", + "VLLM_LOGGING_COLOR", + "VLLM_LOG_STATS_INTERVAL", ++ "VLLM_OPPROF_DIR", + "VLLM_DEBUG_LOG_API_SERVER_RESPONSE", + "VLLM_TUNED_CONFIG_FOLDER", + "VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR", +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/0005-Keep-compile-factor-regression-import-light.patch b/patches/vllm-0.24.0-opprof/0005-Keep-compile-factor-regression-import-light.patch new file mode 100644 index 0000000..9bd5e05 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0005-Keep-compile-factor-regression-import-light.patch @@ -0,0 +1,24 @@ +From bbfa7176a6a3686a88ee66696f1ad8d754559d96 Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sat, 11 Jul 2026 22:38:08 +0800 +Subject: [PATCH 5/5] Keep compile-factor regression import-light + +--- + tests/v1/core/test_opprof.py | 1 + + 1 file changed, 1 insertion(+) + +diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py +index a820e7e..0b8a8a1 100644 +--- a/tests/v1/core/test_opprof.py ++++ b/tests/v1/core/test_opprof.py +@@ -37,6 +37,7 @@ def test_output_dir_is_not_a_compile_factor(monkeypatch: pytest.MonkeyPatch): + config_utils = ModuleType("vllm.config.utils") + config_utils.__dict__["normalize_value"] = lambda value: value + monkeypatch.setitem(sys.modules, "vllm.config.utils", config_utils) ++ monkeypatch.setitem(sys.modules, "torch", ModuleType("torch")) + monkeypatch.setenv("VLLM_OPPROF_DIR", "/tmp/opprof") + + assert "VLLM_OPPROF_DIR" not in envs.compile_factors() +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/0006-Checkpoint-OpProf-accounting-across-hard-kills.patch b/patches/vllm-0.24.0-opprof/0006-Checkpoint-OpProf-accounting-across-hard-kills.patch new file mode 100644 index 0000000..5208500 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0006-Checkpoint-OpProf-accounting-across-hard-kills.patch @@ -0,0 +1,306 @@ +From f8b68f2452c424d22de4a69527a427207dcfbca5 Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sun, 12 Jul 2026 12:56:39 +0800 +Subject: [PATCH] Checkpoint OpProf accounting across hard kills + +--- + tests/v1/core/test_opprof.py | 118 +++++++++++++++++++++++++++++++++++ + vllm/v1/opprof.py | 84 ++++++++++++++++++++++--- + 2 files changed, 195 insertions(+), 7 deletions(-) + +diff --git a/tests/v1/core/test_opprof.py b/tests/v1/core/test_opprof.py +index 0b8a8a1..007c5bb 100644 +--- a/tests/v1/core/test_opprof.py ++++ b/tests/v1/core/test_opprof.py +@@ -5,6 +5,8 @@ + import errno + import importlib.util + import logging ++import os ++import subprocess + import sys + import threading + from pathlib import Path +@@ -121,6 +123,12 @@ def read_jsonl(path): + return [msgspec.json.decode(line) for line in path.read_bytes().splitlines()] + + ++def read_sidecar(path): ++ return msgspec.json.decode( ++ path.with_name(f"{path.name}.footer.json").read_bytes() ++ ) ++ ++ + def test_import_light_and_approved_constants(): + assert "torch" not in sys.modules + assert "vllm" not in sys.modules +@@ -366,6 +374,116 @@ def test_shutdown_flush_is_idempotent(tmp_path): + assert path.stat().st_size > 0 + + ++def test_sidecar_updates_are_atomic(tmp_path, monkeypatch): ++ writer = opprof.JSONLWriter(tmp_path / "atomic.jsonl", start=False) ++ writer._write_sidecar( ++ encoded_records=1, ++ written_records=1, ++ dropped_records=0, ++ last_step_index=0, ++ final=False, ++ ) ++ original = writer.sidecar_path.read_bytes() ++ real_replace = os.replace ++ replacements = [] ++ ++ def inspect_replace(source, destination): ++ assert Path(destination) == writer.sidecar_path ++ assert writer.sidecar_path.read_bytes() == original ++ candidate = msgspec.json.decode(Path(source).read_bytes()) ++ assert candidate["written_records"] == 2 ++ replacements.append(candidate) ++ real_replace(source, destination) ++ ++ monkeypatch.setattr(opprof.os, "replace", inspect_replace) ++ writer._write_sidecar( ++ encoded_records=3, ++ written_records=2, ++ dropped_records=1, ++ last_step_index=2, ++ final=False, ++ ) ++ monkeypatch.setattr(opprof.os, "replace", real_replace) ++ ++ assert len(replacements) == 1 ++ assert read_sidecar(writer.path)["encoded_records"] == 3 ++ assert not list(tmp_path.glob(".*.tmp-*")) ++ writer.close() ++ ++ ++def test_hard_kill_sidecar_balances_last_flush(tmp_path): ++ path = tmp_path / "hard-kill.jsonl" ++ child = """ ++import importlib.util ++import sys ++import time ++from pathlib import Path ++ ++source, output = sys.argv[1:] ++spec = importlib.util.spec_from_file_location("opprof_hard_kill", source) ++module = importlib.util.module_from_spec(spec) ++sys.modules[spec.name] = module ++spec.loader.exec_module(module) ++writer = module.JSONLWriter(Path(output)) ++for step in range(3): ++ assert writer.submit({"schema": 1, "step_index": step}) ++writer._queue.join() ++while not writer.sidecar_path.exists(): ++ time.sleep(0.01) ++print("READY", flush=True) ++time.sleep(60) ++""" ++ process = subprocess.Popen( ++ [ ++ sys.executable, ++ "-c", ++ child, ++ str(_ROOT / "vllm/v1/opprof.py"), ++ str(path), ++ ], ++ stdout=subprocess.PIPE, ++ text=True, ++ ) ++ try: ++ assert process.stdout is not None ++ assert process.stdout.readline().strip() == "READY" ++ process.kill() ++ assert process.wait(timeout=5) < 0 ++ finally: ++ if process.poll() is None: ++ process.kill() ++ process.wait(timeout=5) ++ ++ records = read_jsonl(path) ++ sidecar = read_sidecar(path) ++ assert len(records) == 3 ++ assert all(record.get("record_type") != "footer" for record in records) ++ assert sidecar["final"] is False ++ assert sidecar["written_records"] == len(records) ++ assert sidecar["encoded_records"] == ( ++ sidecar["written_records"] + sidecar["dropped_records"] ++ ) ++ assert sidecar["last_step_index"] == records[-1]["step_index"] == 2 ++ ++ ++def test_clean_footer_and_final_sidecar_agree(tmp_path): ++ sched = scheduler({"decode": request(8, 9)}) ++ rec, path = recorder(tmp_path) ++ assert emit(rec, sched, schedule_output({"decode": 1})) ++ rec.close() ++ ++ record, footer = read_jsonl(path) ++ sidecar = read_sidecar(path) ++ assert sidecar["record_type"] == "footer_checkpoint" ++ assert sidecar["stream"] == path.name ++ assert sidecar["final"] is True ++ assert sidecar["last_step_index"] == record["step_index"] == 0 ++ assert sidecar["checkpoint_wall_ns"] > 0 ++ assert sidecar["flush_interval_seconds"] == opprof.FLUSH_INTERVAL_SECONDS ++ for counter in ("encoded_records", "written_records", "dropped_records"): ++ assert sidecar[counter] == footer[counter] ++ ++ + def test_piecewise_cudagraph_record_preserved(tmp_path): + sched = scheduler({"decode": request(4096, 4097)}) + output = schedule_output({"decode": 1}) +diff --git a/vllm/v1/opprof.py b/vllm/v1/opprof.py +index 75f63de..28d9635 100644 +--- a/vllm/v1/opprof.py ++++ b/vllm/v1/opprof.py +@@ -7,6 +7,7 @@ import queue + import threading + import time + from bisect import bisect_left ++from contextlib import suppress + from pathlib import Path + from typing import Any + +@@ -18,6 +19,7 @@ SCHEMA_VERSION = 1 + CONTEXT_LENGTH_EDGES = tuple(1 << exponent for exponent in range(7, 18)) + CHUNK_SIZE_EDGES = tuple(1 << exponent for exponent in range(4, 12)) + DEFAULT_QUEUE_CAPACITY = 8192 ++FLUSH_INTERVAL_SECONDS = 1.0 + _CLOSE_TIMEOUT_SECONDS = 1.0 + _STOP = object() + _PREFIX_FIELDS = ( # noqa: SIM905 +@@ -69,6 +71,7 @@ class JSONLWriter: + ) -> None: + self._queue: queue.Queue[Any] = queue.Queue(capacity) + self.path = path ++ self.sidecar_path = path.with_name(f"{path.name}.footer.json") + self._encoder = msgspec.json.Encoder() + self._file = path.open("xb", buffering=1 << 20) + self._thread = threading.Thread(target=self._run, daemon=True) +@@ -78,6 +81,9 @@ class JSONLWriter: + self.failure: Exception | None = None + self.encoded_records = self.written_records = 0 + self.dropped_records = self._unreported_drops = 0 ++ self._checkpoint_encoded_records = 0 ++ self._checkpoint_dropped_records = 0 ++ self._last_written_step_index: int | None = None + if start: + self.start() + +@@ -119,33 +125,90 @@ class JSONLWriter: + if self._writer_unavailable(): + return self._drop(pending) + try: +- self._queue.put_nowait(payload) ++ self._queue.put_nowait( ++ ( ++ payload, ++ self.encoded_records, ++ self.dropped_records, ++ int(record["step_index"]), ++ ) ++ ) + except queue.Full: + return self._drop(pending) + self._unreported_drops = 0 + return True + ++ def _write_sidecar( ++ self, ++ *, ++ encoded_records: int, ++ written_records: int, ++ dropped_records: int, ++ last_step_index: int | None, ++ final: bool, ++ ) -> None: ++ sidecar = dict( ++ schema=SCHEMA_VERSION, ++ record_type="footer_checkpoint", ++ stream=self.path.name, ++ encoded_records=encoded_records, ++ written_records=written_records, ++ dropped_records=dropped_records, ++ last_step_index=last_step_index, ++ checkpoint_wall_ns=time.time_ns(), ++ flush_interval_seconds=FLUSH_INTERVAL_SECONDS, ++ final=final, ++ ) ++ temporary_path = self.sidecar_path.with_name( ++ f".{self.sidecar_path.name}.tmp-{os.getpid()}-{threading.get_ident()}" ++ ) ++ try: ++ with temporary_path.open("wb") as temporary_file: ++ temporary_file.write(self._encoder.encode(sidecar) + b"\n") ++ temporary_file.flush() ++ os.replace(temporary_path, self.sidecar_path) ++ finally: ++ with suppress(FileNotFoundError): ++ temporary_path.unlink() ++ ++ def _flush_checkpoint(self) -> None: ++ self._file.flush() ++ self._write_sidecar( ++ encoded_records=self._checkpoint_encoded_records, ++ written_records=self.written_records, ++ dropped_records=self._checkpoint_dropped_records, ++ last_step_index=self._last_written_step_index, ++ final=False, ++ ) ++ + def _run(self) -> None: + buffered = 0 + last_flush = time.monotonic() + try: + while True: + try: +- item = self._queue.get(timeout=1.0) ++ item = self._queue.get(timeout=FLUSH_INTERVAL_SECONDS) + except queue.Empty: +- self._file.flush() ++ self._flush_checkpoint() + buffered = 0 + last_flush = time.monotonic() + continue + try: + if item is _STOP: + break +- self._file.write(item) +- buffered += len(item) ++ payload, encoded, dropped, step_index = item ++ self._file.write(payload) ++ buffered += len(payload) + self.written_records += 1 ++ self._checkpoint_encoded_records = encoded ++ self._checkpoint_dropped_records = dropped ++ self._last_written_step_index = step_index + now = time.monotonic() +- if buffered >= 1 << 20 or now - last_flush >= 1.0: +- self._file.flush() ++ if ( ++ buffered >= 1 << 20 ++ or now - last_flush >= FLUSH_INTERVAL_SECONDS ++ ): ++ self._flush_checkpoint() + buffered = 0 + last_flush = now + finally: +@@ -163,6 +226,13 @@ class JSONLWriter: + try: + self._file.write(self._encoder.encode(footer) + b"\n") + self._file.flush() ++ self._write_sidecar( ++ encoded_records=footer["encoded_records"], ++ written_records=footer["written_records"], ++ dropped_records=footer["dropped_records"], ++ last_step_index=self._last_written_step_index, ++ final=True, ++ ) + except Exception as error: + self._record_failure(error) + finally: +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/0007-Recreate-scheduled-torch-profiler-between-windows.patch b/patches/vllm-0.24.0-opprof/0007-Recreate-scheduled-torch-profiler-between-windows.patch new file mode 100644 index 0000000..80314f3 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/0007-Recreate-scheduled-torch-profiler-between-windows.patch @@ -0,0 +1,27 @@ +From 23450fb21ac255b0cf710f4ee965ee694921975d Mon Sep 17 00:00:00 2001 +From: Gahow Wang +Date: Sun, 12 Jul 2026 13:12:52 +0800 +Subject: [PATCH] Recreate scheduled torch profiler between windows + +--- + vllm/v1/worker/gpu_worker.py | 4 ++++ + 1 file changed, 4 insertions(+) + +diff --git a/vllm/v1/worker/gpu_worker.py b/vllm/v1/worker/gpu_worker.py +index 5e266a3..0058f96 100644 +--- a/vllm/v1/worker/gpu_worker.py ++++ b/vllm/v1/worker/gpu_worker.py +@@ -978,6 +978,10 @@ class Worker(WorkerBase): + logger.warning("Profiler was not started, nothing to stop.") + return + self.profiler.stop() ++ # A scheduled torch.profiler.profile does not reset its schedule ++ # after stop(). Recreate it for the next /start_profile window. ++ if isinstance(self.profiler, TorchProfilerWrapper): ++ self.profiler = None + + def execute_dummy_batch(self) -> None: + num_tokens = getattr(self.model_runner, "uniform_decode_query_len", 1) +-- +2.43.0 + diff --git a/patches/vllm-0.24.0-opprof/README.md b/patches/vllm-0.24.0-opprof/README.md new file mode 100644 index 0000000..33a876b --- /dev/null +++ b/patches/vllm-0.24.0-opprof/README.md @@ -0,0 +1,119 @@ +# vLLM 0.24.0 OpProf patch series + +## Goal + +Apply the accepted OpProf Layer-1 instrumentation to exactly vLLM `v0.24.0` +at base commit `ee0da84ab9e04ac7610e28580af62c365e898389`. The series adds one +scheduler-owned composition record per step without installing new runtime +dependencies or changing GPU kernels. + +## Contents + +- `0001-Add-lightweight-per-step-OpProf-telemetry.patch`: adds the environment + switch, import-light JSONL recorder/writer, scheduler hooks, and reuse of the + existing CUDA-graph stat. Writer failures are exposed without blocking + producers or shutdown, and request histograms are accumulated in-place. +- `0002-Add-standalone-OpProf-telemetry-tests.patch`: adds CPU-only tests that + load the recorder directly without importing or installing vLLM or torch, + including ENOSPC, golden-record, and zero-token regressions. +- `0003-Log-the-OpProf-output-path-at-startup.patch`: logs the resolved JSONL + output path and covers it in the standalone shutdown test. +- `0004-Exclude-OpProf-output-path-from-compile-cache-key.patch`: prevents the + per-run telemetry destination from invalidating vLLM's torch.compile/AOT + cache and adds an import-light regression test. +- `0005-Keep-compile-factor-regression-import-light.patch`: isolates the new + regression from torch in full vLLM test environments. +- `0006-Checkpoint-OpProf-accounting-across-hard-kills.patch`: atomically + checkpoints balanced writer counters beside each JSONL stream once per + flush interval, with clean-close and hard-kill regressions. +- `0007-Recreate-scheduled-torch-profiler-between-windows.patch`: discards a + stopped scheduled torch-profiler wrapper so each subsequent official profile + endpoint call receives a fresh 2+8 schedule and emits its own trace. +- `apply.sh`: verifies the exact base, refuses dirty/wrong revisions, applies + all numbered patches with `git am`, and exits successfully only when the + exact series is already applied directly on the required base. +- `pytest-evidence.txt`: exact isolated test command, dependency versions, and + all-pass summary. + +The source branch tip used to generate the patches is +`23450fb21ac255b0cf710f4ee965ee694921975d` (`opprof`). + +## Apply + +Prerequisite: a clean checkout whose `HEAD` is the exact base commit. + +```bash +./patches/vllm-0.24.0-opprof/apply.sh /path/to/vllm-v0.24.0 +``` + +Running the command again is a no-op only when the five matching patch commits +are rooted directly at the required base. A partially applied series, dirty +tree, unrelated commit, or any other `HEAD` is rejected instead of being +guessed around. + +## Enable and output + +Set an absolute output directory before starting vLLM: + +```bash +export VLLM_OPPROF_DIR=/absolute/path/to/run/opprof +``` + +Unset or empty disables the feature before recorder construction. Combining it +with `--disable-log-stats` fails fast, as approved. + +Each EngineCore/DP scheduler writes one file named approximately +`opprof-v1-dp0-pid1234-.jsonl`. Records contain schema/engine/step and +timestamps; scheduled prefill/decode composition; first/middle/final/unsplit +prefill chunks; 12-bin context and 9-bin chunk-size histograms; preemptions; +running/waiting/deferred queues; KV blocks/usage; local/external prefix deltas; +CUDA-graph hit/mode/bucket/padding; explicit null Layer-1 MoE load; and drop-gap +accounting. A clean close writes a final writer-count footer in the stream. + +Every JSONL flush also atomically replaces +`.footer.json` through a same-directory temporary file. The sidecar +contains the encoded, written, and dropped counts through that durable flush, +the last written step index, a wall-clock timestamp, the one-second flush +interval, and whether it is final. Queue entries carry their submission +ordinal and cumulative drops, so a periodic checkpoint always satisfies +`encoded = written + dropped` without decoding records in the writer thread. +On clean close the in-stream footer is authoritative and the final sidecar must +agree with all three counters. If a hard kill prevents the in-stream footer, +the latest sidecar is authoritative: the decoded data-line count must equal +its `written_records`, its final data-line step must equal +`last_step_index`, and its counters must balance. Data after that checkpoint +may be lost, bounded by at most the configured one-second flush interval. + +The bounded queue holds 8192 encoded records. Producers never wait for disk; +full queues or a failed writer drop the new record and report the gap on the +next successful record. A writer I/O failure is exposed through recorder state, +logged once, and cannot make shutdown wait indefinitely. The writer flushes at +1 MiB, one second, or shutdown. + +## Test + +Only pytest and msgspec are required. `--confcutdir` prevents vLLM's global +test configuration from importing its full dependency stack. + +```bash +cd /path/to/vllm-v0.24.0 +uv run --no-project --with pytest --with msgspec \ + pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q +``` + +Expected summary: + +```text +18 passed in 1.09s +``` + +## Caveats + +- Layer 1 intentionally records no expert-load arrays. Exact routed experts + remain a separate Layer-2 run. +- `PIECEWISE` means graph-wrapped compiled regions, not full-step graph replay. +- Phase 2 must measure the always-on overhead; acceptance requires the upper + bound of the 95% confidence interval to remain below 3% for every primary + serving metric. +- Primary campaign topology is TP1 on community BF16 Qwen3-30B-A3B, with TP2 + and TP4 counterpoints. Record the selected MoE backend log every run. diff --git a/patches/vllm-0.24.0-opprof/apply.sh b/patches/vllm-0.24.0-opprof/apply.sh new file mode 100755 index 0000000..fe6dbf5 --- /dev/null +++ b/patches/vllm-0.24.0-opprof/apply.sh @@ -0,0 +1,58 @@ +#!/usr/bin/env bash +set -euo pipefail + +base_commit=ee0da84ab9e04ac7610e28580af62c365e898389 +script_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd) +repo=${1:-.} +patches=("$script_dir"/0*.patch) + +git -C "$repo" rev-parse --git-dir >/dev/null +if [[ -n $(git -C "$repo" status --porcelain) ]]; then + echo "Refusing to apply to a dirty worktree." >&2 + exit 1 +fi +if ((${#patches[@]} == 0)) || [[ ! -f ${patches[0]} ]]; then + echo "No numbered patch files found in $script_dir" >&2 + exit 1 +fi + +head_commit=$(git -C "$repo" rev-parse HEAD) +mapfile -t recent_commits < <( + git -C "$repo" rev-list --max-count="${#patches[@]}" --reverse HEAD +) +patches_match=true +((${#recent_commits[@]} == ${#patches[@]})) || patches_match=false +for i in "${!patches[@]}"; do + $patches_match || break + patch_id=$(git patch-id --stable <"${patches[$i]}" | awk '{print $1}') + commit_id=$( + git -C "$repo" show --pretty=format: "${recent_commits[$i]}" | + git patch-id --stable | awk '{print $1}' + ) + [[ $patch_id == "$commit_id" ]] || patches_match=false +done +first_parent="" +if ((${#recent_commits[@]} > 0)); then + first_parent=$( + git -C "$repo" rev-parse --verify "${recent_commits[0]}^" 2>/dev/null || true + ) +fi +if $patches_match && [[ $first_parent == "$base_commit" ]]; then + echo "OpProf patch series is already applied." + exit 0 +fi + +if [[ $head_commit == "$base_commit" ]]; then + git -C "$repo" am "${patches[@]}" + echo "Applied OpProf patch series to $repo" + exit 0 +fi +if $patches_match; then + echo "Refusing to treat OpProf patches as already applied:" >&2 + echo "first patch parent is $first_parent, expected $base_commit" >&2 + exit 1 +fi + +echo "Refusing to apply: HEAD is $head_commit; expected $base_commit" >&2 +echo "or the exact OpProf patch series rooted at that base." >&2 +exit 1 diff --git a/patches/vllm-0.24.0-opprof/pytest-evidence.txt b/patches/vllm-0.24.0-opprof/pytest-evidence.txt new file mode 100644 index 0000000..38444cb --- /dev/null +++ b/patches/vllm-0.24.0-opprof/pytest-evidence.txt @@ -0,0 +1,16 @@ +OpProf standalone pytest evidence +Date: 2026-07-12 +Source branch: opprof +Source tip: 23450fb21ac255b0cf710f4ee965ee694921975d +Base: ee0da84ab9e04ac7610e28580af62c365e898389 (v0.24.0) +Environment: Python 3.11.13, pytest 9.1.1, msgspec 0.21.1 +vLLM installed: no +torch installed in isolated test environment: no +GPU/remote access: no + +Command: +uv run --no-project --with pytest --with msgspec pytest --confcutdir=tests/v1/core tests/v1/core/test_opprof.py -q + +Output: +.................. [100%] +18 passed in 1.09s diff --git a/runs/opprof-phase3-ea/provenance/opprof_phase3_client.py b/runs/opprof-phase3-ea/provenance/opprof_phase3_client.py new file mode 100644 index 0000000..7528077 --- /dev/null +++ b/runs/opprof-phase3-ea/provenance/opprof_phase3_client.py @@ -0,0 +1,784 @@ +#!/usr/bin/env python3 +"""Token-exact fixed-duration client for the OpProf Phase-3 protocol.""" + +from __future__ import annotations + +import argparse +import asyncio +import gzip +import hashlib +import json +import math +import os +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import aiohttp + +SCHEMA = 1 +TOKEN_BASE = 1000 +TOKEN_SPAN = 100000 + + +class ManifestExhausted(RuntimeError): + pass + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def atomic_json(path: Path, value: Any, mode: int = 0o640) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode) + with os.fdopen(fd, "w", encoding="utf-8") as f: + json.dump(value, f, sort_keys=True, indent=2) + f.write("\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def atomic_jsonl(path: Path, rows: list[dict[str, Any]], mode: int = 0o640) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode) + with os.fdopen(fd, "w", encoding="utf-8") as f: + for row in rows: + f.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def parse_range(value: str) -> tuple[int, int]: + lo_text, hi_text = value.split(":", 1) + lo, hi = int(lo_text), int(hi_text) + if lo <= 0 or hi < lo: + raise argparse.ArgumentTypeError(f"invalid positive range: {value}") + return lo, hi + + +def _integer_counts(weights: list[float], total: int) -> list[int]: + raw = [w * total for w in weights] + counts = [math.floor(x) for x in raw] + order = sorted( + range(len(raw)), key=lambda i: raw[i] - counts[i], reverse=True + ) + for idx in order[: total - sum(counts)]: + counts[idx] += 1 + return counts + + +def numeric_sanity(values: list[float | int]) -> dict[str, Any]: + finite = [float(x) for x in values if math.isfinite(float(x))] + return { + "n": len(values), + "finite_n": len(finite), + "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, + "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def manifest_summary(rows: list[dict[str, Any]]) -> dict[str, Any]: + return { + "schema": SCHEMA, + "rows": len(rows), + "input_tokens": numeric_sanity([int(r["input_tokens"]) for r in rows]), + "output_tokens": numeric_sanity([int(r["output_tokens"]) for r in rows]), + "arrival_values": sorted({str(r["arrival"]) for r in rows}), + "pattern_values": sorted({str(r["pattern_id"]) for r in rows}), + } + + +def materialize(args: argparse.Namespace) -> dict[str, Any]: + import numpy as np + + rng = np.random.default_rng(args.workload_seed) + n = args.num_requests + if args.kind == "prefix-pool": + if args.num_prefixes <= 0 or args.prefix_len <= 0 or args.suffix_fixed <= 0: + raise ValueError("prefix-pool requires positive pool/prefix/suffix") + lengths = np.full(n, args.prefix_len + args.suffix_fixed, dtype=np.int64) + prefix_ids = np.arange(n, dtype=np.int64) % args.num_prefixes + rng.shuffle(prefix_ids) + else: + prefix_ids = np.full(n, -1, dtype=np.int64) + if args.input_uniform: + lo, hi = parse_range(args.input_uniform) + lengths = rng.integers(lo, hi + 1, n, dtype=np.int64) + elif args.input_fixed: + lengths = np.full(n, args.input_fixed, dtype=np.int64) + elif args.input_mixture: + spec = json.loads(args.input_mixture) + if not isinstance(spec, dict) or not spec: + raise ValueError("input mixture must be a non-empty JSON object") + keys = list(spec) + weights = [float(spec[key]) for key in keys] + if any(w < 0 for w in weights) or not math.isclose(sum(weights), 1.0): + raise ValueError("mixture weights must be non-negative and sum to 1") + pieces = [] + for key, count in zip( + keys, _integer_counts(weights, n), strict=True + ): + kind, lo_text, hi_text = key.split(":") + if kind != "uniform": + raise ValueError(f"unsupported mixture component: {key}") + pieces.append( + rng.integers( + int(lo_text), int(hi_text) + 1, count, dtype=np.int64 + ) + ) + lengths = np.concatenate(pieces) + rng.shuffle(lengths) + else: + raise ValueError("exactly one input distribution is required") + if args.output_fixed <= 0 or args.arrival not in {"steady", "burst:8"}: + raise ValueError("invalid output length or arrival class") + + rows = [] + for i in range(n): + row = { + "schema": SCHEMA, + "request_id": f"{args.id}-{i:05d}", + "pattern_id": args.id, + "kind": args.kind, + "input_tokens": int(lengths[i]), + "output_tokens": args.output_fixed, + "arrival": args.arrival, + "token_seed": int(args.workload_seed * 1000003 + i), + } + if args.kind == "prefix-pool": + row.update( + { + "prefix_id": int(prefix_ids[i]), + "num_prefixes": args.num_prefixes, + "prefix_tokens": args.prefix_len, + } + ) + rows.append(row) + + out = Path(args.out) + atomic_jsonl(out, rows, mode=0o600) + summary = manifest_summary(rows) + summary.update({"sha256": sha256_file(out), "path": str(out)}) + atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600) + print(json.dumps(summary, sort_keys=True)) + return summary + + +def materialize_private(args: argparse.Namespace) -> dict[str, Any]: + from transformers import AutoTokenizer + + source = Path(args.source) + selected: list[dict[str, Any]] = [] + with source.open(encoding="utf-8") as f: + for source_index, line in enumerate(f): + row = json.loads(line) + if ( + float(row["sampling_u"]) <= args.sampling_u_max + and int(row["input_length"]) <= args.max_input_tokens + ): + selected.append( + { + "schema": SCHEMA, + "request_id": f"{args.id}-{len(selected):05d}", + "pattern_id": args.id, + "kind": "private-trace", + "input_tokens": int(row["input_length"]), + "output_tokens": min( + int(row["output_length"]), args.output_cap + ), + "arrival": args.arrival, + "source_index": source_index, + "prompt": row["prompt"], + } + ) + + tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True) + diffs = [ + len(tokenizer.encode(row["prompt"], add_special_tokens=False)) + - row["input_tokens"] + for row in selected + ] + exact = sum(diff == 0 for diff in diffs) + exact_fraction = exact / len(diffs) if diffs else 0.0 + max_abs = max((abs(diff) for diff in diffs), default=-1) + if exact_fraction < 0.99 or max_abs > 1: + raise RuntimeError( + "tokenizer parity gate failed: " + f"exact_fraction={exact_fraction:.6f} max_abs_error={max_abs}" + ) + + out = Path(args.out) + atomic_jsonl(out, selected, mode=0o600) + summary = manifest_summary(selected) + summary.update( + { + "sha256": sha256_file(out), + "source_sha256": sha256_file(source), + "tokenizer_exact_n": exact, + "tokenizer_exact_fraction": exact_fraction, + "tokenizer_max_abs_error": max_abs, + "path": str(out), + } + ) + atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600) + print(json.dumps(summary, sort_keys=True)) + return summary + + +def load_manifest(path: Path) -> list[dict[str, Any]]: + rows = [json.loads(line) for line in path.read_text().splitlines() if line] + required = { + "request_id", + "pattern_id", + "input_tokens", + "output_tokens", + "arrival", + } + if not rows: + raise ValueError("empty manifest") + for row in rows: + if not required.issubset(row): + raise ValueError(f"manifest row lacks {sorted(required - set(row))}") + if len({row["request_id"] for row in rows}) != len(rows): + raise ValueError("duplicate request_id") + return rows + + +def _token_stream(seed: int, count: int) -> list[int]: + state = seed & 0xFFFFFFFF + out = [] + for _ in range(count): + state = (1664525 * state + 1013904223) & 0xFFFFFFFF + out.append(TOKEN_BASE + state % TOKEN_SPAN) + return out + + +def synthetic_prompt(row: dict[str, Any]) -> list[int]: + length = int(row["input_tokens"]) + seed = int(row.get("token_seed", 0)) + if row.get("kind") == "prefix-pool": + prefix_n = int(row["prefix_tokens"]) + tokens = _token_stream(0xA5A50000 + int(row["prefix_id"]), prefix_n) + tokens += _token_stream(seed, length - prefix_n) + offset = prefix_n + else: + tokens = _token_stream(seed, length) + offset = 0 + if length - offset >= 3: + index = int(row["request_id"].rsplit("-", 1)[1]) + tokens[offset : offset + 3] = [ + TOKEN_BASE + index % 100, + TOKEN_BASE + (index // 100) % 100, + TOKEN_BASE + (index // 10000) % 100, + ] + return tokens + + +@dataclass +class RunContext: + args: argparse.Namespace + rows: list[dict[str, Any]] + t0: float + clean_end: float + stop_event: asyncio.Event + lock: asyncio.Lock + next_index: int = 0 + in_flight: int = 0 + max_in_flight: int = 0 + exhausted: bool = False + admission_stop_s: float | None = None + + async def next_row(self) -> dict[str, Any]: + async with self.lock: + if self.next_index >= len(self.rows): + self.exhausted = True + raise ManifestExhausted( + f"manifest exhausted after {self.next_index} admissions" + ) + row = self.rows[self.next_index] + self.next_index += 1 + return row + + +async def request_one( + ctx: RunContext, + session: aiohttp.ClientSession, + row: dict[str, Any], + scheduled: float, +) -> dict[str, Any]: + loop = asyncio.get_running_loop() + admitted = loop.time() + ctx.in_flight += 1 + ctx.max_in_flight = max(ctx.max_in_flight, ctx.in_flight) + status = 0 + actual_output: int | None = None + first_token: float | None = None + error_kind: str | None = None + try: + prompt: str | list[int] = ( + row["prompt"] + if row.get("kind") == "private-trace" + else synthetic_prompt(row) + ) + if not isinstance(prompt, str) and len(prompt) != int(row["input_tokens"]): + raise AssertionError("synthetic prompt length drift") + payload = { + "model": ctx.args.model, + "prompt": prompt, + "max_tokens": int(row["output_tokens"]), + "temperature": ctx.args.temperature, + "ignore_eos": ctx.args.ignore_eos, + "stream": True, + "stream_options": {"include_usage": True}, + "add_special_tokens": False, + "seed": ctx.args.server_seed, + } + headers = { + "Content-Type": "application/json", + "x-request-id": str(row["request_id"]), + } + async with session.post( + ctx.args.base_url.rstrip("/") + "/v1/completions", + json=payload, + headers=headers, + ) as response: + status = response.status + if status != 200: + error_kind = f"http_{status}" + else: + buf = b"" + async for chunk in response.content.iter_any(): + buf += chunk + while b"\n" in buf: + line, buf = buf.split(b"\n", 1) + line = line.strip() + if not line.startswith(b"data:"): + continue + data = line[5:].strip() + if data == b"[DONE]": + continue + event = json.loads(data) + if event.get("choices") and first_token is None: + first_token = loop.time() + if event.get("usage") is not None: + actual_output = int(event["usage"]["completion_tokens"]) + if actual_output is None: + error_kind = "missing_usage" + except (aiohttp.ClientError, asyncio.TimeoutError) as exc: + error_kind = type(exc).__name__ + except Exception as exc: + error_kind = type(exc).__name__ + finally: + completed = loop.time() + ctx.in_flight -= 1 + success = ( + status == 200 + and error_kind is None + and actual_output == int(row["output_tokens"]) + ) + if status == 200 and actual_output is not None and not success: + error_kind = "output_token_mismatch" + return { + "schema": SCHEMA, + "request_id": row["request_id"], + "scheduled_s": scheduled - ctx.t0, + "admitted_s": admitted - ctx.t0, + "first_token_s": None if first_token is None else first_token - ctx.t0, + "completed_s": completed - ctx.t0, + "input_tokens": int(row["input_tokens"]), + "requested_output_tokens": int(row["output_tokens"]), + "actual_output_tokens": actual_output, + "http_status": status, + "success": success, + "error_kind": error_kind, + } + + +async def saturation_load( + ctx: RunContext, session: aiohttp.ClientSession +) -> list[dict[str, Any]]: + results: list[dict[str, Any]] = [] + + async def worker() -> None: + while not ctx.stop_event.is_set(): + try: + row = await ctx.next_row() + except ManifestExhausted: + ctx.stop_event.set() + return + results.append( + await request_one(ctx, session, row, asyncio.get_running_loop().time()) + ) + + tasks = [ + asyncio.create_task(worker()) for _ in range(ctx.args.max_concurrency) + ] + await asyncio.gather(*tasks) + return results + + +async def finite_load( + ctx: RunContext, session: aiohttp.ClientSession, rate: float +) -> list[dict[str, Any]]: + sem = asyncio.Semaphore(ctx.args.max_concurrency) + tasks: list[asyncio.Task[dict[str, Any]]] = [] + batch = 8 if str(ctx.rows[0]["arrival"]) == "burst:8" else 1 + period = batch / rate + event_index = 0 + + async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]: + async with sem: + return await request_one(ctx, session, row, scheduled) + + while not ctx.stop_event.is_set(): + scheduled = ctx.t0 + event_index * period + delay = scheduled - asyncio.get_running_loop().time() + if delay > 0: + try: + await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay) + break + except asyncio.TimeoutError: + pass + if ctx.stop_event.is_set(): + break + try: + for _ in range(batch): + tasks.append( + asyncio.create_task(limited(await ctx.next_row(), scheduled)) + ) + except ManifestExhausted: + ctx.stop_event.set() + break + event_index += 1 + return await asyncio.gather(*tasks) if tasks else [] + + +async def post_profile( + session: aiohttp.ClientSession, base_url: str, endpoint: str +) -> tuple[float, float, int]: + loop = asyncio.get_running_loop() + before = loop.time() + async with session.post(base_url.rstrip("/") + endpoint) as response: + status = response.status + await response.read() + return before, loop.time(), status + + +def _trace_loadable(path: Path) -> bool: + try: + opener = gzip.open if path.suffix == ".gz" else open + with opener(path, "rt", encoding="utf-8") as f: + parsed = json.load(f) + return isinstance(parsed, dict) and isinstance(parsed.get("traceEvents"), list) + except (OSError, EOFError, json.JSONDecodeError): + return False + + +async def wait_new_trace( + trace_dir: Path, before: set[Path], timeout: float +) -> Path: + deadline = asyncio.get_running_loop().time() + timeout + while asyncio.get_running_loop().time() < deadline: + for path in sorted(set(trace_dir.glob("*.pt.trace.json*")) - before): + if _trace_loadable(path): + return path + await asyncio.sleep(0.25) + raise TimeoutError(f"no new loadable trace within {timeout}s") + + +async def timeline( + ctx: RunContext, session: aiohttp.ClientSession +) -> list[dict[str, Any]]: + args = ctx.args + profiles: list[dict[str, Any]] = [] + await asyncio.sleep(max(0, ctx.clean_end - asyncio.get_running_loop().time())) + if args.profile_after_clean: + trace_dir = Path(args.profile_trace_dir) + for window in range(args.num_profile_windows): + prior = set(trace_dir.glob("*.pt.trace.json*")) + start_before, start_after, start_status = await post_profile( + session, args.base_url, "/start_profile" + ) + trace = await wait_new_trace( + trace_dir, prior, args.profile_timeout_seconds + ) + trace_ready = asyncio.get_running_loop().time() + stop_before, stop_after, stop_status = await post_profile( + session, args.base_url, "/stop_profile" + ) + profiles.append( + { + "window": window + 1, + "start_call_s": start_before - ctx.t0, + "start_return_s": start_after - ctx.t0, + "trace_ready_s": trace_ready - ctx.t0, + "stop_call_s": stop_before - ctx.t0, + "stop_return_s": stop_after - ctx.t0, + "start_status": start_status, + "stop_status": stop_status, + "trace_file": trace.name, + "trace_sha256": sha256_file(trace), + } + ) + if start_status != 200 or stop_status != 200: + raise RuntimeError("profile endpoint returned non-200") + await asyncio.sleep(args.recovery_seconds) + else: + await asyncio.sleep(args.post_clean_seconds) + ctx.admission_stop_s = asyncio.get_running_loop().time() - ctx.t0 + ctx.stop_event.set() + return profiles + + +def segment_summary( + records: list[dict[str, Any]], start: float, end: float +) -> dict[str, Any]: + admitted = [r for r in records if start <= r["admitted_s"] < end] + completed = [r for r in records if start <= r["completed_s"] < end] + successes = [r for r in completed if r["success"]] + duration = end - start + return { + "start_s": start, + "end_s": end, + "duration_s": duration, + "admitted": len(admitted), + "completed": len(successes), + "failed": len(completed) - len(successes), + "offered_rps": len(admitted) / duration, + "completed_throughput_rps": len(successes) / duration, + "input_tokens": sum(r["input_tokens"] for r in successes), + "output_tokens": sum(r["actual_output_tokens"] or 0 for r in successes), + } + + +async def run_load(args: argparse.Namespace) -> dict[str, Any]: + manifest = Path(args.manifest) + rows = load_manifest(manifest) + arrivals = {row["arrival"] for row in rows} + if len(arrivals) != 1: + raise ValueError("a manifest must have one arrival class") + if args.load_point == "saturation": + if args.request_rate != "inf": + raise ValueError("saturation requires --request-rate inf") + rate = math.inf + else: + if not args.saturation_result: + raise ValueError("moderate requires --saturation-result") + sat = json.loads(Path(args.saturation_result).read_text()) + rate = args.rate_fraction * float(sat["clean"]["completed_throughput_rps"]) + if not math.isfinite(rate) or rate <= 0: + raise ValueError("derived moderate rate must be positive and finite") + + loop = asyncio.get_running_loop() + t0 = loop.time() + clean_seconds = args.clean_segment_seconds * args.num_clean_segments + ctx = RunContext( + args=args, + rows=rows, + t0=t0, + clean_end=t0 + args.warmup_seconds + clean_seconds, + stop_event=asyncio.Event(), + lock=asyncio.Lock(), + ) + timeout = aiohttp.ClientTimeout(total=None, connect=30, sock_read=600) + connector = aiohttp.TCPConnector(limit=args.max_concurrency) + async with aiohttp.ClientSession(timeout=timeout, connector=connector) as session: + profile_task = asyncio.create_task(timeline(ctx, session)) + load_task = asyncio.create_task( + saturation_load(ctx, session) + if math.isinf(rate) + else finite_load(ctx, session, rate) + ) + try: + profiles = await profile_task + except Exception: + ctx.stop_event.set() + await load_task + raise + records = await load_task + + clean_start = args.warmup_seconds + clean_end = clean_start + clean_seconds + clean = segment_summary(records, clean_start, clean_end) + segments = [] + for i in range(args.num_clean_segments): + start = clean_start + i * args.clean_segment_seconds + segments.append( + { + "name": chr(ord("A") + i), + **segment_summary( + records, start, start + args.clean_segment_seconds + ), + } + ) + successful = [r for r in records if r["success"]] + elapsed_seconds = loop.time() - t0 + if ctx.admission_stop_s is None: + raise RuntimeError("admission stop timestamp was not recorded") + drain_seconds = elapsed_seconds - ctx.admission_stop_s + result = { + "schema": SCHEMA, + "manifest_sha256": sha256_file(manifest), + "manifest_rows": len(rows), + "manifest_admitted": ctx.next_index, + "manifest_wrapped": False, + "manifest_exhausted": ctx.exhausted, + "load_point": args.load_point, + "request_rate": "inf" if math.isinf(rate) else rate, + "rate_fraction": None if math.isinf(rate) else args.rate_fraction, + "arrival": next(iter(arrivals)), + "warmup_seconds": args.warmup_seconds, + "clean_segment_seconds": args.clean_segment_seconds, + "num_clean_segments": args.num_clean_segments, + "elapsed_seconds": elapsed_seconds, + "admission_stop_s": ctx.admission_stop_s, + "drain_seconds": drain_seconds, + "max_in_flight": ctx.max_in_flight, + "records": len(records), + "successful_records": len(successful), + "failed_records": len(records) - len(successful), + "clean": clean, + "segments": segments, + "profiles": profiles, + } + sanity = { + "schema": SCHEMA, + "numeric": { + "input_tokens": numeric_sanity([r["input_tokens"] for r in records]), + "requested_output_tokens": numeric_sanity( + [r["requested_output_tokens"] for r in records] + ), + "actual_output_tokens": numeric_sanity( + [ + r["actual_output_tokens"] + for r in records + if r["actual_output_tokens"] is not None + ] + ), + "scheduled_s": numeric_sanity([r["scheduled_s"] for r in records]), + "admitted_s": numeric_sanity([r["admitted_s"] for r in records]), + "completed_s": numeric_sanity([r["completed_s"] for r in records]), + }, + "invariants": { + "clean_duration_exact": math.isclose(clean["duration_s"], clean_seconds), + "segment_count_exact": len(segments) == args.num_clean_segments, + "manifest_no_wrap": ctx.next_index <= len(rows), + "manifest_not_exhausted": not ctx.exhausted, + "concurrency_bounded": ctx.max_in_flight <= args.max_concurrency, + "drain_within_timeout": drain_seconds <= args.drain_timeout_seconds, + "output_tokens_exact": all( + r["actual_output_tokens"] == r["requested_output_tokens"] + for r in successful + ), + "clean_failures_zero": clean["failed"] == 0, + "profile_count_exact": len(profiles) + == (args.num_profile_windows if args.profile_after_clean else 0), + "profile_status_ok": all( + p["start_status"] == 200 and p["stop_status"] == 200 + for p in profiles + ), + }, + } + if not math.isinf(rate): + sanity["invariants"]["moderate_offered_within_5pct"] = ( + abs(clean["offered_rps"] / rate - 1) <= 0.05 + ) + out = Path(args.result_dir) + out.mkdir(parents=True, exist_ok=True) + atomic_jsonl(out / "requests.jsonl", sorted(records, key=lambda r: r["admitted_s"])) + atomic_jsonl(out / "segments.jsonl", segments) + atomic_json(out / "result.json", result) + atomic_json(out / "sanity.json", sanity) + if ctx.exhausted: + raise ManifestExhausted("manifest exhausted; result retained for diagnosis") + failed = [name for name, ok in sanity["invariants"].items() if not ok] + if failed: + raise RuntimeError(f"client sanity failure: {failed}") + return result + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + mat = sub.add_parser("materialize") + mat.add_argument("--id", required=True) + mat.add_argument("--kind", choices=("synthetic", "prefix-pool"), required=True) + group = mat.add_mutually_exclusive_group() + group.add_argument("--input-uniform") + group.add_argument("--input-fixed", type=int) + group.add_argument("--input-mixture") + mat.add_argument("--output-fixed", type=int, required=True) + mat.add_argument("--prefix", default="none") + mat.add_argument("--arrival", required=True) + mat.add_argument("--num-requests", type=int, required=True) + mat.add_argument("--workload-seed", type=int, required=True) + mat.add_argument("--num-prefixes", type=int, default=0) + mat.add_argument("--prefix-len", type=int, default=0) + mat.add_argument("--suffix-fixed", type=int, default=0) + mat.add_argument("--out", required=True) + private = sub.add_parser("materialize-private") + private.add_argument("--id", required=True) + private.add_argument("--source", required=True) + private.add_argument("--sampling-u-max", type=float, required=True) + private.add_argument("--max-input-tokens", type=int, required=True) + private.add_argument("--output-cap", type=int, required=True) + private.add_argument("--preserve-prompts", action="store_true", required=True) + private.add_argument("--disable-shuffle", action="store_true", required=True) + private.add_argument("--arrival", required=True) + private.add_argument("--model", required=True) + private.add_argument("--out", required=True) + run = sub.add_parser("run") + run.add_argument("--manifest", required=True) + run.add_argument("--base-url", required=True) + run.add_argument("--model", required=True) + run.add_argument("--load-point", choices=("saturation", "moderate"), required=True) + run.add_argument("--request-rate") + run.add_argument("--saturation-result") + run.add_argument("--rate-fraction", type=float, default=0.60) + run.add_argument("--max-concurrency", type=int, default=256) + run.add_argument("--ignore-eos", action="store_true") + run.add_argument("--temperature", type=float, default=0.0) + run.add_argument("--warmup-seconds", type=float, default=60) + run.add_argument("--clean-segment-seconds", type=float, default=80) + run.add_argument("--num-clean-segments", type=int, default=3) + run.add_argument("--profile-after-clean", action="store_true") + run.add_argument("--num-profile-windows", type=int, default=0) + run.add_argument("--profile-warmup-iterations", type=int, default=2) + run.add_argument("--profile-active-iterations", type=int, default=8) + run.add_argument("--profile-trace-dir") + run.add_argument("--profile-timeout-seconds", type=float, default=120) + run.add_argument("--recovery-seconds", type=float, default=30) + run.add_argument("--post-clean-seconds", type=float, default=0) + run.add_argument("--drain-timeout-seconds", type=float, default=120) + run.add_argument("--workload-seed", type=int, default=20260712) + run.add_argument("--server-seed", type=int, default=20260712) + run.add_argument("--result-dir", required=True) + return parser + + +def main() -> None: + args = build_parser().parse_args() + if args.command == "materialize": + materialize(args) + elif args.command == "materialize-private": + materialize_private(args) + else: + if args.profile_after_clean and not args.profile_trace_dir: + raise ValueError("--profile-after-clean requires --profile-trace-dir") + print(json.dumps(asyncio.run(run_load(args)), sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3-ea/provenance/opprof_phase3_controller.py b/runs/opprof-phase3-ea/provenance/opprof_phase3_controller.py new file mode 100644 index 0000000..b501bd2 --- /dev/null +++ b/runs/opprof-phase3-ea/provenance/opprof_phase3_controller.py @@ -0,0 +1,1045 @@ +#!/usr/bin/env python3 +"""Detached/resumable dash0 controller for the Phase-3 co-location gate.""" + +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import math +import os +import re +import shlex +import shutil +import signal +import subprocess +import sys +import threading +import time +import urllib.request +from collections import defaultdict +from pathlib import Path +from typing import Any + + +SCHEMA = 1 +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/e-a-colocation" +PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +SOURCE = Path( + "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0" +) +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +STATE = RUN_ROOT / "controller-state.json" +CPU_MAP = { + 0: "0-19", + 1: "20-39", + 2: "40-59", + 3: "60-79", + 4: "80-99", + 5: "100-119", + 6: "120-139", + 7: "140-159", +} +BACKGROUND_8 = { + 1: "P01", + 2: "P02", + 3: "P03", + 4: "P04", + 5: "P06", + 6: "P07", + 7: "P08", +} +BACKGROUND_4 = {1: "P01", 2: "P03", 3: "P06"} +PATTERNS = { + "P01": ["--kind", "synthetic", "--input-uniform", "128:512", + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P02": ["--kind", "synthetic", "--input-uniform", "128:512", + "--output-fixed", "512", "--prefix", "none", "--arrival", "steady"], + "P03": ["--kind", "synthetic", "--input-uniform", "4096:8192", + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P04": ["--kind", "synthetic", "--input-uniform", "4096:8192", + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P05": ["--kind", "synthetic", "--input-mixture", + '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P06": ["--kind", "synthetic", "--input-mixture", + '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P07": ["--kind", "synthetic", "--input-fixed", "1280", + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P08": ["--kind", "prefix-pool", "--num-prefixes", "8", + "--prefix-len", "1024", "--suffix-fixed", "256", + "--output-fixed", "512", "--arrival", "burst:8"], +} +PROFILE_CONFIG = { + "profiler": "torch", + "torch_profiler_with_stack": True, + "torch_profiler_record_shapes": True, + "torch_profiler_use_gzip": True, + "ignore_frontend": True, + "wait_iterations": 0, + "warmup_iterations": 2, + "active_iterations": 8, +} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + with tmp.open("w", encoding="utf-8") as f: + json.dump(value, f, sort_keys=True, indent=2) + f.write("\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def run_text(command: list[str], check: bool = True) -> str: + result = subprocess.run( + command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {shlex.join(command)}\n" + f"{result.stdout}" + ) + return result.stdout + + +def command_log(path: Path, label: str, command: list[str], expected: str) -> None: + with path.open("a", encoding="utf-8") as f: + f.write( + f"GPU_COMMAND {label}: {shlex.join(command)} ; expected={expected}\n" + ) + + +def load_state(resume: bool) -> dict[str, Any]: + if STATE.exists(): + if not resume: + raise RuntimeError("state exists; use --resume") + return json.loads(STATE.read_text()) + return { + "schema": SCHEMA, + "status": "created", + "created_at": time.time(), + "controller_pid": os.getpid(), + "stages": {}, + "fingerprint": {}, + } + + +def save_state(state: dict[str, Any]) -> None: + state["updated_at"] = time.time() + state["controller_pid"] = os.getpid() + atomic_json(STATE, state) + + +def make_fingerprint() -> dict[str, Any]: + return { + "client_sha256": sha256_file(CLIENT), + "controller_sha256": sha256_file(Path(__file__)), + "source_commit": run_text( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD"] + ).strip(), + "model_path": str(MODEL), + "venv_python": str(VENV / "bin/python"), + # JSON object keys are strings. Normalize before the first write so a + # read/compare during --resume is byte-semantically stable. + "cpu_map": {str(gpu): cpus for gpu, cpus in CPU_MAP.items()}, + } + + +def ensure_provenance() -> None: + provenance = RUN_ROOT / "provenance" + provenance.mkdir(parents=True, exist_ok=True) + for source in (CLIENT, Path(__file__)): + dest = provenance / source.name + source_sha = sha256_file(source) + if dest.exists() and sha256_file(dest) != source_sha: + # Preserve the prior immutable copy and add the reviewed runtime + # repair as a content-addressed sibling. + dest = provenance / f"{source.stem}.{source_sha[:12]}{source.suffix}" + if dest.exists() and sha256_file(dest) != source_sha: + raise RuntimeError(f"content-addressed provenance changed: {dest}") + if not dest.exists(): + shutil.copy2(source, dest) + checksums = { + path.name: sha256_file(path) + for path in sorted(provenance.glob("*.py")) + } + atomic_json(provenance / "sha256.json", checksums) + + +def ensure_manifests() -> None: + PRIVATE.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(PRIVATE, 0o700) + for pattern, spec in PATTERNS.items(): + out = PRIVATE / f"{pattern}.jsonl" + if out.exists(): + summary = json.loads( + out.with_suffix(out.suffix + ".summary.json").read_text() + ) + if summary["rows"] != 32768 or summary["sha256"] != sha256_file(out): + raise RuntimeError(f"manifest verification failed: {out}") + continue + command = [ + str(VENV / "bin/python"), + str(CLIENT), + "materialize", + "--id", + pattern, + *spec, + "--num-requests", + "32768", + "--workload-seed", + "20260712", + "--out", + str(out), + ] + run_text(command) + + +def gpu_query() -> list[dict[str, Any]]: + text = run_text( + [ + "nvidia-smi", + "--query-gpu=index,uuid,memory.used,utilization.gpu", + "--format=csv,noheader,nounits", + ] + ) + rows = [] + for line in text.strip().splitlines(): + index, uuid, memory, util = [part.strip() for part in line.split(",")] + rows.append( + { + "index": int(index), + "uuid": uuid, + "memory_used_mib": int(memory), + "utilization_pct": int(util), + } + ) + return rows + + +def compute_apps() -> list[dict[str, Any]]: + text = run_text( + [ + "nvidia-smi", + "--query-compute-apps=gpu_uuid,pid,process_name,used_memory", + "--format=csv,noheader,nounits", + ], + check=False, + ) + rows = [] + for line in text.strip().splitlines(): + if not line or "No running" in line: + continue + parts = [part.strip() for part in line.split(",", 3)] + if len(parts) == 4 and parts[1].isdigit(): + rows.append( + { + "gpu_uuid": parts[0], + "pid": int(parts[1]), + "process_name": parts[2], + "used_memory_mib": int(parts[3].split()[0]), + } + ) + return rows + + +def preflight(gpus: list[int], stage_dir: Path) -> None: + snapshots = [row for row in gpu_query() if row["index"] in gpus] + apps = compute_apps() + atomic_json(stage_dir / "gpu-before.json", snapshots) + (stage_dir / "clocks-before.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"]) + ) + (stage_dir / "loadavg-before.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + (stage_dir / "processes-before.txt").write_text( + run_text(["ps", "-eo", "user,pid,ppid,pgid,lstart,args", "--sort=pid"]) + ) + if apps or any(row["memory_used_mib"] != 0 for row in snapshots): + raise RuntimeError(f"selected GPUs not idle: snapshots={snapshots} apps={apps}") + if any(row["utilization_pct"] != 0 for row in snapshots): + raise RuntimeError(f"selected GPUs have nonzero utilization: {snapshots}") + + +class Monitor: + def __init__(self, path: Path, owned_pgids: set[int]) -> None: + self.path = path + self.owned_pgids = owned_pgids + self.stop_event = threading.Event() + self.other_apps: list[dict[str, Any]] = [] + self.thread = threading.Thread(target=self._run, daemon=True) + + def start(self) -> None: + self.thread.start() + + def stop(self) -> None: + self.stop_event.set() + self.thread.join(timeout=20) + + def _run(self) -> None: + with self.path.open("a", encoding="utf-8") as f: + while not self.stop_event.is_set(): + apps = compute_apps() + samples = [] + for app in apps: + try: + pgid = os.getpgid(app["pid"]) + except ProcessLookupError: + pgid = None + sample = {**app, "pgid": pgid} + samples.append(sample) + if pgid not in self.owned_pgids: + self.other_apps.append(sample) + clocks = run_text( + [ + "nvidia-smi", + "--query-gpu=index,clocks.sm,clocks.mem,pstate," + "utilization.gpu,memory.used", + "--format=csv,noheader,nounits", + ], + check=False, + ).strip().splitlines() + row = { + "wall_time": time.time(), + "loadavg": os.getloadavg(), + "gpu_clocks": clocks, + "compute_apps": samples, + } + f.write(json.dumps(row, sort_keys=True) + "\n") + f.flush() + self.stop_event.wait(15) + + +def server_command(gpu: int, stage_dir: Path, trace_dir: Path) -> list[str]: + config = {**PROFILE_CONFIG, "torch_profiler_dir": str(trace_dir)} + return [ + "taskset", + "-c", + CPU_MAP[gpu], + str(VENV / "bin/vllm"), + "serve", + str(MODEL), + "--host", + "127.0.0.1", + "--port", + str(8000 + gpu), + "--tensor-parallel-size", + "1", + "--enable-chunked-prefill", + "--enable-prefix-caching", + "--profiler-config", + json.dumps(config, separators=(",", ":")), + ] + + +def start_server( + gpu: int, stage_dir: Path, state_stage: dict[str, Any] +) -> tuple[subprocess.Popen[Any], Any, Path]: + gpu_dir = stage_dir / f"gpu{gpu}" + gpu_dir.mkdir(parents=True, exist_ok=True) + trace_dir = Path(f"/tmp/wjh-opprof-phase3-ea/{stage_dir.name}/gpu{gpu}") + if trace_dir.exists(): + shutil.rmtree(trace_dir) + trace_dir.mkdir(parents=True) + command = server_command(gpu, stage_dir, trace_dir) + command_log( + stage_dir / "commands.log", + f"{stage_dir.name} server gpu{gpu}", + command, + "startup 60-180s; run 300-450s", + ) + env = os.environ.copy() + env.update( + { + "CUDA_VISIBLE_DEVICES": str(gpu), + "VLLM_OPPROF_DIR": str(gpu_dir / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + log_handle = (gpu_dir / "server.log").open("ab", buffering=0) + process = subprocess.Popen( + command, + cwd=SOURCE, + env=env, + stdout=log_handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + state_stage.setdefault("servers", {})[str(gpu)] = { + "pid": process.pid, + "pgid": process.pid, + "command": command, + "trace_dir": str(trace_dir), + } + return process, log_handle, trace_dir + + +def wait_ready(processes: dict[int, subprocess.Popen[Any]], timeout: float = 300) -> None: + deadline = time.monotonic() + timeout + pending = set(processes) + while pending and time.monotonic() < deadline: + for gpu in list(pending): + if processes[gpu].poll() is not None: + raise RuntimeError(f"server gpu{gpu} exited before ready") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{8000 + gpu}/health", timeout=1 + ) as response: + if response.status == 200: + pending.remove(gpu) + except Exception: + pass + time.sleep(1) + if pending: + raise TimeoutError(f"servers not ready within {timeout}s: {sorted(pending)}") + + +def client_command( + gpu: int, + pattern: str, + stage_dir: Path, + load_point: str, + saturation_result: Path | None, + profile: bool, + trace_dir: Path, + post_clean_seconds: int, +) -> list[str]: + result_dir = stage_dir / f"gpu{gpu}" / "client" + command = [ + "taskset", + "-c", + CPU_MAP[gpu], + str(VENV / "bin/python"), + str(CLIENT), + "run", + "--manifest", + str(PRIVATE / f"{pattern}.jsonl"), + "--base-url", + f"http://127.0.0.1:{8000 + gpu}", + "--model", + str(MODEL), + "--load-point", + load_point, + "--max-concurrency", + "256", + "--ignore-eos", + "--temperature", + "0", + "--warmup-seconds", + "60", + "--clean-segment-seconds", + "80", + "--num-clean-segments", + "3", + "--recovery-seconds", + "30", + "--drain-timeout-seconds", + "120", + "--workload-seed", + "20260712", + "--result-dir", + str(result_dir), + ] + if load_point == "saturation": + command += ["--request-rate", "inf"] + else: + assert saturation_result is not None + command += [ + "--saturation-result", + str(saturation_result), + "--rate-fraction", + "0.60", + ] + if profile: + command += [ + "--profile-after-clean", + "--num-profile-windows", + "1", + "--profile-warmup-iterations", + "2", + "--profile-active-iterations", + "8", + "--profile-trace-dir", + str(trace_dir), + "--profile-timeout-seconds", + "120", + ] + elif post_clean_seconds: + command += ["--post-clean-seconds", str(post_clean_seconds)] + return command + + +def stop_processes(processes: list[subprocess.Popen[Any]]) -> None: + live = [process for process in processes if process.poll() is None] + for process in live: + try: + os.killpg(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 60 + while time.monotonic() < deadline and any(p.poll() is None for p in live): + time.sleep(1) + for sig, wait_seconds in ((signal.SIGTERM, 20), (signal.SIGKILL, 5)): + remaining = [p for p in live if p.poll() is None] + if not remaining: + break + for process in remaining: + try: + os.killpg(process.pid, sig) + except ProcessLookupError: + pass + deadline = time.monotonic() + wait_seconds + while time.monotonic() < deadline and any( + p.poll() is None for p in remaining + ): + time.sleep(0.5) + + +def _process_group_alive(pgid: int) -> bool: + try: + os.killpg(pgid, 0) + return True + except ProcessLookupError: + return False + + +def stop_servers(processes: list[subprocess.Popen[Any]]) -> None: + """Let the API parent ask EngineCore to close before group escalation.""" + groups = [process.pid for process in processes] + for process in processes: + if process.poll() is None: + try: + # Group SIGINT also interrupts EngineCore and loses the OpProf + # footer. The API parent's shutdown path signals it cleanly. + os.kill(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 90 + while time.monotonic() < deadline and any( + _process_group_alive(pgid) for pgid in groups + ): + time.sleep(1) + for sig, wait_seconds in ((signal.SIGTERM, 20), (signal.SIGKILL, 5)): + remaining = [pgid for pgid in groups if _process_group_alive(pgid)] + if not remaining: + break + for pgid in remaining: + try: + os.killpg(pgid, sig) + except ProcessLookupError: + pass + deadline = time.monotonic() + wait_seconds + while time.monotonic() < deadline and any( + _process_group_alive(pgid) for pgid in remaining + ): + time.sleep(0.5) + + +def verify_idle(gpus: list[int], stage_dir: Path) -> None: + samples = [] + consecutive_zero = 0 + deadline = time.monotonic() + 120 + while time.monotonic() < deadline and consecutive_zero < 3: + rows = [row for row in gpu_query() if row["index"] in gpus] + apps = compute_apps() + samples.append({"gpus": rows, "compute_apps": apps, "time": time.time()}) + zero = not apps and all(row["memory_used_mib"] == 0 for row in rows) + consecutive_zero = consecutive_zero + 1 if zero else 0 + if consecutive_zero < 3: + time.sleep(5) + atomic_json(stage_dir / "gpu-after-cleanup.json", samples) + if consecutive_zero < 3: + raise RuntimeError("GPU memory did not reach three stable zero samples") + + +def run_stage( + state: dict[str, Any], + name: str, + backgrounds: dict[int, str], + load_point: str, + saturation_stage: str | None, + profile: bool, +) -> None: + stage_dir = RUN_ROOT / name + stage_record = state["stages"].get(name) + if ( + stage_record + and stage_record.get("status") == "complete" + and (stage_dir / "stage-complete.json").exists() + ): + print(f"RESUME skip complete stage {name}", flush=True) + return + if stage_dir.exists(): + shutil.rmtree(stage_dir) + stage_dir.mkdir(parents=True) + gpus = [0, *sorted(backgrounds)] + state["stages"][name] = { + "status": "preflight", + "started_at": time.time(), + "gpus": gpus, + "backgrounds": backgrounds, + "load_point": load_point, + "profile": profile, + } + save_state(state) + preflight(gpus, stage_dir) + servers: dict[int, subprocess.Popen[Any]] = {} + server_logs = [] + client_processes: dict[int, subprocess.Popen[Any]] = {} + client_logs = [] + trace_dirs: dict[int, Path] = {} + monitor: Monitor | None = None + failure: Exception | None = None + try: + for gpu in gpus: + process, handle, trace_dir = start_server( + gpu, stage_dir, state["stages"][name] + ) + servers[gpu] = process + server_logs.append(handle) + trace_dirs[gpu] = trace_dir + state["stages"][name]["status"] = "starting_servers" + save_state(state) + wait_ready(servers) + state["stages"][name]["servers_ready_at"] = time.time() + state["stages"][name]["status"] = "running_clients" + save_state(state) + monitor = Monitor( + stage_dir / "monitor.jsonl", {process.pid for process in servers.values()} + ) + monitor.start() + + assignments = {0: "P05", **backgrounds} + for gpu, pattern in assignments.items(): + is_target = gpu == 0 + sat_result = ( + RUN_ROOT / saturation_stage / "gpu0/client/result.json" + if is_target and saturation_stage + else None + ) + target_load = load_point if is_target else "saturation" + command = client_command( + gpu, + pattern, + stage_dir, + target_load, + sat_result, + profile=is_target and profile, + trace_dir=trace_dirs[gpu], + post_clean_seconds=60 if not is_target and profile else 0, + ) + command_log( + stage_dir / "commands.log", + f"{name} client gpu{gpu} {pattern}", + command, + "60s warmup + 240s clean + optional 2+8 profile/recovery", + ) + handle = (stage_dir / f"gpu{gpu}/client.log").open("ab", buffering=0) + client_logs.append(handle) + process = subprocess.Popen( + command, + cwd=WORKDIR, + stdout=handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + client_processes[gpu] = process + state["stages"][name].setdefault("clients", {})[str(gpu)] = { + "pid": process.pid, + "pgid": process.pid, + "command": command, + } + save_state(state) + + deadline = time.monotonic() + 780 + while time.monotonic() < deadline and any( + process.poll() is None for process in client_processes.values() + ): + if any(process.poll() is not None for process in servers.values()): + raise RuntimeError("server exited while client was active") + time.sleep(2) + if any(process.poll() is None for process in client_processes.values()): + raise TimeoutError("client stage exceeded 780 seconds") + bad = { + gpu: process.returncode + for gpu, process in client_processes.items() + if process.returncode != 0 + } + if bad: + raise RuntimeError(f"client failures: {bad}") + target_result = stage_dir / "gpu0/client/result.json" + if not target_result.exists(): + raise RuntimeError("target result.json missing") + if profile: + trace_dest = stage_dir / "gpu0/traces" + trace_dest.mkdir() + for path in trace_dirs[0].glob("*"): + if path.is_file(): + shutil.copy2(path, trace_dest / path.name) + traces = list(trace_dest.glob("*.pt.trace.json*")) + if len(traces) != 1: + raise RuntimeError(f"expected one target trace, found {len(traces)}") + state["stages"][name]["status"] = "clients_complete" + save_state(state) + except Exception as exc: + failure = exc + finally: + if monitor is not None: + monitor.stop() + stop_processes(list(client_processes.values())) + stop_servers(list(servers.values())) + for handle in client_logs + server_logs: + handle.close() + (stage_dir / "clocks-after.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"], check=False) + ) + (stage_dir / "loadavg-after.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + (stage_dir / "processes-after.txt").write_text( + run_text( + ["ps", "-eo", "user,pid,ppid,pgid,lstart,args", "--sort=pid"], + check=False, + ) + ) + if monitor is not None: + atomic_json(stage_dir / "other-gpu-processes.json", monitor.other_apps) + try: + verify_idle(gpus, stage_dir) + except Exception as exc: + failure = failure or exc + + if failure is not None: + state["stages"][name]["status"] = "failed" + state["stages"][name]["failure"] = repr(failure) + save_state(state) + raise failure + marker = { + "schema": SCHEMA, + "stage": name, + "completed_at": time.time(), + "target_result_sha256": sha256_file( + stage_dir / "gpu0/client/result.json" + ), + } + atomic_json(stage_dir / "stage-complete.json", marker) + state["stages"][name].update( + {"status": "complete", "completed_at": time.time(), "servers": {}, + "clients": {}} + ) + save_state(state) + + +def classify_kernel(name: str) -> str: + value = name.lower() + if re.search(r"topkgating|moe_align|select_expert|moe.*rout", value): + return "moe_router" + if re.search( + r"nccl|all_?reduce|reduce_scatter|all_?gather|custom.*all.*reduce", value + ): + return "collective" + if re.search( + r"flash|fmha|paged_attention|unified_attention|attention|" + r"reshape_and_cache", value + ): + return "attention" + if re.search(r"fused_moe|moe.*gemm|nvjet", value): + return "moe_gemm" + if re.search( + r"sampling|(? dict[str, Any]: + traces = list((RUN_ROOT / stage / "gpu0/traces").glob("*.pt.trace.json*")) + if len(traces) != 1: + raise RuntimeError(f"{stage}: expected one trace") + trace = traces[0] + opener = gzip.open if trace.suffix == ".gz" else open + with opener(trace, "rt", encoding="utf-8") as f: + data = json.load(f) + durations: defaultdict[str, float] = defaultdict(float) + kernel_count = 0 + steps = set() + executes = 0 + for event in data.get("traceEvents", []): + name = str(event.get("name", "")) + match = re.fullmatch(r"ProfilerStep#(\d+)", name) + if match: + steps.add(int(match.group(1))) + if name.startswith("execute_") and event.get("cat") == "user_annotation": + executes += 1 + if event.get("cat") == "kernel" and float(event.get("dur", 0)) >= 0: + kernel_count += 1 + durations[classify_kernel(name)] += float(event.get("dur", 0)) + total = sum(durations.values()) + shares = { + family: duration / total for family, duration in sorted(durations.items()) + } + result = { + "schema": SCHEMA, + "stage": stage, + "trace_file": trace.name, + "trace_sha256": sha256_file(trace), + "trace_loadable": True, + "kernel_count": kernel_count, + "profiler_steps": sorted(steps), + "execute_annotations": executes, + "duration_us": dict(sorted(durations.items())), + "shares": shares, + "classifiable_fraction": 1.0 - shares.get("other", 0.0), + "valid": ( + kernel_count > 0 + and steps == set(range(2, 10)) + and executes == 8 + and 1.0 - shares.get("other", 0.0) >= 0.70 + ), + } + return result + + +def compare_regime(prefix: str) -> dict[str, Any]: + solo_result = json.loads( + (RUN_ROOT / "solo-moderate/gpu0/client/result.json").read_text() + ) + coloc_result = json.loads( + (RUN_ROOT / f"{prefix}-moderate/gpu0/client/result.json").read_text() + ) + solo_trace = trace_analysis("solo-moderate") + coloc_trace = trace_analysis(f"{prefix}-moderate") + solo_t = float(solo_result["clean"]["completed_throughput_rps"]) + coloc_t = float(coloc_result["clean"]["completed_throughput_rps"]) + top3 = sorted( + solo_trace["shares"], key=solo_trace["shares"].get, reverse=True + )[:3] + share_deltas = { + family: abs( + coloc_trace["shares"].get(family, 0) + - solo_trace["shares"].get(family, 0) + ) + for family in top3 + } + throughput_delta = abs(coloc_t / solo_t - 1) + return { + "schema": SCHEMA, + "regime": prefix, + "solo_throughput_rps": solo_t, + "colocated_throughput_rps": coloc_t, + "throughput_delta_fraction": throughput_delta, + "top3_solo_families": top3, + "solo_operator_shares": { + family: solo_trace["shares"][family] for family in top3 + }, + "colocated_operator_shares": { + family: coloc_trace["shares"].get(family, 0) for family in top3 + }, + "operator_share_delta_fraction": share_deltas, + "solo_trace": solo_trace, + "colocated_trace": coloc_trace, + "pass": ( + solo_trace["valid"] + and coloc_trace["valid"] + and throughput_delta < 0.03 + and all(delta < 0.03 for delta in share_deltas.values()) + ), + } + + +def numeric_sanity(values: list[float]) -> dict[str, Any]: + return { + "n": len(values), + "finite_n": sum(math.isfinite(value) for value in values), + "missing_n": sum(not math.isfinite(value) for value in values), + "min": min(values) if values else None, + "max": max(values) if values else None, + "distinct_n": len(set(values)), + } + + +def write_analysis(comparisons: list[dict[str, Any]]) -> dict[str, Any]: + accepted = next((c["regime"] for c in comparisons if c["pass"]), None) + throughputs = [ + value + for c in comparisons + for value in (c["solo_throughput_rps"], c["colocated_throughput_rps"]) + ] + deltas = [ + delta + for c in comparisons + for delta in c["operator_share_delta_fraction"].values() + ] + result = { + "schema": SCHEMA, + "comparisons": comparisons, + "verdict": ( + f"{accepted.replace('coloc-', '')}-way" + if accepted is not None + else "STOP" + ), + "sanity": { + "throughput_rps": numeric_sanity(throughputs), + "throughput_delta": numeric_sanity( + [c["throughput_delta_fraction"] for c in comparisons] + ), + "operator_share_delta": numeric_sanity(deltas), + "invariants": { + "throughputs_positive": all(x > 0 for x in throughputs), + "share_deltas_in_0_1": all(0 <= x <= 1 for x in deltas), + "top3_exact": all( + len(c["top3_solo_families"]) == 3 for c in comparisons + ), + "traces_valid": all( + c["solo_trace"]["valid"] and c["colocated_trace"]["valid"] + for c in comparisons + ), + }, + }, + } + atomic_json(RUN_ROOT / "colocation-analysis.json", result) + return result + + +def clean_recorded_processes(state: dict[str, Any]) -> None: + for stage in state.get("stages", {}).values(): + if stage.get("status") == "complete": + continue + pgids = [ + entry.get("pgid") + for group in ("clients", "servers") + for entry in stage.get(group, {}).values() + if entry.get("pgid") + ] + for pgid in pgids: + try: + os.killpg(int(pgid), signal.SIGINT) + except ProcessLookupError: + pass + if pgids: + time.sleep(5) + + +def execute(resume: bool) -> None: + RUN_ROOT.mkdir(parents=True, exist_ok=True) + state = load_state(resume) + if resume: + clean_recorded_processes(state) + fingerprint = make_fingerprint() + if state["fingerprint"] and state["fingerprint"] != fingerprint: + raise RuntimeError("resume fingerprint differs from frozen execution") + state["fingerprint"] = fingerprint + state["status"] = "running" + save_state(state) + ensure_provenance() + ensure_manifests() + run_stage(state, "solo-saturation", {}, "saturation", None, False) + run_stage( + state, + "solo-moderate", + {}, + "moderate", + "solo-saturation", + True, + ) + run_stage(state, "coloc-8-saturation", BACKGROUND_8, "saturation", None, False) + run_stage( + state, + "coloc-8-moderate", + BACKGROUND_8, + "moderate", + "coloc-8-saturation", + True, + ) + comparisons = [compare_regime("coloc-8")] + analysis = write_analysis(comparisons) + if not analysis["comparisons"][0]["pass"]: + run_stage( + state, "coloc-4-saturation", BACKGROUND_4, "saturation", None, False + ) + run_stage( + state, + "coloc-4-moderate", + BACKGROUND_4, + "moderate", + "coloc-4-saturation", + True, + ) + comparisons.append(compare_regime("coloc-4")) + analysis = write_analysis(comparisons) + state["status"] = "complete" if analysis["verdict"] != "STOP" else "gate_failed" + state["verdict"] = analysis["verdict"] + state["completed_at"] = time.time() + save_state(state) + print(json.dumps(analysis, sort_keys=True), flush=True) + + +def main() -> None: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + run = sub.add_parser("run") + run.add_argument("--resume", action="store_true") + sub.add_parser("status") + sub.add_parser("analyze") + sub.add_parser("plan") + args = parser.parse_args() + if args.command == "run": + execute(args.resume) + elif args.command == "status": + print(STATE.read_text() if STATE.exists() else '{"status":"absent"}') + elif args.command == "analyze": + comparisons = [compare_regime("coloc-8")] + if (RUN_ROOT / "coloc-4-moderate/stage-complete.json").exists(): + comparisons.append(compare_regime("coloc-4")) + print(json.dumps(write_analysis(comparisons), sort_keys=True, indent=2)) + else: + print( + json.dumps( + { + "target": "P05/C00 GPU0", + "background_8": BACKGROUND_8, + "background_4": BACKGROUND_4, + "cpu_map": CPU_MAP, + "stages": [ + "solo-saturation", + "solo-moderate", + "coloc-8-saturation", + "coloc-8-moderate", + "conditional coloc-4 saturation/moderate", + ], + }, + sort_keys=True, + indent=2, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3-ea/provenance/test_phase3_tools.py b/runs/opprof-phase3-ea/provenance/test_phase3_tools.py new file mode 100644 index 0000000..9d5c7a9 --- /dev/null +++ b/runs/opprof-phase3-ea/provenance/test_phase3_tools.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python3 + +from __future__ import annotations + +import asyncio +import importlib.util +import json +import sys +import tempfile +import unittest +from unittest import mock +from pathlib import Path +from types import SimpleNamespace + +from aiohttp import web + + +HERE = Path(__file__).parent + + +def load_module(name: str, filename: str): + spec = importlib.util.spec_from_file_location(name, HERE / filename) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +client = load_module("phase3_client", "opprof_phase3_client.py") +controller = load_module("phase3_controller", "opprof_phase3_controller.py") + + +def rows(n: int, arrival: str = "steady") -> list[dict]: + return [ + { + "schema": 1, + "request_id": f"T-{i:05d}", + "pattern_id": "T", + "kind": "synthetic", + "input_tokens": 8, + "output_tokens": 2, + "arrival": arrival, + "token_seed": i + 1, + } + for i in range(n) + ] + + +class MockServer: + def __init__(self) -> None: + self.active = 0 + self.max_active = 0 + self.payloads = [] + self.runner = None + self.port = None + + async def completion(self, request): + payload = await request.json() + self.payloads.append(payload) + self.active += 1 + self.max_active = max(self.max_active, self.active) + await asyncio.sleep(0.01) + response = web.StreamResponse( + status=200, headers={"Content-Type": "text/event-stream"} + ) + await response.prepare(request) + await response.write( + b'data: {"choices":[{"text":"x"}],"usage":null}\n\n' + ) + usage = json.dumps( + { + "choices": [], + "usage": { + "prompt_tokens": len(payload["prompt"]), + "completion_tokens": payload["max_tokens"], + }, + } + ).encode() + await response.write(b"data: " + usage + b"\n\n") + await response.write(b"data: [DONE]\n\n") + await response.write_eof() + self.active -= 1 + return response + + async def start(self): + app = web.Application() + app.router.add_post("/v1/completions", self.completion) + self.runner = web.AppRunner(app) + await self.runner.setup() + site = web.TCPSite(self.runner, "127.0.0.1", 0) + await site.start() + self.port = site._server.sockets[0].getsockname()[1] + + async def stop(self): + await self.runner.cleanup() + + +def run_args( + manifest: Path, + result_dir: Path, + port: int, + load_point: str = "saturation", + saturation_result: Path | None = None, +): + return SimpleNamespace( + manifest=str(manifest), + base_url=f"http://127.0.0.1:{port}", + model="mock", + load_point=load_point, + request_rate="inf" if load_point == "saturation" else None, + saturation_result=None if saturation_result is None else str(saturation_result), + rate_fraction=0.60, + max_concurrency=3, + ignore_eos=True, + temperature=0, + warmup_seconds=0.04, + clean_segment_seconds=0.04, + num_clean_segments=3, + profile_after_clean=False, + num_profile_windows=0, + profile_warmup_iterations=2, + profile_active_iterations=8, + profile_trace_dir=None, + profile_timeout_seconds=1, + recovery_seconds=0, + post_clean_seconds=0, + drain_timeout_seconds=1, + workload_seed=20260712, + server_seed=20260712, + result_dir=str(result_dir), + ) + + +class Phase3ToolTests(unittest.TestCase): + def test_synthetic_prompt_exact_and_unique_early_prefix(self): + generated = [client.synthetic_prompt(row) for row in rows(200)] + self.assertTrue(all(len(value) == 8 for value in generated)) + self.assertEqual(len({tuple(value[:3]) for value in generated}), 200) + + def test_p05_manifest_has_exact_half_modes(self): + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "P05.jsonl" + args = SimpleNamespace( + id="P05", + kind="synthetic", + input_uniform=None, + input_fixed=None, + input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + output_fixed=64, + prefix="none", + arrival="steady", + num_requests=100, + workload_seed=20260712, + num_prefixes=0, + prefix_len=0, + suffix_fixed=0, + out=str(out), + ) + client.materialize(args) + manifest = client.load_manifest(out) + self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50) + self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50) + + def test_prefix_pool_balanced(self): + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "P08.jsonl" + args = SimpleNamespace( + id="P08", + kind="prefix-pool", + input_uniform=None, + input_fixed=None, + input_mixture=None, + output_fixed=512, + prefix="none", + arrival="burst:8", + num_requests=80, + workload_seed=20260712, + num_prefixes=8, + prefix_len=1024, + suffix_fixed=256, + out=str(out), + ) + client.materialize(args) + manifest = client.load_manifest(out) + counts = {i: 0 for i in range(8)} + for row in manifest: + counts[row["prefix_id"]] += 1 + self.assertEqual(row["input_tokens"], 1280) + self.assertEqual(set(counts.values()), {10}) + + def test_fixed_duration_saturation_and_redaction(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000)) + mock = MockServer() + await mock.start() + try: + result = await client.run_load( + run_args(manifest, root / "result", mock.port) + ) + finally: + await mock.stop() + self.assertEqual(result["max_in_flight"], 3) + self.assertEqual(mock.max_active, 3) + self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9) + self.assertEqual(len(result["segments"]), 3) + self.assertLessEqual(result["drain_seconds"], 1) + self.assertTrue( + all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads) + ) + text = (root / "result/requests.jsonl").read_text() + self.assertNotIn('"prompt":', text) + self.assertNotIn('"text":', text) + + asyncio.run(case()) + + def test_drain_timeout_is_a_hard_sanity_failure(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000)) + mock = MockServer() + await mock.start() + try: + args = run_args(manifest, root / "result", mock.port) + args.drain_timeout_seconds = 0 + with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"): + await client.run_load(args) + finally: + await mock.stop() + sanity = json.loads((root / "result/sanity.json").read_text()) + self.assertFalse(sanity["invariants"]["drain_within_timeout"]) + + asyncio.run(case()) + + def test_burst_schedule_is_eight_at_once(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000, "burst:8")) + sat = root / "sat.json" + client.atomic_json( + sat, {"clean": {"completed_throughput_rps": 100.0}} + ) + mock = MockServer() + await mock.start() + try: + args = run_args( + manifest, root / "result", mock.port, "moderate", sat + ) + args.warmup_seconds = 0 + args.clean_segment_seconds = 0.4 / 3 + result = await client.run_load(args) + finally: + await mock.stop() + records = [ + json.loads(line) + for line in (root / "result/requests.jsonl").read_text().splitlines() + ] + groups = {} + for record in records: + groups.setdefault(round(record["scheduled_s"], 6), 0) + groups[round(record["scheduled_s"], 6)] += 1 + self.assertTrue(all(value == 8 for value in groups.values())) + starts = sorted(groups) + if len(starts) > 1: + self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5) + self.assertAlmostEqual(result["request_rate"], 60.0) + + asyncio.run(case()) + + def test_manifest_exhaustion_stops_instead_of_wrapping(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(2)) + mock = MockServer() + await mock.start() + try: + with self.assertRaises(client.ManifestExhausted): + await client.run_load( + run_args(manifest, root / "result", mock.port) + ) + finally: + await mock.stop() + + asyncio.run(case()) + + def test_cpu_affinity_sets_are_disjoint_and_cover_host(self): + values = [] + for spec in controller.CPU_MAP.values(): + lo, hi = [int(x) for x in spec.split("-")] + values.extend(range(lo, hi + 1)) + self.assertEqual(sorted(values), list(range(160))) + self.assertEqual(len(values), len(set(values))) + + def test_kernel_mapping_priority(self): + cases = { + "void vllm::moe::topkGating": "moe_router", + "ncclDevKernel_AllReduce": "collective", + "flash_fwd_kernel": "attention", + "nvjet_sm90_tst": "moe_gemm", + "argmax_kernel": "sampler", + "cutlass_gemm": "dense_gemm", + "triton_red_fused_add_rms_norm": "norm_elementwise", + "cache_swap_kernel": "kv_memory", + "unknown": "other", + } + self.assertEqual( + {name: controller.classify_kernel(name) for name in cases}, cases + ) + + def test_atomic_state_replacement(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "state.json" + controller.atomic_json(path, {"schema": 1, "value": 1}) + controller.atomic_json(path, {"schema": 1, "value": 2}) + self.assertEqual(json.loads(path.read_text())["value"], 2) + self.assertFalse(list(path.parent.glob("*.tmp.*"))) + + def test_fingerprint_survives_json_roundtrip(self): + original_run_text = controller.run_text + original_hash = controller.sha256_file + try: + controller.run_text = lambda *args, **kwargs: "deadbeef\n" + controller.sha256_file = lambda path: "a" * 64 + fingerprint = controller.make_fingerprint() + finally: + controller.run_text = original_run_text + controller.sha256_file = original_hash + self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint) + self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)}) + + def test_server_shutdown_signals_parent_before_group(self): + process = SimpleNamespace(pid=12345, poll=lambda: None) + with ( + mock.patch.object(controller.os, "kill") as kill, + mock.patch.object(controller.os, "killpg") as killpg, + mock.patch.object(controller, "_process_group_alive", return_value=False), + ): + controller.stop_servers([process]) + kill.assert_called_once_with(12345, controller.signal.SIGINT) + killpg.assert_not_called() + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/runs/opprof-phase3/phase3/access-blocker-20260712.json b/runs/opprof-phase3/phase3/access-blocker-20260712.json new file mode 100644 index 0000000..45016ef --- /dev/null +++ b/runs/opprof-phase3/phase3/access-blocker-20260712.json @@ -0,0 +1,47 @@ +{ + "schema": 1, + "status": "external_access_blocked", + "host": "dash0", + "last_reachability_probe_local": "2026-07-12T16:16:39+08:00", + "failure": "SSH connection timed out during banner exchange before any remote command executed", + "last_durable_remote_state": { + "observed_before_outage": true, + "controller_pid": 2237019, + "controller_status": "running", + "active_stage": "primary-02-saturation", + "active_stage_phase": "starting_servers", + "completed_measured_runs": 8, + "drain_quarantined_runs": 0, + "clean_window_failures": 0, + "missing_trace_files": 8, + "gpu_hours_total": 6.8156048206885655 + }, + "resume": { + "state": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/controller-state.json", + "log": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/phase3-matrix-controller.log", + "command": "cd /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712 && /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python scripts/opprof_phase3_matrix.py run --resume", + "rule": "inspect state/PID/GPU ownership first; do not relaunch a live controller" + }, + "profile_control_repair": { + "reason": "P03/C01 saturated all data-plane connector slots, so /start_profile timed out before reaching the server", + "change": "dedicated aiohttp TCPConnector(limit=2) and ClientSession for profile control calls", + "old_client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84", + "new_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0", + "remote_record": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/repairs/repair-004-profile-control-connector.json", + "tests": "13/13 unittest PASS; ruff PASS; py_compile PASS" + }, + "analysis_ready": { + "script_sha256": "205d9012d9462d84403a0f1435c8a449389aa4bef448d8d6dbced707a11559b0", + "test_sha256": "eac6f55042865c9a24811eef9c1a13c23a26dd1504b54348513d8e6a7ba2b939", + "tests": "4/4 unittest PASS; ruff PASS; py_compile PASS", + "executed_on_matrix": false + }, + "not_claimed": [ + "matrix completion", + "final quarantine count", + "H1a verdict", + "H1b verdict", + "final GPU hours", + "final GPU cleanup" + ] +} diff --git a/runs/opprof-phase3/phase3/analysis/analyze-ap37-final.log b/runs/opprof-phase3/phase3/analysis/analyze-ap37-final.log new file mode 100644 index 0000000..a64f2c1 --- /dev/null +++ b/runs/opprof-phase3/phase3/analysis/analyze-ap37-final.log @@ -0,0 +1 @@ +{"hypothesis": {"H1a": "INCONCLUSIVE", "H1b": "PASS", "compound": "PARTIAL", "h1b_hits": [{"available": true, "ci95_high": 0.23494534783397408, "ci95_low": 0.2253590527218157, "coincident_efficiency_or_residual": true, "control": "P02", "efficiency_loss": 0.11608997226921058, "evaluable": true, "irregular": "P06", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.2301483528866904, "simultaneous_high": 0.23683442206671917, "simultaneous_low": 0.22345003461028415}, {"available": true, "ci95_high": 0.3588237662068493, "ci95_low": 0.34996301147907066, "coincident_efficiency_or_residual": true, "control": "P04", "efficiency_loss": 0.2284629366493368, "evaluable": true, "irregular": "P06", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.3543970834375052, "simultaneous_high": 0.3605880124391376, "simultaneous_low": 0.34826154238421436}, {"available": true, "ci95_high": 0.07258101928545771, "ci95_low": 0.06107199918894719, "coincident_efficiency_or_residual": true, "control": "P01", "efficiency_loss": 0.08324816211784114, "evaluable": true, "irregular": "P09", "material": true, "metric": "padding_fraction", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.0667290414201105, "simultaneous_high": 0.07492451384312424, "simultaneous_low": 0.058870086990988876}, {"available": true, "ci95_high": 0.4001054362524198, "ci95_low": 0.3922157126916775, "coincident_efficiency_or_residual": true, "control": "P01", "efficiency_loss": 0.08324816211784114, "evaluable": true, "irregular": "P09", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.39616418510901036, "simultaneous_high": 0.4016625621683738, "simultaneous_low": 0.3906536566040832}, {"available": true, "ci95_high": 0.08855777444708128, "ci95_low": 0.027415277773332525, "coincident_efficiency_or_residual": true, "control": "P03", "efficiency_loss": 0.4469310402722787, "evaluable": true, "irregular": "P10", "material": true, "metric": "padding_fraction", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.05565211993295673, "simultaneous_high": 0.10341073102256013, "simultaneous_low": 0.020097017796153225}, {"available": true, "ci95_high": 0.4561736305824604, "ci95_low": 0.43955479496868705, "coincident_efficiency_or_residual": true, "control": "P03", "efficiency_loss": 0.4469310402722787, "evaluable": true, "irregular": "P10", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.4478716685574937, "simultaneous_high": 0.459713780733712, "simultaneous_low": 0.43635435003093426}, {"available": true, "ci95_high": 0.08668052314167546, "ci95_low": 0.025823274229141942, "coincident_efficiency_or_residual": true, "control": "P04", "efficiency_loss": 0.1425966128421564, "evaluable": true, "irregular": "P10", "material": true, "metric": "padding_fraction", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.05398526806052646, "simultaneous_high": 0.10213016668480329, "simultaneous_low": 0.018683381445978635}, {"available": true, "ci95_high": 0.45618061702156487, "ci95_low": 0.4395710696329076, "coincident_efficiency_or_residual": true, "control": "P04", "efficiency_loss": 0.1425966128421564, "evaluable": true, "irregular": "P10", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.4478716685574937, "simultaneous_high": 0.4594884290583298, "simultaneous_low": 0.4363807227134572}], "logical_asymmetry": "A-P3-7 permits existential confirmation but incomplete coverage forbids refutation", "refutation_allowed": false}, "out": "runs/phase3/analysis/metrics-ap37.json", "sanity": {"declared_deviations": {"invalid_primary_layer2_windows": ["P01-C00-moderate: window 1", "P01-C00-moderate: window 2", "P02-C00-moderate: window 1", "P02-C00-moderate: window 2", "P03-C00-moderate: window 1", "P03-C00-moderate: window 2", "P06-C00-moderate: window 1", "P06-C00-moderate: window 2", "P07-C00-moderate: window 1", "P07-C00-moderate: window 2", "P08-C00-moderate: window 1", "P08-C00-moderate: window 2", "P09-C00-moderate: window 1", "P09-C00-moderate: window 2", "P10-C00-moderate: window 1", "P10-C00-moderate: window 2"], "missing_cells": ["P03/C11", "P05/C00", "P10/C00-TP2", "P11/C00"], "missing_confirmations": ["P10", "P06", "P03", "P01"], "missing_saturation_traces": 8, "moe_layer_cv": "N/A: layer scopes do not cover >=80% of MoE GEMM time", "unaccepted_canonical_run_markers_excluded": ["P11-C00-saturation"]}, "invariants": {"accepted_run_ids_unique": true, "ap36_operational_finding_reproduced": true, "ap37_confirmation_runs_0": true, "ap37_primary_runs_40": true, "clean_duration_240": true, "clean_failures_zero": true, "clock_and_load_snapshots_complete": true, "complete_cells_20": true, "complete_stage_count_12": true, "completed_c00_patterns_9": true, "controller_frozen_at_ap36_failure": true, "drain_quarantine_under_20pct": true, "h1b_evaluable_contrasts_6": true, "h1b_missing_contrasts_exact": true, "layer1_footer_invariants": true, "missing_cells_exact": true, "missing_trace_count_accepted_8": true, "moderate_rate_within_5pct": true, "no_layer2_parser_issues": true, "other_gpu_processes_absent": true, "patterns_not_all_identical_throughput": true, "ratios_in_unit_interval": true, "trace_count_accepted_72": true}, "numeric": {"completed_throughput_rps": {"distinct_n": 37, "finite_n": 40, "max": 43.3, "min": 0.44583333333333336, "missing_n": 0, "n": 40}, "drain_seconds": {"distinct_n": 40, "finite_n": 40, "max": 288.6191079240234, "min": 0.3054194079886656, "missing_n": 0, "n": 40}, "kendall_tau_b": {"distinct_n": 0, "finite_n": 0, "max": null, "min": null, "missing_n": 0, "n": 0}, "layer1_clean_steps": {"distinct_n": 40, "finite_n": 40, "max": 20171.0, "min": 718.0, "missing_n": 0, "n": 40}, "operator_classifiable_fraction": {"distinct_n": 72, "finite_n": 72, "max": 0.9963781377235988, "min": 0.9704584620048622, "missing_n": 0, "n": 72}, "operator_share": {"distinct_n": 348, "finite_n": 576, "max": 0.8734165832313857, "min": 0.0, "missing_n": 0, "n": 576}, "token_efficiency_per_ms": {"distinct_n": 40, "finite_n": 40, "max": 8.29584869763246, "min": 2.147827352932598, "missing_n": 0, "n": 40}, "waste_contrast_effect": {"distinct_n": 17, "finite_n": 24, "max": 0.5204129156598252, "min": -0.20308199280724987, "missing_n": 12, "n": 36}, "waste_ratios": {"distinct_n": 168, "finite_n": 240, "max": 1.0, "min": 0.0, "missing_n": 0, "n": 240}}}} diff --git a/runs/opprof-phase3/phase3/analysis/analyze-ap37.log b/runs/opprof-phase3/phase3/analysis/analyze-ap37.log new file mode 100644 index 0000000..629925f --- /dev/null +++ b/runs/opprof-phase3/phase3/analysis/analyze-ap37.log @@ -0,0 +1 @@ +{"hypothesis": {"H1a": "INCONCLUSIVE", "H1b": "PASS", "compound": "PARTIAL", "h1b_hits": [{"available": true, "ci95_high": 0.23494534783397408, "ci95_low": 0.2253590527218157, "coincident_efficiency_or_residual": true, "control": "P02", "efficiency_loss": 0.11608997226921058, "evaluable": true, "irregular": "P06", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.2301483528866904, "simultaneous_high": 0.23683442206671917, "simultaneous_low": 0.22345003461028415}, {"available": true, "ci95_high": 0.3588237662068493, "ci95_low": 0.34996301147907066, "coincident_efficiency_or_residual": true, "control": "P04", "efficiency_loss": 0.2284629366493368, "evaluable": true, "irregular": "P06", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.3543970834375052, "simultaneous_high": 0.3605880124391376, "simultaneous_low": 0.34826154238421436}, {"available": true, "ci95_high": 0.07258101928545771, "ci95_low": 0.06107199918894719, "coincident_efficiency_or_residual": true, "control": "P01", "efficiency_loss": 0.08324816211784114, "evaluable": true, "irregular": "P09", "material": true, "metric": "padding_fraction", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.0667290414201105, "simultaneous_high": 0.07492451384312424, "simultaneous_low": 0.058870086990988876}, {"available": true, "ci95_high": 0.4001054362524198, "ci95_low": 0.3922157126916775, "coincident_efficiency_or_residual": true, "control": "P01", "efficiency_loss": 0.08324816211784114, "evaluable": true, "irregular": "P09", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.39616418510901036, "simultaneous_high": 0.4016625621683738, "simultaneous_low": 0.3906536566040832}, {"available": true, "ci95_high": 0.08855777444708128, "ci95_low": 0.027415277773332525, "coincident_efficiency_or_residual": true, "control": "P03", "efficiency_loss": 0.4469310402722787, "evaluable": true, "irregular": "P10", "material": true, "metric": "padding_fraction", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.05565211993295673, "simultaneous_high": 0.10341073102256013, "simultaneous_low": 0.020097017796153225}, {"available": true, "ci95_high": 0.4561736305824604, "ci95_low": 0.43955479496868705, "coincident_efficiency_or_residual": true, "control": "P03", "efficiency_loss": 0.4469310402722787, "evaluable": true, "irregular": "P10", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.4478716685574937, "simultaneous_high": 0.459713780733712, "simultaneous_low": 0.43635435003093426}, {"available": true, "ci95_high": 0.08668052314167546, "ci95_low": 0.025823274229141942, "coincident_efficiency_or_residual": true, "control": "P04", "efficiency_loss": 0.1425966128421564, "evaluable": true, "irregular": "P10", "material": true, "metric": "padding_fraction", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.05398526806052646, "simultaneous_high": 0.10213016668480329, "simultaneous_low": 0.018683381445978635}, {"available": true, "ci95_high": 0.45618061702156487, "ci95_low": 0.4395710696329076, "coincident_efficiency_or_residual": true, "control": "P04", "efficiency_loss": 0.1425966128421564, "evaluable": true, "irregular": "P10", "material": true, "metric": "R64", "p": 0.0, "p_holm": 0.0, "passes_h1b": true, "point": 0.4478716685574937, "simultaneous_high": 0.4594884290583298, "simultaneous_low": 0.4363807227134572}], "logical_asymmetry": "A-P3-7 permits existential confirmation but incomplete coverage forbids refutation", "refutation_allowed": false}, "out": "runs/phase3/analysis/metrics-ap37.json", "sanity": {"declared_deviations": {"invalid_primary_layer2_windows": ["P01-C00-moderate: window 1", "P01-C00-moderate: window 2", "P02-C00-moderate: window 1", "P02-C00-moderate: window 2", "P03-C00-moderate: window 1", "P03-C00-moderate: window 2", "P06-C00-moderate: window 1", "P06-C00-moderate: window 2", "P07-C00-moderate: window 1", "P07-C00-moderate: window 2", "P08-C00-moderate: window 1", "P08-C00-moderate: window 2", "P09-C00-moderate: window 1", "P09-C00-moderate: window 2", "P10-C00-moderate: window 1", "P10-C00-moderate: window 2"], "missing_cells": ["P03/C11", "P05/C00", "P10/C00-TP2", "P11/C00"], "missing_confirmations": ["P10", "P06", "P03", "P01"], "missing_saturation_traces": 8, "moe_layer_cv": "N/A: layer scopes do not cover >=80% of MoE GEMM time", "unaccepted_canonical_run_markers_excluded": ["P11-C00-saturation"]}, "invariants": {"accepted_run_ids_unique": true, "ap37_confirmation_runs_0": true, "ap37_primary_runs_40": true, "clean_duration_240": true, "clean_failures_zero": true, "clock_and_load_snapshots_complete": true, "complete_cells_20": true, "complete_stage_count_12": true, "completed_c00_patterns_9": true, "controller_frozen_at_ap36_failure": true, "drain_quarantine_under_20pct": true, "h1b_evaluable_contrasts_6": true, "h1b_missing_contrasts_exact": true, "layer1_footer_invariants": true, "missing_cells_exact": true, "missing_trace_count_accepted_8": true, "moderate_rate_within_5pct": true, "no_layer2_parser_issues": true, "other_gpu_processes_absent": true, "patterns_not_all_identical_throughput": true, "ratios_in_unit_interval": true, "trace_count_accepted_72": true}, "numeric": {"completed_throughput_rps": {"distinct_n": 37, "finite_n": 40, "max": 43.3, "min": 0.44583333333333336, "missing_n": 0, "n": 40}, "drain_seconds": {"distinct_n": 40, "finite_n": 40, "max": 288.6191079240234, "min": 0.3054194079886656, "missing_n": 0, "n": 40}, "kendall_tau_b": {"distinct_n": 0, "finite_n": 0, "max": null, "min": null, "missing_n": 0, "n": 0}, "layer1_clean_steps": {"distinct_n": 40, "finite_n": 40, "max": 20171.0, "min": 718.0, "missing_n": 0, "n": 40}, "operator_classifiable_fraction": {"distinct_n": 72, "finite_n": 72, "max": 0.9963781377235988, "min": 0.9704584620048622, "missing_n": 0, "n": 72}, "operator_share": {"distinct_n": 348, "finite_n": 576, "max": 0.8734165832313857, "min": 0.0, "missing_n": 0, "n": 576}, "token_efficiency_per_ms": {"distinct_n": 40, "finite_n": 40, "max": 8.29584869763246, "min": 2.147827352932598, "missing_n": 0, "n": 40}, "waste_contrast_effect": {"distinct_n": 17, "finite_n": 24, "max": 0.5204129156598252, "min": -0.20308199280724987, "missing_n": 12, "n": 36}, "waste_ratios": {"distinct_n": 168, "finite_n": 240, "max": 1.0, "min": 0.0, "missing_n": 0, "n": 240}}}} diff --git a/runs/opprof-phase3/phase3/analysis/launch-ap37.log b/runs/opprof-phase3/phase3/analysis/launch-ap37.log new file mode 100644 index 0000000..15755f6 --- /dev/null +++ b/runs/opprof-phase3/phase3/analysis/launch-ap37.log @@ -0,0 +1 @@ +CPU_ANALYSIS A-P3-7: accepted_runs=40, run_root=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3, private_manifests=/home/admin/cpfs/wjh/opprof-phase3-private/manifests, bootstrap=100000 seed=20260714, output=runs/phase3/analysis/metrics-ap37.json, expected=10-30m CPU-only, GPU_cost=0 diff --git a/runs/opprof-phase3/phase3/controller-state-stop-ap36.json b/runs/opprof-phase3/phase3/controller-state-stop-ap36.json new file mode 100644 index 0000000..6945ca9 --- /dev/null +++ b/runs/opprof-phase3/phase3/controller-state-stop-ap36.json @@ -0,0 +1,737 @@ +{ + "clean_window_failures": 0, + "completed_burnins": 5, + "completed_measured_runs": 40, + "controller_pid": 2438791, + "created_at": 1783833885.960389, + "drain_quarantined_runs": 0, + "fingerprint": { + "cells": [ + "P08-C00", + "P03-C10", + "P07-C00", + "P10-C01", + "P01-C10", + "P01-C01", + "P10-C10", + "P03-C01", + "P09-C00", + "P06-C01", + "P10-C11", + "P01-C00", + "P01-C11", + "P10-C00", + "P04-C00", + "P03-C00", + "P06-C10", + "P02-C00", + "P06-C00", + "P06-C11", + "P11-C00", + "P10-C00-TP2", + "P03-C11", + "P05-C00" + ], + "client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0", + "common_controller_sha256": "95f7169a1771e385aab40fcaecd967dc5cff0c21ea67bdd139382774ba43f01f", + "controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f", + "cpu_map": { + "0": "0-19", + "1": "20-39", + "2": "40-59", + "3": "60-79", + "4": "80-99", + "5": "100-119", + "6": "120-139", + "7": "140-159" + }, + "manifests": { + "P01": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl", + "rows": 32768, + "sha256": "13ffb226c83373f54c4a7afea6c78cb7cd29720f1858d56728826fc1367b31a4" + }, + "P02": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P02.jsonl", + "rows": 32768, + "sha256": "0138ada3fccc98298daee66c26bd1952c987cb42ed8d5341d66b698a597417f9" + }, + "P03": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P03.jsonl", + "rows": 32768, + "sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1" + }, + "P04": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P04.jsonl", + "rows": 32768, + "sha256": "caf8d0941b093956a81e1413adc4a4ea9d92460f1b2ed0f6a9b118d0119d6247" + }, + "P05": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P05.jsonl", + "rows": 32768, + "sha256": "192e213109f8cb429b99d9eb0f227bb4390fc03f63b02d5456569369bff5a3d7" + }, + "P06": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P06.jsonl", + "rows": 32768, + "sha256": "65954bc6e47de9e7be07b8975f97f0bc4639979ef6d33e8c664557ded34b9f96" + }, + "P07": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P07.jsonl", + "rows": 32768, + "sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6" + }, + "P08": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P08.jsonl", + "rows": 32768, + "sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9" + }, + "P09": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", + "rows": 32768, + "sha256": "7af92ee3c27dc7d2cf895d6ff3a6e737ec4b6da13d6841ca59e1166f28a0ae1e" + }, + "P10": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl", + "rows": 4011, + "sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33" + }, + "P11": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P11.jsonl", + "rows": 32768, + "sha256": "7d196df38963528ff181cf72ce39c8ad913c8f61d40b1425410d3c6c30b6be18" + } + }, + "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B", + "source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05", + "source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1" + }, + "gpu_hours_this_stage": 0.7173924536837472, + "gpu_hours_total": 14.025875418755744, + "missing_trace_files": 8, + "repairs": [ + { + "change": "dedicated aiohttp TCPConnector(limit=2) for profile endpoint session; request stream unchanged", + "evidence": "/start_profile connection acquisition timed out before server receipt while 256 data-plane connections were occupied", + "failed_run": "P03-C01-saturation", + "failed_stage": "primary-02-saturation", + "new_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0", + "old_client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84", + "repair": "profile-control-connector-isolation", + "retry": "one exact whole-wave retry; failed directories retained as .interrupted-*", + "schema": 1, + "tests": { + "py_compile": "PASS", + "ruff": "PASS", + "unittest": "13/13 PASS" + }, + "timestamp": 1783837763.431071 + }, + { + "budget": { + "hard_cap": 16.0, + "projected_remaining_h20_hours": 6.3857560787267165, + "projected_total_h20_hours": 14.513826778670154, + "projected_total_with_15pct_remaining_contingency": 15.471690190479162, + "used_h20_hours": 8.128070699943438 + }, + "cause": "controller required failed_records==0 across excluded profile/recovery time although the registered hard gate is zero clean-window failures", + "change": "replace all_failures_zero with clean-window boundary validation plus failed-record accounting; excluded failures remain reported", + "evidence": { + "clean_completed": 5828, + "clean_failed": 0, + "error_kind": "ServerDisconnectedError", + "excluded_window_failures": 2, + "failure_completion_s": [ + 373.1505718010012, + 373.9740898209857 + ], + "run": "P01-C01-moderate" + }, + "failed_stage": "primary-02-moderate", + "gpu_rerun": false, + "new_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f", + "old_controller_sha256": "167e48f98f307e16ee44b321068a82a13813b3bf9a4d76882f774f15fe85e595", + "repair": "clean-window-failure-scope", + "schema": 1, + "tests": { + "py_compile": "PASS", + "ruff": "PASS", + "unittest": "14/14 PASS" + }, + "timestamp": 1783847092.6920242 + }, + { + "cause": "single preflight NVML sample observed 4 MiB on GPU0-3 with 0% utilization and no compute apps immediately after a verified cleanup", + "change": "bounded 60s preflight requiring three consecutive samples with zero memory, zero utilization, and no compute apps; no nonzero-memory tolerance", + "failed_stage": "primary-06-saturation", + "gpu_hours_total_unchanged": 11.920448313262728, + "gpu_launch_before_failure": false, + "new_common_controller_sha256": "95f7169a1771e385aab40fcaecd967dc5cff0c21ea67bdd139382774ba43f01f", + "old_common_controller_sha256": "15ad254298a38c4a9318468db89ab32707b6196bb38d5e2a007f2267529397a5", + "repair": "stable-zero-preflight", + "schema": 1, + "tests": { + "py_compile": "PASS", + "ruff": "PASS", + "unittest": "15/15 PASS" + }, + "timestamp": 1783850898.1496143 + }, + { + "code_change": false, + "gpu_hours_used": 12.606695837948058, + "hard_cap": 16.0, + "projected_remaining_h20_hours": 2.1455133807990285, + "projected_total_with_15pct_contingency": 15.07403622586694, + "protocol_action": "single exact whole-wave infrastructure retry under identical 4-way placement; gate unchanged", + "reason": "first TP2 P10 inference-shape cold start/autotune after readiness yielded 15 successful warmup completions against required 32; clean window and artifacts otherwise valid", + "repair": "tp2-warmup-exact-retry", + "run": "P10-C00-TP2-saturation", + "schema": 1, + "stage": "primary-06-saturation", + "timestamp": 1783851710.3100822, + "warmup_required": 32, + "warmup_success": 15 + }, + { + "accepted_runs_preserved": 40, + "amendment": "A-P3-6", + "changed_fingerprint_keys": [ + "controller_sha256" + ], + "created_at": 1783853258.3098204, + "failed_stage_preserved": "primary-06-saturation", + "new_controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f", + "old_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f", + "projected_remaining_h20_hours": 2.1, + "projected_total_h20_hours": 15.408482965071997, + "repair_id": "repair-008-ap36", + "retained_attempt_re_adjudicated": false, + "retained_attempt_reason": "21 completions but A-P3-6 normalized drift 2.3385345997286295 and bin step counts 13/9/44", + "schema": 1 + } + ], + "schema": 1, + "stages": { + "burnin-01": { + "assignments": [ + { + "cell": "P06-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P06-C10", + "gpus": [ + 1 + ] + }, + { + "cell": "P06-C01", + "gpus": [ + 2 + ] + }, + { + "cell": "P06-C11", + "gpus": [ + 3 + ] + } + ], + "burnin": true, + "clients": {}, + "completed_at": 1783834288.1238313, + "confirmation": false, + "gpu_hours": 0.30808498481909435, + "load_point": "saturation", + "profile": false, + "servers": {}, + "started_at": 1783833886.4448946, + "status": "complete", + "validation_attempt1_failure": "RuntimeError(\"server config/backend failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/burnins/C01: {'triton_moe': True, 'chunked_mbt': False, 'tp_effective': True, 'drain_shutdown': True, 'mns_effective': True}\")" + }, + "burnin-02": { + "assignments": [ + { + "cell": "P06-C00-TP2", + "gpus": [ + 0, + 1 + ] + } + ], + "burnin": true, + "clients": {}, + "completed_at": 1783834671.921921, + "confirmation": false, + "gpu_hours": 0.19903395679261948, + "load_point": "saturation", + "profile": false, + "servers": {}, + "started_at": 1783834301.7813473, + "status": "complete" + }, + "primary-01-moderate": { + "assignments": [ + { + "cell": "P08-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P03-C10", + "gpus": [ + 1 + ] + }, + { + "cell": "P07-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P10-C01", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783836723.3639715, + "confirmation": false, + "gpu_hours": 0.5195140059126748, + "load_point": "moderate", + "profile": true, + "servers": {}, + "started_at": 1783836150.2169476, + "status": "complete", + "validation_attempt1_failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P10-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")", + "warmup_feasibility": "P10 target=min(32,floor(0.445*60))=26; observed=27" + }, + "primary-01-saturation": { + "assignments": [ + { + "cell": "P08-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P03-C10", + "gpus": [ + 1 + ] + }, + { + "cell": "P07-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P10-C01", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783836130.8498049, + "confirmation": false, + "drain_readjudication": "P10/C01 288.619107924s <= 600s", + "gpu_hours": 0.8836704309119119, + "load_point": "saturation", + "missing_trace_files": 8, + "profile": true, + "servers": {}, + "started_at": 1783834671.9563339, + "status": "complete", + "validation_attempt1_failure": "RuntimeError(\"client failures: {'P10-C01-saturation': 1}\")" + }, + "primary-02-moderate": { + "assignments": [ + { + "cell": "P01-C10", + "gpus": [ + 0 + ] + }, + { + "cell": "P01-C01", + "gpus": [ + 1 + ] + }, + { + "cell": "P10-C10", + "gpus": [ + 2 + ] + }, + { + "cell": "P03-C01", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783847092.7229948, + "confirmation": false, + "failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P01-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': False, 'manifest_no_wrap': True, 'warmup_completions': True, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")", + "gpu_hours": 0.5151156750652525, + "load_point": "moderate", + "profile": true, + "readjudicated_without_gpu": true, + "servers": {}, + "started_at": 1783838513.4472992, + "status": "complete", + "validation_attempt1_failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P01-C01/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'all_failures_zero': False, 'manifest_no_wrap': True, 'warmup_completions': True, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]\")" + }, + "primary-02-saturation": { + "assignments": [ + { + "cell": "P01-C10", + "gpus": [ + 0 + ] + }, + { + "cell": "P01-C01", + "gpus": [ + 1 + ] + }, + { + "cell": "P10-C10", + "gpus": [ + 2 + ] + }, + { + "cell": "P03-C01", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783838513.4027848, + "confirmation": false, + "gpu_hours": 0.7973502041896184, + "load_point": "saturation", + "profile": true, + "servers": {}, + "started_at": 1783837773.6088765, + "status": "complete" + }, + "primary-03-moderate": { + "assignments": [ + { + "cell": "P09-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P06-C01", + "gpus": [ + 1 + ] + }, + { + "cell": "P10-C11", + "gpus": [ + 2 + ] + }, + { + "cell": "P01-C00", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783848369.3507695, + "confirmation": false, + "gpu_hours": 0.5205460974905226, + "load_point": "moderate", + "profile": true, + "servers": {}, + "started_at": 1783847881.06146, + "status": "complete" + }, + "primary-03-saturation": { + "assignments": [ + { + "cell": "P09-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P06-C01", + "gpus": [ + 1 + ] + }, + { + "cell": "P10-C11", + "gpus": [ + 2 + ] + }, + { + "cell": "P01-C00", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783847881.0197804, + "confirmation": false, + "gpu_hours": 0.8273645816908942, + "load_point": "saturation", + "profile": true, + "servers": {}, + "started_at": 1783847109.5683346, + "status": "complete" + }, + "primary-04-moderate": { + "assignments": [ + { + "cell": "P01-C11", + "gpus": [ + 0 + ] + }, + { + "cell": "P10-C00", + "gpus": [ + 1 + ] + }, + { + "cell": "P04-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P03-C00", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783849583.5228572, + "confirmation": false, + "gpu_hours": 0.5205216948853598, + "load_point": "moderate", + "profile": true, + "servers": {}, + "started_at": 1783849096.4363403, + "status": "complete" + }, + "primary-04-saturation": { + "assignments": [ + { + "cell": "P01-C11", + "gpus": [ + 0 + ] + }, + { + "cell": "P10-C00", + "gpus": [ + 1 + ] + }, + { + "cell": "P04-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P03-C00", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783849096.3948114, + "confirmation": false, + "gpu_hours": 0.7872003297011058, + "load_point": "saturation", + "profile": true, + "servers": {}, + "started_at": 1783848369.390857, + "status": "complete" + }, + "primary-05-moderate": { + "assignments": [ + { + "cell": "P06-C10", + "gpus": [ + 0 + ] + }, + { + "cell": "P02-C00", + "gpus": [ + 1 + ] + }, + { + "cell": "P06-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P06-C11", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783850641.6781306, + "confirmation": false, + "gpu_hours": 0.5138198028008143, + "load_point": "moderate", + "profile": true, + "servers": {}, + "started_at": 1783850163.1211865, + "status": "complete" + }, + "primary-05-saturation": { + "assignments": [ + { + "cell": "P06-C10", + "gpus": [ + 0 + ] + }, + { + "cell": "P02-C00", + "gpus": [ + 1 + ] + }, + { + "cell": "P06-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P06-C11", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": {}, + "completed_at": 1783850163.0472832, + "confirmation": false, + "gpu_hours": 0.6229251067505942, + "load_point": "saturation", + "profile": true, + "servers": {}, + "started_at": 1783849583.5652869, + "status": "complete" + }, + "primary-06-saturation": { + "assignments": [ + { + "cell": "P11-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P10-C00-TP2", + "gpus": [ + 1, + 2 + ] + }, + { + "cell": "P03-C11", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": { + "P03-C11-saturation": { + "pgid": 2440965, + "pid": 2440965 + }, + "P10-C00-TP2-saturation": { + "pgid": 2440964, + "pid": 2440964 + }, + "P11-C00-saturation": { + "pgid": 2440962, + "pid": 2440962 + } + }, + "confirmation": false, + "failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P10-C00-TP2/saturation: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'failed_records_accounted': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]; warmup_completions=17; warmup_gate_branch=failed; warmup_stability={'passed': False, 'reason': 'A-P3-6 stabilization criterion not met', 'window_seconds': [45.0, 60.0], 'bin_seconds': 5.0, 'step_counts': [11, 10, 16], 'scheduled_tokens': [90112, 81920, 70313], 'scheduled_token_throughput': [18022.4, 16384.0, 14062.6], 'mean_scheduled_token_throughput': 16156.333333333334, 'slope_tokens_per_second_squared': -395.98000000000013, 'normalized_drift': 0.367639109533929, 'normalized_drift_limit': 0.1, 'step_indices_continuous': True}\")", + "gpu_hours": 0.7173924536837472, + "load_point": "saturation", + "profile": true, + "servers": { + "P03-C11-saturation": { + "gpus": [ + 3 + ], + "pgid": 2438868, + "pid": 2438868 + }, + "P10-C00-TP2-saturation": { + "gpus": [ + 1, + 2 + ], + "pgid": 2438867, + "pid": 2438867 + }, + "P11-C00-saturation": { + "gpus": [ + 0 + ], + "pgid": 2438866, + "pid": 2438866 + } + }, + "started_at": 1783853279.9695158, + "status": "failed" + } + }, + "status": "failed", + "updated_at": 1783853946.066117 +} diff --git a/runs/opprof-phase3/phase3/metrics.json b/runs/opprof-phase3/phase3/metrics.json new file mode 100644 index 0000000..c74e7e8 --- /dev/null +++ b/runs/opprof-phase3/phase3/metrics.json @@ -0,0 +1,39015 @@ +{ + "all_operator_windows": { + "P01-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.11987238161795156, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9905881019946707, + "mode_shares": { + "NONE": { + "attention": 0.1007462687465188, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8471040152242396, + "moe_router": 0.008477215998495907, + "norm_elementwise": 0.034119643236757626, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.13605518456441312, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019444790834074754, + "moe_gemm": 0.8102627123318585, + "moe_router": 0.01080794700368473, + "norm_elementwise": 0.03163704570787867, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 3, + "PIECEWISE": 5 + }, + "other_share": 0.009411898005329294, + "representativeness": { + "smd": { + "decode_batch_size": 2.4721524781047854, + "mode_FULL": -0.2325584995849942, + "mode_NONE": 0.3041456447839143, + "mode_PIECEWISE": -0.24468169875030038, + "prefill_fraction": 0.18099057277285577, + "scheduled_tokens": 0.4474608400278538 + }, + "valid": false + }, + "shares": { + "attention": 0.11987238161795156, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010532842927605177, + "moe_gemm": 0.8271478398184453, + "moe_router": 0.009739725006365179, + "norm_elementwise": 0.032774871259148075, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1567.9510000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 933.1070000000002, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 827.689, + "name": "triton_red_fused_2" + }, + { + "duration_us": 33.057, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.368999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 17.247, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.647999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848264235733542.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.008867955248024707, + "attention_mixed": 0.11302508576620553, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.990127660582764, + "mode_shares": { + "FULL": { + "attention": 0.16315875281832684, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031741793017298413, + "moe_gemm": 0.7536332801624461, + "moe_router": 0.02179864485086512, + "norm_elementwise": 0.045434533244984684, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.09828250109360281, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8488043657595763, + "moe_router": 0.008302816259724543, + "norm_elementwise": 0.0348733229605455, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.1472178594517614, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002166218689199354, + "moe_gemm": 0.7946960932441856, + "moe_router": 0.012792489660640699, + "norm_elementwise": 0.033466283510587794, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 3, + "PIECEWISE": 4 + }, + "other_share": 0.009872339417236002, + "representativeness": { + "smd": { + "decode_batch_size": 1.4422983740600546, + "mode_FULL": 0.3594518697735859, + "mode_NONE": 0.3041456447839143, + "mode_PIECEWISE": -0.49598146699059914, + "prefill_fraction": -0.3699982864929203, + "scheduled_tokens": 0.1504241077629697 + }, + "valid": false + }, + "shares": { + "attention": 0.12189304101423025, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010615972550528194, + "moe_gemm": 0.8214241451495896, + "moe_router": 0.010879021850869232, + "norm_elementwise": 0.034869855313022065, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1449.5139999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 884.6050000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 826.8859999999994, + "name": "triton_red_fused_2" + }, + { + "duration_us": 30.943000000000005, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.431999999999995, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 17.441, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.328999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848305040542105.pt.trace.json.gz" + } + ], + "P01-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.24646421810404764, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9913999320344749, + "mode_shares": { + "FULL": { + "attention": 0.24646421810404764, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0018861985126382374, + "moe_gemm": 0.6957077658710443, + "moe_router": 0.012274637353427154, + "norm_elementwise": 0.03506711219331752, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.008600067965525126, + "representativeness": { + "smd": { + "decode_batch_size": 0.658013698411068, + "mode_FULL": 0.6877531107808548, + "mode_NONE": -0.6802380609168648, + "mode_PIECEWISE": -0.08229232202065807, + "prefill_fraction": -0.6816133519618747, + "scheduled_tokens": -0.6333120429828512 + }, + "valid": false + }, + "shares": { + "attention": 0.24646421810404764, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0018861985126382374, + "moe_gemm": 0.6957077658710443, + "moe_router": 0.012274637353427154, + "norm_elementwise": 0.03506711219331752, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1013.7599999999983, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 701.0369999999989, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 598.1809999999995, + "name": "triton_red_fused_2" + }, + { + "duration_us": 25.409, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 22.721999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 13.697, + "name": "triton_red_fused_1" + }, + { + "duration_us": 13.632, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847491175653217.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0033278995815980927, + "attention_mixed": 0.07664950781433502, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9882022706147409, + "mode_shares": { + "FULL": { + "attention": 0.2372289894559244, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001973103373413011, + "moe_gemm": 0.7024833850537113, + "moe_router": 0.0127233500493961, + "norm_elementwise": 0.03660441698068597, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.07774006222398774, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8628349211839768, + "moe_router": 0.004560917765838791, + "norm_elementwise": 0.04302637544092451, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 7 + }, + "other_share": 0.011797729385259094, + "representativeness": { + "smd": { + "decode_batch_size": -1.184350878588216, + "mode_FULL": -1.8269976055369814, + "mode_NONE": 1.8429476277592753, + "mode_PIECEWISE": -0.08229232202065807, + "prefill_fraction": 1.7302789555698612, + "scheduled_tokens": 1.471377771158277 + }, + "valid": false + }, + "shares": { + "attention": 0.0799774073959331, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 2.7679120945085497e-05, + "moe_gemm": 0.8605854751478055, + "moe_router": 0.0046754221316145424, + "norm_elementwise": 0.04293628681844257, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 13829.692000000014, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 6941.902000000004, + "name": "triton_red_fused_2" + }, + { + "duration_us": 6363.376999999999, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 297.34499999999997, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 147.58499999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 24.673, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.264999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847533554126659.pt.trace.json.gz" + } + ], + "P01-C01-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.09652918148381516, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9901512141928626, + "mode_shares": { + "NONE": { + "attention": 0.08897688209726898, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8575678619411076, + "moe_router": 0.00844824275387592, + "norm_elementwise": 0.035009188256680354, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.11530968327931657, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001889902178480346, + "moe_gemm": 0.830686715724326, + "moe_router": 0.011422443441794888, + "norm_elementwise": 0.031213089133518845, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 5, + "PIECEWISE": 3 + }, + "other_share": 0.009848785807137474, + "representativeness": { + "smd": { + "decode_batch_size": 0.9854889320929141, + "mode_FULL": -0.5993507732751886, + "mode_NONE": 1.1348233880978151, + "mode_PIECEWISE": -0.7069751104796668, + "prefill_fraction": 0.8188657544214422, + "scheduled_tokens": 1.1521871833461559 + }, + "valid": false + }, + "shares": { + "attention": 0.09652918148381516, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0005420276773630952, + "moe_gemm": 0.8498582961156129, + "moe_router": 0.009301249385245966, + "norm_elementwise": 0.033920459530825446, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1815.9390000000008, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1066.9890000000005, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 990.7689999999997, + "name": "triton_red_fused_2" + }, + { + "duration_us": 38.336, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 21.344, + "name": "triton_red_fused_1" + }, + { + "duration_us": 20.768, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.296999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838883939067909.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.006726776525469258, + "attention_mixed": 0.09532695434384081, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9902577441629138, + "mode_shares": { + "FULL": { + "attention": 0.14374966861821964, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0032887654854086147, + "moe_gemm": 0.7722281273880269, + "moe_router": 0.02184573687437963, + "norm_elementwise": 0.04582940754683107, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.08786076611447088, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8586892334129893, + "moe_router": 0.008388182084023156, + "norm_elementwise": 0.03506617147547036, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.11217913793023779, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001786978933831843, + "moe_gemm": 0.8363300944674595, + "moe_router": 0.010036162117713828, + "norm_elementwise": 0.03050524878909313, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 3, + "PIECEWISE": 4 + }, + "other_share": 0.009742255837086222, + "representativeness": { + "smd": { + "decode_batch_size": 0.7899190491545085, + "mode_FULL": -0.07663743346636, + "mode_NONE": 0.5619254019326344, + "mode_PIECEWISE": -0.4409586407032101, + "prefill_fraction": 0.2296273370258069, + "scheduled_tokens": 0.8462072381054137 + }, + "valid": false + }, + "shares": { + "attention": 0.10205373086931008, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010046539011378537, + "moe_gemm": 0.843998403683613, + "moe_router": 0.009802509795881178, + "norm_elementwise": 0.03339844591297178, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1620.390000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 980.7070000000003, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 882.1790000000001, + "name": "triton_red_fused_2" + }, + { + "duration_us": 34.656, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 20.480999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 19.329, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.488999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838927022053880.pt.trace.json.gz" + } + ], + "P01-C01-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.13796857782087196, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9898086965802668, + "mode_shares": { + "NONE": { + "attention": 0.13796857782087196, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8097353073456708, + "moe_router": 0.005662986065242196, + "norm_elementwise": 0.03644182534848179, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.0101913034197332, + "representativeness": { + "smd": { + "decode_batch_size": -0.8655364049171133, + "mode_FULL": -0.6056347203602891, + "mode_NONE": 0.6248229775281566, + "mode_PIECEWISE": -0.1297191892430149, + "prefill_fraction": 0.6919360042362913, + "scheduled_tokens": 0.8182182157919601 + }, + "valid": false + }, + "shares": { + "attention": 0.13796857782087196, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8097353073456708, + "moe_router": 0.005662986065242196, + "norm_elementwise": 0.03644182534848179, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4331.866000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2358.544999999997, + "name": "triton_red_fused_2" + }, + { + "duration_us": 2195.539, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 92.25900000000001, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 51.296, + "name": "triton_red_fused_1" + }, + { + "duration_us": 25.217, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.408, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838155733630380.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.2400737379378586, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9913784351134121, + "mode_shares": { + "FULL": { + "attention": 0.2400737379378586, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001883134648982493, + "moe_gemm": 0.7024243548240628, + "moe_router": 0.012220876164257787, + "norm_elementwise": 0.03477633153825034, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.008621564886587925, + "representativeness": { + "smd": { + "decode_batch_size": 3.1158472389796006, + "mode_FULL": 3.2983798616544977, + "mode_NONE": -3.197086914213414, + "mode_PIECEWISE": -0.1297191892430149, + "prefill_fraction": -3.24128548180199, + "scheduled_tokens": -2.8103428714456196 + }, + "valid": false + }, + "shares": { + "attention": 0.2400737379378586, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001883134648982493, + "moe_gemm": 0.7024243548240628, + "moe_router": 0.012220876164257787, + "norm_elementwise": 0.03477633153825034, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 982.8509999999973, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 705.1959999999982, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 592.7990000000009, + "name": "triton_red_fused_2" + }, + { + "duration_us": 25.921, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 22.368999999999996, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 14.016999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 12.417, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838206570748178.pt.trace.json.gz" + } + ], + "P01-C10-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.07690712187247944, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9897759832839516, + "mode_shares": { + "NONE": { + "attention": 0.07690712187247944, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8669760431839948, + "moe_router": 0.009429489436313557, + "norm_elementwise": 0.03646332879116368, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.010224016716048499, + "representativeness": { + "smd": { + "decode_batch_size": 1.7903696522371881, + "mode_FULL": -1.4371597160911773, + "mode_NONE": 1.4371597160911769, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.5249388209670753, + "scheduled_tokens": 1.6614670732013832 + }, + "valid": false + }, + "shares": { + "attention": 0.07690712187247944, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8669760431839948, + "moe_router": 0.009429489436313557, + "norm_elementwise": 0.03646332879116368, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1890.1099999999983, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1095.9750000000006, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 1058.7089999999996, + "name": "triton_red_fused_2" + }, + { + "duration_us": 40.384, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.752000000000002, + "name": "triton_red_fused_1" + }, + { + "duration_us": 20.032000000000004, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.072, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838884504530948.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.049266143541160906, + "attention_mixed": 0.08377752045067025, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9891853536707774, + "mode_shares": { + "FULL": { + "attention": 0.15457303786338591, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003489366068720859, + "moe_gemm": 0.7587034266365785, + "moe_router": 0.022859966949438017, + "norm_elementwise": 0.046720964018498824, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.12297149131658515, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8225726880507142, + "moe_router": 0.012431913811185054, + "norm_elementwise": 0.0325372503493313, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 4, + "NONE": 4 + }, + "other_share": 0.01081464632922258, + "representativeness": { + "smd": { + "decode_batch_size": 24.775039622970557, + "mode_FULL": -0.015698813991273975, + "mode_NONE": 0.015698813991273868, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.0963234706088523, + "scheduled_tokens": 0.051398685719152515 + }, + "valid": false + }, + "shares": { + "attention": 0.13304366399183115, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0011121448603552246, + "moe_gemm": 0.8022160188398926, + "moe_router": 0.015755585066310956, + "norm_elementwise": 0.03705794091238747, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 922.1419999999987, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 640.8700000000005, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 594.631000000001, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.12, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 19.169, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 13.408999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.721, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838930721906534.pt.trace.json.gz" + } + ], + "P01-C10-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.17009543151870654, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9841498976767467, + "mode_shares": { + "FULL": { + "attention": 0.17009543151870654, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0039228461117771524, + "moe_gemm": 0.7298923248097096, + "moe_router": 0.026541989019097344, + "norm_elementwise": 0.05369730621745603, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.015850102323253314, + "representativeness": { + "smd": { + "decode_batch_size": 0.346060557246963, + "mode_FULL": 0.3620750335204151, + "mode_NONE": -0.3620750335204151, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.361159573685948, + "scheduled_tokens": -0.3136156264486445 + }, + "valid": false + }, + "shares": { + "attention": 0.17009543151870654, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0039228461117771524, + "moe_gemm": 0.7298923248097096, + "moe_router": 0.026541989019097344, + "norm_elementwise": 0.05369730621745603, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 668.4249999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 540.3219999999997, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 519.6199999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.162, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.146, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.769, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.911999999999997, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838153027510750.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.18529303661511093, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9842190227520464, + "mode_shares": { + "FULL": { + "attention": 0.18529303661511093, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0039014000835443053, + "moe_gemm": 0.7150112365990193, + "moe_router": 0.026542171689060874, + "norm_elementwise": 0.053471177765311045, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.015780977247953545, + "representativeness": { + "smd": { + "decode_batch_size": 0.346060557246963, + "mode_FULL": 0.3620750335204151, + "mode_NONE": -0.3620750335204151, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.361159573685948, + "scheduled_tokens": -0.3136156264486445 + }, + "valid": false + }, + "shares": { + "attention": 0.18529303661511093, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0039014000835443053, + "moe_gemm": 0.7150112365990193, + "moe_router": 0.026542171689060874, + "norm_elementwise": 0.053471177765311045, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 670.1069999999988, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 540.7329999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 521.2969999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.871, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.303999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.639, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.815, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838186373520707.pt.trace.json.gz" + } + ], + "P01-C11-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.07147321055558287, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9898790022112421, + "mode_shares": { + "NONE": { + "attention": 0.07147321055558287, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8734165832313857, + "moe_router": 0.009015253683509436, + "norm_elementwise": 0.03597395474076414, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.010120997788757882, + "representativeness": { + "smd": { + "decode_batch_size": 1.5977202714579513, + "mode_FULL": -1.5988956800998717, + "mode_NONE": 1.5988956800998717, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.7194658992699898, + "scheduled_tokens": 2.889470183704151 + }, + "valid": false + }, + "shares": { + "attention": 0.07147321055558287, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8734165832313857, + "moe_router": 0.009015253683509436, + "norm_elementwise": 0.03597395474076414, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1951.778999999998, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1101.107, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 1087.848, + "name": "triton_red_fused_2" + }, + { + "duration_us": 41.377, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.752, + "name": "triton_red_fused_1" + }, + { + "duration_us": 19.905, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.263999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849468308907391.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.07815008583685061, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9900845346277651, + "mode_shares": { + "NONE": { + "attention": 0.07815008583685061, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8665104217463119, + "moe_router": 0.01003059260991946, + "norm_elementwise": 0.035393434434683305, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.00991546537223486, + "representativeness": { + "smd": { + "decode_batch_size": 3.6469387885636997, + "mode_FULL": -1.5988956800998717, + "mode_NONE": 1.5988956800998717, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.6333752995057749, + "scheduled_tokens": 1.932993318175888 + }, + "valid": false + }, + "shares": { + "attention": 0.07815008583685061, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8665104217463119, + "moe_router": 0.01003059260991946, + "norm_elementwise": 0.035393434434683305, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1646.947999999998, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 960.6999999999999, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 911.4259999999997, + "name": "triton_red_fused_2" + }, + { + "duration_us": 35.168, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 20.32, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 19.008, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.325999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849515320434790.pt.trace.json.gz" + } + ], + "P01-C11-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.016576587438731645, + "attention_mixed": 0.07063959803466414, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9887471770371591, + "mode_shares": { + "FULL": { + "attention": 0.16949408215942174, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003719519173496834, + "moe_gemm": 0.7347698666441799, + "moe_router": 0.02526425285717597, + "norm_elementwise": 0.051657468475250934, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.07829708337077482, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8654362098361494, + "moe_router": 0.006897111319710678, + "norm_elementwise": 0.03853325229671188, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 3, + "NONE": 5 + }, + "other_share": 0.011252822962840962, + "representativeness": { + "smd": { + "decode_batch_size": -0.6805120548902551, + "mode_FULL": -0.7994927227550991, + "mode_NONE": 0.7994927227550991, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 0.8067495635539911, + "scheduled_tokens": 0.78379501498442 + }, + "valid": false + }, + "shares": { + "attention": 0.08721618547339578, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0003637704279935639, + "moe_gemm": 0.8526569904343553, + "moe_router": 0.008693424936188346, + "norm_elementwise": 0.03981680576522616, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 2296.86200000001, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1370.555999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 1296.3929999999978, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 48.382999999999996, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 29.056000000000004, + "name": "triton_red_fused_1" + }, + { + "duration_us": 20.737, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.072999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848740680228913.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.16100163906049575, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9857318180258278, + "mode_shares": { + "FULL": { + "attention": 0.16100163906049575, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0035030082867205997, + "moe_gemm": 0.7487144065958767, + "moe_router": 0.023795240157795577, + "norm_elementwise": 0.048717523924939055, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.014268181974172239, + "representativeness": { + "smd": { + "decode_batch_size": 0.792348678707734, + "mode_FULL": 0.8046290452001797, + "mode_NONE": -0.8046290452001796, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.8036484371137015, + "scheduled_tokens": -0.761537726891679 + }, + "valid": false + }, + "shares": { + "attention": 0.16100163906049575, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0035030082867205997, + "moe_gemm": 0.7487144065958767, + "moe_router": 0.023795240157795577, + "norm_elementwise": 0.048717523924939055, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 662.919999999999, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 536.0999999999999, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 521.2559999999994, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.152, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.952, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.287999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.974999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848781627748251.pt.trace.json.gz" + } + ], + "P02-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.16431648329180198, + "attention_mixed": 0.040375747240313245, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9899496306808163, + "mode_shares": { + "FULL": { + "attention": 0.2070416346282377, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002614300625672357, + "moe_gemm": 0.72402183586612, + "moe_router": 0.017564069072735583, + "norm_elementwise": 0.03839442593161487, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.19565666700967585, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017527292718792177, + "moe_gemm": 0.7535224022949394, + "moe_router": 0.011441473126433909, + "norm_elementwise": 0.02878152643291581, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "other_share": 0.010050369319183737, + "representativeness": { + "smd": { + "decode_batch_size": -0.673985188837054, + "mode_FULL": 0.11926397031787984, + "mode_NONE": -0.2790993472934029, + "mode_PIECEWISE": -0.0171348471830914, + "prefill_fraction": -0.13753859346790945, + "scheduled_tokens": -0.22985353117145157 + }, + "valid": false + }, + "shares": { + "attention": 0.20469223053211524, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0024365065962538455, + "moe_gemm": 0.7301095783988965, + "moe_router": 0.016300609001782734, + "norm_elementwise": 0.036410706151768, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 775.5859999999996, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 562.3350000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 541.5999999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.592, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.384, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.864999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 12.447999999999999, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850544932244469.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.17098573558976576, + "attention_mixed": 0.04244961271753521, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9900472542616138, + "mode_shares": { + "FULL": { + "attention": 0.20867035579310442, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025849579196340765, + "moe_gemm": 0.7236684498934784, + "moe_router": 0.017385853018663355, + "norm_elementwise": 0.03743160202992195, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.23505546138588368, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025209396629618792, + "moe_gemm": 0.7060220856055816, + "moe_router": 0.012259376230707445, + "norm_elementwise": 0.035577961118325376, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "other_share": 0.00995274573838624, + "representativeness": { + "smd": { + "decode_batch_size": 14.182091081480554, + "mode_FULL": 0.11926397031787984, + "mode_NONE": -0.2790993472934029, + "mode_PIECEWISE": -0.0171348471830914, + "prefill_fraction": -0.20596870108215656, + "scheduled_tokens": -0.2935716394203467 + }, + "valid": false + }, + "shares": { + "attention": 0.21343534830730096, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025733966047573075, + "moe_gemm": 0.7204816218764769, + "moe_router": 0.016460041922970896, + "norm_elementwise": 0.03709684555010769, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 767.5569999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 572.6729999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 527.8730000000002, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.624999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.192999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.735999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.902999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850578506315216.pt.trace.json.gz" + } + ], + "P02-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.3066236078963298, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9922817455017883, + "mode_shares": { + "FULL": { + "attention": 0.3066236078963298, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0016715827167837374, + "moe_gemm": 0.6421673334694945, + "moe_router": 0.010874019669713833, + "norm_elementwise": 0.030945201749466352, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.007718254498211715, + "representativeness": { + "smd": { + "decode_batch_size": 0.20138501290112165, + "mode_FULL": 0.20658265838024312, + "mode_NONE": -0.20080594382714848, + "mode_PIECEWISE": -0.04755171103421909, + "prefill_fraction": -0.2043665285851284, + "scheduled_tokens": -0.19254510985844397 + }, + "valid": true + }, + "shares": { + "attention": 0.3066236078963298, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0016715827167837374, + "moe_gemm": 0.6421673334694945, + "moe_router": 0.010874019669713833, + "norm_elementwise": 0.030945201749466352, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 996.7139999999991, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 703.9869999999992, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 606.911, + "name": "triton_red_fused_2" + }, + { + "duration_us": 24.991999999999997, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 21.758999999999997, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 13.247, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 13.184, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849956178681726.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.34678283359255974, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9928295517926563, + "mode_shares": { + "FULL": { + "attention": 0.34678283359255974, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015432330235483139, + "moe_gemm": 0.6057013640578384, + "moe_router": 0.010146306602355423, + "norm_elementwise": 0.028655814516354454, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.0071704482073436755, + "representativeness": { + "smd": { + "decode_batch_size": 0.20138501290112165, + "mode_FULL": 0.20658265838024312, + "mode_NONE": -0.20080594382714848, + "mode_PIECEWISE": -0.04755171103421909, + "prefill_fraction": -0.2043665285851284, + "scheduled_tokens": -0.19254510985844397 + }, + "valid": true + }, + "shares": { + "attention": 0.34678283359255974, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015432330235483139, + "moe_gemm": 0.6057013640578384, + "moe_router": 0.010146306602355423, + "norm_elementwise": 0.028655814516354454, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1013.8929999999987, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 711.0099999999981, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 613.2400000000005, + "name": "triton_red_fused_2" + }, + { + "duration_us": 25.055, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 22.112, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 14.049, + "name": "triton_red_fused_1" + }, + { + "duration_us": 13.855, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849993946669416.pt.trace.json.gz" + } + ], + "P03-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.23677440452100923, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9773142909813943, + "mode_shares": { + "FULL": { + "attention": 0.23677440452100923, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.007300880477096696, + "moe_gemm": 0.5916730425467414, + "moe_router": 0.030667589074404745, + "norm_elementwise": 0.11089837436214241, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.022685709018605626, + "representativeness": { + "smd": { + "decode_batch_size": 3.72009366381467, + "mode_FULL": 0.27793577928319474, + "mode_NONE": -0.27793577928319446, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1868864933066391, + "scheduled_tokens": -0.182101722961305 + }, + "valid": false + }, + "shares": { + "attention": 0.23677440452100923, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.007300880477096696, + "moe_gemm": 0.5916730425467414, + "moe_router": 0.030667589074404745, + "norm_elementwise": 0.11089837436214241, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 576.23, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 464.7449999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.744, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.575, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.040999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 9.41, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849465841450297.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.2295917342415148, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9711974069819732, + "mode_shares": { + "FULL": { + "attention": 0.2295917342415148, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009291871432449663, + "moe_gemm": 0.5573510710995793, + "moe_router": 0.04015146373050413, + "norm_elementwise": 0.13481126647792535, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.02880259301802682, + "representativeness": { + "smd": { + "decode_batch_size": -0.27750213252727524, + "mode_FULL": 0.27793577928319474, + "mode_NONE": -0.27793577928319446, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1868864933066391, + "scheduled_tokens": -0.1838557206293655 + }, + "valid": false + }, + "shares": { + "attention": 0.2295917342415148, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009291871432449663, + "moe_gemm": 0.5573510710995793, + "moe_router": 0.04015146373050413, + "norm_elementwise": 0.13481126647792535, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 571.9689999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 469.85699999999855, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.529, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.383, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.761000000000001, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849497248548609.pt.trace.json.gz" + } + ], + "P03-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.15040135314337208, + "attention_mixed": 0.28933706610779675, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9918450008730845, + "mode_shares": { + "FULL": { + "attention": 0.6859966420485143, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019314646391670124, + "moe_gemm": 0.2659878061842072, + "moe_router": 0.012555897863413516, + "norm_elementwise": 0.025494810657793975, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.3425246256579563, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147373284349258, + "moe_router": 0.003205686402689487, + "norm_elementwise": 0.030942924035345834, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.61102927558752, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0013975676747442667, + "moe_gemm": 0.3571425323248457, + "moe_router": 0.006697455630309823, + "norm_elementwise": 0.018973805351208753, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 6, + "NONE": 1, + "PIECEWISE": 1 + }, + "other_share": 0.008154999126915493, + "representativeness": { + "smd": { + "decode_batch_size": 1.22313193325346, + "mode_FULL": 0.7720860330486908, + "mode_NONE": -1.155709946053278, + "mode_PIECEWISE": 0.5, + "prefill_fraction": -0.8564873113454512, + "scheduled_tokens": -1.021286605051562 + }, + "valid": false + }, + "shares": { + "attention": 0.4397384192511688, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.000537501805597198, + "moe_gemm": 0.5172566828279285, + "moe_router": 0.0055405928130157975, + "norm_elementwise": 0.028771804175374255, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 2966.5110000000122, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1605.9449999999956, + "name": "triton_red_fused_2" + }, + { + "duration_us": 1552.6929999999936, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 63.32699999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 34.559000000000005, + "name": "triton_red_fused_1" + }, + { + "duration_us": 28.895999999999994, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.679, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848741144884870.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3425145548872434, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9914119224463182, + "mode_shares": { + "NONE": { + "attention": 0.3425145548872434, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147853532198007, + "moe_router": 0.0032173993050482958, + "norm_elementwise": 0.030894615034225702, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.008588077553681895, + "representativeness": { + "smd": { + "decode_batch_size": -1.5373920998057986, + "mode_FULL": -1.1124192104801658, + "mode_NONE": 1.1124192104801658, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.1145676479021394, + "scheduled_tokens": 1.1485324061566928 + }, + "valid": false + }, + "shares": { + "attention": 0.3425145548872434, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147853532198007, + "moe_router": 0.0032173993050482958, + "norm_elementwise": 0.030894615034225702, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 17168.532999999996, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 8516.018999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 7858.939000000003, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 365.792, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 181.474, + "name": "triton_red_fused_1" + }, + { + "duration_us": 107.04, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.894999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848782480504538.pt.trace.json.gz" + } + ], + "P03-C01-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.2472611589117568, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9711954941240638, + "mode_shares": { + "FULL": { + "attention": 0.2472611589117568, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008981204160230193, + "moe_gemm": 0.5449779176549342, + "moe_router": 0.03864212767674718, + "norm_elementwise": 0.13133308572039554, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.028804505875936144, + "representativeness": { + "smd": { + "decode_batch_size": 0.12761411907604883, + "mode_FULL": 0.40277438708242275, + "mode_NONE": -0.39239725725147123, + "mode_PIECEWISE": -0.08433231534384081, + "prefill_fraction": -0.3360840189860844, + "scheduled_tokens": -0.3186082852329951 + }, + "valid": false + }, + "shares": { + "attention": 0.2472611589117568, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008981204160230193, + "moe_gemm": 0.5449779176549342, + "moe_router": 0.03864212767674718, + "norm_elementwise": 0.13133308572039554, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 608.2109999999999, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 459.13399999999984, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.080000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.448999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.886, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838881774721247.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.030493797355859827, + "attention_mixed": 0.0, + "attention_prefill": 0.2337756657652817 + }, + "classifiable_fraction": 0.9876621178230831, + "mode_shares": { + "FULL": { + "attention": 0.20796901066629175, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00949416034152379, + "moe_gemm": 0.5730563859384595, + "moe_router": 0.04094999186682201, + "norm_elementwise": 0.138651777663304, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.2847935541112115, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6698501096001059, + "moe_router": 0.004797173211359188, + "norm_elementwise": 0.031704685814038104, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.19039146853788674, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003165041910063069, + "moe_gemm": 0.7266700356495558, + "moe_router": 0.021201509715176935, + "norm_elementwise": 0.04563231947124987, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 6, + "NONE": 1, + "PIECEWISE": 1 + }, + "other_share": 0.012337882176916845, + "representativeness": { + "smd": { + "decode_batch_size": -0.5656913025413245, + "mode_FULL": -0.4645415813164409, + "mode_NONE": 0.17296137292420416, + "mode_PIECEWISE": 0.47910292145473704, + "prefill_fraction": 0.5401368889615827, + "scheduled_tokens": 0.29654274109388945 + }, + "valid": false + }, + "shares": { + "attention": 0.2642694631211415, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001702544587935503, + "moe_gemm": 0.6612308364176214, + "moe_router": 0.011707182831279314, + "norm_elementwise": 0.04875209086510538, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1139.9199999999971, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 713.7360000000015, + "name": "triton_red_fused_2" + }, + { + "duration_us": 380.48100000000034, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 24.064000000000007, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 20.0, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.997999999999998, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838913535208997.pt.trace.json.gz" + } + ], + "P03-C01-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3625175101430537, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9921751035942372, + "mode_shares": { + "NONE": { + "attention": 0.3625175101430537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5972917860703632, + "moe_router": 0.004233348375179386, + "norm_elementwise": 0.028132459005640997, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.007824896405762824, + "representativeness": { + "smd": { + "decode_batch_size": 0.03842752249272106, + "mode_FULL": 0.0, + "mode_NONE": 0.0, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.03842752249261314, + "scheduled_tokens": 0.0 + }, + "valid": true + }, + "shares": { + "attention": 0.3625175101430537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5972917860703632, + "moe_router": 0.004233348375179386, + "norm_elementwise": 0.028132459005640997, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4829.081000000003, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2461.234000000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 2440.887999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 103.19999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 54.239, + "name": "triton_red_fused_1" + }, + { + "duration_us": 34.912000000000006, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 10.815999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838155509202718.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3233680934686369, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9917064998087011, + "mode_shares": { + "NONE": { + "attention": 0.3233680934686369, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6340271091943758, + "moe_router": 0.004488327860901843, + "norm_elementwise": 0.029822969284786536, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.008293500191298969, + "representativeness": { + "smd": { + "decode_batch_size": 0.03842752249272106, + "mode_FULL": 0.0, + "mode_NONE": 0.0, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.03842752249261314, + "scheduled_tokens": 0.0 + }, + "valid": true + }, + "shares": { + "attention": 0.3233680934686369, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6340271091943758, + "moe_router": 0.004488327860901843, + "norm_elementwise": 0.029822969284786536, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4825.282999999999, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2464.268999999999, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 2440.0180000000014, + "name": "triton_red_fused_2" + }, + { + "duration_us": 103.362, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 54.37, + "name": "triton_red_fused_1" + }, + { + "duration_us": 34.081, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.167, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838204169798936.pt.trace.json.gz" + } + ], + "P03-C10-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.22514759732195092, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9713506698327266, + "mode_shares": { + "FULL": { + "attention": 0.22514759732195092, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009442025728651632, + "moe_gemm": 0.5602617147563997, + "moe_router": 0.03949200310681736, + "norm_elementwise": 0.13700732891890693, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.02864933016727339, + "representativeness": { + "smd": { + "decode_batch_size": -0.18984540767127392, + "mode_FULL": 0.2782013992817753, + "mode_NONE": -0.27820139928177545, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.18350522021486032, + "scheduled_tokens": -0.18063803827124955 + }, + "valid": false + }, + "shares": { + "attention": 0.22514759732195092, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009442025728651632, + "moe_gemm": 0.5602617147563997, + "moe_router": 0.03949200310681736, + "norm_elementwise": 0.13700732891890693, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 581.8899999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 449.5400000000006, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.656, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.120999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.725999999999999, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836530738313739.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.22556467485036294, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9704584620048622, + "mode_shares": { + "FULL": { + "attention": 0.22556467485036294, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009411480644856082, + "moe_gemm": 0.5591648344398723, + "moe_router": 0.03945587906666355, + "norm_elementwise": 0.13686159300310727, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.02954153799513786, + "representativeness": { + "smd": { + "decode_batch_size": -0.18984540767127392, + "mode_FULL": 0.2782013992817753, + "mode_NONE": -0.27820139928177545, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.18350522021486032, + "scheduled_tokens": -0.18063803827124955 + }, + "valid": false + }, + "shares": { + "attention": 0.22556467485036294, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009411480644856082, + "moe_gemm": 0.5591648344398723, + "moe_router": 0.03945587906666355, + "norm_elementwise": 0.13686159300310727, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 616.4500000000008, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 449.8830000000004, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.168, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.6, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.630999999999998, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836562201333740.pt.trace.json.gz" + } + ], + "P04-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.45337846230845036, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9811078968569179, + "mode_shares": { + "FULL": { + "attention": 0.45337846230845036, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00447249169315189, + "moe_gemm": 0.4320822710822647, + "moe_router": 0.03260980879841117, + "norm_elementwise": 0.05856486297463963, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.01889210314308214, + "representativeness": { + "smd": { + "decode_batch_size": 0.20844725339793926, + "mode_FULL": 0.18226661968758462, + "mode_NONE": -0.18226661968758412, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.15943564286260098, + "scheduled_tokens": -0.15585724120282418 + }, + "valid": true + }, + "shares": { + "attention": 0.45337846230845036, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00447249169315189, + "moe_gemm": 0.4320822710822647, + "moe_router": 0.03260980879841117, + "norm_elementwise": 0.05856486297463963, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 600.4589999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 503.8849999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 467.6160000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.391000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.958999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.197999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.726999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849465791477789.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.5042949115154224, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.980970138873411, + "mode_shares": { + "FULL": { + "attention": 0.5042949115154224, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004490478325551558, + "moe_gemm": 0.3806685401425692, + "moe_router": 0.032988086256013634, + "norm_elementwise": 0.058528122633854276, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.019029861126589043, + "representativeness": { + "smd": { + "decode_batch_size": 0.20844725339793926, + "mode_FULL": 0.18226661968758462, + "mode_NONE": -0.18226661968758412, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.15943564286260098, + "scheduled_tokens": -0.15585724120282418 + }, + "valid": true + }, + "shares": { + "attention": 0.5042949115154224, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004490478325551558, + "moe_gemm": 0.3806685401425692, + "moe_router": 0.032988086256013634, + "norm_elementwise": 0.058528122633854276, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 600.0620000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 501.89399999999864, + "name": "triton_red_fused_2" + }, + { + "duration_us": 467.97200000000095, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.328, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.800999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.870999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.014999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849497082189253.pt.trace.json.gz" + } + ], + "P04-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.6545383651006825, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.992872517456777, + "mode_shares": { + "FULL": { + "attention": 0.6545383651006825, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001725224109977307, + "moe_gemm": 0.30244289390345225, + "moe_router": 0.011446277854389742, + "norm_elementwise": 0.022719756488275074, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.007127482543223056, + "representativeness": { + "smd": { + "decode_batch_size": -2.090753569627607, + "mode_FULL": 0.3444176950404237, + "mode_NONE": -0.3444176950404235, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.3279408074303144, + "scheduled_tokens": -0.3384806476596461 + }, + "valid": false + }, + "shares": { + "attention": 0.6545383651006825, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001725224109977307, + "moe_gemm": 0.30244289390345225, + "moe_router": 0.011446277854389742, + "norm_elementwise": 0.022719756488275074, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 702.7299999999998, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 538.8529999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.6859999999996, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.480999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.599, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.903999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.879999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848739183825698.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.6629935506677639, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9927205146123725, + "mode_shares": { + "FULL": { + "attention": 0.6629935506677639, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017655694129604385, + "moe_gemm": 0.29304677622038244, + "moe_router": 0.01168833427441094, + "norm_elementwise": 0.02322628403685483, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.007279485387627501, + "representativeness": { + "smd": { + "decode_batch_size": -3.0407898355557177, + "mode_FULL": 0.3444176950404237, + "mode_NONE": -0.3444176950404235, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.3279408074303144, + "scheduled_tokens": -0.3392815338864717 + }, + "valid": false + }, + "shares": { + "attention": 0.6629935506677639, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017655694129604385, + "moe_gemm": 0.29304677622038244, + "moe_router": 0.01168833427441094, + "norm_elementwise": 0.02322628403685483, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 697.0249999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 539.4639999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 512.0369999999991, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.703, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.440999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.392999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.231999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848772357502908.pt.trace.json.gz" + } + ], + "P06-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.2490290455613554, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9782676099166654, + "mode_shares": { + "FULL": { + "attention": 0.2490290455613554, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005145523506508112, + "moe_gemm": 0.6200403635488907, + "moe_router": 0.03550127037050399, + "norm_elementwise": 0.06855140692940716, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.021732390083334552, + "representativeness": { + "smd": { + "decode_batch_size": -0.7216535064826213, + "mode_FULL": 0.13539969781837916, + "mode_NONE": -0.1341287527105229, + "mode_PIECEWISE": -0.018343220956176326, + "prefill_fraction": -0.13284443594178524, + "scheduled_tokens": -0.12475886497271786 + }, + "valid": false + }, + "shares": { + "attention": 0.2490290455613554, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005145523506508112, + "moe_gemm": 0.6200403635488907, + "moe_router": 0.03550127037050399, + "norm_elementwise": 0.06855140692940716, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 597.553, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 503.07699999999915, + "name": "triton_red_fused_2" + }, + { + "duration_us": 466.9400000000012, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.869999999999997, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.854, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.744, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.727, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850544482514161.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.382897364177646, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9857570160520932, + "mode_shares": { + "FULL": { + "attention": 0.382897364177646, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003409296822253497, + "moe_gemm": 0.5303535709645394, + "moe_router": 0.023879803771651553, + "norm_elementwise": 0.04521698031600275, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.014242983947906827, + "representativeness": { + "smd": { + "decode_batch_size": 2.7828468550637027, + "mode_FULL": 0.13539969781837916, + "mode_NONE": -0.1341287527105229, + "mode_PIECEWISE": -0.018343220956176326, + "prefill_fraction": -0.13284443594178524, + "scheduled_tokens": -0.10682572566427663 + }, + "valid": false + }, + "shares": { + "attention": 0.382897364177646, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003409296822253497, + "moe_gemm": 0.5303535709645394, + "moe_router": 0.023879803771651553, + "norm_elementwise": 0.04521698031600275, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 620.7480000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 515.2009999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 493.54399999999913, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.936, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.312, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.751999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.4, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850576196502468.pt.trace.json.gz" + } + ], + "P06-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.5606386878931048, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9937049249586525, + "mode_shares": { + "FULL": { + "attention": 0.5606386878931048, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015728835856848925, + "moe_gemm": 0.3985345690982096, + "moe_router": 0.010374903112678729, + "norm_elementwise": 0.022583881268974364, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.006295075041347565, + "representativeness": { + "smd": { + "decode_batch_size": -0.19939502094061695, + "mode_FULL": 0.3784962133407253, + "mode_NONE": -0.3784962133407256, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.34967744618584184, + "scheduled_tokens": -0.3730920628084988 + }, + "valid": false + }, + "shares": { + "attention": 0.5606386878931048, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015728835856848925, + "moe_gemm": 0.3985345690982096, + "moe_router": 0.010374903112678729, + "norm_elementwise": 0.022583881268974364, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 730.718, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 532.1659999999997, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 517.7589999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.624, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.944999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.351999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 11.359999999999998, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849955632275097.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.5711221921286438, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9938941650105397, + "mode_shares": { + "FULL": { + "attention": 0.5711221921286438, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001510179204207269, + "moe_gemm": 0.3884028034591522, + "moe_router": 0.010456005468211384, + "norm_elementwise": 0.022402984750325126, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.006105834989460318, + "representativeness": { + "smd": { + "decode_batch_size": -3.0651651743014807, + "mode_FULL": 0.3784962133407253, + "mode_NONE": -0.3784962133407256, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.34967744618584184, + "scheduled_tokens": -0.381504751465261 + }, + "valid": false + }, + "shares": { + "attention": 0.5711221921286438, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001510179204207269, + "moe_gemm": 0.3884028034591522, + "moe_router": 0.010456005468211384, + "norm_elementwise": 0.022402984750325126, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 735.0089999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 530.0759999999996, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.90599999999927, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.855999999999995, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.872, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.360999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.878999999999998, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849989625437379.pt.trace.json.gz" + } + ], + "P06-C01-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.016563491820063184, + "attention_mixed": 0.3001356839582549, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9913361171706583, + "mode_shares": { + "FULL": { + "attention": 0.3160414926288235, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0038264604478624375, + "moe_gemm": 0.5878127540909809, + "moe_router": 0.025157477959619073, + "norm_elementwise": 0.05121416016089138, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.31673555083570937, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6406959135573278, + "moe_router": 0.004599858882822663, + "norm_elementwise": 0.029707643839264736, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 3, + "NONE": 5 + }, + "other_share": 0.008663882829341732, + "representativeness": { + "smd": { + "decode_batch_size": 1.2368003091321618, + "mode_FULL": -1.5398684829814582, + "mode_NONE": 1.543433248193871, + "mode_PIECEWISE": -0.03766886010245595, + "prefill_fraction": 1.5423098869457519, + "scheduled_tokens": 1.5062889639493047 + }, + "valid": false + }, + "shares": { + "attention": 0.3166991757783181, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00020054185227634405, + "moe_gemm": 0.637924347798813, + "moe_router": 0.00567726790461336, + "norm_elementwise": 0.03083478383663744, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 3051.4100000000076, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1636.7499999999984, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 1627.5970000000007, + "name": "triton_red_fused_2" + }, + { + "duration_us": 65.154, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 36.064, + "name": "triton_red_fused_1" + }, + { + "duration_us": 24.608, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 10.561, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848264687091429.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.37214450648638964, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9883428882887048, + "mode_shares": { + "FULL": { + "attention": 0.37214450648638964, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00282500541869895, + "moe_gemm": 0.5573743342631299, + "moe_router": 0.017973887722641455, + "norm_elementwise": 0.038025154397844804, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.01165711171129528, + "representativeness": { + "smd": { + "decode_batch_size": 3.172928254461621, + "mode_FULL": 0.25227637804266817, + "mode_NONE": -0.2492686165020934, + "mode_PIECEWISE": -0.03766886010245595, + "prefill_fraction": -0.25064540002066804, + "scheduled_tokens": -0.20625098090671584 + }, + "valid": false + }, + "shares": { + "attention": 0.37214450648638964, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00282500541869895, + "moe_gemm": 0.5573743342631299, + "moe_router": 0.017973887722641455, + "norm_elementwise": 0.038025154397844804, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 656.2589999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 496.6409999999987, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 494.02099999999876, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.584, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.175999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.678999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.911999999999999, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848304448588625.pt.trace.json.gz" + } + ], + "P06-C01-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3895842363152871, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9927290912532949, + "mode_shares": { + "NONE": { + "attention": 0.3895842363152871, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5731444916792802, + "moe_router": 0.003933173990421752, + "norm_elementwise": 0.026067189268305868, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.007270908746705119, + "representativeness": { + "smd": { + "decode_batch_size": -0.4361011161328784, + "mode_FULL": -2.2231293099337295, + "mode_NONE": 2.2283091632883223, + "mode_PIECEWISE": -0.043664375559957246, + "prefill_fraction": 2.341843613561519, + "scheduled_tokens": 2.2566609549134715 + }, + "valid": false + }, + "shares": { + "attention": 0.3895842363152871, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5731444916792802, + "moe_router": 0.003933173990421752, + "norm_elementwise": 0.026067189268305868, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4807.593000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2442.074, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 2441.511, + "name": "triton_red_fused_2" + }, + { + "duration_us": 102.624, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 53.760999999999996, + "name": "triton_red_fused_1" + }, + { + "duration_us": 36.193, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.647999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847493627203394.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3945011262454341, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9927395420899131, + "mode_shares": { + "NONE": { + "attention": 0.3945011262454341, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5682708789371121, + "moe_router": 0.003929026668942022, + "norm_elementwise": 0.02603851023842481, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.007260457910086926, + "representativeness": { + "smd": { + "decode_batch_size": -3.1519864341790544, + "mode_FULL": -2.2231293099337295, + "mode_NONE": 2.2283091632883223, + "mode_PIECEWISE": -0.043664375559957246, + "prefill_fraction": 2.366260389350378, + "scheduled_tokens": 2.2566609549134715 + }, + "valid": false + }, + "shares": { + "attention": 0.3945011262454341, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5682708789371121, + "moe_router": 0.003929026668942022, + "norm_elementwise": 0.02603851023842481, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4813.921000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2446.1619999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 2437.272, + "name": "triton_red_fused_2" + }, + { + "duration_us": 102.78399999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 54.273, + "name": "triton_red_fused_1" + }, + { + "duration_us": 35.68, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.008, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847543255284600.pt.trace.json.gz" + } + ], + "P06-C10-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.3581687545191966, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9772718952687278, + "mode_shares": { + "FULL": { + "attention": 0.3581687545191966, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00525419312260622, + "moe_gemm": 0.5105179315671342, + "moe_router": 0.03567192120640712, + "norm_elementwise": 0.06765909485338374, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.022728104731272125, + "representativeness": { + "smd": { + "decode_batch_size": -0.9404968816184628, + "mode_FULL": 0.1374340854376679, + "mode_NONE": -0.1374340854376676, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.13478349167720893, + "scheduled_tokens": -0.1290275248369585 + }, + "valid": false + }, + "shares": { + "attention": 0.3581687545191966, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00525419312260622, + "moe_gemm": 0.5105179315671342, + "moe_router": 0.03567192120640712, + "norm_elementwise": 0.06765909485338374, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 630.1189999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 521.6069999999993, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 499.6179999999991, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.392, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.6, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.881, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.631, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850544481880252.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.417531453105595, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9841451093044552, + "mode_shares": { + "FULL": { + "attention": 0.417531453105595, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0038280359471430728, + "moe_gemm": 0.48769062887878234, + "moe_router": 0.024961428483772106, + "norm_elementwise": 0.050133562889162694, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.015854890695544816, + "representativeness": { + "smd": { + "decode_batch_size": 2.1353453919214704, + "mode_FULL": 0.1374340854376679, + "mode_NONE": -0.1374340854376676, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.13478349167720893, + "scheduled_tokens": -0.11208567974807711 + }, + "valid": false + }, + "shares": { + "attention": 0.417531453105595, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0038280359471430728, + "moe_gemm": 0.48769062887878234, + "moe_router": 0.024961428483772106, + "norm_elementwise": 0.050133562889162694, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 633.0599999999998, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 495.7439999999989, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 490.0149999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.04, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.791999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.882, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.047, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850575822210846.pt.trace.json.gz" + } + ], + "P06-C10-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3486689259270977, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9915388739029576, + "mode_shares": { + "NONE": { + "attention": 0.3486689259270977, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6093714248402938, + "moe_router": 0.003167635514417327, + "norm_elementwise": 0.030330887621148654, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.008461126097042464, + "representativeness": { + "smd": { + "decode_batch_size": -6.093102704587387, + "mode_FULL": -5.409862359976508, + "mode_NONE": 5.409862359976508, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 5.577249352958417, + "scheduled_tokens": 6.977467251638274 + }, + "valid": false + }, + "shares": { + "attention": 0.3486689259270977, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6093714248402938, + "moe_router": 0.003167635514417327, + "norm_elementwise": 0.030330887621148654, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 18628.55999999999, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 9220.226999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 8416.866999999997, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 397.24800000000005, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 195.808, + "name": "triton_red_fused_1" + }, + { + "duration_us": 109.279, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.024, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849962666875300.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.5719043088501389, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9916731652872516, + "mode_shares": { + "FULL": { + "attention": 0.5719043088501389, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002016827326928409, + "moe_gemm": 0.37609168384078084, + "moe_router": 0.013852953759290538, + "norm_elementwise": 0.02780739151011282, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.008326834712748461, + "representativeness": { + "smd": { + "decode_batch_size": 0.34266016260318394, + "mode_FULL": 0.36945401482766393, + "mode_NONE": -0.36945401482766393, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.36861766414224534, + "scheduled_tokens": -0.3264149440726429 + }, + "valid": false + }, + "shares": { + "attention": 0.5719043088501389, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002016827326928409, + "moe_gemm": 0.37609168384078084, + "moe_router": 0.013852953759290538, + "norm_elementwise": 0.02780739151011282, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 694.7229999999995, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 546.8509999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 520.2939999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.320999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.463, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.448999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.783999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850011262475600.pt.trace.json.gz" + } + ], + "P06-C11-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.31298009715328673, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9837880586014649, + "mode_shares": { + "FULL": { + "attention": 0.31298009715328673, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004023145886862297, + "moe_gemm": 0.5886224548783123, + "moe_router": 0.026592515275261283, + "norm_elementwise": 0.051569845407742246, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.01621194139853512, + "representativeness": { + "smd": { + "decode_batch_size": 2.2622779907056527, + "mode_FULL": 0.2570884920608091, + "mode_NONE": -0.25558416463536143, + "mode_PIECEWISE": -0.026892314000411845, + "prefill_fraction": -0.2554724535338412, + "scheduled_tokens": -0.22611835970475505 + }, + "valid": false + }, + "shares": { + "attention": 0.31298009715328673, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004023145886862297, + "moe_gemm": 0.5886224548783123, + "moe_router": 0.026592515275261283, + "norm_elementwise": 0.051569845407742246, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 633.8630000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 494.91799999999904, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 492.54099999999926, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.038999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.568, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.584, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.719999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850544703259931.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.285815805689067, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9795145559108047, + "mode_shares": { + "FULL": { + "attention": 0.285815805689067, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004890634397475269, + "moe_gemm": 0.594753338701119, + "moe_router": 0.03200944848645275, + "norm_elementwise": 0.062045328636690614, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.020485444089195285, + "representativeness": { + "smd": { + "decode_batch_size": -0.9097707612085933, + "mode_FULL": 0.2570884920608091, + "mode_NONE": -0.25558416463536143, + "mode_PIECEWISE": -0.026892314000411845, + "prefill_fraction": -0.2554724535338412, + "scheduled_tokens": -0.2591423482439635 + }, + "valid": false + }, + "shares": { + "attention": 0.285815805689067, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004890634397475269, + "moe_gemm": 0.594753338701119, + "moe_router": 0.03200944848645275, + "norm_elementwise": 0.062045328636690614, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 616.3130000000001, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 518.4359999999996, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 500.8009999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.976999999999997, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.831999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.774999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.142999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783850576213406484.pt.trace.json.gz" + } + ], + "P06-C11-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.5565323350091496, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9922271393697571, + "mode_shares": { + "FULL": { + "attention": 0.5565323350091496, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019180859437863745, + "moe_gemm": 0.39491275272816917, + "moe_router": 0.012754982760746365, + "norm_elementwise": 0.026108982927905665, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.0077728606302428565, + "representativeness": { + "smd": { + "decode_batch_size": 0.722221197086119, + "mode_FULL": 0.7873442980499663, + "mode_NONE": -0.7857620597700242, + "mode_PIECEWISE": -0.038110796698335295, + "prefill_fraction": -0.7857531825398673, + "scheduled_tokens": -0.7401743374164678 + }, + "valid": false + }, + "shares": { + "attention": 0.5565323350091496, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019180859437863745, + "moe_gemm": 0.39491275272816917, + "moe_router": 0.012754982760746365, + "norm_elementwise": 0.026108982927905665, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 699.4259999999995, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 542.1849999999998, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 520.0949999999991, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.345000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.728, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.902999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.945, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849955389093025.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.5577097612622229, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9919246666563806, + "mode_shares": { + "FULL": { + "attention": 0.5577097612622229, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001982724387439752, + "moe_gemm": 0.3920744686718595, + "moe_router": 0.013242542857418157, + "norm_elementwise": 0.026915169477440156, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.008075333343619472, + "representativeness": { + "smd": { + "decode_batch_size": 0.722221197086119, + "mode_FULL": 0.7873442980499663, + "mode_NONE": -0.7857620597700242, + "mode_PIECEWISE": -0.038110796698335295, + "prefill_fraction": -0.7857531825398673, + "scheduled_tokens": -0.7401743374164678 + }, + "valid": false + }, + "shares": { + "attention": 0.5577097612622229, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001982724387439752, + "moe_gemm": 0.3920744686718595, + "moe_router": 0.013242542857418157, + "norm_elementwise": 0.026915169477440156, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 705.1269999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 544.0159999999997, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 521.0479999999991, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.279999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.824, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.191999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.200999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849988477975368.pt.trace.json.gz" + } + ], + "P07-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.306761436001029, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9841089106220091, + "mode_shares": { + "FULL": { + "attention": 0.306761436001029, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003944802403864725, + "moe_gemm": 0.5964090291561805, + "moe_router": 0.02562456323133302, + "norm_elementwise": 0.051369079829601885, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.015891089377990907, + "representativeness": { + "smd": { + "decode_batch_size": -0.18368608035893982, + "mode_FULL": 0.1535286967084372, + "mode_NONE": -0.15352869670843752, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1535266720575854, + "scheduled_tokens": -0.13391206652423035 + }, + "valid": true + }, + "shares": { + "attention": 0.306761436001029, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003944802403864725, + "moe_gemm": 0.5964090291561805, + "moe_router": 0.02562456323133302, + "norm_elementwise": 0.051369079829601885, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 606.694, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 538.9419999999992, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 496.9549999999989, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.128, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.04, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.56, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.304, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836530765988779.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.2666065580814879, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9860189886811974, + "mode_shares": { + "FULL": { + "attention": 0.2666065580814879, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034992714146667428, + "moe_gemm": 0.6472552645315595, + "moe_router": 0.02277286906470575, + "norm_elementwise": 0.04588502558877749, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.013981011318802606, + "representativeness": { + "smd": { + "decode_batch_size": 3.069910871828644, + "mode_FULL": 0.1535286967084372, + "mode_NONE": -0.15352869670843752, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1535266720575854, + "scheduled_tokens": -0.11608746652035899 + }, + "valid": false + }, + "shares": { + "attention": 0.2666065580814879, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034992714146667428, + "moe_gemm": 0.6472552645315595, + "moe_router": 0.02277286906470575, + "norm_elementwise": 0.04588502558877749, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 621.5659999999995, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 529.2479999999987, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 510.27699999999976, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.607000000000003, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.184, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.230999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.817, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836562828610579.pt.trace.json.gz" + } + ], + "P08-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.3248872069635257, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9877052895017178, + "mode_shares": { + "FULL": { + "attention": 0.3248872069635257, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031072028443770152, + "moe_gemm": 0.5984635601986847, + "moe_router": 0.020260919566213807, + "norm_elementwise": 0.04098639992891659, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.012294710498282197, + "representativeness": { + "smd": { + "decode_batch_size": 1.0901560791930118, + "mode_FULL": 0.16434181266464418, + "mode_NONE": -0.16434181266464443, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.16432997461824633, + "scheduled_tokens": -0.1413297101061453 + }, + "valid": false + }, + "shares": { + "attention": 0.3248872069635257, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031072028443770152, + "moe_gemm": 0.5984635601986847, + "moe_router": 0.020260919566213807, + "norm_elementwise": 0.04098639992891659, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 654.3379999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 552.4199999999996, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 513.1979999999996, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.089000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.015999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.127999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.719, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836530860694701.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.3669917099645223, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9854450802968766, + "mode_shares": { + "FULL": { + "attention": 0.3669917099645223, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003724234761488658, + "moe_gemm": 0.5428235608185195, + "moe_router": 0.024088695306046685, + "norm_elementwise": 0.04781687944629951, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.014554919703123427, + "representativeness": { + "smd": { + "decode_batch_size": -1.4164397504981734, + "mode_FULL": 0.16434181266464418, + "mode_NONE": -0.16434181266464443, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.16432997461824633, + "scheduled_tokens": -0.19075947310069616 + }, + "valid": false + }, + "shares": { + "attention": 0.3669917099645223, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003724234761488658, + "moe_gemm": 0.5428235608185195, + "moe_router": 0.024088695306046685, + "norm_elementwise": 0.04781687944629951, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 632.7909999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 540.1919999999998, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.9399999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.673000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.665, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.713, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.878999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836563379173349.pt.trace.json.gz" + } + ], + "P09-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.08835627115447606, + "attention_mixed": 0.11385037570991731, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.988346330660954, + "mode_shares": { + "FULL": { + "attention": 0.280249688903593, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004209638462393143, + "moe_gemm": 0.6254634471394533, + "moe_router": 0.02523969717175661, + "norm_elementwise": 0.0493304218340256, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.14806901527656063, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7989631873012853, + "moe_router": 0.006717474937844798, + "norm_elementwise": 0.036240204978569675, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.18871401371634564, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0022585830433693877, + "moe_gemm": 0.7546316751137306, + "moe_router": 0.012749732426996046, + "norm_elementwise": 0.031927806640493814, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 5, + "NONE": 1, + "PIECEWISE": 2 + }, + "other_share": 0.011653669339046026, + "representativeness": { + "smd": { + "decode_batch_size": 2.5579606107986748, + "mode_FULL": -0.6775421426296558, + "mode_NONE": 0.2503307644752007, + "mode_PIECEWISE": 0.5907929693818461, + "prefill_fraction": 0.6661033678566376, + "scheduled_tokens": 0.05206272885676596 + }, + "valid": false + }, + "shares": { + "attention": 0.20220664686439338, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002019814379523439, + "moe_gemm": 0.7306681072672004, + "moe_router": 0.014406944379520907, + "norm_elementwise": 0.039044817770315786, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1078.7899999999981, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 718.971000000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 663.9299999999997, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.880000000000003, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.017, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.112, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.974999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848263375341068.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.09568605186600879, + "attention_mixed": 0.025924532025613695, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9819166279177437, + "mode_shares": { + "FULL": { + "attention": 0.14040386061386476, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.006289384167983417, + "moe_gemm": 0.7278572814656825, + "moe_router": 0.03517706269456804, + "norm_elementwise": 0.06867748249282202, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.08139719907850929, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0022138539380632135, + "moe_gemm": 0.8578037652991323, + "moe_router": 0.01403539873195149, + "norm_elementwise": 0.033980351851418185, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "other_share": 0.01808337208225627, + "representativeness": { + "smd": { + "decode_batch_size": -0.19954189900373331, + "mode_FULL": -0.10469513477046205, + "mode_NONE": -0.32861907697425397, + "mode_PIECEWISE": 0.29736821980293143, + "prefill_fraction": 0.10420387771831544, + "scheduled_tokens": -0.23288793337699637 + }, + "valid": false + }, + "shares": { + "attention": 0.12161058389162249, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004991351616949132, + "moe_gemm": 0.7692444768470234, + "moe_router": 0.028443566325178558, + "norm_elementwise": 0.05762664923697027, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 667.7419999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 523.011999999999, + "name": "triton_red_fused_2" + }, + { + "duration_us": 498.98300000000086, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 20.736, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.591, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.645, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.36, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848297311321503.pt.trace.json.gz" + } + ], + "P09-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.24676580346741184, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9902796451206958, + "mode_shares": { + "NONE": { + "attention": 0.24676580346741184, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7048192593007057, + "moe_router": 0.0036328436089412205, + "norm_elementwise": 0.035061738743637105, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.009720354879304175, + "representativeness": { + "smd": { + "decode_batch_size": 1.259998751515166, + "mode_FULL": -1.0890519340869804, + "mode_NONE": 1.0890519340869804, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.1263920173520792, + "scheduled_tokens": 1.3450157218366598 + }, + "valid": false + }, + "shares": { + "attention": 0.24676580346741184, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7048192593007057, + "moe_router": 0.0036328436089412205, + "norm_elementwise": 0.035061738743637105, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 18507.434, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 9166.28299999999, + "name": "triton_red_fused_2" + }, + { + "duration_us": 8451.682000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 396.608, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 195.36100000000002, + "name": "triton_red_fused_1" + }, + { + "duration_us": 75.745, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.616, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847497572190687.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.27787874681050084, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9906768293176031, + "mode_shares": { + "NONE": { + "attention": 0.27787874681050084, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6757237742248838, + "moe_router": 0.0034783848147879478, + "norm_elementwise": 0.03359592346743046, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.009323170682396877, + "representativeness": { + "smd": { + "decode_batch_size": -0.2365580579355552, + "mode_FULL": -1.0890519340869804, + "mode_NONE": 1.0890519340869804, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.1326772391141862, + "scheduled_tokens": 1.3450157218366598 + }, + "valid": false + }, + "shares": { + "attention": 0.27787874681050084, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6757237742248838, + "moe_router": 0.0034783848147879478, + "norm_elementwise": 0.03359592346743046, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 18504.999000000003, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 9192.093999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 8458.98800000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 398.754, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 195.68099999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 84.479, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.392, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847556128452812.pt.trace.json.gz" + } + ], + "P10-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.3087850643091725, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.985252225101006, + "mode_shares": { + "FULL": { + "attention": 0.3087850643091725, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005163136163815447, + "moe_gemm": 0.5777999387887653, + "moe_router": 0.02008575072504131, + "norm_elementwise": 0.07341833511421153, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.014747774898993987, + "representativeness": { + "smd": { + "decode_batch_size": 4.655696267017126, + "mode_FULL": 0.17677689026422408, + "mode_NONE": -0.1682723158645361, + "mode_PIECEWISE": -0.05341411394958018, + "prefill_fraction": -0.12905587442539998, + "scheduled_tokens": -0.09951290348812954 + }, + "valid": false + }, + "shares": { + "attention": 0.3087850643091725, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005163136163815447, + "moe_gemm": 0.5777999387887653, + "moe_router": 0.02008575072504131, + "norm_elementwise": 0.07341833511421153, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 583.5649999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 466.8209999999995, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.617, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.536, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.231999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.049, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849466906401944.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.3033378915990537, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9736215707793747, + "mode_shares": { + "FULL": { + "attention": 0.3033378915990537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008423712471036498, + "moe_gemm": 0.5030357454817748, + "moe_router": 0.03618969859772223, + "norm_elementwise": 0.12263452262978754, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.026378429220625322, + "representativeness": { + "smd": { + "decode_batch_size": -0.5055344205281964, + "mode_FULL": 0.17677689026422408, + "mode_NONE": -0.1682723158645361, + "mode_PIECEWISE": -0.05341411394958018, + "prefill_fraction": -0.12905587442539998, + "scheduled_tokens": -0.10506250419671144 + }, + "valid": false + }, + "shares": { + "attention": 0.3033378915990537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008423712471036498, + "moe_gemm": 0.5030357454817748, + "moe_router": 0.03618969859772223, + "norm_elementwise": 0.12263452262978754, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 590.9459999999992, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 467.6929999999989, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.272, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.152, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.63, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849498465727706.pt.trace.json.gz" + } + ], + "P10-C00-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.7471257503417844, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9942106688319626, + "mode_shares": { + "FULL": { + "attention": 0.7471257503417844, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0013626029325187864, + "moe_gemm": 0.21782046124566493, + "moe_router": 0.009663599861937516, + "norm_elementwise": 0.01823825445005707, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.005789331168037346, + "representativeness": { + "smd": { + "decode_batch_size": 3.613259323224499, + "mode_FULL": 0.6120018940246724, + "mode_NONE": -0.5981417765928579, + "mode_PIECEWISE": -0.10976470169608525, + "prefill_fraction": -0.6117521895228502, + "scheduled_tokens": -0.5755060211990467 + }, + "valid": false + }, + "shares": { + "attention": 0.7471257503417844, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0013626029325187864, + "moe_gemm": 0.21782046124566493, + "moe_router": 0.009663599861937516, + "norm_elementwise": 0.01823825445005707, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 637.9129999999991, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 513.3399999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 484.63699999999847, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.874, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.408, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.879999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.305, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848740596577354.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.2679713777414123, + "attention_mixed": 0.3408621380286316, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9943826035261398, + "mode_shares": { + "FULL": { + "attention": 0.6506807095871076, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001469412163770193, + "moe_gemm": 0.3120351814822058, + "moe_router": 0.01012812440986268, + "norm_elementwise": 0.0195308311121687, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.5795322961533458, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.39390960062584945, + "moe_router": 0.0023577446138654075, + "norm_elementwise": 0.01895990875545615, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "NONE": 1 + }, + "other_share": 0.005617396473860218, + "representativeness": { + "smd": { + "decode_batch_size": 28.040640400381754, + "mode_FULL": 0.09182576274871186, + "mode_NONE": -0.07574082366962798, + "mode_PIECEWISE": -0.10976470169608525, + "prefill_fraction": -0.09431170009009414, + "scheduled_tokens": -0.3287773474224777 + }, + "valid": false + }, + "shares": { + "attention": 0.6088335157700439, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0006051514916514902, + "moe_gemm": 0.36019106467352013, + "moe_router": 0.005557838529095687, + "norm_elementwise": 0.019195033061828647, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1568.3549999999968, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 937.5360000000019, + "name": "triton_red_fused_2" + }, + { + "duration_us": 870.9730000000023, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 32.38400000000001, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 25.343999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 21.376, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.167999999999997, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848774099957127.pt.trace.json.gz" + } + ], + "P10-C01-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.00827308232619546, + "attention_mixed": 0.0, + "attention_prefill": 0.3878891624260789 + }, + "classifiable_fraction": 0.9920412642157855, + "mode_shares": { + "FULL": { + "attention": 0.24862667112394968, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009001841356572508, + "moe_gemm": 0.5445479195579913, + "moe_router": 0.038657214941041004, + "norm_elementwise": 0.1306799512400259, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.3919918323373639, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5695375481303059, + "moe_router": 0.004010888023206571, + "norm_elementwise": 0.027034253877755012, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.47297055300102214, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001160843915130027, + "moe_gemm": 0.49362762525763404, + "moe_router": 0.006709457074185478, + "norm_elementwise": 0.0196235305449198, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 4, + "NONE": 3, + "PIECEWISE": 1 + }, + "other_share": 0.007958735784214513, + "representativeness": { + "smd": { + "decode_batch_size": -1.315856674395644, + "mode_FULL": -1.1873139780953184, + "mode_NONE": 0.904151999012294, + "mode_PIECEWISE": 0.4906969552687828, + "prefill_fraction": 1.2174104966136614, + "scheduled_tokens": 1.0338890364922466 + }, + "valid": false + }, + "shares": { + "attention": 0.39616224475227435, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0004277067707998268, + "moe_gemm": 0.5603247586456352, + "moe_router": 0.005461699212612277, + "norm_elementwise": 0.029664854834463707, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 2284.6260000000075, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1233.3779999999972, + "name": "triton_red_fused_2" + }, + { + "duration_us": 986.3160000000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 48.257000000000005, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 27.777000000000005, + "name": "triton_red_fused_1" + }, + { + "duration_us": 23.712000000000003, + "name": "_compute_slot_mapping_kernel" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836534095748867.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.0063747668464849165, + "attention_mixed": 0.15479011673996892, + "attention_prefill": 0.3931831065712863 + }, + "classifiable_fraction": 0.9945037369811328, + "mode_shares": { + "FULL": { + "attention": 0.3498249631356147, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.006205452478989364, + "moe_gemm": 0.505172700292536, + "moe_router": 0.026126956515180367, + "norm_elementwise": 0.09328180537085141, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.5581441347693616, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.4145765883773621, + "moe_router": 0.0028266883400090486, + "norm_elementwise": 0.019214171797428074, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 3, + "NONE": 5 + }, + "other_share": 0.005496263018867143, + "representativeness": { + "smd": { + "decode_batch_size": -0.21484041020694733, + "mode_FULL": -1.5472770324498917, + "mode_NONE": 1.55520654375562, + "mode_PIECEWISE": -0.055941052979459574, + "prefill_fraction": 1.5826586640498834, + "scheduled_tokens": 1.5981896236261708 + }, + "valid": false + }, + "shares": { + "attention": 0.5543479901577401, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00011308030272031699, + "moe_gemm": 0.41622749712178303, + "moe_router": 0.0032512829022321816, + "norm_elementwise": 0.020563886496657274, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 3222.5210000000084, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1708.5829999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 1400.3889999999992, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 68.76799999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 37.952, + "name": "triton_red_fused_1" + }, + { + "duration_us": 29.728999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 10.174999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783836573272002789.pt.trace.json.gz" + } + ], + "P10-C10-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.2843977937526034, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9824438147308845, + "mode_shares": { + "FULL": { + "attention": 0.2843977937526034, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00405010296319307, + "moe_gemm": 0.6130100529210382, + "moe_router": 0.027604734811478317, + "norm_elementwise": 0.05338113028257156, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.017556185269115537, + "representativeness": { + "smd": { + "decode_batch_size": 5.144106307912562, + "mode_FULL": 0.18130730690818112, + "mode_NONE": -0.1757536966781094, + "mode_PIECEWISE": -0.04385290377913636, + "prefill_fraction": -0.14497536337962438, + "scheduled_tokens": -0.11030736227232732 + }, + "valid": false + }, + "shares": { + "attention": 0.2843977937526034, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00405010296319307, + "moe_gemm": 0.6130100529210382, + "moe_router": 0.027604734811478317, + "norm_elementwise": 0.05338113028257156, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 629.6319999999995, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 515.3909999999998, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 499.33799999999894, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.209, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.758, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.359999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 11.105, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838883199166770.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.3547933594800296, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9860169102260189, + "mode_shares": { + "FULL": { + "attention": 0.3547933594800296, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004769401324813265, + "moe_gemm": 0.534184946336729, + "moe_router": 0.023130978256460693, + "norm_elementwise": 0.06913822482798647, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.013983089773981138, + "representativeness": { + "smd": { + "decode_batch_size": 2.258805078083754, + "mode_FULL": 0.18130730690818112, + "mode_NONE": -0.1757536966781094, + "mode_PIECEWISE": -0.04385290377913636, + "prefill_fraction": -0.14497536337962438, + "scheduled_tokens": -0.11500242829780874 + }, + "valid": false + }, + "shares": { + "attention": 0.3547933594800296, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004769401324813265, + "moe_gemm": 0.534184946336729, + "moe_router": 0.023130978256460693, + "norm_elementwise": 0.06913822482798647, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 620.6479999999996, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 473.21299999999843, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.232, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.408, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.551999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 9.921, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838915336860515.pt.trace.json.gz" + } + ], + "P10-C10-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.041504399483772915, + "attention_mixed": 0.37939509732011933, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9925214098201124, + "mode_shares": { + "FULL": { + "attention": 0.6458307859040855, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001372377547939455, + "moe_gemm": 0.3195152761398567, + "moe_router": 0.008996381149482991, + "norm_elementwise": 0.01843126133535932, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.4054514900827006, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5564249519780634, + "moe_router": 0.0027817925221303996, + "norm_elementwise": 0.027751594727275442, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 4, + "NONE": 4 + }, + "other_share": 0.007478590179887577, + "representativeness": { + "smd": { + "decode_batch_size": 3.1738421486850954, + "mode_FULL": -0.7499688060387298, + "mode_NONE": 0.7499688060387298, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 0.7578037492611197, + "scheduled_tokens": 0.774707112934012 + }, + "valid": false + }, + "shares": { + "attention": 0.4208994968038922, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 8.81960216754039e-05, + "moe_gemm": 0.5411999208365155, + "moe_router": 0.0031811738688339005, + "norm_elementwise": 0.027152622289195533, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 9201.402000000015, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 4652.703000000019, + "name": "triton_red_fused_2" + }, + { + "duration_us": 3948.9770000000117, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 195.13799999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 99.81200000000001, + "name": "triton_red_fused_1" + }, + { + "duration_us": 50.913000000000004, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.383999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838157121058985.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.6315023261702531, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9935622263304319, + "mode_shares": { + "FULL": { + "attention": 0.6315023261702531, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015181468521158754, + "moe_gemm": 0.33046383055037754, + "moe_router": 0.009848784918527526, + "norm_elementwise": 0.020229137839157803, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.006437773669568139, + "representativeness": { + "smd": { + "decode_batch_size": 5.3541833543837525, + "mode_FULL": 0.6097616511341267, + "mode_NONE": -0.6097616511341267, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.6046345482385058, + "scheduled_tokens": -0.573070382452099 + }, + "valid": false + }, + "shares": { + "attention": 0.6315023261702531, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015181468521158754, + "moe_gemm": 0.33046383055037754, + "moe_router": 0.009848784918527526, + "norm_elementwise": 0.020229137839157803, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 707.0429999999991, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 532.921999999999, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 508.57699999999977, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.582, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.630999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.838999999999997, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.496, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783838196792063539.pt.trace.json.gz" + } + ], + "P10-C11-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.0073359442575772465, + "attention_mixed": 0.3947369166548715, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9927989737523812, + "mode_shares": { + "FULL": { + "attention": 0.5045602577486049, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002961104894369635, + "moe_gemm": 0.4165724542443657, + "moe_router": 0.022869450932386057, + "norm_elementwise": 0.040255165110812396, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.40056078314413623, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.5626074540524005, + "moe_router": 0.0039472613290762685, + "norm_elementwise": 0.02576580936708546, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 7 + }, + "other_share": 0.0072010262476187884, + "representativeness": { + "smd": { + "decode_batch_size": 6.315536403064479, + "mode_FULL": -3.018625091009808, + "mode_NONE": 3.026618811087864, + "mode_PIECEWISE": -0.035519441883670656, + "prefill_fraction": 3.0955670336388525, + "scheduled_tokens": 2.775871694111727 + }, + "valid": false + }, + "shares": { + "attention": 0.40207286091244876, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 4.305234134543738e-05, + "moe_gemm": 0.560484209897129, + "moe_router": 0.004222376394183816, + "norm_elementwise": 0.025976474207274176, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4037.0970000000043, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2121.486000000002, + "name": "triton_red_fused_2" + }, + { + "duration_us": 1910.3059999999973, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 85.40900000000002, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 45.473000000000006, + "name": "triton_red_fused_1" + }, + { + "duration_us": 32.478, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.264999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848267170744798.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.2585417693591813, + "attention_mixed": 0.16843822503094136, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9906404216693048, + "mode_shares": { + "FULL": { + "attention": 0.539559922335815, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0024951724086338385, + "moe_gemm": 0.3941567666483815, + "moe_router": 0.019165126164455013, + "norm_elementwise": 0.03398857365164246, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.3234044301174878, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6347896864036598, + "moe_router": 0.004354761640020028, + "norm_elementwise": 0.02926443812587131, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "NONE": 1 + }, + "other_share": 0.009359578330695272, + "representativeness": { + "smd": { + "decode_batch_size": 9.434244196050145, + "mode_FULL": -0.33301959680175525, + "mode_NONE": 0.33591397118576066, + "mode_PIECEWISE": -0.035519441883670656, + "prefill_fraction": 0.3608423156901009, + "scheduled_tokens": 0.38343744134224045 + }, + "valid": false + }, + "shares": { + "attention": 0.42697999439012263, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001195615653941949, + "moe_gemm": 0.5194852350087792, + "moe_router": 0.01145146710403092, + "norm_elementwise": 0.03152810951243005, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1147.1800000000003, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 753.7689999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 745.4559999999998, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 24.256, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 21.409000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.959, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.647999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783848309990988798.pt.trace.json.gz" + } + ], + "P10-C11-saturation": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.6985892713346192, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9963781377235988, + "mode_shares": { + "NONE": { + "attention": 0.6985892713346192, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.28271066266772416, + "moe_router": 0.0019371533585066747, + "norm_elementwise": 0.013141050362748694, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "other_share": 0.0036218622764011715, + "representativeness": { + "smd": { + "decode_batch_size": -1.3424985614091869, + "mode_FULL": -1.2678999201712284, + "mode_NONE": 1.2792167933527154, + "mode_PIECEWISE": -0.09386487859221887, + "prefill_fraction": 1.2748539687576104, + "scheduled_tokens": 1.3073041498709979 + }, + "valid": false + }, + "shares": { + "attention": 0.6985892713346192, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.28271066266772416, + "moe_router": 0.0019371533585066747, + "norm_elementwise": 0.013141050362748694, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 4839.693000000004, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 2523.400000000001, + "name": "triton_red_fused_2" + }, + { + "duration_us": 2258.809, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 102.399, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 53.057, + "name": "triton_red_fused_1" + }, + { + "duration_us": 34.175000000000004, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 10.751999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847496152835877.pt.trace.json.gz" + }, + { + "attention_subshares": { + "attention_decode": 0.623103669396033, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9937738750743762, + "mode_shares": { + "FULL": { + "attention": 0.623103669396033, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0014430357241519828, + "moe_gemm": 0.3399197180195412, + "moe_router": 0.009702122794559957, + "norm_elementwise": 0.019605329140090146, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "other_share": 0.006226124925623746, + "representativeness": { + "smd": { + "decode_batch_size": 15.677343907178292, + "mode_FULL": 1.5739447284884216, + "mode_NONE": -1.5600204796984336, + "mode_PIECEWISE": -0.09386487859221887, + "prefill_fraction": -1.5730835237401557, + "scheduled_tokens": -1.5227269874253568 + }, + "valid": false + }, + "shares": { + "attention": 0.623103669396033, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0014430357241519828, + "moe_gemm": 0.3399197180195412, + "moe_router": 0.009702122794559957, + "norm_elementwise": 0.019605329140090146, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 701.6779999999983, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 536.7749999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.6289999999995, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.16, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.888, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.646999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.720999999999998, + "name": "triton_red_fused_1" + } + ], + "trace": "dp0_pp0_tp0_dcp0_ep0_rank0.1783847541459857339.pt.trace.json.gz" + } + ] + }, + "analysis_seed": 20260714, + "bootstrap_resamples": 100000, + "hypothesis": { + "H1a": "INCONCLUSIVE", + "H1b": "PASS", + "compound": "PARTIAL", + "h1b_hits": [ + { + "available": true, + "ci95_high": 0.23494534783397408, + "ci95_low": 0.2253590527218157, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.2301483528866904, + "simultaneous_high": 0.23683442206671917, + "simultaneous_low": 0.22345003461028415 + }, + { + "available": true, + "ci95_high": 0.3588237662068493, + "ci95_low": 0.34996301147907066, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.3543970834375052, + "simultaneous_high": 0.3605880124391376, + "simultaneous_low": 0.34826154238421436 + }, + { + "available": true, + "ci95_high": 0.07258101928545771, + "ci95_low": 0.06107199918894719, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.0667290414201105, + "simultaneous_high": 0.07492451384312424, + "simultaneous_low": 0.058870086990988876 + }, + { + "available": true, + "ci95_high": 0.4001054362524198, + "ci95_low": 0.3922157126916775, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.39616418510901036, + "simultaneous_high": 0.4016625621683738, + "simultaneous_low": 0.3906536566040832 + }, + { + "available": true, + "ci95_high": 0.08855777444708128, + "ci95_low": 0.027415277773332525, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.05565211993295673, + "simultaneous_high": 0.10341073102256013, + "simultaneous_low": 0.020097017796153225 + }, + { + "available": true, + "ci95_high": 0.4561736305824604, + "ci95_low": 0.43955479496868705, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.4478716685574937, + "simultaneous_high": 0.459713780733712, + "simultaneous_low": 0.43635435003093426 + }, + { + "available": true, + "ci95_high": 0.08668052314167546, + "ci95_low": 0.025823274229141942, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.05398526806052646, + "simultaneous_high": 0.10213016668480329, + "simultaneous_low": 0.018683381445978635 + }, + { + "available": true, + "ci95_high": 0.45618061702156487, + "ci95_low": 0.4395710696329076, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.4478716685574937, + "simultaneous_high": 0.4594884290583298, + "simultaneous_low": 0.4363807227134572 + } + ], + "logical_asymmetry": "A-P3-7 permits existential confirmation but incomplete coverage forbids refutation", + "refutation_allowed": false + }, + "matrix": { + "clean_window_failures": 0, + "complete_cells": [ + "P01/C00", + "P01/C01", + "P01/C10", + "P01/C11", + "P02/C00", + "P03/C00", + "P03/C01", + "P03/C10", + "P04/C00", + "P06/C00", + "P06/C01", + "P06/C10", + "P06/C11", + "P07/C00", + "P08/C00", + "P09/C00", + "P10/C00", + "P10/C01", + "P10/C10", + "P10/C11" + ], + "completed_c00_patterns": [ + "P01", + "P02", + "P03", + "P04", + "P06", + "P07", + "P08", + "P09", + "P10" + ], + "confirmation_runs": 0, + "drain_quarantined_runs": 0, + "gpu_hours_total": 14.025875418755744, + "missing_cells": [ + "P03/C11", + "P05/C00", + "P10/C00-TP2", + "P11/C00" + ], + "primary_runs": 40, + "trace_files": 72 + }, + "operational_findings": { + "failure_boundary": { + "clean_failures": 0, + "excluded_failure_kinds": { + "ServerDisconnectedError": 2 + }, + "excluded_window_failures": 2, + "run_id": "P01-C01-moderate" + }, + "layer2_sampling": { + "classifiable_fraction_min": 0.9704584620048622, + "completed_c00_moderate_patterns": 9, + "evaluable_c00_moderate_patterns": 1, + "invalid_windows": 16 + }, + "long_context_drain": { + "amended_budget_seconds": 600, + "drain_seconds": 288.6191079240234, + "quarantined": false, + "run_id": "P10-C01-saturation" + }, + "p10_tp2_non_stabilization": { + "accepted_measurement": false, + "comparison": "all synthetic pattern runs passed their applicable registered warm-up gates; orchestrator adjudication", + "mean_scheduled_token_throughput": 16156.333333333334, + "normalized_drift": 0.367639109533929, + "normalized_drift_limit": 0.1, + "passes": false, + "same_wave_synthetic_warmup_completions": { + "P03-C11": 102, + "P11-C00": 512 + }, + "scheduled_token_throughput": [ + 18022.4, + 16384.0, + 14062.6 + ], + "scheduled_tokens": [ + 90112, + 81920, + 70313 + ], + "slope_tokens_per_second_squared": -395.98000000000013, + "status": "PATTERN_CONDITIONED_OPERATIONAL_FINDING", + "step_counts": [ + 11, + 10, + 16 + ], + "warmup_completions": 17 + } + }, + "operator_mode_segments": { + "P01-C00-moderate": { + "PIECEWISE": { + "other_share": 0.009442586767856639, + "shares": { + "attention": 0.14059860036997104, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002034731217402245, + "moe_gemm": 0.8039268095204102, + "moe_router": 0.011615692688783654, + "norm_elementwise": 0.03238157943557617, + "sampler": 0.0 + }, + "steps": 9 + } + }, + "P01-C00-saturation": { + "FULL": { + "other_share": 0.008640959425607062, + "shares": { + "attention": 0.24548760946422593, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0018953885436714586, + "moe_gemm": 0.6964242752814832, + "moe_router": 0.01232208790526271, + "norm_elementwise": 0.035229679379749704, + "sampler": 0.0 + }, + "steps": 9 + } + }, + "P02-C00-moderate": { + "FULL": { + "other_share": 0.010310329169884497, + "shares": { + "attention": 0.2078704033504826, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025993696985651725, + "moe_gemm": 0.7238420167249907, + "moe_router": 0.01747338449479615, + "norm_elementwise": 0.03790449656128083, + "sampler": 0.0 + }, + "steps": 14 + } + }, + "P02-C00-saturation": { + "FULL": { + "other_share": 0.007432424187256434, + "shares": { + "attention": 0.3275775913752123, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0016046133640397883, + "moe_gemm": 0.6231403899130352, + "moe_router": 0.010494318932798818, + "norm_elementwise": 0.029750662227657432, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P03-C00-moderate": { + "FULL": { + "other_share": 0.02536523149315264, + "shares": { + "attention": 0.23362801063108302, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00817304104316153, + "moe_gemm": 0.5766381826618499, + "moe_router": 0.03482203360681858, + "norm_elementwise": 0.12137350056393428, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P03-C00-saturation": { + "NONE": { + "other_share": 0.008588238863572845, + "shares": { + "attention": 0.3425157512172391, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147796482452442, + "moe_router": 0.0032160079024607783, + "norm_elementwise": 0.03090035377148312, + "sampler": 0.0 + }, + "steps": 9 + } + }, + "P04-C00-moderate": { + "FULL": { + "other_share": 0.018960704531611056, + "shares": { + "attention": 0.4787340825464985, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004481448763560187, + "moe_gemm": 0.40647901207594744, + "moe_router": 0.03279818524077843, + "norm_elementwise": 0.058546566841604454, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P04-C00-saturation": { + "FULL": { + "other_share": 0.007202580412325669, + "shares": { + "attention": 0.6587156975804976, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017451569361964714, + "moe_gemm": 0.2978006885644627, + "moe_router": 0.0115658672042958, + "norm_elementwise": 0.022970009302221747, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P06-C00-moderate": { + "FULL": { + "other_share": 0.017142137497054708, + "shares": { + "attention": 0.33107685878416826, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004081391199666314, + "moe_gemm": 0.5650713814732576, + "moe_router": 0.028378480372589072, + "norm_elementwise": 0.05424975067326402, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P06-C00-saturation": { + "FULL": { + "other_share": 0.006199138768841979, + "shares": { + "attention": 0.5659533573255217, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015410952588019783, + "moe_gemm": 0.3933982154583866, + "moe_router": 0.010416018392470881, + "norm_elementwise": 0.022492174795976858, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P07-C00-moderate": { + "FULL": { + "other_share": 0.014869887132538163, + "shares": { + "attention": 0.2852930717632125, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003706604156207189, + "moe_gemm": 0.623593410175782, + "moe_router": 0.0240999362809864, + "norm_elementwise": 0.0484370904912738, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P08-C00-moderate": { + "FULL": { + "other_share": 0.013318198191355927, + "shares": { + "attention": 0.3439533361307738, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003386612631774027, + "moe_gemm": 0.5732681692206052, + "moe_router": 0.021994246442847046, + "norm_elementwise": 0.0440794373826441, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P09-C00-moderate": { + "FULL": { + "other_share": 0.018491839460411715, + "shares": { + "attention": 0.21168617810937532, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005229294671508251, + "moe_gemm": 0.675665021622912, + "moe_router": 0.030111781499347623, + "norm_elementwise": 0.05881588463644517, + "sampler": 0.0 + }, + "steps": 12 + } + }, + "P09-C00-saturation": { + "NONE": { + "other_share": 0.0095175085506292, + "shares": { + "attention": 0.26265552511434637, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6899598757335755, + "moe_router": 0.003553959807497156, + "norm_elementwise": 0.03431313079395171, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P10-C00-moderate": { + "FULL": { + "other_share": 0.01890606961699496, + "shares": { + "attention": 0.30683754279558073, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.006328886332379883, + "moe_gemm": 0.5510695811040863, + "moe_router": 0.025843377039614774, + "norm_elementwise": 0.09101454311134333, + "sampler": 0.0 + }, + "steps": 16 + } + }, + "P10-C00-saturation": { + "FULL": { + "other_share": 0.005959860267121688, + "shares": { + "attention": 0.7022397416038559, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001412312487823314, + "moe_gemm": 0.2616684675577343, + "moe_router": 0.009879791931057879, + "norm_elementwise": 0.01883982615240685, + "sampler": 0.0 + }, + "steps": 15 + } + } + }, + "operator_share_table": { + "P01": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.8242859924840175 + }, + { + "family": "attention", + "rank": 2, + "share": 0.12088271131609091 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.033822363286085066 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010309373428617204 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0010574407739066686 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.12088271131609091, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010574407739066686, + "moe_gemm": 0.8242859924840175, + "moe_router": 0.010309373428617204, + "norm_elementwise": 0.033822363286085066, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P01-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.11987238161795156, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9905881019946707, + "mode_shares": { + "NONE": { + "attention": 0.1007462687465188, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8471040152242396, + "moe_router": 0.008477215998495907, + "norm_elementwise": 0.034119643236757626, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.13605518456441312, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019444790834074754, + "moe_gemm": 0.8102627123318585, + "moe_router": 0.01080794700368473, + "norm_elementwise": 0.03163704570787867, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 3, + "PIECEWISE": 5 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.8271478398184453 + }, + { + "family": "attention", + "rank": 2, + "share": 0.11987238161795156 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.032774871259148075 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.009739725006365179 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0010532842927605177 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 2.4721524781047854, + "mode_FULL": -0.2325584995849942, + "mode_NONE": 0.3041456447839143, + "mode_PIECEWISE": -0.24468169875030038, + "prefill_fraction": 0.18099057277285577, + "scheduled_tokens": 0.4474608400278538 + }, + "valid": false + }, + "shares": { + "attention": 0.11987238161795156, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010532842927605177, + "moe_gemm": 0.8271478398184453, + "moe_router": 0.009739725006365179, + "norm_elementwise": 0.032774871259148075, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1567.9510000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 933.1070000000002, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 827.689, + "name": "triton_red_fused_2" + }, + { + "duration_us": 33.057, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.368999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 17.247, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.647999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.008867955248024707, + "attention_mixed": 0.11302508576620553, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.990127660582764, + "mode_shares": { + "FULL": { + "attention": 0.16315875281832684, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031741793017298413, + "moe_gemm": 0.7536332801624461, + "moe_router": 0.02179864485086512, + "norm_elementwise": 0.045434533244984684, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.09828250109360281, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8488043657595763, + "moe_router": 0.008302816259724543, + "norm_elementwise": 0.0348733229605455, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.1472178594517614, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002166218689199354, + "moe_gemm": 0.7946960932441856, + "moe_router": 0.012792489660640699, + "norm_elementwise": 0.033466283510587794, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 3, + "PIECEWISE": 4 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.8214241451495896 + }, + { + "family": "attention", + "rank": 2, + "share": 0.12189304101423025 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.034869855313022065 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010879021850869232 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0010615972550528194 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 1.4422983740600546, + "mode_FULL": 0.3594518697735859, + "mode_NONE": 0.3041456447839143, + "mode_PIECEWISE": -0.49598146699059914, + "prefill_fraction": -0.3699982864929203, + "scheduled_tokens": 0.1504241077629697 + }, + "valid": false + }, + "shares": { + "attention": 0.12189304101423025, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010615972550528194, + "moe_gemm": 0.8214241451495896, + "moe_router": 0.010879021850869232, + "norm_elementwise": 0.034869855313022065, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1449.5139999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 884.6050000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 826.8859999999994, + "name": "triton_red_fused_2" + }, + { + "duration_us": 30.943000000000005, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.431999999999995, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 17.441, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.328999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.778146620509425 + }, + { + "family": "attention", + "rank": 2, + "share": 0.16322081274999037 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03900169950588005 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.008475029742520849 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0009569388167916614 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.16322081274999037, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0009569388167916614, + "moe_gemm": 0.778146620509425, + "moe_router": 0.008475029742520849, + "norm_elementwise": 0.03900169950588005, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P01-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.24646421810404764, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9913999320344749, + "mode_shares": { + "FULL": { + "attention": 0.24646421810404764, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0018861985126382374, + "moe_gemm": 0.6957077658710443, + "moe_router": 0.012274637353427154, + "norm_elementwise": 0.03506711219331752, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6957077658710443 + }, + { + "family": "attention", + "rank": 2, + "share": 0.24646421810404764 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03506711219331752 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.012274637353427154 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0018861985126382374 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.658013698411068, + "mode_FULL": 0.6877531107808548, + "mode_NONE": -0.6802380609168648, + "mode_PIECEWISE": -0.08229232202065807, + "prefill_fraction": -0.6816133519618747, + "scheduled_tokens": -0.6333120429828512 + }, + "valid": false + }, + "shares": { + "attention": 0.24646421810404764, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0018861985126382374, + "moe_gemm": 0.6957077658710443, + "moe_router": 0.012274637353427154, + "norm_elementwise": 0.03506711219331752, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1013.7599999999983, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 701.0369999999989, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 598.1809999999995, + "name": "triton_red_fused_2" + }, + { + "duration_us": 25.409, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 22.721999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 13.697, + "name": "triton_red_fused_1" + }, + { + "duration_us": 13.632, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.0033278995815980927, + "attention_mixed": 0.07664950781433502, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9882022706147409, + "mode_shares": { + "FULL": { + "attention": 0.2372289894559244, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001973103373413011, + "moe_gemm": 0.7024833850537113, + "moe_router": 0.0127233500493961, + "norm_elementwise": 0.03660441698068597, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.07774006222398774, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8628349211839768, + "moe_router": 0.004560917765838791, + "norm_elementwise": 0.04302637544092451, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 7 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.8605854751478055 + }, + { + "family": "attention", + "rank": 2, + "share": 0.0799774073959331 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04293628681844257 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0046754221316145424 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 2.7679120945085497e-05 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -1.184350878588216, + "mode_FULL": -1.8269976055369814, + "mode_NONE": 1.8429476277592753, + "mode_PIECEWISE": -0.08229232202065807, + "prefill_fraction": 1.7302789555698612, + "scheduled_tokens": 1.471377771158277 + }, + "valid": false + }, + "shares": { + "attention": 0.0799774073959331, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 2.7679120945085497e-05, + "moe_gemm": 0.8605854751478055, + "moe_router": 0.0046754221316145424, + "norm_elementwise": 0.04293628681844257, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 13829.692000000014, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 6941.902000000004, + "name": "triton_red_fused_2" + }, + { + "duration_us": 6363.376999999999, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 297.34499999999997, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 147.58499999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 24.673, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.264999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P02": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7252956001376867 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2090637894197081 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03675377585093785 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.016380325462376815 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0025049516005055767 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.2090637894197081, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025049516005055767, + "moe_gemm": 0.7252956001376867, + "moe_router": 0.016380325462376815, + "norm_elementwise": 0.03675377585093785, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P02-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.16431648329180198, + "attention_mixed": 0.040375747240313245, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9899496306808163, + "mode_shares": { + "FULL": { + "attention": 0.2070416346282377, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002614300625672357, + "moe_gemm": 0.72402183586612, + "moe_router": 0.017564069072735583, + "norm_elementwise": 0.03839442593161487, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.19565666700967585, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017527292718792177, + "moe_gemm": 0.7535224022949394, + "moe_router": 0.011441473126433909, + "norm_elementwise": 0.02878152643291581, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7301095783988965 + }, + { + "family": "attention", + "rank": 2, + "share": 0.20469223053211524 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.036410706151768 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.016300609001782734 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0024365065962538455 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.673985188837054, + "mode_FULL": 0.11926397031787984, + "mode_NONE": -0.2790993472934029, + "mode_PIECEWISE": -0.0171348471830914, + "prefill_fraction": -0.13753859346790945, + "scheduled_tokens": -0.22985353117145157 + }, + "valid": false + }, + "shares": { + "attention": 0.20469223053211524, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0024365065962538455, + "moe_gemm": 0.7301095783988965, + "moe_router": 0.016300609001782734, + "norm_elementwise": 0.036410706151768, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 775.5859999999996, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 562.3350000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 541.5999999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.592, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.384, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.864999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 12.447999999999999, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.17098573558976576, + "attention_mixed": 0.04244961271753521, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9900472542616138, + "mode_shares": { + "FULL": { + "attention": 0.20867035579310442, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025849579196340765, + "moe_gemm": 0.7236684498934784, + "moe_router": 0.017385853018663355, + "norm_elementwise": 0.03743160202992195, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.23505546138588368, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025209396629618792, + "moe_gemm": 0.7060220856055816, + "moe_router": 0.012259376230707445, + "norm_elementwise": 0.035577961118325376, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7204816218764769 + }, + { + "family": "attention", + "rank": 2, + "share": 0.21343534830730096 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03709684555010769 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.016460041922970896 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0025733966047573075 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 14.182091081480554, + "mode_FULL": 0.11926397031787984, + "mode_NONE": -0.2790993472934029, + "mode_PIECEWISE": -0.0171348471830914, + "prefill_fraction": -0.20596870108215656, + "scheduled_tokens": -0.2935716394203467 + }, + "valid": false + }, + "shares": { + "attention": 0.21343534830730096, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025733966047573075, + "moe_gemm": 0.7204816218764769, + "moe_router": 0.016460041922970896, + "norm_elementwise": 0.03709684555010769, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 767.5569999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 572.6729999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 527.8730000000002, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.624999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.192999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.735999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.902999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6239343487636664 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3267032207444448 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.0298005081329104 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010510163136034627 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0016074078701660255 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.3267032207444448, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0016074078701660255, + "moe_gemm": 0.6239343487636664, + "moe_router": 0.010510163136034627, + "norm_elementwise": 0.0298005081329104, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P02-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.3066236078963298, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9922817455017883, + "mode_shares": { + "FULL": { + "attention": 0.3066236078963298, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0016715827167837374, + "moe_gemm": 0.6421673334694945, + "moe_router": 0.010874019669713833, + "norm_elementwise": 0.030945201749466352, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6421673334694945 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3066236078963298 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.030945201749466352 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010874019669713833 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0016715827167837374 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.20138501290112165, + "mode_FULL": 0.20658265838024312, + "mode_NONE": -0.20080594382714848, + "mode_PIECEWISE": -0.04755171103421909, + "prefill_fraction": -0.2043665285851284, + "scheduled_tokens": -0.19254510985844397 + }, + "valid": true + }, + "shares": { + "attention": 0.3066236078963298, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0016715827167837374, + "moe_gemm": 0.6421673334694945, + "moe_router": 0.010874019669713833, + "norm_elementwise": 0.030945201749466352, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 996.7139999999991, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 703.9869999999992, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 606.911, + "name": "triton_red_fused_2" + }, + { + "duration_us": 24.991999999999997, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 21.758999999999997, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 13.247, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 13.184, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.34678283359255974, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9928295517926563, + "mode_shares": { + "FULL": { + "attention": 0.34678283359255974, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015432330235483139, + "moe_gemm": 0.6057013640578384, + "moe_router": 0.010146306602355423, + "norm_elementwise": 0.028655814516354454, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6057013640578384 + }, + { + "family": "attention", + "rank": 2, + "share": 0.34678283359255974 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.028655814516354454 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010146306602355423 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0015432330235483139 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.20138501290112165, + "mode_FULL": 0.20658265838024312, + "mode_NONE": -0.20080594382714848, + "mode_PIECEWISE": -0.04755171103421909, + "prefill_fraction": -0.2043665285851284, + "scheduled_tokens": -0.19254510985844397 + }, + "valid": true + }, + "shares": { + "attention": 0.34678283359255974, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015432330235483139, + "moe_gemm": 0.6057013640578384, + "moe_router": 0.010146306602355423, + "norm_elementwise": 0.028655814516354454, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1013.8929999999987, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 711.0099999999981, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 613.2400000000005, + "name": "triton_red_fused_2" + }, + { + "duration_us": 25.055, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 22.112, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 14.049, + "name": "triton_red_fused_1" + }, + { + "duration_us": 13.855, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P03": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5745120568231603 + }, + { + "family": "attention", + "rank": 2, + "share": 0.23318306938126201 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.12285482042003387 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.035409526402454435 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.008296375954773179 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.23318306938126201, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008296375954773179, + "moe_gemm": 0.5745120568231603, + "moe_router": 0.035409526402454435, + "norm_elementwise": 0.12285482042003387, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P03-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.23677440452100923, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9773142909813943, + "mode_shares": { + "FULL": { + "attention": 0.23677440452100923, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.007300880477096696, + "moe_gemm": 0.5916730425467414, + "moe_router": 0.030667589074404745, + "norm_elementwise": 0.11089837436214241, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5916730425467414 + }, + { + "family": "attention", + "rank": 2, + "share": 0.23677440452100923 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.11089837436214241 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.030667589074404745 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.007300880477096696 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 3.72009366381467, + "mode_FULL": 0.27793577928319474, + "mode_NONE": -0.27793577928319446, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1868864933066391, + "scheduled_tokens": -0.182101722961305 + }, + "valid": false + }, + "shares": { + "attention": 0.23677440452100923, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.007300880477096696, + "moe_gemm": 0.5916730425467414, + "moe_router": 0.030667589074404745, + "norm_elementwise": 0.11089837436214241, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 576.23, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 464.7449999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.744, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.575, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.040999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 9.41, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.2295917342415148, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9711974069819732, + "mode_shares": { + "FULL": { + "attention": 0.2295917342415148, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009291871432449663, + "moe_gemm": 0.5573510710995793, + "moe_router": 0.04015146373050413, + "norm_elementwise": 0.13481126647792535, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5573510710995793 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2295917342415148 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.13481126647792535 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.04015146373050413 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.009291871432449663 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.27750213252727524, + "mode_FULL": 0.27793577928319474, + "mode_NONE": -0.27793577928319446, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1868864933066391, + "scheduled_tokens": -0.1838557206293655 + }, + "valid": false + }, + "shares": { + "attention": 0.2295917342415148, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009291871432449663, + "moe_gemm": 0.5573510710995793, + "moe_router": 0.04015146373050413, + "norm_elementwise": 0.13481126647792535, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 571.9689999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 469.85699999999855, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.529, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.383, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.761000000000001, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5660210180238645 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3911264870692061 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.02983320960479998 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.004378996059032046 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.000268750902798599 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.3911264870692061, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.000268750902798599, + "moe_gemm": 0.5660210180238645, + "moe_router": 0.004378996059032046, + "norm_elementwise": 0.02983320960479998, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P03-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.15040135314337208, + "attention_mixed": 0.28933706610779675, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9918450008730845, + "mode_shares": { + "FULL": { + "attention": 0.6859966420485143, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019314646391670124, + "moe_gemm": 0.2659878061842072, + "moe_router": 0.012555897863413516, + "norm_elementwise": 0.025494810657793975, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.3425246256579563, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147373284349258, + "moe_router": 0.003205686402689487, + "norm_elementwise": 0.030942924035345834, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.61102927558752, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0013975676747442667, + "moe_gemm": 0.3571425323248457, + "moe_router": 0.006697455630309823, + "norm_elementwise": 0.018973805351208753, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 6, + "NONE": 1, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5172566828279285 + }, + { + "family": "attention", + "rank": 2, + "share": 0.4397384192511688 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.028771804175374255 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0055405928130157975 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.000537501805597198 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 1.22313193325346, + "mode_FULL": 0.7720860330486908, + "mode_NONE": -1.155709946053278, + "mode_PIECEWISE": 0.5, + "prefill_fraction": -0.8564873113454512, + "scheduled_tokens": -1.021286605051562 + }, + "valid": false + }, + "shares": { + "attention": 0.4397384192511688, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.000537501805597198, + "moe_gemm": 0.5172566828279285, + "moe_router": 0.0055405928130157975, + "norm_elementwise": 0.028771804175374255, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 2966.5110000000122, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 1605.9449999999956, + "name": "triton_red_fused_2" + }, + { + "duration_us": 1552.6929999999936, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 63.32699999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 34.559000000000005, + "name": "triton_red_fused_1" + }, + { + "duration_us": 28.895999999999994, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 11.679, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.3425145548872434, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9914119224463182, + "mode_shares": { + "NONE": { + "attention": 0.3425145548872434, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147853532198007, + "moe_router": 0.0032173993050482958, + "norm_elementwise": 0.030894615034225702, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6147853532198007 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3425145548872434 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.030894615034225702 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0032173993050482958 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -1.5373920998057986, + "mode_FULL": -1.1124192104801658, + "mode_NONE": 1.1124192104801658, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.1145676479021394, + "scheduled_tokens": 1.1485324061566928 + }, + "valid": false + }, + "shares": { + "attention": 0.3425145548872434, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6147853532198007, + "moe_router": 0.0032173993050482958, + "norm_elementwise": 0.030894615034225702, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 17168.532999999996, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 8516.018999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 7858.939000000003, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 365.792, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 181.474, + "name": "triton_red_fused_1" + }, + { + "duration_us": 107.04, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.894999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P04": { + "moderate": { + "mean_ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.47883668691193637 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.4063754056124169 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.05854649280424695 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0327989475272124 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.004481485009351725 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.47883668691193637, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004481485009351725, + "moe_gemm": 0.4063754056124169, + "moe_router": 0.0327989475272124, + "norm_elementwise": 0.05854649280424695, + "sampler": 0.0 + }, + "reason": null, + "run_id": "P04-C00-moderate", + "status": "EVALUABLE", + "valid_window_count": 2, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.45337846230845036, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9811078968569179, + "mode_shares": { + "FULL": { + "attention": 0.45337846230845036, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00447249169315189, + "moe_gemm": 0.4320822710822647, + "moe_router": 0.03260980879841117, + "norm_elementwise": 0.05856486297463963, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.45337846230845036 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.4320822710822647 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.05856486297463963 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.03260980879841117 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.00447249169315189 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.20844725339793926, + "mode_FULL": 0.18226661968758462, + "mode_NONE": -0.18226661968758412, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.15943564286260098, + "scheduled_tokens": -0.15585724120282418 + }, + "valid": true + }, + "shares": { + "attention": 0.45337846230845036, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00447249169315189, + "moe_gemm": 0.4320822710822647, + "moe_router": 0.03260980879841117, + "norm_elementwise": 0.05856486297463963, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 600.4589999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 503.8849999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 467.6160000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.391000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.958999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.197999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.726999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": true, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.5042949115154224, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.980970138873411, + "mode_shares": { + "FULL": { + "attention": 0.5042949115154224, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004490478325551558, + "moe_gemm": 0.3806685401425692, + "moe_router": 0.032988086256013634, + "norm_elementwise": 0.058528122633854276, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.5042949115154224 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.3806685401425692 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.058528122633854276 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.032988086256013634 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.004490478325551558 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.20844725339793926, + "mode_FULL": 0.18226661968758462, + "mode_NONE": -0.18226661968758412, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.15943564286260098, + "scheduled_tokens": -0.15585724120282418 + }, + "valid": true + }, + "shares": { + "attention": 0.5042949115154224, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004490478325551558, + "moe_gemm": 0.3806685401425692, + "moe_router": 0.032988086256013634, + "norm_elementwise": 0.058528122633854276, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 600.0620000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 501.89399999999864, + "name": "triton_red_fused_2" + }, + { + "duration_us": 467.97200000000095, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.328, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.800999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.870999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.014999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": true, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.6587659578842232 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.29774483506191735 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.022973020262564953 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.011567306064400342 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0017453967614688727 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.6587659578842232, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017453967614688727, + "moe_gemm": 0.29774483506191735, + "moe_router": 0.011567306064400342, + "norm_elementwise": 0.022973020262564953, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P04-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.6545383651006825, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.992872517456777, + "mode_shares": { + "FULL": { + "attention": 0.6545383651006825, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001725224109977307, + "moe_gemm": 0.30244289390345225, + "moe_router": 0.011446277854389742, + "norm_elementwise": 0.022719756488275074, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.6545383651006825 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.30244289390345225 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.022719756488275074 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.011446277854389742 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.001725224109977307 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -2.090753569627607, + "mode_FULL": 0.3444176950404237, + "mode_NONE": -0.3444176950404235, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.3279408074303144, + "scheduled_tokens": -0.3384806476596461 + }, + "valid": false + }, + "shares": { + "attention": 0.6545383651006825, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001725224109977307, + "moe_gemm": 0.30244289390345225, + "moe_router": 0.011446277854389742, + "norm_elementwise": 0.022719756488275074, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 702.7299999999998, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 538.8529999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.6859999999996, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.480999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.599, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.903999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.879999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.6629935506677639, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9927205146123725, + "mode_shares": { + "FULL": { + "attention": 0.6629935506677639, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017655694129604385, + "moe_gemm": 0.29304677622038244, + "moe_router": 0.01168833427441094, + "norm_elementwise": 0.02322628403685483, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.6629935506677639 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.29304677622038244 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.02322628403685483 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.01168833427441094 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0017655694129604385 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -3.0407898355557177, + "mode_FULL": 0.3444176950404237, + "mode_NONE": -0.3444176950404235, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.3279408074303144, + "scheduled_tokens": -0.3392815338864717 + }, + "valid": false + }, + "shares": { + "attention": 0.6629935506677639, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017655694129604385, + "moe_gemm": 0.29304677622038244, + "moe_router": 0.01168833427441094, + "norm_elementwise": 0.02322628403685483, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 697.0249999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 539.4639999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 512.0369999999991, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.703, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.440999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.392999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.231999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P05": { + "moderate": { + "mean_ranking": null, + "mean_shares": null, + "reason": "pattern/config cell missing under A-P3-7", + "run_id": "P05-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 0, + "windows": [] + }, + "saturation": { + "mean_ranking": null, + "mean_shares": null, + "reason": "pattern/config cell missing under A-P3-7", + "run_id": "P05-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 0, + "windows": [] + } + }, + "P06": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.575196967256715 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3159632048695007 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.05688419362270496 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02969053707107777 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.004277410164380805 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.3159632048695007, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004277410164380805, + "moe_gemm": 0.575196967256715, + "moe_router": 0.02969053707107777, + "norm_elementwise": 0.05688419362270496, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P06-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.2490290455613554, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9782676099166654, + "mode_shares": { + "FULL": { + "attention": 0.2490290455613554, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005145523506508112, + "moe_gemm": 0.6200403635488907, + "moe_router": 0.03550127037050399, + "norm_elementwise": 0.06855140692940716, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6200403635488907 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2490290455613554 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.06855140692940716 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.03550127037050399 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.005145523506508112 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.7216535064826213, + "mode_FULL": 0.13539969781837916, + "mode_NONE": -0.1341287527105229, + "mode_PIECEWISE": -0.018343220956176326, + "prefill_fraction": -0.13284443594178524, + "scheduled_tokens": -0.12475886497271786 + }, + "valid": false + }, + "shares": { + "attention": 0.2490290455613554, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005145523506508112, + "moe_gemm": 0.6200403635488907, + "moe_router": 0.03550127037050399, + "norm_elementwise": 0.06855140692940716, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 597.553, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 503.07699999999915, + "name": "triton_red_fused_2" + }, + { + "duration_us": 466.9400000000012, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.869999999999997, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.854, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.744, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.727, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.382897364177646, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9857570160520932, + "mode_shares": { + "FULL": { + "attention": 0.382897364177646, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003409296822253497, + "moe_gemm": 0.5303535709645394, + "moe_router": 0.023879803771651553, + "norm_elementwise": 0.04521698031600275, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5303535709645394 + }, + { + "family": "attention", + "rank": 2, + "share": 0.382897364177646 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04521698031600275 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.023879803771651553 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003409296822253497 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 2.7828468550637027, + "mode_FULL": 0.13539969781837916, + "mode_NONE": -0.1341287527105229, + "mode_PIECEWISE": -0.018343220956176326, + "prefill_fraction": -0.13284443594178524, + "scheduled_tokens": -0.10682572566427663 + }, + "valid": false + }, + "shares": { + "attention": 0.382897364177646, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003409296822253497, + "moe_gemm": 0.5303535709645394, + "moe_router": 0.023879803771651553, + "norm_elementwise": 0.04521698031600275, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 620.7480000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 515.2009999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 493.54399999999913, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.936, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.312, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.751999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.4, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.5658804400108743 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.3934686862786809 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.022493433009649746 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010415454290445056 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0015415313949460807 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.5658804400108743, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015415313949460807, + "moe_gemm": 0.3934686862786809, + "moe_router": 0.010415454290445056, + "norm_elementwise": 0.022493433009649746, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P06-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.5606386878931048, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9937049249586525, + "mode_shares": { + "FULL": { + "attention": 0.5606386878931048, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015728835856848925, + "moe_gemm": 0.3985345690982096, + "moe_router": 0.010374903112678729, + "norm_elementwise": 0.022583881268974364, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.5606386878931048 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.3985345690982096 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.022583881268974364 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010374903112678729 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0015728835856848925 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.19939502094061695, + "mode_FULL": 0.3784962133407253, + "mode_NONE": -0.3784962133407256, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.34967744618584184, + "scheduled_tokens": -0.3730920628084988 + }, + "valid": false + }, + "shares": { + "attention": 0.5606386878931048, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0015728835856848925, + "moe_gemm": 0.3985345690982096, + "moe_router": 0.010374903112678729, + "norm_elementwise": 0.022583881268974364, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 730.718, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 532.1659999999997, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 517.7589999999993, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.624, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.944999999999999, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.351999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 11.359999999999998, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.5711221921286438, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9938941650105397, + "mode_shares": { + "FULL": { + "attention": 0.5711221921286438, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001510179204207269, + "moe_gemm": 0.3884028034591522, + "moe_router": 0.010456005468211384, + "norm_elementwise": 0.022402984750325126, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.5711221921286438 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.3884028034591522 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.022402984750325126 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010456005468211384 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.001510179204207269 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -3.0651651743014807, + "mode_FULL": 0.3784962133407253, + "mode_NONE": -0.3784962133407256, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.34967744618584184, + "scheduled_tokens": -0.381504751465261 + }, + "valid": false + }, + "shares": { + "attention": 0.5711221921286438, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001510179204207269, + "moe_gemm": 0.3884028034591522, + "moe_router": 0.010456005468211384, + "norm_elementwise": 0.022402984750325126, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 735.0089999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 530.0759999999996, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.90599999999927, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.855999999999995, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.872, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.360999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.878999999999998, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P07": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.62183214684387 + }, + { + "family": "attention", + "rank": 2, + "share": 0.28668399704125846 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04862705270918968 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.024198716148019384 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003722036909265734 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.28668399704125846, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003722036909265734, + "moe_gemm": 0.62183214684387, + "moe_router": 0.024198716148019384, + "norm_elementwise": 0.04862705270918968, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P07-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.306761436001029, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9841089106220091, + "mode_shares": { + "FULL": { + "attention": 0.306761436001029, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003944802403864725, + "moe_gemm": 0.5964090291561805, + "moe_router": 0.02562456323133302, + "norm_elementwise": 0.051369079829601885, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5964090291561805 + }, + { + "family": "attention", + "rank": 2, + "share": 0.306761436001029 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.051369079829601885 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02562456323133302 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003944802403864725 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.18368608035893982, + "mode_FULL": 0.1535286967084372, + "mode_NONE": -0.15352869670843752, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1535266720575854, + "scheduled_tokens": -0.13391206652423035 + }, + "valid": true + }, + "shares": { + "attention": 0.306761436001029, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003944802403864725, + "moe_gemm": 0.5964090291561805, + "moe_router": 0.02562456323133302, + "norm_elementwise": 0.051369079829601885, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 606.694, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 538.9419999999992, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 496.9549999999989, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.128, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.04, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.56, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.304, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.2666065580814879, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9860189886811974, + "mode_shares": { + "FULL": { + "attention": 0.2666065580814879, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034992714146667428, + "moe_gemm": 0.6472552645315595, + "moe_router": 0.02277286906470575, + "norm_elementwise": 0.04588502558877749, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6472552645315595 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2666065580814879 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04588502558877749 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02277286906470575 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0034992714146667428 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 3.069910871828644, + "mode_FULL": 0.1535286967084372, + "mode_NONE": -0.15352869670843752, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1535266720575854, + "scheduled_tokens": -0.11608746652035899 + }, + "valid": false + }, + "shares": { + "attention": 0.2666065580814879, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034992714146667428, + "moe_gemm": 0.6472552645315595, + "moe_router": 0.02277286906470575, + "norm_elementwise": 0.04588502558877749, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 621.5659999999995, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 529.2479999999987, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 510.27699999999976, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.607000000000003, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.184, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.230999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.817, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": null, + "mean_shares": null, + "reason": "accepted trace windows 0/2", + "run_id": "P07-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 0, + "windows": [] + } + }, + "P08": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5706435605086021 + }, + { + "family": "attention", + "rank": 2, + "share": 0.34593945846402396 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04440163968760805 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.022174807436130248 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0034157188029328364 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.34593945846402396, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034157188029328364, + "moe_gemm": 0.5706435605086021, + "moe_router": 0.022174807436130248, + "norm_elementwise": 0.04440163968760805, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P08-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.3248872069635257, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9877052895017178, + "mode_shares": { + "FULL": { + "attention": 0.3248872069635257, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031072028443770152, + "moe_gemm": 0.5984635601986847, + "moe_router": 0.020260919566213807, + "norm_elementwise": 0.04098639992891659, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5984635601986847 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3248872069635257 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04098639992891659 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.020260919566213807 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0031072028443770152 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 1.0901560791930118, + "mode_FULL": 0.16434181266464418, + "mode_NONE": -0.16434181266464443, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.16432997461824633, + "scheduled_tokens": -0.1413297101061453 + }, + "valid": false + }, + "shares": { + "attention": 0.3248872069635257, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031072028443770152, + "moe_gemm": 0.5984635601986847, + "moe_router": 0.020260919566213807, + "norm_elementwise": 0.04098639992891659, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 654.3379999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 552.4199999999996, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 513.1979999999996, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.089000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.015999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.127999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.719, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.3669917099645223, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9854450802968766, + "mode_shares": { + "FULL": { + "attention": 0.3669917099645223, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003724234761488658, + "moe_gemm": 0.5428235608185195, + "moe_router": 0.024088695306046685, + "norm_elementwise": 0.04781687944629951, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5428235608185195 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3669917099645223 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04781687944629951 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.024088695306046685 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003724234761488658 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -1.4164397504981734, + "mode_FULL": 0.16434181266464418, + "mode_NONE": -0.16434181266464443, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.16432997461824633, + "scheduled_tokens": -0.19075947310069616 + }, + "valid": false + }, + "shares": { + "attention": 0.3669917099645223, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003724234761488658, + "moe_gemm": 0.5428235608185195, + "moe_router": 0.024088695306046685, + "norm_elementwise": 0.04781687944629951, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 632.7909999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 540.1919999999998, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.9399999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.673000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.665, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.713, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.878999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": null, + "mean_shares": null, + "reason": "accepted trace windows 0/2", + "run_id": "P08-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 0, + "windows": [] + } + }, + "P09": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7499562920571119 + }, + { + "family": "attention", + "rank": 2, + "share": 0.16190861537800794 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04833573350364303 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.021425255352349733 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0035055829982362854 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.16190861537800794, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0035055829982362854, + "moe_gemm": 0.7499562920571119, + "moe_router": 0.021425255352349733, + "norm_elementwise": 0.04833573350364303, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P09-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.08835627115447606, + "attention_mixed": 0.11385037570991731, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.988346330660954, + "mode_shares": { + "FULL": { + "attention": 0.280249688903593, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004209638462393143, + "moe_gemm": 0.6254634471394533, + "moe_router": 0.02523969717175661, + "norm_elementwise": 0.0493304218340256, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.14806901527656063, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7989631873012853, + "moe_router": 0.006717474937844798, + "norm_elementwise": 0.036240204978569675, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.18871401371634564, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0022585830433693877, + "moe_gemm": 0.7546316751137306, + "moe_router": 0.012749732426996046, + "norm_elementwise": 0.031927806640493814, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 5, + "NONE": 1, + "PIECEWISE": 2 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7306681072672004 + }, + { + "family": "attention", + "rank": 2, + "share": 0.20220664686439338 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.039044817770315786 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.014406944379520907 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.002019814379523439 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 2.5579606107986748, + "mode_FULL": -0.6775421426296558, + "mode_NONE": 0.2503307644752007, + "mode_PIECEWISE": 0.5907929693818461, + "prefill_fraction": 0.6661033678566376, + "scheduled_tokens": 0.05206272885676596 + }, + "valid": false + }, + "shares": { + "attention": 0.20220664686439338, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002019814379523439, + "moe_gemm": 0.7306681072672004, + "moe_router": 0.014406944379520907, + "norm_elementwise": 0.039044817770315786, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1078.7899999999981, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 718.971000000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 663.9299999999997, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.880000000000003, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.017, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.112, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.974999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.09568605186600879, + "attention_mixed": 0.025924532025613695, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9819166279177437, + "mode_shares": { + "FULL": { + "attention": 0.14040386061386476, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.006289384167983417, + "moe_gemm": 0.7278572814656825, + "moe_router": 0.03517706269456804, + "norm_elementwise": 0.06867748249282202, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.08139719907850929, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0022138539380632135, + "moe_gemm": 0.8578037652991323, + "moe_router": 0.01403539873195149, + "norm_elementwise": 0.033980351851418185, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7692444768470234 + }, + { + "family": "attention", + "rank": 2, + "share": 0.12161058389162249 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.05762664923697027 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.028443566325178558 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.004991351616949132 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.19954189900373331, + "mode_FULL": -0.10469513477046205, + "mode_NONE": -0.32861907697425397, + "mode_PIECEWISE": 0.29736821980293143, + "prefill_fraction": 0.10420387771831544, + "scheduled_tokens": -0.23288793337699637 + }, + "valid": false + }, + "shares": { + "attention": 0.12161058389162249, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004991351616949132, + "moe_gemm": 0.7692444768470234, + "moe_router": 0.028443566325178558, + "norm_elementwise": 0.05762664923697027, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 667.7419999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 523.011999999999, + "name": "triton_red_fused_2" + }, + { + "duration_us": 498.98300000000086, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 20.736, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.591, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.645, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.36, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6902715167627947 + }, + { + "family": "attention", + "rank": 2, + "share": 0.26232227513895634 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03432883110553378 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0035556142118645844 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.26232227513895634, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6902715167627947, + "moe_router": 0.0035556142118645844, + "norm_elementwise": 0.03432883110553378, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P09-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.24676580346741184, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9902796451206958, + "mode_shares": { + "NONE": { + "attention": 0.24676580346741184, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7048192593007057, + "moe_router": 0.0036328436089412205, + "norm_elementwise": 0.035061738743637105, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7048192593007057 + }, + { + "family": "attention", + "rank": 2, + "share": 0.24676580346741184 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.035061738743637105 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0036328436089412205 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 1.259998751515166, + "mode_FULL": -1.0890519340869804, + "mode_NONE": 1.0890519340869804, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.1263920173520792, + "scheduled_tokens": 1.3450157218366598 + }, + "valid": false + }, + "shares": { + "attention": 0.24676580346741184, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7048192593007057, + "moe_router": 0.0036328436089412205, + "norm_elementwise": 0.035061738743637105, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 18507.434, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 9166.28299999999, + "name": "triton_red_fused_2" + }, + { + "duration_us": 8451.682000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 396.608, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 195.36100000000002, + "name": "triton_red_fused_1" + }, + { + "duration_us": 75.745, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.616, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.27787874681050084, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9906768293176031, + "mode_shares": { + "NONE": { + "attention": 0.27787874681050084, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6757237742248838, + "moe_router": 0.0034783848147879478, + "norm_elementwise": 0.03359592346743046, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6757237742248838 + }, + { + "family": "attention", + "rank": 2, + "share": 0.27787874681050084 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03359592346743046 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.0034783848147879478 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.2365580579355552, + "mode_FULL": -1.0890519340869804, + "mode_NONE": 1.0890519340869804, + "mode_PIECEWISE": 0.0, + "prefill_fraction": 1.1326772391141862, + "scheduled_tokens": 1.3450157218366598 + }, + "valid": false + }, + "shares": { + "attention": 0.27787874681050084, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.6757237742248838, + "moe_router": 0.0034783848147879478, + "norm_elementwise": 0.03359592346743046, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 18504.999000000003, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 9192.093999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 8458.98800000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 398.754, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 195.68099999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 84.479, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.392, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P10": { + "moderate": { + "mean_ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5404178421352701 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3060614779541131 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.09802642887199953 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02813772466138177 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.006793424317425972 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.3060614779541131, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.006793424317425972, + "moe_gemm": 0.5404178421352701, + "moe_router": 0.02813772466138177, + "norm_elementwise": 0.09802642887199953, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P10-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.3087850643091725, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.985252225101006, + "mode_shares": { + "FULL": { + "attention": 0.3087850643091725, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005163136163815447, + "moe_gemm": 0.5777999387887653, + "moe_router": 0.02008575072504131, + "norm_elementwise": 0.07341833511421153, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5777999387887653 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3087850643091725 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.07341833511421153 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02008575072504131 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.005163136163815447 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 4.655696267017126, + "mode_FULL": 0.17677689026422408, + "mode_NONE": -0.1682723158645361, + "mode_PIECEWISE": -0.05341411394958018, + "prefill_fraction": -0.12905587442539998, + "scheduled_tokens": -0.09951290348812954 + }, + "valid": false + }, + "shares": { + "attention": 0.3087850643091725, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005163136163815447, + "moe_gemm": 0.5777999387887653, + "moe_router": 0.02008575072504131, + "norm_elementwise": 0.07341833511421153, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 583.5649999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 466.8209999999995, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.617, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.536, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.231999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.049, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.3033378915990537, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9736215707793747, + "mode_shares": { + "FULL": { + "attention": 0.3033378915990537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008423712471036498, + "moe_gemm": 0.5030357454817748, + "moe_router": 0.03618969859772223, + "norm_elementwise": 0.12263452262978754, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5030357454817748 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3033378915990537 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.12263452262978754 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.03618969859772223 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.008423712471036498 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.5055344205281964, + "mode_FULL": 0.17677689026422408, + "mode_NONE": -0.1682723158645361, + "mode_PIECEWISE": -0.05341411394958018, + "prefill_fraction": -0.12905587442539998, + "scheduled_tokens": -0.10506250419671144 + }, + "valid": false + }, + "shares": { + "attention": 0.3033378915990537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008423712471036498, + "moe_gemm": 0.5030357454817748, + "moe_router": 0.03618969859772223, + "norm_elementwise": 0.12263452262978754, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 590.9459999999992, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 467.6929999999989, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.272, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.152, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.63, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "saturation": { + "mean_ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.6779796330559141 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.2890057629595925 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.01871664375594286 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.007610719195516601 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0009838772120851383 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "mean_shares": { + "attention": 0.6779796330559141, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0009838772120851383, + "moe_gemm": 0.2890057629595925, + "moe_router": 0.007610719195516601, + "norm_elementwise": 0.01871664375594286, + "sampler": 0.0 + }, + "reason": "classifiability, representativeness, or recovery gate failed", + "run_id": "P10-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 2, + "windows": [ + { + "attention_subshares": { + "attention_decode": 0.7471257503417844, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9942106688319626, + "mode_shares": { + "FULL": { + "attention": 0.7471257503417844, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0013626029325187864, + "moe_gemm": 0.21782046124566493, + "moe_router": 0.009663599861937516, + "norm_elementwise": 0.01823825445005707, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.7471257503417844 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.21782046124566493 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.01823825445005707 + }, + { + "family": "moe_router", + "rank": 3, + "share": 0.009663599861937516 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0013626029325187864 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 3.613259323224499, + "mode_FULL": 0.6120018940246724, + "mode_NONE": -0.5981417765928579, + "mode_PIECEWISE": -0.10976470169608525, + "prefill_fraction": -0.6117521895228502, + "scheduled_tokens": -0.5755060211990467 + }, + "valid": false + }, + "shares": { + "attention": 0.7471257503417844, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0013626029325187864, + "moe_gemm": 0.21782046124566493, + "moe_router": 0.009663599861937516, + "norm_elementwise": 0.01823825445005707, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 637.9129999999991, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 513.3399999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 484.63699999999847, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.874, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.408, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.879999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.305, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.2679713777414123, + "attention_mixed": 0.3408621380286316, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9943826035261398, + "mode_shares": { + "FULL": { + "attention": 0.6506807095871076, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.001469412163770193, + "moe_gemm": 0.3120351814822058, + "moe_router": 0.01012812440986268, + "norm_elementwise": 0.0195308311121687, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.5795322961533458, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.39390960062584945, + "moe_router": 0.0023577446138654075, + "norm_elementwise": 0.01895990875545615, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "NONE": 1 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.6088335157700439 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.36019106467352013 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.019195033061828647 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.005557838529095687 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0006051514916514902 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 28.040640400381754, + "mode_FULL": 0.09182576274871186, + "mode_NONE": -0.07574082366962798, + "mode_PIECEWISE": -0.10976470169608525, + "prefill_fraction": -0.09431170009009414, + "scheduled_tokens": -0.3287773474224777 + }, + "valid": false + }, + "shares": { + "attention": 0.6088335157700439, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0006051514916514902, + "moe_gemm": 0.36019106467352013, + "moe_router": 0.005557838529095687, + "norm_elementwise": 0.019195033061828647, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1568.3549999999968, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 937.5360000000019, + "name": "triton_red_fused_2" + }, + { + "duration_us": 870.9730000000023, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 32.38400000000001, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 25.343999999999998, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 21.376, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.167999999999997, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ] + } + }, + "P11": { + "moderate": { + "mean_ranking": null, + "mean_shares": null, + "reason": "pattern/config cell missing under A-P3-7", + "run_id": "P11-C00-moderate", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 0, + "windows": [] + }, + "saturation": { + "mean_ranking": null, + "mean_shares": null, + "reason": "pattern/config cell missing under A-P3-7", + "run_id": "P11-C00-saturation", + "status": "NOT_EVALUABLE", + "valid_window_count": 0, + "window_count": 0, + "windows": [] + } + } + }, + "operator_windows": { + "P01-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.0, + "attention_mixed": 0.11987238161795156, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9905881019946707, + "mode_shares": { + "NONE": { + "attention": 0.1007462687465188, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8471040152242396, + "moe_router": 0.008477215998495907, + "norm_elementwise": 0.034119643236757626, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.13605518456441312, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0019444790834074754, + "moe_gemm": 0.8102627123318585, + "moe_router": 0.01080794700368473, + "norm_elementwise": 0.03163704570787867, + "sampler": 0.0 + } + }, + "mode_steps": { + "NONE": 3, + "PIECEWISE": 5 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.8271478398184453 + }, + { + "family": "attention", + "rank": 2, + "share": 0.11987238161795156 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.032774871259148075 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.009739725006365179 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0010532842927605177 + }, + { + "family": "collective", + "rank": 4, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 4, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 4, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 2.4721524781047854, + "mode_FULL": -0.2325584995849942, + "mode_NONE": 0.3041456447839143, + "mode_PIECEWISE": -0.24468169875030038, + "prefill_fraction": 0.18099057277285577, + "scheduled_tokens": 0.4474608400278538 + }, + "valid": false + }, + "shares": { + "attention": 0.11987238161795156, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010532842927605177, + "moe_gemm": 0.8271478398184453, + "moe_router": 0.009739725006365179, + "norm_elementwise": 0.032774871259148075, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1567.9510000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 933.1070000000002, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 827.689, + "name": "triton_red_fused_2" + }, + { + "duration_us": 33.057, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.368999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 17.247, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.647999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.008867955248024707, + "attention_mixed": 0.11302508576620553, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.990127660582764, + "mode_shares": { + "FULL": { + "attention": 0.16315875281832684, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031741793017298413, + "moe_gemm": 0.7536332801624461, + "moe_router": 0.02179864485086512, + "norm_elementwise": 0.045434533244984684, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.09828250109360281, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.8488043657595763, + "moe_router": 0.008302816259724543, + "norm_elementwise": 0.0348733229605455, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.1472178594517614, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002166218689199354, + "moe_gemm": 0.7946960932441856, + "moe_router": 0.012792489660640699, + "norm_elementwise": 0.033466283510587794, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 1, + "NONE": 3, + "PIECEWISE": 4 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.8214241451495896 + }, + { + "family": "attention", + "rank": 2, + "share": 0.12189304101423025 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.034869855313022065 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.010879021850869232 + }, + { + "family": "kv_memory", + "rank": 4, + "share": 0.0010615972550528194 + }, + { + "family": "collective", + "rank": 6, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 6, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 6, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 1.4422983740600546, + "mode_FULL": 0.3594518697735859, + "mode_NONE": 0.3041456447839143, + "mode_PIECEWISE": -0.49598146699059914, + "prefill_fraction": -0.3699982864929203, + "scheduled_tokens": 0.1504241077629697 + }, + "valid": false + }, + "shares": { + "attention": 0.12189304101423025, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0010615972550528194, + "moe_gemm": 0.8214241451495896, + "moe_router": 0.010879021850869232, + "norm_elementwise": 0.034869855313022065, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1449.5139999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 884.6050000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 826.8859999999994, + "name": "triton_red_fused_2" + }, + { + "duration_us": 30.943000000000005, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.431999999999995, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 17.441, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.328999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ], + "P02-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.16431648329180198, + "attention_mixed": 0.040375747240313245, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9899496306808163, + "mode_shares": { + "FULL": { + "attention": 0.2070416346282377, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002614300625672357, + "moe_gemm": 0.72402183586612, + "moe_router": 0.017564069072735583, + "norm_elementwise": 0.03839442593161487, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.19565666700967585, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0017527292718792177, + "moe_gemm": 0.7535224022949394, + "moe_router": 0.011441473126433909, + "norm_elementwise": 0.02878152643291581, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7301095783988965 + }, + { + "family": "attention", + "rank": 2, + "share": 0.20469223053211524 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.036410706151768 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.016300609001782734 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0024365065962538455 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.673985188837054, + "mode_FULL": 0.11926397031787984, + "mode_NONE": -0.2790993472934029, + "mode_PIECEWISE": -0.0171348471830914, + "prefill_fraction": -0.13753859346790945, + "scheduled_tokens": -0.22985353117145157 + }, + "valid": false + }, + "shares": { + "attention": 0.20469223053211524, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0024365065962538455, + "moe_gemm": 0.7301095783988965, + "moe_router": 0.016300609001782734, + "norm_elementwise": 0.036410706151768, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 775.5859999999996, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 562.3350000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 541.5999999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.592, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.384, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.864999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 12.447999999999999, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.17098573558976576, + "attention_mixed": 0.04244961271753521, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9900472542616138, + "mode_shares": { + "FULL": { + "attention": 0.20867035579310442, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025849579196340765, + "moe_gemm": 0.7236684498934784, + "moe_router": 0.017385853018663355, + "norm_elementwise": 0.03743160202992195, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.23505546138588368, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025209396629618792, + "moe_gemm": 0.7060220856055816, + "moe_router": 0.012259376230707445, + "norm_elementwise": 0.035577961118325376, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7204816218764769 + }, + { + "family": "attention", + "rank": 2, + "share": 0.21343534830730096 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.03709684555010769 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.016460041922970896 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0025733966047573075 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 14.182091081480554, + "mode_FULL": 0.11926397031787984, + "mode_NONE": -0.2790993472934029, + "mode_PIECEWISE": -0.0171348471830914, + "prefill_fraction": -0.20596870108215656, + "scheduled_tokens": -0.2935716394203467 + }, + "valid": false + }, + "shares": { + "attention": 0.21343534830730096, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0025733966047573075, + "moe_gemm": 0.7204816218764769, + "moe_router": 0.016460041922970896, + "norm_elementwise": 0.03709684555010769, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 767.5569999999994, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 572.6729999999994, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 527.8730000000002, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.624999999999996, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 16.192999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.735999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.902999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ], + "P03-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.23677440452100923, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9773142909813943, + "mode_shares": { + "FULL": { + "attention": 0.23677440452100923, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.007300880477096696, + "moe_gemm": 0.5916730425467414, + "moe_router": 0.030667589074404745, + "norm_elementwise": 0.11089837436214241, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5916730425467414 + }, + { + "family": "attention", + "rank": 2, + "share": 0.23677440452100923 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.11089837436214241 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.030667589074404745 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.007300880477096696 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 3.72009366381467, + "mode_FULL": 0.27793577928319474, + "mode_NONE": -0.27793577928319446, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1868864933066391, + "scheduled_tokens": -0.182101722961305 + }, + "valid": false + }, + "shares": { + "attention": 0.23677440452100923, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.007300880477096696, + "moe_gemm": 0.5916730425467414, + "moe_router": 0.030667589074404745, + "norm_elementwise": 0.11089837436214241, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 576.23, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 464.7449999999998, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.744, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.575, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.040999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 9.41, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.2295917342415148, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9711974069819732, + "mode_shares": { + "FULL": { + "attention": 0.2295917342415148, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009291871432449663, + "moe_gemm": 0.5573510710995793, + "moe_router": 0.04015146373050413, + "norm_elementwise": 0.13481126647792535, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5573510710995793 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2295917342415148 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.13481126647792535 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.04015146373050413 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.009291871432449663 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.27750213252727524, + "mode_FULL": 0.27793577928319474, + "mode_NONE": -0.27793577928319446, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1868864933066391, + "scheduled_tokens": -0.1838557206293655 + }, + "valid": false + }, + "shares": { + "attention": 0.2295917342415148, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.009291871432449663, + "moe_gemm": 0.5573510710995793, + "moe_router": 0.04015146373050413, + "norm_elementwise": 0.13481126647792535, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 571.9689999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 469.85699999999855, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.529, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.383, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.761000000000001, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 2 + } + ], + "P04-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.45337846230845036, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9811078968569179, + "mode_shares": { + "FULL": { + "attention": 0.45337846230845036, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00447249169315189, + "moe_gemm": 0.4320822710822647, + "moe_router": 0.03260980879841117, + "norm_elementwise": 0.05856486297463963, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.45337846230845036 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.4320822710822647 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.05856486297463963 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.03260980879841117 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.00447249169315189 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.20844725339793926, + "mode_FULL": 0.18226661968758462, + "mode_NONE": -0.18226661968758412, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.15943564286260098, + "scheduled_tokens": -0.15585724120282418 + }, + "valid": true + }, + "shares": { + "attention": 0.45337846230845036, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.00447249169315189, + "moe_gemm": 0.4320822710822647, + "moe_router": 0.03260980879841117, + "norm_elementwise": 0.05856486297463963, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 600.4589999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 503.8849999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 467.6160000000004, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.391000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.958999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.197999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.726999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": true, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.5042949115154224, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.980970138873411, + "mode_shares": { + "FULL": { + "attention": 0.5042949115154224, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004490478325551558, + "moe_gemm": 0.3806685401425692, + "moe_router": 0.032988086256013634, + "norm_elementwise": 0.058528122633854276, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "attention", + "rank": 1, + "share": 0.5042949115154224 + }, + { + "family": "moe_gemm", + "rank": 2, + "share": 0.3806685401425692 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.058528122633854276 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.032988086256013634 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.004490478325551558 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 0.20844725339793926, + "mode_FULL": 0.18226661968758462, + "mode_NONE": -0.18226661968758412, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.15943564286260098, + "scheduled_tokens": -0.15585724120282418 + }, + "valid": true + }, + "shares": { + "attention": 0.5042949115154224, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004490478325551558, + "moe_gemm": 0.3806685401425692, + "moe_router": 0.032988086256013634, + "norm_elementwise": 0.058528122633854276, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 600.0620000000005, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 501.89399999999864, + "name": "triton_red_fused_2" + }, + { + "duration_us": 467.97200000000095, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.328, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 12.800999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.870999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.014999999999999, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": true, + "window": 2 + } + ], + "P06-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.2490290455613554, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9782676099166654, + "mode_shares": { + "FULL": { + "attention": 0.2490290455613554, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005145523506508112, + "moe_gemm": 0.6200403635488907, + "moe_router": 0.03550127037050399, + "norm_elementwise": 0.06855140692940716, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6200403635488907 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2490290455613554 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.06855140692940716 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.03550127037050399 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.005145523506508112 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.7216535064826213, + "mode_FULL": 0.13539969781837916, + "mode_NONE": -0.1341287527105229, + "mode_PIECEWISE": -0.018343220956176326, + "prefill_fraction": -0.13284443594178524, + "scheduled_tokens": -0.12475886497271786 + }, + "valid": false + }, + "shares": { + "attention": 0.2490290455613554, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005145523506508112, + "moe_gemm": 0.6200403635488907, + "moe_router": 0.03550127037050399, + "norm_elementwise": 0.06855140692940716, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 597.553, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 503.07699999999915, + "name": "triton_red_fused_2" + }, + { + "duration_us": 466.9400000000012, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 19.869999999999997, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.854, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.744, + "name": "triton_red_fused_1" + }, + { + "duration_us": 9.727, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.382897364177646, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9857570160520932, + "mode_shares": { + "FULL": { + "attention": 0.382897364177646, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003409296822253497, + "moe_gemm": 0.5303535709645394, + "moe_router": 0.023879803771651553, + "norm_elementwise": 0.04521698031600275, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5303535709645394 + }, + { + "family": "attention", + "rank": 2, + "share": 0.382897364177646 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04521698031600275 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.023879803771651553 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003409296822253497 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 2.7828468550637027, + "mode_FULL": 0.13539969781837916, + "mode_NONE": -0.1341287527105229, + "mode_PIECEWISE": -0.018343220956176326, + "prefill_fraction": -0.13284443594178524, + "scheduled_tokens": -0.10682572566427663 + }, + "valid": false + }, + "shares": { + "attention": 0.382897364177646, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003409296822253497, + "moe_gemm": 0.5303535709645394, + "moe_router": 0.023879803771651553, + "norm_elementwise": 0.04521698031600275, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 620.7480000000002, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 515.2009999999995, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 493.54399999999913, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.936, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.312, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.751999999999999, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.4, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ], + "P07-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.306761436001029, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9841089106220091, + "mode_shares": { + "FULL": { + "attention": 0.306761436001029, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003944802403864725, + "moe_gemm": 0.5964090291561805, + "moe_router": 0.02562456323133302, + "norm_elementwise": 0.051369079829601885, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5964090291561805 + }, + { + "family": "attention", + "rank": 2, + "share": 0.306761436001029 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.051369079829601885 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02562456323133302 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003944802403864725 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.18368608035893982, + "mode_FULL": 0.1535286967084372, + "mode_NONE": -0.15352869670843752, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1535266720575854, + "scheduled_tokens": -0.13391206652423035 + }, + "valid": true + }, + "shares": { + "attention": 0.306761436001029, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003944802403864725, + "moe_gemm": 0.5964090291561805, + "moe_router": 0.02562456323133302, + "norm_elementwise": 0.051369079829601885, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 606.694, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 538.9419999999992, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 496.9549999999989, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.128, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 15.04, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 10.56, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.304, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.2666065580814879, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9860189886811974, + "mode_shares": { + "FULL": { + "attention": 0.2666065580814879, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034992714146667428, + "moe_gemm": 0.6472552645315595, + "moe_router": 0.02277286906470575, + "norm_elementwise": 0.04588502558877749, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.6472552645315595 + }, + { + "family": "attention", + "rank": 2, + "share": 0.2666065580814879 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04588502558877749 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02277286906470575 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0034992714146667428 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 3.069910871828644, + "mode_FULL": 0.1535286967084372, + "mode_NONE": -0.15352869670843752, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.1535266720575854, + "scheduled_tokens": -0.11608746652035899 + }, + "valid": false + }, + "shares": { + "attention": 0.2666065580814879, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0034992714146667428, + "moe_gemm": 0.6472552645315595, + "moe_router": 0.02277286906470575, + "norm_elementwise": 0.04588502558877749, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 621.5659999999995, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 529.2479999999987, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 510.27699999999976, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.607000000000003, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.184, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.230999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.817, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ], + "P08-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.3248872069635257, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9877052895017178, + "mode_shares": { + "FULL": { + "attention": 0.3248872069635257, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031072028443770152, + "moe_gemm": 0.5984635601986847, + "moe_router": 0.020260919566213807, + "norm_elementwise": 0.04098639992891659, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5984635601986847 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3248872069635257 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04098639992891659 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.020260919566213807 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.0031072028443770152 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 1.0901560791930118, + "mode_FULL": 0.16434181266464418, + "mode_NONE": -0.16434181266464443, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.16432997461824633, + "scheduled_tokens": -0.1413297101061453 + }, + "valid": false + }, + "shares": { + "attention": 0.3248872069635257, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0031072028443770152, + "moe_gemm": 0.5984635601986847, + "moe_router": 0.020260919566213807, + "norm_elementwise": 0.04098639992891659, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 654.3379999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 552.4199999999996, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 513.1979999999996, + "name": "triton_red_fused_2" + }, + { + "duration_us": 21.089000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.015999999999998, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 12.127999999999998, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.719, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.3669917099645223, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9854450802968766, + "mode_shares": { + "FULL": { + "attention": 0.3669917099645223, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003724234761488658, + "moe_gemm": 0.5428235608185195, + "moe_router": 0.024088695306046685, + "norm_elementwise": 0.04781687944629951, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5428235608185195 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3669917099645223 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.04781687944629951 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.024088695306046685 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.003724234761488658 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -1.4164397504981734, + "mode_FULL": 0.16434181266464418, + "mode_NONE": -0.16434181266464443, + "mode_PIECEWISE": 0.0, + "prefill_fraction": -0.16432997461824633, + "scheduled_tokens": -0.19075947310069616 + }, + "valid": false + }, + "shares": { + "attention": 0.3669917099645223, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.003724234761488658, + "moe_gemm": 0.5428235608185195, + "moe_router": 0.024088695306046685, + "norm_elementwise": 0.04781687944629951, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 632.7909999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 540.1919999999998, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 511.9399999999992, + "name": "triton_red_fused_2" + }, + { + "duration_us": 20.673000000000002, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.665, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.713, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.878999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ], + "P09-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.08835627115447606, + "attention_mixed": 0.11385037570991731, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.988346330660954, + "mode_shares": { + "FULL": { + "attention": 0.280249688903593, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004209638462393143, + "moe_gemm": 0.6254634471394533, + "moe_router": 0.02523969717175661, + "norm_elementwise": 0.0493304218340256, + "sampler": 0.0 + }, + "NONE": { + "attention": 0.14806901527656063, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0, + "moe_gemm": 0.7989631873012853, + "moe_router": 0.006717474937844798, + "norm_elementwise": 0.036240204978569675, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.18871401371634564, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0022585830433693877, + "moe_gemm": 0.7546316751137306, + "moe_router": 0.012749732426996046, + "norm_elementwise": 0.031927806640493814, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 5, + "NONE": 1, + "PIECEWISE": 2 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7306681072672004 + }, + { + "family": "attention", + "rank": 2, + "share": 0.20220664686439338 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.039044817770315786 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.014406944379520907 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.002019814379523439 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 2.5579606107986748, + "mode_FULL": -0.6775421426296558, + "mode_NONE": 0.2503307644752007, + "mode_PIECEWISE": 0.5907929693818461, + "prefill_fraction": 0.6661033678566376, + "scheduled_tokens": 0.05206272885676596 + }, + "valid": false + }, + "shares": { + "attention": 0.20220664686439338, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.002019814379523439, + "moe_gemm": 0.7306681072672004, + "moe_router": 0.014406944379520907, + "norm_elementwise": 0.039044817770315786, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 1078.7899999999981, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 718.971000000001, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 663.9299999999997, + "name": "triton_red_fused_2" + }, + { + "duration_us": 22.880000000000003, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 22.017, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.112, + "name": "triton_red_fused_1" + }, + { + "duration_us": 10.974999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.09568605186600879, + "attention_mixed": 0.025924532025613695, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9819166279177437, + "mode_shares": { + "FULL": { + "attention": 0.14040386061386476, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.006289384167983417, + "moe_gemm": 0.7278572814656825, + "moe_router": 0.03517706269456804, + "norm_elementwise": 0.06867748249282202, + "sampler": 0.0 + }, + "PIECEWISE": { + "attention": 0.08139719907850929, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.0022138539380632135, + "moe_gemm": 0.8578037652991323, + "moe_router": 0.01403539873195149, + "norm_elementwise": 0.033980351851418185, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 7, + "PIECEWISE": 1 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.7692444768470234 + }, + { + "family": "attention", + "rank": 2, + "share": 0.12161058389162249 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.05762664923697027 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.028443566325178558 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.004991351616949132 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.19954189900373331, + "mode_FULL": -0.10469513477046205, + "mode_NONE": -0.32861907697425397, + "mode_PIECEWISE": 0.29736821980293143, + "prefill_fraction": 0.10420387771831544, + "scheduled_tokens": -0.23288793337699637 + }, + "valid": false + }, + "shares": { + "attention": 0.12161058389162249, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.004991351616949132, + "moe_gemm": 0.7692444768470234, + "moe_router": 0.028443566325178558, + "norm_elementwise": 0.05762664923697027, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 667.7419999999993, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 523.011999999999, + "name": "triton_red_fused_2" + }, + { + "duration_us": 498.98300000000086, + "name": "void vllm::moe::count_and_sort_expert_tokens_kernel(int const*, int*, int*, int*, unsigned long, int, int, int, bool)" + }, + { + "duration_us": 20.736, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 14.591, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.645, + "name": "triton_red_fused_1" + }, + { + "duration_us": 11.36, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + } + ], + "valid": false, + "window": 2 + } + ], + "P10-C00-moderate": [ + { + "attention_subshares": { + "attention_decode": 0.3087850643091725, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.985252225101006, + "mode_shares": { + "FULL": { + "attention": 0.3087850643091725, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005163136163815447, + "moe_gemm": 0.5777999387887653, + "moe_router": 0.02008575072504131, + "norm_elementwise": 0.07341833511421153, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5777999387887653 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3087850643091725 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.07341833511421153 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.02008575072504131 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.005163136163815447 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": 4.655696267017126, + "mode_FULL": 0.17677689026422408, + "mode_NONE": -0.1682723158645361, + "mode_PIECEWISE": -0.05341411394958018, + "prefill_fraction": -0.12905587442539998, + "scheduled_tokens": -0.09951290348812954 + }, + "valid": false + }, + "shares": { + "attention": 0.3087850643091725, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.005163136163815447, + "moe_gemm": 0.5777999387887653, + "moe_router": 0.02008575072504131, + "norm_elementwise": 0.07341833511421153, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 583.5649999999997, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 466.8209999999995, + "name": "triton_red_fused_2" + }, + { + "duration_us": 19.617, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.536, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 11.231999999999998, + "name": "void at::native::vectorized_gather_kernel<16, long>(char*, char*, long*, int, long, long, long, long, bool)" + }, + { + "duration_us": 10.049, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 1 + }, + { + "attention_subshares": { + "attention_decode": 0.3033378915990537, + "attention_mixed": 0.0, + "attention_prefill": 0.0 + }, + "classifiable_fraction": 0.9736215707793747, + "mode_shares": { + "FULL": { + "attention": 0.3033378915990537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008423712471036498, + "moe_gemm": 0.5030357454817748, + "moe_router": 0.03618969859772223, + "norm_elementwise": 0.12263452262978754, + "sampler": 0.0 + } + }, + "mode_steps": { + "FULL": 8 + }, + "ranking": [ + { + "family": "moe_gemm", + "rank": 1, + "share": 0.5030357454817748 + }, + { + "family": "attention", + "rank": 2, + "share": 0.3033378915990537 + }, + { + "family": "norm_elementwise", + "rank": 3, + "share": 0.12263452262978754 + }, + { + "family": "moe_router", + "rank": 4, + "share": 0.03618969859772223 + }, + { + "family": "kv_memory", + "rank": 5, + "share": 0.008423712471036498 + }, + { + "family": "collective", + "rank": 5, + "share": 0.0 + }, + { + "family": "sampler", + "rank": 5, + "share": 0.0 + }, + { + "family": "dense_gemm", + "rank": 5, + "share": 0.0 + } + ], + "representativeness": { + "smd": { + "decode_batch_size": -0.5055344205281964, + "mode_FULL": 0.17677689026422408, + "mode_NONE": -0.1682723158645361, + "mode_PIECEWISE": -0.05341411394958018, + "prefill_fraction": -0.12905587442539998, + "scheduled_tokens": -0.10506250419671144 + }, + "valid": false + }, + "shares": { + "attention": 0.3033378915990537, + "collective": 0.0, + "dense_gemm": 0.0, + "kv_memory": 0.008423712471036498, + "moe_gemm": 0.5030357454817748, + "moe_router": 0.03618969859772223, + "norm_elementwise": 0.12263452262978754, + "sampler": 0.0 + }, + "top_unmatched": [ + { + "duration_us": 590.9459999999992, + "name": "triton_poi_fused_3" + }, + { + "duration_us": 467.6929999999989, + "name": "triton_red_fused_2" + }, + { + "duration_us": 18.272, + "name": "_compute_slot_mapping_kernel" + }, + { + "duration_us": 13.152, + "name": "triton_poi_fused_2" + }, + { + "duration_us": 9.63, + "name": "triton_red_fused_1" + } + ], + "valid": false, + "window": 2 + } + ] + }, + "ranking": { + "accepted_inversions": [], + "candidates": [], + "completed_patterns": [ + "P01", + "P02", + "P03", + "P04", + "P06", + "P07", + "P08", + "P09", + "P10" + ], + "evaluable_patterns": [ + "P04" + ], + "invalid_primary_windows": [ + "P01-C00-moderate: window 1", + "P01-C00-moderate: window 2", + "P02-C00-moderate: window 1", + "P02-C00-moderate: window 2", + "P03-C00-moderate: window 1", + "P03-C00-moderate: window 2", + "P06-C00-moderate: window 1", + "P06-C00-moderate: window 2", + "P07-C00-moderate: window 1", + "P07-C00-moderate: window 2", + "P08-C00-moderate: window 1", + "P08-C00-moderate: window 2", + "P09-C00-moderate: window 1", + "P09-C00-moderate: window 2", + "P10-C00-moderate: window 1", + "P10-C00-moderate: window 2" + ], + "kendall_tau_b": [], + "missing_patterns": [ + "P05", + "P11" + ], + "tests": 1540 + }, + "robustness": { + "confirmation_runs": "NOT_EVALUABLE: all four confirmations are missing", + "mixed_interference": "leave-one-pattern-out fits applied within config/load", + "operator_ranking": "two fixed wall-separated windows; no separate LOAO procedure was preregistered" + }, + "runs": { + "P01-C00-moderate": { + "blocks": [ + { + "bucket": 30576.0, + "duration_ms": 9977.070185999999, + "eligible_miss": 0.0, + "miss": 33.0, + "model": 116.0, + "overflow": 33.0, + "padding": 603.0, + "tokens": 50505.0 + }, + { + "bucket": 29728.0, + "duration_ms": 10025.674661000003, + "eligible_miss": 0.0, + "miss": 33.0, + "model": 119.0, + "overflow": 33.0, + "padding": 537.0, + "tokens": 49844.0 + }, + { + "bucket": 31568.0, + "duration_ms": 9953.621208999997, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 120.0, + "overflow": 29.0, + "padding": 545.0, + "tokens": 49319.0 + }, + { + "bucket": 26832.0, + "duration_ms": 9914.660198999998, + "eligible_miss": 0.0, + "miss": 38.0, + "model": 116.0, + "overflow": 38.0, + "padding": 503.0, + "tokens": 49709.0 + }, + { + "bucket": 31992.0, + "duration_ms": 10027.674603000003, + "eligible_miss": 0.0, + "miss": 27.0, + "model": 126.0, + "overflow": 27.0, + "padding": 574.0, + "tokens": 47825.0 + }, + { + "bucket": 33224.0, + "duration_ms": 9940.844814, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 120.0, + "overflow": 29.0, + "padding": 627.0, + "tokens": 49561.0 + }, + { + "bucket": 33456.0, + "duration_ms": 10001.052717999995, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 128.0, + "overflow": 26.0, + "padding": 565.0, + "tokens": 48243.0 + }, + { + "bucket": 34136.0, + "duration_ms": 9976.993478999995, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 123.0, + "overflow": 24.0, + "padding": 618.0, + "tokens": 49143.0 + }, + { + "bucket": 31224.0, + "duration_ms": 9995.576178, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 118.0, + "overflow": 32.0, + "padding": 610.0, + "tokens": 50258.0 + }, + { + "bucket": 31800.0, + "duration_ms": 9989.647112, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 121.0, + "overflow": 29.0, + "padding": 583.0, + "tokens": 49181.0 + }, + { + "bucket": 36560.0, + "duration_ms": 9966.385023999997, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 125.0, + "overflow": 21.0, + "padding": 660.0, + "tokens": 48591.0 + }, + { + "bucket": 25560.0, + "duration_ms": 10003.187468000002, + "eligible_miss": 0.0, + "miss": 43.0, + "model": 117.0, + "overflow": 43.0, + "padding": 490.0, + "tokens": 51245.0 + }, + { + "bucket": 27448.0, + "duration_ms": 9971.533022999996, + "eligible_miss": 0.0, + "miss": 39.0, + "model": 113.0, + "overflow": 39.0, + "padding": 525.0, + "tokens": 51806.0 + }, + { + "bucket": 29384.0, + "duration_ms": 9983.114686000003, + "eligible_miss": 0.0, + "miss": 35.0, + "model": 115.0, + "overflow": 35.0, + "padding": 564.0, + "tokens": 50967.0 + }, + { + "bucket": 29688.0, + "duration_ms": 9990.978558, + "eligible_miss": 0.0, + "miss": 34.0, + "model": 121.0, + "overflow": 34.0, + "padding": 571.0, + "tokens": 49442.0 + }, + { + "bucket": 33376.0, + "duration_ms": 9995.658681999994, + "eligible_miss": 0.0, + "miss": 27.0, + "model": 121.0, + "overflow": 27.0, + "padding": 615.0, + "tokens": 49129.0 + }, + { + "bucket": 31432.0, + "duration_ms": 10015.605592999998, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 118.0, + "overflow": 31.0, + "padding": 599.0, + "tokens": 50125.0 + }, + { + "bucket": 29048.0, + "duration_ms": 9974.7466, + "eligible_miss": 0.0, + "miss": 35.0, + "model": 116.0, + "overflow": 35.0, + "padding": 529.0, + "tokens": 50381.0 + }, + { + "bucket": 31256.0, + "duration_ms": 9993.147903, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 121.0, + "overflow": 32.0, + "padding": 660.0, + "tokens": 49683.0 + }, + { + "bucket": 33984.0, + "duration_ms": 9963.886105999996, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 121.0, + "overflow": 26.0, + "padding": 658.0, + "tokens": 49368.0 + }, + { + "bucket": 31320.0, + "duration_ms": 10029.250221999997, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 118.0, + "overflow": 32.0, + "padding": 528.0, + "tokens": 50904.0 + }, + { + "bucket": 31120.0, + "duration_ms": 10007.011493000004, + "eligible_miss": 0.0, + "miss": 33.0, + "model": 114.0, + "overflow": 33.0, + "padding": 499.0, + "tokens": 51082.0 + }, + { + "bucket": 33296.0, + "duration_ms": 9953.774958000002, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 121.0, + "overflow": 26.0, + "padding": 607.0, + "tokens": 48871.0 + }, + { + "bucket": 30576.0, + "duration_ms": 9998.836994000001, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 120.0, + "overflow": 32.0, + "padding": 633.0, + "tokens": 50044.0 + }, + { + "bucket": 35040.0, + "duration_ms": 9907.035887999995, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 121.0, + "overflow": 22.0, + "padding": 614.0, + "tokens": 48318.0 + }, + { + "bucket": 31816.0, + "duration_ms": 9997.940913000004, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 114.0, + "overflow": 30.0, + "padding": 557.0, + "tokens": 51711.0 + }, + { + "bucket": 31464.0, + "duration_ms": 10054.742900999996, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 122.0, + "overflow": 30.0, + "padding": 605.0, + "tokens": 48442.0 + }, + { + "bucket": 31064.0, + "duration_ms": 9932.345599, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 120.0, + "overflow": 32.0, + "padding": 589.0, + "tokens": 49398.0 + }, + { + "bucket": 32760.0, + "duration_ms": 10010.170066999997, + "eligible_miss": 0.0, + "miss": 27.0, + "model": 128.0, + "overflow": 27.0, + "padding": 660.0, + "tokens": 47928.0 + }, + { + "bucket": 33432.0, + "duration_ms": 9968.922715000002, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 118.0, + "overflow": 26.0, + "padding": 671.0, + "tokens": 49679.0 + }, + { + "bucket": 29392.0, + "duration_ms": 10013.087841999995, + "eligible_miss": 0.0, + "miss": 34.0, + "model": 118.0, + "overflow": 34.0, + "padding": 541.0, + "tokens": 49880.0 + }, + { + "bucket": 28832.0, + "duration_ms": 9938.452419999998, + "eligible_miss": 0.0, + "miss": 35.0, + "model": 115.0, + "overflow": 35.0, + "padding": 559.0, + "tokens": 51143.0 + }, + { + "bucket": 33672.0, + "duration_ms": 10049.196139999996, + "eligible_miss": 0.0, + "miss": 27.0, + "model": 124.0, + "overflow": 27.0, + "padding": 689.0, + "tokens": 48878.0 + }, + { + "bucket": 35856.0, + "duration_ms": 9952.635497000005, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 123.0, + "overflow": 21.0, + "padding": 647.0, + "tokens": 48258.0 + }, + { + "bucket": 27608.0, + "duration_ms": 9958.917514999996, + "eligible_miss": 0.0, + "miss": 39.0, + "model": 112.0, + "overflow": 39.0, + "padding": 503.0, + "tokens": 51781.0 + }, + { + "bucket": 35256.0, + "duration_ms": 10019.95564, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 126.0, + "overflow": 22.0, + "padding": 667.0, + "tokens": 47965.0 + }, + { + "bucket": 36024.0, + "duration_ms": 9971.399555000004, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 124.0, + "overflow": 21.0, + "padding": 743.0, + "tokens": 47961.0 + }, + { + "bucket": 35056.0, + "duration_ms": 9981.311274000002, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 127.0, + "overflow": 22.0, + "padding": 664.0, + "tokens": 47846.0 + }, + { + "bucket": 27520.0, + "duration_ms": 10008.593500999998, + "eligible_miss": 0.0, + "miss": 39.0, + "model": 115.0, + "overflow": 39.0, + "padding": 460.0, + "tokens": 52024.0 + }, + { + "bucket": 32248.0, + "duration_ms": 9955.697573, + "eligible_miss": 0.0, + "miss": 28.0, + "model": 122.0, + "overflow": 28.0, + "padding": 566.0, + "tokens": 48422.0 + }, + { + "bucket": 31312.0, + "duration_ms": 9951.151515, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 122.0, + "overflow": 31.0, + "padding": 524.0, + "tokens": 48815.0 + }, + { + "bucket": 36640.0, + "duration_ms": 10005.843674999998, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 126.0, + "overflow": 19.0, + "padding": 675.0, + "tokens": 47758.0 + }, + { + "bucket": 36104.0, + "duration_ms": 9967.80646, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 128.0, + "overflow": 21.0, + "padding": 695.0, + "tokens": 47577.0 + }, + { + "bucket": 31296.0, + "duration_ms": 10039.968077000001, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 120.0, + "overflow": 31.0, + "padding": 603.0, + "tokens": 50038.0 + }, + { + "bucket": 28720.0, + "duration_ms": 9947.777837999993, + "eligible_miss": 0.0, + "miss": 37.0, + "model": 115.0, + "overflow": 37.0, + "padding": 521.0, + "tokens": 50984.0 + }, + { + "bucket": 30752.0, + "duration_ms": 9989.05891, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 115.0, + "overflow": 32.0, + "padding": 582.0, + "tokens": 50810.0 + }, + { + "bucket": 31528.0, + "duration_ms": 10007.517981999998, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 117.0, + "overflow": 30.0, + "padding": 631.0, + "tokens": 50638.0 + }, + { + "bucket": 33064.0, + "duration_ms": 10068.188278999998, + "eligible_miss": 0.0, + "miss": 28.0, + "model": 120.0, + "overflow": 28.0, + "padding": 637.0, + "tokens": 49561.0 + } + ], + "clean": { + "admitted": 6235, + "completed": 6232, + "completed_throughput_rps": 25.966666666666665, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1986493, + "offered_rps": 25.979166666666668, + "output_tokens": 398848, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 1.0203027040115558, + "latency_s": { + "e2e": { + "mean": 2.7387289799440433, + "n": 6232, + "p50": 2.737939690996427, + "p95": 2.9280695282504894, + "p99": 2.993426457623136 + }, + "tpot": { + "mean": 0.04167998409546577, + "n": 6232, + "p50": 0.04166355157120427, + "p95": 0.04453765388101428, + "p99": 0.045556931762553245 + }, + "ttft": { + "mean": 0.11288998192969947, + "n": 6232, + "p50": 0.11091382149606943, + "p95": 0.1447070966445608, + "p99": 0.1583496381092118 + } + }, + "layer1": { + "decode_tokens": 392669, + "kv_usage_max": 0.10235198702351989, + "kv_usage_mean": 0.08515698687332914, + "mode_counts": { + "FULL": 114, + "NONE": 1430, + "PIECEWISE": 4216 + }, + "mode_shares": { + "FULL": 0.019791666666666666, + "NONE": 0.2482638888888889, + "PIECEWISE": 0.7319444444444444 + }, + "preemptions": 0, + "prefill_tokens": 1988372, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 8361, + "records_clean": 5760, + "requests_per_step": { + "mean": 69.25416666666666, + "n": 5760, + "p50": 69.0, + "p95": 74.0, + "p99": 76.0 + }, + "scheduled_tokens_per_step": { + "mean": 413.3751736111111, + "n": 5760, + "p50": 402.0, + "p95": 711.0, + "p99": 922.0 + }, + "step_duration_ms": { + "mean": 83.22007816753472, + "n": 5760, + "p50": 83.618554, + "p95": 100.98220115, + "p99": 111.06512396999999 + }, + "token_efficiency_per_ms": 4.967252887925961 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 26.15, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 26.2, + "completion_rate_within_10pct": true, + "tail_end_s": 340.92300570901716, + "tail_start_s": 330.92300570901716, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 26.15, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 26.3, + "completion_rate_within_10pct": true, + "tail_end_s": 381.4778159119887, + "tail_start_s": 371.4778159119887, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 6232, + "run_dir": "runs/phase3/primary/P01-C00/moderate", + "run_id": "P01-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.018691968735246287, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.2482638888888889, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 5646, + "moe_layer_cv": null, + "overflow_rate": 0.2482638888888889, + "padded_tokens_per_useful_token": 0.011972074399390855, + "padding_fraction": 0.018691968735246287 + } + }, + "P01-C00-saturation": { + "blocks": [ + { + "bucket": 13312.0, + "duration_ms": 9700.4556, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 60.0, + "overflow": 8.0, + "padding": 1.0, + "tokens": 73007.0 + }, + { + "bucket": 7536.0, + "duration_ms": 9932.258910999999, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 41.0, + "overflow": 12.0, + "padding": 19.0, + "tokens": 90414.0 + }, + { + "bucket": 6888.0, + "duration_ms": 10178.935983, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 40.0, + "overflow": 13.0, + "padding": 5.0, + "tokens": 94514.0 + }, + { + "bucket": 13960.0, + "duration_ms": 10457.795198000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 7.0, + "tokens": 85573.0 + }, + { + "bucket": 14040.0, + "duration_ms": 10337.022535999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 6.0, + "tokens": 81261.0 + }, + { + "bucket": 13544.0, + "duration_ms": 9714.938590000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 63.0, + "overflow": 10.0, + "padding": 5.0, + "tokens": 75029.0 + }, + { + "bucket": 13544.0, + "duration_ms": 9903.635449, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 63.0, + "overflow": 10.0, + "padding": 5.0, + "tokens": 75289.0 + }, + { + "bucket": 7168.0, + "duration_ms": 9403.655798000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 38.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 85622.0 + }, + { + "bucket": 7800.0, + "duration_ms": 9955.373708000003, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 42.0, + "overflow": 12.0, + "padding": 19.0, + "tokens": 88933.0 + }, + { + "bucket": 12728.0, + "duration_ms": 10579.801631999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 61.0, + "overflow": 11.0, + "padding": 6.0, + "tokens": 90349.0 + }, + { + "bucket": 13768.0, + "duration_ms": 9765.536687999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 63.0, + "overflow": 9.0, + "padding": 16.0, + "tokens": 76354.0 + }, + { + "bucket": 14200.0, + "duration_ms": 10142.375634000004, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 64.0, + "overflow": 9.0, + "padding": 23.0, + "tokens": 80248.0 + }, + { + "bucket": 13768.0, + "duration_ms": 10205.201202, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 13.0, + "tokens": 80905.0 + }, + { + "bucket": 11008.0, + "duration_ms": 9274.425237, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 51.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 74036.0 + }, + { + "bucket": 7624.0, + "duration_ms": 9966.902078, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 42.0, + "overflow": 12.0, + "padding": 13.0, + "tokens": 90176.0 + }, + { + "bucket": 8904.0, + "duration_ms": 10401.74154, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 47.0, + "overflow": 12.0, + "padding": 11.0, + "tokens": 95189.0 + }, + { + "bucket": 13768.0, + "duration_ms": 9996.728811, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 11.0, + "tokens": 78683.0 + }, + { + "bucket": 13768.0, + "duration_ms": 10098.225219, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 11.0, + "tokens": 81135.0 + }, + { + "bucket": 13768.0, + "duration_ms": 10336.009350000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 11.0, + "tokens": 82320.0 + }, + { + "bucket": 14232.0, + "duration_ms": 9508.483533000002, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 63.0, + "overflow": 8.0, + "padding": 25.0, + "tokens": 74058.0 + }, + { + "bucket": 11520.0, + "duration_ms": 9614.615344999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 54.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 77327.0 + }, + { + "bucket": 6344.0, + "duration_ms": 10036.276204, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 38.0, + "overflow": 13.0, + "padding": 11.0, + "tokens": 93120.0 + }, + { + "bucket": 9416.0, + "duration_ms": 10347.189429000002, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 49.0, + "overflow": 12.0, + "padding": 11.0, + "tokens": 93721.0 + }, + { + "bucket": 13512.0, + "duration_ms": 10252.652897, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 64.0, + "overflow": 11.0, + "padding": 11.0, + "tokens": 81378.0 + }, + { + "bucket": 13976.0, + "duration_ms": 10021.668533, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 22.0, + "tokens": 79064.0 + }, + { + "bucket": 13928.0, + "duration_ms": 10279.847324999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 20.0, + "tokens": 82321.0 + }, + { + "bucket": 13512.0, + "duration_ms": 9703.471726999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 63.0, + "overflow": 10.0, + "padding": 11.0, + "tokens": 75946.0 + }, + { + "bucket": 8448.0, + "duration_ms": 9304.457085999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 43.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 78700.0 + }, + { + "bucket": 7112.0, + "duration_ms": 10029.146659999999, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 41.0, + "overflow": 13.0, + "padding": 11.0, + "tokens": 90998.0 + }, + { + "bucket": 11656.0, + "duration_ms": 10649.044713000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 57.0, + "overflow": 11.0, + "padding": 11.0, + "tokens": 93073.0 + }, + { + "bucket": 14176.0, + "duration_ms": 10128.805941, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 64.0, + "overflow": 9.0, + "padding": 15.0, + "tokens": 80561.0 + }, + { + "bucket": 14192.0, + "duration_ms": 10116.131985000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 64.0, + "overflow": 9.0, + "padding": 14.0, + "tokens": 80483.0 + }, + { + "bucket": 13768.0, + "duration_ms": 9503.189379, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 63.0, + "overflow": 9.0, + "padding": 15.0, + "tokens": 73874.0 + }, + { + "bucket": 12032.0, + "duration_ms": 9550.747098, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 55.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 74799.0 + }, + { + "bucket": 9160.0, + "duration_ms": 9953.987155999997, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 48.0, + "overflow": 12.0, + "padding": 14.0, + "tokens": 87139.0 + }, + { + "bucket": 7368.0, + "duration_ms": 10039.995024, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 41.0, + "overflow": 12.0, + "padding": 14.0, + "tokens": 91397.0 + }, + { + "bucket": 13192.0, + "duration_ms": 10544.197295000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 61.0, + "overflow": 10.0, + "padding": 17.0, + "tokens": 89660.0 + }, + { + "bucket": 13768.0, + "duration_ms": 10257.515040999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 14.0, + "tokens": 82262.0 + }, + { + "bucket": 13768.0, + "duration_ms": 9520.285963, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 63.0, + "overflow": 9.0, + "padding": 14.0, + "tokens": 73873.0 + }, + { + "bucket": 14024.0, + "duration_ms": 10389.161113000004, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 15.0, + "tokens": 82860.0 + }, + { + "bucket": 11264.0, + "duration_ms": 9225.102485, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 53.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 72871.0 + }, + { + "bucket": 6360.0, + "duration_ms": 10000.893427, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 38.0, + "overflow": 13.0, + "padding": 2.0, + "tokens": 91753.0 + }, + { + "bucket": 9432.0, + "duration_ms": 10636.165116999997, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 50.0, + "overflow": 13.0, + "padding": 2.0, + "tokens": 93649.0 + }, + { + "bucket": 13528.0, + "duration_ms": 9840.476839, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 63.0, + "overflow": 10.0, + "padding": 2.0, + "tokens": 79400.0 + }, + { + "bucket": 13528.0, + "duration_ms": 10080.211508999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 64.0, + "overflow": 11.0, + "padding": 2.0, + "tokens": 80608.0 + }, + { + "bucket": 14056.0, + "duration_ms": 9981.313267000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 64.0, + "overflow": 10.0, + "padding": 2.0, + "tokens": 78511.0 + }, + { + "bucket": 13304.0, + "duration_ms": 9443.992412999998, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 60.0, + "overflow": 8.0, + "padding": 2.0, + "tokens": 70921.0 + }, + { + "bucket": 6128.0, + "duration_ms": 9931.941679999996, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 37.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 92355.0 + } + ], + "clean": { + "admitted": 10392, + "completed": 10392, + "completed_throughput_rps": 43.3, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3310593, + "offered_rps": 43.3, + "output_tokens": 665088, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 2.2563367690017913, + "latency_s": { + "e2e": { + "mean": 5.900970134643903, + "n": 10392, + "p50": 5.894884387002094, + "p95": 6.19822557131265, + "p99": 6.350330786115083 + }, + "tpot": { + "mean": 0.07734891378714435, + "n": 10392, + "p50": 0.07630683034117933, + "p95": 0.08648301882954362, + "p99": 0.08921301509375178 + }, + "ttft": { + "mean": 1.0279885660538086, + "n": 10392, + "p50": 1.0842870905034943, + "p95": 1.5127993570451501, + "p99": 1.5493279498213086 + } + }, + "layer1": { + "decode_tokens": 652646, + "kv_usage_max": 0.3451743714517437, + "kv_usage_mean": 0.304654382467603, + "mode_counts": { + "FULL": 2164, + "NONE": 498, + "PIECEWISE": 13 + }, + "mode_shares": { + "FULL": 0.8089719626168225, + "NONE": 0.18616822429906543, + "PIECEWISE": 0.00485981308411215 + }, + "preemptions": 0, + "prefill_tokens": 3323073, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 23, + "queue_waiting_mean": 0.2680373831775701, + "records_all": 4187, + "records_clean": 2675, + "requests_per_step": { + "mean": 247.93570093457944, + "n": 2675, + "p50": 256.0, + "p95": 256.0, + "p99": 256.0 + }, + "scheduled_tokens_per_step": { + "mean": 1486.2500934579439, + "n": 2675, + "p50": 256.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 179.15588237308413, + "n": 2675, + "p50": 70.26221, + "p95": 757.4315151999998, + "p99": 772.54264124 + }, + "token_efficiency_per_ms": 8.29584869763246 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 41.35, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 36.7, + "completion_rate_within_10pct": false, + "tail_end_s": 337.969882845995, + "tail_start_s": 327.969882845995, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 41.35, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 38.8, + "completion_rate_within_10pct": true, + "tail_end_s": 392.74431553299655, + "tail_start_s": 382.74431553299655, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 10392, + "run_dir": "runs/phase3/primary/P01-C00/saturation", + "run_id": "P01-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.0008228755826461097, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.18616822429906543, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.18616822429906543, + "padded_tokens_per_useful_token": 0.00011545081531164552, + "padding_fraction": 0.0008228755826461097 + } + }, + "P01-C01-moderate": { + "blocks": [ + { + "bucket": 33608.0, + "duration_ms": 9997.635068999996, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 140.0, + "overflow": 23.0, + "padding": 680.0, + "tokens": 46601.0 + }, + { + "bucket": 37032.0, + "duration_ms": 9961.48037, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 133.0, + "overflow": 19.0, + "padding": 670.0, + "tokens": 47088.0 + }, + { + "bucket": 35760.0, + "duration_ms": 10047.266813000002, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 137.0, + "overflow": 20.0, + "padding": 683.0, + "tokens": 46599.0 + }, + { + "bucket": 35280.0, + "duration_ms": 9931.102751999999, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 137.0, + "overflow": 20.0, + "padding": 695.0, + "tokens": 45552.0 + }, + { + "bucket": 32384.0, + "duration_ms": 10010.852824999993, + "eligible_miss": 0.0, + "miss": 28.0, + "model": 133.0, + "overflow": 28.0, + "padding": 596.0, + "tokens": 47714.0 + }, + { + "bucket": 35808.0, + "duration_ms": 9985.833894000005, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 141.0, + "overflow": 18.0, + "padding": 696.0, + "tokens": 44852.0 + }, + { + "bucket": 34440.0, + "duration_ms": 9966.572230000005, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 138.0, + "overflow": 22.0, + "padding": 753.0, + "tokens": 46096.0 + }, + { + "bucket": 36392.0, + "duration_ms": 10001.279647999996, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 145.0, + "overflow": 16.0, + "padding": 671.0, + "tokens": 44970.0 + }, + { + "bucket": 36648.0, + "duration_ms": 10050.746529000007, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 144.0, + "overflow": 18.0, + "padding": 756.0, + "tokens": 46045.0 + }, + { + "bucket": 36416.0, + "duration_ms": 9897.770381000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 137.0, + "overflow": 17.0, + "padding": 703.0, + "tokens": 45233.0 + }, + { + "bucket": 34080.0, + "duration_ms": 10010.004142999991, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 131.0, + "overflow": 24.0, + "padding": 737.0, + "tokens": 47680.0 + }, + { + "bucket": 36512.0, + "duration_ms": 10016.691993000004, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 144.0, + "overflow": 18.0, + "padding": 656.0, + "tokens": 46337.0 + }, + { + "bucket": 34792.0, + "duration_ms": 9929.375679999992, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 141.0, + "overflow": 20.0, + "padding": 690.0, + "tokens": 45587.0 + }, + { + "bucket": 31120.0, + "duration_ms": 10033.621968, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 130.0, + "overflow": 31.0, + "padding": 628.0, + "tokens": 48792.0 + }, + { + "bucket": 34576.0, + "duration_ms": 9930.906926000003, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 129.0, + "overflow": 24.0, + "padding": 732.0, + "tokens": 48000.0 + }, + { + "bucket": 36664.0, + "duration_ms": 10031.479860999998, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 135.0, + "overflow": 19.0, + "padding": 692.0, + "tokens": 46562.0 + }, + { + "bucket": 33376.0, + "duration_ms": 9948.186458999993, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 135.0, + "overflow": 24.0, + "padding": 727.0, + "tokens": 46551.0 + }, + { + "bucket": 36848.0, + "duration_ms": 10050.385339000004, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 140.0, + "overflow": 18.0, + "padding": 711.0, + "tokens": 46130.0 + }, + { + "bucket": 35712.0, + "duration_ms": 9975.735268999999, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 136.0, + "overflow": 21.0, + "padding": 695.0, + "tokens": 46545.0 + }, + { + "bucket": 33352.0, + "duration_ms": 9940.775106000003, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 130.0, + "overflow": 26.0, + "padding": 682.0, + "tokens": 46963.0 + }, + { + "bucket": 34840.0, + "duration_ms": 9966.589301999995, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 141.0, + "overflow": 22.0, + "padding": 690.0, + "tokens": 46120.0 + }, + { + "bucket": 34928.0, + "duration_ms": 9994.235396, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 141.0, + "overflow": 22.0, + "padding": 747.0, + "tokens": 46500.0 + }, + { + "bucket": 36512.0, + "duration_ms": 10025.756066000002, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 138.0, + "overflow": 18.0, + "padding": 722.0, + "tokens": 47204.0 + }, + { + "bucket": 33312.0, + "duration_ms": 9926.436303000002, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 128.0, + "overflow": 26.0, + "padding": 617.0, + "tokens": 48076.0 + }, + { + "bucket": 35520.0, + "duration_ms": 10003.942259999994, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 139.0, + "overflow": 21.0, + "padding": 715.0, + "tokens": 46573.0 + }, + { + "bucket": 32648.0, + "duration_ms": 9988.033101000003, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 147.0, + "overflow": 22.0, + "padding": 694.0, + "tokens": 44739.0 + }, + { + "bucket": 35728.0, + "duration_ms": 10012.5605, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 134.0, + "overflow": 22.0, + "padding": 742.0, + "tokens": 47332.0 + }, + { + "bucket": 40168.0, + "duration_ms": 9936.560732999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 136.0, + "overflow": 11.0, + "padding": 755.0, + "tokens": 45670.0 + }, + { + "bucket": 34392.0, + "duration_ms": 10016.966579000002, + "eligible_miss": 0.0, + "miss": 27.0, + "model": 124.0, + "overflow": 27.0, + "padding": 694.0, + "tokens": 48772.0 + }, + { + "bucket": 35208.0, + "duration_ms": 10000.276685000003, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 146.0, + "overflow": 18.0, + "padding": 687.0, + "tokens": 45124.0 + }, + { + "bucket": 35912.0, + "duration_ms": 9956.369649000004, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 145.0, + "overflow": 16.0, + "padding": 730.0, + "tokens": 44880.0 + }, + { + "bucket": 38944.0, + "duration_ms": 10032.647223000002, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 144.0, + "overflow": 13.0, + "padding": 751.0, + "tokens": 45459.0 + }, + { + "bucket": 38576.0, + "duration_ms": 9961.166588000004, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 136.0, + "overflow": 15.0, + "padding": 726.0, + "tokens": 46648.0 + }, + { + "bucket": 34832.0, + "duration_ms": 9956.285011999998, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 132.0, + "overflow": 22.0, + "padding": 703.0, + "tokens": 46455.0 + }, + { + "bucket": 34768.0, + "duration_ms": 10025.576010999996, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 130.0, + "overflow": 24.0, + "padding": 710.0, + "tokens": 47930.0 + }, + { + "bucket": 37920.0, + "duration_ms": 9986.476466, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 142.0, + "overflow": 15.0, + "padding": 785.0, + "tokens": 45329.0 + }, + { + "bucket": 36936.0, + "duration_ms": 9928.282509000004, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 139.0, + "overflow": 17.0, + "padding": 750.0, + "tokens": 45710.0 + }, + { + "bucket": 35872.0, + "duration_ms": 10044.772734000007, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 129.0, + "overflow": 23.0, + "padding": 702.0, + "tokens": 48508.0 + }, + { + "bucket": 37512.0, + "duration_ms": 9915.907078000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 131.0, + "overflow": 17.0, + "padding": 709.0, + "tokens": 45909.0 + }, + { + "bucket": 40648.0, + "duration_ms": 9987.487444, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 146.0, + "overflow": 8.0, + "padding": 845.0, + "tokens": 44254.0 + }, + { + "bucket": 38336.0, + "duration_ms": 10042.002508, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 149.0, + "overflow": 12.0, + "padding": 796.0, + "tokens": 45324.0 + }, + { + "bucket": 36144.0, + "duration_ms": 9982.605775999999, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 141.0, + "overflow": 19.0, + "padding": 816.0, + "tokens": 45801.0 + }, + { + "bucket": 31264.0, + "duration_ms": 9969.342419999999, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 130.0, + "overflow": 30.0, + "padding": 591.0, + "tokens": 47244.0 + }, + { + "bucket": 33888.0, + "duration_ms": 9940.104527999998, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 149.0, + "overflow": 18.0, + "padding": 751.0, + "tokens": 44902.0 + }, + { + "bucket": 35936.0, + "duration_ms": 10048.677680999996, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 140.0, + "overflow": 20.0, + "padding": 727.0, + "tokens": 45967.0 + }, + { + "bucket": 40856.0, + "duration_ms": 9962.541356000005, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 141.0, + "overflow": 8.0, + "padding": 765.0, + "tokens": 44470.0 + }, + { + "bucket": 36200.0, + "duration_ms": 9954.206755000001, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 144.0, + "overflow": 16.0, + "padding": 699.0, + "tokens": 44968.0 + }, + { + "bucket": 33544.0, + "duration_ms": 10076.081104, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 140.0, + "overflow": 24.0, + "padding": 648.0, + "tokens": 46552.0 + } + ], + "clean": { + "admitted": 5828, + "completed": 5828, + "completed_throughput_rps": 24.283333333333335, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1855465, + "offered_rps": 24.283333333333335, + "output_tokens": 372992, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 0.9099243679956999, + "latency_s": { + "e2e": { + "mean": 2.385056775077354, + "n": 5828, + "p50": 2.3748430520063266, + "p95": 2.627926318097161, + "p99": 2.698679346006247 + }, + "tpot": { + "mean": 0.03631807814162525, + "n": 5828, + "p50": 0.036205446753661996, + "p95": 0.04003349922697655, + "p99": 0.041193774410538245 + }, + "ttft": { + "mean": 0.09701785215496304, + "n": 5828, + "p50": 0.09653152749524452, + "p95": 0.12175093539408405, + "p99": 0.13423520062555325 + } + }, + "layer1": { + "decode_tokens": 367220, + "kv_usage_max": 0.08770714552139391, + "kv_usage_mean": 0.06821269396718065, + "mode_counts": { + "FULL": 968, + "NONE": 952, + "PIECEWISE": 4698 + }, + "mode_shares": { + "FULL": 0.1462677546086431, + "NONE": 0.1438501057721366, + "PIECEWISE": 0.7098821396192203 + }, + "preemptions": 0, + "prefill_tokens": 1855097, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 9535, + "records_clean": 6618, + "requests_per_step": { + "mean": 56.368691447567244, + "n": 6618, + "p50": 56.0, + "p95": 62.0, + "p99": 64.0 + }, + "scheduled_tokens_per_step": { + "mean": 335.7988818374131, + "n": 6618, + "p50": 345.0, + "p95": 555.0, + "p99": 758.6599999999999 + }, + "step_duration_ms": { + "mean": 72.43269764611665, + "n": 6618, + "p50": 73.99482599999999, + "p95": 92.3781244, + "p99": 96.50266256 + }, + "token_efficiency_per_ms": 4.636012363891521 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 24.05, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 25.5, + "completion_rate_within_10pct": true, + "tail_end_s": 343.72796904700226, + "tail_start_s": 333.72796904700226, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 24.05, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 23.8, + "completion_rate_within_10pct": true, + "tail_end_s": 384.0086190129805, + "tail_start_s": 374.0086190129805, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 5828, + "run_dir": "runs/phase3/primary/P01-C01/moderate", + "run_id": "P01-C01-moderate", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.01992823033576615, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.1438501057721366, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 5650, + "moe_layer_cv": null, + "overflow_rate": 0.1438501057721366, + "padded_tokens_per_useful_token": 0.01535334517982808, + "padding_fraction": 0.01992823033576615 + } + }, + "P01-C01-saturation": { + "blocks": [ + { + "bucket": 4000.0, + "duration_ms": 10203.605463999998, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 56.0, + "overflow": 41.0, + "padding": 15.0, + "tokens": 78120.0 + }, + { + "bucket": 4016.0, + "duration_ms": 9742.766153999997, + "eligible_miss": 0.0, + "miss": 39.0, + "model": 54.0, + "overflow": 39.0, + "padding": 21.0, + "tokens": 73810.0 + }, + { + "bucket": 1536.0, + "duration_ms": 9918.749321999998, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 47.0, + "overflow": 41.0, + "padding": 0.0, + "tokens": 78980.0 + }, + { + "bucket": 2048.0, + "duration_ms": 10299.614912999996, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 50.0, + "overflow": 42.0, + "padding": 6.0, + "tokens": 82678.0 + }, + { + "bucket": 4016.0, + "duration_ms": 9787.362490000001, + "eligible_miss": 0.0, + "miss": 39.0, + "model": 54.0, + "overflow": 39.0, + "padding": 9.0, + "tokens": 74464.0 + }, + { + "bucket": 4096.0, + "duration_ms": 10124.917670999996, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 56.0, + "overflow": 41.0, + "padding": 10.0, + "tokens": 76877.0 + }, + { + "bucket": 3584.0, + "duration_ms": 9860.220744000002, + "eligible_miss": 0.0, + "miss": 39.0, + "model": 53.0, + "overflow": 39.0, + "padding": 6.0, + "tokens": 75880.0 + }, + { + "duration_ms": 10018.159282, + "eligible_miss": 0.0, + "miss": 44.0, + "model": 44.0, + "overflow": 44.0, + "tokens": 81806.0 + }, + { + "bucket": 3584.0, + "duration_ms": 9971.427643, + "eligible_miss": 0.0, + "miss": 40.0, + "model": 54.0, + "overflow": 40.0, + "padding": 6.0, + "tokens": 76676.0 + }, + { + "bucket": 3728.0, + "duration_ms": 9969.959431999998, + "eligible_miss": 0.0, + "miss": 40.0, + "model": 54.0, + "overflow": 40.0, + "padding": 6.0, + "tokens": 75907.0 + }, + { + "bucket": 3328.0, + "duration_ms": 10025.561862000004, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 55.0, + "overflow": 42.0, + "padding": 6.0, + "tokens": 76352.0 + }, + { + "bucket": 3328.0, + "duration_ms": 10026.678248000002, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 54.0, + "overflow": 41.0, + "padding": 6.0, + "tokens": 77493.0 + }, + { + "duration_ms": 9922.874876999997, + "eligible_miss": 0.0, + "miss": 44.0, + "model": 44.0, + "overflow": 44.0, + "tokens": 80650.0 + }, + { + "bucket": 3744.0, + "duration_ms": 10141.583717000001, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 56.0, + "overflow": 42.0, + "padding": 7.0, + "tokens": 77129.0 + }, + { + "bucket": 3328.0, + "duration_ms": 9825.641666, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 54.0, + "overflow": 41.0, + "padding": 6.0, + "tokens": 74853.0 + }, + { + "bucket": 3328.0, + "duration_ms": 10026.996974, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 54.0, + "overflow": 41.0, + "padding": 6.0, + "tokens": 76782.0 + }, + { + "bucket": 3456.0, + "duration_ms": 9989.331988000002, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 54.0, + "overflow": 41.0, + "padding": 13.0, + "tokens": 76731.0 + }, + { + "duration_ms": 10182.908675999997, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 45.0, + "overflow": 45.0, + "tokens": 83111.0 + }, + { + "bucket": 3072.0, + "duration_ms": 9888.974163999997, + "eligible_miss": 0.0, + "miss": 43.0, + "model": 55.0, + "overflow": 43.0, + "padding": 6.0, + "tokens": 74301.0 + }, + { + "bucket": 3488.0, + "duration_ms": 9964.314148999998, + "eligible_miss": 0.0, + "miss": 43.0, + "model": 56.0, + "overflow": 43.0, + "padding": 14.0, + "tokens": 75363.0 + }, + { + "bucket": 3072.0, + "duration_ms": 9960.435744000002, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 53.0, + "overflow": 41.0, + "padding": 6.0, + "tokens": 76792.0 + }, + { + "bucket": 2816.0, + "duration_ms": 9875.558238, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 52.0, + "overflow": 41.0, + "padding": 1.0, + "tokens": 75921.0 + }, + { + "bucket": 704.0, + "duration_ms": 10072.600117999998, + "eligible_miss": 0.0, + "miss": 47.0, + "model": 49.0, + "overflow": 47.0, + "padding": 11.0, + "tokens": 79977.0 + }, + { + "bucket": 3072.0, + "duration_ms": 10144.145703000002, + "eligible_miss": 0.0, + "miss": 43.0, + "model": 55.0, + "overflow": 43.0, + "padding": 6.0, + "tokens": 77811.0 + }, + { + "bucket": 3072.0, + "duration_ms": 10003.565046, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 53.0, + "overflow": 41.0, + "padding": 6.0, + "tokens": 76865.0 + }, + { + "bucket": 3776.0, + "duration_ms": 9818.520830000003, + "eligible_miss": 0.0, + "miss": 40.0, + "model": 53.0, + "overflow": 40.0, + "padding": 20.0, + "tokens": 74777.0 + }, + { + "bucket": 768.0, + "duration_ms": 9910.496845, + "eligible_miss": 0.0, + "miss": 44.0, + "model": 47.0, + "overflow": 44.0, + "padding": 0.0, + "tokens": 78154.0 + }, + { + "bucket": 2496.0, + "duration_ms": 10102.562553000002, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 54.0, + "overflow": 45.0, + "padding": 9.0, + "tokens": 78424.0 + }, + { + "bucket": 3216.0, + "duration_ms": 9924.138751, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 54.0, + "overflow": 42.0, + "padding": 7.0, + "tokens": 75864.0 + }, + { + "bucket": 2816.0, + "duration_ms": 10187.513168, + "eligible_miss": 0.0, + "miss": 44.0, + "model": 55.0, + "overflow": 44.0, + "padding": 6.0, + "tokens": 77625.0 + }, + { + "bucket": 3328.0, + "duration_ms": 9789.426399000002, + "eligible_miss": 0.0, + "miss": 40.0, + "model": 52.0, + "overflow": 40.0, + "padding": 6.0, + "tokens": 74936.0 + }, + { + "duration_ms": 10114.070901, + "eligible_miss": 0.0, + "miss": 46.0, + "model": 46.0, + "overflow": 46.0, + "tokens": 81214.0 + }, + { + "bucket": 2816.0, + "duration_ms": 10088.948949999998, + "eligible_miss": 0.0, + "miss": 44.0, + "model": 55.0, + "overflow": 44.0, + "padding": 6.0, + "tokens": 76677.0 + }, + { + "bucket": 3232.0, + "duration_ms": 9866.455802, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 54.0, + "overflow": 42.0, + "padding": 19.0, + "tokens": 74833.0 + }, + { + "bucket": 3264.0, + "duration_ms": 9894.975111999998, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 54.0, + "overflow": 42.0, + "padding": 21.0, + "tokens": 74962.0 + }, + { + "bucket": 3072.0, + "duration_ms": 10085.450928999997, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 53.0, + "overflow": 42.0, + "padding": 13.0, + "tokens": 78380.0 + }, + { + "duration_ms": 10046.627470999998, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 45.0, + "overflow": 45.0, + "tokens": 81298.0 + }, + { + "bucket": 2560.0, + "duration_ms": 9847.101603000001, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 55.0, + "overflow": 45.0, + "padding": 6.0, + "tokens": 73217.0 + }, + { + "bucket": 2560.0, + "duration_ms": 10024.588917, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 55.0, + "overflow": 45.0, + "padding": 6.0, + "tokens": 76246.0 + }, + { + "bucket": 3008.0, + "duration_ms": 9929.563581000004, + "eligible_miss": 0.0, + "miss": 42.0, + "model": 53.0, + "overflow": 42.0, + "padding": 17.0, + "tokens": 75777.0 + }, + { + "bucket": 2560.0, + "duration_ms": 10077.664327000002, + "eligible_miss": 0.0, + "miss": 44.0, + "model": 54.0, + "overflow": 44.0, + "padding": 6.0, + "tokens": 76980.0 + }, + { + "duration_ms": 9992.777184999997, + "eligible_miss": 0.0, + "miss": 46.0, + "model": 46.0, + "overflow": 46.0, + "tokens": 80209.0 + }, + { + "bucket": 2560.0, + "duration_ms": 10091.163340000003, + "eligible_miss": 0.0, + "miss": 43.0, + "model": 53.0, + "overflow": 43.0, + "padding": 6.0, + "tokens": 77665.0 + }, + { + "bucket": 3072.0, + "duration_ms": 9848.346787, + "eligible_miss": 0.0, + "miss": 43.0, + "model": 54.0, + "overflow": 43.0, + "padding": 15.0, + "tokens": 74253.0 + }, + { + "bucket": 2560.0, + "duration_ms": 10082.291702, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 55.0, + "overflow": 45.0, + "padding": 6.0, + "tokens": 76492.0 + }, + { + "bucket": 2048.0, + "duration_ms": 9842.737886000003, + "eligible_miss": 0.0, + "miss": 41.0, + "model": 48.0, + "overflow": 41.0, + "padding": 6.0, + "tokens": 77465.0 + }, + { + "bucket": 768.0, + "duration_ms": 10038.773943999997, + "eligible_miss": 0.0, + "miss": 47.0, + "model": 50.0, + "overflow": 47.0, + "padding": 6.0, + "tokens": 78339.0 + }, + { + "bucket": 2304.0, + "duration_ms": 10092.552672000002, + "eligible_miss": 0.0, + "miss": 45.0, + "model": 54.0, + "overflow": 45.0, + "padding": 6.0, + "tokens": 76729.0 + } + ], + "clean": { + "admitted": 9713, + "completed": 9713, + "completed_throughput_rps": 40.47083333333333, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3091454, + "offered_rps": 40.47083333333333, + "output_tokens": 621632, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 2.0416924619930796, + "latency_s": { + "e2e": { + "mean": 6.321913773528025, + "n": 9713, + "p50": 6.324586006987374, + "p95": 6.48759062900208, + "p99": 6.554015111649641 + }, + "tpot": { + "mean": 0.09486961022559012, + "n": 9713, + "p50": 0.09484719088909009, + "p95": 0.09758893054289122, + "p99": 0.09857375903008876 + }, + "ttft": { + "mean": 0.34512832931584686, + "n": 9713, + "p50": 0.33935471798758954, + "p95": 0.4816830246010795, + "p99": 0.4956150398438331 + } + }, + "layer1": { + "decode_tokens": 612439, + "kv_usage_max": 0.3233335109500719, + "kv_usage_mean": 0.3000972717007496, + "mode_counts": { + "FULL": 445, + "NONE": 2044, + "PIECEWISE": 21 + }, + "mode_shares": { + "FULL": 0.17729083665338646, + "NONE": 0.8143426294820717, + "PIECEWISE": 0.008366533864541833 + }, + "preemptions": 0, + "prefill_tokens": 3093406, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 5, + "queue_waiting_mean": 0.13266932270916335, + "records_all": 3944, + "records_clean": 2510, + "requests_per_step": { + "mean": 248.12749003984064, + "n": 2510, + "p50": 247.0, + "p95": 256.0, + "p99": 256.0 + }, + "scheduled_tokens_per_step": { + "mean": 1476.4322709163346, + "n": 2510, + "p50": 1699.5, + "p95": 2048.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 191.09350754581675, + "n": 2510, + "p50": 213.4761155, + "p95": 245.4417175, + "p99": 246.39050363 + }, + "token_efficiency_per_ms": 7.726229372614054 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 40.05, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 41.2, + "completion_rate_within_10pct": true, + "tail_end_s": 353.45747823201236, + "tail_start_s": 343.45747823201236, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 40.05, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 42.8, + "completion_rate_within_10pct": true, + "tail_end_s": 391.7979503819952, + "tail_start_s": 381.7979503819952, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 9713, + "run_dir": "runs/phase3/primary/P01-C01/saturation", + "run_id": "P01-C01-saturation", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.002922077922077922, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.8143426294820717, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.8143426294820717, + "padded_tokens_per_useful_token": 9.714383629104834e-05, + "padding_fraction": 0.002922077922077922 + } + }, + "P01-C10-moderate": { + "blocks": [ + { + "bucket": 3288.0, + "duration_ms": 9949.162208000007, + "eligible_miss": 93.0, + "miss": 98.0, + "model": 193.0, + "overflow": 5.0, + "padding": 233.0, + "tokens": 38451.0 + }, + { + "bucket": 3552.0, + "duration_ms": 10012.469207000002, + "eligible_miss": 86.0, + "miss": 98.0, + "model": 195.0, + "overflow": 12.0, + "padding": 382.0, + "tokens": 35916.0 + }, + { + "bucket": 3192.0, + "duration_ms": 10035.694512, + "eligible_miss": 92.0, + "miss": 99.0, + "model": 193.0, + "overflow": 7.0, + "padding": 213.0, + "tokens": 37924.0 + }, + { + "bucket": 3168.0, + "duration_ms": 9936.948090999995, + "eligible_miss": 91.0, + "miss": 97.0, + "model": 178.0, + "overflow": 6.0, + "padding": 292.0, + "tokens": 37083.0 + }, + { + "bucket": 3184.0, + "duration_ms": 10005.574367, + "eligible_miss": 92.0, + "miss": 99.0, + "model": 193.0, + "overflow": 7.0, + "padding": 209.0, + "tokens": 38225.0 + }, + { + "bucket": 3392.0, + "duration_ms": 9952.588998999996, + "eligible_miss": 94.0, + "miss": 98.0, + "model": 190.0, + "overflow": 4.0, + "padding": 363.0, + "tokens": 37680.0 + }, + { + "bucket": 3424.0, + "duration_ms": 10046.252394999994, + "eligible_miss": 86.0, + "miss": 99.0, + "model": 197.0, + "overflow": 13.0, + "padding": 277.0, + "tokens": 37741.0 + }, + { + "bucket": 3232.0, + "duration_ms": 9949.493189000004, + "eligible_miss": 89.0, + "miss": 98.0, + "model": 187.0, + "overflow": 9.0, + "padding": 325.0, + "tokens": 38337.0 + }, + { + "bucket": 3392.0, + "duration_ms": 9946.791941, + "eligible_miss": 85.0, + "miss": 98.0, + "model": 186.0, + "overflow": 13.0, + "padding": 442.0, + "tokens": 37613.0 + }, + { + "bucket": 3712.0, + "duration_ms": 10000.662096999997, + "eligible_miss": 91.0, + "miss": 98.0, + "model": 202.0, + "overflow": 7.0, + "padding": 423.0, + "tokens": 35782.0 + }, + { + "bucket": 3176.0, + "duration_ms": 10035.215516000002, + "eligible_miss": 91.0, + "miss": 99.0, + "model": 196.0, + "overflow": 8.0, + "padding": 136.0, + "tokens": 38369.0 + }, + { + "bucket": 3512.0, + "duration_ms": 9896.367080000002, + "eligible_miss": 94.0, + "miss": 98.0, + "model": 201.0, + "overflow": 4.0, + "padding": 285.0, + "tokens": 36488.0 + }, + { + "bucket": 3488.0, + "duration_ms": 10034.694095, + "eligible_miss": 93.0, + "miss": 99.0, + "model": 208.0, + "overflow": 6.0, + "padding": 209.0, + "tokens": 36296.0 + }, + { + "bucket": 3472.0, + "duration_ms": 9994.456549999997, + "eligible_miss": 88.0, + "miss": 98.0, + "model": 204.0, + "overflow": 10.0, + "padding": 285.0, + "tokens": 37423.0 + }, + { + "bucket": 3528.0, + "duration_ms": 9952.904014, + "eligible_miss": 89.0, + "miss": 98.0, + "model": 201.0, + "overflow": 9.0, + "padding": 267.0, + "tokens": 36217.0 + }, + { + "bucket": 3320.0, + "duration_ms": 10038.123934000003, + "eligible_miss": 86.0, + "miss": 99.0, + "model": 188.0, + "overflow": 13.0, + "padding": 397.0, + "tokens": 40142.0 + }, + { + "bucket": 3512.0, + "duration_ms": 9891.855007000004, + "eligible_miss": 94.0, + "miss": 98.0, + "model": 206.0, + "overflow": 4.0, + "padding": 189.0, + "tokens": 36244.0 + }, + { + "bucket": 3456.0, + "duration_ms": 10094.883914000004, + "eligible_miss": 93.0, + "miss": 99.0, + "model": 207.0, + "overflow": 6.0, + "padding": 165.0, + "tokens": 37001.0 + }, + { + "bucket": 3152.0, + "duration_ms": 9887.934911, + "eligible_miss": 89.0, + "miss": 98.0, + "model": 194.0, + "overflow": 9.0, + "padding": 139.0, + "tokens": 38069.0 + }, + { + "bucket": 3200.0, + "duration_ms": 10016.706595000005, + "eligible_miss": 82.0, + "miss": 98.0, + "model": 181.0, + "overflow": 16.0, + "padding": 396.0, + "tokens": 39585.0 + }, + { + "bucket": 3424.0, + "duration_ms": 9978.465713999998, + "eligible_miss": 90.0, + "miss": 99.0, + "model": 199.0, + "overflow": 9.0, + "padding": 209.0, + "tokens": 38571.0 + }, + { + "bucket": 3328.0, + "duration_ms": 10008.412338, + "eligible_miss": 87.0, + "miss": 98.0, + "model": 194.0, + "overflow": 11.0, + "padding": 248.0, + "tokens": 38128.0 + }, + { + "bucket": 3472.0, + "duration_ms": 9986.329713000001, + "eligible_miss": 86.0, + "miss": 98.0, + "model": 192.0, + "overflow": 12.0, + "padding": 380.0, + "tokens": 37501.0 + }, + { + "bucket": 3312.0, + "duration_ms": 9999.963895000003, + "eligible_miss": 92.0, + "miss": 98.0, + "model": 199.0, + "overflow": 6.0, + "padding": 121.0, + "tokens": 37709.0 + }, + { + "bucket": 3296.0, + "duration_ms": 9915.257924999994, + "eligible_miss": 92.0, + "miss": 98.0, + "model": 197.0, + "overflow": 6.0, + "padding": 190.0, + "tokens": 37317.0 + }, + { + "bucket": 3328.0, + "duration_ms": 10019.317182000004, + "eligible_miss": 92.0, + "miss": 99.0, + "model": 199.0, + "overflow": 7.0, + "padding": 239.0, + "tokens": 37778.0 + }, + { + "bucket": 3408.0, + "duration_ms": 9954.582479999997, + "eligible_miss": 78.0, + "miss": 98.0, + "model": 190.0, + "overflow": 20.0, + "padding": 417.0, + "tokens": 38704.0 + }, + { + "bucket": 3576.0, + "duration_ms": 9995.198032999995, + "eligible_miss": 90.0, + "miss": 98.0, + "model": 195.0, + "overflow": 8.0, + "padding": 391.0, + "tokens": 37028.0 + }, + { + "bucket": 3360.0, + "duration_ms": 10041.384194, + "eligible_miss": 89.0, + "miss": 99.0, + "model": 204.0, + "overflow": 10.0, + "padding": 129.0, + "tokens": 37953.0 + }, + { + "bucket": 3456.0, + "duration_ms": 9891.348972, + "eligible_miss": 94.0, + "miss": 98.0, + "model": 206.0, + "overflow": 4.0, + "padding": 201.0, + "tokens": 37175.0 + }, + { + "bucket": 3440.0, + "duration_ms": 10036.670303999997, + "eligible_miss": 95.0, + "miss": 99.0, + "model": 202.0, + "overflow": 4.0, + "padding": 246.0, + "tokens": 38169.0 + }, + { + "bucket": 3000.0, + "duration_ms": 10017.732388999997, + "eligible_miss": 82.0, + "miss": 98.0, + "model": 186.0, + "overflow": 16.0, + "padding": 152.0, + "tokens": 39972.0 + }, + { + "bucket": 3560.0, + "duration_ms": 9940.346816, + "eligible_miss": 89.0, + "miss": 98.0, + "model": 201.0, + "overflow": 9.0, + "padding": 290.0, + "tokens": 37126.0 + }, + { + "bucket": 3392.0, + "duration_ms": 9977.342724999999, + "eligible_miss": 91.0, + "miss": 99.0, + "model": 205.0, + "overflow": 8.0, + "padding": 121.0, + "tokens": 37406.0 + }, + { + "bucket": 3400.0, + "duration_ms": 9963.435447, + "eligible_miss": 90.0, + "miss": 98.0, + "model": 204.0, + "overflow": 8.0, + "padding": 218.0, + "tokens": 36939.0 + }, + { + "bucket": 3408.0, + "duration_ms": 10002.120796, + "eligible_miss": 89.0, + "miss": 98.0, + "model": 201.0, + "overflow": 9.0, + "padding": 190.0, + "tokens": 38084.0 + }, + { + "bucket": 3352.0, + "duration_ms": 10038.113201000004, + "eligible_miss": 95.0, + "miss": 99.0, + "model": 196.0, + "overflow": 4.0, + "padding": 265.0, + "tokens": 36938.0 + }, + { + "bucket": 3360.0, + "duration_ms": 9896.136273, + "eligible_miss": 86.0, + "miss": 98.0, + "model": 183.0, + "overflow": 12.0, + "padding": 522.0, + "tokens": 39351.0 + }, + { + "bucket": 3640.0, + "duration_ms": 10043.925811, + "eligible_miss": 93.0, + "miss": 99.0, + "model": 203.0, + "overflow": 6.0, + "padding": 316.0, + "tokens": 37318.0 + }, + { + "bucket": 3368.0, + "duration_ms": 9930.598732000006, + "eligible_miss": 92.0, + "miss": 98.0, + "model": 203.0, + "overflow": 6.0, + "padding": 160.0, + "tokens": 36672.0 + }, + { + "bucket": 3488.0, + "duration_ms": 9960.504545000005, + "eligible_miss": 90.0, + "miss": 98.0, + "model": 207.0, + "overflow": 8.0, + "padding": 208.0, + "tokens": 36503.0 + }, + { + "bucket": 3456.0, + "duration_ms": 10007.049996999998, + "eligible_miss": 93.0, + "miss": 98.0, + "model": 205.0, + "overflow": 5.0, + "padding": 237.0, + "tokens": 37001.0 + }, + { + "bucket": 3368.0, + "duration_ms": 10026.919293, + "eligible_miss": 95.0, + "miss": 99.0, + "model": 202.0, + "overflow": 4.0, + "padding": 173.0, + "tokens": 37171.0 + }, + { + "bucket": 3384.0, + "duration_ms": 9957.427777999997, + "eligible_miss": 85.0, + "miss": 98.0, + "model": 194.0, + "overflow": 13.0, + "padding": 296.0, + "tokens": 37784.0 + }, + { + "bucket": 3400.0, + "duration_ms": 9986.793976999998, + "eligible_miss": 89.0, + "miss": 99.0, + "model": 193.0, + "overflow": 10.0, + "padding": 357.0, + "tokens": 38402.0 + }, + { + "bucket": 3448.0, + "duration_ms": 9952.649881000003, + "eligible_miss": 91.0, + "miss": 98.0, + "model": 193.0, + "overflow": 7.0, + "padding": 348.0, + "tokens": 38642.0 + }, + { + "bucket": 3496.0, + "duration_ms": 10045.983043, + "eligible_miss": 94.0, + "miss": 99.0, + "model": 207.0, + "overflow": 5.0, + "padding": 244.0, + "tokens": 37187.0 + }, + { + "bucket": 3376.0, + "duration_ms": 10027.000933999987, + "eligible_miss": 91.0, + "miss": 98.0, + "model": 203.0, + "overflow": 7.0, + "padding": 171.0, + "tokens": 36786.0 + } + ], + "clean": { + "admitted": 4722, + "completed": 4724, + "completed_throughput_rps": 19.683333333333334, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1508829, + "offered_rps": 19.675, + "output_tokens": 302336, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 1.5010973710159305, + "latency_s": { + "e2e": { + "mean": 1.6713845360269364, + "n": 4724, + "p50": 1.6473525780020282, + "p95": 1.8364300984874717, + "p99": 1.9401714695148982 + }, + "tpot": { + "mean": 0.025415942924051336, + "n": 4724, + "p50": 0.025119858555540088, + "p95": 0.02806124114468398, + "p99": 0.029400235347168884 + }, + "ttft": { + "mean": 0.07018013181170209, + "n": 4724, + "p50": 0.06350497498351615, + "p95": 0.10228192825015867, + "p99": 0.11306477584934285 + } + }, + "layer1": { + "decode_tokens": 297570, + "kv_usage_max": 0.050254770706364305, + "kv_usage_mean": 0.03635361272181466, + "mode_counts": { + "FULL": 4743, + "NONE": 4720 + }, + "mode_shares": { + "FULL": 0.5012152594314699, + "NONE": 0.49878474056853006 + }, + "preemptions": 0, + "prefill_tokens": 1508331, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 13476, + "records_clean": 9463, + "requests_per_step": { + "mean": 31.94462643981824, + "n": 9463, + "p50": 32.0, + "p95": 35.0, + "p99": 37.0 + }, + "scheduled_tokens_per_step": { + "mean": 190.83810630878156, + "n": 9463, + "p50": 41.0, + "p95": 507.0, + "p99": 536.0 + }, + "step_duration_ms": { + "mean": 50.647764029377576, + "n": 9463, + "p50": 51.929055, + "p95": 80.0764609, + "p99": 87.45197215999991 + }, + "token_efficiency_per_ms": 3.7679473115158335 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 19.65, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 26.2, + "completion_rate_within_10pct": false, + "tail_end_s": 347.5101968499948, + "tail_start_s": 337.5101968499948, + "valid": false, + "waiting_median": 86.0, + "waiting_within_10pct": false, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 19.65, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 26.4, + "completion_rate_within_10pct": false, + "tail_end_s": 388.53855585400015, + "tail_start_s": 378.53855585400015, + "valid": false, + "waiting_median": 51.0, + "waiting_within_10pct": false, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 4724, + "run_dir": "runs/phase3/primary/P01-C10/moderate", + "run_id": "P01-C10-moderate", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.024418453716312197, + "eligible_miss_rate": 0.45630349783366797, + "graph_miss_rate": 0.49878474056853006, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 4720, + "moe_layer_cv": null, + "overflow_rate": 0.0424812427348621, + "padded_tokens_per_useful_token": 0.007013673507019488, + "padding_fraction": 0.07787369042349122 + } + }, + "P01-C10-saturation": { + "blocks": [ + { + "bucket": 9600.0, + "duration_ms": 9585.937710000002, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 159.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 56941.0 + }, + { + "bucket": 9920.0, + "duration_ms": 10485.887850999992, + "eligible_miss": 2.0, + "miss": 12.0, + "model": 167.0, + "overflow": 10.0, + "padding": 3.0, + "tokens": 69928.0 + }, + { + "bucket": 11200.0, + "duration_ms": 9473.565303, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 183.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 50396.0 + }, + { + "bucket": 8384.0, + "duration_ms": 10083.465294000001, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 143.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 69395.0 + }, + { + "bucket": 11584.0, + "duration_ms": 9877.641023000017, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 189.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 53928.0 + }, + { + "bucket": 8448.0, + "duration_ms": 9991.739116999996, + "eligible_miss": 2.0, + "miss": 12.0, + "model": 144.0, + "overflow": 10.0, + "padding": 3.0, + "tokens": 69003.0 + }, + { + "bucket": 11136.0, + "duration_ms": 10591.129956999997, + "eligible_miss": 2.0, + "miss": 11.0, + "model": 185.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 67429.0 + }, + { + "bucket": 8576.0, + "duration_ms": 9364.851221999996, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 143.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 58746.0 + }, + { + "bucket": 10944.0, + "duration_ms": 10440.677515000005, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 182.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 65678.0 + }, + { + "bucket": 9408.0, + "duration_ms": 9549.062175999998, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 156.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 57336.0 + }, + { + "bucket": 10112.0, + "duration_ms": 10753.035256000001, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 170.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 71899.0 + }, + { + "bucket": 10112.0, + "duration_ms": 9199.662280000004, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 166.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 51847.0 + }, + { + "bucket": 9408.0, + "duration_ms": 10346.985273999993, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 159.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 70688.0 + }, + { + "bucket": 11200.0, + "duration_ms": 9654.620038999998, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 183.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 52925.0 + }, + { + "bucket": 8384.0, + "duration_ms": 10300.524266999999, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 143.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 72078.0 + }, + { + "bucket": 11648.0, + "duration_ms": 9750.345049999998, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 191.0, + "overflow": 7.0, + "padding": 3.0, + "tokens": 52476.0 + }, + { + "bucket": 8320.0, + "duration_ms": 9883.811008, + "eligible_miss": 2.0, + "miss": 11.0, + "model": 141.0, + "overflow": 9.0, + "padding": 2.0, + "tokens": 69969.0 + }, + { + "bucket": 11200.0, + "duration_ms": 10627.722131999999, + "eligible_miss": 2.0, + "miss": 11.0, + "model": 186.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 67543.0 + }, + { + "bucket": 9216.0, + "duration_ms": 9352.733858000001, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 153.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 55624.0 + }, + { + "bucket": 10304.0, + "duration_ms": 10591.480396999996, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 173.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 69690.0 + }, + { + "bucket": 10624.0, + "duration_ms": 9384.526043000003, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 174.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 51441.0 + }, + { + "bucket": 8960.0, + "duration_ms": 10530.41977099999, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 152.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 70762.0 + }, + { + "bucket": 10816.0, + "duration_ms": 9433.685700000002, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 177.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 53860.0 + }, + { + "bucket": 8704.0, + "duration_ms": 10167.772811999997, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 148.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 68677.0 + }, + { + "bucket": 11648.0, + "duration_ms": 9886.508919000004, + "eligible_miss": 3.0, + "miss": 9.0, + "model": 191.0, + "overflow": 6.0, + "padding": 3.0, + "tokens": 53558.0 + }, + { + "bucket": 8320.0, + "duration_ms": 9885.275682000003, + "eligible_miss": 2.0, + "miss": 11.0, + "model": 141.0, + "overflow": 9.0, + "padding": 2.0, + "tokens": 69626.0 + }, + { + "bucket": 11200.0, + "duration_ms": 10338.790289000006, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 186.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 64514.0 + }, + { + "bucket": 10240.0, + "duration_ms": 9647.886849999997, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 169.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 55945.0 + }, + { + "bucket": 9280.0, + "duration_ms": 10466.487114000001, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 157.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 73136.0 + }, + { + "bucket": 10944.0, + "duration_ms": 9522.099796999999, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 179.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 50955.0 + }, + { + "bucket": 8832.0, + "duration_ms": 9977.033838999998, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 150.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 67514.0 + }, + { + "bucket": 11456.0, + "duration_ms": 10609.994196000001, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 190.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 64182.0 + }, + { + "bucket": 8960.0, + "duration_ms": 9355.670728000005, + "eligible_miss": 1.0, + "miss": 9.0, + "model": 149.0, + "overflow": 8.0, + "padding": 2.0, + "tokens": 58884.0 + }, + { + "bucket": 10560.0, + "duration_ms": 10406.566008999995, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 176.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 67259.0 + }, + { + "bucket": 9536.0, + "duration_ms": 9561.564771, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 158.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 57471.0 + }, + { + "bucket": 9984.0, + "duration_ms": 10565.880848999994, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 168.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 70543.0 + }, + { + "bucket": 11008.0, + "duration_ms": 9419.635502000005, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 180.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 50023.0 + }, + { + "bucket": 8576.0, + "duration_ms": 9963.122449000006, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 146.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 68542.0 + }, + { + "bucket": 11648.0, + "duration_ms": 10849.752411000003, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 193.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 68094.0 + }, + { + "bucket": 8576.0, + "duration_ms": 9120.307115999996, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 143.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 56138.0 + }, + { + "bucket": 10944.0, + "duration_ms": 10493.4072, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 182.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 66834.0 + }, + { + "bucket": 9536.0, + "duration_ms": 9470.317570000001, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 158.0, + "overflow": 7.0, + "padding": 2.0, + "tokens": 56673.0 + }, + { + "bucket": 9984.0, + "duration_ms": 10584.995873, + "eligible_miss": 2.0, + "miss": 12.0, + "model": 168.0, + "overflow": 10.0, + "padding": 3.0, + "tokens": 70791.0 + }, + { + "bucket": 10688.0, + "duration_ms": 9395.01719, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 175.0, + "overflow": 6.0, + "padding": 2.0, + "tokens": 51393.0 + }, + { + "bucket": 8896.0, + "duration_ms": 10316.016737, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 151.0, + "overflow": 9.0, + "padding": 3.0, + "tokens": 69610.0 + }, + { + "bucket": 11648.0, + "duration_ms": 9691.449956999992, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 190.0, + "overflow": 6.0, + "padding": 3.0, + "tokens": 52716.0 + }, + { + "bucket": 8192.0, + "duration_ms": 9948.248781, + "eligible_miss": 3.0, + "miss": 12.0, + "model": 140.0, + "overflow": 9.0, + "padding": 2.0, + "tokens": 70776.0 + }, + { + "bucket": 11328.0, + "duration_ms": 10753.227755000007, + "eligible_miss": 3.0, + "miss": 11.0, + "model": 188.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 66675.0 + } + ], + "clean": { + "admitted": 7870, + "completed": 7870, + "completed_throughput_rps": 32.791666666666664, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2504713, + "offered_rps": 32.791666666666664, + "output_tokens": 503680, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 6.23828616499668, + "latency_s": { + "e2e": { + "mean": 7.811715558836478, + "n": 7870, + "p50": 7.815832740001497, + "p95": 7.9524892588422516, + "p99": 7.996230083884439 + }, + "tpot": { + "mean": 0.02288580142111304, + "n": 7870, + "p50": 0.02168167303944568, + "p95": 0.02640617163329836, + "p99": 0.02914771890473766 + }, + "ttft": { + "mean": 6.369910069306356, + "n": 7870, + "p50": 6.40973527150345, + "p95": 6.6001163164910395, + "p99": 6.649974504578276 + } + }, + "layer1": { + "decode_tokens": 495936, + "kv_usage_max": 0.08457388350484563, + "kv_usage_mean": 0.07151143707998636, + "mode_counts": { + "FULL": 7503, + "NONE": 492 + }, + "mode_shares": { + "FULL": 0.9384615384615385, + "NONE": 0.06153846153846154 + }, + "preemptions": 0, + "prefill_tokens": 2504175, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 192, + "queue_waiting_mean": 191.01563477173232, + "records_all": 12286, + "records_clean": 7995, + "requests_per_step": { + "mean": 63.01538461538462, + "n": 7995, + "p50": 64.0, + "p95": 64.0, + "p99": 64.0 + }, + "scheduled_tokens_per_step": { + "mean": 375.2484052532833, + "n": 7995, + "p50": 64.0, + "p95": 421.59999999999854, + "p99": 7181.0599999999995 + }, + "step_duration_ms": { + "mean": 59.99381346328956, + "n": 7995, + "p50": 31.676522, + "p95": 314.1602991999996, + "p99": 626.5985884999999 + }, + "token_efficiency_per_ms": 6.254785011839584 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 32.0, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 32.0, + "completion_rate_within_10pct": true, + "tail_end_s": 333.5596019870136, + "tail_start_s": 323.5596019870136, + "valid": true, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 32.0, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 32.1, + "completion_rate_within_10pct": true, + "tail_end_s": 366.9362416010117, + "tail_start_s": 356.9362416010117, + "valid": true, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 7870, + "run_dir": "runs/phase3/primary/P01-C10/saturation", + "run_id": "P01-C10-saturation", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.0015805949205076409, + "eligible_miss_rate": 0.014509068167604753, + "graph_miss_rate": 0.06153846153846154, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.047029393370856785, + "padded_tokens_per_useful_token": 4.099848305612692e-05, + "padding_fraction": 0.00025614754098360657 + } + }, + "P01-C11-moderate": { + "blocks": [ + { + "bucket": 3712.0, + "duration_ms": 9959.247200000002, + "eligible_miss": 81.0, + "miss": 91.0, + "model": 207.0, + "overflow": 10.0, + "padding": 541.0, + "tokens": 33619.0 + }, + { + "bucket": 3744.0, + "duration_ms": 9977.008518000002, + "eligible_miss": 87.0, + "miss": 90.0, + "model": 207.0, + "overflow": 3.0, + "padding": 449.0, + "tokens": 34868.0 + }, + { + "bucket": 3744.0, + "duration_ms": 9948.611453000003, + "eligible_miss": 80.0, + "miss": 91.0, + "model": 208.0, + "overflow": 11.0, + "padding": 459.0, + "tokens": 34053.0 + }, + { + "bucket": 3712.0, + "duration_ms": 10087.681306, + "eligible_miss": 85.0, + "miss": 91.0, + "model": 207.0, + "overflow": 6.0, + "padding": 552.0, + "tokens": 34447.0 + }, + { + "bucket": 3744.0, + "duration_ms": 9921.699469000001, + "eligible_miss": 84.0, + "miss": 90.0, + "model": 207.0, + "overflow": 6.0, + "padding": 478.0, + "tokens": 34600.0 + }, + { + "bucket": 3616.0, + "duration_ms": 9951.080938000001, + "eligible_miss": 88.0, + "miss": 91.0, + "model": 204.0, + "overflow": 3.0, + "padding": 447.0, + "tokens": 33871.0 + }, + { + "bucket": 3520.0, + "duration_ms": 10009.982611000001, + "eligible_miss": 85.0, + "miss": 91.0, + "model": 201.0, + "overflow": 6.0, + "padding": 315.0, + "tokens": 35572.0 + }, + { + "bucket": 3584.0, + "duration_ms": 9932.805558999993, + "eligible_miss": 81.0, + "miss": 90.0, + "model": 202.0, + "overflow": 9.0, + "padding": 448.0, + "tokens": 34434.0 + }, + { + "bucket": 3744.0, + "duration_ms": 10017.269197000005, + "eligible_miss": 81.0, + "miss": 91.0, + "model": 208.0, + "overflow": 10.0, + "padding": 475.0, + "tokens": 34466.0 + }, + { + "bucket": 3616.0, + "duration_ms": 9983.475279000002, + "eligible_miss": 84.0, + "miss": 90.0, + "model": 203.0, + "overflow": 6.0, + "padding": 417.0, + "tokens": 34861.0 + }, + { + "bucket": 3680.0, + "duration_ms": 10017.097006999997, + "eligible_miss": 81.0, + "miss": 91.0, + "model": 206.0, + "overflow": 10.0, + "padding": 458.0, + "tokens": 35522.0 + }, + { + "bucket": 3808.0, + "duration_ms": 9938.864236999998, + "eligible_miss": 88.0, + "miss": 91.0, + "model": 210.0, + "overflow": 3.0, + "padding": 587.0, + "tokens": 33086.0 + }, + { + "bucket": 3552.0, + "duration_ms": 10024.110196000009, + "eligible_miss": 83.0, + "miss": 91.0, + "model": 202.0, + "overflow": 8.0, + "padding": 378.0, + "tokens": 35313.0 + }, + { + "bucket": 3936.0, + "duration_ms": 9968.116, + "eligible_miss": 88.0, + "miss": 90.0, + "model": 213.0, + "overflow": 2.0, + "padding": 579.0, + "tokens": 33164.0 + }, + { + "bucket": 3808.0, + "duration_ms": 9938.115310000003, + "eligible_miss": 85.0, + "miss": 91.0, + "model": 210.0, + "overflow": 6.0, + "padding": 583.0, + "tokens": 33657.0 + }, + { + "bucket": 4000.0, + "duration_ms": 10033.491619999997, + "eligible_miss": 86.0, + "miss": 91.0, + "model": 216.0, + "overflow": 5.0, + "padding": 656.0, + "tokens": 33781.0 + }, + { + "bucket": 3552.0, + "duration_ms": 9911.066888000005, + "eligible_miss": 81.0, + "miss": 90.0, + "model": 201.0, + "overflow": 9.0, + "padding": 398.0, + "tokens": 34386.0 + }, + { + "bucket": 3744.0, + "duration_ms": 10074.457772999996, + "eligible_miss": 84.0, + "miss": 91.0, + "model": 208.0, + "overflow": 7.0, + "padding": 451.0, + "tokens": 36056.0 + }, + { + "bucket": 3712.0, + "duration_ms": 9942.207149000005, + "eligible_miss": 83.0, + "miss": 90.0, + "model": 206.0, + "overflow": 7.0, + "padding": 484.0, + "tokens": 34082.0 + }, + { + "bucket": 3712.0, + "duration_ms": 9950.111710000001, + "eligible_miss": 86.0, + "miss": 91.0, + "model": 207.0, + "overflow": 5.0, + "padding": 505.0, + "tokens": 34159.0 + }, + { + "bucket": 3904.0, + "duration_ms": 10004.259407000001, + "eligible_miss": 89.0, + "miss": 91.0, + "model": 213.0, + "overflow": 2.0, + "padding": 597.0, + "tokens": 33971.0 + }, + { + "bucket": 3744.0, + "duration_ms": 9984.552112999998, + "eligible_miss": 78.0, + "miss": 90.0, + "model": 207.0, + "overflow": 12.0, + "padding": 471.0, + "tokens": 35554.0 + }, + { + "bucket": 3424.0, + "duration_ms": 9948.117576999995, + "eligible_miss": 80.0, + "miss": 91.0, + "model": 198.0, + "overflow": 11.0, + "padding": 343.0, + "tokens": 36906.0 + }, + { + "bucket": 3584.0, + "duration_ms": 10005.684332000004, + "eligible_miss": 81.0, + "miss": 91.0, + "model": 203.0, + "overflow": 10.0, + "padding": 417.0, + "tokens": 36166.0 + }, + { + "bucket": 3680.0, + "duration_ms": 9982.162338, + "eligible_miss": 82.0, + "miss": 90.0, + "model": 205.0, + "overflow": 8.0, + "padding": 478.0, + "tokens": 34720.0 + }, + { + "bucket": 3648.0, + "duration_ms": 9970.997752000007, + "eligible_miss": 83.0, + "miss": 91.0, + "model": 205.0, + "overflow": 8.0, + "padding": 418.0, + "tokens": 34115.0 + }, + { + "bucket": 3648.0, + "duration_ms": 10001.097707999996, + "eligible_miss": 87.0, + "miss": 91.0, + "model": 205.0, + "overflow": 4.0, + "padding": 431.0, + "tokens": 34821.0 + }, + { + "bucket": 3648.0, + "duration_ms": 9947.801385999994, + "eligible_miss": 85.0, + "miss": 90.0, + "model": 204.0, + "overflow": 5.0, + "padding": 445.0, + "tokens": 34615.0 + }, + { + "bucket": 3712.0, + "duration_ms": 10000.821746, + "eligible_miss": 87.0, + "miss": 91.0, + "model": 207.0, + "overflow": 4.0, + "padding": 494.0, + "tokens": 34429.0 + }, + { + "bucket": 3712.0, + "duration_ms": 10025.943433, + "eligible_miss": 77.0, + "miss": 91.0, + "model": 207.0, + "overflow": 14.0, + "padding": 451.0, + "tokens": 35074.0 + }, + { + "bucket": 3648.0, + "duration_ms": 9968.242391000002, + "eligible_miss": 80.0, + "miss": 90.0, + "model": 204.0, + "overflow": 10.0, + "padding": 431.0, + "tokens": 35314.0 + }, + { + "bucket": 3808.0, + "duration_ms": 9955.46471, + "eligible_miss": 83.0, + "miss": 91.0, + "model": 210.0, + "overflow": 8.0, + "padding": 539.0, + "tokens": 34463.0 + }, + { + "bucket": 3776.0, + "duration_ms": 10017.773912999995, + "eligible_miss": 85.0, + "miss": 91.0, + "model": 209.0, + "overflow": 6.0, + "padding": 506.0, + "tokens": 35333.0 + }, + { + "bucket": 3712.0, + "duration_ms": 9920.078849000001, + "eligible_miss": 88.0, + "miss": 90.0, + "model": 206.0, + "overflow": 2.0, + "padding": 543.0, + "tokens": 34287.0 + }, + { + "bucket": 3616.0, + "duration_ms": 10072.948714000007, + "eligible_miss": 83.0, + "miss": 91.0, + "model": 204.0, + "overflow": 8.0, + "padding": 394.0, + "tokens": 35149.0 + }, + { + "bucket": 3392.0, + "duration_ms": 9921.726875999999, + "eligible_miss": 76.0, + "miss": 90.0, + "model": 196.0, + "overflow": 14.0, + "padding": 293.0, + "tokens": 37023.0 + }, + { + "bucket": 3808.0, + "duration_ms": 9962.493520999991, + "eligible_miss": 86.0, + "miss": 91.0, + "model": 210.0, + "overflow": 5.0, + "padding": 564.0, + "tokens": 34152.0 + }, + { + "bucket": 3840.0, + "duration_ms": 10012.223253000006, + "eligible_miss": 86.0, + "miss": 91.0, + "model": 211.0, + "overflow": 5.0, + "padding": 513.0, + "tokens": 33611.0 + }, + { + "bucket": 3808.0, + "duration_ms": 9982.952109000005, + "eligible_miss": 84.0, + "miss": 90.0, + "model": 209.0, + "overflow": 6.0, + "padding": 570.0, + "tokens": 34152.0 + }, + { + "bucket": 3744.0, + "duration_ms": 9946.108181000001, + "eligible_miss": 83.0, + "miss": 91.0, + "model": 208.0, + "overflow": 8.0, + "padding": 502.0, + "tokens": 35237.0 + }, + { + "bucket": 3744.0, + "duration_ms": 10010.716567999994, + "eligible_miss": 87.0, + "miss": 91.0, + "model": 208.0, + "overflow": 4.0, + "padding": 498.0, + "tokens": 34119.0 + }, + { + "bucket": 3648.0, + "duration_ms": 10004.357205999999, + "eligible_miss": 82.0, + "miss": 90.0, + "model": 204.0, + "overflow": 8.0, + "padding": 455.0, + "tokens": 35705.0 + }, + { + "bucket": 3456.0, + "duration_ms": 9945.860283, + "eligible_miss": 86.0, + "miss": 91.0, + "model": 199.0, + "overflow": 5.0, + "padding": 355.0, + "tokens": 35520.0 + }, + { + "bucket": 3872.0, + "duration_ms": 10024.396264999998, + "eligible_miss": 88.0, + "miss": 91.0, + "model": 212.0, + "overflow": 3.0, + "padding": 531.0, + "tokens": 33351.0 + }, + { + "bucket": 3872.0, + "duration_ms": 9965.571466000003, + "eligible_miss": 82.0, + "miss": 90.0, + "model": 211.0, + "overflow": 8.0, + "padding": 565.0, + "tokens": 35270.0 + }, + { + "bucket": 3872.0, + "duration_ms": 9953.784156999998, + "eligible_miss": 89.0, + "miss": 91.0, + "model": 212.0, + "overflow": 2.0, + "padding": 611.0, + "tokens": 32037.0 + }, + { + "bucket": 3488.0, + "duration_ms": 10030.564947999997, + "eligible_miss": 86.0, + "miss": 91.0, + "model": 200.0, + "overflow": 5.0, + "padding": 451.0, + "tokens": 34202.0 + }, + { + "bucket": 3680.0, + "duration_ms": 9936.440215999995, + "eligible_miss": 84.0, + "miss": 90.0, + "model": 205.0, + "overflow": 6.0, + "padding": 344.0, + "tokens": 34514.0 + } + ], + "clean": { + "admitted": 4351, + "completed": 4351, + "completed_throughput_rps": 18.129166666666666, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1387917, + "offered_rps": 18.129166666666666, + "output_tokens": 278464, + "start_s": 60.0 + }, + "config": "C11", + "drain_quarantined": false, + "drain_seconds": 0.78110175699112, + "latency_s": { + "e2e": { + "mean": 1.5956257307001827, + "n": 4351, + "p50": 1.5987792539817747, + "p95": 1.7036471079918556, + "p99": 1.7430243315029657 + }, + "tpot": { + "mean": 0.024275665845246406, + "n": 4351, + "p50": 0.024345195381015185, + "p95": 0.0258859295316229, + "p99": 0.02641761801570528 + }, + "ttft": { + "mean": 0.06625878244965909, + "n": 4351, + "p50": 0.06313588601187803, + "p95": 0.10306648399273399, + "p99": 0.10799207250238396 + } + }, + "layer1": { + "decode_tokens": 274144, + "kv_usage_max": 0.04047490555855371, + "kv_usage_mean": 0.03138338971244831, + "mode_counts": { + "FULL": 5554, + "NONE": 4351 + }, + "mode_shares": { + "FULL": 0.5607269056032307, + "NONE": 0.4392730943967693 + }, + "preemptions": 0, + "prefill_tokens": 1387663, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 14187, + "records_clean": 9905, + "requests_per_step": { + "mean": 28.11660777385159, + "n": 9905, + "p50": 28.0, + "p95": 30.0, + "p99": 31.0 + }, + "scheduled_tokens_per_step": { + "mean": 167.77455830388692, + "n": 9905, + "p50": 29.0, + "p95": 498.0, + "p99": 531.0 + }, + "step_duration_ms": { + "mean": 48.36846449651691, + "n": 9905, + "p50": 53.217559, + "p95": 64.57586499999995, + "p99": 87.24098867999997 + }, + "token_efficiency_per_ms": 3.4686765447344032 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 18.05, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 26.6, + "completion_rate_within_10pct": false, + "tail_end_s": 347.03789717299514, + "tail_start_s": 337.03789717299514, + "valid": false, + "waiting_median": 13.0, + "waiting_within_10pct": false, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 18.05, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 27.4, + "completion_rate_within_10pct": false, + "tail_end_s": 394.36406223100494, + "tail_start_s": 384.36406223100494, + "valid": false, + "waiting_median": 25.0, + "waiting_within_10pct": false, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 4351, + "run_dir": "runs/phase3/primary/P01-C11/moderate", + "run_id": "P01-C11-moderate", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.031357776531205854, + "eligible_miss_rate": 0.406663301362948, + "graph_miss_rate": 0.4392730943967693, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 4351, + "moe_layer_cv": null, + "overflow_rate": 0.032609793033821305, + "padded_tokens_per_useful_token": 0.013762127611690166, + "padding_fraction": 0.12867978033849478 + } + }, + "P01-C11-saturation": { + "blocks": [ + { + "bucket": 6400.0, + "duration_ms": 9854.135549999999, + "eligible_miss": 2.0, + "miss": 37.0, + "model": 137.0, + "overflow": 35.0, + "padding": 2.0, + "tokens": 59578.0 + }, + { + "bucket": 7552.0, + "duration_ms": 9896.006353999996, + "eligible_miss": 2.0, + "miss": 34.0, + "model": 152.0, + "overflow": 32.0, + "padding": 2.0, + "tokens": 56340.0 + }, + { + "bucket": 8448.0, + "duration_ms": 10198.840811000002, + "eligible_miss": 1.0, + "miss": 35.0, + "model": 167.0, + "overflow": 34.0, + "padding": 3.0, + "tokens": 55828.0 + }, + { + "bucket": 6400.0, + "duration_ms": 9857.422299999998, + "eligible_miss": 2.0, + "miss": 39.0, + "model": 139.0, + "overflow": 37.0, + "padding": 2.0, + "tokens": 60921.0 + }, + { + "bucket": 8896.0, + "duration_ms": 9893.559951000003, + "eligible_miss": 2.0, + "miss": 31.0, + "model": 170.0, + "overflow": 29.0, + "padding": 2.0, + "tokens": 52396.0 + }, + { + "bucket": 7104.0, + "duration_ms": 10051.639851999997, + "eligible_miss": 3.0, + "miss": 38.0, + "model": 149.0, + "overflow": 35.0, + "padding": 3.0, + "tokens": 59830.0 + }, + { + "bucket": 6784.0, + "duration_ms": 9926.596741999998, + "eligible_miss": 1.0, + "miss": 37.0, + "model": 143.0, + "overflow": 36.0, + "padding": 2.0, + "tokens": 59476.0 + }, + { + "bucket": 9216.0, + "duration_ms": 10039.259791, + "eligible_miss": 3.0, + "miss": 32.0, + "model": 176.0, + "overflow": 29.0, + "padding": 3.0, + "tokens": 53246.0 + }, + { + "bucket": 6400.0, + "duration_ms": 10069.991445000001, + "eligible_miss": 1.0, + "miss": 38.0, + "model": 138.0, + "overflow": 37.0, + "padding": 2.0, + "tokens": 63144.0 + }, + { + "bucket": 7104.0, + "duration_ms": 9825.774603999995, + "eligible_miss": 2.0, + "miss": 35.0, + "model": 146.0, + "overflow": 33.0, + "padding": 2.0, + "tokens": 58761.0 + }, + { + "bucket": 8896.0, + "duration_ms": 10039.997257, + "eligible_miss": 3.0, + "miss": 32.0, + "model": 171.0, + "overflow": 29.0, + "padding": 3.0, + "tokens": 53282.0 + }, + { + "bucket": 6336.0, + "duration_ms": 10105.074265000001, + "eligible_miss": 3.0, + "miss": 40.0, + "model": 139.0, + "overflow": 37.0, + "padding": 2.0, + "tokens": 62032.0 + }, + { + "bucket": 7168.0, + "duration_ms": 9824.336723000004, + "eligible_miss": 4.0, + "miss": 37.0, + "model": 149.0, + "overflow": 33.0, + "padding": 2.0, + "tokens": 56409.0 + }, + { + "bucket": 8512.0, + "duration_ms": 10174.319685999995, + "eligible_miss": 5.0, + "miss": 36.0, + "model": 169.0, + "overflow": 31.0, + "padding": 3.0, + "tokens": 56319.0 + }, + { + "bucket": 6272.0, + "duration_ms": 9881.112864000002, + "eligible_miss": 4.0, + "miss": 40.0, + "model": 138.0, + "overflow": 36.0, + "padding": 2.0, + "tokens": 60881.0 + }, + { + "bucket": 7872.0, + "duration_ms": 9879.054422000001, + "eligible_miss": 3.0, + "miss": 36.0, + "model": 159.0, + "overflow": 33.0, + "padding": 2.0, + "tokens": 55569.0 + }, + { + "bucket": 7808.0, + "duration_ms": 10085.14763200001, + "eligible_miss": 4.0, + "miss": 36.0, + "model": 158.0, + "overflow": 32.0, + "padding": 3.0, + "tokens": 57772.0 + }, + { + "bucket": 6272.0, + "duration_ms": 10021.492237000002, + "eligible_miss": 2.0, + "miss": 42.0, + "model": 140.0, + "overflow": 40.0, + "padding": 2.0, + "tokens": 62027.0 + }, + { + "bucket": 8320.0, + "duration_ms": 9862.347805999994, + "eligible_miss": 5.0, + "miss": 34.0, + "model": 164.0, + "overflow": 29.0, + "padding": 2.0, + "tokens": 53646.0 + }, + { + "bucket": 7360.0, + "duration_ms": 10213.140882999996, + "eligible_miss": 4.0, + "miss": 40.0, + "model": 155.0, + "overflow": 36.0, + "padding": 3.0, + "tokens": 59510.0 + }, + { + "bucket": 6272.0, + "duration_ms": 9961.871976000002, + "eligible_miss": 1.0, + "miss": 39.0, + "model": 137.0, + "overflow": 38.0, + "padding": 2.0, + "tokens": 61272.0 + }, + { + "bucket": 8768.0, + "duration_ms": 9788.361232000007, + "eligible_miss": 2.0, + "miss": 33.0, + "model": 170.0, + "overflow": 31.0, + "padding": 2.0, + "tokens": 50522.0 + }, + { + "bucket": 6912.0, + "duration_ms": 10077.373054000002, + "eligible_miss": 3.0, + "miss": 41.0, + "model": 149.0, + "overflow": 38.0, + "padding": 3.0, + "tokens": 59342.0 + }, + { + "bucket": 6656.0, + "duration_ms": 9893.422728000001, + "eligible_miss": 2.0, + "miss": 39.0, + "model": 143.0, + "overflow": 37.0, + "padding": 2.0, + "tokens": 58372.0 + }, + { + "bucket": 8896.0, + "duration_ms": 9963.152674999996, + "eligible_miss": 2.0, + "miss": 32.0, + "model": 171.0, + "overflow": 30.0, + "padding": 2.0, + "tokens": 51851.0 + }, + { + "bucket": 6400.0, + "duration_ms": 10128.076877, + "eligible_miss": 2.0, + "miss": 41.0, + "model": 141.0, + "overflow": 39.0, + "padding": 3.0, + "tokens": 63078.0 + }, + { + "bucket": 6720.0, + "duration_ms": 9840.454017, + "eligible_miss": 3.0, + "miss": 39.0, + "model": 144.0, + "overflow": 36.0, + "padding": 2.0, + "tokens": 56898.0 + }, + { + "bucket": 8960.0, + "duration_ms": 10069.613022000003, + "eligible_miss": 3.0, + "miss": 33.0, + "model": 173.0, + "overflow": 30.0, + "padding": 3.0, + "tokens": 52424.0 + }, + { + "bucket": 6272.0, + "duration_ms": 10086.144182999995, + "eligible_miss": 2.0, + "miss": 41.0, + "model": 139.0, + "overflow": 39.0, + "padding": 2.0, + "tokens": 63091.0 + }, + { + "bucket": 7616.0, + "duration_ms": 9819.59047, + "eligible_miss": 3.0, + "miss": 38.0, + "model": 157.0, + "overflow": 35.0, + "padding": 2.0, + "tokens": 54925.0 + }, + { + "bucket": 8064.0, + "duration_ms": 10084.262563999993, + "eligible_miss": 4.0, + "miss": 38.0, + "model": 164.0, + "overflow": 34.0, + "padding": 3.0, + "tokens": 56586.0 + }, + { + "bucket": 6272.0, + "duration_ms": 9973.057058000002, + "eligible_miss": 3.0, + "miss": 40.0, + "model": 138.0, + "overflow": 37.0, + "padding": 2.0, + "tokens": 62642.0 + }, + { + "bucket": 8576.0, + "duration_ms": 9906.256273999996, + "eligible_miss": 3.0, + "miss": 34.0, + "model": 168.0, + "overflow": 31.0, + "padding": 2.0, + "tokens": 52036.0 + }, + { + "bucket": 7104.0, + "duration_ms": 10073.656216999998, + "eligible_miss": 3.0, + "miss": 40.0, + "model": 151.0, + "overflow": 37.0, + "padding": 3.0, + "tokens": 59041.0 + }, + { + "bucket": 6784.0, + "duration_ms": 9879.548074999997, + "eligible_miss": 2.0, + "miss": 40.0, + "model": 146.0, + "overflow": 38.0, + "padding": 2.0, + "tokens": 57531.0 + }, + { + "bucket": 8896.0, + "duration_ms": 10182.290707999993, + "eligible_miss": 4.0, + "miss": 35.0, + "model": 174.0, + "overflow": 31.0, + "padding": 3.0, + "tokens": 54155.0 + }, + { + "bucket": 6272.0, + "duration_ms": 9961.884863999998, + "eligible_miss": 2.0, + "miss": 40.0, + "model": 138.0, + "overflow": 38.0, + "padding": 2.0, + "tokens": 62473.0 + }, + { + "bucket": 6912.0, + "duration_ms": 9816.541234000002, + "eligible_miss": 2.0, + "miss": 37.0, + "model": 145.0, + "overflow": 35.0, + "padding": 2.0, + "tokens": 57702.0 + }, + { + "bucket": 8768.0, + "duration_ms": 10041.103434999999, + "eligible_miss": 3.0, + "miss": 34.0, + "model": 171.0, + "overflow": 31.0, + "padding": 3.0, + "tokens": 53200.0 + }, + { + "bucket": 6272.0, + "duration_ms": 10059.006235000004, + "eligible_miss": 2.0, + "miss": 41.0, + "model": 139.0, + "overflow": 39.0, + "padding": 2.0, + "tokens": 61771.0 + }, + { + "bucket": 7872.0, + "duration_ms": 9881.051967000003, + "eligible_miss": 2.0, + "miss": 37.0, + "model": 160.0, + "overflow": 35.0, + "padding": 2.0, + "tokens": 54044.0 + }, + { + "bucket": 7808.0, + "duration_ms": 10117.143586, + "eligible_miss": 3.0, + "miss": 39.0, + "model": 161.0, + "overflow": 36.0, + "padding": 3.0, + "tokens": 58050.0 + }, + { + "bucket": 6272.0, + "duration_ms": 9970.732670000003, + "eligible_miss": 2.0, + "miss": 41.0, + "model": 139.0, + "overflow": 39.0, + "padding": 2.0, + "tokens": 60989.0 + }, + { + "bucket": 8640.0, + "duration_ms": 9859.881477999996, + "eligible_miss": 2.0, + "miss": 32.0, + "model": 167.0, + "overflow": 30.0, + "padding": 2.0, + "tokens": 52243.0 + }, + { + "bucket": 7040.0, + "duration_ms": 10080.299878999997, + "eligible_miss": 3.0, + "miss": 40.0, + "model": 150.0, + "overflow": 37.0, + "padding": 3.0, + "tokens": 59585.0 + }, + { + "bucket": 6336.0, + "duration_ms": 10006.960603000003, + "eligible_miss": 2.0, + "miss": 40.0, + "model": 139.0, + "overflow": 38.0, + "padding": 2.0, + "tokens": 60367.0 + }, + { + "bucket": 9280.0, + "duration_ms": 9888.519409999994, + "eligible_miss": 2.0, + "miss": 32.0, + "model": 177.0, + "overflow": 30.0, + "padding": 2.0, + "tokens": 51283.0 + }, + { + "bucket": 6336.0, + "duration_ms": 10039.603299999999, + "eligible_miss": 2.0, + "miss": 41.0, + "model": 140.0, + "overflow": 39.0, + "padding": 3.0, + "tokens": 60761.0 + } + ], + "clean": { + "admitted": 7252, + "completed": 7252, + "completed_throughput_rps": 30.216666666666665, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2308997, + "offered_rps": 30.216666666666665, + "output_tokens": 464128, + "start_s": 60.0 + }, + "config": "C11", + "drain_quarantined": false, + "drain_seconds": 7.466667815984692, + "latency_s": { + "e2e": { + "mean": 8.479005430775388, + "n": 7252, + "p50": 8.488157999003306, + "p95": 8.62182870524848, + "p99": 8.659005985551339 + }, + "tpot": { + "mean": 0.030981617798706566, + "n": 7252, + "p50": 0.03097177065062588, + "p95": 0.032257703036515206, + "p99": 0.032861725646782566 + }, + "ttft": { + "mean": 6.527163509456874, + "n": 7252, + "p50": 6.528871715505375, + "p95": 6.666822149860673, + "p99": 6.698687153424544 + } + }, + "layer1": { + "decode_tokens": 455861, + "kv_usage_max": 0.08202914193200217, + "kv_usage_mean": 0.07020805987115446, + "mode_counts": { + "FULL": 5564, + "NONE": 1786 + }, + "mode_shares": { + "FULL": 0.7570068027210884, + "NONE": 0.24299319727891155 + }, + "preemptions": 0, + "prefill_tokens": 2307340, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 192, + "queue_waiting_mean": 191.0134693877551, + "records_all": 11380, + "records_clean": 7350, + "requests_per_step": { + "mean": 63.01319727891156, + "n": 7350, + "p50": 64.0, + "p95": 64.0, + "p99": 64.0 + }, + "scheduled_tokens_per_step": { + "mean": 375.9457142857143, + "n": 7350, + "p50": 64.0, + "p95": 1647.5499999999993, + "p99": 1965.5100000000002 + }, + "step_duration_ms": { + "mean": 65.19035523346939, + "n": 7350, + "p50": 33.147161, + "p95": 185.55763439999998, + "p99": 202.8066175900001 + }, + "token_efficiency_per_ms": 5.766891635109544 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P01", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 29.9, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 31.6, + "completion_rate_within_10pct": true, + "tail_end_s": 342.5308606699982, + "tail_start_s": 332.5308606699982, + "valid": true, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 29.9, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 28.9, + "completion_rate_within_10pct": true, + "tail_end_s": 375.9222603540111, + "tail_start_s": 365.9222603540111, + "valid": true, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 7252, + "run_dir": "runs/phase3/primary/P01-C11/saturation", + "run_id": "P01-C11-saturation", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.0024497562729742686, + "eligible_miss_rate": 0.017006802721088437, + "graph_miss_rate": 0.24299319727891155, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.22598639455782313, + "padded_tokens_per_useful_token": 4.089460013947592e-05, + "padding_fraction": 0.0003173301581595974 + } + }, + "P02-C00-moderate": { + "blocks": [ + { + "bucket": 24032.0, + "duration_ms": 9990.374930999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 191.0, + "overflow": 4.0, + "padding": 430.0, + "tokens": 25800.0 + }, + { + "bucket": 21776.0, + "duration_ms": 9970.567963999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 186.0, + "overflow": 11.0, + "padding": 334.0, + "tokens": 27607.0 + }, + { + "bucket": 25304.0, + "duration_ms": 9978.748167999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 193.0, + "overflow": 4.0, + "padding": 430.0, + "tokens": 27110.0 + }, + { + "bucket": 24360.0, + "duration_ms": 9948.932990000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 191.0, + "overflow": 5.0, + "padding": 501.0, + "tokens": 26593.0 + }, + { + "bucket": 23512.0, + "duration_ms": 10021.978742000003, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 192.0, + "overflow": 6.0, + "padding": 547.0, + "tokens": 26380.0 + }, + { + "bucket": 22864.0, + "duration_ms": 9986.585967999992, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 187.0, + "overflow": 9.0, + "padding": 455.0, + "tokens": 27383.0 + }, + { + "bucket": 23008.0, + "duration_ms": 9962.632415999997, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 190.0, + "overflow": 8.0, + "padding": 481.0, + "tokens": 26882.0 + }, + { + "bucket": 20856.0, + "duration_ms": 9993.166222000002, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 191.0, + "overflow": 12.0, + "padding": 386.0, + "tokens": 27222.0 + }, + { + "bucket": 23232.0, + "duration_ms": 9980.531696, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 190.0, + "overflow": 5.0, + "padding": 484.0, + "tokens": 25604.0 + }, + { + "bucket": 23568.0, + "duration_ms": 9962.036962000002, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 187.0, + "overflow": 5.0, + "padding": 441.0, + "tokens": 25962.0 + }, + { + "bucket": 23928.0, + "duration_ms": 10001.557425999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 189.0, + "overflow": 4.0, + "padding": 382.0, + "tokens": 25751.0 + }, + { + "bucket": 22448.0, + "duration_ms": 9945.918971, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 187.0, + "overflow": 9.0, + "padding": 259.0, + "tokens": 27301.0 + }, + { + "bucket": 21776.0, + "duration_ms": 10029.021144000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 192.0, + "overflow": 10.0, + "padding": 400.0, + "tokens": 26851.0 + }, + { + "bucket": 22024.0, + "duration_ms": 9953.179873999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 190.0, + "overflow": 9.0, + "padding": 389.0, + "tokens": 26662.0 + }, + { + "bucket": 23864.0, + "duration_ms": 9947.728225000004, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 188.0, + "overflow": 6.0, + "padding": 457.0, + "tokens": 26707.0 + }, + { + "bucket": 22440.0, + "duration_ms": 10006.012027999996, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 189.0, + "overflow": 10.0, + "padding": 464.0, + "tokens": 27499.0 + }, + { + "bucket": 23464.0, + "duration_ms": 9976.957411999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 189.0, + "overflow": 7.0, + "padding": 373.0, + "tokens": 27006.0 + }, + { + "bucket": 21184.0, + "duration_ms": 9958.724134999997, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 189.0, + "overflow": 12.0, + "padding": 304.0, + "tokens": 27389.0 + }, + { + "bucket": 23560.0, + "duration_ms": 9988.031766000007, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 191.0, + "overflow": 5.0, + "padding": 460.0, + "tokens": 25824.0 + }, + { + "bucket": 23512.0, + "duration_ms": 10011.898985999998, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 187.0, + "overflow": 8.0, + "padding": 434.0, + "tokens": 27497.0 + }, + { + "bucket": 23656.0, + "duration_ms": 9966.358194999993, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 190.0, + "overflow": 6.0, + "padding": 455.0, + "tokens": 26595.0 + }, + { + "bucket": 22424.0, + "duration_ms": 10001.583119000004, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 192.0, + "overflow": 9.0, + "padding": 395.0, + "tokens": 27027.0 + }, + { + "bucket": 22496.0, + "duration_ms": 9992.83060800001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 194.0, + "overflow": 7.0, + "padding": 558.0, + "tokens": 25824.0 + }, + { + "bucket": 23264.0, + "duration_ms": 9970.569579999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 195.0, + "overflow": 6.0, + "padding": 710.0, + "tokens": 25915.0 + }, + { + "bucket": 21720.0, + "duration_ms": 9988.029574999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 189.0, + "overflow": 10.0, + "padding": 639.0, + "tokens": 26748.0 + }, + { + "bucket": 23936.0, + "duration_ms": 9961.957320999996, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 190.0, + "overflow": 6.0, + "padding": 614.0, + "tokens": 26614.0 + }, + { + "bucket": 23280.0, + "duration_ms": 9992.756356000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 191.0, + "overflow": 7.0, + "padding": 489.0, + "tokens": 26596.0 + }, + { + "bucket": 23384.0, + "duration_ms": 9956.727936000005, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 190.0, + "overflow": 8.0, + "padding": 487.0, + "tokens": 27340.0 + }, + { + "bucket": 21592.0, + "duration_ms": 10004.012218000007, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 191.0, + "overflow": 11.0, + "padding": 507.0, + "tokens": 27266.0 + }, + { + "bucket": 22712.0, + "duration_ms": 9969.233649999993, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 189.0, + "overflow": 7.0, + "padding": 507.0, + "tokens": 26147.0 + }, + { + "bucket": 23144.0, + "duration_ms": 9995.538473, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 193.0, + "overflow": 4.0, + "padding": 503.0, + "tokens": 24863.0 + }, + { + "bucket": 23232.0, + "duration_ms": 9948.029670000002, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 189.0, + "overflow": 6.0, + "padding": 538.0, + "tokens": 26057.0 + }, + { + "bucket": 23512.0, + "duration_ms": 9983.782809999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 191.0, + "overflow": 6.0, + "padding": 528.0, + "tokens": 26292.0 + }, + { + "bucket": 22096.0, + "duration_ms": 9981.637089999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 189.0, + "overflow": 10.0, + "padding": 490.0, + "tokens": 27165.0 + }, + { + "bucket": 23016.0, + "duration_ms": 9990.229410999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 192.0, + "overflow": 7.0, + "padding": 458.0, + "tokens": 26417.0 + }, + { + "bucket": 23520.0, + "duration_ms": 9964.634507999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 190.0, + "overflow": 7.0, + "padding": 496.0, + "tokens": 27013.0 + }, + { + "bucket": 22984.0, + "duration_ms": 9985.033693000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 191.0, + "overflow": 7.0, + "padding": 561.0, + "tokens": 26179.0 + }, + { + "bucket": 23072.0, + "duration_ms": 10000.245900999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 189.0, + "overflow": 6.0, + "padding": 485.0, + "tokens": 25902.0 + }, + { + "bucket": 23200.0, + "duration_ms": 9981.013887000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 188.0, + "overflow": 7.0, + "padding": 430.0, + "tokens": 26600.0 + }, + { + "bucket": 21640.0, + "duration_ms": 9972.11258, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 190.0, + "overflow": 10.0, + "padding": 369.0, + "tokens": 26891.0 + }, + { + "bucket": 24960.0, + "duration_ms": 10021.411613000006, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 186.0, + "overflow": 6.0, + "padding": 432.0, + "tokens": 27757.0 + }, + { + "bucket": 24344.0, + "duration_ms": 9947.654753000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 191.0, + "overflow": 4.0, + "padding": 377.0, + "tokens": 26135.0 + }, + { + "bucket": 23384.0, + "duration_ms": 9983.115876, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 191.0, + "overflow": 6.0, + "padding": 327.0, + "tokens": 26392.0 + }, + { + "bucket": 23560.0, + "duration_ms": 9962.658046, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 191.0, + "overflow": 6.0, + "padding": 489.0, + "tokens": 26471.0 + }, + { + "bucket": 22432.0, + "duration_ms": 9994.31181, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 188.0, + "overflow": 9.0, + "padding": 458.0, + "tokens": 27072.0 + }, + { + "bucket": 22200.0, + "duration_ms": 9995.201477000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 191.0, + "overflow": 7.0, + "padding": 446.0, + "tokens": 25703.0 + }, + { + "bucket": 21776.0, + "duration_ms": 9962.024166000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 190.0, + "overflow": 10.0, + "padding": 390.0, + "tokens": 26955.0 + }, + { + "bucket": 23576.0, + "duration_ms": 9965.013077000003, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 192.0, + "overflow": 6.0, + "padding": 523.0, + "tokens": 26357.0 + } + ], + "clean": { + "admitted": 1536, + "completed": 1537, + "completed_throughput_rps": 6.404166666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 493355, + "offered_rps": 6.4, + "output_tokens": 786944, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 8.726939176005544, + "latency_s": { + "e2e": { + "mean": 13.527924079167455, + "n": 1537, + "p50": 13.537170164985582, + "p95": 13.641508429596433, + "p99": 13.721163101642158 + }, + "tpot": { + "mean": 0.026301412165737215, + "n": 1537, + "p50": 0.026317087653602897, + "p95": 0.026532306089241387, + "p99": 0.02667916153010606 + }, + "ttft": { + "mean": 0.08790246247573787, + "n": 1537, + "p50": 0.08802004699828103, + "p95": 0.10394775940803809, + "p99": 0.1072390887804795 + } + }, + "layer1": { + "decode_tokens": 785211, + "kv_usage_max": 0.1826439578264396, + "kv_usage_mean": 0.17062570582816017, + "mode_counts": { + "FULL": 7586, + "NONE": 350, + "PIECEWISE": 1186 + }, + "mode_shares": { + "FULL": 0.8316158737119053, + "NONE": 0.03836877877658408, + "PIECEWISE": 0.13001534751151064 + }, + "preemptions": 0, + "prefill_tokens": 492112, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 14298, + "records_clean": 9122, + "requests_per_step": { + "mean": 86.24720456040342, + "n": 9122, + "p50": 86.0, + "p95": 87.0, + "p99": 88.0 + }, + "scheduled_tokens_per_step": { + "mean": 140.02663889497916, + "n": 9122, + "p50": 86.0, + "p95": 487.9499999999989, + "p99": 575.0 + }, + "step_duration_ms": { + "mean": 52.51614705382592, + "n": 9122, + "p50": 45.163168, + "p95": 74.58838244999998, + "p99": 78.05190758999998 + }, + "token_efficiency_per_ms": 2.6663540025405945 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P02", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 6.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 6.8, + "completion_rate_within_10pct": true, + "tail_end_s": 333.57828441497986, + "tail_start_s": 323.57828441497986, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 6.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 7.0, + "completion_rate_within_10pct": true, + "tail_end_s": 367.5501277539879, + "tail_start_s": 357.5501277539879, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 1537, + "run_dir": "runs/phase3/primary/P02-C00/moderate", + "run_id": "P02-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.019977842624707646, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.03836877877658408, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 1536, + "moe_layer_cv": null, + "overflow_rate": 0.03836877877658408, + "padded_tokens_per_useful_token": 0.017279889268415272, + "padding_fraction": 0.019977842624707646 + } + }, + "P02-C00-saturation": { + "blocks": [ + { + "bucket": 31744.0, + "duration_ms": 10002.675380000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 31744.0 + }, + { + "bucket": 30464.0, + "duration_ms": 9977.405133999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 119.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30464.0 + }, + { + "bucket": 10616.0, + "duration_ms": 10968.631501000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 51.0, + "overflow": 10.0, + "padding": 13.0, + "tokens": 91371.0 + }, + { + "bucket": 28928.0, + "duration_ms": 9062.709024, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 114.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 34646.0 + }, + { + "bucket": 32768.0, + "duration_ms": 9948.836104999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 128.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 32768.0 + }, + { + "bucket": 31232.0, + "duration_ms": 9969.489221000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 122.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 31232.0 + }, + { + "bucket": 30208.0, + "duration_ms": 9988.825041999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 118.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30208.0 + }, + { + "bucket": 5616.0, + "duration_ms": 10101.62816, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 34.0, + "overflow": 12.0, + "padding": 8.0, + "tokens": 89499.0 + }, + { + "bucket": 35072.0, + "duration_ms": 9878.621836, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 137.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 35072.0 + }, + { + "bucket": 33024.0, + "duration_ms": 10043.882824999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 129.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 33024.0 + }, + { + "bucket": 31488.0, + "duration_ms": 9993.030219000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 123.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 31488.0 + }, + { + "bucket": 29072.0, + "duration_ms": 10687.022778, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 115.0, + "overflow": 2.0, + "padding": 16.0, + "tokens": 44531.0 + }, + { + "bucket": 9472.0, + "duration_ms": 9269.511275, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 46.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 76327.0 + }, + { + "bucket": 34304.0, + "duration_ms": 9932.089161000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 134.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 34304.0 + }, + { + "bucket": 32768.0, + "duration_ms": 10036.932253, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 128.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 32768.0 + }, + { + "bucket": 30976.0, + "duration_ms": 9951.556923999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 121.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30976.0 + }, + { + "bucket": 21408.0, + "duration_ms": 10437.419686000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 88.0, + "overflow": 5.0, + "padding": 21.0, + "tokens": 59530.0 + }, + { + "bucket": 18176.0, + "duration_ms": 9570.746277, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 77.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 63405.0 + }, + { + "bucket": 34048.0, + "duration_ms": 9934.429371999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 133.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 34048.0 + }, + { + "bucket": 32512.0, + "duration_ms": 10015.332801000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 127.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 32512.0 + }, + { + "bucket": 30976.0, + "duration_ms": 9991.365666000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 121.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30976.0 + }, + { + "bucket": 12784.0, + "duration_ms": 10683.046393999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 60.0, + "overflow": 10.0, + "padding": 8.0, + "tokens": 82291.0 + }, + { + "bucket": 27136.0, + "duration_ms": 9277.709315999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 108.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 42721.0 + }, + { + "bucket": 33792.0, + "duration_ms": 9982.370690000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 132.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 33792.0 + }, + { + "bucket": 32000.0, + "duration_ms": 10015.234685000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 125.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 32000.0 + }, + { + "bucket": 30464.0, + "duration_ms": 9969.169197999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 119.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30464.0 + }, + { + "bucket": 6592.0, + "duration_ms": 9960.992191000003, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 36.0, + "overflow": 11.0, + "padding": 17.0, + "tokens": 88758.0 + }, + { + "bucket": 35840.0, + "duration_ms": 10018.960007, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 140.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 35840.0 + }, + { + "bucket": 33280.0, + "duration_ms": 10004.701410999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 130.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 33280.0 + }, + { + "bucket": 31744.0, + "duration_ms": 9991.484610999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 31744.0 + }, + { + "bucket": 26976.0, + "duration_ms": 10646.21925, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 108.0, + "overflow": 3.0, + "padding": 23.0, + "tokens": 48965.0 + }, + { + "bucket": 11776.0, + "duration_ms": 9300.176934000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 55.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 71947.0 + }, + { + "bucket": 34560.0, + "duration_ms": 9969.634924, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 135.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 34560.0 + }, + { + "bucket": 32768.0, + "duration_ms": 10059.361980999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 128.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 32768.0 + }, + { + "bucket": 30976.0, + "duration_ms": 9932.953102000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 121.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30976.0 + }, + { + "bucket": 18544.0, + "duration_ms": 10893.108333, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 79.0, + "overflow": 7.0, + "padding": 21.0, + "tokens": 71776.0 + }, + { + "bucket": 20992.0, + "duration_ms": 9086.406721000005, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 87.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 51818.0 + }, + { + "bucket": 33536.0, + "duration_ms": 9975.697529000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 131.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 33536.0 + }, + { + "bucket": 31744.0, + "duration_ms": 9990.059341000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 31744.0 + }, + { + "bucket": 30464.0, + "duration_ms": 10009.16256, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 119.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30464.0 + }, + { + "bucket": 12080.0, + "duration_ms": 10643.432062, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 56.0, + "overflow": 9.0, + "padding": 17.0, + "tokens": 85791.0 + }, + { + "bucket": 28928.0, + "duration_ms": 9306.107722, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 115.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 37973.0 + }, + { + "bucket": 33792.0, + "duration_ms": 10023.528799999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 132.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 33792.0 + }, + { + "bucket": 31744.0, + "duration_ms": 10015.802693, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 31744.0 + }, + { + "bucket": 30464.0, + "duration_ms": 9925.296385999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 119.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 30464.0 + }, + { + "bucket": 5072.0, + "duration_ms": 9987.664447000003, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 32.0, + "overflow": 12.0, + "padding": 7.0, + "tokens": 92449.0 + }, + { + "bucket": 35584.0, + "duration_ms": 9973.684404000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 139.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 35584.0 + }, + { + "bucket": 33024.0, + "duration_ms": 10000.565399000006, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 129.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 33024.0 + } + ], + "clean": { + "admitted": 2560, + "completed": 2560, + "completed_throughput_rps": 10.666666666666666, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 815867, + "offered_rps": 10.666666666666666, + "output_tokens": 1310720, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 17.89816882798914, + "latency_s": { + "e2e": { + "mean": 23.776247528639022, + "n": 2560, + "p50": 23.71122523449594, + "p95": 24.185485272797813, + "p99": 24.45216006551811 + }, + "tpot": { + "mean": 0.04448661513129689, + "n": 2560, + "p50": 0.04432050637085125, + "p95": 0.04553604761966451, + "p99": 0.04615120161844849 + }, + "ttft": { + "mean": 1.0435871965463093, + "n": 2560, + "p50": 1.0870295370114036, + "p95": 1.2516038013927755, + "p99": 1.5639317611194565 + } + }, + "layer1": { + "decode_tokens": 1322213, + "kv_usage_max": 0.7301432819680995, + "kv_usage_mean": 0.5031304572048997, + "mode_counts": { + "FULL": 5074, + "NONE": 115, + "PIECEWISE": 7 + }, + "mode_shares": { + "FULL": 0.9765204003079292, + "NONE": 0.022132409545804466, + "PIECEWISE": 0.0013471901462663587 + }, + "preemptions": 0, + "prefill_tokens": 818945, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 18, + "queue_waiting_mean": 0.04157043879907621, + "records_all": 8236, + "records_clean": 5196, + "requests_per_step": { + "mean": 254.9688221709007, + "n": 5196, + "p50": 256.0, + "p95": 256.0, + "p99": 256.0 + }, + "scheduled_tokens_per_step": { + "mean": 412.07813702848347, + "n": 5196, + "p50": 256.0, + "p95": 256.0, + "p99": 7964.100000000004 + }, + "step_duration_ms": { + "mean": 92.26417854715166, + "n": 5196, + "p50": 78.5344495, + "p95": 86.525586, + "p99": 736.0913249500003 + }, + "token_efficiency_per_ms": 4.46628522051915 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P02", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 3.45, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.0, + "completion_rate_within_10pct": false, + "tail_end_s": 337.7681430270022, + "tail_start_s": 327.7681430270022, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 3.45, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 25.6, + "completion_rate_within_10pct": false, + "tail_end_s": 375.69639842701145, + "tail_start_s": 365.69639842701145, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 2560, + "run_dir": "runs/phase3/primary/P02-C00/saturation", + "run_id": "P02-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.3723323346172142, + "R32": 0.3612776759520059, + "R64": 0.36868249852408347, + "bucket_slack": 0.00011601748099157298, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.022132409545804466, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.022132409545804466, + "padded_tokens_per_useful_token": 7.05225863761572e-05, + "padding_fraction": 0.00011601748099157298 + } + }, + "P03-C00-moderate": { + "blocks": [ + { + "bucket": 434.0, + "duration_ms": 9646.774494999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 414.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42391.0 + }, + { + "bucket": 428.0, + "duration_ms": 9324.853701000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 349.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45396.0 + }, + { + "bucket": 451.0, + "duration_ms": 9443.773851999995, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 377.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44779.0 + }, + { + "bucket": 431.0, + "duration_ms": 8789.258909999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 402.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 39827.0 + }, + { + "bucket": 444.0, + "duration_ms": 8735.292183000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 425.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 38560.0 + }, + { + "bucket": 441.0, + "duration_ms": 8991.055925999994, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 417.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40995.0 + }, + { + "bucket": 423.0, + "duration_ms": 9480.767129999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 381.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45023.0 + }, + { + "bucket": 486.0, + "duration_ms": 9729.935500999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 427.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44224.0 + }, + { + "bucket": 416.0, + "duration_ms": 9213.938557999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 384.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45373.0 + }, + { + "bucket": 486.0, + "duration_ms": 8918.107862000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 442.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 39693.0 + }, + { + "bucket": 409.0, + "duration_ms": 9586.225367999996, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 392.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 51207.0 + }, + { + "bucket": 467.0, + "duration_ms": 9515.115281000006, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 368.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44145.0 + }, + { + "bucket": 414.0, + "duration_ms": 8670.001719000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 398.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41341.0 + }, + { + "bucket": 465.0, + "duration_ms": 8769.038874999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 434.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 39805.0 + }, + { + "bucket": 434.0, + "duration_ms": 9390.774539000004, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 375.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46322.0 + }, + { + "bucket": 441.0, + "duration_ms": 9182.554264, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 402.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43923.0 + }, + { + "bucket": 460.0, + "duration_ms": 8867.88750800001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 400.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41579.0 + }, + { + "bucket": 401.0, + "duration_ms": 9142.820255999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 407.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44598.0 + }, + { + "bucket": 485.0, + "duration_ms": 9352.599022999993, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 402.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43986.0 + }, + { + "bucket": 429.0, + "duration_ms": 8637.459765999994, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 425.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40104.0 + }, + { + "bucket": 448.0, + "duration_ms": 9475.931046999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 386.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45960.0 + }, + { + "bucket": 465.0, + "duration_ms": 9328.015840000015, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 430.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43347.0 + }, + { + "bucket": 439.0, + "duration_ms": 9245.173080000004, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 411.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 46942.0 + }, + { + "bucket": 441.0, + "duration_ms": 9031.702460999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 437.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41183.0 + }, + { + "bucket": 439.0, + "duration_ms": 8945.048207000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 426.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41666.0 + }, + { + "bucket": 439.0, + "duration_ms": 8656.631559, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 411.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40487.0 + }, + { + "bucket": 438.0, + "duration_ms": 8917.907948999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 400.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42512.0 + }, + { + "bucket": 440.0, + "duration_ms": 8937.543736000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 426.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42114.0 + }, + { + "bucket": 441.0, + "duration_ms": 8382.712513999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 440.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 37512.0 + }, + { + "bucket": 438.0, + "duration_ms": 9172.430862000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 415.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43506.0 + }, + { + "bucket": 460.0, + "duration_ms": 9511.392210000004, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 407.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45333.0 + }, + { + "bucket": 475.0, + "duration_ms": 9785.702609, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 471.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44172.0 + }, + { + "bucket": 439.0, + "duration_ms": 8901.444294000008, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 426.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40862.0 + }, + { + "bucket": 437.0, + "duration_ms": 9806.976566000003, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 389.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 52715.0 + }, + { + "bucket": 440.0, + "duration_ms": 8958.783222000005, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 409.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 39756.0 + }, + { + "bucket": 422.0, + "duration_ms": 9178.061366000005, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 410.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44271.0 + }, + { + "bucket": 425.0, + "duration_ms": 8984.160134000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 355.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44663.0 + }, + { + "bucket": 466.0, + "duration_ms": 9250.503494999995, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 400.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43886.0 + }, + { + "bucket": 444.0, + "duration_ms": 8412.454728000006, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 449.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 37387.0 + }, + { + "bucket": 444.0, + "duration_ms": 8843.111829, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 445.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40662.0 + }, + { + "bucket": 428.0, + "duration_ms": 9809.472161000005, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 375.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 48958.0 + }, + { + "bucket": 449.0, + "duration_ms": 9674.471301999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 381.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 47368.0 + }, + { + "bucket": 441.0, + "duration_ms": 9450.021538, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 380.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46428.0 + }, + { + "bucket": 471.0, + "duration_ms": 9220.409785000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 463.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41943.0 + }, + { + "bucket": 453.0, + "duration_ms": 8620.783462999989, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 450.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 38619.0 + }, + { + "bucket": 442.0, + "duration_ms": 9088.896370999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 450.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 43799.0 + }, + { + "bucket": 410.0, + "duration_ms": 9024.806824000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 369.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44660.0 + }, + { + "bucket": 445.0, + "duration_ms": 9384.437405000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 357.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46780.0 + } + ], + "clean": { + "admitted": 340, + "completed": 339, + "completed_throughput_rps": 1.4125, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2054190, + "offered_rps": 1.4166666666666667, + "output_tokens": 21696, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 0.4808295039983932, + "latency_s": { + "e2e": { + "mean": 0.8583498531842609, + "n": 339, + "p50": 0.6899766309943516, + "p95": 1.2946852156106616, + "p99": 1.3524367231986252 + }, + "tpot": { + "mean": 0.0073678705388819194, + "n": 339, + "p50": 0.005003864031528965, + "p95": 0.01286850249975003, + "p99": 0.01351572900346207 + }, + "ttft": { + "mean": 0.3941740092347001, + "n": 339, + "p50": 0.38708334998227656, + "p95": 0.5440592002210906, + "p99": 0.5607605650188634 + } + }, + "layer1": { + "decode_tokens": 21402, + "kv_usage_max": 0.053636117869694466, + "kv_usage_mean": 0.0227916201009575, + "mode_counts": { + "FULL": 19249, + "NONE": 726 + }, + "mode_shares": { + "FULL": 0.9636545682102629, + "NONE": 0.03634543178973717 + }, + "preemptions": 0, + "prefill_tokens": 2059360, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 29343, + "records_clean": 19975, + "requests_per_step": { + "mean": 1.0884605757196495, + "n": 19975, + "p50": 1.0, + "p95": 2.0, + "p99": 2.0 + }, + "scheduled_tokens_per_step": { + "mean": 104.16831038798499, + "n": 19975, + "p50": 1.0, + "p95": 2.0, + "p99": 5737.759999999958 + }, + "step_duration_ms": { + "mean": 21.996857135118898, + "n": 19975, + "p50": 9.367757, + "p95": 12.816464299999998, + "p99": 419.37568741999894 + }, + "token_efficiency_per_ms": 4.735599715364607 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P03", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.4, + "completion_rate_within_10pct": true, + "tail_end_s": 331.5279114540026, + "tail_start_s": 321.5279114540026, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.4, + "completion_rate_within_10pct": true, + "tail_end_s": 363.4785102730093, + "tail_start_s": 353.4785102730093, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 339, + "run_dir": "runs/phase3/primary/P03-C00/moderate", + "run_id": "P03-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.0, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.017356679769258258, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 138, + "moe_layer_cv": null, + "overflow_rate": 0.017356679769258258, + "padded_tokens_per_useful_token": 0.0, + "padding_fraction": 0.0 + } + }, + "P03-C00-saturation": { + "blocks": [ + { + "bucket": 864.0, + "duration_ms": 10578.148114000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 28.0, + "overflow": 10.0, + "padding": 2.0, + "tokens": 76404.0 + }, + { + "duration_ms": 10051.067833, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 75648.0 + }, + { + "duration_ms": 10430.060368999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 79159.0 + }, + { + "bucket": 384.0, + "duration_ms": 8947.849064999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 16.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 61135.0 + }, + { + "bucket": 864.0, + "duration_ms": 11122.426039999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 29.0, + "overflow": 11.0, + "padding": 1.0, + "tokens": 78761.0 + }, + { + "duration_ms": 10057.831934, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 76555.0 + }, + { + "duration_ms": 9721.105124000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73139.0 + }, + { + "bucket": 48.0, + "duration_ms": 9329.917999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 1.0, + "tokens": 65693.0 + }, + { + "bucket": 1200.0, + "duration_ms": 10223.449019, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 35.0, + "overflow": 10.0, + "padding": 28.0, + "tokens": 71110.0 + }, + { + "duration_ms": 10493.838531000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 79030.0 + }, + { + "duration_ms": 10439.288879, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81709.0 + }, + { + "bucket": 56.0, + "duration_ms": 9103.318499, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 6.0, + "tokens": 64928.0 + }, + { + "bucket": 1680.0, + "duration_ms": 10367.279989999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 35.0, + "overflow": 11.0, + "padding": 145.0, + "tokens": 72060.0 + }, + { + "duration_ms": 10189.229741, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 75842.0 + }, + { + "duration_ms": 9478.008878, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 73062.0 + }, + { + "bucket": 192.0, + "duration_ms": 9436.550829999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 13.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 65760.0 + }, + { + "bucket": 1328.0, + "duration_ms": 10334.733178, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 31.0, + "overflow": 10.0, + "padding": 89.0, + "tokens": 70413.0 + }, + { + "duration_ms": 10626.730527, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 81172.0 + }, + { + "duration_ms": 9518.246487, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 72556.0 + }, + { + "bucket": 48.0, + "duration_ms": 9749.105485000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 1.0, + "tokens": 68945.0 + }, + { + "bucket": 1152.0, + "duration_ms": 10786.964422000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 34.0, + "overflow": 10.0, + "padding": 29.0, + "tokens": 73567.0 + }, + { + "bucket": 48.0, + "duration_ms": 9785.572214, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 3.0, + "tokens": 76865.0 + }, + { + "duration_ms": 9623.650463, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 72252.0 + }, + { + "bucket": 48.0, + "duration_ms": 10084.750344, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 71946.0 + }, + { + "bucket": 1104.0, + "duration_ms": 10114.801081, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 33.0, + "overflow": 10.0, + "padding": 3.0, + "tokens": 69776.0 + }, + { + "bucket": 48.0, + "duration_ms": 10094.120242, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 2.0, + "tokens": 73552.0 + }, + { + "duration_ms": 10679.833572, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 80671.0 + }, + { + "duration_ms": 9146.444671, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 67917.0 + }, + { + "bucket": 1152.0, + "duration_ms": 10535.819985999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 34.0, + "overflow": 10.0, + "padding": 75.0, + "tokens": 69083.0 + }, + { + "bucket": 48.0, + "duration_ms": 9952.865603, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 4.0, + "tokens": 74679.0 + }, + { + "bucket": 232.0, + "duration_ms": 9606.608713999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 6.0, + "tokens": 71716.0 + }, + { + "duration_ms": 10236.557004000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 78064.0 + }, + { + "bucket": 1152.0, + "duration_ms": 9876.530566, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 34.0, + "overflow": 10.0, + "padding": 18.0, + "tokens": 63632.0 + }, + { + "bucket": 48.0, + "duration_ms": 9665.328617, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 2.0, + "tokens": 72669.0 + }, + { + "duration_ms": 10083.804314, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 76205.0 + }, + { + "duration_ms": 10325.771488, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 77498.0 + }, + { + "bucket": 1200.0, + "duration_ms": 9464.822698999998, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 33.0, + "overflow": 8.0, + "padding": 29.0, + "tokens": 57502.0 + }, + { + "duration_ms": 10398.848107000002, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 78851.0 + }, + { + "duration_ms": 9908.080162, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 73838.0 + }, + { + "duration_ms": 9995.163472999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 76673.0 + }, + { + "bucket": 1152.0, + "duration_ms": 10214.747484, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 33.0, + "overflow": 9.0, + "padding": 74.0, + "tokens": 61305.0 + }, + { + "bucket": 48.0, + "duration_ms": 10312.275375000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 12.0, + "overflow": 11.0, + "padding": 3.0, + "tokens": 79610.0 + }, + { + "duration_ms": 9717.962814999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 74553.0 + }, + { + "duration_ms": 10444.018532999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 82228.0 + }, + { + "bucket": 1320.0, + "duration_ms": 8902.236085, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 32.0, + "overflow": 8.0, + "padding": 148.0, + "tokens": 52523.0 + }, + { + "bucket": 48.0, + "duration_ms": 10528.722166, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 12.0, + "overflow": 11.0, + "padding": 1.0, + "tokens": 82473.0 + }, + { + "duration_ms": 9880.531748000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 73322.0 + }, + { + "duration_ms": 10429.997748999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 79679.0 + } + ], + "clean": { + "admitted": 567, + "completed": 567, + "completed_throughput_rps": 2.3625, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3470825, + "offered_rps": 2.3625, + "output_tokens": 36288, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 93.00942119900719, + "latency_s": { + "e2e": { + "mean": 104.49926800635963, + "n": 567, + "p50": 108.49479436199181, + "p95": 114.96174016330329, + "p99": 125.03032377155616 + }, + "tpot": { + "mean": 0.3016080685133144, + "n": 567, + "p50": 0.301236945777736, + "p95": 0.3107678588207197, + "p99": 0.3154822883366456 + }, + "ttft": { + "mean": 85.49795969002082, + "n": 567, + "p50": 89.48056172500947, + "p95": 96.23374651201881, + "p99": 106.40916011528114 + } + }, + "layer1": { + "decode_tokens": 35902, + "kv_usage_max": 0.9998918626655853, + "kv_usage_mean": 0.9845854764583278, + "mode_counts": { + "FULL": 293, + "NONE": 478, + "PIECEWISE": 4 + }, + "mode_shares": { + "FULL": 0.37806451612903225, + "NONE": 0.6167741935483871, + "PIECEWISE": 0.005161290322580645 + }, + "preemptions": 2, + "prefill_tokens": 3469828, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 211, + "queue_waiting_mean": 207.8516129032258, + "records_all": 1540, + "records_clean": 775, + "requests_per_step": { + "mean": 46.68258064516129, + "n": 775, + "p50": 47.0, + "p95": 49.0, + "p99": 50.0 + }, + "scheduled_tokens_per_step": { + "mean": 4523.522580645162, + "n": 775, + "p50": 6175.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 620.6387306051613, + "n": 775, + "p50": 802.754999, + "p95": 1104.0022976999999, + "p99": 1137.5921699 + }, + "token_efficiency_per_ms": 7.288495476642984 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P03", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 2.3, + "clean_c_waiting_median": 208.0, + "completion_rate_rps": 2.5, + "completion_rate_within_10pct": true, + "tail_end_s": 336.5461640879803, + "tail_start_s": 326.5461640879803, + "valid": true, + "waiting_median": 209.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 2.3, + "clean_c_waiting_median": 208.0, + "completion_rate_rps": 2.1, + "completion_rate_within_10pct": true, + "tail_end_s": 392.17679743198096, + "tail_start_s": 382.17679743198096, + "valid": true, + "waiting_median": 209.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 567, + "run_dir": "runs/phase3/primary/P03-C00/saturation", + "run_id": "P03-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.043326435592343504, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.6167741935483871, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.6167741935483871, + "padded_tokens_per_useful_token": 0.00019111568774549096, + "padding_fraction": 0.043326435592343504 + } + }, + "P03-C01-moderate": { + "blocks": [ + { + "bucket": 954.0, + "duration_ms": 9872.450652000009, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 332.0, + "overflow": 23.0, + "padding": 17.0, + "tokens": 45821.0 + }, + { + "bucket": 419.0, + "duration_ms": 8502.870301999994, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 436.0, + "overflow": 20.0, + "padding": 0.0, + "tokens": 33944.0 + }, + { + "bucket": 537.0, + "duration_ms": 8469.788425000002, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 398.0, + "overflow": 19.0, + "padding": 5.0, + "tokens": 36476.0 + }, + { + "bucket": 1330.0, + "duration_ms": 9275.734582000001, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 353.0, + "overflow": 23.0, + "padding": 7.0, + "tokens": 44107.0 + }, + { + "bucket": 982.0, + "duration_ms": 8895.486666, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 375.0, + "overflow": 20.0, + "padding": 8.0, + "tokens": 40031.0 + }, + { + "bucket": 394.0, + "duration_ms": 8277.386265, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 394.0, + "overflow": 20.0, + "padding": 0.0, + "tokens": 34398.0 + }, + { + "bucket": 572.0, + "duration_ms": 7978.012209000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 431.0, + "overflow": 17.0, + "padding": 8.0, + "tokens": 32212.0 + }, + { + "bucket": 832.0, + "duration_ms": 8694.071453000006, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 403.0, + "overflow": 21.0, + "padding": 12.0, + "tokens": 40122.0 + }, + { + "bucket": 1315.0, + "duration_ms": 8601.290529, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 385.0, + "overflow": 18.0, + "padding": 20.0, + "tokens": 36044.0 + }, + { + "bucket": 749.0, + "duration_ms": 9269.075646999998, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 410.0, + "overflow": 21.0, + "padding": 11.0, + "tokens": 39900.0 + }, + { + "bucket": 374.0, + "duration_ms": 9022.692624000003, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 388.0, + "overflow": 22.0, + "padding": 0.0, + "tokens": 41430.0 + }, + { + "bucket": 876.0, + "duration_ms": 8708.543662000002, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 363.0, + "overflow": 20.0, + "padding": 0.0, + "tokens": 39367.0 + }, + { + "bucket": 1143.0, + "duration_ms": 8555.031544999998, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 383.0, + "overflow": 19.0, + "padding": 0.0, + "tokens": 36768.0 + }, + { + "bucket": 1351.0, + "duration_ms": 8710.897649000002, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 388.0, + "overflow": 18.0, + "padding": 23.0, + "tokens": 37939.0 + }, + { + "bucket": 415.0, + "duration_ms": 9617.347848000003, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 382.0, + "overflow": 24.0, + "padding": 0.0, + "tokens": 44403.0 + }, + { + "bucket": 1146.0, + "duration_ms": 8199.89401900001, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 400.0, + "overflow": 18.0, + "padding": 15.0, + "tokens": 35088.0 + }, + { + "bucket": 690.0, + "duration_ms": 8341.056147, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 377.0, + "overflow": 19.0, + "padding": 2.0, + "tokens": 35322.0 + }, + { + "bucket": 938.0, + "duration_ms": 8602.735985000012, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 384.0, + "overflow": 19.0, + "padding": 14.0, + "tokens": 37364.0 + }, + { + "bucket": 698.0, + "duration_ms": 9105.636625000001, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 404.0, + "overflow": 22.0, + "padding": 3.0, + "tokens": 40949.0 + }, + { + "bucket": 368.0, + "duration_ms": 9234.429309999998, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 357.0, + "overflow": 23.0, + "padding": 0.0, + "tokens": 42032.0 + }, + { + "bucket": 511.0, + "duration_ms": 8338.672249999994, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 410.0, + "overflow": 18.0, + "padding": 5.0, + "tokens": 33517.0 + }, + { + "bucket": 707.0, + "duration_ms": 8830.411349, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 376.0, + "overflow": 20.0, + "padding": 9.0, + "tokens": 39809.0 + }, + { + "bucket": 1006.0, + "duration_ms": 9518.970913000003, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 387.0, + "overflow": 22.0, + "padding": 8.0, + "tokens": 43905.0 + }, + { + "bucket": 1026.0, + "duration_ms": 7924.940408999995, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 399.0, + "overflow": 17.0, + "padding": 14.0, + "tokens": 33149.0 + }, + { + "bucket": 555.0, + "duration_ms": 9312.925641999997, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 409.0, + "overflow": 21.0, + "padding": 6.0, + "tokens": 38795.0 + }, + { + "bucket": 390.0, + "duration_ms": 8887.958122999997, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 408.0, + "overflow": 21.0, + "padding": 0.0, + "tokens": 37762.0 + }, + { + "bucket": 1043.0, + "duration_ms": 9046.173872000007, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 370.0, + "overflow": 22.0, + "padding": 21.0, + "tokens": 43066.0 + }, + { + "bucket": 530.0, + "duration_ms": 8635.939902999999, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 387.0, + "overflow": 19.0, + "padding": 11.0, + "tokens": 36716.0 + }, + { + "bucket": 507.0, + "duration_ms": 8225.677353999987, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 458.0, + "overflow": 17.0, + "padding": 6.0, + "tokens": 32360.0 + }, + { + "bucket": 423.0, + "duration_ms": 8624.490282999994, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 386.0, + "overflow": 21.0, + "padding": 7.0, + "tokens": 39497.0 + }, + { + "bucket": 773.0, + "duration_ms": 8599.456987, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 378.0, + "overflow": 21.0, + "padding": 7.0, + "tokens": 38376.0 + }, + { + "bucket": 795.0, + "duration_ms": 8423.488682999996, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 368.0, + "overflow": 19.0, + "padding": 10.0, + "tokens": 36552.0 + }, + { + "bucket": 551.0, + "duration_ms": 8649.547544999992, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 411.0, + "overflow": 19.0, + "padding": 4.0, + "tokens": 37170.0 + }, + { + "bucket": 1765.0, + "duration_ms": 8547.083584000002, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 429.0, + "overflow": 18.0, + "padding": 22.0, + "tokens": 36596.0 + }, + { + "bucket": 1128.0, + "duration_ms": 8409.201042000006, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 382.0, + "overflow": 19.0, + "padding": 17.0, + "tokens": 36807.0 + }, + { + "bucket": 696.0, + "duration_ms": 8875.711702000002, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 402.0, + "overflow": 20.0, + "padding": 1.0, + "tokens": 36838.0 + }, + { + "bucket": 679.0, + "duration_ms": 9194.931754999994, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 398.0, + "overflow": 21.0, + "padding": 12.0, + "tokens": 40277.0 + }, + { + "bucket": 949.0, + "duration_ms": 9227.201623999996, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 421.0, + "overflow": 22.0, + "padding": 6.0, + "tokens": 41797.0 + }, + { + "bucket": 864.0, + "duration_ms": 8413.728965999999, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 392.0, + "overflow": 19.0, + "padding": 2.0, + "tokens": 35228.0 + }, + { + "bucket": 400.0, + "duration_ms": 9326.991775000006, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 401.0, + "overflow": 22.0, + "padding": 0.0, + "tokens": 40138.0 + }, + { + "bucket": 369.0, + "duration_ms": 8184.575214999996, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 384.0, + "overflow": 20.0, + "padding": 0.0, + "tokens": 35391.0 + }, + { + "bucket": 653.0, + "duration_ms": 8893.912036999998, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 402.0, + "overflow": 21.0, + "padding": 3.0, + "tokens": 40808.0 + }, + { + "bucket": 379.0, + "duration_ms": 8951.537339, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 366.0, + "overflow": 21.0, + "padding": 0.0, + "tokens": 39206.0 + }, + { + "bucket": 494.0, + "duration_ms": 9136.155221999994, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 353.0, + "overflow": 22.0, + "padding": 6.0, + "tokens": 41596.0 + }, + { + "bucket": 1440.0, + "duration_ms": 8599.284305000006, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 444.0, + "overflow": 18.0, + "padding": 12.0, + "tokens": 33942.0 + }, + { + "bucket": 898.0, + "duration_ms": 8410.183011000001, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 401.0, + "overflow": 21.0, + "padding": 14.0, + "tokens": 38166.0 + }, + { + "bucket": 380.0, + "duration_ms": 8305.505917999999, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 396.0, + "overflow": 19.0, + "padding": 1.0, + "tokens": 35656.0 + }, + { + "bucket": 373.0, + "duration_ms": 9378.466820000003, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 368.0, + "overflow": 22.0, + "padding": 0.0, + "tokens": 42144.0 + } + ], + "clean": { + "admitted": 300, + "completed": 300, + "completed_throughput_rps": 1.25, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1816359, + "offered_rps": 1.25, + "output_tokens": 19200, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 0.8021622190135531, + "latency_s": { + "e2e": { + "mean": 0.8661587820320468, + "n": 300, + "p50": 0.72734198148828, + "p95": 1.3885476594397914, + "p99": 1.4521425597564666 + }, + "tpot": { + "mean": 0.006679995698883892, + "n": 300, + "p50": 0.004638713563696319, + "p95": 0.012775019008808564, + "p99": 0.01396566546533168 + }, + "ttft": { + "mean": 0.4453190530023615, + "n": 300, + "p50": 0.43017592300020624, + "p95": 0.6033891130937263, + "p99": 0.6222539604848134 + } + }, + "layer1": { + "decode_tokens": 18903, + "kv_usage_max": 0.052805456386209815, + "kv_usage_mean": 0.021045833023510395, + "mode_counts": { + "FULL": 17787, + "NONE": 1377, + "PIECEWISE": 74 + }, + "mode_shares": { + "FULL": 0.9245763592889074, + "NONE": 0.07157708701528226, + "PIECEWISE": 0.0038465536958103754 + }, + "preemptions": 0, + "prefill_tokens": 1820078, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 28339, + "records_clean": 19238, + "requests_per_step": { + "mean": 1.0367501819315936, + "n": 19238, + "p50": 1.0, + "p95": 2.0, + "p99": 2.0 + }, + "scheduled_tokens_per_step": { + "mean": 95.59106975777108, + "n": 19238, + "p50": 1.0, + "p95": 537.6999999999607, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 21.87376836323942, + "n": 19238, + "p50": 9.320138, + "p95": 120.27652529999995, + "p99": 301.36491193000046 + }, + "token_efficiency_per_ms": 4.370123527431119 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P03", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.2, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.3, + "completion_rate_within_10pct": true, + "tail_end_s": 331.3140532330144, + "tail_start_s": 321.3140532330144, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.2, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.3, + "completion_rate_within_10pct": true, + "tail_end_s": 364.91858237600536, + "tail_start_s": 354.91858237600536, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 300, + "run_dir": "runs/phase3/primary/P03-C01/moderate", + "run_id": "P03-C01-moderate", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.009596612313361013, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.05141005895161719, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 300, + "moe_layer_cv": null, + "overflow_rate": 0.05141005895161719, + "padded_tokens_per_useful_token": 0.0001897790134862731, + "padding_fraction": 0.009596612313361013 + } + }, + "P03-C01-saturation": { + "blocks": [ + { + "duration_ms": 9820.830930999999, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10156.821435, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9768.233745000001, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10029.871413, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10104.615715, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10001.748752999998, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10084.706814999998, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9894.45056, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9969.729338, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10040.877489, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10181.284094999999, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9734.640943000002, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9962.477245, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10118.299278000002, + "eligible_miss": 0.0, + "miss": 33.0, + "model": 33.0, + "overflow": 33.0, + "tokens": 67584.0 + }, + { + "duration_ms": 10158.388776, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9947.161227999999, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9926.129196000002, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9810.286340000002, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10265.503908, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9898.772765000002, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10055.452221000001, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10027.743997000001, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9891.145452, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10007.204138, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10029.236173999998, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9957.308014, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10010.737170999999, + "eligible_miss": 0.0, + "miss": 33.0, + "model": 33.0, + "overflow": 33.0, + "tokens": 67584.0 + }, + { + "duration_ms": 9880.398545000002, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10086.817054999998, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10050.369225999999, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9883.463523999999, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9874.166397, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10038.898957000001, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10046.80094, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10222.667140000001, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9848.842729000002, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10060.680079, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 9929.31106, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9840.875362, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10192.831885999998, + "eligible_miss": 0.0, + "miss": 33.0, + "model": 33.0, + "overflow": 33.0, + "tokens": 67584.0 + }, + { + "duration_ms": 9823.413677999997, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10218.273869000002, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10057.724521999999, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9842.506709000001, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 9849.893720999997, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10006.143056999997, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 31.0, + "overflow": 31.0, + "tokens": 63488.0 + }, + { + "duration_ms": 10209.053572, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + }, + { + "duration_ms": 10185.623811000001, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 32.0, + "overflow": 32.0, + "tokens": 65536.0 + } + ], + "clean": { + "admitted": 501, + "completed": 501, + "completed_throughput_rps": 2.0875, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3073055, + "offered_rps": 2.0875, + "output_tokens": 32064, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 115.08961504499894, + "latency_s": { + "e2e": { + "mean": 113.99268500659687, + "n": 501, + "p50": 121.33762379398104, + "p95": 124.57225271401694, + "p99": 129.85229865298606 + }, + "tpot": { + "mean": 0.15808895612808507, + "n": 501, + "p50": 0.15828559920607535, + "p95": 0.16043450426964445, + "p99": 0.1609802155396051 + }, + "ttft": { + "mean": 104.03308077052752, + "n": 501, + "p50": 111.41430233599385, + "p95": 114.49460457300302, + "p99": 120.11279526297585 + } + }, + "layer1": { + "decode_tokens": 31537, + "kv_usage_max": 0.46842862471359303, + "kv_usage_mean": 0.4521710693825813, + "mode_counts": { + "NONE": 1516 + }, + "mode_shares": { + "NONE": 1.0 + }, + "preemptions": 0, + "prefill_tokens": 3073231, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 236, + "queue_waiting_mean": 233.20712401055408, + "records_all": 3106, + "records_clean": 1516, + "requests_per_step": { + "mean": 22.131926121372032, + "n": 1516, + "p50": 22.0, + "p95": 24.0, + "p99": 24.0 + }, + "scheduled_tokens_per_step": { + "mean": 2048.0, + "n": 1516, + "p50": 2048.0, + "p95": 2048.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 316.6242829643799, + "n": 1516, + "p50": 311.66107999999997, + "p95": 358.29868250000004, + "p99": 374.21161494999996 + }, + "token_efficiency_per_ms": 6.468234150664935 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P03", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 2.0, + "clean_c_waiting_median": 233.0, + "completion_rate_rps": 2.1, + "completion_rate_within_10pct": true, + "tail_end_s": 348.87919582100585, + "tail_start_s": 338.87919582100585, + "valid": true, + "waiting_median": 234.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 2.0, + "clean_c_waiting_median": 233.0, + "completion_rate_rps": 1.9, + "completion_rate_within_10pct": true, + "tail_end_s": 398.6435712639941, + "tail_start_s": 388.6435712639941, + "valid": true, + "waiting_median": 235.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 501, + "run_dir": "runs/phase3/primary/P03-C01/saturation", + "run_id": "P03-C01-saturation", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.0, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 1.0, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 1.0, + "padded_tokens_per_useful_token": 0.0, + "padding_fraction": 0.0 + } + }, + "P03-C10-moderate": { + "blocks": [ + { + "bucket": 432.0, + "duration_ms": 8606.451834000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 426.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 39915.0 + }, + { + "bucket": 390.0, + "duration_ms": 9059.634061, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 364.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45757.0 + }, + { + "bucket": 469.0, + "duration_ms": 9428.323844999995, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 391.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45103.0 + }, + { + "bucket": 430.0, + "duration_ms": 8757.352923000006, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 413.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41620.0 + }, + { + "bucket": 428.0, + "duration_ms": 8500.626963000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 435.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 38894.0 + }, + { + "bucket": 427.0, + "duration_ms": 8880.6502, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 422.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41868.0 + }, + { + "bucket": 447.0, + "duration_ms": 8893.951561999991, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 410.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41639.0 + }, + { + "bucket": 433.0, + "duration_ms": 9634.180625000006, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 406.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46751.0 + }, + { + "bucket": 452.0, + "duration_ms": 9403.595920000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 411.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45103.0 + }, + { + "bucket": 462.0, + "duration_ms": 8918.339332999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 425.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40754.0 + }, + { + "bucket": 394.0, + "duration_ms": 8755.985819000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 401.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42715.0 + }, + { + "bucket": 437.0, + "duration_ms": 9270.625737000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 355.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46240.0 + }, + { + "bucket": 476.0, + "duration_ms": 8676.071426000004, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 456.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 38226.0 + }, + { + "bucket": 425.0, + "duration_ms": 8864.808441000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 389.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43130.0 + }, + { + "bucket": 447.0, + "duration_ms": 8940.273261999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 403.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42214.0 + }, + { + "bucket": 392.0, + "duration_ms": 9269.719275, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 368.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46632.0 + }, + { + "bucket": 477.0, + "duration_ms": 9106.583498999991, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 415.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42267.0 + }, + { + "bucket": 426.0, + "duration_ms": 8750.417243000005, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 418.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41424.0 + }, + { + "bucket": 422.0, + "duration_ms": 9814.590544999997, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 370.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 49275.0 + }, + { + "bucket": 442.0, + "duration_ms": 8302.109783000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 431.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 37646.0 + }, + { + "bucket": 464.0, + "duration_ms": 9556.731577999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 417.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44464.0 + }, + { + "bucket": 433.0, + "duration_ms": 9108.740321000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 416.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43313.0 + }, + { + "bucket": 404.0, + "duration_ms": 8835.236437, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 387.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43111.0 + }, + { + "bucket": 462.0, + "duration_ms": 9117.118130000004, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 435.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42209.0 + }, + { + "bucket": 427.0, + "duration_ms": 8578.996583000004, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 434.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 39814.0 + }, + { + "bucket": 434.0, + "duration_ms": 9440.617024000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 423.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45136.0 + }, + { + "bucket": 467.0, + "duration_ms": 8027.303883, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 444.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 34783.0 + }, + { + "bucket": 418.0, + "duration_ms": 9118.96389, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 385.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 44885.0 + }, + { + "bucket": 439.0, + "duration_ms": 8908.081147, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 423.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41711.0 + }, + { + "bucket": 440.0, + "duration_ms": 8775.029561000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 445.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40126.0 + }, + { + "bucket": 428.0, + "duration_ms": 8871.548213000005, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 395.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42209.0 + }, + { + "bucket": 447.0, + "duration_ms": 9716.337356000007, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 417.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46643.0 + }, + { + "bucket": 439.0, + "duration_ms": 9210.914279, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 442.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42681.0 + }, + { + "bucket": 435.0, + "duration_ms": 9027.704484999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 427.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42068.0 + }, + { + "bucket": 443.0, + "duration_ms": 9020.085927, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 393.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42897.0 + }, + { + "bucket": 437.0, + "duration_ms": 8676.806239999987, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 421.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 40757.0 + }, + { + "bucket": 429.0, + "duration_ms": 9518.529198999995, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 384.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 46872.0 + }, + { + "bucket": 447.0, + "duration_ms": 9277.076311999992, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 395.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 45053.0 + }, + { + "bucket": 437.0, + "duration_ms": 8869.813008000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 409.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41382.0 + }, + { + "bucket": 440.0, + "duration_ms": 8463.263412999995, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 447.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 38229.0 + }, + { + "bucket": 438.0, + "duration_ms": 9138.376169, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 440.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 42630.0 + }, + { + "bucket": 436.0, + "duration_ms": 9575.711637000006, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 373.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 47360.0 + }, + { + "bucket": 411.0, + "duration_ms": 9973.924517000007, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 375.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 49974.0 + }, + { + "bucket": 464.0, + "duration_ms": 9111.179539000006, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 403.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43252.0 + }, + { + "bucket": 441.0, + "duration_ms": 8938.893511999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 448.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41598.0 + }, + { + "bucket": 440.0, + "duration_ms": 8203.012773, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 440.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 36589.0 + }, + { + "bucket": 441.0, + "duration_ms": 8994.393566000004, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 448.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 41980.0 + }, + { + "bucket": 438.0, + "duration_ms": 8992.481718000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 378.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 43403.0 + } + ], + "clean": { + "admitted": 336, + "completed": 335, + "completed_throughput_rps": 1.3958333333333333, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2025114, + "offered_rps": 1.4, + "output_tokens": 21440, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 0.38259596601710655, + "latency_s": { + "e2e": { + "mean": 0.8324871015964326, + "n": 335, + "p50": 0.6860202910029329, + "p95": 1.280704226004309, + "p99": 1.3414597219065765 + }, + "tpot": { + "mean": 0.006963377769765788, + "n": 335, + "p50": 0.004799242444162715, + "p95": 0.012645531087071591, + "p99": 0.013250505485588551 + }, + "ttft": { + "mean": 0.39379430210118793, + "n": 335, + "p50": 0.3876922279887367, + "p95": 0.5416871845984134, + "p99": 0.5613275917002466 + } + }, + "layer1": { + "decode_tokens": 21106, + "kv_usage_max": 0.04955540013987414, + "kv_usage_mean": 0.02050982433691785, + "mode_counts": { + "FULL": 19417, + "NONE": 754 + }, + "mode_shares": { + "FULL": 0.9626196023994844, + "NONE": 0.03738039760051559 + }, + "preemptions": 0, + "prefill_tokens": 2031196, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 29662, + "records_clean": 20171, + "requests_per_step": { + "mean": 1.0630112537801795, + "n": 20171, + "p50": 1.0, + "p95": 2.0, + "p99": 2.0 + }, + "scheduled_tokens_per_step": { + "mean": 101.74517872192752, + "n": 20171, + "p50": 1.0, + "p95": 2.0, + "p99": 5678.799999999996 + }, + "step_duration_ms": { + "mean": 21.460570259927618, + "n": 20171, + "p50": 9.324558, + "p95": 12.3158005, + "p99": 413.3298498999996 + }, + "token_efficiency_per_ms": 4.741028662780309 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P03", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.4, + "completion_rate_within_10pct": true, + "tail_end_s": 331.6514889940154, + "tail_start_s": 321.6514889940154, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.3, + "completion_rate_within_10pct": true, + "tail_end_s": 363.1196642330033, + "tail_start_s": 353.1196642330033, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 335, + "run_dir": "runs/phase3/primary/P03-C10/moderate", + "run_id": "P03-C10-moderate", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.0, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.017010074419075583, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 119, + "moe_layer_cv": null, + "overflow_rate": 0.017010074419075583, + "padded_tokens_per_useful_token": 0.0, + "padding_fraction": 0.0 + } + }, + "P03-C10-saturation": { + "blocks": [ + { + "duration_ms": 9687.455152999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 72210.0 + }, + { + "bucket": 1344.0, + "duration_ms": 9426.609795999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 33.0, + "overflow": 9.0, + "padding": 73.0, + "tokens": 61098.0 + }, + { + "duration_ms": 10645.066627999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 79262.0 + }, + { + "duration_ms": 9608.448556000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73519.0 + }, + { + "duration_ms": 9842.591738, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 1288.0, + "duration_ms": 9692.898608, + "eligible_miss": 1.0, + "miss": 9.0, + "model": 32.0, + "overflow": 8.0, + "padding": 93.0, + "tokens": 60966.0 + }, + { + "bucket": 56.0, + "duration_ms": 10264.24747, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 7.0, + "tokens": 78457.0 + }, + { + "duration_ms": 9965.624573000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 73288.0 + }, + { + "duration_ms": 10783.808192, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81663.0 + }, + { + "bucket": 56.0, + "duration_ms": 9027.21859, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 9.0, + "overflow": 8.0, + "padding": 4.0, + "tokens": 65280.0 + }, + { + "bucket": 1288.0, + "duration_ms": 10933.014086, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 34.0, + "overflow": 11.0, + "padding": 94.0, + "tokens": 76868.0 + }, + { + "duration_ms": 9168.627825000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 66834.0 + }, + { + "duration_ms": 10701.607950000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 84747.0 + }, + { + "duration_ms": 9832.589772000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 1232.0, + "duration_ms": 9859.522224, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 32.0, + "overflow": 10.0, + "padding": 68.0, + "tokens": 63071.0 + }, + { + "duration_ms": 10355.40541, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 77397.0 + }, + { + "duration_ms": 9731.559735, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 72054.0 + }, + { + "duration_ms": 9761.415851000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73079.0 + }, + { + "bucket": 1120.0, + "duration_ms": 9693.590346999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 29.0, + "overflow": 9.0, + "padding": 61.0, + "tokens": 65714.0 + }, + { + "bucket": 112.0, + "duration_ms": 10225.390153999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 13.0, + "overflow": 11.0, + "padding": 9.0, + "tokens": 72902.0 + }, + { + "duration_ms": 10017.791458, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 73933.0 + }, + { + "duration_ms": 9772.353466, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 75807.0 + }, + { + "duration_ms": 10337.735919, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 79175.0 + }, + { + "bucket": 1176.0, + "duration_ms": 10061.821364, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 31.0, + "overflow": 10.0, + "padding": 107.0, + "tokens": 63691.0 + }, + { + "duration_ms": 10157.843276000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 76081.0 + }, + { + "bucket": 56.0, + "duration_ms": 9907.907284, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 6.0, + "tokens": 72726.0 + }, + { + "duration_ms": 10674.836298999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 79153.0 + }, + { + "bucket": 1208.0, + "duration_ms": 8919.901420000002, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 30.0, + "overflow": 8.0, + "padding": 134.0, + "tokens": 57723.0 + }, + { + "duration_ms": 10437.981232999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 78297.0 + }, + { + "bucket": 56.0, + "duration_ms": 9764.594819999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 7.0, + "tokens": 73149.0 + }, + { + "duration_ms": 10628.824848, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 80667.0 + }, + { + "bucket": 392.0, + "duration_ms": 8582.760878000003, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 15.0, + "overflow": 8.0, + "padding": 35.0, + "tokens": 57310.0 + }, + { + "bucket": 784.0, + "duration_ms": 11258.554805000002, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 26.0, + "overflow": 12.0, + "padding": 73.0, + "tokens": 81615.0 + }, + { + "duration_ms": 9436.909324000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 69553.0 + }, + { + "duration_ms": 10436.931808, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 79108.0 + }, + { + "duration_ms": 9560.4529, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 69775.0 + }, + { + "bucket": 1176.0, + "duration_ms": 9651.965396, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 30.0, + "overflow": 9.0, + "padding": 113.0, + "tokens": 61029.0 + }, + { + "duration_ms": 10305.695789, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 78692.0 + }, + { + "bucket": 48.0, + "duration_ms": 10496.442815, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 77771.0 + }, + { + "duration_ms": 9699.25625, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 72736.0 + }, + { + "bucket": 944.0, + "duration_ms": 9765.820548, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 26.0, + "overflow": 9.0, + "padding": 97.0, + "tokens": 61021.0 + }, + { + "bucket": 168.0, + "duration_ms": 10234.118829, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 14.0, + "overflow": 11.0, + "padding": 21.0, + "tokens": 77813.0 + }, + { + "duration_ms": 10091.831287, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 73848.0 + }, + { + "duration_ms": 9733.236188, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 75878.0 + }, + { + "duration_ms": 10696.939206, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 80674.0 + }, + { + "bucket": 1120.0, + "duration_ms": 9454.301014, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 30.0, + "overflow": 10.0, + "padding": 68.0, + "tokens": 59155.0 + }, + { + "duration_ms": 11414.313313, + "eligible_miss": 1.0, + "miss": 9.0, + "model": 9.0, + "overflow": 8.0, + "tokens": 65885.0 + }, + { + "duration_ms": 8901.841632, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 71890.0 + } + ], + "clean": { + "admitted": 559, + "completed": 559, + "completed_throughput_rps": 2.3291666666666666, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3429894, + "offered_rps": 2.3291666666666666, + "output_tokens": 35776, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 91.29178713698639, + "latency_s": { + "e2e": { + "mean": 105.03485774811313, + "n": 559, + "p50": 108.84702699899208, + "p95": 118.28994776179317, + "p99": 126.86703993751374 + }, + "tpot": { + "mean": 0.32836759844498487, + "n": 559, + "p50": 0.3280060368410814, + "p95": 0.33947711540307546, + "p99": 0.3471749682567315 + }, + "ttft": { + "mean": 84.34769904607909, + "n": 559, + "p50": 88.07736104700598, + "p95": 97.8547530898999, + "p99": 106.90823322451381 + } + }, + "layer1": { + "decode_tokens": 35621, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9860734670273078, + "mode_counts": { + "FULL": 244, + "NONE": 474 + }, + "mode_shares": { + "FULL": 0.3398328690807799, + "NONE": 0.6601671309192201 + }, + "preemptions": 1, + "prefill_tokens": 3434399, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 207, + "queue_waiting_mean": 203.81337047353762, + "records_all": 1456, + "records_clean": 718, + "requests_per_step": { + "mean": 50.63091922005571, + "n": 718, + "p50": 51.0, + "p95": 53.0, + "p99": 53.0 + }, + "scheduled_tokens_per_step": { + "mean": 4832.896935933148, + "n": 718, + "p50": 6751.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 667.980022321727, + "n": 718, + "p50": 875.1449805, + "p95": 1118.2583134, + "p99": 1146.5907156899998 + }, + "token_efficiency_per_ms": 7.235092030350308 + }, + "layer2_missing_after_controller_cleanup": true, + "load": "saturation", + "pattern": "P03", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 2.3499999999999996, + "clean_c_waiting_median": 204.0, + "completion_rate_rps": 2.1, + "completion_rate_within_10pct": false, + "tail_end_s": 356.32926020701416, + "tail_start_s": 346.32926020701416, + "valid": false, + "waiting_median": 206.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 2.3499999999999996, + "clean_c_waiting_median": 204.0, + "completion_rate_rps": 2.1, + "completion_rate_within_10pct": false, + "tail_end_s": 417.3262625900097, + "tail_start_s": 407.3262625900097, + "valid": false, + "waiting_median": 205.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 559, + "run_dir": "runs/phase3/primary/P03-C10/saturation", + "run_id": "P03-C10-saturation", + "trace_files": 0, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.07603458049886622, + "eligible_miss_rate": 0.002785515320334262, + "graph_miss_rate": 0.6601671309192201, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.6573816155988857, + "padded_tokens_per_useful_token": 0.0003083555714376286, + "padding_fraction": 0.07853787433940106 + } + }, + "P04-C00-moderate": { + "blocks": [ + { + "bucket": 1456.0, + "duration_ms": 8981.908387999994, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 186.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 33537.0 + }, + { + "bucket": 2618.0, + "duration_ms": 10098.805099999992, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 332.0, + "overflow": 3.0, + "padding": 7.0, + "tokens": 24601.0 + }, + { + "bucket": 1840.0, + "duration_ms": 9484.212540999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 234.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 29683.0 + }, + { + "bucket": 2231.0, + "duration_ms": 9981.909945, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 285.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 33276.0 + }, + { + "bucket": 1936.0, + "duration_ms": 8581.831738000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 245.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 19888.0 + }, + { + "bucket": 2135.0, + "duration_ms": 10071.43481000001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 273.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 34368.0 + }, + { + "bucket": 1408.0, + "duration_ms": 8588.356751000007, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 180.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 27431.0 + }, + { + "bucket": 2368.0, + "duration_ms": 11072.200314999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 300.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 35136.0 + }, + { + "bucket": 2552.0, + "duration_ms": 8904.710039000003, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 288.0, + "overflow": 2.0, + "padding": 28.0, + "tokens": 16184.0 + }, + { + "bucket": 1786.0, + "duration_ms": 8996.101777999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 230.0, + "overflow": 5.0, + "padding": 4.0, + "tokens": 38785.0 + }, + { + "bucket": 3208.0, + "duration_ms": 8631.933416999998, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 341.0, + "overflow": 1.0, + "padding": 6.0, + "tokens": 11394.0 + }, + { + "bucket": 1359.0, + "duration_ms": 7128.594627000001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 176.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 33353.0 + }, + { + "bucket": 2200.0, + "duration_ms": 9367.741511, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 278.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 23940.0 + }, + { + "bucket": 1874.0, + "duration_ms": 9525.214446999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 241.0, + "overflow": 5.0, + "padding": 7.0, + "tokens": 40863.0 + }, + { + "bucket": 2824.0, + "duration_ms": 8884.210989, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 323.0, + "overflow": 1.0, + "padding": 3.0, + "tokens": 11013.0 + }, + { + "bucket": 1503.0, + "duration_ms": 8559.785575, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 195.0, + "overflow": 5.0, + "padding": 7.0, + "tokens": 40929.0 + }, + { + "bucket": 2776.0, + "duration_ms": 9198.640848000005, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 348.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 8403.0 + }, + { + "bucket": 1297.0, + "duration_ms": 8906.830054, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 170.0, + "overflow": 6.0, + "padding": 4.0, + "tokens": 46993.0 + }, + { + "bucket": 3064.0, + "duration_ms": 8544.142515999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 383.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3064.0 + }, + { + "bucket": 1010.0, + "duration_ms": 7998.338001000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 134.0, + "overflow": 6.0, + "padding": 4.0, + "tokens": 47062.0 + }, + { + "bucket": 2744.0, + "duration_ms": 8611.122775000005, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 344.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 7439.0 + }, + { + "bucket": 1327.0, + "duration_ms": 9267.594286, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 174.0, + "overflow": 6.0, + "padding": 6.0, + "tokens": 50473.0 + }, + { + "bucket": 2840.0, + "duration_ms": 8589.381527999998, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 356.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 4600.0 + }, + { + "bucket": 1231.0, + "duration_ms": 8726.466514, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 162.0, + "overflow": 6.0, + "padding": 7.0, + "tokens": 47914.0 + }, + { + "bucket": 3056.0, + "duration_ms": 9110.523627999997, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 383.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 4266.0 + }, + { + "bucket": 1015.0, + "duration_ms": 8217.807083000002, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 135.0, + "overflow": 6.0, + "padding": 6.0, + "tokens": 47537.0 + }, + { + "bucket": 3192.0, + "duration_ms": 9264.992175999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 399.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3192.0 + }, + { + "bucket": 882.0, + "duration_ms": 8246.404675000002, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 118.0, + "overflow": 6.0, + "padding": 4.0, + "tokens": 50030.0 + }, + { + "bucket": 2768.0, + "duration_ms": 9175.787352999994, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 347.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 7379.0 + }, + { + "bucket": 1311.0, + "duration_ms": 9409.615027000003, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 172.0, + "overflow": 6.0, + "padding": 6.0, + "tokens": 46846.0 + }, + { + "bucket": 3352.0, + "duration_ms": 9545.358950999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 419.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3352.0 + }, + { + "bucket": 721.0, + "duration_ms": 7947.838777999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 98.0, + "overflow": 6.0, + "padding": 4.0, + "tokens": 48373.0 + }, + { + "bucket": 3400.0, + "duration_ms": 9494.459786, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 425.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3400.0 + }, + { + "bucket": 666.0, + "duration_ms": 7796.334624, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 92.0, + "overflow": 7.0, + "padding": 4.0, + "tokens": 50886.0 + }, + { + "bucket": 3264.0, + "duration_ms": 9631.711120000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 408.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3264.0 + }, + { + "bucket": 815.0, + "duration_ms": 8377.705931, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 110.0, + "overflow": 6.0, + "padding": 6.0, + "tokens": 49358.0 + }, + { + "bucket": 3800.0, + "duration_ms": 9837.149042000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 475.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3800.0 + }, + { + "bucket": 374.0, + "duration_ms": 6803.713179000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 54.0, + "overflow": 6.0, + "padding": 7.0, + "tokens": 47662.0 + }, + { + "bucket": 3304.0, + "duration_ms": 9963.765209000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 413.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3304.0 + }, + { + "bucket": 673.0, + "duration_ms": 8837.429310000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 93.0, + "overflow": 7.0, + "padding": 4.0, + "tokens": 54138.0 + }, + { + "bucket": 3560.0, + "duration_ms": 9562.608677000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 445.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3560.0 + }, + { + "bucket": 946.0, + "duration_ms": 8044.274325999995, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 126.0, + "overflow": 6.0, + "padding": 7.0, + "tokens": 45628.0 + }, + { + "bucket": 3631.0, + "duration_ms": 9953.335582999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 456.0, + "overflow": 0.0, + "padding": 3.0, + "tokens": 3628.0 + }, + { + "bucket": 232.0, + "duration_ms": 6868.462794000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 36.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 49763.0 + }, + { + "bucket": 3640.0, + "duration_ms": 9983.985477999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 455.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3640.0 + }, + { + "bucket": 449.0, + "duration_ms": 7694.749212000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 65.0, + "overflow": 7.0, + "padding": 7.0, + "tokens": 51796.0 + }, + { + "bucket": 3760.0, + "duration_ms": 9970.747046999993, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 470.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 3760.0 + }, + { + "bucket": 591.0, + "duration_ms": 7556.453625000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 83.0, + "overflow": 7.0, + "padding": 6.0, + "tokens": 48512.0 + } + ], + "clean": { + "admitted": 192, + "completed": 192, + "completed_throughput_rps": 0.8, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1184043, + "offered_rps": 0.8, + "output_tokens": 98304, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 1.4950859160162508, + "latency_s": { + "e2e": { + "mean": 8.979008184640861, + "n": 192, + "p50": 8.750423648001743, + "p95": 9.690806861204328, + "p99": 13.313055781236617 + }, + "tpot": { + "mean": 0.013778383638832437, + "n": 192, + "p50": 0.013384224433496772, + "p95": 0.01721187468572363, + "p99": 0.023013204993655518 + }, + "ttft": { + "mean": 1.9382541451974855, + "n": 192, + "p50": 2.007326473496505, + "p95": 3.2427438382524993, + "p99": 3.608097693949822 + } + }, + "layer1": { + "decode_tokens": 98661, + "kv_usage_max": 0.36836982968369825, + "kv_usage_mean": 0.17221124824207087, + "mode_counts": { + "FULL": 12291, + "NONE": 208, + "PIECEWISE": 2 + }, + "mode_shares": { + "FULL": 0.9832013438924886, + "NONE": 0.01663866890648748, + "PIECEWISE": 0.00015998720102391808 + }, + "preemptions": 0, + "prefill_tokens": 1208742, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 6, + "queue_waiting_mean": 0.024478041756659467, + "records_all": 19184, + "records_clean": 12501, + "requests_per_step": { + "mean": 7.917766578673706, + "n": 12501, + "p50": 8.0, + "p95": 8.0, + "p99": 8.0 + }, + "scheduled_tokens_per_step": { + "mean": 104.5838732901368, + "n": 12501, + "p50": 8.0, + "p95": 8.0, + "p99": 7505.0 + }, + "step_duration_ms": { + "mean": 34.23699552891769, + "n": 12501, + "p50": 22.373712, + "p95": 25.211079, + "p99": 784.700082 + }, + "token_efficiency_per_ms": 3.0547035940055496 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P04", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.8, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": true, + "tail_end_s": 331.31545102299424, + "tail_start_s": 321.31545102299424, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.8, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": true, + "tail_end_s": 362.91449153699796, + "tail_start_s": 352.91449153699796, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 192, + "run_dir": "runs/phase3/primary/P04-C00/moderate", + "run_id": "P04-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.00166685187243027, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.013006824568446406, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 141, + "moe_layer_cv": null, + "overflow_rate": 0.013006824568446406, + "padded_tokens_per_useful_token": 0.00012620439145389755, + "padding_fraction": 0.00166685187243027 + } + }, + "P04-C00-saturation": { + "blocks": [ + { + "bucket": 7760.0, + "duration_ms": 9661.424796000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 156.0, + "overflow": 0.0, + "padding": 664.0, + "tokens": 7096.0 + }, + { + "bucket": 4560.0, + "duration_ms": 11026.035121000004, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 101.0, + "overflow": 6.0, + "padding": 476.0, + "tokens": 41799.0 + }, + { + "duration_ms": 10369.803374000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9400.000758999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10727.988309, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "bucket": 5280.0, + "duration_ms": 8557.027166000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 113.0, + "overflow": 3.0, + "padding": 21.0, + "tokens": 16948.0 + }, + { + "bucket": 8112.0, + "duration_ms": 9916.120584999993, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 169.0, + "overflow": 0.0, + "padding": 231.0, + "tokens": 7881.0 + }, + { + "bucket": 7920.0, + "duration_ms": 9986.177872999995, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 165.0, + "overflow": 0.0, + "padding": 436.0, + "tokens": 7484.0 + }, + { + "bucket": 1488.0, + "duration_ms": 11304.550902000003, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 42.0, + "overflow": 11.0, + "padding": 94.0, + "tokens": 74374.0 + }, + { + "duration_ms": 9583.59484, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 9862.853266, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 48.0, + "duration_ms": 10026.114451, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 73216.0 + }, + { + "bucket": 7152.0, + "duration_ms": 9198.616807000002, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 150.0, + "overflow": 1.0, + "padding": 109.0, + "tokens": 11483.0 + }, + { + "bucket": 7968.0, + "duration_ms": 9991.270598000001, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 167.0, + "overflow": 1.0, + "padding": 294.0, + "tokens": 11577.0 + }, + { + "bucket": 7632.0, + "duration_ms": 10544.012887999994, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 161.0, + "overflow": 2.0, + "padding": 512.0, + "tokens": 18576.0 + }, + { + "duration_ms": 9973.108670000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 76425.0 + }, + { + "duration_ms": 10665.678355, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9722.500454999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 1984.0, + "duration_ms": 9096.570938999997, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 40.0, + "overflow": 6.0, + "padding": 49.0, + "tokens": 50598.0 + }, + { + "bucket": 7872.0, + "duration_ms": 9917.962257000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 164.0, + "overflow": 0.0, + "padding": 250.0, + "tokens": 7622.0 + }, + { + "bucket": 7440.0, + "duration_ms": 10042.182822000002, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 156.0, + "overflow": 1.0, + "padding": 382.0, + "tokens": 9371.0 + }, + { + "bucket": 5856.0, + "duration_ms": 10961.045746999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 127.0, + "overflow": 5.0, + "padding": 428.0, + "tokens": 32912.0 + }, + { + "duration_ms": 9701.789703, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 73342.0 + }, + { + "duration_ms": 10480.6525, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9541.685850000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 4008.0, + "duration_ms": 9259.903107000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 79.0, + "overflow": 4.0, + "padding": 357.0, + "tokens": 36411.0 + }, + { + "bucket": 8016.0, + "duration_ms": 9968.672883000003, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 168.0, + "overflow": 1.0, + "padding": 39.0, + "tokens": 10814.0 + }, + { + "bucket": 7728.0, + "duration_ms": 10029.055906999994, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 162.0, + "overflow": 1.0, + "padding": 232.0, + "tokens": 10443.0 + }, + { + "bucket": 3504.0, + "duration_ms": 11264.764591, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 81.0, + "overflow": 8.0, + "padding": 149.0, + "tokens": 61248.0 + }, + { + "duration_ms": 10113.265851999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 78314.0 + }, + { + "duration_ms": 9743.199005, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 9638.610864000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 6288.0, + "duration_ms": 9178.285882, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 133.0, + "overflow": 2.0, + "padding": 16.0, + "tokens": 13431.0 + }, + { + "bucket": 8064.0, + "duration_ms": 10005.450483999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 168.0, + "overflow": 0.0, + "padding": 226.0, + "tokens": 7838.0 + }, + { + "bucket": 7296.0, + "duration_ms": 10016.172216999998, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 153.0, + "overflow": 1.0, + "padding": 377.0, + "tokens": 10862.0 + }, + { + "bucket": 1152.0, + "duration_ms": 10843.782151, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 34.0, + "overflow": 10.0, + "padding": 73.0, + "tokens": 75324.0 + }, + { + "duration_ms": 10150.313138, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 76025.0 + }, + { + "duration_ms": 9713.204290000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 624.0, + "duration_ms": 9239.420065000002, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 21.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 63071.0 + }, + { + "bucket": 8112.0, + "duration_ms": 10021.796982, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 170.0, + "overflow": 1.0, + "padding": 46.0, + "tokens": 10196.0 + }, + { + "bucket": 8208.0, + "duration_ms": 9994.886259000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 171.0, + "overflow": 0.0, + "padding": 222.0, + "tokens": 7986.0 + }, + { + "bucket": 5904.0, + "duration_ms": 11039.833101999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 128.0, + "overflow": 5.0, + "padding": 284.0, + "tokens": 35573.0 + }, + { + "duration_ms": 9913.298117, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 9716.298324, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10405.118951999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "bucket": 3216.0, + "duration_ms": 8902.433363000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 71.0, + "overflow": 4.0, + "padding": 67.0, + "tokens": 35679.0 + }, + { + "bucket": 7296.0, + "duration_ms": 9963.001459999998, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 153.0, + "overflow": 1.0, + "padding": 282.0, + "tokens": 11615.0 + }, + { + "bucket": 7728.0, + "duration_ms": 10025.764016999992, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 161.0, + "overflow": 0.0, + "padding": 480.0, + "tokens": 7248.0 + } + ], + "clean": { + "admitted": 324, + "completed": 324, + "completed_throughput_rps": 1.35, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1969814, + "offered_rps": 1.35, + "output_tokens": 165888, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 168.52608891300042, + "latency_s": { + "e2e": { + "mean": 153.5840079071512, + "n": 324, + "p50": 176.7334543744946, + "p95": 204.46718762134697, + "p99": 209.50116568645083 + }, + "tpot": { + "mean": 0.06281919152370617, + "n": 324, + "p50": 0.06188693252445537, + "p95": 0.06422231276379772, + "p99": 0.08522842260954068 + }, + "ttft": { + "mean": 121.48340103853735, + "n": 324, + "p50": 145.04611464199843, + "p95": 173.27588533475497, + "p99": 178.38239817070252 + } + }, + "layer1": { + "decode_tokens": 252604, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9870783228921769, + "mode_counts": { + "FULL": 3481, + "NONE": 272, + "PIECEWISE": 2 + }, + "mode_shares": { + "FULL": 0.9270306258322237, + "NONE": 0.07243675099866845, + "PIECEWISE": 0.0005326231691078562 + }, + "preemptions": 24, + "prefill_tokens": 1967058, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 213, + "queue_waiting_mean": 209.8069241011984, + "records_all": 8254, + "records_clean": 3755, + "requests_per_step": { + "mean": 46.02103861517976, + "n": 3755, + "p50": 46.0, + "p95": 48.0, + "p99": 49.0 + }, + "scheduled_tokens_per_step": { + "mean": 591.1217043941411, + "n": 3755, + "p50": 46.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 127.6711850985353, + "n": 3755, + "p50": 59.800215, + "p95": 1020.3509887999998, + "p99": 1101.04686276 + }, + "token_efficiency_per_ms": 4.630032249938932 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P04", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.15, + "clean_c_waiting_median": 210.0, + "completion_rate_rps": 0.1, + "completion_rate_within_10pct": false, + "tail_end_s": 333.2067631710088, + "tail_start_s": 323.2067631710088, + "valid": false, + "waiting_median": 212.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.15, + "clean_c_waiting_median": 210.0, + "completion_rate_rps": 0.7, + "completion_rate_within_10pct": false, + "tail_end_s": 366.1246157819987, + "tail_start_s": 356.1246157819987, + "valid": false, + "waiting_median": 209.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 324, + "run_dir": "runs/phase3/primary/P04-C00/saturation", + "run_id": "P04-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.24734040984998104, + "R32": 0.23857870988687485, + "R64": 0.24443376797326863, + "bucket_slack": 0.04040043753269606, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.07243675099866845, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.07243675099866845, + "padded_tokens_per_useful_token": 0.0030617274161561536, + "padding_fraction": 0.04040043753269606 + } + }, + "P06-C00-moderate": { + "blocks": [ + { + "bucket": 2896.0, + "duration_ms": 9977.739791, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 318.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 33597.0 + }, + { + "bucket": 2656.0, + "duration_ms": 10026.279938000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 326.0, + "overflow": 4.0, + "padding": 7.0, + "tokens": 32460.0 + }, + { + "bucket": 3832.0, + "duration_ms": 9938.875863000001, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 322.0, + "overflow": 2.0, + "padding": 15.0, + "tokens": 19113.0 + }, + { + "bucket": 5096.0, + "duration_ms": 11421.808803999997, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 395.0, + "overflow": 3.0, + "padding": 27.0, + "tokens": 27356.0 + }, + { + "bucket": 4112.0, + "duration_ms": 8522.299042999995, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 421.0, + "overflow": 0.0, + "padding": 2.0, + "tokens": 4110.0 + }, + { + "bucket": 3264.0, + "duration_ms": 9974.524448999991, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 336.0, + "overflow": 3.0, + "padding": 12.0, + "tokens": 26847.0 + }, + { + "bucket": 3472.0, + "duration_ms": 9979.503383000014, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 329.0, + "overflow": 4.0, + "padding": 7.0, + "tokens": 28440.0 + }, + { + "bucket": 3376.0, + "duration_ms": 9994.057406999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 330.0, + "overflow": 4.0, + "padding": 10.0, + "tokens": 29265.0 + }, + { + "bucket": 3680.0, + "duration_ms": 9988.828849000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 335.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20268.0 + }, + { + "bucket": 3288.0, + "duration_ms": 9975.919217, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 336.0, + "overflow": 3.0, + "padding": 11.0, + "tokens": 24992.0 + }, + { + "bucket": 4632.0, + "duration_ms": 9973.066123999999, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 440.0, + "overflow": 1.0, + "padding": 7.0, + "tokens": 12364.0 + }, + { + "bucket": 3712.0, + "duration_ms": 9796.098216, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 422.0, + "overflow": 3.0, + "padding": 17.0, + "tokens": 28271.0 + }, + { + "bucket": 3520.0, + "duration_ms": 10179.04618099999, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 410.0, + "overflow": 3.0, + "padding": 14.0, + "tokens": 28082.0 + }, + { + "bucket": 3296.0, + "duration_ms": 8780.04112, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 313.0, + "overflow": 2.0, + "padding": 20.0, + "tokens": 13630.0 + }, + { + "bucket": 3184.0, + "duration_ms": 9968.883369999992, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 259.0, + "overflow": 5.0, + "padding": 11.0, + "tokens": 32334.0 + }, + { + "bucket": 4288.0, + "duration_ms": 9980.859213000005, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 348.0, + "overflow": 2.0, + "padding": 17.0, + "tokens": 17762.0 + }, + { + "bucket": 3952.0, + "duration_ms": 9992.083239999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 348.0, + "overflow": 3.0, + "padding": 9.0, + "tokens": 22666.0 + }, + { + "bucket": 3688.0, + "duration_ms": 9978.982320999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 380.0, + "overflow": 3.0, + "padding": 11.0, + "tokens": 21289.0 + }, + { + "bucket": 2592.0, + "duration_ms": 10035.271873999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 328.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 30630.0 + }, + { + "bucket": 3576.0, + "duration_ms": 11149.077350000001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 382.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 36338.0 + }, + { + "bucket": 3136.0, + "duration_ms": 9374.885931000008, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 269.0, + "overflow": 4.0, + "padding": 15.0, + "tokens": 23355.0 + }, + { + "bucket": 4768.0, + "duration_ms": 9343.843769999994, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 353.0, + "overflow": 0.0, + "padding": 15.0, + "tokens": 4753.0 + }, + { + "bucket": 4016.0, + "duration_ms": 9988.893537000016, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 318.0, + "overflow": 3.0, + "padding": 20.0, + "tokens": 28572.0 + }, + { + "bucket": 3768.0, + "duration_ms": 9978.159868, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 349.0, + "overflow": 4.0, + "padding": 13.0, + "tokens": 25966.0 + }, + { + "bucket": 3848.0, + "duration_ms": 9978.960759999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 399.0, + "overflow": 3.0, + "padding": 13.0, + "tokens": 22342.0 + }, + { + "bucket": 3648.0, + "duration_ms": 9980.344796999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 424.0, + "overflow": 3.0, + "padding": 8.0, + "tokens": 19594.0 + }, + { + "bucket": 3544.0, + "duration_ms": 9970.853603000001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 363.0, + "overflow": 3.0, + "padding": 6.0, + "tokens": 26235.0 + }, + { + "bucket": 3280.0, + "duration_ms": 10198.273267000004, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 359.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 27480.0 + }, + { + "bucket": 4040.0, + "duration_ms": 10590.225440999997, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 407.0, + "overflow": 3.0, + "padding": 11.0, + "tokens": 26086.0 + }, + { + "bucket": 4384.0, + "duration_ms": 10426.231084000008, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 447.0, + "overflow": 2.0, + "padding": 6.0, + "tokens": 20762.0 + }, + { + "bucket": 3056.0, + "duration_ms": 8686.102160000002, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 322.0, + "overflow": 2.0, + "padding": 6.0, + "tokens": 15162.0 + }, + { + "bucket": 4816.0, + "duration_ms": 9979.046929000002, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 434.0, + "overflow": 2.0, + "padding": 22.0, + "tokens": 15088.0 + }, + { + "bucket": 2752.0, + "duration_ms": 9979.596705000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 348.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 32405.0 + }, + { + "bucket": 3696.0, + "duration_ms": 9974.602910999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 370.0, + "overflow": 4.0, + "padding": 11.0, + "tokens": 24107.0 + }, + { + "bucket": 4248.0, + "duration_ms": 9966.603755999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 454.0, + "overflow": 2.0, + "padding": 18.0, + "tokens": 13044.0 + }, + { + "bucket": 3100.0, + "duration_ms": 8939.906178000005, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 391.0, + "overflow": 3.0, + "padding": 1.0, + "tokens": 22061.0 + }, + { + "bucket": 3416.0, + "duration_ms": 11407.424273999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 431.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 35009.0 + }, + { + "bucket": 3016.0, + "duration_ms": 9221.427410000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 360.0, + "overflow": 4.0, + "padding": 8.0, + "tokens": 18203.0 + }, + { + "bucket": 4488.0, + "duration_ms": 9300.330514999992, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 409.0, + "overflow": 0.0, + "padding": 12.0, + "tokens": 4476.0 + }, + { + "bucket": 3168.0, + "duration_ms": 9992.190109000017, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 336.0, + "overflow": 3.0, + "padding": 10.0, + "tokens": 26458.0 + }, + { + "bucket": 2896.0, + "duration_ms": 9985.548586999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 254.0, + "overflow": 5.0, + "padding": 7.0, + "tokens": 34100.0 + }, + { + "bucket": 3408.0, + "duration_ms": 9982.407799000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 304.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 28456.0 + }, + { + "bucket": 3640.0, + "duration_ms": 9981.351961999993, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 318.0, + "overflow": 4.0, + "padding": 18.0, + "tokens": 28462.0 + }, + { + "bucket": 4400.0, + "duration_ms": 9964.023838000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 405.0, + "overflow": 3.0, + "padding": 14.0, + "tokens": 21380.0 + }, + { + "bucket": 3552.0, + "duration_ms": 10730.976689999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 429.0, + "overflow": 4.0, + "padding": 9.0, + "tokens": 31431.0 + }, + { + "bucket": 3352.0, + "duration_ms": 10643.849985000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 422.0, + "overflow": 4.0, + "padding": 7.0, + "tokens": 28467.0 + }, + { + "bucket": 3272.0, + "duration_ms": 8558.336599999995, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 322.0, + "overflow": 1.0, + "padding": 11.0, + "tokens": 11404.0 + }, + { + "bucket": 4920.0, + "duration_ms": 9976.434318000003, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 389.0, + "overflow": 2.0, + "padding": 21.0, + "tokens": 18903.0 + } + ], + "clean": { + "admitted": 336, + "completed": 336, + "completed_throughput_rps": 1.4, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 955216, + "offered_rps": 1.4, + "output_tokens": 172032, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 5.683892283006571, + "latency_s": { + "e2e": { + "mean": 8.300746099748721, + "n": 336, + "p50": 8.40417995498865, + "p95": 9.786131062501227, + "p99": 10.154464515444124 + }, + "tpot": { + "mean": 0.01437395923223587, + "n": 336, + "p50": 0.01442530346280711, + "p95": 0.017416477798915524, + "p99": 0.018449774422510133 + }, + "ttft": { + "mean": 0.9556529320761911, + "n": 336, + "p50": 0.9955719049903564, + "p95": 1.8687072527463897, + "p99": 2.1914087791024928 + } + }, + "layer1": { + "decode_tokens": 173276, + "kv_usage_max": 0.23579345769126792, + "kv_usage_mean": 0.10409258767106862, + "mode_counts": { + "FULL": 17310, + "NONE": 149, + "PIECEWISE": 9 + }, + "mode_shares": { + "FULL": 0.9909548889397756, + "NONE": 0.008529883215021754, + "PIECEWISE": 0.0005152278452026563 + }, + "preemptions": 0, + "prefill_tokens": 950299, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 6, + "queue_waiting_mean": 0.011735745362949393, + "records_all": 25870, + "records_clean": 17468, + "requests_per_step": { + "mean": 9.943782917334554, + "n": 17468, + "p50": 8.0, + "p95": 16.0, + "p99": 16.0 + }, + "scheduled_tokens_per_step": { + "mean": 64.32190290817495, + "n": 17468, + "p50": 8.0, + "p95": 16.0, + "p99": 16.0 + }, + "step_duration_ms": { + "mean": 27.291852446588045, + "n": 17468, + "p50": 19.243071, + "p95": 28.63492535, + "p99": 93.35969100999505 + }, + "token_efficiency_per_ms": 2.3568170403257582 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": false, + "tail_end_s": 331.6668705250195, + "tail_start_s": 321.6668705250195, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.6, + "completion_rate_within_10pct": true, + "tail_end_s": 363.2625439830008, + "tail_start_s": 353.2625439830008, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 336, + "run_dir": "runs/phase3/primary/P06-C00/moderate", + "run_id": "P06-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.002759758734494139, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.008302794319743472, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 152, + "moe_layer_cv": null, + "overflow_rate": 0.008302794319743472, + "padded_tokens_per_useful_token": 0.00043165787775626904, + "padding_fraction": 0.002759758734494139 + } + }, + "P06-C00-saturation": { + "blocks": [ + { + "bucket": 12200.0, + "duration_ms": 9955.488134000001, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 119.0, + "overflow": 1.0, + "padding": 685.0, + "tokens": 15219.0 + }, + { + "bucket": 11712.0, + "duration_ms": 9996.430858999998, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 124.0, + "overflow": 2.0, + "padding": 346.0, + "tokens": 16607.0 + }, + { + "bucket": 12288.0, + "duration_ms": 10013.737815999999, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 129.0, + "overflow": 1.0, + "padding": 673.0, + "tokens": 13629.0 + }, + { + "bucket": 368.0, + "duration_ms": 10611.232619999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 15.0, + "overflow": 11.0, + "padding": 15.0, + "tokens": 84440.0 + }, + { + "duration_ms": 10559.656503, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9437.028964000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 2184.0, + "duration_ms": 9372.165505999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 28.0, + "overflow": 7.0, + "padding": 53.0, + "tokens": 58674.0 + }, + { + "bucket": 12896.0, + "duration_ms": 9988.717321, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 516.0, + "tokens": 12380.0 + }, + { + "bucket": 12104.0, + "duration_ms": 9950.489277, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 122.0, + "overflow": 1.0, + "padding": 546.0, + "tokens": 12471.0 + }, + { + "bucket": 11904.0, + "duration_ms": 10044.752167999994, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 581.0, + "tokens": 11323.0 + }, + { + "bucket": 7856.0, + "duration_ms": 11363.681751999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 92.0, + "overflow": 6.0, + "padding": 276.0, + "tokens": 45842.0 + }, + { + "duration_ms": 9348.223145, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10283.85525, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9479.791809, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 7952.0, + "duration_ms": 9480.092465, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 75.0, + "overflow": 4.0, + "padding": 305.0, + "tokens": 34686.0 + }, + { + "bucket": 13344.0, + "duration_ms": 9951.624168, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 124.0, + "overflow": 1.0, + "padding": 523.0, + "tokens": 15795.0 + }, + { + "bucket": 13000.0, + "duration_ms": 9992.023440000004, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 126.0, + "overflow": 1.0, + "padding": 431.0, + "tokens": 15494.0 + }, + { + "bucket": 12048.0, + "duration_ms": 10005.716017999994, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 125.0, + "overflow": 1.0, + "padding": 237.0, + "tokens": 18903.0 + }, + { + "bucket": 3360.0, + "duration_ms": 11177.442514, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 44.0, + "overflow": 9.0, + "padding": 104.0, + "tokens": 73653.0 + }, + { + "duration_ms": 9424.047387, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10728.534254, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "bucket": 480.0, + "duration_ms": 9615.067625000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 14.0, + "overflow": 9.0, + "padding": 16.0, + "tokens": 66217.0 + }, + { + "bucket": 11328.0, + "duration_ms": 9024.415422, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 118.0, + "overflow": 0.0, + "padding": 402.0, + "tokens": 10926.0 + }, + { + "bucket": 12272.0, + "duration_ms": 10041.456662999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 132.0, + "overflow": 0.0, + "padding": 585.0, + "tokens": 11687.0 + }, + { + "bucket": 11792.0, + "duration_ms": 9956.999717000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 134.0, + "overflow": 0.0, + "padding": 583.0, + "tokens": 11209.0 + }, + { + "bucket": 7824.0, + "duration_ms": 10999.740231, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 94.0, + "overflow": 5.0, + "padding": 583.0, + "tokens": 42306.0 + }, + { + "duration_ms": 9748.006496000002, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10606.91422, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9383.162393, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 7184.0, + "duration_ms": 9217.577436, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 71.0, + "overflow": 4.0, + "padding": 189.0, + "tokens": 35953.0 + }, + { + "bucket": 12840.0, + "duration_ms": 10052.109649000002, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 118.0, + "overflow": 1.0, + "padding": 421.0, + "tokens": 20487.0 + }, + { + "bucket": 11984.0, + "duration_ms": 9913.599149, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 116.0, + "overflow": 1.0, + "padding": 209.0, + "tokens": 19156.0 + }, + { + "bucket": 12576.0, + "duration_ms": 10053.582624999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 131.0, + "overflow": 0.0, + "padding": 493.0, + "tokens": 12083.0 + }, + { + "bucket": 4512.0, + "duration_ms": 11061.520617999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 55.0, + "overflow": 8.0, + "padding": 252.0, + "tokens": 63278.0 + }, + { + "duration_ms": 9458.263235999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10419.149388, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9654.973689, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73553.0 + }, + { + "bucket": 9592.0, + "duration_ms": 9694.816922000002, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 111.0, + "overflow": 2.0, + "padding": 46.0, + "tokens": 21082.0 + }, + { + "bucket": 11352.0, + "duration_ms": 9669.751934000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 129.0, + "overflow": 0.0, + "padding": 307.0, + "tokens": 11045.0 + }, + { + "bucket": 10912.0, + "duration_ms": 9934.947404, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 127.0, + "overflow": 1.0, + "padding": 609.0, + "tokens": 13237.0 + }, + { + "bucket": 9040.0, + "duration_ms": 10788.090522, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 117.0, + "overflow": 4.0, + "padding": 256.0, + "tokens": 30161.0 + }, + { + "duration_ms": 10598.505613999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9569.238599, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 9677.49048, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 4800.0, + "duration_ms": 9392.868976999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 55.0, + "overflow": 5.0, + "padding": 76.0, + "tokens": 44430.0 + }, + { + "bucket": 11904.0, + "duration_ms": 9982.797705000004, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 125.0, + "overflow": 1.0, + "padding": 396.0, + "tokens": 16678.0 + }, + { + "bucket": 12680.0, + "duration_ms": 9975.084492000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 136.0, + "overflow": 0.0, + "padding": 467.0, + "tokens": 12213.0 + }, + { + "bucket": 10824.0, + "duration_ms": 9999.780528000003, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 124.0, + "overflow": 1.0, + "padding": 107.0, + "tokens": 18627.0 + } + ], + "clean": { + "admitted": 564, + "completed": 564, + "completed_throughput_rps": 2.35, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1633038, + "offered_rps": 2.35, + "output_tokens": 288768, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 95.17034762600088, + "latency_s": { + "e2e": { + "mean": 101.44001434788701, + "n": 564, + "p50": 107.6167936980055, + "p95": 122.415330010849, + "p99": 126.31151792582939 + }, + "tpot": { + "mean": 0.07128777578487226, + "n": 564, + "p50": 0.06864038345594425, + "p95": 0.0934390971569203, + "p99": 0.10815281686064476 + }, + "ttft": { + "mean": 65.01196092181728, + "n": 564, + "p50": 72.56720603049325, + "p95": 86.16573075478665, + "p99": 90.08658842061791 + } + }, + "layer1": { + "decode_tokens": 501569, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9854490932772836, + "mode_counts": { + "FULL": 3186, + "NONE": 237, + "PIECEWISE": 5 + }, + "mode_shares": { + "FULL": 0.9294049008168028, + "NONE": 0.0691365227537923, + "PIECEWISE": 0.0014585764294049008 + }, + "preemptions": 89, + "prefill_tokens": 1616947, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 183, + "queue_waiting_mean": 163.93465577596265, + "records_all": 6649, + "records_clean": 3428, + "requests_per_step": { + "mean": 91.7362893815636, + "n": 3428, + "p50": 92.0, + "p95": 105.0, + "p99": 108.0 + }, + "scheduled_tokens_per_step": { + "mean": 618.0035005834305, + "n": 3428, + "p50": 92.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 139.92259133722288, + "n": 3428, + "p50": 75.726733, + "p95": 1011.79896875, + "p99": 1094.55223217 + }, + "token_efficiency_per_ms": 4.416752825095988 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.65, + "clean_c_waiting_median": 168.0, + "completion_rate_rps": 0.1, + "completion_rate_within_10pct": false, + "tail_end_s": 334.1661366420158, + "tail_start_s": 324.1661366420158, + "valid": false, + "waiting_median": 177.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.65, + "clean_c_waiting_median": 168.0, + "completion_rate_rps": 1.5, + "completion_rate_within_10pct": true, + "tail_end_s": 367.7157571800053, + "tail_start_s": 357.7157571800053, + "valid": true, + "waiting_median": 174.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 564, + "run_dir": "runs/phase3/primary/P06-C00/saturation", + "run_id": "P06-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.03675532053452812, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.0691365227537923, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.0691365227537923, + "padded_tokens_per_useful_token": 0.005328258082544573, + "padding_fraction": 0.03675532053452812 + } + }, + "P06-C01-moderate": { + "blocks": [ + { + "bucket": 2672.0, + "duration_ms": 9983.095475, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 292.0, + "overflow": 15.0, + "padding": 1.0, + "tokens": 33391.0 + }, + { + "bucket": 3080.0, + "duration_ms": 10144.319392000001, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 295.0, + "overflow": 15.0, + "padding": 28.0, + "tokens": 32556.0 + }, + { + "bucket": 3888.0, + "duration_ms": 9834.87609, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 292.0, + "overflow": 8.0, + "padding": 49.0, + "tokens": 19195.0 + }, + { + "bucket": 5000.0, + "duration_ms": 10216.316318000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 356.0, + "overflow": 7.0, + "padding": 88.0, + "tokens": 19248.0 + }, + { + "bucket": 4400.0, + "duration_ms": 10053.599015, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 404.0, + "overflow": 7.0, + "padding": 25.0, + "tokens": 18564.0 + }, + { + "bucket": 3280.0, + "duration_ms": 9645.34066100001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 317.0, + "overflow": 9.0, + "padding": 42.0, + "tokens": 20547.0 + }, + { + "bucket": 3920.0, + "duration_ms": 9973.3789, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 293.0, + "overflow": 12.0, + "padding": 48.0, + "tokens": 28448.0 + }, + { + "bucket": 3368.0, + "duration_ms": 9980.79801399999, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 302.0, + "overflow": 13.0, + "padding": 49.0, + "tokens": 29315.0 + }, + { + "bucket": 3960.0, + "duration_ms": 10001.963437999992, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 325.0, + "overflow": 8.0, + "padding": 13.0, + "tokens": 20331.0 + }, + { + "bucket": 3288.0, + "duration_ms": 9961.200092999981, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 318.0, + "overflow": 11.0, + "padding": 36.0, + "tokens": 25058.0 + }, + { + "bucket": 4608.0, + "duration_ms": 9977.158559999993, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 415.0, + "overflow": 4.0, + "padding": 30.0, + "tokens": 12344.0 + }, + { + "bucket": 3320.0, + "duration_ms": 9407.243992, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 406.0, + "overflow": 10.0, + "padding": 36.0, + "tokens": 22495.0 + }, + { + "bucket": 2992.0, + "duration_ms": 10010.801535999995, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 386.0, + "overflow": 12.0, + "padding": 0.0, + "tokens": 27398.0 + }, + { + "bucket": 3832.0, + "duration_ms": 9827.609744000005, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 336.0, + "overflow": 8.0, + "padding": 68.0, + "tokens": 20148.0 + }, + { + "bucket": 3120.0, + "duration_ms": 9981.477417000004, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 240.0, + "overflow": 15.0, + "padding": 43.0, + "tokens": 32364.0 + }, + { + "bucket": 4688.0, + "duration_ms": 9983.903237999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 333.0, + "overflow": 7.0, + "padding": 74.0, + "tokens": 17800.0 + }, + { + "bucket": 4280.0, + "duration_ms": 9966.367873999996, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 332.0, + "overflow": 9.0, + "padding": 43.0, + "tokens": 22669.0 + }, + { + "bucket": 3632.0, + "duration_ms": 9973.898278999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 349.0, + "overflow": 9.0, + "padding": 34.0, + "tokens": 21274.0 + }, + { + "bucket": 2480.0, + "duration_ms": 10004.851047000002, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 305.0, + "overflow": 15.0, + "padding": 0.0, + "tokens": 30637.0 + }, + { + "bucket": 3112.0, + "duration_ms": 10109.527119000008, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 335.0, + "overflow": 13.0, + "padding": 22.0, + "tokens": 29714.0 + }, + { + "bucket": 3176.0, + "duration_ms": 10042.397934999999, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 238.0, + "overflow": 13.0, + "padding": 58.0, + "tokens": 29549.0 + }, + { + "bucket": 5336.0, + "duration_ms": 9993.305631999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 338.0, + "overflow": 4.0, + "padding": 64.0, + "tokens": 13464.0 + }, + { + "bucket": 3968.0, + "duration_ms": 9765.116452, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 326.0, + "overflow": 8.0, + "padding": 34.0, + "tokens": 20318.0 + }, + { + "bucket": 3712.0, + "duration_ms": 9967.345562000008, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 322.0, + "overflow": 12.0, + "padding": 47.0, + "tokens": 25977.0 + }, + { + "bucket": 4072.0, + "duration_ms": 9988.596370999992, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 378.0, + "overflow": 9.0, + "padding": 52.0, + "tokens": 22452.0 + }, + { + "bucket": 3648.0, + "duration_ms": 9960.308586000008, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 408.0, + "overflow": 9.0, + "padding": 35.0, + "tokens": 19631.0 + }, + { + "bucket": 3016.0, + "duration_ms": 9985.088991000002, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 362.0, + "overflow": 12.0, + "padding": 21.0, + "tokens": 26267.0 + }, + { + "bucket": 3080.0, + "duration_ms": 9984.616373000004, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 350.0, + "overflow": 12.0, + "padding": 3.0, + "tokens": 27653.0 + }, + { + "bucket": 3384.0, + "duration_ms": 10077.784970999994, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 374.0, + "overflow": 11.0, + "padding": 43.0, + "tokens": 25482.0 + }, + { + "bucket": 4112.0, + "duration_ms": 10033.526778999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 412.0, + "overflow": 7.0, + "padding": 35.0, + "tokens": 18413.0 + }, + { + "bucket": 3904.0, + "duration_ms": 9969.954875999993, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 361.0, + "overflow": 9.0, + "padding": 37.0, + "tokens": 22229.0 + }, + { + "bucket": 4752.0, + "duration_ms": 9828.034798999997, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 432.0, + "overflow": 3.0, + "padding": 44.0, + "tokens": 10852.0 + }, + { + "bucket": 2576.0, + "duration_ms": 9986.577423, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 323.0, + "overflow": 15.0, + "padding": 14.0, + "tokens": 32355.0 + }, + { + "bucket": 3816.0, + "duration_ms": 9964.685290999985, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 357.0, + "overflow": 10.0, + "padding": 59.0, + "tokens": 24237.0 + }, + { + "bucket": 4304.0, + "duration_ms": 9969.980953000006, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 466.0, + "overflow": 5.0, + "padding": 61.0, + "tokens": 13096.0 + }, + { + "bucket": 3080.0, + "duration_ms": 9643.314677000006, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 397.0, + "overflow": 10.0, + "padding": 4.0, + "tokens": 22061.0 + }, + { + "bucket": 2856.0, + "duration_ms": 10174.796019999994, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 371.0, + "overflow": 14.0, + "padding": 0.0, + "tokens": 31528.0 + }, + { + "bucket": 3240.0, + "duration_ms": 9782.694661999996, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 386.0, + "overflow": 10.0, + "padding": 24.0, + "tokens": 21500.0 + }, + { + "bucket": 4144.0, + "duration_ms": 10109.441436, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 395.0, + "overflow": 5.0, + "padding": 40.0, + "tokens": 14344.0 + }, + { + "bucket": 3480.0, + "duration_ms": 9843.460232999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 358.0, + "overflow": 7.0, + "padding": 29.0, + "tokens": 16616.0 + }, + { + "bucket": 2760.0, + "duration_ms": 9977.245893999998, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 224.0, + "overflow": 16.0, + "padding": 40.0, + "tokens": 34060.0 + }, + { + "bucket": 3400.0, + "duration_ms": 10009.148130999996, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 262.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 28551.0 + }, + { + "bucket": 4048.0, + "duration_ms": 9959.503941000004, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 280.0, + "overflow": 12.0, + "padding": 73.0, + "tokens": 28551.0 + }, + { + "bucket": 4488.0, + "duration_ms": 9984.051271999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 364.0, + "overflow": 9.0, + "padding": 68.0, + "tokens": 21479.0 + }, + { + "bucket": 2880.0, + "duration_ms": 9985.313251000007, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 346.0, + "overflow": 14.0, + "padding": 20.0, + "tokens": 30861.0 + }, + { + "bucket": 2952.0, + "duration_ms": 10389.374124000002, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 369.0, + "overflow": 14.0, + "padding": 29.0, + "tokens": 30093.0 + }, + { + "bucket": 3488.0, + "duration_ms": 9569.878081999996, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 304.0, + "overflow": 10.0, + "padding": 39.0, + "tokens": 23929.0 + }, + { + "bucket": 5152.0, + "duration_ms": 10210.717404, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 437.0, + "overflow": 4.0, + "padding": 80.0, + "tokens": 13264.0 + } + ], + "clean": { + "admitted": 344, + "completed": 336, + "completed_throughput_rps": 1.4, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 955216, + "offered_rps": 1.4333333333333333, + "output_tokens": 172032, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 4.491966611996759, + "latency_s": { + "e2e": { + "mean": 8.773913019880638, + "n": 336, + "p50": 8.817151172013837, + "p95": 10.46433969050122, + "p99": 10.71811510950938 + }, + "tpot": { + "mean": 0.015270867651661564, + "n": 336, + "p50": 0.015250473021524602, + "p95": 0.018393428898271797, + "p99": 0.019656831172886448 + }, + "ttft": { + "mean": 0.9704996498815793, + "n": 336, + "p50": 0.8652395185054047, + "p95": 2.05128988873912, + "p99": 2.5086708923001417 + } + }, + "layer1": { + "decode_tokens": 173853, + "kv_usage_max": 0.23274897426333463, + "kv_usage_mean": 0.10860699049654005, + "mode_counts": { + "FULL": 16066, + "NONE": 488, + "PIECEWISE": 21 + }, + "mode_shares": { + "FULL": 0.969291101055807, + "NONE": 0.029441930618401207, + "PIECEWISE": 0.0012669683257918551 + }, + "preemptions": 0, + "prefill_tokens": 958455, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 7, + "queue_waiting_mean": 0.0856108597285068, + "records_all": 23686, + "records_clean": 16575, + "requests_per_step": { + "mean": 10.53628959276018, + "n": 16575, + "p50": 8.0, + "p95": 16.0, + "p99": 16.0 + }, + "scheduled_tokens_per_step": { + "mean": 68.31420814479638, + "n": 16575, + "p50": 8.0, + "p95": 16.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 28.850315916923076, + "n": 16575, + "p50": 19.387058, + "p95": 29.4861622, + "p99": 291.6603715399994 + }, + "token_efficiency_per_ms": 2.3678842318923965 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.6, + "completion_rate_within_10pct": true, + "tail_end_s": 341.8995923009934, + "tail_start_s": 331.8995923009934, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.6, + "completion_rate_within_10pct": true, + "tail_end_s": 373.7317875540175, + "tail_start_s": 363.7317875540175, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 336, + "run_dir": "runs/phase3/primary/P06-C01/moderate", + "run_id": "P06-C01-moderate", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.010139748725418792, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.029207651922032465, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 501, + "moe_layer_cv": null, + "overflow_rate": 0.029207651922032465, + "padded_tokens_per_useful_token": 0.001573776746256319, + "padding_fraction": 0.010139748725418792 + } + }, + "P06-C01-saturation": { + "blocks": [ + { + "bucket": 6904.0, + "duration_ms": 9418.292302000007, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 77.0, + "overflow": 13.0, + "padding": 188.0, + "tokens": 31910.0 + }, + { + "bucket": 10648.0, + "duration_ms": 9994.622971000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 109.0, + "overflow": 4.0, + "padding": 398.0, + "tokens": 17851.0 + }, + { + "bucket": 12576.0, + "duration_ms": 10008.333585999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 131.0, + "overflow": 0.0, + "padding": 233.0, + "tokens": 12343.0 + }, + { + "bucket": 7280.0, + "duration_ms": 10245.870207, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 90.0, + "overflow": 14.0, + "padding": 268.0, + "tokens": 34239.0 + }, + { + "duration_ms": 10108.830802, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 9833.619431999998, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 10136.308064999997, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 9738.182761, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "bucket": 4160.0, + "duration_ms": 9903.814556, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 59.0, + "overflow": 19.0, + "padding": 72.0, + "tokens": 42229.0 + }, + { + "bucket": 12896.0, + "duration_ms": 10019.929632999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 124.0, + "overflow": 0.0, + "padding": 551.0, + "tokens": 12345.0 + }, + { + "bucket": 12216.0, + "duration_ms": 9991.564728000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 127.0, + "overflow": 0.0, + "padding": 183.0, + "tokens": 12033.0 + }, + { + "bucket": 7760.0, + "duration_ms": 10099.811674000006, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 89.0, + "overflow": 12.0, + "padding": 419.0, + "tokens": 30410.0 + }, + { + "duration_ms": 9986.679154000001, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 9952.167458000002, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 10194.748773000003, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 9892.233782000003, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "bucket": 5152.0, + "duration_ms": 10279.543559000002, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 66.0, + "overflow": 20.0, + "padding": 277.0, + "tokens": 44672.0 + }, + { + "bucket": 12168.0, + "duration_ms": 9531.385584999995, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 117.0, + "overflow": 1.0, + "padding": 392.0, + "tokens": 13685.0 + }, + { + "bucket": 12376.0, + "duration_ms": 9952.329975, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 120.0, + "overflow": 2.0, + "padding": 636.0, + "tokens": 15205.0 + }, + { + "bucket": 9600.0, + "duration_ms": 10234.776673999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 109.0, + "overflow": 9.0, + "padding": 216.0, + "tokens": 26935.0 + }, + { + "duration_ms": 10098.437773000001, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 10149.765926, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 9781.945214, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 10055.906094999998, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "bucket": 4224.0, + "duration_ms": 9670.510049999997, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 62.0, + "overflow": 18.0, + "padding": 153.0, + "tokens": 39980.0 + }, + { + "bucket": 12288.0, + "duration_ms": 9951.205562000006, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 128.0, + "overflow": 0.0, + "padding": 658.0, + "tokens": 11630.0 + }, + { + "bucket": 12040.0, + "duration_ms": 9984.624423999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 130.0, + "overflow": 0.0, + "padding": 571.0, + "tokens": 11469.0 + }, + { + "bucket": 6160.0, + "duration_ms": 10267.234748, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 85.0, + "overflow": 15.0, + "padding": 104.0, + "tokens": 35685.0 + }, + { + "duration_ms": 10186.587034000002, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 9887.492626, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 9782.122324, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 10100.028454, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "bucket": 5896.0, + "duration_ms": 9748.447406000001, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 71.0, + "overflow": 15.0, + "padding": 225.0, + "tokens": 36391.0 + }, + { + "bucket": 12272.0, + "duration_ms": 10042.153232000008, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 120.0, + "overflow": 2.0, + "padding": 656.0, + "tokens": 14848.0 + }, + { + "bucket": 10760.0, + "duration_ms": 9964.378218999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 111.0, + "overflow": 4.0, + "padding": 439.0, + "tokens": 17856.0 + }, + { + "bucket": 9120.0, + "duration_ms": 10090.992645999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 104.0, + "overflow": 9.0, + "padding": 167.0, + "tokens": 25996.0 + }, + { + "duration_ms": 10061.783895, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 60197.0 + }, + { + "duration_ms": 10226.395404999997, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 9836.990006, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 10073.936996, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "bucket": 3608.0, + "duration_ms": 9672.199914999997, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 61.0, + "overflow": 20.0, + "padding": 0.0, + "tokens": 43169.0 + }, + { + "bucket": 11088.0, + "duration_ms": 10005.363535, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 128.0, + "overflow": 2.0, + "padding": 191.0, + "tokens": 14398.0 + }, + { + "bucket": 11528.0, + "duration_ms": 9979.159297999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 131.0, + "overflow": 0.0, + "padding": 414.0, + "tokens": 11114.0 + }, + { + "bucket": 6488.0, + "duration_ms": 10210.046228000001, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 88.0, + "overflow": 14.0, + "padding": 415.0, + "tokens": 33466.0 + }, + { + "duration_ms": 10130.624972, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 9761.507399999999, + "eligible_miss": 0.0, + "miss": 29.0, + "model": 29.0, + "overflow": 29.0, + "tokens": 59392.0 + }, + { + "duration_ms": 10248.139619, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 30.0, + "overflow": 30.0, + "tokens": 61440.0 + }, + { + "duration_ms": 9801.388455, + "eligible_miss": 0.0, + "miss": 28.0, + "model": 28.0, + "overflow": 28.0, + "tokens": 57344.0 + } + ], + "clean": { + "admitted": 566, + "completed": 566, + "completed_throughput_rps": 2.3583333333333334, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1637559, + "offered_rps": 2.3583333333333334, + "output_tokens": 289792, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 98.6341013039928, + "latency_s": { + "e2e": { + "mean": 106.76549268240902, + "n": 566, + "p50": 112.07339421949291, + "p95": 130.40440611923987, + "p99": 135.57815159696622 + }, + "tpot": { + "mean": 0.07499950729258756, + "n": 566, + "p50": 0.07366898813504892, + "p95": 0.08367265785228196, + "p99": 0.10166048986216235 + }, + "ttft": { + "mean": 68.44074445589678, + "n": 566, + "p50": 74.25358793899068, + "p95": 91.44885190949572, + "p99": 96.69285478206731 + } + }, + "layer1": { + "decode_tokens": 413751, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9691897666726189, + "mode_counts": { + "FULL": 2240, + "NONE": 900, + "PIECEWISE": 4 + }, + "mode_shares": { + "FULL": 0.712468193384224, + "NONE": 0.2862595419847328, + "PIECEWISE": 0.001272264631043257 + }, + "preemptions": 46, + "prefill_tokens": 1622801, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 181, + "queue_waiting_mean": 163.7395038167939, + "records_all": 6363, + "records_clean": 3144, + "requests_per_step": { + "mean": 91.90044529262086, + "n": 3144, + "p50": 92.0, + "p95": 102.0, + "p99": 106.0 + }, + "scheduled_tokens_per_step": { + "mean": 647.7582697201018, + "n": 3144, + "p50": 98.0, + "p95": 2048.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 152.44669628944018, + "n": 3144, + "p50": 79.9125085, + "p95": 365.08051875, + "p99": 395.60820568 + }, + "token_efficiency_per_ms": 4.2490804030954346 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 2.25, + "clean_c_waiting_median": 169.0, + "completion_rate_rps": 4.3, + "completion_rate_within_10pct": false, + "tail_end_s": 349.72332530099084, + "tail_start_s": 339.72332530099084, + "valid": false, + "waiting_median": 176.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 2.25, + "clean_c_waiting_median": 169.0, + "completion_rate_rps": 4.0, + "completion_rate_within_10pct": false, + "tail_end_s": 400.2956078030111, + "tail_start_s": 390.2956078030111, + "valid": false, + "waiting_median": 180.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 566, + "run_dir": "runs/phase3/primary/P06-C01/saturation", + "run_id": "P06-C01-saturation", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.03570125177913215, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.2862595419847328, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.2862595419847328, + "padded_tokens_per_useful_token": 0.003842769543817197, + "padding_fraction": 0.03570125177913215 + } + }, + "P06-C10-moderate": { + "blocks": [ + { + "bucket": 2984.0, + "duration_ms": 9968.688442999994, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 293.0, + "overflow": 5.0, + "padding": 9.0, + "tokens": 33676.0 + }, + { + "bucket": 3072.0, + "duration_ms": 9999.67704899999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 241.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 32895.0 + }, + { + "bucket": 4456.0, + "duration_ms": 9985.151113999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 329.0, + "overflow": 2.0, + "padding": 15.0, + "tokens": 19737.0 + }, + { + "bucket": 3832.0, + "duration_ms": 9958.370330000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 324.0, + "overflow": 3.0, + "padding": 23.0, + "tokens": 26537.0 + }, + { + "bucket": 3440.0, + "duration_ms": 9982.292306000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 340.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 27035.0 + }, + { + "bucket": 3160.0, + "duration_ms": 9981.709315999999, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 325.0, + "overflow": 3.0, + "padding": 7.0, + "tokens": 28132.0 + }, + { + "bucket": 3640.0, + "duration_ms": 9979.026379999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 354.0, + "overflow": 4.0, + "padding": 7.0, + "tokens": 29531.0 + }, + { + "bucket": 3632.0, + "duration_ms": 9998.650812999997, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 339.0, + "overflow": 2.0, + "padding": 14.0, + "tokens": 20209.0 + }, + { + "bucket": 4152.0, + "duration_ms": 10839.769605000001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 362.0, + "overflow": 3.0, + "padding": 10.0, + "tokens": 25857.0 + }, + { + "bucket": 4168.0, + "duration_ms": 9557.040489000003, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 380.0, + "overflow": 1.0, + "padding": 10.0, + "tokens": 11897.0 + }, + { + "bucket": 5128.0, + "duration_ms": 9451.50596200001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 510.0, + "overflow": 0.0, + "padding": 6.0, + "tokens": 5122.0 + }, + { + "bucket": 2296.0, + "duration_ms": 9415.719377999993, + "eligible_miss": 2.0, + "miss": 6.0, + "model": 293.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 35631.0 + }, + { + "bucket": 3376.0, + "duration_ms": 9974.437237000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 291.0, + "overflow": 4.0, + "padding": 20.0, + "tokens": 30094.0 + }, + { + "bucket": 3200.0, + "duration_ms": 9996.377144000007, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 269.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 32361.0 + }, + { + "bucket": 4448.0, + "duration_ms": 9968.941337, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 337.0, + "overflow": 2.0, + "padding": 11.0, + "tokens": 17928.0 + }, + { + "bucket": 3752.0, + "duration_ms": 9978.026393, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 322.0, + "overflow": 3.0, + "padding": 17.0, + "tokens": 22903.0 + }, + { + "bucket": 3784.0, + "duration_ms": 9992.190155999993, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 376.0, + "overflow": 3.0, + "padding": 6.0, + "tokens": 21390.0 + }, + { + "bucket": 2712.0, + "duration_ms": 10144.682894000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 321.0, + "overflow": 5.0, + "padding": 11.0, + "tokens": 30739.0 + }, + { + "bucket": 3512.0, + "duration_ms": 10679.815315, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 345.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 36274.0 + }, + { + "bucket": 3216.0, + "duration_ms": 9468.849742000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 241.0, + "overflow": 4.0, + "padding": 15.0, + "tokens": 23435.0 + }, + { + "bucket": 4872.0, + "duration_ms": 10371.413924999999, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 340.0, + "overflow": 2.0, + "padding": 15.0, + "tokens": 21241.0 + }, + { + "bucket": 4016.0, + "duration_ms": 9233.718109, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 336.0, + "overflow": 1.0, + "padding": 7.0, + "tokens": 12652.0 + }, + { + "bucket": 4000.0, + "duration_ms": 9964.93926299999, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 345.0, + "overflow": 3.0, + "padding": 13.0, + "tokens": 26186.0 + }, + { + "bucket": 3992.0, + "duration_ms": 9983.514276999982, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 390.0, + "overflow": 3.0, + "padding": 9.0, + "tokens": 22490.0 + }, + { + "bucket": 3824.0, + "duration_ms": 9967.431117999991, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 414.0, + "overflow": 3.0, + "padding": 8.0, + "tokens": 19770.0 + }, + { + "bucket": 3104.0, + "duration_ms": 9987.400416000002, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 339.0, + "overflow": 3.0, + "padding": 5.0, + "tokens": 26275.0 + }, + { + "bucket": 3408.0, + "duration_ms": 9969.86757, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 317.0, + "overflow": 3.0, + "padding": 2.0, + "tokens": 27982.0 + }, + { + "bucket": 3800.0, + "duration_ms": 9983.764680000006, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 364.0, + "overflow": 3.0, + "padding": 11.0, + "tokens": 25846.0 + }, + { + "bucket": 3168.0, + "duration_ms": 9992.213853, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 344.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 31647.0 + }, + { + "bucket": 4288.0, + "duration_ms": 9963.092042, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 396.0, + "overflow": 2.0, + "padding": 6.0, + "tokens": 14924.0 + }, + { + "bucket": 4080.0, + "duration_ms": 10617.837592, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 412.0, + "overflow": 3.0, + "padding": 6.0, + "tokens": 28650.0 + }, + { + "bucket": 3264.0, + "duration_ms": 9873.84534800001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 376.0, + "overflow": 3.0, + "padding": 2.0, + "tokens": 24735.0 + }, + { + "bucket": 4440.0, + "duration_ms": 9427.799631999995, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 421.0, + "overflow": 1.0, + "padding": 12.0, + "tokens": 8453.0 + }, + { + "bucket": 4592.0, + "duration_ms": 9966.856386999998, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 492.0, + "overflow": 2.0, + "padding": 13.0, + "tokens": 13393.0 + }, + { + "bucket": 3068.0, + "duration_ms": 9315.430008000005, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 387.0, + "overflow": 3.0, + "padding": 1.0, + "tokens": 22030.0 + }, + { + "bucket": 2544.0, + "duration_ms": 9980.105680000008, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 285.0, + "overflow": 5.0, + "padding": 9.0, + "tokens": 36537.0 + }, + { + "bucket": 4280.0, + "duration_ms": 9989.648308, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 357.0, + "overflow": 2.0, + "padding": 12.0, + "tokens": 17052.0 + }, + { + "bucket": 3496.0, + "duration_ms": 9983.011968999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 340.0, + "overflow": 3.0, + "padding": 8.0, + "tokens": 26788.0 + }, + { + "bucket": 2536.0, + "duration_ms": 9973.913760000001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 283.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 33733.0 + }, + { + "bucket": 3384.0, + "duration_ms": 9981.835801000007, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 292.0, + "overflow": 3.0, + "padding": 7.0, + "tokens": 28419.0 + }, + { + "bucket": 3776.0, + "duration_ms": 10016.567731000003, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 294.0, + "overflow": 3.0, + "padding": 12.0, + "tokens": 28608.0 + }, + { + "bucket": 4504.0, + "duration_ms": 9950.887116000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 380.0, + "overflow": 3.0, + "padding": 8.0, + "tokens": 21490.0 + }, + { + "bucket": 4784.0, + "duration_ms": 11398.240730999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 471.0, + "overflow": 3.0, + "padding": 13.0, + "tokens": 29347.0 + }, + { + "bucket": 3400.0, + "duration_ms": 9854.494030000007, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 391.0, + "overflow": 3.0, + "padding": 12.0, + "tokens": 23089.0 + }, + { + "bucket": 2960.0, + "duration_ms": 8671.937073, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 255.0, + "overflow": 2.0, + "padding": 11.0, + "tokens": 19835.0 + }, + { + "bucket": 4712.0, + "duration_ms": 9964.669994000005, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 377.0, + "overflow": 2.0, + "padding": 20.0, + "tokens": 19049.0 + }, + { + "bucket": 3576.0, + "duration_ms": 9977.808734999995, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 374.0, + "overflow": 3.0, + "padding": 2.0, + "tokens": 25329.0 + }, + { + "bucket": 2728.0, + "duration_ms": 9983.505524999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 263.0, + "overflow": 5.0, + "padding": 7.0, + "tokens": 37252.0 + } + ], + "clean": { + "admitted": 352, + "completed": 352, + "completed_throughput_rps": 1.4666666666666666, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 991279, + "offered_rps": 1.4666666666666666, + "output_tokens": 180224, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 4.380946171004325, + "latency_s": { + "e2e": { + "mean": 8.521981804951297, + "n": 352, + "p50": 8.611102025490254, + "p95": 10.042639616953966, + "p99": 10.376431019741284 + }, + "tpot": { + "mean": 0.014745414302892397, + "n": 352, + "p50": 0.01479352528180147, + "p95": 0.01784971316144328, + "p99": 0.018753809229155375 + }, + "ttft": { + "mean": 0.9870750961732814, + "n": 352, + "p50": 1.0108699320117012, + "p95": 1.8988438446089275, + "p99": 2.218485610165226 + } + }, + "layer1": { + "decode_tokens": 177688, + "kv_usage_max": 0.21875312219002896, + "kv_usage_mean": 0.1036769300750604, + "mode_counts": { + "FULL": 16470, + "NONE": 160, + "PIECEWISE": 1 + }, + "mode_shares": { + "FULL": 0.9903192832661897, + "NONE": 0.009620588058445073, + "PIECEWISE": 6.01286753652817e-05 + }, + "preemptions": 0, + "prefill_tokens": 1006497, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 6, + "queue_waiting_mean": 0.014911911490589862, + "records_all": 24869, + "records_clean": 16631, + "requests_per_step": { + "mean": 10.711201972220552, + "n": 16631, + "p50": 8.0, + "p95": 16.0, + "p99": 16.0 + }, + "scheduled_tokens_per_step": { + "mean": 71.20347543743611, + "n": 16631, + "p50": 8.0, + "p95": 16.0, + "p99": 16.0 + }, + "step_duration_ms": { + "mean": 28.721464258072277, + "n": 16631, + "p50": 19.530679, + "p95": 28.7628175, + "p99": 301.7449877000092 + }, + "token_efficiency_per_ms": 2.479103251757929 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.6, + "completion_rate_within_10pct": true, + "tail_end_s": 331.32366371300304, + "tail_start_s": 321.32366371300304, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.6, + "completion_rate_within_10pct": true, + "tail_end_s": 362.93273995199706, + "tail_start_s": 352.93273995199706, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 352, + "run_dir": "runs/phase3/primary/P06-C10/moderate", + "run_id": "P06-C10-moderate", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.00284059447606813, + "eligible_miss_rate": 0.0006615745474228664, + "graph_miss_rate": 0.009382329945269743, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 155, + "moe_layer_cv": null, + "overflow_rate": 0.008720755397846875, + "padded_tokens_per_useful_token": 0.0003631189383415598, + "padding_fraction": 0.0024354878905276513 + } + }, + "P06-C10-saturation": { + "blocks": [ + { + "duration_ms": 10184.668676000001, + "eligible_miss": 2.0, + "miss": 15.0, + "model": 15.0, + "overflow": 13.0, + "tokens": 75610.0 + }, + { + "bucket": 3712.0, + "duration_ms": 8871.553595999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 63.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 43504.0 + }, + { + "bucket": 10752.0, + "duration_ms": 9944.392154999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 168.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10752.0 + }, + { + "bucket": 10496.0, + "duration_ms": 9989.546154, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 164.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10496.0 + }, + { + "bucket": 5888.0, + "duration_ms": 11067.204580000001, + "eligible_miss": 3.0, + "miss": 10.0, + "model": 102.0, + "overflow": 7.0, + "padding": 1.0, + "tokens": 48983.0 + }, + { + "duration_ms": 9496.656133000002, + "eligible_miss": 1.0, + "miss": 10.0, + "model": 10.0, + "overflow": 9.0, + "tokens": 73458.0 + }, + { + "bucket": 896.0, + "duration_ms": 9422.229260000002, + "eligible_miss": 1.0, + "miss": 11.0, + "model": 25.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 64392.0 + }, + { + "bucket": 11200.0, + "duration_ms": 10010.468302999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 175.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11200.0 + }, + { + "bucket": 10944.0, + "duration_ms": 9975.989317999994, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 171.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10944.0 + }, + { + "bucket": 7808.0, + "duration_ms": 10552.635057000001, + "eligible_miss": 2.0, + "miss": 7.0, + "model": 129.0, + "overflow": 5.0, + "padding": 1.0, + "tokens": 34660.0 + }, + { + "duration_ms": 9884.066383000001, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 75141.0 + }, + { + "bucket": 64.0, + "duration_ms": 9923.407183999998, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 14.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 72823.0 + }, + { + "bucket": 11008.0, + "duration_ms": 9593.384500999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 172.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11008.0 + }, + { + "bucket": 11072.0, + "duration_ms": 9989.240422999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 173.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11072.0 + }, + { + "bucket": 8704.0, + "duration_ms": 10628.489381000001, + "eligible_miss": 2.0, + "miss": 6.0, + "model": 142.0, + "overflow": 4.0, + "padding": 4.0, + "tokens": 30665.0 + }, + { + "duration_ms": 9825.908395999999, + "eligible_miss": 1.0, + "miss": 13.0, + "model": 13.0, + "overflow": 12.0, + "tokens": 75749.0 + }, + { + "duration_ms": 10510.417456000001, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 13.0, + "overflow": 13.0, + "tokens": 78441.0 + }, + { + "bucket": 10112.0, + "duration_ms": 9049.764061999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 158.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10112.0 + }, + { + "bucket": 11136.0, + "duration_ms": 9938.462914999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 174.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11136.0 + }, + { + "bucket": 9472.0, + "duration_ms": 10972.880893000001, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 152.0, + "overflow": 3.0, + "padding": 1.0, + "tokens": 31022.0 + }, + { + "duration_ms": 10338.176429, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 13.0, + "overflow": 13.0, + "tokens": 79069.0 + }, + { + "duration_ms": 9218.303395, + "eligible_miss": 1.0, + "miss": 11.0, + "model": 11.0, + "overflow": 10.0, + "tokens": 66925.0 + }, + { + "bucket": 6272.0, + "duration_ms": 9481.264913000005, + "eligible_miss": 1.0, + "miss": 5.0, + "model": 103.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 35279.0 + }, + { + "bucket": 11392.0, + "duration_ms": 9944.188837000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 178.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11392.0 + }, + { + "bucket": 11136.0, + "duration_ms": 10002.603861000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 174.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11136.0 + }, + { + "bucket": 1920.0, + "duration_ms": 10572.825789999999, + "eligible_miss": 2.0, + "miss": 12.0, + "model": 42.0, + "overflow": 10.0, + "padding": 1.0, + "tokens": 69902.0 + }, + { + "duration_ms": 10003.447610000001, + "eligible_miss": 1.0, + "miss": 13.0, + "model": 13.0, + "overflow": 12.0, + "tokens": 69136.0 + }, + { + "bucket": 1984.0, + "duration_ms": 9449.084334, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 40.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 60721.0 + }, + { + "bucket": 10880.0, + "duration_ms": 9949.095603, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 170.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10880.0 + }, + { + "bucket": 10816.0, + "duration_ms": 10024.887996000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 169.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10816.0 + }, + { + "bucket": 6976.0, + "duration_ms": 10451.003526999995, + "eligible_miss": 3.0, + "miss": 10.0, + "model": 119.0, + "overflow": 7.0, + "padding": 1.0, + "tokens": 35730.0 + }, + { + "duration_ms": 10420.409311, + "eligible_miss": 2.0, + "miss": 14.0, + "model": 14.0, + "overflow": 12.0, + "tokens": 81494.0 + }, + { + "bucket": 64.0, + "duration_ms": 9135.379603, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 11.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 64970.0 + }, + { + "bucket": 11520.0, + "duration_ms": 9928.713835, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 180.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11520.0 + }, + { + "bucket": 11328.0, + "duration_ms": 9972.419044999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 177.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11328.0 + }, + { + "bucket": 7744.0, + "duration_ms": 10875.907865000003, + "eligible_miss": 1.0, + "miss": 7.0, + "model": 128.0, + "overflow": 6.0, + "padding": 1.0, + "tokens": 40810.0 + }, + { + "duration_ms": 9818.483750000001, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 74921.0 + }, + { + "duration_ms": 10055.352617999999, + "eligible_miss": 2.0, + "miss": 13.0, + "model": 13.0, + "overflow": 11.0, + "tokens": 76325.0 + }, + { + "bucket": 6080.0, + "duration_ms": 9237.526175, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 98.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 29721.0 + }, + { + "bucket": 10688.0, + "duration_ms": 9993.652559, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 167.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10688.0 + }, + { + "bucket": 10496.0, + "duration_ms": 10020.391325999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 164.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10496.0 + }, + { + "bucket": 3328.0, + "duration_ms": 10926.115058, + "eligible_miss": 1.0, + "miss": 11.0, + "model": 63.0, + "overflow": 10.0, + "padding": 1.0, + "tokens": 62223.0 + }, + { + "duration_ms": 9090.596137999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 67808.0 + }, + { + "bucket": 1664.0, + "duration_ms": 9942.095380999996, + "eligible_miss": 4.0, + "miss": 14.0, + "model": 40.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 63097.0 + }, + { + "bucket": 10816.0, + "duration_ms": 9981.973696000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 169.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10816.0 + }, + { + "bucket": 10560.0, + "duration_ms": 9974.256657000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 165.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10560.0 + }, + { + "bucket": 7552.0, + "duration_ms": 10682.932023000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 123.0, + "overflow": 5.0, + "padding": 1.0, + "tokens": 35997.0 + }, + { + "duration_ms": 9975.763851000002, + "eligible_miss": 3.0, + "miss": 13.0, + "model": 13.0, + "overflow": 10.0, + "tokens": 73687.0 + } + ], + "clean": { + "admitted": 583, + "completed": 583, + "completed_throughput_rps": 2.4291666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1706195, + "offered_rps": 2.4291666666666667, + "output_tokens": 298496, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 95.53387634200044, + "latency_s": { + "e2e": { + "mean": 101.3565315140522, + "n": 583, + "p50": 104.58615519001614, + "p95": 114.11084459880075, + "p99": 118.06476088725671 + }, + "tpot": { + "mean": 0.04984293034401163, + "n": 583, + "p50": 0.049699737217209855, + "p95": 0.0539970578847181, + "p99": 0.05502665809392811 + }, + "ttft": { + "mean": 75.88679410826225, + "n": 583, + "p50": 79.60525833698921, + "p95": 89.27045010388537, + "p99": 93.5858921495825 + } + }, + "layer1": { + "decode_tokens": 294516, + "kv_usage_max": 0.7428314516934759, + "kv_usage_mean": 0.6299692841321703, + "mode_counts": { + "FULL": 4320, + "NONE": 304 + }, + "mode_shares": { + "FULL": 0.9342560553633218, + "NONE": 0.0657439446366782 + }, + "preemptions": 0, + "prefill_tokens": 1694099, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 197, + "queue_waiting_mean": 191.90376297577853, + "records_all": 8763, + "records_clean": 4624, + "requests_per_step": { + "mean": 63.84407439446367, + "n": 4624, + "p50": 64.0, + "p95": 64.0, + "p99": 64.0 + }, + "scheduled_tokens_per_step": { + "mean": 430.06379757785464, + "n": 4624, + "p50": 64.0, + "p95": 2907.5999999999585, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 103.6544584801038, + "n": 4624, + "p50": 58.3942475, + "p95": 540.7021005499993, + "p99": 1052.19412524 + }, + "token_efficiency_per_ms": 4.149013982455991 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.7, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 3.7, + "completion_rate_within_10pct": false, + "tail_end_s": 355.943668923981, + "tail_start_s": 345.943668923981, + "valid": false, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.7, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 2.5, + "completion_rate_within_10pct": false, + "tail_end_s": 389.65710150499945, + "tail_start_s": 379.65710150499945, + "valid": false, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 583, + "run_dir": "runs/phase3/primary/P06-C10/saturation", + "run_id": "P06-C10-saturation", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.0009259002385840316, + "eligible_miss_rate": 0.007352941176470588, + "graph_miss_rate": 0.0657439446366782, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.05839100346020761, + "padded_tokens_per_useful_token": 6.03435054045152e-06, + "padding_fraction": 4.340277777777778e-05 + } + }, + "P06-C11-moderate": { + "blocks": [ + { + "bucket": 2784.0, + "duration_ms": 9991.795590999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 289.0, + "overflow": 15.0, + "padding": 1.0, + "tokens": 33503.0 + }, + { + "bucket": 2496.0, + "duration_ms": 9984.000759000004, + "eligible_miss": 1.0, + "miss": 16.0, + "model": 285.0, + "overflow": 15.0, + "padding": 34.0, + "tokens": 32430.0 + }, + { + "bucket": 4056.0, + "duration_ms": 9979.876596999997, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 300.0, + "overflow": 8.0, + "padding": 49.0, + "tokens": 19363.0 + }, + { + "bucket": 4240.0, + "duration_ms": 10347.565718000002, + "eligible_miss": 1.0, + "miss": 10.0, + "model": 338.0, + "overflow": 9.0, + "padding": 78.0, + "tokens": 23048.0 + }, + { + "bucket": 4024.0, + "duration_ms": 9814.699700000014, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 390.0, + "overflow": 7.0, + "padding": 14.0, + "tokens": 18199.0 + }, + { + "bucket": 3608.0, + "duration_ms": 9772.572115999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 342.0, + "overflow": 7.0, + "padding": 42.0, + "tokens": 17030.0 + }, + { + "bucket": 3504.0, + "duration_ms": 9976.144115999994, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 301.0, + "overflow": 13.0, + "padding": 44.0, + "tokens": 28542.0 + }, + { + "bucket": 3320.0, + "duration_ms": 9971.772360999998, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 295.0, + "overflow": 13.0, + "padding": 45.0, + "tokens": 29271.0 + }, + { + "bucket": 3776.0, + "duration_ms": 10005.948374999994, + "eligible_miss": 1.0, + "miss": 9.0, + "model": 313.0, + "overflow": 8.0, + "padding": 7.0, + "tokens": 20428.0 + }, + { + "bucket": 3296.0, + "duration_ms": 9975.181751, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 312.0, + "overflow": 11.0, + "padding": 29.0, + "tokens": 25073.0 + }, + { + "bucket": 4680.0, + "duration_ms": 9966.712054000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 417.0, + "overflow": 4.0, + "padding": 63.0, + "tokens": 12383.0 + }, + { + "bucket": 2872.0, + "duration_ms": 9521.160925, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 372.0, + "overflow": 13.0, + "padding": 3.0, + "tokens": 28224.0 + }, + { + "bucket": 2968.0, + "duration_ms": 10163.028479999995, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 384.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 29422.0 + }, + { + "bucket": 3560.0, + "duration_ms": 9863.206437, + "eligible_miss": 1.0, + "miss": 10.0, + "model": 329.0, + "overflow": 9.0, + "padding": 62.0, + "tokens": 20722.0 + }, + { + "bucket": 3504.0, + "duration_ms": 9738.622831, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 271.0, + "overflow": 10.0, + "padding": 43.0, + "tokens": 23939.0 + }, + { + "bucket": 4400.0, + "duration_ms": 9974.671073000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 335.0, + "overflow": 7.0, + "padding": 60.0, + "tokens": 17889.0 + }, + { + "bucket": 3480.0, + "duration_ms": 9981.279874999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 336.0, + "overflow": 10.0, + "padding": 24.0, + "tokens": 22694.0 + }, + { + "bucket": 3656.0, + "duration_ms": 9969.817867000003, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 353.0, + "overflow": 9.0, + "padding": 33.0, + "tokens": 21299.0 + }, + { + "bucket": 2688.0, + "duration_ms": 9981.607506999997, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 290.0, + "overflow": 15.0, + "padding": 0.0, + "tokens": 30845.0 + }, + { + "bucket": 2456.0, + "duration_ms": 10210.062520999998, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 299.0, + "overflow": 17.0, + "padding": 22.0, + "tokens": 37250.0 + }, + { + "bucket": 3600.0, + "duration_ms": 9763.962257999996, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 269.0, + "overflow": 9.0, + "padding": 58.0, + "tokens": 21899.0 + }, + { + "bucket": 4344.0, + "duration_ms": 10347.414697999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 309.0, + "overflow": 9.0, + "padding": 55.0, + "tokens": 22721.0 + }, + { + "bucket": 4088.0, + "duration_ms": 10040.395758999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 356.0, + "overflow": 9.0, + "padding": 38.0, + "tokens": 19686.0 + }, + { + "bucket": 4048.0, + "duration_ms": 9530.296545999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 361.0, + "overflow": 7.0, + "padding": 43.0, + "tokens": 17378.0 + }, + { + "bucket": 3760.0, + "duration_ms": 9977.245639000013, + "eligible_miss": 1.0, + "miss": 10.0, + "model": 363.0, + "overflow": 9.0, + "padding": 50.0, + "tokens": 22300.0 + }, + { + "bucket": 3784.0, + "duration_ms": 9964.184954999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 397.0, + "overflow": 8.0, + "padding": 35.0, + "tokens": 19756.0 + }, + { + "bucket": 2960.0, + "duration_ms": 9980.599558000002, + "eligible_miss": 2.0, + "miss": 13.0, + "model": 347.0, + "overflow": 11.0, + "padding": 18.0, + "tokens": 26224.0 + }, + { + "bucket": 3136.0, + "duration_ms": 9993.791949, + "eligible_miss": 1.0, + "miss": 13.0, + "model": 322.0, + "overflow": 12.0, + "padding": 0.0, + "tokens": 27865.0 + }, + { + "bucket": 3424.0, + "duration_ms": 9958.501031, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 361.0, + "overflow": 11.0, + "padding": 45.0, + "tokens": 25520.0 + }, + { + "bucket": 3312.0, + "duration_ms": 10133.055848, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 359.0, + "overflow": 12.0, + "padding": 31.0, + "tokens": 27857.0 + }, + { + "bucket": 3904.0, + "duration_ms": 9830.097450000003, + "eligible_miss": 1.0, + "miss": 8.0, + "model": 373.0, + "overflow": 7.0, + "padding": 33.0, + "tokens": 18494.0 + }, + { + "bucket": 4840.0, + "duration_ms": 10346.134011999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 469.0, + "overflow": 4.0, + "padding": 37.0, + "tokens": 12995.0 + }, + { + "bucket": 2808.0, + "duration_ms": 9589.825538000006, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 346.0, + "overflow": 11.0, + "padding": 14.0, + "tokens": 24395.0 + }, + { + "bucket": 3904.0, + "duration_ms": 9980.938999999978, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 366.0, + "overflow": 10.0, + "padding": 59.0, + "tokens": 24325.0 + }, + { + "bucket": 4296.0, + "duration_ms": 9970.991288999987, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 464.0, + "overflow": 5.0, + "padding": 61.0, + "tokens": 13088.0 + }, + { + "bucket": 3104.0, + "duration_ms": 9654.592499000011, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 400.0, + "overflow": 10.0, + "padding": 4.0, + "tokens": 22085.0 + }, + { + "bucket": 2320.0, + "duration_ms": 9985.013716000003, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 301.0, + "overflow": 17.0, + "padding": 30.0, + "tokens": 36415.0 + }, + { + "bucket": 3800.0, + "duration_ms": 9975.918895999994, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 397.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 16638.0 + }, + { + "bucket": 3248.0, + "duration_ms": 10256.340521999995, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 327.0, + "overflow": 12.0, + "padding": 40.0, + "tokens": 26613.0 + }, + { + "bucket": 3624.0, + "duration_ms": 9876.904798999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 361.0, + "overflow": 7.0, + "padding": 30.0, + "tokens": 17930.0 + }, + { + "bucket": 3320.0, + "duration_ms": 10086.872195000005, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 286.0, + "overflow": 12.0, + "padding": 40.0, + "tokens": 26428.0 + }, + { + "bucket": 3816.0, + "duration_ms": 9703.728442000009, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 263.0, + "overflow": 10.0, + "padding": 59.0, + "tokens": 22764.0 + }, + { + "bucket": 3784.0, + "duration_ms": 9980.674858999995, + "eligible_miss": 1.0, + "miss": 13.0, + "model": 288.0, + "overflow": 12.0, + "padding": 45.0, + "tokens": 28685.0 + }, + { + "bucket": 4288.0, + "duration_ms": 9975.193013000007, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 389.0, + "overflow": 9.0, + "padding": 43.0, + "tokens": 21304.0 + }, + { + "bucket": 2888.0, + "duration_ms": 9985.337101999998, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 336.0, + "overflow": 14.0, + "padding": 29.0, + "tokens": 30860.0 + }, + { + "bucket": 2368.0, + "duration_ms": 9982.161468000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 284.0, + "overflow": 17.0, + "padding": 0.0, + "tokens": 35682.0 + }, + { + "bucket": 4384.0, + "duration_ms": 9980.502426000003, + "eligible_miss": 1.0, + "miss": 8.0, + "model": 366.0, + "overflow": 7.0, + "padding": 39.0, + "tokens": 18763.0 + }, + { + "bucket": 3800.0, + "duration_ms": 10233.927156, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 356.0, + "overflow": 11.0, + "padding": 80.0, + "tokens": 25562.0 + } + ], + "clean": { + "admitted": 344, + "completed": 336, + "completed_throughput_rps": 1.4, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 955216, + "offered_rps": 1.4333333333333333, + "output_tokens": 172032, + "start_s": 60.0 + }, + "config": "C11", + "drain_quarantined": false, + "drain_seconds": 3.233019929000875, + "latency_s": { + "e2e": { + "mean": 8.811718997470356, + "n": 336, + "p50": 8.860830737001379, + "p95": 10.438232306754799, + "p99": 10.661098391610722 + }, + "tpot": { + "mean": 0.015341037308058707, + "n": 336, + "p50": 0.015262965586132356, + "p95": 0.018323992875227187, + "p99": 0.019567817679344728 + }, + "ttft": { + "mean": 0.9724489330523544, + "n": 336, + "p50": 0.8726355374965351, + "p95": 2.050784301252861, + "p99": 2.5037182233965716 + } + }, + "layer1": { + "decode_tokens": 173763, + "kv_usage_max": 0.21429622724819708, + "kv_usage_mean": 0.1012056417269781, + "mode_counts": { + "FULL": 15859, + "NONE": 507, + "PIECEWISE": 5 + }, + "mode_shares": { + "FULL": 0.968725184777961, + "NONE": 0.030969397104636247, + "PIECEWISE": 0.0003054181174027243 + }, + "preemptions": 0, + "prefill_tokens": 972023, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 7, + "queue_waiting_mean": 0.08826583592938733, + "records_all": 23855, + "records_clean": 16371, + "requests_per_step": { + "mean": 10.662696231140432, + "n": 16371, + "p50": 8.0, + "p95": 16.0, + "p99": 16.0 + }, + "scheduled_tokens_per_step": { + "mean": 69.98876061327958, + "n": 16371, + "p50": 8.0, + "p95": 16.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 29.214729049966408, + "n": 16371, + "p50": 19.582729, + "p95": 29.3386855, + "p99": 293.3203081 + }, + "token_efficiency_per_ms": 2.3956669423008075 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 1.6, + "completion_rate_within_10pct": true, + "tail_end_s": 331.74715706999996, + "tail_start_s": 321.74715706999996, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 1.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": false, + "tail_end_s": 363.41947703200276, + "tail_start_s": 353.41947703200276, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 336, + "run_dir": "runs/phase3/primary/P06-C11/moderate", + "run_id": "P06-C11-moderate", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.010014462145906983, + "eligible_miss_rate": 0.0006720840716075029, + "graph_miss_rate": 0.030732571638052177, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 504, + "moe_layer_cv": null, + "overflow_rate": 0.030060487566444676, + "padded_tokens_per_useful_token": 0.0014566419907382356, + "padding_fraction": 0.009799201503053077 + } + }, + "P06-C11-saturation": { + "blocks": [ + { + "bucket": 192.0, + "duration_ms": 10082.961571000002, + "eligible_miss": 3.0, + "miss": 33.0, + "model": 36.0, + "overflow": 30.0, + "padding": 3.0, + "tokens": 60243.0 + }, + { + "bucket": 960.0, + "duration_ms": 9944.095406, + "eligible_miss": 3.0, + "miss": 32.0, + "model": 47.0, + "overflow": 29.0, + "padding": 10.0, + "tokens": 54480.0 + }, + { + "bucket": 832.0, + "duration_ms": 9730.454249000002, + "eligible_miss": 0.0, + "miss": 28.0, + "model": 41.0, + "overflow": 28.0, + "padding": 0.0, + "tokens": 56909.0 + }, + { + "bucket": 10688.0, + "duration_ms": 10026.303037000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 167.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10688.0 + }, + { + "bucket": 10432.0, + "duration_ms": 9986.693028999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 163.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10432.0 + }, + { + "bucket": 3648.0, + "duration_ms": 10226.678839000002, + "eligible_miss": 2.0, + "miss": 24.0, + "model": 81.0, + "overflow": 22.0, + "padding": 12.0, + "tokens": 45900.0 + }, + { + "bucket": 384.0, + "duration_ms": 9925.763535999999, + "eligible_miss": 1.0, + "miss": 30.0, + "model": 36.0, + "overflow": 29.0, + "padding": 5.0, + "tokens": 59966.0 + }, + { + "bucket": 576.0, + "duration_ms": 10225.903192999996, + "eligible_miss": 2.0, + "miss": 32.0, + "model": 41.0, + "overflow": 30.0, + "padding": 8.0, + "tokens": 59654.0 + }, + { + "bucket": 6464.0, + "duration_ms": 9584.442423999999, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 113.0, + "overflow": 12.0, + "padding": 2.0, + "tokens": 29715.0 + }, + { + "bucket": 10880.0, + "duration_ms": 9960.685231999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 170.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10880.0 + }, + { + "bucket": 8128.0, + "duration_ms": 10558.448065000004, + "eligible_miss": 2.0, + "miss": 12.0, + "model": 139.0, + "overflow": 10.0, + "padding": 9.0, + "tokens": 26886.0 + }, + { + "bucket": 256.0, + "duration_ms": 9654.425447000003, + "eligible_miss": 0.0, + "miss": 30.0, + "model": 34.0, + "overflow": 30.0, + "padding": 6.0, + "tokens": 58128.0 + }, + { + "bucket": 768.0, + "duration_ms": 10016.586984000001, + "eligible_miss": 0.0, + "miss": 32.0, + "model": 44.0, + "overflow": 32.0, + "padding": 13.0, + "tokens": 59666.0 + }, + { + "bucket": 3136.0, + "duration_ms": 9749.143122000003, + "eligible_miss": 2.0, + "miss": 24.0, + "model": 73.0, + "overflow": 22.0, + "padding": 7.0, + "tokens": 46471.0 + }, + { + "bucket": 11264.0, + "duration_ms": 10010.441336999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 176.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11264.0 + }, + { + "bucket": 10688.0, + "duration_ms": 9979.167980999999, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 169.0, + "overflow": 1.0, + "padding": 2.0, + "tokens": 11707.0 + }, + { + "bucket": 1088.0, + "duration_ms": 10111.994275000005, + "eligible_miss": 2.0, + "miss": 31.0, + "model": 48.0, + "overflow": 29.0, + "padding": 18.0, + "tokens": 57957.0 + }, + { + "bucket": 640.0, + "duration_ms": 10101.266452000002, + "eligible_miss": 2.0, + "miss": 33.0, + "model": 43.0, + "overflow": 31.0, + "padding": 7.0, + "tokens": 58422.0 + }, + { + "bucket": 256.0, + "duration_ms": 9945.284613999998, + "eligible_miss": 0.0, + "miss": 31.0, + "model": 35.0, + "overflow": 31.0, + "padding": 2.0, + "tokens": 62645.0 + }, + { + "bucket": 10432.0, + "duration_ms": 9836.099997999996, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 165.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 12659.0 + }, + { + "bucket": 11008.0, + "duration_ms": 9939.759173, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 172.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 11008.0 + }, + { + "bucket": 3776.0, + "duration_ms": 10179.827958999998, + "eligible_miss": 3.0, + "miss": 24.0, + "model": 83.0, + "overflow": 21.0, + "padding": 10.0, + "tokens": 45969.0 + }, + { + "bucket": 320.0, + "duration_ms": 10209.444457000001, + "eligible_miss": 1.0, + "miss": 31.0, + "model": 36.0, + "overflow": 30.0, + "padding": 3.0, + "tokens": 61439.0 + }, + { + "bucket": 576.0, + "duration_ms": 9864.660417000001, + "eligible_miss": 2.0, + "miss": 30.0, + "model": 39.0, + "overflow": 28.0, + "padding": 6.0, + "tokens": 57207.0 + }, + { + "bucket": 3200.0, + "duration_ms": 9742.562434000005, + "eligible_miss": 1.0, + "miss": 22.0, + "model": 72.0, + "overflow": 21.0, + "padding": 3.0, + "tokens": 44364.0 + }, + { + "bucket": 10880.0, + "duration_ms": 10003.151557000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 170.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10880.0 + }, + { + "bucket": 10176.0, + "duration_ms": 10321.144609999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 162.0, + "overflow": 3.0, + "padding": 1.0, + "tokens": 15280.0 + }, + { + "bucket": 896.0, + "duration_ms": 10135.580694999999, + "eligible_miss": 3.0, + "miss": 31.0, + "model": 45.0, + "overflow": 28.0, + "padding": 9.0, + "tokens": 55560.0 + }, + { + "bucket": 192.0, + "duration_ms": 9969.717403999999, + "eligible_miss": 1.0, + "miss": 30.0, + "model": 33.0, + "overflow": 29.0, + "padding": 3.0, + "tokens": 58681.0 + }, + { + "bucket": 896.0, + "duration_ms": 9658.526538000002, + "eligible_miss": 2.0, + "miss": 30.0, + "model": 44.0, + "overflow": 28.0, + "padding": 11.0, + "tokens": 53798.0 + }, + { + "bucket": 5376.0, + "duration_ms": 9859.479598999997, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 99.0, + "overflow": 15.0, + "padding": 1.0, + "tokens": 34693.0 + }, + { + "bucket": 10688.0, + "duration_ms": 10015.031904, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 167.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10688.0 + }, + { + "bucket": 8896.0, + "duration_ms": 10145.63891, + "eligible_miss": 2.0, + "miss": 8.0, + "model": 147.0, + "overflow": 6.0, + "padding": 6.0, + "tokens": 19153.0 + }, + { + "bucket": 960.0, + "duration_ms": 9868.575407999999, + "eligible_miss": 4.0, + "miss": 31.0, + "model": 46.0, + "overflow": 27.0, + "padding": 12.0, + "tokens": 53598.0 + }, + { + "bucket": 448.0, + "duration_ms": 10157.341323999999, + "eligible_miss": 2.0, + "miss": 33.0, + "model": 40.0, + "overflow": 31.0, + "padding": 5.0, + "tokens": 62763.0 + }, + { + "bucket": 960.0, + "duration_ms": 9793.570380999998, + "eligible_miss": 2.0, + "miss": 29.0, + "model": 44.0, + "overflow": 27.0, + "padding": 2.0, + "tokens": 55034.0 + }, + { + "bucket": 10944.0, + "duration_ms": 9956.056209, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 171.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10944.0 + }, + { + "bucket": 10816.0, + "duration_ms": 9998.52987, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 169.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10816.0 + }, + { + "bucket": 2304.0, + "duration_ms": 10328.584660000002, + "eligible_miss": 2.0, + "miss": 29.0, + "model": 65.0, + "overflow": 27.0, + "padding": 8.0, + "tokens": 52481.0 + }, + { + "bucket": 256.0, + "duration_ms": 9895.535402000001, + "eligible_miss": 2.0, + "miss": 32.0, + "model": 36.0, + "overflow": 30.0, + "padding": 3.0, + "tokens": 58504.0 + }, + { + "bucket": 704.0, + "duration_ms": 10133.923506000001, + "eligible_miss": 4.0, + "miss": 33.0, + "model": 44.0, + "overflow": 29.0, + "padding": 12.0, + "tokens": 57857.0 + }, + { + "bucket": 1728.0, + "duration_ms": 9642.673120000001, + "eligible_miss": 1.0, + "miss": 24.0, + "model": 51.0, + "overflow": 23.0, + "padding": 0.0, + "tokens": 49241.0 + }, + { + "bucket": 10496.0, + "duration_ms": 9969.511140999995, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 164.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10496.0 + }, + { + "bucket": 10304.0, + "duration_ms": 9971.499231000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 161.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 10304.0 + }, + { + "bucket": 1856.0, + "duration_ms": 10479.133187, + "eligible_miss": 0.0, + "miss": 28.0, + "model": 57.0, + "overflow": 28.0, + "padding": 7.0, + "tokens": 55542.0 + }, + { + "bucket": 192.0, + "duration_ms": 9845.170644999998, + "eligible_miss": 1.0, + "miss": 30.0, + "model": 33.0, + "overflow": 29.0, + "padding": 4.0, + "tokens": 58977.0 + }, + { + "bucket": 952.0, + "duration_ms": 9922.926027000003, + "eligible_miss": 2.0, + "miss": 31.0, + "model": 45.0, + "overflow": 29.0, + "padding": 11.0, + "tokens": 54863.0 + }, + { + "bucket": 5568.0, + "duration_ms": 9766.356098, + "eligible_miss": 3.0, + "miss": 17.0, + "model": 104.0, + "overflow": 14.0, + "padding": 7.0, + "tokens": 31661.0 + } + ], + "clean": { + "admitted": 568, + "completed": 568, + "completed_throughput_rps": 2.3666666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1643843, + "offered_rps": 2.3666666666666667, + "output_tokens": 290816, + "start_s": 60.0 + }, + "config": "C11", + "drain_quarantined": false, + "drain_seconds": 101.96125678598764, + "latency_s": { + "e2e": { + "mean": 105.60085378511151, + "n": 568, + "p50": 109.96654033550294, + "p95": 121.63363380308728, + "p99": 126.2996933970525 + }, + "tpot": { + "mean": 0.053216996705190735, + "n": 568, + "p50": 0.053181282284747264, + "p95": 0.05820221032573622, + "p99": 0.05901893801076068 + }, + "ttft": { + "mean": 78.40696846875905, + "n": 568, + "p50": 83.40565797999443, + "p95": 95.05179660475405, + "p99": 100.46216534071517 + } + }, + "layer1": { + "decode_tokens": 273535, + "kv_usage_max": 0.7332090467546485, + "kv_usage_mean": 0.6163612764631382, + "mode_counts": { + "FULL": 3390, + "NONE": 929, + "PIECEWISE": 1 + }, + "mode_shares": { + "FULL": 0.7847222222222222, + "NONE": 0.2150462962962963, + "PIECEWISE": 0.0002314814814814815 + }, + "preemptions": 0, + "prefill_tokens": 1642938, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 201, + "queue_waiting_mean": 192.1122685185185, + "records_all": 8397, + "records_clean": 4320, + "requests_per_step": { + "mean": 63.62453703703704, + "n": 4320, + "p50": 64.0, + "p95": 64.0, + "p99": 64.0 + }, + "scheduled_tokens_per_step": { + "mean": 443.6280092592593, + "n": 4320, + "p50": 64.0, + "p95": 2048.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 110.97934042986111, + "n": 4320, + "p50": 59.9869195, + "p95": 338.55503250000004, + "p99": 371.98406197 + }, + "token_efficiency_per_ms": 3.9973927358095263 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P06", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 3.1, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 2.3, + "completion_rate_within_10pct": false, + "tail_end_s": 333.1218382779916, + "tail_start_s": 323.1218382779916, + "valid": false, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 3.1, + "clean_c_waiting_median": 192.0, + "completion_rate_rps": 2.3, + "completion_rate_within_10pct": false, + "tail_end_s": 366.2731193389918, + "tail_start_s": 356.2731193389918, + "valid": false, + "waiting_median": 192.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 568, + "run_dir": "runs/phase3/primary/P06-C11/saturation", + "run_id": "P06-C11-saturation", + "trace_files": 2, + "waste": { + "R128": 0.6023504575478206, + "R32": 0.5930739622699981, + "R64": 0.5988308514107739, + "bucket_slack": 0.002611677510228721, + "eligible_miss_rate": 0.013657407407407408, + "graph_miss_rate": 0.2150462962962963, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.2013888888888889, + "padded_tokens_per_useful_token": 0.00011896854273449195, + "padding_fraction": 0.0010503040353786622 + } + }, + "P07-C00-moderate": { + "blocks": [ + { + "bucket": 7296.0, + "duration_ms": 9971.907498999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 290.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 27872.0 + }, + { + "bucket": 7536.0, + "duration_ms": 9985.334541999999, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 285.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28112.0 + }, + { + "bucket": 7664.0, + "duration_ms": 9979.916429000008, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 283.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28240.0 + }, + { + "bucket": 8112.0, + "duration_ms": 9987.633896000001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 307.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28688.0 + }, + { + "bucket": 7664.0, + "duration_ms": 9980.657777999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 304.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28240.0 + }, + { + "bucket": 7640.0, + "duration_ms": 9962.675201000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 301.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28216.0 + }, + { + "bucket": 7256.0, + "duration_ms": 9975.330927, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 283.0, + "overflow": 5.0, + "padding": 6.0, + "tokens": 27864.0 + }, + { + "bucket": 7912.0, + "duration_ms": 9979.580427, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 303.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28488.0 + }, + { + "bucket": 8152.0, + "duration_ms": 9976.500125999999, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 323.0, + "overflow": 4.0, + "padding": 14.0, + "tokens": 28726.0 + }, + { + "bucket": 8104.0, + "duration_ms": 10161.867221999993, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 344.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28658.0 + }, + { + "bucket": 7720.0, + "duration_ms": 10428.90538799999, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 327.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 28286.0 + }, + { + "bucket": 7712.0, + "duration_ms": 9321.163778999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 312.0, + "overflow": 2.0, + "padding": 12.0, + "tokens": 17994.0 + }, + { + "bucket": 7432.0, + "duration_ms": 9997.364576000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 290.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28008.0 + }, + { + "bucket": 7704.0, + "duration_ms": 9968.157033000007, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 292.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28280.0 + }, + { + "bucket": 8016.0, + "duration_ms": 9983.975116999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 313.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28592.0 + }, + { + "bucket": 7344.0, + "duration_ms": 9979.674523000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 293.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 27920.0 + }, + { + "bucket": 7672.0, + "duration_ms": 9975.575549000001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 292.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28247.0 + }, + { + "bucket": 7864.0, + "duration_ms": 9992.28741, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 302.0, + "overflow": 4.0, + "padding": 11.0, + "tokens": 28441.0 + }, + { + "bucket": 7952.0, + "duration_ms": 9958.252177000008, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 318.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28528.0 + }, + { + "bucket": 7744.0, + "duration_ms": 9988.906290000008, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 328.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28300.0 + }, + { + "bucket": 7392.0, + "duration_ms": 9994.193122000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 311.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 27958.0 + }, + { + "bucket": 7808.0, + "duration_ms": 10226.192321999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 316.0, + "overflow": 4.0, + "padding": 8.0, + "tokens": 28388.0 + }, + { + "bucket": 8248.0, + "duration_ms": 10417.59643, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 328.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28824.0 + }, + { + "bucket": 8008.0, + "duration_ms": 9272.779190000003, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 323.0, + "overflow": 2.0, + "padding": 12.0, + "tokens": 18290.0 + }, + { + "bucket": 7696.0, + "duration_ms": 9975.396886000011, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 316.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28272.0 + }, + { + "bucket": 7688.0, + "duration_ms": 9974.740527999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 318.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28264.0 + }, + { + "bucket": 7608.0, + "duration_ms": 10001.287686999993, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 315.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28184.0 + }, + { + "bucket": 7768.0, + "duration_ms": 9973.692631000007, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 324.0, + "overflow": 5.0, + "padding": 12.0, + "tokens": 28370.0 + }, + { + "bucket": 7528.0, + "duration_ms": 9966.926483999992, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 317.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 28096.0 + }, + { + "bucket": 7320.0, + "duration_ms": 9979.419324999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 295.0, + "overflow": 4.0, + "padding": 14.0, + "tokens": 27894.0 + }, + { + "bucket": 7592.0, + "duration_ms": 9984.026474999999, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 296.0, + "overflow": 4.0, + "padding": 6.0, + "tokens": 28170.0 + }, + { + "bucket": 7736.0, + "duration_ms": 9996.897663000005, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 304.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28312.0 + }, + { + "bucket": 8032.0, + "duration_ms": 9989.042325, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 324.0, + "overflow": 4.0, + "padding": 8.0, + "tokens": 28612.0 + }, + { + "bucket": 8728.0, + "duration_ms": 10264.222522000005, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 382.0, + "overflow": 4.0, + "padding": 18.0, + "tokens": 29266.0 + }, + { + "bucket": 7776.0, + "duration_ms": 10385.723340000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 356.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28320.0 + }, + { + "bucket": 7616.0, + "duration_ms": 9266.232714000002, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 323.0, + "overflow": 2.0, + "padding": 6.0, + "tokens": 17888.0 + }, + { + "bucket": 7504.0, + "duration_ms": 9968.152762999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 313.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28064.0 + }, + { + "bucket": 8136.0, + "duration_ms": 9974.840383000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 336.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28712.0 + }, + { + "bucket": 8064.0, + "duration_ms": 9986.361588000007, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 359.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28604.0 + }, + { + "bucket": 7224.0, + "duration_ms": 9958.172716, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 338.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 27768.0 + }, + { + "bucket": 7200.0, + "duration_ms": 9997.393850000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 311.0, + "overflow": 4.0, + "padding": 8.0, + "tokens": 27763.0 + }, + { + "bucket": 7376.0, + "duration_ms": 9988.796274000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 292.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 27948.0 + }, + { + "bucket": 7464.0, + "duration_ms": 9971.854588999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 292.0, + "overflow": 4.0, + "padding": 5.0, + "tokens": 28047.0 + }, + { + "bucket": 7280.0, + "duration_ms": 9992.991605999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 276.0, + "overflow": 4.0, + "padding": 8.0, + "tokens": 27860.0 + }, + { + "bucket": 7888.0, + "duration_ms": 10047.721750999994, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 289.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 28460.0 + }, + { + "bucket": 8672.0, + "duration_ms": 10296.579270999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 330.0, + "overflow": 4.0, + "padding": 12.0, + "tokens": 29248.0 + }, + { + "bucket": 7912.0, + "duration_ms": 10413.904203, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 320.0, + "overflow": 4.0, + "padding": 8.0, + "tokens": 28492.0 + }, + { + "bucket": 7792.0, + "duration_ms": 9163.911243, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 309.0, + "overflow": 2.0, + "padding": 12.0, + "tokens": 18074.0 + } + ], + "clean": { + "admitted": 736, + "completed": 736, + "completed_throughput_rps": 3.066666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 942080, + "offered_rps": 3.066666666666667, + "output_tokens": 376832, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 5.035221696016379, + "latency_s": { + "e2e": { + "mean": 8.60263692463037, + "n": 736, + "p50": 8.720922641005018, + "p95": 9.229017912512063, + "p99": 9.427626249768945 + }, + "tpot": { + "mean": 0.015932894322490253, + "n": 736, + "p50": 0.016117811239738683, + "p95": 0.01720207957583063, + "p99": 0.017540519277731717 + }, + "ttft": { + "mean": 0.4609279258378513, + "n": 736, + "p50": 0.4356144489865983, + "p95": 0.554922532508499, + "p99": 0.5631193216860991 + } + }, + "layer1": { + "decode_tokens": 375768, + "kv_usage_max": 0.17042443903757776, + "kv_usage_mean": 0.13114972391662066, + "mode_counts": { + "FULL": 14817, + "NONE": 186 + }, + "mode_shares": { + "FULL": 0.9876024795040992, + "NONE": 0.012397520495900819 + }, + "preemptions": 0, + "prefill_tokens": 942080, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 1, + "queue_waiting_mean": 0.005398920215956809, + "records_all": 22882, + "records_clean": 15003, + "requests_per_step": { + "mean": 25.100779844031194, + "n": 15003, + "p50": 24.0, + "p95": 32.0, + "p99": 32.0 + }, + "scheduled_tokens_per_step": { + "mean": 87.83896554022529, + "n": 15003, + "p50": 24.0, + "p95": 32.0, + "p99": 2102.0 + }, + "step_duration_ms": { + "mean": 31.925929198826903, + "n": 15003, + "p50": 25.808318, + "p95": 29.5452691, + "p99": 412.22777862000004 + }, + "token_efficiency_per_ms": 2.7513362255859692 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P07", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 3.2, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 2.4, + "completion_rate_within_10pct": false, + "tail_end_s": 331.94992835301673, + "tail_start_s": 321.94992835301673, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 3.2, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 3.2, + "completion_rate_within_10pct": true, + "tail_end_s": 364.21091389100184, + "tail_start_s": 354.21091389100184, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 736, + "run_dir": "runs/phase3/primary/P07-C00/moderate", + "run_id": "P07-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.0, + "R32": 0.0, + "R64": 0.0, + "bucket_slack": 0.0014049177504090948, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.012397520495900819, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 186, + "moe_layer_cv": null, + "overflow_rate": 0.012397520495900819, + "padded_tokens_per_useful_token": 0.00039610030898859353, + "padding_fraction": 0.0014049177504090948 + } + }, + "P07-C00-saturation": { + "blocks": [ + { + "bucket": 16752.0, + "duration_ms": 9950.153387999995, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 99.0, + "overflow": 12.0, + "padding": 325.0, + "tokens": 31732.0 + }, + { + "bucket": 15208.0, + "duration_ms": 9991.850533, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 96.0, + "overflow": 13.0, + "padding": 289.0, + "tokens": 33555.0 + }, + { + "bucket": 14240.0, + "duration_ms": 10002.270118999997, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 96.0, + "overflow": 15.0, + "padding": 297.0, + "tokens": 33650.0 + }, + { + "bucket": 1040.0, + "duration_ms": 11075.8932, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 20.0, + "overflow": 14.0, + "padding": 30.0, + "tokens": 106042.0 + }, + { + "duration_ms": 10086.661052, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "bucket": 2696.0, + "duration_ms": 8805.740211999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 21.0, + "overflow": 9.0, + "padding": 12.0, + "tokens": 74012.0 + }, + { + "bucket": 24512.0, + "duration_ms": 9992.72727, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 114.0, + "overflow": 1.0, + "padding": 386.0, + "tokens": 25608.0 + }, + { + "bucket": 23256.0, + "duration_ms": 10053.696391000003, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 117.0, + "overflow": 1.0, + "padding": 406.0, + "tokens": 24716.0 + }, + { + "bucket": 20208.0, + "duration_ms": 9956.686812000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 112.0, + "overflow": 4.0, + "padding": 432.0, + "tokens": 25111.0 + }, + { + "bucket": 19280.0, + "duration_ms": 9997.674086, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 112.0, + "overflow": 5.0, + "padding": 349.0, + "tokens": 25614.0 + }, + { + "bucket": 1984.0, + "duration_ms": 10921.154089999998, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 24.0, + "overflow": 13.0, + "padding": 31.0, + "tokens": 100985.0 + }, + { + "duration_ms": 10088.02937, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "duration_ms": 10118.796231, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "bucket": 20336.0, + "duration_ms": 8830.944203000001, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 96.0, + "overflow": 2.0, + "padding": 322.0, + "tokens": 22057.0 + }, + { + "bucket": 19904.0, + "duration_ms": 10019.577912999996, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 105.0, + "overflow": 7.0, + "padding": 396.0, + "tokens": 30394.0 + }, + { + "bucket": 17688.0, + "duration_ms": 9972.803946999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 104.0, + "overflow": 11.0, + "padding": 272.0, + "tokens": 29529.0 + }, + { + "bucket": 15184.0, + "duration_ms": 9973.274595999994, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 97.0, + "overflow": 13.0, + "padding": 273.0, + "tokens": 33543.0 + }, + { + "bucket": 12312.0, + "duration_ms": 10407.001862, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 86.0, + "overflow": 15.0, + "padding": 265.0, + "tokens": 51277.0 + }, + { + "duration_ms": 10120.670290999999, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "duration_ms": 10056.795141, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "bucket": 7392.0, + "duration_ms": 9387.771273, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 41.0, + "overflow": 8.0, + "padding": 85.0, + "tokens": 65871.0 + }, + { + "bucket": 24560.0, + "duration_ms": 10008.093556999993, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 116.0, + "overflow": 1.0, + "padding": 433.0, + "tokens": 24978.0 + }, + { + "bucket": 21704.0, + "duration_ms": 9986.943306, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 113.0, + "overflow": 3.0, + "padding": 350.0, + "tokens": 25656.0 + }, + { + "bucket": 20592.0, + "duration_ms": 10009.293340000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 113.0, + "overflow": 3.0, + "padding": 404.0, + "tokens": 25315.0 + }, + { + "bucket": 16872.0, + "duration_ms": 9990.23609, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 103.0, + "overflow": 9.0, + "padding": 297.0, + "tokens": 29596.0 + }, + { + "duration_ms": 10501.451498, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 14.0, + "overflow": 14.0, + "tokens": 103534.0 + }, + { + "duration_ms": 10057.099084000001, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "bucket": 224.0, + "duration_ms": 9523.209128, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 12.0, + "overflow": 11.0, + "padding": 4.0, + "tokens": 87906.0 + }, + { + "bucket": 23048.0, + "duration_ms": 9911.289427, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 110.0, + "overflow": 3.0, + "padding": 364.0, + "tokens": 26660.0 + }, + { + "bucket": 19912.0, + "duration_ms": 9967.263191999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 107.0, + "overflow": 8.0, + "padding": 367.0, + "tokens": 27474.0 + }, + { + "bucket": 18392.0, + "duration_ms": 9988.550586000003, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 105.0, + "overflow": 9.0, + "padding": 302.0, + "tokens": 28252.0 + }, + { + "bucket": 15400.0, + "duration_ms": 9924.671659000003, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 99.0, + "overflow": 13.0, + "padding": 300.0, + "tokens": 30650.0 + }, + { + "bucket": 9304.0, + "duration_ms": 10994.186665000001, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 68.0, + "overflow": 14.0, + "padding": 198.0, + "tokens": 70027.0 + }, + { + "duration_ms": 10127.156386, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "duration_ms": 10078.856573000001, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "bucket": 11384.0, + "duration_ms": 8814.812563000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 57.0, + "overflow": 6.0, + "padding": 170.0, + "tokens": 48368.0 + }, + { + "bucket": 23984.0, + "duration_ms": 10032.058657000003, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 115.0, + "overflow": 1.0, + "padding": 384.0, + "tokens": 25458.0 + }, + { + "bucket": 21704.0, + "duration_ms": 9986.41377, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 114.0, + "overflow": 3.0, + "padding": 418.0, + "tokens": 24611.0 + }, + { + "bucket": 20344.0, + "duration_ms": 9950.299295000006, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 112.0, + "overflow": 3.0, + "padding": 324.0, + "tokens": 25289.0 + }, + { + "bucket": 13440.0, + "duration_ms": 10610.674172999996, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 90.0, + "overflow": 13.0, + "padding": 266.0, + "tokens": 50097.0 + }, + { + "duration_ms": 10146.211426, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "duration_ms": 10060.054723, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 12.0, + "overflow": 12.0, + "tokens": 98304.0 + }, + { + "bucket": 3584.0, + "duration_ms": 9127.613545, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 25.0, + "overflow": 9.0, + "padding": 67.0, + "tokens": 74197.0 + }, + { + "bucket": 23472.0, + "duration_ms": 9983.624336, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 112.0, + "overflow": 2.0, + "padding": 390.0, + "tokens": 25682.0 + }, + { + "bucket": 19264.0, + "duration_ms": 10066.432618, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 105.0, + "overflow": 8.0, + "padding": 305.0, + "tokens": 30308.0 + }, + { + "bucket": 17736.0, + "duration_ms": 10044.883365000007, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 103.0, + "overflow": 10.0, + "padding": 331.0, + "tokens": 30141.0 + }, + { + "bucket": 15312.0, + "duration_ms": 10041.116643000001, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 99.0, + "overflow": 13.0, + "padding": 314.0, + "tokens": 31613.0 + }, + { + "bucket": 7192.0, + "duration_ms": 10456.028967, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 56.0, + "overflow": 14.0, + "padding": 129.0, + "tokens": 72857.0 + } + ], + "clean": { + "admitted": 1226, + "completed": 1226, + "completed_throughput_rps": 5.108333333333333, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1569280, + "offered_rps": 5.108333333333333, + "output_tokens": 627712, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 26.098238860984566, + "latency_s": { + "e2e": { + "mean": 50.88784179576417, + "n": 1226, + "p50": 42.76737600599881, + "p95": 66.36966083350853, + "p99": 76.04152522151708 + }, + "tpot": { + "mean": 0.07373855953213852, + "n": 1226, + "p50": 0.06754604706850764, + "p95": 0.11021927271477756, + "p99": 0.11510430237866912 + }, + "ttft": { + "mean": 13.207437874841386, + "n": 1226, + "p50": 7.195238005995634, + "p95": 31.643819290002284, + "p99": 41.397227533998375 + } + }, + "layer1": { + "decode_tokens": 1086437, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9898433722094666, + "mode_counts": { + "FULL": 2964, + "NONE": 433, + "PIECEWISE": 11 + }, + "mode_shares": { + "FULL": 0.8697183098591549, + "NONE": 0.12705399061032863, + "PIECEWISE": 0.003227699530516432 + }, + "preemptions": 391, + "prefill_tokens": 1528968, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 91, + "queue_waiting_mean": 66.81338028169014, + "records_all": 5714, + "records_clean": 3408, + "requests_per_step": { + "mean": 188.46537558685446, + "n": 3408, + "p50": 186.5, + "p95": 218.0, + "p99": 223.0 + }, + "scheduled_tokens_per_step": { + "mean": 767.431044600939, + "n": 3408, + "p50": 192.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 140.90337340140846, + "n": 3408, + "p50": 84.995821, + "p95": 838.4765309999999, + "p99": 847.72110561 + }, + "token_efficiency_per_ms": 5.44650582931514 + }, + "layer2_missing_after_controller_cleanup": true, + "load": "saturation", + "pattern": "P07", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 4.05, + "clean_c_waiting_median": 70.0, + "completion_rate_rps": 8.3, + "completion_rate_within_10pct": false, + "tail_end_s": 358.0940292410087, + "tail_start_s": 348.0940292410087, + "valid": false, + "waiting_median": 85.0, + "waiting_within_10pct": false, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 4.05, + "clean_c_waiting_median": 70.0, + "completion_rate_rps": 8.8, + "completion_rate_within_10pct": false, + "tail_end_s": 417.66287675101194, + "tail_start_s": 407.66287675101194, + "valid": false, + "waiting_median": 85.0, + "waiting_within_10pct": false, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 1226, + "run_dir": "runs/phase3/primary/P07-C00/saturation", + "run_id": "P07-C00-saturation", + "trace_files": 0, + "waste": { + "R128": 0.0, + "R32": 0.0, + "R64": 0.0, + "bucket_slack": 0.017754083421928288, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.12705399061032863, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.12705399061032863, + "padded_tokens_per_useful_token": 0.003933234049793436, + "padding_fraction": 0.017754083421928288 + } + }, + "P08-C00-moderate": { + "blocks": [ + { + "bucket": 14056.0, + "duration_ms": 9973.848525000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 263.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22440.0 + }, + { + "bucket": 14216.0, + "duration_ms": 9967.381556999999, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 265.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20504.0 + }, + { + "bucket": 13688.0, + "duration_ms": 9966.267621999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 257.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22072.0 + }, + { + "bucket": 14104.0, + "duration_ms": 9990.000707, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 259.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20392.0 + }, + { + "bucket": 13912.0, + "duration_ms": 9988.124715999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 254.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22296.0 + }, + { + "bucket": 14400.0, + "duration_ms": 9978.191433, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 263.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 20738.0 + }, + { + "bucket": 14136.0, + "duration_ms": 9975.772872000003, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 265.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 22621.0 + }, + { + "bucket": 15152.0, + "duration_ms": 9957.495739999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 290.0, + "overflow": 3.0, + "padding": 4.0, + "tokens": 21436.0 + }, + { + "bucket": 13672.0, + "duration_ms": 10003.284594000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 274.0, + "overflow": 4.0, + "padding": 3.0, + "tokens": 22053.0 + }, + { + "bucket": 13912.0, + "duration_ms": 9952.839312, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 274.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20200.0 + }, + { + "bucket": 13656.0, + "duration_ms": 9985.693855000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 264.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22040.0 + }, + { + "bucket": 14016.0, + "duration_ms": 9984.229559999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 263.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20304.0 + }, + { + "bucket": 13808.0, + "duration_ms": 10000.379454999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 258.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22192.0 + }, + { + "bucket": 14400.0, + "duration_ms": 9949.354869000004, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 264.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20688.0 + }, + { + "bucket": 14200.0, + "duration_ms": 9995.867115, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 267.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 22635.0 + }, + { + "bucket": 15216.0, + "duration_ms": 9971.602727000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 288.0, + "overflow": 3.0, + "padding": 12.0, + "tokens": 21236.0 + }, + { + "bucket": 13944.0, + "duration_ms": 9976.335151999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 281.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22328.0 + }, + { + "bucket": 14112.0, + "duration_ms": 9982.280347, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 282.0, + "overflow": 3.0, + "padding": 7.0, + "tokens": 20393.0 + }, + { + "bucket": 13624.0, + "duration_ms": 9981.804391, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 272.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22008.0 + }, + { + "bucket": 14528.0, + "duration_ms": 9978.589668999997, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 282.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20816.0 + }, + { + "bucket": 14176.0, + "duration_ms": 9974.379933999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 284.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22560.0 + }, + { + "bucket": 14168.0, + "duration_ms": 9961.784644000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 285.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20456.0 + }, + { + "bucket": 13720.0, + "duration_ms": 9975.473181, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 277.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22104.0 + }, + { + "bucket": 14168.0, + "duration_ms": 9993.255385, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 280.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 20506.0 + }, + { + "bucket": 13760.0, + "duration_ms": 10024.908194000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 273.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22144.0 + }, + { + "bucket": 14544.0, + "duration_ms": 9930.993723000005, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 283.0, + "overflow": 4.0, + "padding": 2.0, + "tokens": 20880.0 + }, + { + "bucket": 14088.0, + "duration_ms": 10068.104888999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 285.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 22522.0 + }, + { + "bucket": 14128.0, + "duration_ms": 9882.121841999988, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 284.0, + "overflow": 3.0, + "padding": 4.0, + "tokens": 20412.0 + }, + { + "bucket": 13616.0, + "duration_ms": 10159.141668999995, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 273.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22000.0 + }, + { + "bucket": 13832.0, + "duration_ms": 9816.491638000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 263.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20120.0 + }, + { + "bucket": 13952.0, + "duration_ms": 10078.082938999993, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 266.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22336.0 + }, + { + "bucket": 13728.0, + "duration_ms": 9864.037892999986, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 254.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20016.0 + }, + { + "bucket": 14008.0, + "duration_ms": 9994.704993000005, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 255.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20296.0 + }, + { + "bucket": 13704.0, + "duration_ms": 9969.414991000005, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 247.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22120.0 + }, + { + "bucket": 14040.0, + "duration_ms": 9992.463381999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 249.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20352.0 + }, + { + "bucket": 14536.0, + "duration_ms": 9982.517607999996, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 257.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22952.0 + }, + { + "bucket": 15968.0, + "duration_ms": 9962.079757999993, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 301.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 22256.0 + }, + { + "bucket": 14504.0, + "duration_ms": 9986.519805000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 304.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22872.0 + }, + { + "bucket": 13800.0, + "duration_ms": 9982.44843, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 298.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 20064.0 + }, + { + "bucket": 13528.0, + "duration_ms": 9954.52768, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 282.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 21904.0 + }, + { + "bucket": 13584.0, + "duration_ms": 9994.419997000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 274.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 19872.0 + }, + { + "bucket": 13288.0, + "duration_ms": 9966.098695, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 255.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 21672.0 + }, + { + "bucket": 15040.0, + "duration_ms": 9972.185662000004, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 282.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 21328.0 + }, + { + "bucket": 14256.0, + "duration_ms": 10008.302290000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 277.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22640.0 + }, + { + "bucket": 13504.0, + "duration_ms": 9971.076230000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 268.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 19842.0 + }, + { + "bucket": 13720.0, + "duration_ms": 9979.042713999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 261.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22104.0 + }, + { + "bucket": 14888.0, + "duration_ms": 9956.323997, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 282.0, + "overflow": 3.0, + "padding": 2.0, + "tokens": 21174.0 + }, + { + "bucket": 13376.0, + "duration_ms": 9991.657548999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 260.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 21810.0 + } + ], + "clean": { + "admitted": 1344, + "completed": 1344, + "completed_throughput_rps": 5.6, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1720320, + "offered_rps": 5.6, + "output_tokens": 688128, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 5.560244205989875, + "latency_s": { + "e2e": { + "mean": 9.539366701375652, + "n": 1344, + "p50": 9.522452891003923, + "p95": 10.283733322568878, + "p99": 10.47988052393106 + }, + "tpot": { + "mean": 0.018335971554982657, + "n": 1344, + "p50": 0.0183028954266283, + "p95": 0.019802372696771993, + "p99": 0.020153929372081728 + }, + "ttft": { + "mean": 0.16968523677951453, + "n": 1344, + "p50": 0.1693532774952473, + "p95": 0.1819040500369738, + "p99": 0.19063215127098374 + } + }, + "layer1": { + "decode_tokens": 684642, + "kv_usage_max": 0.14144363341443633, + "kv_usage_mean": 0.12019157982892181, + "mode_counts": { + "FULL": 12866, + "NONE": 177, + "PIECEWISE": 1 + }, + "mode_shares": { + "FULL": 0.9863538791781662, + "NONE": 0.013569457221711132, + "PIECEWISE": 7.666360012266176e-05 + }, + "preemptions": 0, + "prefill_tokens": 344064, + "prefix_query_hit_ratio": 0.8, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 19842, + "records_clean": 13044, + "requests_per_step": { + "mean": 52.59015639374425, + "n": 13044, + "p50": 56.0, + "p95": 56.0, + "p99": 56.0 + }, + "scheduled_tokens_per_step": { + "mean": 78.8643054277829, + "n": 13044, + "p50": 56.0, + "p95": 56.0, + "p99": 2096.0 + }, + "step_duration_ms": { + "mean": 36.71817724087703, + "n": 13044, + "p50": 33.9972625, + "p95": 37.57236414999999, + "p99": 144.8586379 + }, + "token_efficiency_per_ms": 2.147827352932598 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P08", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 5.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 7.2, + "completion_rate_within_10pct": false, + "tail_end_s": 332.508747830987, + "tail_start_s": 322.508747830987, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 5.6, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 5.6, + "completion_rate_within_10pct": true, + "tail_end_s": 365.04628598099225, + "tail_start_s": 355.04628598099225, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 1344, + "run_dir": "runs/phase3/primary/P08-C00/moderate", + "run_id": "P08-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.0, + "R32": 0.0, + "R64": 0.0, + "bucket_slack": 5.026789832874023e-05, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.013569457221711132, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 178, + "moe_layer_cv": null, + "overflow_rate": 0.013569457221711132, + "padded_tokens_per_useful_token": 3.3051231352786896e-05, + "padding_fraction": 5.026789832874023e-05 + } + }, + "P08-C00-saturation": { + "blocks": [ + { + "bucket": 9216.0, + "duration_ms": 9340.304726, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 43.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 62688.0 + }, + { + "bucket": 26624.0, + "duration_ms": 10001.778646000006, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 104.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26624.0 + }, + { + "bucket": 25344.0, + "duration_ms": 9956.614504999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 99.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25344.0 + }, + { + "bucket": 24832.0, + "duration_ms": 10030.431760000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 97.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24832.0 + }, + { + "bucket": 24064.0, + "duration_ms": 9933.479672999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 94.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24064.0 + }, + { + "bucket": 19160.0, + "duration_ms": 10748.071276, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 79.0, + "overflow": 5.0, + "padding": 9.0, + "tokens": 45939.0 + }, + { + "bucket": 12288.0, + "duration_ms": 9267.026669000006, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 53.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 52621.0 + }, + { + "bucket": 26368.0, + "duration_ms": 9943.246027999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 103.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26368.0 + }, + { + "bucket": 25856.0, + "duration_ms": 10050.698422000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 101.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25856.0 + }, + { + "bucket": 24832.0, + "duration_ms": 9989.595579000003, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 97.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24832.0 + }, + { + "bucket": 24064.0, + "duration_ms": 9906.974473999995, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 94.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24064.0 + }, + { + "bucket": 15832.0, + "duration_ms": 11031.223406, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 68.0, + "overflow": 7.0, + "padding": 9.0, + "tokens": 58554.0 + }, + { + "bucket": 16128.0, + "duration_ms": 8962.881495000001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 66.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 40518.0 + }, + { + "bucket": 26368.0, + "duration_ms": 10047.384447999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 103.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26368.0 + }, + { + "bucket": 25344.0, + "duration_ms": 9939.983968000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 99.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25344.0 + }, + { + "bucket": 24576.0, + "duration_ms": 9980.957590000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 96.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24576.0 + }, + { + "bucket": 24064.0, + "duration_ms": 10063.673846999998, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 94.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24064.0 + }, + { + "bucket": 12760.0, + "duration_ms": 10636.981154, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 57.0, + "overflow": 8.0, + "padding": 9.0, + "tokens": 63581.0 + }, + { + "bucket": 19200.0, + "duration_ms": 9239.554301999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 77.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 35491.0 + }, + { + "bucket": 26368.0, + "duration_ms": 10018.135894000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 103.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26368.0 + }, + { + "bucket": 25600.0, + "duration_ms": 10004.704074, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 100.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25600.0 + }, + { + "bucket": 24832.0, + "duration_ms": 9988.921407000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 97.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24832.0 + }, + { + "bucket": 24064.0, + "duration_ms": 9980.550060999996, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 94.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24064.0 + }, + { + "bucket": 9176.0, + "duration_ms": 10845.426534999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 45.0, + "overflow": 10.0, + "padding": 9.0, + "tokens": 76288.0 + }, + { + "bucket": 23552.0, + "duration_ms": 9122.136448, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 92.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 23552.0 + }, + { + "bucket": 26112.0, + "duration_ms": 10039.638599000002, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 102.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26112.0 + }, + { + "bucket": 25088.0, + "duration_ms": 9935.394242, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 98.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25088.0 + }, + { + "bucket": 24576.0, + "duration_ms": 9981.978842999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 96.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24576.0 + }, + { + "bucket": 24064.0, + "duration_ms": 10074.602884000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 94.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24064.0 + }, + { + "bucket": 6104.0, + "duration_ms": 9997.324195000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 33.0, + "overflow": 10.0, + "padding": 9.0, + "tokens": 73216.0 + }, + { + "bucket": 26880.0, + "duration_ms": 9919.636349999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 105.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26880.0 + }, + { + "bucket": 26112.0, + "duration_ms": 10016.483458, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 102.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26112.0 + }, + { + "bucket": 25344.0, + "duration_ms": 9973.412112000004, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 99.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25344.0 + }, + { + "bucket": 24832.0, + "duration_ms": 10002.756156, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 97.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24832.0 + }, + { + "bucket": 24064.0, + "duration_ms": 9929.880055000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 94.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24064.0 + }, + { + "bucket": 6872.0, + "duration_ms": 10052.947775999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 36.0, + "overflow": 10.0, + "padding": 9.0, + "tokens": 73984.0 + }, + { + "bucket": 26624.0, + "duration_ms": 9946.62487499999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 104.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26624.0 + }, + { + "bucket": 25600.0, + "duration_ms": 9982.362614000001, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 100.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25600.0 + }, + { + "bucket": 25088.0, + "duration_ms": 10020.922033, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 98.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25088.0 + }, + { + "bucket": 24320.0, + "duration_ms": 9969.521037, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 95.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24320.0 + }, + { + "bucket": 22488.0, + "duration_ms": 10837.747847000004, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 90.0, + "overflow": 3.0, + "padding": 9.0, + "tokens": 36127.0 + }, + { + "bucket": 8448.0, + "duration_ms": 9141.736970000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 40.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 61921.0 + }, + { + "bucket": 26624.0, + "duration_ms": 9981.340275999999, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 104.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 26624.0 + }, + { + "bucket": 25600.0, + "duration_ms": 9997.061726999997, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 100.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 25600.0 + }, + { + "bucket": 24832.0, + "duration_ms": 9999.201582, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 97.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24832.0 + }, + { + "bucket": 24320.0, + "duration_ms": 10035.232025000005, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 95.0, + "overflow": 0.0, + "padding": 0.0, + "tokens": 24320.0 + }, + { + "bucket": 19416.0, + "duration_ms": 10835.279315, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 80.0, + "overflow": 5.0, + "padding": 9.0, + "tokens": 46195.0 + }, + { + "bucket": 11776.0, + "duration_ms": 9096.011046000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 51.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 52109.0 + } + ], + "clean": { + "admitted": 2233, + "completed": 2233, + "completed_throughput_rps": 9.304166666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2858240, + "offered_rps": 9.304166666666667, + "output_tokens": 1143296, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 14.923095890000695, + "latency_s": { + "e2e": { + "mean": 29.33866877817903, + "n": 2233, + "p50": 29.366875634004828, + "p95": 29.640699471993138, + "p99": 29.840447392413626 + }, + "tpot": { + "mean": 0.055211153321731224, + "n": 2233, + "p50": 0.05503479847167354, + "p95": 0.05621860527475374, + "p99": 0.05679465504637017 + }, + "ttft": { + "mean": 1.1257694307743766, + "n": 2233, + "p50": 1.2680383940169122, + "p95": 1.3710993590124416, + "p99": 1.7439167306513992 + } + }, + "layer1": { + "decode_tokens": 1059552, + "kv_usage_max": 0.6920789402541228, + "kv_usage_mean": 0.47382357971574685, + "mode_counts": { + "FULL": 4070, + "NONE": 87, + "PIECEWISE": 8 + }, + "mode_shares": { + "FULL": 0.9771908763505402, + "NONE": 0.020888355342136854, + "PIECEWISE": 0.0019207683073229293 + }, + "preemptions": 0, + "prefill_tokens": 576512, + "prefix_query_hit_ratio": 0.8, + "queue_waiting_max": 0, + "queue_waiting_mean": 0.0, + "records_all": 8229, + "records_clean": 4165, + "requests_per_step": { + "mean": 254.93517406962786, + "n": 4165, + "p50": 256.0, + "p95": 256.0, + "p99": 256.0 + }, + "scheduled_tokens_per_step": { + "mean": 392.8124849939976, + "n": 4165, + "p50": 256.0, + "p95": 256.0, + "p99": 8099.0 + }, + "step_duration_ms": { + "mean": 114.95746276206482, + "n": 4165, + "p50": 101.934486, + "p95": 107.8044754, + "p99": 858.72215388 + }, + "token_efficiency_per_ms": 3.4170246589995465 + }, + "layer2_missing_after_controller_cleanup": true, + "load": "saturation", + "pattern": "P08", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.0, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 5.2, + "completion_rate_within_10pct": false, + "tail_end_s": 411.30330635100836, + "tail_start_s": 401.30330635100836, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.0, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 10.1, + "completion_rate_within_10pct": false, + "tail_end_s": 493.2576180040196, + "tail_start_s": 483.2576180040196, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 2233, + "run_dir": "runs/phase3/primary/P08-C00/saturation", + "run_id": "P08-C00-saturation", + "trace_files": 0, + "waste": { + "R128": 0.0, + "R32": 0.0, + "R64": 0.0, + "bucket_slack": 6.885366301487239e-05, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.020888355342136854, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.020888355342136854, + "padded_tokens_per_useful_token": 4.4008058364464957e-05, + "padding_fraction": 6.885366301487239e-05 + } + }, + "P09-C00-moderate": { + "blocks": [ + { + "bucket": 4842.0, + "duration_ms": 9958.405207000005, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 208.0, + "overflow": 13.0, + "padding": 352.0, + "tokens": 50247.0 + }, + { + "bucket": 6232.0, + "duration_ms": 10019.690176, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 379.0, + "overflow": 9.0, + "padding": 398.0, + "tokens": 34279.0 + }, + { + "bucket": 6130.0, + "duration_ms": 10012.610543, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 347.0, + "overflow": 9.0, + "padding": 569.0, + "tokens": 38180.0 + }, + { + "bucket": 4522.0, + "duration_ms": 10068.259875999993, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 218.0, + "overflow": 13.0, + "padding": 309.0, + "tokens": 51545.0 + }, + { + "bucket": 4488.0, + "duration_ms": 9882.649579000003, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 278.0, + "overflow": 14.0, + "padding": 449.0, + "tokens": 44110.0 + }, + { + "bucket": 6070.0, + "duration_ms": 9947.566525999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 398.0, + "overflow": 10.0, + "padding": 462.0, + "tokens": 30960.0 + }, + { + "bucket": 5762.0, + "duration_ms": 10407.604067000006, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 389.0, + "overflow": 11.0, + "padding": 413.0, + "tokens": 39848.0 + }, + { + "bucket": 5364.0, + "duration_ms": 9508.797856999996, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 286.0, + "overflow": 12.0, + "padding": 461.0, + "tokens": 36418.0 + }, + { + "bucket": 4626.0, + "duration_ms": 9989.894830999996, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 207.0, + "overflow": 11.0, + "padding": 542.0, + "tokens": 50432.0 + }, + { + "bucket": 6263.0, + "duration_ms": 9985.924533000003, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 383.0, + "overflow": 9.0, + "padding": 479.0, + "tokens": 34787.0 + }, + { + "bucket": 4496.0, + "duration_ms": 10077.274496999995, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 295.0, + "overflow": 13.0, + "padding": 663.0, + "tokens": 44594.0 + }, + { + "bucket": 4492.0, + "duration_ms": 9866.059131999997, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 353.0, + "overflow": 13.0, + "padding": 445.0, + "tokens": 34497.0 + }, + { + "bucket": 4804.0, + "duration_ms": 10142.781146999996, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 434.0, + "overflow": 13.0, + "padding": 450.0, + "tokens": 31998.0 + }, + { + "bucket": 3332.0, + "duration_ms": 9844.485115999998, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 203.0, + "overflow": 13.0, + "padding": 238.0, + "tokens": 51123.0 + }, + { + "bucket": 6188.0, + "duration_ms": 10027.362657999993, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 225.0, + "overflow": 13.0, + "padding": 699.0, + "tokens": 44635.0 + }, + { + "bucket": 4724.0, + "duration_ms": 9965.154617999997, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 173.0, + "overflow": 14.0, + "padding": 434.0, + "tokens": 51340.0 + }, + { + "bucket": 5552.0, + "duration_ms": 9927.860614999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 312.0, + "overflow": 10.0, + "padding": 558.0, + "tokens": 37401.0 + }, + { + "bucket": 2860.0, + "duration_ms": 10211.194994999996, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 192.0, + "overflow": 14.0, + "padding": 182.0, + "tokens": 59914.0 + }, + { + "bucket": 5386.0, + "duration_ms": 9765.457841999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 265.0, + "overflow": 11.0, + "padding": 453.0, + "tokens": 40425.0 + }, + { + "bucket": 4166.0, + "duration_ms": 10272.187767999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 385.0, + "overflow": 11.0, + "padding": 242.0, + "tokens": 44978.0 + }, + { + "bucket": 5616.0, + "duration_ms": 9760.582664000001, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 225.0, + "overflow": 13.0, + "padding": 551.0, + "tokens": 45331.0 + }, + { + "bucket": 6718.0, + "duration_ms": 9900.015582000007, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 388.0, + "overflow": 10.0, + "padding": 359.0, + "tokens": 34828.0 + }, + { + "bucket": 4962.0, + "duration_ms": 10067.267675999994, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 272.0, + "overflow": 15.0, + "padding": 370.0, + "tokens": 48400.0 + }, + { + "bucket": 4784.0, + "duration_ms": 9941.889420000003, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 151.0, + "overflow": 14.0, + "padding": 566.0, + "tokens": 52131.0 + }, + { + "bucket": 4152.0, + "duration_ms": 9947.915998000008, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 222.0, + "overflow": 14.0, + "padding": 414.0, + "tokens": 49116.0 + }, + { + "bucket": 5470.0, + "duration_ms": 9983.611937999998, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 364.0, + "overflow": 14.0, + "padding": 547.0, + "tokens": 34668.0 + }, + { + "bucket": 3540.0, + "duration_ms": 11197.859721, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 221.0, + "overflow": 16.0, + "padding": 221.0, + "tokens": 66725.0 + }, + { + "bucket": 6640.0, + "duration_ms": 9417.335203000006, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 272.0, + "overflow": 9.0, + "padding": 578.0, + "tokens": 32531.0 + }, + { + "bucket": 5842.0, + "duration_ms": 9879.985574999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 329.0, + "overflow": 9.0, + "padding": 450.0, + "tokens": 40048.0 + }, + { + "bucket": 2872.0, + "duration_ms": 9451.456130999999, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 80.0, + "overflow": 14.0, + "padding": 146.0, + "tokens": 61227.0 + }, + { + "bucket": 5776.0, + "duration_ms": 9994.594990999996, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 136.0, + "overflow": 10.0, + "padding": 378.0, + "tokens": 55901.0 + }, + { + "bucket": 5339.0, + "duration_ms": 10200.117377999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 422.0, + "overflow": 10.0, + "padding": 289.0, + "tokens": 38209.0 + }, + { + "bucket": 2976.0, + "duration_ms": 10224.24287, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 82.0, + "overflow": 16.0, + "padding": 225.0, + "tokens": 64384.0 + }, + { + "bucket": 4264.0, + "duration_ms": 9567.238165000006, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 152.0, + "overflow": 14.0, + "padding": 401.0, + "tokens": 52662.0 + }, + { + "bucket": 5566.0, + "duration_ms": 9932.33232, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 341.0, + "overflow": 11.0, + "padding": 537.0, + "tokens": 37176.0 + }, + { + "bucket": 6158.0, + "duration_ms": 10517.70363300001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 423.0, + "overflow": 10.0, + "padding": 355.0, + "tokens": 36732.0 + }, + { + "bucket": 4512.0, + "duration_ms": 9502.686013999999, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 253.0, + "overflow": 12.0, + "padding": 379.0, + "tokens": 45470.0 + }, + { + "bucket": 4606.0, + "duration_ms": 10520.496230000002, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 226.0, + "overflow": 14.0, + "padding": 506.0, + "tokens": 55036.0 + }, + { + "bucket": 3856.0, + "duration_ms": 10017.898350999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 99.0, + "overflow": 15.0, + "padding": 311.0, + "tokens": 62102.0 + }, + { + "bucket": 4112.0, + "duration_ms": 9365.698862000003, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 105.0, + "overflow": 11.0, + "padding": 388.0, + "tokens": 52921.0 + }, + { + "bucket": 5820.0, + "duration_ms": 10483.163068999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 219.0, + "overflow": 11.0, + "padding": 689.0, + "tokens": 52494.0 + }, + { + "bucket": 5088.0, + "duration_ms": 9710.597793999992, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 127.0, + "overflow": 11.0, + "padding": 361.0, + "tokens": 51839.0 + }, + { + "bucket": 5676.0, + "duration_ms": 9812.009743999994, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 390.0, + "overflow": 9.0, + "padding": 382.0, + "tokens": 33697.0 + }, + { + "bucket": 5322.0, + "duration_ms": 10481.612746, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 271.0, + "overflow": 12.0, + "padding": 493.0, + "tokens": 50862.0 + }, + { + "bucket": 5042.0, + "duration_ms": 9438.804079000005, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 363.0, + "overflow": 11.0, + "padding": 542.0, + "tokens": 32980.0 + }, + { + "bucket": 4616.0, + "duration_ms": 9963.877349000002, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 420.0, + "overflow": 13.0, + "padding": 285.0, + "tokens": 33032.0 + }, + { + "bucket": 4592.0, + "duration_ms": 10306.957352000003, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 260.0, + "overflow": 13.0, + "padding": 466.0, + "tokens": 49197.0 + }, + { + "bucket": 3688.0, + "duration_ms": 9934.53882, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 94.0, + "overflow": 13.0, + "padding": 335.0, + "tokens": 61669.0 + } + ], + "clean": { + "admitted": 1181, + "completed": 1185, + "completed_throughput_rps": 4.9375, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2116361, + "offered_rps": 4.920833333333333, + "output_tokens": 75840, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 0.6209534369991161, + "latency_s": { + "e2e": { + "mean": 1.6859668752037082, + "n": 1185, + "p50": 1.3681276970019098, + "p95": 3.9139900154143104, + "p99": 4.631161756748335 + }, + "tpot": { + "mean": 0.023014267673890598, + "n": 1185, + "p50": 0.017877640635041252, + "p95": 0.05483829206029425, + "p99": 0.06860999865810366 + }, + "ttft": { + "mean": 0.23606801174860054, + "n": 1185, + "p50": 0.12713102900306694, + "p95": 0.7102715160115618, + "p99": 1.0138114924693946 + } + }, + "layer1": { + "decode_tokens": 74436, + "kv_usage_max": 0.23887537172208706, + "kv_usage_mean": 0.03684185997310298, + "mode_counts": { + "FULL": 11783, + "NONE": 580, + "PIECEWISE": 474 + }, + "mode_shares": { + "FULL": 0.9178935888447457, + "NONE": 0.045181896081639014, + "PIECEWISE": 0.036924515073615334 + }, + "preemptions": 0, + "prefill_tokens": 2108643, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 1, + "queue_waiting_mean": 7.789982083041208e-05, + "records_all": 19341, + "records_clean": 12837, + "requests_per_step": { + "mean": 5.891485549583236, + "n": 12837, + "p50": 4.0, + "p95": 16.0, + "p99": 21.0 + }, + "scheduled_tokens_per_step": { + "mean": 170.0614629586352, + "n": 12837, + "p50": 4.0, + "p95": 456.0, + "p99": 5951.919999999998 + }, + "step_duration_ms": { + "mean": 37.34546321243281, + "n": 12837, + "p50": 17.244659, + "p95": 108.07043219999986, + "p99": 478.49575835999946 + }, + "token_efficiency_per_ms": 4.553738214231586 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P09", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 4.949999999999999, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 5.3, + "completion_rate_within_10pct": true, + "tail_end_s": 334.5889142579981, + "tail_start_s": 324.5889142579981, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 4.949999999999999, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 4.8, + "completion_rate_within_10pct": true, + "tail_end_s": 366.81370019100723, + "tail_start_s": 356.81370019100723, + "valid": true, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": true, + "requests_completed": 1185, + "run_dir": "runs/phase3/primary/P09-C00/moderate", + "run_id": "P09-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.7696283944181126, + "R32": 0.7555897047229047, + "R64": 0.7648466836330938, + "bucket_slack": 0.08542101015535679, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.045181896081639014, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 1054, + "moe_layer_cv": null, + "overflow_rate": 0.045181896081639014, + "padded_tokens_per_useful_token": 0.009308870636381002, + "padding_fraction": 0.08542101015535679 + } + }, + "P09-C00-saturation": { + "blocks": [ + { + "duration_ms": 9749.202351999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 10176.620378000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "duration_ms": 9362.997435, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "bucket": 4936.0, + "duration_ms": 10439.874369999998, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 37.0, + "overflow": 9.0, + "padding": 129.0, + "tokens": 66116.0 + }, + { + "duration_ms": 9925.238021000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 82014.0 + }, + { + "duration_ms": 9704.610116999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "bucket": 160.0, + "duration_ms": 9388.846731, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 7.0, + "tokens": 73377.0 + }, + { + "bucket": 4152.0, + "duration_ms": 10519.513901, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 37.0, + "overflow": 11.0, + "padding": 175.0, + "tokens": 71992.0 + }, + { + "duration_ms": 10267.690021, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 88360.0 + }, + { + "duration_ms": 9614.783594, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 76869.0 + }, + { + "bucket": 2400.0, + "duration_ms": 9588.755232000001, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 23.0, + "overflow": 8.0, + "padding": 30.0, + "tokens": 62556.0 + }, + { + "bucket": 1912.0, + "duration_ms": 10646.490665, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 23.0, + "overflow": 11.0, + "padding": 33.0, + "tokens": 80491.0 + }, + { + "duration_ms": 10009.880642, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 78425.0 + }, + { + "duration_ms": 10656.483834999999, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 89372.0 + }, + { + "bucket": 4144.0, + "duration_ms": 9085.264818000003, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 33.0, + "overflow": 7.0, + "padding": 151.0, + "tokens": 56724.0 + }, + { + "bucket": 312.0, + "duration_ms": 10100.92011, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 13.0, + "overflow": 11.0, + "padding": 5.0, + "tokens": 84841.0 + }, + { + "duration_ms": 10627.625256, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 84284.0 + }, + { + "duration_ms": 10260.804656999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "bucket": 3264.0, + "duration_ms": 9273.320073000003, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 33.0, + "overflow": 9.0, + "padding": 63.0, + "tokens": 54343.0 + }, + { + "bucket": 144.0, + "duration_ms": 10147.99783, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 13.0, + "overflow": 12.0, + "padding": 4.0, + "tokens": 85321.0 + }, + { + "duration_ms": 10021.874463, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 86244.0 + }, + { + "duration_ms": 9849.016171000001, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 78795.0 + }, + { + "bucket": 4216.0, + "duration_ms": 9761.302597000004, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 34.0, + "overflow": 10.0, + "padding": 44.0, + "tokens": 64574.0 + }, + { + "bucket": 304.0, + "duration_ms": 10243.365921, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 13.0, + "overflow": 11.0, + "padding": 5.0, + "tokens": 84741.0 + }, + { + "duration_ms": 9702.099989999999, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 78863.0 + }, + { + "duration_ms": 10873.143569999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 90112.0 + }, + { + "bucket": 4192.0, + "duration_ms": 9480.605753999998, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 35.0, + "overflow": 10.0, + "padding": 44.0, + "tokens": 60971.0 + }, + { + "bucket": 152.0, + "duration_ms": 10362.664876, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 12.0, + "overflow": 11.0, + "padding": 5.0, + "tokens": 86984.0 + }, + { + "duration_ms": 10277.260139000002, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 86407.0 + }, + { + "bucket": 160.0, + "duration_ms": 9326.807399, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 2.0, + "tokens": 72897.0 + }, + { + "bucket": 3960.0, + "duration_ms": 9700.36302, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 35.0, + "overflow": 12.0, + "padding": 71.0, + "tokens": 66285.0 + }, + { + "duration_ms": 10365.53776, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 83470.0 + }, + { + "duration_ms": 9817.621681, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 80495.0 + }, + { + "bucket": 960.0, + "duration_ms": 10456.115671000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 17.0, + "overflow": 11.0, + "padding": 14.0, + "tokens": 79230.0 + }, + { + "bucket": 3184.0, + "duration_ms": 9856.080246000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 30.0, + "overflow": 11.0, + "padding": 104.0, + "tokens": 71349.0 + }, + { + "duration_ms": 9921.03899, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 78306.0 + }, + { + "duration_ms": 10466.651198, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81920.0 + }, + { + "bucket": 856.0, + "duration_ms": 8754.815939, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 15.0, + "overflow": 9.0, + "padding": 33.0, + "tokens": 63314.0 + }, + { + "bucket": 2688.0, + "duration_ms": 11187.383653000003, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 32.0, + "overflow": 12.0, + "padding": 59.0, + "tokens": 82299.0 + }, + { + "duration_ms": 9584.902875, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 79102.0 + }, + { + "duration_ms": 10227.432321, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 81437.0 + }, + { + "bucket": 3528.0, + "duration_ms": 8850.584567999998, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 29.0, + "overflow": 8.0, + "padding": 100.0, + "tokens": 59108.0 + }, + { + "bucket": 488.0, + "duration_ms": 10674.824159000002, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 17.0, + "overflow": 14.0, + "padding": 18.0, + "tokens": 82423.0 + }, + { + "duration_ms": 10659.708045999998, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 89002.0 + }, + { + "duration_ms": 9632.058042, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 10.0, + "overflow": 10.0, + "tokens": 76260.0 + }, + { + "bucket": 3168.0, + "duration_ms": 9123.113560000002, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 29.0, + "overflow": 7.0, + "padding": 90.0, + "tokens": 52910.0 + }, + { + "bucket": 456.0, + "duration_ms": 10462.624554999999, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 17.0, + "overflow": 14.0, + "padding": 6.0, + "tokens": 89979.0 + }, + { + "duration_ms": 10867.76237, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 11.0, + "overflow": 11.0, + "tokens": 88279.0 + } + ], + "clean": { + "admitted": 1969, + "completed": 1969, + "completed_throughput_rps": 8.204166666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3580022, + "offered_rps": 8.204166666666667, + "output_tokens": 126016, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 16.845643790991744, + "latency_s": { + "e2e": { + "mean": 31.15778886316919, + "n": 1969, + "p50": 31.64944553602254, + "p95": 34.37905165641568, + "p99": 35.391083135553636 + }, + "tpot": { + "mean": 0.28642918251555166, + "n": 1969, + "p50": 0.28561478876231594, + "p95": 0.29734348012356177, + "p99": 0.30687090564979286 + }, + "ttft": { + "mean": 13.112750364689438, + "n": 1969, + "p50": 13.525480992015218, + "p95": 16.44797444838332, + "p99": 17.28292753903661 + } + }, + "layer1": { + "decode_tokens": 130074, + "kv_usage_max": 0.9999459313327926, + "kv_usage_mean": 0.9840891207267473, + "mode_counts": { + "FULL": 309, + "NONE": 495, + "PIECEWISE": 2 + }, + "mode_shares": { + "FULL": 0.38337468982630274, + "NONE": 0.6141439205955335, + "PIECEWISE": 0.0024813895781637717 + }, + "preemptions": 14, + "prefill_tokens": 3582175, + "prefix_query_hit_ratio": 0.0, + "queue_waiting_max": 130, + "queue_waiting_mean": 98.75930521091811, + "records_all": 1390, + "records_clean": 806, + "requests_per_step": { + "mean": 152.36352357320098, + "n": 806, + "p50": 153.0, + "p95": 170.0, + "p99": 176.0 + }, + "scheduled_tokens_per_step": { + "mean": 4605.7679900744415, + "n": 806, + "p50": 7108.5, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 595.6000987245658, + "n": 806, + "p50": 742.1835209999999, + "p95": 1068.00519675, + "p99": 1104.1620354000001 + }, + "token_efficiency_per_ms": 7.732987284483932 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P09", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 7.85, + "clean_c_waiting_median": 98.0, + "completion_rate_rps": 9.1, + "completion_rate_within_10pct": false, + "tail_end_s": 358.2478721680236, + "tail_start_s": 348.2478721680236, + "valid": false, + "waiting_median": 90.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 7.85, + "clean_c_waiting_median": 98.0, + "completion_rate_rps": 8.0, + "completion_rate_within_10pct": true, + "tail_end_s": 417.95203437501914, + "tail_start_s": 407.95203437501914, + "valid": false, + "waiting_median": 88.0, + "waiting_within_10pct": false, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 1969, + "run_dir": "runs/phase3/primary/P09-C00/saturation", + "run_id": "P09-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.7696283944181126, + "R32": 0.7555897047229047, + "R64": 0.7648466836330938, + "bucket_slack": 0.023966543348882097, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.6141439205955335, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.6141439205955335, + "padded_tokens_per_useful_token": 0.000321099150407206, + "padding_fraction": 0.023966543348882097 + } + }, + "P10-C00-moderate": { + "blocks": [ + { + "bucket": 617.0, + "duration_ms": 8382.640889000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 620.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 14884.0 + }, + { + "bucket": 126.0, + "duration_ms": 11045.799707, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 128.0, + "overflow": 6.0, + "padding": 1.0, + "tokens": 44858.0 + }, + { + "bucket": 152.0, + "duration_ms": 8970.843604999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 27.0, + "overflow": 6.0, + "padding": 51.0, + "tokens": 42577.0 + }, + { + "bucket": 1504.0, + "duration_ms": 10324.448034000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 192.0, + "overflow": 4.0, + "padding": 488.0, + "tokens": 24681.0 + }, + { + "bucket": 770.0, + "duration_ms": 11102.520114999996, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 231.0, + "overflow": 6.0, + "padding": 111.0, + "tokens": 43629.0 + }, + { + "bucket": 1160.0, + "duration_ms": 8529.057380000006, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 166.0, + "overflow": 3.0, + "padding": 117.0, + "tokens": 15737.0 + }, + { + "bucket": 688.0, + "duration_ms": 5848.787868000003, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 476.0, + "overflow": 2.0, + "padding": 46.0, + "tokens": 4992.0 + }, + { + "bucket": 255.0, + "duration_ms": 9870.306260000001, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 260.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 35214.0 + }, + { + "bucket": 272.0, + "duration_ms": 7990.321079999999, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 72.0, + "overflow": 4.0, + "padding": 7.0, + "tokens": 28400.0 + }, + { + "bucket": 1302.0, + "duration_ms": 9966.716543000008, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 237.0, + "overflow": 4.0, + "padding": 243.0, + "tokens": 23279.0 + }, + { + "bucket": 448.0, + "duration_ms": 11609.913073999996, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 132.0, + "overflow": 6.0, + "padding": 9.0, + "tokens": 43441.0 + }, + { + "bucket": 1565.0, + "duration_ms": 8345.582649000005, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 496.0, + "overflow": 2.0, + "padding": 83.0, + "tokens": 12026.0 + }, + { + "bucket": 211.0, + "duration_ms": 5952.138712999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 215.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 21433.0 + }, + { + "bucket": 488.0, + "duration_ms": 8490.797738000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 354.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 25206.0 + }, + { + "bucket": 610.0, + "duration_ms": 6998.942739000001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 489.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 14062.0 + }, + { + "bucket": 423.0, + "duration_ms": 8712.311025000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 428.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 33204.0 + }, + { + "bucket": 822.0, + "duration_ms": 7311.043669000002, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 465.0, + "overflow": 2.0, + "padding": 3.0, + "tokens": 9758.0 + }, + { + "bucket": 699.0, + "duration_ms": 3225.8468359999983, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 333.0, + "overflow": 1.0, + "padding": 1.0, + "tokens": 1714.0 + }, + { + "bucket": 990.0, + "duration_ms": 9264.891035000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 445.0, + "overflow": 4.0, + "padding": 4.0, + "tokens": 28155.0 + }, + { + "bucket": 466.0, + "duration_ms": 9110.875763999999, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 401.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 31929.0 + }, + { + "bucket": 1071.0, + "duration_ms": 6715.519883000001, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 382.0, + "overflow": 2.0, + "padding": 18.0, + "tokens": 7012.0 + }, + { + "bucket": 550.0, + "duration_ms": 3031.6539789999974, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 329.0, + "overflow": 1.0, + "padding": 8.0, + "tokens": 1755.0 + }, + { + "bucket": 376.0, + "duration_ms": 5516.428338, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 379.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 14766.0 + }, + { + "bucket": 1075.0, + "duration_ms": 7219.809630999999, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 480.0, + "overflow": 3.0, + "padding": 13.0, + "tokens": 19060.0 + }, + { + "bucket": 337.0, + "duration_ms": 9648.578009, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 341.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 34101.0 + }, + { + "bucket": 946.0, + "duration_ms": 8697.491265, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 480.0, + "overflow": 3.0, + "padding": 86.0, + "tokens": 17701.0 + }, + { + "bucket": 554.0, + "duration_ms": 9683.282806000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 559.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 32266.0 + }, + { + "bucket": 678.0, + "duration_ms": 7048.451871000006, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 545.0, + "overflow": 1.0, + "padding": 3.0, + "tokens": 7876.0 + }, + { + "bucket": 646.0, + "duration_ms": 8348.867864999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 649.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 15839.0 + }, + { + "bucket": 256.0, + "duration_ms": 7786.6034809999965, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 260.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 25371.0 + }, + { + "bucket": 740.0, + "duration_ms": 9608.457868999996, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 191.0, + "overflow": 6.0, + "padding": 182.0, + "tokens": 33852.0 + }, + { + "bucket": 1045.0, + "duration_ms": 9948.261445000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 399.0, + "overflow": 4.0, + "padding": 101.0, + "tokens": 21987.0 + }, + { + "bucket": 499.0, + "duration_ms": 5516.342829999998, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 502.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 10171.0 + }, + { + "bucket": 414.0, + "duration_ms": 8107.245063000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 418.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 26442.0 + }, + { + "bucket": 920.0, + "duration_ms": 7048.238840000001, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 501.0, + "overflow": 1.0, + "padding": 1.0, + "tokens": 4144.0 + }, + { + "bucket": 335.0, + "duration_ms": 5664.3669340000015, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 338.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 16438.0 + }, + { + "bucket": 984.0, + "duration_ms": 8897.687465000003, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 340.0, + "overflow": 4.0, + "padding": 34.0, + "tokens": 26851.0 + }, + { + "bucket": 951.0, + "duration_ms": 5888.0405209999935, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 494.0, + "overflow": 1.0, + "padding": 13.0, + "tokens": 7545.0 + }, + { + "bucket": 635.0, + "duration_ms": 6843.962990000004, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 637.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 10109.0 + }, + { + "bucket": 326.0, + "duration_ms": 10501.115293, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 332.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 36220.0 + }, + { + "bucket": 935.0, + "duration_ms": 6618.986783999992, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 494.0, + "overflow": 3.0, + "padding": 95.0, + "tokens": 3782.0 + }, + { + "bucket": 512.0, + "duration_ms": 8131.997052, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 490.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 22549.0 + }, + { + "bucket": 742.0, + "duration_ms": 9767.929607000004, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 414.0, + "overflow": 3.0, + "padding": 5.0, + "tokens": 23222.0 + }, + { + "bucket": 639.0, + "duration_ms": 5619.278247999995, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 609.0, + "overflow": 2.0, + "padding": 4.0, + "tokens": 2804.0 + }, + { + "bucket": 887.0, + "duration_ms": 3543.163819999999, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 250.0, + "overflow": 1.0, + "padding": 3.0, + "tokens": 9076.0 + }, + { + "bucket": 576.0, + "duration_ms": 8349.781138999993, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 580.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 21628.0 + }, + { + "bucket": 457.0, + "duration_ms": 11131.830392999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 432.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 39509.0 + }, + { + "bucket": 564.0, + "duration_ms": 7224.973540000003, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 138.0, + "overflow": 2.0, + "padding": 63.0, + "tokens": 12290.0 + } + ], + "clean": { + "admitted": 113, + "completed": 111, + "completed_throughput_rps": 0.4625, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 968714, + "offered_rps": 0.4708333333333333, + "output_tokens": 25409, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 1.3658369119802956, + "latency_s": { + "e2e": { + "mean": 3.3829411591981593, + "n": 111, + "p50": 2.0798101979889907, + "p95": 10.049355082010152, + "p99": 13.440132959096816 + }, + "tpot": { + "mean": 0.010226489478255945, + "n": 111, + "p50": 0.0059010086391650726, + "p95": 0.029376378439209316, + "p99": 0.04181571422473475 + }, + "ttft": { + "mean": 0.9392844953882059, + "n": 111, + "p50": 0.6116193659836426, + "p95": 3.0904414410033496, + "p99": 4.187785376003013 + } + }, + "layer1": { + "decode_tokens": 25704, + "kv_usage_max": 0.38664503919978377, + "kv_usage_mean": 0.04570683788470549, + "mode_counts": { + "FULL": 17941, + "NONE": 283, + "PIECEWISE": 22 + }, + "mode_shares": { + "FULL": 0.9832840074536885, + "NONE": 0.015510248821659542, + "PIECEWISE": 0.0012057437246519786 + }, + "preemptions": 0, + "prefill_tokens": 977841, + "prefix_query_hit_ratio": 0.017412117941776724, + "queue_waiting_max": 1, + "queue_waiting_mean": 5.4806532938726295e-05, + "records_all": 26343, + "records_clean": 18246, + "requests_per_step": { + "mean": 1.4192151704483174, + "n": 18246, + "p50": 1.0, + "p95": 4.0, + "p99": 6.0 + }, + "scheduled_tokens_per_step": { + "mean": 55.00082209799408, + "n": 18246, + "p50": 1.0, + "p95": 4.0, + "p99": 59.29999999998108 + }, + "step_duration_ms": { + "mean": 20.99978799101173, + "n": 18246, + "p50": 9.6838665, + "p95": 22.50289075, + "p99": 282.73305594999846 + }, + "token_efficiency_per_ms": 2.6191132082635966 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.5, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.6, + "completion_rate_within_10pct": false, + "tail_end_s": 331.62960390897933, + "tail_start_s": 321.62960390897933, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.5, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": false, + "tail_end_s": 363.15318555399426, + "tail_start_s": 353.15318555399426, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 111, + "run_dir": "runs/phase3/primary/P10-C00/moderate", + "run_id": "P10-C00-moderate", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.05565211993295673, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.009211252068394925, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 92, + "moe_layer_cv": null, + "overflow_rate": 0.009211252068394925, + "padded_tokens_per_useful_token": 0.0017866662680796576, + "padding_fraction": 0.05565211993295673 + } + }, + "P10-C00-saturation": { + "blocks": [ + { + "bucket": 1176.0, + "duration_ms": 10009.811425999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 53.0, + "overflow": 4.0, + "padding": 343.0, + "tokens": 26342.0 + }, + { + "bucket": 2128.0, + "duration_ms": 9992.710429, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 92.0, + "overflow": 3.0, + "padding": 600.0, + "tokens": 24128.0 + }, + { + "bucket": 528.0, + "duration_ms": 11089.164078, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 31.0, + "overflow": 9.0, + "padding": 111.0, + "tokens": 67385.0 + }, + { + "bucket": 48.0, + "duration_ms": 9556.7498, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 6.0, + "tokens": 70830.0 + }, + { + "duration_ms": 10951.74559, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 9388.825067, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 59522.0 + }, + { + "bucket": 6592.0, + "duration_ms": 9039.374405999999, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 107.0, + "overflow": 1.0, + "padding": 249.0, + "tokens": 8794.0 + }, + { + "bucket": 4312.0, + "duration_ms": 11316.996691999995, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 84.0, + "overflow": 7.0, + "padding": 188.0, + "tokens": 43138.0 + }, + { + "bucket": 1736.0, + "duration_ms": 9279.988393, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 38.0, + "overflow": 7.0, + "padding": 99.0, + "tokens": 50096.0 + }, + { + "duration_ms": 9763.796564999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 8.0, + "overflow": 8.0, + "tokens": 61672.0 + }, + { + "bucket": 48.0, + "duration_ms": 10376.027167, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 10.0, + "overflow": 9.0, + "padding": 2.0, + "tokens": 70629.0 + }, + { + "duration_ms": 10483.151265999999, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 68074.0 + }, + { + "bucket": 5544.0, + "duration_ms": 8734.690442000003, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 105.0, + "overflow": 2.0, + "padding": 266.0, + "tokens": 10348.0 + }, + { + "bucket": 2784.0, + "duration_ms": 10805.742734, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 62.0, + "overflow": 4.0, + "padding": 226.0, + "tokens": 29614.0 + }, + { + "bucket": 1208.0, + "duration_ms": 9140.551409, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 30.0, + "overflow": 4.0, + "padding": 140.0, + "tokens": 27875.0 + }, + { + "bucket": 1184.0, + "duration_ms": 11310.472485, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 36.0, + "overflow": 6.0, + "padding": 74.0, + "tokens": 44653.0 + }, + { + "duration_ms": 9714.580936, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "bucket": 320.0, + "duration_ms": 10471.643823999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 8.0, + "overflow": 5.0, + "padding": 7.0, + "tokens": 41273.0 + }, + { + "duration_ms": 10020.970553999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "duration_ms": 10602.244435999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 6.0, + "overflow": 6.0, + "tokens": 49152.0 + }, + { + "bucket": 144.0, + "duration_ms": 8836.206753999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 14.0, + "overflow": 5.0, + "padding": 20.0, + "tokens": 40196.0 + }, + { + "bucket": 2032.0, + "duration_ms": 9019.412542, + "eligible_miss": 0.0, + "miss": 0.0, + "model": 127.0, + "overflow": 0.0, + "padding": 329.0, + "tokens": 1703.0 + }, + { + "bucket": 944.0, + "duration_ms": 10821.127354, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 65.0, + "overflow": 6.0, + "padding": 173.0, + "tokens": 44294.0 + }, + { + "bucket": 624.0, + "duration_ms": 9594.954417999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 25.0, + "overflow": 5.0, + "padding": 20.0, + "tokens": 35181.0 + }, + { + "bucket": 48.0, + "duration_ms": 11813.587782999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 10.0, + "overflow": 7.0, + "padding": 3.0, + "tokens": 56898.0 + }, + { + "bucket": 80.0, + "duration_ms": 9131.614881000001, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 12.0, + "overflow": 7.0, + "padding": 1.0, + "tokens": 55677.0 + }, + { + "duration_ms": 9896.023617, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 6.0, + "overflow": 6.0, + "tokens": 49152.0 + }, + { + "bucket": 168.0, + "duration_ms": 9331.333203999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 13.0, + "overflow": 6.0, + "padding": 43.0, + "tokens": 46277.0 + }, + { + "bucket": 1752.0, + "duration_ms": 9416.513950000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 76.0, + "overflow": 3.0, + "padding": 439.0, + "tokens": 18675.0 + }, + { + "bucket": 1848.0, + "duration_ms": 10004.734847999993, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 80.0, + "overflow": 3.0, + "padding": 463.0, + "tokens": 21615.0 + }, + { + "bucket": 736.0, + "duration_ms": 11113.904184, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 38.0, + "overflow": 7.0, + "padding": 186.0, + "tokens": 51247.0 + }, + { + "bucket": 424.0, + "duration_ms": 10244.33515, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 24.0, + "overflow": 6.0, + "padding": 119.0, + "tokens": 44715.0 + }, + { + "bucket": 144.0, + "duration_ms": 10147.747402, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 9.0, + "overflow": 5.0, + "padding": 21.0, + "tokens": 41083.0 + }, + { + "bucket": 112.0, + "duration_ms": 10365.436959, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 11.0, + "overflow": 5.0, + "padding": 15.0, + "tokens": 40790.0 + }, + { + "bucket": 64.0, + "duration_ms": 9517.532987999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 9.0, + "overflow": 5.0, + "padding": 4.0, + "tokens": 41020.0 + }, + { + "bucket": 832.0, + "duration_ms": 8581.742997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 26.0, + "overflow": 4.0, + "padding": 54.0, + "tokens": 26461.0 + }, + { + "bucket": 1536.0, + "duration_ms": 10026.462834999997, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 99.0, + "overflow": 3.0, + "padding": 221.0, + "tokens": 19754.0 + }, + { + "bucket": 128.0, + "duration_ms": 10718.594011999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 14.0, + "overflow": 6.0, + "padding": 24.0, + "tokens": 44891.0 + }, + { + "bucket": 960.0, + "duration_ms": 11535.182276, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 66.0, + "overflow": 6.0, + "padding": 166.0, + "tokens": 38927.0 + }, + { + "bucket": 752.0, + "duration_ms": 7739.217905, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 23.0, + "overflow": 3.0, + "padding": 82.0, + "tokens": 25246.0 + }, + { + "bucket": 128.0, + "duration_ms": 11694.124687000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 14.0, + "overflow": 6.0, + "padding": 33.0, + "tokens": 48548.0 + }, + { + "bucket": 144.0, + "duration_ms": 8799.382035999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 14.0, + "overflow": 5.0, + "padding": 38.0, + "tokens": 37550.0 + }, + { + "duration_ms": 12679.302825, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 6.0, + "overflow": 6.0, + "tokens": 49152.0 + }, + { + "bucket": 64.0, + "duration_ms": 8103.778363, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 8.0, + "overflow": 4.0, + "padding": 17.0, + "tokens": 29351.0 + }, + { + "bucket": 1584.0, + "duration_ms": 8692.517480999999, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 100.0, + "overflow": 1.0, + "padding": 429.0, + "tokens": 2044.0 + }, + { + "bucket": 352.0, + "duration_ms": 11405.081409999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 28.0, + "overflow": 6.0, + "padding": 111.0, + "tokens": 49393.0 + }, + { + "bucket": 848.0, + "duration_ms": 11003.100921, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 58.0, + "overflow": 5.0, + "padding": 213.0, + "tokens": 31053.0 + }, + { + "bucket": 144.0, + "duration_ms": 8373.303938000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 15.0, + "overflow": 6.0, + "padding": 18.0, + "tokens": 43909.0 + } + ], + "clean": { + "admitted": 189, + "completed": 189, + "completed_throughput_rps": 0.7875, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1905018, + "offered_rps": 0.7875, + "output_tokens": 43531, + "start_s": 60.0 + }, + "config": "C00", + "drain_quarantined": false, + "drain_seconds": 264.76528122398304, + "latency_s": { + "e2e": { + "mean": 147.21125181689527, + "n": 189, + "p50": 127.2441149209917, + "p95": 264.33665179300004, + "p99": 293.5073658543558 + }, + "tpot": { + "mean": 0.13403168434781512, + "n": 189, + "p50": 0.12877863171766055, + "p95": 0.17072597033296066, + "p99": 0.34752630997851625 + }, + "ttft": { + "mean": 117.40516835215816, + "n": 189, + "p50": 105.56007634400157, + "p95": 230.40205931000526, + "p99": 254.36873530238168 + } + }, + "layer1": { + "decode_tokens": 49088, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9654548095061397, + "mode_counts": { + "FULL": 1444, + "NONE": 258, + "PIECEWISE": 5 + }, + "mode_shares": { + "FULL": 0.8459285295840656, + "NONE": 0.15114235500878734, + "PIECEWISE": 0.0029291154071470417 + }, + "preemptions": 3, + "prefill_tokens": 1894887, + "prefix_query_hit_ratio": 0.00428220027136565, + "queue_waiting_max": 245, + "queue_waiting_mean": 230.05799648506152, + "records_all": 4556, + "records_clean": 1707, + "requests_per_step": { + "mean": 25.708845928529584, + "n": 1707, + "p50": 17.0, + "p95": 57.0, + "p99": 63.0 + }, + "scheduled_tokens_per_step": { + "mean": 1138.825424721734, + "n": 1707, + "p50": 18.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 281.5790834323375, + "n": 1707, + "p50": 74.93868, + "p95": 1529.4904996999996, + "p99": 2280.0959513600005 + }, + "token_efficiency_per_ms": 4.044424787664991 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.35, + "clean_c_waiting_median": 243.0, + "completion_rate_rps": 0.7, + "completion_rate_within_10pct": false, + "tail_end_s": 332.293171702011, + "tail_start_s": 322.293171702011, + "valid": false, + "waiting_median": 218.0, + "waiting_within_10pct": false, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.35, + "clean_c_waiting_median": 243.0, + "completion_rate_rps": 0.6, + "completion_rate_within_10pct": false, + "tail_end_s": 367.90418185602175, + "tail_start_s": 357.90418185602175, + "valid": false, + "waiting_median": 233.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 189, + "run_dir": "runs/phase3/primary/P10-C00/saturation", + "run_id": "P10-C00-saturation", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.12563348416289594, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.15114235500878734, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.15114235500878734, + "padded_tokens_per_useful_token": 0.0028565182165408505, + "padding_fraction": 0.12563348416289594 + } + }, + "P10-C01-moderate": { + "blocks": [ + { + "bucket": 443.0, + "duration_ms": 7410.423085999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 453.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 19475.0 + }, + { + "bucket": 521.0, + "duration_ms": 8271.043127999998, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 533.0, + "overflow": 12.0, + "padding": 0.0, + "tokens": 22979.0 + }, + { + "duration_ms": 9669.505019, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 19.0, + "overflow": 19.0, + "tokens": 38912.0 + }, + { + "bucket": 148.0, + "duration_ms": 9878.223238999995, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 40.0, + "overflow": 21.0, + "padding": 54.0, + "tokens": 40191.0 + }, + { + "bucket": 1408.0, + "duration_ms": 10418.856203999998, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 188.0, + "overflow": 12.0, + "padding": 452.0, + "tokens": 24661.0 + }, + { + "bucket": 720.0, + "duration_ms": 9851.350958, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 245.0, + "overflow": 16.0, + "padding": 105.0, + "tokens": 31329.0 + }, + { + "bucket": 736.0, + "duration_ms": 9674.858718000003, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 127.0, + "overflow": 14.0, + "padding": 95.0, + "tokens": 27804.0 + }, + { + "bucket": 827.0, + "duration_ms": 6290.920664999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 505.0, + "overflow": 3.0, + "padding": 46.0, + "tokens": 5131.0 + }, + { + "bucket": 715.0, + "duration_ms": 6676.0982399999975, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 326.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 21195.0 + }, + { + "duration_ms": 10065.836480000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 34476.0 + }, + { + "bucket": 784.0, + "duration_ms": 9484.886980999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 137.0, + "overflow": 15.0, + "padding": 24.0, + "tokens": 30505.0 + }, + { + "bucket": 1144.0, + "duration_ms": 10797.808420000001, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 166.0, + "overflow": 13.0, + "padding": 249.0, + "tokens": 27519.0 + }, + { + "bucket": 1316.0, + "duration_ms": 9174.807360999997, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 235.0, + "overflow": 12.0, + "padding": 74.0, + "tokens": 25760.0 + }, + { + "bucket": 1121.0, + "duration_ms": 6238.404124999998, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 547.0, + "overflow": 2.0, + "padding": 28.0, + "tokens": 3805.0 + }, + { + "bucket": 326.0, + "duration_ms": 7573.451611000008, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 267.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 24980.0 + }, + { + "bucket": 986.0, + "duration_ms": 10151.806548, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 331.0, + "overflow": 16.0, + "padding": 3.0, + "tokens": 33019.0 + }, + { + "bucket": 515.0, + "duration_ms": 5298.552034000004, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 518.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 4947.0 + }, + { + "bucket": 345.0, + "duration_ms": 8100.068032000006, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 239.0, + "overflow": 15.0, + "padding": 2.0, + "tokens": 30160.0 + }, + { + "bucket": 936.0, + "duration_ms": 7808.381264000005, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 601.0, + "overflow": 5.0, + "padding": 3.0, + "tokens": 9872.0 + }, + { + "bucket": 544.0, + "duration_ms": 2734.374267999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 182.0, + "overflow": 5.0, + "padding": 1.0, + "tokens": 9751.0 + }, + { + "bucket": 1125.0, + "duration_ms": 9574.814693000002, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 516.0, + "overflow": 10.0, + "padding": 4.0, + "tokens": 20098.0 + }, + { + "bucket": 331.0, + "duration_ms": 7593.752275999999, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 340.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 25650.0 + }, + { + "bucket": 1070.0, + "duration_ms": 7976.790719999997, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 385.0, + "overflow": 6.0, + "padding": 18.0, + "tokens": 13156.0 + }, + { + "bucket": 503.0, + "duration_ms": 2601.4658710000012, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 282.0, + "overflow": 1.0, + "padding": 8.0, + "tokens": 1708.0 + }, + { + "bucket": 302.0, + "duration_ms": 4862.400702999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 310.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 14692.0 + }, + { + "bucket": 844.0, + "duration_ms": 7449.2967190000045, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 602.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 10650.0 + }, + { + "bucket": 608.0, + "duration_ms": 7460.192439999997, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 270.0, + "overflow": 13.0, + "padding": 13.0, + "tokens": 26433.0 + }, + { + "bucket": 794.0, + "duration_ms": 9665.416551, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 383.0, + "overflow": 12.0, + "padding": 86.0, + "tokens": 24648.0 + }, + { + "bucket": 545.0, + "duration_ms": 8619.111659999997, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 527.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 25532.0 + }, + { + "bucket": 719.0, + "duration_ms": 9468.548906999986, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 608.0, + "overflow": 8.0, + "padding": 3.0, + "tokens": 16480.0 + }, + { + "bucket": 432.0, + "duration_ms": 6951.234554999997, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 442.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 18453.0 + }, + { + "bucket": 938.0, + "duration_ms": 6602.597068, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 654.0, + "overflow": 3.0, + "padding": 11.0, + "tokens": 5931.0 + }, + { + "duration_ms": 9777.494176999999, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 43529.0 + }, + { + "bucket": 1010.0, + "duration_ms": 10071.462241000005, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 281.0, + "overflow": 13.0, + "padding": 188.0, + "tokens": 27103.0 + }, + { + "bucket": 958.0, + "duration_ms": 8959.625635999993, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 528.0, + "overflow": 7.0, + "padding": 92.0, + "tokens": 13677.0 + }, + { + "bucket": 444.0, + "duration_ms": 4595.177759999996, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 447.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 6068.0 + }, + { + "bucket": 255.0, + "duration_ms": 7323.852219999997, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 267.0, + "overflow": 12.0, + "padding": 0.0, + "tokens": 24473.0 + }, + { + "bucket": 927.0, + "duration_ms": 7674.425592000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 587.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 6033.0 + }, + { + "bucket": 255.0, + "duration_ms": 4658.851812999998, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 263.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 15325.0 + }, + { + "bucket": 952.0, + "duration_ms": 9628.718707000005, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 323.0, + "overflow": 14.0, + "padding": 45.0, + "tokens": 27847.0 + }, + { + "bucket": 1378.0, + "duration_ms": 5780.386532000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 462.0, + "overflow": 3.0, + "padding": 13.0, + "tokens": 7509.0 + }, + { + "bucket": 698.0, + "duration_ms": 6068.889615000002, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 545.0, + "overflow": 5.0, + "padding": 6.0, + "tokens": 10027.0 + }, + { + "bucket": 562.0, + "duration_ms": 8098.781943, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 494.0, + "overflow": 11.0, + "padding": 7.0, + "tokens": 23083.0 + }, + { + "bucket": 791.0, + "duration_ms": 9644.798028000012, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 375.0, + "overflow": 8.0, + "padding": 85.0, + "tokens": 15246.0 + }, + { + "bucket": 466.0, + "duration_ms": 6499.454319999998, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 473.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 14168.0 + }, + { + "bucket": 447.0, + "duration_ms": 10338.326678999998, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 354.0, + "overflow": 16.0, + "padding": 5.0, + "tokens": 32959.0 + }, + { + "bucket": 905.0, + "duration_ms": 6872.5209749999885, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 637.0, + "overflow": 1.0, + "padding": 4.0, + "tokens": 1848.0 + }, + { + "bucket": 520.0, + "duration_ms": 2768.948879, + "eligible_miss": 0.0, + "miss": 1.0, + "model": 298.0, + "overflow": 1.0, + "padding": 1.0, + "tokens": 1741.0 + } + ], + "clean": { + "admitted": 107, + "completed": 107, + "completed_throughput_rps": 0.44583333333333336, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 933292, + "offered_rps": 0.44583333333333336, + "output_tokens": 24329, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 0.3054194079886656, + "latency_s": { + "e2e": { + "mean": 3.4583488524498125, + "n": 107, + "p50": 2.139935596002033, + "p95": 9.892479610099684, + "p99": 14.278773261667808 + }, + "tpot": { + "mean": 0.010096695599975438, + "n": 107, + "p50": 0.00563074907849488, + "p95": 0.029468602368621807, + "p99": 0.04342289214143664 + }, + "ttft": { + "mean": 1.0536771315690152, + "n": 107, + "p50": 0.7089335940254387, + "p95": 3.353996323794126, + "p99": 4.558524873343412 + } + }, + "layer1": { + "decode_tokens": 24222, + "kv_usage_max": 0.3808813342569404, + "kv_usage_mean": 0.044735391515669486, + "mode_counts": { + "FULL": 16854, + "NONE": 585, + "PIECEWISE": 28 + }, + "mode_shares": { + "FULL": 0.9649052498998111, + "NONE": 0.03349172725711341, + "PIECEWISE": 0.0016030228430755139 + }, + "preemptions": 0, + "prefill_tokens": 916316, + "prefix_query_hit_ratio": 0.018189376958122432, + "queue_waiting_max": 1, + "queue_waiting_mean": 0.0010877655006583844, + "records_all": 24251, + "records_clean": 17467, + "requests_per_step": { + "mean": 1.4158126753306235, + "n": 17467, + "p50": 1.0, + "p95": 3.0, + "p99": 6.0 + }, + "scheduled_tokens_per_step": { + "mean": 53.84656781359134, + "n": 17467, + "p50": 1.0, + "p95": 4.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 21.36354231757028, + "n": 17467, + "p50": 9.592219, + "p95": 28.20071580000003, + "p99": 409.62446344 + }, + "token_efficiency_per_ms": 2.520488737923657 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.45, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": false, + "tail_end_s": 338.4903358910233, + "tail_start_s": 328.4903358910233, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.45, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.9, + "completion_rate_within_10pct": false, + "tail_end_s": 381.45741498301504, + "tail_start_s": 371.45741498301504, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 107, + "run_dir": "runs/phase3/primary/P10-C01/moderate", + "run_id": "P10-C01-moderate", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.0533824348579563, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.0270301423549075, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 226, + "moe_layer_cv": null, + "overflow_rate": 0.0270301423549075, + "padded_tokens_per_useful_token": 0.0018340566781990732, + "padding_fraction": 0.0533824348579563 + } + }, + "P10-C01-saturation": { + "blocks": [ + { + "duration_ms": 9618.323389999998, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 34816.0 + }, + { + "bucket": 568.0, + "duration_ms": 9715.450135999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 32.0, + "overflow": 15.0, + "padding": 83.0, + "tokens": 31205.0 + }, + { + "bucket": 2416.0, + "duration_ms": 10303.149564000001, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 88.0, + "overflow": 11.0, + "padding": 471.0, + "tokens": 24119.0 + }, + { + "duration_ms": 9830.213772, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 18.0, + "overflow": 18.0, + "tokens": 36864.0 + }, + { + "duration_ms": 10602.143477999998, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 45056.0 + }, + { + "duration_ms": 9267.446006, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 33346.0 + }, + { + "bucket": 320.0, + "duration_ms": 10856.284119, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 35.0, + "overflow": 15.0, + "padding": 41.0, + "tokens": 30939.0 + }, + { + "bucket": 256.0, + "duration_ms": 9228.995142999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 31.0, + "overflow": 15.0, + "padding": 46.0, + "tokens": 30912.0 + }, + { + "duration_ms": 10118.542377999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 15.0, + "overflow": 15.0, + "tokens": 30720.0 + }, + { + "duration_ms": 10201.908828999998, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 18.0, + "overflow": 18.0, + "tokens": 36864.0 + }, + { + "duration_ms": 9753.736368, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 44505.0 + }, + { + "bucket": 152.0, + "duration_ms": 9841.631798, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 30.0, + "overflow": 22.0, + "padding": 16.0, + "tokens": 44528.0 + }, + { + "bucket": 1512.0, + "duration_ms": 10504.794488, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 76.0, + "overflow": 13.0, + "padding": 311.0, + "tokens": 27062.0 + }, + { + "bucket": 48.0, + "duration_ms": 9442.621208999999, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 21.0, + "overflow": 19.0, + "padding": 6.0, + "tokens": 37606.0 + }, + { + "bucket": 760.0, + "duration_ms": 10248.626628999997, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 47.0, + "overflow": 22.0, + "padding": 108.0, + "tokens": 44547.0 + }, + { + "bucket": 1240.0, + "duration_ms": 10165.940051, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 47.0, + "overflow": 15.0, + "padding": 99.0, + "tokens": 31861.0 + }, + { + "duration_ms": 9696.808636, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 23.0, + "overflow": 23.0, + "tokens": 47104.0 + }, + { + "duration_ms": 10186.123948, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 19.0, + "overflow": 19.0, + "tokens": 38912.0 + }, + { + "duration_ms": 10415.876993000002, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 44975.0 + }, + { + "bucket": 1760.0, + "duration_ms": 9292.007447999998, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 59.0, + "overflow": 15.0, + "padding": 53.0, + "tokens": 31127.0 + }, + { + "bucket": 720.0, + "duration_ms": 10396.926209, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 38.0, + "overflow": 20.0, + "padding": 79.0, + "tokens": 41152.0 + }, + { + "duration_ms": 10035.738295999998, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 16.0, + "overflow": 16.0, + "tokens": 31763.0 + }, + { + "bucket": 1088.0, + "duration_ms": 10139.922182, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 50.0, + "overflow": 18.0, + "padding": 51.0, + "tokens": 36827.0 + }, + { + "bucket": 1000.0, + "duration_ms": 9732.766099999997, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 38.0, + "overflow": 14.0, + "padding": 161.0, + "tokens": 29511.0 + }, + { + "bucket": 192.0, + "duration_ms": 10131.727447999998, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 23.0, + "overflow": 18.0, + "padding": 28.0, + "tokens": 36374.0 + }, + { + "bucket": 144.0, + "duration_ms": 9822.238614, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 26.0, + "overflow": 24.0, + "padding": 5.0, + "tokens": 49291.0 + }, + { + "duration_ms": 10355.003429999999, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "bucket": 1312.0, + "duration_ms": 9328.351725, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 53.0, + "overflow": 12.0, + "padding": 170.0, + "tokens": 24406.0 + }, + { + "bucket": 640.0, + "duration_ms": 10598.633892, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 35.0, + "overflow": 15.0, + "padding": 107.0, + "tokens": 31253.0 + }, + { + "duration_ms": 9896.93457, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 45056.0 + }, + { + "bucket": 920.0, + "duration_ms": 9851.330482999998, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 49.0, + "overflow": 18.0, + "padding": 157.0, + "tokens": 36871.0 + }, + { + "bucket": 312.0, + "duration_ms": 9662.383065000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 30.0, + "overflow": 17.0, + "padding": 6.0, + "tokens": 34549.0 + }, + { + "bucket": 384.0, + "duration_ms": 10609.498962, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 37.0, + "overflow": 21.0, + "padding": 17.0, + "tokens": 43375.0 + }, + { + "duration_ms": 9741.637745000002, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 18.0, + "overflow": 18.0, + "tokens": 36864.0 + }, + { + "duration_ms": 9892.446897000002, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 45056.0 + }, + { + "bucket": 896.0, + "duration_ms": 9733.218633999997, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 46.0, + "overflow": 18.0, + "padding": 196.0, + "tokens": 36052.0 + }, + { + "bucket": 984.0, + "duration_ms": 10332.206858000001, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 51.0, + "overflow": 20.0, + "padding": 210.0, + "tokens": 41734.0 + }, + { + "bucket": 96.0, + "duration_ms": 10533.172222, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 25.0, + "overflow": 22.0, + "padding": 10.0, + "tokens": 43712.0 + }, + { + "bucket": 448.0, + "duration_ms": 9116.097409999998, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 28.0, + "overflow": 14.0, + "padding": 42.0, + "tokens": 27607.0 + }, + { + "bucket": 256.0, + "duration_ms": 10697.825888, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 32.0, + "overflow": 24.0, + "padding": 28.0, + "tokens": 49380.0 + }, + { + "bucket": 640.0, + "duration_ms": 9922.849348, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 38.0, + "overflow": 18.0, + "padding": 97.0, + "tokens": 37145.0 + }, + { + "bucket": 480.0, + "duration_ms": 9694.890612999998, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 34.0, + "overflow": 19.0, + "padding": 90.0, + "tokens": 38612.0 + }, + { + "duration_ms": 9985.187363, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "bucket": 1728.0, + "duration_ms": 10054.963302000002, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 68.0, + "overflow": 14.0, + "padding": 299.0, + "tokens": 29357.0 + }, + { + "bucket": 160.0, + "duration_ms": 9795.101537999999, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 27.0, + "overflow": 22.0, + "padding": 31.0, + "tokens": 44721.0 + }, + { + "bucket": 128.0, + "duration_ms": 10214.997361, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 23.0, + "overflow": 19.0, + "padding": 17.0, + "tokens": 38461.0 + }, + { + "bucket": 448.0, + "duration_ms": 9722.990461, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 32.0, + "overflow": 18.0, + "padding": 7.0, + "tokens": 35935.0 + }, + { + "duration_ms": 10366.926333999998, + "eligible_miss": 0.0, + "miss": 24.0, + "model": 24.0, + "overflow": 24.0, + "tokens": 49152.0 + } + ], + "clean": { + "admitted": 178, + "completed": 178, + "completed_throughput_rps": 0.7416666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1755158, + "offered_rps": 0.7416666666666667, + "output_tokens": 40658, + "start_s": 60.0 + }, + "config": "C01", + "drain_quarantined": false, + "drain_seconds": 288.6191079240234, + "latency_s": { + "e2e": { + "mean": 183.04354939324844, + "n": 178, + "p50": 186.8344520800165, + "p95": 290.3771271756501, + "p99": 296.9446104261672 + }, + "tpot": { + "mean": 0.14786293761943992, + "n": 178, + "p50": 0.14534626581766596, + "p95": 0.17603060793336645, + "p99": 0.22781931117242266 + }, + "ttft": { + "mean": 149.49741162677012, + "n": 178, + "p50": 150.95995998900617, + "p95": 258.07463337168736, + "p99": 280.9327837695714 + } + }, + "layer1": { + "decode_tokens": 39630, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9423858731719604, + "mode_counts": { + "FULL": 697, + "NONE": 882, + "PIECEWISE": 5 + }, + "mode_shares": { + "FULL": 0.44002525252525254, + "NONE": 0.5568181818181818, + "PIECEWISE": 0.0031565656565656565 + }, + "preemptions": 1, + "prefill_tokens": 1763574, + "prefix_query_hit_ratio": 0.004381548750661162, + "queue_waiting_max": 244, + "queue_waiting_mean": 230.1294191919192, + "records_all": 4742, + "records_clean": 1584, + "requests_per_step": { + "mean": 25.63888888888889, + "n": 1584, + "p50": 26.0, + "p95": 35.0, + "p99": 39.0 + }, + "scheduled_tokens_per_step": { + "mean": 1138.3863636363637, + "n": 1584, + "p50": 2048.0, + "p95": 2048.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 302.8135046294192, + "n": 1584, + "p50": 324.637002, + "p95": 706.7920114499998, + "p99": 844.95236557 + }, + "token_efficiency_per_ms": 3.7593645799566042 + }, + "layer2_missing_after_controller_cleanup": true, + "load": "saturation", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.8, + "clean_c_waiting_median": 229.0, + "completion_rate_rps": 0.7, + "completion_rate_within_10pct": false, + "tail_end_s": 368.55468072599615, + "tail_start_s": 358.55468072599615, + "valid": false, + "waiting_median": 219.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.8, + "clean_c_waiting_median": 229.0, + "completion_rate_rps": 0.9, + "completion_rate_within_10pct": false, + "tail_end_s": 420.6437278209778, + "tail_start_s": 410.6437278209778, + "valid": false, + "waiting_median": 234.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 178, + "run_dir": "runs/phase3/primary/P10-C01/saturation", + "run_id": "P10-C01-saturation", + "trace_files": 0, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.13822246455834242, + "eligible_miss_rate": 0.0, + "graph_miss_rate": 0.5568181818181818, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.5568181818181818, + "padded_tokens_per_useful_token": 0.0016869971450817545, + "padding_fraction": 0.13822246455834242 + } + }, + "P10-C10-moderate": { + "blocks": [ + { + "bucket": 343.0, + "duration_ms": 10106.476507000005, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 347.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 34424.0 + }, + { + "bucket": 4.0, + "duration_ms": 11007.921244, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 8.0, + "overflow": 7.0, + "padding": 1.0, + "tokens": 53699.0 + }, + { + "bucket": 1136.0, + "duration_ms": 7384.894006999999, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 145.0, + "overflow": 3.0, + "padding": 339.0, + "tokens": 16975.0 + }, + { + "bucket": 1018.0, + "duration_ms": 11292.837419000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 194.0, + "overflow": 6.0, + "padding": 203.0, + "tokens": 43013.0 + }, + { + "bucket": 96.0, + "duration_ms": 8682.53226, + "eligible_miss": 1.0, + "miss": 6.0, + "model": 26.0, + "overflow": 5.0, + "padding": 28.0, + "tokens": 29653.0 + }, + { + "bucket": 1386.0, + "duration_ms": 7021.152411000001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 317.0, + "overflow": 3.0, + "padding": 287.0, + "tokens": 10081.0 + }, + { + "bucket": 509.0, + "duration_ms": 9115.488792000006, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 513.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 27533.0 + }, + { + "bucket": 4.0, + "duration_ms": 8336.767972, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 5.0, + "overflow": 4.0, + "padding": 1.0, + "tokens": 28327.0 + }, + { + "bucket": 828.0, + "duration_ms": 9504.892607000002, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 162.0, + "overflow": 5.0, + "padding": 101.0, + "tokens": 30693.0 + }, + { + "bucket": 936.0, + "duration_ms": 9976.587113999994, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 136.0, + "overflow": 4.0, + "padding": 184.0, + "tokens": 27456.0 + }, + { + "bucket": 1212.0, + "duration_ms": 9979.223648999996, + "eligible_miss": 1.0, + "miss": 5.0, + "model": 262.0, + "overflow": 4.0, + "padding": 192.0, + "tokens": 28329.0 + }, + { + "bucket": 472.0, + "duration_ms": 7446.108408999998, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 377.0, + "overflow": 4.0, + "padding": 19.0, + "tokens": 20125.0 + }, + { + "bucket": 391.0, + "duration_ms": 8153.021131000002, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 315.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 26659.0 + }, + { + "bucket": 739.0, + "duration_ms": 7842.656145999993, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 538.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 14193.0 + }, + { + "bucket": 408.0, + "duration_ms": 8552.172508000003, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 413.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 33189.0 + }, + { + "bucket": 874.0, + "duration_ms": 7561.893929999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 477.0, + "overflow": 2.0, + "padding": 12.0, + "tokens": 9802.0 + }, + { + "bucket": 302.0, + "duration_ms": 4306.8247440000005, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 306.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 12522.0 + }, + { + "bucket": 646.0, + "duration_ms": 8740.812153000004, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 495.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 17456.0 + }, + { + "bucket": 328.0, + "duration_ms": 9669.982888, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 324.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 32644.0 + }, + { + "bucket": 812.0, + "duration_ms": 6619.143540000003, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 513.0, + "overflow": 2.0, + "padding": 24.0, + "tokens": 7524.0 + }, + { + "bucket": 198.0, + "duration_ms": 1584.5358729999998, + "eligible_miss": 1.0, + "miss": 1.0, + "model": 176.0, + "overflow": 0.0, + "padding": 5.0, + "tokens": 390.0 + }, + { + "bucket": 499.0, + "duration_ms": 8041.176958999996, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 504.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 24697.0 + }, + { + "bucket": 536.0, + "duration_ms": 6431.412983999996, + "eligible_miss": 2.0, + "miss": 3.0, + "model": 530.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 9315.0 + }, + { + "bucket": 289.0, + "duration_ms": 9811.863633000004, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 217.0, + "overflow": 6.0, + "padding": 1.0, + "tokens": 41862.0 + }, + { + "bucket": 1109.0, + "duration_ms": 9629.157217999988, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 638.0, + "overflow": 3.0, + "padding": 111.0, + "tokens": 15744.0 + }, + { + "bucket": 538.0, + "duration_ms": 8930.29845700001, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 448.0, + "overflow": 4.0, + "padding": 3.0, + "tokens": 26534.0 + }, + { + "bucket": 607.0, + "duration_ms": 7727.710782000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 537.0, + "overflow": 3.0, + "padding": 0.0, + "tokens": 18628.0 + }, + { + "bucket": 710.0, + "duration_ms": 7073.704011, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 699.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 5992.0 + }, + { + "bucket": 169.0, + "duration_ms": 9870.707891999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 50.0, + "overflow": 7.0, + "padding": 42.0, + "tokens": 43650.0 + }, + { + "bucket": 836.0, + "duration_ms": 10003.146918, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 203.0, + "overflow": 5.0, + "padding": 149.0, + "tokens": 31768.0 + }, + { + "bucket": 1200.0, + "duration_ms": 8354.661415000008, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 574.0, + "overflow": 3.0, + "padding": 173.0, + "tokens": 10969.0 + }, + { + "bucket": 564.0, + "duration_ms": 5582.916378000002, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 566.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 5935.0 + }, + { + "bucket": 454.0, + "duration_ms": 9035.325433000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 254.0, + "overflow": 4.0, + "padding": 1.0, + "tokens": 28016.0 + }, + { + "bucket": 560.0, + "duration_ms": 5052.833035999991, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 558.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 1534.0 + }, + { + "bucket": 259.0, + "duration_ms": 9378.153805999998, + "eligible_miss": 1.0, + "miss": 7.0, + "model": 203.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 41925.0 + }, + { + "bucket": 837.0, + "duration_ms": 6503.318629000009, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 435.0, + "overflow": 1.0, + "padding": 75.0, + "tokens": 7678.0 + }, + { + "bucket": 616.0, + "duration_ms": 6539.7740289999965, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 603.0, + "overflow": 2.0, + "padding": 1.0, + "tokens": 10089.0 + }, + { + "bucket": 509.0, + "duration_ms": 10607.03604, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 514.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 35398.0 + }, + { + "bucket": 880.0, + "duration_ms": 7213.780934000007, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 393.0, + "overflow": 3.0, + "padding": 118.0, + "tokens": 3015.0 + }, + { + "bucket": 482.0, + "duration_ms": 8018.256782999996, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 480.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 24213.0 + }, + { + "bucket": 751.0, + "duration_ms": 9984.175938, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 438.0, + "overflow": 3.0, + "padding": 5.0, + "tokens": 23234.0 + }, + { + "bucket": 752.0, + "duration_ms": 6134.946822999995, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 635.0, + "overflow": 2.0, + "padding": 21.0, + "tokens": 2900.0 + }, + { + "bucket": 272.0, + "duration_ms": 3775.7959439999972, + "eligible_miss": 2.0, + "miss": 3.0, + "model": 275.0, + "overflow": 1.0, + "padding": 0.0, + "tokens": 9101.0 + }, + { + "bucket": 595.0, + "duration_ms": 8498.819438, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 570.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 21647.0 + }, + { + "bucket": 400.0, + "duration_ms": 10498.661573000001, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 390.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 39452.0 + }, + { + "bucket": 700.0, + "duration_ms": 9937.671109000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 174.0, + "overflow": 4.0, + "padding": 83.0, + "tokens": 28790.0 + }, + { + "bucket": 188.0, + "duration_ms": 10682.796628, + "eligible_miss": 1.0, + "miss": 6.0, + "model": 53.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 37936.0 + }, + { + "bucket": 1652.0, + "duration_ms": 7574.382696000001, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 234.0, + "overflow": 1.0, + "padding": 431.0, + "tokens": 9741.0 + } + ], + "clean": { + "admitted": 117, + "completed": 114, + "completed_throughput_rps": 0.475, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1011895, + "offered_rps": 0.4875, + "output_tokens": 26177, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 1.0034805840114132, + "latency_s": { + "e2e": { + "mean": 4.036499573850163, + "n": 114, + "p50": 2.565201995501411, + "p95": 11.757720911737124, + "p99": 15.403192080657472 + }, + "tpot": { + "mean": 0.012512976698821513, + "n": 114, + "p50": 0.006702121849636516, + "p95": 0.03670983474116142, + "p99": 0.04342372289472191 + }, + "ttft": { + "mean": 1.008607643877775, + "n": 114, + "p50": 0.6484979435044806, + "p95": 3.168752788902201, + "p99": 4.3141448924437285 + } + }, + "layer1": { + "decode_tokens": 26784, + "kv_usage_max": 0.35747826955739836, + "kv_usage_mean": 0.04747642230487483, + "mode_counts": { + "FULL": 17025, + "NONE": 293, + "PIECEWISE": 10 + }, + "mode_shares": { + "FULL": 0.9825138504155124, + "NONE": 0.016909048938134812, + "PIECEWISE": 0.0005771006463527239 + }, + "preemptions": 0, + "prefill_tokens": 1037666, + "prefix_query_hit_ratio": 0.01670807677073206, + "queue_waiting_max": 1, + "queue_waiting_mean": 5.771006463527239e-05, + "records_all": 25426, + "records_clean": 17328, + "requests_per_step": { + "mean": 1.5573060941828254, + "n": 17328, + "p50": 1.0, + "p95": 4.0, + "p99": 6.0 + }, + "scheduled_tokens_per_step": { + "mean": 61.4294783010157, + "n": 17328, + "p50": 1.0, + "p95": 5.0, + "p99": 560.6799999999494 + }, + "step_duration_ms": { + "mean": 22.724977538665744, + "n": 17328, + "p50": 9.6194405, + "p95": 25.861211249999997, + "p99": 402.42142650999864 + }, + "token_efficiency_per_ms": 2.7031700337875195 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.3, + "completion_rate_within_10pct": false, + "tail_end_s": 331.677197498997, + "tail_start_s": 321.677197498997, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.4, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.6, + "completion_rate_within_10pct": false, + "tail_end_s": 364.01766435598256, + "tail_start_s": 354.01766435598256, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 114, + "run_dir": "runs/phase3/primary/P10-C10/moderate", + "run_id": "P10-C10-moderate", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.07714860592124173, + "eligible_miss_rate": 0.0008708778448676265, + "graph_miss_rate": 0.010973060845332094, + "mixed_interference": null, + "mixed_supported_steps": 0, + "mixed_total_steps": 114, + "moe_layer_cv": null, + "overflow_rate": 0.010102183000464468, + "padded_tokens_per_useful_token": 0.0024519705011978015, + "padding_fraction": 0.08815780585016551 + } + }, + "P10-C10-saturation": { + "blocks": [ + { + "bucket": 6616.0, + "duration_ms": 9902.418747000002, + "eligible_miss": 3.0, + "miss": 5.0, + "model": 117.0, + "overflow": 2.0, + "padding": 499.0, + "tokens": 8832.0 + }, + { + "bucket": 4656.0, + "duration_ms": 11073.679255, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 104.0, + "overflow": 7.0, + "padding": 133.0, + "tokens": 34294.0 + }, + { + "duration_ms": 9774.869166999999, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 8.0, + "overflow": 8.0, + "tokens": 65536.0 + }, + { + "bucket": 112.0, + "duration_ms": 9892.910486, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 10.0, + "overflow": 8.0, + "padding": 2.0, + "tokens": 64002.0 + }, + { + "duration_ms": 10001.235723, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 9.0, + "overflow": 9.0, + "tokens": 73728.0 + }, + { + "duration_ms": 10119.018779000002, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 8.0, + "overflow": 8.0, + "tokens": 65536.0 + }, + { + "bucket": 4920.0, + "duration_ms": 9125.751893, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 81.0, + "overflow": 2.0, + "padding": 287.0, + "tokens": 19456.0 + }, + { + "bucket": 5600.0, + "duration_ms": 9984.559546000002, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 103.0, + "overflow": 3.0, + "padding": 320.0, + "tokens": 19133.0 + }, + { + "bucket": 1928.0, + "duration_ms": 12199.917546000002, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 43.0, + "overflow": 6.0, + "padding": 129.0, + "tokens": 48899.0 + }, + { + "duration_ms": 9820.404919, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "bucket": 32.0, + "duration_ms": 8602.427712, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 5.0, + "overflow": 4.0, + "padding": 0.0, + "tokens": 29474.0 + }, + { + "bucket": 32.0, + "duration_ms": 10829.544716999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 8.0, + "overflow": 7.0, + "padding": 5.0, + "tokens": 49947.0 + }, + { + "duration_ms": 11234.986391, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "duration_ms": 8382.891783, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "duration_ms": 9916.166873999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 6.0, + "overflow": 6.0, + "tokens": 42794.0 + }, + { + "bucket": 1472.0, + "duration_ms": 10955.237687999997, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 67.0, + "overflow": 5.0, + "padding": 420.0, + "tokens": 34910.0 + }, + { + "bucket": 472.0, + "duration_ms": 7994.8781, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 23.0, + "overflow": 3.0, + "padding": 133.0, + "tokens": 24594.0 + }, + { + "bucket": 896.0, + "duration_ms": 10001.742973000004, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 60.0, + "overflow": 4.0, + "padding": 55.0, + "tokens": 32428.0 + }, + { + "bucket": 1040.0, + "duration_ms": 11356.648647999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 70.0, + "overflow": 5.0, + "padding": 67.0, + "tokens": 33503.0 + }, + { + "bucket": 80.0, + "duration_ms": 9773.165826999999, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 12.0, + "overflow": 7.0, + "padding": 6.0, + "tokens": 57418.0 + }, + { + "bucket": 16.0, + "duration_ms": 9263.606479999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 7.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 39803.0 + }, + { + "bucket": 136.0, + "duration_ms": 11027.643027, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 13.0, + "overflow": 7.0, + "padding": 35.0, + "tokens": 57445.0 + }, + { + "bucket": 1032.0, + "duration_ms": 9844.611378999998, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 48.0, + "overflow": 5.0, + "padding": 259.0, + "tokens": 37190.0 + }, + { + "bucket": 504.0, + "duration_ms": 10220.950637, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 26.0, + "overflow": 5.0, + "padding": 106.0, + "tokens": 36540.0 + }, + { + "bucket": 1032.0, + "duration_ms": 8496.597829000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 47.0, + "overflow": 4.0, + "padding": 217.0, + "tokens": 29147.0 + }, + { + "bucket": 1320.0, + "duration_ms": 9977.614238999997, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 59.0, + "overflow": 4.0, + "padding": 276.0, + "tokens": 31867.0 + }, + { + "bucket": 720.0, + "duration_ms": 10088.512911999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 36.0, + "overflow": 6.0, + "padding": 151.0, + "tokens": 38179.0 + }, + { + "bucket": 384.0, + "duration_ms": 12066.516112, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 22.0, + "overflow": 6.0, + "padding": 75.0, + "tokens": 49461.0 + }, + { + "duration_ms": 10250.579179999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "bucket": 112.0, + "duration_ms": 10331.163991999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 11.0, + "overflow": 6.0, + "padding": 28.0, + "tokens": 41306.0 + }, + { + "duration_ms": 7934.342012, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 35929.0 + }, + { + "bucket": 624.0, + "duration_ms": 9344.787628000002, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 30.0, + "overflow": 4.0, + "padding": 182.0, + "tokens": 27843.0 + }, + { + "bucket": 792.0, + "duration_ms": 10257.251678999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 47.0, + "overflow": 6.0, + "padding": 120.0, + "tokens": 36425.0 + }, + { + "bucket": 912.0, + "duration_ms": 12426.491586999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 63.0, + "overflow": 6.0, + "padding": 48.0, + "tokens": 45258.0 + }, + { + "bucket": 1120.0, + "duration_ms": 7300.928393, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 72.0, + "overflow": 1.0, + "padding": 56.0, + "tokens": 7547.0 + }, + { + "bucket": 272.0, + "duration_ms": 11811.300586999998, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 23.0, + "overflow": 6.0, + "padding": 35.0, + "tokens": 47753.0 + }, + { + "duration_ms": 11055.91304, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 40960.0 + }, + { + "duration_ms": 8944.076589, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 5.0, + "overflow": 5.0, + "tokens": 39604.0 + }, + { + "bucket": 80.0, + "duration_ms": 9594.855515000001, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 11.0, + "overflow": 6.0, + "padding": 16.0, + "tokens": 47376.0 + }, + { + "bucket": 24.0, + "duration_ms": 9620.566718, + "eligible_miss": 0.0, + "miss": 7.0, + "model": 8.0, + "overflow": 7.0, + "padding": 7.0, + "tokens": 54810.0 + }, + { + "bucket": 1512.0, + "duration_ms": 9199.070156999998, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 66.0, + "overflow": 2.0, + "padding": 296.0, + "tokens": 11630.0 + }, + { + "bucket": 432.0, + "duration_ms": 10189.773732, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 24.0, + "overflow": 6.0, + "padding": 37.0, + "tokens": 47726.0 + }, + { + "bucket": 1576.0, + "duration_ms": 9587.827011, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 64.0, + "overflow": 4.0, + "padding": 163.0, + "tokens": 30393.0 + }, + { + "bucket": 1608.0, + "duration_ms": 9988.892738000002, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 63.0, + "overflow": 5.0, + "padding": 210.0, + "tokens": 36002.0 + }, + { + "bucket": 384.0, + "duration_ms": 11564.252193, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 20.0, + "overflow": 8.0, + "padding": 71.0, + "tokens": 64411.0 + }, + { + "bucket": 192.0, + "duration_ms": 10965.250629999999, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 12.0, + "overflow": 6.0, + "padding": 19.0, + "tokens": 41855.0 + }, + { + "duration_ms": 9069.412812, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 6.0, + "overflow": 6.0, + "tokens": 49152.0 + }, + { + "bucket": 1376.0, + "duration_ms": 8453.388135000001, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 46.0, + "overflow": 3.0, + "padding": 77.0, + "tokens": 23205.0 + } + ], + "clean": { + "admitted": 196, + "completed": 196, + "completed_throughput_rps": 0.8166666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1899551, + "offered_rps": 0.8166666666666667, + "output_tokens": 44004, + "start_s": 60.0 + }, + "config": "C10", + "drain_quarantined": false, + "drain_seconds": 256.107847391977, + "latency_s": { + "e2e": { + "mean": 144.9189226589494, + "n": 196, + "p50": 117.59595297249325, + "p95": 282.36017964924395, + "p99": 292.70826233768895 + }, + "tpot": { + "mean": 0.1367971753786876, + "n": 196, + "p50": 0.1362825763050598, + "p95": 0.18970892391759359, + "p99": 0.26770929005363897 + }, + "ttft": { + "mean": 114.34430171160724, + "n": 196, + "p50": 83.6976999160106, + "p95": 257.0734725165021, + "p99": 268.6144477106413 + } + }, + "layer1": { + "decode_tokens": 53087, + "kv_usage_max": 1.0, + "kv_usage_mean": 0.9697357124350795, + "mode_counts": { + "FULL": 1334, + "NONE": 259 + }, + "mode_shares": { + "FULL": 0.837413684871312, + "NONE": 0.162586315128688 + }, + "preemptions": 16, + "prefill_tokens": 1864054, + "prefix_query_hit_ratio": 0.004917128512585017, + "queue_waiting_max": 243, + "queue_waiting_mean": 226.5593220338983, + "records_all": 4180, + "records_clean": 1593, + "requests_per_step": { + "mean": 29.18706842435656, + "n": 1593, + "p50": 19.0, + "p95": 58.0, + "p99": 61.0 + }, + "scheduled_tokens_per_step": { + "mean": 1203.4783427495292, + "n": 1593, + "p50": 24.0, + "p95": 8192.0, + "p99": 8192.0 + }, + "step_duration_ms": { + "mean": 301.2044153904583, + "n": 1593, + "p50": 79.154278, + "p95": 1583.7362489999996, + "p99": 2282.78115692 + }, + "token_efficiency_per_ms": 3.9955534555807635 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.55, + "clean_c_waiting_median": 237.0, + "completion_rate_rps": 0.8, + "completion_rate_within_10pct": false, + "tail_end_s": 343.2937143140007, + "tail_start_s": 333.2937143140007, + "valid": false, + "waiting_median": 216.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.55, + "clean_c_waiting_median": 237.0, + "completion_rate_rps": 0.3, + "completion_rate_within_10pct": false, + "tail_end_s": 376.3072694450093, + "tail_start_s": 366.3072694450093, + "valid": false, + "waiting_median": 228.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 196, + "run_dir": "runs/phase3/primary/P10-C10/saturation", + "run_id": "P10-C10-saturation", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.10055066079295154, + "eligible_miss_rate": 0.003138731952291274, + "graph_miss_rate": 0.162586315128688, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.15944758317639673, + "padded_tokens_per_useful_token": 0.0023681095965294155, + "padding_fraction": 0.10314431115957834 + } + }, + "P10-C11-moderate": { + "blocks": [ + { + "bucket": 503.0, + "duration_ms": 8991.754106999995, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 510.0, + "overflow": 14.0, + "padding": 0.0, + "tokens": 27059.0 + }, + { + "duration_ms": 9836.927439, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "bucket": 120.0, + "duration_ms": 9811.257092, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 35.0, + "overflow": 20.0, + "padding": 44.0, + "tokens": 39413.0 + }, + { + "bucket": 1152.0, + "duration_ms": 9848.213607, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 157.0, + "overflow": 13.0, + "padding": 283.0, + "tokens": 26731.0 + }, + { + "bucket": 424.0, + "duration_ms": 10637.645715999999, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 73.0, + "overflow": 20.0, + "padding": 57.0, + "tokens": 39993.0 + }, + { + "bucket": 1512.0, + "duration_ms": 9332.632807999998, + "eligible_miss": 1.0, + "miss": 8.0, + "model": 225.0, + "overflow": 7.0, + "padding": 375.0, + "tokens": 13027.0 + }, + { + "bucket": 611.0, + "duration_ms": 6026.108009999988, + "eligible_miss": 1.0, + "miss": 4.0, + "model": 575.0, + "overflow": 3.0, + "padding": 10.0, + "tokens": 6608.0 + }, + { + "duration_ms": 9745.453402, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 34816.0 + }, + { + "bucket": 228.0, + "duration_ms": 9988.957895999996, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 78.0, + "overflow": 21.0, + "padding": 0.0, + "tokens": 42459.0 + }, + { + "bucket": 904.0, + "duration_ms": 10263.974203999996, + "eligible_miss": 1.0, + "miss": 15.0, + "model": 128.0, + "overflow": 14.0, + "padding": 226.0, + "tokens": 29075.0 + }, + { + "bucket": 528.0, + "duration_ms": 9166.581995999995, + "eligible_miss": 2.0, + "miss": 16.0, + "model": 72.0, + "overflow": 14.0, + "padding": 94.0, + "tokens": 29752.0 + }, + { + "bucket": 1842.0, + "duration_ms": 8078.520900999999, + "eligible_miss": 1.0, + "miss": 3.0, + "model": 437.0, + "overflow": 2.0, + "padding": 470.0, + "tokens": 4395.0 + }, + { + "bucket": 326.0, + "duration_ms": 8712.581120999988, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 270.0, + "overflow": 16.0, + "padding": 0.0, + "tokens": 31124.0 + }, + { + "bucket": 543.0, + "duration_ms": 9440.142476, + "eligible_miss": 1.0, + "miss": 14.0, + "model": 308.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 26918.0 + }, + { + "bucket": 510.0, + "duration_ms": 5693.341847, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 516.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 11086.0 + }, + { + "bucket": 755.0, + "duration_ms": 9750.186765999988, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 357.0, + "overflow": 13.0, + "padding": 49.0, + "tokens": 25127.0 + }, + { + "bucket": 628.0, + "duration_ms": 5739.3034739999985, + "eligible_miss": 1.0, + "miss": 5.0, + "model": 466.0, + "overflow": 4.0, + "padding": 3.0, + "tokens": 9184.0 + }, + { + "bucket": 316.0, + "duration_ms": 6905.508828000001, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 329.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 24457.0 + }, + { + "bucket": 707.0, + "duration_ms": 8045.125614000001, + "eligible_miss": 1.0, + "miss": 7.0, + "model": 516.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 12122.0 + }, + { + "bucket": 307.0, + "duration_ms": 9190.669360999995, + "eligible_miss": 1.0, + "miss": 16.0, + "model": 147.0, + "overflow": 15.0, + "padding": 36.0, + "tokens": 31224.0 + }, + { + "bucket": 644.0, + "duration_ms": 4659.236079000004, + "eligible_miss": 1.0, + "miss": 2.0, + "model": 452.0, + "overflow": 1.0, + "padding": 22.0, + "tokens": 2032.0 + }, + { + "bucket": 374.0, + "duration_ms": 3697.778438000003, + "eligible_miss": 0.0, + "miss": 3.0, + "model": 354.0, + "overflow": 3.0, + "padding": 5.0, + "tokens": 5989.0 + }, + { + "bucket": 590.0, + "duration_ms": 8927.317469999984, + "eligible_miss": 1.0, + "miss": 11.0, + "model": 575.0, + "overflow": 10.0, + "padding": 0.0, + "tokens": 19420.0 + }, + { + "bucket": 256.0, + "duration_ms": 7494.414503999998, + "eligible_miss": 1.0, + "miss": 14.0, + "model": 270.0, + "overflow": 13.0, + "padding": 0.0, + "tokens": 26433.0 + }, + { + "bucket": 749.0, + "duration_ms": 10068.411879000008, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 263.0, + "overflow": 17.0, + "padding": 147.0, + "tokens": 33581.0 + }, + { + "bucket": 739.0, + "duration_ms": 8692.409100999997, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 541.0, + "overflow": 9.0, + "padding": 37.0, + "tokens": 18533.0 + }, + { + "bucket": 722.0, + "duration_ms": 8879.877492, + "eligible_miss": 0.0, + "miss": 10.0, + "model": 529.0, + "overflow": 10.0, + "padding": 3.0, + "tokens": 20752.0 + }, + { + "bucket": 650.0, + "duration_ms": 8948.98821000001, + "eligible_miss": 1.0, + "miss": 10.0, + "model": 636.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 16903.0 + }, + { + "bucket": 367.0, + "duration_ms": 7598.144262000002, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 378.0, + "overflow": 11.0, + "padding": 0.0, + "tokens": 21755.0 + }, + { + "bucket": 484.0, + "duration_ms": 9681.970063000002, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 138.0, + "overflow": 17.0, + "padding": 121.0, + "tokens": 33652.0 + }, + { + "bucket": 952.0, + "duration_ms": 9716.605327000007, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 196.0, + "overflow": 13.0, + "padding": 201.0, + "tokens": 25588.0 + }, + { + "bucket": 869.0, + "duration_ms": 6559.674515999999, + "eligible_miss": 0.0, + "miss": 5.0, + "model": 473.0, + "overflow": 5.0, + "padding": 128.0, + "tokens": 10417.0 + }, + { + "bucket": 495.0, + "duration_ms": 7574.075480000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 504.0, + "overflow": 9.0, + "padding": 0.0, + "tokens": 18569.0 + }, + { + "bucket": 597.0, + "duration_ms": 8411.306683000006, + "eligible_miss": 0.0, + "miss": 6.0, + "model": 432.0, + "overflow": 6.0, + "padding": 0.0, + "tokens": 11847.0 + }, + { + "bucket": 345.0, + "duration_ms": 5015.054257000001, + "eligible_miss": 1.0, + "miss": 8.0, + "model": 353.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 13607.0 + }, + { + "bucket": 538.0, + "duration_ms": 9609.595639999996, + "eligible_miss": 1.0, + "miss": 16.0, + "model": 251.0, + "overflow": 15.0, + "padding": 87.0, + "tokens": 29841.0 + }, + { + "bucket": 832.0, + "duration_ms": 6853.801568000002, + "eligible_miss": 2.0, + "miss": 5.0, + "model": 528.0, + "overflow": 3.0, + "padding": 70.0, + "tokens": 7678.0 + }, + { + "bucket": 648.0, + "duration_ms": 7249.697015, + "eligible_miss": 1.0, + "miss": 6.0, + "model": 654.0, + "overflow": 5.0, + "padding": 0.0, + "tokens": 10122.0 + }, + { + "bucket": 352.0, + "duration_ms": 9321.284261999997, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 289.0, + "overflow": 16.0, + "padding": 7.0, + "tokens": 33113.0 + }, + { + "bucket": 1092.0, + "duration_ms": 8666.118680999994, + "eligible_miss": 0.0, + "miss": 4.0, + "model": 556.0, + "overflow": 4.0, + "padding": 143.0, + "tokens": 6943.0 + }, + { + "bucket": 491.0, + "duration_ms": 8220.941466999995, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 453.0, + "overflow": 11.0, + "padding": 0.0, + "tokens": 22531.0 + }, + { + "bucket": 821.0, + "duration_ms": 9977.426195000007, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 358.0, + "overflow": 11.0, + "padding": 61.0, + "tokens": 23256.0 + }, + { + "bucket": 511.0, + "duration_ms": 4679.773215999997, + "eligible_miss": 0.0, + "miss": 2.0, + "model": 513.0, + "overflow": 2.0, + "padding": 0.0, + "tokens": 2680.0 + }, + { + "bucket": 344.0, + "duration_ms": 5247.637218999998, + "eligible_miss": 2.0, + "miss": 9.0, + "model": 353.0, + "overflow": 7.0, + "padding": 0.0, + "tokens": 14579.0 + }, + { + "bucket": 665.0, + "duration_ms": 8615.953720000001, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 617.0, + "overflow": 8.0, + "padding": 0.0, + "tokens": 16312.0 + }, + { + "bucket": 239.0, + "duration_ms": 9411.176352999995, + "eligible_miss": 1.0, + "miss": 18.0, + "model": 257.0, + "overflow": 17.0, + "padding": 0.0, + "tokens": 35195.0 + }, + { + "bucket": 508.0, + "duration_ms": 9621.412921, + "eligible_miss": 0.0, + "miss": 13.0, + "model": 140.0, + "overflow": 13.0, + "padding": 31.0, + "tokens": 26633.0 + }, + { + "bucket": 116.0, + "duration_ms": 11128.783442000002, + "eligible_miss": 1.0, + "miss": 19.0, + "model": 48.0, + "overflow": 18.0, + "padding": 0.0, + "tokens": 37285.0 + } + ], + "clean": { + "admitted": 116, + "completed": 111, + "completed_throughput_rps": 0.4625, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 968714, + "offered_rps": 0.48333333333333334, + "output_tokens": 25409, + "start_s": 60.0 + }, + "config": "C11", + "drain_quarantined": false, + "drain_seconds": 2.1805057660094462, + "latency_s": { + "e2e": { + "mean": 4.364980708677919, + "n": 111, + "p50": 2.5644153020111844, + "p95": 15.528192899510032, + "p99": 17.251885565518755 + }, + "tpot": { + "mean": 0.013476979717565554, + "n": 111, + "p50": 0.007207736172530727, + "p95": 0.048615745335301455, + "p99": 0.05342051563131643 + }, + "ttft": { + "mean": 1.111188190306174, + "n": 111, + "p50": 0.7105740209808573, + "p95": 3.396348365509766, + "p99": 4.767139900015901 + } + }, + "layer1": { + "decode_tokens": 25661, + "kv_usage_max": 0.39101211794142177, + "kv_usage_mean": 0.04664282350048209, + "mode_counts": { + "FULL": 15795, + "NONE": 640, + "PIECEWISE": 8 + }, + "mode_shares": { + "FULL": 0.9605911330049262, + "NONE": 0.038922337772912485, + "PIECEWISE": 0.00048652922216140606 + }, + "preemptions": 0, + "prefill_tokens": 1025115, + "prefix_query_hit_ratio": 0.0167845310754508, + "queue_waiting_max": 1, + "queue_waiting_mean": 0.0024326461108070303, + "records_all": 22073, + "records_clean": 16443, + "requests_per_step": { + "mean": 1.59514687100894, + "n": 16443, + "p50": 1.0, + "p95": 5.0, + "p99": 7.0 + }, + "scheduled_tokens_per_step": { + "mean": 63.904153743234204, + "n": 16443, + "p50": 1.0, + "p95": 6.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 24.31026893662957, + "n": 16443, + "p50": 9.632865, + "p95": 31.7124496, + "p99": 461.83891844 + }, + "token_efficiency_per_ms": 2.6286897076217217 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "moderate", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.5, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.1, + "completion_rate_within_10pct": false, + "tail_end_s": 345.9574947899964, + "tail_start_s": 335.9574947899964, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.5, + "clean_c_waiting_median": 0.0, + "completion_rate_rps": 0.3, + "completion_rate_within_10pct": false, + "tail_end_s": 379.9735003169917, + "tail_start_s": 369.9735003169917, + "valid": false, + "waiting_median": 0.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 111, + "run_dir": "runs/phase3/primary/P10-C11/moderate", + "run_id": "P10-C11-moderate", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.0805340710212838, + "eligible_miss_rate": 0.0014069859913133909, + "graph_miss_rate": 0.0332782773597602, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.03187129136844681, + "padded_tokens_per_useful_token": 0.002579046342893252, + "padding_fraction": 0.09711173224396187 + } + }, + "P10-C11-saturation": { + "blocks": [ + { + "duration_ms": 9803.640058999998, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 26.0, + "overflow": 26.0, + "tokens": 53248.0 + }, + { + "bucket": 5880.0, + "duration_ms": 9835.950327000006, + "eligible_miss": 1.0, + "miss": 7.0, + "model": 100.0, + "overflow": 6.0, + "padding": 462.0, + "tokens": 15976.0 + }, + { + "bucket": 392.0, + "duration_ms": 10336.127500000002, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 33.0, + "overflow": 26.0, + "padding": 1.0, + "tokens": 53639.0 + }, + { + "duration_ms": 9842.753817000003, + "eligible_miss": 0.0, + "miss": 27.0, + "model": 27.0, + "overflow": 27.0, + "tokens": 55296.0 + }, + { + "duration_ms": 10500.215170000001, + "eligible_miss": 0.0, + "miss": 26.0, + "model": 26.0, + "overflow": 26.0, + "tokens": 53248.0 + }, + { + "duration_ms": 9819.693657000002, + "eligible_miss": 0.0, + "miss": 25.0, + "model": 25.0, + "overflow": 25.0, + "tokens": 51200.0 + }, + { + "duration_ms": 9582.212909000002, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 23.0, + "overflow": 23.0, + "tokens": 47104.0 + }, + { + "bucket": 112.0, + "duration_ms": 10275.247018, + "eligible_miss": 0.0, + "miss": 25.0, + "model": 27.0, + "overflow": 25.0, + "padding": 1.0, + "tokens": 50799.0 + }, + { + "bucket": 2760.0, + "duration_ms": 9618.081895000001, + "eligible_miss": 2.0, + "miss": 17.0, + "model": 62.0, + "overflow": 15.0, + "padding": 177.0, + "tokens": 31342.0 + }, + { + "bucket": 2912.0, + "duration_ms": 11031.054516, + "eligible_miss": 1.0, + "miss": 13.0, + "model": 66.0, + "overflow": 12.0, + "padding": 239.0, + "tokens": 27706.0 + }, + { + "duration_ms": 9609.575980999998, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 16.0, + "overflow": 16.0, + "tokens": 31606.0 + }, + { + "duration_ms": 9580.234407, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 34816.0 + }, + { + "bucket": 40.0, + "duration_ms": 10043.451591, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 21.0, + "overflow": 20.0, + "padding": 4.0, + "tokens": 39801.0 + }, + { + "duration_ms": 10257.613201999999, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 19.0, + "overflow": 19.0, + "tokens": 38912.0 + }, + { + "duration_ms": 9893.364799, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 19.0, + "overflow": 19.0, + "tokens": 38912.0 + }, + { + "duration_ms": 9812.629688, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "duration_ms": 10162.704605, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 18.0, + "overflow": 18.0, + "tokens": 36864.0 + }, + { + "duration_ms": 10096.544445, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 21.0, + "overflow": 21.0, + "tokens": 43008.0 + }, + { + "bucket": 192.0, + "duration_ms": 9439.287972999999, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 21.0, + "overflow": 15.0, + "padding": 38.0, + "tokens": 30874.0 + }, + { + "bucket": 1440.0, + "duration_ms": 10398.481440000001, + "eligible_miss": 0.0, + "miss": 9.0, + "model": 96.0, + "overflow": 9.0, + "padding": 43.0, + "tokens": 19829.0 + }, + { + "bucket": 64.0, + "duration_ms": 10069.123950999998, + "eligible_miss": 0.0, + "miss": 21.0, + "model": 24.0, + "overflow": 21.0, + "padding": 14.0, + "tokens": 42743.0 + }, + { + "duration_ms": 10163.353964999998, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 22.0, + "overflow": 22.0, + "tokens": 45056.0 + }, + { + "duration_ms": 9601.490734, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 19.0, + "overflow": 19.0, + "tokens": 38912.0 + }, + { + "duration_ms": 9938.422147000001, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40389.0 + }, + { + "bucket": 72.0, + "duration_ms": 10031.912896999998, + "eligible_miss": 0.0, + "miss": 23.0, + "model": 25.0, + "overflow": 23.0, + "padding": 6.0, + "tokens": 47170.0 + }, + { + "bucket": 240.0, + "duration_ms": 9938.585003, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 28.0, + "overflow": 18.0, + "padding": 52.0, + "tokens": 35610.0 + }, + { + "bucket": 96.0, + "duration_ms": 10070.516326, + "eligible_miss": 0.0, + "miss": 22.0, + "model": 26.0, + "overflow": 22.0, + "padding": 21.0, + "tokens": 44167.0 + }, + { + "bucket": 1248.0, + "duration_ms": 9806.104926000007, + "eligible_miss": 0.0, + "miss": 12.0, + "model": 64.0, + "overflow": 12.0, + "padding": 209.0, + "tokens": 23747.0 + }, + { + "bucket": 648.0, + "duration_ms": 10603.34154, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 45.0, + "overflow": 18.0, + "padding": 111.0, + "tokens": 37401.0 + }, + { + "duration_ms": 9530.284238, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "duration_ms": 10103.438895999998, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "duration_ms": 10733.006420000002, + "eligible_miss": 1.0, + "miss": 21.0, + "model": 21.0, + "overflow": 20.0, + "tokens": 41432.0 + }, + { + "duration_ms": 9224.846354000001, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 16.0, + "overflow": 16.0, + "tokens": 32516.0 + }, + { + "duration_ms": 10388.344382, + "eligible_miss": 0.0, + "miss": 19.0, + "model": 19.0, + "overflow": 19.0, + "tokens": 37733.0 + }, + { + "bucket": 544.0, + "duration_ms": 10253.314467000004, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 39.0, + "overflow": 14.0, + "padding": 88.0, + "tokens": 29128.0 + }, + { + "bucket": 32.0, + "duration_ms": 9439.055381000002, + "eligible_miss": 0.0, + "miss": 14.0, + "model": 16.0, + "overflow": 14.0, + "padding": 3.0, + "tokens": 28677.0 + }, + { + "duration_ms": 10483.385640999999, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 33881.0 + }, + { + "bucket": 944.0, + "duration_ms": 10235.9488, + "eligible_miss": 0.0, + "miss": 11.0, + "model": 70.0, + "overflow": 11.0, + "padding": 114.0, + "tokens": 23358.0 + }, + { + "bucket": 144.0, + "duration_ms": 9260.176585000001, + "eligible_miss": 0.0, + "miss": 18.0, + "model": 27.0, + "overflow": 18.0, + "padding": 20.0, + "tokens": 35688.0 + }, + { + "bucket": 208.0, + "duration_ms": 10099.584046999998, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 24.0, + "overflow": 16.0, + "padding": 22.0, + "tokens": 32954.0 + }, + { + "bucket": 240.0, + "duration_ms": 9581.310761, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 31.0, + "overflow": 16.0, + "padding": 46.0, + "tokens": 32833.0 + }, + { + "duration_ms": 10818.639540999997, + "eligible_miss": 0.0, + "miss": 20.0, + "model": 20.0, + "overflow": 20.0, + "tokens": 40960.0 + }, + { + "duration_ms": 9592.901207, + "eligible_miss": 0.0, + "miss": 15.0, + "model": 15.0, + "overflow": 15.0, + "tokens": 30720.0 + }, + { + "bucket": 496.0, + "duration_ms": 10098.093245999999, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 47.0, + "overflow": 16.0, + "padding": 94.0, + "tokens": 32618.0 + }, + { + "duration_ms": 9722.888504, + "eligible_miss": 0.0, + "miss": 16.0, + "model": 16.0, + "overflow": 16.0, + "tokens": 32768.0 + }, + { + "duration_ms": 10337.847011, + "eligible_miss": 0.0, + "miss": 17.0, + "model": 17.0, + "overflow": 17.0, + "tokens": 34816.0 + }, + { + "bucket": 880.0, + "duration_ms": 9603.342421999998, + "eligible_miss": 0.0, + "miss": 8.0, + "model": 63.0, + "overflow": 8.0, + "padding": 166.0, + "tokens": 16514.0 + }, + { + "bucket": 16.0, + "duration_ms": 10377.703795, + "eligible_miss": 1.0, + "miss": 17.0, + "model": 18.0, + "overflow": 16.0, + "padding": 4.0, + "tokens": 33158.0 + } + ], + "clean": { + "admitted": 193, + "completed": 193, + "completed_throughput_rps": 0.8041666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1762708, + "offered_rps": 0.8041666666666667, + "output_tokens": 44197, + "start_s": 60.0 + }, + "config": "C11", + "drain_quarantined": false, + "drain_seconds": 273.14732490401366, + "latency_s": { + "e2e": { + "mean": 134.18431260923774, + "n": 193, + "p50": 111.3837862299988, + "p95": 258.4493683842126, + "p99": 287.67599090651015 + }, + "tpot": { + "mean": 0.15051965069877637, + "n": 193, + "p50": 0.1509121470940847, + "p95": 0.1897459957210793, + "p99": 0.2325328770395698 + }, + "ttft": { + "mean": 100.05380060465345, + "n": 193, + "p50": 83.08665158701479, + "p95": 216.04882907219576, + "p99": 245.99976527046636 + } + }, + "layer1": { + "decode_tokens": 42219, + "kv_usage_max": 0.9999509395084138, + "kv_usage_mean": 0.9401752966026676, + "mode_counts": { + "FULL": 592, + "NONE": 875, + "PIECEWISE": 5 + }, + "mode_shares": { + "FULL": 0.40217391304347827, + "NONE": 0.5944293478260869, + "PIECEWISE": 0.0033967391304347825 + }, + "preemptions": 3, + "prefill_tokens": 1739790, + "prefix_query_hit_ratio": 0.004341961412847301, + "queue_waiting_max": 244, + "queue_waiting_mean": 227.1576086956522, + "records_all": 4155, + "records_clean": 1472, + "requests_per_step": { + "mean": 28.575407608695652, + "n": 1472, + "p50": 19.0, + "p95": 59.0, + "p99": 62.0 + }, + "scheduled_tokens_per_step": { + "mean": 1210.6039402173913, + "n": 1472, + "p50": 2048.0, + "p95": 2048.0, + "p99": 2048.0 + }, + "step_duration_ms": { + "mean": 326.1180898002717, + "n": 1472, + "p50": 347.0396935, + "p95": 729.4108498, + "p99": 859.3618077799999 + }, + "token_efficiency_per_ms": 3.7121643296721607 + }, + "layer2_missing_after_controller_cleanup": false, + "load": "saturation", + "pattern": "P10", + "profile_recovery": [ + { + "clean_c_completion_rate_median_rps": 0.3, + "clean_c_waiting_median": 243.0, + "completion_rate_rps": 0.4, + "completion_rate_within_10pct": false, + "tail_end_s": 349.9593778490089, + "tail_start_s": 339.9593778490089, + "valid": false, + "waiting_median": 227.0, + "waiting_within_10pct": true, + "window": 1 + }, + { + "clean_c_completion_rate_median_rps": 0.3, + "clean_c_waiting_median": 243.0, + "completion_rate_rps": 0.9, + "completion_rate_within_10pct": false, + "tail_end_s": 383.3783031209896, + "tail_start_s": 373.3783031209896, + "valid": false, + "waiting_median": 219.0, + "waiting_within_10pct": true, + "window": 2 + } + ], + "profile_recovery_valid": false, + "requests_completed": 193, + "run_dir": "runs/phase3/primary/P10-C11/saturation", + "run_id": "P10-C11-saturation", + "trace_files": 2, + "waste": { + "R128": 0.7047173851989379, + "R32": 0.6764453342592629, + "R64": 0.6923054365307624, + "bucket_slack": 0.08936285097192224, + "eligible_miss_rate": 0.004076086956521739, + "graph_miss_rate": 0.5944293478260869, + "mixed_interference": null, + "moe_layer_cv": null, + "overflow_rate": 0.5903532608695652, + "padded_tokens_per_useful_token": 0.0010858531017520115, + "padding_fraction": 0.09872448979591837 + } + } + }, + "sanity": { + "declared_deviations": { + "invalid_primary_layer2_windows": [ + "P01-C00-moderate: window 1", + "P01-C00-moderate: window 2", + "P02-C00-moderate: window 1", + "P02-C00-moderate: window 2", + "P03-C00-moderate: window 1", + "P03-C00-moderate: window 2", + "P06-C00-moderate: window 1", + "P06-C00-moderate: window 2", + "P07-C00-moderate: window 1", + "P07-C00-moderate: window 2", + "P08-C00-moderate: window 1", + "P08-C00-moderate: window 2", + "P09-C00-moderate: window 1", + "P09-C00-moderate: window 2", + "P10-C00-moderate: window 1", + "P10-C00-moderate: window 2" + ], + "missing_cells": [ + "P03/C11", + "P05/C00", + "P10/C00-TP2", + "P11/C00" + ], + "missing_confirmations": [ + "P10", + "P06", + "P03", + "P01" + ], + "missing_saturation_traces": 8, + "moe_layer_cv": "N/A: layer scopes do not cover >=80% of MoE GEMM time", + "unaccepted_canonical_run_markers_excluded": [ + "P11-C00-saturation" + ] + }, + "invariants": { + "accepted_run_ids_unique": true, + "ap36_operational_finding_reproduced": true, + "ap37_confirmation_runs_0": true, + "ap37_primary_runs_40": true, + "clean_duration_240": true, + "clean_failures_zero": true, + "clock_and_load_snapshots_complete": true, + "complete_cells_20": true, + "complete_stage_count_12": true, + "completed_c00_patterns_9": true, + "controller_frozen_at_ap36_failure": true, + "drain_quarantine_under_20pct": true, + "h1b_evaluable_contrasts_6": true, + "h1b_missing_contrasts_exact": true, + "layer1_footer_invariants": true, + "missing_cells_exact": true, + "missing_trace_count_accepted_8": true, + "moderate_rate_within_5pct": true, + "no_layer2_parser_issues": true, + "other_gpu_processes_absent": true, + "patterns_not_all_identical_throughput": true, + "ratios_in_unit_interval": true, + "trace_count_accepted_72": true + }, + "numeric": { + "completed_throughput_rps": { + "distinct_n": 37, + "finite_n": 40, + "max": 43.3, + "min": 0.44583333333333336, + "missing_n": 0, + "n": 40 + }, + "drain_seconds": { + "distinct_n": 40, + "finite_n": 40, + "max": 288.6191079240234, + "min": 0.3054194079886656, + "missing_n": 0, + "n": 40 + }, + "kendall_tau_b": { + "distinct_n": 0, + "finite_n": 0, + "max": null, + "min": null, + "missing_n": 0, + "n": 0 + }, + "layer1_clean_steps": { + "distinct_n": 40, + "finite_n": 40, + "max": 20171.0, + "min": 718.0, + "missing_n": 0, + "n": 40 + }, + "operator_classifiable_fraction": { + "distinct_n": 72, + "finite_n": 72, + "max": 0.9963781377235988, + "min": 0.9704584620048622, + "missing_n": 0, + "n": 72 + }, + "operator_share": { + "distinct_n": 348, + "finite_n": 576, + "max": 0.8734165832313857, + "min": 0.0, + "missing_n": 0, + "n": 576 + }, + "token_efficiency_per_ms": { + "distinct_n": 40, + "finite_n": 40, + "max": 8.29584869763246, + "min": 2.147827352932598, + "missing_n": 0, + "n": 40 + }, + "waste_contrast_effect": { + "distinct_n": 17, + "finite_n": 24, + "max": 0.5204129156598252, + "min": -0.20308199280724987, + "missing_n": 12, + "n": 36 + }, + "waste_ratios": { + "distinct_n": 168, + "finite_n": 240, + "max": 1.0, + "min": 0.0, + "missing_n": 0, + "n": 240 + } + } + }, + "schema": 1, + "waste_contrast_status": [ + { + "control": "P01", + "efficiency_loss": null, + "evaluable": false, + "irregular": "P05", + "metrics": [], + "missing_cells": [ + "P05-C00" + ], + "passes_h1b": false, + "passing_metrics": [], + "verdict": "NOT_EVALUABLE" + }, + { + "control": "P03", + "efficiency_loss": null, + "evaluable": false, + "irregular": "P05", + "metrics": [], + "missing_cells": [ + "P05-C00" + ], + "passes_h1b": false, + "passing_metrics": [], + "verdict": "NOT_EVALUABLE" + }, + { + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "metrics": [ + { + "available": true, + "ci95_high": -0.01613620902704422, + "ci95_low": -0.018335446670663714, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.017218083890213506, + "simultaneous_high": -0.0157356399373619, + "simultaneous_low": -0.0187887535849442 + }, + { + "available": true, + "ci95_high": -0.026647777870177056, + "ci95_low": -0.03356127646776247, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.03006598445684061, + "simultaneous_high": -0.02530594241869426, + "simultaneous_low": -0.03499903509242671 + }, + { + "available": true, + "ci95_high": -0.026633809539104453, + "ci95_low": -0.033569133608294076, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.03006598445684061, + "simultaneous_high": -0.025351137872591654, + "simultaneous_low": -0.0349968653294319 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.23494534783397408, + "ci95_low": 0.2253590527218157, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.2301483528866904, + "simultaneous_high": 0.23683442206671917, + "simultaneous_low": 0.22345003461028415 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "missing_cells": [], + "passes_h1b": true, + "passing_metrics": [ + "R64" + ], + "verdict": "PASS" + }, + { + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "metrics": [ + { + "available": true, + "ci95_high": 0.0018881170209094382, + "ci95_low": 0.00013272036050889215, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "padding_fraction", + "p": 0.02776, + "p_holm": 0.08327999999999999, + "passes_h1b": false, + "point": 0.001092906862063869, + "simultaneous_high": 0.002162520644872445, + "simultaneous_low": -0.0002871891115189649 + }, + { + "available": true, + "ci95_high": -0.000493778899366458, + "ci95_low": -0.010138790500287403, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "graph_miss_rate", + "p": 0.02632, + "p_holm": 0.10528, + "passes_h1b": false, + "point": -0.004704030248702935, + "simultaneous_high": 0.0009672756203671407, + "simultaneous_low": -0.012632664546359583 + }, + { + "available": true, + "ci95_high": -0.0005068842298477604, + "ci95_low": -0.010089468512164912, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "overflow_rate", + "p": 0.02672, + "p_holm": 0.10688, + "passes_h1b": false, + "point": -0.004704030248702935, + "simultaneous_high": 0.0009123995960551078, + "simultaneous_low": -0.012541358318390415 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.3588237662068493, + "ci95_low": 0.34996301147907066, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.3543970834375052, + "simultaneous_high": 0.3605880124391376, + "simultaneous_low": 0.34826154238421436 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "missing_cells": [], + "passes_h1b": true, + "passing_metrics": [ + "R64" + ], + "verdict": "PASS" + }, + { + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "metrics": [ + { + "available": true, + "ci95_high": 0.07258101928545771, + "ci95_low": 0.06107199918894719, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.0667290414201105, + "simultaneous_high": 0.07492451384312424, + "simultaneous_low": 0.058870086990988876 + }, + { + "available": true, + "ci95_high": -0.18669616533208697, + "ci95_low": -0.21919298047891625, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.20308199280724987, + "simultaneous_high": -0.18011971739412205, + "simultaneous_low": -0.2256276809071987 + }, + { + "available": true, + "ci95_high": -0.18660802863031692, + "ci95_low": -0.21934591413132573, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.20308199280724987, + "simultaneous_high": -0.18035129413637946, + "simultaneous_low": -0.22589311960316016 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.4001054362524198, + "ci95_low": 0.3922157126916775, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.39616418510901036, + "simultaneous_high": 0.4016625621683738, + "simultaneous_low": 0.3906536566040832 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "missing_cells": [], + "passes_h1b": true, + "passing_metrics": [ + "padding_fraction", + "R64" + ], + "verdict": "PASS" + }, + { + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "metrics": [ + { + "available": true, + "ci95_high": 0.09130100661903304, + "ci95_low": 0.07976380555258109, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.08542101015535679, + "simultaneous_high": 0.09354749622519089, + "simultaneous_low": 0.0776469398212851 + }, + { + "available": true, + "ci95_high": 0.03477003572235889, + "ci95_low": 0.022150480290407913, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.027825216312380757, + "simultaneous_high": 0.0379073029527558, + "simultaneous_low": 0.020262740610606924 + }, + { + "available": true, + "ci95_high": 0.03478537546264273, + "ci95_low": 0.022157030355767448, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.027825216312380757, + "simultaneous_high": 0.038155569680251286, + "simultaneous_low": 0.020260913333777324 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.5238722090454428, + "ci95_low": 0.5169233936643203, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.5204129156598252, + "simultaneous_high": 0.5252564301232907, + "simultaneous_low": 0.5155191519515977 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "missing_cells": [], + "passes_h1b": false, + "passing_metrics": [], + "verdict": "NO_QUALIFYING_METRIC" + }, + { + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "metrics": [ + { + "available": true, + "ci95_high": 0.08855777444708128, + "ci95_low": 0.027415277773332525, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.05565211993295673, + "simultaneous_high": 0.10341073102256013, + "simultaneous_low": 0.020097017796153225 + }, + { + "available": true, + "ci95_high": -0.00601669947733407, + "ci95_low": -0.009872845681590466, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.008145427700863333, + "simultaneous_high": -0.0049986468588015505, + "simultaneous_low": -0.010459290275607078 + }, + { + "available": true, + "ci95_high": -0.006012535696644192, + "ci95_low": -0.00989506245698499, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.008145427700863333, + "simultaneous_high": -0.005013934447667286, + "simultaneous_low": -0.010471471437559795 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.4561736305824604, + "ci95_low": 0.43955479496868705, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.4478716685574937, + "simultaneous_high": 0.459713780733712, + "simultaneous_low": 0.43635435003093426 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "missing_cells": [], + "passes_h1b": true, + "passing_metrics": [ + "padding_fraction", + "R64" + ], + "verdict": "PASS" + }, + { + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "metrics": [ + { + "available": true, + "ci95_high": 0.08668052314167546, + "ci95_low": 0.025823274229141942, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.05398526806052646, + "simultaneous_high": 0.10213016668480329, + "simultaneous_low": 0.018683381445978635 + }, + { + "available": true, + "ci95_high": 0.0007610647800286464, + "ci95_low": -0.00936641415445055, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "graph_miss_rate", + "p": 0.10722, + "p_holm": 0.32166, + "passes_h1b": false, + "point": -0.0037955725000514817, + "simultaneous_high": 0.0023548687154005257, + "simultaneous_low": -0.011962189207249601 + }, + { + "available": true, + "ci95_high": 0.0007651798910721392, + "ci95_low": -0.009350365146339001, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "overflow_rate", + "p": 0.10804, + "p_holm": 0.32411999999999996, + "passes_h1b": false, + "point": -0.0037955725000514817, + "simultaneous_high": 0.0023480385252717118, + "simultaneous_low": -0.011930615598005356 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.45618061702156487, + "ci95_low": 0.4395710696329076, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.4478716685574937, + "simultaneous_high": 0.4594884290583298, + "simultaneous_low": 0.4363807227134572 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "missing_cells": [], + "passes_h1b": true, + "passing_metrics": [ + "padding_fraction", + "R64" + ], + "verdict": "PASS" + } + ], + "waste_contrasts": [ + { + "available": true, + "ci95_high": -0.01613620902704422, + "ci95_low": -0.018335446670663714, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.017218083890213506, + "simultaneous_high": -0.0157356399373619, + "simultaneous_low": -0.0187887535849442 + }, + { + "available": true, + "ci95_high": -0.026647777870177056, + "ci95_low": -0.03356127646776247, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.03006598445684061, + "simultaneous_high": -0.02530594241869426, + "simultaneous_low": -0.03499903509242671 + }, + { + "available": true, + "ci95_high": -0.026633809539104453, + "ci95_low": -0.033569133608294076, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.03006598445684061, + "simultaneous_high": -0.025351137872591654, + "simultaneous_low": -0.0349968653294319 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.23494534783397408, + "ci95_low": 0.2253590527218157, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.2301483528866904, + "simultaneous_high": 0.23683442206671917, + "simultaneous_low": 0.22345003461028415 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P02", + "efficiency_loss": 0.11608997226921058, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.0018881170209094382, + "ci95_low": 0.00013272036050889215, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "padding_fraction", + "p": 0.02776, + "p_holm": 0.08327999999999999, + "passes_h1b": false, + "point": 0.001092906862063869, + "simultaneous_high": 0.002162520644872445, + "simultaneous_low": -0.0002871891115189649 + }, + { + "available": true, + "ci95_high": -0.000493778899366458, + "ci95_low": -0.010138790500287403, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "graph_miss_rate", + "p": 0.02632, + "p_holm": 0.10528, + "passes_h1b": false, + "point": -0.004704030248702935, + "simultaneous_high": 0.0009672756203671407, + "simultaneous_low": -0.012632664546359583 + }, + { + "available": true, + "ci95_high": -0.0005068842298477604, + "ci95_low": -0.010089468512164912, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "overflow_rate", + "p": 0.02672, + "p_holm": 0.10688, + "passes_h1b": false, + "point": -0.004704030248702935, + "simultaneous_high": 0.0009123995960551078, + "simultaneous_low": -0.012541358318390415 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.3588237662068493, + "ci95_low": 0.34996301147907066, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.3543970834375052, + "simultaneous_high": 0.3605880124391376, + "simultaneous_low": 0.34826154238421436 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.2284629366493368, + "evaluable": true, + "irregular": "P06", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.07258101928545771, + "ci95_low": 0.06107199918894719, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.0667290414201105, + "simultaneous_high": 0.07492451384312424, + "simultaneous_low": 0.058870086990988876 + }, + { + "available": true, + "ci95_high": -0.18669616533208697, + "ci95_low": -0.21919298047891625, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.20308199280724987, + "simultaneous_high": -0.18011971739412205, + "simultaneous_low": -0.2256276809071987 + }, + { + "available": true, + "ci95_high": -0.18660802863031692, + "ci95_low": -0.21934591413132573, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.20308199280724987, + "simultaneous_high": -0.18035129413637946, + "simultaneous_low": -0.22589311960316016 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.4001054362524198, + "ci95_low": 0.3922157126916775, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.39616418510901036, + "simultaneous_high": 0.4016625621683738, + "simultaneous_low": 0.3906536566040832 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P01", + "efficiency_loss": 0.08324816211784114, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.09130100661903304, + "ci95_low": 0.07976380555258109, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.08542101015535679, + "simultaneous_high": 0.09354749622519089, + "simultaneous_low": 0.0776469398212851 + }, + { + "available": true, + "ci95_high": 0.03477003572235889, + "ci95_low": 0.022150480290407913, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.027825216312380757, + "simultaneous_high": 0.0379073029527558, + "simultaneous_low": 0.020262740610606924 + }, + { + "available": true, + "ci95_high": 0.03478537546264273, + "ci95_low": 0.022157030355767448, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.027825216312380757, + "simultaneous_high": 0.038155569680251286, + "simultaneous_low": 0.020260913333777324 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.5238722090454428, + "ci95_low": 0.5169233936643203, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": 0.5204129156598252, + "simultaneous_high": 0.5252564301232907, + "simultaneous_low": 0.5155191519515977 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": false, + "control": "P03", + "efficiency_loss": 0.03840305601484284, + "evaluable": true, + "irregular": "P09", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.08855777444708128, + "ci95_low": 0.027415277773332525, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.05565211993295673, + "simultaneous_high": 0.10341073102256013, + "simultaneous_low": 0.020097017796153225 + }, + { + "available": true, + "ci95_high": -0.00601669947733407, + "ci95_low": -0.009872845681590466, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "graph_miss_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.008145427700863333, + "simultaneous_high": -0.0049986468588015505, + "simultaneous_low": -0.010459290275607078 + }, + { + "available": true, + "ci95_high": -0.006012535696644192, + "ci95_low": -0.00989506245698499, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "overflow_rate", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": false, + "point": -0.008145427700863333, + "simultaneous_high": -0.005013934447667286, + "simultaneous_low": -0.010471471437559795 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.4561736305824604, + "ci95_low": 0.43955479496868705, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.4478716685574937, + "simultaneous_high": 0.459713780733712, + "simultaneous_low": 0.43635435003093426 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P03", + "efficiency_loss": 0.4469310402722787, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.08668052314167546, + "ci95_low": 0.025823274229141942, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "padding_fraction", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.05398526806052646, + "simultaneous_high": 0.10213016668480329, + "simultaneous_low": 0.018683381445978635 + }, + { + "available": true, + "ci95_high": 0.0007610647800286464, + "ci95_low": -0.00936641415445055, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "graph_miss_rate", + "p": 0.10722, + "p_holm": 0.32166, + "passes_h1b": false, + "point": -0.0037955725000514817, + "simultaneous_high": 0.0023548687154005257, + "simultaneous_low": -0.011962189207249601 + }, + { + "available": true, + "ci95_high": 0.0007651798910721392, + "ci95_low": -0.009350365146339001, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "overflow_rate", + "p": 0.10804, + "p_holm": 0.32411999999999996, + "passes_h1b": false, + "point": -0.0037955725000514817, + "simultaneous_high": 0.0023480385252717118, + "simultaneous_low": -0.011930615598005356 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "mixed_interference", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + }, + { + "available": true, + "ci95_high": 0.45618061702156487, + "ci95_low": 0.4395710696329076, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": true, + "metric": "R64", + "p": 0.0, + "p_holm": 0.0, + "passes_h1b": true, + "point": 0.4478716685574937, + "simultaneous_high": 0.4594884290583298, + "simultaneous_low": 0.4363807227134572 + }, + { + "available": false, + "ci95_high": null, + "ci95_low": null, + "coincident_efficiency_or_residual": true, + "control": "P04", + "efficiency_loss": 0.1425966128421564, + "evaluable": true, + "irregular": "P10", + "material": false, + "metric": "moe_layer_cv", + "p": 1.0, + "p_holm": 1.0, + "passes_h1b": false, + "point": null, + "simultaneous_high": null, + "simultaneous_low": null + } + ], + "waste_thresholds": { + "R64": 0.15, + "graph_miss_rate": 0.1, + "mixed_interference": 0.1, + "moe_layer_cv": 0.15, + "overflow_rate": 0.1, + "padding_fraction": 0.05 + } +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P03-C10/result.json b/runs/opprof-phase3/phase3/remote-evidence/P03-C10/result.json new file mode 100644 index 0000000..1ea40b3 --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P03-C10/result.json @@ -0,0 +1,103 @@ +{ + "admission_stop_s": 417.3262625900097, + "arrival": "steady", + "clean": { + "admitted": 559, + "completed": 559, + "completed_throughput_rps": 2.3291666666666666, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 3429894, + "offered_rps": 2.3291666666666666, + "output_tokens": 35776, + "start_s": 60.0 + }, + "clean_segment_seconds": 80.0, + "drain_seconds": 91.29178713698639, + "elapsed_seconds": 508.6180497269961, + "failed_records": 0, + "load_point": "saturation", + "manifest_admitted": 1094, + "manifest_exhausted": false, + "manifest_rows": 32768, + "manifest_sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1", + "manifest_wrapped": false, + "max_in_flight": 256, + "num_clean_segments": 3, + "profiles": [ + { + "start_call_s": 300.0043962339987, + "start_return_s": 301.3819164079905, + "start_status": 200, + "stop_call_s": 321.8697677869932, + "stop_return_s": 326.31530767300865, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835059474707213.pt.trace.json.gz", + "trace_ready_s": 321.86976291501196, + "trace_sha256": "527ddb81b1d54a12a463508bc163d85a2c443d94c8a897959a39735e030b90a2", + "window": 1 + }, + { + "start_call_s": 356.32926020701416, + "start_return_s": 361.34978177401354, + "start_status": 200, + "stop_call_s": 381.577829301008, + "stop_return_s": 387.31277802900877, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835119198018909.pt.trace.json.gz", + "trace_ready_s": 381.57782580400817, + "trace_sha256": "0882d9705f1208580d5e693f94ba64d89ab12ef0aae2731a82d460a46a3e4cb1", + "window": 2 + } + ], + "rate_fraction": null, + "records": 1094, + "request_rate": "inf", + "schema": 1, + "segments": [ + { + "admitted": 191, + "completed": 191, + "completed_throughput_rps": 2.3875, + "duration_s": 80.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 1154628, + "name": "A", + "offered_rps": 2.3875, + "output_tokens": 12224, + "start_s": 60.0 + }, + { + "admitted": 184, + "completed": 184, + "completed_throughput_rps": 2.3, + "duration_s": 80.0, + "end_s": 220.0, + "failed": 0, + "input_tokens": 1135217, + "name": "B", + "offered_rps": 2.3, + "output_tokens": 11776, + "start_s": 140.0 + }, + { + "admitted": 184, + "completed": 184, + "completed_throughput_rps": 2.3, + "duration_s": 80.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1140049, + "name": "C", + "offered_rps": 2.3, + "output_tokens": 11776, + "start_s": 220.0 + } + ], + "successful_records": 1094, + "t0_mono_ns": 166323993366724, + "t0_wall_ns": 1783834751119796663, + "warmup_seconds": 60.0 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P03-C10/sanity.json b/runs/opprof-phase3/phase3/remote-evidence/P03-C10/sanity.json new file mode 100644 index 0000000..ad33c7b --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P03-C10/sanity.json @@ -0,0 +1,65 @@ +{ + "invariants": { + "clean_duration_exact": true, + "clean_failures_zero": true, + "concurrency_bounded": true, + "drain_within_timeout": true, + "manifest_no_wrap": true, + "manifest_not_exhausted": true, + "output_tokens_exact": true, + "profile_count_exact": true, + "profile_status_ok": true, + "segment_count_exact": true + }, + "numeric": { + "actual_output_tokens": { + "distinct_n": 1, + "finite_n": 1094, + "max": 64.0, + "min": 64.0, + "missing_n": 0, + "n": 1094 + }, + "admitted_s": { + "distinct_n": 1094, + "finite_n": 1094, + "max": 416.7779763180006, + "min": 0.0008771540015004575, + "missing_n": 0, + "n": 1094 + }, + "completed_s": { + "distinct_n": 1094, + "finite_n": 1094, + "max": 508.61568894199445, + "min": 22.18080371801625, + "missing_n": 0, + "n": 1094 + }, + "input_tokens": { + "distinct_n": 964, + "finite_n": 1094, + "max": 8190.0, + "min": 4097.0, + "missing_n": 0, + "n": 1094 + }, + "requested_output_tokens": { + "distinct_n": 1, + "finite_n": 1094, + "max": 64.0, + "min": 64.0, + "missing_n": 0, + "n": 1094 + }, + "scheduled_s": { + "distinct_n": 1094, + "finite_n": 1094, + "max": 416.7779758319957, + "min": 0.0008756940078455955, + "missing_n": 0, + "n": 1094 + } + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P07-C00/result.json b/runs/opprof-phase3/phase3/remote-evidence/P07-C00/result.json new file mode 100644 index 0000000..410a0f9 --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P07-C00/result.json @@ -0,0 +1,103 @@ +{ + "admission_stop_s": 417.66287675101194, + "arrival": "burst:8", + "clean": { + "admitted": 1226, + "completed": 1226, + "completed_throughput_rps": 5.108333333333333, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1569280, + "offered_rps": 5.108333333333333, + "output_tokens": 627712, + "start_s": 60.0 + }, + "clean_segment_seconds": 80.0, + "drain_seconds": 26.098238860984566, + "elapsed_seconds": 443.7611156119965, + "failed_records": 0, + "load_point": "saturation", + "manifest_admitted": 2078, + "manifest_exhausted": false, + "manifest_rows": 32768, + "manifest_sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6", + "manifest_wrapped": false, + "max_in_flight": 256, + "num_clean_segments": 3, + "profiles": [ + { + "start_call_s": 300.0039349270228, + "start_return_s": 300.9924320260179, + "start_status": 200, + "stop_call_s": 323.90002606701455, + "stop_return_s": 328.078528626007, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835058771060551.pt.trace.json.gz", + "trace_ready_s": 323.9000201699964, + "trace_sha256": "2d12038b03c6f9c28d1a488abd880ff2763e1f3c0ab502cfdd2507b649de2af7", + "window": 1 + }, + { + "start_call_s": 358.0940292410087, + "start_return_s": 358.8905565890018, + "start_status": 200, + "stop_call_s": 382.04741394601297, + "stop_return_s": 387.64796055300394, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835116964699401.pt.trace.json.gz", + "trace_ready_s": 382.04741024502437, + "trace_sha256": "2184d4f83ea22c0e5455dd3914922548ce8d3d5eba34f89562df69326e3832b5", + "window": 2 + } + ], + "rate_fraction": null, + "records": 2078, + "request_rate": "inf", + "schema": 1, + "segments": [ + { + "admitted": 394, + "completed": 394, + "completed_throughput_rps": 4.925, + "duration_s": 80.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 504320, + "name": "A", + "offered_rps": 4.925, + "output_tokens": 201728, + "start_s": 60.0 + }, + { + "admitted": 397, + "completed": 397, + "completed_throughput_rps": 4.9625, + "duration_s": 80.0, + "end_s": 220.0, + "failed": 0, + "input_tokens": 508160, + "name": "B", + "offered_rps": 4.9625, + "output_tokens": 203264, + "start_s": 140.0 + }, + { + "admitted": 435, + "completed": 435, + "completed_throughput_rps": 5.4375, + "duration_s": 80.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 556800, + "name": "C", + "offered_rps": 5.4375, + "output_tokens": 222720, + "start_s": 220.0 + } + ], + "successful_records": 2078, + "t0_mono_ns": 166323999932716, + "t0_wall_ns": 1783834751126361482, + "warmup_seconds": 60.0 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P07-C00/sanity.json b/runs/opprof-phase3/phase3/remote-evidence/P07-C00/sanity.json new file mode 100644 index 0000000..3594ef7 --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P07-C00/sanity.json @@ -0,0 +1,65 @@ +{ + "invariants": { + "clean_duration_exact": true, + "clean_failures_zero": true, + "concurrency_bounded": true, + "drain_within_timeout": true, + "manifest_no_wrap": true, + "manifest_not_exhausted": true, + "output_tokens_exact": true, + "profile_count_exact": true, + "profile_status_ok": true, + "segment_count_exact": true + }, + "numeric": { + "actual_output_tokens": { + "distinct_n": 1, + "finite_n": 2078, + "max": 512.0, + "min": 512.0, + "missing_n": 0, + "n": 2078 + }, + "admitted_s": { + "distinct_n": 2078, + "finite_n": 2078, + "max": 417.40899016702315, + "min": 0.0008179180149454623, + "missing_n": 0, + "n": 2078 + }, + "completed_s": { + "distinct_n": 2078, + "finite_n": 2078, + "max": 443.75592052700813, + "min": 35.15151289102505, + "missing_n": 0, + "n": 2078 + }, + "input_tokens": { + "distinct_n": 1, + "finite_n": 2078, + "max": 1280.0, + "min": 1280.0, + "missing_n": 0, + "n": 2078 + }, + "requested_output_tokens": { + "distinct_n": 1, + "finite_n": 2078, + "max": 512.0, + "min": 512.0, + "missing_n": 0, + "n": 2078 + }, + "scheduled_s": { + "distinct_n": 2078, + "finite_n": 2078, + "max": 417.4089898300008, + "min": 0.0008165129984263331, + "missing_n": 0, + "n": 2078 + } + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P08-C00/result.json b/runs/opprof-phase3/phase3/remote-evidence/P08-C00/result.json new file mode 100644 index 0000000..25d6e7a --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P08-C00/result.json @@ -0,0 +1,103 @@ +{ + "admission_stop_s": 493.2576180040196, + "arrival": "burst:8", + "clean": { + "admitted": 2233, + "completed": 2233, + "completed_throughput_rps": 9.304166666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 2858240, + "offered_rps": 9.304166666666667, + "output_tokens": 1143296, + "start_s": 60.0 + }, + "clean_segment_seconds": 80.0, + "drain_seconds": 14.923095890000695, + "elapsed_seconds": 508.1807138940203, + "failed_records": 0, + "load_point": "saturation", + "manifest_admitted": 3941, + "manifest_exhausted": false, + "manifest_rows": 32768, + "manifest_sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9", + "manifest_wrapped": false, + "max_in_flight": 256, + "num_clean_segments": 3, + "profiles": [ + { + "start_call_s": 300.0054247789958, + "start_return_s": 323.45228312400286, + "start_status": 200, + "stop_call_s": 354.6066670610162, + "stop_return_s": 381.28106322701205, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835082061801278.pt.trace.json.gz", + "trace_ready_s": 354.6066622030048, + "trace_sha256": "53d36238e82e5d031d8ed7841094c56d715b7cd5009368b64e380bf89a0f0dec", + "window": 1 + }, + { + "start_call_s": 411.30330635100836, + "start_return_s": 411.4401313569979, + "start_status": 200, + "stop_call_s": 436.0742275560042, + "stop_return_s": 463.23921855099616, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835169024306574.pt.trace.json.gz", + "trace_ready_s": 436.0742245099973, + "trace_sha256": "e088bd800a4e43446effbbcd4fce7d9ba1defaceb378a6e0db04d64780e1d09a", + "window": 2 + } + ], + "rate_fraction": null, + "records": 3941, + "request_rate": "inf", + "schema": 1, + "segments": [ + { + "admitted": 697, + "completed": 697, + "completed_throughput_rps": 8.7125, + "duration_s": 80.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 892160, + "name": "A", + "offered_rps": 8.7125, + "output_tokens": 356864, + "start_s": 60.0 + }, + { + "admitted": 768, + "completed": 768, + "completed_throughput_rps": 9.6, + "duration_s": 80.0, + "end_s": 220.0, + "failed": 0, + "input_tokens": 983040, + "name": "B", + "offered_rps": 9.6, + "output_tokens": 393216, + "start_s": 140.0 + }, + { + "admitted": 768, + "completed": 768, + "completed_throughput_rps": 9.6, + "duration_s": 80.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 983040, + "name": "C", + "offered_rps": 9.6, + "output_tokens": 393216, + "start_s": 220.0 + } + ], + "successful_records": 3941, + "t0_mono_ns": 166324009670273, + "t0_wall_ns": 1783834751136099481, + "warmup_seconds": 60.0 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P08-C00/sanity.json b/runs/opprof-phase3/phase3/remote-evidence/P08-C00/sanity.json new file mode 100644 index 0000000..6bc2dc9 --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P08-C00/sanity.json @@ -0,0 +1,65 @@ +{ + "invariants": { + "clean_duration_exact": true, + "clean_failures_zero": true, + "concurrency_bounded": true, + "drain_within_timeout": true, + "manifest_no_wrap": true, + "manifest_not_exhausted": true, + "output_tokens_exact": true, + "profile_count_exact": true, + "profile_status_ok": true, + "segment_count_exact": true + }, + "numeric": { + "actual_output_tokens": { + "distinct_n": 1, + "finite_n": 3941, + "max": 512.0, + "min": 512.0, + "missing_n": 0, + "n": 3941 + }, + "admitted_s": { + "distinct_n": 3941, + "finite_n": 3941, + "max": 493.25117621201207, + "min": 0.0009094980196096003, + "missing_n": 0, + "n": 3941 + }, + "completed_s": { + "distinct_n": 3941, + "finite_n": 3941, + "max": 508.1755032700021, + "min": 30.030000179016497, + "missing_n": 0, + "n": 3941 + }, + "input_tokens": { + "distinct_n": 1, + "finite_n": 3941, + "max": 1280.0, + "min": 1280.0, + "missing_n": 0, + "n": 3941 + }, + "requested_output_tokens": { + "distinct_n": 1, + "finite_n": 3941, + "max": 512.0, + "min": 512.0, + "missing_n": 0, + "n": 3941 + }, + "scheduled_s": { + "distinct_n": 3941, + "finite_n": 3941, + "max": 493.25117589501315, + "min": 0.000907983019715175, + "missing_n": 0, + "n": 3941 + } + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P10-C01/result.json b/runs/opprof-phase3/phase3/remote-evidence/P10-C01/result.json new file mode 100644 index 0000000..7e2306e --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P10-C01/result.json @@ -0,0 +1,103 @@ +{ + "admission_stop_s": 420.6437278209778, + "arrival": "steady", + "clean": { + "admitted": 178, + "completed": 178, + "completed_throughput_rps": 0.7416666666666667, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 1755158, + "offered_rps": 0.7416666666666667, + "output_tokens": 40658, + "start_s": 60.0 + }, + "clean_segment_seconds": 80.0, + "drain_seconds": 288.6191079240234, + "elapsed_seconds": 709.2628357450012, + "failed_records": 0, + "load_point": "saturation", + "manifest_admitted": 558, + "manifest_exhausted": false, + "manifest_rows": 4011, + "manifest_sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33", + "manifest_wrapped": false, + "max_in_flight": 256, + "num_clean_segments": 3, + "profiles": [ + { + "start_call_s": 300.0035294489935, + "start_return_s": 310.3520467719936, + "start_status": 200, + "stop_call_s": 325.0655262139917, + "stop_return_s": 338.54296391498065, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835066783986388.pt.trace.json.gz", + "trace_ready_s": 325.0655210709956, + "trace_sha256": "d457af416fa91f44e617c7259167fca467bea8466a3e4c4b916f811fb5899c73", + "window": 1 + }, + { + "start_call_s": 368.55468072599615, + "start_return_s": 372.31818850699347, + "start_status": 200, + "stop_call_s": 386.0271748309897, + "stop_return_s": 390.6334540609969, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783835128341345321.pt.trace.json.gz", + "trace_ready_s": 386.02717122397735, + "trace_sha256": "41e8b34a0464cfd92d16b21c6242113b23594081697f5d1e88cdc3647542a96a", + "window": 2 + } + ], + "rate_fraction": null, + "records": 558, + "request_rate": "inf", + "schema": 1, + "segments": [ + { + "admitted": 48, + "completed": 48, + "completed_throughput_rps": 0.6, + "duration_s": 80.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 548329, + "name": "A", + "offered_rps": 0.6, + "output_tokens": 11027, + "start_s": 60.0 + }, + { + "admitted": 70, + "completed": 70, + "completed_throughput_rps": 0.875, + "duration_s": 80.0, + "end_s": 220.0, + "failed": 0, + "input_tokens": 574251, + "name": "B", + "offered_rps": 0.875, + "output_tokens": 16298, + "start_s": 140.0 + }, + { + "admitted": 60, + "completed": 60, + "completed_throughput_rps": 0.75, + "duration_s": 80.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 632578, + "name": "C", + "offered_rps": 0.75, + "output_tokens": 13333, + "start_s": 220.0 + } + ], + "successful_records": 558, + "t0_mono_ns": 166325149689000, + "t0_wall_ns": 1783834752276118250, + "warmup_seconds": 60.0 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/P10-C01/sanity.json b/runs/opprof-phase3/phase3/remote-evidence/P10-C01/sanity.json new file mode 100644 index 0000000..d8ffc3d --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/P10-C01/sanity.json @@ -0,0 +1,65 @@ +{ + "invariants": { + "clean_duration_exact": true, + "clean_failures_zero": true, + "concurrency_bounded": true, + "drain_within_timeout": false, + "manifest_no_wrap": true, + "manifest_not_exhausted": true, + "output_tokens_exact": true, + "profile_count_exact": true, + "profile_status_ok": true, + "segment_count_exact": true + }, + "numeric": { + "actual_output_tokens": { + "distinct_n": 92, + "finite_n": 558, + "max": 256.0, + "min": 6.0, + "missing_n": 0, + "n": 558 + }, + "admitted_s": { + "distinct_n": 558, + "finite_n": 558, + "max": 420.08466253298684, + "min": 0.000850101001560688, + "missing_n": 0, + "n": 558 + }, + "completed_s": { + "distinct_n": 558, + "finite_n": 558, + "max": 709.2613093179825, + "min": 14.001143716974184, + "missing_n": 0, + "n": 558 + }, + "input_tokens": { + "distinct_n": 535, + "finite_n": 558, + "max": 32527.0, + "min": 70.0, + "missing_n": 0, + "n": 558 + }, + "requested_output_tokens": { + "distinct_n": 92, + "finite_n": 558, + "max": 256.0, + "min": 6.0, + "missing_n": 0, + "n": 558 + }, + "scheduled_s": { + "distinct_n": 558, + "finite_n": 558, + "max": 420.08466206499725, + "min": 0.0008485929865855724, + "missing_n": 0, + "n": 558 + } + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/controller-state.json b/runs/opprof-phase3/phase3/remote-evidence/controller-state.json new file mode 100644 index 0000000..2d9bdca --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/controller-state.json @@ -0,0 +1,258 @@ +{ + "completed_burnins": 5, + "completed_measured_runs": 0, + "controller_pid": 2187729, + "created_at": 1783833885.960389, + "fingerprint": { + "cells": [ + "P08-C00", + "P03-C10", + "P07-C00", + "P10-C01", + "P01-C10", + "P01-C01", + "P10-C10", + "P03-C01", + "P09-C00", + "P06-C01", + "P10-C11", + "P01-C00", + "P01-C11", + "P10-C00", + "P04-C00", + "P03-C00", + "P06-C10", + "P02-C00", + "P06-C00", + "P06-C11", + "P11-C00", + "P10-C00-TP2", + "P03-C11", + "P05-C00" + ], + "client_sha256": "a87d92efecd5a8765b51067800b6382f9b174a2ede65f8933fcc9f846ff03d84", + "common_controller_sha256": "15ad254298a38c4a9318468db89ab32707b6196bb38d5e2a007f2267529397a5", + "controller_sha256": "c2d3232fb99c66cea55e30e6cb1aa6a84d3a816434ed8155a309f41df2837f73", + "cpu_map": { + "0": "0-19", + "1": "20-39", + "2": "40-59", + "3": "60-79", + "4": "80-99", + "5": "100-119", + "6": "120-139", + "7": "140-159" + }, + "manifests": { + "P01": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl", + "rows": 32768, + "sha256": "13ffb226c83373f54c4a7afea6c78cb7cd29720f1858d56728826fc1367b31a4" + }, + "P02": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P02.jsonl", + "rows": 32768, + "sha256": "0138ada3fccc98298daee66c26bd1952c987cb42ed8d5341d66b698a597417f9" + }, + "P03": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P03.jsonl", + "rows": 32768, + "sha256": "432cfbc26d36f105c179c83f3bb0f3b24b8b3f205788263b4171797f0a4d6fa1" + }, + "P04": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P04.jsonl", + "rows": 32768, + "sha256": "caf8d0941b093956a81e1413adc4a4ea9d92460f1b2ed0f6a9b118d0119d6247" + }, + "P05": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P05.jsonl", + "rows": 32768, + "sha256": "192e213109f8cb429b99d9eb0f227bb4390fc03f63b02d5456569369bff5a3d7" + }, + "P06": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P06.jsonl", + "rows": 32768, + "sha256": "65954bc6e47de9e7be07b8975f97f0bc4639979ef6d33e8c664557ded34b9f96" + }, + "P07": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P07.jsonl", + "rows": 32768, + "sha256": "74df66e21a705cd583493199a875e226e93d2da7cfede9bca81dfd9bcb8c9cc6" + }, + "P08": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P08.jsonl", + "rows": 32768, + "sha256": "0c4f835aae099265c3eb596d06c6c2fa7070dd280558eb289c602e8c3434dfe9" + }, + "P09": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", + "rows": 32768, + "sha256": "7af92ee3c27dc7d2cf895d6ff3a6e737ec4b6da13d6841ca59e1166f28a0ae1e" + }, + "P10": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P10.jsonl", + "rows": 4011, + "sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33" + }, + "P11": { + "path": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P11.jsonl", + "rows": 32768, + "sha256": "7d196df38963528ff181cf72ce39c8ad913c8f61d40b1425410d3c6c30b6be18" + } + }, + "model": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B", + "source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05", + "source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1" + }, + "gpu_hours_this_stage": 0.8836704309119119, + "gpu_hours_total": 5.473521912336349, + "schema": 1, + "stages": { + "burnin-01": { + "assignments": [ + { + "cell": "P06-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P06-C10", + "gpus": [ + 1 + ] + }, + { + "cell": "P06-C01", + "gpus": [ + 2 + ] + }, + { + "cell": "P06-C11", + "gpus": [ + 3 + ] + } + ], + "burnin": true, + "clients": {}, + "completed_at": 1783834288.1238313, + "confirmation": false, + "gpu_hours": 0.30808498481909435, + "load_point": "saturation", + "profile": false, + "servers": {}, + "started_at": 1783833886.4448946, + "status": "complete", + "validation_attempt1_failure": "RuntimeError(\"server config/backend failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/burnins/C01: {'triton_moe': True, 'chunked_mbt': False, 'tp_effective': True, 'drain_shutdown': True, 'mns_effective': True}\")" + }, + "burnin-02": { + "assignments": [ + { + "cell": "P06-C00-TP2", + "gpus": [ + 0, + 1 + ] + } + ], + "burnin": true, + "clients": {}, + "completed_at": 1783834671.921921, + "confirmation": false, + "gpu_hours": 0.19903395679261948, + "load_point": "saturation", + "profile": false, + "servers": {}, + "started_at": 1783834301.7813473, + "status": "complete" + }, + "primary-01-saturation": { + "assignments": [ + { + "cell": "P08-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "P03-C10", + "gpus": [ + 1 + ] + }, + { + "cell": "P07-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "P10-C01", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": { + "P03-C10-saturation": { + "pgid": 2195599, + "pid": 2195599 + }, + "P07-C00-saturation": { + "pgid": 2195600, + "pid": 2195600 + }, + "P08-C00-saturation": { + "pgid": 2195597, + "pid": 2195597 + }, + "P10-C01-saturation": { + "pgid": 2195601, + "pid": 2195601 + } + }, + "confirmation": false, + "failure": "RuntimeError(\"client failures: {'P10-C01-saturation': 1}\")", + "gpu_hours": 0.8836704309119119, + "load_point": "saturation", + "profile": true, + "servers": { + "P03-C10-saturation": { + "gpus": [ + 1 + ], + "pgid": 2193680, + "pid": 2193680 + }, + "P07-C00-saturation": { + "gpus": [ + 2 + ], + "pgid": 2193681, + "pid": 2193681 + }, + "P08-C00-saturation": { + "gpus": [ + 0 + ], + "pgid": 2193679, + "pid": 2193679 + }, + "P10-C01-saturation": { + "gpus": [ + 3 + ], + "pgid": 2193682, + "pid": 2193682 + } + }, + "started_at": 1783834671.9563339, + "status": "failed" + } + }, + "status": "failed", + "updated_at": 1783835479.1651883 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/repair-001-explicit-mbt-log.json b/runs/opprof-phase3/phase3/remote-evidence/repair-001-explicit-mbt-log.json new file mode 100644 index 0000000..39c75ef --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/repair-001-explicit-mbt-log.json @@ -0,0 +1,16 @@ +{ + "at": 1783834288.13871, + "finding": "Explicit MBT=2048 is logged in non-default args; scheduler.py emits the separate Chunked-prefill line only for default MBT.", + "gpu_rerun": false, + "kind": "validator-only", + "new_controller_sha256": "c2d3232fb99c66cea55e30e6cb1aa6a84d3a816434ed8155a309f41df2837f73", + "old_controller_sha256": "74859614a8da3341bf827bb75b667db7d219f059d1d29c1182b9a77c2ee59e92", + "protocol_or_workload_changed": false, + "revalidated_runs": [ + "P06-C00-burnin", + "P06-C10-burnin", + "P06-C01-burnin", + "P06-C11-burnin" + ], + "schema": 1 +} diff --git a/runs/opprof-phase3/phase3/remote-evidence/sidecar-verification-state.json b/runs/opprof-phase3/phase3/remote-evidence/sidecar-verification-state.json new file mode 100644 index 0000000..292dcf7 --- /dev/null +++ b/runs/opprof-phase3/phase3/remote-evidence/sidecar-verification-state.json @@ -0,0 +1,355 @@ +{ + "gpu_seconds": 427.43714332580566, + "results": { + "graceful": { + "accounting": { + "bytes": 1462585, + "checkpoint_age_seconds": -0.382308841, + "counters": { + "dropped_records": 0, + "encoded_records": 1598, + "written_records": 1598 + }, + "footer_count": 1, + "invariants": { + "all_schema_1": true, + "encoded_balanced": true, + "final_sidecar": true, + "footer_sidecar_agree": true, + "last_step_matches": true, + "one_footer_last": true, + "steps_contiguous": true, + "written_matches_records": true, + "zero_drops": true + }, + "last_step_index": 1597, + "records": 1598, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/graceful/opprof/opprof-v1-dp0-pid2163417-1783832631507578426.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/graceful/opprof/opprof-v1-dp0-pid2163417-1783832631507578426.jsonl" + }, + "clean": { + "admitted": 5202, + "completed": 5202, + "completed_throughput_rps": 43.35, + "duration_s": 120.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 1662118, + "offered_rps": 43.35, + "output_tokens": 332928, + "start_s": 20.0 + }, + "failed_records": 0, + "gpu_seconds": 219.92383646965027, + "gpu_zero_samples": [ + [ + { + "index": 0, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 1, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 2, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 3, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 4, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 5, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 6, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 7, + "memory_mib": 0, + "utilization_pct": 0 + } + ], + [ + { + "index": 0, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 1, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 2, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 3, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 4, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 5, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 6, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 7, + "memory_mib": 0, + "utilization_pct": 0 + } + ], + [ + { + "index": 0, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 1, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 2, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 3, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 4, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 5, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 6, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 7, + "memory_mib": 0, + "utilization_pct": 0 + } + ] + ], + "mode": "graceful", + "schema": 1, + "server_returncode": 0, + "status": "pass", + "termination_wall_ns": 1783832775735239240 + }, + "hard-kill": { + "accounting": { + "bytes": 1460860, + "checkpoint_age_seconds": 0.02436319, + "counters": { + "dropped_records": 0, + "encoded_records": 1596, + "written_records": 1596 + }, + "footer_count": 0, + "invariants": { + "all_schema_1": true, + "checkpoint_sidecar": true, + "checkpoint_within_bound": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "written_matches_records": true, + "zero_drops": true + }, + "last_step_index": 1595, + "records": 1596, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/hard-kill/opprof/opprof-v1-dp0-pid2166360-1783832841962667333.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/e-b-sidecar-verification/hard-kill/opprof/opprof-v1-dp0-pid2166360-1783832841962667333.jsonl" + }, + "clean": { + "admitted": 5179, + "completed": 5179, + "completed_throughput_rps": 43.15833333333333, + "duration_s": 120.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 1655384, + "offered_rps": 43.15833333333333, + "output_tokens": 331456, + "start_s": 20.0 + }, + "failed_records": 0, + "gpu_seconds": 207.5133068561554, + "gpu_zero_samples": [ + [ + { + "index": 0, + "memory_mib": 0, + "utilization_pct": 1 + }, + { + "index": 1, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 2, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 3, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 4, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 5, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 6, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 7, + "memory_mib": 0, + "utilization_pct": 0 + } + ], + [ + { + "index": 0, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 1, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 2, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 3, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 4, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 5, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 6, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 7, + "memory_mib": 0, + "utilization_pct": 0 + } + ], + [ + { + "index": 0, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 1, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 2, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 3, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 4, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 5, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 6, + "memory_mib": 0, + "utilization_pct": 0 + }, + { + "index": 7, + "memory_mib": 0, + "utilization_pct": 0 + } + ] + ], + "mode": "hard-kill", + "schema": 1, + "server_returncode": -9, + "status": "pass", + "termination_wall_ns": 1783832986410202706 + } + }, + "schema": 1, + "status": "complete" +} diff --git a/runs/opprof-phase3/phase3/repair-008-ap36.json b/runs/opprof-phase3/phase3/repair-008-ap36.json new file mode 100644 index 0000000..e2ca344 --- /dev/null +++ b/runs/opprof-phase3/phase3/repair-008-ap36.json @@ -0,0 +1,17 @@ +{ + "accepted_runs_preserved": 40, + "amendment": "A-P3-6", + "changed_fingerprint_keys": [ + "controller_sha256" + ], + "created_at": 1783853258.3098204, + "failed_stage_preserved": "primary-06-saturation", + "new_controller_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f", + "old_controller_sha256": "becfe00889274b51023016b1e7edb866d10e477249504f2032859a4d621f295f", + "projected_remaining_h20_hours": 2.1, + "projected_total_h20_hours": 15.408482965071997, + "repair_id": "repair-008-ap36", + "retained_attempt_re_adjudicated": false, + "retained_attempt_reason": "21 completions but A-P3-6 normalized drift 2.3385345997286295 and bin step counts 13/9/44", + "schema": 1 +} diff --git a/runs/opprof-phase3/phase4/capture-p09/controller-state.json b/runs/opprof-phase3/phase4/capture-p09/controller-state.json new file mode 100644 index 0000000..89be26a --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/controller-state.json @@ -0,0 +1,469 @@ +{ + "arms": { + "OFF": { + "client_pid": 2487398, + "server_pid": 2486295, + "started_at": 1783856664.5207698, + "status": "complete", + "summary": { + "arm": "OFF", + "bucket_tokens": 237911, + "capture_sizes": "default", + "clean_completed": 1184, + "clean_completed_throughput_rps": 4.933333333333334, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7853007119847462, + "e2e_latency_mean_s": 1.681152764688729, + "e2e_latency_p95_s": 3.8763178953449815, + "gpu_hours": 0.13751606033907995, + "graph_hit_steps": 12252, + "graph_miss_rate": 0.045199501246882795, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 16491, + "model_step_duration_ms": 479367.749483, + "model_steps": 12832, + "padding_fraction": 0.08545632610514016, + "padding_tokens": 20331, + "schema": 1, + "token_efficiency_per_ms": 4.5540506267984115, + "useful_tokens": 2183065 + } + }, + "ON": { + "client_pid": 2479800, + "server_pid": 2477652, + "started_at": 1783856086.549906, + "status": "complete", + "summary": { + "arm": "ON", + "bucket_tokens": 227714, + "capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "clean_completed": 1190, + "clean_completed_throughput_rps": 4.958333333333333, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7876331160077825, + "e2e_latency_mean_s": 1.612280680370388, + "e2e_latency_p95_s": 3.9930475192406423, + "gpu_hours": 0.15887338949574364, + "graph_hit_steps": 14253, + "graph_miss_rate": 0.0389724226282786, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 17579, + "model_step_duration_ms": 478655.543996, + "model_steps": 14831, + "padding_fraction": 0.035658764941988635, + "padding_tokens": 8120, + "schema": 1, + "token_efficiency_per_ms": 4.562222306607711, + "useful_tokens": 2183733 + } + } + }, + "controller_pid": 2477456, + "created_at": 1783856081.3481774, + "gpu_hours_increment": 0.2963894498348236, + "plan": { + "added_capture_sizes": [ + 3, + 5, + 6, + 7, + 9 + ], + "clean_seconds": 240, + "gpu": 0, + "gpu_hour_limit": 16.0, + "load": "moderate", + "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", + "measured_padding_recovery_bound": 0.049776380388728225, + "on_capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "order": [ + "ON", + "OFF" + ], + "pattern": "P09", + "primary_metric": "clean graph-hit padding_fraction", + "prior_gpu_hours": 14.025875418755744, + "profile": false, + "projected_increment_gpu_hours": 0.5, + "projected_total_gpu_hours": 14.525875418755744, + "rate_fraction": 0.6, + "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json", + "schema": 1, + "secondary_metrics": [ + "Layer-1 useful scheduled tokens/model-step millisecond", + "clean completed request throughput", + "clean request latency" + ], + "warmup_seconds": 60 + }, + "result": { + "delta": { + "completed_throughput_relative": 0.0050675675675675436, + "e2e_mean_latency_relative": -0.04096717785851722, + "e2e_p95_latency_relative": 0.03011353223527924, + "padding_fraction_points": -0.049797561163151524, + "padding_reduction_fraction": 0.5827252753866776, + "token_efficiency_relative": 0.0017943761453185214 + }, + "gpu_hours_increment": 0.2963894498348236, + "gpu_hours_total": 14.322264868590567, + "off": { + "arm": "OFF", + "bucket_tokens": 237911, + "capture_sizes": "default", + "clean_completed": 1184, + "clean_completed_throughput_rps": 4.933333333333334, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7853007119847462, + "e2e_latency_mean_s": 1.681152764688729, + "e2e_latency_p95_s": 3.8763178953449815, + "gpu_hours": 0.13751606033907995, + "graph_hit_steps": 12252, + "graph_miss_rate": 0.045199501246882795, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 16491, + "model_step_duration_ms": 479367.749483, + "model_steps": 12832, + "padding_fraction": 0.08545632610514016, + "padding_tokens": 20331, + "schema": 1, + "token_efficiency_per_ms": 4.5540506267984115, + "useful_tokens": 2183065 + }, + "on": { + "arm": "ON", + "bucket_tokens": 227714, + "capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "clean_completed": 1190, + "clean_completed_throughput_rps": 4.958333333333333, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7876331160077825, + "e2e_latency_mean_s": 1.612280680370388, + "e2e_latency_p95_s": 3.9930475192406423, + "gpu_hours": 0.15887338949574364, + "graph_hit_steps": 14253, + "graph_miss_rate": 0.0389724226282786, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 17579, + "model_step_duration_ms": 478655.543996, + "model_steps": 14831, + "padding_fraction": 0.035658764941988635, + "padding_tokens": 8120, + "schema": 1, + "token_efficiency_per_ms": 4.562222306607711, + "useful_tokens": 2183733 + }, + "plan": { + "added_capture_sizes": [ + 3, + 5, + 6, + 7, + 9 + ], + "clean_seconds": 240, + "gpu": 0, + "gpu_hour_limit": 16.0, + "load": "moderate", + "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", + "measured_padding_recovery_bound": 0.049776380388728225, + "on_capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "order": [ + "ON", + "OFF" + ], + "pattern": "P09", + "primary_metric": "clean graph-hit padding_fraction", + "prior_gpu_hours": 14.025875418755744, + "profile": false, + "projected_increment_gpu_hours": 0.5, + "projected_total_gpu_hours": 14.525875418755744, + "rate_fraction": 0.6, + "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json", + "schema": 1, + "secondary_metrics": [ + "Layer-1 useful scheduled tokens/model-step millisecond", + "clean completed request throughput", + "clean request latency" + ], + "warmup_seconds": 60 + }, + "schema": 1 + }, + "schema": 1, + "status": "complete", + "updated_at": 1783857160.3890784 +} diff --git a/runs/opprof-phase3/phase4/capture-p09/controller.log b/runs/opprof-phase3/phase4/capture-p09/controller.log new file mode 100644 index 0000000..bc26bd7 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/controller.log @@ -0,0 +1,5 @@ +GPU_COMMAND P09-capture-ON-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512; expected=6-9m +GPU_COMMAND P09-capture-ON-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/on/client; expected=5-7m +GPU_COMMAND P09-capture-OFF-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120; expected=6-9m +GPU_COMMAND P09-capture-OFF-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/off/client; expected=5-7m +{"delta": {"completed_throughput_relative": 0.0050675675675675436, "e2e_mean_latency_relative": -0.04096717785851722, "e2e_p95_latency_relative": 0.03011353223527924, "padding_fraction_points": -0.049797561163151524, "padding_reduction_fraction": 0.5827252753866776, "token_efficiency_relative": 0.0017943761453185214}, "gpu_hours_increment": 0.2963894498348236, "gpu_hours_total": 14.322264868590567, "off": {"arm": "OFF", "bucket_tokens": 237911, "capture_sizes": "default", "clean_completed": 1184, "clean_completed_throughput_rps": 4.933333333333334, "clean_failed": 0, "clean_offered_rps": 4.920833333333333, "drain_seconds": 0.7853007119847462, "e2e_latency_mean_s": 1.681152764688729, "e2e_latency_p95_s": 3.8763178953449815, "gpu_hours": 0.13751606033907995, "graph_hit_steps": 12252, "graph_miss_rate": 0.045199501246882795, "layer1_invariants": {"cudagraph_identity": true, "footer_balanced": true, "footer_written_matches": true, "schema_1": true, "sidecar_agrees": true, "sidecar_final": true, "steps_unique_contiguous": true, "token_composition": true, "zero_drops": true}, "layer1_records": 16491, "model_step_duration_ms": 479367.749483, "model_steps": 12832, "padding_fraction": 0.08545632610514016, "padding_tokens": 20331, "schema": 1, "token_efficiency_per_ms": 4.5540506267984115, "useful_tokens": 2183065}, "on": {"arm": "ON", "bucket_tokens": 227714, "capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], "clean_completed": 1190, "clean_completed_throughput_rps": 4.958333333333333, "clean_failed": 0, "clean_offered_rps": 4.920833333333333, "drain_seconds": 0.7876331160077825, "e2e_latency_mean_s": 1.612280680370388, "e2e_latency_p95_s": 3.9930475192406423, "gpu_hours": 0.15887338949574364, "graph_hit_steps": 14253, "graph_miss_rate": 0.0389724226282786, "layer1_invariants": {"cudagraph_identity": true, "footer_balanced": true, "footer_written_matches": true, "schema_1": true, "sidecar_agrees": true, "sidecar_final": true, "steps_unique_contiguous": true, "token_composition": true, "zero_drops": true}, "layer1_records": 17579, "model_step_duration_ms": 478655.543996, "model_steps": 14831, "padding_fraction": 0.035658764941988635, "padding_tokens": 8120, "schema": 1, "token_efficiency_per_ms": 4.562222306607711, "useful_tokens": 2183733}, "plan": {"added_capture_sizes": [3, 5, 6, 7, 9], "clean_seconds": 240, "gpu": 0, "gpu_hour_limit": 16.0, "load": "moderate", "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", "measured_padding_recovery_bound": 0.049776380388728225, "on_capture_sizes": [1, 2, 3, 4, 5, 6, 7, 8, 9, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], "order": ["ON", "OFF"], "pattern": "P09", "primary_metric": "clean graph-hit padding_fraction", "prior_gpu_hours": 14.025875418755744, "profile": false, "projected_increment_gpu_hours": 0.5, "projected_total_gpu_hours": 14.525875418755744, "rate_fraction": 0.6, "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json", "schema": 1, "secondary_metrics": ["Layer-1 useful scheduled tokens/model-step millisecond", "clean completed request throughput", "clean request latency"], "warmup_seconds": 60}, "schema": 1} diff --git a/runs/opprof-phase3/phase4/capture-p09/launch.log b/runs/opprof-phase3/phase4/capture-p09/launch.log new file mode 100644 index 0000000..64c83ea --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/launch.log @@ -0,0 +1 @@ +GPU_VALIDATION Phase4 P09 capture sizes: order=ON,OFF, GPU0 TP1, manifest=/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl, saturation_source=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json, added_sizes=3,5,6,7,9, warmup=60s, clean=240s/arm, projected=0.50 H20-hours, cumulative_projection=14.525875/16, expected_wall=15-22m diff --git a/runs/opprof-phase3/phase4/capture-p09/off/client-sanity.json b/runs/opprof-phase3/phase4/capture-p09/off/client-sanity.json new file mode 100644 index 0000000..a62b0c9 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/off/client-sanity.json @@ -0,0 +1,66 @@ +{ + "invariants": { + "clean_duration_exact": true, + "clean_failures_zero": true, + "concurrency_bounded": true, + "drain_within_timeout": true, + "manifest_no_wrap": true, + "manifest_not_exhausted": true, + "moderate_offered_within_5pct": true, + "output_tokens_exact": true, + "profile_count_exact": true, + "profile_status_ok": true, + "segment_count_exact": true + }, + "numeric": { + "actual_output_tokens": { + "distinct_n": 1, + "finite_n": 1477, + "max": 64.0, + "min": 64.0, + "missing_n": 0, + "n": 1477 + }, + "admitted_s": { + "distinct_n": 1477, + "finite_n": 1477, + "max": 299.8483776419889, + "min": 0.00022695399820804596, + "missing_n": 0, + "n": 1477 + }, + "completed_s": { + "distinct_n": 1477, + "finite_n": 1477, + "max": 300.78334441498737, + "min": 0.6682249770092312, + "missing_n": 0, + "n": 1477 + }, + "input_tokens": { + "distinct_n": 944, + "finite_n": 1477, + "max": 8161.0, + "min": 128.0, + "missing_n": 0, + "n": 1477 + }, + "requested_output_tokens": { + "distinct_n": 1, + "finite_n": 1477, + "max": 64.0, + "min": 64.0, + "missing_n": 0, + "n": 1477 + }, + "scheduled_s": { + "distinct_n": 1477, + "finite_n": 1477, + "max": 299.84763839512016, + "min": 0.0, + "missing_n": 0, + "n": 1477 + } + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase4/capture-p09/off/commands.log b/runs/opprof-phase3/phase4/capture-p09/off/commands.log new file mode 100644 index 0000000..f408830 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/off/commands.log @@ -0,0 +1,2 @@ +GPU_COMMAND P09-capture-OFF-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 ; expected=6-9m +GPU_COMMAND P09-capture-OFF-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/off/client ; expected=5-7m diff --git a/runs/opprof-phase3/phase4/capture-p09/off/summary.json b/runs/opprof-phase3/phase4/capture-p09/off/summary.json new file mode 100644 index 0000000..326739f --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/off/summary.json @@ -0,0 +1,34 @@ +{ + "arm": "OFF", + "bucket_tokens": 237911, + "capture_sizes": "default", + "clean_completed": 1184, + "clean_completed_throughput_rps": 4.933333333333334, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7853007119847462, + "e2e_latency_mean_s": 1.681152764688729, + "e2e_latency_p95_s": 3.8763178953449815, + "gpu_hours": 0.13751606033907995, + "graph_hit_steps": 12252, + "graph_miss_rate": 0.045199501246882795, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 16491, + "model_step_duration_ms": 479367.749483, + "model_steps": 12832, + "padding_fraction": 0.08545632610514016, + "padding_tokens": 20331, + "schema": 1, + "token_efficiency_per_ms": 4.5540506267984115, + "useful_tokens": 2183065 +} diff --git a/runs/opprof-phase3/phase4/capture-p09/on/client-sanity.json b/runs/opprof-phase3/phase4/capture-p09/on/client-sanity.json new file mode 100644 index 0000000..9cf8a24 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/on/client-sanity.json @@ -0,0 +1,66 @@ +{ + "invariants": { + "clean_duration_exact": true, + "clean_failures_zero": true, + "concurrency_bounded": true, + "drain_within_timeout": true, + "manifest_no_wrap": true, + "manifest_not_exhausted": true, + "moderate_offered_within_5pct": true, + "output_tokens_exact": true, + "profile_count_exact": true, + "profile_status_ok": true, + "segment_count_exact": true + }, + "numeric": { + "actual_output_tokens": { + "distinct_n": 1, + "finite_n": 1477, + "max": 64.0, + "min": 64.0, + "missing_n": 0, + "n": 1477 + }, + "admitted_s": { + "distinct_n": 1477, + "finite_n": 1477, + "max": 299.8487557930057, + "min": 0.00024887899053283036, + "missing_n": 0, + "n": 1477 + }, + "completed_s": { + "distinct_n": 1477, + "finite_n": 1477, + "max": 300.78616064199014, + "min": 15.648636110010557, + "missing_n": 0, + "n": 1477 + }, + "input_tokens": { + "distinct_n": 944, + "finite_n": 1477, + "max": 8161.0, + "min": 128.0, + "missing_n": 0, + "n": 1477 + }, + "requested_output_tokens": { + "distinct_n": 1, + "finite_n": 1477, + "max": 64.0, + "min": 64.0, + "missing_n": 0, + "n": 1477 + }, + "scheduled_s": { + "distinct_n": 1477, + "finite_n": 1477, + "max": 299.84763839512016, + "min": 0.0, + "missing_n": 0, + "n": 1477 + } + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase4/capture-p09/on/commands.log b/runs/opprof-phase3/phase4/capture-p09/on/commands.log new file mode 100644 index 0000000..a4f3e7d --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/on/commands.log @@ -0,0 +1,2 @@ +GPU_COMMAND P09-capture-ON-server: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8200 --tensor-parallel-size 1 --enable-chunked-prefill --enable-prefix-caching --shutdown-timeout 120 --cudagraph-capture-sizes 1 2 3 4 5 6 7 8 9 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248 256 272 288 304 320 336 352 368 384 400 416 432 448 464 480 496 512 ; expected=6-9m +GPU_COMMAND P09-capture-ON-client: taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/scripts/opprof_phase3_client.py run --manifest /home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl --base-url http://127.0.0.1:8200 --model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --load-point moderate --saturation-result /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json --rate-fraction 0.60 --max-concurrency 256 --ignore-eos --temperature 0 --warmup-seconds 60 --clean-segment-seconds 80 --num-clean-segments 3 --recovery-seconds 30 --drain-timeout-seconds 120 --workload-seed 20260712 --server-seed 20260712 --result-dir /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase4-capture-p09/on/client ; expected=5-7m diff --git a/runs/opprof-phase3/phase4/capture-p09/on/summary.json b/runs/opprof-phase3/phase4/capture-p09/on/summary.json new file mode 100644 index 0000000..82e7a94 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/on/summary.json @@ -0,0 +1,91 @@ +{ + "arm": "ON", + "bucket_tokens": 227714, + "capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "clean_completed": 1190, + "clean_completed_throughput_rps": 4.958333333333333, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7876331160077825, + "e2e_latency_mean_s": 1.612280680370388, + "e2e_latency_p95_s": 3.9930475192406423, + "gpu_hours": 0.15887338949574364, + "graph_hit_steps": 14253, + "graph_miss_rate": 0.0389724226282786, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 17579, + "model_step_duration_ms": 478655.543996, + "model_steps": 14831, + "padding_fraction": 0.035658764941988635, + "padding_tokens": 8120, + "schema": 1, + "token_efficiency_per_ms": 4.562222306607711, + "useful_tokens": 2183733 +} diff --git a/runs/opprof-phase3/phase4/capture-p09/result.json b/runs/opprof-phase3/phase4/capture-p09/result.json new file mode 100644 index 0000000..99fd735 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture-p09/result.json @@ -0,0 +1,230 @@ +{ + "delta": { + "completed_throughput_relative": 0.0050675675675675436, + "e2e_mean_latency_relative": -0.04096717785851722, + "e2e_p95_latency_relative": 0.03011353223527924, + "padding_fraction_points": -0.049797561163151524, + "padding_reduction_fraction": 0.5827252753866776, + "token_efficiency_relative": 0.0017943761453185214 + }, + "gpu_hours_increment": 0.2963894498348236, + "gpu_hours_total": 14.322264868590567, + "off": { + "arm": "OFF", + "bucket_tokens": 237911, + "capture_sizes": "default", + "clean_completed": 1184, + "clean_completed_throughput_rps": 4.933333333333334, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7853007119847462, + "e2e_latency_mean_s": 1.681152764688729, + "e2e_latency_p95_s": 3.8763178953449815, + "gpu_hours": 0.13751606033907995, + "graph_hit_steps": 12252, + "graph_miss_rate": 0.045199501246882795, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 16491, + "model_step_duration_ms": 479367.749483, + "model_steps": 12832, + "padding_fraction": 0.08545632610514016, + "padding_tokens": 20331, + "schema": 1, + "token_efficiency_per_ms": 4.5540506267984115, + "useful_tokens": 2183065 + }, + "on": { + "arm": "ON", + "bucket_tokens": 227714, + "capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "clean_completed": 1190, + "clean_completed_throughput_rps": 4.958333333333333, + "clean_failed": 0, + "clean_offered_rps": 4.920833333333333, + "drain_seconds": 0.7876331160077825, + "e2e_latency_mean_s": 1.612280680370388, + "e2e_latency_p95_s": 3.9930475192406423, + "gpu_hours": 0.15887338949574364, + "graph_hit_steps": 14253, + "graph_miss_rate": 0.0389724226282786, + "layer1_invariants": { + "cudagraph_identity": true, + "footer_balanced": true, + "footer_written_matches": true, + "schema_1": true, + "sidecar_agrees": true, + "sidecar_final": true, + "steps_unique_contiguous": true, + "token_composition": true, + "zero_drops": true + }, + "layer1_records": 17579, + "model_step_duration_ms": 478655.543996, + "model_steps": 14831, + "padding_fraction": 0.035658764941988635, + "padding_tokens": 8120, + "schema": 1, + "token_efficiency_per_ms": 4.562222306607711, + "useful_tokens": 2183733 + }, + "plan": { + "added_capture_sizes": [ + 3, + 5, + 6, + 7, + 9 + ], + "clean_seconds": 240, + "gpu": 0, + "gpu_hour_limit": 16.0, + "load": "moderate", + "manifest": "/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P09.jsonl", + "measured_padding_recovery_bound": 0.049776380388728225, + "on_capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 9, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "order": [ + "ON", + "OFF" + ], + "pattern": "P09", + "primary_metric": "clean graph-hit padding_fraction", + "prior_gpu_hours": 14.025875418755744, + "profile": false, + "projected_increment_gpu_hours": 0.5, + "projected_total_gpu_hours": 14.525875418755744, + "rate_fraction": 0.6, + "saturation_result": "/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase3/primary/P09-C00/saturation/client/result.json", + "schema": 1, + "secondary_metrics": [ + "Layer-1 useful scheduled tokens/model-step millisecond", + "clean completed request throughput", + "clean request latency" + ], + "warmup_seconds": 60 + }, + "schema": 1 +} diff --git a/runs/opprof-phase3/phase4/capture_validation.py b/runs/opprof-phase3/phase4/capture_validation.py new file mode 100644 index 0000000..8dcbc70 --- /dev/null +++ b/runs/opprof-phase3/phase4/capture_validation.py @@ -0,0 +1,373 @@ +#!/usr/bin/env python3 +"""One-pair P09 CUDAGraph capture-size validation for OpProf Phase 4.""" + +from __future__ import annotations + +import argparse +import json +import math +import os +import shlex +import signal +import subprocess +import sys +import time +import urllib.request +from pathlib import Path +from typing import Any + +import numpy as np + +import opprof_phase3_controller as common +import opprof_phase3_matrix as matrix + + +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +ROOT = WORKDIR / "runs/phase4-capture-p09" +PHASE3 = WORKDIR / "runs/phase3" +PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0") +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +STATE = ROOT / "controller-state.json" +GPU = 0 +PORT = 8200 +PRIOR_GPU_HOURS = 14.025875418755744 +GPU_HOUR_LIMIT = 16.0 +EXPECTED_INCREMENT_HOURS = 0.5 +ADDED_CAPTURE_SIZES = (3, 5, 6, 7, 9) +DEFAULT_CAPTURE_SIZES = ( + 1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, + 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, + 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, + 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, +) +ON_CAPTURE_SIZES = tuple(sorted(set(DEFAULT_CAPTURE_SIZES + ADDED_CAPTURE_SIZES))) + + +def plan() -> dict[str, Any]: + return { + "schema": 1, + "pattern": "P09", + "load": "moderate", + "order": ["ON", "OFF"], + "gpu": GPU, + "warmup_seconds": 60, + "clean_seconds": 240, + "profile": False, + "manifest": str(PRIVATE / "P09.jsonl"), + "saturation_result": str( + PHASE3 / "primary/P09-C00/saturation/client/result.json" + ), + "rate_fraction": 0.60, + "added_capture_sizes": list(ADDED_CAPTURE_SIZES), + "on_capture_sizes": list(ON_CAPTURE_SIZES), + "measured_padding_recovery_bound": 0.049776380388728225, + "primary_metric": "clean graph-hit padding_fraction", + "secondary_metrics": [ + "Layer-1 useful scheduled tokens/model-step millisecond", + "clean completed request throughput", + "clean request latency", + ], + "prior_gpu_hours": PRIOR_GPU_HOURS, + "projected_increment_gpu_hours": EXPECTED_INCREMENT_HOURS, + "projected_total_gpu_hours": PRIOR_GPU_HOURS + EXPECTED_INCREMENT_HOURS, + "gpu_hour_limit": GPU_HOUR_LIMIT, + } + + +def save_state(state: dict[str, Any]) -> None: + state["updated_at"] = time.time() + state["controller_pid"] = os.getpid() + common.atomic_json(STATE, state) + + +def wait_ready(server: subprocess.Popen[Any]) -> None: + deadline = time.monotonic() + 300 + while time.monotonic() < deadline: + if server.poll() is not None: + raise RuntimeError("server exited before readiness") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{PORT}/health", timeout=1 + ) as response: + if response.status == 200: + return + except Exception: + pass + time.sleep(1) + raise TimeoutError("server readiness timeout") + + +def server_command(arm: str) -> list[str]: + command = [ + "taskset", "-c", "0-19", str(VENV / "bin/vllm"), "serve", str(MODEL), + "--host", "127.0.0.1", "--port", str(PORT), + "--tensor-parallel-size", "1", "--enable-chunked-prefill", + "--enable-prefix-caching", "--shutdown-timeout", "120", + ] + if arm == "ON": + command.extend(("--cudagraph-capture-sizes", *map(str, ON_CAPTURE_SIZES))) + return command + + +def client_command(run_dir: Path) -> list[str]: + return [ + "taskset", "-c", "0-19", str(VENV / "bin/python"), str(CLIENT), "run", + "--manifest", str(PRIVATE / "P09.jsonl"), + "--base-url", f"http://127.0.0.1:{PORT}", "--model", str(MODEL), + "--load-point", "moderate", "--saturation-result", + str(PHASE3 / "primary/P09-C00/saturation/client/result.json"), + "--rate-fraction", "0.60", "--max-concurrency", "256", "--ignore-eos", + "--temperature", "0", "--warmup-seconds", "60", + "--clean-segment-seconds", "80", "--num-clean-segments", "3", + "--recovery-seconds", "30", "--drain-timeout-seconds", "120", + "--workload-seed", "20260712", "--server-seed", "20260712", + "--result-dir", str(run_dir / "client"), + ] + + +def summarize(run_dir: Path, arm: str) -> dict[str, Any]: + result = json.loads((run_dir / "client/result.json").read_text()) + requests = [ + json.loads(line) + for line in (run_dir / "client/requests.jsonl").read_text().splitlines() + ] + t0 = int(result["t0_mono_ns"]) + start, end = t0 + int(60e9), t0 + int(300e9) + stream = next((run_dir / "opprof").glob("*.jsonl")) + records = [] + for line in stream.read_text().splitlines(): + record = json.loads(line) + if ( + "step_index" in record + and start <= int(record["submit_mono_ns"]) < end + ): + records.append(record) + model = [record for record in records if record["model_executed"]] + hits = [ + record + for record in model + if record["cudagraph"]["hit"] + and int(record["cudagraph"]["bucket_tokens"]) > 0 + ] + pad = sum(int(record["cudagraph"]["padding_tokens"]) for record in hits) + bucket = sum(int(record["cudagraph"]["bucket_tokens"]) for record in hits) + useful = sum( + int(record["prefill_tokens"]) + int(record["decode_tokens"]) + for record in records + ) + duration_ms = sum( + (int(record["complete_mono_ns"]) - int(record["submit_mono_ns"])) / 1e6 + for record in records + ) + completed = [ + request + for request in requests + if request["success"] and 60 <= float(request["completed_s"]) < 300 + ] + e2e = np.asarray( + [float(request["completed_s"] - request["admitted_s"]) for request in completed] + ) + layer1 = matrix.validate_layer1(run_dir) + return { + "schema": 1, + "arm": arm, + "capture_sizes": list(ON_CAPTURE_SIZES) if arm == "ON" else "default", + "clean_completed": len(completed), + "clean_failed": int(result["clean"]["failed"]), + "clean_completed_throughput_rps": float( + result["clean"]["completed_throughput_rps"] + ), + "clean_offered_rps": float(result["clean"]["offered_rps"]), + "e2e_latency_mean_s": float(e2e.mean()), + "e2e_latency_p95_s": float(np.quantile(e2e, 0.95)), + "model_steps": len(model), + "graph_hit_steps": len(hits), + "padding_tokens": pad, + "bucket_tokens": bucket, + "padding_fraction": pad / bucket, + "graph_miss_rate": sum(not record["cudagraph"]["hit"] for record in model) + / len(model), + "useful_tokens": useful, + "model_step_duration_ms": duration_ms, + "token_efficiency_per_ms": useful / duration_ms, + "layer1_records": layer1["records"], + "layer1_invariants": layer1["invariants"], + "drain_seconds": float(result["drain_seconds"]), + } + + +def run_arm(state: dict[str, Any], arm: str) -> None: + if state["arms"].get(arm, {}).get("status") == "complete": + return + run_dir = ROOT / arm.lower() + if run_dir.exists(): + run_dir.rename(run_dir.with_name(f"{run_dir.name}.interrupted-{int(time.time())}")) + run_dir.mkdir(parents=True) + common.preflight([GPU], run_dir) + server_cmd = server_command(arm) + client_cmd = client_command(run_dir) + commands_path = run_dir / "commands.log" + common.command_log(commands_path, f"P09-capture-{arm}-server", server_cmd, "6-9m") + common.command_log(commands_path, f"P09-capture-{arm}-client", client_cmd, "5-7m") + print( + f"GPU_COMMAND P09-capture-{arm}-server: {shlex.join(server_cmd)}; " + "expected=6-9m", + flush=True, + ) + print( + f"GPU_COMMAND P09-capture-{arm}-client: {shlex.join(client_cmd)}; " + "expected=5-7m", + flush=True, + ) + environment = os.environ.copy() + environment.update( + { + "CUDA_VISIBLE_DEVICES": str(GPU), + "VLLM_OPPROF_DIR": str(run_dir / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + server_log = (run_dir / "server.log").open("ab", buffering=0) + server_started = time.time() + server = subprocess.Popen( + server_cmd, + cwd=SOURCE, + env=environment, + stdout=server_log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + client = None + client_log = None + monitor = None + owned = {server.pid} + state["arms"][arm] = { + "status": "starting", + "server_pid": server.pid, + "started_at": server_started, + } + save_state(state) + failure = None + try: + wait_ready(server) + monitor = common.Monitor(run_dir / "monitor.jsonl", owned) + monitor.start() + client_log = (run_dir / "client.log").open("ab", buffering=0) + client = subprocess.Popen( + client_cmd, + cwd=WORKDIR, + stdout=client_log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + owned.add(client.pid) + state["arms"][arm].update(status="running", client_pid=client.pid) + save_state(state) + deadline = time.monotonic() + 900 + while client.poll() is None and time.monotonic() < deadline: + if server.poll() is not None: + raise RuntimeError("server exited during client load") + if monitor.other_apps: + raise RuntimeError(f"other GPU process appeared: {monitor.other_apps}") + time.sleep(2) + if client.poll() is None: + raise TimeoutError("client exceeded 900 seconds") + if client.returncode: + raise RuntimeError(f"client failed with {client.returncode}") + except Exception as error: + failure = error + finally: + if client is not None and client.poll() is None: + try: + os.killpg(client.pid, signal.SIGKILL) + except ProcessLookupError: + pass + common.stop_servers([server]) + server_log.close() + if client_log is not None: + client_log.close() + if monitor is not None: + monitor.stop() + common.verify_idle([GPU], run_dir) + gpu_hours = (time.time() - server_started) / 3600 + state["gpu_hours_increment"] += gpu_hours + if PRIOR_GPU_HOURS + state["gpu_hours_increment"] >= GPU_HOUR_LIMIT: + failure = failure or RuntimeError("GPU-hour limit reached") + if failure is not None: + state["arms"][arm].update(status="failed", failure=repr(failure)) + state["status"] = "failed" + save_state(state) + raise failure + summary = summarize(run_dir, arm) + summary["gpu_hours"] = gpu_hours + common.atomic_json(run_dir / "summary.json", summary) + state["arms"][arm].update(status="complete", summary=summary) + save_state(state) + + +def run() -> None: + ROOT.mkdir(parents=True, exist_ok=True) + if PRIOR_GPU_HOURS + EXPECTED_INCREMENT_HOURS >= GPU_HOUR_LIMIT: + raise RuntimeError("projected validation exceeds GPU-hour budget") + if STATE.exists(): + state = json.loads(STATE.read_text()) + else: + state = { + "schema": 1, + "status": "running", + "created_at": time.time(), + "plan": plan(), + "arms": {}, + "gpu_hours_increment": 0.0, + } + for arm in ("ON", "OFF"): + run_arm(state, arm) + on, off = state["arms"]["ON"]["summary"], state["arms"]["OFF"]["summary"] + result = { + "schema": 1, + "plan": plan(), + "on": on, + "off": off, + "delta": { + "padding_fraction_points": on["padding_fraction"] - off["padding_fraction"], + "padding_reduction_fraction": 1 - on["padding_fraction"] / off["padding_fraction"], + "token_efficiency_relative": on["token_efficiency_per_ms"] + / off["token_efficiency_per_ms"] + - 1, + "completed_throughput_relative": on["clean_completed_throughput_rps"] + / off["clean_completed_throughput_rps"] + - 1, + "e2e_mean_latency_relative": on["e2e_latency_mean_s"] + / off["e2e_latency_mean_s"] + - 1, + "e2e_p95_latency_relative": on["e2e_latency_p95_s"] + / off["e2e_latency_p95_s"] + - 1, + }, + "gpu_hours_increment": state["gpu_hours_increment"], + "gpu_hours_total": PRIOR_GPU_HOURS + state["gpu_hours_increment"], + } + common.atomic_json(ROOT / "result.json", result) + state["status"] = "complete" + state["result"] = result + save_state(state) + print(json.dumps(result, sort_keys=True)) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("command", choices=("plan", "run")) + args = parser.parse_args() + if args.command == "plan": + print(json.dumps(plan(), indent=2, sort_keys=True)) + else: + run() + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/analyze_phase3.py b/runs/opprof-phase3/provenance/analyze_phase3.py new file mode 100644 index 0000000..a74ce54 --- /dev/null +++ b/runs/opprof-phase3/provenance/analyze_phase3.py @@ -0,0 +1,1611 @@ +#!/usr/bin/env python3 +"""Frozen Phase-3 OpProf analysis; emits aggregate, prompt-free JSON only.""" + +from __future__ import annotations + +import argparse +import bisect +import gzip +import json +import math +import re +from collections import Counter, defaultdict +from pathlib import Path +from typing import Any + +import numpy as np + +import opprof_phase3_controller as common + + +SEED = 20260714 +BOOTSTRAPS = 100_000 +FAMILIES = ( + "attention", + "moe_gemm", + "moe_router", + "collective", + "sampler", + "dense_gemm", + "norm_elementwise", + "kv_memory", +) +IRREGULAR_CONTROLS = ( + ("P05", "P01"), + ("P05", "P03"), + ("P06", "P02"), + ("P06", "P04"), + ("P09", "P01"), + ("P09", "P03"), + ("P10", "P03"), + ("P10", "P04"), +) +AP37_MISSING_CELLS = ( + "P03/C11", + "P05/C00", + "P10/C00-TP2", + "P11/C00", +) +AP37_MISSING_CONTRASTS = ( + ("P05", "P01"), + ("P05", "P03"), +) +CAPTURE_BUCKETS = ( + 1, + 2, + 4, + 8, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512, +) + + +def load_json(path: Path) -> Any: + return json.loads(path.read_text()) + + +def numeric_sanity(values: list[float | int | None]) -> dict[str, Any]: + finite = [float(value) for value in values if value is not None and math.isfinite(value)] + return { + "n": len(values), + "finite_n": len(finite), + "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, + "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def percentile_summary(values: list[float]) -> dict[str, Any]: + if not values: + return {"n": 0, "mean": None, "p50": None, "p95": None, "p99": None} + array = np.asarray(values, dtype=float) + return { + "n": len(values), + "mean": float(array.mean()), + "p50": float(np.quantile(array, 0.50)), + "p95": float(np.quantile(array, 0.95)), + "p99": float(np.quantile(array, 0.99)), + } + + +def jsonl(path: Path) -> list[dict[str, Any]]: + with path.open() as source: + return [json.loads(line) for line in source] + + +def expected_cells() -> set[str]: + cells = {f"P{index:02d}/C00" for index in range(1, 12)} + for pattern in ("P01", "P03", "P06", "P10"): + cells.update(f"{pattern}/{config}" for config in ("C10", "C01", "C11")) + cells.add("P10/C00-TP2") + return cells + + +def accepted_marker_paths( + root: Path, +) -> tuple[list[Path], list[Path], list[Path], list[str]]: + complete_stages = sorted( + path.parent for path in root.glob("stages/*/stage-complete.json") + ) + accepted_ids = [] + for stage in complete_stages: + stage_name = stage.name + if not (stage_name.startswith("primary-") or stage_name == "confirmations"): + continue + marker = load_json(stage / "stage-complete.json") + accepted_ids.extend(str(run_id) for run_id in marker["runs"]) + if len(accepted_ids) != len(set(accepted_ids)): + raise RuntimeError("accepted run ID appears in more than one complete stage") + + canonical = {} + for path in root.glob("primary/*/*/run-complete.json"): + if path.parent.name not in {"saturation", "moderate"}: + continue + run_id = str(load_json(path)["run_id"]) + if run_id in canonical: + raise RuntimeError(f"duplicate canonical run marker: {run_id}") + canonical[run_id] = path + for path in root.glob("confirmations/*/run-complete.json"): + run_id = str(load_json(path)["run_id"]) + if run_id in canonical: + raise RuntimeError(f"duplicate canonical run marker: {run_id}") + canonical[run_id] = path + missing = sorted(set(accepted_ids) - set(canonical)) + if missing: + raise RuntimeError(f"complete-stage run markers missing: {missing}") + selected = [canonical[run_id] for run_id in accepted_ids] + primary = sorted(path for path in selected if "primary" in path.parts) + confirmations = sorted(path for path in selected if "confirmations" in path.parts) + unaccepted = sorted(set(canonical) - set(accepted_ids)) + return primary, confirmations, complete_stages, unaccepted + + +def partial_verdict(has_hit: bool) -> str: + return "PASS" if has_hit else "INCONCLUSIVE" + + +def run_records(run_dir: Path) -> tuple[list[dict[str, Any]], list[dict[str, Any]]]: + result = load_json(run_dir / "client/result.json") + t0 = int(result["t0_mono_ns"]) + start = t0 + int(60e9) + end = t0 + int(300e9) + stream = next((run_dir / "opprof").glob("*.jsonl")) + all_records = [] + clean = [] + for record in jsonl(stream): + if "step_index" not in record: + continue + all_records.append(record) + if start <= int(record["submit_mono_ns"]) < end: + clean.append(record) + return all_records, clean + + +def ap36_warmup_stability(run_dir: Path) -> dict[str, Any]: + result = load_json(run_dir / "client/result.json") + t0 = int(result["t0_mono_ns"]) + requests = jsonl(run_dir / "client/requests.jsonl") + completions = sum( + bool(item["success"]) + and 0 <= float(item["completed_s"]) < float(result["warmup_seconds"]) + for item in requests + ) + all_records, _ = run_records(run_dir) + counts = [0, 0, 0] + tokens = [0, 0, 0] + for item in all_records: + if not item["model_executed"]: + continue + relative_s = (int(item["submit_mono_ns"]) - t0) / 1e9 + if not 45 <= relative_s < 60: + continue + index = min(2, int((relative_s - 45) // 5)) + counts[index] += 1 + tokens[index] += int(item["prefill_tokens"]) + int(item["decode_tokens"]) + rates = [value / 5.0 for value in tokens] + mean = sum(rates) / 3 + slope = (rates[2] - rates[0]) / 10.0 + drift = abs(slope) * 15 / mean if mean > 0 else math.inf + return { + "warmup_completions": completions, + "step_counts": counts, + "scheduled_tokens": tokens, + "scheduled_token_throughput": rates, + "mean_scheduled_token_throughput": mean, + "slope_tokens_per_second_squared": slope, + "normalized_drift": drift, + "normalized_drift_limit": 0.10, + "passes": completions >= 16 + and all(value >= 16 for value in counts) + and drift <= 0.10, + } + + +def mode(record: dict[str, Any]) -> str: + return str(record["cudagraph"]["runtime_mode"]) + + +def clean_block(record: dict[str, Any], t0_ns: int) -> int: + return min(47, max(0, int((int(record["submit_mono_ns"]) - t0_ns - 60e9) // 5e9))) + + +def summarize_run(run_dir: Path, marker: dict[str, Any]) -> tuple[dict[str, Any], list[dict[str, Any]]]: + client = load_json(run_dir / "client/result.json") + requests = jsonl(run_dir / "client/requests.jsonl") + all_records, records = run_records(run_dir) + completed = [ + item + for item in requests + if item["success"] and 60 <= float(item["completed_s"]) < 300 + ] + e2e = [float(item["completed_s"] - item["admitted_s"]) for item in completed] + ttft = [float(item["first_token_s"] - item["admitted_s"]) for item in completed] + tpot = [ + float(item["completed_s"] - item["first_token_s"]) + / max(1, int(item["actual_output_tokens"]) - 1) + for item in completed + ] + duration_ms = [ + (int(item["complete_mono_ns"]) - int(item["submit_mono_ns"])) / 1e6 + for item in records + ] + useful = [int(item["prefill_tokens"]) + int(item["decode_tokens"]) for item in records] + hit_records = [ + item + for item in records + if item["model_executed"] + and item["cudagraph"]["hit"] + and int(item["cudagraph"]["bucket_tokens"]) > 0 + ] + model_records = [item for item in records if item["model_executed"]] + misses = [item for item in model_records if not item["cudagraph"]["hit"]] + eligible_misses = [ + item for item in misses if int(item["cudagraph"]["unpadded_tokens"]) <= CAPTURE_BUCKETS[-1] + ] + overflow = [ + item for item in misses if int(item["cudagraph"]["unpadded_tokens"]) > CAPTURE_BUCKETS[-1] + ] + theoretical_buckets = [ + CAPTURE_BUCKETS[bisect.bisect_left(CAPTURE_BUCKETS, int(item["cudagraph"]["unpadded_tokens"]))] + for item in model_records + if 0 < int(item["cudagraph"]["unpadded_tokens"]) <= CAPTURE_BUCKETS[-1] + ] + theoretical_unpadded = [ + int(item["cudagraph"]["unpadded_tokens"]) + for item in model_records + if 0 < int(item["cudagraph"]["unpadded_tokens"]) <= CAPTURE_BUCKETS[-1] + ] + prefixes = [] + for item in records: + for source in (item["prefix"].get("local"), item["prefix"].get("external")): + if source: + prefixes.append(source) + blocks = [defaultdict(float) for _ in range(48)] + t0 = int(client["t0_mono_ns"]) + for item, step_ms in zip(records, duration_ms, strict=True): + block = blocks[clean_block(item, t0)] + token = int(item["prefill_tokens"]) + int(item["decode_tokens"]) + graph = item["cudagraph"] + block["tokens"] += token + block["duration_ms"] += step_ms + block["model"] += bool(item["model_executed"]) + block["miss"] += bool(item["model_executed"] and not graph["hit"]) + block["eligible_miss"] += bool( + item["model_executed"] + and not graph["hit"] + and int(graph["unpadded_tokens"]) <= CAPTURE_BUCKETS[-1] + ) + block["overflow"] += bool( + item["model_executed"] + and not graph["hit"] + and int(graph["unpadded_tokens"]) > CAPTURE_BUCKETS[-1] + ) + if item["model_executed"] and graph["hit"] and int(graph["bucket_tokens"]) > 0: + block["padding"] += int(graph["padding_tokens"]) + block["bucket"] += int(graph["bucket_tokens"]) + denominator = sum(duration_ms) + bucket_sum = sum(int(item["cudagraph"]["bucket_tokens"]) for item in hit_records) + pad_sum = sum(int(item["cudagraph"]["padding_tokens"]) for item in hit_records) + model_n = len(model_records) + modes = Counter(mode(item) for item in records) + completion_times = [ + float(item["completed_s"]) + for item in requests + if item["success"] + ] + clean_completion_rates = [ + sum(start <= value < start + 10 for value in completion_times) / 10 + for start in np.arange(220, 300, 10) + ] + clean_waiting = [ + item["queues"]["waiting"] + for item in records + if t0 + int(220e9) <= int(item["submit_mono_ns"]) < t0 + int(300e9) + ] + clean_completion_median = float(np.median(clean_completion_rates)) + clean_waiting_median = float(np.median(clean_waiting)) if clean_waiting else None + recoveries = [] + profiles = client.get("profiles", []) + for index, profile in enumerate(profiles): + recovery_end = ( + float(profiles[index + 1]["start_call_s"]) + if index + 1 < len(profiles) + else float(client["admission_stop_s"]) + ) + tail_start = recovery_end - 10 + completion_rate = ( + sum(tail_start <= value < recovery_end for value in completion_times) / 10 + ) + tail_waiting = [ + item["queues"]["waiting"] + for item in all_records + if t0 + int(tail_start * 1e9) + <= int(item["submit_mono_ns"]) + < t0 + int(recovery_end * 1e9) + ] + waiting_median = float(np.median(tail_waiting)) if tail_waiting else None + rate_ok = ( + completion_rate == clean_completion_median + if clean_completion_median == 0 + else abs(completion_rate / clean_completion_median - 1) <= 0.10 + ) + waiting_ok = ( + waiting_median == clean_waiting_median + if clean_waiting_median == 0 + else ( + waiting_median is not None + and clean_waiting_median is not None + and abs(waiting_median / clean_waiting_median - 1) <= 0.10 + ) + ) + recoveries.append( + { + "window": index + 1, + "tail_start_s": tail_start, + "tail_end_s": recovery_end, + "completion_rate_rps": completion_rate, + "clean_c_completion_rate_median_rps": clean_completion_median, + "waiting_median": waiting_median, + "clean_c_waiting_median": clean_waiting_median, + "completion_rate_within_10pct": rate_ok, + "waiting_within_10pct": waiting_ok, + "valid": rate_ok and waiting_ok, + } + ) + summary = { + "run_id": marker["run_id"], + "pattern": marker["pattern"], + "config": marker["config"], + "load": client["load_point"], + "drain_seconds": float(client["drain_seconds"]), + "drain_quarantined": bool(marker["drain_quarantined"]), + "clean": client["clean"], + "requests_completed": len(completed), + "latency_s": { + "e2e": percentile_summary(e2e), + "ttft": percentile_summary(ttft), + "tpot": percentile_summary(tpot), + }, + "layer1": { + "records_all": len(all_records), + "records_clean": len(records), + "step_duration_ms": percentile_summary(duration_ms), + "scheduled_tokens_per_step": percentile_summary([float(value) for value in useful]), + "requests_per_step": percentile_summary( + [float(item["scheduled_requests"]) for item in records] + ), + "prefill_tokens": sum(int(item["prefill_tokens"]) for item in records), + "decode_tokens": sum(int(item["decode_tokens"]) for item in records), + "token_efficiency_per_ms": sum(useful) / denominator if denominator else None, + "queue_waiting_mean": float(np.mean([item["queues"]["waiting"] for item in records])), + "queue_waiting_max": max((item["queues"]["waiting"] for item in records), default=0), + "kv_usage_mean": float(np.mean([item["kv"]["usage"] for item in records])), + "kv_usage_max": max((item["kv"]["usage"] for item in records), default=0), + "preemptions": sum(int(item["preemptions"]) for item in records), + "prefix_query_hit_ratio": ( + sum(int(item["hits"]) for item in prefixes) + / sum(int(item["queries"]) for item in prefixes) + if sum(int(item["queries"]) for item in prefixes) + else 0.0 + ), + "mode_counts": dict(sorted(modes.items())), + "mode_shares": {key: value / len(records) for key, value in sorted(modes.items())}, + }, + "waste": { + "padding_fraction": pad_sum / bucket_sum if bucket_sum else 0.0, + "padded_tokens_per_useful_token": pad_sum / sum(useful) if sum(useful) else 0.0, + "graph_miss_rate": len(misses) / model_n if model_n else 0.0, + "eligible_miss_rate": len(eligible_misses) / model_n if model_n else 0.0, + "overflow_rate": len(overflow) / model_n if model_n else 0.0, + "bucket_slack": ( + (sum(theoretical_buckets) - sum(theoretical_unpadded)) + / sum(theoretical_buckets) + if theoretical_buckets + else 0.0 + ), + "mixed_interference": None, + "moe_layer_cv": None, + }, + "trace_files": len(marker.get("traces", [])), + "profile_recovery": recoveries, + "profile_recovery_valid": all(item["valid"] for item in recoveries), + "layer2_missing_after_controller_cleanup": bool( + marker.get("layer2_missing_after_controller_cleanup") + ), + "blocks": [dict(block) for block in blocks], + } + return summary, records + + +def manifest_raggedness(path: Path, cohort: int) -> tuple[float, list[tuple[float, float]]]: + lengths = [] + with path.open() as source: + for line in source: + lengths.append(int(json.loads(line)["input_tokens"])) + pieces = [] + for start in range(0, len(lengths) - cohort + 1, cohort): + group = lengths[start : start + cohort] + denominator = float(cohort * max(group)) + pieces.append((denominator - sum(group), denominator)) + numerator = sum(item[0] for item in pieces) + denominator = sum(item[1] for item in pieces) + return numerator / denominator, pieces + + +EXECUTE = re.compile(r"execute_context_\d+\((\d+)\)_generation_\d+\((\d+)\)") + + +def trace_steps(path: Path, layer1: list[dict[str, Any]]) -> dict[str, Any]: + opener = gzip.open if path.suffix == ".gz" else open + with opener(path, "rt", encoding="utf-8") as source: + payload = json.load(source) + base = int(payload["baseTimeNanoseconds"]) + events = payload["traceEvents"] + cpu = [] + gpu_by_external: dict[int, list[dict[str, Any]]] = defaultdict(list) + kernels = [] + for event in events: + name = str(event.get("name", "")) + if event.get("cat") == "user_annotation" and EXECUTE.fullmatch(name): + cpu.append(event) + elif event.get("cat") == "gpu_user_annotation" and EXECUTE.fullmatch(name): + gpu_by_external[int(event.get("args", {}).get("External id", -1))].append(event) + elif event.get("cat") == "kernel": + kernels.append(event) + cpu.sort(key=lambda item: float(item["ts"])) + if len(cpu) != 8: + raise RuntimeError(f"active execute count {len(cpu)} != 8: {path}") + walls = [int(item["submit_wall_ns"]) for item in layer1] + joined = [] + used_steps = set() + unmatched: dict[str, float] = defaultdict(float) + for event in cpu: + external = int(event["args"]["External id"]) + candidates = gpu_by_external.get(external, []) + if not candidates: + raise RuntimeError(f"GPU execute annotation absent: {path}: {external}") + gpu = max(candidates, key=lambda item: float(item.get("dur", 0))) + start = float(gpu["ts"]) + end = start + float(gpu["dur"]) + durations: dict[str, float] = defaultdict(float) + for kernel in kernels: + midpoint = float(kernel["ts"]) + float(kernel.get("dur", 0)) / 2 + if start <= midpoint <= end: + kernel_name = str(kernel.get("name", "")) + duration = float(kernel.get("dur", 0)) + family = common.classify_kernel(kernel_name) + durations[family] += duration + if family == "other": + unmatched[kernel_name] += duration + total = sum(durations.values()) + if total <= 0: + raise RuntimeError(f"active execute has no kernels: {path}: {external}") + wall = base + int(float(event["ts"]) * 1000) + index = bisect.bisect_left(walls, wall) + choices = layer1[max(0, index - 3) : index + 3] + record = min(choices, key=lambda item: abs(int(item["submit_wall_ns"]) - wall)) + match = EXECUTE.fullmatch(str(event["name"])) + assert match + expected = (int(match.group(1)), int(match.group(2))) + actual = (int(record["prefill_tokens"]), int(record["decode_tokens"])) + delta_ms = (int(record["submit_wall_ns"]) - wall) / 1e6 + if actual != expected or abs(delta_ms) > 100 or record["step_index"] in used_steps: + raise RuntimeError( + f"ambiguous Layer-1 join: {path}: expected={expected} actual={actual} " + f"delta_ms={delta_ms}" + ) + used_steps.add(record["step_index"]) + shares = {family: durations.get(family, 0.0) / total for family in FAMILIES} + shares["other"] = durations.get("other", 0.0) / total + joined.append( + { + "step_index": int(record["step_index"]), + "join_delta_ms": delta_ms, + "prefill_tokens": actual[0], + "decode_tokens": actual[1], + "scheduled_requests": int(record["scheduled_requests"]), + "decode_batch_size": int(record["decode_batch_size"]), + "runtime_mode": mode(record), + "duration_us": dict(durations), + "shares": shares, + "classifiable_fraction": 1.0 - shares["other"], + } + ) + aggregate = defaultdict(float) + for step in joined: + for family, duration in step["duration_us"].items(): + aggregate[family] += duration + total = sum(aggregate.values()) + attention_subduration = defaultdict(float) + mode_duration: dict[str, dict[str, float]] = defaultdict(lambda: defaultdict(float)) + mode_steps = Counter() + for step in joined: + if step["prefill_tokens"] and step["decode_tokens"]: + attention_key = "attention_mixed" + elif step["prefill_tokens"]: + attention_key = "attention_prefill" + else: + attention_key = "attention_decode" + attention_subduration[attention_key] += step["duration_us"].get("attention", 0.0) + mode_steps[step["runtime_mode"]] += 1 + for family, duration in step["duration_us"].items(): + mode_duration[step["runtime_mode"]][family] += duration + return { + "path": path.name, + "steps": joined, + "shares": {family: aggregate.get(family, 0.0) / total for family in FAMILIES}, + "other_share": aggregate.get("other", 0.0) / total, + "classifiable_fraction": 1.0 - aggregate.get("other", 0.0) / total, + "attention_subshares": { + key: attention_subduration.get(key, 0.0) / total + for key in ("attention_prefill", "attention_decode", "attention_mixed") + }, + "mode_steps": dict(sorted(mode_steps.items())), + "mode_shares": { + key: { + family: durations.get(family, 0.0) / sum(durations.values()) + for family in FAMILIES + } + for key, durations in sorted(mode_duration.items()) + }, + "top_unmatched": [ + {"name": name, "duration_us": duration} + for name, duration in sorted(unmatched.items(), key=lambda item: item[1], reverse=True)[ + :20 + ] + ], + } + + +def smd(profile: list[float], clean: list[float]) -> float: + a = np.asarray(profile, dtype=float) + b = np.asarray(clean, dtype=float) + if np.all(a == a[0]) and np.all(b == b[0]): + return 0.0 if a[0] == b[0] else math.inf + denominator = math.sqrt((float(a.var(ddof=1)) + float(b.var(ddof=1))) / 2) + if denominator == 0: + return math.inf + return float((a.mean() - b.mean()) / denominator) + + +def representativeness(window: dict[str, Any], clean_c: list[dict[str, Any]]) -> dict[str, Any]: + steps = window["steps"] + features: dict[str, tuple[list[float], list[float]]] = { + "scheduled_tokens": ( + [step["prefill_tokens"] + step["decode_tokens"] for step in steps], + [item["prefill_tokens"] + item["decode_tokens"] for item in clean_c], + ), + "prefill_fraction": ( + [ + step["prefill_tokens"] + / max(1, step["prefill_tokens"] + step["decode_tokens"]) + for step in steps + ], + [ + item["prefill_tokens"] + / max(1, item["prefill_tokens"] + item["decode_tokens"]) + for item in clean_c + ], + ), + "decode_batch_size": ( + [step["decode_batch_size"] for step in steps], + [item["decode_batch_size"] for item in clean_c], + ), + } + modes = ("FULL", "PIECEWISE", "NONE") + for key in modes: + features[f"mode_{key}"] = ( + [float(step["runtime_mode"] == key) for step in steps], + [float(mode(item) == key) for item in clean_c], + ) + values = {key: smd(*pair) for key, pair in features.items()} + return { + "smd": values, + "valid": all(math.isfinite(value) and abs(value) <= 0.25 for value in values.values()), + } + + +def convex_hull(points: list[tuple[float, float]]) -> list[tuple[float, float]]: + unique = sorted(set(points)) + if len(unique) <= 1: + return unique + + def cross(origin, left, right): + return (left[0] - origin[0]) * (right[1] - origin[1]) - ( + left[1] - origin[1] + ) * (right[0] - origin[0]) + + lower = [] + for point in unique: + while len(lower) >= 2 and cross(lower[-2], lower[-1], point) <= 0: + lower.pop() + lower.append(point) + upper = [] + for point in reversed(unique): + while len(upper) >= 2 and cross(upper[-2], upper[-1], point) <= 0: + upper.pop() + upper.append(point) + return lower[:-1] + upper[:-1] + + +def inside_convex(hull: list[tuple[float, float]], point: tuple[float, float]) -> bool: + if not hull: + return False + if len(hull) == 1: + return point == hull[0] + if len(hull) == 2: + left, right = hull + cross = (right[0] - left[0]) * (point[1] - left[1]) - ( + right[1] - left[1] + ) * (point[0] - left[0]) + return ( + abs(cross) <= 1e-9 + and min(left[0], right[0]) <= point[0] <= max(left[0], right[0]) + and min(left[1], right[1]) <= point[1] <= max(left[1], right[1]) + ) + signs = [] + for index, left in enumerate(hull): + right = hull[(index + 1) % len(hull)] + cross = (right[0] - left[0]) * (point[1] - left[1]) - ( + right[1] - left[1] + ) * (point[0] - left[0]) + if abs(cross) > 1e-9: + signs.append(math.copysign(1, cross)) + return not signs or min(signs) == max(signs) + + +def fit_nonnegative_robust(rows: list[tuple[float, float, float]]): + values = np.asarray(rows, dtype=float) + x = np.log1p(values[:, 0]) + n = np.log1p(values[:, 1]) + y = values[:, 2] + x_knots = np.quantile(x, [0.25, 0.5, 0.75]) + n_knots = np.quantile(n, [0.25, 0.5, 0.75]) + + def design(raw_x, raw_n): + a = np.log1p(np.asarray(raw_x, dtype=float)) + b = np.log1p(np.asarray(raw_n, dtype=float)) + columns = [np.ones_like(a), a, b, a * b] + columns.extend(np.maximum(0, a - knot) for knot in x_knots) + columns.extend(np.maximum(0, b - knot) for knot in n_knots) + return np.column_stack(columns) + + matrix = design(values[:, 0], values[:, 1]) + weights = np.ones(len(y)) + coef = np.zeros(matrix.shape[1]) + for _ in range(20): + weighted = np.sqrt(weights) + coef = np.linalg.lstsq(matrix * weighted[:, None], y * weighted, rcond=None)[0] + coef = np.maximum(0, coef) + residual = y - matrix @ coef + scale = 1.4826 * np.median(np.abs(residual - np.median(residual))) + 1e-9 + weights = np.minimum(1.0, 1.345 * scale / np.maximum(np.abs(residual), 1e-9)) + + def predict(raw_x: float, raw_n: float) -> float: + return float(max(0.0, design([raw_x], [raw_n])[0] @ coef)) + + return predict, convex_hull( + [(float(row[0]), float(row[1])) for row in values] + ) + + +def add_mixed_interference( + summaries: dict[str, dict[str, Any]], + records: dict[str, list[dict[str, Any]]], +) -> None: + groups: dict[tuple[str, str], list[str]] = defaultdict(list) + for run_id, summary in summaries.items(): + if run_id.endswith("-confirmation"): + continue + groups[(summary["config"], summary["load"])].append(run_id) + for run_ids in groups.values(): + for target in run_ids: + pattern = summaries[target]["pattern"] + training = [item for item in run_ids if summaries[item]["pattern"] != pattern] + prefill = [] + decode = [] + zero = [] + for run_id in training: + for item in records[run_id]: + p = int(item["prefill_tokens"]) + d = int(item["decode_tokens"]) + n = int(item["scheduled_requests"]) + duration = (item["complete_mono_ns"] - item["submit_mono_ns"]) / 1e6 + if p > 0 and d == 0: + prefill.append((p, n, duration)) + elif d > 0 and p == 0: + decode.append((d, n, duration)) + elif p == 0 and d == 0: + zero.append(duration) + if len(prefill) < 30 or len(decode) < 30: + continue + fp, p_support = fit_nonnegative_robust(prefill) + fd, d_support = fit_nonnegative_robust(decode) + alpha = float(np.median(zero)) if zero else 0.0 + supported = [] + for item in records[target]: + p = int(item["prefill_tokens"]) + d = int(item["decode_tokens"]) + n = int(item["scheduled_requests"]) + if not (p > 0 and d > 0): + continue + if not inside_convex(p_support, (p, n)) or not inside_convex( + d_support, (d, n) + ): + continue + predicted = fp(p, n) + fd(d, n) - alpha + if predicted <= 0: + continue + observed = (item["complete_mono_ns"] - item["submit_mono_ns"]) / 1e6 + supported.append((item, observed - predicted, predicted)) + summaries[target]["waste"]["mixed_supported_steps"] = len(supported) + summaries[target]["waste"]["mixed_total_steps"] = sum( + item["prefill_tokens"] > 0 and item["decode_tokens"] > 0 + for item in records[target] + ) + if len(supported) < 30: + continue + numerator = sum(item[1] for item in supported) + denominator = sum(item[2] for item in supported) + summaries[target]["waste"]["mixed_interference"] = numerator / denominator + blocks = summaries[target]["blocks"] + t0 = int(load_json(Path(summaries[target]["run_dir"]) / "client/result.json")["t0_mono_ns"]) + for record, residual, predicted in supported: + block = blocks[clean_block(record, t0)] + block["mix_residual"] = block.get("mix_residual", 0.0) + residual + block["mix_predicted"] = block.get("mix_predicted", 0.0) + predicted + + +def ratio_blocks(summary: dict[str, Any], metric: str) -> np.ndarray | None: + keys = { + "padding_fraction": ("padding", "bucket"), + "graph_miss_rate": ("miss", "model"), + "overflow_rate": ("overflow", "model"), + "mixed_interference": ("mix_residual", "mix_predicted"), + "efficiency": ("tokens", "duration_ms"), + } + numerator, denominator = keys[metric] + values = np.asarray( + [[block.get(numerator, 0.0), block.get(denominator, 0.0)] for block in summary["blocks"]], + dtype=float, + ) + if values[:, 1].sum() <= 0: + return None + return values + + +def bootstrap_difference( + left: np.ndarray, right: np.ndarray, rng: np.random.Generator +) -> dict[str, float]: + point = left[:, 0].sum() / left[:, 1].sum() - right[:, 0].sum() / right[:, 1].sum() + draws = np.empty(BOOTSTRAPS) + chunk = 5000 + for start in range(0, BOOTSTRAPS, chunk): + count = min(chunk, BOOTSTRAPS - start) + li = rng.integers(0, len(left), size=(count, len(left))) + ri = rng.integers(0, len(right), size=(count, len(right))) + ln = left[li, 0].sum(axis=1) + ld = left[li, 1].sum(axis=1) + rn = right[ri, 0].sum(axis=1) + rd = right[ri, 1].sum(axis=1) + draws[start : start + count] = ln / np.maximum(ld, 1e-12) - rn / np.maximum(rd, 1e-12) + p = min(1.0, 2 * min(float(np.mean(draws <= 0)), float(np.mean(draws >= 0)))) + return { + "point": float(point), + "ci95_low": float(np.quantile(draws, 0.025)), + "ci95_high": float(np.quantile(draws, 0.975)), + "simultaneous_low": float(np.quantile(draws, 0.05 / 16)), + "simultaneous_high": float(np.quantile(draws, 1 - 0.05 / 16)), + "p": p, + } + + +def holm(results: list[dict[str, Any]], total_tests: int | None = None) -> None: + ordered = sorted(results, key=lambda item: item["p"]) + count = total_tests or len(ordered) + running = 0.0 + for index, item in enumerate(ordered): + running = max(running, min(1.0, (count - index) * item["p"])) + item["p_holm"] = running + + +def permutation_p( + left: list[dict[str, Any]], + right: list[dict[str, Any]], + family_a: str, + family_b: str, + orientation: float, + rng: np.random.Generator, +) -> float: + observed_windows = [] + null_windows = [] + for window in range(2): + a = np.asarray( + [ + orientation * (step["shares"][family_a] - step["shares"][family_b]) + for step in left[window]["steps"] + ] + ) + b = np.asarray( + [ + orientation * (step["shares"][family_a] - step["shares"][family_b]) + for step in right[window]["steps"] + ] + ) + observed_windows.append(float(a.mean() - b.mean())) + pooled = np.concatenate([a, b]) + values = np.empty(BOOTSTRAPS) + for start in range(0, BOOTSTRAPS, 5000): + count = min(5000, BOOTSTRAPS - start) + random = rng.random((count, 16)) + indices = np.argpartition(random, 8, axis=1)[:, :8] + selected = np.take_along_axis(np.broadcast_to(pooled, (count, 16)), indices, axis=1) + values[start : start + count] = 2 * selected.mean(axis=1) - pooled.mean() * 2 + null_windows.append(values) + observed = min(observed_windows) + null = np.minimum(null_windows[0], null_windows[1]) + return float((np.count_nonzero(null >= observed) + 1) / (BOOTSTRAPS + 1)) + + +def kendall_tau_b(left: list[float], right: list[float]) -> float: + concordant = discordant = tie_left = tie_right = 0 + for i in range(len(left)): + for j in range(i + 1, len(left)): + a = np.sign(left[i] - left[j]) + b = np.sign(right[i] - right[j]) + if a == 0 and b != 0: + tie_left += 1 + elif b == 0 and a != 0: + tie_right += 1 + elif a * b > 0: + concordant += 1 + elif a * b < 0: + discordant += 1 + denominator = math.sqrt( + (concordant + discordant + tie_left) * (concordant + discordant + tie_right) + ) + return (concordant - discordant) / denominator if denominator else 0.0 + + +def ranked_families(shares: dict[str, float]) -> list[dict[str, Any]]: + ordered = sorted(FAMILIES, key=lambda family: shares[family], reverse=True) + result = [] + rank = 1 + group_top = None + for index, family in enumerate(ordered): + if group_top is None or group_top - shares[family] > 0.01: + rank = index + 1 + group_top = shares[family] + result.append({"family": family, "share": shares[family], "rank": rank}) + return result + + +def operator_window_valid( + window: dict[str, Any], summary: dict[str, Any] +) -> bool: + representative = window.get("representativeness") + return bool( + window["classifiable_fraction"] >= 0.70 + and representative + and representative["valid"] + and summary["profile_recovery_valid"] + ) + + +def analyze(root: Path, private: Path) -> dict[str, Any]: + markers, confirmation_markers, complete_stages, unaccepted_markers = ( + accepted_marker_paths(root) + ) + if len(markers) != 40 or confirmation_markers: + raise RuntimeError( + f"A-P3-7 run count before analysis: primary={len(markers)} " + f"confirm={len(confirmation_markers)}" + ) + summaries: dict[str, dict[str, Any]] = {} + layer_records: dict[str, list[dict[str, Any]]] = {} + for path in markers + confirmation_markers: + marker = load_json(path) + run_id = marker["run_id"] + summary, records = summarize_run(path.parent, marker) + summary["run_dir"] = str(path.parent) + summaries[run_id] = summary + layer_records[run_id] = records + cell_loads: dict[str, set[str]] = defaultdict(set) + for summary in summaries.values(): + cell_loads[f"{summary['pattern']}/{summary['config']}"].add(summary["load"]) + complete_cells = sorted( + cell for cell, loads in cell_loads.items() if loads == {"saturation", "moderate"} + ) + missing_cells = sorted(expected_cells() - set(complete_cells)) + if missing_cells != sorted(AP37_MISSING_CELLS): + raise RuntimeError(f"A-P3-7 missing-cell mismatch: {missing_cells}") + completed_c00_patterns = sorted( + cell.split("/", 1)[0] for cell in complete_cells if cell.endswith("/C00") + ) + ragged = {} + ragged_pieces = {} + for pattern in (f"P{index:02d}" for index in range(1, 12)): + values = {} + for cohort in (32, 64, 128): + value, pieces = manifest_raggedness(private / f"{pattern}.jsonl", cohort) + values[f"R{cohort}"] = value + if cohort == 64: + ragged_pieces[pattern] = np.asarray(pieces) + ragged[pattern] = values + for summary in summaries.values(): + summary["waste"].update(ragged[summary["pattern"]]) + add_mixed_interference(summaries, layer_records) + + layer2: dict[str, list[dict[str, Any]]] = {} + layer2_issues = [] + for run_id, summary in summaries.items(): + run_dir = Path(summary["run_dir"]) + traces = sorted((run_dir / "traces").glob("*.pt.trace.json*")) + if not traces: + if not summary["layer2_missing_after_controller_cleanup"] and not run_id.endswith("confirmation"): + layer2_issues.append(f"unexpected missing traces: {run_id}") + continue + all_layer, _ = run_records(run_dir) + windows = [trace_steps(path, all_layer) for path in traces] + if summary["config"] != "C00-TP2": + clean_result = load_json(run_dir / "client/result.json") + t0 = int(clean_result["t0_mono_ns"]) + clean_c = [ + item + for item in all_layer + if t0 + int(220e9) <= int(item["submit_mono_ns"]) < t0 + int(300e9) + ] + for window in windows: + window["representativeness"] = representativeness(window, clean_c) + layer2[run_id] = windows + + all_operator_windows = { + run_id: [ + { + "trace": window["path"], + "shares": window["shares"], + "other_share": window["other_share"], + "classifiable_fraction": window["classifiable_fraction"], + "attention_subshares": window["attention_subshares"], + "mode_steps": window["mode_steps"], + "mode_shares": window["mode_shares"], + "top_unmatched": window["top_unmatched"], + "representativeness": window.get("representativeness"), + } + for window in windows + ] + for run_id, windows in sorted(layer2.items()) + } + + patterns = [f"P{i:02d}" for i in range(1, 12)] + primary_ids = [f"{pattern}-C00-moderate" for pattern in completed_c00_patterns] + operator_table = {} + operator_share_table = {} + operator_mode_segments = {} + invalid_primary_windows = [] + for pattern in patterns: + operator_share_table[pattern] = {} + for load in ("saturation", "moderate"): + run_id = f"{pattern}-C00-{load}" + windows = layer2.get(run_id, []) + rows = [] + for index, window in enumerate(windows, 1): + valid = operator_window_valid(window, summaries[run_id]) + rows.append( + { + "window": index, + "shares": window["shares"], + "ranking": ranked_families(window["shares"]), + "classifiable_fraction": window["classifiable_fraction"], + "representativeness": window["representativeness"], + "attention_subshares": window["attention_subshares"], + "mode_steps": window["mode_steps"], + "mode_shares": window["mode_shares"], + "top_unmatched": window["top_unmatched"], + "valid": valid, + } + ) + mean_shares = ( + { + family: float( + np.mean([window["shares"][family] for window in windows]) + ) + for family in FAMILIES + } + if windows + else None + ) + status = ( + "EVALUABLE" + if len(rows) == 2 and all(row["valid"] for row in rows) + else "NOT_EVALUABLE" + ) + reason = None + if run_id not in summaries: + reason = "pattern/config cell missing under A-P3-7" + elif len(rows) != 2: + reason = f"accepted trace windows {len(rows)}/2" + elif status != "EVALUABLE": + reason = "classifiability, representativeness, or recovery gate failed" + operator_share_table[pattern][load] = { + "run_id": run_id, + "status": status, + "reason": reason, + "window_count": len(rows), + "valid_window_count": sum(row["valid"] for row in rows), + "mean_shares": mean_shares, + "mean_ranking": ranked_families(mean_shares) if mean_shares else None, + "windows": rows, + } + if run_id in primary_ids: + if len(rows) != 2: + invalid_primary_windows.append( + f"{run_id}: trace count {len(rows)}" + ) + else: + operator_table[run_id] = rows + invalid_primary_windows.extend( + f"{run_id}: window {row['window']}" + for row in rows + if not row["valid"] + ) + if not windows: + continue + by_mode: dict[str, list[dict[str, Any]]] = defaultdict(list) + for window in windows: + for step in window["steps"]: + by_mode[step["runtime_mode"]].append(step) + operator_mode_segments[run_id] = {} + for runtime_mode, steps in sorted(by_mode.items()): + if len(steps) < 8: + continue + durations = defaultdict(float) + for step in steps: + for family, duration in step["duration_us"].items(): + durations[family] += duration + total = sum(durations.values()) + operator_mode_segments[run_id][runtime_mode] = { + "steps": len(steps), + "shares": { + family: durations.get(family, 0.0) / total + for family in FAMILIES + }, + "other_share": durations.get("other", 0.0) / total, + } + + rng = np.random.default_rng(SEED) + inversions = [] + total_ranking_tests = math.comb(11, 2) * math.comb(len(FAMILIES), 2) + for pi, left_pattern in enumerate(patterns): + left_id = f"{left_pattern}-C00-moderate" + if len(operator_table.get(left_id, [])) != 2 or any( + not item["valid"] for item in operator_table.get(left_id, []) + ): + continue + for right_pattern in patterns[pi + 1 :]: + right_id = f"{right_pattern}-C00-moderate" + if len(operator_table.get(right_id, [])) != 2 or any( + not item["valid"] for item in operator_table.get(right_id, []) + ): + continue + for ai, family_a in enumerate(FAMILIES): + for family_b in FAMILIES[ai + 1 :]: + left_gaps = [ + window["shares"][family_a] - window["shares"][family_b] + for window in layer2[left_id] + ] + right_gaps = [ + window["shares"][family_a] - window["shares"][family_b] + for window in layer2[right_id] + ] + orientation = None + if min(left_gaps) >= 0.05 and max(right_gaps) <= -0.05: + orientation = 1.0 + elif max(left_gaps) <= -0.05 and min(right_gaps) >= 0.05: + orientation = -1.0 + if orientation is None: + continue + p = permutation_p( + layer2[left_id], + layer2[right_id], + family_a, + family_b, + orientation, + rng, + ) + inversions.append( + { + "left": left_pattern, + "right": right_pattern, + "family_a": family_a, + "family_b": family_b, + "left_gaps": left_gaps, + "right_gaps": right_gaps, + "orientation": orientation, + "p": p, + } + ) + holm(inversions, total_tests=total_ranking_tests) + accepted_inversions = [item for item in inversions if item["p_holm"] < 0.05] + + tau = [] + for index, left in enumerate(patterns): + left_id = f"{left}-C00-moderate" + if len(operator_table.get(left_id, [])) != 2 or any( + not item["valid"] for item in operator_table.get(left_id, []) + ): + continue + left_mean = { + family: float(np.mean([window["shares"][family] for window in layer2[left_id]])) + for family in FAMILIES + } + left_rank = {item["family"]: item["rank"] for item in ranked_families(left_mean)} + for right in patterns[index + 1 :]: + right_id = f"{right}-C00-moderate" + if len(operator_table.get(right_id, [])) != 2 or any( + not item["valid"] for item in operator_table.get(right_id, []) + ): + continue + right_mean = { + family: float( + np.mean([window["shares"][family] for window in layer2[right_id]]) + ) + for family in FAMILIES + } + right_rank = { + item["family"]: item["rank"] for item in ranked_families(right_mean) + } + tau.append( + { + "left": left, + "right": right, + "tau_b": kendall_tau_b( + [left_rank[family] for family in FAMILIES], + [right_rank[family] for family in FAMILIES], + ), + } + ) + + waste_contrasts = [] + contrast_status = [] + by_metric: dict[str, list[dict[str, Any]]] = defaultdict(list) + for irregular, control in IRREGULAR_CONTROLS: + left_id = f"{irregular}-C00-moderate" + right_id = f"{control}-C00-moderate" + missing = [ + run_id.removesuffix("-moderate") + for run_id in (left_id, right_id) + if run_id not in summaries + ] + if missing: + contrast_status.append( + { + "irregular": irregular, + "control": control, + "evaluable": False, + "verdict": "NOT_EVALUABLE", + "missing_cells": missing, + "efficiency_loss": None, + "metrics": [], + "passing_metrics": [], + "passes_h1b": False, + } + ) + continue + left = summaries[left_id] + right = summaries[right_id] + efficiency_loss = 1 - ( + left["layer1"]["token_efficiency_per_ms"] + / right["layer1"]["token_efficiency_per_ms"] + ) + metric_results = [] + for metric in ("padding_fraction", "graph_miss_rate", "overflow_rate", "mixed_interference"): + a = ratio_blocks(left, metric) + b = ratio_blocks(right, metric) + if a is None or b is None: + result = { + "point": None, + "ci95_low": None, + "ci95_high": None, + "simultaneous_low": None, + "simultaneous_high": None, + "p": 1.0, + "available": False, + } + else: + result = bootstrap_difference(a, b, rng) + result["available"] = True + result.update( + { + "irregular": irregular, + "control": control, + "metric": metric, + "efficiency_loss": efficiency_loss, + "evaluable": True, + } + ) + by_metric[metric].append(result) + waste_contrasts.append(result) + metric_results.append(result) + result = bootstrap_difference(ragged_pieces[irregular], ragged_pieces[control], rng) + result["available"] = True + result.update( + { + "irregular": irregular, + "control": control, + "metric": "R64", + "efficiency_loss": efficiency_loss, + "evaluable": True, + } + ) + by_metric["R64"].append(result) + waste_contrasts.append(result) + metric_results.append(result) + result = { + "irregular": irregular, + "control": control, + "metric": "moe_layer_cv", + "efficiency_loss": efficiency_loss, + "evaluable": True, + "point": None, + "ci95_low": None, + "ci95_high": None, + "simultaneous_low": None, + "simultaneous_high": None, + "p": 1.0, + "available": False, + } + by_metric["moe_layer_cv"].append(result) + waste_contrasts.append(result) + metric_results.append(result) + contrast_status.append( + { + "irregular": irregular, + "control": control, + "evaluable": True, + "verdict": None, + "missing_cells": [], + "efficiency_loss": efficiency_loss, + "metrics": metric_results, + "passing_metrics": [], + "passes_h1b": False, + } + ) + for values in by_metric.values(): + holm(values, total_tests=len(IRREGULAR_CONTROLS)) + thresholds = { + "padding_fraction": 0.05, + "graph_miss_rate": 0.10, + "overflow_rate": 0.10, + "R64": 0.15, + "mixed_interference": 0.10, + "moe_layer_cv": 0.15, + } + for item in waste_contrasts: + item["material"] = ( + item["point"] is not None + and item["point"] >= thresholds[item["metric"]] + and item["simultaneous_low"] is not None + and item["simultaneous_low"] > 0 + and item["p_holm"] < 0.05 + ) + item["coincident_efficiency_or_residual"] = ( + item["efficiency_loss"] >= 0.05 + or ( + summaries[f"{item['irregular']}-C00-moderate"]["waste"].get( + "mixed_interference" + ) + or -math.inf + ) + > 0 + ) + item["passes_h1b"] = item["material"] and item["coincident_efficiency_or_residual"] + for contrast in contrast_status: + if not contrast["evaluable"]: + continue + hits = [item for item in contrast["metrics"] if item["passes_h1b"]] + contrast["passing_metrics"] = [item["metric"] for item in hits] + contrast["passes_h1b"] = bool(hits) + contrast["verdict"] = "PASS" if hits else "NO_QUALIFYING_METRIC" + observed_missing_contrasts = sorted( + (item["irregular"], item["control"]) + for item in contrast_status + if not item["evaluable"] + ) + if observed_missing_contrasts != sorted(AP37_MISSING_CONTRASTS): + raise RuntimeError( + f"A-P3-7 missing-contrast mismatch: {observed_missing_contrasts}" + ) + h1b_hits = [item for item in waste_contrasts if item["passes_h1b"]] + + controller = load_json(root / "controller-state.json") + trace_count = sum( + len(list((Path(summary["run_dir"]) / "traces").glob("*.pt.trace.json*"))) + for summary in summaries.values() + ) + other_gpu_processes = [] + for stage in complete_stages: + path = stage / "other-gpu-processes.json" + if path.exists(): + other_gpu_processes.extend(load_json(path)) + clean_failures = sum(int(item["clean"]["failed"]) for item in summaries.values()) + moderate_rate_ok = [] + for run_id, summary in summaries.items(): + if summary["load"] == "moderate": + offered = float(summary["clean"]["offered_rps"]) + requested = float(load_json(Path(summary["run_dir"]) / "client/result.json")["request_rate"]) + moderate_rate_ok.append(abs(offered / requested - 1) <= 0.05) + ratios = [] + for summary in summaries.values(): + ratios.extend( + [ + summary["waste"]["padding_fraction"], + summary["waste"]["graph_miss_rate"], + summary["waste"]["overflow_rate"], + summary["waste"]["R64"], + summary["layer1"]["kv_usage_mean"], + summary["layer1"]["kv_usage_max"], + ] + ) + p10_ap36 = ap36_warmup_stability( + root / "primary/P10-C00-TP2/saturation" + ) + same_wave_warmup = {} + for cell in ("P11-C00", "P03-C11"): + result = load_json(root / f"primary/{cell}/saturation/client/result.json") + requests = jsonl(root / f"primary/{cell}/saturation/client/requests.jsonl") + same_wave_warmup[cell] = sum( + bool(item["success"]) + and 0 + <= float(item["completed_s"]) + < float(result["warmup_seconds"]) + for item in requests + ) + boundary_marker = load_json( + Path(summaries["P01-C01-moderate"]["run_dir"]) / "run-complete.json" + ) + operational_findings = { + "p10_tp2_non_stabilization": { + **p10_ap36, + "status": "PATTERN_CONDITIONED_OPERATIONAL_FINDING", + "accepted_measurement": False, + "comparison": ( + "all synthetic pattern runs passed their applicable registered " + "warm-up gates; orchestrator adjudication" + ), + "same_wave_synthetic_warmup_completions": same_wave_warmup, + }, + "long_context_drain": { + "run_id": "P10-C01-saturation", + "drain_seconds": summaries["P10-C01-saturation"]["drain_seconds"], + "amended_budget_seconds": 600, + "quarantined": False, + }, + "failure_boundary": { + "run_id": "P01-C01-moderate", + "clean_failures": summaries["P01-C01-moderate"]["clean"]["failed"], + "excluded_window_failures": boundary_marker["client"][ + "excluded_window_failures" + ], + "excluded_failure_kinds": boundary_marker["client"][ + "excluded_window_failure_kinds" + ], + }, + "layer2_sampling": { + "completed_c00_moderate_patterns": len(completed_c00_patterns), + "evaluable_c00_moderate_patterns": sum( + operator_share_table[pattern]["moderate"]["status"] + == "EVALUABLE" + for pattern in completed_c00_patterns + ), + "invalid_windows": len(invalid_primary_windows), + "classifiable_fraction_min": min( + window["classifiable_fraction"] + for windows in layer2.values() + for window in windows + ), + }, + } + sanity = { + "numeric": { + "completed_throughput_rps": numeric_sanity( + [item["clean"]["completed_throughput_rps"] for item in summaries.values()] + ), + "token_efficiency_per_ms": numeric_sanity( + [item["layer1"]["token_efficiency_per_ms"] for item in summaries.values()] + ), + "drain_seconds": numeric_sanity([item["drain_seconds"] for item in summaries.values()]), + "layer1_clean_steps": numeric_sanity( + [item["layer1"]["records_clean"] for item in summaries.values()] + ), + "operator_classifiable_fraction": numeric_sanity( + [window["classifiable_fraction"] for windows in layer2.values() for window in windows] + ), + "waste_ratios": numeric_sanity(ratios), + "kendall_tau_b": numeric_sanity([item["tau_b"] for item in tau]), + "operator_share": numeric_sanity( + [ + share + for windows in layer2.values() + for window in windows + for share in window["shares"].values() + ] + ), + "waste_contrast_effect": numeric_sanity( + [item["point"] for item in waste_contrasts] + ), + }, + "invariants": { + "ap37_primary_runs_40": len(markers) == 40, + "ap37_confirmation_runs_0": len(confirmation_markers) == 0, + "accepted_run_ids_unique": len(summaries) == len(markers), + "complete_cells_20": len(complete_cells) == 20, + "missing_cells_exact": missing_cells == sorted(AP37_MISSING_CELLS), + "completed_c00_patterns_9": len(completed_c00_patterns) == 9, + "clean_duration_240": all(item["clean"]["duration_s"] == 240 for item in summaries.values()), + "clean_failures_zero": clean_failures == 0, + "moderate_rate_within_5pct": all(moderate_rate_ok), + "layer1_footer_invariants": all( + all(load_json(Path(item["run_dir"]) / "run-complete.json")["layer1"]["invariants"].values()) + for item in summaries.values() + ), + "ratios_in_unit_interval": all(0 <= value <= 1 for value in ratios), + "trace_count_accepted_72": trace_count == 72, + "missing_trace_count_accepted_8": controller.get("missing_trace_files") == 8, + "controller_frozen_at_ap36_failure": controller.get("status") == "failed" + and controller.get("completed_measured_runs") == 40, + "complete_stage_count_12": len(complete_stages) == 12, + "clock_and_load_snapshots_complete": all( + (stage / "clocks-before.txt").exists() + and (stage / "clocks-after.txt").exists() + and (stage / "loadavg-before.txt").exists() + and (stage / "loadavg-after.txt").exists() + for stage in complete_stages + ), + "other_gpu_processes_absent": not other_gpu_processes, + "drain_quarantine_under_20pct": controller.get("drain_quarantined_runs", 0) / 40 <= 0.20, + "no_layer2_parser_issues": not layer2_issues, + "h1b_evaluable_contrasts_6": sum( + item["evaluable"] for item in contrast_status + ) + == 6, + "h1b_missing_contrasts_exact": observed_missing_contrasts + == sorted(AP37_MISSING_CONTRASTS), + "ap36_operational_finding_reproduced": math.isclose( + p10_ap36["normalized_drift"], 0.367639109533929 + ) + and p10_ap36["warmup_completions"] == 17, + "patterns_not_all_identical_throughput": len( + {round(item["clean"]["completed_throughput_rps"], 9) for item in summaries.values()} + ) + > 1, + }, + "declared_deviations": { + "missing_saturation_traces": 8, + "missing_cells": missing_cells, + "missing_confirmations": ["P10", "P06", "P03", "P01"], + "unaccepted_canonical_run_markers_excluded": unaccepted_markers, + "invalid_primary_layer2_windows": invalid_primary_windows, + "moe_layer_cv": "N/A: layer scopes do not cover >=80% of MoE GEMM time", + }, + } + if not all(sanity["invariants"].values()): + raise RuntimeError(f"data sanity red flag: {sanity['invariants']}") + return { + "schema": 1, + "analysis_seed": SEED, + "bootstrap_resamples": BOOTSTRAPS, + "matrix": { + "primary_runs": len(markers), + "confirmation_runs": len(confirmation_markers), + "complete_cells": complete_cells, + "missing_cells": missing_cells, + "completed_c00_patterns": completed_c00_patterns, + "trace_files": trace_count, + "drain_quarantined_runs": controller.get("drain_quarantined_runs", 0), + "clean_window_failures": clean_failures, + "gpu_hours_total": controller["gpu_hours_total"], + }, + "runs": summaries, + "all_operator_windows": all_operator_windows, + "operator_windows": operator_table, + "operator_share_table": operator_share_table, + "operator_mode_segments": operator_mode_segments, + "ranking": { + "tests": total_ranking_tests, + "completed_patterns": completed_c00_patterns, + "evaluable_patterns": [ + pattern + for pattern in completed_c00_patterns + if operator_share_table[pattern]["moderate"]["status"] + == "EVALUABLE" + ], + "missing_patterns": ["P05", "P11"], + "candidates": inversions, + "accepted_inversions": accepted_inversions, + "kendall_tau_b": tau, + "invalid_primary_windows": invalid_primary_windows, + }, + "waste_contrasts": waste_contrasts, + "waste_contrast_status": contrast_status, + "waste_thresholds": thresholds, + "robustness": { + "mixed_interference": "leave-one-pattern-out fits applied within config/load", + "operator_ranking": "two fixed wall-separated windows; no separate LOAO procedure was preregistered", + "confirmation_runs": "NOT_EVALUABLE: all four confirmations are missing", + }, + "operational_findings": operational_findings, + "hypothesis": { + "H1a": partial_verdict(bool(accepted_inversions)), + "H1b": partial_verdict(bool(h1b_hits)), + "compound": "CONFIRMED" if accepted_inversions and h1b_hits else "PARTIAL" if accepted_inversions or h1b_hits else "INCONCLUSIVE", + "h1b_hits": h1b_hits, + "refutation_allowed": False, + "logical_asymmetry": "A-P3-7 permits existential confirmation but incomplete coverage forbids refutation", + }, + "sanity": sanity, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--private", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + result = analyze(args.root, args.private) + args.out.parent.mkdir(parents=True, exist_ok=True) + temporary = args.out.with_suffix(args.out.suffix + ".tmp") + temporary.write_text(json.dumps(result, indent=2, sort_keys=True, allow_nan=False) + "\n") + temporary.replace(args.out) + print(json.dumps({"out": str(args.out), "hypothesis": result["hypothesis"], "sanity": result["sanity"]}, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/opprof_phase3_client.py b/runs/opprof-phase3/provenance/opprof_phase3_client.py new file mode 100644 index 0000000..b1f8f3e --- /dev/null +++ b/runs/opprof-phase3/provenance/opprof_phase3_client.py @@ -0,0 +1,794 @@ +#!/usr/bin/env python3 +"""Token-exact fixed-duration client for the OpProf Phase-3 protocol.""" + +from __future__ import annotations + +import argparse +import asyncio +import gzip +import hashlib +import json +import math +import os +import time +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import aiohttp + +SCHEMA = 1 +TOKEN_BASE = 1000 +TOKEN_SPAN = 100000 + + +class ManifestExhausted(RuntimeError): + pass + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def atomic_json(path: Path, value: Any, mode: int = 0o640) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode) + with os.fdopen(fd, "w", encoding="utf-8") as f: + json.dump(value, f, sort_keys=True, indent=2) + f.write("\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def atomic_jsonl(path: Path, rows: list[dict[str, Any]], mode: int = 0o640) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + fd = os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, mode) + with os.fdopen(fd, "w", encoding="utf-8") as f: + for row in rows: + f.write(json.dumps(row, sort_keys=True, separators=(",", ":")) + "\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def parse_range(value: str) -> tuple[int, int]: + lo_text, hi_text = value.split(":", 1) + lo, hi = int(lo_text), int(hi_text) + if lo <= 0 or hi < lo: + raise argparse.ArgumentTypeError(f"invalid positive range: {value}") + return lo, hi + + +def _integer_counts(weights: list[float], total: int) -> list[int]: + raw = [w * total for w in weights] + counts = [math.floor(x) for x in raw] + order = sorted( + range(len(raw)), key=lambda i: raw[i] - counts[i], reverse=True + ) + for idx in order[: total - sum(counts)]: + counts[idx] += 1 + return counts + + +def numeric_sanity(values: list[float | int]) -> dict[str, Any]: + finite = [float(x) for x in values if math.isfinite(float(x))] + return { + "n": len(values), + "finite_n": len(finite), + "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, + "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def manifest_summary(rows: list[dict[str, Any]]) -> dict[str, Any]: + return { + "schema": SCHEMA, + "rows": len(rows), + "input_tokens": numeric_sanity([int(r["input_tokens"]) for r in rows]), + "output_tokens": numeric_sanity([int(r["output_tokens"]) for r in rows]), + "arrival_values": sorted({str(r["arrival"]) for r in rows}), + "pattern_values": sorted({str(r["pattern_id"]) for r in rows}), + } + + +def materialize(args: argparse.Namespace) -> dict[str, Any]: + import numpy as np + + rng = np.random.default_rng(args.workload_seed) + n = args.num_requests + if args.kind == "prefix-pool": + if args.num_prefixes <= 0 or args.prefix_len <= 0 or args.suffix_fixed <= 0: + raise ValueError("prefix-pool requires positive pool/prefix/suffix") + lengths = np.full(n, args.prefix_len + args.suffix_fixed, dtype=np.int64) + prefix_ids = np.arange(n, dtype=np.int64) % args.num_prefixes + rng.shuffle(prefix_ids) + else: + prefix_ids = np.full(n, -1, dtype=np.int64) + if args.input_uniform: + lo, hi = parse_range(args.input_uniform) + lengths = rng.integers(lo, hi + 1, n, dtype=np.int64) + elif args.input_fixed: + lengths = np.full(n, args.input_fixed, dtype=np.int64) + elif args.input_mixture: + spec = json.loads(args.input_mixture) + if not isinstance(spec, dict) or not spec: + raise ValueError("input mixture must be a non-empty JSON object") + keys = list(spec) + weights = [float(spec[key]) for key in keys] + if any(w < 0 for w in weights) or not math.isclose(sum(weights), 1.0): + raise ValueError("mixture weights must be non-negative and sum to 1") + pieces = [] + for key, count in zip( + keys, _integer_counts(weights, n), strict=True + ): + kind, lo_text, hi_text = key.split(":") + if kind != "uniform": + raise ValueError(f"unsupported mixture component: {key}") + pieces.append( + rng.integers( + int(lo_text), int(hi_text) + 1, count, dtype=np.int64 + ) + ) + lengths = np.concatenate(pieces) + rng.shuffle(lengths) + else: + raise ValueError("exactly one input distribution is required") + if args.output_fixed <= 0 or args.arrival not in {"steady", "burst:8"}: + raise ValueError("invalid output length or arrival class") + + rows = [] + for i in range(n): + row = { + "schema": SCHEMA, + "request_id": f"{args.id}-{i:05d}", + "pattern_id": args.id, + "kind": args.kind, + "input_tokens": int(lengths[i]), + "output_tokens": args.output_fixed, + "arrival": args.arrival, + "token_seed": int(args.workload_seed * 1000003 + i), + } + if args.kind == "prefix-pool": + row.update( + { + "prefix_id": int(prefix_ids[i]), + "num_prefixes": args.num_prefixes, + "prefix_tokens": args.prefix_len, + } + ) + rows.append(row) + + out = Path(args.out) + atomic_jsonl(out, rows, mode=0o600) + summary = manifest_summary(rows) + summary.update({"sha256": sha256_file(out), "path": str(out)}) + atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600) + print(json.dumps(summary, sort_keys=True)) + return summary + + +def materialize_private(args: argparse.Namespace) -> dict[str, Any]: + from transformers import AutoTokenizer + + source = Path(args.source) + selected: list[dict[str, Any]] = [] + with source.open(encoding="utf-8") as f: + for source_index, line in enumerate(f): + row = json.loads(line) + if ( + float(row["sampling_u"]) <= args.sampling_u_max + and int(row["input_length"]) <= args.max_input_tokens + ): + selected.append( + { + "schema": SCHEMA, + "request_id": f"{args.id}-{len(selected):05d}", + "pattern_id": args.id, + "kind": "private-trace", + "input_tokens": int(row["input_length"]), + "output_tokens": min( + int(row["output_length"]), args.output_cap + ), + "arrival": args.arrival, + "source_index": source_index, + "prompt": row["prompt"], + } + ) + + tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True) + diffs = [ + len(tokenizer.encode(row["prompt"], add_special_tokens=False)) + - row["input_tokens"] + for row in selected + ] + exact = sum(diff == 0 for diff in diffs) + exact_fraction = exact / len(diffs) if diffs else 0.0 + max_abs = max((abs(diff) for diff in diffs), default=-1) + if exact_fraction < 0.99 or max_abs > 1: + raise RuntimeError( + "tokenizer parity gate failed: " + f"exact_fraction={exact_fraction:.6f} max_abs_error={max_abs}" + ) + + out = Path(args.out) + atomic_jsonl(out, selected, mode=0o600) + summary = manifest_summary(selected) + summary.update( + { + "sha256": sha256_file(out), + "source_sha256": sha256_file(source), + "tokenizer_exact_n": exact, + "tokenizer_exact_fraction": exact_fraction, + "tokenizer_max_abs_error": max_abs, + "path": str(out), + } + ) + atomic_json(out.with_suffix(out.suffix + ".summary.json"), summary, mode=0o600) + print(json.dumps(summary, sort_keys=True)) + return summary + + +def load_manifest(path: Path) -> list[dict[str, Any]]: + rows = [json.loads(line) for line in path.read_text().splitlines() if line] + required = { + "request_id", + "pattern_id", + "input_tokens", + "output_tokens", + "arrival", + } + if not rows: + raise ValueError("empty manifest") + for row in rows: + if not required.issubset(row): + raise ValueError(f"manifest row lacks {sorted(required - set(row))}") + if len({row["request_id"] for row in rows}) != len(rows): + raise ValueError("duplicate request_id") + return rows + + +def _token_stream(seed: int, count: int) -> list[int]: + state = seed & 0xFFFFFFFF + out = [] + for _ in range(count): + state = (1664525 * state + 1013904223) & 0xFFFFFFFF + out.append(TOKEN_BASE + state % TOKEN_SPAN) + return out + + +def synthetic_prompt(row: dict[str, Any]) -> list[int]: + length = int(row["input_tokens"]) + seed = int(row.get("token_seed", 0)) + if row.get("kind") == "prefix-pool": + prefix_n = int(row["prefix_tokens"]) + tokens = _token_stream(0xA5A50000 + int(row["prefix_id"]), prefix_n) + tokens += _token_stream(seed, length - prefix_n) + offset = prefix_n + else: + tokens = _token_stream(seed, length) + offset = 0 + if length - offset >= 3: + index = int(row["request_id"].rsplit("-", 1)[1]) + tokens[offset : offset + 3] = [ + TOKEN_BASE + index % 100, + TOKEN_BASE + (index // 100) % 100, + TOKEN_BASE + (index // 10000) % 100, + ] + return tokens + + +@dataclass +class RunContext: + args: argparse.Namespace + rows: list[dict[str, Any]] + t0: float + clean_end: float + stop_event: asyncio.Event + lock: asyncio.Lock + next_index: int = 0 + in_flight: int = 0 + max_in_flight: int = 0 + exhausted: bool = False + admission_stop_s: float | None = None + + async def next_row(self) -> dict[str, Any]: + async with self.lock: + if self.next_index >= len(self.rows): + self.exhausted = True + raise ManifestExhausted( + f"manifest exhausted after {self.next_index} admissions" + ) + row = self.rows[self.next_index] + self.next_index += 1 + return row + + +async def request_one( + ctx: RunContext, + session: aiohttp.ClientSession, + row: dict[str, Any], + scheduled: float, +) -> dict[str, Any]: + loop = asyncio.get_running_loop() + admitted = loop.time() + ctx.in_flight += 1 + ctx.max_in_flight = max(ctx.max_in_flight, ctx.in_flight) + status = 0 + actual_output: int | None = None + first_token: float | None = None + error_kind: str | None = None + try: + prompt: str | list[int] = ( + row["prompt"] + if row.get("kind") == "private-trace" + else synthetic_prompt(row) + ) + if not isinstance(prompt, str) and len(prompt) != int(row["input_tokens"]): + raise AssertionError("synthetic prompt length drift") + payload = { + "model": ctx.args.model, + "prompt": prompt, + "max_tokens": int(row["output_tokens"]), + "temperature": ctx.args.temperature, + "ignore_eos": ctx.args.ignore_eos, + "stream": True, + "stream_options": {"include_usage": True}, + "add_special_tokens": False, + "seed": ctx.args.server_seed, + } + headers = { + "Content-Type": "application/json", + "x-request-id": str(row["request_id"]), + } + async with session.post( + ctx.args.base_url.rstrip("/") + "/v1/completions", + json=payload, + headers=headers, + ) as response: + status = response.status + if status != 200: + error_kind = f"http_{status}" + else: + buf = b"" + async for chunk in response.content.iter_any(): + buf += chunk + while b"\n" in buf: + line, buf = buf.split(b"\n", 1) + line = line.strip() + if not line.startswith(b"data:"): + continue + data = line[5:].strip() + if data == b"[DONE]": + continue + event = json.loads(data) + if event.get("choices") and first_token is None: + first_token = loop.time() + if event.get("usage") is not None: + actual_output = int(event["usage"]["completion_tokens"]) + if actual_output is None: + error_kind = "missing_usage" + except (aiohttp.ClientError, asyncio.TimeoutError) as exc: + error_kind = type(exc).__name__ + except Exception as exc: + error_kind = type(exc).__name__ + finally: + completed = loop.time() + ctx.in_flight -= 1 + success = ( + status == 200 + and error_kind is None + and actual_output == int(row["output_tokens"]) + ) + if status == 200 and actual_output is not None and not success: + error_kind = "output_token_mismatch" + return { + "schema": SCHEMA, + "request_id": row["request_id"], + "scheduled_s": scheduled - ctx.t0, + "admitted_s": admitted - ctx.t0, + "first_token_s": None if first_token is None else first_token - ctx.t0, + "completed_s": completed - ctx.t0, + "input_tokens": int(row["input_tokens"]), + "requested_output_tokens": int(row["output_tokens"]), + "actual_output_tokens": actual_output, + "http_status": status, + "success": success, + "error_kind": error_kind, + } + + +async def saturation_load( + ctx: RunContext, session: aiohttp.ClientSession +) -> list[dict[str, Any]]: + results: list[dict[str, Any]] = [] + + async def worker() -> None: + while not ctx.stop_event.is_set(): + try: + row = await ctx.next_row() + except ManifestExhausted: + ctx.stop_event.set() + return + results.append( + await request_one(ctx, session, row, asyncio.get_running_loop().time()) + ) + + tasks = [ + asyncio.create_task(worker()) for _ in range(ctx.args.max_concurrency) + ] + await asyncio.gather(*tasks) + return results + + +async def finite_load( + ctx: RunContext, session: aiohttp.ClientSession, rate: float +) -> list[dict[str, Any]]: + sem = asyncio.Semaphore(ctx.args.max_concurrency) + tasks: list[asyncio.Task[dict[str, Any]]] = [] + batch = 8 if str(ctx.rows[0]["arrival"]) == "burst:8" else 1 + period = batch / rate + event_index = 0 + + async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]: + async with sem: + return await request_one(ctx, session, row, scheduled) + + while not ctx.stop_event.is_set(): + scheduled = ctx.t0 + event_index * period + delay = scheduled - asyncio.get_running_loop().time() + if delay > 0: + try: + await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay) + break + except asyncio.TimeoutError: + pass + if ctx.stop_event.is_set(): + break + try: + for _ in range(batch): + tasks.append( + asyncio.create_task(limited(await ctx.next_row(), scheduled)) + ) + except ManifestExhausted: + ctx.stop_event.set() + break + event_index += 1 + return await asyncio.gather(*tasks) if tasks else [] + + +async def post_profile( + session: aiohttp.ClientSession, base_url: str, endpoint: str +) -> tuple[float, float, int]: + loop = asyncio.get_running_loop() + before = loop.time() + async with session.post(base_url.rstrip("/") + endpoint) as response: + status = response.status + await response.read() + return before, loop.time(), status + + +def _trace_loadable(path: Path) -> bool: + try: + opener = gzip.open if path.suffix == ".gz" else open + with opener(path, "rt", encoding="utf-8") as f: + parsed = json.load(f) + return isinstance(parsed, dict) and isinstance(parsed.get("traceEvents"), list) + except (OSError, EOFError, json.JSONDecodeError): + return False + + +async def wait_new_trace( + trace_dir: Path, before: set[Path], timeout: float +) -> Path: + deadline = asyncio.get_running_loop().time() + timeout + while asyncio.get_running_loop().time() < deadline: + for path in sorted(set(trace_dir.glob("*.pt.trace.json*")) - before): + if _trace_loadable(path): + return path + await asyncio.sleep(0.25) + raise TimeoutError(f"no new loadable trace within {timeout}s") + + +async def timeline( + ctx: RunContext, session: aiohttp.ClientSession +) -> list[dict[str, Any]]: + args = ctx.args + profiles: list[dict[str, Any]] = [] + await asyncio.sleep(max(0, ctx.clean_end - asyncio.get_running_loop().time())) + if args.profile_after_clean: + trace_dir = Path(args.profile_trace_dir) + for window in range(args.num_profile_windows): + prior = set(trace_dir.glob("*.pt.trace.json*")) + start_before, start_after, start_status = await post_profile( + session, args.base_url, "/start_profile" + ) + trace = await wait_new_trace( + trace_dir, prior, args.profile_timeout_seconds + ) + trace_ready = asyncio.get_running_loop().time() + stop_before, stop_after, stop_status = await post_profile( + session, args.base_url, "/stop_profile" + ) + profiles.append( + { + "window": window + 1, + "start_call_s": start_before - ctx.t0, + "start_return_s": start_after - ctx.t0, + "trace_ready_s": trace_ready - ctx.t0, + "stop_call_s": stop_before - ctx.t0, + "stop_return_s": stop_after - ctx.t0, + "start_status": start_status, + "stop_status": stop_status, + "trace_file": trace.name, + "trace_sha256": sha256_file(trace), + } + ) + if start_status != 200 or stop_status != 200: + raise RuntimeError("profile endpoint returned non-200") + await asyncio.sleep(args.recovery_seconds) + else: + await asyncio.sleep(args.post_clean_seconds) + ctx.admission_stop_s = asyncio.get_running_loop().time() - ctx.t0 + ctx.stop_event.set() + return profiles + + +def segment_summary( + records: list[dict[str, Any]], start: float, end: float +) -> dict[str, Any]: + admitted = [r for r in records if start <= r["admitted_s"] < end] + completed = [r for r in records if start <= r["completed_s"] < end] + successes = [r for r in completed if r["success"]] + duration = end - start + return { + "start_s": start, + "end_s": end, + "duration_s": duration, + "admitted": len(admitted), + "completed": len(successes), + "failed": len(completed) - len(successes), + "offered_rps": len(admitted) / duration, + "completed_throughput_rps": len(successes) / duration, + "input_tokens": sum(r["input_tokens"] for r in successes), + "output_tokens": sum(r["actual_output_tokens"] or 0 for r in successes), + } + + +async def run_load(args: argparse.Namespace) -> dict[str, Any]: + manifest = Path(args.manifest) + rows = load_manifest(manifest) + arrivals = {row["arrival"] for row in rows} + if len(arrivals) != 1: + raise ValueError("a manifest must have one arrival class") + if args.load_point == "saturation": + if args.request_rate != "inf": + raise ValueError("saturation requires --request-rate inf") + rate = math.inf + else: + if not args.saturation_result: + raise ValueError("moderate requires --saturation-result") + sat = json.loads(Path(args.saturation_result).read_text()) + rate = args.rate_fraction * float(sat["clean"]["completed_throughput_rps"]) + if not math.isfinite(rate) or rate <= 0: + raise ValueError("derived moderate rate must be positive and finite") + + loop = asyncio.get_running_loop() + t0 = loop.time() + t0_mono_ns = int(t0 * 1e9) + t0_wall_ns = time.time_ns() + clean_seconds = args.clean_segment_seconds * args.num_clean_segments + ctx = RunContext( + args=args, + rows=rows, + t0=t0, + clean_end=t0 + args.warmup_seconds + clean_seconds, + stop_event=asyncio.Event(), + lock=asyncio.Lock(), + ) + timeout = aiohttp.ClientTimeout(total=None, connect=30, sock_read=600) + connector = aiohttp.TCPConnector(limit=args.max_concurrency) + control_connector = aiohttp.TCPConnector(limit=2) + async with ( + aiohttp.ClientSession(timeout=timeout, connector=connector) as session, + aiohttp.ClientSession( + timeout=timeout, connector=control_connector + ) as control_session, + ): + profile_task = asyncio.create_task(timeline(ctx, control_session)) + load_task = asyncio.create_task( + saturation_load(ctx, session) + if math.isinf(rate) + else finite_load(ctx, session, rate) + ) + try: + profiles = await profile_task + except Exception: + ctx.stop_event.set() + await load_task + raise + records = await load_task + + clean_start = args.warmup_seconds + clean_end = clean_start + clean_seconds + clean = segment_summary(records, clean_start, clean_end) + segments = [] + for i in range(args.num_clean_segments): + start = clean_start + i * args.clean_segment_seconds + segments.append( + { + "name": chr(ord("A") + i), + **segment_summary( + records, start, start + args.clean_segment_seconds + ), + } + ) + successful = [r for r in records if r["success"]] + elapsed_seconds = loop.time() - t0 + if ctx.admission_stop_s is None: + raise RuntimeError("admission stop timestamp was not recorded") + drain_seconds = elapsed_seconds - ctx.admission_stop_s + result = { + "schema": SCHEMA, + "manifest_sha256": sha256_file(manifest), + "manifest_rows": len(rows), + "manifest_admitted": ctx.next_index, + "manifest_wrapped": False, + "manifest_exhausted": ctx.exhausted, + "load_point": args.load_point, + "t0_mono_ns": t0_mono_ns, + "t0_wall_ns": t0_wall_ns, + "request_rate": "inf" if math.isinf(rate) else rate, + "rate_fraction": None if math.isinf(rate) else args.rate_fraction, + "arrival": next(iter(arrivals)), + "warmup_seconds": args.warmup_seconds, + "clean_segment_seconds": args.clean_segment_seconds, + "num_clean_segments": args.num_clean_segments, + "elapsed_seconds": elapsed_seconds, + "admission_stop_s": ctx.admission_stop_s, + "drain_seconds": drain_seconds, + "max_in_flight": ctx.max_in_flight, + "records": len(records), + "successful_records": len(successful), + "failed_records": len(records) - len(successful), + "clean": clean, + "segments": segments, + "profiles": profiles, + } + sanity = { + "schema": SCHEMA, + "numeric": { + "input_tokens": numeric_sanity([r["input_tokens"] for r in records]), + "requested_output_tokens": numeric_sanity( + [r["requested_output_tokens"] for r in records] + ), + "actual_output_tokens": numeric_sanity( + [ + r["actual_output_tokens"] + for r in records + if r["actual_output_tokens"] is not None + ] + ), + "scheduled_s": numeric_sanity([r["scheduled_s"] for r in records]), + "admitted_s": numeric_sanity([r["admitted_s"] for r in records]), + "completed_s": numeric_sanity([r["completed_s"] for r in records]), + }, + "invariants": { + "clean_duration_exact": math.isclose(clean["duration_s"], clean_seconds), + "segment_count_exact": len(segments) == args.num_clean_segments, + "manifest_no_wrap": ctx.next_index <= len(rows), + "manifest_not_exhausted": not ctx.exhausted, + "concurrency_bounded": ctx.max_in_flight <= args.max_concurrency, + "drain_within_timeout": drain_seconds <= args.drain_timeout_seconds, + "output_tokens_exact": all( + r["actual_output_tokens"] == r["requested_output_tokens"] + for r in successful + ), + "clean_failures_zero": clean["failed"] == 0, + "profile_count_exact": len(profiles) + == (args.num_profile_windows if args.profile_after_clean else 0), + "profile_status_ok": all( + p["start_status"] == 200 and p["stop_status"] == 200 + for p in profiles + ), + }, + } + if not math.isinf(rate): + sanity["invariants"]["moderate_offered_within_5pct"] = ( + abs(clean["offered_rps"] / rate - 1) <= 0.05 + ) + out = Path(args.result_dir) + out.mkdir(parents=True, exist_ok=True) + atomic_jsonl(out / "requests.jsonl", sorted(records, key=lambda r: r["admitted_s"])) + atomic_jsonl(out / "segments.jsonl", segments) + atomic_json(out / "result.json", result) + atomic_json(out / "sanity.json", sanity) + if ctx.exhausted: + raise ManifestExhausted("manifest exhausted; result retained for diagnosis") + failed = [name for name, ok in sanity["invariants"].items() if not ok] + if failed: + raise RuntimeError(f"client sanity failure: {failed}") + return result + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + mat = sub.add_parser("materialize") + mat.add_argument("--id", required=True) + mat.add_argument("--kind", choices=("synthetic", "prefix-pool"), required=True) + group = mat.add_mutually_exclusive_group() + group.add_argument("--input-uniform") + group.add_argument("--input-fixed", type=int) + group.add_argument("--input-mixture") + mat.add_argument("--output-fixed", type=int, required=True) + mat.add_argument("--prefix", default="none") + mat.add_argument("--arrival", required=True) + mat.add_argument("--num-requests", type=int, required=True) + mat.add_argument("--workload-seed", type=int, required=True) + mat.add_argument("--num-prefixes", type=int, default=0) + mat.add_argument("--prefix-len", type=int, default=0) + mat.add_argument("--suffix-fixed", type=int, default=0) + mat.add_argument("--out", required=True) + private = sub.add_parser("materialize-private") + private.add_argument("--id", required=True) + private.add_argument("--source", required=True) + private.add_argument("--sampling-u-max", type=float, required=True) + private.add_argument("--max-input-tokens", type=int, required=True) + private.add_argument("--output-cap", type=int, required=True) + private.add_argument("--preserve-prompts", action="store_true", required=True) + private.add_argument("--disable-shuffle", action="store_true", required=True) + private.add_argument("--arrival", required=True) + private.add_argument("--model", required=True) + private.add_argument("--out", required=True) + run = sub.add_parser("run") + run.add_argument("--manifest", required=True) + run.add_argument("--base-url", required=True) + run.add_argument("--model", required=True) + run.add_argument("--load-point", choices=("saturation", "moderate"), required=True) + run.add_argument("--request-rate") + run.add_argument("--saturation-result") + run.add_argument("--rate-fraction", type=float, default=0.60) + run.add_argument("--max-concurrency", type=int, default=256) + run.add_argument("--ignore-eos", action="store_true") + run.add_argument("--temperature", type=float, default=0.0) + run.add_argument("--warmup-seconds", type=float, default=60) + run.add_argument("--clean-segment-seconds", type=float, default=80) + run.add_argument("--num-clean-segments", type=int, default=3) + run.add_argument("--profile-after-clean", action="store_true") + run.add_argument("--num-profile-windows", type=int, default=0) + run.add_argument("--profile-warmup-iterations", type=int, default=2) + run.add_argument("--profile-active-iterations", type=int, default=8) + run.add_argument("--profile-trace-dir") + run.add_argument("--profile-timeout-seconds", type=float, default=120) + run.add_argument("--recovery-seconds", type=float, default=30) + run.add_argument("--post-clean-seconds", type=float, default=0) + run.add_argument("--drain-timeout-seconds", type=float, default=120) + run.add_argument("--workload-seed", type=int, default=20260712) + run.add_argument("--server-seed", type=int, default=20260712) + run.add_argument("--result-dir", required=True) + return parser + + +def main() -> None: + args = build_parser().parse_args() + if args.command == "materialize": + materialize(args) + elif args.command == "materialize-private": + materialize_private(args) + else: + if args.profile_after_clean and not args.profile_trace_dir: + raise ValueError("--profile-after-clean requires --profile-trace-dir") + print(json.dumps(asyncio.run(run_load(args)), sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/opprof_phase3_controller.py b/runs/opprof-phase3/provenance/opprof_phase3_controller.py new file mode 100644 index 0000000..8c88cfb --- /dev/null +++ b/runs/opprof-phase3/provenance/opprof_phase3_controller.py @@ -0,0 +1,1056 @@ +#!/usr/bin/env python3 +"""Detached/resumable dash0 controller for the Phase-3 co-location gate.""" + +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import math +import os +import re +import shlex +import shutil +import signal +import subprocess +import threading +import time +import urllib.request +from collections import defaultdict +from pathlib import Path +from typing import Any + + +SCHEMA = 1 +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/e-a-colocation" +PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +SOURCE = Path( + "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0" +) +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +STATE = RUN_ROOT / "controller-state.json" +CPU_MAP = { + 0: "0-19", + 1: "20-39", + 2: "40-59", + 3: "60-79", + 4: "80-99", + 5: "100-119", + 6: "120-139", + 7: "140-159", +} +BACKGROUND_8 = { + 1: "P01", + 2: "P02", + 3: "P03", + 4: "P04", + 5: "P06", + 6: "P07", + 7: "P08", +} +BACKGROUND_4 = {1: "P01", 2: "P03", 3: "P06"} +PATTERNS = { + "P01": ["--kind", "synthetic", "--input-uniform", "128:512", + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P02": ["--kind", "synthetic", "--input-uniform", "128:512", + "--output-fixed", "512", "--prefix", "none", "--arrival", "steady"], + "P03": ["--kind", "synthetic", "--input-uniform", "4096:8192", + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P04": ["--kind", "synthetic", "--input-uniform", "4096:8192", + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P05": ["--kind", "synthetic", "--input-mixture", + '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P06": ["--kind", "synthetic", "--input-mixture", + '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P07": ["--kind", "synthetic", "--input-fixed", "1280", + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P08": ["--kind", "prefix-pool", "--num-prefixes", "8", + "--prefix-len", "1024", "--suffix-fixed", "256", + "--output-fixed", "512", "--arrival", "burst:8"], +} +PROFILE_CONFIG = { + "profiler": "torch", + "torch_profiler_with_stack": True, + "torch_profiler_record_shapes": True, + "torch_profiler_use_gzip": True, + "ignore_frontend": True, + "wait_iterations": 0, + "warmup_iterations": 2, + "active_iterations": 8, +} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + with tmp.open("w", encoding="utf-8") as f: + json.dump(value, f, sort_keys=True, indent=2) + f.write("\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def run_text(command: list[str], check: bool = True) -> str: + result = subprocess.run( + command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {shlex.join(command)}\n" + f"{result.stdout}" + ) + return result.stdout + + +def command_log(path: Path, label: str, command: list[str], expected: str) -> None: + with path.open("a", encoding="utf-8") as f: + f.write( + f"GPU_COMMAND {label}: {shlex.join(command)} ; expected={expected}\n" + ) + + +def load_state(resume: bool) -> dict[str, Any]: + if STATE.exists(): + if not resume: + raise RuntimeError("state exists; use --resume") + return json.loads(STATE.read_text()) + return { + "schema": SCHEMA, + "status": "created", + "created_at": time.time(), + "controller_pid": os.getpid(), + "stages": {}, + "fingerprint": {}, + } + + +def save_state(state: dict[str, Any]) -> None: + state["updated_at"] = time.time() + state["controller_pid"] = os.getpid() + atomic_json(STATE, state) + + +def make_fingerprint() -> dict[str, Any]: + return { + "client_sha256": sha256_file(CLIENT), + "controller_sha256": sha256_file(Path(__file__)), + "source_commit": run_text( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD"] + ).strip(), + "model_path": str(MODEL), + "venv_python": str(VENV / "bin/python"), + # JSON object keys are strings. Normalize before the first write so a + # read/compare during --resume is byte-semantically stable. + "cpu_map": {str(gpu): cpus for gpu, cpus in CPU_MAP.items()}, + } + + +def ensure_provenance() -> None: + provenance = RUN_ROOT / "provenance" + provenance.mkdir(parents=True, exist_ok=True) + for source in (CLIENT, Path(__file__)): + dest = provenance / source.name + source_sha = sha256_file(source) + if dest.exists() and sha256_file(dest) != source_sha: + # Preserve the prior immutable copy and add the reviewed runtime + # repair as a content-addressed sibling. + dest = provenance / f"{source.stem}.{source_sha[:12]}{source.suffix}" + if dest.exists() and sha256_file(dest) != source_sha: + raise RuntimeError(f"content-addressed provenance changed: {dest}") + if not dest.exists(): + shutil.copy2(source, dest) + checksums = { + path.name: sha256_file(path) + for path in sorted(provenance.glob("*.py")) + } + atomic_json(provenance / "sha256.json", checksums) + + +def ensure_manifests() -> None: + PRIVATE.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(PRIVATE, 0o700) + for pattern, spec in PATTERNS.items(): + out = PRIVATE / f"{pattern}.jsonl" + if out.exists(): + summary = json.loads( + out.with_suffix(out.suffix + ".summary.json").read_text() + ) + if summary["rows"] != 32768 or summary["sha256"] != sha256_file(out): + raise RuntimeError(f"manifest verification failed: {out}") + continue + command = [ + str(VENV / "bin/python"), + str(CLIENT), + "materialize", + "--id", + pattern, + *spec, + "--num-requests", + "32768", + "--workload-seed", + "20260712", + "--out", + str(out), + ] + run_text(command) + + +def gpu_query() -> list[dict[str, Any]]: + text = run_text( + [ + "nvidia-smi", + "--query-gpu=index,uuid,memory.used,utilization.gpu", + "--format=csv,noheader,nounits", + ] + ) + rows = [] + for line in text.strip().splitlines(): + index, uuid, memory, util = [part.strip() for part in line.split(",")] + rows.append( + { + "index": int(index), + "uuid": uuid, + "memory_used_mib": int(memory), + "utilization_pct": int(util), + } + ) + return rows + + +def compute_apps() -> list[dict[str, Any]]: + text = run_text( + [ + "nvidia-smi", + "--query-compute-apps=gpu_uuid,pid,process_name,used_memory", + "--format=csv,noheader,nounits", + ], + check=False, + ) + rows = [] + for line in text.strip().splitlines(): + if not line or "No running" in line: + continue + parts = [part.strip() for part in line.split(",", 3)] + if len(parts) == 4 and parts[1].isdigit(): + rows.append( + { + "gpu_uuid": parts[0], + "pid": int(parts[1]), + "process_name": parts[2], + "used_memory_mib": int(parts[3].split()[0]), + } + ) + return rows + + +def preflight(gpus: list[int], stage_dir: Path) -> None: + samples = [] + consecutive_idle = 0 + deadline = time.monotonic() + 60 + while time.monotonic() < deadline and consecutive_idle < 3: + snapshots = [row for row in gpu_query() if row["index"] in gpus] + apps = compute_apps() + samples.append({"gpus": snapshots, "compute_apps": apps, "time": time.time()}) + idle = ( + not apps + and all(row["memory_used_mib"] == 0 for row in snapshots) + and all(row["utilization_pct"] == 0 for row in snapshots) + ) + consecutive_idle = consecutive_idle + 1 if idle else 0 + if consecutive_idle < 3: + time.sleep(2) + atomic_json(stage_dir / "gpu-before.json", snapshots) + atomic_json(stage_dir / "gpu-before-samples.json", samples) + (stage_dir / "clocks-before.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"]) + ) + (stage_dir / "loadavg-before.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + (stage_dir / "processes-before.txt").write_text( + run_text(["ps", "-eo", "user,pid,ppid,pgid,lstart,args", "--sort=pid"]) + ) + if consecutive_idle < 3: + raise RuntimeError(f"selected GPUs not stably idle: samples={samples}") + + +class Monitor: + def __init__(self, path: Path, owned_pgids: set[int]) -> None: + self.path = path + self.owned_pgids = owned_pgids + self.stop_event = threading.Event() + self.other_apps: list[dict[str, Any]] = [] + self.thread = threading.Thread(target=self._run, daemon=True) + + def start(self) -> None: + self.thread.start() + + def stop(self) -> None: + self.stop_event.set() + self.thread.join(timeout=20) + + def _run(self) -> None: + with self.path.open("a", encoding="utf-8") as f: + while not self.stop_event.is_set(): + apps = compute_apps() + samples = [] + for app in apps: + try: + pgid = os.getpgid(app["pid"]) + except ProcessLookupError: + pgid = None + sample = {**app, "pgid": pgid} + samples.append(sample) + if pgid not in self.owned_pgids: + self.other_apps.append(sample) + clocks = run_text( + [ + "nvidia-smi", + "--query-gpu=index,clocks.sm,clocks.mem,pstate," + "utilization.gpu,memory.used", + "--format=csv,noheader,nounits", + ], + check=False, + ).strip().splitlines() + row = { + "wall_time": time.time(), + "loadavg": os.getloadavg(), + "gpu_clocks": clocks, + "compute_apps": samples, + } + f.write(json.dumps(row, sort_keys=True) + "\n") + f.flush() + self.stop_event.wait(15) + + +def server_command(gpu: int, stage_dir: Path, trace_dir: Path) -> list[str]: + config = {**PROFILE_CONFIG, "torch_profiler_dir": str(trace_dir)} + return [ + "taskset", + "-c", + CPU_MAP[gpu], + str(VENV / "bin/vllm"), + "serve", + str(MODEL), + "--host", + "127.0.0.1", + "--port", + str(8000 + gpu), + "--tensor-parallel-size", + "1", + "--enable-chunked-prefill", + "--enable-prefix-caching", + "--profiler-config", + json.dumps(config, separators=(",", ":")), + ] + + +def start_server( + gpu: int, stage_dir: Path, state_stage: dict[str, Any] +) -> tuple[subprocess.Popen[Any], Any, Path]: + gpu_dir = stage_dir / f"gpu{gpu}" + gpu_dir.mkdir(parents=True, exist_ok=True) + trace_dir = Path(f"/tmp/wjh-opprof-phase3-ea/{stage_dir.name}/gpu{gpu}") + if trace_dir.exists(): + shutil.rmtree(trace_dir) + trace_dir.mkdir(parents=True) + command = server_command(gpu, stage_dir, trace_dir) + command_log( + stage_dir / "commands.log", + f"{stage_dir.name} server gpu{gpu}", + command, + "startup 60-180s; run 300-450s", + ) + env = os.environ.copy() + env.update( + { + "CUDA_VISIBLE_DEVICES": str(gpu), + "VLLM_OPPROF_DIR": str(gpu_dir / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + log_handle = (gpu_dir / "server.log").open("ab", buffering=0) + process = subprocess.Popen( + command, + cwd=SOURCE, + env=env, + stdout=log_handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + state_stage.setdefault("servers", {})[str(gpu)] = { + "pid": process.pid, + "pgid": process.pid, + "command": command, + "trace_dir": str(trace_dir), + } + return process, log_handle, trace_dir + + +def wait_ready(processes: dict[int, subprocess.Popen[Any]], timeout: float = 300) -> None: + deadline = time.monotonic() + timeout + pending = set(processes) + while pending and time.monotonic() < deadline: + for gpu in list(pending): + if processes[gpu].poll() is not None: + raise RuntimeError(f"server gpu{gpu} exited before ready") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{8000 + gpu}/health", timeout=1 + ) as response: + if response.status == 200: + pending.remove(gpu) + except Exception: + pass + time.sleep(1) + if pending: + raise TimeoutError(f"servers not ready within {timeout}s: {sorted(pending)}") + + +def client_command( + gpu: int, + pattern: str, + stage_dir: Path, + load_point: str, + saturation_result: Path | None, + profile: bool, + trace_dir: Path, + post_clean_seconds: int, +) -> list[str]: + result_dir = stage_dir / f"gpu{gpu}" / "client" + command = [ + "taskset", + "-c", + CPU_MAP[gpu], + str(VENV / "bin/python"), + str(CLIENT), + "run", + "--manifest", + str(PRIVATE / f"{pattern}.jsonl"), + "--base-url", + f"http://127.0.0.1:{8000 + gpu}", + "--model", + str(MODEL), + "--load-point", + load_point, + "--max-concurrency", + "256", + "--ignore-eos", + "--temperature", + "0", + "--warmup-seconds", + "60", + "--clean-segment-seconds", + "80", + "--num-clean-segments", + "3", + "--recovery-seconds", + "30", + "--drain-timeout-seconds", + "120", + "--workload-seed", + "20260712", + "--result-dir", + str(result_dir), + ] + if load_point == "saturation": + command += ["--request-rate", "inf"] + else: + assert saturation_result is not None + command += [ + "--saturation-result", + str(saturation_result), + "--rate-fraction", + "0.60", + ] + if profile: + command += [ + "--profile-after-clean", + "--num-profile-windows", + "1", + "--profile-warmup-iterations", + "2", + "--profile-active-iterations", + "8", + "--profile-trace-dir", + str(trace_dir), + "--profile-timeout-seconds", + "120", + ] + elif post_clean_seconds: + command += ["--post-clean-seconds", str(post_clean_seconds)] + return command + + +def stop_processes(processes: list[subprocess.Popen[Any]]) -> None: + live = [process for process in processes if process.poll() is None] + for process in live: + try: + os.killpg(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 60 + while time.monotonic() < deadline and any(p.poll() is None for p in live): + time.sleep(1) + for sig, wait_seconds in ((signal.SIGTERM, 20), (signal.SIGKILL, 5)): + remaining = [p for p in live if p.poll() is None] + if not remaining: + break + for process in remaining: + try: + os.killpg(process.pid, sig) + except ProcessLookupError: + pass + deadline = time.monotonic() + wait_seconds + while time.monotonic() < deadline and any( + p.poll() is None for p in remaining + ): + time.sleep(0.5) + + +def _process_group_alive(pgid: int) -> bool: + try: + os.killpg(pgid, 0) + return True + except ProcessLookupError: + return False + + +def stop_servers(processes: list[subprocess.Popen[Any]]) -> None: + """Let the API parent ask EngineCore to close before group escalation.""" + groups = [process.pid for process in processes] + for process in processes: + if process.poll() is None: + try: + # Group SIGINT also interrupts EngineCore and loses the OpProf + # footer. The API parent's shutdown path signals it cleanly. + os.kill(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 90 + while time.monotonic() < deadline and any( + _process_group_alive(pgid) for pgid in groups + ): + time.sleep(1) + for sig, wait_seconds in ((signal.SIGTERM, 20), (signal.SIGKILL, 5)): + remaining = [pgid for pgid in groups if _process_group_alive(pgid)] + if not remaining: + break + for pgid in remaining: + try: + os.killpg(pgid, sig) + except ProcessLookupError: + pass + deadline = time.monotonic() + wait_seconds + while time.monotonic() < deadline and any( + _process_group_alive(pgid) for pgid in remaining + ): + time.sleep(0.5) + + +def verify_idle(gpus: list[int], stage_dir: Path) -> None: + samples = [] + consecutive_zero = 0 + deadline = time.monotonic() + 120 + while time.monotonic() < deadline and consecutive_zero < 3: + rows = [row for row in gpu_query() if row["index"] in gpus] + apps = compute_apps() + samples.append({"gpus": rows, "compute_apps": apps, "time": time.time()}) + zero = not apps and all(row["memory_used_mib"] == 0 for row in rows) + consecutive_zero = consecutive_zero + 1 if zero else 0 + if consecutive_zero < 3: + time.sleep(5) + atomic_json(stage_dir / "gpu-after-cleanup.json", samples) + if consecutive_zero < 3: + raise RuntimeError("GPU memory did not reach three stable zero samples") + + +def run_stage( + state: dict[str, Any], + name: str, + backgrounds: dict[int, str], + load_point: str, + saturation_stage: str | None, + profile: bool, +) -> None: + stage_dir = RUN_ROOT / name + stage_record = state["stages"].get(name) + if ( + stage_record + and stage_record.get("status") == "complete" + and (stage_dir / "stage-complete.json").exists() + ): + print(f"RESUME skip complete stage {name}", flush=True) + return + if stage_dir.exists(): + shutil.rmtree(stage_dir) + stage_dir.mkdir(parents=True) + gpus = [0, *sorted(backgrounds)] + state["stages"][name] = { + "status": "preflight", + "started_at": time.time(), + "gpus": gpus, + "backgrounds": backgrounds, + "load_point": load_point, + "profile": profile, + } + save_state(state) + preflight(gpus, stage_dir) + servers: dict[int, subprocess.Popen[Any]] = {} + server_logs = [] + client_processes: dict[int, subprocess.Popen[Any]] = {} + client_logs = [] + trace_dirs: dict[int, Path] = {} + monitor: Monitor | None = None + failure: Exception | None = None + try: + for gpu in gpus: + process, handle, trace_dir = start_server( + gpu, stage_dir, state["stages"][name] + ) + servers[gpu] = process + server_logs.append(handle) + trace_dirs[gpu] = trace_dir + state["stages"][name]["status"] = "starting_servers" + save_state(state) + wait_ready(servers) + state["stages"][name]["servers_ready_at"] = time.time() + state["stages"][name]["status"] = "running_clients" + save_state(state) + monitor = Monitor( + stage_dir / "monitor.jsonl", {process.pid for process in servers.values()} + ) + monitor.start() + + assignments = {0: "P05", **backgrounds} + for gpu, pattern in assignments.items(): + is_target = gpu == 0 + sat_result = ( + RUN_ROOT / saturation_stage / "gpu0/client/result.json" + if is_target and saturation_stage + else None + ) + target_load = load_point if is_target else "saturation" + command = client_command( + gpu, + pattern, + stage_dir, + target_load, + sat_result, + profile=is_target and profile, + trace_dir=trace_dirs[gpu], + post_clean_seconds=60 if not is_target and profile else 0, + ) + command_log( + stage_dir / "commands.log", + f"{name} client gpu{gpu} {pattern}", + command, + "60s warmup + 240s clean + optional 2+8 profile/recovery", + ) + handle = (stage_dir / f"gpu{gpu}/client.log").open("ab", buffering=0) + client_logs.append(handle) + process = subprocess.Popen( + command, + cwd=WORKDIR, + stdout=handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + client_processes[gpu] = process + state["stages"][name].setdefault("clients", {})[str(gpu)] = { + "pid": process.pid, + "pgid": process.pid, + "command": command, + } + save_state(state) + + deadline = time.monotonic() + 780 + while time.monotonic() < deadline and any( + process.poll() is None for process in client_processes.values() + ): + if any(process.poll() is not None for process in servers.values()): + raise RuntimeError("server exited while client was active") + time.sleep(2) + if any(process.poll() is None for process in client_processes.values()): + raise TimeoutError("client stage exceeded 780 seconds") + bad = { + gpu: process.returncode + for gpu, process in client_processes.items() + if process.returncode != 0 + } + if bad: + raise RuntimeError(f"client failures: {bad}") + target_result = stage_dir / "gpu0/client/result.json" + if not target_result.exists(): + raise RuntimeError("target result.json missing") + if profile: + trace_dest = stage_dir / "gpu0/traces" + trace_dest.mkdir() + for path in trace_dirs[0].glob("*"): + if path.is_file(): + shutil.copy2(path, trace_dest / path.name) + traces = list(trace_dest.glob("*.pt.trace.json*")) + if len(traces) != 1: + raise RuntimeError(f"expected one target trace, found {len(traces)}") + state["stages"][name]["status"] = "clients_complete" + save_state(state) + except Exception as exc: + failure = exc + finally: + if monitor is not None: + monitor.stop() + stop_processes(list(client_processes.values())) + stop_servers(list(servers.values())) + for handle in client_logs + server_logs: + handle.close() + (stage_dir / "clocks-after.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"], check=False) + ) + (stage_dir / "loadavg-after.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + (stage_dir / "processes-after.txt").write_text( + run_text( + ["ps", "-eo", "user,pid,ppid,pgid,lstart,args", "--sort=pid"], + check=False, + ) + ) + if monitor is not None: + atomic_json(stage_dir / "other-gpu-processes.json", monitor.other_apps) + try: + verify_idle(gpus, stage_dir) + except Exception as exc: + failure = failure or exc + + if failure is not None: + state["stages"][name]["status"] = "failed" + state["stages"][name]["failure"] = repr(failure) + save_state(state) + raise failure + marker = { + "schema": SCHEMA, + "stage": name, + "completed_at": time.time(), + "target_result_sha256": sha256_file( + stage_dir / "gpu0/client/result.json" + ), + } + atomic_json(stage_dir / "stage-complete.json", marker) + state["stages"][name].update( + {"status": "complete", "completed_at": time.time(), "servers": {}, + "clients": {}} + ) + save_state(state) + + +def classify_kernel(name: str) -> str: + value = name.lower() + if re.search(r"topkgating|moe_align|select_expert|moe.*rout", value): + return "moe_router" + if re.search( + r"nccl|all_?reduce|reduce_scatter|all_?gather|custom.*all.*reduce", value + ): + return "collective" + if re.search( + r"flash|fmha|paged_attention|unified_attention|attention|" + r"reshape_and_cache", value + ): + return "attention" + if re.search(r"fused_moe|moe.*gemm|nvjet", value): + return "moe_gemm" + if re.search( + r"sampling|(? dict[str, Any]: + traces = list((RUN_ROOT / stage / "gpu0/traces").glob("*.pt.trace.json*")) + if len(traces) != 1: + raise RuntimeError(f"{stage}: expected one trace") + trace = traces[0] + opener = gzip.open if trace.suffix == ".gz" else open + with opener(trace, "rt", encoding="utf-8") as f: + data = json.load(f) + durations: defaultdict[str, float] = defaultdict(float) + kernel_count = 0 + steps = set() + executes = 0 + for event in data.get("traceEvents", []): + name = str(event.get("name", "")) + match = re.fullmatch(r"ProfilerStep#(\d+)", name) + if match: + steps.add(int(match.group(1))) + if name.startswith("execute_") and event.get("cat") == "user_annotation": + executes += 1 + if event.get("cat") == "kernel" and float(event.get("dur", 0)) >= 0: + kernel_count += 1 + durations[classify_kernel(name)] += float(event.get("dur", 0)) + total = sum(durations.values()) + shares = { + family: duration / total for family, duration in sorted(durations.items()) + } + result = { + "schema": SCHEMA, + "stage": stage, + "trace_file": trace.name, + "trace_sha256": sha256_file(trace), + "trace_loadable": True, + "kernel_count": kernel_count, + "profiler_steps": sorted(steps), + "execute_annotations": executes, + "duration_us": dict(sorted(durations.items())), + "shares": shares, + "classifiable_fraction": 1.0 - shares.get("other", 0.0), + "valid": ( + kernel_count > 0 + and steps == set(range(2, 10)) + and executes == 8 + and 1.0 - shares.get("other", 0.0) >= 0.70 + ), + } + return result + + +def compare_regime(prefix: str) -> dict[str, Any]: + solo_result = json.loads( + (RUN_ROOT / "solo-moderate/gpu0/client/result.json").read_text() + ) + coloc_result = json.loads( + (RUN_ROOT / f"{prefix}-moderate/gpu0/client/result.json").read_text() + ) + solo_trace = trace_analysis("solo-moderate") + coloc_trace = trace_analysis(f"{prefix}-moderate") + solo_t = float(solo_result["clean"]["completed_throughput_rps"]) + coloc_t = float(coloc_result["clean"]["completed_throughput_rps"]) + top3 = sorted( + solo_trace["shares"], key=solo_trace["shares"].get, reverse=True + )[:3] + share_deltas = { + family: abs( + coloc_trace["shares"].get(family, 0) + - solo_trace["shares"].get(family, 0) + ) + for family in top3 + } + throughput_delta = abs(coloc_t / solo_t - 1) + return { + "schema": SCHEMA, + "regime": prefix, + "solo_throughput_rps": solo_t, + "colocated_throughput_rps": coloc_t, + "throughput_delta_fraction": throughput_delta, + "top3_solo_families": top3, + "solo_operator_shares": { + family: solo_trace["shares"][family] for family in top3 + }, + "colocated_operator_shares": { + family: coloc_trace["shares"].get(family, 0) for family in top3 + }, + "operator_share_delta_fraction": share_deltas, + "solo_trace": solo_trace, + "colocated_trace": coloc_trace, + "pass": ( + solo_trace["valid"] + and coloc_trace["valid"] + and throughput_delta < 0.03 + and all(delta < 0.03 for delta in share_deltas.values()) + ), + } + + +def numeric_sanity(values: list[float]) -> dict[str, Any]: + return { + "n": len(values), + "finite_n": sum(math.isfinite(value) for value in values), + "missing_n": sum(not math.isfinite(value) for value in values), + "min": min(values) if values else None, + "max": max(values) if values else None, + "distinct_n": len(set(values)), + } + + +def write_analysis(comparisons: list[dict[str, Any]]) -> dict[str, Any]: + accepted = next((c["regime"] for c in comparisons if c["pass"]), None) + throughputs = [ + value + for c in comparisons + for value in (c["solo_throughput_rps"], c["colocated_throughput_rps"]) + ] + deltas = [ + delta + for c in comparisons + for delta in c["operator_share_delta_fraction"].values() + ] + result = { + "schema": SCHEMA, + "comparisons": comparisons, + "verdict": ( + f"{accepted.replace('coloc-', '')}-way" + if accepted is not None + else "STOP" + ), + "sanity": { + "throughput_rps": numeric_sanity(throughputs), + "throughput_delta": numeric_sanity( + [c["throughput_delta_fraction"] for c in comparisons] + ), + "operator_share_delta": numeric_sanity(deltas), + "invariants": { + "throughputs_positive": all(x > 0 for x in throughputs), + "share_deltas_in_0_1": all(0 <= x <= 1 for x in deltas), + "top3_exact": all( + len(c["top3_solo_families"]) == 3 for c in comparisons + ), + "traces_valid": all( + c["solo_trace"]["valid"] and c["colocated_trace"]["valid"] + for c in comparisons + ), + }, + }, + } + atomic_json(RUN_ROOT / "colocation-analysis.json", result) + return result + + +def clean_recorded_processes(state: dict[str, Any]) -> None: + for stage in state.get("stages", {}).values(): + if stage.get("status") == "complete": + continue + pgids = [ + entry.get("pgid") + for group in ("clients", "servers") + for entry in stage.get(group, {}).values() + if entry.get("pgid") + ] + for pgid in pgids: + try: + os.killpg(int(pgid), signal.SIGINT) + except ProcessLookupError: + pass + if pgids: + time.sleep(5) + + +def execute(resume: bool) -> None: + RUN_ROOT.mkdir(parents=True, exist_ok=True) + state = load_state(resume) + if resume: + clean_recorded_processes(state) + fingerprint = make_fingerprint() + if state["fingerprint"] and state["fingerprint"] != fingerprint: + raise RuntimeError("resume fingerprint differs from frozen execution") + state["fingerprint"] = fingerprint + state["status"] = "running" + save_state(state) + ensure_provenance() + ensure_manifests() + run_stage(state, "solo-saturation", {}, "saturation", None, False) + run_stage( + state, + "solo-moderate", + {}, + "moderate", + "solo-saturation", + True, + ) + run_stage(state, "coloc-8-saturation", BACKGROUND_8, "saturation", None, False) + run_stage( + state, + "coloc-8-moderate", + BACKGROUND_8, + "moderate", + "coloc-8-saturation", + True, + ) + comparisons = [compare_regime("coloc-8")] + analysis = write_analysis(comparisons) + if not analysis["comparisons"][0]["pass"]: + run_stage( + state, "coloc-4-saturation", BACKGROUND_4, "saturation", None, False + ) + run_stage( + state, + "coloc-4-moderate", + BACKGROUND_4, + "moderate", + "coloc-4-saturation", + True, + ) + comparisons.append(compare_regime("coloc-4")) + analysis = write_analysis(comparisons) + state["status"] = "complete" if analysis["verdict"] != "STOP" else "gate_failed" + state["verdict"] = analysis["verdict"] + state["completed_at"] = time.time() + save_state(state) + print(json.dumps(analysis, sort_keys=True), flush=True) + + +def main() -> None: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + run = sub.add_parser("run") + run.add_argument("--resume", action="store_true") + sub.add_parser("status") + sub.add_parser("analyze") + sub.add_parser("plan") + args = parser.parse_args() + if args.command == "run": + execute(args.resume) + elif args.command == "status": + print(STATE.read_text() if STATE.exists() else '{"status":"absent"}') + elif args.command == "analyze": + comparisons = [compare_regime("coloc-8")] + if (RUN_ROOT / "coloc-4-moderate/stage-complete.json").exists(): + comparisons.append(compare_regime("coloc-4")) + print(json.dumps(write_analysis(comparisons), sort_keys=True, indent=2)) + else: + print( + json.dumps( + { + "target": "P05/C00 GPU0", + "background_8": BACKGROUND_8, + "background_4": BACKGROUND_4, + "cpu_map": CPU_MAP, + "stages": [ + "solo-saturation", + "solo-moderate", + "coloc-8-saturation", + "coloc-8-moderate", + "conditional coloc-4 saturation/moderate", + ], + }, + sort_keys=True, + indent=2, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/opprof_phase3_controller_ea.py b/runs/opprof-phase3/provenance/opprof_phase3_controller_ea.py new file mode 100644 index 0000000..b501bd2 --- /dev/null +++ b/runs/opprof-phase3/provenance/opprof_phase3_controller_ea.py @@ -0,0 +1,1045 @@ +#!/usr/bin/env python3 +"""Detached/resumable dash0 controller for the Phase-3 co-location gate.""" + +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import math +import os +import re +import shlex +import shutil +import signal +import subprocess +import sys +import threading +import time +import urllib.request +from collections import defaultdict +from pathlib import Path +from typing import Any + + +SCHEMA = 1 +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/e-a-colocation" +PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +SOURCE = Path( + "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0" +) +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +STATE = RUN_ROOT / "controller-state.json" +CPU_MAP = { + 0: "0-19", + 1: "20-39", + 2: "40-59", + 3: "60-79", + 4: "80-99", + 5: "100-119", + 6: "120-139", + 7: "140-159", +} +BACKGROUND_8 = { + 1: "P01", + 2: "P02", + 3: "P03", + 4: "P04", + 5: "P06", + 6: "P07", + 7: "P08", +} +BACKGROUND_4 = {1: "P01", 2: "P03", 3: "P06"} +PATTERNS = { + "P01": ["--kind", "synthetic", "--input-uniform", "128:512", + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P02": ["--kind", "synthetic", "--input-uniform", "128:512", + "--output-fixed", "512", "--prefix", "none", "--arrival", "steady"], + "P03": ["--kind", "synthetic", "--input-uniform", "4096:8192", + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P04": ["--kind", "synthetic", "--input-uniform", "4096:8192", + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P05": ["--kind", "synthetic", "--input-mixture", + '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + "--output-fixed", "64", "--prefix", "none", "--arrival", "steady"], + "P06": ["--kind", "synthetic", "--input-mixture", + '{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P07": ["--kind", "synthetic", "--input-fixed", "1280", + "--output-fixed", "512", "--prefix", "none", "--arrival", "burst:8"], + "P08": ["--kind", "prefix-pool", "--num-prefixes", "8", + "--prefix-len", "1024", "--suffix-fixed", "256", + "--output-fixed", "512", "--arrival", "burst:8"], +} +PROFILE_CONFIG = { + "profiler": "torch", + "torch_profiler_with_stack": True, + "torch_profiler_record_shapes": True, + "torch_profiler_use_gzip": True, + "ignore_frontend": True, + "wait_iterations": 0, + "warmup_iterations": 2, + "active_iterations": 8, +} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + with tmp.open("w", encoding="utf-8") as f: + json.dump(value, f, sort_keys=True, indent=2) + f.write("\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def run_text(command: list[str], check: bool = True) -> str: + result = subprocess.run( + command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {shlex.join(command)}\n" + f"{result.stdout}" + ) + return result.stdout + + +def command_log(path: Path, label: str, command: list[str], expected: str) -> None: + with path.open("a", encoding="utf-8") as f: + f.write( + f"GPU_COMMAND {label}: {shlex.join(command)} ; expected={expected}\n" + ) + + +def load_state(resume: bool) -> dict[str, Any]: + if STATE.exists(): + if not resume: + raise RuntimeError("state exists; use --resume") + return json.loads(STATE.read_text()) + return { + "schema": SCHEMA, + "status": "created", + "created_at": time.time(), + "controller_pid": os.getpid(), + "stages": {}, + "fingerprint": {}, + } + + +def save_state(state: dict[str, Any]) -> None: + state["updated_at"] = time.time() + state["controller_pid"] = os.getpid() + atomic_json(STATE, state) + + +def make_fingerprint() -> dict[str, Any]: + return { + "client_sha256": sha256_file(CLIENT), + "controller_sha256": sha256_file(Path(__file__)), + "source_commit": run_text( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD"] + ).strip(), + "model_path": str(MODEL), + "venv_python": str(VENV / "bin/python"), + # JSON object keys are strings. Normalize before the first write so a + # read/compare during --resume is byte-semantically stable. + "cpu_map": {str(gpu): cpus for gpu, cpus in CPU_MAP.items()}, + } + + +def ensure_provenance() -> None: + provenance = RUN_ROOT / "provenance" + provenance.mkdir(parents=True, exist_ok=True) + for source in (CLIENT, Path(__file__)): + dest = provenance / source.name + source_sha = sha256_file(source) + if dest.exists() and sha256_file(dest) != source_sha: + # Preserve the prior immutable copy and add the reviewed runtime + # repair as a content-addressed sibling. + dest = provenance / f"{source.stem}.{source_sha[:12]}{source.suffix}" + if dest.exists() and sha256_file(dest) != source_sha: + raise RuntimeError(f"content-addressed provenance changed: {dest}") + if not dest.exists(): + shutil.copy2(source, dest) + checksums = { + path.name: sha256_file(path) + for path in sorted(provenance.glob("*.py")) + } + atomic_json(provenance / "sha256.json", checksums) + + +def ensure_manifests() -> None: + PRIVATE.mkdir(parents=True, exist_ok=True, mode=0o700) + os.chmod(PRIVATE, 0o700) + for pattern, spec in PATTERNS.items(): + out = PRIVATE / f"{pattern}.jsonl" + if out.exists(): + summary = json.loads( + out.with_suffix(out.suffix + ".summary.json").read_text() + ) + if summary["rows"] != 32768 or summary["sha256"] != sha256_file(out): + raise RuntimeError(f"manifest verification failed: {out}") + continue + command = [ + str(VENV / "bin/python"), + str(CLIENT), + "materialize", + "--id", + pattern, + *spec, + "--num-requests", + "32768", + "--workload-seed", + "20260712", + "--out", + str(out), + ] + run_text(command) + + +def gpu_query() -> list[dict[str, Any]]: + text = run_text( + [ + "nvidia-smi", + "--query-gpu=index,uuid,memory.used,utilization.gpu", + "--format=csv,noheader,nounits", + ] + ) + rows = [] + for line in text.strip().splitlines(): + index, uuid, memory, util = [part.strip() for part in line.split(",")] + rows.append( + { + "index": int(index), + "uuid": uuid, + "memory_used_mib": int(memory), + "utilization_pct": int(util), + } + ) + return rows + + +def compute_apps() -> list[dict[str, Any]]: + text = run_text( + [ + "nvidia-smi", + "--query-compute-apps=gpu_uuid,pid,process_name,used_memory", + "--format=csv,noheader,nounits", + ], + check=False, + ) + rows = [] + for line in text.strip().splitlines(): + if not line or "No running" in line: + continue + parts = [part.strip() for part in line.split(",", 3)] + if len(parts) == 4 and parts[1].isdigit(): + rows.append( + { + "gpu_uuid": parts[0], + "pid": int(parts[1]), + "process_name": parts[2], + "used_memory_mib": int(parts[3].split()[0]), + } + ) + return rows + + +def preflight(gpus: list[int], stage_dir: Path) -> None: + snapshots = [row for row in gpu_query() if row["index"] in gpus] + apps = compute_apps() + atomic_json(stage_dir / "gpu-before.json", snapshots) + (stage_dir / "clocks-before.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"]) + ) + (stage_dir / "loadavg-before.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + (stage_dir / "processes-before.txt").write_text( + run_text(["ps", "-eo", "user,pid,ppid,pgid,lstart,args", "--sort=pid"]) + ) + if apps or any(row["memory_used_mib"] != 0 for row in snapshots): + raise RuntimeError(f"selected GPUs not idle: snapshots={snapshots} apps={apps}") + if any(row["utilization_pct"] != 0 for row in snapshots): + raise RuntimeError(f"selected GPUs have nonzero utilization: {snapshots}") + + +class Monitor: + def __init__(self, path: Path, owned_pgids: set[int]) -> None: + self.path = path + self.owned_pgids = owned_pgids + self.stop_event = threading.Event() + self.other_apps: list[dict[str, Any]] = [] + self.thread = threading.Thread(target=self._run, daemon=True) + + def start(self) -> None: + self.thread.start() + + def stop(self) -> None: + self.stop_event.set() + self.thread.join(timeout=20) + + def _run(self) -> None: + with self.path.open("a", encoding="utf-8") as f: + while not self.stop_event.is_set(): + apps = compute_apps() + samples = [] + for app in apps: + try: + pgid = os.getpgid(app["pid"]) + except ProcessLookupError: + pgid = None + sample = {**app, "pgid": pgid} + samples.append(sample) + if pgid not in self.owned_pgids: + self.other_apps.append(sample) + clocks = run_text( + [ + "nvidia-smi", + "--query-gpu=index,clocks.sm,clocks.mem,pstate," + "utilization.gpu,memory.used", + "--format=csv,noheader,nounits", + ], + check=False, + ).strip().splitlines() + row = { + "wall_time": time.time(), + "loadavg": os.getloadavg(), + "gpu_clocks": clocks, + "compute_apps": samples, + } + f.write(json.dumps(row, sort_keys=True) + "\n") + f.flush() + self.stop_event.wait(15) + + +def server_command(gpu: int, stage_dir: Path, trace_dir: Path) -> list[str]: + config = {**PROFILE_CONFIG, "torch_profiler_dir": str(trace_dir)} + return [ + "taskset", + "-c", + CPU_MAP[gpu], + str(VENV / "bin/vllm"), + "serve", + str(MODEL), + "--host", + "127.0.0.1", + "--port", + str(8000 + gpu), + "--tensor-parallel-size", + "1", + "--enable-chunked-prefill", + "--enable-prefix-caching", + "--profiler-config", + json.dumps(config, separators=(",", ":")), + ] + + +def start_server( + gpu: int, stage_dir: Path, state_stage: dict[str, Any] +) -> tuple[subprocess.Popen[Any], Any, Path]: + gpu_dir = stage_dir / f"gpu{gpu}" + gpu_dir.mkdir(parents=True, exist_ok=True) + trace_dir = Path(f"/tmp/wjh-opprof-phase3-ea/{stage_dir.name}/gpu{gpu}") + if trace_dir.exists(): + shutil.rmtree(trace_dir) + trace_dir.mkdir(parents=True) + command = server_command(gpu, stage_dir, trace_dir) + command_log( + stage_dir / "commands.log", + f"{stage_dir.name} server gpu{gpu}", + command, + "startup 60-180s; run 300-450s", + ) + env = os.environ.copy() + env.update( + { + "CUDA_VISIBLE_DEVICES": str(gpu), + "VLLM_OPPROF_DIR": str(gpu_dir / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + log_handle = (gpu_dir / "server.log").open("ab", buffering=0) + process = subprocess.Popen( + command, + cwd=SOURCE, + env=env, + stdout=log_handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + state_stage.setdefault("servers", {})[str(gpu)] = { + "pid": process.pid, + "pgid": process.pid, + "command": command, + "trace_dir": str(trace_dir), + } + return process, log_handle, trace_dir + + +def wait_ready(processes: dict[int, subprocess.Popen[Any]], timeout: float = 300) -> None: + deadline = time.monotonic() + timeout + pending = set(processes) + while pending and time.monotonic() < deadline: + for gpu in list(pending): + if processes[gpu].poll() is not None: + raise RuntimeError(f"server gpu{gpu} exited before ready") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{8000 + gpu}/health", timeout=1 + ) as response: + if response.status == 200: + pending.remove(gpu) + except Exception: + pass + time.sleep(1) + if pending: + raise TimeoutError(f"servers not ready within {timeout}s: {sorted(pending)}") + + +def client_command( + gpu: int, + pattern: str, + stage_dir: Path, + load_point: str, + saturation_result: Path | None, + profile: bool, + trace_dir: Path, + post_clean_seconds: int, +) -> list[str]: + result_dir = stage_dir / f"gpu{gpu}" / "client" + command = [ + "taskset", + "-c", + CPU_MAP[gpu], + str(VENV / "bin/python"), + str(CLIENT), + "run", + "--manifest", + str(PRIVATE / f"{pattern}.jsonl"), + "--base-url", + f"http://127.0.0.1:{8000 + gpu}", + "--model", + str(MODEL), + "--load-point", + load_point, + "--max-concurrency", + "256", + "--ignore-eos", + "--temperature", + "0", + "--warmup-seconds", + "60", + "--clean-segment-seconds", + "80", + "--num-clean-segments", + "3", + "--recovery-seconds", + "30", + "--drain-timeout-seconds", + "120", + "--workload-seed", + "20260712", + "--result-dir", + str(result_dir), + ] + if load_point == "saturation": + command += ["--request-rate", "inf"] + else: + assert saturation_result is not None + command += [ + "--saturation-result", + str(saturation_result), + "--rate-fraction", + "0.60", + ] + if profile: + command += [ + "--profile-after-clean", + "--num-profile-windows", + "1", + "--profile-warmup-iterations", + "2", + "--profile-active-iterations", + "8", + "--profile-trace-dir", + str(trace_dir), + "--profile-timeout-seconds", + "120", + ] + elif post_clean_seconds: + command += ["--post-clean-seconds", str(post_clean_seconds)] + return command + + +def stop_processes(processes: list[subprocess.Popen[Any]]) -> None: + live = [process for process in processes if process.poll() is None] + for process in live: + try: + os.killpg(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 60 + while time.monotonic() < deadline and any(p.poll() is None for p in live): + time.sleep(1) + for sig, wait_seconds in ((signal.SIGTERM, 20), (signal.SIGKILL, 5)): + remaining = [p for p in live if p.poll() is None] + if not remaining: + break + for process in remaining: + try: + os.killpg(process.pid, sig) + except ProcessLookupError: + pass + deadline = time.monotonic() + wait_seconds + while time.monotonic() < deadline and any( + p.poll() is None for p in remaining + ): + time.sleep(0.5) + + +def _process_group_alive(pgid: int) -> bool: + try: + os.killpg(pgid, 0) + return True + except ProcessLookupError: + return False + + +def stop_servers(processes: list[subprocess.Popen[Any]]) -> None: + """Let the API parent ask EngineCore to close before group escalation.""" + groups = [process.pid for process in processes] + for process in processes: + if process.poll() is None: + try: + # Group SIGINT also interrupts EngineCore and loses the OpProf + # footer. The API parent's shutdown path signals it cleanly. + os.kill(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 90 + while time.monotonic() < deadline and any( + _process_group_alive(pgid) for pgid in groups + ): + time.sleep(1) + for sig, wait_seconds in ((signal.SIGTERM, 20), (signal.SIGKILL, 5)): + remaining = [pgid for pgid in groups if _process_group_alive(pgid)] + if not remaining: + break + for pgid in remaining: + try: + os.killpg(pgid, sig) + except ProcessLookupError: + pass + deadline = time.monotonic() + wait_seconds + while time.monotonic() < deadline and any( + _process_group_alive(pgid) for pgid in remaining + ): + time.sleep(0.5) + + +def verify_idle(gpus: list[int], stage_dir: Path) -> None: + samples = [] + consecutive_zero = 0 + deadline = time.monotonic() + 120 + while time.monotonic() < deadline and consecutive_zero < 3: + rows = [row for row in gpu_query() if row["index"] in gpus] + apps = compute_apps() + samples.append({"gpus": rows, "compute_apps": apps, "time": time.time()}) + zero = not apps and all(row["memory_used_mib"] == 0 for row in rows) + consecutive_zero = consecutive_zero + 1 if zero else 0 + if consecutive_zero < 3: + time.sleep(5) + atomic_json(stage_dir / "gpu-after-cleanup.json", samples) + if consecutive_zero < 3: + raise RuntimeError("GPU memory did not reach three stable zero samples") + + +def run_stage( + state: dict[str, Any], + name: str, + backgrounds: dict[int, str], + load_point: str, + saturation_stage: str | None, + profile: bool, +) -> None: + stage_dir = RUN_ROOT / name + stage_record = state["stages"].get(name) + if ( + stage_record + and stage_record.get("status") == "complete" + and (stage_dir / "stage-complete.json").exists() + ): + print(f"RESUME skip complete stage {name}", flush=True) + return + if stage_dir.exists(): + shutil.rmtree(stage_dir) + stage_dir.mkdir(parents=True) + gpus = [0, *sorted(backgrounds)] + state["stages"][name] = { + "status": "preflight", + "started_at": time.time(), + "gpus": gpus, + "backgrounds": backgrounds, + "load_point": load_point, + "profile": profile, + } + save_state(state) + preflight(gpus, stage_dir) + servers: dict[int, subprocess.Popen[Any]] = {} + server_logs = [] + client_processes: dict[int, subprocess.Popen[Any]] = {} + client_logs = [] + trace_dirs: dict[int, Path] = {} + monitor: Monitor | None = None + failure: Exception | None = None + try: + for gpu in gpus: + process, handle, trace_dir = start_server( + gpu, stage_dir, state["stages"][name] + ) + servers[gpu] = process + server_logs.append(handle) + trace_dirs[gpu] = trace_dir + state["stages"][name]["status"] = "starting_servers" + save_state(state) + wait_ready(servers) + state["stages"][name]["servers_ready_at"] = time.time() + state["stages"][name]["status"] = "running_clients" + save_state(state) + monitor = Monitor( + stage_dir / "monitor.jsonl", {process.pid for process in servers.values()} + ) + monitor.start() + + assignments = {0: "P05", **backgrounds} + for gpu, pattern in assignments.items(): + is_target = gpu == 0 + sat_result = ( + RUN_ROOT / saturation_stage / "gpu0/client/result.json" + if is_target and saturation_stage + else None + ) + target_load = load_point if is_target else "saturation" + command = client_command( + gpu, + pattern, + stage_dir, + target_load, + sat_result, + profile=is_target and profile, + trace_dir=trace_dirs[gpu], + post_clean_seconds=60 if not is_target and profile else 0, + ) + command_log( + stage_dir / "commands.log", + f"{name} client gpu{gpu} {pattern}", + command, + "60s warmup + 240s clean + optional 2+8 profile/recovery", + ) + handle = (stage_dir / f"gpu{gpu}/client.log").open("ab", buffering=0) + client_logs.append(handle) + process = subprocess.Popen( + command, + cwd=WORKDIR, + stdout=handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + client_processes[gpu] = process + state["stages"][name].setdefault("clients", {})[str(gpu)] = { + "pid": process.pid, + "pgid": process.pid, + "command": command, + } + save_state(state) + + deadline = time.monotonic() + 780 + while time.monotonic() < deadline and any( + process.poll() is None for process in client_processes.values() + ): + if any(process.poll() is not None for process in servers.values()): + raise RuntimeError("server exited while client was active") + time.sleep(2) + if any(process.poll() is None for process in client_processes.values()): + raise TimeoutError("client stage exceeded 780 seconds") + bad = { + gpu: process.returncode + for gpu, process in client_processes.items() + if process.returncode != 0 + } + if bad: + raise RuntimeError(f"client failures: {bad}") + target_result = stage_dir / "gpu0/client/result.json" + if not target_result.exists(): + raise RuntimeError("target result.json missing") + if profile: + trace_dest = stage_dir / "gpu0/traces" + trace_dest.mkdir() + for path in trace_dirs[0].glob("*"): + if path.is_file(): + shutil.copy2(path, trace_dest / path.name) + traces = list(trace_dest.glob("*.pt.trace.json*")) + if len(traces) != 1: + raise RuntimeError(f"expected one target trace, found {len(traces)}") + state["stages"][name]["status"] = "clients_complete" + save_state(state) + except Exception as exc: + failure = exc + finally: + if monitor is not None: + monitor.stop() + stop_processes(list(client_processes.values())) + stop_servers(list(servers.values())) + for handle in client_logs + server_logs: + handle.close() + (stage_dir / "clocks-after.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"], check=False) + ) + (stage_dir / "loadavg-after.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + (stage_dir / "processes-after.txt").write_text( + run_text( + ["ps", "-eo", "user,pid,ppid,pgid,lstart,args", "--sort=pid"], + check=False, + ) + ) + if monitor is not None: + atomic_json(stage_dir / "other-gpu-processes.json", monitor.other_apps) + try: + verify_idle(gpus, stage_dir) + except Exception as exc: + failure = failure or exc + + if failure is not None: + state["stages"][name]["status"] = "failed" + state["stages"][name]["failure"] = repr(failure) + save_state(state) + raise failure + marker = { + "schema": SCHEMA, + "stage": name, + "completed_at": time.time(), + "target_result_sha256": sha256_file( + stage_dir / "gpu0/client/result.json" + ), + } + atomic_json(stage_dir / "stage-complete.json", marker) + state["stages"][name].update( + {"status": "complete", "completed_at": time.time(), "servers": {}, + "clients": {}} + ) + save_state(state) + + +def classify_kernel(name: str) -> str: + value = name.lower() + if re.search(r"topkgating|moe_align|select_expert|moe.*rout", value): + return "moe_router" + if re.search( + r"nccl|all_?reduce|reduce_scatter|all_?gather|custom.*all.*reduce", value + ): + return "collective" + if re.search( + r"flash|fmha|paged_attention|unified_attention|attention|" + r"reshape_and_cache", value + ): + return "attention" + if re.search(r"fused_moe|moe.*gemm|nvjet", value): + return "moe_gemm" + if re.search( + r"sampling|(? dict[str, Any]: + traces = list((RUN_ROOT / stage / "gpu0/traces").glob("*.pt.trace.json*")) + if len(traces) != 1: + raise RuntimeError(f"{stage}: expected one trace") + trace = traces[0] + opener = gzip.open if trace.suffix == ".gz" else open + with opener(trace, "rt", encoding="utf-8") as f: + data = json.load(f) + durations: defaultdict[str, float] = defaultdict(float) + kernel_count = 0 + steps = set() + executes = 0 + for event in data.get("traceEvents", []): + name = str(event.get("name", "")) + match = re.fullmatch(r"ProfilerStep#(\d+)", name) + if match: + steps.add(int(match.group(1))) + if name.startswith("execute_") and event.get("cat") == "user_annotation": + executes += 1 + if event.get("cat") == "kernel" and float(event.get("dur", 0)) >= 0: + kernel_count += 1 + durations[classify_kernel(name)] += float(event.get("dur", 0)) + total = sum(durations.values()) + shares = { + family: duration / total for family, duration in sorted(durations.items()) + } + result = { + "schema": SCHEMA, + "stage": stage, + "trace_file": trace.name, + "trace_sha256": sha256_file(trace), + "trace_loadable": True, + "kernel_count": kernel_count, + "profiler_steps": sorted(steps), + "execute_annotations": executes, + "duration_us": dict(sorted(durations.items())), + "shares": shares, + "classifiable_fraction": 1.0 - shares.get("other", 0.0), + "valid": ( + kernel_count > 0 + and steps == set(range(2, 10)) + and executes == 8 + and 1.0 - shares.get("other", 0.0) >= 0.70 + ), + } + return result + + +def compare_regime(prefix: str) -> dict[str, Any]: + solo_result = json.loads( + (RUN_ROOT / "solo-moderate/gpu0/client/result.json").read_text() + ) + coloc_result = json.loads( + (RUN_ROOT / f"{prefix}-moderate/gpu0/client/result.json").read_text() + ) + solo_trace = trace_analysis("solo-moderate") + coloc_trace = trace_analysis(f"{prefix}-moderate") + solo_t = float(solo_result["clean"]["completed_throughput_rps"]) + coloc_t = float(coloc_result["clean"]["completed_throughput_rps"]) + top3 = sorted( + solo_trace["shares"], key=solo_trace["shares"].get, reverse=True + )[:3] + share_deltas = { + family: abs( + coloc_trace["shares"].get(family, 0) + - solo_trace["shares"].get(family, 0) + ) + for family in top3 + } + throughput_delta = abs(coloc_t / solo_t - 1) + return { + "schema": SCHEMA, + "regime": prefix, + "solo_throughput_rps": solo_t, + "colocated_throughput_rps": coloc_t, + "throughput_delta_fraction": throughput_delta, + "top3_solo_families": top3, + "solo_operator_shares": { + family: solo_trace["shares"][family] for family in top3 + }, + "colocated_operator_shares": { + family: coloc_trace["shares"].get(family, 0) for family in top3 + }, + "operator_share_delta_fraction": share_deltas, + "solo_trace": solo_trace, + "colocated_trace": coloc_trace, + "pass": ( + solo_trace["valid"] + and coloc_trace["valid"] + and throughput_delta < 0.03 + and all(delta < 0.03 for delta in share_deltas.values()) + ), + } + + +def numeric_sanity(values: list[float]) -> dict[str, Any]: + return { + "n": len(values), + "finite_n": sum(math.isfinite(value) for value in values), + "missing_n": sum(not math.isfinite(value) for value in values), + "min": min(values) if values else None, + "max": max(values) if values else None, + "distinct_n": len(set(values)), + } + + +def write_analysis(comparisons: list[dict[str, Any]]) -> dict[str, Any]: + accepted = next((c["regime"] for c in comparisons if c["pass"]), None) + throughputs = [ + value + for c in comparisons + for value in (c["solo_throughput_rps"], c["colocated_throughput_rps"]) + ] + deltas = [ + delta + for c in comparisons + for delta in c["operator_share_delta_fraction"].values() + ] + result = { + "schema": SCHEMA, + "comparisons": comparisons, + "verdict": ( + f"{accepted.replace('coloc-', '')}-way" + if accepted is not None + else "STOP" + ), + "sanity": { + "throughput_rps": numeric_sanity(throughputs), + "throughput_delta": numeric_sanity( + [c["throughput_delta_fraction"] for c in comparisons] + ), + "operator_share_delta": numeric_sanity(deltas), + "invariants": { + "throughputs_positive": all(x > 0 for x in throughputs), + "share_deltas_in_0_1": all(0 <= x <= 1 for x in deltas), + "top3_exact": all( + len(c["top3_solo_families"]) == 3 for c in comparisons + ), + "traces_valid": all( + c["solo_trace"]["valid"] and c["colocated_trace"]["valid"] + for c in comparisons + ), + }, + }, + } + atomic_json(RUN_ROOT / "colocation-analysis.json", result) + return result + + +def clean_recorded_processes(state: dict[str, Any]) -> None: + for stage in state.get("stages", {}).values(): + if stage.get("status") == "complete": + continue + pgids = [ + entry.get("pgid") + for group in ("clients", "servers") + for entry in stage.get(group, {}).values() + if entry.get("pgid") + ] + for pgid in pgids: + try: + os.killpg(int(pgid), signal.SIGINT) + except ProcessLookupError: + pass + if pgids: + time.sleep(5) + + +def execute(resume: bool) -> None: + RUN_ROOT.mkdir(parents=True, exist_ok=True) + state = load_state(resume) + if resume: + clean_recorded_processes(state) + fingerprint = make_fingerprint() + if state["fingerprint"] and state["fingerprint"] != fingerprint: + raise RuntimeError("resume fingerprint differs from frozen execution") + state["fingerprint"] = fingerprint + state["status"] = "running" + save_state(state) + ensure_provenance() + ensure_manifests() + run_stage(state, "solo-saturation", {}, "saturation", None, False) + run_stage( + state, + "solo-moderate", + {}, + "moderate", + "solo-saturation", + True, + ) + run_stage(state, "coloc-8-saturation", BACKGROUND_8, "saturation", None, False) + run_stage( + state, + "coloc-8-moderate", + BACKGROUND_8, + "moderate", + "coloc-8-saturation", + True, + ) + comparisons = [compare_regime("coloc-8")] + analysis = write_analysis(comparisons) + if not analysis["comparisons"][0]["pass"]: + run_stage( + state, "coloc-4-saturation", BACKGROUND_4, "saturation", None, False + ) + run_stage( + state, + "coloc-4-moderate", + BACKGROUND_4, + "moderate", + "coloc-4-saturation", + True, + ) + comparisons.append(compare_regime("coloc-4")) + analysis = write_analysis(comparisons) + state["status"] = "complete" if analysis["verdict"] != "STOP" else "gate_failed" + state["verdict"] = analysis["verdict"] + state["completed_at"] = time.time() + save_state(state) + print(json.dumps(analysis, sort_keys=True), flush=True) + + +def main() -> None: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + run = sub.add_parser("run") + run.add_argument("--resume", action="store_true") + sub.add_parser("status") + sub.add_parser("analyze") + sub.add_parser("plan") + args = parser.parse_args() + if args.command == "run": + execute(args.resume) + elif args.command == "status": + print(STATE.read_text() if STATE.exists() else '{"status":"absent"}') + elif args.command == "analyze": + comparisons = [compare_regime("coloc-8")] + if (RUN_ROOT / "coloc-4-moderate/stage-complete.json").exists(): + comparisons.append(compare_regime("coloc-4")) + print(json.dumps(write_analysis(comparisons), sort_keys=True, indent=2)) + else: + print( + json.dumps( + { + "target": "P05/C00 GPU0", + "background_8": BACKGROUND_8, + "background_4": BACKGROUND_4, + "cpu_map": CPU_MAP, + "stages": [ + "solo-saturation", + "solo-moderate", + "coloc-8-saturation", + "coloc-8-moderate", + "conditional coloc-4 saturation/moderate", + ], + }, + sort_keys=True, + indent=2, + ) + ) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/opprof_phase3_matrix.py b/runs/opprof-phase3/provenance/opprof_phase3_matrix.py new file mode 100644 index 0000000..630b0c4 --- /dev/null +++ b/runs/opprof-phase3/provenance/opprof_phase3_matrix.py @@ -0,0 +1,1252 @@ +#!/usr/bin/env python3 +"""Detached, resumable four-GPU controller for the Phase-3 matrix.""" + +from __future__ import annotations + +import argparse +import gzip +import hashlib +import json +import math +import os +import re +import shlex +import shutil +import signal +import subprocess +import time +import urllib.request +from dataclasses import dataclass +from pathlib import Path +from typing import Any + +import opprof_phase3_controller as common + +SCHEMA = 1 +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/phase3" +PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +SOURCE = Path( + "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0" +) +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +STATE = RUN_ROOT / "controller-state.json" +GPU_LIMIT = 4 +GPU_HOUR_LIMIT = 16.0 +PRIOR_GPU_HOURS = 3.964 + 427.43714332580566 / 3600 +CPU_MAP = common.CPU_MAP +PROFILE_CONFIG = { + "profiler": "torch", + "torch_profiler_with_stack": True, + "torch_profiler_record_shapes": True, + "torch_profiler_use_gzip": True, + "ignore_frontend": True, + "wait_iterations": 0, + "warmup_iterations": 2, + "active_iterations": 8, +} +CONFIGS = { + "C00": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []}, + "C10": { + "tp": 1, + "mns": 64, + "mbt": 8192, + "flags": ["--max-num-seqs", "64"], + }, + "C01": { + "tp": 1, + "mns": 1024, + "mbt": 2048, + "flags": ["--max-num-batched-tokens", "2048"], + }, + "C11": { + "tp": 1, + "mns": 64, + "mbt": 2048, + "flags": [ + "--max-num-seqs", + "64", + "--max-num-batched-tokens", + "2048", + ], + }, + "C00-TP2": {"tp": 2, "mns": 1024, "mbt": 8192, "flags": []}, +} +LONG_BURST = {"P04", "P06", "P07", "P08", "P11"} +SENTINELS = ("P01", "P03", "P06", "P10") +PATTERNS = tuple(f"P{index:02d}" for index in range(1, 12)) + + +@dataclass(frozen=True) +class Cell: + pattern: str + config: str + + @property + def cell_id(self) -> str: + return f"{self.pattern}-{self.config}" + + @property + def width(self) -> int: + return int(CONFIGS[self.config]["tp"]) + + +@dataclass(frozen=True) +class Assignment: + cell: Cell + gpus: tuple[int, ...] + + +def drain_budget(pattern: str) -> int: + if pattern == "P10": + return 600 + if pattern in LONG_BURST: + return 240 + return 120 + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def atomic_json(path: Path, value: Any) -> None: + common.atomic_json(path, value) + + +def run_text(command: list[str], check: bool = True) -> str: + return common.run_text(command, check=check) + + +def numeric_sanity(values: list[float | int]) -> dict[str, Any]: + finite = [float(value) for value in values if math.isfinite(float(value))] + return { + "n": len(values), + "finite_n": len(finite), + "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, + "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def cells() -> list[Cell]: + result = [Cell(pattern, "C00") for pattern in PATTERNS] + for pattern in SENTINELS: + result.extend(Cell(pattern, config) for config in ("C10", "C01", "C11")) + result.append(Cell("P10", "C00-TP2")) + return sorted( + result, + key=lambda cell: hashlib.sha256( + f"20260713:{cell.pattern}:{cell.config}".encode() + ).hexdigest(), + ) + + +def pack_cells(items: list[Cell]) -> list[list[Assignment]]: + waves: list[list[Assignment]] = [] + current: list[Assignment] = [] + slot = 0 + for cell in items: + if slot + cell.width > GPU_LIMIT: + waves.append(current) + current = [] + slot = 0 + current.append(Assignment(cell, tuple(range(slot, slot + cell.width)))) + slot += cell.width + if current: + waves.append(current) + assert sum(len(wave) for wave in waves) == len(items) + assert all(sum(item.cell.width for item in wave) <= GPU_LIMIT for wave in waves) + return waves + + +def load_state(resume: bool) -> dict[str, Any]: + if STATE.exists(): + if not resume: + raise RuntimeError("controller state exists; use --resume") + return json.loads(STATE.read_text()) + return { + "schema": SCHEMA, + "status": "created", + "created_at": time.time(), + "controller_pid": os.getpid(), + "gpu_hours_total": PRIOR_GPU_HOURS, + "gpu_hours_this_stage": 0.0, + "completed_measured_runs": 0, + "completed_burnins": 0, + "drain_quarantined_runs": 0, + "clean_window_failures": 0, + "missing_trace_files": 0, + "stages": {}, + "fingerprint": {}, + } + + +def save_state(state: dict[str, Any]) -> None: + state["controller_pid"] = os.getpid() + state["updated_at"] = time.time() + atomic_json(STATE, state) + + +def manifest_fingerprint() -> dict[str, Any]: + result = {} + for pattern in PATTERNS: + path = PRIVATE / f"{pattern}.jsonl" + summary_path = path.with_suffix(path.suffix + ".summary.json") + if not path.exists() or not summary_path.exists(): + raise RuntimeError(f"manifest missing: {path}") + summary = json.loads(summary_path.read_text()) + expected_rows = 4011 if pattern == "P10" else 32768 + if summary["rows"] != expected_rows or summary["sha256"] != sha256_file(path): + raise RuntimeError(f"manifest verification failed: {pattern}") + result[pattern] = { + "path": str(path), + "rows": summary["rows"], + "sha256": summary["sha256"], + } + return result + + +def fingerprint() -> dict[str, Any]: + return { + "source_commit": run_text( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD"] + ).strip(), + "source_tree": run_text( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD^{tree}"] + ).strip(), + "client_sha256": sha256_file(CLIENT), + "controller_sha256": sha256_file(Path(__file__)), + "common_controller_sha256": sha256_file( + Path(common.__file__).resolve() + ), + "model": str(MODEL), + "cpu_map": {str(key): value for key, value in CPU_MAP.items()}, + "manifests": manifest_fingerprint(), + "cells": [cell.cell_id for cell in cells()], + } + + +def ensure_provenance() -> None: + destination = RUN_ROOT / "provenance" + destination.mkdir(parents=True, exist_ok=True) + sources = [CLIENT, Path(__file__).resolve(), Path(common.__file__).resolve()] + checksums = {} + for source in sources: + target = destination / source.name + digest = sha256_file(source) + if target.exists() and sha256_file(target) != digest: + target = destination / f"{source.stem}.{digest[:12]}{source.suffix}" + if target.exists() and sha256_file(target) != digest: + raise RuntimeError(f"content-addressed provenance mismatch: {target}") + if not target.exists(): + shutil.copy2(source, target) + checksums[target.name] = digest + patch_dir = WORKDIR / "provenance/patches-vllm-0.24.0-opprof" + checksums["patches"] = { + path.name: sha256_file(path) for path in sorted(patch_dir.glob("0*.patch")) + } + atomic_json(destination / "sha256.json", checksums) + + +def preflight(stage_dir: Path, gpus: list[int]) -> None: + stage_dir.mkdir(parents=True, exist_ok=True) + common.preflight(gpus, stage_dir) + free_kb = shutil.disk_usage(RUN_ROOT.parent).free + if free_kb < 100 * (1 << 30): + raise RuntimeError("CPFS free space below 100 GiB") + + +def cpu_mask(gpus: tuple[int, ...]) -> str: + ranges = [CPU_MAP[gpu] for gpu in gpus] + return ",".join(ranges) + + +def server_command( + assignment: Assignment, port: int, trace_dir: Path +) -> list[str]: + config = { + **PROFILE_CONFIG, + "torch_profiler_dir": str(trace_dir), + } + details = CONFIGS[assignment.cell.config] + return [ + "taskset", + "-c", + cpu_mask(assignment.gpus), + str(VENV / "bin/vllm"), + "serve", + str(MODEL), + "--host", + "127.0.0.1", + "--port", + str(port), + "--tensor-parallel-size", + str(details["tp"]), + "--enable-chunked-prefill", + "--enable-prefix-caching", + "--shutdown-timeout", + "120", + "--profiler-config", + json.dumps(config, separators=(",", ":")), + *details["flags"], + ] + + +def client_command( + assignment: Assignment, + port: int, + run_dir: Path, + load_point: str, + profile: bool, + burnin: bool, + saturation_result: Path | None, +) -> list[str]: + if burnin: + warmup, segment, segments = 0, 20, 3 + else: + warmup, segment, segments = 60, 80, 3 + drain = drain_budget(assignment.cell.pattern) + command = [ + "taskset", + "-c", + cpu_mask(assignment.gpus), + str(VENV / "bin/python"), + str(CLIENT), + "run", + "--manifest", + str(PRIVATE / f"{assignment.cell.pattern}.jsonl"), + "--base-url", + f"http://127.0.0.1:{port}", + "--model", + str(MODEL), + "--load-point", + load_point, + "--max-concurrency", + "256", + "--ignore-eos", + "--temperature", + "0", + "--warmup-seconds", + str(warmup), + "--clean-segment-seconds", + str(segment), + "--num-clean-segments", + str(segments), + "--recovery-seconds", + "30", + "--drain-timeout-seconds", + str(drain), + "--workload-seed", + "20260712", + "--server-seed", + "20260712", + "--result-dir", + str(run_dir / "client"), + ] + if load_point == "saturation": + command.extend(("--request-rate", "inf")) + else: + if saturation_result is None: + raise RuntimeError("moderate run lacks saturation result") + command.extend( + ( + "--saturation-result", + str(saturation_result), + "--rate-fraction", + "0.60", + ) + ) + if profile: + command.extend( + ( + "--profile-after-clean", + "--num-profile-windows", + "2", + "--profile-warmup-iterations", + "2", + "--profile-active-iterations", + "8", + "--profile-trace-dir", + str(run_dir / "trace-tmp"), + "--profile-timeout-seconds", + "120", + ) + ) + return command + + +def wait_ready(entries: list[dict[str, Any]]) -> None: + pending = {entry["port"]: entry for entry in entries} + deadline = time.monotonic() + 300 + while pending and time.monotonic() < deadline: + for port, entry in list(pending.items()): + if entry["server"].poll() is not None: + raise RuntimeError(f"server exited before ready: {entry['run_id']}") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{port}/health", timeout=1 + ) as response: + if response.status == 200: + del pending[port] + except Exception: + pass + time.sleep(1) + if pending: + raise TimeoutError(f"server readiness timeout: {sorted(pending)}") + + +def stop_servers(entries: list[dict[str, Any]]) -> None: + live = [entry for entry in entries if entry["server"].poll() is None] + for entry in live: + try: + os.kill(entry["server"].pid, signal.SIGINT) + except ProcessLookupError: + pass + deadline = time.monotonic() + 150 + while time.monotonic() < deadline and any( + entry["server"].poll() is None for entry in live + ): + time.sleep(1) + remaining = [entry for entry in live if entry["server"].poll() is None] + for signum in (signal.SIGTERM, signal.SIGKILL): + if not remaining: + break + for entry in remaining: + try: + os.killpg(entry["server"].pid, signum) + except ProcessLookupError: + pass + time.sleep(5) + remaining = [entry for entry in remaining if entry["server"].poll() is None] + if remaining: + raise RuntimeError("server process groups survived SIGKILL") + + +def trace_summary(path: Path) -> dict[str, Any]: + opener = gzip.open if path.suffix == ".gz" else open + with opener(path, "rt", encoding="utf-8") as source: + payload = json.load(source) + durations: dict[str, float] = {} + kernel_count = 0 + steps = set() + executes = 0 + for event in payload.get("traceEvents", []): + name = str(event.get("name", "")) + if event.get("cat") == "user_annotation": + match = re.fullmatch(r"ProfilerStep#(\d+)", name) + if match: + steps.add(int(match.group(1))) + if name.startswith("execute_"): + executes += 1 + if event.get("cat") == "kernel": + duration = float(event.get("dur", 0)) + if duration < 0: + raise RuntimeError(f"negative kernel duration in {path}") + family = common.classify_kernel(name) + durations[family] = durations.get(family, 0.0) + duration + kernel_count += 1 + total = sum(durations.values()) + shares = { + family: duration / total for family, duration in sorted(durations.items()) + } + classifiable = 1.0 - shares.get("other", 0.0) + valid = ( + kernel_count > 0 + and steps == set(range(2, 10)) + and executes == 8 + and classifiable >= 0.70 + ) + return { + "path": str(path), + "sha256": sha256_file(path), + "bytes": path.stat().st_size, + "kernel_count": kernel_count, + "profiler_steps": sorted(steps), + "execute_annotations": executes, + "duration_us": durations, + "shares": shares, + "classifiable_fraction": classifiable, + "valid": valid, + } + + +def validate_layer1(run_dir: Path) -> dict[str, Any]: + streams = sorted((run_dir / "opprof").glob("*.jsonl")) + sidecars = sorted((run_dir / "opprof").glob("*.jsonl.footer.json")) + if len(streams) != 1 or len(sidecars) != 1: + raise RuntimeError( + f"{run_dir}: expected one Layer-1 stream/sidecar, " + f"got {len(streams)}/{len(sidecars)}" + ) + raw = streams[0].read_bytes() + if not raw.endswith(b"\n"): + raise RuntimeError(f"partial Layer-1 line: {streams[0]}") + decoded = [json.loads(line) for line in raw.splitlines()] + if not decoded or decoded[-1].get("record_type") != "footer": + raise RuntimeError(f"clean-close footer missing: {streams[0]}") + records, footer = decoded[:-1], decoded[-1] + sidecar = json.loads(sidecars[0].read_text()) + indices = [int(record["step_index"]) for record in records] + invariants = { + "schema_1": all(item.get("schema") == 1 for item in decoded) + and sidecar.get("schema") == 1, + "steps_unique_contiguous": sorted(indices) == list(range(len(indices))), + "footer_written_matches": footer["written_records"] == len(records), + "footer_balanced": footer["encoded_records"] + == footer["written_records"] + footer["dropped_records"], + "zero_drops": footer["dropped_records"] == 0 + and all(record["dropped_records_before"] == 0 for record in records), + "sidecar_final": sidecar.get("final") is True, + "sidecar_agrees": all( + sidecar[counter] == footer[counter] + for counter in ("encoded_records", "written_records", "dropped_records") + ), + "token_composition": all( + record["prefill_tokens"] + record["decode_tokens"] + >= 0 + for record in records + ), + "cudagraph_identity": all( + record["cudagraph"]["bucket_tokens"] + == record["cudagraph"]["unpadded_tokens"] + + record["cudagraph"]["padding_tokens"] + for record in records + ), + } + if not all(invariants.values()): + raise RuntimeError(f"Layer-1 invariant failure: {run_dir}: {invariants}") + return { + "stream": str(streams[0]), + "stream_sha256": sha256_file(streams[0]), + "sidecar": str(sidecars[0]), + "records": len(records), + "last_step_index": max(indices), + "footer": footer, + "invariants": invariants, + "numeric": { + "prefill_tokens": numeric_sanity( + [record["prefill_tokens"] for record in records] + ), + "decode_tokens": numeric_sanity( + [record["decode_tokens"] for record in records] + ), + "scheduled_requests": numeric_sanity( + [record["scheduled_requests"] for record in records] + ), + "padding_tokens": numeric_sanity( + [record["cudagraph"]["padding_tokens"] for record in records] + ), + }, + } + + +def summarize_request_failures( + requests: list[dict[str, Any]], clean_start: float, clean_end: float +) -> dict[str, Any]: + failed = [request for request in requests if not request["success"]] + clean_failed = [ + request + for request in failed + if clean_start <= float(request["completed_s"]) < clean_end + ] + excluded_kinds: dict[str, int] = {} + for request in failed: + if clean_start <= float(request["completed_s"]) < clean_end: + continue + key = str(request.get("error_kind") or "unknown") + excluded_kinds[key] = excluded_kinds.get(key, 0) + 1 + return { + "failed": len(failed), + "clean_failed": len(clean_failed), + "excluded": len(failed) - len(clean_failed), + "excluded_kinds": excluded_kinds, + } + + +def p10_warmup_stability(run_dir: Path, t0_mono_ns: int) -> dict[str, Any]: + streams = sorted((run_dir / "opprof").glob("*.jsonl")) + if len(streams) != 1: + return { + "passed": False, + "reason": f"expected one Layer-1 stream, found {len(streams)}", + "step_counts": [0, 0, 0], + "scheduled_token_throughput": [None, None, None], + "mean_scheduled_token_throughput": None, + "slope_tokens_per_second_squared": None, + "normalized_drift": None, + } + records = [] + try: + for line in streams[0].read_text().splitlines(): + item = json.loads(line) + if "step_index" in item: + records.append(item) + except (OSError, ValueError) as error: + return { + "passed": False, + "reason": f"Layer-1 decode failed: {error}", + "step_counts": [0, 0, 0], + "scheduled_token_throughput": [None, None, None], + "mean_scheduled_token_throughput": None, + "slope_tokens_per_second_squared": None, + "normalized_drift": None, + } + indices = [int(item["step_index"]) for item in records] + continuous = indices == list(range(len(indices))) + step_counts = [0, 0, 0] + token_counts = [0, 0, 0] + for item in records: + if not item.get("model_executed"): + continue + relative_s = (int(item["submit_mono_ns"]) - t0_mono_ns) / 1e9 + if not 45 <= relative_s < 60: + continue + bin_index = min(2, int((relative_s - 45) // 5)) + step_counts[bin_index] += 1 + token_counts[bin_index] += int(item["prefill_tokens"]) + int( + item["decode_tokens"] + ) + rates = [tokens / 5.0 for tokens in token_counts] + mean_rate = sum(rates) / len(rates) + midpoints = [47.5, 52.5, 57.5] + mean_midpoint = sum(midpoints) / len(midpoints) + slope = sum( + (point - mean_midpoint) * (rate - mean_rate) + for point, rate in zip(midpoints, rates, strict=True) + ) / sum((point - mean_midpoint) ** 2 for point in midpoints) + normalized_drift = ( + abs(slope) * 15.0 / mean_rate if mean_rate > 0 else math.inf + ) + finite_positive = all(math.isfinite(rate) and rate > 0 for rate in rates) + passed = ( + continuous + and all(count >= 16 for count in step_counts) + and finite_positive + and math.isfinite(normalized_drift) + and normalized_drift <= 0.10 + ) + return { + "passed": passed, + "reason": None if passed else "A-P3-6 stabilization criterion not met", + "window_seconds": [45.0, 60.0], + "bin_seconds": 5.0, + "step_counts": step_counts, + "scheduled_tokens": token_counts, + "scheduled_token_throughput": rates, + "mean_scheduled_token_throughput": mean_rate, + "slope_tokens_per_second_squared": slope, + "normalized_drift": normalized_drift, + "normalized_drift_limit": 0.10, + "step_indices_continuous": continuous, + } + + +def validate_client( + run_dir: Path, pattern: str, profile: bool, burnin: bool +) -> dict[str, Any]: + result = json.loads((run_dir / "client/result.json").read_text()) + sanity = json.loads((run_dir / "client/sanity.json").read_text()) + failed = [ + key + for key, value in sanity["invariants"].items() + if not value and key != "drain_within_timeout" + ] + quarantined = float(result["drain_seconds"]) > drain_budget(pattern) + expected_duration = 60 if burnin else 240 + warmup_completions = 0 + requests = [] + for line in (run_dir / "client/requests.jsonl").read_text().splitlines(): + request = json.loads(line) + requests.append(request) + if 0 <= request["completed_s"] < result["warmup_seconds"] and request["success"]: + warmup_completions += 1 + clean_start = float(result["clean"]["start_s"]) + clean_end = float(result["clean"]["end_s"]) + failure_summary = summarize_request_failures(requests, clean_start, clean_end) + request_rate = result["request_rate"] + warmup_required = 32 + if pattern != "P10" and request_rate != "inf": + warmup_required = min( + 32, + max(1, math.floor(float(request_rate) * result["warmup_seconds"])), + ) + warmup_stability = ( + p10_warmup_stability(run_dir, int(result["t0_mono_ns"])) + if pattern == "P10" and not burnin + else None + ) + if burnin: + warmup_passed = True + warmup_gate_branch = "burnin" + elif pattern == "P10" and warmup_completions >= 32: + warmup_passed = True + warmup_gate_branch = "count" + elif ( + pattern == "P10" + and warmup_completions >= 16 + and warmup_stability is not None + and warmup_stability["passed"] + ): + warmup_passed = True + warmup_gate_branch = "telemetry" + else: + warmup_passed = warmup_completions >= warmup_required + warmup_gate_branch = "count" if warmup_passed else "failed" + invariants = { + "client_sanity": not failed, + "clean_duration": math.isclose( + float(result["clean"]["duration_s"]), expected_duration + ), + "clean_failures_zero": result["clean"]["failed"] == 0 + and failure_summary["clean_failed"] == 0, + "failed_records_accounted": result["failed_records"] + == failure_summary["failed"], + "manifest_no_wrap": not result["manifest_wrapped"] + and not result["manifest_exhausted"], + "warmup_completions": warmup_passed, + "profile_count": len(result["profiles"]) == (2 if profile else 0), + "profile_after_clean": all( + item["start_call_s"] >= result["warmup_seconds"] + expected_duration + for item in result["profiles"] + ), + "drain_re_adjudicated": not quarantined, + } + non_drain_invariants = { + key: value for key, value in invariants.items() if key != "drain_re_adjudicated" + } + if not all(non_drain_invariants.values()): + raise RuntimeError( + f"client invariant failure: {run_dir}: {invariants}; failed={failed}; " + f"warmup_completions={warmup_completions}; " + f"warmup_gate_branch={warmup_gate_branch}; " + f"warmup_stability={warmup_stability}" + ) + return { + "result": result, + "sanity": sanity, + "request_count": len(requests), + "warmup_completions": warmup_completions, + "warmup_required": warmup_required, + "warmup_gate_branch": warmup_gate_branch, + "warmup_stability": warmup_stability, + "drain_budget_seconds": drain_budget(pattern), + "drain_quarantined": quarantined, + "excluded_window_failures": failure_summary["excluded"], + "excluded_window_failure_kinds": failure_summary["excluded_kinds"], + "invariants": invariants, + } + + +def validate_run( + entry: dict[str, Any], + profile: bool, + burnin: bool, + allow_missing_traces: bool = False, +) -> dict[str, Any]: + run_dir = entry["run_dir"] + client = validate_client( + run_dir, entry["assignment"].cell.pattern, profile, burnin + ) + layer1 = validate_layer1(run_dir) + log = (run_dir / "server.log").read_text(errors="replace") + details = CONFIGS[entry["assignment"].cell.config] + server_invariants = { + "triton_moe": "Using TRITON Unquantized MoE backend" in log, + "chunked_mbt": ( + ( + details["mbt"] == 8192 + and "Chunked prefill is enabled with max_num_batched_tokens=8192" + in log + ) + or ( + details["mbt"] == 2048 + and "'max_num_batched_tokens': 2048" in log + and "'enable_chunked_prefill': True" in log + ) + ), + "tp_effective": f"tensor_parallel_size={details['tp']}" in log, + "drain_shutdown": "mode=drain timeout=120s" in log, + "mns_effective": details["mns"] == 1024 + or "'max_num_seqs': 64" in log, + } + if not all(server_invariants.values()): + raise RuntimeError( + f"server config/backend failure: {run_dir}: {server_invariants}" + ) + trace_summaries: list[dict[str, Any]] = [] + if profile: + trace_dir = run_dir / "traces" + traces = sorted(trace_dir.glob("*.pt.trace.json*")) + expected = 2 * int(details["tp"]) + if len(traces) != expected and not allow_missing_traces: + raise RuntimeError( + f"trace count mismatch: {run_dir}: {len(traces)} != {expected}" + ) + if traces: + trace_summaries = [trace_summary(path) for path in traces] + if not all(summary["valid"] for summary in trace_summaries): + raise RuntimeError(f"invalid Layer-2 trace: {run_dir}") + for rank in range(int(details["tp"])): + if sum(f"rank{rank}" in path.name for path in traces) != 2: + raise RuntimeError(f"rank trace mismatch: {run_dir}: rank{rank}") + forbidden = re.compile(r'"(?:prompt|messages|content|text)"\s*:') + for path in ( + run_dir / "client/requests.jsonl", + run_dir / "client/result.json", + Path(layer1["stream"]), + ): + if forbidden.search(path.read_text(errors="replace")): + raise RuntimeError(f"private/generated text field leaked: {path}") + summary = { + "schema": SCHEMA, + "run_id": entry["run_id"], + "pattern": entry["assignment"].cell.pattern, + "config": entry["assignment"].cell.config, + "gpus": entry["assignment"].gpus, + "client": client, + "layer1": layer1, + "traces": trace_summaries, + "missing_trace_files": ( + 2 * int(details["tp"]) - len(trace_summaries) if profile else 0 + ), + "layer2_missing_after_controller_cleanup": bool( + profile and not trace_summaries and allow_missing_traces + ), + "drain_quarantined": client["drain_quarantined"], + "server_invariants": server_invariants, + } + atomic_json(run_dir / "run-complete.json", summary) + return summary + + +def run_stage( + state: dict[str, Any], + stage_name: str, + assignments: list[Assignment], + load_point: str, + *, + profile: bool, + burnin: bool = False, + confirmation: bool = False, +) -> None: + stage_dir = RUN_ROOT / "stages" / stage_name + existing = state["stages"].get(stage_name) + if existing and existing.get("status") == "complete": + if not (stage_dir / "stage-complete.json").exists(): + raise RuntimeError(f"complete stage marker missing: {stage_name}") + print(f"RESUME skip {stage_name}", flush=True) + return + if stage_dir.exists(): + os.replace(stage_dir, stage_dir.with_name(f"{stage_dir.name}.interrupted-{int(time.time())}")) + stage_dir.mkdir(parents=True) + physical_gpus = sorted({gpu for item in assignments for gpu in item.gpus}) + reserve = sum(item.cell.width for item in assignments) * 1200 / 3600 + if state["gpu_hours_total"] + reserve > GPU_HOUR_LIMIT: + raise RuntimeError( + f"GPU budget reservation exceeds {GPU_HOUR_LIMIT}: " + f"used={state['gpu_hours_total']:.6f} reserve={reserve:.6f}" + ) + state["stages"][stage_name] = { + "status": "preflight", + "started_at": time.time(), + "load_point": load_point, + "profile": profile, + "burnin": burnin, + "confirmation": confirmation, + "assignments": [ + {"cell": item.cell.cell_id, "gpus": item.gpus} for item in assignments + ], + } + save_state(state) + preflight(stage_dir, physical_gpus) + entries: list[dict[str, Any]] = [] + handles: list[Any] = [] + owned_pgids: set[int] = set() + monitor: common.Monitor | None = None + failure: Exception | None = None + stage_gpu_seconds = 0.0 + try: + for assignment in assignments: + suffix = "burnin" if burnin else ("confirmation" if confirmation else load_point) + run_id = f"{assignment.cell.cell_id}-{suffix}" + if burnin: + run_dir = RUN_ROOT / "burnins" / assignment.cell.config + elif confirmation: + run_dir = RUN_ROOT / "confirmations" / assignment.cell.pattern + else: + run_dir = RUN_ROOT / "primary" / assignment.cell.cell_id / load_point + if run_dir.exists(): + os.replace( + run_dir, + run_dir.with_name(f"{run_dir.name}.interrupted-{int(time.time())}"), + ) + run_dir.mkdir(parents=True) + trace_tmp = Path(f"/tmp/wjh-opprof-phase3-matrix/{run_id}") + if trace_tmp.exists(): + shutil.rmtree(trace_tmp) + trace_tmp.mkdir(parents=True) + (run_dir / "trace-tmp").symlink_to(trace_tmp, target_is_directory=True) + port = 8100 + assignment.gpus[0] + command = server_command(assignment, port, trace_tmp) + with (run_dir / "commands.log").open("w", encoding="utf-8") as output: + output.write( + f"GPU_COMMAND {run_id} server: {shlex.join(command)}; " + "expected=startup 60-180s + fixed load 300-600s\n" + ) + environment = os.environ.copy() + environment.update( + { + "CUDA_VISIBLE_DEVICES": ",".join(map(str, assignment.gpus)), + "VLLM_OPPROF_DIR": str(run_dir / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + handle = (run_dir / "server.log").open("ab", buffering=0) + handles.append(handle) + spawned_at = time.time() + server = subprocess.Popen( + command, + cwd=SOURCE, + env=environment, + stdout=handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + owned_pgids.add(server.pid) + entries.append( + { + "assignment": assignment, + "run_id": run_id, + "run_dir": run_dir, + "trace_tmp": trace_tmp, + "port": port, + "server": server, + "spawned_at": spawned_at, + } + ) + state["stages"][stage_name]["status"] = "starting_servers" + state["stages"][stage_name]["servers"] = { + entry["run_id"]: { + "pid": entry["server"].pid, + "pgid": entry["server"].pid, + "gpus": entry["assignment"].gpus, + } + for entry in entries + } + save_state(state) + wait_ready(entries) + monitor = common.Monitor(stage_dir / "monitor.jsonl", owned_pgids) + monitor.start() + for entry in entries: + assignment = entry["assignment"] + saturation_result = None + if load_point == "moderate": + saturation_result = ( + RUN_ROOT + / "primary" + / assignment.cell.cell_id + / "saturation/client/result.json" + ) + command = client_command( + assignment, + entry["port"], + entry["run_dir"], + load_point, + profile, + burnin, + saturation_result, + ) + with (entry["run_dir"] / "commands.log").open("a", encoding="utf-8") as output: + output.write( + f"GPU_COMMAND {entry['run_id']} client: {shlex.join(command)}; " + "expected=fixed protocol timeline\n" + ) + handle = (entry["run_dir"] / "client.log").open("ab", buffering=0) + handles.append(handle) + client = subprocess.Popen( + command, + cwd=WORKDIR, + stdout=handle, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + entry["client"] = client + owned_pgids.add(client.pid) + state["stages"][stage_name]["status"] = "running_clients" + state["stages"][stage_name]["clients"] = { + entry["run_id"]: {"pid": entry["client"].pid, "pgid": entry["client"].pid} + for entry in entries + } + save_state(state) + deadline = time.monotonic() + 1200 + while time.monotonic() < deadline and any( + entry["client"].poll() is None for entry in entries + ): + if any(entry["server"].poll() is not None for entry in entries): + raise RuntimeError("server exited while a client was active") + if monitor.other_apps: + raise RuntimeError(f"other GPU process appeared: {monitor.other_apps}") + live_hours = sum( + (time.time() - entry["spawned_at"]) * entry["assignment"].cell.width + for entry in entries + ) / 3600 + if state["gpu_hours_total"] + live_hours >= GPU_HOUR_LIMIT: + raise RuntimeError("cumulative GPU-hour hard stop reached") + time.sleep(2) + if any(entry["client"].poll() is None for entry in entries): + raise TimeoutError(f"stage exceeded 1200 seconds: {stage_name}") + bad = {} + for entry in entries: + if entry["client"].returncode == 0: + continue + sanity_path = entry["run_dir"] / "client/sanity.json" + if not sanity_path.exists(): + bad[entry["run_id"]] = entry["client"].returncode + continue + client_sanity = json.loads(sanity_path.read_text())["invariants"] + failed = [key for key, value in client_sanity.items() if not value] + if failed != ["drain_within_timeout"]: + bad[entry["run_id"]] = { + "returncode": entry["client"].returncode, + "failed_invariants": failed, + } + if bad: + raise RuntimeError(f"client failures: {bad}") + for entry in entries: + if profile: + destination = entry["run_dir"] / "traces" + destination.mkdir() + for path in sorted(entry["trace_tmp"].glob("*.pt.trace.json*")): + shutil.copy2(path, destination / path.name) + state["stages"][stage_name]["status"] = "clients_complete" + save_state(state) + except Exception as error: + failure = error + finally: + for entry in entries: + client = entry.get("client") + if client is not None and client.poll() is None: + try: + os.killpg(client.pid, signal.SIGKILL) + except ProcessLookupError: + pass + try: + stop_servers(entries) + except Exception as error: + failure = failure or error + ended_at = time.time() + stage_gpu_seconds = sum( + (ended_at - entry["spawned_at"]) * entry["assignment"].cell.width + for entry in entries + ) + if monitor is not None: + monitor.stop() + atomic_json(stage_dir / "other-gpu-processes.json", monitor.other_apps) + if monitor.other_apps: + failure = failure or RuntimeError( + f"other GPU process appeared: {monitor.other_apps}" + ) + for entry in entries: + trace_link = entry["run_dir"] / "trace-tmp" + if trace_link.is_symlink(): + trace_link.unlink() + if entry["trace_tmp"].exists(): + shutil.rmtree(entry["trace_tmp"]) + for handle in handles: + handle.close() + (stage_dir / "clocks-after.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"], check=False) + ) + (stage_dir / "loadavg-after.txt").write_text( + " ".join(str(value) for value in os.getloadavg()) + "\n" + ) + try: + common.verify_idle(physical_gpus, stage_dir) + except Exception as error: + failure = failure or error + state["gpu_hours_total"] += stage_gpu_seconds / 3600 + state["gpu_hours_this_stage"] = stage_gpu_seconds / 3600 + if failure is None: + try: + summaries = [validate_run(entry, profile, burnin) for entry in entries] + except Exception as error: + failure = error + if failure is not None: + state["stages"][stage_name]["status"] = "failed" + state["stages"][stage_name]["failure"] = repr(failure) + state["stages"][stage_name]["gpu_hours"] = stage_gpu_seconds / 3600 + state["status"] = "failed" + save_state(state) + raise failure + marker = { + "schema": SCHEMA, + "stage": stage_name, + "completed_at": time.time(), + "gpu_hours": stage_gpu_seconds / 3600, + "runs": [summary["run_id"] for summary in summaries], + } + atomic_json(stage_dir / "stage-complete.json", marker) + state["stages"][stage_name].update( + { + "status": "complete", + "completed_at": marker["completed_at"], + "gpu_hours": marker["gpu_hours"], + "servers": {}, + "clients": {}, + } + ) + if burnin: + state["completed_burnins"] += len(assignments) + else: + state["completed_measured_runs"] += len(assignments) + state.setdefault("drain_quarantined_runs", 0) + state["drain_quarantined_runs"] += sum( + summary["drain_quarantined"] for summary in summaries + ) + if ( + state["completed_measured_runs"] > 0 + and state["drain_quarantined_runs"] + / state["completed_measured_runs"] + > 0.20 + ): + state["status"] = "failed" + save_state(state) + raise RuntimeError("drain-quarantine rate exceeded 20%") + save_state(state) + + +def cleanup_recorded(state: dict[str, Any]) -> None: + for stage in state.get("stages", {}).values(): + if stage.get("status") == "complete": + continue + for group in ("clients", "servers"): + for item in stage.get(group, {}).values(): + pgid = item.get("pgid") + if pgid: + try: + os.killpg(int(pgid), signal.SIGKILL) + except ProcessLookupError: + pass + time.sleep(2) + + +def execute(resume: bool) -> None: + RUN_ROOT.mkdir(parents=True, exist_ok=True) + state = load_state(resume) + if resume: + cleanup_recorded(state) + current = fingerprint() + if state["fingerprint"] and state["fingerprint"] != current: + raise RuntimeError("resume fingerprint differs from frozen matrix") + state["fingerprint"] = current + state["status"] = "running" + save_state(state) + ensure_provenance() + + burnin_cells = [Cell("P06", config) for config in CONFIGS] + for index, wave in enumerate(pack_cells(burnin_cells), 1): + run_stage( + state, + f"burnin-{index:02d}", + wave, + "saturation", + profile=False, + burnin=True, + ) + + matrix_waves = pack_cells(cells()) + for index, wave in enumerate(matrix_waves, 1): + run_stage( + state, + f"primary-{index:02d}-saturation", + wave, + "saturation", + profile=True, + ) + run_stage( + state, + f"primary-{index:02d}-moderate", + wave, + "moderate", + profile=True, + ) + + confirmations = [ + Assignment(Cell(pattern, "C00"), (gpu,)) + for gpu, pattern in enumerate(("P10", "P06", "P03", "P01")) + ] + run_stage( + state, + "confirmations", + confirmations, + "moderate", + profile=False, + confirmation=True, + ) + if state["completed_measured_runs"] != 52 or state["completed_burnins"] != 5: + raise RuntimeError( + f"completion count mismatch: measured={state['completed_measured_runs']} " + f"burnins={state['completed_burnins']}" + ) + trace_count = len(list((RUN_ROOT / "primary").glob("*/*/traces/*.pt.trace.json*"))) + expected_trace_count = 100 - state.get("missing_trace_files", 0) + if trace_count != expected_trace_count: + raise RuntimeError( + f"trace aggregate mismatch: {trace_count} != {expected_trace_count}" + ) + state["trace_files"] = trace_count + state["status"] = "complete" + state["completed_at"] = time.time() + save_state(state) + print(json.dumps(state, sort_keys=True), flush=True) + + +def plan() -> dict[str, Any]: + matrix_waves = pack_cells(cells()) + return { + "schema": SCHEMA, + "placement": "4-way GPU0-3", + "cells": len(cells()), + "primary_runs": 48, + "confirmation_runs": 4, + "burnins": 5, + "matrix_waves": [ + [ + {"cell": item.cell.cell_id, "gpus": item.gpus} + for item in wave + ] + for wave in matrix_waves + ], + "expected_trace_files": 100, + "prior_gpu_hours": PRIOR_GPU_HOURS, + "gpu_hour_limit": GPU_HOUR_LIMIT, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(dest="command", required=True) + run = subparsers.add_parser("run") + run.add_argument("--resume", action="store_true") + subparsers.add_parser("status") + subparsers.add_parser("plan") + args = parser.parse_args() + if args.command == "run": + execute(args.resume) + elif args.command == "status": + print(STATE.read_text() if STATE.exists() else '{"status":"absent"}') + else: + print(json.dumps(plan(), sort_keys=True, indent=2)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/test_phase3_analysis.py b/runs/opprof-phase3/provenance/test_phase3_analysis.py new file mode 100644 index 0000000..b70d2b0 --- /dev/null +++ b/runs/opprof-phase3/provenance/test_phase3_analysis.py @@ -0,0 +1,109 @@ +from __future__ import annotations + +import json +import tempfile +import unittest +from pathlib import Path + +import analyze_phase3 as analysis + + +class Phase3AnalysisTests(unittest.TestCase): + def test_ap36_stability_formula(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + (root / "client").mkdir() + (root / "opprof").mkdir() + (root / "client/result.json").write_text( + json.dumps({"t0_mono_ns": 0, "warmup_seconds": 60}) + ) + (root / "client/requests.jsonl").write_text( + "".join( + json.dumps({"success": True, "completed_s": index + 1}) + "\n" + for index in range(16) + ) + ) + records = [] + for bin_index in range(3): + for step in range(16): + records.append( + { + "step_index": len(records), + "model_executed": True, + "submit_mono_ns": int( + (45 + 5 * bin_index + (step + 0.5) / 16 * 5) + * 1e9 + ), + "prefill_tokens": 100, + "decode_tokens": 0, + } + ) + (root / "opprof/test.jsonl").write_text( + "".join(json.dumps(item) + "\n" for item in records) + ) + result = analysis.ap36_warmup_stability(root) + self.assertTrue(result["passes"]) + self.assertEqual(result["normalized_drift"], 0) + + def test_ap37_partial_verdict_can_confirm_but_not_refute(self): + self.assertEqual(analysis.partial_verdict(True), "PASS") + self.assertEqual(analysis.partial_verdict(False), "INCONCLUSIVE") + + def test_accepted_markers_come_only_from_complete_stages(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + complete = root / "stages/primary-01-saturation" + complete.mkdir(parents=True) + (complete / "stage-complete.json").write_text( + json.dumps({"runs": ["P01-C00-saturation"]}) + ) + accepted = root / "primary/P01-C00/saturation" + accepted.mkdir(parents=True) + (accepted / "run-complete.json").write_text( + json.dumps({"run_id": "P01-C00-saturation"}) + ) + unaccepted = root / "primary/P05-C00/saturation" + unaccepted.mkdir(parents=True) + (unaccepted / "run-complete.json").write_text( + json.dumps({"run_id": "P05-C00-saturation"}) + ) + primary, confirmations, stages, excluded = analysis.accepted_marker_paths( + root + ) + self.assertEqual(primary, [accepted / "run-complete.json"]) + self.assertEqual(confirmations, []) + self.assertEqual(stages, [complete]) + self.assertEqual(excluded, ["P05-C00-saturation"]) + + def test_r64_is_ratio_of_cohort_sums(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "manifest.jsonl" + rows = [{"input_tokens": value} for value in (1, 3, 2, 2)] + path.write_text("".join(json.dumps(row) + "\n" for row in rows)) + value, pieces = analysis.manifest_raggedness(path, 2) + self.assertEqual(pieces, [(2.0, 6.0), (0.0, 4.0)]) + self.assertAlmostEqual(value, 0.2) + + def test_one_percentage_point_ranking_ties(self): + shares = dict.fromkeys(analysis.FAMILIES, 0.0) + shares.update(attention=0.40, moe_gemm=0.395, moe_router=0.20) + ranked = {item["family"]: item["rank"] for item in analysis.ranked_families(shares)} + self.assertEqual(ranked["attention"], ranked["moe_gemm"]) + self.assertGreater(ranked["moe_router"], ranked["attention"]) + + def test_holm_uses_declared_total_test_family(self): + values = [{"p": 0.001}, {"p": 0.01}] + analysis.holm(values, total_tests=10) + self.assertAlmostEqual(values[0]["p_holm"], 0.01) + self.assertAlmostEqual(values[1]["p_holm"], 0.09) + + def test_robust_spline_prediction_is_nonnegative(self): + rows = [(float(x), float(n), float(2 * x + n)) for x in range(1, 20) for n in (1, 4)] + predict, hull = analysis.fit_nonnegative_robust(rows) + self.assertGreaterEqual(predict(3, 2), 0) + self.assertTrue(analysis.inside_convex(hull, (3, 2))) + self.assertFalse(analysis.inside_convex(hull, (100, 2))) + + +if __name__ == "__main__": + unittest.main() diff --git a/runs/opprof-phase3/provenance/test_phase3_tools.py b/runs/opprof-phase3/provenance/test_phase3_tools.py new file mode 100644 index 0000000..5a5a9ed --- /dev/null +++ b/runs/opprof-phase3/provenance/test_phase3_tools.py @@ -0,0 +1,486 @@ +#!/usr/bin/env python3 + +from __future__ import annotations + +import asyncio +import importlib.util +import json +import sys +import tempfile +import unittest +from unittest import mock +from pathlib import Path +from types import SimpleNamespace + +from aiohttp import web + + +HERE = Path(__file__).parent + + +def load_module(name: str, filename: str): + spec = importlib.util.spec_from_file_location(name, HERE / filename) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +client = load_module("phase3_client", "opprof_phase3_client.py") +controller = load_module("phase3_controller", "opprof_phase3_controller.py") +sys.modules["opprof_phase3_controller"] = controller +matrix = load_module("phase3_matrix", "opprof_phase3_matrix.py") + + +def rows(n: int, arrival: str = "steady") -> list[dict]: + return [ + { + "schema": 1, + "request_id": f"T-{i:05d}", + "pattern_id": "T", + "kind": "synthetic", + "input_tokens": 8, + "output_tokens": 2, + "arrival": arrival, + "token_seed": i + 1, + } + for i in range(n) + ] + + +class MockServer: + def __init__(self, delay: float = 0.01, trace_dir: Path | None = None) -> None: + self.active = 0 + self.max_active = 0 + self.payloads = [] + self.runner = None + self.port = None + self.delay = delay + self.trace_dir = trace_dir + self.profile_count = 0 + + async def completion(self, request): + payload = await request.json() + self.payloads.append(payload) + self.active += 1 + self.max_active = max(self.max_active, self.active) + await asyncio.sleep(self.delay) + response = web.StreamResponse( + status=200, headers={"Content-Type": "text/event-stream"} + ) + await response.prepare(request) + await response.write( + b'data: {"choices":[{"text":"x"}],"usage":null}\n\n' + ) + usage = json.dumps( + { + "choices": [], + "usage": { + "prompt_tokens": len(payload["prompt"]), + "completion_tokens": payload["max_tokens"], + }, + } + ).encode() + await response.write(b"data: " + usage + b"\n\n") + await response.write(b"data: [DONE]\n\n") + await response.write_eof() + self.active -= 1 + return response + + async def start_profile(self, request): + self.profile_count += 1 + path = self.trace_dir / f"window-{self.profile_count}.pt.trace.json" + path.write_text('{"traceEvents": []}') + return web.Response(status=200) + + async def stop_profile(self, request): + return web.Response(status=200) + + async def start(self): + app = web.Application() + app.router.add_post("/v1/completions", self.completion) + if self.trace_dir is not None: + app.router.add_post("/start_profile", self.start_profile) + app.router.add_post("/stop_profile", self.stop_profile) + self.runner = web.AppRunner(app) + await self.runner.setup() + site = web.TCPSite(self.runner, "127.0.0.1", 0) + await site.start() + self.port = site._server.sockets[0].getsockname()[1] + + async def stop(self): + await self.runner.cleanup() + + +def run_args( + manifest: Path, + result_dir: Path, + port: int, + load_point: str = "saturation", + saturation_result: Path | None = None, +): + return SimpleNamespace( + manifest=str(manifest), + base_url=f"http://127.0.0.1:{port}", + model="mock", + load_point=load_point, + request_rate="inf" if load_point == "saturation" else None, + saturation_result=None if saturation_result is None else str(saturation_result), + rate_fraction=0.60, + max_concurrency=3, + ignore_eos=True, + temperature=0, + warmup_seconds=0.04, + clean_segment_seconds=0.04, + num_clean_segments=3, + profile_after_clean=False, + num_profile_windows=0, + profile_warmup_iterations=2, + profile_active_iterations=8, + profile_trace_dir=None, + profile_timeout_seconds=1, + recovery_seconds=0, + post_clean_seconds=0, + drain_timeout_seconds=1, + workload_seed=20260712, + server_seed=20260712, + result_dir=str(result_dir), + ) + + +class Phase3ToolTests(unittest.TestCase): + def write_warmup_stream(self, root: Path, tokens: list[int]) -> None: + stream_dir = root / "opprof" + stream_dir.mkdir() + records = [] + step = 0 + for bin_index, token_count in enumerate(tokens): + per_step, remainder = divmod(token_count, 16) + for in_bin in range(16): + records.append( + { + "step_index": step, + "model_executed": True, + "submit_mono_ns": int( + 1e9 + + (45 + bin_index * 5 + (in_bin + 0.5) / 16 * 5) + * 1e9 + ), + "prefill_tokens": per_step + (in_bin < remainder), + "decode_tokens": 0, + } + ) + step += 1 + (stream_dir / "test.jsonl").write_text( + "".join(json.dumps(record) + "\n" for record in records) + ) + + def test_p10_warmup_stability_accepts_flat_trailing_quartile(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self.write_warmup_stream(root, [1600, 1600, 1600]) + result = matrix.p10_warmup_stability(root, int(1e9)) + self.assertTrue(result["passed"]) + self.assertEqual(result["step_counts"], [16, 16, 16]) + self.assertEqual(result["scheduled_token_throughput"], [320, 320, 320]) + self.assertEqual(result["normalized_drift"], 0) + + def test_p10_warmup_stability_rejects_large_drift(self): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + self.write_warmup_stream(root, [1600, 3200, 4800]) + result = matrix.p10_warmup_stability(root, int(1e9)) + self.assertFalse(result["passed"]) + self.assertGreater(result["normalized_drift"], 0.10) + + def test_only_clean_window_request_failures_are_hard_failures(self): + requests = [ + {"success": False, "completed_s": 59.9, "error_kind": "warmup"}, + {"success": False, "completed_s": 60.0, "error_kind": "clean"}, + {"success": False, "completed_s": 299.9, "error_kind": "clean"}, + {"success": False, "completed_s": 300.0, "error_kind": "recovery"}, + {"success": True, "completed_s": 100.0, "error_kind": None}, + ] + summary = matrix.summarize_request_failures(requests, 60.0, 300.0) + self.assertEqual(summary["failed"], 4) + self.assertEqual(summary["clean_failed"], 2) + self.assertEqual(summary["excluded"], 2) + self.assertEqual(summary["excluded_kinds"], {"warmup": 1, "recovery": 1}) + + def test_synthetic_prompt_exact_and_unique_early_prefix(self): + generated = [client.synthetic_prompt(row) for row in rows(200)] + self.assertTrue(all(len(value) == 8 for value in generated)) + self.assertEqual(len({tuple(value[:3]) for value in generated}), 200) + + def test_p05_manifest_has_exact_half_modes(self): + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "P05.jsonl" + args = SimpleNamespace( + id="P05", + kind="synthetic", + input_uniform=None, + input_fixed=None, + input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + output_fixed=64, + prefix="none", + arrival="steady", + num_requests=100, + workload_seed=20260712, + num_prefixes=0, + prefix_len=0, + suffix_fixed=0, + out=str(out), + ) + client.materialize(args) + manifest = client.load_manifest(out) + self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50) + self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50) + + def test_prefix_pool_balanced(self): + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "P08.jsonl" + args = SimpleNamespace( + id="P08", + kind="prefix-pool", + input_uniform=None, + input_fixed=None, + input_mixture=None, + output_fixed=512, + prefix="none", + arrival="burst:8", + num_requests=80, + workload_seed=20260712, + num_prefixes=8, + prefix_len=1024, + suffix_fixed=256, + out=str(out), + ) + client.materialize(args) + manifest = client.load_manifest(out) + counts = {i: 0 for i in range(8)} + for row in manifest: + counts[row["prefix_id"]] += 1 + self.assertEqual(row["input_tokens"], 1280) + self.assertEqual(set(counts.values()), {10}) + + def test_fixed_duration_saturation_and_redaction(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000)) + mock = MockServer() + await mock.start() + try: + result = await client.run_load( + run_args(manifest, root / "result", mock.port) + ) + finally: + await mock.stop() + self.assertEqual(result["max_in_flight"], 3) + self.assertEqual(mock.max_active, 3) + self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9) + self.assertEqual(len(result["segments"]), 3) + self.assertLessEqual(result["drain_seconds"], 1) + self.assertTrue( + all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads) + ) + text = (root / "result/requests.jsonl").read_text() + self.assertNotIn('"prompt":', text) + self.assertNotIn('"text":', text) + + asyncio.run(case()) + + def test_profile_control_plane_is_not_starved_by_data_connector(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + trace_dir = root / "traces" + trace_dir.mkdir() + client.atomic_jsonl(manifest, rows(1000)) + server = MockServer(delay=0.5, trace_dir=trace_dir) + await server.start() + args = run_args(manifest, root / "result", server.port) + args.max_concurrency = 1 + args.warmup_seconds = 0.01 + args.clean_segment_seconds = 0.01 + args.num_clean_segments = 1 + args.profile_after_clean = True + args.num_profile_windows = 1 + args.profile_trace_dir = str(trace_dir) + args.recovery_seconds = 0 + try: + result = await client.run_load(args) + finally: + await server.stop() + self.assertEqual(server.max_active, 1) + self.assertEqual(server.profile_count, 1) + self.assertLess(result["profiles"][0]["start_return_s"], 0.25) + + asyncio.run(case()) + + def test_drain_timeout_is_a_hard_sanity_failure(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000)) + mock = MockServer() + await mock.start() + try: + args = run_args(manifest, root / "result", mock.port) + args.drain_timeout_seconds = 0 + with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"): + await client.run_load(args) + finally: + await mock.stop() + sanity = json.loads((root / "result/sanity.json").read_text()) + self.assertFalse(sanity["invariants"]["drain_within_timeout"]) + + asyncio.run(case()) + + def test_burst_schedule_is_eight_at_once(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000, "burst:8")) + sat = root / "sat.json" + client.atomic_json( + sat, {"clean": {"completed_throughput_rps": 100.0}} + ) + mock = MockServer() + await mock.start() + try: + args = run_args( + manifest, root / "result", mock.port, "moderate", sat + ) + args.warmup_seconds = 0 + args.clean_segment_seconds = 0.4 / 3 + result = await client.run_load(args) + finally: + await mock.stop() + records = [ + json.loads(line) + for line in (root / "result/requests.jsonl").read_text().splitlines() + ] + groups = {} + for record in records: + groups.setdefault(round(record["scheduled_s"], 6), 0) + groups[round(record["scheduled_s"], 6)] += 1 + self.assertTrue(all(value == 8 for value in groups.values())) + starts = sorted(groups) + if len(starts) > 1: + self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5) + self.assertAlmostEqual(result["request_rate"], 60.0) + + asyncio.run(case()) + + def test_manifest_exhaustion_stops_instead_of_wrapping(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(2)) + mock = MockServer() + await mock.start() + try: + with self.assertRaises(client.ManifestExhausted): + await client.run_load( + run_args(manifest, root / "result", mock.port) + ) + finally: + await mock.stop() + + asyncio.run(case()) + + def test_cpu_affinity_sets_are_disjoint_and_cover_host(self): + values = [] + for spec in controller.CPU_MAP.values(): + lo, hi = [int(x) for x in spec.split("-")] + values.extend(range(lo, hi + 1)) + self.assertEqual(sorted(values), list(range(160))) + self.assertEqual(len(values), len(set(values))) + + def test_preflight_waits_for_three_stable_zero_samples(self): + def sample(memory): + return [ + { + "index": 0, + "uuid": "GPU-test", + "memory_used_mib": memory, + "utilization_pct": 0, + } + ] + + with tempfile.TemporaryDirectory() as tmp: + with ( + mock.patch.object( + controller, + "gpu_query", + side_effect=[sample(4), sample(0), sample(0), sample(0)], + ), + mock.patch.object(controller, "compute_apps", return_value=[]), + mock.patch.object(controller, "run_text", return_value=""), + mock.patch.object(controller.time, "sleep"), + ): + root = Path(tmp) + controller.preflight([0], root) + samples = json.loads((root / "gpu-before-samples.json").read_text()) + self.assertEqual(len(samples), 4) + self.assertEqual(samples[0]["gpus"][0]["memory_used_mib"], 4) + self.assertEqual(samples[-1]["gpus"][0]["memory_used_mib"], 0) + + def test_kernel_mapping_priority(self): + cases = { + "void vllm::moe::topkGating": "moe_router", + "ncclDevKernel_AllReduce": "collective", + "flash_fwd_kernel": "attention", + "nvjet_sm90_tst": "moe_gemm", + "argmax_kernel": "sampler", + "cutlass_gemm": "dense_gemm", + "triton_red_fused_add_rms_norm": "norm_elementwise", + "cache_swap_kernel": "kv_memory", + "unknown": "other", + } + self.assertEqual( + {name: controller.classify_kernel(name) for name in cases}, cases + ) + + def test_atomic_state_replacement(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "state.json" + controller.atomic_json(path, {"schema": 1, "value": 1}) + controller.atomic_json(path, {"schema": 1, "value": 2}) + self.assertEqual(json.loads(path.read_text())["value"], 2) + self.assertFalse(list(path.parent.glob("*.tmp.*"))) + + def test_fingerprint_survives_json_roundtrip(self): + original_run_text = controller.run_text + original_hash = controller.sha256_file + try: + controller.run_text = lambda *args, **kwargs: "deadbeef\n" + controller.sha256_file = lambda path: "a" * 64 + fingerprint = controller.make_fingerprint() + finally: + controller.run_text = original_run_text + controller.sha256_file = original_hash + self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint) + self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)}) + + def test_server_shutdown_signals_parent_before_group(self): + process = SimpleNamespace(pid=12345, poll=lambda: None) + with ( + mock.patch.object(controller.os, "kill") as kill, + mock.patch.object(controller.os, "killpg") as killpg, + mock.patch.object(controller, "_process_group_alive", return_value=False), + ): + controller.stop_servers([process]) + kill.assert_called_once_with(12345, controller.signal.SIGINT) + killpg.assert_not_called() + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/runs/opprof-phase3/provenance/test_phase3_tools_ea.py b/runs/opprof-phase3/provenance/test_phase3_tools_ea.py new file mode 100644 index 0000000..9d5c7a9 --- /dev/null +++ b/runs/opprof-phase3/provenance/test_phase3_tools_ea.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python3 + +from __future__ import annotations + +import asyncio +import importlib.util +import json +import sys +import tempfile +import unittest +from unittest import mock +from pathlib import Path +from types import SimpleNamespace + +from aiohttp import web + + +HERE = Path(__file__).parent + + +def load_module(name: str, filename: str): + spec = importlib.util.spec_from_file_location(name, HERE / filename) + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +client = load_module("phase3_client", "opprof_phase3_client.py") +controller = load_module("phase3_controller", "opprof_phase3_controller.py") + + +def rows(n: int, arrival: str = "steady") -> list[dict]: + return [ + { + "schema": 1, + "request_id": f"T-{i:05d}", + "pattern_id": "T", + "kind": "synthetic", + "input_tokens": 8, + "output_tokens": 2, + "arrival": arrival, + "token_seed": i + 1, + } + for i in range(n) + ] + + +class MockServer: + def __init__(self) -> None: + self.active = 0 + self.max_active = 0 + self.payloads = [] + self.runner = None + self.port = None + + async def completion(self, request): + payload = await request.json() + self.payloads.append(payload) + self.active += 1 + self.max_active = max(self.max_active, self.active) + await asyncio.sleep(0.01) + response = web.StreamResponse( + status=200, headers={"Content-Type": "text/event-stream"} + ) + await response.prepare(request) + await response.write( + b'data: {"choices":[{"text":"x"}],"usage":null}\n\n' + ) + usage = json.dumps( + { + "choices": [], + "usage": { + "prompt_tokens": len(payload["prompt"]), + "completion_tokens": payload["max_tokens"], + }, + } + ).encode() + await response.write(b"data: " + usage + b"\n\n") + await response.write(b"data: [DONE]\n\n") + await response.write_eof() + self.active -= 1 + return response + + async def start(self): + app = web.Application() + app.router.add_post("/v1/completions", self.completion) + self.runner = web.AppRunner(app) + await self.runner.setup() + site = web.TCPSite(self.runner, "127.0.0.1", 0) + await site.start() + self.port = site._server.sockets[0].getsockname()[1] + + async def stop(self): + await self.runner.cleanup() + + +def run_args( + manifest: Path, + result_dir: Path, + port: int, + load_point: str = "saturation", + saturation_result: Path | None = None, +): + return SimpleNamespace( + manifest=str(manifest), + base_url=f"http://127.0.0.1:{port}", + model="mock", + load_point=load_point, + request_rate="inf" if load_point == "saturation" else None, + saturation_result=None if saturation_result is None else str(saturation_result), + rate_fraction=0.60, + max_concurrency=3, + ignore_eos=True, + temperature=0, + warmup_seconds=0.04, + clean_segment_seconds=0.04, + num_clean_segments=3, + profile_after_clean=False, + num_profile_windows=0, + profile_warmup_iterations=2, + profile_active_iterations=8, + profile_trace_dir=None, + profile_timeout_seconds=1, + recovery_seconds=0, + post_clean_seconds=0, + drain_timeout_seconds=1, + workload_seed=20260712, + server_seed=20260712, + result_dir=str(result_dir), + ) + + +class Phase3ToolTests(unittest.TestCase): + def test_synthetic_prompt_exact_and_unique_early_prefix(self): + generated = [client.synthetic_prompt(row) for row in rows(200)] + self.assertTrue(all(len(value) == 8 for value in generated)) + self.assertEqual(len({tuple(value[:3]) for value in generated}), 200) + + def test_p05_manifest_has_exact_half_modes(self): + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "P05.jsonl" + args = SimpleNamespace( + id="P05", + kind="synthetic", + input_uniform=None, + input_fixed=None, + input_mixture='{"uniform:128:512":0.5,"uniform:4096:8192":0.5}', + output_fixed=64, + prefix="none", + arrival="steady", + num_requests=100, + workload_seed=20260712, + num_prefixes=0, + prefix_len=0, + suffix_fixed=0, + out=str(out), + ) + client.materialize(args) + manifest = client.load_manifest(out) + self.assertEqual(sum(r["input_tokens"] <= 512 for r in manifest), 50) + self.assertEqual(sum(r["input_tokens"] >= 4096 for r in manifest), 50) + + def test_prefix_pool_balanced(self): + with tempfile.TemporaryDirectory() as tmp: + out = Path(tmp) / "P08.jsonl" + args = SimpleNamespace( + id="P08", + kind="prefix-pool", + input_uniform=None, + input_fixed=None, + input_mixture=None, + output_fixed=512, + prefix="none", + arrival="burst:8", + num_requests=80, + workload_seed=20260712, + num_prefixes=8, + prefix_len=1024, + suffix_fixed=256, + out=str(out), + ) + client.materialize(args) + manifest = client.load_manifest(out) + counts = {i: 0 for i in range(8)} + for row in manifest: + counts[row["prefix_id"]] += 1 + self.assertEqual(row["input_tokens"], 1280) + self.assertEqual(set(counts.values()), {10}) + + def test_fixed_duration_saturation_and_redaction(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000)) + mock = MockServer() + await mock.start() + try: + result = await client.run_load( + run_args(manifest, root / "result", mock.port) + ) + finally: + await mock.stop() + self.assertEqual(result["max_in_flight"], 3) + self.assertEqual(mock.max_active, 3) + self.assertAlmostEqual(result["clean"]["duration_s"], 0.12, places=9) + self.assertEqual(len(result["segments"]), 3) + self.assertLessEqual(result["drain_seconds"], 1) + self.assertTrue( + all(p["max_tokens"] == 2 and p["ignore_eos"] for p in mock.payloads) + ) + text = (root / "result/requests.jsonl").read_text() + self.assertNotIn('"prompt":', text) + self.assertNotIn('"text":', text) + + asyncio.run(case()) + + def test_drain_timeout_is_a_hard_sanity_failure(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000)) + mock = MockServer() + await mock.start() + try: + args = run_args(manifest, root / "result", mock.port) + args.drain_timeout_seconds = 0 + with self.assertRaisesRegex(RuntimeError, "drain_within_timeout"): + await client.run_load(args) + finally: + await mock.stop() + sanity = json.loads((root / "result/sanity.json").read_text()) + self.assertFalse(sanity["invariants"]["drain_within_timeout"]) + + asyncio.run(case()) + + def test_burst_schedule_is_eight_at_once(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(1000, "burst:8")) + sat = root / "sat.json" + client.atomic_json( + sat, {"clean": {"completed_throughput_rps": 100.0}} + ) + mock = MockServer() + await mock.start() + try: + args = run_args( + manifest, root / "result", mock.port, "moderate", sat + ) + args.warmup_seconds = 0 + args.clean_segment_seconds = 0.4 / 3 + result = await client.run_load(args) + finally: + await mock.stop() + records = [ + json.loads(line) + for line in (root / "result/requests.jsonl").read_text().splitlines() + ] + groups = {} + for record in records: + groups.setdefault(round(record["scheduled_s"], 6), 0) + groups[round(record["scheduled_s"], 6)] += 1 + self.assertTrue(all(value == 8 for value in groups.values())) + starts = sorted(groups) + if len(starts) > 1: + self.assertAlmostEqual(starts[1] - starts[0], 8 / 60, places=5) + self.assertAlmostEqual(result["request_rate"], 60.0) + + asyncio.run(case()) + + def test_manifest_exhaustion_stops_instead_of_wrapping(self): + async def case(): + with tempfile.TemporaryDirectory() as tmp: + root = Path(tmp) + manifest = root / "m.jsonl" + client.atomic_jsonl(manifest, rows(2)) + mock = MockServer() + await mock.start() + try: + with self.assertRaises(client.ManifestExhausted): + await client.run_load( + run_args(manifest, root / "result", mock.port) + ) + finally: + await mock.stop() + + asyncio.run(case()) + + def test_cpu_affinity_sets_are_disjoint_and_cover_host(self): + values = [] + for spec in controller.CPU_MAP.values(): + lo, hi = [int(x) for x in spec.split("-")] + values.extend(range(lo, hi + 1)) + self.assertEqual(sorted(values), list(range(160))) + self.assertEqual(len(values), len(set(values))) + + def test_kernel_mapping_priority(self): + cases = { + "void vllm::moe::topkGating": "moe_router", + "ncclDevKernel_AllReduce": "collective", + "flash_fwd_kernel": "attention", + "nvjet_sm90_tst": "moe_gemm", + "argmax_kernel": "sampler", + "cutlass_gemm": "dense_gemm", + "triton_red_fused_add_rms_norm": "norm_elementwise", + "cache_swap_kernel": "kv_memory", + "unknown": "other", + } + self.assertEqual( + {name: controller.classify_kernel(name) for name in cases}, cases + ) + + def test_atomic_state_replacement(self): + with tempfile.TemporaryDirectory() as tmp: + path = Path(tmp) / "state.json" + controller.atomic_json(path, {"schema": 1, "value": 1}) + controller.atomic_json(path, {"schema": 1, "value": 2}) + self.assertEqual(json.loads(path.read_text())["value"], 2) + self.assertFalse(list(path.parent.glob("*.tmp.*"))) + + def test_fingerprint_survives_json_roundtrip(self): + original_run_text = controller.run_text + original_hash = controller.sha256_file + try: + controller.run_text = lambda *args, **kwargs: "deadbeef\n" + controller.sha256_file = lambda path: "a" * 64 + fingerprint = controller.make_fingerprint() + finally: + controller.run_text = original_run_text + controller.sha256_file = original_hash + self.assertEqual(json.loads(json.dumps(fingerprint)), fingerprint) + self.assertEqual(set(fingerprint["cpu_map"]), {str(i) for i in range(8)}) + + def test_server_shutdown_signals_parent_before_group(self): + process = SimpleNamespace(pid=12345, poll=lambda: None) + with ( + mock.patch.object(controller.os, "kill") as kill, + mock.patch.object(controller.os, "killpg") as killpg, + mock.patch.object(controller, "_process_group_alive", return_value=False), + ): + controller.stop_servers([process]) + kill.assert_called_once_with(12345, controller.signal.SIGINT) + killpg.assert_not_called() + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/runs/opprof-phase3/provenance/verify_shutdown_footer.py b/runs/opprof-phase3/provenance/verify_shutdown_footer.py new file mode 100644 index 0000000..9463818 --- /dev/null +++ b/runs/opprof-phase3/provenance/verify_shutdown_footer.py @@ -0,0 +1,353 @@ +#!/usr/bin/env python3 +"""One-GPU verification of API-parent-first OpProf shutdown.""" + +from __future__ import annotations + +import json +import os +import shlex +import shutil +import signal +import subprocess +import time +import urllib.request +from pathlib import Path +from typing import Any + +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_DIR = WORKDIR / "runs/e-b-shutdown-verification" +SOURCE = Path( + "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0" +) +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +MANIFEST = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl") +PORT = 8010 + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_name(path.name + f".tmp.{os.getpid()}") + with tmp.open("w", encoding="utf-8") as f: + json.dump(value, f, sort_keys=True, indent=2) + f.write("\n") + f.flush() + os.fsync(f.fileno()) + os.replace(tmp, path) + + +def run_text(command: list[str], check: bool = True) -> str: + result = subprocess.run( + command, text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {shlex.join(command)}\n" + f"{result.stdout}" + ) + return result.stdout + + +def gpu_rows() -> list[dict[str, int]]: + output = run_text( + [ + "nvidia-smi", + "--query-gpu=index,memory.used,utilization.gpu", + "--format=csv,noheader,nounits", + ] + ) + rows = [] + for line in output.strip().splitlines(): + index, memory, utilization = [int(part.strip()) for part in line.split(",")] + rows.append( + {"index": index, "memory_mib": memory, "utilization_pct": utilization} + ) + return rows + + +def compute_apps() -> str: + return run_text( + [ + "nvidia-smi", + "--query-compute-apps=gpu_uuid,pid,process_name,used_memory", + "--format=csv,noheader,nounits", + ], + check=False, + ).strip() + + +def group_alive(pgid: int, process: subprocess.Popen[Any] | None = None) -> bool: + if process is not None: + process.poll() # reap an exited API parent before probing its group + try: + os.killpg(pgid, 0) + return True + except ProcessLookupError: + return False + + +def wait_ready(process: subprocess.Popen[Any], timeout: float = 300) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + if process.poll() is not None: + raise RuntimeError("server exited before readiness") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{PORT}/health", timeout=1 + ) as response: + if response.status == 200: + return + except Exception: + pass + time.sleep(1) + raise TimeoutError("server readiness timeout") + + +def shutdown_parent_first(process: subprocess.Popen[Any]) -> dict[str, Any]: + called_at = time.time() + os.kill(process.pid, signal.SIGINT) + deadline = time.monotonic() + 120 + while time.monotonic() < deadline and group_alive(process.pid, process): + time.sleep(1) + graceful = not group_alive(process.pid, process) + if not graceful: + for sig in (signal.SIGTERM, signal.SIGKILL): + try: + os.killpg(process.pid, sig) + except ProcessLookupError: + break + time.sleep(5) + return { + "api_parent_pid": process.pid, + "sigint_wall_time": called_at, + "group_gone_wall_time": time.time(), + "graceful": graceful, + "returncode": process.poll(), + } + + +def wait_stable_zero() -> list[list[dict[str, int]]]: + samples = [] + consecutive = 0 + deadline = time.monotonic() + 120 + while time.monotonic() < deadline and consecutive < 3: + rows = gpu_rows() + samples.append(rows) + zero = rows[0]["memory_mib"] == 0 and not compute_apps() + consecutive = consecutive + 1 if zero else 0 + if consecutive < 3: + time.sleep(5) + if consecutive < 3: + raise RuntimeError("GPU0 did not reach three stable zero samples") + return samples + + +def validate_footer() -> dict[str, Any]: + files = sorted((RUN_DIR / "opprof").glob("*.jsonl")) + if len(files) != 1: + raise RuntimeError(f"expected one OpProf JSONL, found {len(files)}") + lines = files[0].read_text().splitlines() + decoded = [json.loads(line) for line in lines] + if not decoded or decoded[-1].get("record_type") != "footer": + raise RuntimeError("Layer-1 footer missing") + records, footer = decoded[:-1], decoded[-1] + indices = [record["step_index"] for record in records] + invariants = { + "all_schema_1": all(item.get("schema") == 1 for item in decoded), + "one_footer_last": sum( + item.get("record_type") == "footer" for item in decoded + ) + == 1, + "steps_contiguous": indices == list(range(len(indices))), + "written_matches_records": footer["written_records"] == len(records), + "encoded_balanced": footer["encoded_records"] + == footer["written_records"] + footer["dropped_records"], + "zero_drops": footer["dropped_records"] == 0 + and all(record["dropped_records_before"] == 0 for record in records), + } + if not all(invariants.values()): + raise RuntimeError(f"footer accounting invalid: {invariants}, {footer}") + return { + "path": str(files[0]), + "bytes": files[0].stat().st_size, + "records": len(records), + "footer": footer, + "invariants": invariants, + } + + +def main() -> None: + if RUN_DIR.exists(): + raise RuntimeError(f"refusing to overwrite {RUN_DIR}") + RUN_DIR.mkdir(parents=True) + state: dict[str, Any] = { + "schema": 1, + "status": "preflight", + "started_at": time.time(), + "gpu": 0, + } + atomic_json(RUN_DIR / "state.json", state) + before = gpu_rows() + if before[0]["memory_mib"] != 0 or before[0]["utilization_pct"] != 0: + raise RuntimeError(f"GPU0 is not idle: {before[0]}") + if compute_apps(): + raise RuntimeError("a compute process exists before verification") + (RUN_DIR / "gpu-before.json").write_text(json.dumps(before, indent=2) + "\n") + (RUN_DIR / "clocks-before.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"]) + ) + profile_config = { + "profiler": "torch", + "torch_profiler_dir": "/tmp/wjh-opprof-p3-footer-verify", + "ignore_frontend": True, + "wait_iterations": 0, + "warmup_iterations": 2, + "active_iterations": 8, + } + trace_dir = Path(profile_config["torch_profiler_dir"]) + if trace_dir.exists(): + shutil.rmtree(trace_dir) + trace_dir.mkdir() + server_command = [ + "taskset", + "-c", + "0-19", + str(VENV / "bin/vllm"), + "serve", + str(MODEL), + "--host", + "127.0.0.1", + "--port", + str(PORT), + "--tensor-parallel-size", + "1", + "--enable-chunked-prefill", + "--enable-prefix-caching", + "--profiler-config", + json.dumps(profile_config, separators=(",", ":")), + ] + client_command = [ + "taskset", + "-c", + "0-19", + str(VENV / "bin/python"), + str(CLIENT), + "run", + "--manifest", + str(MANIFEST), + "--base-url", + f"http://127.0.0.1:{PORT}", + "--model", + str(MODEL), + "--load-point", + "saturation", + "--request-rate", + "inf", + "--max-concurrency", + "256", + "--ignore-eos", + "--temperature", + "0", + "--warmup-seconds", + "30", + "--clean-segment-seconds", + "40", + "--num-clean-segments", + "3", + "--drain-timeout-seconds", + "120", + "--workload-seed", + "20260712", + "--result-dir", + str(RUN_DIR / "client"), + ] + with (RUN_DIR / "commands.log").open("w") as f: + f.write( + "GPU_COMMAND shutdown footer server: " + + shlex.join(server_command) + + " ; expected=60-180s startup + 150-180s load\n" + ) + f.write( + "GPU_COMMAND shutdown footer client: " + + shlex.join(client_command) + + " ; expected=30s warmup + 120s clean + drain\n" + ) + server_log = (RUN_DIR / "server.log").open("ab", buffering=0) + env = os.environ.copy() + env.update( + { + "CUDA_VISIBLE_DEVICES": "0", + "VLLM_OPPROF_DIR": str(RUN_DIR / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + server: subprocess.Popen[Any] | None = None + try: + print( + "GPU_COMMAND shutdown-fix verification: P01/C00 GPU0, " + "30s warmup + 120s clean, expected 4-6 wall-min", + flush=True, + ) + server = subprocess.Popen( + server_command, + cwd=SOURCE, + env=env, + stdout=server_log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + state.update({"status": "server_starting", "server_pid": server.pid}) + atomic_json(RUN_DIR / "state.json", state) + wait_ready(server) + state.update({"status": "client_running", "server_ready_at": time.time()}) + atomic_json(RUN_DIR / "state.json", state) + with (RUN_DIR / "client.log").open("ab", buffering=0) as client_log: + result = subprocess.run( + client_command, + cwd=WORKDIR, + stdout=client_log, + stderr=subprocess.STDOUT, + ) + if result.returncode: + raise RuntimeError(f"client exited {result.returncode}") + state["status"] = "parent_first_shutdown" + atomic_json(RUN_DIR / "state.json", state) + shutdown = shutdown_parent_first(server) + if not shutdown["graceful"]: + raise RuntimeError(f"server required escalation: {shutdown}") + footer = validate_footer() + zero_samples = wait_stable_zero() + client_result = json.loads((RUN_DIR / "client/result.json").read_text()) + result_json = { + "schema": 1, + "status": "pass", + "gpu": 0, + "shutdown": shutdown, + "footer": footer, + "clean": client_result["clean"], + "failed_records": client_result["failed_records"], + "gpu_zero_samples": zero_samples, + "gpu_seconds": zero_samples and time.time() - state["started_at"], + } + atomic_json(RUN_DIR / "result.json", result_json) + state.update({"status": "complete", "completed_at": time.time()}) + atomic_json(RUN_DIR / "state.json", state) + print(json.dumps(result_json, sort_keys=True), flush=True) + except Exception as error: + state.update({"status": "failed", "failure": repr(error)}) + atomic_json(RUN_DIR / "state.json", state) + if server is not None and group_alive(server.pid, server): + try: + os.killpg(server.pid, signal.SIGKILL) + except ProcessLookupError: + pass + raise + finally: + server_log.close() + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase3/provenance/verify_sidecar_shutdown.py b/runs/opprof-phase3/provenance/verify_sidecar_shutdown.py new file mode 100644 index 0000000..084e403 --- /dev/null +++ b/runs/opprof-phase3/provenance/verify_sidecar_shutdown.py @@ -0,0 +1,392 @@ +#!/usr/bin/env python3 +"""GPU verification of graceful and hard-kill OpProf accounting.""" + +from __future__ import annotations + +import json +import os +import shlex +import signal +import subprocess +import time +import urllib.request +from pathlib import Path +from typing import Any + +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/e-b-sidecar-verification" +SOURCE = Path( + "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0" +) +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +MANIFEST = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests/P01.jsonl") +FLUSH_INTERVAL_SECONDS = 1.0 +CHECKPOINT_TOLERANCE_SECONDS = 0.1 + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_name(f"{path.name}.tmp-{os.getpid()}") + with temporary.open("w", encoding="utf-8") as output: + json.dump(value, output, indent=2, sort_keys=True) + output.write("\n") + output.flush() + os.fsync(output.fileno()) + os.replace(temporary, path) + + +def run_text(command: list[str], check: bool = True) -> str: + result = subprocess.run( + command, + text=True, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + ) + if check and result.returncode: + raise RuntimeError( + f"command failed ({result.returncode}): {shlex.join(command)}\n" + f"{result.stdout}" + ) + return result.stdout + + +def gpu_rows() -> list[dict[str, int]]: + output = run_text( + [ + "nvidia-smi", + "--query-gpu=index,memory.used,utilization.gpu", + "--format=csv,noheader,nounits", + ] + ) + rows = [] + for line in output.strip().splitlines(): + index, memory, utilization = (int(value.strip()) for value in line.split(",")) + rows.append( + { + "index": index, + "memory_mib": memory, + "utilization_pct": utilization, + } + ) + return rows + + +def compute_apps() -> str: + return run_text( + [ + "nvidia-smi", + "--query-compute-apps=gpu_uuid,pid,process_name,used_memory", + "--format=csv,noheader,nounits", + ], + check=False, + ).strip() + + +def assert_idle() -> list[dict[str, int]]: + rows = gpu_rows() + if any(row["memory_mib"] or row["utilization_pct"] for row in rows): + raise RuntimeError(f"dash0 is not GPU-idle: {rows}") + applications = compute_apps() + if applications: + raise RuntimeError(f"compute applications present: {applications}") + return rows + + +def wait_ready(process: subprocess.Popen[Any], port: int) -> None: + deadline = time.monotonic() + 300 + while time.monotonic() < deadline: + if process.poll() is not None: + raise RuntimeError("server exited before readiness") + try: + with urllib.request.urlopen( + f"http://127.0.0.1:{port}/health", timeout=1 + ) as response: + if response.status == 200: + return + except Exception: + pass + time.sleep(1) + raise TimeoutError("server readiness timeout") + + +def wait_zero() -> tuple[list[list[dict[str, int]]], float]: + samples: list[list[dict[str, int]]] = [] + consecutive = 0 + deadline = time.monotonic() + 180 + while time.monotonic() < deadline and consecutive < 3: + rows = gpu_rows() + samples.append(rows) + zero = all(row["memory_mib"] == 0 for row in rows) and not compute_apps() + consecutive = consecutive + 1 if zero else 0 + if consecutive < 3: + time.sleep(2) + if consecutive < 3: + raise RuntimeError("GPUs did not reach three stable zero samples") + return samples, time.time() + + +def wait_fresh_sidecar(run_dir: Path) -> dict[str, Any]: + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + files = sorted((run_dir / "opprof").glob("*.jsonl.footer.json")) + if len(files) == 1: + sidecar = json.loads(files[0].read_text()) + age = time.time_ns() - sidecar["checkpoint_wall_ns"] + if 0 <= age <= 250_000_000: + return sidecar + time.sleep(0.02) + raise TimeoutError("could not observe a fresh OpProf sidecar") + + +def validate_accounting( + run_dir: Path, mode: str, termination_wall_ns: int +) -> dict[str, Any]: + streams = sorted((run_dir / "opprof").glob("*.jsonl")) + sidecars = sorted((run_dir / "opprof").glob("*.jsonl.footer.json")) + if len(streams) != 1 or len(sidecars) != 1: + raise RuntimeError( + f"expected one stream/sidecar, got {len(streams)}/{len(sidecars)}" + ) + raw = streams[0].read_bytes() + if not raw.endswith(b"\n"): + raise RuntimeError("partial final JSONL line") + decoded = [json.loads(line) for line in raw.splitlines()] + footers = [row for row in decoded if row.get("record_type") == "footer"] + records = [row for row in decoded if row.get("record_type") != "footer"] + sidecar = json.loads(sidecars[0].read_text()) + indices = [row["step_index"] for row in records] + checkpoint_age = ( + termination_wall_ns - sidecar["checkpoint_wall_ns"] + ) / 1e9 + common = { + "all_schema_1": all(row.get("schema") == 1 for row in decoded) + and sidecar.get("schema") == 1, + "steps_contiguous": indices == list(range(len(indices))), + "written_matches_records": sidecar["written_records"] == len(records), + "encoded_balanced": sidecar["encoded_records"] + == sidecar["written_records"] + sidecar["dropped_records"], + "last_step_matches": bool(records) + and sidecar["last_step_index"] == records[-1]["step_index"], + "zero_drops": sidecar["dropped_records"] == 0 + and all(row["dropped_records_before"] == 0 for row in records), + } + if mode == "graceful": + footer_ok = len(footers) == 1 and decoded[-1] is footers[0] + agreement = footer_ok and all( + footers[0][counter] == sidecar[counter] + for counter in ("encoded_records", "written_records", "dropped_records") + ) + specific = { + "one_footer_last": footer_ok, + "final_sidecar": sidecar["final"] is True, + "footer_sidecar_agree": agreement, + } + else: + specific = { + "no_in_stream_footer": not footers, + "checkpoint_sidecar": sidecar["final"] is False, + "checkpoint_within_bound": checkpoint_age + <= FLUSH_INTERVAL_SECONDS + CHECKPOINT_TOLERANCE_SECONDS, + } + invariants = {**common, **specific} + if not all(invariants.values()): + raise RuntimeError( + f"{mode} accounting invalid: {invariants}; sidecar={sidecar}" + ) + return { + "stream": str(streams[0]), + "sidecar": str(sidecars[0]), + "bytes": len(raw), + "records": len(records), + "footer_count": len(footers), + "checkpoint_age_seconds": checkpoint_age, + "counters": { + key: sidecar[key] + for key in ("encoded_records", "written_records", "dropped_records") + }, + "last_step_index": sidecar["last_step_index"], + "invariants": invariants, + } + + +def run_trial(mode: str, port: int) -> dict[str, Any]: + run_dir = RUN_ROOT / mode + if run_dir.exists(): + raise RuntimeError(f"refusing to overwrite {run_dir}") + run_dir.mkdir(parents=True) + before = assert_idle() + atomic_json(run_dir / "gpu-before.json", before) + (run_dir / "clocks-before.txt").write_text( + run_text(["nvidia-smi", "-q", "-d", "CLOCK"]) + ) + server_command = [ + "taskset", + "-c", + "0-19", + str(VENV / "bin/vllm"), + "serve", + str(MODEL), + "--host", + "127.0.0.1", + "--port", + str(port), + "--tensor-parallel-size", + "1", + "--enable-chunked-prefill", + "--enable-prefix-caching", + "--shutdown-timeout", + "120", + ] + client_command = [ + "taskset", + "-c", + "0-19", + str(VENV / "bin/python"), + str(CLIENT), + "run", + "--manifest", + str(MANIFEST), + "--base-url", + f"http://127.0.0.1:{port}", + "--model", + str(MODEL), + "--load-point", + "saturation", + "--request-rate", + "inf", + "--max-concurrency", + "256", + "--ignore-eos", + "--temperature", + "0", + "--warmup-seconds", + "20", + "--clean-segment-seconds", + "40", + "--num-clean-segments", + "3", + "--drain-timeout-seconds", + "120", + "--workload-seed", + "20260712", + "--result-dir", + str(run_dir / "client"), + ] + with (run_dir / "commands.log").open("w", encoding="utf-8") as output: + output.write( + f"GPU_COMMAND {mode} server: {shlex.join(server_command)}; " + "expected=60-180s startup + 140-180s load\n" + ) + output.write( + f"GPU_COMMAND {mode} client: {shlex.join(client_command)}; " + "expected=20s warmup + 120s clean + drain\n" + ) + environment = os.environ.copy() + environment.update( + { + "CUDA_VISIBLE_DEVICES": "0", + "VLLM_OPPROF_DIR": str(run_dir / "opprof"), + "HF_HUB_OFFLINE": "1", + "TRANSFORMERS_OFFLINE": "1", + "PYTHONUNBUFFERED": "1", + } + ) + started = time.time() + server_log = (run_dir / "server.log").open("ab", buffering=0) + server: subprocess.Popen[Any] | None = None + try: + print( + f"GPU_COMMAND sidecar-{mode}: P01/C00 GPU0, 20s warmup + " + "120s clean, expected 4-6 wall-min", + flush=True, + ) + server = subprocess.Popen( + server_command, + cwd=SOURCE, + env=environment, + stdout=server_log, + stderr=subprocess.STDOUT, + start_new_session=True, + ) + atomic_json( + run_dir / "state.json", + {"status": "server_starting", "server_pid": server.pid}, + ) + wait_ready(server, port) + with (run_dir / "client.log").open("ab", buffering=0) as client_log: + client = subprocess.run( + client_command, + cwd=WORKDIR, + stdout=client_log, + stderr=subprocess.STDOUT, + ) + if client.returncode: + raise RuntimeError(f"client exited {client.returncode}") + if mode == "graceful": + termination_wall_ns = time.time_ns() + os.kill(server.pid, signal.SIGINT) + server.wait(timeout=150) + log_text = (run_dir / "server.log").read_text(errors="replace") + if "mode=drain timeout=120s" not in log_text: + raise RuntimeError("official drain-mode log not observed") + else: + wait_fresh_sidecar(run_dir) + termination_wall_ns = time.time_ns() + os.killpg(server.pid, signal.SIGKILL) + server.wait(timeout=30) + zero_samples, zero_at = wait_zero() + accounting = validate_accounting(run_dir, mode, termination_wall_ns) + client_result = json.loads((run_dir / "client/result.json").read_text()) + result = { + "schema": 1, + "status": "pass", + "mode": mode, + "server_returncode": server.returncode, + "termination_wall_ns": termination_wall_ns, + "accounting": accounting, + "clean": client_result["clean"], + "failed_records": client_result["failed_records"], + "gpu_zero_samples": zero_samples, + "gpu_seconds": zero_at - started, + } + atomic_json(run_dir / "result.json", result) + atomic_json(run_dir / "state.json", {"status": "complete"}) + return result + except Exception as error: + atomic_json( + run_dir / "state.json", + {"status": "failed", "failure": repr(error)}, + ) + if server is not None and server.poll() is None: + try: + os.killpg(server.pid, signal.SIGKILL) + except ProcessLookupError: + pass + wait_zero() + raise + finally: + server_log.close() + + +def main() -> None: + if RUN_ROOT.exists(): + raise RuntimeError(f"refusing to overwrite {RUN_ROOT}") + RUN_ROOT.mkdir(parents=True) + state: dict[str, Any] = {"schema": 1, "status": "running", "results": {}} + atomic_json(RUN_ROOT / "state.json", state) + for offset, mode in enumerate(("graceful", "hard-kill")): + result = run_trial(mode, 8010 + offset) + state["results"][mode] = result + atomic_json(RUN_ROOT / "state.json", state) + state["status"] = "complete" + state["gpu_seconds"] = sum( + result["gpu_seconds"] for result in state["results"].values() + ) + atomic_json(RUN_ROOT / "state.json", state) + print(json.dumps(state, sort_keys=True), flush=True) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase5/analyze_phase5.py b/runs/opprof-phase5/analyze_phase5.py new file mode 100644 index 0000000..5c3fcf7 --- /dev/null +++ b/runs/opprof-phase5/analyze_phase5.py @@ -0,0 +1,537 @@ +#!/usr/bin/env python3 +"""Frozen Phase-5 bridge/control ledger analysis.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +from pathlib import Path +from typing import Any + +import numpy as np + + +SEED = 20260716 +RESAMPLES = 100_000 +RATE = 0.4725 +ARMS = ("base", "A1", "A2", "A3", "A4") +MECHANISMS = ("A1", "A2", "A3", "A4") +CAPTURE_SIZES = { + 1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, + 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, + 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, + 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, +} + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def numeric(values: list[float | int | None]) -> dict[str, Any]: + finite = [float(value) for value in values if value is not None and math.isfinite(float(value))] + return { + "n": len(values), + "finite_n": len(finite), + "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, + "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def ci(draws: np.ndarray) -> list[float]: + low, high = np.quantile(draws, [0.025, 0.975]) + return [float(low), float(high)] + + +def cv(values: list[float]) -> float: + array = np.asarray(values, dtype=np.float64) + mean = float(array.mean()) + if mean == 0: + return 0.0 if np.all(array == 0) else math.inf + return float(array.std(ddof=0) / mean) + + +def parse_run(run_dir: Path) -> dict[str, Any]: + result = json.loads((run_dir / "client/result.json").read_text()) + complete = json.loads((run_dir / "run-complete.json").read_text()) + t0 = int(result["t0_mono_ns"]) + clean_start = float(result["clean"]["start_s"]) + clean_end = float(result["clean"]["end_s"]) + stream = next((run_dir / "opprof").glob("*.jsonl")) + records = [] + for line in stream.read_text().splitlines(): + item = json.loads(line) + if "step_index" not in item: + continue + relative = (int(item["submit_mono_ns"]) - t0) / 1e9 + if clean_start <= relative < clean_end: + item["relative_s"] = relative + records.append(item) + blocks = [] + decode_means = [] + waiting_means = [] + for block_index in range(48): + start = clean_start + 5 * block_index + selected = [item for item in records if start <= item["relative_s"] < start + 5] + if not selected: + # Fixed-time blocks are the resampling unit. A rate-following trace + # can legitimately have an idle block, which contributes no tokens + # and no model-step time but must remain in the arrival-variance + # distribution. + blocks.append([0, 0.0]) + decode_means.append(0.0) + waiting_means.append(0.0) + continue + tokens = sum(int(item["prefill_tokens"]) + int(item["decode_tokens"]) for item in selected) + duration = sum( + (int(item["complete_mono_ns"]) - int(item["submit_mono_ns"])) / 1e6 + for item in selected + ) + blocks.append([tokens, duration]) + decode_means.append(float(np.mean([int(item["decode_batch_size"]) for item in selected]))) + waiting_means.append(float(np.mean([int(item["queues"]["waiting"]) for item in selected]))) + pure_decode = [ + item for item in records + if int(item["prefill_tokens"]) == 0 and int(item["decode_batch_size"]) > 0 + ] + pure_bucket = sum(int(item["cudagraph"]["bucket_tokens"]) for item in pure_decode) + pure_padding = sum(int(item["cudagraph"]["padding_tokens"]) for item in pure_decode) + support = sorted({int(item["decode_batch_size"]) for item in pure_decode}) + covered = sum(int(item["decode_batch_size"]) in CAPTURE_SIZES for item in pure_decode) + prefix_queries = 0 + prefix_hits = 0 + prefix_present = 0 + for item in records: + local = item.get("prefix", {}).get("local") + if local is None: + continue + prefix_present += 1 + prefix_queries += int(local.get("queries", 0)) + prefix_hits += int(local.get("hits", 0)) + block_array = np.asarray(blocks, dtype=np.float64) + return { + "run_id": complete["run_id"], + "run_dir": str(run_dir), + "stream_sha256": sha256_file(stream), + "blocks": block_array, + "token_efficiency_per_ms": float(block_array[:, 0].sum() / block_array[:, 1].sum()), + "clean_steps": len(records), + "clean_duration_s": clean_end - clean_start, + "clean_failed": int(result["clean"]["failed"]), + "offered_rps": float(result["clean"]["offered_rps"]), + "drain_seconds": float(result["drain_seconds"]), + "warmup_completions": int(complete["client"]["warmup_completions"]), + "warmup_gate_branch": complete["client"].get("warmup_gate_branch", "P3-pre-A-P5-1"), + "warmup_stability": complete["client"].get("warmup_stability"), + "cold_start_gate": complete["client"].get("cold_start_gate"), + "decode_B_block_cv": cv(decode_means), + "waiting_block_cv": cv(waiting_means), + "pure_decode_steps": len(pure_decode), + "pure_decode_support": support, + "pure_decode_support_coverage": covered / len(pure_decode), + "pure_decode_padding_tokens": pure_padding, + "pure_decode_bucket_tokens": pure_bucket, + "pure_decode_padding_fraction": pure_padding / pure_bucket, + "prefix_present_steps": prefix_present, + "prefix_queries": prefix_queries, + "prefix_hits": prefix_hits, + "prefix_hit_ratio": prefix_hits / prefix_queries if prefix_queries else 0.0, + "layer1_invariants": complete["layer1"]["invariants"], + "client_invariants": complete["client"]["invariants"], + "server_invariants": complete["server_invariants"], + "drain_quarantined": bool(complete["drain_quarantined"]), + } + + +def hierarchical_draws(runs: list[dict[str, Any]], rng: np.random.Generator) -> np.ndarray: + count = len(runs) + output = np.empty(RESAMPLES, dtype=np.float64) + for start in range(0, RESAMPLES, 5000): + size = min(5000, RESAMPLES - start) + token_sum = np.zeros(size, dtype=np.float64) + duration_sum = np.zeros(size, dtype=np.float64) + selected_runs = rng.integers(0, count, size=(size, count)) + for position in range(count): + block_indices = rng.integers(0, 48, size=(size, 48)) + for run_index, run in enumerate(runs): + mask = selected_runs[:, position] == run_index + if not np.any(mask): + continue + blocks = run["blocks"] + choices = block_indices[mask] + token_sum[mask] += blocks[choices, 0].sum(axis=1) + duration_sum[mask] += blocks[choices, 1].sum(axis=1) + output[start : start + size] = token_sum / duration_sum + return output + + +def point_efficiency(runs: list[dict[str, Any]]) -> float: + tokens = sum(float(run["blocks"][:, 0].sum()) for run in runs) + duration = sum(float(run["blocks"][:, 1].sum()) for run in runs) + return tokens / duration + + +def two_sided_p(draws: np.ndarray) -> float: + return float(min(1.0, 2 * min(np.mean(draws <= 0), np.mean(draws >= 0)))) + + +def holm(pvalues: dict[str, float]) -> dict[str, float]: + ordered = sorted(pvalues, key=pvalues.get) + adjusted: dict[str, float] = {} + running = 0.0 + total = len(ordered) + for rank, key in enumerate(ordered): + running = max(running, (total - rank) * pvalues[key]) + adjusted[key] = min(1.0, running) + return adjusted + + +def discover_primary(root: Path) -> dict[str, list[dict[str, Any]]]: + result = {arm: [] for arm in ARMS} + for marker in sorted((root / "primary").glob("*/moderate/run-complete.json")): + if "background" in str(marker): + continue + parsed = parse_run(marker.parent) + arm = parsed["run_id"].split("-r", 1)[0] + if arm in result: + result[arm].append(parsed) + if any(len(result[arm]) != 3 for arm in ARMS): + raise RuntimeError(f"primary replication mismatch: { {key:len(value) for key,value in result.items()} }") + return result + + +def control_runs(root: Path, p3root: Path, pattern: str) -> tuple[list[dict[str, Any]], str]: + fresh = sorted((root / "primary").glob(f"control-{pattern}-r*-C00/moderate/run-complete.json")) + if fresh: + if len(fresh) != 3: + raise RuntimeError(f"partial fresh control set: {pattern}: {len(fresh)}") + return [parse_run(path.parent) for path in fresh], "P5-rerun" + return [parse_run(p3root / f"primary/{pattern}-C00/moderate")], "P3-reused" + + +def aggregate_aux(runs: list[dict[str, Any]], key: str) -> float: + return float(np.mean([float(run[key]) for run in runs])) + + +def analyze(root: Path, p3root: Path, private: Path) -> dict[str, Any]: + primary = discover_primary(root) + p3_base = [parse_run(p3root / "primary/P10-C00/moderate")] + controls = {} + control_source = {} + for pattern in ("P03", "P04"): + controls[pattern], control_source[pattern] = control_runs(root, p3root, pattern) + rng = np.random.default_rng(SEED) + draws = {arm: hierarchical_draws(primary[arm], rng) for arm in ARMS} + p3_base_draws = hierarchical_draws(p3_base, rng) + control_draws = {pattern: hierarchical_draws(controls[pattern], rng) for pattern in controls} + points = {arm: point_efficiency(primary[arm]) for arm in ARMS} + p3_base_point = point_efficiency(p3_base) + control_points = {pattern: point_efficiency(controls[pattern]) for pattern in controls} + + bridge_draws = draws["A3"] - p3_base_draws + bridge_point = points["A3"] - p3_base_point + bridge_ci = ci(bridge_draws) + bridge = { + "point_delta": bridge_point, + "relative_abs_delta": abs(bridge_point) / p3_base_point, + "ci95": bridge_ci, + "within_3pct": abs(bridge_point) / p3_base_point <= 0.03, + "ci_contains_zero": bridge_ci[0] <= 0 <= bridge_ci[1], + } + bridge["reuse_passed"] = bridge["within_3pct"] and bridge["ci_contains_zero"] + + deltas = {arm: draws[arm] - draws["base"] for arm in MECHANISMS} + raw_p = {arm: two_sided_p(deltas[arm]) for arm in MECHANISMS} + holm_p = holm(raw_p) + manifests = { + name: json.loads((private / f"P10-{name}.jsonl.summary.json").read_text()) + for name in ("base", "A1", "A3") + } + base_prefix = aggregate_aux(primary["base"], "prefix_hit_ratio") + a1_prefix = aggregate_aux(primary["A1"], "prefix_hit_ratio") + base_padding = aggregate_aux(primary["base"], "pure_decode_padding_fraction") + a2_padding = aggregate_aux(primary["A2"], "pure_decode_padding_fraction") + padding_reduction = (base_padding - a2_padding) / base_padding if base_padding > 0 else 0.0 + manipulations = { + "A1": { + "passed": ( + manifests["base"]["r16"] - manifests["A1"]["r16"] >= 0.15 + and (manifests["base"]["r16"] - manifests["A1"]["r16"]) / manifests["base"]["r16"] >= 0.20 + and manifests["A1"]["max_added_delay_seconds"] <= 64 + and abs(a1_prefix - base_prefix) <= 0.01 + ), + "base_R16": manifests["base"]["r16"], + "ablated_R16": manifests["A1"]["r16"], + "prefix_hit_ratio_delta": a1_prefix - base_prefix, + "max_added_delay_seconds": manifests["A1"]["max_added_delay_seconds"], + }, + "A2": { + "passed": aggregate_aux(primary["A2"], "pure_decode_support_coverage") >= 0.99 and padding_reduction >= 0.90, + "support_coverage": aggregate_aux(primary["A2"], "pure_decode_support_coverage"), + "base_padding_fraction": base_padding, + "ablated_padding_fraction": a2_padding, + "padding_reduction": padding_reduction, + "observed_support": sorted({value for run in primary["A2"] for value in run["pure_decode_support"]}), + }, + "A3": { + "passed": ( + aggregate_aux(primary["A3"], "decode_B_block_cv") < aggregate_aux(primary["base"], "decode_B_block_cv") + and aggregate_aux(primary["A3"], "waiting_block_cv") < aggregate_aux(primary["base"], "waiting_block_cv") + ), + "base_decode_B_cv": aggregate_aux(primary["base"], "decode_B_block_cv"), + "ablated_decode_B_cv": aggregate_aux(primary["A3"], "decode_B_block_cv"), + "base_waiting_cv": aggregate_aux(primary["base"], "waiting_block_cv"), + "ablated_waiting_cv": aggregate_aux(primary["A3"], "waiting_block_cv"), + }, + "A4": { + "passed": sum(run["prefix_queries"] for run in primary["A4"]) == 0 and sum(run["prefix_hits"] for run in primary["A4"]) == 0, + "prefix_queries": sum(run["prefix_queries"] for run in primary["A4"]), + "prefix_hits": sum(run["prefix_hits"] for run in primary["A4"]), + }, + } + + ledgers = {} + for control in ("P03", "P04"): + denominator = control_draws[control] - draws["base"] + point_denominator = control_points[control] - points["base"] + denominator_ci = ci(denominator) + stable_denominator = bool( + point_denominator > 0 + and denominator_ci[0] > 0 + and np.mean(denominator <= 0) <= 0.05 + ) + rows = {} + share_draws = {} + for arm in MECHANISMS: + share_draws[arm] = deltas[arm] / denominator + point = (points[arm] - points["base"]) / point_denominator + interval = ci(share_draws[arm]) + reportable = stable_denominator and manipulations[arm]["passed"] + rows[arm] = { + "delta_E": points[arm] - points["base"], + "delta_E_ci95": ci(deltas[arm]), + "share": point if reportable else None, + "share_ci95": interval if reportable else None, + "diagnostic_share": point, + "diagnostic_share_ci95": interval, + "share_status": ( + "REPORTABLE" + if reportable + else ( + "N/A—unstable control denominator" + if not stable_denominator + else "N/A—manipulation failed" + ) + ), + "p_two_sided": raw_p[arm], + "p_holm": holm_p[arm], + "manipulation_passed": manipulations[arm]["passed"], + } + residual_draws = 1.0 - sum(share_draws.values()) + residual_point = 1.0 - sum(row["diagnostic_share"] for row in rows.values()) + ledgers[control] = { + "status": "EVALUABLE" if stable_denominator else "INCONCLUSIVE—unstable denominator", + "control_source": control_source[control], + "control_E": control_points[control], + "base_E": points["base"], + "gap_E": point_denominator, + "gap_ci95": denominator_ci, + "denominator_nonpositive_fraction": float(np.mean(denominator <= 0)), + "stable_denominator": stable_denominator, + "mechanisms": rows, + "diagnostic_share_sum": sum(row["diagnostic_share"] for row in rows.values()), + "residual_interaction": residual_point, + "residual_interaction_ci95": ci(residual_draws), + "residual_status": ( + "REPORTABLE" + if stable_denominator and all(item["passed"] for item in manipulations.values()) + else "DIAGNOSTIC_ONLY—incomplete official share ledger" + ), + } + + dominance = {} + for arm in MECHANISMS: + per_control = {} + for control in ("P03", "P04"): + row = ledgers[control]["mechanisms"][arm] + if not ledgers[control]["stable_denominator"] or not row["manipulation_passed"]: + per_control[control] = "NOT_EVALUABLE" + continue + direction_ok = row["delta_E"] > 0 if arm != "A4" else True + per_control[control] = ( + row["share"] >= 0.30 + and row["share_ci95"][0] > 0.15 + and row["p_holm"] < 0.05 + and direction_ok + and row["manipulation_passed"] + ) + dominance[arm] = { + "per_control": per_control, + "verdict": ( + "NOT EVALUABLE" + if any(value == "NOT_EVALUABLE" for value in per_control.values()) + else ( + "DOMINANT" + if all(per_control.values()) + else ("CONTROL-SENSITIVE" if any(per_control.values()) else "NOT DOMINANT") + ) + ), + } + + all_runs = [run for arm in ARMS for run in primary[arm]] + share_widths = [ + ledgers[control]["mechanisms"][arm]["diagnostic_share_ci95"][1] + - ledgers[control]["mechanisms"][arm]["diagnostic_share_ci95"][0] + for control in ("P03", "P04") for arm in MECHANISMS + ] + state = json.loads((root / "controller-state.json").read_text()) + amendment_evidence_path = root / "a-p5-1-retained-audit.jsonl" + amendment_evidence = [] + if amendment_evidence_path.exists(): + amendment_evidence = [ + json.loads(line) for line in amendment_evidence_path.read_text().splitlines() + ] + invariants = { + "primary_runs_15": len(all_runs) == 15, + "three_replicates_per_arm": all(len(primary[arm]) == 3 for arm in ARMS), + "clean_duration_240": all(math.isclose(run["clean_duration_s"], 240.0) for run in all_runs), + "clean_failures_zero": all(run["clean_failed"] == 0 for run in all_runs), + "offered_rate_within_5pct": all(abs(run["offered_rps"] / RATE - 1) <= 0.05 for run in all_runs), + "layer1_accounting": all(all(run["layer1_invariants"].values()) for run in all_runs), + "client_invariants": all(all(value for key, value in run["client_invariants"].items() if key != "drain_re_adjudicated") for run in all_runs), + "server_invariants": all(all(run["server_invariants"].values()) for run in all_runs), + "a_p5_1_cold_start_gates": all( + run["cold_start_gate"] is not None + and run["cold_start_gate"]["passed"] + for run in all_runs + ), + "drain_quarantine_under_20pct": sum(run["drain_quarantined"] for run in all_runs) / 15 <= 0.20, + "gpu_budget_below_6": float(state["gpu_hours_total"]) < 6.0, + "manifests_same_ids": len({manifests[name]["request_id_set_sha256"] for name in manifests}) == 1, + "manifests_same_token_sums": len({manifests[name]["input_tokens"]["sum"] for name in manifests}) == 1 and len({manifests[name]["output_tokens"]["sum"] for name in manifests}) == 1, + "control_denominators_stable": all(ledgers[control]["stable_denominator"] for control in ledgers), + "bridge_decision_resolved": bridge["reuse_passed"] or all(source == "P5-rerun" for source in control_source.values()), + "ratios_finite": all(math.isfinite(ledgers[c]["mechanisms"][a]["diagnostic_share"]) for c in ledgers for a in MECHANISMS), + "per_arm_results_not_all_identical": len({round(points[arm], 12) for arm in ARMS}) > 1, + } + red_flags = [key for key, value in invariants.items() if not value] + publishable = ( + not red_flags + and all(item["passed"] for item in manipulations.values()) + and sum(width > 0.50 for width in share_widths) < 2 + ) + return { + "schema": 1, + "status": "PUBLISHABLE" if publishable else "INCONCLUSIVE_OR_PARTIAL", + "limitation": "Recorded-arrival P5 bridge ledger anchored to P3 controls; not a literal decomposition of P3's already-uniform P10 gap.", + "analysis_seed": SEED, + "bootstrap_resamples": RESAMPLES, + "efficiency": { + arm: { + "point": points[arm], + "ci95": ci(draws[arm]), + "runs": [ + {key: value for key, value in run.items() if key not in {"blocks", "warmup_stability"}} + for run in primary[arm] + ], + } + for arm in ARMS + }, + "p3_base_E": p3_base_point, + "bridge": bridge, + "control_sources": control_source, + "manipulations": manipulations, + "holm": {"family": list(MECHANISMS), "raw_p": raw_p, "adjusted_p": holm_p}, + "ledgers": ledgers, + "dominance": dominance, + "config_tier_A2": { + "delta_E": points["A2"] - points["base"], + "relative_E_delta": points["A2"] / points["base"] - 1, + "delta_E_ci95": ci(deltas["A2"]), + "base_padding_fraction": base_padding, + "A2_padding_fraction": a2_padding, + "padding_reduction": padding_reduction, + }, + "amendment_A_P5_1": { + "reason": "Rate-following throughput drift tracks arrival shape and is not a cold-start stationarity test.", + "retained_failed_run_evidence": amendment_evidence, + "recorded_drift_range": ( + [ + min( + item["superseded_drift_evidence"]["normalized_drift"] + for item in amendment_evidence + if item["run"] != "A3-r1-C00" + ), + max( + item["superseded_drift_evidence"]["normalized_drift"] + for item in amendment_evidence + if item["run"] != "A3-r1-C00" + ), + ] + if amendment_evidence else None + ), + "uniform_A3_drift": ( + next( + item["superseded_drift_evidence"]["normalized_drift"] + for item in amendment_evidence + if item["run"] == "A3-r1-C00" + ) + if amendment_evidence else None + ), + }, + "gpu": { + "new_h20_hours": float(state["gpu_hours_total"]), + "hard_cap": 6.0, + "controller_status": state["status"], + "completed_measured_runs_including_background": state["completed_measured_runs"], + "completed_burnins": state["completed_burnins"], + }, + "sanity": { + "red_flags": red_flags, + "invariants": invariants, + "numeric": { + "primary_E": numeric([run["token_efficiency_per_ms"] for run in all_runs]), + "clean_steps": numeric([run["clean_steps"] for run in all_runs]), + "offered_rps": numeric([run["offered_rps"] for run in all_runs]), + "drain_seconds": numeric([run["drain_seconds"] for run in all_runs]), + "diagnostic_share": numeric([ledgers[c]["mechanisms"][a]["diagnostic_share"] for c in ledgers for a in MECHANISMS]), + "residual_interaction": numeric([ledgers[c]["residual_interaction"] for c in ledgers]), + "share_ci_width": numeric(share_widths), + }, + "declared": { + "manipulation_failures": [arm for arm, item in manipulations.items() if not item["passed"]], + "control_sources": control_source, + "bridge_reuse_passed": bridge["reuse_passed"], + }, + }, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--p3-root", type=Path, required=True) + parser.add_argument("--private", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + result = analyze(args.root, args.p3_root, args.private) + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n") + print(json.dumps({ + "status": result["status"], + "bridge": result["bridge"], + "red_flags": result["sanity"]["red_flags"], + "gpu": result["gpu"], + }, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase5/opprof_phase5_client.py b/runs/opprof-phase5/opprof_phase5_client.py new file mode 100644 index 0000000..40343fb --- /dev/null +++ b/runs/opprof-phase5/opprof_phase5_client.py @@ -0,0 +1,242 @@ +#!/usr/bin/env python3 +"""Phase-5 private-manifest transforms and timestamp-scheduled P3 client wrapper.""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import math +import sys +from pathlib import Path +from typing import Any + +import aiohttp + +import opprof_phase3_client as p3 + + +def _numeric(values: list[float | int]) -> dict[str, Any]: + finite = [float(value) for value in values if math.isfinite(float(value))] + return { + "n": len(values), + "finite_n": len(finite), + "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, + "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + "sum": sum(finite), + } + + +def _r16(rows: list[dict[str, Any]]) -> float: + groups = [rows[index : index + 16] for index in range(0, len(rows) - 15, 16)] + useful = sum(sum(int(row["input_tokens"]) for row in group) for group in groups) + rectangular = sum(16 * max(int(row["input_tokens"]) for row in group) for group in groups) + return 1.0 - useful / rectangular + + +def _source_timestamps(path: Path, indices: set[int], field: str) -> dict[int, float]: + result: dict[int, float] = {} + maximum = max(indices) + with path.open(encoding="utf-8") as source: + for index, line in enumerate(source): + if index in indices: + value = float(json.loads(line)[field]) + if not math.isfinite(value): + raise ValueError(f"non-finite source timestamp at {index}") + result[index] = value + if index >= maximum: + break + if set(result) != indices: + raise ValueError("timestamp source did not cover all source_index values") + return result + + +def transform(args: argparse.Namespace) -> dict[str, Any]: + rows = p3.load_manifest(Path(args.input))[: args.take_first] + if len(rows) != args.take_first: + raise ValueError(f"requested {args.take_first} rows, found {len(rows)}") + indices = [int(row[args.join_key]) for row in rows] + timestamps = _source_timestamps( + Path(args.timestamp_source), set(indices), args.timestamp_field + ) + source_times = [timestamps[index] for index in indices] + if any(right < left for left, right in zip(source_times, source_times[1:])): + raise ValueError("selected timestamps are not nondecreasing") + if source_times[-1] <= source_times[0]: + raise ValueError("selected timestamp span is not positive") + end_s = (len(rows) - 1) / args.target_rate + scale = end_s / (source_times[-1] - source_times[0]) + recorded_slots = [(value - source_times[0]) * scale for value in source_times] + uniform_slots = [index / args.target_rate for index in range(len(rows))] + slots = recorded_slots if args.arrival == "recorded-scaled" else uniform_slots + + for index, row in enumerate(rows): + row["arrival"] = args.arrival + row["arrival_s"] = slots[index] + row["original_index"] = index + row["source_timestamp"] = source_times[index] + + original = list(rows) + max_added_delay = 0.0 + if args.service_order == "length-binned": + edges = [int(item) for item in args.length_bin_edges.split(",")] + + def bin_id(row: dict[str, Any]) -> int: + length = int(row["input_tokens"]) + for index, edge in enumerate(edges): + if length <= edge: + return index + raise ValueError(f"input length {length} exceeds final edge") + + reordered: list[dict[str, Any]] = [] + for offset in range(0, len(rows), args.reorder_block_size): + block = rows[offset : offset + args.reorder_block_size] + ordered = sorted( + block, + key=lambda row: ( + bin_id(row), + int(row["input_tokens"]), + int(row["original_index"]), + ), + ) + block_slots = slots[offset : offset + len(block)] + for position, row in enumerate(ordered): + added = max(0.0, block_slots[position] - float(row["arrival_s"])) + max_added_delay = max(max_added_delay, added) + row["arrival_s"] = block_slots[position] + reordered.extend(ordered) + rows = reordered + if max_added_delay > args.max_added_delay_seconds + 1e-9: + raise ValueError( + f"fairness cap exceeded: {max_added_delay} > {args.max_added_delay_seconds}" + ) + elif args.service_order != "original": + raise ValueError(f"unsupported service order: {args.service_order}") + + if sorted(row["request_id"] for row in rows) != sorted( + row["request_id"] for row in original + ): + raise AssertionError("request identity changed") + for key in ("input_tokens", "output_tokens"): + if sum(int(row[key]) for row in rows) != sum(int(row[key]) for row in original): + raise AssertionError(f"{key} total changed") + arrival_values = [float(row["arrival_s"]) for row in rows] + if any(right < left for left, right in zip(arrival_values, arrival_values[1:])): + raise AssertionError("assigned arrival slots are not nondecreasing") + + output = Path(args.out) + p3.atomic_jsonl(output, rows, mode=0o600) + summary = { + "schema": 1, + "path": str(output), + "sha256": p3.sha256_file(output), + "rows": len(rows), + "arrival": args.arrival, + "service_order": args.service_order, + "target_rate": args.target_rate, + "input_tokens": _numeric([int(row["input_tokens"]) for row in rows]), + "output_tokens": _numeric([int(row["output_tokens"]) for row in rows]), + "arrival_s": _numeric(arrival_values), + "source_timestamp": _numeric(source_times), + "r16": _r16(rows), + "max_added_delay_seconds": max_added_delay, + "request_id_set_sha256": p3.hashlib.sha256( + "\n".join(sorted(str(row["request_id"]) for row in rows)).encode() + ).hexdigest(), + "invariants": { + "same_request_ids": True, + "same_input_tokens": True, + "same_output_tokens": True, + "arrival_nondecreasing": True, + "fairness_cap": max_added_delay <= args.max_added_delay_seconds + 1e-9, + "no_prompt_in_summary": True, + }, + } + p3.atomic_json(output.with_suffix(output.suffix + ".summary.json"), summary, mode=0o600) + print(json.dumps(summary, sort_keys=True)) + return summary + + +async def finite_timestamp_load( + ctx: p3.RunContext, session: aiohttp.ClientSession, rate: float +) -> list[dict[str, Any]]: + if "arrival_s" not in ctx.rows[0]: + return await _ORIGINAL_FINITE_LOAD(ctx, session, rate) + sem = asyncio.Semaphore(ctx.args.max_concurrency) + tasks: list[asyncio.Task[dict[str, Any]]] = [] + + async def limited(row: dict[str, Any], scheduled: float) -> dict[str, Any]: + async with sem: + return await p3.request_one(ctx, session, row, scheduled) + + for expected in ctx.rows: + scheduled = ctx.t0 + float(expected["arrival_s"]) + delay = scheduled - asyncio.get_running_loop().time() + if delay > 0: + try: + await asyncio.wait_for(ctx.stop_event.wait(), timeout=delay) + break + except asyncio.TimeoutError: + pass + if ctx.stop_event.is_set(): + break + row = await ctx.next_row() + if row["request_id"] != expected["request_id"]: + raise AssertionError("timestamp scheduler row drift") + tasks.append(asyncio.create_task(limited(row, scheduled))) + return await asyncio.gather(*tasks) if tasks else [] + + +_ORIGINAL_FINITE_LOAD = p3.finite_load +p3.finite_load = finite_timestamp_load + + +def build_transform_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser() + parser.add_argument("--in", dest="input", required=True) + parser.add_argument("--take-first", type=int, required=True) + parser.add_argument("--timestamp-source", required=True) + parser.add_argument("--join-key", default="source_index") + parser.add_argument("--timestamp-field", default="timestamp") + parser.add_argument("--arrival", choices=("recorded-scaled", "uniform"), required=True) + parser.add_argument("--target-rate", type=float, required=True) + parser.add_argument("--service-order", choices=("original", "length-binned"), required=True) + parser.add_argument("--reorder-block-size", type=int, default=32) + parser.add_argument("--analysis-cohort-size", type=int, default=16) + parser.add_argument("--length-bin-edges", default="512,1024,2048,4096,8192,16384,32768") + parser.add_argument("--max-added-delay-seconds", type=float, default=64) + parser.add_argument("--out", required=True) + return parser + + +def main() -> None: + if len(sys.argv) > 1 and sys.argv[1] == "transform": + transform(build_transform_parser().parse_args(sys.argv[2:])) + return + fixed_rate = None + if "--fixed-request-rate" in sys.argv: + index = sys.argv.index("--fixed-request-rate") + fixed_rate = float(sys.argv[index + 1]) + del sys.argv[index : index + 2] + args = p3.build_parser().parse_args() + if args.command != "run": + p3.main() + return + if fixed_rate is not None: + if args.load_point != "moderate" or fixed_rate <= 0 or not math.isfinite(fixed_rate): + raise ValueError("--fixed-request-rate requires positive finite moderate rate") + result_dir = Path(args.result_dir) + result_dir.mkdir(parents=True, exist_ok=True) + source = result_dir / "fixed-rate-source.json" + p3.atomic_json(source, {"clean": {"completed_throughput_rps": fixed_rate}}) + args.saturation_result = str(source) + args.rate_fraction = 1.0 + if args.profile_after_clean and not args.profile_trace_dir: + raise ValueError("--profile-after-clean requires --profile-trace-dir") + print(json.dumps(asyncio.run(p3.run_load(args)), sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase5/opprof_phase5_controller.py b/runs/opprof-phase5/opprof_phase5_controller.py new file mode 100644 index 0000000..0d8a7e6 --- /dev/null +++ b/runs/opprof-phase5/opprof_phase5_controller.py @@ -0,0 +1,670 @@ +#!/usr/bin/env python3 +"""Detached, resumable Phase-5 four-way primary/control controller.""" + +from __future__ import annotations + +import argparse +import datetime as dt +import hashlib +import json +import math +import os +import re +import shlex +import shutil +import subprocess +import time +from pathlib import Path +from typing import Any + +import opprof_phase3_matrix as m + + +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/phase5" +PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase5-private/manifests") +P3_PRIVATE = Path("/home/admin/cpfs/wjh/opprof-phase3-private/manifests") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0") +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +CLIENT = WORKDIR / "scripts/opprof_phase5_client.py" +P3_CLIENT = WORKDIR / "scripts/opprof_phase3_client.py" +STATE = RUN_ROOT / "controller-state.json" +RATE = 0.4725 +CAPTURE_SIZES = ( + 1, 2, 3, 4, 5, 6, 7, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, + 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, + 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, + 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512, +) + + +m.WORKDIR = WORKDIR +m.RUN_ROOT = RUN_ROOT +m.PRIVATE = PRIVATE +m.MODEL = MODEL +m.SOURCE = SOURCE +m.VENV = VENV +m.CLIENT = CLIENT +m.STATE = STATE +m.GPU_HOUR_LIMIT = 6.0 +m.PRIOR_GPU_HOURS = 0.0 +m.CONFIGS = { + "C00": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []}, + "A2": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []}, + "A4": {"tp": 1, "mns": 1024, "mbt": 8192, "flags": []}, +} + + +def sha256_file(path: Path) -> str: + return m.sha256_file(path) + + +def arm_name(pattern: str) -> str: + return pattern.split("-r", 1)[0] + + +def manifest_for(pattern: str, burnin: bool) -> Path: + if burnin or pattern.startswith("background"): + return P3_PRIVATE / "P06.jsonl" + if pattern.startswith("control-P03"): + return P3_PRIVATE / "P03.jsonl" + if pattern.startswith("control-P04"): + return P3_PRIVATE / "P04.jsonl" + arm = arm_name(pattern) + return PRIVATE / ("P10-base.jsonl" if arm in {"base", "A2", "A4"} else f"P10-{arm}.jsonl") + + +def saturation_result_for(pattern: str) -> Path: + if pattern.startswith("control-P03"): + cell = "P03-C00" + elif pattern.startswith("control-P04"): + cell = "P04-C00" + else: + cell = "P10-C00" + return WORKDIR / f"runs/phase3/primary/{cell}/saturation/client/result.json" + + +def drain_budget(_pattern: str) -> int: + return 600 + + +m.drain_budget = drain_budget + + +def server_command(assignment: m.Assignment, port: int, _trace_dir: Path) -> list[str]: + arm = arm_name(assignment.cell.pattern) + command = [ + "taskset", "-c", m.cpu_mask(assignment.gpus), str(VENV / "bin/vllm"), + "serve", str(MODEL), "--host", "127.0.0.1", "--port", str(port), + "--tensor-parallel-size", "1", "--enable-chunked-prefill", + ] + if arm != "A4" and assignment.cell.config != "A4": + command.append("--enable-prefix-caching") + command.extend(("--shutdown-timeout", "600")) + if arm == "A2" or assignment.cell.config == "A2": + command.extend(("--cudagraph-capture-sizes", *map(str, CAPTURE_SIZES))) + return command + + +def client_command( + assignment: m.Assignment, + port: int, + run_dir: Path, + _load_point: str, + _profile: bool, + burnin: bool, + _saturation_result: Path | None, +) -> list[str]: + pattern = assignment.cell.pattern + common = [ + "taskset", "-c", m.cpu_mask(assignment.gpus), str(VENV / "bin/python"), + str(CLIENT), "run", "--manifest", str(manifest_for(pattern, burnin)), + "--base-url", f"http://127.0.0.1:{port}", "--model", str(MODEL), + "--max-concurrency", "256", "--ignore-eos", "--temperature", "0", + "--workload-seed", "20260712", "--server-seed", "20260712", + "--result-dir", str(run_dir / "client"), + ] + if burnin: + return common + [ + "--load-point", "saturation", "--request-rate", "inf", + "--warmup-seconds", "0", "--clean-segment-seconds", "20", + "--num-clean-segments", "3", "--post-clean-seconds", "0", + "--drain-timeout-seconds", "600", + ] + if pattern.startswith("background"): + return common + [ + "--load-point", "saturation", "--request-rate", "inf", + "--warmup-seconds", "60", "--clean-segment-seconds", "80", + "--num-clean-segments", "3", "--post-clean-seconds", "0", + "--drain-timeout-seconds", "600", + ] + if pattern.startswith("control-"): + return common + [ + "--load-point", "moderate", "--saturation-result", + str(saturation_result_for(pattern)), "--rate-fraction", "0.60", + "--warmup-seconds", "60", "--clean-segment-seconds", "80", + "--num-clean-segments", "3", "--post-clean-seconds", "0", + "--drain-timeout-seconds", "600", + ] + return common + [ + "--load-point", "moderate", "--fixed-request-rate", str(RATE), + "--warmup-seconds", "60", "--clean-segment-seconds", "80", + "--num-clean-segments", "3", "--post-clean-seconds", "0", + "--drain-timeout-seconds", "600", + ] + + +m.server_command = server_command +m.client_command = client_command + + +_ORIGINAL_VALIDATE_CLIENT = m.validate_client + + +def _log_event_time(line: str, t0_wall_ns: int) -> float | None: + match = re.search(r"INFO\s+(\d{2})-(\d{2})\s+(\d{2}):(\d{2}):(\d{2})", line) + if match is None: + return None + month, day, hour, minute, second = map(int, match.groups()) + year = dt.datetime.fromtimestamp(t0_wall_ns / 1e9, tz=dt.timezone.utc).year + stamp = dt.datetime(year, month, day, hour, minute, second, tzinfo=dt.timezone.utc) + return stamp.timestamp() + + +def cold_start_gate(run_dir: Path) -> dict[str, Any]: + result = json.loads((run_dir / "client/result.json").read_text()) + requests = [ + json.loads(line) + for line in (run_dir / "client/requests.jsonl").read_text().splitlines() + ] + warm = [ + request for request in requests + if request["success"] and 0 <= float(request["completed_s"]) < 60 + ] + warm_long = [request for request in warm if int(request["input_tokens"]) >= 8192] + t0_mono_ns = int(result["t0_mono_ns"]) + stream = next((run_dir / "opprof").glob("*.jsonl")) + warm_descriptors: set[tuple[str, int]] = set() + clean_descriptors: set[tuple[str, int]] = set() + for line in stream.read_text().splitlines(): + item = json.loads(line) + if "step_index" not in item or not item.get("model_executed"): + continue + graph = item["cudagraph"] + if not graph.get("hit"): + continue + relative_s = (int(item["submit_mono_ns"]) - t0_mono_ns) / 1e9 + descriptor = (str(graph["runtime_mode"]), int(graph["bucket_tokens"])) + if 0 <= relative_s < 60: + warm_descriptors.add(descriptor) + elif 60 <= relative_s < 300: + clean_descriptors.add(descriptor) + + log_lines = (run_dir / "server.log").read_text(errors="replace").splitlines() + ready_indices = [ + index for index, line in enumerate(log_lines) + if "Application startup complete" in line + ] + if len(ready_indices) != 1: + raise RuntimeError(f"expected one server ready marker: {run_dir}: {ready_indices}") + ready_index = ready_indices[0] + event_pattern = re.compile( + r"torch\.compile took|Directly load AOT compilation|\bCompiling\b|Capturing CUDA graphs", + re.IGNORECASE, + ) + events = [] + clean_boundary_wall_s = int(result["t0_wall_ns"]) / 1e9 + 60 + clean_end_wall_s = int(result["t0_wall_ns"]) / 1e9 + 300 + event_gate = True + clean_events = 0 + for index, line in enumerate(log_lines): + if not event_pattern.search(line): + continue + timestamp = _log_event_time(line, int(result["t0_wall_ns"])) + phase = "pre-ready" if index < ready_index else "post-ready" + if index >= ready_index: + if timestamp is None: + event_gate = False + phase = "post-ready-unparseable" + elif timestamp >= clean_boundary_wall_s: + event_gate = False + phase = "clean" if timestamp < clean_end_wall_s else "post-clean" + if timestamp < clean_end_wall_s: + clean_events += 1 + else: + phase = "warmup" + events.append( + { + "line": index + 1, + "phase": phase, + "timestamp_s": timestamp, + "message_prefix": line[:200], + } + ) + + config_match = re.search( + r"cudagraph_capture_sizes['\"]?:\s*\[([^\]]+)\]", + "\n".join(log_lines[:ready_index]), + ) + startup_sizes = ( + {int(value.strip()) for value in config_match.group(1).split(",")} + if config_match else set() + ) + startup_modes = set() + for line in log_lines[:ready_index]: + if "Capturing CUDA graphs" not in line: + continue + if "PIECEWISE" in line: + startup_modes.add("PIECEWISE") + if "FULL" in line: + startup_modes.add("FULL") + uncovered = sorted( + descriptor for descriptor in clean_descriptors + if descriptor[0] not in startup_modes or descriptor[1] not in startup_sizes + ) + passed = ( + event_gate + and len(warm) >= 16 + and len(warm_long) >= 1 + and startup_modes == {"FULL", "PIECEWISE"} + and bool(startup_sizes) + and not uncovered + and clean_events == 0 + ) + return { + "amendment": "A-P5-1", + "passed": passed, + "warmup_completions": len(warm), + "warmup_long_completions": len(warm_long), + "warmup_long_min_tokens": min( + (int(request["input_tokens"]) for request in warm_long), default=None + ), + "server_ready_line": ready_index + 1, + "compile_capture_events": events, + "event_gate_passed": event_gate, + "clean_capture_events": clean_events, + "startup_capture_sizes": sorted(startup_sizes), + "startup_capture_modes": sorted(startup_modes), + "warmup_descriptors": [list(item) for item in sorted(warm_descriptors)], + "clean_descriptors": [list(item) for item in sorted(clean_descriptors)], + "uncovered_clean_descriptors": [list(item) for item in uncovered], + "invariants": { + "events_preclean": event_gate, + "warmup_completions_ge_16": len(warm) >= 16, + "warmup_long_completion": len(warm_long) >= 1, + "startup_modes_complete": startup_modes == {"FULL", "PIECEWISE"}, + "startup_sizes_present": bool(startup_sizes), + "clean_descriptors_covered": not uncovered, + "zero_clean_capture_events": clean_events == 0, + }, + } + + +def validate_rate_client(run_dir: Path) -> dict[str, Any]: + result = json.loads((run_dir / "client/result.json").read_text()) + sanity = json.loads((run_dir / "client/sanity.json").read_text()) + requests = [ + json.loads(line) + for line in (run_dir / "client/requests.jsonl").read_text().splitlines() + ] + failed_sanity = [ + key for key, value in sanity["invariants"].items() + if not value and key != "drain_within_timeout" + ] + failure_summary = m.summarize_request_failures( + requests, float(result["clean"]["start_s"]), float(result["clean"]["end_s"]) + ) + cold = cold_start_gate(run_dir) + quarantined = float(result["drain_seconds"]) > 600 + invariants = { + "client_sanity": not failed_sanity, + "clean_duration": math.isclose(float(result["clean"]["duration_s"]), 240.0), + "clean_failures_zero": result["clean"]["failed"] == 0 + and failure_summary["clean_failed"] == 0, + "failed_records_accounted": result["failed_records"] == failure_summary["failed"], + "manifest_no_wrap": not result["manifest_wrapped"] + and not result["manifest_exhausted"], + "warmup_cold_start_gate": cold["passed"], + "profile_count": len(result["profiles"]) == 0, + "drain_re_adjudicated": not quarantined, + } + non_drain = {key: value for key, value in invariants.items() if key != "drain_re_adjudicated"} + if not all(non_drain.values()): + raise RuntimeError( + f"A-P5-1 rate-client invariant failure: {run_dir}: " + f"invariants={invariants}; failed_sanity={failed_sanity}; cold={cold}" + ) + return { + "result": result, + "sanity": sanity, + "request_count": len(requests), + "warmup_completions": cold["warmup_completions"], + "warmup_required": 16, + "warmup_gate_branch": "A-P5-1-cold-start", + "warmup_stability": None, + "cold_start_gate": cold, + "drain_budget_seconds": 600, + "drain_quarantined": quarantined, + "excluded_window_failures": failure_summary["excluded"], + "excluded_window_failure_kinds": failure_summary["excluded_kinds"], + "invariants": invariants, + } + + +def validate_run( + entry: dict[str, Any], profile: bool, burnin: bool, allow_missing_traces: bool = False +) -> dict[str, Any]: + del profile, allow_missing_traces + pattern = entry["assignment"].cell.pattern + if burnin or pattern.startswith("background"): + validation_pattern = "P06" + elif pattern.startswith("control-P03"): + validation_pattern = "P03" + elif pattern.startswith("control-P04"): + validation_pattern = "P04" + else: + validation_pattern = "P10" + if burnin or pattern.startswith("background"): + client = _ORIGINAL_VALIDATE_CLIENT( + entry["run_dir"], validation_pattern, False, burnin + ) + else: + client = validate_rate_client(entry["run_dir"]) + layer1 = m.validate_layer1(entry["run_dir"]) + log = (entry["run_dir"] / "server.log").read_text(errors="replace") + invariants = { + "triton_moe": "Using TRITON Unquantized MoE backend" in log, + "chunked_mbt": "Chunked prefill is enabled with max_num_batched_tokens=8192" in log, + "tp1": "tensor_parallel_size=1" in log, + "drain_shutdown": "mode=drain timeout=600s" in log, + "a2_sizes": entry["assignment"].cell.config != "A2" + or all(str(size) in log for size in (3, 5, 6, 7)), + } + if not all(invariants.values()): + raise RuntimeError(f"server invariant failure: {entry['run_id']}: {invariants}") + forbidden = re.compile(r'"(?:prompt|messages|content|text)"\s*:') + for path in ( + entry["run_dir"] / "client/requests.jsonl", + entry["run_dir"] / "client/result.json", + Path(layer1["stream"]), + ): + if forbidden.search(path.read_text(errors="replace")): + raise RuntimeError(f"private text leaked: {path}") + summary = { + "schema": 1, + "run_id": entry["run_id"], + "pattern": pattern, + "config": entry["assignment"].cell.config, + "gpus": entry["assignment"].gpus, + "client": client, + "layer1": layer1, + "traces": [], + "missing_trace_files": 0, + "layer2_missing_after_controller_cleanup": False, + "drain_quarantined": client["drain_quarantined"], + "server_invariants": invariants, + } + m.atomic_json(entry["run_dir"] / "run-complete.json", summary) + return summary + + +m.validate_run = validate_run + + +def manifests() -> dict[str, Any]: + result = {} + for name in ("base", "A1", "A3"): + path = PRIVATE / f"P10-{name}.jsonl" + summary = json.loads(path.with_suffix(path.suffix + ".summary.json").read_text()) + if summary["rows"] != 142 or summary["sha256"] != sha256_file(path): + raise RuntimeError(f"manifest verification failed: {name}") + result[name] = {"path": str(path), "sha256": summary["sha256"]} + return result + + +def fingerprint() -> dict[str, Any]: + return { + "source_commit": subprocess.check_output( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD"], text=True + ).strip(), + "source_tree": subprocess.check_output( + ["git", "-C", str(SOURCE), "rev-parse", "HEAD^{tree}"], text=True + ).strip(), + "client_sha256": sha256_file(CLIENT), + "controller_sha256": sha256_file(Path(__file__)), + "p3_client_sha256": sha256_file(P3_CLIENT), + "p3_matrix_sha256": sha256_file(Path(m.__file__).resolve()), + "manifests": manifests(), + "capture_sizes": list(CAPTURE_SIZES), + "rate": RATE, + } + + +def load_state(resume: bool) -> dict[str, Any]: + if STATE.exists(): + if not resume: + raise RuntimeError("controller state exists; use --resume") + return json.loads(STATE.read_text()) + return { + "schema": 1, + "status": "created", + "created_at": time.time(), + "controller_pid": os.getpid(), + "gpu_hours_total": 0.0, + "gpu_hours_this_stage": 0.0, + "completed_measured_runs": 0, + "completed_burnins": 0, + "drain_quarantined_runs": 0, + "clean_window_failures": 0, + "missing_trace_files": 0, + "stages": {}, + "fingerprint": {}, + } + + +def save_state(state: dict[str, Any]) -> None: + state["controller_pid"] = os.getpid() + state["updated_at"] = time.time() + m.atomic_json(STATE, state) + + +m.save_state = save_state + + +def ensure_provenance() -> None: + destination = RUN_ROOT / "provenance" + destination.mkdir(parents=True, exist_ok=True) + sources = [CLIENT, Path(__file__).resolve(), Path(m.__file__).resolve(), Path(m.common.__file__).resolve()] + hashes = {} + for source in sources: + target = destination / source.name + digest = sha256_file(source) + if target.exists() and sha256_file(target) != digest: + target = destination / f"{source.stem}.{digest[:12]}{source.suffix}" + if target.exists() and sha256_file(target) != digest: + raise RuntimeError(f"content-addressed provenance mismatch: {target}") + if not target.exists(): + shutil.copy2(source, target) + hashes[target.name] = digest + m.atomic_json(destination / "sha256.json", hashes) + + +def primary_cells() -> list[m.Cell]: + config = {"base": "C00", "A1": "C00", "A2": "A2", "A3": "C00", "A4": "A4"} + items = [m.Cell(f"{arm}-r{replicate}", config[arm]) for arm in config for replicate in range(1, 4)] + return sorted( + items, + key=lambda cell: hashlib.sha256( + f"20260715:{cell.pattern.rsplit('-r',1)[1]}:{arm_name(cell.pattern)}".encode() + ).hexdigest(), + ) + + +def pack_unique(items: list[m.Cell], prefix: str) -> list[list[m.Assignment]]: + remaining = list(items) + waves: list[list[m.Assignment]] = [] + wave_index = 0 + while remaining: + selected: list[m.Cell] = [] + used: set[str] = set() + for cell in list(remaining): + key = arm_name(cell.pattern) + if key in used: + continue + selected.append(cell) + used.add(key) + remaining.remove(cell) + if len(selected) == 4: + break + while len(selected) < 4: + selected.append(m.Cell(f"background-{prefix}-{wave_index}-{len(selected)}", "C00")) + assignments = [] + for slot, cell in enumerate(selected): + gpu = (slot + wave_index) % 4 + assignments.append(m.Assignment(cell, (gpu,))) + waves.append(assignments) + wave_index += 1 + return waves + + +def pack_primary(items: list[m.Cell]) -> list[list[m.Assignment]]: + """Pack SHA-ordered cells as 4/4/4/3 without duplicate arms per wave.""" + capacities = (4, 4, 4, 3) + waves: list[list[m.Cell]] = [[] for _ in capacities] + + def place(index: int) -> bool: + if index == len(items): + return all(len(wave) == capacity for wave, capacity in zip(waves, capacities, strict=True)) + cell = items[index] + arm = arm_name(cell.pattern) + for wave_index, capacity in enumerate(capacities): + if len(waves[wave_index]) >= capacity: + continue + if any(arm_name(existing.pattern) == arm for existing in waves[wave_index]): + continue + waves[wave_index].append(cell) + if place(index + 1): + return True + waves[wave_index].pop() + return False + + if not place(0): + raise RuntimeError("cannot pack frozen primary assignments into 4/4/4/3") + result: list[list[m.Assignment]] = [] + for wave_index, cells in enumerate(waves): + if len(cells) == 3: + cells.append(m.Cell("background-primary-final", "C00")) + result.append( + [ + m.Assignment(cell, ((slot + wave_index) % 4,)) + for slot, cell in enumerate(cells) + ] + ) + return result + + +def execute_primary(resume: bool, amendment_a_p5_1: bool = False) -> None: + RUN_ROOT.mkdir(parents=True, exist_ok=True) + state = load_state(resume) + if resume: + m.cleanup_recorded(state) + current = fingerprint() + if state["fingerprint"] and state["fingerprint"] != current: + failure = str(state.get("stages", {}).get("primary-01", {}).get("failure", "")) + if not ( + amendment_a_p5_1 + and state.get("status") == "failed" + and "warmup" in failure.lower() + ): + raise RuntimeError("resume fingerprint differs from frozen Phase-5 plan") + state.setdefault("amendments", {})["A-P5-1"] = { + "approved": True, + "applied_at": time.time(), + "reason": "replace rate-following drift gate with cold-start gates", + "prior_fingerprint": state["fingerprint"], + "replacement_fingerprint": current, + "retained_gpu_hours": state["gpu_hours_total"], + "burnins_reused": state["completed_burnins"], + } + state["fingerprint"] = current + state["status"] = "amended_resume_A-P5-1" + save_state(state) + state["fingerprint"] = current + state["status"] = "running_primary" + save_state(state) + ensure_provenance() + burnins = [ + m.Assignment(m.Cell("burnin-C00", "C00"), (0,)), + m.Assignment(m.Cell("burnin-A2", "A2"), (1,)), + m.Assignment(m.Cell("burnin-A4", "A4"), (2,)), + ] + m.run_stage(state, "burnins", burnins, "saturation", profile=False, burnin=True) + waves = pack_primary(primary_cells()) + for index, wave in enumerate(waves, 1): + m.run_stage(state, f"primary-{index:02d}", wave, "moderate", profile=False) + primary = [ + path for path in (RUN_ROOT / "primary").glob("*-r*-*/moderate/run-complete.json") + if "background" not in str(path) + ] + if len(primary) != 15: + raise RuntimeError(f"primary completion mismatch: {len(primary)} != 15") + state["primary_runs"] = 15 + state["background_runs"] = state["completed_measured_runs"] - 15 + state["status"] = "primary_complete" + state["completed_at"] = time.time() + save_state(state) + + +def execute_controls(resume: bool) -> None: + state = load_state(resume) + m.cleanup_recorded(state) + if state.get("fingerprint") != fingerprint(): + raise RuntimeError("control resume fingerprint mismatch") + cells = [ + m.Cell(f"control-{pattern}-r{replicate}", "C00") + for pattern in ("P03", "P04") for replicate in range(1, 4) + ] + for index, wave in enumerate(pack_unique(cells, "controls"), 1): + m.run_stage(state, f"controls-{index:02d}", wave, "moderate", profile=False) + state["status"] = "controls_complete" + state["controls_completed_at"] = time.time() + save_state(state) + + +def plan() -> dict[str, Any]: + return { + "schema": 1, + "primary_runs": 15, + "burnins": 3, + "waves": [[{"cell": a.cell.cell_id, "gpus": a.gpus} for a in wave] for wave in pack_primary(primary_cells())], + "rate": RATE, + "clean_seconds": 240, + "drain_seconds": 600, + "gpu_hour_limit": 6.0, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + sub = parser.add_subparsers(dest="command", required=True) + for name in ("primary", "controls"): + item = sub.add_parser(name) + item.add_argument("--resume", action="store_true") + if name == "primary": + item.add_argument("--amendment-a-p5-1", action="store_true") + sub.add_parser("plan") + sub.add_parser("status") + args = parser.parse_args() + if args.command == "primary": + execute_primary(args.resume, args.amendment_a_p5_1) + elif args.command == "controls": + execute_controls(args.resume) + elif args.command == "plan": + print(json.dumps(plan(), sort_keys=True, indent=2)) + else: + print(STATE.read_text() if STATE.exists() else '{"status":"absent"}') + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase5/phase5/controller-state.json b/runs/opprof-phase5/phase5/controller-state.json new file mode 100644 index 0000000..4f2915f --- /dev/null +++ b/runs/opprof-phase5/phase5/controller-state.json @@ -0,0 +1,212 @@ +{ + "clean_window_failures": 0, + "completed_burnins": 3, + "completed_measured_runs": 0, + "controller_pid": 2505724, + "created_at": 1783858074.2830184, + "drain_quarantined_runs": 0, + "fingerprint": { + "capture_sizes": [ + 1, + 2, + 3, + 4, + 5, + 6, + 7, + 8, + 16, + 24, + 32, + 40, + 48, + 56, + 64, + 72, + 80, + 88, + 96, + 104, + 112, + 120, + 128, + 136, + 144, + 152, + 160, + 168, + 176, + 184, + 192, + 200, + 208, + 216, + 224, + 232, + 240, + 248, + 256, + 272, + 288, + 304, + 320, + 336, + 352, + 368, + 384, + 400, + 416, + 432, + 448, + 464, + 480, + 496, + 512 + ], + "client_sha256": "f935486ab2588c1fca514be29b59361f53118eb000ea037863de48ce4fc76b16", + "controller_sha256": "ea2e2066a8b6d9427c59fe80db44cf87a86776570ca9066bfdff0c0b0e7f4a46", + "manifests": { + "A1": { + "path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-A1.jsonl", + "sha256": "cab8468983fb7397ec88eb5e88a44c8b1b53d8021b7867e7bec6f58d3d903806" + }, + "A3": { + "path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-A3.jsonl", + "sha256": "ec5eaa29fcd3ec421e4f68a902c7e7fea9c0bc6b16f1013cef30ccc1f97bae24" + }, + "base": { + "path": "/home/admin/cpfs/wjh/opprof-phase5-private/manifests/P10-base.jsonl", + "sha256": "d3b4f540ddd629dd0e34fff55c3b3837f359bdd6e5947309a1ea2a358f811ffc" + } + }, + "p3_client_sha256": "ab937a5f28252559c2fd97e848a500f1094cef232823ce4b90da8c0ece7554a0", + "p3_matrix_sha256": "6ac565ff35ead305f7b2e39e6a754389d03c27ea6511b2c9e8ebc0c868c9519f", + "rate": 0.4725, + "source_commit": "4b253fd8619764b6971a7f2e3a3aa7545f6ace05", + "source_tree": "a3d536b287a724e60abbec68b45eed7e088a15d1" + }, + "gpu_hours_this_stage": 0.4224752351972792, + "gpu_hours_total": 0.6476566231913037, + "missing_trace_files": 0, + "schema": 1, + "stages": { + "burnins": { + "assignments": [ + { + "cell": "burnin-C00-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "burnin-A2-A2", + "gpus": [ + 1 + ] + }, + { + "cell": "burnin-A4-A4", + "gpus": [ + 2 + ] + } + ], + "burnin": true, + "clients": {}, + "completed_at": 1783858361.2828288, + "confirmation": false, + "gpu_hours": 0.22518138799402448, + "load_point": "saturation", + "profile": false, + "servers": {}, + "started_at": 1783858074.4911764, + "status": "complete" + }, + "primary-01": { + "assignments": [ + { + "cell": "base-r2-C00", + "gpus": [ + 0 + ] + }, + { + "cell": "A3-r1-C00", + "gpus": [ + 1 + ] + }, + { + "cell": "A1-r2-C00", + "gpus": [ + 2 + ] + }, + { + "cell": "A4-r1-A4", + "gpus": [ + 3 + ] + } + ], + "burnin": false, + "clients": { + "A1-r2-C00-moderate": { + "pgid": 2512122, + "pid": 2512122 + }, + "A3-r1-C00-moderate": { + "pgid": 2512121, + "pid": 2512121 + }, + "A4-r1-A4-moderate": { + "pgid": 2512123, + "pid": 2512123 + }, + "base-r2-C00-moderate": { + "pgid": 2512119, + "pid": 2512119 + } + }, + "confirmation": false, + "failure": "RuntimeError(\"client invariant failure: /home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5/primary/base-r2-C00/moderate: {'client_sanity': True, 'clean_duration': True, 'clean_failures_zero': True, 'failed_records_accounted': True, 'manifest_no_wrap': True, 'warmup_completions': False, 'profile_count': True, 'profile_after_clean': True, 'drain_re_adjudicated': True}; failed=[]; warmup_completions=25; warmup_gate_branch=failed; warmup_stability={'passed': False, 'reason': 'A-P3-6 stabilization criterion not met', 'window_seconds': [45.0, 60.0], 'bin_seconds': 5.0, 'step_counts': [380, 201, 187], 'scheduled_tokens': [26927, 27616, 463], 'scheduled_token_throughput': [5385.4, 5523.2, 92.6], 'mean_scheduled_token_throughput': 3667.066666666666, 'slope_tokens_per_second_squared': -529.28, 'normalized_drift': 2.1650001817983497, 'normalized_drift_limit': 0.1, 'step_indices_continuous': True}\")", + "gpu_hours": 0.4224752351972792, + "load_point": "moderate", + "profile": false, + "servers": { + "A1-r2-C00-moderate": { + "gpus": [ + 2 + ], + "pgid": 2510424, + "pid": 2510424 + }, + "A3-r1-C00-moderate": { + "gpus": [ + 1 + ], + "pgid": 2510423, + "pid": 2510423 + }, + "A4-r1-A4-moderate": { + "gpus": [ + 3 + ], + "pgid": 2510425, + "pid": 2510425 + }, + "base-r2-C00-moderate": { + "gpus": [ + 0 + ], + "pgid": 2510422, + "pid": 2510422 + } + }, + "started_at": 1783858361.3184204, + "status": "failed" + } + }, + "status": "failed", + "updated_at": 1783858758.067235 +} diff --git a/runs/opprof-phase5/phase5/launch-echo.log b/runs/opprof-phase5/phase5/launch-echo.log new file mode 100644 index 0000000..5cc9ce1 --- /dev/null +++ b/runs/opprof-phase5/phase5/launch-echo.log @@ -0,0 +1 @@ +LAUNCH_ECHO utc=2026-07-12T12:07:54Z host=dash0 gpus=0-3 cpus=0-79 source=/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0@4b253fd manifests=/home/admin/cpfs/wjh/opprof-phase5-private/manifests outputs=/home/admin/cpfs/wjh/opprof-phase3-dash0-20260712/runs/phase5 runs=3burnin+15primary+1background rate=0.4725 warmup=60s clean=240s drain=600s est_wall=35-60min est_gpu=1.7-2.1_H20h hard_cap=6.0_H20h conditional_controls=6_if_bridge_fails diff --git a/runs/opprof-phase5/phase5/metrics.json b/runs/opprof-phase5/phase5/metrics.json new file mode 100644 index 0000000..6a1c9f4 --- /dev/null +++ b/runs/opprof-phase5/phase5/metrics.json @@ -0,0 +1,14178 @@ +{ + "amendment_A_P5_1": { + "reason": "Rate-following throughput drift tracks arrival shape and is not a cold-start stationarity test.", + "recorded_drift_range": [ + 1.90551033490161, + 2.1650001817983497 + ], + "retained_failed_run_evidence": [ + { + "cold_start_gate": { + "amendment": "A-P5-1", + "clean_capture_events": 0, + "clean_descriptors": [ + [ + "FULL", + 1 + ], + [ + "FULL", + 2 + ], + [ + "FULL", + 4 + ], + [ + "FULL", + 8 + ], + [ + "FULL", + 16 + ], + [ + "PIECEWISE", + 80 + ], + [ + "PIECEWISE", + 144 + ], + [ + "PIECEWISE", + 216 + ], + [ + "PIECEWISE", + 240 + ], + [ + "PIECEWISE", + 248 + ], + [ + "PIECEWISE", + 272 + ], + [ + "PIECEWISE", + 368 + ], + [ + "PIECEWISE", + 384 + ], + [ + "PIECEWISE", + 448 + ], + [ + "PIECEWISE", + 480 + ] + ], + "compile_capture_events": [ + { + "line": 66, + "message_prefix": "(EngineCore pid=2510807) INFO 07-12 12:13:38 [decorators.py:311] Directly load AOT compilation from path /home/admin/cpfs/wjh/.cache/vllm/torch_compile_cache/torch_aot_compile/7a0273078f7c7f7f0453a4a0", + "phase": "pre-ready", + "timestamp_s": 1783858418.0 + }, + { + "line": 67, + "message_prefix": "(EngineCore pid=2510807) INFO 07-12 12:13:38 [monitor.py:53] torch.compile took 5.74 s in total", + "phase": "pre-ready", + "timestamp_s": 1783858418.0 + }, + { + "line": 81, + "message_prefix": "Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 0%| | 0/51 [00:00 None: + adjusted = a.holm({"A1": 0.001, "A2": 0.02, "A3": 0.04, "A4": 0.5}) + assert adjusted == {"A1": 0.004, "A2": 0.06, "A3": 0.08, "A4": 0.5} + assert a.ci(np.arange(100, dtype=np.float64)) == [2.475, 96.52499999999999] + runs = [ + {"blocks": np.asarray([[10.0, 2.0]] * 48)}, + {"blocks": np.asarray([[20.0, 4.0]] * 48)}, + {"blocks": np.asarray([[30.0, 6.0]] * 48)}, + ] + draws = a.hierarchical_draws(runs, np.random.default_rng(a.SEED)) + assert draws.shape == (a.RESAMPLES,) + assert np.allclose(draws, 5.0) + assert a.point_efficiency(runs) == 5.0 + idle_blocks = np.asarray([[0.0, 0.0]] + [[10.0, 2.0]] * 47) + idle_draws = a.hierarchical_draws( + [{"blocks": idle_blocks}], np.random.default_rng(a.SEED) + ) + assert np.all(np.isfinite(idle_draws)) + assert np.allclose(idle_draws, 5.0) + assert a.point_efficiency([{"blocks": idle_blocks}]) == 5.0 + assert a.two_sided_p(np.asarray([-1.0, 1.0])) == 1.0 + print("phase5 analysis: PASS") + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase5/test_phase5_tools.py b/runs/opprof-phase5/test_phase5_tools.py new file mode 100644 index 0000000..e01e3db --- /dev/null +++ b/runs/opprof-phase5/test_phase5_tools.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import json +import subprocess +import sys +import tempfile +from pathlib import Path + + +HERE = Path(__file__).resolve().parent +CLIENT = HERE / "opprof_phase5_client.py" + + +def write_jsonl(path: Path, rows: list[dict]) -> None: + path.write_text("".join(json.dumps(row) + "\n" for row in rows)) + + +def run(command: list[str]) -> None: + completed = subprocess.run(command, text=True, capture_output=True) + if completed.returncode: + raise RuntimeError(f"command failed: {completed.stderr}") + + +def main() -> None: + with tempfile.TemporaryDirectory() as tmp_text: + tmp = Path(tmp_text) + manifest = tmp / "p3.jsonl" + source = tmp / "source.jsonl" + base = tmp / "base.jsonl" + a1 = tmp / "a1.jsonl" + rows = [] + source_rows = [] + lengths = [128, 8192, 256, 4096] * 8 + for index, length in enumerate(lengths): + rows.append({ + "request_id": f"P10-{index}", "pattern_id": "P10", + "input_tokens": length, "output_tokens": 8, + "arrival": "steady", "kind": "private-trace", + "source_index": index, "prompt": f"private-{index}", + }) + source_rows.append({"timestamp": index * 0.1, "prompt": f"private-{index}"}) + write_jsonl(manifest, rows) + write_jsonl(source, source_rows) + common = [ + sys.executable, str(CLIENT), "transform", "--in", str(manifest), + "--take-first", "32", "--timestamp-source", str(source), + "--join-key", "source_index", "--timestamp-field", "timestamp", + "--arrival", "recorded-scaled", "--target-rate", "0.5", + ] + run(common + ["--service-order", "original", "--out", str(base)]) + run(common + [ + "--service-order", "length-binned", "--reorder-block-size", "32", + "--analysis-cohort-size", "16", "--max-added-delay-seconds", "64", + "--out", str(a1), + ]) + base_summary = json.loads((base.with_suffix(".jsonl.summary.json")).read_text()) + a1_summary = json.loads((a1.with_suffix(".jsonl.summary.json")).read_text()) + assert base_summary["rows"] == a1_summary["rows"] == 32 + assert base_summary["request_id_set_sha256"] == a1_summary["request_id_set_sha256"] + assert a1_summary["r16"] < base_summary["r16"] + assert a1_summary["max_added_delay_seconds"] <= 64 + assert "private-" not in json.dumps(a1_summary) + print("phase5 tools: PASS") + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase6/analyze_phase6.py b/runs/opprof-phase6/analyze_phase6.py new file mode 100644 index 0000000..f78ef80 --- /dev/null +++ b/runs/opprof-phase6/analyze_phase6.py @@ -0,0 +1,466 @@ +#!/usr/bin/env python3 +"""Frozen Phase-6 paired-anchor, frontier, rank, and Layer-1 analysis.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +from collections import Counter +from pathlib import Path +from typing import Any + +import numpy as np + + +OLD_BEST = "tp2_mns32" +TOP20 = {"tp2_mns32", "tp2_mns64", "tp4_mns16", "tp4_mns32", "tp4_mns64"} + + +def sha256_file(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def numeric(values: list[float | int | None]) -> dict[str, Any]: + finite = [float(x) for x in values if x is not None and math.isfinite(float(x))] + return { + "n": len(values), "finite_n": len(finite), "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def cv(values: list[float]) -> float: + if not values: + return 0.0 + array = np.asarray(values, dtype=np.float64) + mean = float(array.mean()) + return float(array.std(ddof=0) / mean) if mean else 0.0 + + +def floor_buckets(scores: dict[str, float]) -> tuple[float, dict[str, int]]: + tol = max(1e-9, 1e-6 * max(abs(x) for x in scores.values())) + return tol, {key: math.floor(value / tol) for key, value in scores.items()} + + +def kendall_tau_b(x: dict[str, int], y: dict[str, int]) -> dict[str, Any]: + keys = sorted(x) + c = d = tx = ty = joint = 0 + for i, left in enumerate(keys): + for right in keys[i + 1:]: + dx = (x[left] > x[right]) - (x[left] < x[right]) + dy = (y[left] > y[right]) - (y[left] < y[right]) + if dx == 0 and dy == 0: + joint += 1 + elif dx == 0: + tx += 1 + elif dy == 0: + ty += 1 + elif dx == dy: + c += 1 + else: + d += 1 + denom = math.sqrt((c + d + tx) * (c + d + ty)) + return { + "tau_b": (c - d) / denom if denom else None, "concordant": c, + "discordant": d, "v20_only_ties": tx, "v24_only_ties": ty, + "joint_ties": joint, "pairs": len(keys) * (len(keys) - 1) // 2, + } + + +def layer_metrics(records: list[dict[str, Any]]) -> dict[str, Any]: + model = [x for x in records if x.get("model_executed")] + mode = Counter(str(x["cudagraph"]["runtime_mode"]) for x in model) + total_bucket = sum(int(x["cudagraph"]["bucket_tokens"]) for x in model) + total_padding = sum(int(x["cudagraph"]["padding_tokens"]) for x in model) + kv = [float(x["kv"]["usage"]) for x in model] + waiting = [float(x["queues"]["waiting"]) for x in model] + decode_b = [float(x["decode_batch_size"]) for x in model] + return { + "steps": len(records), "model_steps": len(model), + "prefill_only_steps": sum(int(x["prefill_tokens"]) > 0 and int(x["decode_tokens"]) == 0 for x in model), + "decode_only_steps": sum(int(x["prefill_tokens"]) == 0 and int(x["decode_tokens"]) > 0 for x in model), + "mixed_steps": sum(int(x["prefill_tokens"]) > 0 and int(x["decode_tokens"]) > 0 for x in model), + "prefill_tokens": sum(int(x["prefill_tokens"]) for x in model), + "decode_tokens": sum(int(x["decode_tokens"]) for x in model), + "preemptions": sum(int(x["preemptions"]) for x in model), + "kv_usage_mean": float(np.mean(kv)) if kv else None, + "kv_usage_max": max(kv) if kv else None, + "waiting_mean": float(np.mean(waiting)) if waiting else None, + "waiting_max": max(waiting) if waiting else None, + "waiting_cv": cv(waiting), "decode_B_mean": float(np.mean(decode_b)) if decode_b else None, + "decode_B_cv": cv(decode_b), "graph_mode_counts": dict(sorted(mode.items())), + "graph_mode_shares": {key: value / len(model) for key, value in sorted(mode.items())} if model else {}, + "padding_fraction": total_padding / total_bucket if total_bucket else 0.0, + } + + +def load_stream(path: Path) -> list[dict[str, Any]]: + return [json.loads(line) for line in path.read_text().splitlines() if "step_index" in json.loads(line)] + + +def p3_reference(p3_root: Path) -> dict[str, Any]: + run = p3_root / "primary/P10-C00/moderate" + result = json.loads((run / "client/result.json").read_text()) + t0 = int(result["t0_mono_ns"]) + lo = t0 + int(float(result["clean"]["start_s"]) * 1e9) + hi = t0 + int(float(result["clean"]["end_s"]) * 1e9) + stream = next((run / "opprof").glob("*.jsonl")) + records = [x for x in load_stream(stream) if lo <= int(x["submit_mono_ns"]) < hi] + return { + "limitation": "P3 P10 reference is TP1/MNS-default, steady arrival, sampling_u<=0.125, output<=256; it is contextual rather than a matched v0.20 composition baseline.", + "stream_sha256": sha256_file(stream), "metrics": layer_metrics(records), + } + + +def accepted_anchor(anchor_dir: Path) -> dict[str, Any]: + primary = json.loads((anchor_dir / "result.json").read_text()) + cell_dir = anchor_dir.parent + anchor = float(primary["anchor"]) + confirms = [] + for path in sorted(cell_dir.glob("confirm-*-anchor-*/result.json")): + item = json.loads(path.read_text()) + if math.isclose(float(item["anchor"]), anchor, rel_tol=0, abs_tol=1e-15): + confirms.append(item) + trials = [primary, *confirms] + votes = [bool(x["feasible"]) for x in trials] + if len(votes) == 1 or len(set(votes)) == 1: + verdict = votes[0] + resolved = True + elif len(votes) >= 3: + verdict = sum(votes) >= 2 + resolved = True + else: + verdict = None + resolved = False + return { + "anchor": anchor, "primary": primary, "confirmations": confirms, + "trial_count": len(trials), "pass_rates": [x["pass_rate"] for x in trials], + "feasible_votes": votes, "accepted_feasible": verdict, "resolved": resolved, + "accepted_pass_rate": float(np.median([x["pass_rate"] for x in trials])), + } + + +def matching_anchor_dir(cell_dir: Path, anchor: float) -> Path | None: + for path in cell_dir.glob("anchor-*/result.json"): + item = json.loads(path.read_text()) + if math.isclose(float(item["anchor"]), anchor, rel_tol=0, abs_tol=1e-15): + return path.parent + return None + + +def analyze(root: Path, ground_path: Path, p3_root: Path) -> dict[str, Any]: + ground = json.loads(ground_path.read_text()) + old_cells = {x["cell_id"]: x for x in ground["cells"]} + old_probe = { + (x["cell_id"], float(p["sampling_u"])): p + for x in ground["cells"] for p in x["probe_history"] + } + cells: dict[str, Any] = {} + all_primary = [] + all_client_invariants = [] + selection_matches = [] + for cell, old in old_cells.items(): + cell_dir = root / "solo-authoritative/cells" / cell + colocated_dir = root / "cells" / cell + f20 = float(old["measured_objective"]["slo_feasible_req_s_per_gpu"]) + if not (cell_dir / "cell-valid.json").exists(): + cells[cell] = { + "tp": old["tensor_parallel_size"], "mns": old["max_num_seqs"], + "measurement_status": "UNMEASURED_SOLO", + "f20": f20, "f24": None, "drift": None, + "observed_max_feasible_rate": None, + "material_frontier_moved": None, "boundary": "UNMEASURED", + "bounded": False, "censor": "UNMEASURED_SOLO", + "anchors": [], "cell_valid": None, + } + continue + valid = json.loads((cell_dir / "cell-valid.json").read_text()) + stream = next((cell_dir / "opprof").glob("*.jsonl")) + stream_records = load_stream(stream) + anchors = [] + for path in sorted(cell_dir.glob("anchor-*/result.json")): + accepted = accepted_anchor(path.parent) + primary = accepted["primary"] + key = (cell, float(primary["anchor"])) + historical = old_probe[key] + lo = int(primary["interval"]["start_mono_ns"]) + hi = int(primary["interval"]["end_mono_ns"]) + records = [x for x in stream_records if lo <= int(x["submit_mono_ns"]) <= hi] + accepted["layer1"] = layer_metrics(records) + for confirmation in accepted["confirmations"]: + confirm_lo = int(confirmation["interval"]["start_mono_ns"]) + confirm_hi = int(confirmation["interval"]["end_mono_ns"]) + confirmation["layer1"] = layer_metrics([ + x for x in stream_records + if confirm_lo <= int(x["submit_mono_ns"]) <= confirm_hi + ]) + accepted["v20"] = { + "pass_rate": historical["pass_rate"], "feasible": historical["feasible"], + "request_count": historical["request_count"], + "rate_per_gpu": historical["request_rate_per_gpu_req_s_gpu"], + } + accepted["v24"] = { + "pass_rate": accepted["accepted_pass_rate"], + "feasible": accepted["accepted_feasible"], + "request_count": primary["selection"]["count"], + "rate_per_gpu": primary["selection"]["offered_req_s_per_gpu"], + } + colocated_anchor_dir = matching_anchor_dir(colocated_dir, float(primary["anchor"])) + if colocated_anchor_dir is not None: + colocated = accepted_anchor(colocated_anchor_dir) + accepted["colocated"] = { + "primary_pass_rate": colocated["primary"]["pass_rate"], + "primary_feasible": colocated["primary"]["feasible"], + "trial_pass_rates": colocated["pass_rates"], + "accepted_feasible": colocated["accepted_feasible"], + "solo_minus_colocated_primary_pass_rate": ( + accepted["accepted_pass_rate"] - colocated["primary"]["pass_rate"] + ), + } + else: + accepted["colocated"] = None + accepted["feasibility_flip"] = ( + accepted["accepted_feasible"] is not None + and accepted["accepted_feasible"] != historical["feasible"] + ) + anchors.append(accepted) + all_primary.append(primary) + all_client_invariants.extend(primary["invariants"].values()) + selection_matches.append(primary["selection"]["count"] == historical["request_count"]) + peak_u = float(old["measured_objective"]["best_sampling_u"]) + peak = next(x for x in anchors if math.isclose(x["anchor"], peak_u, abs_tol=1e-15)) + feasible = [x for x in anchors if x["accepted_feasible"] is True] + infeasible = [x for x in anchors if x["accepted_feasible"] is False] + unresolved = [x for x in anchors if x["accepted_feasible"] is None] + observed_frontier = max((x["v24"]["rate_per_gpu"] for x in feasible), default=None) + lower = [x for x in anchors if x["anchor"] < peak_u] + upper = [x for x in anchors if x["anchor"] > peak_u] + if unresolved: + bounded, censor = False, "UNRESOLVED_SOLO_ANCHOR" + elif feasible and infeasible: + monotonic = max(x["anchor"] for x in feasible) < min(x["anchor"] for x in infeasible) + bounded = monotonic + censor = None if bounded else "NONMONOTONIC_SOLO_ANCHORS" + elif feasible: + bounded, censor = False, "RIGHT_CENSORED_HISTORY_EDGE" + elif infeasible: + bounded, censor = False, "LEFT_CENSORED_HISTORY_EDGE" + else: + bounded, censor = False, "NO_RESOLVED_SOLO_ANCHOR" + frontier = ( + None if censor in { + "UNRESOLVED_SOLO_ANCHOR", "NONMONOTONIC_SOLO_ANCHORS", + "NO_RESOLVED_SOLO_ANCHOR", + } else observed_frontier + ) + drift = (frontier / f20 - 1) if frontier is not None else None + if censor == "NONMONOTONIC_SOLO_ANCHORS": + boundary = "NONMONOTONIC" + elif peak["accepted_feasible"] is False: + boundary = "DOWN" + elif peak["accepted_feasible"] is None: + boundary = "UNRESOLVED" + elif any(x["accepted_feasible"] is True for x in upper): + boundary = "UP" + else: + boundary = "STABLE" + cells[cell] = { + "tp": old["tensor_parallel_size"], "mns": old["max_num_seqs"], + "measurement_status": "SOLO_AUTHORITATIVE", + "f20": f20, "f24": frontier, "drift": drift, + "observed_max_feasible_rate": observed_frontier, + "material_frontier_moved": bounded and drift is not None and abs(drift) > .05, + "boundary": boundary, "bounded": bounded, "censor": censor, + "anchors": anchors, "cell_valid": valid, + } + + scores20 = {key: value["f20"] for key, value in cells.items()} + scores24 = {key: value["f24"] for key, value in cells.items() if value["f24"] is not None} + tol20, buckets20 = floor_buckets(scores20) + tol24, buckets24 = floor_buckets(scores24) + full_bounded = len(scores24) == 12 and all(x["bounded"] for x in cells.values()) + max24 = max(buckets24.values()) + if not full_bounded: + argmax = "INCONCLUSIVE" + else: + argmax = "SURVIVED" if buckets24[OLD_BEST] == max24 else "MOVED" + tau = kendall_tau_b(buckets20, buckets24) if full_bounded else None + reversals = [] + for left in sorted(TOP20): + for right in sorted(TOP20): + if left >= right: + continue + if not cells[left]["bounded"] or not cells[right]["bounded"]: + continue + if left not in scores24 or right not in scores24: + continue + old_delta = scores20[left] - scores20[right] + if abs(old_delta) / max(scores20[left], scores20[right]) <= .05: + continue + new_delta = scores24[left] - scores24[right] + if old_delta * new_delta < 0: + reversals.append([left, right]) + if not full_bounded or argmax == "INCONCLUSIVE" or tau is None: + ranking = "INCONCLUSIVE" + elif argmax == "MOVED" or tau["tau_b"] < .8 or reversals: + ranking = "MOVED" + else: + ranking = "SURVIVED" + trap_inputs = ["tp4_mns8", "tp4_mns16", "tp4_mns32"] + if not all(cells[x]["bounded"] for x in trap_inputs): + trap = "INCONCLUSIVE" + elif buckets24["tp4_mns16"] == max24: + trap = "CEASES_TO_BE_A_TRAP" + elif buckets24["tp4_mns16"] >= max(buckets24["tp4_mns8"], buckets24["tp4_mns32"]): + trap = "PERSISTS" + else: + trap = "ESCAPES" + + colocated_state = json.loads((root / "controller-state.json").read_text()) + solo_state_path = root / "solo-authoritative/controller-state.json" + solo_state = json.loads(solo_state_path.read_text()) if solo_state_path.exists() else None + state = solo_state or colocated_state + measured_cells = [ + x for x in cells.values() if x["measurement_status"] == "SOLO_AUTHORITATIVE" + ] + coverage = { + "solo_primary_anchors_at_least_25": len(all_primary) >= 25, + "solo_measured_cells_12": len(measured_cells) == 12, + } + invariants = { + "surface_rows_12": len(cells) == 12, + "selection_counts_match": all(selection_matches), + "client_invariants": all(all_client_invariants), + "cell_validity": all( + all(x["cell_valid"]["invariants"].values()) for x in measured_cells + ), + "gpu_below_6": float(state["gpu_hours_total"]) < 6.0, + "rates_nonnegative": all(x["f24"] is None or x["f24"] >= 0 for x in cells.values()), + "surface_not_identical": len(set(scores24.values())) > 1, + } + red_flags = [key for key, value in {**coverage, **invariants}.items() if not value] + drifted = [key for key, value in cells.items() if value["material_frontier_moved"] is True] + flip_cells = [ + key for key, value in cells.items() + if any(anchor["feasibility_flip"] for anchor in value["anchors"]) + ] + partial = bool([key for key, value in coverage.items() if not value]) + w1_audit_path = root / "w1-readjudication-A-P6-1.json" + w1_audit = json.loads(w1_audit_path.read_text()) if w1_audit_path.exists() else None + censored = {key: value["censor"] for key, value in cells.items() if not value["bounded"]} + colocated_deltas = [ + { + "cell": cell, "anchor": anchor["anchor"], + "solo_pass_rate": anchor["accepted_pass_rate"], + "solo_feasible": anchor["accepted_feasible"], + **anchor["colocated"], + } + for cell, value in cells.items() for anchor in value["anchors"] + if anchor.get("colocated") is not None + ] + return { + "schema": 1, + "status": "BUDGET_STOP_PARTIAL" if partial else ("VALID" if not red_flags else "INVALID"), + "authoritative_tier": "A-P6-2 solo host placement", + "limitation": "Upgrade-path churn includes dash1->dash0 and resolved-default changes. Co-located W1-W3 values are indicative only; P3 composition is contextual, not a matched v0.20 Layer-1 baseline.", + "ground_truth_sha256": sha256_file(ground_path), "cells": cells, + "floor_buckets": {"v20_tol": tol20, "v20": buckets20, "v24_tol": tol24, "v24": buckets24}, + "verdicts": { + "argmax": argmax, "ranking": ranking, "trap": trap, + "full_surface_bounded": full_bounded, "tau_b": tau, + "top_pair_reversals_gt5pct": reversals, + }, + "materially_drifted_cells": drifted, + "feasibility_flip_cells": flip_cells, + "decision_blockers": { + "coverage": [key for key, value in coverage.items() if not value], + "unbounded_or_unresolved_cells": censored, + }, + "solo_vs_colocated": colocated_deltas, + "w1_readjudication": w1_audit, + "run_stats": { + "measured_cells": len(measured_cells), + "surface_cells": len(cells), + "primary_anchor_runs": len(all_primary), + "confirmation_runs": sum( + len(anchor["confirmations"]) + for cell in cells.values() for anchor in cell["anchors"] + ), + "accepted_anchor_trials": sum( + anchor["trial_count"] + for cell in cells.values() for anchor in cell["anchors"] + ), + "warmup_runs": len(measured_cells), + "solo_cell_gpu_hours": { + key: value.get("gpu_hours") for key, value in (solo_state or {}).get("cells", {}).items() + }, + "colocated_wave_gpu_hours": { + key: value.get("gpu_hours") for key, value in colocated_state["waves"].items() + }, + "launch_echo": ( + (root / "launch-echo.log").read_text().splitlines() + + ((root / "solo-authoritative/launch-echo.log").read_text().splitlines() + if (root / "solo-authoritative/launch-echo.log").exists() else []) + ), + }, + "attempt_history": { + "colocated_status": colocated_state["status"], + "colocated_h20_hours": colocated_state["gpu_hours_total"], + "colocated_primary_anchors": colocated_state["completed_primary_anchors"], + "colocated_confirmations": colocated_state["completed_confirmations"], + "colocated_budget_stop": colocated_state.get("budget_stop"), + "solo_status": (solo_state or {}).get("status"), + "solo_repairs": (solo_state or {}).get("repairs", []), + "solo_failures": (solo_state or {}).get("failures", []), + "raw_roots": { + "colocated": str(root / "cells"), + "solo_authoritative": str(root / "solo-authoritative/cells"), + }, + }, + "p3_composition_reference": p3_reference(p3_root), + "gpu": { + "new_h20_hours": state["gpu_hours_total"], "hard_cap": 6.0, + "prior_colocated_h20_hours": colocated_state["gpu_hours_total"], + "solo_h20_hours": (solo_state or {}).get("solo_gpu_hours", 0.0), + "completed_primary_anchors": (solo_state or {}).get("primary_anchors", 0), + "confirmations": (solo_state or {}).get("confirmations", 0), + "controller_status": state["status"], + "budget_stop": state.get("budget_stop"), + }, + "sanity": { + "red_flags": red_flags, "coverage": coverage, "invariants": invariants, + "numeric": { + "v20_score": numeric(list(scores20.values())), + "v24_score": numeric(list(scores24.values())), + "drift": numeric([x["drift"] for x in cells.values()]), + "primary_pass_rate": numeric([x["pass_rate"] for x in all_primary]), + "selected_count": numeric([x["selection"]["count"] for x in all_primary]), + "layer1_steps": numeric([ + anchor["layer1"]["steps"] for cell in cells.values() for anchor in cell["anchors"] + ]), + }, + }, + } + + +def main() -> None: + p = argparse.ArgumentParser() + p.add_argument("--root", type=Path, required=True) + p.add_argument("--ground-truth", type=Path, required=True) + p.add_argument("--p3-root", type=Path, required=True) + p.add_argument("--out", type=Path, required=True) + args = p.parse_args() + result = analyze(args.root, args.ground_truth, args.p3_root) + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(result, sort_keys=True, indent=2) + "\n") + print(json.dumps({"status": result["status"], "verdicts": result["verdicts"], "red_flags": result["sanity"]["red_flags"], "gpu": result["gpu"]}, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase6/opprof_phase6_client.py b/runs/opprof-phase6/opprof_phase6_client.py new file mode 100644 index 0000000..11c5e3c --- /dev/null +++ b/runs/opprof-phase6/opprof_phase6_client.py @@ -0,0 +1,250 @@ +#!/usr/bin/env python3 +"""Exact C1 anchor replay using the pinned AITuner trace/worker/SLO paths.""" + +from __future__ import annotations + +import argparse +import dataclasses +import hashlib +import json +import math +import os +import sys +import time +from pathlib import Path +from typing import Any + + +AITUNER_ROOT = Path(os.environ.get("AITUNER_ROOT", Path(__file__).resolve().parents[2])) +sys.path.insert(0, str(AITUNER_ROOT / "src")) +os.environ.setdefault("AITUNER_CODEX_BASE_URL", "http://127.0.0.1:1") + +from aituner.slo import evaluate_request, summarize_evaluations # noqa: E402 +from aituner.spec import load_study_spec # noqa: E402 +from aituner.trace import load_trace_requests, select_requests_for_threshold # noqa: E402 +from aituner.worker import _probe_drain_deadline, _replay_requests # noqa: E402 + + +def atomic_json(path: Path, payload: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(payload, sort_keys=True, indent=2) + "\n") + os.replace(tmp, path) + + +def sha256_file(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1 << 20), b""): + h.update(chunk) + return h.hexdigest() + + +def numeric(values: list[float | int | None]) -> dict[str, Any]: + finite = [float(x) for x in values if x is not None and math.isfinite(float(x))] + return { + "n": len(values), "finite_n": len(finite), "missing_n": len(values) - len(finite), + "min": min(finite) if finite else None, "max": max(finite) if finite else None, + "distinct_n": len(set(finite)), + } + + +def load_selected(study_path: Path, anchor: float): + study = load_study_spec(study_path) + window, requests = load_trace_requests(study, study_spec_path=study_path) + selected = select_requests_for_threshold(requests, threshold=anchor) + return study, window, requests, selected + + +def selected_summary(selected, duration_s: float, tp: int) -> dict[str, Any]: + ids = "\n".join(item.row_id for item in selected) + arrival = "\n".join(f"{item.arrival_s:.12f}" for item in selected) + lengths = "\n".join(str(item.prompt_tokens_hint) for item in selected) + return { + "count": len(selected), + "offered_req_s": len(selected) / duration_s, + "offered_req_s_per_gpu": len(selected) / duration_s / tp, + "request_id_order_sha256": hashlib.sha256(ids.encode()).hexdigest(), + "arrival_order_sha256": hashlib.sha256(arrival.encode()).hexdigest(), + "raw_length_order_sha256": hashlib.sha256(lengths.encode()).hexdigest(), + "arrival_s": numeric([item.arrival_s for item in selected]), + "raw_input_tokens": numeric([item.prompt_tokens_hint for item in selected]), + "long_gt4096": sum(int(item.prompt_tokens_hint or 0) > 4096 for item in selected), + } + + +def run_replay(args: argparse.Namespace, *, warmup: bool) -> dict[str, Any]: + study_path = Path(args.study) + study, window, _requests, selected = load_selected(study_path, args.anchor) + if warmup: + first = selected[:16] + if not any(int(item.prompt_tokens_hint or 0) > 4096 for item in first): + long_item = next(item for item in selected if int(item.prompt_tokens_hint or 0) > 4096) + first = [*selected[:15], long_item] + first = sorted({item.row_id: item for item in first}.values(), key=lambda item: item.arrival_s) + if len(first) < 16: + raise RuntimeError("warmup set has fewer than 16 unique requests") + start = first[0].arrival_s + selected = [dataclasses.replace(item, arrival_s=item.arrival_s - start) for item in first] + + duration_s = float(window.window_end - window.window_start) + interval_start_mono_ns = time.monotonic_ns() + interval_start_wall_ns = time.time_ns() + outcomes, early_stopped, early_stop_reason = _replay_requests( + selected, + base_url=args.base_url, + timeout_s=study.engine.request_timeout_s, + max_concurrency=study.trace.max_concurrency, + target_pass_rate=(0.0 if warmup else study.slo.target_pass_rate), + max_lag_s=study.trace.early_stop_max_lag_s, + max_elapsed_s=( + 120.0 if warmup else _probe_drain_deadline( + selected, study.slo, ceiling=study.trace.early_stop_max_elapsed_s + ) + ), + evaluate_outcome=lambda outcome: evaluate_request(outcome, study.slo), + drain_inflight_on_early_stop=True, + ) + interval_end_mono_ns = time.monotonic_ns() + interval_end_wall_ns = time.time_ns() + evaluations, slo_summary = summarize_evaluations(outcomes, study.slo) + by_id = {item.row_id: item for item in selected} + details = [] + for outcome, evaluation in zip(outcomes, evaluations): + request = by_id[outcome.request_id] + details.append({ + "request_id": outcome.request_id, + "sampling_u": request.sampling_u, + "arrival_s": request.arrival_s, + "raw_input_tokens": request.prompt_tokens_hint, + "success": outcome.success, + "ttft_ms": outcome.ttft_ms, + "tpot_ms": outcome.tpot_ms, + "completion_tokens": outcome.completion_tokens, + "completion_tokens_source": outcome.completion_tokens_source, + "slo_pass": evaluation.passed, + "reasons": evaluation.reasons, + "error": outcome.error, + }) + out = Path(args.result_dir) + out.mkdir(parents=True, exist_ok=True) + with (out / "requests.jsonl").open("w") as f: + for item in details: + f.write(json.dumps(item, sort_keys=True) + "\n") + summary = selected_summary(selected, duration_s, args.tp) + exact = sum( + item.success and item.completion_tokens_source == "usage" and item.completion_tokens == 128 + for item in outcomes + ) + result = { + "schema": 1, + "kind": "warmup" if warmup else "anchor", + "cell": args.cell, + "anchor": args.anchor, + "tp": args.tp, + "mns": args.mns, + "study_sha256": sha256_file(study_path), + "interval": { + "start_mono_ns": interval_start_mono_ns, "end_mono_ns": interval_end_mono_ns, + "start_wall_ns": interval_start_wall_ns, "end_wall_ns": interval_end_wall_ns, + "elapsed_s": (interval_end_mono_ns - interval_start_mono_ns) / 1e9, + }, + "selection": summary, + "observed_count": len(outcomes), + "exact_output_count": exact, + "slo_pass_count": slo_summary["slo_pass_count"], + "pass_rate": slo_summary["slo_pass_rate"], + "feasible": bool(slo_summary["feasible"]), + "early_stopped": early_stopped, + "early_stop_reason": early_stop_reason, + "ttft_ms": numeric([item.ttft_ms for item in outcomes]), + "tpot_ms": numeric([item.tpot_ms for item in outcomes]), + "invariants": { + "selected_nonempty": bool(selected), + "outcomes_cover_selected": len(outcomes) == len(selected), + "exact_output_or_failed": all( + (not item.success) or ( + item.completion_tokens_source == "usage" and item.completion_tokens == 128 + ) for item in outcomes + ), + "raw_lengths_present": all(item.prompt_tokens_hint is not None for item in selected), + "arrival_nondecreasing": all( + b.arrival_s >= a.arrival_s for a, b in zip(selected, selected[1:]) + ), + "warmup_16": (len(outcomes) >= 16 if warmup else True), + "warmup_exact_16": (exact >= 16 if warmup else True), + "warmup_long": ( + any(int(item.prompt_tokens_hint or 0) > 4096 for item in selected) + if warmup else True + ), + }, + } + atomic_json(out / "result.json", result) + print(json.dumps({k: result[k] for k in ("cell", "anchor", "kind", "pass_rate", "feasible")})) + if not all(result["invariants"].values()): + raise RuntimeError(f"client invariants failed: {result['invariants']}") + return result + + +def preflight(args: argparse.Namespace) -> None: + ground = json.loads(Path(args.ground_truth).read_text()) + studies = {1: Path(args.primary_study), 2: Path(args.primary_study), 4: Path(args.tp4_study)} + loaded = {} + mismatches = [] + values = [] + for cell in ground["cells"]: + tp = int(cell["tensor_parallel_size"]) + if tp not in loaded: + _study, _window, requests, _selected = load_selected(studies[tp], 0.0) + loaded[tp] = requests + for historical in cell["probe_history"]: + selected = select_requests_for_threshold( + loaded[tp], threshold=float(historical["sampling_u"]) + ) + values.append(len(selected)) + if len(selected) != int(historical["request_count"]): + mismatches.append({ + "cell": cell["cell_id"], "anchor": historical["sampling_u"], + "expected": historical["request_count"], "actual": len(selected), + }) + result = { + "schema": 1, "observations": len(values), "mismatches": mismatches, + "request_counts": numeric(values), + "invariants": {"observations_92": len(values) == 92, "counts_match": not mismatches}, + } + atomic_json(Path(args.out), result) + print(json.dumps(result, sort_keys=True)) + if not all(result["invariants"].values()): + raise RuntimeError("preflight count reconstruction failed") + + +def parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser() + sub = p.add_subparsers(dest="command", required=True) + pf = sub.add_parser("preflight") + pf.add_argument("--ground-truth", required=True) + pf.add_argument("--primary-study", required=True) + pf.add_argument("--tp4-study", required=True) + pf.add_argument("--out", required=True) + for name in ("warmup", "run-anchor"): + q = sub.add_parser(name) + q.add_argument("--study", required=True) + q.add_argument("--cell", required=True) + q.add_argument("--anchor", type=float, required=True) + q.add_argument("--tp", type=int, required=True) + q.add_argument("--mns", type=int, required=True) + q.add_argument("--base-url", required=True) + q.add_argument("--result-dir", required=True) + return p + + +def main() -> None: + args = parser().parse_args() + if args.command == "preflight": + preflight(args) + else: + run_replay(args, warmup=args.command == "warmup") + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase6/opprof_phase6_controller.py b/runs/opprof-phase6/opprof_phase6_controller.py new file mode 100644 index 0000000..706c833 --- /dev/null +++ b/runs/opprof-phase6/opprof_phase6_controller.py @@ -0,0 +1,492 @@ +#!/usr/bin/env python3 +"""Detached adaptive Phase-6 controller for the frozen 25-anchor surface.""" + +from __future__ import annotations + +import argparse +import json +import os +import re +import shlex +import signal +import subprocess +import time +import urllib.request +from pathlib import Path +from typing import Any + + +WORKDIR = Path("/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712") +RUN_ROOT = WORKDIR / "runs/phase6" +STATE = RUN_ROOT / "controller-state.json" +SOURCE = Path("/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0") +VENV = Path("/tmp/wjh-opprof-phase2-dash0-20260711/.venv") +AITUNER = Path("/home/admin/cpfs/wjh/aituner/aituner") +MODEL = Path("/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B") +CLIENT = WORKDIR / "scripts/opprof_phase6_client.py" +PRIMARY_STUDY = WORKDIR / "provenance/study-primary.json" +TP4_STUDY = WORKDIR / "provenance/study-tp4.json" +GROUND = WORKDIR / "provenance/ground_truth.json" +GPU_LIMIT = 3.0 +CPU_MAP = {i: f"{20*i}-{20*i+19}" for i in range(8)} +MARKER = "opprof-phase6-20260712" +OWNED_PGIDS: set[int] = set() + + +CELLS = { + "tp1_mns8": {"tp": 1, "mns": 8, "lower": .21875, "peak": .2265625, "upper": .23046875}, + "tp1_mns16": {"tp": 1, "mns": 16, "lower": .2421875, "peak": .24609375, "upper": .25}, + "tp1_mns32": {"tp": 1, "mns": 32, "lower": .234375, "peak": .2421875, "upper": .24609375}, + "tp1_mns64": {"tp": 1, "mns": 64, "lower": .234375, "peak": .2421875, "upper": .24609375}, + "tp2_mns8": {"tp": 2, "mns": 8, "lower": .4921875, "peak": .49609375, "upper": .5}, + "tp2_mns16": {"tp": 2, "mns": 16, "lower": .4921875, "peak": .49609375, "upper": .5}, + "tp2_mns32": {"tp": 2, "mns": 32, "lower": .75, "peak": .75390625, "upper": .7578125}, + "tp2_mns64": {"tp": 2, "mns": 64, "lower": .5, "peak": .75, "upper": .75390625}, + "tp4_mns8": {"tp": 4, "mns": 8, "lower": .016055910008, "peak": .016591107009, "upper": .017126304009}, + "tp4_mns16": {"tp": 4, "mns": 16, "lower": .033182214016, "peak": .033717411016, "upper": .034252608017, "trap": True}, + "tp4_mns32": {"tp": 4, "mns": 32, "lower": .033182214016, "peak": .033717411016, "upper": .034252608017}, + "tp4_mns64": {"tp": 4, "mns": 64, "lower": .033182214016, "peak": .033717411016, "upper": .034252608017}, +} +WAVES = [ + ("W1-tp1", [("tp1_mns8", (0,)), ("tp1_mns16", (1,)), ("tp1_mns32", (2,)), ("tp1_mns64", (3,))], .35), + ("W2-tp2", [("tp2_mns8", (0,1)), ("tp2_mns16", (2,3)), ("tp2_mns32", (4,5)), ("tp2_mns64", (6,7))], .65), + ("W3-tp4-trap", [("tp4_mns8", (0,1,2,3)), ("tp4_mns16", (4,5,6,7))], .85), + ("W4-tp4", [("tp4_mns32", (0,1,2,3)), ("tp4_mns64", (4,5,6,7))], .65), +] + + +def atomic_json(path: Path, value: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(value, sort_keys=True, indent=2) + "\n") + os.replace(tmp, path) + + +def load_state() -> dict[str, Any]: + if STATE.exists(): + return json.loads(STATE.read_text()) + return { + "schema": 1, "status": "initialized", "gpu_hours_total": 0.0, + "completed_primary_anchors": 0, "completed_confirmations": 0, + "waves": {}, "failures": [], "started_at": time.time(), + } + + +def save_state(state: dict[str, Any]) -> None: + atomic_json(STATE, state) + + +def cpu_mask(gpus: tuple[int, ...]) -> str: + return ",".join(CPU_MAP[g] for g in gpus) + + +def study_for(tp: int) -> Path: + return TP4_STUDY if tp == 4 else PRIMARY_STUDY + + +def run_text(command: list[str], check: bool = True) -> str: + result = subprocess.run(command, text=True, capture_output=True) + if check and result.returncode: + raise RuntimeError(f"command failed {command}: {result.stderr}") + return result.stdout + + +def compute_pids() -> list[int]: + text = run_text([ + "nvidia-smi", "--query-compute-apps=pid", "--format=csv,noheader,nounits" + ], check=False) + return sorted({int(x.strip()) for x in text.splitlines() if x.strip().isdigit()}) + + +def pid_owned(pid: int) -> bool: + try: + if os.getpgid(pid) in OWNED_PGIDS: + return True + except ProcessLookupError: + return True + try: + env = Path(f"/proc/{pid}/environ").read_bytes().split(b"\0") + except (FileNotFoundError, PermissionError): + return False + return f"OPPROF_PHASE6_MARKER={MARKER}".encode() in env + + +def assert_no_other_compute() -> None: + other = [pid for pid in compute_pids() if not pid_owned(pid)] + if other: + raise RuntimeError(f"outside GPU processes detected: {other}") + + +def assert_all_idle() -> None: + if compute_pids(): + raise RuntimeError(f"GPU compute processes remain: {compute_pids()}") + rows = run_text([ + "nvidia-smi", "--query-gpu=index,memory.used,utilization.gpu", + "--format=csv,noheader,nounits", + ]) + bad = [] + for line in rows.splitlines(): + index, memory, util = [int(x.strip()) for x in line.split(",")] + if memory != 0 or util != 0: + bad.append((index, memory, util)) + if bad: + raise RuntimeError(f"GPU cleanup failure: {bad}") + + +def wait_ready(entry: dict[str, Any], timeout: float = 300.0) -> None: + deadline = time.monotonic() + timeout + url = f"http://127.0.0.1:{entry['port']}/v1/models" + while time.monotonic() < deadline: + if entry["server"].poll() is not None: + raise RuntimeError(f"server exited before ready: {entry['cell']}") + try: + with urllib.request.urlopen(url, timeout=2) as response: + if response.status < 500: + return + except Exception: + pass + assert_no_other_compute() + time.sleep(1) + raise TimeoutError(f"server ready timeout: {entry['cell']}") + + +def server_command(cell: str, gpus: tuple[int, ...], port: int) -> list[str]: + cfg = CELLS[cell] + return [ + "taskset", "-c", cpu_mask(gpus), str(VENV / "bin/vllm"), "serve", str(MODEL), + "--host", "127.0.0.1", "--port", str(port), + "--served-model-name", "qwen3-30b-a3b-community", + "--max-num-batched-tokens", "8192", "--max-num-seqs", str(cfg["mns"]), + "--tensor-parallel-size", str(cfg["tp"]), "--shutdown-timeout", "120", + ] + + +def client_command(entry: dict[str, Any], anchor: float, out: Path, warmup: bool) -> list[str]: + cfg = CELLS[entry["cell"]] + return [ + "taskset", "-c", cpu_mask(entry["gpus"]), str(VENV / "bin/python"), str(CLIENT), + "warmup" if warmup else "run-anchor", "--study", str(study_for(cfg["tp"])), + "--cell", entry["cell"], "--anchor", str(anchor), "--tp", str(cfg["tp"]), + "--mns", str(cfg["mns"]), "--base-url", f"http://127.0.0.1:{entry['port']}", + "--result-dir", str(out), + ] + + +def live_gpu_hours(entries: list[dict[str, Any]]) -> float: + now = time.time() + return sum( + len(e["gpus"]) * ((e.get("stopped_at") or now) - e["spawned_at"]) + for e in entries + ) / 3600 + + +def run_clients( + entries: list[dict[str, Any]], assignments: list[tuple[dict[str, Any], float, Path]], + state: dict[str, Any], wave_name: str, warmup: bool = False, +) -> list[dict[str, Any]]: + processes = [] + handles = [] + for entry, anchor, out in assignments: + command = client_command(entry, anchor, out, warmup) + with (entry["dir"] / "commands.log").open("a") as f: + f.write(f"CLIENT {shlex.join(command)}\n") + handle = (out.parent / f"{out.name}.log").open("ab", buffering=0) + handles.append(handle) + client_env = os.environ.copy() + client_env.update({"AITUNER_ROOT": str(AITUNER), "PYTHONUNBUFFERED": "1"}) + p = subprocess.Popen( + command, cwd=WORKDIR, env=client_env, stdout=handle, + stderr=subprocess.STDOUT, start_new_session=True, + ) + processes.append((entry, anchor, out, p)) + deadline = time.monotonic() + 180 + while any(p.poll() is None for *_rest, p in processes): + if time.monotonic() > deadline: + raise TimeoutError(f"client batch timeout: {wave_name}") + for entry in entries: + if entry.get("stopped_at") is None and entry["server"].poll() is not None: + raise RuntimeError(f"server exited during client: {entry['cell']}") + assert_no_other_compute() + if state["gpu_hours_total"] + live_gpu_hours(entries) >= GPU_LIMIT: + raise RuntimeError("3.0 H20-hour hard stop reached") + time.sleep(1) + for handle in handles: + handle.close() + bad = [(e["cell"], a, p.returncode) for e, a, _o, p in processes if p.returncode] + if bad: + raise RuntimeError(f"client failures: {bad}") + results = [] + for entry, anchor, out, _p in processes: + result = json.loads((out / "result.json").read_text()) + results.append(result) + entry.setdefault("results", []).append({"anchor": anchor, "dir": str(out), "kind": result["kind"]}) + return results + + +def stop_entry(entry: dict[str, Any]) -> None: + if entry.get("stopped_at") is not None: + return + process = entry["server"] + try: + # Official vLLM shutdown: signal the API parent so EngineCore drains and + # emits the in-stream footer/final sidecar. Process-group signals are + # fallback only. + os.kill(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + try: + process.wait(timeout=150) + except subprocess.TimeoutExpired: + try: + os.killpg(process.pid, signal.SIGTERM) + except ProcessLookupError: + pass + try: + process.wait(timeout=10) + except subprocess.TimeoutExpired: + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + process.wait(timeout=30) + entry["stopped_at"] = time.time() + entry["server_handle"].close() + + +def validate_cell(entry: dict[str, Any]) -> dict[str, Any]: + log = (entry["dir"] / "server.log").read_text(errors="replace").splitlines() + ready = [i for i, line in enumerate(log) if "Application startup complete" in line] + event = re.compile(r"torch\.compile took|Directly load AOT compilation|\bCompiling\b|Capturing CUDA graphs", re.I) + post_ready_events = [i + 1 for i, line in enumerate(log) if event.search(line) and ready and i > ready[0]] + streams = sorted((entry["dir"] / "opprof").glob("*.jsonl")) + sidecars = sorted((entry["dir"] / "opprof").glob("*.jsonl.footer.json")) + if len(streams) != 1 or len(sidecars) != 1: + raise RuntimeError(f"Layer1 stream/sidecar mismatch: {entry['cell']}") + decoded = [json.loads(line) for line in streams[0].read_text().splitlines()] + footers = [item for item in decoded if item.get("record_type") == "footer"] + records = [item for item in decoded if "step_index" in item] + sidecar = json.loads(sidecars[0].read_text()) + indices = [int(item["step_index"]) for item in records] + warm = json.loads((entry["dir"] / "warmup/result.json").read_text()) + intervals_ok = True + for item in entry.get("results", []): + if item["kind"] != "anchor": + continue + result = json.loads((Path(item["dir"]) / "result.json").read_text()) + lo, hi = result["interval"]["start_mono_ns"], result["interval"]["end_mono_ns"] + intervals_ok &= any(lo <= int(r["submit_mono_ns"]) <= hi for r in records) + common = { + "one_ready_marker": len(ready) == 1, + "compile_capture_pre_ready": not post_ready_events, + "warmup_exact_16": warm["exact_output_count"] >= 16, + "warmup_long": warm["selection"]["long_gt4096"] >= 1, + "layer1_contiguous": indices == list(range(len(indices))), + "written_matches_records": sidecar.get("written_records") == len(records), + "encoded_balanced": sidecar.get("encoded_records") == sidecar.get("written_records") + sidecar.get("dropped_records"), + "last_step_matches": bool(records) and sidecar.get("last_step_index") == records[-1]["step_index"], + "layer1_zero_drops": sidecar.get("dropped_records") == 0, + "anchor_intervals_present": intervals_ok, + } + if footers: + accounting = { + "one_footer_last": len(footers) == 1 and decoded[-1] is footers[0], + "sidecar_final": sidecar.get("final") is True, + "footer_sidecar_agrees": all( + footers[0].get(key) == sidecar.get(key) + for key in ("encoded_records", "written_records", "dropped_records") + ), + } + accounting_mode = "graceful-footer" + else: + delta = abs(streams[0].stat().st_mtime_ns - int(sidecar["checkpoint_wall_ns"])) / 1e9 + accounting = { + "checkpoint_sidecar": sidecar.get("final") is False, + "checkpoint_within_flush_of_stream": delta <= float(sidecar["flush_interval_seconds"]) + .1, + } + accounting_mode = "checkpoint-sidecar-fallback" + invariants = {**common, **accounting} + if not all(invariants.values()): + raise RuntimeError(f"cell validity failure {entry['cell']}: {invariants}") + result = { + "cell": entry["cell"], "invariants": invariants, "layer1_records": len(records), + "stream": str(streams[0]), "post_ready_capture_events": post_ready_events, + "accounting_mode": accounting_mode, + } + atomic_json(entry["dir"] / "cell-valid.json", result) + return result + + +def historical_expected() -> dict[tuple[str, float], dict[str, Any]]: + ground = json.loads(GROUND.read_text()) + result = {} + for cell in ground["cells"]: + for probe in cell["probe_history"]: + result[(cell["cell_id"], float(probe["sampling_u"]))] = probe + return result + + +def execute_wave(index: int, state: dict[str, Any], expected: dict[tuple[str, float], dict[str, Any]]) -> None: + wave_name, assignments, estimate = WAVES[index] + if state["waves"].get(wave_name, {}).get("status") == "complete": + return + future = sum(w[2] for w in WAVES[index:]) + .10 + if state["gpu_hours_total"] + future >= GPU_LIMIT: + raise RuntimeError(f"projected budget exceeds cap before {wave_name}: {state['gpu_hours_total']+future}") + echo = ( + f"WAVE_ECHO wave={wave_name} assignments=" + + ",".join(f"{cell}:gpu{'+'.join(map(str, gpus))}" for cell, gpus in assignments) + + f" spent_h20h={state['gpu_hours_total']:.6f} wave_est_h20h={estimate:.3f} " + + f"remaining_projection_h20h={future:.3f} cap_h20h={GPU_LIMIT:.1f} " + + f"ground_truth={GROUND} workload=chat_w20260311_1000.jsonl" + ) + with (RUN_ROOT / "launch-echo.log").open("a") as handle: + handle.write(echo + "\n") + print(echo, flush=True) + assert_all_idle() + wave_dir = RUN_ROOT / "waves" / wave_name + wave_dir.mkdir(parents=True, exist_ok=True) + entries = [] + state["status"] = "running" + state["waves"][wave_name] = {"status": "starting", "estimate_h20_hours": estimate, "started_at": time.time()} + save_state(state) + failure = None + try: + for offset, (cell, gpus) in enumerate(assignments): + cell_dir = RUN_ROOT / "cells" / cell + cell_dir.mkdir(parents=True, exist_ok=True) + port = 8500 + index * 10 + offset + command = server_command(cell, gpus, port) + with (cell_dir / "commands.log").open("a") as f: + f.write(f"SERVER {shlex.join(command)}\n") + handle = (cell_dir / "server.log").open("ab", buffering=0) + env = os.environ.copy() + env.update({ + "CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)), + "VLLM_OPPROF_DIR": str(cell_dir / "opprof"), + "OPPROF_PHASE6_MARKER": MARKER, "AITUNER_ROOT": str(AITUNER), + "HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1", "PYTHONUNBUFFERED": "1", + }) + server = subprocess.Popen(command, cwd=SOURCE, env=env, stdout=handle, stderr=subprocess.STDOUT, start_new_session=True) + OWNED_PGIDS.add(server.pid) + entries.append({ + "cell": cell, "gpus": gpus, "port": port, "dir": cell_dir, + "server": server, "server_handle": handle, "spawned_at": time.time(), + }) + for entry in entries: + wait_ready(entry) + state["waves"][wave_name]["status"] = "warmup" + save_state(state) + run_clients(entries, [ + (e, CELLS[e["cell"]]["peak"], e["dir"] / "warmup") for e in entries + ], state, wave_name, warmup=True) + state["waves"][wave_name]["status"] = "peaks" + save_state(state) + peaks = run_clients(entries, [ + (e, CELLS[e["cell"]]["peak"], e["dir"] / f"anchor-{CELLS[e['cell']]['peak']}") + for e in entries + ], state, wave_name) + for result in peaks: + old = expected[(result["cell"], float(result["anchor"]))] + if result["selection"]["count"] != old["request_count"]: + raise RuntimeError(f"selection mismatch {result['cell']} peak") + neighbor_jobs = [] + for entry, result in zip(entries, peaks, strict=True): + key = "upper" if result["feasible"] else "lower" + anchor = CELLS[entry["cell"]][key] + neighbor_jobs.append((entry, anchor, entry["dir"] / f"anchor-{anchor}")) + neighbors = run_clients(entries, neighbor_jobs, state, wave_name) + for result in neighbors: + old = expected[(result["cell"], float(result["anchor"]))] + if result["selection"]["count"] != old["request_count"]: + raise RuntimeError(f"selection mismatch {result['cell']} neighbor") + trap_entry = next((e for e in entries if CELLS[e["cell"]].get("trap")), None) + if trap_entry is not None: + used = {float(item["anchor"]) for item in trap_entry["results"] if item["kind"] == "anchor"} + extra = next(CELLS[trap_entry["cell"]][k] for k in ("lower", "upper") if CELLS[trap_entry["cell"]][k] not in used) + extra_result = run_clients(entries, [(trap_entry, extra, trap_entry["dir"] / f"anchor-{extra}")], state, wave_name)[0] + if extra_result["selection"]["count"] != expected[(extra_result["cell"], float(extra))]["request_count"]: + raise RuntimeError("trap extra selection mismatch") + + # Confirm only protocol-triggered anchors while the relevant server is hot. + triggers = [] + for entry in entries: + for item in entry.get("results", []): + if item["kind"] != "anchor": + continue + result = json.loads((Path(item["dir"]) / "result.json").read_text()) + old = expected[(entry["cell"], float(result["anchor"]))] + flip = bool(result["feasible"]) != bool(old["feasible"]) + if .93 <= float(result["pass_rate"]) <= .97 or (entry["cell"] in {"tp2_mns32", "tp4_mns16"} and flip): + priority = 0 if entry["cell"] == "tp2_mns32" else (1 if entry["cell"] == "tp4_mns16" else 2) + triggers.append((priority, entry, float(result["anchor"]))) + triggers.sort(key=lambda x: x[0]) + for _priority, entry, anchor in triggers: + projected_extra = len(entry["gpus"]) * 80 / 3600 + future_primary = sum(w[2] for w in WAVES[index + 1:]) + if state["gpu_hours_total"] + live_gpu_hours(entries) + future_primary + projected_extra + .03 >= GPU_LIMIT: + state.setdefault("unconfirmed_triggers", []).append({"cell": entry["cell"], "anchor": anchor}) + continue + confirm_index = 1 + sum(1 for item in entry.get("results", []) if item["kind"] == "anchor" and Path(item["dir"]).name.startswith("confirm")) + out = entry["dir"] / f"confirm-{confirm_index}-anchor-{anchor}" + run_clients(entries, [(entry, anchor, out)], state, wave_name) + state["completed_confirmations"] += 1 + state["waves"][wave_name]["status"] = "stopping" + save_state(state) + except Exception as error: + failure = error + finally: + for entry in entries: + try: + stop_entry(entry) + except Exception as error: + failure = failure or error + time.sleep(2) + try: + assert_all_idle() + except Exception as error: + failure = failure or error + wave_hours = live_gpu_hours(entries) + state["gpu_hours_total"] += wave_hours + state["waves"][wave_name]["gpu_hours"] = wave_hours + if failure is not None: + state["waves"][wave_name]["status"] = "failed" + state["waves"][wave_name]["failure"] = repr(failure) + state["status"] = "failed" + state["failures"].append({"wave": wave_name, "failure": repr(failure)}) + save_state(state) + raise failure + validations = [validate_cell(entry) for entry in entries] + primary_count = sum( + 1 for entry in entries for item in entry.get("results", []) + if item["kind"] == "anchor" and not Path(item["dir"]).name.startswith("confirm") + ) + state["completed_primary_anchors"] += primary_count + state["waves"][wave_name].update({ + "status": "complete", "completed_at": time.time(), "primary_anchors": primary_count, + "validations": validations, + }) + save_state(state) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--resume", action="store_true") + args = parser.parse_args() + RUN_ROOT.mkdir(parents=True, exist_ok=True) + state = load_state() + expected = historical_expected() + state["status"] = "running" + save_state(state) + for index in range(len(WAVES)): + execute_wave(index, state, expected) + state["status"] = "primary_complete" + state["completed_at"] = time.time() + save_state(state) + print(json.dumps({ + "status": state["status"], "primary_anchors": state["completed_primary_anchors"], + "confirmations": state["completed_confirmations"], "gpu_hours": state["gpu_hours_total"], + }, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase6/opprof_phase6_solo_controller.py b/runs/opprof-phase6/opprof_phase6_solo_controller.py new file mode 100644 index 0000000..c4170af --- /dev/null +++ b/runs/opprof-phase6/opprof_phase6_solo_controller.py @@ -0,0 +1,419 @@ +#!/usr/bin/env python3 +"""A-P6-2 serialized solo controller for authoritative Phase-6 frontiers.""" + +from __future__ import annotations + +import json +import math +import os +import shlex +import subprocess +import time +from pathlib import Path +from typing import Any + +import opprof_phase6_controller as base + + +SOLO_ROOT = base.RUN_ROOT / "solo-authoritative" +STATE = SOLO_ROOT / "controller-state.json" +CAMPAIGN_STATE = base.RUN_ROOT / "controller-state.json" +GPU_LIMIT = 6.0 +SAFETY_HOURS = 0.20 +MARKER = "opprof-phase6-solo-A-P6-2" +TRACE = base.AITUNER / "trace_windows/traces/chat_w20260311_1000.jsonl" + +ORDER = [ + "tp4_mns32", "tp4_mns64", "tp2_mns32", "tp2_mns64", + "tp4_mns16", "tp2_mns8", "tp2_mns16", "tp4_mns8", + "tp1_mns8", "tp1_mns16", "tp1_mns32", "tp1_mns64", +] + +# Exact co-located primaries are remeasured before adaptive crawl. W4 had no +# prior primary, so it starts at P and selects L/U from the solo result. +CORE = { + "tp1_mns8": [base.CELLS["tp1_mns8"]["peak"], base.CELLS["tp1_mns8"]["lower"]], + "tp1_mns16": [base.CELLS["tp1_mns16"]["peak"], base.CELLS["tp1_mns16"]["upper"]], + "tp1_mns32": [base.CELLS["tp1_mns32"]["peak"], base.CELLS["tp1_mns32"]["upper"]], + "tp1_mns64": [base.CELLS["tp1_mns64"]["peak"], base.CELLS["tp1_mns64"]["upper"]], + "tp2_mns8": [base.CELLS["tp2_mns8"]["peak"], base.CELLS["tp2_mns8"]["lower"]], + "tp2_mns16": [base.CELLS["tp2_mns16"]["peak"], base.CELLS["tp2_mns16"]["lower"]], + "tp2_mns32": [base.CELLS["tp2_mns32"]["peak"], base.CELLS["tp2_mns32"]["lower"]], + "tp2_mns64": [base.CELLS["tp2_mns64"]["peak"], base.CELLS["tp2_mns64"]["lower"]], + "tp4_mns8": [base.CELLS["tp4_mns8"]["peak"], base.CELLS["tp4_mns8"]["lower"]], + "tp4_mns16": [ + base.CELLS["tp4_mns16"]["peak"], base.CELLS["tp4_mns16"]["lower"], + base.CELLS["tp4_mns16"]["upper"], + ], + "tp4_mns32": [base.CELLS["tp4_mns32"]["peak"]], + "tp4_mns64": [base.CELLS["tp4_mns64"]["peak"]], +} + +CELL_ESTIMATE = {cell: {1: .11, 2: .22, 4: .48}[cfg["tp"]] for cell, cfg in base.CELLS.items()} + + +def atomic_json(path: Path, value: Any) -> None: + base.atomic_json(path, value) + + +def load_state() -> dict[str, Any]: + if STATE.exists(): + return json.loads(STATE.read_text()) + campaign = json.loads(CAMPAIGN_STATE.read_text()) + return { + "schema": 1, "amendment": "A-P6-2", "status": "initialized", + "hard_cap_h20_hours": GPU_LIMIT, + "prior_h20_hours": float(campaign["gpu_hours_total"]), + "gpu_hours_total": float(campaign["gpu_hours_total"]), + "solo_gpu_hours": 0.0, "completed_cells": 0, + "primary_anchors": 0, "confirmations": 0, + "cells": {}, "failures": [], "started_at": time.time(), + } + + +def save_state(state: dict[str, Any]) -> None: + atomic_json(STATE, state) + + +def historical() -> tuple[dict[tuple[str, float], dict[str, Any]], dict[str, list[float]]]: + ground = json.loads(base.GROUND.read_text()) + expected = {} + histories = {} + for cell in ground["cells"]: + anchors = [] + for probe in cell["probe_history"]: + anchor = float(probe["sampling_u"]) + expected[(cell["cell_id"], anchor)] = probe + anchors.append(anchor) + histories[cell["cell_id"]] = sorted(anchors) + return expected, histories + + +def same_anchor(left: float, right: float) -> bool: + return math.isclose(left, right, rel_tol=0, abs_tol=1e-15) + + +def colocated_primary(cell: str, anchor: float) -> dict[str, Any] | None: + cell_dir = base.RUN_ROOT / "cells" / cell + for path in cell_dir.glob("anchor-*/result.json"): + item = json.loads(path.read_text()) + if same_anchor(float(item["anchor"]), anchor): + return item + return None + + +def append_echo(line: str) -> None: + SOLO_ROOT.mkdir(parents=True, exist_ok=True) + with (SOLO_ROOT / "launch-echo.log").open("a") as handle: + handle.write(line + "\n") + print(line, flush=True) + + +def wait_all_idle(timeout: float = 30.0) -> None: + deadline = time.monotonic() + timeout + error = None + while time.monotonic() < deadline: + try: + base.assert_all_idle() + return + except RuntimeError as current: + error = current + time.sleep(1) + raise error or RuntimeError("GPU cleanup did not reach idle") + + +def remaining_projection(index: int) -> float: + return sum(CELL_ESTIMATE[cell] for cell in ORDER[index:]) + SAFETY_HOURS + + +def start_entry(cell: str, index: int) -> dict[str, Any]: + cfg = base.CELLS[cell] + gpus = tuple(range(int(cfg["tp"]))) + cell_dir = SOLO_ROOT / "cells" / cell + cell_dir.mkdir(parents=True, exist_ok=True) + port = 8700 + index + command = base.server_command(cell, gpus, port) + with (cell_dir / "commands.log").open("a") as handle: + handle.write(f"SERVER {shlex.join(command)}\n") + server_handle = (cell_dir / "server.log").open("ab", buffering=0) + env = os.environ.copy() + env.update({ + "CUDA_VISIBLE_DEVICES": ",".join(map(str, gpus)), + "VLLM_OPPROF_DIR": str(cell_dir / "opprof"), + "OPPROF_PHASE6_MARKER": MARKER, "AITUNER_ROOT": str(base.AITUNER), + "HF_HUB_OFFLINE": "1", "TRANSFORMERS_OFFLINE": "1", "PYTHONUNBUFFERED": "1", + }) + server = subprocess.Popen( + command, cwd=base.SOURCE, env=env, stdout=server_handle, + stderr=subprocess.STDOUT, start_new_session=True, + ) + base.OWNED_PGIDS.add(server.pid) + return { + "cell": cell, "gpus": gpus, "port": port, "dir": cell_dir, + "server": server, "server_handle": server_handle, + "spawned_at": time.time(), "results": [], + } + + +def run_one( + entry: dict[str, Any], anchor: float, out: Path, state: dict[str, Any], + cell_state: dict[str, Any], role: str, +) -> dict[str, Any]: + result = base.run_clients( + [entry], [(entry, anchor, out)], state, f"solo-{entry['cell']}" + )[0] + expected_count = cell_state["expected_counts"][str(anchor)] + if int(result["selection"]["count"]) != int(expected_count): + raise RuntimeError( + f"selection mismatch {entry['cell']} {anchor}: " + f"{result['selection']['count']} != {expected_count}" + ) + cell_state.setdefault("runs", []).append({ + "anchor": anchor, "role": role, "dir": str(out), + "pass_rate": result["pass_rate"], "feasible": result["feasible"], + }) + save_state(state) + return result + + +def anchor_trials(cell_state: dict[str, Any], anchor: float) -> list[dict[str, Any]]: + return [ + item for item in cell_state.get("runs", []) + if same_anchor(float(item["anchor"]), anchor) + ] + + +def accepted_feasible(cell_state: dict[str, Any], anchor: float) -> bool | None: + trials = anchor_trials(cell_state, anchor) + votes = [bool(item["feasible"]) for item in trials] + if not votes: + return None + if len(votes) == 1 or len(set(votes)) == 1: + return votes[0] + if len(votes) >= 3: + return sum(votes) >= 2 + return None + + +def optional_fits( + state: dict[str, Any], entry: dict[str, Any], future_after: float, +) -> bool: + replay = len(entry["gpus"]) * 80 / 3600 + projected = ( + float(state["gpu_hours_total"]) + base.live_gpu_hours([entry]) + + future_after + replay + SAFETY_HOURS + ) + return projected < GPU_LIMIT + + +def maybe_confirm( + entry: dict[str, Any], anchor: float, primary: dict[str, Any], + state: dict[str, Any], cell_state: dict[str, Any], expected: dict[tuple[str, float], dict[str, Any]], + future_after: float, +) -> None: + old = expected[(entry["cell"], anchor)] + coloc = colocated_primary(entry["cell"], anchor) + disagreement = ( + bool(primary["feasible"]) != bool(old["feasible"]) + or (coloc is not None and bool(primary["feasible"]) != bool(coloc["feasible"])) + ) + boundary = .93 <= float(primary["pass_rate"]) <= .97 + if not (disagreement or boundary): + return + while len(anchor_trials(cell_state, anchor)) < 3: + trials = anchor_trials(cell_state, anchor) + if len(trials) >= 2 and len({bool(item["feasible"]) for item in trials}) == 1: + return + if not optional_fits(state, entry, future_after): + cell_state.setdefault("deferred_confirmations", []).append(anchor) + return + trial = len(trials) + 1 + out = entry["dir"] / f"confirm-{trial - 1}-anchor-{anchor}" + run_one(entry, anchor, out, state, cell_state, f"confirmation-{trial}") + state["confirmations"] += 1 + + +def run_primary( + entry: dict[str, Any], anchor: float, state: dict[str, Any], + cell_state: dict[str, Any], expected: dict[tuple[str, float], dict[str, Any]], + future_after: float, role: str, +) -> dict[str, Any]: + existing = [item for item in cell_state.get("runs", []) if item["role"].startswith("primary")] + if any(same_anchor(float(item["anchor"]), anchor) for item in existing): + path = next( + Path(item["dir"]) for item in existing + if same_anchor(float(item["anchor"]), anchor) + ) + return json.loads((path / "result.json").read_text()) + out = entry["dir"] / f"anchor-{anchor}" + result = run_one(entry, anchor, out, state, cell_state, role) + state["primary_anchors"] += 1 + maybe_confirm(entry, anchor, result, state, cell_state, expected, future_after) + return result + + +def next_below(history: list[float], tested: set[float]) -> float | None: + if not tested: + return None + candidates = [x for x in history if x < min(tested) and x not in tested] + return max(candidates) if candidates else None + + +def next_above(history: list[float], tested: set[float]) -> float | None: + if not tested: + return None + candidates = [x for x in history if x > max(tested) and x not in tested] + return min(candidates) if candidates else None + + +def execute_cell( + index: int, cell: str, state: dict[str, Any], + expected: dict[tuple[str, float], dict[str, Any]], histories: dict[str, list[float]], +) -> None: + if state["cells"].get(cell, {}).get("status") == "complete": + return + future = remaining_projection(index) + if float(state["gpu_hours_total"]) + future >= GPU_LIMIT: + state["status"] = "budget_projection_stop" + state["budget_stop"] = { + "before_cell": cell, "spent_h20_hours": state["gpu_hours_total"], + "remaining_projection_h20_hours": future, + "projected_total_h20_hours": state["gpu_hours_total"] + future, + "hard_cap_h20_hours": GPU_LIMIT, + } + save_state(state) + raise RuntimeError(f"projected budget exceeds cap before {cell}") + cfg = base.CELLS[cell] + echo = ( + f"SOLO_WAVE_ECHO cell={cell} tp={cfg['tp']} mns={cfg['mns']} " + f"gpus=0-{int(cfg['tp'])-1} mandatory={','.join(map(str, CORE[cell]))} " + f"spent_h20h={state['gpu_hours_total']:.6f} cell_est_h20h={CELL_ESTIMATE[cell]:.3f} " + f"remaining_projection_h20h={future:.3f} cap_h20h={GPU_LIMIT:.1f} " + f"ground_truth={base.GROUND} trace={TRACE}" + ) + append_echo(echo) + wait_all_idle() + cell_state = { + "status": "starting", "started_at": time.time(), "tp": cfg["tp"], "mns": cfg["mns"], + "mandatory": CORE[cell], + "expected_counts": { + str(anchor): expected[(cell, anchor)]["request_count"] for anchor in histories[cell] + }, + "runs": [], + } + state["status"] = "running" + state["cells"][cell] = cell_state + save_state(state) + entry = start_entry(cell, index) + failure = None + future_after = sum(CELL_ESTIMATE[item] for item in ORDER[index + 1:]) + try: + base.wait_ready(entry) + cell_state["status"] = "warmup" + save_state(state) + warm = base.run_clients( + [entry], [(entry, cfg["peak"], entry["dir"] / "warmup")], + state, f"solo-{cell}", warmup=True, + )[0] + cell_state["warmup"] = { + "exact_output_count": warm["exact_output_count"], + "long_gt4096": warm["selection"]["long_gt4096"], + } + cell_state["status"] = "mandatory" + save_state(state) + for anchor in CORE[cell]: + run_primary(entry, anchor, state, cell_state, expected, future_after, "primary-mandatory") + + peak = float(cfg["peak"]) + peak_vote = accepted_feasible(cell_state, peak) + if cell in {"tp4_mns32", "tp4_mns64"} and peak_vote is not None: + direction = float(cfg["upper"] if peak_vote else cfg["lower"]) + run_primary(entry, direction, state, cell_state, expected, future_after, "primary-direction") + + cell_state["status"] = "crawl" + save_state(state) + while True: + primary_anchors = { + float(item["anchor"]) for item in cell_state["runs"] + if item["role"].startswith("primary") + } + votes = {anchor: accepted_feasible(cell_state, anchor) for anchor in primary_anchors} + pass_anchors = [anchor for anchor, vote in votes.items() if vote is True] + fail_anchors = [anchor for anchor, vote in votes.items() if vote is False] + if pass_anchors and fail_anchors and max(pass_anchors) < min(fail_anchors): + break + if any(vote is None for vote in votes.values()): + cell_state["censor"] = "UNRESOLVED_SOLO_ANCHOR" + break + if pass_anchors and not fail_anchors: + candidate = next_above(histories[cell], primary_anchors) + elif fail_anchors and not pass_anchors: + candidate = next_below(histories[cell], primary_anchors) + else: + # Non-monotonic anchors already contain both states but no valid bracket. + cell_state["censor"] = "NONMONOTONIC_SOLO_ANCHORS" + break + if candidate is None: + cell_state["censor"] = "HISTORY_EDGE" + break + if not optional_fits(state, entry, future_after): + cell_state["censor"] = "BUDGET_CENSORED" + break + run_primary(entry, candidate, state, cell_state, expected, future_after, "primary-crawl") + cell_state["status"] = "stopping" + save_state(state) + except Exception as error: + failure = error + finally: + try: + base.stop_entry(entry) + except Exception as error: + failure = failure or error + time.sleep(2) + try: + wait_all_idle() + except Exception as error: + failure = failure or error + hours = base.live_gpu_hours([entry]) + state["gpu_hours_total"] += hours + state["solo_gpu_hours"] += hours + cell_state["gpu_hours"] = hours + if failure is not None: + cell_state["status"] = "failed" + cell_state["failure"] = repr(failure) + state["status"] = "failed" + state["failures"].append({"cell": cell, "failure": repr(failure)}) + save_state(state) + raise failure + validation = base.validate_cell(entry) + cell_state["validation"] = validation + cell_state["status"] = "complete" + cell_state["completed_at"] = time.time() + state["completed_cells"] += 1 + save_state(state) + + +def main() -> None: + SOLO_ROOT.mkdir(parents=True, exist_ok=True) + base.GPU_LIMIT = GPU_LIMIT + base.MARKER = MARKER + expected, histories = historical() + state = load_state() + state["status"] = "running" + save_state(state) + for index, cell in enumerate(ORDER): + execute_cell(index, cell, state, expected, histories) + state["status"] = "complete" + state["completed_at"] = time.time() + save_state(state) + print(json.dumps({ + "status": state["status"], "cells": state["completed_cells"], + "primary_anchors": state["primary_anchors"], + "confirmations": state["confirmations"], + "solo_gpu_hours": state["solo_gpu_hours"], + "campaign_gpu_hours": state["gpu_hours_total"], + }, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase6/p3-reference/primary/P10-C00/moderate/client/result.json b/runs/opprof-phase6/p3-reference/primary/P10-C00/moderate/client/result.json new file mode 100644 index 0000000..8a0dff9 --- /dev/null +++ b/runs/opprof-phase6/p3-reference/primary/P10-C00/moderate/client/result.json @@ -0,0 +1,103 @@ +{ + "admission_stop_s": 363.15318555399426, + "arrival": "steady", + "clean": { + "admitted": 113, + "completed": 111, + "completed_throughput_rps": 0.4625, + "duration_s": 240.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 968714, + "offered_rps": 0.4708333333333333, + "output_tokens": 25409, + "start_s": 60.0 + }, + "clean_segment_seconds": 80.0, + "drain_seconds": 1.3658369119802956, + "elapsed_seconds": 364.51902246597456, + "failed_records": 0, + "load_point": "moderate", + "manifest_admitted": 172, + "manifest_exhausted": false, + "manifest_rows": 4011, + "manifest_sha256": "f51b7a1cc657d62b9ea81823c754408732326b06e03439452433cd8ed481bf33", + "manifest_wrapped": false, + "max_in_flight": 7, + "num_clean_segments": 3, + "profiles": [ + { + "start_call_s": 300.00394913199125, + "start_return_s": 300.03973603399936, + "start_status": 200, + "stop_call_s": 300.89519165997626, + "stop_return_s": 301.6286224719952, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849466906401944.pt.trace.json.gz", + "trace_ready_s": 300.8951868820004, + "trace_sha256": "a15670f86d843d40e7c4b6f354dd0193ea1259824a96440392d923f5270ba095", + "window": 1 + }, + { + "start_call_s": 331.62960390897933, + "start_return_s": 331.63382844498847, + "start_status": 200, + "stop_call_s": 332.4971265429922, + "stop_return_s": 333.15040952397976, + "stop_status": 200, + "trace_file": "dp0_pp0_tp0_dcp0_ep0_rank0.1783849498465727706.pt.trace.json.gz", + "trace_ready_s": 332.4971234249824, + "trace_sha256": "0f5962e193e09a9b2f1df6cea23f92419dcf2b322ee4ef2135a4fc3e1fe0617c", + "window": 2 + } + ], + "rate_fraction": 0.6, + "records": 172, + "request_rate": 0.4725, + "schema": 1, + "segments": [ + { + "admitted": 38, + "completed": 37, + "completed_throughput_rps": 0.4625, + "duration_s": 80.0, + "end_s": 140.0, + "failed": 0, + "input_tokens": 404192, + "name": "A", + "offered_rps": 0.475, + "output_tokens": 8525, + "start_s": 60.0 + }, + { + "admitted": 37, + "completed": 38, + "completed_throughput_rps": 0.475, + "duration_s": 80.0, + "end_s": 220.0, + "failed": 0, + "input_tokens": 327301, + "name": "B", + "offered_rps": 0.4625, + "output_tokens": 8617, + "start_s": 140.0 + }, + { + "admitted": 38, + "completed": 36, + "completed_throughput_rps": 0.45, + "duration_s": 80.0, + "end_s": 300.0, + "failed": 0, + "input_tokens": 237221, + "name": "C", + "offered_rps": 0.475, + "output_tokens": 8267, + "start_s": 220.0 + } + ], + "successful_records": 172, + "t0_mono_ns": 180739547167834, + "t0_wall_ns": 1783849166673595809, + "warmup_seconds": 60.0 +} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375.log b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375.log new file mode 100644 index 0000000..2acdd5d --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375.log @@ -0,0 +1 @@ +{"cell": "tp1_mns16", "anchor": 0.24609375, "kind": "anchor", "pass_rate": 1.0, "feasible": true} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375/result.json b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375/result.json new file mode 100644 index 0000000..6f9fa80 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.24609375/result.json @@ -0,0 +1,74 @@ +{ + "anchor": 0.24609375, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.328174978, + "end_mono_ns": 199601637648726, + "end_wall_ns": 1783868028764076133, + "start_mono_ns": 199540309473748, + "start_wall_ns": 1783867967435901551 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 39.29021051171873, + "min": 4.445230456776771, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1787.3226249939762, + "min": 37.47074800776318, + "missing_n": 0, + "n": 141 + } +} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25.log b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25.log new file mode 100644 index 0000000..79aba70 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25.log @@ -0,0 +1 @@ +{"cell": "tp1_mns16", "anchor": 0.25, "kind": "anchor", "pass_rate": 1.0, "feasible": true} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25/result.json b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25/result.json new file mode 100644 index 0000000..5bb1774 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/anchor-0.25/result.json @@ -0,0 +1,74 @@ +{ + "anchor": 0.25, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.471211524, + "end_mono_ns": 199668842271892, + "end_wall_ns": 1783868095968699176, + "start_mono_ns": 199607371060368, + "start_wall_ns": 1783868034497487431 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 44.43295162989699, + "min": 4.492281684945301, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1784.063341008732, + "min": 57.29520702152513, + "missing_n": 0, + "n": 143 + } +} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/cell-valid.json b/runs/opprof-phase6/phase6/cells/tp1_mns16/cell-valid.json new file mode 100644 index 0000000..fd79348 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/cell-valid.json @@ -0,0 +1,21 @@ +{ + "accounting_mode": "A-P6-1-checkpoint-sidecar", + "cell": "tp1_mns16", + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "layer1_records": 9212, + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl" +} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/commands.log b/runs/opprof-phase6/phase6/cells/tp1_mns16/commands.log new file mode 100644 index 0000000..ce1d5f6 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/commands.log @@ -0,0 +1,4 @@ +SERVER taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8501 --served-model-name qwen3-30b-a3b-community --max-num-batched-tokens 8192 --max-num-seqs 16 --tensor-parallel-size 1 --shutdown-timeout 120 +CLIENT taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py warmup --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.24609375 --tp 1 --mns 16 --base-url http://127.0.0.1:8501 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/warmup +CLIENT taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.24609375 --tp 1 --mns 16 --base-url http://127.0.0.1:8501 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.24609375 +CLIENT taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.25 --tp 1 --mns 16 --base-url http://127.0.0.1:8501 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.25 diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl.footer.json b/runs/opprof-phase6/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl.footer.json new file mode 100644 index 0000000..f9c08c7 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl.footer.json @@ -0,0 +1 @@ +{"schema":1,"record_type":"footer_checkpoint","stream":"opprof-v1-dp0-pid2638886-1783867898065648658.jsonl","encoded_records":9212,"written_records":9212,"dropped_records":0,"last_step_index":9211,"checkpoint_wall_ns":1783868096969027153,"flush_interval_seconds":1.0,"final":false} diff --git a/runs/opprof-phase6/phase6/cells/tp1_mns16/server.log b/runs/opprof-phase6/phase6/cells/tp1_mns16/server.log new file mode 100644 index 0000000..ca4bb58 --- /dev/null +++ b/runs/opprof-phase6/phase6/cells/tp1_mns16/server.log @@ -0,0 +1,442 @@ +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] █ █ █▄ ▄█ +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2 +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:339] +(APIServer pid=2638187) INFO 07-12 14:50:41 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8501, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120} +(APIServer pid=2638187) INFO 07-12 14:50:52 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM +(APIServer pid=2638187) INFO 07-12 14:50:52 [model.py:1725] Using max model len 40960 +(APIServer pid=2638187) INFO 07-12 14:50:52 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192. +(APIServer pid=2638187) INFO 07-12 14:50:52 [vllm.py:1006] Asynchronous scheduling is enabled. +(APIServer pid=2638187) INFO 07-12 14:50:52 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) +(EngineCore pid=2638886) INFO 07-12 14:51:03 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2638886) INFO 07-12 14:51:05 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:37021 backend=nccl +(EngineCore pid=2638886) INFO 07-12 14:51:05 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2638886) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2638886) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2638886) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2638886) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2638886) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2638886) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB. +(EngineCore pid=2638886) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2638886) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2638900) INFO 07-12 14:51:05 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:50247 backend=nccl +(EngineCore pid=2638900) INFO 07-12 14:51:05 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2638900) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2638900) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2638900) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2638900) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2638900) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2638900) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB. +(EngineCore pid=2638900) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2638900) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2638898) INFO 07-12 14:51:06 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:39471 backend=nccl +(EngineCore pid=2638898) INFO 07-12 14:51:06 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2638898) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2638898) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2638898) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2638898) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2638898) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2638898) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB. +(EngineCore pid=2638898) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2638898) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2638884) INFO 07-12 14:51:06 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:48769 backend=nccl +(EngineCore pid=2638884) INFO 07-12 14:51:06 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2638884) INFO 07-12 14:51:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2638884) INFO 07-12 14:51:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2638884) INFO 07-12 14:51:07 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2638884) INFO 07-12 14:51:07 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2638884) INFO 07-12 14:51:07 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2638884) INFO 07-12 14:51:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1283.36 GiB. +(EngineCore pid=2638884) INFO 07-12 14:51:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2638884) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2661159) WARNING 07-12 15:16:53 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2661159) INFO 07-12 15:16:53 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2661557) INFO 07-12 15:17:04 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:48921 backend=nccl +(Worker pid=2661558) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:48921 backend=nccl +(Worker pid=2661557) INFO 07-12 15:17:06 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2661557) INFO 07-12 15:17:08 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661557) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661557) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2661557) INFO 07-12 15:17:09 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2661557) INFO 07-12 15:17:09 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.69 GiB. +(Worker_TP0 pid=2661557) INFO 07-12 15:17:10 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2661557) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2661056) WARNING 07-12 15:16:52 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2661056) INFO 07-12 15:16:52 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2661549) INFO 07-12 15:17:04 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:34189 backend=nccl +(Worker pid=2661550) INFO 07-12 15:17:04 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:34189 backend=nccl +(Worker pid=2661549) INFO 07-12 15:17:05 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2661549) INFO 07-12 15:17:08 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661549) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661549) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2661549) INFO 07-12 15:17:09 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2661549) INFO 07-12 15:17:09 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.68 GiB. +(Worker_TP0 pid=2661549) INFO 07-12 15:17:10 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2661549) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2661205) WARNING 07-12 15:16:54 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2661205) INFO 07-12 15:16:54 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2661569) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41317 backend=nccl +(Worker pid=2661570) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41317 backend=nccl +(Worker pid=2661569) INFO 07-12 15:17:06 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2661569) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661569) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661569) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2661569) INFO 07-12 15:17:10 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2661569) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2661569) INFO 07-12 15:17:11 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.25 GiB. +(Worker_TP0 pid=2661569) INFO 07-12 15:17:11 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2661569) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2661256) WARNING 07-12 15:16:55 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2661256) INFO 07-12 15:16:55 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2661580) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:42171 backend=nccl +(Worker pid=2661581) INFO 07-12 15:17:05 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:42171 backend=nccl +(Worker pid=2661580) INFO 07-12 15:17:06 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2661580) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661580) INFO 07-12 15:17:09 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2661580) INFO 07-12 15:17:09 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2661580) INFO 07-12 15:17:10 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2661580) INFO 07-12 15:17:10 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2661580) INFO 07-12 15:17:10 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2661580) INFO 07-12 15:17:10 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2661580) INFO 07-12 15:17:10 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2661580) INFO 07-12 15:17:10 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1275.26 GiB. +(Worker_TP0 pid=2661580) INFO 07-12 15:17:10 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2661580) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2670831) WARNING 07-12 15:23:59 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2670831) INFO 07-12 15:23:59 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4 +(Worker pid=2671149) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl +(Worker pid=2671150) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl +(Worker pid=2671152) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl +(Worker pid=2671151) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:44927 backend=nccl +(Worker pid=2671149) INFO 07-12 15:24:13 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2671149) INFO 07-12 15:24:16 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2671149) INFO 07-12 15:24:17 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2671149) INFO 07-12 15:24:17 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2671149) INFO 07-12 15:24:17 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2671149) INFO 07-12 15:24:17 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1278.20 GiB. +(Worker_TP0 pid=2671149) INFO 07-12 15:24:18 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2671149) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2670761) WARNING 07-12 15:23:58 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2670761) INFO 07-12 15:23:58 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4 +(Worker pid=2671136) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl +(Worker pid=2671138) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl +(Worker pid=2671139) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl +(Worker pid=2671137) INFO 07-12 15:24:12 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:42089 backend=nccl +(Worker pid=2671136) INFO 07-12 15:24:13 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2671136) INFO 07-12 15:24:16 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2671136) INFO 07-12 15:24:17 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2671136) INFO 07-12 15:24:17 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2671136) INFO 07-12 15:24:17 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2671136) INFO 07-12 15:24:17 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1278.21 GiB. +(Worker_TP0 pid=2671136) INFO 07-12 15:24:18 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2671136) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00 + main() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 434, in main + execute_wave(index, state, expected) + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 410, in execute_wave + raise failure + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 329, in execute_wave + wait_ready(entry) + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 142, in wait_ready + assert_no_other_compute() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 111, in assert_no_other_compute + raise RuntimeError(f"outside GPU processes detected: {other}") +RuntimeError: outside GPU processes detected: [2635013, 2635031, 2635032, 2635034] +Traceback (most recent call last): + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 452, in + main() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 441, in main + execute_wave(index, state, expected) + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 418, in execute_wave + validations = [validate_cell(entry) for entry in entries] + ^^^^^^^^^^^^^^^^^^^^ + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 280, in validate_cell + raise RuntimeError(f"cell validity failure {entry['cell']}: {invariants}") +RuntimeError: cell validity failure tp1_mns8: {'one_ready_marker': True, 'compile_capture_pre_ready': True, 'warmup_exact_16': True, 'warmup_long': True, 'layer1_footer': False, 'layer1_contiguous': True, 'layer1_zero_drops': False, 'sidecar_final': False, 'sidecar_agrees': False, 'anchor_intervals_present': True} +WAVE_ECHO wave=W2-tp2 assignments=tp2_mns8:gpu0+1,tp2_mns16:gpu2+3,tp2_mns32:gpu4+5,tp2_mns64:gpu6+7 spent_h20h=0.432311 wave_est_h20h=0.650 remaining_projection_h20h=2.250 cap_h20h=3.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json workload=chat_w20260311_1000.jsonl +WAVE_ECHO wave=W3-tp4-trap assignments=tp4_mns8:gpu0+1+2+3,tp4_mns16:gpu4+5+6+7 spent_h20h=1.352676 wave_est_h20h=0.850 remaining_projection_h20h=1.600 cap_h20h=3.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json workload=chat_w20260311_1000.jsonl +Traceback (most recent call last): + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 492, in + main() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 481, in main + execute_wave(index, state, expected) + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 334, in execute_wave + raise RuntimeError(f"projected budget exceeds cap before {wave_name}: {state['gpu_hours_total']+future}") +RuntimeError: projected budget exceeds cap before W4-tp4: 3.0411728494699863 diff --git a/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/commands.log b/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/commands.log new file mode 100644 index 0000000..d0434a4 --- /dev/null +++ b/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/commands.log @@ -0,0 +1 @@ +SERVER taskset -c 20-39 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8501 --served-model-name qwen3-30b-a3b-community --max-num-batched-tokens 8192 --max-num-seqs 16 --tensor-parallel-size 1 --shutdown-timeout 120 diff --git a/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/opprof/opprof-v1-dp0-pid2635032-1783867760160342792.jsonl.footer.json b/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/opprof/opprof-v1-dp0-pid2635032-1783867760160342792.jsonl.footer.json new file mode 100644 index 0000000..c13f998 --- /dev/null +++ b/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/opprof/opprof-v1-dp0-pid2635032-1783867760160342792.jsonl.footer.json @@ -0,0 +1 @@ +{"schema":1,"record_type":"footer_checkpoint","stream":"opprof-v1-dp0-pid2635032-1783867760160342792.jsonl","encoded_records":0,"written_records":0,"dropped_records":0,"last_step_index":null,"checkpoint_wall_ns":1783867763197526318,"flush_interval_seconds":1.0,"final":false} diff --git a/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/server.log b/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/server.log new file mode 100644 index 0000000..aded6fb --- /dev/null +++ b/runs/opprof-phase6/phase6/failed-attempts/W1-monitor-marker-false-positive/tp1_mns16/server.log @@ -0,0 +1,126 @@ +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:339] +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:339] █ █ █▄ ▄█ +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2 +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:339] +(APIServer pid=2634246) INFO 07-12 14:47:36 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8501, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120} +(APIServer pid=2634246) INFO 07-12 14:47:48 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM +(APIServer pid=2634246) INFO 07-12 14:47:48 [model.py:1725] Using max model len 40960 +(APIServer pid=2634246) INFO 07-12 14:47:48 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192. +(APIServer pid=2634246) INFO 07-12 14:47:48 [vllm.py:1006] Asynchronous scheduling is enabled. +(APIServer pid=2634246) INFO 07-12 14:47:48 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) +(EngineCore pid=2635032) INFO 07-12 14:47:59 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2635032) INFO 07-12 14:48:01 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:56823 backend=nccl +(EngineCore pid=2635032) INFO 07-12 14:48:01 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2635032) INFO 07-12 14:48:03 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2635032) INFO 07-12 14:48:03 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2635032) INFO 07-12 14:48:04 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2635032) INFO 07-12 14:48:04 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2635032) INFO 07-12 14:48:04 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2635032) INFO 07-12 14:48:04 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.19 GiB. +(EngineCore pid=2635032) INFO 07-12 14:48:04 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2635032) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2635031) INFO 07-12 14:48:02 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:47899 backend=nccl +(EngineCore pid=2635031) INFO 07-12 14:48:02 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2635031) INFO 07-12 14:48:03 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2635031) INFO 07-12 14:48:03 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2635031) INFO 07-12 14:48:04 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2635031) INFO 07-12 14:48:04 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2635031) INFO 07-12 14:48:04 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2635031) INFO 07-12 14:48:04 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.19 GiB. +(EngineCore pid=2635031) INFO 07-12 14:48:04 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2635031) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2635034) INFO 07-12 14:48:02 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:40669 backend=nccl +(EngineCore pid=2635034) INFO 07-12 14:48:02 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2635034) INFO 07-12 14:48:03 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2635034) INFO 07-12 14:48:03 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2635034) INFO 07-12 14:48:04 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2635034) INFO 07-12 14:48:04 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2635034) INFO 07-12 14:48:04 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2635034) INFO 07-12 14:48:04 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.19 GiB. +(EngineCore pid=2635034) INFO 07-12 14:48:04 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2635034) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2635013) INFO 07-12 14:48:02 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:57917 backend=nccl +(EngineCore pid=2635013) INFO 07-12 14:48:02 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2635013) INFO 07-12 14:48:03 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2635013) INFO 07-12 14:48:03 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(APIServer pid=2634245) Traceback (most recent call last): +(APIServer pid=2634245) File "/tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm", line 8, in +(APIServer pid=2634245) sys.exit(main()) +(APIServer pid=2634245) ^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/cli/main.py", line 95, in main +(APIServer pid=2634245) args.dispatch_function(args) +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/cli/serve.py", line 148, in cmd +(APIServer pid=2634245) uvloop.run(run_server(args)) +(APIServer pid=2634245) File "/tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run +(APIServer pid=2634245) return __asyncio.run( +(APIServer pid=2634245) ^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run +(APIServer pid=2634245) return runner.run(main) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run +(APIServer pid=2634245) return self._loop.run_until_complete(task) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "uvloop/loop.pyx", line 1512, in uvloop.loop.Loop.run_until_complete +(APIServer pid=2634245) File "uvloop/loop.pyx", line 1505, in uvloop.loop.Loop.run_until_complete +(APIServer pid=2634245) File "uvloop/loop.pyx", line 1379, in uvloop.loop.Loop.run_forever +(APIServer pid=2634245) File "uvloop/loop.pyx", line 557, in uvloop.loop.Loop._run +(APIServer pid=2634245) File "uvloop/loop.pyx", line 476, in uvloop.loop.Loop._on_idle +(APIServer pid=2634245) File "uvloop/cbhandles.pyx", line 83, in uvloop.loop.Handle._run +(APIServer pid=2634245) File "uvloop/cbhandles.pyx", line 61, in uvloop.loop.Handle._run +(APIServer pid=2634245) File "/tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper +(APIServer pid=2634245) return await main +(APIServer pid=2634245) ^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/openai/api_server.py", line 663, in run_server +(APIServer pid=2634245) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs) +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/openai/api_server.py", line 677, in run_server_worker +(APIServer pid=2634245) async with build_async_engine_client( +(APIServer pid=2634245) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=2634245) return await anext(self.gen) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/openai/api_server.py", line 99, in build_async_engine_client +(APIServer pid=2634245) async with build_async_engine_client_from_engine_args( +(APIServer pid=2634245) File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ +(APIServer pid=2634245) return await anext(self.gen) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/openai/api_server.py", line 135, in build_async_engine_client_from_engine_args +(APIServer pid=2634245) async_llm = AsyncLLM.from_vllm_config( +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config +(APIServer pid=2634245) return cls( +(APIServer pid=2634245) ^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/v1/engine/async_llm.py", line 146, in __init__ +(APIServer pid=2634245) self.engine_core = EngineCoreClient.make_async_mp_client( +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=2634245) return func(*args, **kwargs) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/v1/engine/core_client.py", line 132, in make_async_mp_client +(APIServer pid=2634245) return AsyncMPClient(*client_args) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/tracing/otel.py", line 178, in sync_wrapper +(APIServer pid=2634245) return func(*args, **kwargs) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/v1/engine/core_client.py", line 963, in __init__ +(APIServer pid=2634245) super().__init__( +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/v1/engine/core_client.py", line 619, in __init__ +(APIServer pid=2634245) if not sync_input_socket.poll( +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/zmq/sugar/socket.py", line 1062, in poll +(APIServer pid=2634245) evts = dict(p.poll(timeout)) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "/tmp/wjh-opprof-phase2-dash0-20260711/.venv/lib/python3.12/site-packages/zmq/sugar/poll.py", line 106, in poll +(APIServer pid=2634245) return zmq_poll(self.sockets, timeout=timeout) +(APIServer pid=2634245) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +(APIServer pid=2634245) File "zmq/backend/cython/_zmq.py", line 1680, in zmq.backend.cython._zmq.zmq_poll +(APIServer pid=2634245) File "zmq/backend/cython/_zmq.py", line 179, in zmq.backend.cython._zmq._check_rc +(APIServer pid=2634245) File "/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0/vllm/entrypoints/openai/api_server.py", line 658, in _interrupt_init +(APIServer pid=2634245) raise KeyboardInterrupt("terminated") +(APIServer pid=2634245) KeyboardInterrupt: terminated diff --git a/runs/opprof-phase6/phase6/launch-echo.log b/runs/opprof-phase6/phase6/launch-echo.log new file mode 100644 index 0000000..cb8d54b --- /dev/null +++ b/runs/opprof-phase6/phase6/launch-echo.log @@ -0,0 +1,4 @@ +LAUNCH_ECHO utc=2026-07-12T14:47:24Z host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd model=Qwen3-30B-A3B trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl trace_sha=f539f38e cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16_raw_gt4096 horizon=60s drain=derived_le120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h +RESUME_ECHO utc=2026-07-12T14:50:30Z repair=phase6-owned-pgid-monitor prior_spend=0.1329181679_H20h projected_total_with_reserve=2.7829181679_H20h hard_cap=3.0_H20h restart=W1 +WAVE_ECHO wave=W2-tp2 assignments=tp2_mns8:gpu0+1,tp2_mns16:gpu2+3,tp2_mns32:gpu4+5,tp2_mns64:gpu6+7 spent_h20h=0.432311 wave_est_h20h=0.650 remaining_projection_h20h=2.250 cap_h20h=3.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json workload=chat_w20260311_1000.jsonl +WAVE_ECHO wave=W3-tp4-trap assignments=tp4_mns8:gpu0+1+2+3,tp4_mns16:gpu4+5+6+7 spent_h20h=1.352676 wave_est_h20h=0.850 remaining_projection_h20h=1.600 cap_h20h=3.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json workload=chat_w20260311_1000.jsonl diff --git a/runs/opprof-phase6/phase6/metrics.json b/runs/opprof-phase6/phase6/metrics.json new file mode 100644 index 0000000..06f5d3f --- /dev/null +++ b/runs/opprof-phase6/phase6/metrics.json @@ -0,0 +1,9193 @@ +{ + "attempt_history": { + "colocated_budget_stop": { + "before_wave": "W4-tp4", + "excess_h20_hours": 0.04117284946998634, + "hard_cap_h20_hours": 3.0, + "projected_total_h20_hours": 3.0411728494699863, + "remaining_cells": [ + "tp4_mns32", + "tp4_mns64" + ], + "remaining_primary_anchors": 4, + "safety_buffer_h20_hours": 0.1, + "spent_h20_hours": 2.2911728494699863, + "wave_estimate_h20_hours": 0.65 + }, + "colocated_confirmations": 3, + "colocated_h20_hours": 2.2911728494699863, + "colocated_primary_anchors": 21, + "colocated_status": "budget_projection_stop_before_W4", + "raw_roots": { + "colocated": "runs/opprof-phase6/phase6/cells", + "solo_authoritative": "runs/opprof-phase6/phase6/solo-authoritative/cells" + }, + "solo_failures": [ + { + "cell": "tp4_mns64", + "failure": "RuntimeError('GPU cleanup failure: [(0, 4, 0), (1, 4, 0), (2, 4, 0), (3, 4, 0), (4, 4, 0), (5, 4, 0), (6, 4, 0), (7, 1, 0)]')" + } + ], + "solo_repairs": [ + { + "at": 1783873873, + "id": "A-P6-2-transient-driver-memory-poll", + "reason": "1-4 MiB at 0% decayed to 0 MiB with zero compute PIDs; footer validation passed", + "rerun": false + }, + { + "added_h20_hours": 0, + "at": 1783877778, + "id": "A-P6-2-prelaunch-driver-memory-poll", + "reason": "prelaunch 1-4 MiB at 0% with zero compute PIDs; no server allocated" + } + ], + "solo_status": "complete" + }, + "authoritative_tier": "A-P6-2 solo host placement", + "cells": { + "tp1_mns16": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.24609375, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + true + ], + "layer1": { + "decode_B_cv": 0.887214258064248, + "decode_B_mean": 3.9555997349237906, + "decode_only_steps": 4394, + "decode_tokens": 17907, + "graph_mode_counts": { + "FULL": 4394, + "NONE": 109, + "PIECEWISE": 24 + }, + "graph_mode_shares": { + "FULL": 0.9706207201237023, + "NONE": 0.02407775568809366, + "PIECEWISE": 0.005301524188204109 + }, + "kv_usage_max": 0.1876057740169238, + "kv_usage_mean": 0.03770906995392833, + "mixed_steps": 122, + "model_steps": 4527, + "padding_fraction": 0.009525377680813404, + "preemptions": 0, + "prefill_only_steps": 11, + "prefill_tokens": 343017, + "steps": 4548, + "waiting_cv": 6.829878371500364, + "waiting_max": 3.0, + "waiting_mean": 0.041528606140932185 + }, + "pass_rates": [ + 1.0 + ], + "primary": { + "anchor": 0.24609375, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.285616071, + "end_mono_ns": 208628026400377, + "end_wall_ns": 1783877055152827630, + "start_mono_ns": 208566740784306, + "start_wall_ns": 1783876993867211598 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 39.26863328358488, + "min": 4.784946039540869, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1786.641047016019, + "min": 38.60557498410344, + "missing_n": 0, + "n": 141 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": true, + "pass_rate": 0.9574468085106383, + "rate_per_gpu": 2.35, + "request_count": 141 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.35, + "request_count": 141 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.25, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [ + { + "anchor": 0.25, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.468118415, + "end_mono_ns": 208763972089008, + "end_wall_ns": 1783877191098516077, + "start_mono_ns": 208702503970593, + "start_wall_ns": 1783877129630397769 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.8471743605753291, + "decode_B_mean": 4.446865817825661, + "decode_only_steps": 3951, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 3951, + "NONE": 122, + "PIECEWISE": 11 + }, + "graph_mode_shares": { + "FULL": 0.9674338883447601, + "NONE": 0.029872673849167482, + "PIECEWISE": 0.0026934378060724778 + }, + "kv_usage_max": 0.21259333001493286, + "kv_usage_mean": 0.04239238802470177, + "mixed_steps": 124, + "model_steps": 4084, + "padding_fraction": 0.008283430064048115, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4102, + "waiting_cv": 6.540526719992036, + "waiting_max": 2.0, + "waiting_mean": 0.03893241919686582 + }, + "mns": 16, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 44.40597959039696, + "min": 4.490311023564716, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1776.1785559996497, + "min": 49.54166599782184, + "missing_n": 0, + "n": 143 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.8476702434656673, + "decode_B_mean": 4.452316744300074, + "decode_only_steps": 3946, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 3946, + "NONE": 122, + "PIECEWISE": 11 + }, + "graph_mode_shares": { + "FULL": 0.967393969110076, + "NONE": 0.029909291493012993, + "PIECEWISE": 0.0026967393969110076 + }, + "kv_usage_max": 0.21259333001493286, + "kv_usage_mean": 0.0424465609586476, + "mixed_steps": 124, + "model_steps": 4079, + "padding_fraction": 0.008325663817313311, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4097, + "waiting_cv": 6.446796801454737, + "waiting_max": 2.0, + "waiting_mean": 0.04094140720764893 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.25, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.484334594, + "end_mono_ns": 208696038541120, + "end_wall_ns": 1783877123164968015, + "start_mono_ns": 208634554206526, + "start_wall_ns": 1783877061680633802 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 44.45809762976243, + "min": 4.488573606239676, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1778.0322160106152, + "min": 47.90777899324894, + "missing_n": 0, + "n": 143 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.8461538461538461, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.15579710144927536, + "anchor": 0.5, + "colocated": null, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + false + ], + "layer1": { + "decode_B_cv": 0.2627716896409309, + "decode_B_mean": 14.094089264173704, + "decode_only_steps": 748, + "decode_tokens": 11684, + "graph_mode_counts": { + "FULL": 748, + "NONE": 79, + "PIECEWISE": 2 + }, + "graph_mode_shares": { + "FULL": 0.902291917973462, + "NONE": 0.09529553679131483, + "PIECEWISE": 0.0024125452352231603 + }, + "kv_usage_max": 0.17585863613738173, + "kv_usage_mean": 0.12595119309308353, + "mixed_steps": 80, + "model_steps": 829, + "padding_fraction": 0.0025652464867276376, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 247663, + "steps": 831, + "waiting_cv": 0.8636814052672781, + "waiting_max": 24.0, + "waiting_mean": 9.182147165259348 + }, + "pass_rates": [ + 0.15579710144927536 + ], + "primary": { + "anchor": 0.5, + "cell": "tp1_mns16", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 92, + "feasible": false, + "interval": { + "elapsed_s": 26.818138737, + "end_mono_ns": 208796704305524, + "end_wall_ns": 1783877223830732816, + "start_mono_ns": 208769886166787, + "start_wall_ns": 1783877197012593929 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 276, + "pass_rate": 0.15579710144927536, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 4.6, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 43, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 92, + "finite_n": 92, + "max": 40.052486527445325, + "min": 11.32848871666498, + "missing_n": 184, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 92, + "finite_n": 92, + "max": 6538.087566004833, + "min": 66.57706701662391, + "missing_n": 184, + "n": 276 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": false, + "pass_rate": 0.09057971014492754, + "rate_per_gpu": 4.6, + "request_count": 276 + }, + "v24": { + "feasible": false, + "pass_rate": 0.15579710144927536, + "rate_per_gpu": 4.6, + "request_count": 276 + } + } + ], + "boundary": "UP", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp1_mns16", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 14174, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2781473-1783876972093434233.jsonl" + }, + "censor": null, + "drift": 0.014184397163120588, + "f20": 2.35, + "f24": 2.3833333333333333, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 16, + "observed_max_feasible_rate": 2.3833333333333333, + "tp": 1 + }, + "tp1_mns32": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.2421875, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + true + ], + "layer1": { + "decode_B_cv": 0.8594012026472565, + "decode_B_mean": 3.6052631578947367, + "decode_only_steps": 4697, + "decode_tokens": 17399, + "graph_mode_counts": { + "FULL": 4697, + "NONE": 99, + "PIECEWISE": 30 + }, + "graph_mode_shares": { + "FULL": 0.9732697886448405, + "NONE": 0.020513883133029424, + "PIECEWISE": 0.006216328222130129 + }, + "kv_usage_max": 0.16263079222720478, + "kv_usage_mean": 0.033693593351574506, + "mixed_steps": 116, + "model_steps": 4826, + "padding_fraction": 0.009021021758645999, + "preemptions": 0, + "prefill_only_steps": 13, + "prefill_tokens": 321495, + "steps": 4849, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0 + ], + "primary": { + "anchor": 0.2421875, + "cell": "tp1_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 137, + "feasible": true, + "interval": { + "elapsed_s": 61.276838561, + "end_mono_ns": 208934443492979, + "end_wall_ns": 1783877361569920155, + "start_mono_ns": 208873166654418, + "start_wall_ns": 1783877300293081595 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 137, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "e8e91fd47d9152811ed7bd79e20c0f45ba6677560a419542d9c17abd82bebb4b", + "arrival_s": { + "distinct_n": 137, + "finite_n": 137, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 137 + }, + "count": 137, + "long_gt4096": 47, + "offered_req_s": 2.283333333333333, + "offered_req_s_per_gpu": 2.283333333333333, + "raw_input_tokens": { + "distinct_n": 129, + "finite_n": 137, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 137 + }, + "raw_length_order_sha256": "93176915562ff9a118f3550157ddd1025d73a6b70cf4d03be9765de9a1b5d744", + "request_id_order_sha256": "adb62bea7f7a12c1e33fa1572ec1d0e274100013ad90e14c6ef5c549b0e0d017" + }, + "slo_pass_count": 137, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 137, + "finite_n": 137, + "max": 35.652001740137756, + "min": 4.437448913378157, + "missing_n": 0, + "n": 137 + }, + "ttft_ms": { + "distinct_n": 137, + "finite_n": 137, + "max": 1483.3475630148314, + "min": 36.238638014765456, + "missing_n": 0, + "n": 137 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": true, + "pass_rate": 0.9635036496350365, + "rate_per_gpu": 2.283333333333333, + "request_count": 137 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.283333333333333, + "request_count": 137 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.24609375, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [ + { + "anchor": 0.24609375, + "cell": "tp1_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.273475712, + "end_mono_ns": 209070710390123, + "end_wall_ns": 1783877497836817221, + "start_mono_ns": 209009436914411, + "start_wall_ns": 1783877436563341828 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.8559440602375088, + "decode_B_mean": 4.302498798654493, + "decode_only_steps": 4031, + "decode_tokens": 17907, + "graph_mode_counts": { + "FULL": 4031, + "NONE": 117, + "PIECEWISE": 14 + }, + "graph_mode_shares": { + "FULL": 0.9685247477174436, + "NONE": 0.028111484863046612, + "PIECEWISE": 0.003363767419509851 + }, + "kv_usage_max": 0.19890383657199806, + "kv_usage_mean": 0.0410361188986074, + "mixed_steps": 122, + "model_steps": 4162, + "padding_fraction": 0.009242801279772485, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 394569, + "steps": 4179, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 32, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 43.52338799214959, + "min": 4.487348055168179, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1779.3545550084673, + "min": 48.642217007, + "missing_n": 0, + "n": 141 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.8576207728623054, + "decode_B_mean": 4.314939759036145, + "decode_only_steps": 4019, + "decode_tokens": 17907, + "graph_mode_counts": { + "FULL": 4019, + "NONE": 117, + "PIECEWISE": 14 + }, + "graph_mode_shares": { + "FULL": 0.968433734939759, + "NONE": 0.028192771084337348, + "PIECEWISE": 0.0033734939759036144 + }, + "kv_usage_max": 0.22017937219730943, + "kv_usage_mean": 0.04115210017949225, + "mixed_steps": 121, + "model_steps": 4150, + "padding_fraction": 0.009261839109172488, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 394569, + "steps": 4168, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.24609375, + "cell": "tp1_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.286066924, + "end_mono_ns": 209002647122468, + "end_wall_ns": 1783877429773549925, + "start_mono_ns": 208941361055544, + "start_wall_ns": 1783877368487482719 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 45.35468105514585, + "min": 4.485542401506412, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1783.296645997325, + "min": 46.25170599319972, + "missing_n": 0, + "n": 141 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.723404255319149, + "rate_per_gpu": 2.35, + "request_count": 141 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.35, + "request_count": 141 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.25, + "colocated": null, + "confirmations": [ + { + "anchor": 0.25, + "cell": "tp1_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.498563674, + "end_mono_ns": 209206240674999, + "end_wall_ns": 1783877633367101961, + "start_mono_ns": 209144742111325, + "start_wall_ns": 1783877571868538807 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.8820507730831504, + "decode_B_mean": 4.484197530864198, + "decode_only_steps": 3918, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 3918, + "NONE": 118, + "PIECEWISE": 14 + }, + "graph_mode_shares": { + "FULL": 0.9674074074074074, + "NONE": 0.029135802469135802, + "PIECEWISE": 0.0034567901234567903 + }, + "kv_usage_max": 0.22017937219730943, + "kv_usage_mean": 0.04277839906007984, + "mixed_steps": 123, + "model_steps": 4050, + "padding_fraction": 0.00986005990683259, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4068, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 32, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 45.34078648040717, + "min": 4.47612570064311, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1783.0426309956238, + "min": 52.536423987476155, + "missing_n": 0, + "n": 143 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.8847865907155498, + "decode_B_mean": 4.5053336641032, + "decode_only_steps": 3900, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 3900, + "NONE": 118, + "PIECEWISE": 13 + }, + "graph_mode_shares": { + "FULL": 0.9675018605805011, + "NONE": 0.02927313321756388, + "PIECEWISE": 0.0032250062019350038 + }, + "kv_usage_max": 0.22919780767314402, + "kv_usage_mean": 0.04297144316400907, + "mixed_steps": 122, + "model_steps": 4031, + "padding_fraction": 0.009878771271720542, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4049, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.25, + "cell": "tp1_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.483328476, + "end_mono_ns": 209138933842058, + "end_wall_ns": 1783877566060269438, + "start_mono_ns": 209077450513582, + "start_wall_ns": 1783877504576941248 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 47.06143370878105, + "min": 4.492237472331049, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1786.8237869988661, + "min": 48.965324996970594, + "missing_n": 0, + "n": 143 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.7132867132867133, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.09057971014492754, + "anchor": 0.5, + "colocated": null, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + false + ], + "layer1": { + "decode_B_cv": 0.5164333981562619, + "decode_B_mean": 19.341954022988507, + "decode_only_steps": 301, + "decode_tokens": 6731, + "graph_mode_counts": { + "FULL": 301, + "NONE": 40, + "PIECEWISE": 7 + }, + "graph_mode_shares": { + "FULL": 0.8649425287356322, + "NONE": 0.11494252873563218, + "PIECEWISE": 0.020114942528735632 + }, + "kv_usage_max": 0.30597907324364726, + "kv_usage_mean": 0.1712594997966886, + "mixed_steps": 46, + "model_steps": 348, + "padding_fraction": 0.006935025889427092, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 141906, + "steps": 350, + "waiting_cv": 2.356711308876946, + "waiting_max": 8.0, + "waiting_mean": 0.8304597701149425 + }, + "pass_rates": [ + 0.09057971014492754 + ], + "primary": { + "anchor": 0.5, + "cell": "tp1_mns32", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 53, + "feasible": false, + "interval": { + "elapsed_s": 14.566282774, + "end_mono_ns": 209227196674186, + "end_wall_ns": 1783877654323101400, + "start_mono_ns": 209212630391412, + "start_wall_ns": 1783877639756818908 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 276, + "pass_rate": 0.09057971014492754, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 4.6, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 25, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 53, + "finite_n": 53, + "max": 64.01803995268126, + "min": 15.782681448700448, + "missing_n": 223, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 53, + "finite_n": 53, + "max": 1837.5349910056684, + "min": 66.4229110116139, + "missing_n": 223, + "n": 276 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": false, + "pass_rate": 0.014492753623188406, + "rate_per_gpu": 4.6, + "request_count": 276 + }, + "v24": { + "feasible": false, + "pass_rate": 0.09057971014492754, + "rate_per_gpu": 4.6, + "request_count": 276 + } + } + ], + "boundary": "UP", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp1_mns32", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 22265, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2786356-1783877278912197439.jsonl" + }, + "censor": null, + "drift": 0.04379562043795615, + "f20": 2.283333333333333, + "f24": 2.3833333333333333, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 32, + "observed_max_feasible_rate": 2.3833333333333333, + "tp": 1 + }, + "tp1_mns64": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.2421875, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + true + ], + "layer1": { + "decode_B_cv": 0.8404309797113223, + "decode_B_mean": 3.7546396201985326, + "decode_only_steps": 4506, + "decode_tokens": 17399, + "graph_mode_counts": { + "FULL": 4506, + "NONE": 94, + "PIECEWISE": 34 + }, + "graph_mode_shares": { + "FULL": 0.9723780750971083, + "NONE": 0.02028485110056107, + "PIECEWISE": 0.0073370738023306 + }, + "kv_usage_max": 0.16300329703267058, + "kv_usage_mean": 0.03513139426134951, + "mixed_steps": 117, + "model_steps": 4634, + "padding_fraction": 0.009548136695882932, + "preemptions": 0, + "prefill_only_steps": 11, + "prefill_tokens": 321495, + "steps": 4655, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0 + ], + "primary": { + "anchor": 0.2421875, + "cell": "tp1_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 137, + "feasible": true, + "interval": { + "elapsed_s": 61.278139352, + "end_mono_ns": 209486742535416, + "end_wall_ns": 1783877913868962814, + "start_mono_ns": 209425464396064, + "start_wall_ns": 1783877852590823366 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 137, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "e8e91fd47d9152811ed7bd79e20c0f45ba6677560a419542d9c17abd82bebb4b", + "arrival_s": { + "distinct_n": 137, + "finite_n": 137, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 137 + }, + "count": 137, + "long_gt4096": 47, + "offered_req_s": 2.283333333333333, + "offered_req_s_per_gpu": 2.283333333333333, + "raw_input_tokens": { + "distinct_n": 129, + "finite_n": 137, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 137 + }, + "raw_length_order_sha256": "93176915562ff9a118f3550157ddd1025d73a6b70cf4d03be9765de9a1b5d744", + "request_id_order_sha256": "adb62bea7f7a12c1e33fa1572ec1d0e274100013ad90e14c6ef5c549b0e0d017" + }, + "slo_pass_count": 137, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 137, + "finite_n": 137, + "max": 34.97855922053238, + "min": 4.792881425099785, + "missing_n": 0, + "n": 137 + }, + "ttft_ms": { + "distinct_n": 137, + "finite_n": 137, + "max": 1473.6714000173379, + "min": 39.49895699042827, + "missing_n": 0, + "n": 137 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": true, + "pass_rate": 0.9635036496350365, + "rate_per_gpu": 2.283333333333333, + "request_count": 137 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.283333333333333, + "request_count": 137 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.24609375, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [ + { + "anchor": 0.24609375, + "cell": "tp1_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.282251823, + "end_mono_ns": 209622963827674, + "end_wall_ns": 1783878050090255139, + "start_mono_ns": 209561681575851, + "start_wall_ns": 1783877988808003321 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.8583238385674631, + "decode_B_mean": 4.310784785748676, + "decode_only_steps": 4023, + "decode_tokens": 17907, + "graph_mode_counts": { + "FULL": 4023, + "NONE": 109, + "PIECEWISE": 22 + }, + "graph_mode_shares": { + "FULL": 0.9684641309581127, + "NONE": 0.026239768897448244, + "PIECEWISE": 0.005296100144439095 + }, + "kv_usage_max": 0.22075132380857232, + "kv_usage_mean": 0.04121597495330755, + "mixed_steps": 122, + "model_steps": 4154, + "padding_fraction": 0.009435525510020294, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 394569, + "steps": 4171, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 64, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 48.01008399226703, + "min": 4.492254126138662, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1784.66858502361, + "min": 49.72933098906651, + "missing_n": 0, + "n": 141 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.8575530551138815, + "decode_B_mean": 4.331640058055152, + "decode_only_steps": 4004, + "decode_tokens": 17907, + "graph_mode_counts": { + "FULL": 4004, + "NONE": 109, + "PIECEWISE": 21 + }, + "graph_mode_shares": { + "FULL": 0.9685534591194969, + "NONE": 0.026366715045960328, + "PIECEWISE": 0.0050798258345428155 + }, + "kv_usage_max": 0.22075132380857232, + "kv_usage_mean": 0.041403411820508135, + "mixed_steps": 121, + "model_steps": 4134, + "padding_fraction": 0.009433146655651722, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 394569, + "steps": 4151, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.24609375, + "cell": "tp1_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.282007271, + "end_mono_ns": 209554942853886, + "end_wall_ns": 1783877982069281476, + "start_mono_ns": 209493660846615, + "start_wall_ns": 1783877920787273687 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 48.099267582718916, + "min": 4.4906551889638875, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1780.4366719792597, + "min": 50.33687801915221, + "missing_n": 0, + "n": 141 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.723404255319149, + "rate_per_gpu": 2.35, + "request_count": 141 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.35, + "request_count": 141 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.25, + "colocated": null, + "confirmations": [ + { + "anchor": 0.25, + "cell": "tp1_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.478525423, + "end_mono_ns": 209758399063288, + "end_wall_ns": 1783878185525489953, + "start_mono_ns": 209696920537865, + "start_wall_ns": 1783878124046965275 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.8846988959321377, + "decode_B_mean": 4.478668310727497, + "decode_only_steps": 3923, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 3923, + "NONE": 110, + "PIECEWISE": 22 + }, + "graph_mode_shares": { + "FULL": 0.9674475955610358, + "NONE": 0.027127003699136867, + "PIECEWISE": 0.005425400739827373 + }, + "kv_usage_max": 0.22075132380857232, + "kv_usage_mean": 0.04283284378214971, + "mixed_steps": 123, + "model_steps": 4055, + "padding_fraction": 0.009930223878834262, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4073, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 64, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 48.10000394492073, + "min": 4.492832637643148, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1776.7020479950588, + "min": 48.44904300989583, + "missing_n": 0, + "n": 143 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.8845805558797767, + "decode_B_mean": 4.50421626984127, + "decode_only_steps": 3901, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 3901, + "NONE": 110, + "PIECEWISE": 21 + }, + "graph_mode_shares": { + "FULL": 0.9675099206349206, + "NONE": 0.027281746031746032, + "PIECEWISE": 0.005208333333333333 + }, + "kv_usage_max": 0.22974323109201722, + "kv_usage_mean": 0.04306607684036748, + "mixed_steps": 122, + "model_steps": 4032, + "padding_fraction": 0.010049479812472692, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4050, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.25, + "cell": "tp1_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.473047427, + "end_mono_ns": 209690427848013, + "end_wall_ns": 1783878117554274971, + "start_mono_ns": 209628954800586, + "start_wall_ns": 1783878056081227964 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 48.026050653631295, + "min": 4.490948598428563, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1785.189588990761, + "min": 47.32625198084861, + "missing_n": 0, + "n": 143 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.7132867132867133, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.06159420289855073, + "anchor": 0.5, + "colocated": null, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + false + ], + "layer1": { + "decode_B_cv": 0.6547417209277581, + "decode_B_mean": 29.794721407624632, + "decode_only_steps": 279, + "decode_tokens": 10160, + "graph_mode_counts": { + "FULL": 279, + "NONE": 53, + "PIECEWISE": 9 + }, + "graph_mode_shares": { + "FULL": 0.8181818181818182, + "NONE": 0.15542521994134897, + "PIECEWISE": 0.026392961876832845 + }, + "kv_usage_max": 0.5738335498051754, + "kv_usage_mean": 0.2696644867126917, + "mixed_steps": 61, + "model_steps": 341, + "padding_fraction": 0.004259919991350416, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 220080, + "steps": 343, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 0.06159420289855073 + ], + "primary": { + "anchor": 0.5, + "cell": "tp1_mns64", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 80, + "feasible": false, + "interval": { + "elapsed_s": 20.200314331, + "end_mono_ns": 209785367162483, + "end_wall_ns": 1783878212493589584, + "start_mono_ns": 209765166848152, + "start_wall_ns": 1783878192293275756 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 276, + "pass_rate": 0.06159420289855073, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 4.6, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 17, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 80, + "finite_n": 80, + "max": 105.92603693692395, + "min": 21.502651858343338, + "missing_n": 196, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 80, + "finite_n": 80, + "max": 1093.1891240179539, + "min": 69.8842880083248, + "missing_n": 196, + "n": 276 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": false, + "pass_rate": 0.057971014492753624, + "rate_per_gpu": 4.6, + "request_count": 276 + }, + "v24": { + "feasible": false, + "pass_rate": 0.06159420289855073, + "rate_per_gpu": 4.6, + "request_count": 276 + } + } + ], + "boundary": "UP", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp1_mns64", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 22050, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2795014-1783877830485564578.jsonl" + }, + "censor": null, + "drift": 0.04379562043795615, + "f20": 2.283333333333333, + "f24": 2.3833333333333333, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 64, + "observed_max_feasible_rate": 2.3833333333333333, + "tp": 1 + }, + "tp1_mns8": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 0.9917355371900827, + "anchor": 0.21875, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 0.9917355371900827, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 0.9917355371900827 + ] + }, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + true + ], + "layer1": { + "decode_B_cv": 0.6893222962038346, + "decode_B_mean": 3.3075764098148945, + "decode_only_steps": 4528, + "decode_tokens": 15367, + "graph_mode_counts": { + "FULL": 4528, + "NONE": 118 + }, + "graph_mode_shares": { + "FULL": 0.9746018080068877, + "NONE": 0.025398191993112353 + }, + "kv_usage_max": 0.1212995671426439, + "kv_usage_mean": 0.031261334740002124, + "mixed_steps": 106, + "model_steps": 4646, + "padding_fraction": 0.005704275062658432, + "preemptions": 0, + "prefill_only_steps": 12, + "prefill_tokens": 332550, + "steps": 4669, + "waiting_cv": 4.382949971464802, + "waiting_max": 6.0, + "waiting_mean": 0.16659492036160137 + }, + "pass_rates": [ + 0.9917355371900827 + ], + "primary": { + "anchor": 0.21875, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 121, + "feasible": true, + "interval": { + "elapsed_s": 61.272714175, + "end_mono_ns": 208049949471616, + "end_wall_ns": 1783876477075898873, + "start_mono_ns": 207988676757441, + "start_wall_ns": 1783876415803184668 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 121, + "pass_rate": 0.9917355371900827, + "schema": 1, + "selection": { + "arrival_order_sha256": "f585e4793a8a6a621dbfe622514743ac316e34e3e93aaa1c6ba5f1eb58e8ff2d", + "arrival_s": { + "distinct_n": 121, + "finite_n": 121, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 121 + }, + "count": 121, + "long_gt4096": 42, + "offered_req_s": 2.0166666666666666, + "offered_req_s_per_gpu": 2.0166666666666666, + "raw_input_tokens": { + "distinct_n": 114, + "finite_n": 121, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 121 + }, + "raw_length_order_sha256": "061592e591871368954f4f80ee75cde2cfc399bd719f3f5deaad275e9af72e8b", + "request_id_order_sha256": "9b8c177ea00e5e42bd3294328201cb57addfd823913a5721714ce69646d715f4" + }, + "slo_pass_count": 120, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 121, + "finite_n": 121, + "max": 30.270937291215958, + "min": 4.492692204786274, + "missing_n": 0, + "n": 121 + }, + "ttft_ms": { + "distinct_n": 121, + "finite_n": 121, + "max": 2006.5461480116937, + "min": 70.57557700318284, + "missing_n": 0, + "n": 121 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.0166666666666666, + "request_count": 121 + }, + "v24": { + "feasible": true, + "pass_rate": 0.9917355371900827, + "rate_per_gpu": 2.0166666666666666, + "request_count": 121 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 0.9920634920634921, + "anchor": 0.2265625, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.20634920634920634, + "solo_minus_colocated_primary_pass_rate": 0.7857142857142858, + "trial_pass_rates": [ + 0.20634920634920634 + ] + }, + "confirmations": [ + { + "anchor": 0.2265625, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 126, + "feasible": true, + "interval": { + "elapsed_s": 61.283628232, + "end_mono_ns": 207982157816433, + "end_wall_ns": 1783876409284243533, + "start_mono_ns": 207920874188201, + "start_wall_ns": 1783876348000615426 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.662970700451707, + "decode_B_mean": 3.5473287519397028, + "decode_only_steps": 4388, + "decode_tokens": 16002, + "graph_mode_counts": { + "FULL": 4388, + "NONE": 123 + }, + "graph_mode_shares": { + "FULL": 0.9727333185546442, + "NONE": 0.0272666814453558 + }, + "kv_usage_max": 0.1212995671426439, + "kv_usage_mean": 0.03382010985325586, + "mixed_steps": 112, + "model_steps": 4511, + "padding_fraction": 0.006645804345354434, + "preemptions": 0, + "prefill_only_steps": 11, + "prefill_tokens": 350949, + "steps": 4532, + "waiting_cv": 4.2697345691988176, + "waiting_max": 6.0, + "waiting_mean": 0.17401906450897806 + }, + "mns": 8, + "observed_count": 126, + "pass_rate": 0.9920634920634921, + "schema": 1, + "selection": { + "arrival_order_sha256": "1b67241583bd6f75b1ad3f6d8c86bf0c815306a28f18ea49b1511fed6c091abe", + "arrival_s": { + "distinct_n": 126, + "finite_n": 126, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 126 + }, + "count": 126, + "long_gt4096": 44, + "offered_req_s": 2.1, + "offered_req_s_per_gpu": 2.1, + "raw_input_tokens": { + "distinct_n": 119, + "finite_n": 126, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 126 + }, + "raw_length_order_sha256": "29e3240c20df0cc68172d39aeae84d79908acf564c1b1e2edd447a984e7bdb0e", + "request_id_order_sha256": "a05848006335fc9f595432bab793a13d427244886e4f04e51ba74d495453ef90" + }, + "slo_pass_count": 125, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 126, + "finite_n": 126, + "max": 30.423986094391456, + "min": 4.49271833865104, + "missing_n": 0, + "n": 126 + }, + "ttft_ms": { + "distinct_n": 126, + "finite_n": 126, + "max": 2015.6794769864064, + "min": 71.10144800390117, + "missing_n": 0, + "n": 126 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.710953356633502, + "decode_B_mean": 3.3845177664974617, + "decode_only_steps": 4605, + "decode_tokens": 16002, + "graph_mode_counts": { + "FULL": 4606, + "NONE": 112, + "PIECEWISE": 10 + }, + "graph_mode_shares": { + "FULL": 0.9741962774957699, + "NONE": 0.023688663282571912, + "PIECEWISE": 0.0021150592216582064 + }, + "kv_usage_max": 0.1212000597044629, + "kv_usage_mean": 0.03228697515824587, + "mixed_steps": 113, + "model_steps": 4728, + "padding_fraction": 0.006803989473072891, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 308933, + "steps": 4746, + "waiting_cv": 4.12058629788992, + "waiting_max": 6.0, + "waiting_mean": 0.17956852791878172 + }, + "pass_rates": [ + 0.9920634920634921, + 0.9920634920634921 + ], + "primary": { + "anchor": 0.2265625, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 126, + "feasible": true, + "interval": { + "elapsed_s": 61.279043859, + "end_mono_ns": 207914196717219, + "end_wall_ns": 1783876341323144094, + "start_mono_ns": 207852917673360, + "start_wall_ns": 1783876280044100637 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 126, + "pass_rate": 0.9920634920634921, + "schema": 1, + "selection": { + "arrival_order_sha256": "1b67241583bd6f75b1ad3f6d8c86bf0c815306a28f18ea49b1511fed6c091abe", + "arrival_s": { + "distinct_n": 126, + "finite_n": 126, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 126 + }, + "count": 126, + "long_gt4096": 44, + "offered_req_s": 2.1, + "offered_req_s_per_gpu": 2.1, + "raw_input_tokens": { + "distinct_n": 119, + "finite_n": 126, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 126 + }, + "raw_length_order_sha256": "29e3240c20df0cc68172d39aeae84d79908acf564c1b1e2edd447a984e7bdb0e", + "request_id_order_sha256": "a05848006335fc9f595432bab793a13d427244886e4f04e51ba74d495453ef90" + }, + "slo_pass_count": 125, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 126, + "finite_n": 126, + "max": 30.22168563792526, + "min": 4.490380700884797, + "missing_n": 0, + "n": 126 + }, + "ttft_ms": { + "distinct_n": 126, + "finite_n": 126, + "max": 2057.7456309983972, + "min": 41.72074800590053, + "missing_n": 0, + "n": 126 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.1, + "request_count": 126 + }, + "v24": { + "feasible": true, + "pass_rate": 0.9920634920634921, + "rate_per_gpu": 2.1, + "request_count": 126 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 0.9923076923076923, + "anchor": 0.23046875, + "colocated": null, + "confirmations": [ + { + "anchor": 0.23046875, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 130, + "feasible": true, + "interval": { + "elapsed_s": 61.281326625, + "end_mono_ns": 208184418966888, + "end_wall_ns": 1783876611545393714, + "start_mono_ns": 208123137640263, + "start_wall_ns": 1783876550264067787 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.6705225594648561, + "decode_B_mean": 3.5697297297297297, + "decode_only_steps": 4500, + "decode_tokens": 16510, + "graph_mode_counts": { + "FULL": 4500, + "NONE": 125 + }, + "graph_mode_shares": { + "FULL": 0.972972972972973, + "NONE": 0.02702702702702703 + }, + "kv_usage_max": 0.1212995671426439, + "kv_usage_mean": 0.033224780579375335, + "mixed_steps": 115, + "model_steps": 4625, + "padding_fraction": 0.006263622434666264, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 351880, + "steps": 4644, + "waiting_cv": 3.926195018056357, + "waiting_max": 6.0, + "waiting_mean": 0.1922162162162162 + }, + "mns": 8, + "observed_count": 130, + "pass_rate": 0.9923076923076923, + "schema": 1, + "selection": { + "arrival_order_sha256": "57bbe1eb7c1bb474cc7e2b0200b6e92bf84e8c5a21849b9aa6589fcb8ad650d5", + "arrival_s": { + "distinct_n": 130, + "finite_n": 130, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 130 + }, + "count": 130, + "long_gt4096": 44, + "offered_req_s": 2.1666666666666665, + "offered_req_s_per_gpu": 2.1666666666666665, + "raw_input_tokens": { + "distinct_n": 122, + "finite_n": 130, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 130 + }, + "raw_length_order_sha256": "30beac66a67ec987e5f99102936df78950092f1d7eaef89738be547a01fd8954", + "request_id_order_sha256": "0f4c973153935c8abcd421eb631900670e31af0dc8a33bdf7e4d3d85e7ed7711" + }, + "slo_pass_count": 129, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 130, + "finite_n": 130, + "max": 30.304548842328284, + "min": 4.494556952744547, + "missing_n": 0, + "n": 130 + }, + "ttft_ms": { + "distinct_n": 130, + "finite_n": 130, + "max": 2015.1741319859866, + "min": 67.51175099634565, + "missing_n": 0, + "n": 130 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.6684768566852552, + "decode_B_mean": 3.5891304347826085, + "decode_only_steps": 4475, + "decode_tokens": 16510, + "graph_mode_counts": { + "FULL": 4475, + "NONE": 125 + }, + "graph_mode_shares": { + "FULL": 0.9728260869565217, + "NONE": 0.02717391304347826 + }, + "kv_usage_max": 0.12124981342355345, + "kv_usage_mean": 0.03340155361395873, + "mixed_steps": 115, + "model_steps": 4600, + "padding_fraction": 0.006226898691487927, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 351832, + "steps": 4619, + "waiting_cv": 3.9040334267501087, + "waiting_max": 6.0, + "waiting_mean": 0.19369565217391305 + }, + "pass_rates": [ + 0.9923076923076923, + 0.9923076923076923 + ], + "primary": { + "anchor": 0.23046875, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 130, + "feasible": true, + "interval": { + "elapsed_s": 61.272080003, + "end_mono_ns": 208117172178971, + "end_wall_ns": 1783876544298605842, + "start_mono_ns": 208055900098968, + "start_wall_ns": 1783876483026526659 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 130, + "pass_rate": 0.9923076923076923, + "schema": 1, + "selection": { + "arrival_order_sha256": "57bbe1eb7c1bb474cc7e2b0200b6e92bf84e8c5a21849b9aa6589fcb8ad650d5", + "arrival_s": { + "distinct_n": 130, + "finite_n": 130, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 130 + }, + "count": 130, + "long_gt4096": 44, + "offered_req_s": 2.1666666666666665, + "offered_req_s_per_gpu": 2.1666666666666665, + "raw_input_tokens": { + "distinct_n": 122, + "finite_n": 130, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 130 + }, + "raw_length_order_sha256": "30beac66a67ec987e5f99102936df78950092f1d7eaef89738be547a01fd8954", + "request_id_order_sha256": "0f4c973153935c8abcd421eb631900670e31af0dc8a33bdf7e4d3d85e7ed7711" + }, + "slo_pass_count": 129, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 130, + "finite_n": 130, + "max": 30.301648842475164, + "min": 4.491634480380196, + "missing_n": 0, + "n": 130 + }, + "ttft_ms": { + "distinct_n": 130, + "finite_n": 130, + "max": 2005.0652289937716, + "min": 68.62633599666879, + "missing_n": 0, + "n": 130 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.8846153846153846, + "rate_per_gpu": 2.1666666666666665, + "request_count": 130 + }, + "v24": { + "feasible": true, + "pass_rate": 0.9923076923076923, + "rate_per_gpu": 2.1666666666666665, + "request_count": 130 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 0.9924242424242424, + "anchor": 0.234375, + "colocated": null, + "confirmations": [ + { + "anchor": 0.234375, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 132, + "feasible": true, + "interval": { + "elapsed_s": 61.282402784, + "end_mono_ns": 208319682497617, + "end_wall_ns": 1783876746808924426, + "start_mono_ns": 208258400094833, + "start_wall_ns": 1783876685526522242 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.6667355379026283, + "decode_B_mean": 3.700662251655629, + "decode_only_steps": 4403, + "decode_tokens": 16764, + "graph_mode_counts": { + "FULL": 4403, + "NONE": 127 + }, + "graph_mode_shares": { + "FULL": 0.9719646799116998, + "NONE": 0.02803532008830022 + }, + "kv_usage_max": 0.12234439524354446, + "kv_usage_mean": 0.034346420099096664, + "mixed_steps": 117, + "model_steps": 4530, + "padding_fraction": 0.006442136522204217, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 356622, + "steps": 4549, + "waiting_cv": 3.428239831071161, + "waiting_max": 7.0, + "waiting_mean": 0.28719646799117 + }, + "mns": 8, + "observed_count": 132, + "pass_rate": 0.9924242424242424, + "schema": 1, + "selection": { + "arrival_order_sha256": "3ade8cc1a55a8a75ea339c66a69fae18f60e5a39097714bc35e4eb728d717aeb", + "arrival_s": { + "distinct_n": 132, + "finite_n": 132, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 132 + }, + "count": 132, + "long_gt4096": 45, + "offered_req_s": 2.2, + "offered_req_s_per_gpu": 2.2, + "raw_input_tokens": { + "distinct_n": 124, + "finite_n": 132, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 132 + }, + "raw_length_order_sha256": "14c37ab0a19d72208c80b7ba7c72fd57614f0763f82e0f04453f0e18cb74bc99", + "request_id_order_sha256": "14d2a74fffe25e7d9b3e7490c392eae0dc31bb1f3fe6566c9c0a567543bdd6ac" + }, + "slo_pass_count": 131, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 132, + "finite_n": 132, + "max": 30.46692667715068, + "min": 4.496486598578113, + "missing_n": 0, + "n": 132 + }, + "ttft_ms": { + "distinct_n": 132, + "finite_n": 132, + "max": 2019.6287339786068, + "min": 68.34458399680443, + "missing_n": 0, + "n": 132 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.665245698535184, + "decode_B_mean": 3.7146022601373807, + "decode_only_steps": 4386, + "decode_tokens": 16764, + "graph_mode_counts": { + "FULL": 4386, + "NONE": 127 + }, + "graph_mode_shares": { + "FULL": 0.971859073786838, + "NONE": 0.028140926213161978 + }, + "kv_usage_max": 0.12234439524354446, + "kv_usage_mean": 0.03447585460170693, + "mixed_steps": 117, + "model_steps": 4513, + "padding_fraction": 0.006420985686497303, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 356622, + "steps": 4532, + "waiting_cv": 3.411179152233226, + "waiting_max": 7.0, + "waiting_mean": 0.28960779968978506 + }, + "pass_rates": [ + 0.9924242424242424, + 0.9924242424242424 + ], + "primary": { + "anchor": 0.234375, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 132, + "feasible": true, + "interval": { + "elapsed_s": 61.276628632, + "end_mono_ns": 208252399388309, + "end_wall_ns": 1783876679525815422, + "start_mono_ns": 208191122759677, + "start_wall_ns": 1783876618249186880 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 132, + "pass_rate": 0.9924242424242424, + "schema": 1, + "selection": { + "arrival_order_sha256": "3ade8cc1a55a8a75ea339c66a69fae18f60e5a39097714bc35e4eb728d717aeb", + "arrival_s": { + "distinct_n": 132, + "finite_n": 132, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 132 + }, + "count": 132, + "long_gt4096": 45, + "offered_req_s": 2.2, + "offered_req_s_per_gpu": 2.2, + "raw_input_tokens": { + "distinct_n": 124, + "finite_n": 132, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 132 + }, + "raw_length_order_sha256": "14c37ab0a19d72208c80b7ba7c72fd57614f0763f82e0f04453f0e18cb74bc99", + "request_id_order_sha256": "14d2a74fffe25e7d9b3e7490c392eae0dc31bb1f3fe6566c9c0a567543bdd6ac" + }, + "slo_pass_count": 131, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 132, + "finite_n": 132, + "max": 30.4299091888632, + "min": 4.494227968545, + "missing_n": 0, + "n": 132 + }, + "ttft_ms": { + "distinct_n": 132, + "finite_n": 132, + "max": 2019.111283996608, + "min": 68.64958599908277, + "missing_n": 0, + "n": 132 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.7121212121212122, + "rate_per_gpu": 2.2, + "request_count": 132 + }, + "v24": { + "feasible": true, + "pass_rate": 0.9924242424242424, + "rate_per_gpu": 2.2, + "request_count": 132 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 0.9545454545454546, + "anchor": 0.25, + "colocated": null, + "confirmations": [ + { + "anchor": 0.25, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 62.091254073, + "end_mono_ns": 208457089228587, + "end_wall_ns": 1783876884215656001, + "start_mono_ns": 208394997974514, + "start_wall_ns": 1783876822124401690 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.6763471639371686, + "decode_B_mean": 4.24520804114072, + "decode_only_steps": 4139, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 4139, + "NONE": 139 + }, + "graph_mode_shares": { + "FULL": 0.9675081813931744, + "NONE": 0.032491818606825616 + }, + "kv_usage_max": 0.14015622667794414, + "kv_usage_mean": 0.04057879834231494, + "mixed_steps": 130, + "model_steps": 4278, + "padding_fraction": 0.003411914357383169, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4295, + "waiting_cv": 2.1493134856714793, + "waiting_max": 8.0, + "waiting_mean": 0.7344553529686769 + }, + "mns": 8, + "observed_count": 143, + "pass_rate": 0.951048951048951, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 136, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 32.83134977150833, + "min": 4.493729432991626, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 2528.6180730035994, + "min": 70.94443199457601, + "missing_n": 0, + "n": 143 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.6750552222545031, + "decode_B_mean": 4.254157882408058, + "decode_only_steps": 4130, + "decode_tokens": 18161, + "graph_mode_counts": { + "FULL": 4130, + "NONE": 139 + }, + "graph_mode_shares": { + "FULL": 0.9674396814242211, + "NONE": 0.032560318575778874 + }, + "kv_usage_max": 0.1402059803970347, + "kv_usage_mean": 0.0406622962412423, + "mixed_steps": 130, + "model_steps": 4269, + "padding_fraction": 0.00343323965629562, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 400989, + "steps": 4286, + "waiting_cv": 2.1402634177657514, + "waiting_max": 8.0, + "waiting_mean": 0.7364722417427969 + }, + "pass_rates": [ + 0.958041958041958, + 0.951048951048951 + ], + "primary": { + "anchor": 0.25, + "cell": "tp1_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 62.126477495, + "end_mono_ns": 208388459138035, + "end_wall_ns": 1783876815585565186, + "start_mono_ns": 208326332660540, + "start_wall_ns": 1783876753459087651 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 143, + "pass_rate": 0.958041958041958, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 137, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 32.858955346627205, + "min": 4.492996810975771, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 2534.6092930121813, + "min": 69.11832699552178, + "missing_n": 0, + "n": 143 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.6993006993006993, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + }, + "v24": { + "feasible": true, + "pass_rate": 0.9545454545454546, + "rate_per_gpu": 2.3833333333333333, + "request_count": 143 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.09057971014492754, + "anchor": 0.5, + "colocated": null, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + false + ], + "layer1": { + "decode_B_cv": 0.2914556273099967, + "decode_B_mean": 7.192397207137316, + "decode_only_steps": 1224, + "decode_tokens": 9271, + "graph_mode_counts": { + "FULL": 1224, + "NONE": 65 + }, + "graph_mode_shares": { + "FULL": 0.9495733126454616, + "NONE": 0.0504266873545384 + }, + "kv_usage_max": 0.11015473406637144, + "kv_usage_mean": 0.06439327809885674, + "mixed_steps": 64, + "model_steps": 1289, + "padding_fraction": 0.00017727091427474038, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 193772, + "steps": 1291, + "waiting_cv": 0.7722950850545821, + "waiting_max": 27.0, + "waiting_mean": 11.664856477889836 + }, + "pass_rates": [ + 0.09057971014492754 + ], + "primary": { + "anchor": 0.5, + "cell": "tp1_mns8", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 73, + "feasible": false, + "interval": { + "elapsed_s": 24.795686401, + "end_mono_ns": 208488668302515, + "end_wall_ns": 1783876915794729601, + "start_mono_ns": 208463872616114, + "start_wall_ns": 1783876890999043094 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 276, + "pass_rate": 0.09057971014492754, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 4.6, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 25, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 73, + "finite_n": 73, + "max": 23.976862346469314, + "min": 4.817576614193853, + "missing_n": 203, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 73, + "finite_n": 73, + "max": 9597.04709201469, + "min": 359.1133150039241, + "missing_n": 203, + "n": 276 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": false, + "pass_rate": 0.06521739130434782, + "rate_per_gpu": 4.6, + "request_count": 276 + }, + "v24": { + "feasible": false, + "pass_rate": 0.09057971014492754, + "rate_per_gpu": 4.6, + "request_count": 276 + } + } + ], + "boundary": "UP", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp1_mns8", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 42704, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns8/opprof/opprof-v1-dp0-pid2770271-1783876258461072221.jsonl" + }, + "censor": null, + "drift": 0.13492063492063489, + "f20": 2.1, + "f24": 2.3833333333333333, + "material_frontier_moved": true, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 8, + "observed_max_feasible_rate": 2.3833333333333333, + "tp": 1 + }, + "tp2_mns16": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.4921875, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + true + ], + "layer1": { + "decode_B_cv": 0.4590338982798913, + "decode_B_mean": 3.190716353787242, + "decode_only_steps": 10438, + "decode_tokens": 34163, + "graph_mode_counts": { + "FULL": 10456, + "PIECEWISE": 251 + }, + "graph_mode_shares": { + "FULL": 0.9765573923601383, + "PIECEWISE": 0.023442607639861772 + }, + "kv_usage_max": 0.027596242213826172, + "kv_usage_mean": 0.008071671724621204, + "mixed_steps": 259, + "model_steps": 10707, + "padding_fraction": 0.18738718662952647, + "preemptions": 0, + "prefill_only_steps": 10, + "prefill_tokens": 2303, + "steps": 10726, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0 + ], + "primary": { + "anchor": 0.4921875, + "cell": "tp2_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 269, + "feasible": true, + "interval": { + "elapsed_s": 60.6037635, + "end_mono_ns": 207177385667514, + "end_wall_ns": 1783875604512094524, + "start_mono_ns": 207116781904014, + "start_wall_ns": 1783875543908331340 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 269, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "ffbc7b8dd324c8e745654110a2412b03ddade891beb2bfe022ffde6502b0184b", + "arrival_s": { + "distinct_n": 269, + "finite_n": 269, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 269 + }, + "count": 269, + "long_gt4096": 95, + "offered_req_s": 4.483333333333333, + "offered_req_s_per_gpu": 2.2416666666666667, + "raw_input_tokens": { + "distinct_n": 250, + "finite_n": 269, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 269 + }, + "raw_length_order_sha256": "bc951350376a487d34d2def78fd32a2286e23bf8a7f481ff578009e12e3f1fb8", + "request_id_order_sha256": "cd227aee3be472a35e8c35af60a44a1411c4dc689cbd98746b3fb8a8d329b44c" + }, + "slo_pass_count": 269, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 269, + "finite_n": 269, + "max": 7.384683173259822, + "min": 4.122870834681799, + "missing_n": 0, + "n": 269 + }, + "ttft_ms": { + "distinct_n": 269, + "finite_n": 269, + "max": 81.90969700808637, + "min": 36.26534101204015, + "missing_n": 0, + "n": 269 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.2416666666666667, + "request_count": 269 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.2416666666666667, + "request_count": 269 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.49609375, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.08791208791208792, + "solo_minus_colocated_primary_pass_rate": 0.9120879120879121, + "trial_pass_rates": [ + 0.08791208791208792 + ] + }, + "confirmations": [ + { + "anchor": 0.49609375, + "cell": "tp2_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 273, + "feasible": true, + "interval": { + "elapsed_s": 60.606811621, + "end_mono_ns": 207110687436898, + "end_wall_ns": 1783875537813863970, + "start_mono_ns": 207050080625277, + "start_wall_ns": 1783875477207052621 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.4557177084525861, + "decode_B_mean": 3.22131376010406, + "decode_only_steps": 10490, + "decode_tokens": 34671, + "graph_mode_counts": { + "FULL": 10508, + "PIECEWISE": 255 + }, + "graph_mode_shares": { + "FULL": 0.9763077208956611, + "PIECEWISE": 0.02369227910433894 + }, + "kv_usage_max": 0.027583477994485905, + "kv_usage_mean": 0.008177114927138187, + "mixed_steps": 265, + "model_steps": 10763, + "padding_fraction": 0.18570580083151852, + "preemptions": 0, + "prefill_only_steps": 8, + "prefill_tokens": 2346, + "steps": 10776, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 16, + "observed_count": 273, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "41168f4bd02f3ef4e83b7440a3dab7458f08df9c2bbd4452fad076b4a2e0eaed", + "arrival_s": { + "distinct_n": 273, + "finite_n": 273, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 273 + }, + "count": 273, + "long_gt4096": 97, + "offered_req_s": 4.55, + "offered_req_s_per_gpu": 2.275, + "raw_input_tokens": { + "distinct_n": 253, + "finite_n": 273, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 273 + }, + "raw_length_order_sha256": "9e38ed1766e3933dc05a7ced5d0a73fed47ce769800f0d2c10014af0ef823f28", + "request_id_order_sha256": "a78992ce59bb3b857a7ef8844e0a690ea9dd529a659cf5edd599b63fef8b6a80" + }, + "slo_pass_count": 273, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 273, + "finite_n": 273, + "max": 7.352796283515727, + "min": 4.158690614180316, + "missing_n": 0, + "n": 273 + }, + "ttft_ms": { + "distinct_n": 273, + "finite_n": 273, + "max": 76.74413800123148, + "min": 35.70285899331793, + "missing_n": 0, + "n": 273 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.6022655439791298, + "decode_B_mean": 7.877982276755283, + "decode_only_steps": 4135, + "decode_tokens": 34671, + "graph_mode_counts": { + "FULL": 4135, + "NONE": 241, + "PIECEWISE": 25 + }, + "graph_mode_shares": { + "FULL": 0.9395591910929334, + "NONE": 0.05476028175414679, + "PIECEWISE": 0.005680527152919791 + }, + "kv_usage_max": 0.04904013070560609, + "kv_usage_mean": 0.019502306005154225, + "mixed_steps": 264, + "model_steps": 4401, + "padding_fraction": 0.008382838710511516, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 732330, + "steps": 4404, + "waiting_cv": 3.6068182123674535, + "waiting_max": 4.0, + "waiting_mean": 0.169052488070893 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.49609375, + "cell": "tp2_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 273, + "feasible": true, + "interval": { + "elapsed_s": 61.288445601, + "end_mono_ns": 207043660468133, + "end_wall_ns": 1783875470786895019, + "start_mono_ns": 206982372022532, + "start_wall_ns": 1783875409498449983 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 273, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "41168f4bd02f3ef4e83b7440a3dab7458f08df9c2bbd4452fad076b4a2e0eaed", + "arrival_s": { + "distinct_n": 273, + "finite_n": 273, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 273 + }, + "count": 273, + "long_gt4096": 97, + "offered_req_s": 4.55, + "offered_req_s_per_gpu": 2.275, + "raw_input_tokens": { + "distinct_n": 253, + "finite_n": 273, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 273 + }, + "raw_length_order_sha256": "9e38ed1766e3933dc05a7ced5d0a73fed47ce769800f0d2c10014af0ef823f28", + "request_id_order_sha256": "a78992ce59bb3b857a7ef8844e0a690ea9dd529a659cf5edd599b63fef8b6a80" + }, + "slo_pass_count": 273, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 273, + "finite_n": 273, + "max": 25.645890527525843, + "min": 4.867463031422537, + "missing_n": 0, + "n": 273 + }, + "ttft_ms": { + "distinct_n": 273, + "finite_n": 273, + "max": 785.8217659813818, + "min": 40.98877400974743, + "missing_n": 0, + "n": 273 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.275, + "request_count": 273 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.275, + "request_count": 273 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.5, + "colocated": null, + "confirmations": [ + { + "anchor": 0.5, + "cell": "tp2_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 276, + "feasible": true, + "interval": { + "elapsed_s": 60.597814702, + "end_mono_ns": 207311802163383, + "end_wall_ns": 1783875738928591380, + "start_mono_ns": 207251204348681, + "start_wall_ns": 1783875678330776089 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.43927663118671056, + "decode_B_mean": 3.287563308947665, + "decode_only_steps": 10386, + "decode_tokens": 35052, + "graph_mode_counts": { + "FULL": 10404, + "PIECEWISE": 258 + }, + "graph_mode_shares": { + "FULL": 0.975801913337085, + "PIECEWISE": 0.024198086662915026 + }, + "kv_usage_max": 0.027596242213826172, + "kv_usage_mean": 0.008404164574732987, + "mixed_steps": 267, + "model_steps": 10662, + "padding_fraction": 0.19011036572170525, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 2373, + "steps": 10676, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 16, + "observed_count": 276, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 2.3, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 276, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 7.367673905669, + "min": 4.163846921260051, + "missing_n": 0, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 76.49152100202627, + "min": 36.295153986429796, + "missing_n": 0, + "n": 276 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.43488654756672906, + "decode_B_mean": 3.3278268299629734, + "decode_only_steps": 10257, + "decode_tokens": 35052, + "graph_mode_counts": { + "FULL": 10275, + "NONE": 3, + "PIECEWISE": 255 + }, + "graph_mode_shares": { + "FULL": 0.975505553973227, + "NONE": 0.0002848191398461977, + "PIECEWISE": 0.0242096268869268 + }, + "kv_usage_max": 0.027583477994485905, + "kv_usage_mean": 0.008506688712689907, + "mixed_steps": 270, + "model_steps": 10533, + "padding_fraction": 0.1414077258810501, + "preemptions": 0, + "prefill_only_steps": 6, + "prefill_tokens": 17669, + "steps": 10544, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.5, + "cell": "tp2_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 276, + "feasible": true, + "interval": { + "elapsed_s": 60.59665648, + "end_mono_ns": 207244584143802, + "end_wall_ns": 1783875671710570778, + "start_mono_ns": 207183987487322, + "start_wall_ns": 1783875611113914502 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 276, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 2.3, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 276, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 8.08854837806558, + "min": 4.161210228344352, + "missing_n": 0, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 341.36167902033776, + "min": 35.80997997778468, + "missing_n": 0, + "n": 276 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.9492753623188406, + "rate_per_gpu": 2.3, + "request_count": 276 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3, + "request_count": 276 + } + } + ], + "boundary": "UP", + "bounded": false, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp2_mns16", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 47449, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns16/opprof/opprof-v1-dp0-pid2755628-1783875389707263062.jsonl" + }, + "censor": "RIGHT_CENSORED_HISTORY_EDGE", + "drift": 0.01098901098901095, + "f20": 2.275, + "f24": 2.3, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 16, + "observed_max_feasible_rate": 2.3, + "tp": 2 + }, + "tp2_mns32": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.75, + "colocated": { + "accepted_feasible": null, + "primary_feasible": false, + "primary_pass_rate": 0.4117647058823529, + "solo_minus_colocated_primary_pass_rate": 0.5882352941176471, + "trial_pass_rates": [ + 0.4117647058823529, + 1.0 + ] + }, + "confirmations": [ + { + "anchor": 0.75, + "cell": "tp2_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 391, + "feasible": true, + "interval": { + "elapsed_s": 60.736454527, + "end_mono_ns": 205757115140726, + "end_wall_ns": 1783874184241567823, + "start_mono_ns": 205696378686199, + "start_wall_ns": 1783874123505113386 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.30141395475883415, + "decode_B_mean": 5.579438202247191, + "decode_only_steps": 8509, + "decode_tokens": 49657, + "graph_mode_counts": { + "FULL": 8528, + "PIECEWISE": 372 + }, + "graph_mode_shares": { + "FULL": 0.9582022471910112, + "PIECEWISE": 0.041797752808988765 + }, + "kv_usage_max": 0.033790082623520235, + "kv_usage_mean": 0.014114627776614742, + "mixed_steps": 390, + "model_steps": 8900, + "padding_fraction": 0.21425186589266673, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 3403, + "steps": 8902, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 32, + "observed_count": 391, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "d3ed714128e985d0e0a0ca6b3767210c181114ab6694fb066a60aecf5afa9f9a", + "arrival_s": { + "distinct_n": 391, + "finite_n": 391, + "max": 59.91880000000001, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 391 + }, + "count": 391, + "long_gt4096": 140, + "offered_req_s": 6.516666666666667, + "offered_req_s_per_gpu": 3.2583333333333333, + "raw_input_tokens": { + "distinct_n": 361, + "finite_n": 391, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 391 + }, + "raw_length_order_sha256": "8152bbaf9f27e84e6fbd6359174d17de4b9c96ea8b8c58e3760290be3469b2a9", + "request_id_order_sha256": "4f4c792e90f5c1bf33978fea26debcadfce5f8b872a31be86a88a1bf6b09a21c" + }, + "slo_pass_count": 391, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 391, + "finite_n": 391, + "max": 9.374451102353413, + "min": 5.516279039411314, + "missing_n": 0, + "n": 391 + }, + "ttft_ms": { + "distinct_n": 391, + "finite_n": 391, + "max": 77.84111899673007, + "min": 37.644838012056425, + "missing_n": 0, + "n": 391 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.3012988926514766, + "decode_B_mean": 5.571300347806575, + "decode_only_steps": 8522, + "decode_tokens": 49657, + "graph_mode_counts": { + "FULL": 8541, + "PIECEWISE": 372 + }, + "graph_mode_shares": { + "FULL": 0.9582632110400539, + "PIECEWISE": 0.04173678895994615 + }, + "kv_usage_max": 0.03380285287394491, + "kv_usage_mean": 0.01409497800467283, + "mixed_steps": 390, + "model_steps": 8913, + "padding_fraction": 0.2127713238675984, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 3403, + "steps": 8914, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.75, + "cell": "tp2_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 391, + "feasible": true, + "interval": { + "elapsed_s": 60.741404276, + "end_mono_ns": 205690325731221, + "end_wall_ns": 1783874117452158150, + "start_mono_ns": 205629584326945, + "start_wall_ns": 1783874056710754232 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 391, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "d3ed714128e985d0e0a0ca6b3767210c181114ab6694fb066a60aecf5afa9f9a", + "arrival_s": { + "distinct_n": 391, + "finite_n": 391, + "max": 59.91880000000001, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 391 + }, + "count": 391, + "long_gt4096": 140, + "offered_req_s": 6.516666666666667, + "offered_req_s_per_gpu": 3.2583333333333333, + "raw_input_tokens": { + "distinct_n": 361, + "finite_n": 391, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 391 + }, + "raw_length_order_sha256": "8152bbaf9f27e84e6fbd6359174d17de4b9c96ea8b8c58e3760290be3469b2a9", + "request_id_order_sha256": "4f4c792e90f5c1bf33978fea26debcadfce5f8b872a31be86a88a1bf6b09a21c" + }, + "slo_pass_count": 391, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 391, + "finite_n": 391, + "max": 9.36014444877093, + "min": 5.401530496017113, + "missing_n": 0, + "n": 391 + }, + "ttft_ms": { + "distinct_n": 391, + "finite_n": 391, + "max": 78.69401399511844, + "min": 39.25309798796661, + "missing_n": 0, + "n": 391 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 3.2583333333333333, + "request_count": 391 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 3.2583333333333333, + "request_count": 391 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.5748730964467006, + "anchor": 0.75390625, + "colocated": { + "accepted_feasible": null, + "primary_feasible": false, + "primary_pass_rate": 0.027918781725888325, + "solo_minus_colocated_primary_pass_rate": 0.5469543147208122, + "trial_pass_rates": [ + 0.027918781725888325, + 0.9568527918781726 + ] + }, + "confirmations": [ + { + "anchor": 0.75390625, + "cell": "tp2_mns32", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 394, + "feasible": false, + "interval": { + "elapsed_s": 65.610579626, + "end_mono_ns": 205623164922621, + "end_wall_ns": 1783874050291349598, + "start_mono_ns": 205557554342995, + "start_wall_ns": 1783873984680770337 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.8410188583626338, + "decode_B_mean": 12.345916604983962, + "decode_only_steps": 3677, + "decode_tokens": 50038, + "graph_mode_counts": { + "FULL": 3681, + "NONE": 228, + "PIECEWISE": 144 + }, + "graph_mode_shares": { + "FULL": 0.9082161361954109, + "NONE": 0.056254626202812734, + "PIECEWISE": 0.03552923760177646 + }, + "kv_usage_max": 0.11002847765844692, + "kv_usage_mean": 0.03062022163451763, + "mixed_steps": 375, + "model_steps": 4053, + "padding_fraction": 0.00952631630978508, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 751485, + "steps": 4054, + "waiting_cv": 2.902773549928104, + "waiting_max": 21.0, + "waiting_mean": 1.2871946706143598 + }, + "mns": 32, + "observed_count": 394, + "pass_rate": 0.9441624365482234, + "schema": 1, + "selection": { + "arrival_order_sha256": "60fc43518e79fd46abe59afde67663a762ca17e817202fa52803040ac9056796", + "arrival_s": { + "distinct_n": 394, + "finite_n": 394, + "max": 59.91880000000001, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 394 + }, + "count": 394, + "long_gt4096": 142, + "offered_req_s": 6.566666666666666, + "offered_req_s_per_gpu": 3.283333333333333, + "raw_input_tokens": { + "distinct_n": 364, + "finite_n": 394, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 394 + }, + "raw_length_order_sha256": "4a9cc03e351cce8c558b30732418c878fc3b6d24e64bef6717ce7c87c384384f", + "request_id_order_sha256": "2677a6a12ff0e104d6246dd121d0667ff4fe3d25f8d68699547906420246ebdd" + }, + "slo_pass_count": 372, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 394, + "finite_n": 394, + "max": 49.94228484263042, + "min": 5.9553904016135535, + "missing_n": 0, + "n": 394 + }, + "ttft_ms": { + "distinct_n": 394, + "finite_n": 394, + "max": 4393.049476988381, + "min": 39.858332980656996, + "missing_n": 0, + "n": 394 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + false, + false + ], + "layer1": { + "decode_B_cv": 0.6931371977878618, + "decode_B_mean": 18.279269602577873, + "decode_only_steps": 827, + "decode_tokens": 17018, + "graph_mode_counts": { + "FULL": 827, + "NONE": 78, + "PIECEWISE": 26 + }, + "graph_mode_shares": { + "FULL": 0.8882921589688507, + "NONE": 0.08378088077336197, + "PIECEWISE": 0.027926960257787327 + }, + "kv_usage_max": 0.09019627874902625, + "kv_usage_mean": 0.04515560550142388, + "mixed_steps": 103, + "model_steps": 931, + "padding_fraction": 0.003433293669421307, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 348136, + "steps": 933, + "waiting_cv": 1.380157838358137, + "waiting_max": 22.0, + "waiting_mean": 5.59828141783029 + }, + "pass_rates": [ + 0.20558375634517767, + 0.9441624365482234 + ], + "primary": { + "anchor": 0.75390625, + "cell": "tp2_mns32", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 134, + "feasible": false, + "interval": { + "elapsed_s": 24.702770822, + "end_mono_ns": 205551739158061, + "end_wall_ns": 1783873978865585407, + "start_mono_ns": 205527036387239, + "start_wall_ns": 1783873954162814682 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 394, + "pass_rate": 0.20558375634517767, + "schema": 1, + "selection": { + "arrival_order_sha256": "60fc43518e79fd46abe59afde67663a762ca17e817202fa52803040ac9056796", + "arrival_s": { + "distinct_n": 394, + "finite_n": 394, + "max": 59.91880000000001, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 394 + }, + "count": 394, + "long_gt4096": 142, + "offered_req_s": 6.566666666666666, + "offered_req_s_per_gpu": 3.283333333333333, + "raw_input_tokens": { + "distinct_n": 364, + "finite_n": 394, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 394 + }, + "raw_length_order_sha256": "4a9cc03e351cce8c558b30732418c878fc3b6d24e64bef6717ce7c87c384384f", + "request_id_order_sha256": "2677a6a12ff0e104d6246dd121d0667ff4fe3d25f8d68699547906420246ebdd" + }, + "slo_pass_count": 81, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 134, + "finite_n": 134, + "max": 58.49868067706143, + "min": 5.973745897590131, + "missing_n": 260, + "n": 394 + }, + "ttft_ms": { + "distinct_n": 134, + "finite_n": 134, + "max": 4140.989850013284, + "min": 41.99217198765837, + "missing_n": 260, + "n": 394 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 3.283333333333333, + "request_count": 394 + }, + "v24": { + "feasible": false, + "pass_rate": 0.5748730964467006, + "rate_per_gpu": 3.283333333333333, + "request_count": 394 + } + } + ], + "boundary": "DOWN", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp2_mns32", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 22996, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns32/opprof/opprof-v1-dp0-pid2730662-1783873935616837232.jsonl" + }, + "censor": null, + "drift": -0.0076142131979695105, + "f20": 3.283333333333333, + "f24": 3.2583333333333333, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 32, + "observed_max_feasible_rate": 3.2583333333333333, + "tp": 2 + }, + "tp2_mns64": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.5, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.4601449275362319, + "solo_minus_colocated_primary_pass_rate": 0.5398550724637681, + "trial_pass_rates": [ + 0.4601449275362319 + ] + }, + "confirmations": [ + { + "anchor": 0.5, + "cell": "tp2_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 276, + "feasible": true, + "interval": { + "elapsed_s": 60.595378457, + "end_mono_ns": 206073272935295, + "end_wall_ns": 1783874500399363603, + "start_mono_ns": 206012677556838, + "start_wall_ns": 1783874439803984826 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.43938933197651325, + "decode_B_mean": 3.2814079760344503, + "decode_only_steps": 10406, + "decode_tokens": 35052, + "graph_mode_counts": { + "FULL": 10424, + "PIECEWISE": 258 + }, + "graph_mode_shares": { + "FULL": 0.9758472196217937, + "PIECEWISE": 0.02415278037820633 + }, + "kv_usage_max": 0.02765094834311732, + "kv_usage_mean": 0.008405260750030348, + "mixed_steps": 268, + "model_steps": 10682, + "padding_fraction": 0.1896895163036418, + "preemptions": 0, + "prefill_only_steps": 8, + "prefill_tokens": 2373, + "steps": 10696, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 64, + "observed_count": 276, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 2.3, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 276, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 7.359818944764346, + "min": 4.136619818769028, + "missing_n": 0, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 77.53581900033168, + "min": 35.34514401690103, + "missing_n": 0, + "n": 276 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.4389125542508539, + "decode_B_mean": 3.2792590513612123, + "decode_only_steps": 10413, + "decode_tokens": 35052, + "graph_mode_counts": { + "FULL": 10431, + "PIECEWISE": 258 + }, + "graph_mode_shares": { + "FULL": 0.9758630367667696, + "PIECEWISE": 0.024136963233230425 + }, + "kv_usage_max": 0.02765094834311732, + "kv_usage_mean": 0.008399969301794988, + "mixed_steps": 268, + "model_steps": 10689, + "padding_fraction": 0.1888465039663618, + "preemptions": 0, + "prefill_only_steps": 8, + "prefill_tokens": 2373, + "steps": 10704, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.5, + "cell": "tp2_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 276, + "feasible": true, + "interval": { + "elapsed_s": 60.600044298, + "end_mono_ns": 206005852949422, + "end_wall_ns": 1783874432979377438, + "start_mono_ns": 205945252905124, + "start_wall_ns": 1783874372379332336 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 276, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 2.3, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 276, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 7.361844818861275, + "min": 4.134220165552527, + "missing_n": 0, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 276, + "finite_n": 276, + "max": 80.72970999637619, + "min": 35.60976800508797, + "missing_n": 0, + "n": 276 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 0.9855072463768116, + "rate_per_gpu": 2.3, + "request_count": 276 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3, + "request_count": 276 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.561381074168798, + "anchor": 0.75, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.14066496163682865, + "solo_minus_colocated_primary_pass_rate": 0.4207161125319694, + "trial_pass_rates": [ + 0.14066496163682865 + ] + }, + "confirmations": [ + { + "anchor": 0.75, + "cell": "tp2_mns64", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 391, + "feasible": false, + "interval": { + "elapsed_s": 63.069024051, + "end_mono_ns": 205939344506671, + "end_wall_ns": 1783874366470934060, + "start_mono_ns": 205876275482620, + "start_wall_ns": 1783874303401910107 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 1.0159259489996566, + "decode_B_mean": 11.521345707656613, + "decode_only_steps": 3960, + "decode_tokens": 49657, + "graph_mode_counts": { + "FULL": 3964, + "NONE": 197, + "PIECEWISE": 149 + }, + "graph_mode_shares": { + "FULL": 0.9197215777262181, + "NONE": 0.045707656612529, + "PIECEWISE": 0.0345707656612529 + }, + "kv_usage_max": 0.2012431416183862, + "kv_usage_mean": 0.028511089747761235, + "mixed_steps": 349, + "model_steps": 4310, + "padding_fraction": 0.011283134354128766, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 740571, + "steps": 4312, + "waiting_cv": 65.64297372910525, + "waiting_max": 1.0, + "waiting_mean": 0.0002320185614849188 + }, + "mns": 64, + "observed_count": 391, + "pass_rate": 0.8925831202046036, + "schema": 1, + "selection": { + "arrival_order_sha256": "d3ed714128e985d0e0a0ca6b3767210c181114ab6694fb066a60aecf5afa9f9a", + "arrival_s": { + "distinct_n": 391, + "finite_n": 391, + "max": 59.91880000000001, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 391 + }, + "count": 391, + "long_gt4096": 140, + "offered_req_s": 6.516666666666667, + "offered_req_s_per_gpu": 3.2583333333333333, + "raw_input_tokens": { + "distinct_n": 361, + "finite_n": 391, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 391 + }, + "raw_length_order_sha256": "8152bbaf9f27e84e6fbd6359174d17de4b9c96ea8b8c58e3760290be3469b2a9", + "request_id_order_sha256": "4f4c792e90f5c1bf33978fea26debcadfce5f8b872a31be86a88a1bf6b09a21c" + }, + "slo_pass_count": 349, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 391, + "finite_n": 391, + "max": 82.47090074005463, + "min": 5.57776618112744, + "missing_n": 0, + "n": 391 + }, + "ttft_ms": { + "distinct_n": 391, + "finite_n": 391, + "max": 973.1985879770946, + "min": 38.53105800226331, + "missing_n": 0, + "n": 391 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + false, + false + ], + "layer1": { + "decode_B_cv": 0.9873521668722912, + "decode_B_mean": 18.045209903121638, + "decode_only_steps": 815, + "decode_tokens": 16764, + "graph_mode_counts": { + "FULL": 815, + "NONE": 88, + "PIECEWISE": 26 + }, + "graph_mode_shares": { + "FULL": 0.8772874058127018, + "NONE": 0.09472551130247578, + "PIECEWISE": 0.02798708288482239 + }, + "kv_usage_max": 0.1673764851833378, + "kv_usage_mean": 0.0448404249585349, + "mixed_steps": 113, + "model_steps": 929, + "padding_fraction": 0.005237520594669355, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 344293, + "steps": 931, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 0.23017902813299232, + 0.8925831202046036 + ], + "primary": { + "anchor": 0.75, + "cell": "tp2_mns64", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 132, + "feasible": false, + "interval": { + "elapsed_s": 22.208300455, + "end_mono_ns": 205869691965843, + "end_wall_ns": 1783874296818392926, + "start_mono_ns": 205847483665388, + "start_wall_ns": 1783874274610092601 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 391, + "pass_rate": 0.23017902813299232, + "schema": 1, + "selection": { + "arrival_order_sha256": "d3ed714128e985d0e0a0ca6b3767210c181114ab6694fb066a60aecf5afa9f9a", + "arrival_s": { + "distinct_n": 391, + "finite_n": 391, + "max": 59.91880000000001, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 391 + }, + "count": 391, + "long_gt4096": 140, + "offered_req_s": 6.516666666666667, + "offered_req_s_per_gpu": 3.2583333333333333, + "raw_input_tokens": { + "distinct_n": 361, + "finite_n": 391, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 391 + }, + "raw_length_order_sha256": "8152bbaf9f27e84e6fbd6359174d17de4b9c96ea8b8c58e3760290be3469b2a9", + "request_id_order_sha256": "4f4c792e90f5c1bf33978fea26debcadfce5f8b872a31be86a88a1bf6b09a21c" + }, + "slo_pass_count": 90, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 132, + "finite_n": 132, + "max": 73.44510653546615, + "min": 5.941438047292341, + "missing_n": 259, + "n": 391 + }, + "ttft_ms": { + "distinct_n": 132, + "finite_n": 132, + "max": 704.1629290033597, + "min": 40.83754599560052, + "missing_n": 259, + "n": 391 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 3.2583333333333333, + "request_count": 391 + }, + "v24": { + "feasible": false, + "pass_rate": 0.561381074168798, + "rate_per_gpu": 3.2583333333333333, + "request_count": 391 + } + } + ], + "boundary": "DOWN", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp2_mns64", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 26882, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns64/opprof/opprof-v1-dp0-pid2736153-1783874256706634444.jsonl" + }, + "censor": null, + "drift": -0.2941176470588236, + "f20": 3.2583333333333333, + "f24": 2.3, + "material_frontier_moved": true, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 64, + "observed_max_feasible_rate": 2.3, + "tp": 2 + }, + "tp2_mns8": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.4921875, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.6914498141263941, + "solo_minus_colocated_primary_pass_rate": 0.30855018587360594, + "trial_pass_rates": [ + 0.6914498141263941 + ] + }, + "confirmations": [ + { + "anchor": 0.4921875, + "cell": "tp2_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 269, + "feasible": true, + "interval": { + "elapsed_s": 60.601494927, + "end_mono_ns": 206893279230894, + "end_wall_ns": 1783875320405658014, + "start_mono_ns": 206832677735967, + "start_wall_ns": 1783875259804163348 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.47181373537222526, + "decode_B_mean": 3.3661444477288405, + "decode_only_steps": 9880, + "decode_tokens": 34163, + "graph_mode_counts": { + "FULL": 9898, + "NONE": 52, + "PIECEWISE": 199 + }, + "graph_mode_shares": { + "FULL": 0.9752684993595429, + "NONE": 0.005123657503202286, + "PIECEWISE": 0.0196078431372549 + }, + "kv_usage_max": 0.02758955119125095, + "kv_usage_mean": 0.008499553958525155, + "mixed_steps": 260, + "model_steps": 10149, + "padding_fraction": 0.19511764446210214, + "preemptions": 0, + "prefill_only_steps": 9, + "prefill_tokens": 2303, + "steps": 10164, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 8, + "observed_count": 269, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "ffbc7b8dd324c8e745654110a2412b03ddade891beb2bfe022ffde6502b0184b", + "arrival_s": { + "distinct_n": 269, + "finite_n": 269, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 269 + }, + "count": 269, + "long_gt4096": 95, + "offered_req_s": 4.483333333333333, + "offered_req_s_per_gpu": 2.2416666666666667, + "raw_input_tokens": { + "distinct_n": 250, + "finite_n": 269, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 269 + }, + "raw_length_order_sha256": "bc951350376a487d34d2def78fd32a2286e23bf8a7f481ff578009e12e3f1fb8", + "request_id_order_sha256": "cd227aee3be472a35e8c35af60a44a1411c4dc689cbd98746b3fb8a8d329b44c" + }, + "slo_pass_count": 269, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 269, + "finite_n": 269, + "max": 7.978645503964627, + "min": 4.193213968592747, + "missing_n": 0, + "n": 269 + }, + "ttft_ms": { + "distinct_n": 269, + "finite_n": 269, + "max": 136.03269401937723, + "min": 35.76707499451004, + "missing_n": 0, + "n": 269 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.48627204434656124, + "decode_B_mean": 3.451156682493181, + "decode_only_steps": 9630, + "decode_tokens": 34163, + "graph_mode_counts": { + "FULL": 9645, + "NONE": 70, + "PIECEWISE": 184 + }, + "graph_mode_shares": { + "FULL": 0.9743408425093444, + "NONE": 0.007071421355692494, + "PIECEWISE": 0.018587736134963128 + }, + "kv_usage_max": 0.02758955119125095, + "kv_usage_mean": 0.008708181568414703, + "mixed_steps": 261, + "model_steps": 9899, + "padding_fraction": 0.08521454941596562, + "preemptions": 0, + "prefill_only_steps": 8, + "prefill_tokens": 56527, + "steps": 9914, + "waiting_cv": 9.543086602680969, + "waiting_max": 5.0, + "waiting_mean": 0.03909485806647136 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.4921875, + "cell": "tp2_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 269, + "feasible": true, + "interval": { + "elapsed_s": 61.694519409, + "end_mono_ns": 206826428376802, + "end_wall_ns": 1783875253554803495, + "start_mono_ns": 206764733857393, + "start_wall_ns": 1783875191860284710 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 269, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "ffbc7b8dd324c8e745654110a2412b03ddade891beb2bfe022ffde6502b0184b", + "arrival_s": { + "distinct_n": 269, + "finite_n": 269, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 269 + }, + "count": 269, + "long_gt4096": 95, + "offered_req_s": 4.483333333333333, + "offered_req_s_per_gpu": 2.2416666666666667, + "raw_input_tokens": { + "distinct_n": 250, + "finite_n": 269, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 269 + }, + "raw_length_order_sha256": "bc951350376a487d34d2def78fd32a2286e23bf8a7f481ff578009e12e3f1fb8", + "request_id_order_sha256": "cd227aee3be472a35e8c35af60a44a1411c4dc689cbd98746b3fb8a8d329b44c" + }, + "slo_pass_count": 269, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 269, + "finite_n": 269, + "max": 14.617633236311816, + "min": 4.180891763814908, + "missing_n": 0, + "n": 269 + }, + "ttft_ms": { + "distinct_n": 269, + "finite_n": 269, + "max": 1039.087387995096, + "min": 37.062153016449884, + "missing_n": 0, + "n": 269 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.2416666666666667, + "request_count": 269 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.2416666666666667, + "request_count": 269 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.5805860805860805, + "anchor": 0.49609375, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.09157509157509157, + "solo_minus_colocated_primary_pass_rate": 0.48901098901098894, + "trial_pass_rates": [ + 0.09157509157509157 + ] + }, + "confirmations": [ + { + "anchor": 0.49609375, + "cell": "tp2_mns8", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 258, + "feasible": false, + "interval": { + "elapsed_s": 59.397687517, + "end_mono_ns": 206758345547488, + "end_wall_ns": 1783875185471974373, + "start_mono_ns": 206698947859971, + "start_wall_ns": 1783875126074287316 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.42342382793116656, + "decode_B_mean": 5.461, + "decode_only_steps": 5742, + "decode_tokens": 32766, + "graph_mode_counts": { + "FULL": 5746, + "NONE": 190, + "PIECEWISE": 64 + }, + "graph_mode_shares": { + "FULL": 0.9576666666666667, + "NONE": 0.03166666666666667, + "PIECEWISE": 0.010666666666666666 + }, + "kv_usage_max": 0.037377333690644776, + "kv_usage_mean": 0.013459032536613793, + "mixed_steps": 256, + "model_steps": 6000, + "padding_fraction": 0.008552057016312654, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 475823, + "steps": 6004, + "waiting_cv": 1.6234139342293743, + "waiting_max": 11.0, + "waiting_mean": 1.8581666666666667 + }, + "mns": 8, + "observed_count": 273, + "pass_rate": 0.8901098901098901, + "schema": 1, + "selection": { + "arrival_order_sha256": "41168f4bd02f3ef4e83b7440a3dab7458f08df9c2bbd4452fad076b4a2e0eaed", + "arrival_s": { + "distinct_n": 273, + "finite_n": 273, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 273 + }, + "count": 273, + "long_gt4096": 97, + "offered_req_s": 4.55, + "offered_req_s_per_gpu": 2.275, + "raw_input_tokens": { + "distinct_n": 253, + "finite_n": 273, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 273 + }, + "raw_length_order_sha256": "9e38ed1766e3933dc05a7ced5d0a73fed47ce769800f0d2c10014af0ef823f28", + "request_id_order_sha256": "a78992ce59bb3b857a7ef8844e0a690ea9dd529a659cf5edd599b63fef8b6a80" + }, + "slo_pass_count": 243, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 258, + "finite_n": 258, + "max": 18.377687952776373, + "min": 4.628447409468065, + "missing_n": 15, + "n": 273 + }, + "ttft_ms": { + "distinct_n": 258, + "finite_n": 258, + "max": 2424.360237986548, + "min": 39.415776991518214, + "missing_n": 15, + "n": 273 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + false, + false + ], + "layer1": { + "decode_B_cv": 0.3706255790761462, + "decode_B_mean": 6.0536246276067525, + "decode_only_steps": 1918, + "decode_tokens": 12192, + "graph_mode_counts": { + "FULL": 1918, + "NONE": 87, + "PIECEWISE": 9 + }, + "graph_mode_shares": { + "FULL": 0.9523336643495531, + "NONE": 0.04319761668321748, + "PIECEWISE": 0.004468718967229395 + }, + "kv_usage_max": 0.024909715044089675, + "kv_usage_mean": 0.013887512347528483, + "mixed_steps": 95, + "model_steps": 2014, + "padding_fraction": 0.0036008349493892927, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 203091, + "steps": 2016, + "waiting_cv": 1.0644646575061532, + "waiting_max": 13.0, + "waiting_mean": 4.344091360476663 + }, + "pass_rates": [ + 0.27106227106227104, + 0.8901098901098901 + ], + "primary": { + "anchor": 0.49609375, + "cell": "tp2_mns8", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 96, + "feasible": false, + "interval": { + "elapsed_s": 23.212238551, + "end_mono_ns": 206692540366228, + "end_wall_ns": 1783875119666793202, + "start_mono_ns": 206669328127677, + "start_wall_ns": 1783875096454555226 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 273, + "pass_rate": 0.27106227106227104, + "schema": 1, + "selection": { + "arrival_order_sha256": "41168f4bd02f3ef4e83b7440a3dab7458f08df9c2bbd4452fad076b4a2e0eaed", + "arrival_s": { + "distinct_n": 273, + "finite_n": 273, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 273 + }, + "count": 273, + "long_gt4096": 97, + "offered_req_s": 4.55, + "offered_req_s_per_gpu": 2.275, + "raw_input_tokens": { + "distinct_n": 253, + "finite_n": 273, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 273 + }, + "raw_length_order_sha256": "9e38ed1766e3933dc05a7ced5d0a73fed47ce769800f0d2c10014af0ef823f28", + "request_id_order_sha256": "a78992ce59bb3b857a7ef8844e0a690ea9dd529a659cf5edd599b63fef8b6a80" + }, + "slo_pass_count": 74, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 2, + "tpot_ms": { + "distinct_n": 96, + "finite_n": 96, + "max": 15.690051220626916, + "min": 6.1712947953217405, + "missing_n": 177, + "n": 273 + }, + "ttft_ms": { + "distinct_n": 96, + "finite_n": 96, + "max": 2895.5836099921726, + "min": 42.191794025711715, + "missing_n": 177, + "n": 273 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.275, + "request_count": 273 + }, + "v24": { + "feasible": false, + "pass_rate": 0.5805860805860805, + "rate_per_gpu": 2.275, + "request_count": 273 + } + } + ], + "boundary": "DOWN", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp2_mns8", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 28404, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp2_mns8/opprof/opprof-v1-dp0-pid2750298-1783875076865595283.jsonl" + }, + "censor": null, + "drift": -0.0146520146520146, + "f20": 2.275, + "f24": 2.2416666666666667, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 8, + "observed_max_feasible_rate": 2.2416666666666667, + "tp": 2 + }, + "tp4_mns16": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.033182214016, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.6086956521739131, + "solo_minus_colocated_primary_pass_rate": 0.3913043478260869, + "trial_pass_rates": [ + 0.6086956521739131 + ] + }, + "confirmations": [ + { + "anchor": 0.033182214016, + "cell": "tp4_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 575, + "feasible": true, + "interval": { + "elapsed_s": 62.401356779, + "end_mono_ns": 206373778206418, + "end_wall_ns": 1783874800904633584, + "start_mono_ns": 206311376849639, + "start_wall_ns": 1783874738503276864 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.4803846550950009, + "decode_B_mean": 7.062379110251451, + "decode_only_steps": 9786, + "decode_tokens": 73025, + "graph_mode_counts": { + "FULL": 9815, + "NONE": 62, + "PIECEWISE": 463 + }, + "graph_mode_shares": { + "FULL": 0.9492263056092843, + "NONE": 0.005996131528046422, + "PIECEWISE": 0.04477756286266925 + }, + "kv_usage_max": 0.02085830736034522, + "kv_usage_mean": 0.006703151279153486, + "mixed_steps": 552, + "model_steps": 10340, + "padding_fraction": 0.0800442841521147, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 197867, + "steps": 10344, + "waiting_cv": 6.673872175860442, + "waiting_max": 15.0, + "waiting_mean": 0.236073500967118 + }, + "mns": 16, + "observed_count": 575, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "86ee457cd91595c76454d54ebc69890836ef603ceb96deaf9c7cf05e91de9407", + "arrival_s": { + "distinct_n": 574, + "finite_n": 575, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 575 + }, + "count": 575, + "long_gt4096": 178, + "offered_req_s": 9.583333333333334, + "offered_req_s_per_gpu": 2.3958333333333335, + "raw_input_tokens": { + "distinct_n": 524, + "finite_n": 575, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 575 + }, + "raw_length_order_sha256": "8354300b1680841ab43ef8164307d7c1f28bc877994b8d1e7584aff030677560", + "request_id_order_sha256": "e6b624a0212807df3733c54b4a2160fba24f19ca4776b14f34918d83428bba7f" + }, + "slo_pass_count": 575, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 575, + "finite_n": 575, + "max": 16.395648126032086, + "min": 3.8425548977904227, + "missing_n": 0, + "n": 575 + }, + "ttft_ms": { + "distinct_n": 575, + "finite_n": 575, + "max": 1883.0044260248542, + "min": 34.824717993615195, + "missing_n": 0, + "n": 575 + } + }, + { + "anchor": 0.033182214016, + "cell": "tp4_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 575, + "feasible": true, + "interval": { + "elapsed_s": 60.609467854, + "end_mono_ns": 206440578236362, + "end_wall_ns": 1783874867704663470, + "start_mono_ns": 206379968768508, + "start_wall_ns": 1783874807095196033 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.4504546097224705, + "decode_B_mean": 6.67138680796638, + "decode_only_steps": 10383, + "decode_tokens": 73025, + "graph_mode_counts": { + "FULL": 10415, + "NONE": 2, + "PIECEWISE": 529 + }, + "graph_mode_shares": { + "FULL": 0.9514891284487484, + "NONE": 0.0001827151470856934, + "PIECEWISE": 0.048328156404165906 + }, + "kv_usage_max": 0.020235975861699562, + "kv_usage_mean": 0.00634432601196062, + "mixed_steps": 561, + "model_steps": 10946, + "padding_fraction": 0.24434226736121215, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 4875, + "steps": 10950, + "waiting_cv": 22.80872848797353, + "waiting_max": 1.0, + "waiting_mean": 0.0019185090443997808 + }, + "mns": 16, + "observed_count": 575, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "86ee457cd91595c76454d54ebc69890836ef603ceb96deaf9c7cf05e91de9407", + "arrival_s": { + "distinct_n": 574, + "finite_n": 575, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 575 + }, + "count": 575, + "long_gt4096": 178, + "offered_req_s": 9.583333333333334, + "offered_req_s_per_gpu": 2.3958333333333335, + "raw_input_tokens": { + "distinct_n": 524, + "finite_n": 575, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 575 + }, + "raw_length_order_sha256": "8354300b1680841ab43ef8164307d7c1f28bc877994b8d1e7584aff030677560", + "request_id_order_sha256": "e6b624a0212807df3733c54b4a2160fba24f19ca4776b14f34918d83428bba7f" + }, + "slo_pass_count": 575, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 575, + "finite_n": 575, + "max": 7.079952629970562, + "min": 3.8461172678251203, + "missing_n": 0, + "n": 575 + }, + "ttft_ms": { + "distinct_n": 575, + "finite_n": 575, + "max": 181.3676160236355, + "min": 33.920757996384054, + "missing_n": 0, + "n": 575 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + false, + true, + true + ], + "layer1": { + "decode_B_cv": 0.56014861811598, + "decode_B_mean": 8.163244484121924, + "decode_only_steps": 7355, + "decode_tokens": 64008, + "graph_mode_counts": { + "FULL": 7370, + "NONE": 194, + "PIECEWISE": 277 + }, + "graph_mode_shares": { + "FULL": 0.9399311312332611, + "NONE": 0.024741742124728988, + "PIECEWISE": 0.035327126642009946 + }, + "kv_usage_max": 0.02132760652325838, + "kv_usage_mean": 0.007707377988685883, + "mixed_steps": 484, + "model_steps": 7841, + "padding_fraction": 0.02360025908891416, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 534444, + "steps": 7845, + "waiting_cv": 2.4771546994509084, + "waiting_max": 40.0, + "waiting_mean": 3.417166177783446 + }, + "pass_rates": [ + 0.7530434782608696, + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.033182214016, + "cell": "tp4_mns16", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 504, + "feasible": false, + "interval": { + "elapsed_s": 57.926653309, + "end_mono_ns": 206305019222036, + "end_wall_ns": 1783874732145649173, + "start_mono_ns": 206247092568727, + "start_wall_ns": 1783874674218995807 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 575, + "pass_rate": 0.7530434782608696, + "schema": 1, + "selection": { + "arrival_order_sha256": "86ee457cd91595c76454d54ebc69890836ef603ceb96deaf9c7cf05e91de9407", + "arrival_s": { + "distinct_n": 574, + "finite_n": 575, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 575 + }, + "count": 575, + "long_gt4096": 178, + "offered_req_s": 9.583333333333334, + "offered_req_s_per_gpu": 2.3958333333333335, + "raw_input_tokens": { + "distinct_n": 524, + "finite_n": 575, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 575 + }, + "raw_length_order_sha256": "8354300b1680841ab43ef8164307d7c1f28bc877994b8d1e7584aff030677560", + "request_id_order_sha256": "e6b624a0212807df3733c54b4a2160fba24f19ca4776b14f34918d83428bba7f" + }, + "slo_pass_count": 433, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 504, + "finite_n": 504, + "max": 17.29787870078333, + "min": 3.856502622081788, + "missing_n": 71, + "n": 575 + }, + "ttft_ms": { + "distinct_n": 504, + "finite_n": 504, + "max": 4620.4556869925, + "min": 35.387877986067906, + "missing_n": 71, + "n": 575 + } + }, + "resolved": true, + "trial_count": 3, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3958333333333335, + "request_count": 575 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.3958333333333335, + "request_count": 575 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.2380546075085324, + "anchor": 0.033717411016, + "colocated": { + "accepted_feasible": null, + "primary_feasible": false, + "primary_pass_rate": 0.03583617747440273, + "solo_minus_colocated_primary_pass_rate": 0.20221843003412968, + "trial_pass_rates": [ + 0.03583617747440273, + 1.0 + ] + }, + "confirmations": [ + { + "anchor": 0.033717411016, + "cell": "tp4_mns16", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 299, + "feasible": false, + "interval": { + "elapsed_s": 36.241514217, + "end_mono_ns": 206240554322036, + "end_wall_ns": 1783874667680749525, + "start_mono_ns": 206204312807819, + "start_wall_ns": 1783874631439235350 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.5002690999504984, + "decode_B_mean": 9.4272591857001, + "decode_only_steps": 3736, + "decode_tokens": 37973, + "graph_mode_counts": { + "FULL": 3748, + "NONE": 144, + "PIECEWISE": 136 + }, + "graph_mode_shares": { + "FULL": 0.9304865938430983, + "NONE": 0.03574975173783516, + "PIECEWISE": 0.033763654419066536 + }, + "kv_usage_max": 0.027770267857598285, + "kv_usage_mean": 0.009047796233306362, + "mixed_steps": 291, + "model_steps": 4028, + "padding_fraction": 0.014732144579056788, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 472914, + "steps": 4030, + "waiting_cv": 1.8755427871196464, + "waiting_max": 33.0, + "waiting_mean": 4.938182720953327 + }, + "mns": 16, + "observed_count": 586, + "pass_rate": 0.4129692832764505, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 242, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 299, + "finite_n": 299, + "max": 19.551449724588558, + "min": 4.617206692946781, + "missing_n": 287, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 299, + "finite_n": 299, + "max": 4369.375169015257, + "min": 35.537771997042, + "missing_n": 287, + "n": 586 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + false, + false + ], + "layer1": { + "decode_B_cv": 0.40183749315339584, + "decode_B_mean": 12.478345184227537, + "decode_only_steps": 1419, + "decode_tokens": 19304, + "graph_mode_counts": { + "FULL": 1419, + "NONE": 112, + "PIECEWISE": 16 + }, + "graph_mode_shares": { + "FULL": 0.9172592113768584, + "NONE": 0.07239819004524888, + "PIECEWISE": 0.010342598577892695 + }, + "kv_usage_max": 0.01979728316593299, + "kv_usage_mean": 0.009849502078910156, + "mixed_steps": 127, + "model_steps": 1547, + "padding_fraction": 0.0025222796189245523, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 301815, + "steps": 1549, + "waiting_cv": 0.9394117903870358, + "waiting_max": 48.0, + "waiting_mean": 21.13251454427925 + }, + "pass_rates": [ + 0.06313993174061433, + 0.4129692832764505 + ], + "primary": { + "anchor": 0.033717411016, + "cell": "tp4_mns16", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 152, + "feasible": false, + "interval": { + "elapsed_s": 25.795635813, + "end_mono_ns": 206197322315089, + "end_wall_ns": 1783874624448742328, + "start_mono_ns": 206171526679276, + "start_wall_ns": 1783874598653106506 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 586, + "pass_rate": 0.06313993174061433, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 37, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 152, + "finite_n": 152, + "max": 38.16994751173834, + "min": 4.869590023684675, + "missing_n": 434, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 152, + "finite_n": 152, + "max": 7810.731097008102, + "min": 37.92213799897581, + "missing_n": 434, + "n": 586 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.441666666666667, + "request_count": 586 + }, + "v24": { + "feasible": false, + "pass_rate": 0.2380546075085324, + "rate_per_gpu": 2.441666666666667, + "request_count": 586 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.034252608017, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.8466666666666667, + "solo_minus_colocated_primary_pass_rate": 0.15333333333333332, + "trial_pass_rates": [ + 0.8466666666666667 + ] + }, + "confirmations": [ + { + "anchor": 0.034252608017, + "cell": "tp4_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 600, + "feasible": true, + "interval": { + "elapsed_s": 60.607371178, + "end_mono_ns": 206573458638053, + "end_wall_ns": 1783875000585066169, + "start_mono_ns": 206512851266875, + "start_wall_ns": 1783874939977694056 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.4392997651461427, + "decode_B_mean": 7.083759412475597, + "decode_only_steps": 10175, + "decode_tokens": 76200, + "graph_mode_counts": { + "FULL": 10207, + "NONE": 5, + "PIECEWISE": 545 + }, + "graph_mode_shares": { + "FULL": 0.9488705029283258, + "NONE": 0.00046481360974249327, + "PIECEWISE": 0.050664683461931766 + }, + "kv_usage_max": 0.023199702097053132, + "kv_usage_mean": 0.0067668545830499825, + "mixed_steps": 580, + "model_steps": 10757, + "padding_fraction": 0.23808720270739, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 5074, + "steps": 10761, + "waiting_cv": 16.231307168862653, + "waiting_max": 3.0, + "waiting_mean": 0.007065166868085898 + }, + "mns": 16, + "observed_count": 600, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 600, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 7.261857527672995, + "min": 3.852719755927614, + "missing_n": 0, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 210.09885301464237, + "min": 35.12379902531393, + "missing_n": 0, + "n": 600 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.43553630007582306, + "decode_B_mean": 7.3332691752478105, + "decode_only_steps": 9816, + "decode_tokens": 76200, + "graph_mode_counts": { + "FULL": 9844, + "NONE": 23, + "PIECEWISE": 524 + }, + "graph_mode_shares": { + "FULL": 0.9473582908286017, + "NONE": 0.002213453950534116, + "PIECEWISE": 0.05042825522086421 + }, + "kv_usage_max": 0.023245611797772847, + "kv_usage_mean": 0.006997659194358993, + "mixed_steps": 573, + "model_steps": 10391, + "padding_fraction": 0.1403000197011061, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 76530, + "steps": 10395, + "waiting_cv": 9.789921043363753, + "waiting_max": 3.0, + "waiting_mean": 0.02367433355788663 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.034252608017, + "cell": "tp4_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 600, + "feasible": true, + "interval": { + "elapsed_s": 60.606381404, + "end_mono_ns": 206507070554626, + "end_wall_ns": 1783874934196982187, + "start_mono_ns": 206446464173222, + "start_wall_ns": 1783874873590600542 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 600, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 600, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 8.318593196880704, + "min": 3.8498818111701274, + "missing_n": 0, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 367.71361899445765, + "min": 35.09980899980292, + "missing_n": 0, + "n": 600 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.16, + "rate_per_gpu": 2.5, + "request_count": 600 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.5, + "request_count": 600 + } + } + ], + "boundary": "NONMONOTONIC", + "bounded": false, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp4_mns16", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 56011, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns16/opprof/opprof-v1-dp0-pid2741574-1783874579483097755.jsonl" + }, + "censor": "NONMONOTONIC_SOLO_ANCHORS", + "drift": null, + "f20": 2.441666666666667, + "f24": null, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 16, + "observed_max_feasible_rate": 2.5, + "tp": 4 + }, + "tp4_mns32": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.033717411016, + "colocated": null, + "confirmations": [ + { + "anchor": 0.033717411016, + "cell": "tp4_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 586, + "feasible": true, + "interval": { + "elapsed_s": 62.020765229, + "end_mono_ns": 204687293289695, + "end_wall_ns": 1783873114419717059, + "start_mono_ns": 204625272524466, + "start_wall_ns": 1783873052398951749 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.7728594063694523, + "decode_B_mean": 14.757485623636724, + "decode_only_steps": 4528, + "decode_tokens": 74422, + "graph_mode_counts": { + "FULL": 4538, + "NONE": 353, + "PIECEWISE": 152 + }, + "graph_mode_shares": { + "FULL": 0.8998611937338885, + "NONE": 0.06999801705334127, + "PIECEWISE": 0.030140789212770178 + }, + "kv_usage_max": 0.0474979704994869, + "kv_usage_mean": 0.013829397882131243, + "mixed_steps": 514, + "model_steps": 5043, + "padding_fraction": 0.00724689319740189, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 1249040, + "steps": 5044, + "waiting_cv": 2.1417261616924654, + "waiting_max": 14.0, + "waiting_mean": 1.331945270672219 + }, + "mns": 32, + "observed_count": 586, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 586, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 30.96185862196712, + "min": 4.221293874044616, + "missing_n": 0, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 1400.6275889987592, + "min": 36.0499199887272, + "missing_n": 0, + "n": 586 + } + }, + { + "anchor": 0.033717411016, + "cell": "tp4_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 586, + "feasible": true, + "interval": { + "elapsed_s": 60.604883143, + "end_mono_ns": 204754670204385, + "end_wall_ns": 1783873181796631942, + "start_mono_ns": 204694065321242, + "start_wall_ns": 1783873121191748842 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.45040651352782296, + "decode_B_mean": 6.820823022637705, + "decode_only_steps": 10338, + "decode_tokens": 74422, + "graph_mode_counts": { + "FULL": 10370, + "PIECEWISE": 541 + }, + "graph_mode_shares": { + "FULL": 0.9504170103565209, + "PIECEWISE": 0.049582989643479056 + }, + "kv_usage_max": 0.02171948473662444, + "kv_usage_mean": 0.0065474296776537955, + "mixed_steps": 571, + "model_steps": 10911, + "padding_fraction": 0.24547562922971636, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 4960, + "steps": 10914, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 32, + "observed_count": 586, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 586, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 6.865110700926194, + "min": 3.877922850404316, + "missing_n": 0, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 149.36435397248715, + "min": 35.21961401565932, + "missing_n": 0, + "n": 586 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + false, + true, + true + ], + "layer1": { + "decode_B_cv": 0.7499245353192122, + "decode_B_mean": 18.03623898139079, + "decode_only_steps": 919, + "decode_tokens": 18415, + "graph_mode_counts": { + "FULL": 919, + "NONE": 84, + "PIECEWISE": 18 + }, + "graph_mode_shares": { + "FULL": 0.9000979431929481, + "NONE": 0.08227228207639568, + "PIECEWISE": 0.01762977473065622 + }, + "kv_usage_max": 0.03680160930455778, + "kv_usage_mean": 0.014215112207466527, + "mixed_steps": 101, + "model_steps": 1021, + "padding_fraction": 0.002578912799907275, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 291380, + "steps": 1023, + "waiting_cv": 1.3083314310129481, + "waiting_max": 32.0, + "waiting_mean": 10.449559255631733 + }, + "pass_rates": [ + 0.030716723549488054, + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.033717411016, + "cell": "tp4_mns32", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 145, + "feasible": false, + "interval": { + "elapsed_s": 26.619371633, + "end_mono_ns": 204618619172708, + "end_wall_ns": 1783873045745600041, + "start_mono_ns": 204591999801075, + "start_wall_ns": 1783873019126228288 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 586, + "pass_rate": 0.030716723549488054, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 18, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 145, + "finite_n": 145, + "max": 73.72474424402581, + "min": 4.843318141679697, + "missing_n": 441, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 145, + "finite_n": 145, + "max": 8646.978151984513, + "min": 38.25148000032641, + "missing_n": 441, + "n": 586 + } + }, + "resolved": true, + "trial_count": 3, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.441666666666667, + "request_count": 586 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.441666666666667, + "request_count": 586 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.034252608017, + "colocated": null, + "confirmations": [ + { + "anchor": 0.034252608017, + "cell": "tp4_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 600, + "feasible": true, + "interval": { + "elapsed_s": 60.604396598, + "end_mono_ns": 204887731841084, + "end_wall_ns": 1783873314858268660, + "start_mono_ns": 204827127444486, + "start_wall_ns": 1783873254253871898 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.43810333602442786, + "decode_B_mean": 7.013345605154165, + "decode_only_steps": 10279, + "decode_tokens": 76200, + "graph_mode_counts": { + "FULL": 10312, + "PIECEWISE": 553 + }, + "graph_mode_shares": { + "FULL": 0.9491026231017027, + "PIECEWISE": 0.050897376898297285 + }, + "kv_usage_max": 0.02171948473662444, + "kv_usage_mean": 0.006707113892272084, + "mixed_steps": 584, + "model_steps": 10865, + "padding_fraction": 0.23717900585673524, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 5074, + "steps": 10868, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 32, + "observed_count": 600, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 600, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 6.888400803147778, + "min": 3.8725274959194116, + "missing_n": 0, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 180.16752798575908, + "min": 36.45357800996862, + "missing_n": 0, + "n": 600 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.43758222525190354, + "decode_B_mean": 7.132827857343443, + "decode_only_steps": 10102, + "decode_tokens": 76200, + "graph_mode_counts": { + "FULL": 10134, + "NONE": 12, + "PIECEWISE": 537 + }, + "graph_mode_shares": { + "FULL": 0.9486099410278012, + "NONE": 0.0011232799775344005, + "PIECEWISE": 0.05026677899466442 + }, + "kv_usage_max": 0.02170927341328799, + "kv_usage_mean": 0.006817761170149559, + "mixed_steps": 579, + "model_steps": 10683, + "padding_fraction": 0.18119643194947688, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 36338, + "steps": 10686, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.034252608017, + "cell": "tp4_mns32", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 600, + "feasible": true, + "interval": { + "elapsed_s": 60.611230768, + "end_mono_ns": 204821255565263, + "end_wall_ns": 1783873248381992322, + "start_mono_ns": 204760644334495, + "start_wall_ns": 1783873187770761898 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 32, + "observed_count": 600, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 600, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 8.065448354335691, + "min": 3.8434486376950296, + "missing_n": 0, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 213.7685909983702, + "min": 36.311193980509415, + "missing_n": 0, + "n": 600 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.3466666666666667, + "rate_per_gpu": 2.5, + "request_count": 600 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.5, + "request_count": 600 + } + } + ], + "boundary": "UP", + "bounded": false, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp4_mns32", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 38705, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns32/opprof/opprof-v1-dp0-pid2712385-1783873000008589576.jsonl" + }, + "censor": "RIGHT_CENSORED_HISTORY_EDGE", + "drift": 0.023890784982935065, + "f20": 2.441666666666667, + "f24": 2.5, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 32, + "observed_max_feasible_rate": 2.5, + "tp": 4 + }, + "tp4_mns64": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 0.9573378839590444, + "anchor": 0.033717411016, + "colocated": null, + "confirmations": [ + { + "anchor": 0.033717411016, + "cell": "tp4_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 586, + "feasible": true, + "interval": { + "elapsed_s": 61.252125704, + "end_mono_ns": 205150002958549, + "end_wall_ns": 1783873577129386048, + "start_mono_ns": 205088750832845, + "start_wall_ns": 1783873515877260392 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.990784912736828, + "decode_B_mean": 17.199445343193897, + "decode_only_steps": 3887, + "decode_tokens": 74422, + "graph_mode_counts": { + "FULL": 3900, + "NONE": 277, + "PIECEWISE": 150 + }, + "graph_mode_shares": { + "FULL": 0.9013173099144904, + "NONE": 0.06401663970418303, + "PIECEWISE": 0.03466605038132656 + }, + "kv_usage_max": 0.07642403607251314, + "kv_usage_mean": 0.016070506041063987, + "mixed_steps": 439, + "model_steps": 4327, + "padding_fraction": 0.008507526868831557, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 1249040, + "steps": 4328, + "waiting_cv": 15.388934053580373, + "waiting_max": 14.0, + "waiting_mean": 0.025190663277097295 + }, + "mns": 64, + "observed_count": 586, + "pass_rate": 0.9573378839590444, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 561, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 57.01185407082176, + "min": 4.5877628188349275, + "missing_n": 0, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 1760.64259299892, + "min": 36.9716240093112, + "missing_n": 0, + "n": 586 + } + }, + { + "anchor": 0.033717411016, + "cell": "tp4_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 586, + "feasible": true, + "interval": { + "elapsed_s": 60.60792841, + "end_mono_ns": 205216585331572, + "end_wall_ns": 1783873643711758987, + "start_mono_ns": 205155977403162, + "start_wall_ns": 1783873583103830301 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.45282168963732516, + "decode_B_mean": 6.808966148215919, + "decode_only_steps": 10362, + "decode_tokens": 74422, + "graph_mode_counts": { + "FULL": 10394, + "PIECEWISE": 536 + }, + "graph_mode_shares": { + "FULL": 0.95096065873742, + "PIECEWISE": 0.049039341262580055 + }, + "kv_usage_max": 0.02175291146488345, + "kv_usage_mean": 0.006544635598350098, + "mixed_steps": 566, + "model_steps": 10930, + "padding_fraction": 0.24457090653013835, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 4960, + "steps": 10933, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 64, + "observed_count": 586, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 586, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 6.877395669300857, + "min": 3.8451504644728813, + "missing_n": 0, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 586, + "finite_n": 586, + "max": 156.2074380053673, + "min": 35.24657100206241, + "missing_n": 0, + "n": 586 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + false, + true, + true + ], + "layer1": { + "decode_B_cv": 1.0913841089520357, + "decode_B_mean": 23.85362694300518, + "decode_only_steps": 697, + "decode_tokens": 18415, + "graph_mode_counts": { + "FULL": 697, + "NONE": 57, + "PIECEWISE": 18 + }, + "graph_mode_shares": { + "FULL": 0.9028497409326425, + "NONE": 0.07383419689119171, + "PIECEWISE": 0.023316062176165803 + }, + "kv_usage_max": 0.05553510628508329, + "kv_usage_mean": 0.018575513354830497, + "mixed_steps": 74, + "model_steps": 772, + "padding_fraction": 0.0030892503740889126, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 291380, + "steps": 774, + "waiting_cv": 8.047543908731592, + "waiting_max": 15.0, + "waiting_mean": 0.11658031088082901 + }, + "pass_rates": [ + 0.11433447098976109, + 0.9573378839590444, + 1.0 + ], + "primary": { + "anchor": 0.033717411016, + "cell": "tp4_mns64", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 145, + "feasible": false, + "interval": { + "elapsed_s": 21.629262541, + "end_mono_ns": 205082240360979, + "end_wall_ns": 1783873509366788019, + "start_mono_ns": 205060611098438, + "start_wall_ns": 1783873487737525923 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 586, + "pass_rate": 0.11433447098976109, + "schema": 1, + "selection": { + "arrival_order_sha256": "fe44bf6b0efd6e6ab4d19676e34ace608827e419dbb5c19c8595992f0339c9b1", + "arrival_s": { + "distinct_n": 585, + "finite_n": 586, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 586 + }, + "count": 586, + "long_gt4096": 184, + "offered_req_s": 9.766666666666667, + "offered_req_s_per_gpu": 2.441666666666667, + "raw_input_tokens": { + "distinct_n": 534, + "finite_n": 586, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 586 + }, + "raw_length_order_sha256": "e2d040068b279e99073f6965d1136eb0ca6e8534c883f331b4cb85c5fc7a4650", + "request_id_order_sha256": "472a874ecc0e844b5b36a715bbc309b63e73074d39327cd734869f76721d71db" + }, + "slo_pass_count": 67, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 145, + "finite_n": 145, + "max": 91.25109214961756, + "min": 4.869500480135756, + "missing_n": 441, + "n": 586 + }, + "ttft_ms": { + "distinct_n": 145, + "finite_n": 145, + "max": 2708.699943992542, + "min": 38.82970198174007, + "missing_n": 441, + "n": 586 + } + }, + "resolved": true, + "trial_count": 3, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.441666666666667, + "request_count": 586 + }, + "v24": { + "feasible": true, + "pass_rate": 0.9573378839590444, + "rate_per_gpu": 2.441666666666667, + "request_count": 586 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.034252608017, + "colocated": null, + "confirmations": [ + { + "anchor": 0.034252608017, + "cell": "tp4_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 600, + "feasible": true, + "interval": { + "elapsed_s": 60.610679916, + "end_mono_ns": 205349689243783, + "end_wall_ns": 1783873776815670721, + "start_mono_ns": 205289078563867, + "start_wall_ns": 1783873716204991417 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.43808700909802734, + "decode_B_mean": 7.037960653920754, + "decode_only_steps": 10241, + "decode_tokens": 76200, + "graph_mode_counts": { + "FULL": 10275, + "PIECEWISE": 552 + }, + "graph_mode_shares": { + "FULL": 0.9490163480188418, + "PIECEWISE": 0.050983651981158215 + }, + "kv_usage_max": 0.021747799147265434, + "kv_usage_mean": 0.006738918545855606, + "mixed_steps": 584, + "model_steps": 10827, + "padding_fraction": 0.23898611384214913, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 5074, + "steps": 10830, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "mns": 64, + "observed_count": 600, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 600, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 6.913960456672147, + "min": 3.8400337796229285, + "missing_n": 0, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 166.59539798274636, + "min": 34.91001299698837, + "missing_n": 0, + "n": 600 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.43723497129874916, + "decode_B_mean": 7.1435267647886, + "decode_only_steps": 10093, + "decode_tokens": 76200, + "graph_mode_counts": { + "FULL": 10123, + "NONE": 11, + "PIECEWISE": 533 + }, + "graph_mode_shares": { + "FULL": 0.9490015937001969, + "NONE": 0.0010312177744445487, + "PIECEWISE": 0.049967188525358586 + }, + "kv_usage_max": 0.02175291146488345, + "kv_usage_mean": 0.006837303780040747, + "mixed_steps": 572, + "model_steps": 10667, + "padding_fraction": 0.18233272543648688, + "preemptions": 0, + "prefill_only_steps": 2, + "prefill_tokens": 36338, + "steps": 10671, + "waiting_cv": 0.0, + "waiting_max": 0.0, + "waiting_mean": 0.0 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.034252608017, + "cell": "tp4_mns64", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 600, + "feasible": true, + "interval": { + "elapsed_s": 60.605306393, + "end_mono_ns": 205283157213222, + "end_wall_ns": 1783873710283642970, + "start_mono_ns": 205222551906829, + "start_wall_ns": 1783873649678334142 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 64, + "observed_count": 600, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 600, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 7.9323868347445545, + "min": 3.8530240866619594, + "missing_n": 0, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 600, + "finite_n": 600, + "max": 200.9163040202111, + "min": 37.27158301626332, + "missing_n": 0, + "n": 600 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.3466666666666667, + "rate_per_gpu": 2.5, + "request_count": 600 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 2.5, + "request_count": 600 + } + } + ], + "boundary": "UP", + "bounded": false, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp4_mns64", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 37695, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns64/opprof/opprof-v1-dp0-pid2721119-1783873467764669604.jsonl" + }, + "censor": "RIGHT_CENSORED_HISTORY_EDGE", + "drift": 0.023890784982935065, + "f20": 2.441666666666667, + "f24": 2.5, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 64, + "observed_max_feasible_rate": 2.5, + "tp": 4 + }, + "tp4_mns8": { + "anchors": [ + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.016055910008, + "colocated": { + "accepted_feasible": true, + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_minus_colocated_primary_pass_rate": 0.0, + "trial_pass_rates": [ + 1.0 + ] + }, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + true + ], + "layer1": { + "decode_B_cv": 0.6336819511764374, + "decode_B_mean": 3.168421052631579, + "decode_only_steps": 11771, + "decode_tokens": 38227, + "graph_mode_counts": { + "FULL": 11787, + "NONE": 58, + "PIECEWISE": 220 + }, + "graph_mode_shares": { + "FULL": 0.976958143389971, + "NONE": 0.004807293825113966, + "PIECEWISE": 0.018234562784915042 + }, + "kv_usage_max": 0.014136726419664902, + "kv_usage_mean": 0.003088824282364161, + "mixed_steps": 274, + "model_steps": 12065, + "padding_fraction": 0.14902320921689766, + "preemptions": 0, + "prefill_only_steps": 20, + "prefill_tokens": 2545, + "steps": 12103, + "waiting_cv": 6.602592185052594, + "waiting_max": 4.0, + "waiting_mean": 0.048984666390385415 + }, + "pass_rates": [ + 1.0 + ], + "primary": { + "anchor": 0.016055910008, + "cell": "tp4_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 301, + "feasible": true, + "interval": { + "elapsed_s": 60.445619611, + "end_mono_ns": 207596969411260, + "end_wall_ns": 1783876024095838081, + "start_mono_ns": 207536523791649, + "start_wall_ns": 1783875963650218795 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 301, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "42c88c970510ae1b49ebcf10d2a9fb9bbc07c2ec2761dfd19193a6962cab1273", + "arrival_s": { + "distinct_n": 300, + "finite_n": 301, + "max": 59.75439999999999, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 301 + }, + "count": 301, + "long_gt4096": 95, + "offered_req_s": 5.016666666666667, + "offered_req_s_per_gpu": 1.2541666666666667, + "raw_input_tokens": { + "distinct_n": 283, + "finite_n": 301, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 301 + }, + "raw_length_order_sha256": "d1765a9ae2785d90f135e114ce1d802dec58e86fd36152182dee987a73d1604c", + "request_id_order_sha256": "2d680e08c068237fadfff567b3fb7f55a8e0a0121c009cefa07ee2653093e700" + }, + "slo_pass_count": 301, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 301, + "finite_n": 301, + "max": 7.6074999528863705, + "min": 3.489643448778597, + "missing_n": 0, + "n": 301 + }, + "ttft_ms": { + "distinct_n": 301, + "finite_n": 301, + "max": 479.98269999516197, + "min": 36.43614600878209, + "missing_n": 0, + "n": 301 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 1.2541666666666667, + "request_count": 301 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 1.2541666666666667, + "request_count": 301 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.016591107009, + "colocated": { + "accepted_feasible": false, + "primary_feasible": false, + "primary_pass_rate": 0.07142857142857142, + "solo_minus_colocated_primary_pass_rate": 0.9285714285714286, + "trial_pass_rates": [ + 0.07142857142857142 + ] + }, + "confirmations": [ + { + "anchor": 0.016591107009, + "cell": "tp4_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 308, + "feasible": true, + "interval": { + "elapsed_s": 60.446717466, + "end_mono_ns": 207530401973415, + "end_wall_ns": 1783875957528400301, + "start_mono_ns": 207469955255949, + "start_wall_ns": 1783875897081683055 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.6262272955855006, + "decode_B_mean": 3.317445509286744, + "decode_only_steps": 11489, + "decode_tokens": 39116, + "graph_mode_counts": { + "FULL": 11507, + "NONE": 60, + "PIECEWISE": 224 + }, + "graph_mode_shares": { + "FULL": 0.9759138325841744, + "NONE": 0.005088626918836401, + "PIECEWISE": 0.018997540496989228 + }, + "kv_usage_max": 0.014136726419664902, + "kv_usage_mean": 0.0032106540833222847, + "mixed_steps": 285, + "model_steps": 11791, + "padding_fraction": 0.14593952793301807, + "preemptions": 0, + "prefill_only_steps": 17, + "prefill_tokens": 2604, + "steps": 11825, + "waiting_cv": 4.97853639909132, + "waiting_max": 4.0, + "waiting_mean": 0.07709269782037147 + }, + "mns": 8, + "observed_count": 308, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "860d2ac188790a401e93d3f99b9f37d6d69f0dfda1af94c37bc0067278444632", + "arrival_s": { + "distinct_n": 307, + "finite_n": 308, + "max": 59.75439999999999, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 308 + }, + "count": 308, + "long_gt4096": 96, + "offered_req_s": 5.133333333333334, + "offered_req_s_per_gpu": 1.2833333333333334, + "raw_input_tokens": { + "distinct_n": 288, + "finite_n": 308, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 308 + }, + "raw_length_order_sha256": "c533dffff00fdea5f5492826549ca09fa62fe46ef98942928115816f92a38771", + "request_id_order_sha256": "2407501cda53bf9c761c448dc50ab4780b4a55958cc00ba1b19cd09de1cf73e1" + }, + "slo_pass_count": 308, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 308, + "finite_n": 308, + "max": 7.508626362165861, + "min": 3.4711365354262056, + "missing_n": 0, + "n": 308 + }, + "ttft_ms": { + "distinct_n": 308, + "finite_n": 308, + "max": 484.5911349984817, + "min": 35.676457977388054, + "missing_n": 0, + "n": 308 + } + } + ], + "feasibility_flip": false, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.48839477826210115, + "decode_B_mean": 5.379727685325265, + "decode_only_steps": 6971, + "decode_tokens": 39116, + "graph_mode_counts": { + "FULL": 6971, + "NONE": 285, + "PIECEWISE": 15 + }, + "graph_mode_shares": { + "FULL": 0.9587402007976894, + "NONE": 0.03919680924219502, + "PIECEWISE": 0.0020629899601155273 + }, + "kv_usage_max": 0.014172425223755059, + "kv_usage_mean": 0.005133820067639756, + "mixed_steps": 296, + "model_steps": 7271, + "padding_fraction": 0.0039644157567773385, + "preemptions": 0, + "prefill_only_steps": 4, + "prefill_tokens": 774412, + "steps": 7278, + "waiting_cv": 1.5096011764183093, + "waiting_max": 10.0, + "waiting_mean": 1.5337642690138908 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.016591107009, + "cell": "tp4_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 308, + "feasible": true, + "interval": { + "elapsed_s": 61.479843593, + "end_mono_ns": 207463713361534, + "end_wall_ns": 1783875890839788527, + "start_mono_ns": 207402233517941, + "start_wall_ns": 1783875829359945005 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 308, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "860d2ac188790a401e93d3f99b9f37d6d69f0dfda1af94c37bc0067278444632", + "arrival_s": { + "distinct_n": 307, + "finite_n": 308, + "max": 59.75439999999999, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 308 + }, + "count": 308, + "long_gt4096": 96, + "offered_req_s": 5.133333333333334, + "offered_req_s_per_gpu": 1.2833333333333334, + "raw_input_tokens": { + "distinct_n": 288, + "finite_n": 308, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 308 + }, + "raw_length_order_sha256": "c533dffff00fdea5f5492826549ca09fa62fe46ef98942928115816f92a38771", + "request_id_order_sha256": "2407501cda53bf9c761c448dc50ab4780b4a55958cc00ba1b19cd09de1cf73e1" + }, + "slo_pass_count": 308, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 308, + "finite_n": 308, + "max": 12.594227795259968, + "min": 3.8338481023720665, + "missing_n": 0, + "n": 308 + }, + "ttft_ms": { + "distinct_n": 308, + "finite_n": 308, + "max": 1612.671570997918, + "min": 38.93829099251889, + "missing_n": 0, + "n": 308 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 1.2833333333333334, + "request_count": 308 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 1.2833333333333334, + "request_count": 308 + } + }, + { + "accepted_feasible": true, + "accepted_pass_rate": 1.0, + "anchor": 0.017126304009, + "colocated": null, + "confirmations": [ + { + "anchor": 0.017126304009, + "cell": "tp4_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 317, + "feasible": true, + "interval": { + "elapsed_s": 60.494741705, + "end_mono_ns": 207730936466431, + "end_wall_ns": 1783876158062893709, + "start_mono_ns": 207670441724726, + "start_wall_ns": 1783876097568152371 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "layer1": { + "decode_B_cv": 0.6217052681226126, + "decode_B_mean": 3.3507282563462337, + "decode_only_steps": 11706, + "decode_tokens": 40259, + "graph_mode_counts": { + "FULL": 11723, + "NONE": 65, + "PIECEWISE": 227 + }, + "graph_mode_shares": { + "FULL": 0.9756970453599667, + "NONE": 0.005409904286308781, + "PIECEWISE": 0.01889305035372451 + }, + "kv_usage_max": 0.01413162659050926, + "kv_usage_mean": 0.003239002729363623, + "mixed_steps": 291, + "model_steps": 12015, + "padding_fraction": 0.14168915542228885, + "preemptions": 0, + "prefill_only_steps": 18, + "prefill_tokens": 2678, + "steps": 12048, + "waiting_cv": 5.080978847786386, + "waiting_max": 4.0, + "waiting_mean": 0.07265917602996255 + }, + "mns": 8, + "observed_count": 317, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "37e0795a537a347b17e01cbe56e581172f28f709c8a9fc527cd2e149ef427a11", + "arrival_s": { + "distinct_n": 316, + "finite_n": 317, + "max": 59.75439999999999, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 317 + }, + "count": 317, + "long_gt4096": 98, + "offered_req_s": 5.283333333333333, + "offered_req_s_per_gpu": 1.3208333333333333, + "raw_input_tokens": { + "distinct_n": 295, + "finite_n": 317, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 317 + }, + "raw_length_order_sha256": "4edec9417636579f3ebf9140d418b8072a89f80362751426a578737870179a9c", + "request_id_order_sha256": "93cd99ebf2a760cc440dce0cb7e51273686dcfbfb07650d27e9e16d2ff1d2eed" + }, + "slo_pass_count": 317, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 317, + "finite_n": 317, + "max": 7.324393826727499, + "min": 3.5252888424791338, + "missing_n": 0, + "n": 317 + }, + "ttft_ms": { + "distinct_n": 317, + "finite_n": 317, + "max": 458.84576899698004, + "min": 35.29350899043493, + "missing_n": 0, + "n": 317 + } + } + ], + "feasibility_flip": true, + "feasible_votes": [ + true, + true + ], + "layer1": { + "decode_B_cv": 0.6191035929724532, + "decode_B_mean": 3.3865242261103634, + "decode_only_steps": 11577, + "decode_tokens": 40259, + "graph_mode_counts": { + "FULL": 11594, + "NONE": 74, + "PIECEWISE": 220 + }, + "graph_mode_shares": { + "FULL": 0.9752691790040376, + "NONE": 0.006224764468371467, + "PIECEWISE": 0.01850605652759085 + }, + "kv_usage_max": 0.014126526761353508, + "kv_usage_mean": 0.003272512922970512, + "mixed_steps": 294, + "model_steps": 11888, + "padding_fraction": 0.09524192144416321, + "preemptions": 0, + "prefill_only_steps": 17, + "prefill_tokens": 27302, + "steps": 11921, + "waiting_cv": 4.70523208451703, + "waiting_max": 4.0, + "waiting_mean": 0.08310901749663527 + }, + "pass_rates": [ + 1.0, + 1.0 + ], + "primary": { + "anchor": 0.017126304009, + "cell": "tp4_mns8", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 317, + "feasible": true, + "interval": { + "elapsed_s": 60.482683353, + "end_mono_ns": 207663639929087, + "end_wall_ns": 1783876090766355953, + "start_mono_ns": 207603157245734, + "start_wall_ns": 1783876030283672912 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 317, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "37e0795a537a347b17e01cbe56e581172f28f709c8a9fc527cd2e149ef427a11", + "arrival_s": { + "distinct_n": 316, + "finite_n": 317, + "max": 59.75439999999999, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 317 + }, + "count": 317, + "long_gt4096": 98, + "offered_req_s": 5.283333333333333, + "offered_req_s_per_gpu": 1.3208333333333333, + "raw_input_tokens": { + "distinct_n": 295, + "finite_n": 317, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 317 + }, + "raw_length_order_sha256": "4edec9417636579f3ebf9140d418b8072a89f80362751426a578737870179a9c", + "request_id_order_sha256": "93cd99ebf2a760cc440dce0cb7e51273686dcfbfb07650d27e9e16d2ff1d2eed" + }, + "slo_pass_count": 317, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 317, + "finite_n": 317, + "max": 7.4181628267291755, + "min": 3.5225878738684533, + "missing_n": 0, + "n": 317 + }, + "ttft_ms": { + "distinct_n": 317, + "finite_n": 317, + "max": 437.27293200208806, + "min": 36.18798102252185, + "missing_n": 0, + "n": 317 + } + }, + "resolved": true, + "trial_count": 2, + "v20": { + "feasible": false, + "pass_rate": 0.8517350157728707, + "rate_per_gpu": 1.3208333333333333, + "request_count": 317 + }, + "v24": { + "feasible": true, + "pass_rate": 1.0, + "rate_per_gpu": 1.3208333333333333, + "request_count": 317 + } + }, + { + "accepted_feasible": false, + "accepted_pass_rate": 0.17833333333333334, + "anchor": 0.034252608017, + "colocated": null, + "confirmations": [], + "feasibility_flip": false, + "feasible_votes": [ + false + ], + "layer1": { + "decode_B_cv": 0.18954813856475664, + "decode_B_mean": 7.449155243863564, + "decode_only_steps": 2953, + "decode_tokens": 23368, + "graph_mode_counts": { + "FULL": 2958, + "NONE": 133, + "PIECEWISE": 46 + }, + "graph_mode_shares": { + "FULL": 0.9429391138029964, + "NONE": 0.04239719477207523, + "PIECEWISE": 0.014663691424928276 + }, + "kv_usage_max": 0.014697707626794454, + "kv_usage_mean": 0.006744695570071691, + "mixed_steps": 183, + "model_steps": 3137, + "padding_fraction": 0.002938639059635501, + "preemptions": 0, + "prefill_only_steps": 1, + "prefill_tokens": 235513, + "steps": 3139, + "waiting_cv": 0.7801609570717858, + "waiting_max": 42.0, + "waiting_mean": 15.969716289448519 + }, + "pass_rates": [ + 0.17833333333333334 + ], + "primary": { + "anchor": 0.034252608017, + "cell": "tp4_mns8", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 184, + "feasible": false, + "interval": { + "elapsed_s": 25.123306736, + "end_mono_ns": 207762064800655, + "end_wall_ns": 1783876189191227609, + "start_mono_ns": 207736941493919, + "start_wall_ns": 1783876164067921262 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 8, + "observed_count": 600, + "pass_rate": 0.17833333333333334, + "schema": 1, + "selection": { + "arrival_order_sha256": "50f1b34c344c451c6d7501290d0b53f147d0c0b6a4bc078b303d5c33576ebe2d", + "arrival_s": { + "distinct_n": 599, + "finite_n": 600, + "max": 59.90990000000002, + "min": 0.13000000000001818, + "missing_n": 0, + "n": 600 + }, + "count": 600, + "long_gt4096": 187, + "offered_req_s": 10.0, + "offered_req_s_per_gpu": 2.5, + "raw_input_tokens": { + "distinct_n": 546, + "finite_n": 600, + "max": 8157.0, + "min": 69.0, + "missing_n": 0, + "n": 600 + }, + "raw_length_order_sha256": "08eff8eecd74dc4646aa45945bd752432ee5eb795006755527dd32d7178c1140", + "request_id_order_sha256": "21c1f3b7d972b19c952a894863a70d1e85bd0e24a7e7b6b0fcc088360d0e01bd" + }, + "slo_pass_count": 107, + "study_sha256": "76e9d642ba9570917cfeeea848ed92f27904e87760eedabd48805b84e82eca26", + "tp": 4, + "tpot_ms": { + "distinct_n": 184, + "finite_n": 184, + "max": 10.008890590608472, + "min": 4.802999944897281, + "missing_n": 416, + "n": 600 + }, + "ttft_ms": { + "distinct_n": 184, + "finite_n": 184, + "max": 5891.477338009281, + "min": 41.87891198671423, + "missing_n": 416, + "n": 600 + } + }, + "resolved": true, + "trial_count": 1, + "v20": { + "feasible": false, + "pass_rate": 0.056666666666666664, + "rate_per_gpu": 2.5, + "request_count": 600 + }, + "v24": { + "feasible": false, + "pass_rate": 0.17833333333333334, + "rate_per_gpu": 2.5, + "request_count": 600 + } + } + ], + "boundary": "UP", + "bounded": true, + "cell_valid": { + "accounting_mode": "graceful-footer", + "cell": "tp4_mns8", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 58725, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp4_mns8/opprof/opprof-v1-dp0-pid2762505-1783875811578126796.jsonl" + }, + "censor": null, + "drift": 0.029220779220779036, + "f20": 1.2833333333333334, + "f24": 1.3208333333333333, + "material_frontier_moved": false, + "measurement_status": "SOLO_AUTHORITATIVE", + "mns": 8, + "observed_max_feasible_rate": 1.3208333333333333, + "tp": 4 + } + }, + "decision_blockers": { + "coverage": [], + "unbounded_or_unresolved_cells": { + "tp2_mns16": "RIGHT_CENSORED_HISTORY_EDGE", + "tp4_mns16": "NONMONOTONIC_SOLO_ANCHORS", + "tp4_mns32": "RIGHT_CENSORED_HISTORY_EDGE", + "tp4_mns64": "RIGHT_CENSORED_HISTORY_EDGE" + } + }, + "feasibility_flip_cells": [ + "tp1_mns8", + "tp1_mns16", + "tp1_mns32", + "tp1_mns64", + "tp2_mns8", + "tp2_mns16", + "tp2_mns32", + "tp2_mns64", + "tp4_mns8", + "tp4_mns16", + "tp4_mns32", + "tp4_mns64" + ], + "floor_buckets": { + "v20": { + "tp1_mns16": 715736, + "tp1_mns32": 695431, + "tp1_mns64": 695431, + "tp1_mns8": 639593, + "tp2_mns16": 692893, + "tp2_mns32": 1000000, + "tp2_mns64": 992385, + "tp2_mns8": 692893, + "tp4_mns16": 743654, + "tp4_mns32": 743654, + "tp4_mns64": 743654, + "tp4_mns8": 390862 + }, + "v20_tol": 3.283333333333333e-06, + "v24": { + "tp1_mns16": 731457, + "tp1_mns32": 731457, + "tp1_mns64": 731457, + "tp1_mns8": 731457, + "tp2_mns16": 705882, + "tp2_mns32": 1000000, + "tp2_mns64": 705882, + "tp2_mns8": 687979, + "tp4_mns32": 767263, + "tp4_mns64": 767263, + "tp4_mns8": 405370 + }, + "v24_tol": 3.258333333333333e-06 + }, + "gpu": { + "budget_stop": null, + "completed_primary_anchors": 37, + "confirmations": 29, + "controller_status": "complete", + "hard_cap": 6.0, + "new_h20_hours": 5.644544164273745, + "prior_colocated_h20_hours": 2.2911728494699863, + "solo_h20_hours": 3.3533713148037596 + }, + "ground_truth_sha256": "23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670", + "limitation": "Upgrade-path churn includes dash1->dash0 and resolved-default changes. Co-located W1-W3 values are indicative only; P3 composition is contextual, not a matched v0.20 Layer-1 baseline.", + "materially_drifted_cells": [ + "tp1_mns8", + "tp2_mns64" + ], + "p3_composition_reference": { + "limitation": "P3 P10 reference is TP1/MNS-default, steady arrival, sampling_u<=0.125, output<=256; it is contextual rather than a matched v0.20 composition baseline.", + "metrics": { + "decode_B_cv": 0.6605754665660243, + "decode_B_mean": 1.4177606177606177, + "decode_only_steps": 17941, + "decode_tokens": 25704, + "graph_mode_counts": { + "FULL": 17941, + "NONE": 167, + "PIECEWISE": 22 + }, + "graph_mode_shares": { + "FULL": 0.9895752895752896, + "NONE": 0.009211252068394925, + "PIECEWISE": 0.0012134583563154992 + }, + "kv_usage_max": 0.38664503919978377, + "kv_usage_mean": 0.04595307358602457, + "mixed_steps": 92, + "model_steps": 18130, + "padding_fraction": 0.0017834797849081603, + "preemptions": 0, + "prefill_only_steps": 97, + "prefill_tokens": 977841, + "steps": 18246, + "waiting_cv": 134.6439749858864, + "waiting_max": 1.0, + "waiting_mean": 5.515719801434087e-05 + }, + "stream_sha256": "51ad4be12178da91d2af484d0946a2274afd3bcbbee33f37940cfe0ff2ea7fa7" + }, + "run_stats": { + "accepted_anchor_trials": 66, + "colocated_wave_gpu_hours": { + "W1-tp1": 0.299392895169, + "W2-tp2": 0.9203650120894115, + "W3-tp4-trap": 0.9384967743025886 + }, + "confirmation_runs": 29, + "launch_echo": [ + "LAUNCH_ECHO utc=2026-07-12T14:47:24Z host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd model=Qwen3-30B-A3B trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl trace_sha=f539f38e cells=12 primary_anchors=25 waves=4 gpus=0-7 warmup=16_raw_gt4096 horizon=60s drain=derived_le120s est_wall=25-35min est_gpu=2.65_H20h confirm_reserve=0.35_H20h hard_cap=3.0_H20h", + "RESUME_ECHO utc=2026-07-12T14:50:30Z repair=phase6-owned-pgid-monitor prior_spend=0.1329181679_H20h projected_total_with_reserve=2.7829181679_H20h hard_cap=3.0_H20h restart=W1", + "WAVE_ECHO wave=W2-tp2 assignments=tp2_mns8:gpu0+1,tp2_mns16:gpu2+3,tp2_mns32:gpu4+5,tp2_mns64:gpu6+7 spent_h20h=0.432311 wave_est_h20h=0.650 remaining_projection_h20h=2.250 cap_h20h=3.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json workload=chat_w20260311_1000.jsonl", + "WAVE_ECHO wave=W3-tp4-trap assignments=tp4_mns8:gpu0+1+2+3,tp4_mns16:gpu4+5+6+7 spent_h20h=1.352676 wave_est_h20h=0.850 remaining_projection_h20h=1.600 cap_h20h=3.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json workload=chat_w20260311_1000.jsonl", + "A-P6-2_LAUNCH_ECHO utc=2026-07-12T16:14:18Z host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.2911728495 new_est_h20h=3.440 cumulative_est_h20h=5.7311728495 cap_h20h=6.0 est_wall=45-70min ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp4_mns32 tp=4 mns=32 gpus=0-3 mandatory=0.033717411016 spent_h20h=2.291173 cell_est_h20h=0.480 remaining_projection_h20h=3.440 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp4_mns64 tp=4 mns=64 gpus=0-3 mandatory=0.033717411016 spent_h20h=2.805712 cell_est_h20h=0.480 remaining_projection_h20h=2.960 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "A-P6-2_RESUME_ECHO utc=2026-07-12T16:31:13Z repair=transient-driver-memory-poll completed_cells=2 spent_h20h=3.315063744253854 restart=tp2_mns32 cap_h20h=6.0", + "SOLO_WAVE_ECHO cell=tp2_mns32 tp=2 mns=32 gpus=0-1 mandatory=0.75390625,0.75 spent_h20h=3.315064 cell_est_h20h=0.220 remaining_projection_h20h=2.480 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp2_mns64 tp=2 mns=64 gpus=0-1 mandatory=0.75,0.5 spent_h20h=3.490967 cell_est_h20h=0.220 remaining_projection_h20h=2.260 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp4_mns16 tp=4 mns=16 gpus=0-3 mandatory=0.033717411016,0.033182214016,0.034252608017 spent_h20h=3.664183 cell_est_h20h=0.480 remaining_projection_h20h=2.040 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp2_mns8 tp=2 mns=8 gpus=0-1 mandatory=0.49609375,0.4921875 spent_h20h=4.217542 cell_est_h20h=0.220 remaining_projection_h20h=1.560 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp2_mns16 tp=2 mns=16 gpus=0-1 mandatory=0.49609375,0.4921875 spent_h20h=4.392992 cell_est_h20h=0.220 remaining_projection_h20h=1.340 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp4_mns8 tp=4 mns=8 gpus=0-3 mandatory=0.016591107009,0.016055910008 spent_h20h=4.623573 cell_est_h20h=0.480 remaining_projection_h20h=1.120 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp1_mns8 tp=1 mns=8 gpus=0-0 mandatory=0.2265625,0.21875 spent_h20h=5.120182 cell_est_h20h=0.110 remaining_projection_h20h=0.640 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp1_mns16 tp=1 mns=16 gpus=0-0 mandatory=0.24609375,0.25 spent_h20h=5.319979 cell_est_h20h=0.110 remaining_projection_h20h=0.530 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp1_mns32 tp=1 mns=32 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.404469 cell_est_h20h=0.110 remaining_projection_h20h=0.420 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "SOLO_WAVE_ECHO cell=tp1_mns64 tp=1 mns=64 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.523280 cell_est_h20h=0.110 remaining_projection_h20h=0.310 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl", + "A-P6-2_RESUME_ECHO utc=2026-07-12T17:36:18Z repair=prelaunch-driver-memory-poll spent_h20h=5.523279991083946 restart=tp1_mns64 cap_h20h=6.0", + "SOLO_WAVE_ECHO cell=tp1_mns64 tp=1 mns=64 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.523280 cell_est_h20h=0.110 remaining_projection_h20h=0.310 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl" + ], + "measured_cells": 12, + "primary_anchor_runs": 37, + "solo_cell_gpu_hours": { + "tp1_mns16": 0.08449018637339274, + "tp1_mns32": 0.11881072620550791, + "tp1_mns64": 0.12126417318979898, + "tp1_mns8": 0.19979675398932564, + "tp2_mns16": 0.23058102700445388, + "tp2_mns32": 0.17590342534912956, + "tp2_mns64": 0.17321541958385045, + "tp2_mns8": 0.17544995917214287, + "tp4_mns16": 0.5533596136834886, + "tp4_mns32": 0.5145386989911397, + "tp4_mns64": 0.509352195792728, + "tp4_mns8": 0.49660913546880087 + }, + "surface_cells": 12, + "warmup_runs": 12 + }, + "sanity": { + "coverage": { + "solo_measured_cells_12": true, + "solo_primary_anchors_at_least_25": true + }, + "invariants": { + "cell_validity": true, + "client_invariants": true, + "gpu_below_6": true, + "rates_nonnegative": true, + "selection_counts_match": true, + "surface_not_identical": true, + "surface_rows_12": true + }, + "numeric": { + "drift": { + "distinct_n": 9, + "finite_n": 11, + "max": 0.13492063492063489, + "min": -0.2941176470588236, + "missing_n": 1, + "n": 12 + }, + "layer1_steps": { + "distinct_n": 37, + "finite_n": 37, + "max": 12103.0, + "min": 343.0, + "missing_n": 0, + "n": 37 + }, + "primary_pass_rate": { + "distinct_n": 17, + "finite_n": 37, + "max": 1.0, + "min": 0.030716723549488054, + "missing_n": 0, + "n": 37 + }, + "selected_count": { + "distinct_n": 18, + "finite_n": 37, + "max": 600.0, + "min": 121.0, + "missing_n": 0, + "n": 37 + }, + "v20_score": { + "distinct_n": 8, + "finite_n": 12, + "max": 3.283333333333333, + "min": 1.2833333333333334, + "missing_n": 0, + "n": 12 + }, + "v24_score": { + "distinct_n": 6, + "finite_n": 11, + "max": 3.2583333333333333, + "min": 1.3208333333333333, + "missing_n": 0, + "n": 11 + } + }, + "red_flags": [] + }, + "schema": 1, + "solo_vs_colocated": [ + { + "accepted_feasible": true, + "anchor": 0.21875, + "cell": "tp1_mns8", + "primary_feasible": true, + "primary_pass_rate": 0.9917355371900827, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 0.9917355371900827, + "trial_pass_rates": [ + 0.9917355371900827 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.2265625, + "cell": "tp1_mns8", + "primary_feasible": false, + "primary_pass_rate": 0.20634920634920634, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.7857142857142858, + "solo_pass_rate": 0.9920634920634921, + "trial_pass_rates": [ + 0.20634920634920634 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.24609375, + "cell": "tp1_mns16", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.25, + "cell": "tp1_mns16", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.2421875, + "cell": "tp1_mns32", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.24609375, + "cell": "tp1_mns32", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.2421875, + "cell": "tp1_mns64", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.24609375, + "cell": "tp1_mns64", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.4921875, + "cell": "tp2_mns8", + "primary_feasible": false, + "primary_pass_rate": 0.6914498141263941, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.30855018587360594, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.6914498141263941 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.49609375, + "cell": "tp2_mns8", + "primary_feasible": false, + "primary_pass_rate": 0.09157509157509157, + "solo_feasible": false, + "solo_minus_colocated_primary_pass_rate": 0.48901098901098894, + "solo_pass_rate": 0.5805860805860805, + "trial_pass_rates": [ + 0.09157509157509157 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.4921875, + "cell": "tp2_mns16", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.49609375, + "cell": "tp2_mns16", + "primary_feasible": false, + "primary_pass_rate": 0.08791208791208792, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.9120879120879121, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.08791208791208792 + ] + }, + { + "accepted_feasible": null, + "anchor": 0.75, + "cell": "tp2_mns32", + "primary_feasible": false, + "primary_pass_rate": 0.4117647058823529, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.5882352941176471, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.4117647058823529, + 1.0 + ] + }, + { + "accepted_feasible": null, + "anchor": 0.75390625, + "cell": "tp2_mns32", + "primary_feasible": false, + "primary_pass_rate": 0.027918781725888325, + "solo_feasible": false, + "solo_minus_colocated_primary_pass_rate": 0.5469543147208122, + "solo_pass_rate": 0.5748730964467006, + "trial_pass_rates": [ + 0.027918781725888325, + 0.9568527918781726 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.5, + "cell": "tp2_mns64", + "primary_feasible": false, + "primary_pass_rate": 0.4601449275362319, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.5398550724637681, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.4601449275362319 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.75, + "cell": "tp2_mns64", + "primary_feasible": false, + "primary_pass_rate": 0.14066496163682865, + "solo_feasible": false, + "solo_minus_colocated_primary_pass_rate": 0.4207161125319694, + "solo_pass_rate": 0.561381074168798, + "trial_pass_rates": [ + 0.14066496163682865 + ] + }, + { + "accepted_feasible": true, + "anchor": 0.016055910008, + "cell": "tp4_mns8", + "primary_feasible": true, + "primary_pass_rate": 1.0, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.0, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 1.0 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.016591107009, + "cell": "tp4_mns8", + "primary_feasible": false, + "primary_pass_rate": 0.07142857142857142, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.9285714285714286, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.07142857142857142 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.033182214016, + "cell": "tp4_mns16", + "primary_feasible": false, + "primary_pass_rate": 0.6086956521739131, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.3913043478260869, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.6086956521739131 + ] + }, + { + "accepted_feasible": null, + "anchor": 0.033717411016, + "cell": "tp4_mns16", + "primary_feasible": false, + "primary_pass_rate": 0.03583617747440273, + "solo_feasible": false, + "solo_minus_colocated_primary_pass_rate": 0.20221843003412968, + "solo_pass_rate": 0.2380546075085324, + "trial_pass_rates": [ + 0.03583617747440273, + 1.0 + ] + }, + { + "accepted_feasible": false, + "anchor": 0.034252608017, + "cell": "tp4_mns16", + "primary_feasible": false, + "primary_pass_rate": 0.8466666666666667, + "solo_feasible": true, + "solo_minus_colocated_primary_pass_rate": 0.15333333333333332, + "solo_pass_rate": 1.0, + "trial_pass_rates": [ + 0.8466666666666667 + ] + } + ], + "status": "VALID", + "verdicts": { + "argmax": "INCONCLUSIVE", + "full_surface_bounded": false, + "ranking": "INCONCLUSIVE", + "tau_b": null, + "top_pair_reversals_gt5pct": [], + "trap": "INCONCLUSIVE" + }, + "w1_readjudication": { + "amendment": "A-P6-1", + "cells": [ + { + "anchors": [ + { + "anchor": 0.24609375, + "covered": true, + "interval_end_mono_ns": 199601637648726, + "interval_start_mono_ns": 199540309473748, + "layer1_records": 4706, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.24609375/result.json" + }, + { + "anchor": 0.25, + "covered": true, + "interval_end_mono_ns": 199668842271892, + "interval_start_mono_ns": 199607371060368, + "layer1_records": 3883, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.25/result.json" + } + ], + "cell": "tp1_mns16", + "checkpoint_after_anchor_s": 1.000327977, + "checkpoint_stream_delta_s": 0.001439359, + "counters": { + "dropped_records": 0, + "encoded_records": 9212, + "last_step_index": 9211, + "written_records": 9212 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 9212, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl" + }, + { + "anchors": [ + { + "anchor": 0.2421875, + "covered": true, + "interval_end_mono_ns": 199601604247251, + "interval_start_mono_ns": 199540309397799, + "layer1_records": 4844, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/anchor-0.2421875/result.json" + }, + { + "anchor": 0.24609375, + "covered": true, + "interval_end_mono_ns": 199668653603418, + "interval_start_mono_ns": 199607365362010, + "layer1_records": 4144, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/anchor-0.24609375/result.json" + } + ], + "cell": "tp1_mns32", + "checkpoint_after_anchor_s": 1.000711369, + "checkpoint_stream_delta_s": 0.001155708, + "counters": { + "dropped_records": 0, + "encoded_records": 9633, + "last_step_index": 9632, + "written_records": 9633 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 9633, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl" + }, + { + "anchors": [ + { + "anchor": 0.2421875, + "covered": true, + "interval_end_mono_ns": 199601593594258, + "interval_start_mono_ns": 199540314883238, + "layer1_records": 4855, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/anchor-0.2421875/result.json" + }, + { + "anchor": 0.24609375, + "covered": true, + "interval_end_mono_ns": 199668751241940, + "interval_start_mono_ns": 199607467392165, + "layer1_records": 4165, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/anchor-0.24609375/result.json" + } + ], + "cell": "tp1_mns64", + "checkpoint_after_anchor_s": 2.013733755, + "checkpoint_stream_delta_s": 1.014815835, + "counters": { + "dropped_records": 0, + "encoded_records": 9654, + "last_step_index": 9653, + "written_records": 9654 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 9654, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2638898-1783867899798067669.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2638898-1783867899798067669.jsonl" + }, + { + "anchors": [ + { + "anchor": 0.21875, + "covered": true, + "interval_end_mono_ns": 199668659416799, + "interval_start_mono_ns": 199607365130468, + "layer1_records": 5756, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/anchor-0.21875/result.json" + }, + { + "anchor": 0.2265625, + "covered": true, + "interval_end_mono_ns": 199568971776020, + "interval_start_mono_ns": 199540309654499, + "layer1_records": 1663, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/anchor-0.2265625/result.json" + } + ], + "cell": "tp1_mns8", + "checkpoint_after_anchor_s": 1.000563608, + "checkpoint_stream_delta_s": 0.12747079, + "counters": { + "dropped_records": 0, + "encoded_records": 7683, + "last_step_index": 7682, + "written_records": 7683 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 7683, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/opprof/opprof-v1-dp0-pid2638884-1783867942846550713.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/opprof/opprof-v1-dp0-pid2638884-1783867942846550713.jsonl" + } + ], + "mode": "checkpoint-sidecar", + "passed": true, + "sanity": { + "anchor_count": 8, + "cell_count": 4, + "records_distinct": 4, + "records_max": 9654, + "records_min": 7683 + }, + "schema": 1 + } +} diff --git a/runs/opprof-phase6/phase6/selection-preflight.json b/runs/opprof-phase6/phase6/selection-preflight.json new file mode 100644 index 0000000..6898645 --- /dev/null +++ b/runs/opprof-phase6/phase6/selection-preflight.json @@ -0,0 +1,17 @@ +{ + "invariants": { + "counts_match": true, + "observations_92": true + }, + "mismatches": [], + "observations": 92, + "request_counts": { + "distinct_n": 34, + "finite_n": 92, + "max": 600.0, + "min": 66.0, + "missing_n": 0, + "n": 92 + }, + "schema": 1 +} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375.log b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375.log new file mode 100644 index 0000000..2acdd5d --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375.log @@ -0,0 +1 @@ +{"cell": "tp1_mns16", "anchor": 0.24609375, "kind": "anchor", "pass_rate": 1.0, "feasible": true} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375/result.json b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375/result.json new file mode 100644 index 0000000..9470e0f --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375/result.json @@ -0,0 +1,74 @@ +{ + "anchor": 0.24609375, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 141, + "feasible": true, + "interval": { + "elapsed_s": 61.285616071, + "end_mono_ns": 208628026400377, + "end_wall_ns": 1783877055152827630, + "start_mono_ns": 208566740784306, + "start_wall_ns": 1783876993867211598 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 141, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "7c8f4ce3fc2db40d6329e8ead59378f564608ad6df09bc95854baef2dd0703d4", + "arrival_s": { + "distinct_n": 141, + "finite_n": 141, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 141 + }, + "count": 141, + "long_gt4096": 50, + "offered_req_s": 2.35, + "offered_req_s_per_gpu": 2.35, + "raw_input_tokens": { + "distinct_n": 133, + "finite_n": 141, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 141 + }, + "raw_length_order_sha256": "a30d27aea9630ed3623c73237867fbd3a6b7eec9bedec0f1c390610fd16ea5ba", + "request_id_order_sha256": "7e48c6bfc00eeadd6011111170c31123fb117c9fa75c18ccc8d8d505fd7fcff3" + }, + "slo_pass_count": 141, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 39.26863328358488, + "min": 4.784946039540869, + "missing_n": 0, + "n": 141 + }, + "ttft_ms": { + "distinct_n": 141, + "finite_n": 141, + "max": 1786.641047016019, + "min": 38.60557498410344, + "missing_n": 0, + "n": 141 + } +} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25.log b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25.log new file mode 100644 index 0000000..79aba70 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25.log @@ -0,0 +1 @@ +{"cell": "tp1_mns16", "anchor": 0.25, "kind": "anchor", "pass_rate": 1.0, "feasible": true} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25/result.json b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25/result.json new file mode 100644 index 0000000..43dd400 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25/result.json @@ -0,0 +1,74 @@ +{ + "anchor": 0.25, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.484334594, + "end_mono_ns": 208696038541120, + "end_wall_ns": 1783877123164968015, + "start_mono_ns": 208634554206526, + "start_wall_ns": 1783877061680633802 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 44.45809762976243, + "min": 4.488573606239676, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1778.0322160106152, + "min": 47.90777899324894, + "missing_n": 0, + "n": 143 + } +} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5.log b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5.log new file mode 100644 index 0000000..d3500d7 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5.log @@ -0,0 +1 @@ +{"cell": "tp1_mns16", "anchor": 0.5, "kind": "anchor", "pass_rate": 0.15579710144927536, "feasible": false} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5/result.json b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5/result.json new file mode 100644 index 0000000..693e85e --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5/result.json @@ -0,0 +1,74 @@ +{ + "anchor": 0.5, + "cell": "tp1_mns16", + "early_stop_reason": "slo_pass_rate_unrecoverable", + "early_stopped": true, + "exact_output_count": 92, + "feasible": false, + "interval": { + "elapsed_s": 26.818138737, + "end_mono_ns": 208796704305524, + "end_wall_ns": 1783877223830732816, + "start_mono_ns": 208769886166787, + "start_wall_ns": 1783877197012593929 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 276, + "pass_rate": 0.15579710144927536, + "schema": 1, + "selection": { + "arrival_order_sha256": "aeea4a8e54cc7de279804b8c353f748a480aedcbb21e249a90a60267daee8dff", + "arrival_s": { + "distinct_n": 276, + "finite_n": 276, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 276 + }, + "count": 276, + "long_gt4096": 99, + "offered_req_s": 4.6, + "offered_req_s_per_gpu": 4.6, + "raw_input_tokens": { + "distinct_n": 256, + "finite_n": 276, + "max": 8149.0, + "min": 71.0, + "missing_n": 0, + "n": 276 + }, + "raw_length_order_sha256": "983df528039301bd41a6684fecea75cdcf8822d214094d6d35def7fc7740c2ba", + "request_id_order_sha256": "3160017dbd8249f711eb115f1e72ddbb37eab3a3333a48762cc3104e94bd2f15" + }, + "slo_pass_count": 43, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 92, + "finite_n": 92, + "max": 40.052486527445325, + "min": 11.32848871666498, + "missing_n": 184, + "n": 276 + }, + "ttft_ms": { + "distinct_n": 92, + "finite_n": 92, + "max": 6538.087566004833, + "min": 66.57706701662391, + "missing_n": 184, + "n": 276 + } +} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/cell-valid.json b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/cell-valid.json new file mode 100644 index 0000000..202e48a --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/cell-valid.json @@ -0,0 +1,22 @@ +{ + "accounting_mode": "graceful-footer", + "cell": "tp1_mns16", + "invariants": { + "anchor_intervals_present": true, + "compile_capture_pre_ready": true, + "encoded_balanced": true, + "footer_sidecar_agrees": true, + "last_step_matches": true, + "layer1_contiguous": true, + "layer1_zero_drops": true, + "one_footer_last": true, + "one_ready_marker": true, + "sidecar_final": true, + "warmup_exact_16": true, + "warmup_long": true, + "written_matches_records": true + }, + "layer1_records": 14174, + "post_ready_capture_events": [], + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2781473-1783876972093434233.jsonl" +} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/commands.log b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/commands.log new file mode 100644 index 0000000..0605870 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/commands.log @@ -0,0 +1,6 @@ +SERVER taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/vllm serve /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B --host 127.0.0.1 --port 8709 --served-model-name qwen3-30b-a3b-community --max-num-batched-tokens 8192 --max-num-seqs 16 --tensor-parallel-size 1 --shutdown-timeout 120 +CLIENT taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py warmup --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.24609375 --tp 1 --mns 16 --base-url http://127.0.0.1:8709 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/warmup +CLIENT taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.24609375 --tp 1 --mns 16 --base-url http://127.0.0.1:8709 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.24609375 +CLIENT taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.25 --tp 1 --mns 16 --base-url http://127.0.0.1:8709 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.25 +CLIENT taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.25 --tp 1 --mns 16 --base-url http://127.0.0.1:8709 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25 +CLIENT taskset -c 0-19 /tmp/wjh-opprof-phase2-dash0-20260711/.venv/bin/python /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_client.py run-anchor --study /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/study-primary.json --cell tp1_mns16 --anchor 0.5 --tp 1 --mns 16 --base-url http://127.0.0.1:8709 --result-dir /home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/solo-authoritative/cells/tp1_mns16/anchor-0.5 diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25.log b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25.log new file mode 100644 index 0000000..79aba70 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25.log @@ -0,0 +1 @@ +{"cell": "tp1_mns16", "anchor": 0.25, "kind": "anchor", "pass_rate": 1.0, "feasible": true} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25/result.json b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25/result.json new file mode 100644 index 0000000..3887749 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/confirm-1-anchor-0.25/result.json @@ -0,0 +1,74 @@ +{ + "anchor": 0.25, + "cell": "tp1_mns16", + "early_stop_reason": "", + "early_stopped": false, + "exact_output_count": 143, + "feasible": true, + "interval": { + "elapsed_s": 61.468118415, + "end_mono_ns": 208763972089008, + "end_wall_ns": 1783877191098516077, + "start_mono_ns": 208702503970593, + "start_wall_ns": 1783877129630397769 + }, + "invariants": { + "arrival_nondecreasing": true, + "exact_output_or_failed": true, + "outcomes_cover_selected": true, + "raw_lengths_present": true, + "selected_nonempty": true, + "warmup_16": true, + "warmup_exact_16": true, + "warmup_long": true + }, + "kind": "anchor", + "mns": 16, + "observed_count": 143, + "pass_rate": 1.0, + "schema": 1, + "selection": { + "arrival_order_sha256": "932babe37b75c53f4a0eee029df66fd3e8acd3972d925f4a68714faa827b9366", + "arrival_s": { + "distinct_n": 143, + "finite_n": 143, + "max": 59.78950000000005, + "min": 0.008300000000008368, + "missing_n": 0, + "n": 143 + }, + "count": 143, + "long_gt4096": 51, + "offered_req_s": 2.3833333333333333, + "offered_req_s_per_gpu": 2.3833333333333333, + "raw_input_tokens": { + "distinct_n": 135, + "finite_n": 143, + "max": 8149.0, + "min": 72.0, + "missing_n": 0, + "n": 143 + }, + "raw_length_order_sha256": "eac9a2d39e762892f716fde086ada79aa87161d04819c24bbeaabbf9e8839af0", + "request_id_order_sha256": "26629995f38f2c013d5aa5e4b9d7344311ab1f951c9d83e68212c67ca8076fda" + }, + "slo_pass_count": 143, + "study_sha256": "9474f0d0b53579f1db852ca68abfb0b96ba43ae4e17738118bf8e3209eb09ece", + "tp": 1, + "tpot_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 44.40597959039696, + "min": 4.490311023564716, + "missing_n": 0, + "n": 143 + }, + "ttft_ms": { + "distinct_n": 143, + "finite_n": 143, + "max": 1776.1785559996497, + "min": 49.54166599782184, + "missing_n": 0, + "n": 143 + } +} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2781473-1783876972093434233.jsonl.footer.json b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2781473-1783876972093434233.jsonl.footer.json new file mode 100644 index 0000000..03227f6 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2781473-1783876972093434233.jsonl.footer.json @@ -0,0 +1 @@ +{"schema":1,"record_type":"footer_checkpoint","stream":"opprof-v1-dp0-pid2781473-1783876972093434233.jsonl","encoded_records":14174,"written_records":14174,"dropped_records":0,"last_step_index":14173,"checkpoint_wall_ns":1783877224563096809,"flush_interval_seconds":1.0,"final":true} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/server.log b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/server.log new file mode 100644 index 0000000..edc4f30 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/cells/tp1_mns16/server.log @@ -0,0 +1,686 @@ +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:339] +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:339] █ █ █▄ ▄█ +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:339] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.24.1.dev3+g668cfb7e2 +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:339] █▄█▀ █ █ █ █ model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:339] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀ +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:339] +(APIServer pid=2781293) INFO 07-12 17:22:11 [api_utils.py:273] non-default args: {'model_tag': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'host': '127.0.0.1', 'port': 8709, 'model': '/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', 'served_model_name': ['qwen3-30b-a3b-community'], 'max_num_batched_tokens': 8192, 'max_num_seqs': 16, 'shutdown_timeout': 120} +(APIServer pid=2781293) INFO 07-12 17:22:11 [model.py:598] Resolved architecture: Qwen3MoeForCausalLM +(APIServer pid=2781293) INFO 07-12 17:22:11 [model.py:1725] Using max model len 40960 +(APIServer pid=2781293) INFO 07-12 17:22:11 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192. +(APIServer pid=2781293) INFO 07-12 17:22:11 [vllm.py:1006] Asynchronous scheduling is enabled. +(APIServer pid=2781293) INFO 07-12 17:22:11 [kernel.py:276] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']) +(EngineCore pid=2781473) INFO 07-12 17:22:21 [core.py:114] Initializing a V1 LLM engine (v0.24.1.dev3+g668cfb7e2) with config: model='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', speculative_config=None, tokenizer='/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_verbose=False), seed=0, served_model_name=qwen3-30b-a3b-community, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': , 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2781473) INFO 07-12 17:22:23 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:35361 backend=nccl +(EngineCore pid=2781473) INFO 07-12 17:22:23 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2781473) INFO 07-12 17:22:24 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2781473) INFO 07-12 17:22:24 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2781473) INFO 07-12 17:22:25 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2781473) INFO 07-12 17:22:25 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2781473) INFO 07-12 17:22:25 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2781473) INFO 07-12 17:22:25 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.56 GiB. +(EngineCore pid=2781473) INFO 07-12 17:22:25 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2781473) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2786356) INFO 07-12 17:27:30 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:56647 backend=nccl +(EngineCore pid=2786356) INFO 07-12 17:27:30 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2786356) INFO 07-12 17:27:31 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2786356) INFO 07-12 17:27:31 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2786356) INFO 07-12 17:27:31 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2786356) INFO 07-12 17:27:31 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2786356) INFO 07-12 17:27:31 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2786356) INFO 07-12 17:27:32 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.53 GiB. +(EngineCore pid=2786356) INFO 07-12 17:27:32 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2786356) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2795014) INFO 07-12 17:36:40 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:53983 backend=nccl +(EngineCore pid=2795014) INFO 07-12 17:36:40 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2795014) INFO 07-12 17:36:41 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2795014) INFO 07-12 17:36:41 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2795014) INFO 07-12 17:36:41 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2795014) INFO 07-12 17:36:41 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2795014) INFO 07-12 17:36:41 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2795014) INFO 07-12 17:36:41 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.62 GiB. +(EngineCore pid=2795014) INFO 07-12 17:36:41 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2795014) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2770271) INFO 07-12 17:10:29 [parallel_state.py:1588] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.27.132.244:41283 backend=nccl +(EngineCore pid=2770271) INFO 07-12 17:10:29 [parallel_state.py:1923] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(EngineCore pid=2770271) INFO 07-12 17:10:30 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(EngineCore pid=2770271) INFO 07-12 17:10:30 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(EngineCore pid=2770271) INFO 07-12 17:10:31 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(EngineCore pid=2770271) INFO 07-12 17:10:31 [flash_attn.py:670] Using FlashAttention version 3 +(EngineCore pid=2770271) INFO 07-12 17:10:31 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(EngineCore pid=2770271) INFO 07-12 17:10:31 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1288.54 GiB. +(EngineCore pid=2770271) INFO 07-12 17:10:31 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(EngineCore pid=2770271) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2755628) WARNING 07-12 16:55:50 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2755628) INFO 07-12 16:55:50 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2755779) INFO 07-12 16:56:00 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41575 backend=nccl +(Worker pid=2755780) INFO 07-12 16:56:00 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41575 backend=nccl +(Worker pid=2755779) INFO 07-12 16:56:01 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2755779) INFO 07-12 16:56:02 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2755779) INFO 07-12 16:56:03 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2755779) INFO 07-12 16:56:03 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2755779) INFO 07-12 16:56:03 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2755779) INFO 07-12 16:56:03 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.52 GiB. +(Worker_TP0 pid=2755779) INFO 07-12 16:56:04 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2755779) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2730662) WARNING 07-12 16:31:35 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2730662) INFO 07-12 16:31:35 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2730817) INFO 07-12 16:31:45 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:54663 backend=nccl +(Worker pid=2730818) INFO 07-12 16:31:45 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:54663 backend=nccl +(Worker pid=2730817) INFO 07-12 16:31:46 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2730817) INFO 07-12 16:31:47 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2730817) INFO 07-12 16:31:47 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2730817) INFO 07-12 16:31:47 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2730817) INFO 07-12 16:31:48 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2730817) INFO 07-12 16:31:48 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2730817) INFO 07-12 16:31:49 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.57 GiB. +(Worker_TP0 pid=2730817) INFO 07-12 16:31:49 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2730817) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2736153) WARNING 07-12 16:36:54 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2736153) INFO 07-12 16:36:54 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2736308) INFO 07-12 16:37:04 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:51703 backend=nccl +(Worker pid=2736309) INFO 07-12 16:37:04 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:51703 backend=nccl +(Worker pid=2736308) INFO 07-12 16:37:05 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2736308) INFO 07-12 16:37:06 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2736308) INFO 07-12 16:37:07 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2736308) INFO 07-12 16:37:07 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2736308) INFO 07-12 16:37:07 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2736308) INFO 07-12 16:37:07 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.55 GiB. +(Worker_TP0 pid=2736308) INFO 07-12 16:37:08 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2736308) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2750298) WARNING 07-12 16:50:31 [multiproc_executor.py:1063] Reducing Torch parallelism from 40 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2750298) INFO 07-12 16:50:31 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=2, local_world_size=2 +(Worker pid=2750550) INFO 07-12 16:50:41 [parallel_state.py:1588] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:45665 backend=nccl +(Worker pid=2750551) INFO 07-12 16:50:41 [parallel_state.py:1588] world_size=2 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:45665 backend=nccl +(Worker pid=2750550) INFO 07-12 16:50:42 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2750550) INFO 07-12 16:50:43 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2750550) INFO 07-12 16:50:44 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2750550) INFO 07-12 16:50:44 [parallel_state.py:1923] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2750550) INFO 07-12 16:50:44 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2750550) INFO 07-12 16:50:44 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1286.53 GiB. +(Worker_TP0 pid=2750550) INFO 07-12 16:50:45 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2750550) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 32, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2741574) WARNING 07-12 16:42:10 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2741574) INFO 07-12 16:42:10 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4 +(Worker pid=2741794) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl +(Worker pid=2741795) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl +(Worker pid=2741796) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl +(Worker pid=2741797) INFO 07-12 16:42:21 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:44217 backend=nccl +(Worker pid=2741794) INFO 07-12 16:42:22 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2741794) INFO 07-12 16:42:24 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2741794) INFO 07-12 16:42:25 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2741794) INFO 07-12 16:42:25 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2741794) INFO 07-12 16:42:25 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2741794) INFO 07-12 16:42:25 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.26 GiB. +(Worker_TP0 pid=2741794) INFO 07-12 16:42:26 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2741794) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 64, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2712385) WARNING 07-12 16:14:49 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2712385) INFO 07-12 16:14:49 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4 +(Worker pid=2712610) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl +(Worker pid=2712613) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl +(Worker pid=2712612) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl +(Worker pid=2712611) INFO 07-12 16:15:00 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:41619 backend=nccl +(Worker pid=2712610) INFO 07-12 16:15:01 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2712610) INFO 07-12 16:15:03 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2712610) INFO 07-12 16:15:04 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2712610) INFO 07-12 16:15:04 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2712610) INFO 07-12 16:15:04 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2712610) INFO 07-12 16:15:04 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.29 GiB. +(Worker_TP0 pid=2712610) INFO 07-12 16:15:05 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2712610) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 128, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2721119) WARNING 07-12 16:22:36 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2721119) INFO 07-12 16:22:36 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4 +(Worker pid=2721339) INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl +(Worker pid=2721340) INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl +(Worker pid=2721341) (Worker pid=2721342) INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl +INFO 07-12 16:22:47 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:52671 backend=nccl +(Worker pid=2721339) INFO 07-12 16:22:48 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2721339) INFO 07-12 16:22:50 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2721339) INFO 07-12 16:22:50 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2721339) INFO 07-12 16:22:50 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2721339) INFO 07-12 16:22:51 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.21 GiB. +(Worker_TP0 pid=2721339) INFO 07-12 16:22:51 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2721339) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [512, 8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': , 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 16, 'dynamic_shapes_config': {'type': , 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, moe_backend='auto', linear_backend='auto') +(EngineCore pid=2762505) WARNING 07-12 17:02:49 [multiproc_executor.py:1063] Reducing Torch parallelism from 80 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. +(EngineCore pid=2762505) INFO 07-12 17:02:49 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=127.0.0.1, mq_connect_ip=172.27.132.244 (local), world_size=4, local_world_size=4 +(Worker pid=2762723) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=0 local_rank=0 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl +(Worker pid=2762726) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=3 local_rank=3 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl +(Worker pid=2762724) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=1 local_rank=1 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl +(Worker pid=2762725) INFO 07-12 17:03:01 [parallel_state.py:1588] world_size=4 rank=2 local_rank=2 distributed_init_method=tcp://127.0.0.1:59219 backend=nccl +(Worker pid=2762723) INFO 07-12 17:03:02 [pynccl.py:113] vLLM is using nccl==2.28.9 +(Worker pid=2762723) INFO 07-12 17:03:04 [cuda_communicator.py:245] Using ['CUSTOM', 'SYMM_MEM', 'PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2762723) INFO 07-12 17:03:04 [cuda_communicator.py:245] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'CUSTOM', 'SYMM_MEM', 'PYNCCL']. +(Worker pid=2762723) INFO 07-12 17:03:04 [parallel_state.py:1923] rank 0 in world size 4 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A +(Worker pid=2762723) INFO 07-12 17:03:05 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling. +(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [gpu_model_runner.py:5164] Starting to load model /home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B... +(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [cuda.py:480] Using FLASH_ATTN attention backend out of potential backends: ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION']. +(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [flash_attn.py:670] Using FlashAttention version 3 +(Worker_TP0 pid=2762723) INFO 07-12 17:03:05 [unquantized.py:247] Using TRITON Unquantized MoE backend out of potential backends: ['TRITON', 'BATCHED_TRITON', 'FlashInfer TRTLLM', 'FlashInfer CUTLASS']. +(Worker_TP0 pid=2762723) INFO 07-12 17:03:06 [weight_utils.py:849] Filesystem type for checkpoints: FUSE.ALIYUN-ALINAS-EFC. Checkpoint size: 56.87 GiB. Available RAM: 1284.19 GiB. +(Worker_TP0 pid=2762723) INFO 07-12 17:03:06 [weight_utils.py:872] Auto-prefetch is disabled because the filesystem (FUSE.ALIYUN-ALINAS-EFC) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch. +(Worker_TP0 pid=2762723) Loading safetensors checkpoint shards: 0% Completed | 0/16 [00:00 + main() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_solo_controller.py", line 392, in main + execute_cell(index, cell, state, expected, histories) + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_solo_controller.py", line 374, in execute_cell + raise failure + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_solo_controller.py", line 361, in execute_cell + base.assert_all_idle() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 133, in assert_all_idle + raise RuntimeError(f"GPU cleanup failure: {bad}") +RuntimeError: GPU cleanup failure: [(0, 4, 0), (1, 4, 0), (2, 4, 0), (3, 4, 0), (4, 4, 0), (5, 4, 0), (6, 4, 0), (7, 1, 0)] +SOLO_WAVE_ECHO cell=tp2_mns32 tp=2 mns=32 gpus=0-1 mandatory=0.75390625,0.75 spent_h20h=3.315064 cell_est_h20h=0.220 remaining_projection_h20h=2.480 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp2_mns64 tp=2 mns=64 gpus=0-1 mandatory=0.75,0.5 spent_h20h=3.490967 cell_est_h20h=0.220 remaining_projection_h20h=2.260 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp4_mns16 tp=4 mns=16 gpus=0-3 mandatory=0.033717411016,0.033182214016,0.034252608017 spent_h20h=3.664183 cell_est_h20h=0.480 remaining_projection_h20h=2.040 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp2_mns8 tp=2 mns=8 gpus=0-1 mandatory=0.49609375,0.4921875 spent_h20h=4.217542 cell_est_h20h=0.220 remaining_projection_h20h=1.560 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp2_mns16 tp=2 mns=16 gpus=0-1 mandatory=0.49609375,0.4921875 spent_h20h=4.392992 cell_est_h20h=0.220 remaining_projection_h20h=1.340 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp4_mns8 tp=4 mns=8 gpus=0-3 mandatory=0.016591107009,0.016055910008 spent_h20h=4.623573 cell_est_h20h=0.480 remaining_projection_h20h=1.120 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns8 tp=1 mns=8 gpus=0-0 mandatory=0.2265625,0.21875 spent_h20h=5.120182 cell_est_h20h=0.110 remaining_projection_h20h=0.640 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns16 tp=1 mns=16 gpus=0-0 mandatory=0.24609375,0.25 spent_h20h=5.319979 cell_est_h20h=0.110 remaining_projection_h20h=0.530 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns32 tp=1 mns=32 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.404469 cell_est_h20h=0.110 remaining_projection_h20h=0.420 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns64 tp=1 mns=64 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.523280 cell_est_h20h=0.110 remaining_projection_h20h=0.310 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +Traceback (most recent call last): + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_solo_controller.py", line 419, in + main() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_solo_controller.py", line 405, in main + execute_cell(index, cell, state, expected, histories) + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_solo_controller.py", line 295, in execute_cell + base.assert_all_idle() + File "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/scripts/opprof_phase6_controller.py", line 133, in assert_all_idle + raise RuntimeError(f"GPU cleanup failure: {bad}") +RuntimeError: GPU cleanup failure: [(0, 4, 0), (1, 4, 0), (2, 1, 0), (3, 1, 0), (4, 1, 0), (5, 1, 0), (6, 1, 0), (7, 1, 0)] +SOLO_WAVE_ECHO cell=tp1_mns64 tp=1 mns=64 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.523280 cell_est_h20h=0.110 remaining_projection_h20h=0.310 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +{"campaign_gpu_hours": 5.644544164273745, "cells": 12, "confirmations": 29, "primary_anchors": 37, "solo_gpu_hours": 3.3533713148037596, "status": "complete"} diff --git a/runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log b/runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log new file mode 100644 index 0000000..68b464e --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/launch-echo.log @@ -0,0 +1,16 @@ +A-P6-2_LAUNCH_ECHO utc=2026-07-12T16:14:18Z host=dash0 engine=vllm-0.24.1.dev3+g668cfb7e2@4b253fd authoritative=solo cells=12 mandatory_anchors=25 crawl=recorded-only confirmations=2of3 prior_h20h=2.2911728495 new_est_h20h=3.440 cumulative_est_h20h=5.7311728495 cap_h20h=6.0 est_wall=45-70min ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp4_mns32 tp=4 mns=32 gpus=0-3 mandatory=0.033717411016 spent_h20h=2.291173 cell_est_h20h=0.480 remaining_projection_h20h=3.440 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp4_mns64 tp=4 mns=64 gpus=0-3 mandatory=0.033717411016 spent_h20h=2.805712 cell_est_h20h=0.480 remaining_projection_h20h=2.960 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +A-P6-2_RESUME_ECHO utc=2026-07-12T16:31:13Z repair=transient-driver-memory-poll completed_cells=2 spent_h20h=3.315063744253854 restart=tp2_mns32 cap_h20h=6.0 +SOLO_WAVE_ECHO cell=tp2_mns32 tp=2 mns=32 gpus=0-1 mandatory=0.75390625,0.75 spent_h20h=3.315064 cell_est_h20h=0.220 remaining_projection_h20h=2.480 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp2_mns64 tp=2 mns=64 gpus=0-1 mandatory=0.75,0.5 spent_h20h=3.490967 cell_est_h20h=0.220 remaining_projection_h20h=2.260 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp4_mns16 tp=4 mns=16 gpus=0-3 mandatory=0.033717411016,0.033182214016,0.034252608017 spent_h20h=3.664183 cell_est_h20h=0.480 remaining_projection_h20h=2.040 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp2_mns8 tp=2 mns=8 gpus=0-1 mandatory=0.49609375,0.4921875 spent_h20h=4.217542 cell_est_h20h=0.220 remaining_projection_h20h=1.560 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp2_mns16 tp=2 mns=16 gpus=0-1 mandatory=0.49609375,0.4921875 spent_h20h=4.392992 cell_est_h20h=0.220 remaining_projection_h20h=1.340 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp4_mns8 tp=4 mns=8 gpus=0-3 mandatory=0.016591107009,0.016055910008 spent_h20h=4.623573 cell_est_h20h=0.480 remaining_projection_h20h=1.120 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns8 tp=1 mns=8 gpus=0-0 mandatory=0.2265625,0.21875 spent_h20h=5.120182 cell_est_h20h=0.110 remaining_projection_h20h=0.640 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns16 tp=1 mns=16 gpus=0-0 mandatory=0.24609375,0.25 spent_h20h=5.319979 cell_est_h20h=0.110 remaining_projection_h20h=0.530 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns32 tp=1 mns=32 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.404469 cell_est_h20h=0.110 remaining_projection_h20h=0.420 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +SOLO_WAVE_ECHO cell=tp1_mns64 tp=1 mns=64 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.523280 cell_est_h20h=0.110 remaining_projection_h20h=0.310 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl +A-P6-2_RESUME_ECHO utc=2026-07-12T17:36:18Z repair=prelaunch-driver-memory-poll spent_h20h=5.523279991083946 restart=tp1_mns64 cap_h20h=6.0 +SOLO_WAVE_ECHO cell=tp1_mns64 tp=1 mns=64 gpus=0-0 mandatory=0.2421875,0.24609375 spent_h20h=5.523280 cell_est_h20h=0.110 remaining_projection_h20h=0.310 cap_h20h=6.0 ground_truth=/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/provenance/ground_truth.json trace=/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/chat_w20260311_1000.jsonl diff --git a/runs/opprof-phase6/phase6/solo-authoritative/selection-preflight.json b/runs/opprof-phase6/phase6/solo-authoritative/selection-preflight.json new file mode 100644 index 0000000..6898645 --- /dev/null +++ b/runs/opprof-phase6/phase6/solo-authoritative/selection-preflight.json @@ -0,0 +1,17 @@ +{ + "invariants": { + "counts_match": true, + "observations_92": true + }, + "mismatches": [], + "observations": 92, + "request_counts": { + "distinct_n": 34, + "finite_n": 92, + "max": 600.0, + "min": 66.0, + "missing_n": 0, + "n": 92 + }, + "schema": 1 +} diff --git a/runs/opprof-phase6/phase6/w1-readjudication-A-P6-1.json b/runs/opprof-phase6/phase6/w1-readjudication-A-P6-1.json new file mode 100644 index 0000000..10e2726 --- /dev/null +++ b/runs/opprof-phase6/phase6/w1-readjudication-A-P6-1.json @@ -0,0 +1,211 @@ +{ + "amendment": "A-P6-1", + "cells": [ + { + "anchors": [ + { + "anchor": 0.24609375, + "covered": true, + "interval_end_mono_ns": 199601637648726, + "interval_start_mono_ns": 199540309473748, + "layer1_records": 4706, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.24609375/result.json" + }, + { + "anchor": 0.25, + "covered": true, + "interval_end_mono_ns": 199668842271892, + "interval_start_mono_ns": 199607371060368, + "layer1_records": 3883, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/anchor-0.25/result.json" + } + ], + "cell": "tp1_mns16", + "checkpoint_after_anchor_s": 1.000327977, + "checkpoint_stream_delta_s": 0.001439359, + "counters": { + "dropped_records": 0, + "encoded_records": 9212, + "last_step_index": 9211, + "written_records": 9212 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 9212, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns16/opprof/opprof-v1-dp0-pid2638886-1783867898065648658.jsonl" + }, + { + "anchors": [ + { + "anchor": 0.2421875, + "covered": true, + "interval_end_mono_ns": 199601604247251, + "interval_start_mono_ns": 199540309397799, + "layer1_records": 4844, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/anchor-0.2421875/result.json" + }, + { + "anchor": 0.24609375, + "covered": true, + "interval_end_mono_ns": 199668653603418, + "interval_start_mono_ns": 199607365362010, + "layer1_records": 4144, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/anchor-0.24609375/result.json" + } + ], + "cell": "tp1_mns32", + "checkpoint_after_anchor_s": 1.000711369, + "checkpoint_stream_delta_s": 0.001155708, + "counters": { + "dropped_records": 0, + "encoded_records": 9633, + "last_step_index": 9632, + "written_records": 9633 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 9633, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns32/opprof/opprof-v1-dp0-pid2638900-1783867898685556703.jsonl" + }, + { + "anchors": [ + { + "anchor": 0.2421875, + "covered": true, + "interval_end_mono_ns": 199601593594258, + "interval_start_mono_ns": 199540314883238, + "layer1_records": 4855, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/anchor-0.2421875/result.json" + }, + { + "anchor": 0.24609375, + "covered": true, + "interval_end_mono_ns": 199668751241940, + "interval_start_mono_ns": 199607467392165, + "layer1_records": 4165, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/anchor-0.24609375/result.json" + } + ], + "cell": "tp1_mns64", + "checkpoint_after_anchor_s": 2.013733755, + "checkpoint_stream_delta_s": 1.014815835, + "counters": { + "dropped_records": 0, + "encoded_records": 9654, + "last_step_index": 9653, + "written_records": 9654 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 9654, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2638898-1783867899798067669.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns64/opprof/opprof-v1-dp0-pid2638898-1783867899798067669.jsonl" + }, + { + "anchors": [ + { + "anchor": 0.21875, + "covered": true, + "interval_end_mono_ns": 199668659416799, + "interval_start_mono_ns": 199607365130468, + "layer1_records": 5756, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/anchor-0.21875/result.json" + }, + { + "anchor": 0.2265625, + "covered": true, + "interval_end_mono_ns": 199568971776020, + "interval_start_mono_ns": 199540309654499, + "layer1_records": 1663, + "result": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/anchor-0.2265625/result.json" + } + ], + "cell": "tp1_mns8", + "checkpoint_after_anchor_s": 1.000563608, + "checkpoint_stream_delta_s": 0.12747079, + "counters": { + "dropped_records": 0, + "encoded_records": 7683, + "last_step_index": 7682, + "written_records": 7683 + }, + "footer_count": 0, + "invariants": { + "all_anchor_intervals_covered": true, + "all_schema_1": true, + "checkpoint_after_all_anchor_intervals": true, + "checkpoint_sidecar": true, + "checkpoint_within_flush_of_stream": true, + "complete_final_newline": true, + "encoded_balanced": true, + "last_step_matches": true, + "no_in_stream_footer": true, + "steps_contiguous": true, + "two_anchor_intervals": true, + "written_matches_records": true, + "zero_drops": true + }, + "passed": true, + "records": 7683, + "sidecar": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/opprof/opprof-v1-dp0-pid2638884-1783867942846550713.jsonl.footer.json", + "stream": "/home/admin/cpfs/wjh/opprof-phase6-dash0-20260712/runs/phase6/cells/tp1_mns8/opprof/opprof-v1-dp0-pid2638884-1783867942846550713.jsonl" + } + ], + "mode": "checkpoint-sidecar", + "passed": true, + "sanity": { + "anchor_count": 8, + "cell_count": 4, + "records_distinct": 4, + "records_max": 9654, + "records_min": 7683 + }, + "schema": 1 +} diff --git a/runs/opprof-phase6/readjudicate_w1.py b/runs/opprof-phase6/readjudicate_w1.py new file mode 100644 index 0000000..e095404 --- /dev/null +++ b/runs/opprof-phase6/readjudicate_w1.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +"""CPU-only A-P6-1 checkpoint-sidecar adjudication for the four W1 cells.""" + +from __future__ import annotations + +import argparse +import json +import os +from pathlib import Path +from typing import Any + + +def atomic_json(path: Path, value: Any) -> None: + tmp = path.with_suffix(path.suffix + ".tmp") + tmp.write_text(json.dumps(value, sort_keys=True, indent=2) + "\n") + os.replace(tmp, path) + + +def audit_cell(cell_dir: Path) -> dict[str, Any]: + streams = sorted((cell_dir / "opprof").glob("*.jsonl")) + sidecars = sorted((cell_dir / "opprof").glob("*.jsonl.footer.json")) + if len(streams) != 1 or len(sidecars) != 1: + raise RuntimeError(f"{cell_dir}: stream/sidecar count {len(streams)}/{len(sidecars)}") + stream, sidecar_path = streams[0], sidecars[0] + raw = stream.read_bytes() + complete_newline = raw.endswith(b"\n") + decoded = [json.loads(line) for line in raw.splitlines()] + footers = [x for x in decoded if x.get("record_type") == "footer"] + records = [x for x in decoded if "step_index" in x] + sidecar = json.loads(sidecar_path.read_text()) + indices = [int(x["step_index"]) for x in records] + checkpoint_stream_delta_s = abs(stream.stat().st_mtime_ns - int(sidecar["checkpoint_wall_ns"])) / 1e9 + anchor_results = [] + coverage = [] + for path in sorted(cell_dir.glob("anchor-*/result.json")): + result = json.loads(path.read_text()) + lo = int(result["interval"]["start_mono_ns"]) + hi = int(result["interval"]["end_mono_ns"]) + selected = [x for x in records if lo <= int(x["submit_mono_ns"]) <= hi] + # Layer-1 intervals are keyed by submit_mono_ns. A final async model step + # may complete sub-millisecond after the client has received its last + # response; the post-interval checkpoint proves that record is durable. + covered = bool(selected) and max(int(x["submit_mono_ns"]) for x in selected) <= hi + anchor_results.append({ + "anchor": result["anchor"], "result": str(path), + "interval_start_mono_ns": lo, "interval_end_mono_ns": hi, + "layer1_records": len(selected), "covered": covered, + }) + coverage.append(covered) + latest_anchor_wall_ns = max( + int(json.loads(path.read_text())["interval"]["end_wall_ns"]) + for path in cell_dir.glob("anchor-*/result.json") + ) + invariants = { + "complete_final_newline": complete_newline, + "all_schema_1": all(x.get("schema") == 1 for x in decoded) and sidecar.get("schema") == 1, + "no_in_stream_footer": not footers, + "checkpoint_sidecar": sidecar.get("final") is False, + "steps_contiguous": indices == list(range(len(indices))), + "written_matches_records": int(sidecar["written_records"]) == len(records), + "encoded_balanced": int(sidecar["encoded_records"]) == int(sidecar["written_records"]) + int(sidecar["dropped_records"]), + "last_step_matches": bool(records) and int(sidecar["last_step_index"]) == indices[-1], + "zero_drops": int(sidecar["dropped_records"]) == 0 and all(int(x["dropped_records_before"]) == 0 for x in records), + "checkpoint_within_flush_of_stream": checkpoint_stream_delta_s <= float(sidecar["flush_interval_seconds"]) + .1, + "checkpoint_after_all_anchor_intervals": int(sidecar["checkpoint_wall_ns"]) >= latest_anchor_wall_ns, + "two_anchor_intervals": len(anchor_results) == 2, + "all_anchor_intervals_covered": len(coverage) == 2 and all(coverage), + } + return { + "cell": cell_dir.name, "passed": all(invariants.values()), + "stream": str(stream), "sidecar": str(sidecar_path), "records": len(records), + "footer_count": len(footers), "checkpoint_stream_delta_s": checkpoint_stream_delta_s, + "checkpoint_after_anchor_s": (int(sidecar["checkpoint_wall_ns"]) - latest_anchor_wall_ns) / 1e9, + "counters": {key: sidecar[key] for key in ("encoded_records", "written_records", "dropped_records", "last_step_index")}, + "anchors": anchor_results, "invariants": invariants, + } + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--out", type=Path, required=True) + args = parser.parse_args() + cells = [audit_cell(path) for path in sorted((args.root / "cells").glob("tp1_mns*"))] + result = { + "schema": 1, "amendment": "A-P6-1", "mode": "checkpoint-sidecar", + "cells": cells, "passed": len(cells) == 4 and all(x["passed"] for x in cells), + "sanity": { + "cell_count": len(cells), "anchor_count": sum(len(x["anchors"]) for x in cells), + "records_min": min(x["records"] for x in cells), "records_max": max(x["records"] for x in cells), + "records_distinct": len({x["records"] for x in cells}), + }, + } + for cell in cells: + cell_dir = Path(cell["stream"]).parent.parent + atomic_json(cell_dir / "cell-valid.json", { + "cell": cell["cell"], "invariants": cell["invariants"], + "layer1_records": cell["records"], "stream": cell["stream"], + "accounting_mode": "A-P6-1-checkpoint-sidecar", + }) + atomic_json(args.out, result) + print(json.dumps({"passed": result["passed"], "sanity": result["sanity"], "cells": [{"cell": x["cell"], "passed": x["passed"], "invariants": x["invariants"]} for x in cells]}, sort_keys=True)) + if not result["passed"]: + raise RuntimeError("W1 checkpoint-sidecar re-adjudication failed") + + +if __name__ == "__main__": + main() diff --git a/runs/opprof-phase6/test_phase6_tools.py b/runs/opprof-phase6/test_phase6_tools.py new file mode 100644 index 0000000..0cc338d --- /dev/null +++ b/runs/opprof-phase6/test_phase6_tools.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import importlib.util +from pathlib import Path + + +HERE = Path(__file__).resolve().parent + + +def load(name: str, path: Path): + spec = importlib.util.spec_from_file_location(name, path) + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + spec.loader.exec_module(module) + return module + + +def main() -> None: + controller = load("p6controller", HERE / "opprof_phase6_controller.py") + solo = load("p6solo", HERE / "opprof_phase6_solo_controller.py") + analysis = load("p6analysis", HERE / "analyze_phase6.py") + assert len(controller.CELLS) == 12 + primary = sum(3 if item.get("trap") else 2 for item in controller.CELLS.values()) + assert primary == 25 + assert sum(len(gpus) for _name, wave, _cost in controller.WAVES for _cell, gpus in wave) == 28 + scores = {"a": 1.0, "b": 1.0, "c": 2.0} + tol, buckets = analysis.floor_buckets(scores) + assert tol == 2e-6 and buckets["a"] == buckets["b"] < buckets["c"] + tau = analysis.kendall_tau_b({"a": 0, "b": 1, "c": 2}, {"a": 0, "b": 1, "c": 2}) + assert tau["tau_b"] == 1.0 and tau["concordant"] == 3 + assert len(solo.ORDER) == 12 and set(solo.ORDER) == set(controller.CELLS) + assert sum(len(items) for items in solo.CORE.values()) == 23 + assert sum(2 if len(items) == 1 else len(items) for items in solo.CORE.values()) == 25 + assert solo.GPU_LIMIT == 6.0 + assert sum(solo.CELL_ESTIMATE.values()) + solo.SAFETY_HOURS == 3.44 + assert solo.next_below([.1, .2, .3], {.2, .3}) == .1 + assert solo.next_above([.1, .2, .3], {.1, .2}) == .3 + print("phase6 tools: PASS") + + +if __name__ == "__main__": + main()