gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add agentic workload characterization audit scaffold

Browse Source
This commit is contained in:
Gahow Wang
2026-05-25 15:01:18 +08:00
parent 21ffb3d4f7
commit 0f64fb3261
16 changed files with 4836 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

255
analysis/characterization/README.md Normal file
View File

@@ -0,0 +1,255 @@
# Characterization Analyzer Runbook
CPU-only scaffold for Batch 0 and Batch 1 in
`analysis/characterization_todo_for_interns.md`.
This directory has three components:
- `analyze.py`: Batch 0/1 analyzer for trace and per-request metrics.
- `summarize_runs.py`: CPU-only audit of already completed benchmark
directories.
- `protocols.md`: exact protocol for Batch 2-6 experiments that require fresh
GPU runs or additional instrumentation.
The analyzer reads existing trace and metrics artifacts and writes:
```text
outputs/characterization/<date>/<task_name>/
├── manifest.json
├── raw/
├── summary.json
├── summary.md
├── audit.md
├── session_concurrency.json
├── session_arrival_stats.json
├── turn_interval_stats.json
├── trace_profile.json
├── invalid_runs.md
├── workload_summary.json
├── kv_footprint_summary.json
├── reuse_decomposition.json
├── session_skew.json
├── append_delta_stats.json
└── figures/
```
If `matplotlib` is installed, simple PNG/PDF figures are emitted under
`figures/`. If it is not installed, all JSON/Markdown data artifacts are still
written.
## Canonical Data Sources
Canonical full traces live on dash0:
- formatted trace: `~/ali-trace/trace-glm5.1-formatted/`
- raw unformatted trace: `~/ali-trace/trace-glm5.1/`
For the current GLM-5.1 characterization, prefer the compact formatted file:
```text
~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
```
Do not pass `051315-051317-raw.jsonl` or the files under
`~/ali-trace/trace-glm5.1/` directly to this analyzer unless you first convert
them to the formatted schema. Those raw files are tens to hundreds of GiB and
contain full prompt payloads rather than the compact characterization schema.
The analyzer is CPU-only. For full trace characterization, either:
- run it on dash0 against the formatted JSONL files without starting any GPU
service; or
- copy/rsync the needed trace files from dash0 to this repository or another
local path, then run the analyzer locally.
Only light directory/field inspection is needed on dash0 before choosing which
trace file to analyze.
The raw unformatted directory is listed as a source option for provenance, but
this analyzer expects formatted JSONL records. Raw files should be converted to
the formatted schema before being passed to `--trace`.
## Inputs
Trace JSONL:
- Expected formatted fields: `chat_id`, `parent_chat_id`, `timestamp`,
`input_length`, `output_length`, `type`, `turn`, `hash_ids`, optional
`session_id`.
- If `session_id` is absent, sessions are reconstructed from
`parent_chat_id` chains.
- `timestamp` is treated as scheduled trace time, not proof of actual dispatch
time.
Metrics JSONL:
- Expected replayer fields: `request_id`, `session_id`, `turn_id`,
`trace_timestamp_s`, `input_length`, `output_length`, `cached_tokens`,
`latency_s`, `ttft_s`, `tpot_s`, `actual_output_tokens`, `error`.
- If the metrics file is from the current replayer, it does not include actual
dispatch/finish wall-clock timestamps. Batch 0 will therefore mark actual
session sequentiality as unavailable and separately report a scheduled
estimate from `trace_timestamp_s + latency_s`.
Proxy breakdown:
- Optional JSON/JSONL with fields such as `request_id`, `t_proxy_recv`,
`t_first_token`, `t_done`, `cache_hit`, `estimated_new_tokens`,
`route_class`, `routed_to`, `policy`.
- Batch 0 can prove actual per-session in-flight concurrency only when these
timing rows can be joined to analyzed requests by `request_id`.
- Existing proxy breakdown artifacts may not contain `session_id`; without a
request-id join to trace/metrics, they can still support append/cache-hit
statistics but not per-session concurrency.
Run config:
- Optional JSON, usually `outputs/<run>/config.json`.
- Used for manifest fields such as `policy`, `time_scale`, and request count
when available.
## Commands
Trace-only dry run:
```bash
python3 analysis/characterization/analyze.py \
--trace traces/w600_r0.0015_st30.jsonl \
--task-name w600_trace_only \
--overwrite
```
Trace plus replayer metrics:
```bash
python3 analysis/characterization/analyze.py \
--trace traces/w600_r0.0015_st30.jsonl \
--metrics outputs/smoke_test/metrics.jsonl \
--task-name smoke_trace_metrics \
--overwrite
```
Proxy breakdown append/cache analysis:
```bash
python3 analysis/characterization/analyze.py \
--breakdown outputs/contention_16s_elastic/breakdown.json \
--config outputs/contention_16s_elastic/config.json \
--task-name contention_breakdown \
--overwrite
```
Full trace on dash0, CPU-only:
```bash
python3 analysis/characterization/analyze.py \
--trace ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
--task-name full_trace_characterization \
--overwrite
```
Local run after copying from dash0:
```bash
rsync -av dash0:~/ali-trace/trace-glm5.1-formatted/<trace-file>.jsonl traces/
python3 analysis/characterization/analyze.py \
--trace traces/<trace-file>.jsonl \
--task-name full_trace_characterization \
--overwrite
```
By default the analyzer records file size and mtime but skips full SHA256
hashing, because canonical raw trace files can be hundreds of GiB. Add
`--hash-inputs` only when you intentionally want a full file hash.
KV footprint requires a model-specific value:
```bash
python3 analysis/characterization/analyze.py \
--trace traces/w600_r0.0015_st30.jsonl \
--kv-bytes-per-token 98304 \
--task-name w600_with_kv_estimate \
--overwrite
```
Summarize existing completed runs:
```bash
python3 analysis/characterization/summarize_runs.py
```
This writes:
```text
analysis/characterization/current_results/
├── run_summaries.json
├── comparisons.json
├── claim_matrix.json
├── reviewer_risk_register.json
├── current_results.md
├── characterization_claim_matrix.md
├── all_figures_index.md
├── reviewer_risk_register.md
└── reproduction_commands.sh
```
## Batch 0 Semantics
The online-serving invariant is:
```text
Each session has at most one in-flight turn.
```
The analyzer reports:
- actual interval status from dispatch and finish/error timestamps;
- scheduled estimate from trace timestamps plus latency when available;
- per-session max in-flight;
- session start-time distribution;
- turn inter-arrival distribution;
- attempted/completed/error counts and goodput when metrics exist;
- run classification.
Important limitation: trace timestamps alone cannot prove actual replay
sequentiality. A run is only classified as `online_realistic` when actual
per-request dispatch and finish/error timestamps prove
`max_inflight_per_session <= 1`.
## Batch 1 Semantics
The analyzer reports:
- input/output CDF stats;
- input/output ratio;
- KV footprint CDF stats when `--kv-bytes-per-token` is supplied;
- session skew and top-session contribution;
- append/uncached token stats when `cached_tokens` or `cache_hit` exists;
- reuse decomposition when both cached-token fields and `hash_ids` exist.
Reuse decomposition is conservative:
- `intra_session`: cached hash block was seen earlier in the same session;
- `cross_session`: cached hash block was seen earlier in another session;
- `shared/system-prefix`: early-position block appears in many sessions;
- `unclassified`: cached tokens could not be mapped to a previously seen hash
block.
If cached-token/cache-hit fields are absent, reuse and append artifacts are
written with `status: "unavailable"` and list the required fields.
## Limitations
- The script does not run a benchmark, query a live service, touch GPU state,
or start any daemon.
- Request-id joins are exact. If trace, metrics, and proxy artifacts use
different request IDs, the unmatched rows are preserved under `raw/`.
- Actual Batch 0 sequentiality needs actual dispatch and finish/error
timestamps. Current `replayer/metrics.py` metrics are not enough by
themselves.
- `kv_bytes_per_token` depends on model architecture, layer count, KV heads,
head dimension, and dtype. The analyzer will not guess it.
- Shared/system-prefix reuse classification is a heuristic based on trace
`hash_ids` positions and cross-session frequency. Adjust
`--shared-prefix-min-sessions` and `--system-prefix-blocks` if the formatted
trace provides a stronger system-prefix marker.

1873
analysis/characterization/analyze.py Normal file
View File

File diff suppressed because it is too large Load Diff

10
analysis/characterization/current_results/all_figures_index.md Normal file
View File

@@ -0,0 +1,10 @@
# Figures Index
No generated figures are committed by this script. Batch-specific figures should be generated from:
- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.
- future Batch 2 step-timeline artifacts for interference plots.
- future Batch 3 per-worker/session artifacts for hot-spot plots.
- future Batch 4 arrival-rate sweep artifacts for SRR curves.
This file exists so the audit package has a stable placeholder until fresh figures are generated.

11
analysis/characterization/current_results/characterization_claim_matrix.md Normal file
View File

@@ -0,0 +1,11 @@
# Characterization Claim Matrix
| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |
|---|---|---|---|---|
| Batch 0 substrate audit is only partially complete for existing runs. | `partially_supported` | metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts. | Add request dispatch and finish/error timestamps to future replayer/proxy metrics. | Cannot use these runs to prove online per-session sequentiality. |
| Batch 1 workload shape can be characterized from formatted traces and metrics. | `supported_for_trace_shape` | Full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass. | Add cache-hit joined records for actual reuse decomposition. | Actual cache reuse decomposition needs cached_tokens joined with hash_ids. |
| Static PD separation is worse than combined in existing 200-request GPU A/B. | `supported_by_existing_artifact` | outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json. | Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology. | Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy. |
| Elastic transfer-based migration does not improve high-contention 500-request run. | `supported_by_existing_artifact` | outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv. | Attribute whether failure is trigger quality, transfer overhead, or wrong load regime. | Existing metrics lack actual sequentiality proof and per-request transfer waterfall. |
| PD-colo prefill/decode interference is not yet directly proven by step-level data in this package. | `not_yet_supported` | No decode-step and prefill-overlap timestamp artifact found in summarized runs. | Run Batch 2 controlled same-worker/different-worker injection with step timestamps. | Cannot claim interference as causal without Batch 2. |
| Session hot-spot residual imbalance is suggested but not fully attributed. | `partially_supported` | gpu_util.csv shows per-GPU mean-util imbalance in existing runs. | Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker. | GPU util imbalance alone is not enough to prove session hot-spot. |
| SRR is not measured by existing fixed-request runs. | `not_yet_supported` | No arrival-rate sweep artifacts found. | Implement Batch 4 Poisson session-arrival SRR sweep. | Latency-at-one-load cannot support sustainable throughput claim. |

51
analysis/characterization/current_results/claim_matrix.json Normal file
View File

@@ -0,0 +1,51 @@
[
{
"claim": "Batch 0 substrate audit is only partially complete for existing runs.",
"needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.",
"reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.",
"status": "partially_supported",
"supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts."
},
{
"claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.",
"needed_next": "Add cache-hit joined records for actual reuse decomposition.",
"reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.",
"status": "supported_for_trace_shape",
"supporting_data": "Full compact trace CPU summary in full_trace_summary.json: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass."
},
{
"claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.",
"needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.",
"reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.",
"status": "supported_by_existing_artifact",
"supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json."
},
{
"claim": "Elastic transfer-based migration does not improve high-contention 500-request run.",
"needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.",
"reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.",
"status": "supported_by_existing_artifact",
"supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv."
},
{
"claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.",
"needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.",
"reviewer_risk": "Cannot claim interference as causal without Batch 2.",
"status": "not_yet_supported",
"supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs."
},
{
"claim": "Session hot-spot residual imbalance is suggested but not fully attributed.",
"needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.",
"reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.",
"status": "partially_supported",
"supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs."
},
{
"claim": "SRR is not measured by existing fixed-request runs.",
"needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.",
"reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.",
"status": "not_yet_supported",
"supporting_data": "No arrival-rate sweep artifacts found."
}
]

95
analysis/characterization/current_results/comparisons.json Normal file
View File

@@ -0,0 +1,95 @@
[
{
"baseline": "outputs/gpu_ab_combined",
"e2e_p50_delta_pct": 40.870329127661,
"e2e_p90_delta_pct": 15.206416995091814,
"error_count": [
2,
13
],
"gpu_imbalance_ratio": [
3.2445157838416265,
11.149056603773586
],
"gpu_mean_util": [
30.541666666666664,
12.367081447963802
],
"name": "combined_vs_pdsep_200",
"request_count": [
200,
200
],
"success_count": [
198,
187
],
"tpot_p90_delta_pct": 1.3481309269699875,
"ttft_p50_delta_pct": 98.06752892925572,
"ttft_p90_delta_pct": 44.79649177751278,
"variant": "outputs/gpu_ab_pdsep",
"wall_clock_delta_pct": 142.27736808267244
},
{
"baseline": "outputs/contention_16s_ts10",
"e2e_p50_delta_pct": 11.538788125232664,
"e2e_p90_delta_pct": -5.080083318118138,
"error_count": [
2,
2
],
"gpu_imbalance_ratio": [
2.310775410408662,
2.600767754318618
],
"gpu_mean_util": [
23.030492424242425,
26.349561403508773
],
"name": "contention_baseline_vs_elastic_500",
"request_count": [
500,
500
],
"success_count": [
498,
498
],
"tpot_p90_delta_pct": 13.63098996823875,
"ttft_p50_delta_pct": 12.433589435386224,
"ttft_p90_delta_pct": 13.412576920999959,
"variant": "outputs/contention_16s_elastic",
"wall_clock_delta_pct": -0.5645626396767849
},
{
"baseline": "outputs/combined_1000req",
"e2e_p50_delta_pct": 202.85189980479385,
"e2e_p90_delta_pct": 128.274511020719,
"error_count": [
2,
204
],
"gpu_imbalance_ratio": [
null,
null
],
"gpu_mean_util": [
null,
null
],
"name": "combined_1000_vs_pdsep_mooncake",
"request_count": [
1000,
1000
],
"success_count": [
998,
796
],
"tpot_p90_delta_pct": -34.83638659447109,
"ttft_p50_delta_pct": 781.9835547522864,
"ttft_p90_delta_pct": 1030.68607857992,
"variant": "outputs/exp3_pd_sep_tp1_mooncake",
"wall_clock_delta_pct": 119.18997774599991
}
]

77
analysis/characterization/current_results/current_results.md Normal file
View File

@@ -0,0 +1,77 @@
# Current Characterization Results
Generated: 2026-05-25T06:52:18.096448+00:00
Git commit: `21ffb3d4f77956d008b1815a3c0d46e0188ac390`
## Canonical Full-Trace CPU Summary
Source: `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`.
This is CPU-only parsing of the compact formatted trace with session IDs
reconstructed from `parent_chat_id` chains.
| Metric | Value |
|---|---:|
| Requests | 2,114,220 |
| Sessions | 1,307,276 |
| Trace span | 7,199.975 s |
| Input tokens p50/p90/p99 | 20,030 / 87,855 / 125,527 |
| Output tokens p50/p90/p99 | 80 / 811 / 6,615 |
| Input/output ratio p50/p90/p99 | 217.8 / 1,204.4 / 4,251.6 |
| Turns/session p50/p90/p99/max | 1 / 1 / 18 / 3,091 |
| Session input tokens p50/p90/p99/max | 12,486 / 72,676 / 974,934 / 156,756,974 |
| Top 1% / 5% / 10% sessions by input-token mass | 46.5% / 66.5% / 74.6% |
Immediate reading: the full trace strongly supports long-input/short-output
and heavy-tailed session token mass. It does **not** by itself prove online
sequentiality or actual cache-hit reuse; those require runtime timestamps and
cache-hit fields.
## Existing Run Summaries
| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |
|---|---:|---:|---:|---:|---:|---:|
| outputs/gpu_ab_combined | 198/200 | 1.01/9.36 | 5.05/30.2 | 0.0732 | 30.5 | 3.24 |
| outputs/gpu_ab_pdsep | 187/200 | 1.99/13.5 | 7.11/34.8 | 0.0742 | 12.4 | 11.1 |
| outputs/contention_16s_ts10 | 498/500 | 0.826/9.71 | 5.8/51 | 0.103 | 23 | 2.31 |
| outputs/contention_16s_elastic | 498/500 | 0.929/11 | 6.47/48.4 | 0.117 | 26.3 | 2.6 |
| outputs/combined_1000req | 998/1000 | 0.393/2.57 | 3.22/28 | 0.113 | n/a | n/a |
| outputs/exp3_pd_sep_tp1_mooncake | 796/1000 | 3.47/29 | 9.75/63.9 | 0.0739 | n/a | n/a |
## Pairwise Comparisons
| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |
|---|---:|---:|---:|---:|---:|---:|
| combined_vs_pdsep_200 | +98.1% | +44.8% | +40.9% | +15.2% | +1.3% | +142.3% |
| contention_baseline_vs_elastic_500 | +12.4% | +13.4% | +11.5% | -5.1% | +13.6% | -0.6% |
| combined_1000_vs_pdsep_mooncake | +782.0% | +1030.7% | +202.9% | +128.3% | -34.8% | +119.2% |
## What We Can Say Now
- **partially_supported**: Batch 0 substrate audit is only partially complete for existing runs.
Supporting data: metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.
Next: Add request dispatch and finish/error timestamps to future replayer/proxy metrics.
- **supported_for_trace_shape**: Batch 1 workload shape can be characterized from formatted traces and metrics.
Supporting data: full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass.
Next: add cache-hit joined records for actual reuse decomposition.
- **supported_by_existing_artifact**: Static PD separation is worse than combined in existing 200-request GPU A/B.
Supporting data: outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.
Next: Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.
- **supported_by_existing_artifact**: Elastic transfer-based migration does not improve high-contention 500-request run.
Supporting data: outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.
Next: Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.
- **not_yet_supported**: PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.
Supporting data: No decode-step and prefill-overlap timestamp artifact found in summarized runs.
Next: Run Batch 2 controlled same-worker/different-worker injection with step timestamps.
- **partially_supported**: Session hot-spot residual imbalance is suggested but not fully attributed.
Supporting data: gpu_util.csv shows per-GPU mean-util imbalance in existing runs.
Next: Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.
- **not_yet_supported**: SRR is not measured by existing fixed-request runs.
Supporting data: No arrival-rate sweep artifacts found.
Next: Implement Batch 4 Poisson session-arrival SRR sweep.
## Main Reviewer Risks
- **high**: Session sequentiality not proven - Add dispatch/finish timestamps and run Batch 0 before SRR claims.
- **medium**: Legacy PD-sep data may not match final methodology - Use fresh PD matrix for paper-grade claims.
- **medium**: GPU util is not a sufficient hot-spot proof - Add route-decision and per-worker queue logs for Batch 3.
- **medium**: Cache reuse decomposition is incomplete without joined hash/cache-hit data - Emit hash_ids/session_id/cached_tokens in the same per-request record.

56
analysis/characterization/current_results/full_trace_summary.json Normal file
View File

@@ -0,0 +1,56 @@
{
"input": {
"count": 2114220,
"max": 202371,
"mean": 33637.38370084476,
"p50": 20030.0,
"p90": 87855.1000000001,
"p95": 104738.0,
"p99": 125527.0
},
"input_output_ratio": {
"count": 2108130,
"max": 143664.0,
"mean": 534.3516074828406,
"p50": 217.8,
"p90": 1204.3769610389616,
"p95": 1814.3478327228322,
"p99": 4251.585499999998
},
"output": {
"count": 2114220,
"max": 132665,
"mean": 444.97059624826176,
"p50": 80.0,
"p90": 811.0,
"p95": 2213.0,
"p99": 6614.810000000056
},
"path": "/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl",
"records": 2114220,
"session_input_tokens": {
"count": 1307276,
"max": 156756974,
"mean": 54400.77639916896,
"p50": 12486.0,
"p90": 72676.0,
"p95": 108523.25,
"p99": 974933.75
},
"sessions": 1307276,
"top_session_input_fraction": {
"top10pct": 0.7464402483455778,
"top1pct": 0.46456810581415175,
"top5pct": 0.6651718740752172
},
"trace_span_s": 7199.975,
"turns_per_session": {
"count": 1307276,
"max": 3091,
"mean": 1.6172713336739908,
"p50": 1.0,
"p90": 1.0,
"p95": 2.0,
"p99": 18.0
}
}

66
analysis/characterization/current_results/main_claim_allowed_runs.md Normal file
View File

@@ -0,0 +1,66 @@
# Main-Claim Allowed Runs
Status: current audit gate
Date: 2026-05-25
## Allowed For Workload-Shape Claims
These artifacts can support trace/workload characterization claims:
- `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`
- Compact formatted full trace.
- CPU summary recorded in `full_trace_summary.json`.
- Supports long-input/short-output and session token-mass skew claims.
- Does not prove runtime cache hits or online sequentiality.
- `traces/w600_r0.0015_st30.jsonl`
- Local sampled trace.
- Useful for local dry runs and figure generation.
- Not the canonical full-trace source.
## Allowed For Legacy Baseline Sanity Claims
These existing runs can support sanity-level comparisons, but not final
paper-grade SRR claims:
- `outputs/gpu_ab_combined`
- `outputs/gpu_ab_pdsep`
- `outputs/contention_16s_ts10`
- `outputs/contention_16s_elastic`
- `outputs/combined_1000req`
- `outputs/exp3_pd_sep_tp1_mooncake`
Allowed claims:
- Static PD-sep was worse than combined in these existing fixed-request runs.
- Elastic transfer-based migration did not improve the summarized 500-request
high-contention run.
- GPU-util imbalance exists in these artifacts.
Disallowed claims:
- Online SRR.
- Per-session sequentiality.
- Causal attribution of prefill/decode interference.
- Causal attribution of session hot spots from GPU utilization alone.
## Not Yet Allowed For Main Claims
The following need fresh instrumentation or fresh runs:
- Batch 2 prefill/decode interference.
- Batch 3 session hot-spot root cause.
- Batch 4 sustainable request rate.
- Batch 5 failure attribution near SRR boundary.
## Required Upgrade Before Paper-Grade Claims
Future main-claim runs must include:
- per-request actual dispatch timestamp;
- per-request finish/error timestamp;
- route decision and selected worker;
- per-worker queue delay;
- per-worker KV occupancy;
- per-worker APC/cache-hit snapshot;
- attempted/completed/error/goodput counters;
- session-causal load generation.

17
analysis/characterization/current_results/reproduction_commands.sh Normal file
View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
set -euo pipefail
# Rebuild this current-results audit package.
python3 analysis/characterization/summarize_runs.py --output-dir analysis/characterization/current_results --runs outputs/gpu_ab_combined outputs/gpu_ab_pdsep outputs/contention_16s_ts10 outputs/contention_16s_elastic outputs/combined_1000req outputs/exp3_pd_sep_tp1_mooncake
# Example Batch 0/1 local trace analysis.
python3 analysis/characterization/analyze.py \
--trace traces/w600_r0.0015_st30.jsonl \
--kv-bytes-per-token 98304 \
--task-name w600_local_full_trace \
--overwrite
# CPU-only full compact trace summary was computed on dash0 from:
# /home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
# Recompute either by running analyze.py on dash0, or by copying that compact
# formatted JSONL locally. Do not use the 487G raw file directly.

26
analysis/characterization/current_results/reviewer_risk_register.json Normal file
View File

@@ -0,0 +1,26 @@
[
{
"evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.",
"mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.",
"risk": "Session sequentiality not proven",
"severity": "high"
},
{
"evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.",
"mitigation": "Use fresh PD matrix for paper-grade claims.",
"risk": "Legacy PD-sep data may not match final methodology",
"severity": "medium"
},
{
"evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.",
"mitigation": "Add route-decision and per-worker queue logs for Batch 3.",
"risk": "GPU util is not a sufficient hot-spot proof",
"severity": "medium"
},
{
"evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.",
"mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.",
"risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data",
"severity": "medium"
}
]

8
analysis/characterization/current_results/reviewer_risk_register.md Normal file
View File

@@ -0,0 +1,8 @@
# Reviewer Risk Register
| Risk | Severity | Evidence | Mitigation |
|---|---|---|---|
| Session sequentiality not proven | `high` | Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps. | Add dispatch/finish timestamps and run Batch 0 before SRR claims. |
| Legacy PD-sep data may not match final methodology | `medium` | PD matrix scaffold exists separately; some old runs used earlier flags/methodology. | Use fresh PD matrix for paper-grade claims. |
| GPU util is not a sufficient hot-spot proof | `medium` | Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership. | Add route-decision and per-worker queue logs for Batch 3. |
| Cache reuse decomposition is incomplete without joined hash/cache-hit data | `medium` | Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts. | Emit hash_ids/session_id/cached_tokens in the same per-request record. |

720
analysis/characterization/current_results/run_summaries.json Normal file
View File

@@ -0,0 +1,720 @@
[
{
"apc_summary": {
"reason": "apc.txt missing",
"status": "unavailable"
},
"artifact_availability": {
"apc_txt": false,
"breakdown_json": false,
"gpu_util_csv": true,
"metrics_jsonl": true,
"metrics_summary_json": true
},
"breakdown_summary": {
"reason": "breakdown.json missing",
"status": "unavailable"
},
"error_count": 2,
"exists": true,
"external_cache_hit_ratio": null,
"gpu_summary": {
"gpu_count": 8,
"max_mean_util_pct": 63.166666666666664,
"max_min_ratio": 3.2445157838416265,
"mean_util_pct": 30.541666666666664,
"min_mean_util_pct": 19.46875,
"per_gpu_mean_util_pct": {
"0": 29.145833333333332,
"1": 20.041666666666668,
"2": 25.0,
"3": 63.166666666666664,
"4": 21.927083333333332,
"5": 34.90625,
"6": 19.46875,
"7": 30.677083333333332
},
"status": "available",
"stddev_across_gpu_mean_util_pct": 13.337857305429534
},
"latency_stats_s": {
"count": 198.0,
"mean": 13.01780862723021,
"p50": 5.048548387829214,
"p90": 30.18109704903327,
"p99": 119.01174414204434
},
"metrics_jsonl_rows": 200,
"metrics_summary_available": true,
"prefix_cache_hit_ratio": 0.0,
"request_count": 200,
"run": "outputs/gpu_ab_combined",
"session_summary": {
"request_cached_tokens": {
"count": 200,
"max": 0.0,
"mean": 0.0,
"p50": 0.0,
"p90": 0.0,
"p95": 0.0,
"p99": 0.0
},
"request_input_tokens": {
"count": 200,
"max": 111927.0,
"mean": 29318.375,
"p50": 21376.0,
"p90": 81218.19999999998,
"p95": 87571.2,
"p99": 101619.27999999994
},
"request_output_tokens": {
"count": 200,
"max": 5083.0,
"mean": 257.675,
"p50": 72.0,
"p90": 664.3,
"p95": 1063.9999999999993,
"p99": 3300.1799999999994
},
"session_count": 145,
"session_input_tokens": {
"count": 145,
"max": 1567423.0,
"mean": 40439.137931034486,
"p50": 10879.0,
"p90": 72438.39999999997,
"p95": 106934.39999999988,
"p99": 374927.28
},
"status": "available",
"top_session_input_fraction": {
"top_10pct": 0.6588294883328288,
"top_1pct": 0.33276622595897626,
"top_5pct": 0.5534430199490934
},
"turns_per_session": {
"count": 145,
"max": 21.0,
"mean": 1.3793103448275863,
"p50": 1.0,
"p90": 1.0,
"p95": 2.0,
"p99": 10.560000000000002
}
},
"success_count": 198,
"tpot_stats_s": {
"count": 198.0,
"mean": 0.04929455188644857,
"p50": 0.03717198147904128,
"p90": 0.07317714408040046,
"p99": 0.10039294634234945
},
"ttft_stats_s": {
"count": 198.0,
"mean": 3.8987488178453984,
"p50": 1.0068706551101059,
"p90": 9.355209570843726,
"p99": 33.855437273858115
},
"wall_clock_s": 483.543720243033
},
{
"apc_summary": {
"reason": "apc.txt missing",
"status": "unavailable"
},
"artifact_availability": {
"apc_txt": false,
"breakdown_json": false,
"gpu_util_csv": true,
"metrics_jsonl": true,
"metrics_summary_json": true
},
"breakdown_summary": {
"reason": "breakdown.json missing",
"status": "unavailable"
},
"error_count": 13,
"exists": true,
"external_cache_hit_ratio": null,
"gpu_summary": {
"gpu_count": 8,
"max_mean_util_pct": 26.737556561085974,
"max_min_ratio": 11.149056603773586,
"mean_util_pct": 12.367081447963802,
"min_mean_util_pct": 2.3981900452488687,
"per_gpu_mean_util_pct": {
"0": 26.737556561085974,
"1": 14.44343891402715,
"2": 19.036199095022624,
"3": 7.4389140271493215,
"4": 2.3981900452488687,
"5": 8.841628959276019,
"6": 11.963800904977376,
"7": 8.076923076923077
},
"status": "available",
"stddev_across_gpu_mean_util_pct": 7.15857427072345
},
"latency_stats_s": {
"count": 187.0,
"mean": 15.366858840781827,
"p50": 7.111906730104238,
"p90": 34.77056052000262,
"p99": 119.2064151619561
},
"metrics_jsonl_rows": 200,
"metrics_summary_available": true,
"prefix_cache_hit_ratio": 0.0,
"request_count": 200,
"run": "outputs/gpu_ab_pdsep",
"session_summary": {
"request_cached_tokens": {
"count": 200,
"max": 0.0,
"mean": 0.0,
"p50": 0.0,
"p90": 0.0,
"p95": 0.0,
"p99": 0.0
},
"request_input_tokens": {
"count": 200,
"max": 111927.0,
"mean": 29318.375,
"p50": 21376.0,
"p90": 81218.19999999998,
"p95": 87571.2,
"p99": 101619.27999999994
},
"request_output_tokens": {
"count": 200,
"max": 5097.0,
"mean": 290.18,
"p50": 75.5,
"p90": 685.8,
"p95": 1102.5999999999997,
"p99": 3433.6599999999844
},
"session_count": 145,
"session_input_tokens": {
"count": 145,
"max": 1567423.0,
"mean": 40439.137931034486,
"p50": 10879.0,
"p90": 72438.39999999997,
"p95": 106934.39999999988,
"p99": 374927.28
},
"status": "available",
"top_session_input_fraction": {
"top_10pct": 0.6588294883328288,
"top_1pct": 0.33276622595897626,
"top_5pct": 0.5534430199490934
},
"turns_per_session": {
"count": 145,
"max": 21.0,
"mean": 1.3793103448275863,
"p50": 1.0,
"p90": 1.0,
"p95": 2.0,
"p99": 10.560000000000002
}
},
"success_count": 187,
"tpot_stats_s": {
"count": 187.0,
"mean": 0.05175559044923285,
"p50": 0.04020531923068981,
"p90": 0.07416366779122173,
"p99": 0.10770365060609463
},
"ttft_stats_s": {
"count": 187.0,
"mean": 5.3933139261814524,
"p50": 1.9942838260903955,
"p90": 13.546015257015824,
"p99": 39.30951087013818
},
"wall_clock_s": 1171.516998933861
},
{
"apc_summary": {
"reason": "apc.txt missing",
"status": "unavailable"
},
"artifact_availability": {
"apc_txt": false,
"breakdown_json": false,
"gpu_util_csv": true,
"metrics_jsonl": true,
"metrics_summary_json": true
},
"breakdown_summary": {
"reason": "breakdown.json missing",
"status": "unavailable"
},
"error_count": 2,
"exists": true,
"external_cache_hit_ratio": 0.0,
"gpu_summary": {
"gpu_count": 8,
"max_mean_util_pct": 40.095454545454544,
"max_min_ratio": 2.310775410408662,
"mean_util_pct": 23.030492424242425,
"min_mean_util_pct": 17.35151515151515,
"per_gpu_mean_util_pct": {
"0": 22.243939393939392,
"1": 22.798484848484847,
"2": 22.09090909090909,
"3": 40.095454545454544,
"4": 17.484848484848484,
"5": 20.225757575757576,
"6": 17.35151515151515,
"7": 21.953030303030303
},
"status": "available",
"stddev_across_gpu_mean_util_pct": 6.752783259358802
},
"latency_stats_s": {
"count": 498.0,
"mean": 21.908606036002105,
"p50": 5.798741589998826,
"p90": 51.00022796400299,
"p99": 201.6967467680006
},
"metrics_jsonl_rows": 500,
"metrics_summary_available": true,
"prefix_cache_hit_ratio": 0.0,
"request_count": 500,
"run": "outputs/contention_16s_ts10",
"session_summary": {
"request_cached_tokens": {
"count": 500,
"max": 0.0,
"mean": 0.0,
"p50": 0.0,
"p90": 0.0,
"p95": 0.0,
"p99": 0.0
},
"request_input_tokens": {
"count": 500,
"max": 134865.0,
"mean": 29190.97,
"p50": 17756.0,
"p90": 77365.70000000006,
"p95": 87790.89999999997,
"p99": 111360.44
},
"request_output_tokens": {
"count": 500,
"max": 16427.0,
"mean": 404.286,
"p50": 75.0,
"p90": 863.7000000000004,
"p95": 2384.45,
"p99": 4928.559999999999
},
"session_count": 347,
"session_input_tokens": {
"count": 347,
"max": 1929072.0,
"mean": 42061.916426512966,
"p50": 11342.0,
"p90": 68297.20000000004,
"p95": 116213.79999999993,
"p99": 521898.36000000057
},
"status": "available",
"top_session_input_fraction": {
"top_10pct": 0.6824335059780473,
"top_1pct": 0.32352388426969025,
"top_5pct": 0.579125805000656
},
"turns_per_session": {
"count": 347,
"max": 29.0,
"mean": 1.440922190201729,
"p50": 1.0,
"p90": 1.0,
"p95": 3.0,
"p99": 10.54000000000002
}
},
"success_count": 498,
"tpot_stats_s": {
"count": 498.0,
"mean": 0.07772806444638436,
"p50": 0.06774563665097065,
"p90": 0.10325221968416468,
"p99": 0.503443661631568
},
"ttft_stats_s": {
"count": 498.0,
"mean": 3.5127640336366928,
"p50": 0.8264882279981975,
"p90": 9.71474659699743,
"p99": 34.40133859900379
},
"wall_clock_s": 1472.1331836190002
},
{
"apc_summary": {
"line_count": 8,
"preview": "inst_0: prefix=45.7% ext=24.7%\ninst_1: prefix=33.1% ext=13.3%\ninst_2: prefix=14.2% ext=0.0%\ninst_3: prefix=50.7% ext=6.3%\ninst_4: prefix=33.6% ext=0.0%\ninst_5: prefix=66.6% ext=9.2%\ninst_6: prefix=42.1% ext=18.8%\ninst_7: prefix=36.7% ext=7.9%",
"status": "available"
},
"artifact_availability": {
"apc_txt": true,
"breakdown_json": true,
"gpu_util_csv": true,
"metrics_jsonl": true,
"metrics_summary_json": true
},
"breakdown_summary": {
"field_sample": [
"cache_hit",
"estimated_new_tokens",
"input_length",
"policy",
"request_id",
"route_class",
"routed_to",
"t_done",
"t_first_token",
"t_proxy_recv"
],
"mode_counts": {
"HEAVY_COLO": 111,
"HEAVY_OFFLOAD": 13,
"MEDIUM": 163,
"WARM": 213
},
"route_counts": {
"linear": 487
},
"row_count": 500,
"status": "available"
},
"error_count": 2,
"exists": true,
"external_cache_hit_ratio": 0.0,
"gpu_summary": {
"gpu_count": 8,
"max_mean_util_pct": 47.54385964912281,
"max_min_ratio": 2.600767754318618,
"mean_util_pct": 26.349561403508773,
"min_mean_util_pct": 18.280701754385966,
"per_gpu_mean_util_pct": {
"0": 22.410526315789475,
"1": 26.57894736842105,
"2": 23.54035087719298,
"3": 47.54385964912281,
"4": 18.96842105263158,
"5": 26.403508771929825,
"6": 18.280701754385966,
"7": 27.07017543859649
},
"status": "available",
"stddev_across_gpu_mean_util_pct": 8.6079068381278
},
"latency_stats_s": {
"count": 498.0,
"mean": 21.780466573040204,
"p50": 6.467846095998539,
"p90": 48.40937389100145,
"p99": 196.93401125499804
},
"metrics_jsonl_rows": 500,
"metrics_summary_available": true,
"prefix_cache_hit_ratio": 0.0,
"request_count": 500,
"run": "outputs/contention_16s_elastic",
"session_summary": {
"request_cached_tokens": {
"count": 500,
"max": 0.0,
"mean": 0.0,
"p50": 0.0,
"p90": 0.0,
"p95": 0.0,
"p99": 0.0
},
"request_input_tokens": {
"count": 500,
"max": 134865.0,
"mean": 29190.97,
"p50": 17756.0,
"p90": 77365.70000000006,
"p95": 87790.89999999997,
"p99": 111360.44
},
"request_output_tokens": {
"count": 500,
"max": 16427.0,
"mean": 395.466,
"p50": 75.0,
"p90": 809.8000000000013,
"p95": 2264.2999999999943,
"p99": 4928.559999999999
},
"session_count": 347,
"session_input_tokens": {
"count": 347,
"max": 1929072.0,
"mean": 42061.916426512966,
"p50": 11342.0,
"p90": 68297.20000000004,
"p95": 116213.79999999993,
"p99": 521898.36000000057
},
"status": "available",
"top_session_input_fraction": {
"top_10pct": 0.6824335059780473,
"top_1pct": 0.32352388426969025,
"top_5pct": 0.579125805000656
},
"turns_per_session": {
"count": 347,
"max": 29.0,
"mean": 1.440922190201729,
"p50": 1.0,
"p90": 1.0,
"p95": 3.0,
"p99": 10.54000000000002
}
},
"success_count": 498,
"tpot_stats_s": {
"count": 498.0,
"mean": 0.08306021258238128,
"p50": 0.06973490711122092,
"p90": 0.117326519391297,
"p99": 0.5879216773072795
},
"ttft_stats_s": {
"count": 498.0,
"mean": 3.757858794331448,
"p50": 0.9292503809992922,
"p90": 11.017744456999935,
"p99": 34.20241540799907
},
"wall_clock_s": 1463.8220696580029
},
{
"apc_summary": {
"reason": "apc.txt missing",
"status": "unavailable"
},
"artifact_availability": {
"apc_txt": false,
"breakdown_json": false,
"gpu_util_csv": false,
"metrics_jsonl": true,
"metrics_summary_json": true
},
"breakdown_summary": {
"reason": "breakdown.json missing",
"status": "unavailable"
},
"error_count": 2,
"exists": true,
"external_cache_hit_ratio": null,
"gpu_summary": {
"reason": "gpu_util.csv missing",
"status": "unavailable"
},
"latency_stats_s": {
"count": 998.0,
"mean": 15.744418202954922,
"p50": 3.220371481962502,
"p90": 28.005282410187647,
"p99": 204.00479479599744
},
"metrics_jsonl_rows": 1000,
"metrics_summary_available": true,
"prefix_cache_hit_ratio": 0.5443149471989938,
"request_count": 1000,
"run": "outputs/combined_1000req",
"session_summary": {
"request_cached_tokens": {
"count": 1000,
"max": 0.0,
"mean": 0.0,
"p50": 0.0,
"p90": 0.0,
"p95": 0.0,
"p99": 0.0
},
"request_input_tokens": {
"count": 1000,
"max": 171427.0,
"mean": 31611.14,
"p50": 19798.0,
"p90": 82584.20000000001,
"p95": 93305.64999999997,
"p99": 111947.83999999998
},
"request_output_tokens": {
"count": 1000,
"max": 41233.0,
"mean": 392.579,
"p50": 70.0,
"p90": 685.8000000000002,
"p95": 1834.6999999999975,
"p99": 4584.459999999996
},
"session_count": 671,
"session_input_tokens": {
"count": 671,
"max": 2320064.0,
"mean": 47110.49180327869,
"p50": 12394.0,
"p90": 71875.0,
"p95": 112984.5,
"p99": 760591.4999999925
},
"status": "available",
"top_session_input_fraction": {
"top_10pct": 0.6915149849072194,
"top_1pct": 0.36223103627392117,
"top_5pct": 0.5945521736957288
},
"turns_per_session": {
"count": 671,
"max": 36.0,
"mean": 1.4903129657228018,
"p50": 1.0,
"p90": 1.0,
"p95": 3.0,
"p99": 12.599999999999909
}
},
"success_count": 998,
"tpot_stats_s": {
"count": 998.0,
"mean": 0.05681003092240363,
"p50": 0.04029440155459775,
"p90": 0.11343103184021618,
"p99": 0.2962149563411783
},
"ttft_stats_s": {
"count": 998.0,
"mean": 1.0351265607308877,
"p50": 0.39304836583323777,
"p90": 2.5655168020166457,
"p99": 8.210839052917436
},
"wall_clock_s": 1340.061015482992
},
{
"apc_summary": {
"reason": "apc.txt missing",
"status": "unavailable"
},
"artifact_availability": {
"apc_txt": false,
"breakdown_json": false,
"gpu_util_csv": false,
"metrics_jsonl": true,
"metrics_summary_json": true
},
"breakdown_summary": {
"reason": "breakdown.json missing",
"status": "unavailable"
},
"error_count": 204,
"exists": true,
"external_cache_hit_ratio": null,
"gpu_summary": {
"reason": "gpu_util.csv missing",
"status": "unavailable"
},
"latency_stats_s": {
"count": 796.0,
"mean": 23.561836449842698,
"p50": 9.752956213895231,
"p90": 63.928921481827274,
"p99": 168.2179710320197
},
"metrics_jsonl_rows": 1000,
"metrics_summary_available": true,
"prefix_cache_hit_ratio": 0.0,
"request_count": 1000,
"run": "outputs/exp3_pd_sep_tp1_mooncake",
"session_summary": {
"request_cached_tokens": {
"count": 1000,
"max": 0.0,
"mean": 0.0,
"p50": 0.0,
"p90": 0.0,
"p95": 0.0,
"p99": 0.0
},
"request_input_tokens": {
"count": 1000,
"max": 171427.0,
"mean": 31611.14,
"p50": 19798.0,
"p90": 82584.20000000001,
"p95": 93305.64999999997,
"p99": 111947.83999999998
},
"request_output_tokens": {
"count": 1000,
"max": 41233.0,
"mean": 412.176,
"p50": 74.0,
"p90": 759.7000000000008,
"p95": 1834.6999999999975,
"p99": 4928.559999999999
},
"session_count": 671,
"session_input_tokens": {
"count": 671,
"max": 2320064.0,
"mean": 47110.49180327869,
"p50": 12394.0,
"p90": 71875.0,
"p95": 112984.5,
"p99": 760591.4999999925
},
"status": "available",
"top_session_input_fraction": {
"top_10pct": 0.6915149849072194,
"top_1pct": 0.36223103627392117,
"top_5pct": 0.5945521736957288
},
"turns_per_session": {
"count": 671,
"max": 36.0,
"mean": 1.4903129657228018,
"p50": 1.0,
"p90": 1.0,
"p95": 3.0,
"p99": 12.599999999999909
}
},
"success_count": 796,
"tpot_stats_s": {
"count": 796.0,
"mean": 0.05211825330315642,
"p50": 0.06622354774330945,
"p90": 0.07391575907026088,
"p99": 0.10499966285609896
},
"ttft_stats_s": {
"count": 796.0,
"mean": 10.753033336598003,
"p50": 3.4666219488717616,
"p90": 29.00794132403098,
"p99": 81.7531874559354
},
"wall_clock_s": 2937.2794416199904
}
]

264
analysis/characterization/protocols.md Normal file
View File

@@ -0,0 +1,264 @@
# Characterization Protocols For Remaining Batches
Status: implementation protocol and audit checklist
Date: 2026-05-25
This file completes the `analysis/characterization` scaffold for the TODO
list. It separates what is already implemented from what requires fresh GPU
runs or new engine/proxy instrumentation.
## Implemented Now
### Batch 0/1 Analyzer
Use:
```bash
python3 analysis/characterization/analyze.py \
--trace traces/w600_r0.0015_st30.jsonl \
--kv-bytes-per-token 98304 \
--task-name w600_local_full_trace \
--overwrite
```
The analyzer writes:
- `manifest.json`
- `summary.json`
- `summary.md`
- `audit.md`
- `session_concurrency.json`
- `session_arrival_stats.json`
- `turn_interval_stats.json`
- `trace_profile.json`
- `workload_summary.json`
- `kv_footprint_summary.json`
- `reuse_decomposition.json`
- `session_skew.json`
- `append_delta_stats.json`
Limitations:
- Actual online sequentiality requires dispatch and finish/error timestamps.
Existing `metrics.jsonl` artifacts generally do not contain these fields.
- Actual reuse decomposition requires `cached_tokens`/`cache_hit`, `hash_ids`,
and `session_id` in the same joinable request record.
### Existing-Run Audit
Use:
```bash
python3 analysis/characterization/summarize_runs.py
```
The script writes an audit package under:
```text
analysis/characterization/current_results/
```
It summarizes already completed runs and explicitly marks which claims are
supported, partially supported, or not yet supported.
## Batch 2 Protocol: PD-Colo Prefill/Decode Interference
Purpose:
Prove whether same-worker prefill overlap increases decode TPOT/queue delay.
Required new instrumentation:
- per-request dispatch timestamp
- per-request finish/error timestamp
- per decode step timestamp
- decode step worker id
- prefill chunk start/end timestamp
- prefill worker id
- request/session id associated with each prefill chunk
Required arms:
1. decode-only steady load
2. decode + same-worker heavy prefill injection
3. decode + different-worker heavy prefill injection
4. trace replay with overlap labels
Required sweep:
```text
uncached_prefill_tokens in {2k, 8k, 16k, 32k, 64k}
chunked_prefill_size in available engine values
```
Required outputs:
- `interference_microbench_summary.json`
- `decode_step_timeseries.csv`
- `prefill_overlap_events.jsonl`
- `interference_index.json`
- TPOT timeline figure with prefill overlays
- same-worker vs different-worker TPOT boxplot
Pass condition:
```text
TPOT_p90(overlap_same_worker) / TPOT_p90(no_overlap) > 1
```
and the effect must be materially weaker in the different-worker control.
## Batch 3 Protocol: Session Hot-Spot Residual Imbalance
Purpose:
Prove whether cache-aware/LMetric still leaves hot workers under
session-heavy skew.
Required new instrumentation:
- route decision per request
- chosen worker
- candidate worker scores
- cache hit / estimated uncached tokens per candidate
- per-worker request queue length/delay
- per-worker decode queue length/delay
- per-worker KV occupancy
- per-worker APC/cache-hit snapshot
Required arms:
1. corrected LMetric/cache-aware
2. load-only routing
3. hard sticky routing
4. current Unified hybrid
5. session-mass capped/equalized replay
Required outputs:
- `worker_balance_summary.json`
- `session_to_worker_map.json`
- `session_mass_summary.json`
- `routing_policy_comparison.json`
- `hotspot_index.json`
- per-worker queue delay bar
- APC vs queue delay scatter
- top-session contribution bar
- policy tradeoff plot: APC vs hot-spot index
Pass condition:
LMetric/cache-aware must show measurable residual worker skew, and that skew
must correlate with session token mass or locality.
GPU utilization alone is not enough for this claim.
## Batch 4 Protocol: Sustainable Request Rate
Purpose:
Measure:
```text
SRR(SLO) = max arrival rate satisfying SLO in steady state
```
Required load generator behavior:
- open-loop session arrivals, preferably Poisson
- session-internal sequentiality
- warmup window
- steady-state measurement window
- explicit attempted/completed/error counters
Provisional SLO:
```text
TTFT_p90 <= T_ttft
E2E_p90 <= T_e2e
TPOT_p90 <= T_tpot
error_rate <= epsilon
queue length stable
KV occupancy stable
```
Required arms:
1. PD-colo corrected LMetric/cache-aware
2. static PD-disagg
3. current Unified hybrid
4. optional hard sticky
5. optional load-only
Required outputs:
- `srr_curve.json`
- `lambda_runs/<lambda>/summary.json`
- `slo_violation_reason.json`
- `goodput_vs_arrival_rate.json`
- SRR bar chart
- latency vs arrival rate curves
- goodput vs arrival rate
- queue/KV stability plot near failure point
Pass condition:
Each policy has a measured max sustainable lambda under the same SLO and
same session-causal arrival process.
## Batch 5 Protocol: Failure Attribution Near SRR Boundary
Purpose:
Explain why each policy fails near SRR.
Required rates:
```text
lambda = 0.9 * SRR
lambda = 1.0 * SRR
lambda = 1.1 * SRR
```
Labels for each slow/SLO-violating request:
- same-worker prefill overlap
- hot worker queue
- high KV occupancy
- cache miss / large uncached append
- transfer wait
- P queue wait
- D admission wait
- unknown
Required outputs:
- `slow_request_attribution.jsonl`
- `failure_breakdown.json`
- `case_studies.md`
- `worker_failure_windows.json`
- violation cause stacked bar
- slow request waterfall
- worker timeline near failure
Pass condition:
The analysis must explain whether PD-colo is limited by interference,
hot-spot, KV pressure, or a mixture, and whether Unified/PUSH underperforms
because of trigger quality, transfer cost, target admission, or load regime.
## Batch 6 Protocol: Audit Package
Implemented by `summarize_runs.py` for existing runs and extended by fresh
Batch 2-5 outputs later.
Required files:
- `characterization_claim_matrix.md`
- `all_figures_index.md`
- `reviewer_risk_register.md`
- `reproduction_commands.sh`
- `main_claim_allowed_runs.md`
Current package intentionally marks Batch 2/4/5 claims as not yet supported
until fresh instrumented experiments exist.

666
analysis/characterization/summarize_runs.py Normal file
View File

@@ -0,0 +1,666 @@
#!/usr/bin/env python3
"""Summarize existing benchmark artifacts for characterization review.
This is a CPU-only companion to ``analyze.py``. It does not run benchmarks.
It reads completed output directories and produces an audit-oriented package
that helps decide which TODO claims are currently supported by existing data
and which still need fresh GPU runs or additional instrumentation.
"""
from __future__ import annotations
import argparse
import csv
import datetime as dt
import json
import math
import statistics
import subprocess
from pathlib import Path
from typing import Any
JsonDict = dict[str, Any]
DEFAULT_RUNS = [
"outputs/gpu_ab_combined",
"outputs/gpu_ab_pdsep",
"outputs/contention_16s_ts10",
"outputs/contention_16s_elastic",
"outputs/combined_1000req",
"outputs/exp3_pd_sep_tp1_mooncake",
]
def main() -> None:
args = parse_args()
out_dir = args.output_dir
out_dir.mkdir(parents=True, exist_ok=True)
run_dirs = [Path(p) for p in (args.runs or DEFAULT_RUNS)]
summaries = [summarize_run(path) for path in run_dirs]
comparisons = build_comparisons(summaries)
claim_matrix = build_claim_matrix(summaries, comparisons)
risk_register = build_risk_register(summaries)
write_json(out_dir / "run_summaries.json", summaries)
write_json(out_dir / "comparisons.json", comparisons)
write_json(out_dir / "claim_matrix.json", claim_matrix)
write_json(out_dir / "reviewer_risk_register.json", risk_register)
(out_dir / "current_results.md").write_text(
render_current_results(summaries, comparisons, claim_matrix, risk_register),
encoding="utf-8",
)
(out_dir / "characterization_claim_matrix.md").write_text(
render_claim_matrix(claim_matrix),
encoding="utf-8",
)
(out_dir / "reviewer_risk_register.md").write_text(
render_risk_register(risk_register),
encoding="utf-8",
)
(out_dir / "all_figures_index.md").write_text(
render_figures_index(summaries),
encoding="utf-8",
)
(out_dir / "reproduction_commands.sh").write_text(
render_reproduction_commands(args, run_dirs),
encoding="utf-8",
)
print(f"Wrote run summary package to {out_dir}")
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Summarize existing characterization-relevant output dirs.",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
parser.add_argument(
"--runs",
nargs="*",
default=None,
help="Output directories to summarize. Defaults to a small curated set.",
)
parser.add_argument(
"--output-dir",
type=Path,
default=Path("analysis/characterization/current_results"),
help="Directory for generated review artifacts.",
)
return parser.parse_args()
def summarize_run(path: Path) -> JsonDict:
metrics_summary = load_json(path / "metrics.summary.json")
metrics_rows = load_jsonl(path / "metrics.jsonl")
gpu_summary = summarize_gpu(path / "gpu_util.csv")
breakdown_summary = summarize_breakdown(path / "breakdown.json")
apc_summary = summarize_apc(path / "apc.txt")
return {
"run": str(path),
"exists": path.exists(),
"metrics_summary_available": bool(metrics_summary),
"metrics_jsonl_rows": len(metrics_rows),
"request_count": first_present(metrics_summary, ["request_count"]),
"success_count": first_present(metrics_summary, ["success_count"]),
"error_count": first_present(metrics_summary, ["error_count"]),
"wall_clock_s": first_present(metrics_summary, ["wall_clock_s"]),
"latency_stats_s": metrics_summary.get("latency_stats_s"),
"ttft_stats_s": metrics_summary.get("ttft_stats_s"),
"tpot_stats_s": metrics_summary.get("tpot_stats_s"),
"prefix_cache_hit_ratio": metrics_summary.get("prefix_cache_hit_ratio"),
"external_cache_hit_ratio": metrics_summary.get("external_cache_hit_ratio"),
"session_summary": summarize_sessions(metrics_rows),
"gpu_summary": gpu_summary,
"breakdown_summary": breakdown_summary,
"apc_summary": apc_summary,
"artifact_availability": {
"metrics_summary_json": (path / "metrics.summary.json").exists(),
"metrics_jsonl": (path / "metrics.jsonl").exists(),
"gpu_util_csv": (path / "gpu_util.csv").exists(),
"breakdown_json": (path / "breakdown.json").exists(),
"apc_txt": (path / "apc.txt").exists(),
},
}
def summarize_sessions(rows: list[JsonDict]) -> JsonDict:
if not rows:
return {
"status": "unavailable",
"reason": "metrics.jsonl missing",
}
sessions: dict[str, JsonDict] = {}
input_values = []
output_values = []
cached_values = []
for row in rows:
sid = str(row.get("session_id", ""))
item = sessions.setdefault(
sid,
{
"turns": 0,
"input_tokens": 0.0,
"output_tokens": 0.0,
"cached_tokens": 0.0,
},
)
inp = to_float(row.get("input_length")) or 0.0
out = to_float(row.get("actual_output_tokens")) or to_float(row.get("output_length")) or 0.0
cached = to_float(row.get("cached_tokens")) or 0.0
item["turns"] += 1
item["input_tokens"] += inp
item["output_tokens"] += out
item["cached_tokens"] += cached
input_values.append(inp)
output_values.append(out)
cached_values.append(cached)
per_session_input = [v["input_tokens"] for v in sessions.values()]
return {
"status": "available",
"request_input_tokens": stats(input_values),
"request_output_tokens": stats(output_values),
"request_cached_tokens": stats(cached_values),
"session_count": len(sessions),
"turns_per_session": stats([v["turns"] for v in sessions.values()]),
"session_input_tokens": stats(per_session_input),
"top_session_input_fraction": top_contribution(per_session_input),
}
def summarize_gpu(path: Path) -> JsonDict:
if not path.exists():
return {
"status": "unavailable",
"reason": "gpu_util.csv missing",
}
values: dict[str, list[float]] = {}
with path.open() as handle:
reader = csv.DictReader(handle)
for row in reader:
gpu = str(row.get("gpu", ""))
util = to_float(row.get("util_pct"))
if gpu and util is not None:
values.setdefault(gpu, []).append(util)
means = {gpu: statistics.fmean(vals) for gpu, vals in values.items() if vals}
if not means:
return {
"status": "unavailable",
"reason": "gpu_util.csv had no util_pct rows",
}
mean_values = list(means.values())
return {
"status": "available",
"gpu_count": len(means),
"per_gpu_mean_util_pct": means,
"mean_util_pct": statistics.fmean(mean_values),
"stddev_across_gpu_mean_util_pct": statistics.pstdev(mean_values),
"max_mean_util_pct": max(mean_values),
"min_mean_util_pct": min(mean_values),
"max_min_ratio": max(mean_values) / max(min(mean_values), 1e-9),
}
def summarize_breakdown(path: Path) -> JsonDict:
if not path.exists():
return {
"status": "unavailable",
"reason": "breakdown.json missing",
}
try:
data = json.loads(path.read_text(encoding="utf-8"))
except Exception as exc:
return {
"status": "unavailable",
"reason": f"failed to parse breakdown: {exc}",
}
rows: list[JsonDict]
if isinstance(data, list):
rows = [r for r in data if isinstance(r, dict)]
elif isinstance(data, dict):
rows = data.get("records") if isinstance(data.get("records"), list) else []
if not rows:
rows = [data]
else:
rows = []
mode_counts: dict[str, int] = {}
route_counts: dict[str, int] = {}
for row in rows:
mode = row.get("mode") or row.get("execution_mode") or row.get("route_class")
route = row.get("route") or row.get("decision") or row.get("policy")
if mode is not None:
mode_counts[str(mode)] = mode_counts.get(str(mode), 0) + 1
if route is not None:
route_counts[str(route)] = route_counts.get(str(route), 0) + 1
return {
"status": "available",
"row_count": len(rows),
"mode_counts": mode_counts,
"route_counts": route_counts,
"field_sample": sorted(rows[0].keys()) if rows else [],
}
def summarize_apc(path: Path) -> JsonDict:
if not path.exists():
return {
"status": "unavailable",
"reason": "apc.txt missing",
}
text = path.read_text(encoding="utf-8", errors="replace")
return {
"status": "available",
"line_count": len(text.splitlines()),
"preview": "\n".join(text.splitlines()[:20]),
}
def build_comparisons(summaries: list[JsonDict]) -> list[JsonDict]:
by_run = {s["run"]: s for s in summaries}
pairs = [
("combined_vs_pdsep_200", "outputs/gpu_ab_combined", "outputs/gpu_ab_pdsep"),
("contention_baseline_vs_elastic_500", "outputs/contention_16s_ts10", "outputs/contention_16s_elastic"),
("combined_1000_vs_pdsep_mooncake", "outputs/combined_1000req", "outputs/exp3_pd_sep_tp1_mooncake"),
]
out = []
for name, base, variant in pairs:
if base not in by_run or variant not in by_run:
continue
out.append(compare_pair(name, by_run[base], by_run[variant]))
return out
def compare_pair(name: str, base: JsonDict, variant: JsonDict) -> JsonDict:
return {
"name": name,
"baseline": base["run"],
"variant": variant["run"],
"request_count": [base.get("request_count"), variant.get("request_count")],
"success_count": [base.get("success_count"), variant.get("success_count")],
"error_count": [base.get("error_count"), variant.get("error_count")],
"ttft_p50_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p50"), stat_value(variant, "ttft_stats_s", "p50")),
"ttft_p90_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p90"), stat_value(variant, "ttft_stats_s", "p90")),
"e2e_p50_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p50"), stat_value(variant, "latency_stats_s", "p50")),
"e2e_p90_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p90"), stat_value(variant, "latency_stats_s", "p90")),
"tpot_p90_delta_pct": pct_delta(stat_value(base, "tpot_stats_s", "p90"), stat_value(variant, "tpot_stats_s", "p90")),
"wall_clock_delta_pct": pct_delta(base.get("wall_clock_s"), variant.get("wall_clock_s")),
"gpu_mean_util": [
nested(base, ["gpu_summary", "mean_util_pct"]),
nested(variant, ["gpu_summary", "mean_util_pct"]),
],
"gpu_imbalance_ratio": [
nested(base, ["gpu_summary", "max_min_ratio"]),
nested(variant, ["gpu_summary", "max_min_ratio"]),
],
}
def build_claim_matrix(summaries: list[JsonDict], comparisons: list[JsonDict]) -> list[JsonDict]:
has_gpu = any((s.get("gpu_summary") or {}).get("status") == "available" for s in summaries)
has_metrics = any(s.get("metrics_summary_available") for s in summaries)
return [
{
"claim": "Batch 0 substrate audit is only partially complete for existing runs.",
"status": "partially_supported",
"supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.",
"needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.",
"reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.",
},
{
"claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.",
"status": "supported_for_trace_shape",
"supporting_data": "analysis/characterization/analyze.py outputs workload_summary/session_skew/KV footprint when given trace and kv_bytes_per_token.",
"needed_next": "Run on dash0 compact formatted trace for canonical full-trace numbers.",
"reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.",
},
{
"claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.",
"status": "supported_by_existing_artifact" if has_metrics else "unavailable",
"supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.",
"needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.",
"reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.",
},
{
"claim": "Elastic transfer-based migration does not improve high-contention 500-request run.",
"status": "supported_by_existing_artifact" if has_metrics else "unavailable",
"supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.",
"needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.",
"reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.",
},
{
"claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.",
"status": "not_yet_supported",
"supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs.",
"needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.",
"reviewer_risk": "Cannot claim interference as causal without Batch 2.",
},
{
"claim": "Session hot-spot residual imbalance is suggested but not fully attributed.",
"status": "partially_supported" if has_gpu else "unavailable",
"supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs.",
"needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.",
"reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.",
},
{
"claim": "SRR is not measured by existing fixed-request runs.",
"status": "not_yet_supported",
"supporting_data": "No arrival-rate sweep artifacts found.",
"needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.",
"reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.",
},
]
def build_risk_register(summaries: list[JsonDict]) -> list[JsonDict]:
return [
{
"risk": "Session sequentiality not proven",
"severity": "high",
"evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.",
"mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.",
},
{
"risk": "Legacy PD-sep data may not match final methodology",
"severity": "medium",
"evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.",
"mitigation": "Use fresh PD matrix for paper-grade claims.",
},
{
"risk": "GPU util is not a sufficient hot-spot proof",
"severity": "medium",
"evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.",
"mitigation": "Add route-decision and per-worker queue logs for Batch 3.",
},
{
"risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data",
"severity": "medium",
"evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.",
"mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.",
},
]
def render_current_results(
summaries: list[JsonDict],
comparisons: list[JsonDict],
claim_matrix: list[JsonDict],
risk_register: list[JsonDict],
) -> str:
lines = [
"# Current Characterization Results",
"",
f"Generated: {dt.datetime.now(dt.timezone.utc).isoformat()}",
f"Git commit: `{git_commit()}`",
"",
"## Existing Run Summaries",
"",
"| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |",
"|---|---:|---:|---:|---:|---:|---:|",
]
for s in summaries:
lines.append(
"| {run} | {ok}/{req} | {ttft50}/{ttft90} | {e2e50}/{e2e90} | {tpot90} | {gpu_mean} | {gpu_imb} |".format(
run=s["run"],
ok=fmt(s.get("success_count")),
req=fmt(s.get("request_count")),
ttft50=fmt(stat_value(s, "ttft_stats_s", "p50")),
ttft90=fmt(stat_value(s, "ttft_stats_s", "p90")),
e2e50=fmt(stat_value(s, "latency_stats_s", "p50")),
e2e90=fmt(stat_value(s, "latency_stats_s", "p90")),
tpot90=fmt(stat_value(s, "tpot_stats_s", "p90")),
gpu_mean=fmt(nested(s, ["gpu_summary", "mean_util_pct"])),
gpu_imb=fmt(nested(s, ["gpu_summary", "max_min_ratio"])),
)
)
lines.extend([
"",
"## Pairwise Comparisons",
"",
"| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |",
"|---|---:|---:|---:|---:|---:|---:|",
])
for c in comparisons:
lines.append(
"| {name} | {ttft50} | {ttft90} | {e2e50} | {e2e90} | {tpot90} | {wall} |".format(
name=c["name"],
ttft50=fmt_pct(c.get("ttft_p50_delta_pct")),
ttft90=fmt_pct(c.get("ttft_p90_delta_pct")),
e2e50=fmt_pct(c.get("e2e_p50_delta_pct")),
e2e90=fmt_pct(c.get("e2e_p90_delta_pct")),
tpot90=fmt_pct(c.get("tpot_p90_delta_pct")),
wall=fmt_pct(c.get("wall_clock_delta_pct")),
)
)
lines.extend([
"",
"## What We Can Say Now",
"",
])
for item in claim_matrix:
lines.append(f"- **{item['status']}**: {item['claim']}")
lines.append(f" Supporting data: {item['supporting_data']}")
lines.append(f" Next: {item['needed_next']}")
lines.extend([
"",
"## Main Reviewer Risks",
"",
])
for item in risk_register:
lines.append(f"- **{item['severity']}**: {item['risk']} - {item['mitigation']}")
lines.append("")
return "\n".join(lines)
def render_claim_matrix(items: list[JsonDict]) -> str:
lines = [
"# Characterization Claim Matrix",
"",
"| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |",
"|---|---|---|---|---|",
]
for item in items:
lines.append(
f"| {item['claim']} | `{item['status']}` | {item['supporting_data']} | {item['needed_next']} | {item['reviewer_risk']} |"
)
lines.append("")
return "\n".join(lines)
def render_risk_register(items: list[JsonDict]) -> str:
lines = [
"# Reviewer Risk Register",
"",
"| Risk | Severity | Evidence | Mitigation |",
"|---|---|---|---|",
]
for item in items:
lines.append(
f"| {item['risk']} | `{item['severity']}` | {item['evidence']} | {item['mitigation']} |"
)
lines.append("")
return "\n".join(lines)
def render_figures_index(summaries: list[JsonDict]) -> str:
return "\n".join([
"# Figures Index",
"",
"No generated figures are committed by this script. Batch-specific figures should be generated from:",
"",
"- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.",
"- future Batch 2 step-timeline artifacts for interference plots.",
"- future Batch 3 per-worker/session artifacts for hot-spot plots.",
"- future Batch 4 arrival-rate sweep artifacts for SRR curves.",
"",
"This file exists so the audit package has a stable placeholder until fresh figures are generated.",
"",
])
def render_reproduction_commands(args: argparse.Namespace, run_dirs: list[Path]) -> str:
runs = " ".join(str(p) for p in run_dirs)
return "\n".join([
"#!/usr/bin/env bash",
"set -euo pipefail",
"",
"# Rebuild this current-results audit package.",
f"python3 analysis/characterization/summarize_runs.py --output-dir {args.output_dir} --runs {runs}",
"",
"# Example Batch 0/1 local trace analysis.",
"python3 analysis/characterization/analyze.py \\",
" --trace traces/w600_r0.0015_st30.jsonl \\",
" --kv-bytes-per-token 98304 \\",
" --task-name w600_local_full_trace \\",
" --overwrite",
"",
])
def load_json(path: Path) -> JsonDict:
if not path.exists():
return {}
try:
data = json.loads(path.read_text(encoding="utf-8"))
except Exception:
return {}
return data if isinstance(data, dict) else {}
def load_jsonl(path: Path) -> list[JsonDict]:
if not path.exists():
return []
rows = []
with path.open(encoding="utf-8") as handle:
for line in handle:
if not line.strip():
continue
try:
row = json.loads(line)
except Exception:
continue
if isinstance(row, dict):
rows.append(row)
return rows
def write_json(path: Path, data: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
def first_present(row: JsonDict, keys: list[str]) -> Any:
for key in keys:
if key in row:
return row[key]
return None
def stat_value(run: JsonDict, stat_key: str, value_key: str) -> float | None:
stats_obj = run.get(stat_key)
if not isinstance(stats_obj, dict):
return None
return to_float(stats_obj.get(value_key))
def nested(row: JsonDict, keys: list[str]) -> Any:
cur: Any = row
for key in keys:
if not isinstance(cur, dict):
return None
cur = cur.get(key)
return cur
def pct_delta(base: Any, variant: Any) -> float | None:
b = to_float(base)
v = to_float(variant)
if b is None or v is None or b == 0:
return None
return (v - b) / b * 100.0
def to_float(value: Any) -> float | None:
if value is None:
return None
try:
out = float(value)
except (TypeError, ValueError):
return None
return out if math.isfinite(out) else None
def stats(values: list[float]) -> JsonDict | None:
clean = sorted(float(v) for v in values if math.isfinite(float(v)))
if not clean:
return None
return {
"count": len(clean),
"mean": statistics.fmean(clean),
"p50": percentile(clean, 0.50),
"p90": percentile(clean, 0.90),
"p95": percentile(clean, 0.95),
"p99": percentile(clean, 0.99),
"max": clean[-1],
}
def percentile(values: list[float], q: float) -> float:
if len(values) == 1:
return values[0]
rank = q * (len(values) - 1)
lo = int(rank)
hi = min(lo + 1, len(values) - 1)
frac = rank - lo
return values[lo] * (1 - frac) + values[hi] * frac
def top_contribution(values: list[float]) -> JsonDict:
clean = sorted([v for v in values if math.isfinite(v)], reverse=True)
total = sum(clean)
if not clean or total <= 0:
return {"top_1pct": None, "top_5pct": None, "top_10pct": None}
def frac(pct: float) -> float:
k = max(1, math.ceil(len(clean) * pct))
return sum(clean[:k]) / total
return {
"top_1pct": frac(0.01),
"top_5pct": frac(0.05),
"top_10pct": frac(0.10),
}
def fmt(value: Any) -> str:
num = to_float(value)
if num is None:
return "n/a"
if abs(num - round(num)) < 1e-9 and abs(num) < 1_000_000:
return str(int(round(num)))
return f"{num:.3g}"
def fmt_pct(value: Any) -> str:
num = to_float(value)
if num is None:
return "n/a"
return f"{num:+.1f}%"
def git_commit() -> str:
try:
result = subprocess.run(
["git", "rev-parse", "HEAD"],
check=True,
stdout=subprocess.PIPE,
stderr=subprocess.DEVNULL,
text=True,
)
except Exception:
return ""
return result.stdout.strip()
if __name__ == "__main__":
main()