Add agentic workload characterization audit scaffold
This commit is contained in:
255
analysis/characterization/README.md
Normal file
255
analysis/characterization/README.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# Characterization Analyzer Runbook
|
||||
|
||||
CPU-only scaffold for Batch 0 and Batch 1 in
|
||||
`analysis/characterization_todo_for_interns.md`.
|
||||
|
||||
This directory has three components:
|
||||
|
||||
- `analyze.py`: Batch 0/1 analyzer for trace and per-request metrics.
|
||||
- `summarize_runs.py`: CPU-only audit of already completed benchmark
|
||||
directories.
|
||||
- `protocols.md`: exact protocol for Batch 2-6 experiments that require fresh
|
||||
GPU runs or additional instrumentation.
|
||||
|
||||
The analyzer reads existing trace and metrics artifacts and writes:
|
||||
|
||||
```text
|
||||
outputs/characterization/<date>/<task_name>/
|
||||
├── manifest.json
|
||||
├── raw/
|
||||
├── summary.json
|
||||
├── summary.md
|
||||
├── audit.md
|
||||
├── session_concurrency.json
|
||||
├── session_arrival_stats.json
|
||||
├── turn_interval_stats.json
|
||||
├── trace_profile.json
|
||||
├── invalid_runs.md
|
||||
├── workload_summary.json
|
||||
├── kv_footprint_summary.json
|
||||
├── reuse_decomposition.json
|
||||
├── session_skew.json
|
||||
├── append_delta_stats.json
|
||||
└── figures/
|
||||
```
|
||||
|
||||
If `matplotlib` is installed, simple PNG/PDF figures are emitted under
|
||||
`figures/`. If it is not installed, all JSON/Markdown data artifacts are still
|
||||
written.
|
||||
|
||||
## Canonical Data Sources
|
||||
|
||||
Canonical full traces live on dash0:
|
||||
|
||||
- formatted trace: `~/ali-trace/trace-glm5.1-formatted/`
|
||||
- raw unformatted trace: `~/ali-trace/trace-glm5.1/`
|
||||
|
||||
For the current GLM-5.1 characterization, prefer the compact formatted file:
|
||||
|
||||
```text
|
||||
~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
|
||||
```
|
||||
|
||||
Do not pass `051315-051317-raw.jsonl` or the files under
|
||||
`~/ali-trace/trace-glm5.1/` directly to this analyzer unless you first convert
|
||||
them to the formatted schema. Those raw files are tens to hundreds of GiB and
|
||||
contain full prompt payloads rather than the compact characterization schema.
|
||||
|
||||
The analyzer is CPU-only. For full trace characterization, either:
|
||||
|
||||
- run it on dash0 against the formatted JSONL files without starting any GPU
|
||||
service; or
|
||||
- copy/rsync the needed trace files from dash0 to this repository or another
|
||||
local path, then run the analyzer locally.
|
||||
|
||||
Only light directory/field inspection is needed on dash0 before choosing which
|
||||
trace file to analyze.
|
||||
|
||||
The raw unformatted directory is listed as a source option for provenance, but
|
||||
this analyzer expects formatted JSONL records. Raw files should be converted to
|
||||
the formatted schema before being passed to `--trace`.
|
||||
|
||||
## Inputs
|
||||
|
||||
Trace JSONL:
|
||||
|
||||
- Expected formatted fields: `chat_id`, `parent_chat_id`, `timestamp`,
|
||||
`input_length`, `output_length`, `type`, `turn`, `hash_ids`, optional
|
||||
`session_id`.
|
||||
- If `session_id` is absent, sessions are reconstructed from
|
||||
`parent_chat_id` chains.
|
||||
- `timestamp` is treated as scheduled trace time, not proof of actual dispatch
|
||||
time.
|
||||
|
||||
Metrics JSONL:
|
||||
|
||||
- Expected replayer fields: `request_id`, `session_id`, `turn_id`,
|
||||
`trace_timestamp_s`, `input_length`, `output_length`, `cached_tokens`,
|
||||
`latency_s`, `ttft_s`, `tpot_s`, `actual_output_tokens`, `error`.
|
||||
- If the metrics file is from the current replayer, it does not include actual
|
||||
dispatch/finish wall-clock timestamps. Batch 0 will therefore mark actual
|
||||
session sequentiality as unavailable and separately report a scheduled
|
||||
estimate from `trace_timestamp_s + latency_s`.
|
||||
|
||||
Proxy breakdown:
|
||||
|
||||
- Optional JSON/JSONL with fields such as `request_id`, `t_proxy_recv`,
|
||||
`t_first_token`, `t_done`, `cache_hit`, `estimated_new_tokens`,
|
||||
`route_class`, `routed_to`, `policy`.
|
||||
- Batch 0 can prove actual per-session in-flight concurrency only when these
|
||||
timing rows can be joined to analyzed requests by `request_id`.
|
||||
- Existing proxy breakdown artifacts may not contain `session_id`; without a
|
||||
request-id join to trace/metrics, they can still support append/cache-hit
|
||||
statistics but not per-session concurrency.
|
||||
|
||||
Run config:
|
||||
|
||||
- Optional JSON, usually `outputs/<run>/config.json`.
|
||||
- Used for manifest fields such as `policy`, `time_scale`, and request count
|
||||
when available.
|
||||
|
||||
## Commands
|
||||
|
||||
Trace-only dry run:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace traces/w600_r0.0015_st30.jsonl \
|
||||
--task-name w600_trace_only \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Trace plus replayer metrics:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace traces/w600_r0.0015_st30.jsonl \
|
||||
--metrics outputs/smoke_test/metrics.jsonl \
|
||||
--task-name smoke_trace_metrics \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Proxy breakdown append/cache analysis:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--breakdown outputs/contention_16s_elastic/breakdown.json \
|
||||
--config outputs/contention_16s_elastic/config.json \
|
||||
--task-name contention_breakdown \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Full trace on dash0, CPU-only:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
|
||||
--task-name full_trace_characterization \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Local run after copying from dash0:
|
||||
|
||||
```bash
|
||||
rsync -av dash0:~/ali-trace/trace-glm5.1-formatted/<trace-file>.jsonl traces/
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace traces/<trace-file>.jsonl \
|
||||
--task-name full_trace_characterization \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
By default the analyzer records file size and mtime but skips full SHA256
|
||||
hashing, because canonical raw trace files can be hundreds of GiB. Add
|
||||
`--hash-inputs` only when you intentionally want a full file hash.
|
||||
|
||||
KV footprint requires a model-specific value:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace traces/w600_r0.0015_st30.jsonl \
|
||||
--kv-bytes-per-token 98304 \
|
||||
--task-name w600_with_kv_estimate \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Summarize existing completed runs:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/summarize_runs.py
|
||||
```
|
||||
|
||||
This writes:
|
||||
|
||||
```text
|
||||
analysis/characterization/current_results/
|
||||
├── run_summaries.json
|
||||
├── comparisons.json
|
||||
├── claim_matrix.json
|
||||
├── reviewer_risk_register.json
|
||||
├── current_results.md
|
||||
├── characterization_claim_matrix.md
|
||||
├── all_figures_index.md
|
||||
├── reviewer_risk_register.md
|
||||
└── reproduction_commands.sh
|
||||
```
|
||||
|
||||
## Batch 0 Semantics
|
||||
|
||||
The online-serving invariant is:
|
||||
|
||||
```text
|
||||
Each session has at most one in-flight turn.
|
||||
```
|
||||
|
||||
The analyzer reports:
|
||||
|
||||
- actual interval status from dispatch and finish/error timestamps;
|
||||
- scheduled estimate from trace timestamps plus latency when available;
|
||||
- per-session max in-flight;
|
||||
- session start-time distribution;
|
||||
- turn inter-arrival distribution;
|
||||
- attempted/completed/error counts and goodput when metrics exist;
|
||||
- run classification.
|
||||
|
||||
Important limitation: trace timestamps alone cannot prove actual replay
|
||||
sequentiality. A run is only classified as `online_realistic` when actual
|
||||
per-request dispatch and finish/error timestamps prove
|
||||
`max_inflight_per_session <= 1`.
|
||||
|
||||
## Batch 1 Semantics
|
||||
|
||||
The analyzer reports:
|
||||
|
||||
- input/output CDF stats;
|
||||
- input/output ratio;
|
||||
- KV footprint CDF stats when `--kv-bytes-per-token` is supplied;
|
||||
- session skew and top-session contribution;
|
||||
- append/uncached token stats when `cached_tokens` or `cache_hit` exists;
|
||||
- reuse decomposition when both cached-token fields and `hash_ids` exist.
|
||||
|
||||
Reuse decomposition is conservative:
|
||||
|
||||
- `intra_session`: cached hash block was seen earlier in the same session;
|
||||
- `cross_session`: cached hash block was seen earlier in another session;
|
||||
- `shared/system-prefix`: early-position block appears in many sessions;
|
||||
- `unclassified`: cached tokens could not be mapped to a previously seen hash
|
||||
block.
|
||||
|
||||
If cached-token/cache-hit fields are absent, reuse and append artifacts are
|
||||
written with `status: "unavailable"` and list the required fields.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The script does not run a benchmark, query a live service, touch GPU state,
|
||||
or start any daemon.
|
||||
- Request-id joins are exact. If trace, metrics, and proxy artifacts use
|
||||
different request IDs, the unmatched rows are preserved under `raw/`.
|
||||
- Actual Batch 0 sequentiality needs actual dispatch and finish/error
|
||||
timestamps. Current `replayer/metrics.py` metrics are not enough by
|
||||
themselves.
|
||||
- `kv_bytes_per_token` depends on model architecture, layer count, KV heads,
|
||||
head dimension, and dtype. The analyzer will not guess it.
|
||||
- Shared/system-prefix reuse classification is a heuristic based on trace
|
||||
`hash_ids` positions and cross-session frequency. Adjust
|
||||
`--shared-prefix-min-sessions` and `--system-prefix-blocks` if the formatted
|
||||
trace provides a stronger system-prefix marker.
|
||||
1873
analysis/characterization/analyze.py
Normal file
1873
analysis/characterization/analyze.py
Normal file
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,10 @@
|
||||
# Figures Index
|
||||
|
||||
No generated figures are committed by this script. Batch-specific figures should be generated from:
|
||||
|
||||
- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.
|
||||
- future Batch 2 step-timeline artifacts for interference plots.
|
||||
- future Batch 3 per-worker/session artifacts for hot-spot plots.
|
||||
- future Batch 4 arrival-rate sweep artifacts for SRR curves.
|
||||
|
||||
This file exists so the audit package has a stable placeholder until fresh figures are generated.
|
||||
@@ -0,0 +1,11 @@
|
||||
# Characterization Claim Matrix
|
||||
|
||||
| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |
|
||||
|---|---|---|---|---|
|
||||
| Batch 0 substrate audit is only partially complete for existing runs. | `partially_supported` | metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts. | Add request dispatch and finish/error timestamps to future replayer/proxy metrics. | Cannot use these runs to prove online per-session sequentiality. |
|
||||
| Batch 1 workload shape can be characterized from formatted traces and metrics. | `supported_for_trace_shape` | Full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass. | Add cache-hit joined records for actual reuse decomposition. | Actual cache reuse decomposition needs cached_tokens joined with hash_ids. |
|
||||
| Static PD separation is worse than combined in existing 200-request GPU A/B. | `supported_by_existing_artifact` | outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json. | Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology. | Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy. |
|
||||
| Elastic transfer-based migration does not improve high-contention 500-request run. | `supported_by_existing_artifact` | outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv. | Attribute whether failure is trigger quality, transfer overhead, or wrong load regime. | Existing metrics lack actual sequentiality proof and per-request transfer waterfall. |
|
||||
| PD-colo prefill/decode interference is not yet directly proven by step-level data in this package. | `not_yet_supported` | No decode-step and prefill-overlap timestamp artifact found in summarized runs. | Run Batch 2 controlled same-worker/different-worker injection with step timestamps. | Cannot claim interference as causal without Batch 2. |
|
||||
| Session hot-spot residual imbalance is suggested but not fully attributed. | `partially_supported` | gpu_util.csv shows per-GPU mean-util imbalance in existing runs. | Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker. | GPU util imbalance alone is not enough to prove session hot-spot. |
|
||||
| SRR is not measured by existing fixed-request runs. | `not_yet_supported` | No arrival-rate sweep artifacts found. | Implement Batch 4 Poisson session-arrival SRR sweep. | Latency-at-one-load cannot support sustainable throughput claim. |
|
||||
51
analysis/characterization/current_results/claim_matrix.json
Normal file
51
analysis/characterization/current_results/claim_matrix.json
Normal file
@@ -0,0 +1,51 @@
|
||||
[
|
||||
{
|
||||
"claim": "Batch 0 substrate audit is only partially complete for existing runs.",
|
||||
"needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.",
|
||||
"reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.",
|
||||
"status": "partially_supported",
|
||||
"supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts."
|
||||
},
|
||||
{
|
||||
"claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.",
|
||||
"needed_next": "Add cache-hit joined records for actual reuse decomposition.",
|
||||
"reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.",
|
||||
"status": "supported_for_trace_shape",
|
||||
"supporting_data": "Full compact trace CPU summary in full_trace_summary.json: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass."
|
||||
},
|
||||
{
|
||||
"claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.",
|
||||
"needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.",
|
||||
"reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.",
|
||||
"status": "supported_by_existing_artifact",
|
||||
"supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json."
|
||||
},
|
||||
{
|
||||
"claim": "Elastic transfer-based migration does not improve high-contention 500-request run.",
|
||||
"needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.",
|
||||
"reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.",
|
||||
"status": "supported_by_existing_artifact",
|
||||
"supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv."
|
||||
},
|
||||
{
|
||||
"claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.",
|
||||
"needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.",
|
||||
"reviewer_risk": "Cannot claim interference as causal without Batch 2.",
|
||||
"status": "not_yet_supported",
|
||||
"supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs."
|
||||
},
|
||||
{
|
||||
"claim": "Session hot-spot residual imbalance is suggested but not fully attributed.",
|
||||
"needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.",
|
||||
"reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.",
|
||||
"status": "partially_supported",
|
||||
"supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs."
|
||||
},
|
||||
{
|
||||
"claim": "SRR is not measured by existing fixed-request runs.",
|
||||
"needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.",
|
||||
"reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.",
|
||||
"status": "not_yet_supported",
|
||||
"supporting_data": "No arrival-rate sweep artifacts found."
|
||||
}
|
||||
]
|
||||
95
analysis/characterization/current_results/comparisons.json
Normal file
95
analysis/characterization/current_results/comparisons.json
Normal file
@@ -0,0 +1,95 @@
|
||||
[
|
||||
{
|
||||
"baseline": "outputs/gpu_ab_combined",
|
||||
"e2e_p50_delta_pct": 40.870329127661,
|
||||
"e2e_p90_delta_pct": 15.206416995091814,
|
||||
"error_count": [
|
||||
2,
|
||||
13
|
||||
],
|
||||
"gpu_imbalance_ratio": [
|
||||
3.2445157838416265,
|
||||
11.149056603773586
|
||||
],
|
||||
"gpu_mean_util": [
|
||||
30.541666666666664,
|
||||
12.367081447963802
|
||||
],
|
||||
"name": "combined_vs_pdsep_200",
|
||||
"request_count": [
|
||||
200,
|
||||
200
|
||||
],
|
||||
"success_count": [
|
||||
198,
|
||||
187
|
||||
],
|
||||
"tpot_p90_delta_pct": 1.3481309269699875,
|
||||
"ttft_p50_delta_pct": 98.06752892925572,
|
||||
"ttft_p90_delta_pct": 44.79649177751278,
|
||||
"variant": "outputs/gpu_ab_pdsep",
|
||||
"wall_clock_delta_pct": 142.27736808267244
|
||||
},
|
||||
{
|
||||
"baseline": "outputs/contention_16s_ts10",
|
||||
"e2e_p50_delta_pct": 11.538788125232664,
|
||||
"e2e_p90_delta_pct": -5.080083318118138,
|
||||
"error_count": [
|
||||
2,
|
||||
2
|
||||
],
|
||||
"gpu_imbalance_ratio": [
|
||||
2.310775410408662,
|
||||
2.600767754318618
|
||||
],
|
||||
"gpu_mean_util": [
|
||||
23.030492424242425,
|
||||
26.349561403508773
|
||||
],
|
||||
"name": "contention_baseline_vs_elastic_500",
|
||||
"request_count": [
|
||||
500,
|
||||
500
|
||||
],
|
||||
"success_count": [
|
||||
498,
|
||||
498
|
||||
],
|
||||
"tpot_p90_delta_pct": 13.63098996823875,
|
||||
"ttft_p50_delta_pct": 12.433589435386224,
|
||||
"ttft_p90_delta_pct": 13.412576920999959,
|
||||
"variant": "outputs/contention_16s_elastic",
|
||||
"wall_clock_delta_pct": -0.5645626396767849
|
||||
},
|
||||
{
|
||||
"baseline": "outputs/combined_1000req",
|
||||
"e2e_p50_delta_pct": 202.85189980479385,
|
||||
"e2e_p90_delta_pct": 128.274511020719,
|
||||
"error_count": [
|
||||
2,
|
||||
204
|
||||
],
|
||||
"gpu_imbalance_ratio": [
|
||||
null,
|
||||
null
|
||||
],
|
||||
"gpu_mean_util": [
|
||||
null,
|
||||
null
|
||||
],
|
||||
"name": "combined_1000_vs_pdsep_mooncake",
|
||||
"request_count": [
|
||||
1000,
|
||||
1000
|
||||
],
|
||||
"success_count": [
|
||||
998,
|
||||
796
|
||||
],
|
||||
"tpot_p90_delta_pct": -34.83638659447109,
|
||||
"ttft_p50_delta_pct": 781.9835547522864,
|
||||
"ttft_p90_delta_pct": 1030.68607857992,
|
||||
"variant": "outputs/exp3_pd_sep_tp1_mooncake",
|
||||
"wall_clock_delta_pct": 119.18997774599991
|
||||
}
|
||||
]
|
||||
77
analysis/characterization/current_results/current_results.md
Normal file
77
analysis/characterization/current_results/current_results.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# Current Characterization Results
|
||||
|
||||
Generated: 2026-05-25T06:52:18.096448+00:00
|
||||
Git commit: `21ffb3d4f77956d008b1815a3c0d46e0188ac390`
|
||||
|
||||
## Canonical Full-Trace CPU Summary
|
||||
|
||||
Source: `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`.
|
||||
This is CPU-only parsing of the compact formatted trace with session IDs
|
||||
reconstructed from `parent_chat_id` chains.
|
||||
|
||||
| Metric | Value |
|
||||
|---|---:|
|
||||
| Requests | 2,114,220 |
|
||||
| Sessions | 1,307,276 |
|
||||
| Trace span | 7,199.975 s |
|
||||
| Input tokens p50/p90/p99 | 20,030 / 87,855 / 125,527 |
|
||||
| Output tokens p50/p90/p99 | 80 / 811 / 6,615 |
|
||||
| Input/output ratio p50/p90/p99 | 217.8 / 1,204.4 / 4,251.6 |
|
||||
| Turns/session p50/p90/p99/max | 1 / 1 / 18 / 3,091 |
|
||||
| Session input tokens p50/p90/p99/max | 12,486 / 72,676 / 974,934 / 156,756,974 |
|
||||
| Top 1% / 5% / 10% sessions by input-token mass | 46.5% / 66.5% / 74.6% |
|
||||
|
||||
Immediate reading: the full trace strongly supports long-input/short-output
|
||||
and heavy-tailed session token mass. It does **not** by itself prove online
|
||||
sequentiality or actual cache-hit reuse; those require runtime timestamps and
|
||||
cache-hit fields.
|
||||
|
||||
## Existing Run Summaries
|
||||
|
||||
| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| outputs/gpu_ab_combined | 198/200 | 1.01/9.36 | 5.05/30.2 | 0.0732 | 30.5 | 3.24 |
|
||||
| outputs/gpu_ab_pdsep | 187/200 | 1.99/13.5 | 7.11/34.8 | 0.0742 | 12.4 | 11.1 |
|
||||
| outputs/contention_16s_ts10 | 498/500 | 0.826/9.71 | 5.8/51 | 0.103 | 23 | 2.31 |
|
||||
| outputs/contention_16s_elastic | 498/500 | 0.929/11 | 6.47/48.4 | 0.117 | 26.3 | 2.6 |
|
||||
| outputs/combined_1000req | 998/1000 | 0.393/2.57 | 3.22/28 | 0.113 | n/a | n/a |
|
||||
| outputs/exp3_pd_sep_tp1_mooncake | 796/1000 | 3.47/29 | 9.75/63.9 | 0.0739 | n/a | n/a |
|
||||
|
||||
## Pairwise Comparisons
|
||||
|
||||
| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| combined_vs_pdsep_200 | +98.1% | +44.8% | +40.9% | +15.2% | +1.3% | +142.3% |
|
||||
| contention_baseline_vs_elastic_500 | +12.4% | +13.4% | +11.5% | -5.1% | +13.6% | -0.6% |
|
||||
| combined_1000_vs_pdsep_mooncake | +782.0% | +1030.7% | +202.9% | +128.3% | -34.8% | +119.2% |
|
||||
|
||||
## What We Can Say Now
|
||||
|
||||
- **partially_supported**: Batch 0 substrate audit is only partially complete for existing runs.
|
||||
Supporting data: metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.
|
||||
Next: Add request dispatch and finish/error timestamps to future replayer/proxy metrics.
|
||||
- **supported_for_trace_shape**: Batch 1 workload shape can be characterized from formatted traces and metrics.
|
||||
Supporting data: full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass.
|
||||
Next: add cache-hit joined records for actual reuse decomposition.
|
||||
- **supported_by_existing_artifact**: Static PD separation is worse than combined in existing 200-request GPU A/B.
|
||||
Supporting data: outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.
|
||||
Next: Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.
|
||||
- **supported_by_existing_artifact**: Elastic transfer-based migration does not improve high-contention 500-request run.
|
||||
Supporting data: outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.
|
||||
Next: Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.
|
||||
- **not_yet_supported**: PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.
|
||||
Supporting data: No decode-step and prefill-overlap timestamp artifact found in summarized runs.
|
||||
Next: Run Batch 2 controlled same-worker/different-worker injection with step timestamps.
|
||||
- **partially_supported**: Session hot-spot residual imbalance is suggested but not fully attributed.
|
||||
Supporting data: gpu_util.csv shows per-GPU mean-util imbalance in existing runs.
|
||||
Next: Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.
|
||||
- **not_yet_supported**: SRR is not measured by existing fixed-request runs.
|
||||
Supporting data: No arrival-rate sweep artifacts found.
|
||||
Next: Implement Batch 4 Poisson session-arrival SRR sweep.
|
||||
|
||||
## Main Reviewer Risks
|
||||
|
||||
- **high**: Session sequentiality not proven - Add dispatch/finish timestamps and run Batch 0 before SRR claims.
|
||||
- **medium**: Legacy PD-sep data may not match final methodology - Use fresh PD matrix for paper-grade claims.
|
||||
- **medium**: GPU util is not a sufficient hot-spot proof - Add route-decision and per-worker queue logs for Batch 3.
|
||||
- **medium**: Cache reuse decomposition is incomplete without joined hash/cache-hit data - Emit hash_ids/session_id/cached_tokens in the same per-request record.
|
||||
@@ -0,0 +1,56 @@
|
||||
{
|
||||
"input": {
|
||||
"count": 2114220,
|
||||
"max": 202371,
|
||||
"mean": 33637.38370084476,
|
||||
"p50": 20030.0,
|
||||
"p90": 87855.1000000001,
|
||||
"p95": 104738.0,
|
||||
"p99": 125527.0
|
||||
},
|
||||
"input_output_ratio": {
|
||||
"count": 2108130,
|
||||
"max": 143664.0,
|
||||
"mean": 534.3516074828406,
|
||||
"p50": 217.8,
|
||||
"p90": 1204.3769610389616,
|
||||
"p95": 1814.3478327228322,
|
||||
"p99": 4251.585499999998
|
||||
},
|
||||
"output": {
|
||||
"count": 2114220,
|
||||
"max": 132665,
|
||||
"mean": 444.97059624826176,
|
||||
"p50": 80.0,
|
||||
"p90": 811.0,
|
||||
"p95": 2213.0,
|
||||
"p99": 6614.810000000056
|
||||
},
|
||||
"path": "/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl",
|
||||
"records": 2114220,
|
||||
"session_input_tokens": {
|
||||
"count": 1307276,
|
||||
"max": 156756974,
|
||||
"mean": 54400.77639916896,
|
||||
"p50": 12486.0,
|
||||
"p90": 72676.0,
|
||||
"p95": 108523.25,
|
||||
"p99": 974933.75
|
||||
},
|
||||
"sessions": 1307276,
|
||||
"top_session_input_fraction": {
|
||||
"top10pct": 0.7464402483455778,
|
||||
"top1pct": 0.46456810581415175,
|
||||
"top5pct": 0.6651718740752172
|
||||
},
|
||||
"trace_span_s": 7199.975,
|
||||
"turns_per_session": {
|
||||
"count": 1307276,
|
||||
"max": 3091,
|
||||
"mean": 1.6172713336739908,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 2.0,
|
||||
"p99": 18.0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,66 @@
|
||||
# Main-Claim Allowed Runs
|
||||
|
||||
Status: current audit gate
|
||||
Date: 2026-05-25
|
||||
|
||||
## Allowed For Workload-Shape Claims
|
||||
|
||||
These artifacts can support trace/workload characterization claims:
|
||||
|
||||
- `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`
|
||||
- Compact formatted full trace.
|
||||
- CPU summary recorded in `full_trace_summary.json`.
|
||||
- Supports long-input/short-output and session token-mass skew claims.
|
||||
- Does not prove runtime cache hits or online sequentiality.
|
||||
- `traces/w600_r0.0015_st30.jsonl`
|
||||
- Local sampled trace.
|
||||
- Useful for local dry runs and figure generation.
|
||||
- Not the canonical full-trace source.
|
||||
|
||||
## Allowed For Legacy Baseline Sanity Claims
|
||||
|
||||
These existing runs can support sanity-level comparisons, but not final
|
||||
paper-grade SRR claims:
|
||||
|
||||
- `outputs/gpu_ab_combined`
|
||||
- `outputs/gpu_ab_pdsep`
|
||||
- `outputs/contention_16s_ts10`
|
||||
- `outputs/contention_16s_elastic`
|
||||
- `outputs/combined_1000req`
|
||||
- `outputs/exp3_pd_sep_tp1_mooncake`
|
||||
|
||||
Allowed claims:
|
||||
|
||||
- Static PD-sep was worse than combined in these existing fixed-request runs.
|
||||
- Elastic transfer-based migration did not improve the summarized 500-request
|
||||
high-contention run.
|
||||
- GPU-util imbalance exists in these artifacts.
|
||||
|
||||
Disallowed claims:
|
||||
|
||||
- Online SRR.
|
||||
- Per-session sequentiality.
|
||||
- Causal attribution of prefill/decode interference.
|
||||
- Causal attribution of session hot spots from GPU utilization alone.
|
||||
|
||||
## Not Yet Allowed For Main Claims
|
||||
|
||||
The following need fresh instrumentation or fresh runs:
|
||||
|
||||
- Batch 2 prefill/decode interference.
|
||||
- Batch 3 session hot-spot root cause.
|
||||
- Batch 4 sustainable request rate.
|
||||
- Batch 5 failure attribution near SRR boundary.
|
||||
|
||||
## Required Upgrade Before Paper-Grade Claims
|
||||
|
||||
Future main-claim runs must include:
|
||||
|
||||
- per-request actual dispatch timestamp;
|
||||
- per-request finish/error timestamp;
|
||||
- route decision and selected worker;
|
||||
- per-worker queue delay;
|
||||
- per-worker KV occupancy;
|
||||
- per-worker APC/cache-hit snapshot;
|
||||
- attempted/completed/error/goodput counters;
|
||||
- session-causal load generation.
|
||||
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# Rebuild this current-results audit package.
|
||||
python3 analysis/characterization/summarize_runs.py --output-dir analysis/characterization/current_results --runs outputs/gpu_ab_combined outputs/gpu_ab_pdsep outputs/contention_16s_ts10 outputs/contention_16s_elastic outputs/combined_1000req outputs/exp3_pd_sep_tp1_mooncake
|
||||
|
||||
# Example Batch 0/1 local trace analysis.
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace traces/w600_r0.0015_st30.jsonl \
|
||||
--kv-bytes-per-token 98304 \
|
||||
--task-name w600_local_full_trace \
|
||||
--overwrite
|
||||
|
||||
# CPU-only full compact trace summary was computed on dash0 from:
|
||||
# /home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
|
||||
# Recompute either by running analyze.py on dash0, or by copying that compact
|
||||
# formatted JSONL locally. Do not use the 487G raw file directly.
|
||||
@@ -0,0 +1,26 @@
|
||||
[
|
||||
{
|
||||
"evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.",
|
||||
"mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.",
|
||||
"risk": "Session sequentiality not proven",
|
||||
"severity": "high"
|
||||
},
|
||||
{
|
||||
"evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.",
|
||||
"mitigation": "Use fresh PD matrix for paper-grade claims.",
|
||||
"risk": "Legacy PD-sep data may not match final methodology",
|
||||
"severity": "medium"
|
||||
},
|
||||
{
|
||||
"evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.",
|
||||
"mitigation": "Add route-decision and per-worker queue logs for Batch 3.",
|
||||
"risk": "GPU util is not a sufficient hot-spot proof",
|
||||
"severity": "medium"
|
||||
},
|
||||
{
|
||||
"evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.",
|
||||
"mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.",
|
||||
"risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data",
|
||||
"severity": "medium"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,8 @@
|
||||
# Reviewer Risk Register
|
||||
|
||||
| Risk | Severity | Evidence | Mitigation |
|
||||
|---|---|---|---|
|
||||
| Session sequentiality not proven | `high` | Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps. | Add dispatch/finish timestamps and run Batch 0 before SRR claims. |
|
||||
| Legacy PD-sep data may not match final methodology | `medium` | PD matrix scaffold exists separately; some old runs used earlier flags/methodology. | Use fresh PD matrix for paper-grade claims. |
|
||||
| GPU util is not a sufficient hot-spot proof | `medium` | Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership. | Add route-decision and per-worker queue logs for Batch 3. |
|
||||
| Cache reuse decomposition is incomplete without joined hash/cache-hit data | `medium` | Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts. | Emit hash_ids/session_id/cached_tokens in the same per-request record. |
|
||||
720
analysis/characterization/current_results/run_summaries.json
Normal file
720
analysis/characterization/current_results/run_summaries.json
Normal file
@@ -0,0 +1,720 @@
|
||||
[
|
||||
{
|
||||
"apc_summary": {
|
||||
"reason": "apc.txt missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"artifact_availability": {
|
||||
"apc_txt": false,
|
||||
"breakdown_json": false,
|
||||
"gpu_util_csv": true,
|
||||
"metrics_jsonl": true,
|
||||
"metrics_summary_json": true
|
||||
},
|
||||
"breakdown_summary": {
|
||||
"reason": "breakdown.json missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"error_count": 2,
|
||||
"exists": true,
|
||||
"external_cache_hit_ratio": null,
|
||||
"gpu_summary": {
|
||||
"gpu_count": 8,
|
||||
"max_mean_util_pct": 63.166666666666664,
|
||||
"max_min_ratio": 3.2445157838416265,
|
||||
"mean_util_pct": 30.541666666666664,
|
||||
"min_mean_util_pct": 19.46875,
|
||||
"per_gpu_mean_util_pct": {
|
||||
"0": 29.145833333333332,
|
||||
"1": 20.041666666666668,
|
||||
"2": 25.0,
|
||||
"3": 63.166666666666664,
|
||||
"4": 21.927083333333332,
|
||||
"5": 34.90625,
|
||||
"6": 19.46875,
|
||||
"7": 30.677083333333332
|
||||
},
|
||||
"status": "available",
|
||||
"stddev_across_gpu_mean_util_pct": 13.337857305429534
|
||||
},
|
||||
"latency_stats_s": {
|
||||
"count": 198.0,
|
||||
"mean": 13.01780862723021,
|
||||
"p50": 5.048548387829214,
|
||||
"p90": 30.18109704903327,
|
||||
"p99": 119.01174414204434
|
||||
},
|
||||
"metrics_jsonl_rows": 200,
|
||||
"metrics_summary_available": true,
|
||||
"prefix_cache_hit_ratio": 0.0,
|
||||
"request_count": 200,
|
||||
"run": "outputs/gpu_ab_combined",
|
||||
"session_summary": {
|
||||
"request_cached_tokens": {
|
||||
"count": 200,
|
||||
"max": 0.0,
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"request_input_tokens": {
|
||||
"count": 200,
|
||||
"max": 111927.0,
|
||||
"mean": 29318.375,
|
||||
"p50": 21376.0,
|
||||
"p90": 81218.19999999998,
|
||||
"p95": 87571.2,
|
||||
"p99": 101619.27999999994
|
||||
},
|
||||
"request_output_tokens": {
|
||||
"count": 200,
|
||||
"max": 5083.0,
|
||||
"mean": 257.675,
|
||||
"p50": 72.0,
|
||||
"p90": 664.3,
|
||||
"p95": 1063.9999999999993,
|
||||
"p99": 3300.1799999999994
|
||||
},
|
||||
"session_count": 145,
|
||||
"session_input_tokens": {
|
||||
"count": 145,
|
||||
"max": 1567423.0,
|
||||
"mean": 40439.137931034486,
|
||||
"p50": 10879.0,
|
||||
"p90": 72438.39999999997,
|
||||
"p95": 106934.39999999988,
|
||||
"p99": 374927.28
|
||||
},
|
||||
"status": "available",
|
||||
"top_session_input_fraction": {
|
||||
"top_10pct": 0.6588294883328288,
|
||||
"top_1pct": 0.33276622595897626,
|
||||
"top_5pct": 0.5534430199490934
|
||||
},
|
||||
"turns_per_session": {
|
||||
"count": 145,
|
||||
"max": 21.0,
|
||||
"mean": 1.3793103448275863,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 2.0,
|
||||
"p99": 10.560000000000002
|
||||
}
|
||||
},
|
||||
"success_count": 198,
|
||||
"tpot_stats_s": {
|
||||
"count": 198.0,
|
||||
"mean": 0.04929455188644857,
|
||||
"p50": 0.03717198147904128,
|
||||
"p90": 0.07317714408040046,
|
||||
"p99": 0.10039294634234945
|
||||
},
|
||||
"ttft_stats_s": {
|
||||
"count": 198.0,
|
||||
"mean": 3.8987488178453984,
|
||||
"p50": 1.0068706551101059,
|
||||
"p90": 9.355209570843726,
|
||||
"p99": 33.855437273858115
|
||||
},
|
||||
"wall_clock_s": 483.543720243033
|
||||
},
|
||||
{
|
||||
"apc_summary": {
|
||||
"reason": "apc.txt missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"artifact_availability": {
|
||||
"apc_txt": false,
|
||||
"breakdown_json": false,
|
||||
"gpu_util_csv": true,
|
||||
"metrics_jsonl": true,
|
||||
"metrics_summary_json": true
|
||||
},
|
||||
"breakdown_summary": {
|
||||
"reason": "breakdown.json missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"error_count": 13,
|
||||
"exists": true,
|
||||
"external_cache_hit_ratio": null,
|
||||
"gpu_summary": {
|
||||
"gpu_count": 8,
|
||||
"max_mean_util_pct": 26.737556561085974,
|
||||
"max_min_ratio": 11.149056603773586,
|
||||
"mean_util_pct": 12.367081447963802,
|
||||
"min_mean_util_pct": 2.3981900452488687,
|
||||
"per_gpu_mean_util_pct": {
|
||||
"0": 26.737556561085974,
|
||||
"1": 14.44343891402715,
|
||||
"2": 19.036199095022624,
|
||||
"3": 7.4389140271493215,
|
||||
"4": 2.3981900452488687,
|
||||
"5": 8.841628959276019,
|
||||
"6": 11.963800904977376,
|
||||
"7": 8.076923076923077
|
||||
},
|
||||
"status": "available",
|
||||
"stddev_across_gpu_mean_util_pct": 7.15857427072345
|
||||
},
|
||||
"latency_stats_s": {
|
||||
"count": 187.0,
|
||||
"mean": 15.366858840781827,
|
||||
"p50": 7.111906730104238,
|
||||
"p90": 34.77056052000262,
|
||||
"p99": 119.2064151619561
|
||||
},
|
||||
"metrics_jsonl_rows": 200,
|
||||
"metrics_summary_available": true,
|
||||
"prefix_cache_hit_ratio": 0.0,
|
||||
"request_count": 200,
|
||||
"run": "outputs/gpu_ab_pdsep",
|
||||
"session_summary": {
|
||||
"request_cached_tokens": {
|
||||
"count": 200,
|
||||
"max": 0.0,
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"request_input_tokens": {
|
||||
"count": 200,
|
||||
"max": 111927.0,
|
||||
"mean": 29318.375,
|
||||
"p50": 21376.0,
|
||||
"p90": 81218.19999999998,
|
||||
"p95": 87571.2,
|
||||
"p99": 101619.27999999994
|
||||
},
|
||||
"request_output_tokens": {
|
||||
"count": 200,
|
||||
"max": 5097.0,
|
||||
"mean": 290.18,
|
||||
"p50": 75.5,
|
||||
"p90": 685.8,
|
||||
"p95": 1102.5999999999997,
|
||||
"p99": 3433.6599999999844
|
||||
},
|
||||
"session_count": 145,
|
||||
"session_input_tokens": {
|
||||
"count": 145,
|
||||
"max": 1567423.0,
|
||||
"mean": 40439.137931034486,
|
||||
"p50": 10879.0,
|
||||
"p90": 72438.39999999997,
|
||||
"p95": 106934.39999999988,
|
||||
"p99": 374927.28
|
||||
},
|
||||
"status": "available",
|
||||
"top_session_input_fraction": {
|
||||
"top_10pct": 0.6588294883328288,
|
||||
"top_1pct": 0.33276622595897626,
|
||||
"top_5pct": 0.5534430199490934
|
||||
},
|
||||
"turns_per_session": {
|
||||
"count": 145,
|
||||
"max": 21.0,
|
||||
"mean": 1.3793103448275863,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 2.0,
|
||||
"p99": 10.560000000000002
|
||||
}
|
||||
},
|
||||
"success_count": 187,
|
||||
"tpot_stats_s": {
|
||||
"count": 187.0,
|
||||
"mean": 0.05175559044923285,
|
||||
"p50": 0.04020531923068981,
|
||||
"p90": 0.07416366779122173,
|
||||
"p99": 0.10770365060609463
|
||||
},
|
||||
"ttft_stats_s": {
|
||||
"count": 187.0,
|
||||
"mean": 5.3933139261814524,
|
||||
"p50": 1.9942838260903955,
|
||||
"p90": 13.546015257015824,
|
||||
"p99": 39.30951087013818
|
||||
},
|
||||
"wall_clock_s": 1171.516998933861
|
||||
},
|
||||
{
|
||||
"apc_summary": {
|
||||
"reason": "apc.txt missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"artifact_availability": {
|
||||
"apc_txt": false,
|
||||
"breakdown_json": false,
|
||||
"gpu_util_csv": true,
|
||||
"metrics_jsonl": true,
|
||||
"metrics_summary_json": true
|
||||
},
|
||||
"breakdown_summary": {
|
||||
"reason": "breakdown.json missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"error_count": 2,
|
||||
"exists": true,
|
||||
"external_cache_hit_ratio": 0.0,
|
||||
"gpu_summary": {
|
||||
"gpu_count": 8,
|
||||
"max_mean_util_pct": 40.095454545454544,
|
||||
"max_min_ratio": 2.310775410408662,
|
||||
"mean_util_pct": 23.030492424242425,
|
||||
"min_mean_util_pct": 17.35151515151515,
|
||||
"per_gpu_mean_util_pct": {
|
||||
"0": 22.243939393939392,
|
||||
"1": 22.798484848484847,
|
||||
"2": 22.09090909090909,
|
||||
"3": 40.095454545454544,
|
||||
"4": 17.484848484848484,
|
||||
"5": 20.225757575757576,
|
||||
"6": 17.35151515151515,
|
||||
"7": 21.953030303030303
|
||||
},
|
||||
"status": "available",
|
||||
"stddev_across_gpu_mean_util_pct": 6.752783259358802
|
||||
},
|
||||
"latency_stats_s": {
|
||||
"count": 498.0,
|
||||
"mean": 21.908606036002105,
|
||||
"p50": 5.798741589998826,
|
||||
"p90": 51.00022796400299,
|
||||
"p99": 201.6967467680006
|
||||
},
|
||||
"metrics_jsonl_rows": 500,
|
||||
"metrics_summary_available": true,
|
||||
"prefix_cache_hit_ratio": 0.0,
|
||||
"request_count": 500,
|
||||
"run": "outputs/contention_16s_ts10",
|
||||
"session_summary": {
|
||||
"request_cached_tokens": {
|
||||
"count": 500,
|
||||
"max": 0.0,
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"request_input_tokens": {
|
||||
"count": 500,
|
||||
"max": 134865.0,
|
||||
"mean": 29190.97,
|
||||
"p50": 17756.0,
|
||||
"p90": 77365.70000000006,
|
||||
"p95": 87790.89999999997,
|
||||
"p99": 111360.44
|
||||
},
|
||||
"request_output_tokens": {
|
||||
"count": 500,
|
||||
"max": 16427.0,
|
||||
"mean": 404.286,
|
||||
"p50": 75.0,
|
||||
"p90": 863.7000000000004,
|
||||
"p95": 2384.45,
|
||||
"p99": 4928.559999999999
|
||||
},
|
||||
"session_count": 347,
|
||||
"session_input_tokens": {
|
||||
"count": 347,
|
||||
"max": 1929072.0,
|
||||
"mean": 42061.916426512966,
|
||||
"p50": 11342.0,
|
||||
"p90": 68297.20000000004,
|
||||
"p95": 116213.79999999993,
|
||||
"p99": 521898.36000000057
|
||||
},
|
||||
"status": "available",
|
||||
"top_session_input_fraction": {
|
||||
"top_10pct": 0.6824335059780473,
|
||||
"top_1pct": 0.32352388426969025,
|
||||
"top_5pct": 0.579125805000656
|
||||
},
|
||||
"turns_per_session": {
|
||||
"count": 347,
|
||||
"max": 29.0,
|
||||
"mean": 1.440922190201729,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 3.0,
|
||||
"p99": 10.54000000000002
|
||||
}
|
||||
},
|
||||
"success_count": 498,
|
||||
"tpot_stats_s": {
|
||||
"count": 498.0,
|
||||
"mean": 0.07772806444638436,
|
||||
"p50": 0.06774563665097065,
|
||||
"p90": 0.10325221968416468,
|
||||
"p99": 0.503443661631568
|
||||
},
|
||||
"ttft_stats_s": {
|
||||
"count": 498.0,
|
||||
"mean": 3.5127640336366928,
|
||||
"p50": 0.8264882279981975,
|
||||
"p90": 9.71474659699743,
|
||||
"p99": 34.40133859900379
|
||||
},
|
||||
"wall_clock_s": 1472.1331836190002
|
||||
},
|
||||
{
|
||||
"apc_summary": {
|
||||
"line_count": 8,
|
||||
"preview": "inst_0: prefix=45.7% ext=24.7%\ninst_1: prefix=33.1% ext=13.3%\ninst_2: prefix=14.2% ext=0.0%\ninst_3: prefix=50.7% ext=6.3%\ninst_4: prefix=33.6% ext=0.0%\ninst_5: prefix=66.6% ext=9.2%\ninst_6: prefix=42.1% ext=18.8%\ninst_7: prefix=36.7% ext=7.9%",
|
||||
"status": "available"
|
||||
},
|
||||
"artifact_availability": {
|
||||
"apc_txt": true,
|
||||
"breakdown_json": true,
|
||||
"gpu_util_csv": true,
|
||||
"metrics_jsonl": true,
|
||||
"metrics_summary_json": true
|
||||
},
|
||||
"breakdown_summary": {
|
||||
"field_sample": [
|
||||
"cache_hit",
|
||||
"estimated_new_tokens",
|
||||
"input_length",
|
||||
"policy",
|
||||
"request_id",
|
||||
"route_class",
|
||||
"routed_to",
|
||||
"t_done",
|
||||
"t_first_token",
|
||||
"t_proxy_recv"
|
||||
],
|
||||
"mode_counts": {
|
||||
"HEAVY_COLO": 111,
|
||||
"HEAVY_OFFLOAD": 13,
|
||||
"MEDIUM": 163,
|
||||
"WARM": 213
|
||||
},
|
||||
"route_counts": {
|
||||
"linear": 487
|
||||
},
|
||||
"row_count": 500,
|
||||
"status": "available"
|
||||
},
|
||||
"error_count": 2,
|
||||
"exists": true,
|
||||
"external_cache_hit_ratio": 0.0,
|
||||
"gpu_summary": {
|
||||
"gpu_count": 8,
|
||||
"max_mean_util_pct": 47.54385964912281,
|
||||
"max_min_ratio": 2.600767754318618,
|
||||
"mean_util_pct": 26.349561403508773,
|
||||
"min_mean_util_pct": 18.280701754385966,
|
||||
"per_gpu_mean_util_pct": {
|
||||
"0": 22.410526315789475,
|
||||
"1": 26.57894736842105,
|
||||
"2": 23.54035087719298,
|
||||
"3": 47.54385964912281,
|
||||
"4": 18.96842105263158,
|
||||
"5": 26.403508771929825,
|
||||
"6": 18.280701754385966,
|
||||
"7": 27.07017543859649
|
||||
},
|
||||
"status": "available",
|
||||
"stddev_across_gpu_mean_util_pct": 8.6079068381278
|
||||
},
|
||||
"latency_stats_s": {
|
||||
"count": 498.0,
|
||||
"mean": 21.780466573040204,
|
||||
"p50": 6.467846095998539,
|
||||
"p90": 48.40937389100145,
|
||||
"p99": 196.93401125499804
|
||||
},
|
||||
"metrics_jsonl_rows": 500,
|
||||
"metrics_summary_available": true,
|
||||
"prefix_cache_hit_ratio": 0.0,
|
||||
"request_count": 500,
|
||||
"run": "outputs/contention_16s_elastic",
|
||||
"session_summary": {
|
||||
"request_cached_tokens": {
|
||||
"count": 500,
|
||||
"max": 0.0,
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"request_input_tokens": {
|
||||
"count": 500,
|
||||
"max": 134865.0,
|
||||
"mean": 29190.97,
|
||||
"p50": 17756.0,
|
||||
"p90": 77365.70000000006,
|
||||
"p95": 87790.89999999997,
|
||||
"p99": 111360.44
|
||||
},
|
||||
"request_output_tokens": {
|
||||
"count": 500,
|
||||
"max": 16427.0,
|
||||
"mean": 395.466,
|
||||
"p50": 75.0,
|
||||
"p90": 809.8000000000013,
|
||||
"p95": 2264.2999999999943,
|
||||
"p99": 4928.559999999999
|
||||
},
|
||||
"session_count": 347,
|
||||
"session_input_tokens": {
|
||||
"count": 347,
|
||||
"max": 1929072.0,
|
||||
"mean": 42061.916426512966,
|
||||
"p50": 11342.0,
|
||||
"p90": 68297.20000000004,
|
||||
"p95": 116213.79999999993,
|
||||
"p99": 521898.36000000057
|
||||
},
|
||||
"status": "available",
|
||||
"top_session_input_fraction": {
|
||||
"top_10pct": 0.6824335059780473,
|
||||
"top_1pct": 0.32352388426969025,
|
||||
"top_5pct": 0.579125805000656
|
||||
},
|
||||
"turns_per_session": {
|
||||
"count": 347,
|
||||
"max": 29.0,
|
||||
"mean": 1.440922190201729,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 3.0,
|
||||
"p99": 10.54000000000002
|
||||
}
|
||||
},
|
||||
"success_count": 498,
|
||||
"tpot_stats_s": {
|
||||
"count": 498.0,
|
||||
"mean": 0.08306021258238128,
|
||||
"p50": 0.06973490711122092,
|
||||
"p90": 0.117326519391297,
|
||||
"p99": 0.5879216773072795
|
||||
},
|
||||
"ttft_stats_s": {
|
||||
"count": 498.0,
|
||||
"mean": 3.757858794331448,
|
||||
"p50": 0.9292503809992922,
|
||||
"p90": 11.017744456999935,
|
||||
"p99": 34.20241540799907
|
||||
},
|
||||
"wall_clock_s": 1463.8220696580029
|
||||
},
|
||||
{
|
||||
"apc_summary": {
|
||||
"reason": "apc.txt missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"artifact_availability": {
|
||||
"apc_txt": false,
|
||||
"breakdown_json": false,
|
||||
"gpu_util_csv": false,
|
||||
"metrics_jsonl": true,
|
||||
"metrics_summary_json": true
|
||||
},
|
||||
"breakdown_summary": {
|
||||
"reason": "breakdown.json missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"error_count": 2,
|
||||
"exists": true,
|
||||
"external_cache_hit_ratio": null,
|
||||
"gpu_summary": {
|
||||
"reason": "gpu_util.csv missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"latency_stats_s": {
|
||||
"count": 998.0,
|
||||
"mean": 15.744418202954922,
|
||||
"p50": 3.220371481962502,
|
||||
"p90": 28.005282410187647,
|
||||
"p99": 204.00479479599744
|
||||
},
|
||||
"metrics_jsonl_rows": 1000,
|
||||
"metrics_summary_available": true,
|
||||
"prefix_cache_hit_ratio": 0.5443149471989938,
|
||||
"request_count": 1000,
|
||||
"run": "outputs/combined_1000req",
|
||||
"session_summary": {
|
||||
"request_cached_tokens": {
|
||||
"count": 1000,
|
||||
"max": 0.0,
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"request_input_tokens": {
|
||||
"count": 1000,
|
||||
"max": 171427.0,
|
||||
"mean": 31611.14,
|
||||
"p50": 19798.0,
|
||||
"p90": 82584.20000000001,
|
||||
"p95": 93305.64999999997,
|
||||
"p99": 111947.83999999998
|
||||
},
|
||||
"request_output_tokens": {
|
||||
"count": 1000,
|
||||
"max": 41233.0,
|
||||
"mean": 392.579,
|
||||
"p50": 70.0,
|
||||
"p90": 685.8000000000002,
|
||||
"p95": 1834.6999999999975,
|
||||
"p99": 4584.459999999996
|
||||
},
|
||||
"session_count": 671,
|
||||
"session_input_tokens": {
|
||||
"count": 671,
|
||||
"max": 2320064.0,
|
||||
"mean": 47110.49180327869,
|
||||
"p50": 12394.0,
|
||||
"p90": 71875.0,
|
||||
"p95": 112984.5,
|
||||
"p99": 760591.4999999925
|
||||
},
|
||||
"status": "available",
|
||||
"top_session_input_fraction": {
|
||||
"top_10pct": 0.6915149849072194,
|
||||
"top_1pct": 0.36223103627392117,
|
||||
"top_5pct": 0.5945521736957288
|
||||
},
|
||||
"turns_per_session": {
|
||||
"count": 671,
|
||||
"max": 36.0,
|
||||
"mean": 1.4903129657228018,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 3.0,
|
||||
"p99": 12.599999999999909
|
||||
}
|
||||
},
|
||||
"success_count": 998,
|
||||
"tpot_stats_s": {
|
||||
"count": 998.0,
|
||||
"mean": 0.05681003092240363,
|
||||
"p50": 0.04029440155459775,
|
||||
"p90": 0.11343103184021618,
|
||||
"p99": 0.2962149563411783
|
||||
},
|
||||
"ttft_stats_s": {
|
||||
"count": 998.0,
|
||||
"mean": 1.0351265607308877,
|
||||
"p50": 0.39304836583323777,
|
||||
"p90": 2.5655168020166457,
|
||||
"p99": 8.210839052917436
|
||||
},
|
||||
"wall_clock_s": 1340.061015482992
|
||||
},
|
||||
{
|
||||
"apc_summary": {
|
||||
"reason": "apc.txt missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"artifact_availability": {
|
||||
"apc_txt": false,
|
||||
"breakdown_json": false,
|
||||
"gpu_util_csv": false,
|
||||
"metrics_jsonl": true,
|
||||
"metrics_summary_json": true
|
||||
},
|
||||
"breakdown_summary": {
|
||||
"reason": "breakdown.json missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"error_count": 204,
|
||||
"exists": true,
|
||||
"external_cache_hit_ratio": null,
|
||||
"gpu_summary": {
|
||||
"reason": "gpu_util.csv missing",
|
||||
"status": "unavailable"
|
||||
},
|
||||
"latency_stats_s": {
|
||||
"count": 796.0,
|
||||
"mean": 23.561836449842698,
|
||||
"p50": 9.752956213895231,
|
||||
"p90": 63.928921481827274,
|
||||
"p99": 168.2179710320197
|
||||
},
|
||||
"metrics_jsonl_rows": 1000,
|
||||
"metrics_summary_available": true,
|
||||
"prefix_cache_hit_ratio": 0.0,
|
||||
"request_count": 1000,
|
||||
"run": "outputs/exp3_pd_sep_tp1_mooncake",
|
||||
"session_summary": {
|
||||
"request_cached_tokens": {
|
||||
"count": 1000,
|
||||
"max": 0.0,
|
||||
"mean": 0.0,
|
||||
"p50": 0.0,
|
||||
"p90": 0.0,
|
||||
"p95": 0.0,
|
||||
"p99": 0.0
|
||||
},
|
||||
"request_input_tokens": {
|
||||
"count": 1000,
|
||||
"max": 171427.0,
|
||||
"mean": 31611.14,
|
||||
"p50": 19798.0,
|
||||
"p90": 82584.20000000001,
|
||||
"p95": 93305.64999999997,
|
||||
"p99": 111947.83999999998
|
||||
},
|
||||
"request_output_tokens": {
|
||||
"count": 1000,
|
||||
"max": 41233.0,
|
||||
"mean": 412.176,
|
||||
"p50": 74.0,
|
||||
"p90": 759.7000000000008,
|
||||
"p95": 1834.6999999999975,
|
||||
"p99": 4928.559999999999
|
||||
},
|
||||
"session_count": 671,
|
||||
"session_input_tokens": {
|
||||
"count": 671,
|
||||
"max": 2320064.0,
|
||||
"mean": 47110.49180327869,
|
||||
"p50": 12394.0,
|
||||
"p90": 71875.0,
|
||||
"p95": 112984.5,
|
||||
"p99": 760591.4999999925
|
||||
},
|
||||
"status": "available",
|
||||
"top_session_input_fraction": {
|
||||
"top_10pct": 0.6915149849072194,
|
||||
"top_1pct": 0.36223103627392117,
|
||||
"top_5pct": 0.5945521736957288
|
||||
},
|
||||
"turns_per_session": {
|
||||
"count": 671,
|
||||
"max": 36.0,
|
||||
"mean": 1.4903129657228018,
|
||||
"p50": 1.0,
|
||||
"p90": 1.0,
|
||||
"p95": 3.0,
|
||||
"p99": 12.599999999999909
|
||||
}
|
||||
},
|
||||
"success_count": 796,
|
||||
"tpot_stats_s": {
|
||||
"count": 796.0,
|
||||
"mean": 0.05211825330315642,
|
||||
"p50": 0.06622354774330945,
|
||||
"p90": 0.07391575907026088,
|
||||
"p99": 0.10499966285609896
|
||||
},
|
||||
"ttft_stats_s": {
|
||||
"count": 796.0,
|
||||
"mean": 10.753033336598003,
|
||||
"p50": 3.4666219488717616,
|
||||
"p90": 29.00794132403098,
|
||||
"p99": 81.7531874559354
|
||||
},
|
||||
"wall_clock_s": 2937.2794416199904
|
||||
}
|
||||
]
|
||||
264
analysis/characterization/protocols.md
Normal file
264
analysis/characterization/protocols.md
Normal file
@@ -0,0 +1,264 @@
|
||||
# Characterization Protocols For Remaining Batches
|
||||
|
||||
Status: implementation protocol and audit checklist
|
||||
Date: 2026-05-25
|
||||
|
||||
This file completes the `analysis/characterization` scaffold for the TODO
|
||||
list. It separates what is already implemented from what requires fresh GPU
|
||||
runs or new engine/proxy instrumentation.
|
||||
|
||||
## Implemented Now
|
||||
|
||||
### Batch 0/1 Analyzer
|
||||
|
||||
Use:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/analyze.py \
|
||||
--trace traces/w600_r0.0015_st30.jsonl \
|
||||
--kv-bytes-per-token 98304 \
|
||||
--task-name w600_local_full_trace \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
The analyzer writes:
|
||||
|
||||
- `manifest.json`
|
||||
- `summary.json`
|
||||
- `summary.md`
|
||||
- `audit.md`
|
||||
- `session_concurrency.json`
|
||||
- `session_arrival_stats.json`
|
||||
- `turn_interval_stats.json`
|
||||
- `trace_profile.json`
|
||||
- `workload_summary.json`
|
||||
- `kv_footprint_summary.json`
|
||||
- `reuse_decomposition.json`
|
||||
- `session_skew.json`
|
||||
- `append_delta_stats.json`
|
||||
|
||||
Limitations:
|
||||
|
||||
- Actual online sequentiality requires dispatch and finish/error timestamps.
|
||||
Existing `metrics.jsonl` artifacts generally do not contain these fields.
|
||||
- Actual reuse decomposition requires `cached_tokens`/`cache_hit`, `hash_ids`,
|
||||
and `session_id` in the same joinable request record.
|
||||
|
||||
### Existing-Run Audit
|
||||
|
||||
Use:
|
||||
|
||||
```bash
|
||||
python3 analysis/characterization/summarize_runs.py
|
||||
```
|
||||
|
||||
The script writes an audit package under:
|
||||
|
||||
```text
|
||||
analysis/characterization/current_results/
|
||||
```
|
||||
|
||||
It summarizes already completed runs and explicitly marks which claims are
|
||||
supported, partially supported, or not yet supported.
|
||||
|
||||
## Batch 2 Protocol: PD-Colo Prefill/Decode Interference
|
||||
|
||||
Purpose:
|
||||
|
||||
Prove whether same-worker prefill overlap increases decode TPOT/queue delay.
|
||||
|
||||
Required new instrumentation:
|
||||
|
||||
- per-request dispatch timestamp
|
||||
- per-request finish/error timestamp
|
||||
- per decode step timestamp
|
||||
- decode step worker id
|
||||
- prefill chunk start/end timestamp
|
||||
- prefill worker id
|
||||
- request/session id associated with each prefill chunk
|
||||
|
||||
Required arms:
|
||||
|
||||
1. decode-only steady load
|
||||
2. decode + same-worker heavy prefill injection
|
||||
3. decode + different-worker heavy prefill injection
|
||||
4. trace replay with overlap labels
|
||||
|
||||
Required sweep:
|
||||
|
||||
```text
|
||||
uncached_prefill_tokens in {2k, 8k, 16k, 32k, 64k}
|
||||
chunked_prefill_size in available engine values
|
||||
```
|
||||
|
||||
Required outputs:
|
||||
|
||||
- `interference_microbench_summary.json`
|
||||
- `decode_step_timeseries.csv`
|
||||
- `prefill_overlap_events.jsonl`
|
||||
- `interference_index.json`
|
||||
- TPOT timeline figure with prefill overlays
|
||||
- same-worker vs different-worker TPOT boxplot
|
||||
|
||||
Pass condition:
|
||||
|
||||
```text
|
||||
TPOT_p90(overlap_same_worker) / TPOT_p90(no_overlap) > 1
|
||||
```
|
||||
|
||||
and the effect must be materially weaker in the different-worker control.
|
||||
|
||||
## Batch 3 Protocol: Session Hot-Spot Residual Imbalance
|
||||
|
||||
Purpose:
|
||||
|
||||
Prove whether cache-aware/LMetric still leaves hot workers under
|
||||
session-heavy skew.
|
||||
|
||||
Required new instrumentation:
|
||||
|
||||
- route decision per request
|
||||
- chosen worker
|
||||
- candidate worker scores
|
||||
- cache hit / estimated uncached tokens per candidate
|
||||
- per-worker request queue length/delay
|
||||
- per-worker decode queue length/delay
|
||||
- per-worker KV occupancy
|
||||
- per-worker APC/cache-hit snapshot
|
||||
|
||||
Required arms:
|
||||
|
||||
1. corrected LMetric/cache-aware
|
||||
2. load-only routing
|
||||
3. hard sticky routing
|
||||
4. current Unified hybrid
|
||||
5. session-mass capped/equalized replay
|
||||
|
||||
Required outputs:
|
||||
|
||||
- `worker_balance_summary.json`
|
||||
- `session_to_worker_map.json`
|
||||
- `session_mass_summary.json`
|
||||
- `routing_policy_comparison.json`
|
||||
- `hotspot_index.json`
|
||||
- per-worker queue delay bar
|
||||
- APC vs queue delay scatter
|
||||
- top-session contribution bar
|
||||
- policy tradeoff plot: APC vs hot-spot index
|
||||
|
||||
Pass condition:
|
||||
|
||||
LMetric/cache-aware must show measurable residual worker skew, and that skew
|
||||
must correlate with session token mass or locality.
|
||||
|
||||
GPU utilization alone is not enough for this claim.
|
||||
|
||||
## Batch 4 Protocol: Sustainable Request Rate
|
||||
|
||||
Purpose:
|
||||
|
||||
Measure:
|
||||
|
||||
```text
|
||||
SRR(SLO) = max arrival rate satisfying SLO in steady state
|
||||
```
|
||||
|
||||
Required load generator behavior:
|
||||
|
||||
- open-loop session arrivals, preferably Poisson
|
||||
- session-internal sequentiality
|
||||
- warmup window
|
||||
- steady-state measurement window
|
||||
- explicit attempted/completed/error counters
|
||||
|
||||
Provisional SLO:
|
||||
|
||||
```text
|
||||
TTFT_p90 <= T_ttft
|
||||
E2E_p90 <= T_e2e
|
||||
TPOT_p90 <= T_tpot
|
||||
error_rate <= epsilon
|
||||
queue length stable
|
||||
KV occupancy stable
|
||||
```
|
||||
|
||||
Required arms:
|
||||
|
||||
1. PD-colo corrected LMetric/cache-aware
|
||||
2. static PD-disagg
|
||||
3. current Unified hybrid
|
||||
4. optional hard sticky
|
||||
5. optional load-only
|
||||
|
||||
Required outputs:
|
||||
|
||||
- `srr_curve.json`
|
||||
- `lambda_runs/<lambda>/summary.json`
|
||||
- `slo_violation_reason.json`
|
||||
- `goodput_vs_arrival_rate.json`
|
||||
- SRR bar chart
|
||||
- latency vs arrival rate curves
|
||||
- goodput vs arrival rate
|
||||
- queue/KV stability plot near failure point
|
||||
|
||||
Pass condition:
|
||||
|
||||
Each policy has a measured max sustainable lambda under the same SLO and
|
||||
same session-causal arrival process.
|
||||
|
||||
## Batch 5 Protocol: Failure Attribution Near SRR Boundary
|
||||
|
||||
Purpose:
|
||||
|
||||
Explain why each policy fails near SRR.
|
||||
|
||||
Required rates:
|
||||
|
||||
```text
|
||||
lambda = 0.9 * SRR
|
||||
lambda = 1.0 * SRR
|
||||
lambda = 1.1 * SRR
|
||||
```
|
||||
|
||||
Labels for each slow/SLO-violating request:
|
||||
|
||||
- same-worker prefill overlap
|
||||
- hot worker queue
|
||||
- high KV occupancy
|
||||
- cache miss / large uncached append
|
||||
- transfer wait
|
||||
- P queue wait
|
||||
- D admission wait
|
||||
- unknown
|
||||
|
||||
Required outputs:
|
||||
|
||||
- `slow_request_attribution.jsonl`
|
||||
- `failure_breakdown.json`
|
||||
- `case_studies.md`
|
||||
- `worker_failure_windows.json`
|
||||
- violation cause stacked bar
|
||||
- slow request waterfall
|
||||
- worker timeline near failure
|
||||
|
||||
Pass condition:
|
||||
|
||||
The analysis must explain whether PD-colo is limited by interference,
|
||||
hot-spot, KV pressure, or a mixture, and whether Unified/PUSH underperforms
|
||||
because of trigger quality, transfer cost, target admission, or load regime.
|
||||
|
||||
## Batch 6 Protocol: Audit Package
|
||||
|
||||
Implemented by `summarize_runs.py` for existing runs and extended by fresh
|
||||
Batch 2-5 outputs later.
|
||||
|
||||
Required files:
|
||||
|
||||
- `characterization_claim_matrix.md`
|
||||
- `all_figures_index.md`
|
||||
- `reviewer_risk_register.md`
|
||||
- `reproduction_commands.sh`
|
||||
- `main_claim_allowed_runs.md`
|
||||
|
||||
Current package intentionally marks Batch 2/4/5 claims as not yet supported
|
||||
until fresh instrumented experiments exist.
|
||||
666
analysis/characterization/summarize_runs.py
Normal file
666
analysis/characterization/summarize_runs.py
Normal file
@@ -0,0 +1,666 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Summarize existing benchmark artifacts for characterization review.
|
||||
|
||||
This is a CPU-only companion to ``analyze.py``. It does not run benchmarks.
|
||||
It reads completed output directories and produces an audit-oriented package
|
||||
that helps decide which TODO claims are currently supported by existing data
|
||||
and which still need fresh GPU runs or additional instrumentation.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import datetime as dt
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
|
||||
JsonDict = dict[str, Any]
|
||||
|
||||
|
||||
DEFAULT_RUNS = [
|
||||
"outputs/gpu_ab_combined",
|
||||
"outputs/gpu_ab_pdsep",
|
||||
"outputs/contention_16s_ts10",
|
||||
"outputs/contention_16s_elastic",
|
||||
"outputs/combined_1000req",
|
||||
"outputs/exp3_pd_sep_tp1_mooncake",
|
||||
]
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
out_dir = args.output_dir
|
||||
out_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
run_dirs = [Path(p) for p in (args.runs or DEFAULT_RUNS)]
|
||||
summaries = [summarize_run(path) for path in run_dirs]
|
||||
comparisons = build_comparisons(summaries)
|
||||
claim_matrix = build_claim_matrix(summaries, comparisons)
|
||||
risk_register = build_risk_register(summaries)
|
||||
|
||||
write_json(out_dir / "run_summaries.json", summaries)
|
||||
write_json(out_dir / "comparisons.json", comparisons)
|
||||
write_json(out_dir / "claim_matrix.json", claim_matrix)
|
||||
write_json(out_dir / "reviewer_risk_register.json", risk_register)
|
||||
(out_dir / "current_results.md").write_text(
|
||||
render_current_results(summaries, comparisons, claim_matrix, risk_register),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(out_dir / "characterization_claim_matrix.md").write_text(
|
||||
render_claim_matrix(claim_matrix),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(out_dir / "reviewer_risk_register.md").write_text(
|
||||
render_risk_register(risk_register),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(out_dir / "all_figures_index.md").write_text(
|
||||
render_figures_index(summaries),
|
||||
encoding="utf-8",
|
||||
)
|
||||
(out_dir / "reproduction_commands.sh").write_text(
|
||||
render_reproduction_commands(args, run_dirs),
|
||||
encoding="utf-8",
|
||||
)
|
||||
print(f"Wrote run summary package to {out_dir}")
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser(
|
||||
description="Summarize existing characterization-relevant output dirs.",
|
||||
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
|
||||
)
|
||||
parser.add_argument(
|
||||
"--runs",
|
||||
nargs="*",
|
||||
default=None,
|
||||
help="Output directories to summarize. Defaults to a small curated set.",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output-dir",
|
||||
type=Path,
|
||||
default=Path("analysis/characterization/current_results"),
|
||||
help="Directory for generated review artifacts.",
|
||||
)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def summarize_run(path: Path) -> JsonDict:
|
||||
metrics_summary = load_json(path / "metrics.summary.json")
|
||||
metrics_rows = load_jsonl(path / "metrics.jsonl")
|
||||
gpu_summary = summarize_gpu(path / "gpu_util.csv")
|
||||
breakdown_summary = summarize_breakdown(path / "breakdown.json")
|
||||
apc_summary = summarize_apc(path / "apc.txt")
|
||||
return {
|
||||
"run": str(path),
|
||||
"exists": path.exists(),
|
||||
"metrics_summary_available": bool(metrics_summary),
|
||||
"metrics_jsonl_rows": len(metrics_rows),
|
||||
"request_count": first_present(metrics_summary, ["request_count"]),
|
||||
"success_count": first_present(metrics_summary, ["success_count"]),
|
||||
"error_count": first_present(metrics_summary, ["error_count"]),
|
||||
"wall_clock_s": first_present(metrics_summary, ["wall_clock_s"]),
|
||||
"latency_stats_s": metrics_summary.get("latency_stats_s"),
|
||||
"ttft_stats_s": metrics_summary.get("ttft_stats_s"),
|
||||
"tpot_stats_s": metrics_summary.get("tpot_stats_s"),
|
||||
"prefix_cache_hit_ratio": metrics_summary.get("prefix_cache_hit_ratio"),
|
||||
"external_cache_hit_ratio": metrics_summary.get("external_cache_hit_ratio"),
|
||||
"session_summary": summarize_sessions(metrics_rows),
|
||||
"gpu_summary": gpu_summary,
|
||||
"breakdown_summary": breakdown_summary,
|
||||
"apc_summary": apc_summary,
|
||||
"artifact_availability": {
|
||||
"metrics_summary_json": (path / "metrics.summary.json").exists(),
|
||||
"metrics_jsonl": (path / "metrics.jsonl").exists(),
|
||||
"gpu_util_csv": (path / "gpu_util.csv").exists(),
|
||||
"breakdown_json": (path / "breakdown.json").exists(),
|
||||
"apc_txt": (path / "apc.txt").exists(),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def summarize_sessions(rows: list[JsonDict]) -> JsonDict:
|
||||
if not rows:
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"reason": "metrics.jsonl missing",
|
||||
}
|
||||
sessions: dict[str, JsonDict] = {}
|
||||
input_values = []
|
||||
output_values = []
|
||||
cached_values = []
|
||||
for row in rows:
|
||||
sid = str(row.get("session_id", ""))
|
||||
item = sessions.setdefault(
|
||||
sid,
|
||||
{
|
||||
"turns": 0,
|
||||
"input_tokens": 0.0,
|
||||
"output_tokens": 0.0,
|
||||
"cached_tokens": 0.0,
|
||||
},
|
||||
)
|
||||
inp = to_float(row.get("input_length")) or 0.0
|
||||
out = to_float(row.get("actual_output_tokens")) or to_float(row.get("output_length")) or 0.0
|
||||
cached = to_float(row.get("cached_tokens")) or 0.0
|
||||
item["turns"] += 1
|
||||
item["input_tokens"] += inp
|
||||
item["output_tokens"] += out
|
||||
item["cached_tokens"] += cached
|
||||
input_values.append(inp)
|
||||
output_values.append(out)
|
||||
cached_values.append(cached)
|
||||
per_session_input = [v["input_tokens"] for v in sessions.values()]
|
||||
return {
|
||||
"status": "available",
|
||||
"request_input_tokens": stats(input_values),
|
||||
"request_output_tokens": stats(output_values),
|
||||
"request_cached_tokens": stats(cached_values),
|
||||
"session_count": len(sessions),
|
||||
"turns_per_session": stats([v["turns"] for v in sessions.values()]),
|
||||
"session_input_tokens": stats(per_session_input),
|
||||
"top_session_input_fraction": top_contribution(per_session_input),
|
||||
}
|
||||
|
||||
|
||||
def summarize_gpu(path: Path) -> JsonDict:
|
||||
if not path.exists():
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"reason": "gpu_util.csv missing",
|
||||
}
|
||||
values: dict[str, list[float]] = {}
|
||||
with path.open() as handle:
|
||||
reader = csv.DictReader(handle)
|
||||
for row in reader:
|
||||
gpu = str(row.get("gpu", ""))
|
||||
util = to_float(row.get("util_pct"))
|
||||
if gpu and util is not None:
|
||||
values.setdefault(gpu, []).append(util)
|
||||
means = {gpu: statistics.fmean(vals) for gpu, vals in values.items() if vals}
|
||||
if not means:
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"reason": "gpu_util.csv had no util_pct rows",
|
||||
}
|
||||
mean_values = list(means.values())
|
||||
return {
|
||||
"status": "available",
|
||||
"gpu_count": len(means),
|
||||
"per_gpu_mean_util_pct": means,
|
||||
"mean_util_pct": statistics.fmean(mean_values),
|
||||
"stddev_across_gpu_mean_util_pct": statistics.pstdev(mean_values),
|
||||
"max_mean_util_pct": max(mean_values),
|
||||
"min_mean_util_pct": min(mean_values),
|
||||
"max_min_ratio": max(mean_values) / max(min(mean_values), 1e-9),
|
||||
}
|
||||
|
||||
|
||||
def summarize_breakdown(path: Path) -> JsonDict:
|
||||
if not path.exists():
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"reason": "breakdown.json missing",
|
||||
}
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"reason": f"failed to parse breakdown: {exc}",
|
||||
}
|
||||
rows: list[JsonDict]
|
||||
if isinstance(data, list):
|
||||
rows = [r for r in data if isinstance(r, dict)]
|
||||
elif isinstance(data, dict):
|
||||
rows = data.get("records") if isinstance(data.get("records"), list) else []
|
||||
if not rows:
|
||||
rows = [data]
|
||||
else:
|
||||
rows = []
|
||||
mode_counts: dict[str, int] = {}
|
||||
route_counts: dict[str, int] = {}
|
||||
for row in rows:
|
||||
mode = row.get("mode") or row.get("execution_mode") or row.get("route_class")
|
||||
route = row.get("route") or row.get("decision") or row.get("policy")
|
||||
if mode is not None:
|
||||
mode_counts[str(mode)] = mode_counts.get(str(mode), 0) + 1
|
||||
if route is not None:
|
||||
route_counts[str(route)] = route_counts.get(str(route), 0) + 1
|
||||
return {
|
||||
"status": "available",
|
||||
"row_count": len(rows),
|
||||
"mode_counts": mode_counts,
|
||||
"route_counts": route_counts,
|
||||
"field_sample": sorted(rows[0].keys()) if rows else [],
|
||||
}
|
||||
|
||||
|
||||
def summarize_apc(path: Path) -> JsonDict:
|
||||
if not path.exists():
|
||||
return {
|
||||
"status": "unavailable",
|
||||
"reason": "apc.txt missing",
|
||||
}
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
return {
|
||||
"status": "available",
|
||||
"line_count": len(text.splitlines()),
|
||||
"preview": "\n".join(text.splitlines()[:20]),
|
||||
}
|
||||
|
||||
|
||||
def build_comparisons(summaries: list[JsonDict]) -> list[JsonDict]:
|
||||
by_run = {s["run"]: s for s in summaries}
|
||||
pairs = [
|
||||
("combined_vs_pdsep_200", "outputs/gpu_ab_combined", "outputs/gpu_ab_pdsep"),
|
||||
("contention_baseline_vs_elastic_500", "outputs/contention_16s_ts10", "outputs/contention_16s_elastic"),
|
||||
("combined_1000_vs_pdsep_mooncake", "outputs/combined_1000req", "outputs/exp3_pd_sep_tp1_mooncake"),
|
||||
]
|
||||
out = []
|
||||
for name, base, variant in pairs:
|
||||
if base not in by_run or variant not in by_run:
|
||||
continue
|
||||
out.append(compare_pair(name, by_run[base], by_run[variant]))
|
||||
return out
|
||||
|
||||
|
||||
def compare_pair(name: str, base: JsonDict, variant: JsonDict) -> JsonDict:
|
||||
return {
|
||||
"name": name,
|
||||
"baseline": base["run"],
|
||||
"variant": variant["run"],
|
||||
"request_count": [base.get("request_count"), variant.get("request_count")],
|
||||
"success_count": [base.get("success_count"), variant.get("success_count")],
|
||||
"error_count": [base.get("error_count"), variant.get("error_count")],
|
||||
"ttft_p50_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p50"), stat_value(variant, "ttft_stats_s", "p50")),
|
||||
"ttft_p90_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p90"), stat_value(variant, "ttft_stats_s", "p90")),
|
||||
"e2e_p50_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p50"), stat_value(variant, "latency_stats_s", "p50")),
|
||||
"e2e_p90_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p90"), stat_value(variant, "latency_stats_s", "p90")),
|
||||
"tpot_p90_delta_pct": pct_delta(stat_value(base, "tpot_stats_s", "p90"), stat_value(variant, "tpot_stats_s", "p90")),
|
||||
"wall_clock_delta_pct": pct_delta(base.get("wall_clock_s"), variant.get("wall_clock_s")),
|
||||
"gpu_mean_util": [
|
||||
nested(base, ["gpu_summary", "mean_util_pct"]),
|
||||
nested(variant, ["gpu_summary", "mean_util_pct"]),
|
||||
],
|
||||
"gpu_imbalance_ratio": [
|
||||
nested(base, ["gpu_summary", "max_min_ratio"]),
|
||||
nested(variant, ["gpu_summary", "max_min_ratio"]),
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
def build_claim_matrix(summaries: list[JsonDict], comparisons: list[JsonDict]) -> list[JsonDict]:
|
||||
has_gpu = any((s.get("gpu_summary") or {}).get("status") == "available" for s in summaries)
|
||||
has_metrics = any(s.get("metrics_summary_available") for s in summaries)
|
||||
return [
|
||||
{
|
||||
"claim": "Batch 0 substrate audit is only partially complete for existing runs.",
|
||||
"status": "partially_supported",
|
||||
"supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.",
|
||||
"needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.",
|
||||
"reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.",
|
||||
},
|
||||
{
|
||||
"claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.",
|
||||
"status": "supported_for_trace_shape",
|
||||
"supporting_data": "analysis/characterization/analyze.py outputs workload_summary/session_skew/KV footprint when given trace and kv_bytes_per_token.",
|
||||
"needed_next": "Run on dash0 compact formatted trace for canonical full-trace numbers.",
|
||||
"reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.",
|
||||
},
|
||||
{
|
||||
"claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.",
|
||||
"status": "supported_by_existing_artifact" if has_metrics else "unavailable",
|
||||
"supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.",
|
||||
"needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.",
|
||||
"reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.",
|
||||
},
|
||||
{
|
||||
"claim": "Elastic transfer-based migration does not improve high-contention 500-request run.",
|
||||
"status": "supported_by_existing_artifact" if has_metrics else "unavailable",
|
||||
"supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.",
|
||||
"needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.",
|
||||
"reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.",
|
||||
},
|
||||
{
|
||||
"claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.",
|
||||
"status": "not_yet_supported",
|
||||
"supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs.",
|
||||
"needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.",
|
||||
"reviewer_risk": "Cannot claim interference as causal without Batch 2.",
|
||||
},
|
||||
{
|
||||
"claim": "Session hot-spot residual imbalance is suggested but not fully attributed.",
|
||||
"status": "partially_supported" if has_gpu else "unavailable",
|
||||
"supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs.",
|
||||
"needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.",
|
||||
"reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.",
|
||||
},
|
||||
{
|
||||
"claim": "SRR is not measured by existing fixed-request runs.",
|
||||
"status": "not_yet_supported",
|
||||
"supporting_data": "No arrival-rate sweep artifacts found.",
|
||||
"needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.",
|
||||
"reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def build_risk_register(summaries: list[JsonDict]) -> list[JsonDict]:
|
||||
return [
|
||||
{
|
||||
"risk": "Session sequentiality not proven",
|
||||
"severity": "high",
|
||||
"evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.",
|
||||
"mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.",
|
||||
},
|
||||
{
|
||||
"risk": "Legacy PD-sep data may not match final methodology",
|
||||
"severity": "medium",
|
||||
"evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.",
|
||||
"mitigation": "Use fresh PD matrix for paper-grade claims.",
|
||||
},
|
||||
{
|
||||
"risk": "GPU util is not a sufficient hot-spot proof",
|
||||
"severity": "medium",
|
||||
"evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.",
|
||||
"mitigation": "Add route-decision and per-worker queue logs for Batch 3.",
|
||||
},
|
||||
{
|
||||
"risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data",
|
||||
"severity": "medium",
|
||||
"evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.",
|
||||
"mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.",
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def render_current_results(
|
||||
summaries: list[JsonDict],
|
||||
comparisons: list[JsonDict],
|
||||
claim_matrix: list[JsonDict],
|
||||
risk_register: list[JsonDict],
|
||||
) -> str:
|
||||
lines = [
|
||||
"# Current Characterization Results",
|
||||
"",
|
||||
f"Generated: {dt.datetime.now(dt.timezone.utc).isoformat()}",
|
||||
f"Git commit: `{git_commit()}`",
|
||||
"",
|
||||
"## Existing Run Summaries",
|
||||
"",
|
||||
"| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |",
|
||||
"|---|---:|---:|---:|---:|---:|---:|",
|
||||
]
|
||||
for s in summaries:
|
||||
lines.append(
|
||||
"| {run} | {ok}/{req} | {ttft50}/{ttft90} | {e2e50}/{e2e90} | {tpot90} | {gpu_mean} | {gpu_imb} |".format(
|
||||
run=s["run"],
|
||||
ok=fmt(s.get("success_count")),
|
||||
req=fmt(s.get("request_count")),
|
||||
ttft50=fmt(stat_value(s, "ttft_stats_s", "p50")),
|
||||
ttft90=fmt(stat_value(s, "ttft_stats_s", "p90")),
|
||||
e2e50=fmt(stat_value(s, "latency_stats_s", "p50")),
|
||||
e2e90=fmt(stat_value(s, "latency_stats_s", "p90")),
|
||||
tpot90=fmt(stat_value(s, "tpot_stats_s", "p90")),
|
||||
gpu_mean=fmt(nested(s, ["gpu_summary", "mean_util_pct"])),
|
||||
gpu_imb=fmt(nested(s, ["gpu_summary", "max_min_ratio"])),
|
||||
)
|
||||
)
|
||||
lines.extend([
|
||||
"",
|
||||
"## Pairwise Comparisons",
|
||||
"",
|
||||
"| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |",
|
||||
"|---|---:|---:|---:|---:|---:|---:|",
|
||||
])
|
||||
for c in comparisons:
|
||||
lines.append(
|
||||
"| {name} | {ttft50} | {ttft90} | {e2e50} | {e2e90} | {tpot90} | {wall} |".format(
|
||||
name=c["name"],
|
||||
ttft50=fmt_pct(c.get("ttft_p50_delta_pct")),
|
||||
ttft90=fmt_pct(c.get("ttft_p90_delta_pct")),
|
||||
e2e50=fmt_pct(c.get("e2e_p50_delta_pct")),
|
||||
e2e90=fmt_pct(c.get("e2e_p90_delta_pct")),
|
||||
tpot90=fmt_pct(c.get("tpot_p90_delta_pct")),
|
||||
wall=fmt_pct(c.get("wall_clock_delta_pct")),
|
||||
)
|
||||
)
|
||||
lines.extend([
|
||||
"",
|
||||
"## What We Can Say Now",
|
||||
"",
|
||||
])
|
||||
for item in claim_matrix:
|
||||
lines.append(f"- **{item['status']}**: {item['claim']}")
|
||||
lines.append(f" Supporting data: {item['supporting_data']}")
|
||||
lines.append(f" Next: {item['needed_next']}")
|
||||
lines.extend([
|
||||
"",
|
||||
"## Main Reviewer Risks",
|
||||
"",
|
||||
])
|
||||
for item in risk_register:
|
||||
lines.append(f"- **{item['severity']}**: {item['risk']} - {item['mitigation']}")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_claim_matrix(items: list[JsonDict]) -> str:
|
||||
lines = [
|
||||
"# Characterization Claim Matrix",
|
||||
"",
|
||||
"| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |",
|
||||
"|---|---|---|---|---|",
|
||||
]
|
||||
for item in items:
|
||||
lines.append(
|
||||
f"| {item['claim']} | `{item['status']}` | {item['supporting_data']} | {item['needed_next']} | {item['reviewer_risk']} |"
|
||||
)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_risk_register(items: list[JsonDict]) -> str:
|
||||
lines = [
|
||||
"# Reviewer Risk Register",
|
||||
"",
|
||||
"| Risk | Severity | Evidence | Mitigation |",
|
||||
"|---|---|---|---|",
|
||||
]
|
||||
for item in items:
|
||||
lines.append(
|
||||
f"| {item['risk']} | `{item['severity']}` | {item['evidence']} | {item['mitigation']} |"
|
||||
)
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def render_figures_index(summaries: list[JsonDict]) -> str:
|
||||
return "\n".join([
|
||||
"# Figures Index",
|
||||
"",
|
||||
"No generated figures are committed by this script. Batch-specific figures should be generated from:",
|
||||
"",
|
||||
"- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.",
|
||||
"- future Batch 2 step-timeline artifacts for interference plots.",
|
||||
"- future Batch 3 per-worker/session artifacts for hot-spot plots.",
|
||||
"- future Batch 4 arrival-rate sweep artifacts for SRR curves.",
|
||||
"",
|
||||
"This file exists so the audit package has a stable placeholder until fresh figures are generated.",
|
||||
"",
|
||||
])
|
||||
|
||||
|
||||
def render_reproduction_commands(args: argparse.Namespace, run_dirs: list[Path]) -> str:
|
||||
runs = " ".join(str(p) for p in run_dirs)
|
||||
return "\n".join([
|
||||
"#!/usr/bin/env bash",
|
||||
"set -euo pipefail",
|
||||
"",
|
||||
"# Rebuild this current-results audit package.",
|
||||
f"python3 analysis/characterization/summarize_runs.py --output-dir {args.output_dir} --runs {runs}",
|
||||
"",
|
||||
"# Example Batch 0/1 local trace analysis.",
|
||||
"python3 analysis/characterization/analyze.py \\",
|
||||
" --trace traces/w600_r0.0015_st30.jsonl \\",
|
||||
" --kv-bytes-per-token 98304 \\",
|
||||
" --task-name w600_local_full_trace \\",
|
||||
" --overwrite",
|
||||
"",
|
||||
])
|
||||
|
||||
|
||||
def load_json(path: Path) -> JsonDict:
|
||||
if not path.exists():
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except Exception:
|
||||
return {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def load_jsonl(path: Path) -> list[JsonDict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
rows = []
|
||||
with path.open(encoding="utf-8") as handle:
|
||||
for line in handle:
|
||||
if not line.strip():
|
||||
continue
|
||||
try:
|
||||
row = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if isinstance(row, dict):
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def write_json(path: Path, data: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def first_present(row: JsonDict, keys: list[str]) -> Any:
|
||||
for key in keys:
|
||||
if key in row:
|
||||
return row[key]
|
||||
return None
|
||||
|
||||
|
||||
def stat_value(run: JsonDict, stat_key: str, value_key: str) -> float | None:
|
||||
stats_obj = run.get(stat_key)
|
||||
if not isinstance(stats_obj, dict):
|
||||
return None
|
||||
return to_float(stats_obj.get(value_key))
|
||||
|
||||
|
||||
def nested(row: JsonDict, keys: list[str]) -> Any:
|
||||
cur: Any = row
|
||||
for key in keys:
|
||||
if not isinstance(cur, dict):
|
||||
return None
|
||||
cur = cur.get(key)
|
||||
return cur
|
||||
|
||||
|
||||
def pct_delta(base: Any, variant: Any) -> float | None:
|
||||
b = to_float(base)
|
||||
v = to_float(variant)
|
||||
if b is None or v is None or b == 0:
|
||||
return None
|
||||
return (v - b) / b * 100.0
|
||||
|
||||
|
||||
def to_float(value: Any) -> float | None:
|
||||
if value is None:
|
||||
return None
|
||||
try:
|
||||
out = float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return out if math.isfinite(out) else None
|
||||
|
||||
|
||||
def stats(values: list[float]) -> JsonDict | None:
|
||||
clean = sorted(float(v) for v in values if math.isfinite(float(v)))
|
||||
if not clean:
|
||||
return None
|
||||
return {
|
||||
"count": len(clean),
|
||||
"mean": statistics.fmean(clean),
|
||||
"p50": percentile(clean, 0.50),
|
||||
"p90": percentile(clean, 0.90),
|
||||
"p95": percentile(clean, 0.95),
|
||||
"p99": percentile(clean, 0.99),
|
||||
"max": clean[-1],
|
||||
}
|
||||
|
||||
|
||||
def percentile(values: list[float], q: float) -> float:
|
||||
if len(values) == 1:
|
||||
return values[0]
|
||||
rank = q * (len(values) - 1)
|
||||
lo = int(rank)
|
||||
hi = min(lo + 1, len(values) - 1)
|
||||
frac = rank - lo
|
||||
return values[lo] * (1 - frac) + values[hi] * frac
|
||||
|
||||
|
||||
def top_contribution(values: list[float]) -> JsonDict:
|
||||
clean = sorted([v for v in values if math.isfinite(v)], reverse=True)
|
||||
total = sum(clean)
|
||||
if not clean or total <= 0:
|
||||
return {"top_1pct": None, "top_5pct": None, "top_10pct": None}
|
||||
|
||||
def frac(pct: float) -> float:
|
||||
k = max(1, math.ceil(len(clean) * pct))
|
||||
return sum(clean[:k]) / total
|
||||
|
||||
return {
|
||||
"top_1pct": frac(0.01),
|
||||
"top_5pct": frac(0.05),
|
||||
"top_10pct": frac(0.10),
|
||||
}
|
||||
|
||||
|
||||
def fmt(value: Any) -> str:
|
||||
num = to_float(value)
|
||||
if num is None:
|
||||
return "n/a"
|
||||
if abs(num - round(num)) < 1e-9 and abs(num) < 1_000_000:
|
||||
return str(int(round(num)))
|
||||
return f"{num:.3g}"
|
||||
|
||||
|
||||
def fmt_pct(value: Any) -> str:
|
||||
num = to_float(value)
|
||||
if num is None:
|
||||
return "n/a"
|
||||
return f"{num:+.1f}%"
|
||||
|
||||
|
||||
def git_commit() -> str:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["git", "rev-parse", "HEAD"],
|
||||
check=True,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.DEVNULL,
|
||||
text=True,
|
||||
)
|
||||
except Exception:
|
||||
return ""
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
641
analysis/characterization_todo_for_interns.md
Normal file
641
analysis/characterization_todo_for_interns.md
Normal file
@@ -0,0 +1,641 @@
|
||||
# Agentic Workload Characterization TODO
|
||||
|
||||
Status: execution checklist for interns
|
||||
Date: 2026-05-25
|
||||
|
||||
## 0. Purpose
|
||||
|
||||
We are not starting from the assumption that Unified routing or PUSH
|
||||
migration is already the answer.
|
||||
|
||||
The first goal is to build a rigorous characterization package that proves:
|
||||
|
||||
1. which dimensions make agentic serving different;
|
||||
2. where static PD-disaggregation works poorly;
|
||||
3. where PD-colocation/cache-aware routing still has residual failure modes;
|
||||
4. how these failure modes reduce sustainable request rate under SLO.
|
||||
|
||||
Only after these facts are established should we refine the positive system
|
||||
design.
|
||||
|
||||
Primary system goal:
|
||||
|
||||
```text
|
||||
maximize sustainable request rate under SLO
|
||||
```
|
||||
|
||||
Prefill-decode interference and session hot-spot imbalance are mechanisms
|
||||
that may reduce SRR. They are not the final metric by themselves.
|
||||
|
||||
## 1. Global Delivery Rules
|
||||
|
||||
Every task must produce data, figures, and an audit trail. A task is not
|
||||
complete if it only produces a written conclusion.
|
||||
|
||||
Use this output layout:
|
||||
|
||||
```text
|
||||
outputs/characterization/<date>/<task_name>/
|
||||
├── manifest.json
|
||||
├── raw/
|
||||
├── summary.json
|
||||
├── summary.md
|
||||
├── figures/
|
||||
└── audit.md
|
||||
```
|
||||
|
||||
Required fields in `manifest.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"git_commit": "",
|
||||
"host": "",
|
||||
"gpu_type": "",
|
||||
"gpu_count": 0,
|
||||
"trace_path": "",
|
||||
"trace_sha256": "",
|
||||
"policy": "",
|
||||
"launch_command": "",
|
||||
"request_limit": null,
|
||||
"time_scale": null,
|
||||
"session_sampling_method": "",
|
||||
"session_sequential": true,
|
||||
"start_time": "",
|
||||
"end_time": ""
|
||||
}
|
||||
```
|
||||
|
||||
Every comparison must report:
|
||||
|
||||
- attempted requests
|
||||
- completed requests
|
||||
- errors / timeouts
|
||||
- goodput
|
||||
- TTFT p50/p90/p99
|
||||
- E2E p50/p90/p99
|
||||
- TPOT p50/p90/p99
|
||||
- per-worker queue metrics
|
||||
- per-worker GPU utilization
|
||||
- per-worker KV occupancy if available
|
||||
- per-worker APC / cache-hit metrics
|
||||
|
||||
Every figure must be reproducible from raw data by a script committed or
|
||||
saved alongside the artifact.
|
||||
|
||||
## 2. Batch 0: Benchmark Substrate Audit
|
||||
|
||||
### Goal
|
||||
|
||||
Prove the load generator and trace replay are valid before trusting any
|
||||
performance result.
|
||||
|
||||
The most important invariant:
|
||||
|
||||
```text
|
||||
For online agentic serving, each session must have at most one in-flight turn.
|
||||
Turn N+1 must not be sent before turn N completes.
|
||||
```
|
||||
|
||||
### TODO
|
||||
|
||||
1. Implement or run an analyzer that reconstructs per-session request
|
||||
intervals:
|
||||
- dispatch timestamp
|
||||
- first-token timestamp
|
||||
- finish timestamp
|
||||
- error / timeout timestamp
|
||||
2. Compute max concurrent in-flight turns per session.
|
||||
3. Compute session start-time distribution.
|
||||
4. Compute turn inter-arrival distribution.
|
||||
5. Classify each existing run as one of:
|
||||
- `online_realistic`
|
||||
- `burst_stress`
|
||||
- `synthetic_microbench`
|
||||
- `invalid_for_online_claim`
|
||||
6. For any run where session sequentiality is violated, write down exactly
|
||||
which claim it can still support.
|
||||
|
||||
### Data Artifacts
|
||||
|
||||
- `session_concurrency.json`
|
||||
- `session_arrival_stats.json`
|
||||
- `turn_interval_stats.json`
|
||||
- `trace_profile.json`
|
||||
- `invalid_runs.md`
|
||||
|
||||
### Figures
|
||||
|
||||
- session start-time CDF
|
||||
- per-session max in-flight histogram
|
||||
- turns per session CDF
|
||||
- turn inter-arrival CDF
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The `audit.md` must answer:
|
||||
|
||||
1. Does the main trace satisfy `max_inflight_per_session == 1`?
|
||||
2. If not, is the run explicitly labeled as stress or invalid?
|
||||
3. Are attempted/completed/error counts included?
|
||||
4. Are latency percentiles computed only over successes, and if so, is
|
||||
goodput also reported?
|
||||
|
||||
### Pass Criteria
|
||||
|
||||
- Main online-serving experiments must have `max_inflight_per_session == 1`.
|
||||
- Any violation must be clearly labeled and excluded from SRR claims.
|
||||
|
||||
## 3. Batch 1: Workload Characterization
|
||||
|
||||
### Goal
|
||||
|
||||
Establish agentic workload facts independent of any proposed system.
|
||||
|
||||
Required facts:
|
||||
|
||||
1. long input, short output;
|
||||
2. large per-request KV footprint;
|
||||
3. reuse is mostly intra-session;
|
||||
4. session token mass is heavy-tailed;
|
||||
5. total prompt length and effective uncached prefill work are different.
|
||||
|
||||
### TODO
|
||||
|
||||
1. Compute input token CDF.
|
||||
2. Compute output token CDF.
|
||||
3. Compute input/output ratio.
|
||||
4. Estimate KV footprint per request:
|
||||
|
||||
```text
|
||||
kv_bytes_per_request = input_tokens * kv_bytes_per_token
|
||||
```
|
||||
|
||||
5. Decompose reusable KV into:
|
||||
- intra-session reuse
|
||||
- cross-session reuse
|
||||
- shared/system-prefix reuse
|
||||
6. Compute session-level skew:
|
||||
- turns per session
|
||||
- cumulative input tokens per session
|
||||
- cumulative output tokens per session
|
||||
- cumulative uncached tokens per session
|
||||
- top-k session contribution
|
||||
7. Compute append / effective-prefill distribution:
|
||||
|
||||
```text
|
||||
uncached_tokens = input_tokens - cached_tokens
|
||||
```
|
||||
|
||||
8. Compare total input length vs uncached tokens.
|
||||
|
||||
### Data Artifacts
|
||||
|
||||
- `workload_summary.json`
|
||||
- `kv_footprint_summary.json`
|
||||
- `reuse_decomposition.json`
|
||||
- `session_skew.json`
|
||||
- `append_delta_stats.json`
|
||||
|
||||
### Figures
|
||||
|
||||
- input/output token CDF
|
||||
- input/output ratio CDF
|
||||
- KV footprint CDF
|
||||
- reuse decomposition stacked bar
|
||||
- turns per session CDF
|
||||
- per-session token mass Lorenz curve
|
||||
- top-k sessions token contribution bar
|
||||
- total input vs uncached tokens scatter
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The `audit.md` must answer:
|
||||
|
||||
1. What are input p50/p90/p99?
|
||||
2. What are output p50/p90/p99?
|
||||
3. What is the estimated KV footprint p50/p90/p99?
|
||||
4. What fraction of reuse is intra-session?
|
||||
5. What fraction of total token mass comes from top 1% / 5% sessions?
|
||||
6. Are long prompts often small appends after cache reuse?
|
||||
|
||||
### Pass Criteria
|
||||
|
||||
The batch passes only if these facts can be stated numerically with raw data
|
||||
links and plotted figures.
|
||||
|
||||
## 4. Batch 2: PD-Colo Prefill-Decode Interference Proof
|
||||
|
||||
### Goal
|
||||
|
||||
Prove that PD-colocation can suffer from prefill-decode interference under
|
||||
high load, and quantify how much this affects TPOT, decode queueing, and SLO.
|
||||
|
||||
Hypothesis:
|
||||
|
||||
```text
|
||||
When heavy uncached prefill overlaps with active decode on the same worker,
|
||||
decode TPOT and/or decode queue delay increases.
|
||||
```
|
||||
|
||||
### TODO
|
||||
|
||||
1. Run controlled microbenchmarks:
|
||||
- decode-only steady load;
|
||||
- decode load plus same-worker heavy prefill injection;
|
||||
- decode load plus different-worker heavy prefill injection.
|
||||
2. Sweep uncached prefill sizes:
|
||||
- 2k
|
||||
- 8k
|
||||
- 16k
|
||||
- 32k
|
||||
- 64k
|
||||
3. If supported, sweep chunked prefill size.
|
||||
4. Log timestamps for:
|
||||
- decode steps;
|
||||
- prefill start/end;
|
||||
- prefill chunks;
|
||||
- queue admission;
|
||||
- request completion.
|
||||
5. In trace replay, label decode steps by whether they overlap with
|
||||
same-worker prefill.
|
||||
6. Compute:
|
||||
|
||||
```text
|
||||
interference_index =
|
||||
TPOT_p90(decode steps overlapping same-worker prefill)
|
||||
/ TPOT_p90(decode steps without same-worker prefill)
|
||||
```
|
||||
|
||||
7. Compare same-worker vs different-worker controls.
|
||||
|
||||
### Data Artifacts
|
||||
|
||||
- `interference_microbench_summary.json`
|
||||
- `decode_step_timeseries.csv`
|
||||
- `prefill_overlap_events.jsonl`
|
||||
- `interference_index.json`
|
||||
- `trace_overlap_summary.json`
|
||||
|
||||
### Figures
|
||||
|
||||
- TPOT time series with prefill overlap annotation
|
||||
- interference index vs uncached prefill size
|
||||
- same-worker vs different-worker TPOT boxplot
|
||||
- chunk size vs TTFT/TPOT tradeoff
|
||||
- trace replay overlap vs non-overlap TPOT comparison
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The `audit.md` must answer:
|
||||
|
||||
1. Is the interference observed on the same worker?
|
||||
2. Is the different-worker control significantly weaker?
|
||||
3. Does interference grow with uncached prefill size?
|
||||
4. Does the phenomenon appear in real trace replay, not only microbench?
|
||||
5. Could the result be explained by global load instead of local colocation?
|
||||
|
||||
### Pass Criteria
|
||||
|
||||
- Same-worker overlap must measurably increase TPOT or decode queue delay.
|
||||
- The effect must be weaker or absent in the different-worker control.
|
||||
- The effect must be visible in at least one trace replay setting.
|
||||
|
||||
## 5. Batch 3: Session Hot-Spot Residual Imbalance Proof
|
||||
|
||||
### Goal
|
||||
|
||||
Prove that cache-aware/LMetric is a strong baseline but still leaves residual
|
||||
hot-worker imbalance due to session skew and locality.
|
||||
|
||||
Hypothesis:
|
||||
|
||||
```text
|
||||
Cache-aware routing preserves locality by attracting future turns to cached
|
||||
workers. This is usually good, but heavy-tailed sessions can create hot
|
||||
workers whose queue delay/SLO violations are much worse than the median
|
||||
worker even when other workers still have headroom.
|
||||
```
|
||||
|
||||
### TODO
|
||||
|
||||
1. Run the same session-causal trace with:
|
||||
- corrected LMetric/cache-aware;
|
||||
- load-only routing;
|
||||
- hard sticky routing;
|
||||
- current Unified hybrid, if available.
|
||||
2. For each worker, record:
|
||||
- assigned session count;
|
||||
- cumulative input tokens;
|
||||
- cumulative uncached tokens;
|
||||
- cumulative output tokens;
|
||||
- request queue delay;
|
||||
- decode queue delay;
|
||||
- GPU utilization;
|
||||
- KV occupancy;
|
||||
- APC / cache-hit rate;
|
||||
- SLO violations.
|
||||
3. For each session, record:
|
||||
- worker set used;
|
||||
- primary worker;
|
||||
- cumulative token mass;
|
||||
- number of turns;
|
||||
- latency contribution;
|
||||
- whether it appears in slow-request set.
|
||||
4. Create a session-mass capped or equalized replay:
|
||||
- cap max session turns or token mass;
|
||||
- rerun LMetric/cache-aware;
|
||||
- compare hot-spot index.
|
||||
5. Compute:
|
||||
|
||||
```text
|
||||
hotspot_index =
|
||||
max_worker_queue_delay_p90 / median_worker_queue_delay_p90
|
||||
```
|
||||
|
||||
6. Compute locality/load tradeoff:
|
||||
|
||||
```text
|
||||
locality_gain = APC(policy) - APC(load_only)
|
||||
imbalance_cost =
|
||||
max_worker_latency_p90(policy) - median_worker_latency_p90(policy)
|
||||
```
|
||||
|
||||
### Data Artifacts
|
||||
|
||||
- `worker_balance_summary.json`
|
||||
- `session_to_worker_map.json`
|
||||
- `session_mass_summary.json`
|
||||
- `routing_policy_comparison.json`
|
||||
- `hotspot_index.json`
|
||||
- `capped_session_replay_summary.json`
|
||||
|
||||
### Figures
|
||||
|
||||
- per-worker queue delay bar
|
||||
- per-worker token mass bar
|
||||
- GPU utilization timeline by worker
|
||||
- KV occupancy timeline by worker
|
||||
- APC vs queue delay scatter
|
||||
- top sessions contribution bar
|
||||
- policy tradeoff plot: APC vs hotspot_index
|
||||
- original vs session-capped hot-spot comparison
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The `audit.md` must answer:
|
||||
|
||||
1. Does LMetric/cache-aware still show worker-level skew?
|
||||
2. Are SLO violations concentrated on hot workers or hot sessions?
|
||||
3. Does load-only routing improve balance but reduce APC/locality?
|
||||
4. Does hard sticky improve locality but worsen hot-spot/HOL?
|
||||
5. Does session-mass capping reduce hot spots?
|
||||
|
||||
### Pass Criteria
|
||||
|
||||
- LMetric/cache-aware must be shown as strong but imperfect.
|
||||
- There must be measurable residual hot-worker imbalance.
|
||||
- The imbalance must correlate with session token mass or locality.
|
||||
|
||||
## 6. Batch 4: Sustainable Request Rate Sweep
|
||||
|
||||
### Goal
|
||||
|
||||
Connect interference and hot-spot mechanisms to the final metric:
|
||||
|
||||
```text
|
||||
SRR(SLO) = max arrival rate satisfying SLO in steady state
|
||||
```
|
||||
|
||||
### TODO
|
||||
|
||||
1. Define provisional SLO thresholds. Use configurable values, for example:
|
||||
|
||||
```text
|
||||
TTFT_p90 <= T_ttft
|
||||
E2E_p90 <= T_e2e
|
||||
TPOT_p90 <= T_tpot
|
||||
error_rate <= epsilon
|
||||
queue length stable
|
||||
KV occupancy stable
|
||||
```
|
||||
|
||||
2. Implement arrival-rate sweep:
|
||||
- Poisson session arrivals;
|
||||
- session-internal sequentiality;
|
||||
- warmup window;
|
||||
- steady-state measurement window.
|
||||
3. For each arrival rate `lambda`, run:
|
||||
- PD-colo cache-aware/LMetric;
|
||||
- static PD-disagg;
|
||||
- current Unified hybrid;
|
||||
- optional hard sticky;
|
||||
- optional load-only.
|
||||
4. Find maximum sustainable lambda for each policy.
|
||||
5. Report instability reasons:
|
||||
- SLO violation;
|
||||
- queue growth;
|
||||
- KV occupancy growth;
|
||||
- error/timeout growth.
|
||||
|
||||
### Data Artifacts
|
||||
|
||||
- `srr_curve.json`
|
||||
- `lambda_runs/<lambda>/summary.json`
|
||||
- `slo_violation_reason.json`
|
||||
- `goodput_vs_arrival_rate.json`
|
||||
- `stability_summary.json`
|
||||
|
||||
### Figures
|
||||
|
||||
- SRR bar chart
|
||||
- TTFT p90 vs arrival rate
|
||||
- E2E p90 vs arrival rate
|
||||
- TPOT p90 vs arrival rate
|
||||
- goodput vs arrival rate
|
||||
- error rate vs arrival rate
|
||||
- queue length over time near failure point
|
||||
- KV occupancy over time near failure point
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The `audit.md` must answer:
|
||||
|
||||
1. Are session arrivals open-loop and Poisson?
|
||||
2. Is session-internal sequentiality enforced?
|
||||
3. How long are warmup and steady-state windows?
|
||||
4. Is SRR failure persistent rather than transient?
|
||||
5. Are completed/requested counts reported at every lambda?
|
||||
6. Are policies compared on the same trace and same arrival process?
|
||||
|
||||
### Pass Criteria
|
||||
|
||||
- Each policy must have a measured SRR under the same SLO.
|
||||
- Failure must be attributed to persistent SLO violation, queue growth, KV
|
||||
growth, or error growth.
|
||||
- Data must be session-causal.
|
||||
|
||||
## 7. Batch 5: Failure Attribution Near SRR Boundary
|
||||
|
||||
### Goal
|
||||
|
||||
At and around the PD-colo/LMetric failure point, determine whether SLO
|
||||
violations are caused by prefill-decode interference, session hot spots, KV
|
||||
pressure, cache misses, or other mechanisms.
|
||||
|
||||
### TODO
|
||||
|
||||
1. Select three arrival rates:
|
||||
|
||||
```text
|
||||
lambda = 0.9 * SRR
|
||||
lambda = 1.0 * SRR
|
||||
lambda = 1.1 * SRR
|
||||
```
|
||||
|
||||
2. For every slow or SLO-violating request, assign labels:
|
||||
- same-worker prefill overlap;
|
||||
- hot worker queue;
|
||||
- high KV occupancy;
|
||||
- cache miss / large uncached append;
|
||||
- transfer wait;
|
||||
- P queue wait;
|
||||
- D admission wait;
|
||||
- unknown.
|
||||
3. Produce per-request waterfall for representative slow requests.
|
||||
4. Produce per-worker timeline around failure windows.
|
||||
5. Summarize cause distribution.
|
||||
|
||||
### Data Artifacts
|
||||
|
||||
- `slow_request_attribution.jsonl`
|
||||
- `failure_breakdown.json`
|
||||
- `case_studies.md`
|
||||
- `worker_failure_windows.json`
|
||||
|
||||
### Figures
|
||||
|
||||
- SLO violation cause stacked bar
|
||||
- slow request waterfall
|
||||
- worker timeline near failure
|
||||
- prefill/decode/KV/queue stacked breakdown
|
||||
- failure cause vs arrival rate
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The `audit.md` must answer:
|
||||
|
||||
1. What fraction of slow requests overlap same-worker prefill?
|
||||
2. What fraction are on hot workers?
|
||||
3. What fraction happen under high KV occupancy?
|
||||
4. What fraction are large uncached append requests?
|
||||
5. For PD-disagg/Unified migration, how much time is transfer/P queue/D wait?
|
||||
6. What remains unexplained?
|
||||
|
||||
### Pass Criteria
|
||||
|
||||
The batch must answer:
|
||||
|
||||
1. Why PD-colo/LMetric hits its SRR limit.
|
||||
2. Why static PD-disagg hits its SRR limit.
|
||||
3. If Unified/PUSH underperforms, whether the cause is trigger quality, cost
|
||||
model, transfer overhead, wrong load regime, or something else.
|
||||
|
||||
## 8. Batch 6: Audit Package
|
||||
|
||||
### Goal
|
||||
|
||||
Make the whole characterization package reviewable by a strict systems
|
||||
reviewer.
|
||||
|
||||
### TODO
|
||||
|
||||
1. Write a claim matrix:
|
||||
|
||||
```text
|
||||
claim -> data artifact -> figure -> script -> caveat -> reviewer risk
|
||||
```
|
||||
|
||||
2. Write a figure index:
|
||||
- figure filename;
|
||||
- source data;
|
||||
- generation command;
|
||||
- intended claim.
|
||||
3. Write a reviewer risk register:
|
||||
- loadgen validity risks;
|
||||
- trace representativeness risks;
|
||||
- metric bias risks;
|
||||
- implementation-specific risks;
|
||||
- generalization risks.
|
||||
4. Write a reproduction script or command list.
|
||||
5. Mark experiments that cannot support main claims.
|
||||
|
||||
### Final Artifacts
|
||||
|
||||
- `characterization_claim_matrix.md`
|
||||
- `all_figures_index.md`
|
||||
- `reviewer_risk_register.md`
|
||||
- `reproduction_commands.sh`
|
||||
- `main_claim_allowed_runs.md`
|
||||
|
||||
### Audit Checks
|
||||
|
||||
The final package must satisfy:
|
||||
|
||||
1. Every claim links to raw data.
|
||||
2. Every figure can be regenerated.
|
||||
3. Every experiment has a manifest.
|
||||
4. Every caveat is explicit.
|
||||
5. Invalid or stress-only runs are not used for online-serving claims.
|
||||
|
||||
## 9. Priority Order
|
||||
|
||||
### Priority 1
|
||||
|
||||
Do these first:
|
||||
|
||||
1. Batch 0: Benchmark Substrate Audit
|
||||
2. Batch 1: Workload Characterization
|
||||
3. Batch 3: Session Hot-Spot Residual Imbalance Proof
|
||||
|
||||
Reason:
|
||||
|
||||
These define whether the trace and routing problem are real. Without them,
|
||||
SRR sweeps and system experiments are not trustworthy.
|
||||
|
||||
### Priority 2
|
||||
|
||||
Do these after the substrate and workload facts are stable:
|
||||
|
||||
1. Batch 2: PD-Colo Prefill-Decode Interference Proof
|
||||
2. Batch 5: Failure Attribution Near SRR Boundary
|
||||
|
||||
Reason:
|
||||
|
||||
These explain the mechanisms behind SLO/SRR failure and determine what the
|
||||
positive system should actually fix.
|
||||
|
||||
### Priority 3
|
||||
|
||||
Do these after instrumentation and attribution are ready:
|
||||
|
||||
1. Batch 4: Sustainable Request Rate Sweep
|
||||
2. Batch 6: Audit Package
|
||||
|
||||
Reason:
|
||||
|
||||
SRR sweeps are expensive. They should run only after trace validity,
|
||||
logging, and attribution labels are ready.
|
||||
|
||||
## 10. Non-Negotiable Reviewer Rules
|
||||
|
||||
1. Do not use session-nonsequential loadgen for online-serving claims.
|
||||
2. Do not compare latency percentiles without attempted/completed/error counts.
|
||||
3. Do not use APC alone as a success metric.
|
||||
4. Do not use average GPU utilization as proof of load balance.
|
||||
5. Do not compare policies on different traces unless explicitly labeled.
|
||||
6. Do not hide failed requests or timeouts.
|
||||
7. Do not claim Unified/PUSH is the answer before failure attribution proves
|
||||
the relevant bottleneck and cost budget.
|
||||
8. Treat corrected LMetric/cache-aware PD-colo as the main baseline.
|
||||
9. Treat static PD-disagg as an important baseline, not a strawman.
|
||||
10. Every result must be reproducible from raw artifacts and commands.
|
||||
Reference in New Issue
Block a user