256 lines
8.2 KiB
Markdown
256 lines
8.2 KiB
Markdown
# Characterization Analyzer Runbook
|
|
|
|
CPU-only scaffold for Batch 0 and Batch 1 in
|
|
`analysis/characterization_todo_for_interns.md`.
|
|
|
|
This directory has three components:
|
|
|
|
- `analyze.py`: Batch 0/1 analyzer for trace and per-request metrics.
|
|
- `summarize_runs.py`: CPU-only audit of already completed benchmark
|
|
directories.
|
|
- `protocols.md`: exact protocol for Batch 2-6 experiments that require fresh
|
|
GPU runs or additional instrumentation.
|
|
|
|
The analyzer reads existing trace and metrics artifacts and writes:
|
|
|
|
```text
|
|
outputs/characterization/<date>/<task_name>/
|
|
├── manifest.json
|
|
├── raw/
|
|
├── summary.json
|
|
├── summary.md
|
|
├── audit.md
|
|
├── session_concurrency.json
|
|
├── session_arrival_stats.json
|
|
├── turn_interval_stats.json
|
|
├── trace_profile.json
|
|
├── invalid_runs.md
|
|
├── workload_summary.json
|
|
├── kv_footprint_summary.json
|
|
├── reuse_decomposition.json
|
|
├── session_skew.json
|
|
├── append_delta_stats.json
|
|
└── figures/
|
|
```
|
|
|
|
If `matplotlib` is installed, simple PNG/PDF figures are emitted under
|
|
`figures/`. If it is not installed, all JSON/Markdown data artifacts are still
|
|
written.
|
|
|
|
## Canonical Data Sources
|
|
|
|
Canonical full traces live on dash0:
|
|
|
|
- formatted trace: `~/ali-trace/trace-glm5.1-formatted/`
|
|
- raw unformatted trace: `~/ali-trace/trace-glm5.1/`
|
|
|
|
For the current GLM-5.1 characterization, prefer the compact formatted file:
|
|
|
|
```text
|
|
~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl
|
|
```
|
|
|
|
Do not pass `051315-051317-raw.jsonl` or the files under
|
|
`~/ali-trace/trace-glm5.1/` directly to this analyzer unless you first convert
|
|
them to the formatted schema. Those raw files are tens to hundreds of GiB and
|
|
contain full prompt payloads rather than the compact characterization schema.
|
|
|
|
The analyzer is CPU-only. For full trace characterization, either:
|
|
|
|
- run it on dash0 against the formatted JSONL files without starting any GPU
|
|
service; or
|
|
- copy/rsync the needed trace files from dash0 to this repository or another
|
|
local path, then run the analyzer locally.
|
|
|
|
Only light directory/field inspection is needed on dash0 before choosing which
|
|
trace file to analyze.
|
|
|
|
The raw unformatted directory is listed as a source option for provenance, but
|
|
this analyzer expects formatted JSONL records. Raw files should be converted to
|
|
the formatted schema before being passed to `--trace`.
|
|
|
|
## Inputs
|
|
|
|
Trace JSONL:
|
|
|
|
- Expected formatted fields: `chat_id`, `parent_chat_id`, `timestamp`,
|
|
`input_length`, `output_length`, `type`, `turn`, `hash_ids`, optional
|
|
`session_id`.
|
|
- If `session_id` is absent, sessions are reconstructed from
|
|
`parent_chat_id` chains.
|
|
- `timestamp` is treated as scheduled trace time, not proof of actual dispatch
|
|
time.
|
|
|
|
Metrics JSONL:
|
|
|
|
- Expected replayer fields: `request_id`, `session_id`, `turn_id`,
|
|
`trace_timestamp_s`, `input_length`, `output_length`, `cached_tokens`,
|
|
`latency_s`, `ttft_s`, `tpot_s`, `actual_output_tokens`, `error`.
|
|
- If the metrics file is from the current replayer, it does not include actual
|
|
dispatch/finish wall-clock timestamps. Batch 0 will therefore mark actual
|
|
session sequentiality as unavailable and separately report a scheduled
|
|
estimate from `trace_timestamp_s + latency_s`.
|
|
|
|
Proxy breakdown:
|
|
|
|
- Optional JSON/JSONL with fields such as `request_id`, `t_proxy_recv`,
|
|
`t_first_token`, `t_done`, `cache_hit`, `estimated_new_tokens`,
|
|
`route_class`, `routed_to`, `policy`.
|
|
- Batch 0 can prove actual per-session in-flight concurrency only when these
|
|
timing rows can be joined to analyzed requests by `request_id`.
|
|
- Existing proxy breakdown artifacts may not contain `session_id`; without a
|
|
request-id join to trace/metrics, they can still support append/cache-hit
|
|
statistics but not per-session concurrency.
|
|
|
|
Run config:
|
|
|
|
- Optional JSON, usually `outputs/<run>/config.json`.
|
|
- Used for manifest fields such as `policy`, `time_scale`, and request count
|
|
when available.
|
|
|
|
## Commands
|
|
|
|
Trace-only dry run:
|
|
|
|
```bash
|
|
python3 analysis/characterization/analyze.py \
|
|
--trace traces/w600_r0.0015_st30.jsonl \
|
|
--task-name w600_trace_only \
|
|
--overwrite
|
|
```
|
|
|
|
Trace plus replayer metrics:
|
|
|
|
```bash
|
|
python3 analysis/characterization/analyze.py \
|
|
--trace traces/w600_r0.0015_st30.jsonl \
|
|
--metrics outputs/smoke_test/metrics.jsonl \
|
|
--task-name smoke_trace_metrics \
|
|
--overwrite
|
|
```
|
|
|
|
Proxy breakdown append/cache analysis:
|
|
|
|
```bash
|
|
python3 analysis/characterization/analyze.py \
|
|
--breakdown outputs/contention_16s_elastic/breakdown.json \
|
|
--config outputs/contention_16s_elastic/config.json \
|
|
--task-name contention_breakdown \
|
|
--overwrite
|
|
```
|
|
|
|
Full trace on dash0, CPU-only:
|
|
|
|
```bash
|
|
python3 analysis/characterization/analyze.py \
|
|
--trace ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \
|
|
--task-name full_trace_characterization \
|
|
--overwrite
|
|
```
|
|
|
|
Local run after copying from dash0:
|
|
|
|
```bash
|
|
rsync -av dash0:~/ali-trace/trace-glm5.1-formatted/<trace-file>.jsonl traces/
|
|
python3 analysis/characterization/analyze.py \
|
|
--trace traces/<trace-file>.jsonl \
|
|
--task-name full_trace_characterization \
|
|
--overwrite
|
|
```
|
|
|
|
By default the analyzer records file size and mtime but skips full SHA256
|
|
hashing, because canonical raw trace files can be hundreds of GiB. Add
|
|
`--hash-inputs` only when you intentionally want a full file hash.
|
|
|
|
KV footprint requires a model-specific value:
|
|
|
|
```bash
|
|
python3 analysis/characterization/analyze.py \
|
|
--trace traces/w600_r0.0015_st30.jsonl \
|
|
--kv-bytes-per-token 98304 \
|
|
--task-name w600_with_kv_estimate \
|
|
--overwrite
|
|
```
|
|
|
|
Summarize existing completed runs:
|
|
|
|
```bash
|
|
python3 analysis/characterization/summarize_runs.py
|
|
```
|
|
|
|
This writes:
|
|
|
|
```text
|
|
analysis/characterization/current_results/
|
|
├── run_summaries.json
|
|
├── comparisons.json
|
|
├── claim_matrix.json
|
|
├── reviewer_risk_register.json
|
|
├── current_results.md
|
|
├── characterization_claim_matrix.md
|
|
├── all_figures_index.md
|
|
├── reviewer_risk_register.md
|
|
└── reproduction_commands.sh
|
|
```
|
|
|
|
## Batch 0 Semantics
|
|
|
|
The online-serving invariant is:
|
|
|
|
```text
|
|
Each session has at most one in-flight turn.
|
|
```
|
|
|
|
The analyzer reports:
|
|
|
|
- actual interval status from dispatch and finish/error timestamps;
|
|
- scheduled estimate from trace timestamps plus latency when available;
|
|
- per-session max in-flight;
|
|
- session start-time distribution;
|
|
- turn inter-arrival distribution;
|
|
- attempted/completed/error counts and goodput when metrics exist;
|
|
- run classification.
|
|
|
|
Important limitation: trace timestamps alone cannot prove actual replay
|
|
sequentiality. A run is only classified as `online_realistic` when actual
|
|
per-request dispatch and finish/error timestamps prove
|
|
`max_inflight_per_session <= 1`.
|
|
|
|
## Batch 1 Semantics
|
|
|
|
The analyzer reports:
|
|
|
|
- input/output CDF stats;
|
|
- input/output ratio;
|
|
- KV footprint CDF stats when `--kv-bytes-per-token` is supplied;
|
|
- session skew and top-session contribution;
|
|
- append/uncached token stats when `cached_tokens` or `cache_hit` exists;
|
|
- reuse decomposition when both cached-token fields and `hash_ids` exist.
|
|
|
|
Reuse decomposition is conservative:
|
|
|
|
- `intra_session`: cached hash block was seen earlier in the same session;
|
|
- `cross_session`: cached hash block was seen earlier in another session;
|
|
- `shared/system-prefix`: early-position block appears in many sessions;
|
|
- `unclassified`: cached tokens could not be mapped to a previously seen hash
|
|
block.
|
|
|
|
If cached-token/cache-hit fields are absent, reuse and append artifacts are
|
|
written with `status: "unavailable"` and list the required fields.
|
|
|
|
## Limitations
|
|
|
|
- The script does not run a benchmark, query a live service, touch GPU state,
|
|
or start any daemon.
|
|
- Request-id joins are exact. If trace, metrics, and proxy artifacts use
|
|
different request IDs, the unmatched rows are preserved under `raw/`.
|
|
- Actual Batch 0 sequentiality needs actual dispatch and finish/error
|
|
timestamps. Current `replayer/metrics.py` metrics are not enough by
|
|
themselves.
|
|
- `kv_bytes_per_token` depends on model architecture, layer count, KV heads,
|
|
head dimension, and dtype. The analyzer will not guess it.
|
|
- Shared/system-prefix reuse classification is a heuristic based on trace
|
|
`hash_ids` positions and cross-session frequency. Adjust
|
|
`--shared-prefix-min-sessions` and `--system-prefix-blocks` if the formatted
|
|
trace provides a stronger system-prefix marker.
|