diff --git a/analysis/characterization/README.md b/analysis/characterization/README.md new file mode 100644 index 0000000..e19b824 --- /dev/null +++ b/analysis/characterization/README.md @@ -0,0 +1,255 @@ +# Characterization Analyzer Runbook + +CPU-only scaffold for Batch 0 and Batch 1 in +`analysis/characterization_todo_for_interns.md`. + +This directory has three components: + +- `analyze.py`: Batch 0/1 analyzer for trace and per-request metrics. +- `summarize_runs.py`: CPU-only audit of already completed benchmark + directories. +- `protocols.md`: exact protocol for Batch 2-6 experiments that require fresh + GPU runs or additional instrumentation. + +The analyzer reads existing trace and metrics artifacts and writes: + +```text +outputs/characterization/// +├── manifest.json +├── raw/ +├── summary.json +├── summary.md +├── audit.md +├── session_concurrency.json +├── session_arrival_stats.json +├── turn_interval_stats.json +├── trace_profile.json +├── invalid_runs.md +├── workload_summary.json +├── kv_footprint_summary.json +├── reuse_decomposition.json +├── session_skew.json +├── append_delta_stats.json +└── figures/ +``` + +If `matplotlib` is installed, simple PNG/PDF figures are emitted under +`figures/`. If it is not installed, all JSON/Markdown data artifacts are still +written. + +## Canonical Data Sources + +Canonical full traces live on dash0: + +- formatted trace: `~/ali-trace/trace-glm5.1-formatted/` +- raw unformatted trace: `~/ali-trace/trace-glm5.1/` + +For the current GLM-5.1 characterization, prefer the compact formatted file: + +```text +~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl +``` + +Do not pass `051315-051317-raw.jsonl` or the files under +`~/ali-trace/trace-glm5.1/` directly to this analyzer unless you first convert +them to the formatted schema. Those raw files are tens to hundreds of GiB and +contain full prompt payloads rather than the compact characterization schema. + +The analyzer is CPU-only. For full trace characterization, either: + +- run it on dash0 against the formatted JSONL files without starting any GPU + service; or +- copy/rsync the needed trace files from dash0 to this repository or another + local path, then run the analyzer locally. + +Only light directory/field inspection is needed on dash0 before choosing which +trace file to analyze. + +The raw unformatted directory is listed as a source option for provenance, but +this analyzer expects formatted JSONL records. Raw files should be converted to +the formatted schema before being passed to `--trace`. + +## Inputs + +Trace JSONL: + +- Expected formatted fields: `chat_id`, `parent_chat_id`, `timestamp`, + `input_length`, `output_length`, `type`, `turn`, `hash_ids`, optional + `session_id`. +- If `session_id` is absent, sessions are reconstructed from + `parent_chat_id` chains. +- `timestamp` is treated as scheduled trace time, not proof of actual dispatch + time. + +Metrics JSONL: + +- Expected replayer fields: `request_id`, `session_id`, `turn_id`, + `trace_timestamp_s`, `input_length`, `output_length`, `cached_tokens`, + `latency_s`, `ttft_s`, `tpot_s`, `actual_output_tokens`, `error`. +- If the metrics file is from the current replayer, it does not include actual + dispatch/finish wall-clock timestamps. Batch 0 will therefore mark actual + session sequentiality as unavailable and separately report a scheduled + estimate from `trace_timestamp_s + latency_s`. + +Proxy breakdown: + +- Optional JSON/JSONL with fields such as `request_id`, `t_proxy_recv`, + `t_first_token`, `t_done`, `cache_hit`, `estimated_new_tokens`, + `route_class`, `routed_to`, `policy`. +- Batch 0 can prove actual per-session in-flight concurrency only when these + timing rows can be joined to analyzed requests by `request_id`. +- Existing proxy breakdown artifacts may not contain `session_id`; without a + request-id join to trace/metrics, they can still support append/cache-hit + statistics but not per-session concurrency. + +Run config: + +- Optional JSON, usually `outputs//config.json`. +- Used for manifest fields such as `policy`, `time_scale`, and request count + when available. + +## Commands + +Trace-only dry run: + +```bash +python3 analysis/characterization/analyze.py \ + --trace traces/w600_r0.0015_st30.jsonl \ + --task-name w600_trace_only \ + --overwrite +``` + +Trace plus replayer metrics: + +```bash +python3 analysis/characterization/analyze.py \ + --trace traces/w600_r0.0015_st30.jsonl \ + --metrics outputs/smoke_test/metrics.jsonl \ + --task-name smoke_trace_metrics \ + --overwrite +``` + +Proxy breakdown append/cache analysis: + +```bash +python3 analysis/characterization/analyze.py \ + --breakdown outputs/contention_16s_elastic/breakdown.json \ + --config outputs/contention_16s_elastic/config.json \ + --task-name contention_breakdown \ + --overwrite +``` + +Full trace on dash0, CPU-only: + +```bash +python3 analysis/characterization/analyze.py \ + --trace ~/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl \ + --task-name full_trace_characterization \ + --overwrite +``` + +Local run after copying from dash0: + +```bash +rsync -av dash0:~/ali-trace/trace-glm5.1-formatted/.jsonl traces/ +python3 analysis/characterization/analyze.py \ + --trace traces/.jsonl \ + --task-name full_trace_characterization \ + --overwrite +``` + +By default the analyzer records file size and mtime but skips full SHA256 +hashing, because canonical raw trace files can be hundreds of GiB. Add +`--hash-inputs` only when you intentionally want a full file hash. + +KV footprint requires a model-specific value: + +```bash +python3 analysis/characterization/analyze.py \ + --trace traces/w600_r0.0015_st30.jsonl \ + --kv-bytes-per-token 98304 \ + --task-name w600_with_kv_estimate \ + --overwrite +``` + +Summarize existing completed runs: + +```bash +python3 analysis/characterization/summarize_runs.py +``` + +This writes: + +```text +analysis/characterization/current_results/ +├── run_summaries.json +├── comparisons.json +├── claim_matrix.json +├── reviewer_risk_register.json +├── current_results.md +├── characterization_claim_matrix.md +├── all_figures_index.md +├── reviewer_risk_register.md +└── reproduction_commands.sh +``` + +## Batch 0 Semantics + +The online-serving invariant is: + +```text +Each session has at most one in-flight turn. +``` + +The analyzer reports: + +- actual interval status from dispatch and finish/error timestamps; +- scheduled estimate from trace timestamps plus latency when available; +- per-session max in-flight; +- session start-time distribution; +- turn inter-arrival distribution; +- attempted/completed/error counts and goodput when metrics exist; +- run classification. + +Important limitation: trace timestamps alone cannot prove actual replay +sequentiality. A run is only classified as `online_realistic` when actual +per-request dispatch and finish/error timestamps prove +`max_inflight_per_session <= 1`. + +## Batch 1 Semantics + +The analyzer reports: + +- input/output CDF stats; +- input/output ratio; +- KV footprint CDF stats when `--kv-bytes-per-token` is supplied; +- session skew and top-session contribution; +- append/uncached token stats when `cached_tokens` or `cache_hit` exists; +- reuse decomposition when both cached-token fields and `hash_ids` exist. + +Reuse decomposition is conservative: + +- `intra_session`: cached hash block was seen earlier in the same session; +- `cross_session`: cached hash block was seen earlier in another session; +- `shared/system-prefix`: early-position block appears in many sessions; +- `unclassified`: cached tokens could not be mapped to a previously seen hash + block. + +If cached-token/cache-hit fields are absent, reuse and append artifacts are +written with `status: "unavailable"` and list the required fields. + +## Limitations + +- The script does not run a benchmark, query a live service, touch GPU state, + or start any daemon. +- Request-id joins are exact. If trace, metrics, and proxy artifacts use + different request IDs, the unmatched rows are preserved under `raw/`. +- Actual Batch 0 sequentiality needs actual dispatch and finish/error + timestamps. Current `replayer/metrics.py` metrics are not enough by + themselves. +- `kv_bytes_per_token` depends on model architecture, layer count, KV heads, + head dimension, and dtype. The analyzer will not guess it. +- Shared/system-prefix reuse classification is a heuristic based on trace + `hash_ids` positions and cross-session frequency. Adjust + `--shared-prefix-min-sessions` and `--system-prefix-blocks` if the formatted + trace provides a stronger system-prefix marker. diff --git a/analysis/characterization/analyze.py b/analysis/characterization/analyze.py new file mode 100644 index 0000000..ccb510d --- /dev/null +++ b/analysis/characterization/analyze.py @@ -0,0 +1,1873 @@ +#!/usr/bin/env python3 +"""CPU-only Batch 0/1 characterization analyzer. + +The script is intentionally conservative about timestamp claims. Actual +per-session in-flight concurrency is only marked available when each analyzed +request has both a dispatch timestamp and a finish/error timestamp. When a run +only has trace timestamps, or only has trace timestamps plus latency, the script +emits an explicit unavailable/estimate status instead of treating it as proof of +online sequentiality. +""" + +from __future__ import annotations + +import argparse +import collections +import datetime as dt +import hashlib +import json +import math +import platform +import statistics +import subprocess +import sys +from pathlib import Path +from typing import Any + + +JsonDict = dict[str, Any] + + +DISPATCH_FIELDS = [ + "dispatch_timestamp_s", + "start_timestamp_s", + "request_start_s", + "t_dispatch", + "t_proxy_recv", +] +FIRST_TOKEN_FIELDS = [ + "first_token_timestamp_s", + "t_first_token", +] +FINISH_FIELDS = [ + "finish_timestamp_s", + "completion_timestamp_s", + "end_timestamp_s", + "t_done", +] +ERROR_TIME_FIELDS = [ + "error_timestamp_s", + "timeout_timestamp_s", + "t_error", + "t_timeout", +] +CACHED_TOKEN_FIELDS = [ + "cached_tokens", + "cache_hit", + "prefix_cache_hit_tokens", +] + + +def main() -> None: + args = parse_args() + started_at = dt.datetime.now(dt.timezone.utc) + + config = load_optional_json_object(args.config) + metrics_summary = load_metrics_summary(args.metrics, args.metrics_summary) + task_name = args.task_name or infer_task_name(args) + run_date = args.date or dt.datetime.now().strftime("%Y-%m-%d") + out_dir = args.output_root / run_date / task_name + prepare_output_dir(out_dir, args.overwrite) + (out_dir / "raw").mkdir(parents=True, exist_ok=True) + (out_dir / "figures").mkdir(parents=True, exist_ok=True) + + trace_records, trace_warnings = load_trace_records(args.trace, args.request_limit) + metrics_records = load_metrics_records(args.metrics) + breakdown_records = load_breakdown_records(args.breakdown) + + merge_result = merge_records(trace_records, metrics_records, breakdown_records) + records = merge_result["records"] + unmatched_metrics = merge_result["unmatched_metrics"] + unmatched_breakdown = merge_result["unmatched_breakdown"] + + write_jsonl(out_dir / "raw" / "merged_requests.jsonl", records) + write_jsonl(out_dir / "raw" / "unmatched_metrics.jsonl", unmatched_metrics) + write_jsonl(out_dir / "raw" / "unmatched_breakdown.jsonl", unmatched_breakdown) + + batch0 = analyze_batch0(records, metrics_summary, config, args.classification) + batch1 = analyze_batch1( + records=records, + unmatched_breakdown=unmatched_breakdown, + kv_bytes_per_token=args.kv_bytes_per_token, + block_size=args.block_size, + shared_prefix_min_sessions=args.shared_prefix_min_sessions, + system_prefix_blocks=args.system_prefix_blocks, + ) + + write_json(out_dir / "session_concurrency.json", batch0["session_concurrency"]) + write_json(out_dir / "session_arrival_stats.json", batch0["session_arrival_stats"]) + write_json(out_dir / "turn_interval_stats.json", batch0["turn_interval_stats"]) + write_json(out_dir / "trace_profile.json", batch0["trace_profile"]) + (out_dir / "invalid_runs.md").write_text( + render_invalid_runs(batch0), + encoding="utf-8", + ) + + write_json(out_dir / "workload_summary.json", batch1["workload_summary"]) + write_json(out_dir / "kv_footprint_summary.json", batch1["kv_footprint_summary"]) + write_json(out_dir / "reuse_decomposition.json", batch1["reuse_decomposition"]) + write_json(out_dir / "session_skew.json", batch1["session_skew"]) + write_json(out_dir / "append_delta_stats.json", batch1["append_delta_stats"]) + + figure_status = generate_figures(out_dir, records, batch0, batch1, args.no_figures) + finished_at = dt.datetime.now(dt.timezone.utc) + + manifest = build_manifest( + args=args, + config=config, + out_dir=out_dir, + started_at=started_at, + finished_at=finished_at, + batch0=batch0, + ) + manifest["figure_status"] = figure_status + manifest["input_status"] = { + "trace_records": len(trace_records), + "metrics_records": len(metrics_records), + "breakdown_records": len(breakdown_records), + "analyzed_records": len(records), + "unmatched_metrics": len(unmatched_metrics), + "unmatched_breakdown": len(unmatched_breakdown), + "trace_warnings": trace_warnings, + "merge_warnings": merge_result["warnings"], + } + write_json(out_dir / "manifest.json", manifest) + + summary = { + "classification": batch0["classification"], + "analyzed_records": len(records), + "manifest": manifest, + "batch0": { + "session_concurrency_status": batch0["session_concurrency"]["status"], + "session_sequential": batch0["session_concurrency"]["session_sequential"], + "max_inflight_per_session": batch0["session_concurrency"]["max_inflight_per_session"], + "attempted_requests": batch0["trace_profile"]["attempted_requests"], + "completed_requests": batch0["trace_profile"]["completed_requests"], + "error_requests": batch0["trace_profile"]["error_requests"], + }, + "batch1": { + "input_stats": batch1["workload_summary"]["input_tokens"], + "output_stats": batch1["workload_summary"]["output_tokens"], + "kv_footprint_status": batch1["kv_footprint_summary"]["status"], + "reuse_status": batch1["reuse_decomposition"]["status"], + "append_status": batch1["append_delta_stats"]["status"], + }, + "outputs": relative_output_list(out_dir), + } + write_json(out_dir / "summary.json", summary) + (out_dir / "summary.md").write_text( + render_summary_md(summary, batch0, batch1), + encoding="utf-8", + ) + (out_dir / "audit.md").write_text( + render_audit_md(batch0, batch1), + encoding="utf-8", + ) + + print(f"Wrote characterization outputs to {out_dir}") + print(f"Run classification: {batch0['classification']['label']}") + print(f"Session sequentiality: {batch0['session_concurrency']['session_sequential']}") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Analyze Batch 0/1 workload characterization artifacts.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + parser.add_argument("--trace", type=Path, default=None, + help="Trace JSONL path. Supports Ali trace fields used by replayer.trace.") + parser.add_argument("--metrics", type=Path, default=None, + help="Per-request metrics JSONL path.") + parser.add_argument("--metrics-summary", type=Path, default=None, + help="metrics.summary.json path. If omitted, inferred from --metrics when possible.") + parser.add_argument("--breakdown", type=Path, default=None, + help="Proxy breakdown JSON/JSONL path with timing/cache fields.") + parser.add_argument("--config", type=Path, default=None, + help="Optional run config JSON.") + parser.add_argument("--output-root", type=Path, + default=Path("outputs/characterization"), + help="Output root. Final path is //.") + parser.add_argument("--date", default=None, + help="Date component for output path. Defaults to local YYYY-MM-DD.") + parser.add_argument("--task-name", default=None, + help="Task/run name for output path.") + parser.add_argument("--request-limit", type=int, default=None, + help="Limit trace records read for a small dry run.") + parser.add_argument("--policy", default=None, + help="Policy label to record in manifest.") + parser.add_argument("--launch-command", default=None, + help="Original launch command to record in manifest.") + parser.add_argument("--time-scale", type=float, default=None, + help="Replay time scale, if known.") + parser.add_argument("--session-sampling-method", default="", + help="Session sampling method label.") + parser.add_argument("--hash-inputs", action="store_true", + help=( + "Compute full SHA256 for input files. Disabled by default because " + "canonical raw traces can be hundreds of GiB." + )) + parser.add_argument("--kv-bytes-per-token", type=float, default=0.0, + help="Model-specific KV bytes per input token. 0 marks KV footprint unavailable.") + parser.add_argument("--block-size", type=int, default=512, + help="Trace hash block size in tokens.") + parser.add_argument("--shared-prefix-min-sessions", type=int, default=8, + help="Hash block session-count threshold for shared/system-prefix reuse.") + parser.add_argument("--system-prefix-blocks", type=int, default=4, + help="Only block positions below this index can be classified as shared/system-prefix.") + parser.add_argument("--classification", default="auto", + choices=[ + "auto", + "online_realistic", + "burst_stress", + "synthetic_microbench", + "invalid_for_online_claim", + ], + help="Run classification override.") + parser.add_argument("--gpu-type", default="", + help="GPU type for manifest. Leave empty for CPU-only trace analysis.") + parser.add_argument("--gpu-count", type=int, default=0, + help="GPU count for manifest. Use 0 for CPU-only trace analysis.") + parser.add_argument("--overwrite", action="store_true", + help="Overwrite an existing output directory.") + parser.add_argument("--no-figures", action="store_true", + help="Skip matplotlib figure generation.") + args = parser.parse_args() + if args.trace is None and args.metrics is None and args.breakdown is None: + parser.error("provide at least one of --trace, --metrics, or --breakdown") + if args.block_size <= 0: + parser.error("--block-size must be positive") + return args + + +def prepare_output_dir(out_dir: Path, overwrite: bool) -> None: + if out_dir.exists() and any(out_dir.iterdir()) and not overwrite: + raise SystemExit( + f"output directory already exists and is not empty: {out_dir}\n" + "pass --overwrite or choose a different --task-name" + ) + out_dir.mkdir(parents=True, exist_ok=True) + + +def infer_task_name(args: argparse.Namespace) -> str: + for path in [args.trace, args.metrics, args.breakdown]: + if path is not None: + return path.stem.replace(".", "_") + return "characterization" + + +def load_optional_json_object(path: Path | None) -> JsonDict: + if path is None or not path.exists(): + return {} + data = json.loads(path.read_text(encoding="utf-8")) + return data if isinstance(data, dict) else {} + + +def load_metrics_summary(metrics: Path | None, explicit: Path | None) -> JsonDict: + candidates: list[Path] = [] + if explicit is not None: + candidates.append(explicit) + if metrics is not None: + if metrics.name == "metrics.jsonl": + candidates.append(metrics.with_name("metrics.summary.json")) + else: + candidates.append(metrics.with_suffix(".summary.json")) + for path in candidates: + if path.exists(): + data = json.loads(path.read_text(encoding="utf-8")) + return data if isinstance(data, dict) else {} + return {} + + +def load_trace_records(path: Path | None, request_limit: int | None) -> tuple[list[JsonDict], list[str]]: + if path is None: + return [], [] + warnings: list[str] = [] + records: list[JsonDict] = [] + chat_to_session: dict[int, str] = {} + with path.open("r", encoding="utf-8") as handle: + for idx, line in enumerate(handle): + if request_limit is not None and len(records) >= request_limit: + break + if not line.strip(): + continue + row = json.loads(line) + chat_id_raw = first_present(row, ["chat_id", "id"]) + parent_raw = first_present(row, ["parent_chat_id", "parent_id"], -1) + chat_id = to_int(chat_id_raw, idx) + parent_chat_id = to_int(parent_raw, -1) + session_id = row.get("session_id") + if session_id is None: + if parent_chat_id < 0: + session_id = str(chat_id) + else: + session_id = chat_to_session.get(parent_chat_id, str(parent_chat_id)) + chat_to_session[chat_id] = str(session_id) + + turn_id = to_int(first_present(row, ["turn", "turn_id"], idx), idx) + scheduled_dispatch = to_float(first_present(row, ["timestamp", "trace_timestamp_s"])) + input_tokens = to_int(first_present(row, ["input_length", "input_tokens", "prompt_tokens"])) + output_tokens = to_int(first_present(row, ["output_length", "output_tokens", "completion_tokens"])) + hash_ids = row.get("hash_ids", []) + if not isinstance(hash_ids, list): + warnings.append(f"row {idx}: hash_ids is not a list; ignoring") + hash_ids = [] + + request_id = str( + row.get("request_id") + or f"{session_id}:{turn_id}:{chat_id}:{idx}" + ) + records.append({ + "request_id": request_id, + "session_id": str(session_id), + "turn_id": turn_id, + "chat_id": chat_id, + "parent_chat_id": parent_chat_id, + "scheduled_dispatch_s": scheduled_dispatch, + "input_tokens": input_tokens, + "output_tokens": output_tokens, + "request_type": str(row.get("type", row.get("request_type", ""))), + "hash_ids": [to_int(h) for h in hash_ids if to_int(h) is not None], + "trace_present": True, + "metrics_present": False, + "breakdown_present": False, + "source_index": idx, + }) + return records, warnings + + +def load_metrics_records(path: Path | None) -> list[JsonDict]: + if path is None or not path.exists(): + return [] + records: list[JsonDict] = [] + with path.open("r", encoding="utf-8") as handle: + for idx, line in enumerate(handle): + if not line.strip(): + continue + row = json.loads(line) + records.append(metrics_row_to_record(row, idx)) + return records + + +def metrics_row_to_record(row: JsonDict, idx: int) -> JsonDict: + request_id = str(row.get("request_id") or f"metrics:{idx}") + session_id = row.get("session_id") + scheduled_dispatch = to_float(first_present(row, ["trace_timestamp_s", "timestamp"])) + input_tokens = to_int(first_present(row, ["input_length", "input_tokens", "prompt_tokens"])) + output_tokens = to_int(first_present(row, ["output_length", "requested_output_tokens", "output_tokens"])) + actual_dispatch = to_float(first_present(row, DISPATCH_FIELDS)) + ttft = to_float(row.get("ttft_s")) + latency = to_float(row.get("latency_s")) + first_token = to_float(first_present(row, FIRST_TOKEN_FIELDS)) + finish = to_float(first_present(row, FINISH_FIELDS)) + error_time = to_float(first_present(row, ERROR_TIME_FIELDS)) + if actual_dispatch is not None and first_token is None and ttft is not None: + first_token = actual_dispatch + ttft + if actual_dispatch is not None and finish is None and latency is not None: + finish = actual_dispatch + latency + + prompt_details = row.get("prompt_tokens_details") + cached_from_details = None + if isinstance(prompt_details, dict): + cached_from_details = to_int(prompt_details.get("cached_tokens")) + cached_tokens = to_int(first_present(row, CACHED_TOKEN_FIELDS, cached_from_details)) + + record: JsonDict = { + "request_id": request_id, + "session_id": str(session_id) if session_id is not None else None, + "turn_id": to_int(row.get("turn_id")), + "scheduled_dispatch_s": scheduled_dispatch, + "input_tokens": input_tokens, + "output_tokens": output_tokens, + "request_type": str(row.get("request_type", "")), + "hash_ids": [], + "trace_present": False, + "metrics_present": True, + "breakdown_present": False, + "source_index": idx, + "effective_input_length": to_int(row.get("effective_input_length")), + "cached_tokens": cached_tokens, + "latency_s": latency, + "ttft_s": ttft, + "tpot_s": to_float(row.get("tpot_s")), + "actual_output_tokens": to_int(row.get("actual_output_tokens")), + "requested_output_tokens": to_int(row.get("requested_output_tokens")), + "finish_reason": row.get("finish_reason"), + "error": row.get("error"), + "actual_dispatch_s": actual_dispatch, + "actual_first_token_s": first_token, + "actual_finish_s": finish, + "actual_error_s": error_time, + } + if scheduled_dispatch is not None and latency is not None: + record["scheduled_finish_estimate_s"] = scheduled_dispatch + latency + if scheduled_dispatch is not None and ttft is not None: + record["scheduled_first_token_estimate_s"] = scheduled_dispatch + ttft + return record + + +def load_breakdown_records(path: Path | None) -> list[JsonDict]: + if path is None or not path.exists(): + return [] + text = path.read_text(encoding="utf-8").strip() + if not text: + return [] + raw: Any + if text[0] in "[{": + raw = json.loads(text) + if isinstance(raw, dict): + raw = raw.get("records", [raw]) + else: + raw = [json.loads(line) for line in text.splitlines() if line.strip()] + records: list[JsonDict] = [] + for idx, row in enumerate(raw if isinstance(raw, list) else []): + if not isinstance(row, dict): + continue + request_id = str(row.get("request_id") or f"breakdown:{idx}") + actual_dispatch = to_float(first_present(row, DISPATCH_FIELDS)) + first_token = to_float(first_present(row, FIRST_TOKEN_FIELDS)) + finish = to_float(first_present(row, FINISH_FIELDS)) + error_time = to_float(first_present(row, ERROR_TIME_FIELDS)) + cached_tokens = to_int(first_present(row, CACHED_TOKEN_FIELDS)) + input_tokens = to_int(first_present(row, ["input_length", "input_tokens", "prompt_tokens"])) + estimated_new = to_int(first_present(row, ["estimated_new_tokens", "uncached_tokens", "new_tokens"])) + records.append({ + "request_id": request_id, + "session_id": str(row["session_id"]) if "session_id" in row else None, + "turn_id": to_int(first_present(row, ["turn_id", "turn"])), + "scheduled_dispatch_s": to_float(first_present(row, ["trace_timestamp_s", "timestamp"])), + "input_tokens": input_tokens, + "output_tokens": to_int(first_present(row, ["output_length", "output_tokens"])), + "hash_ids": [], + "trace_present": False, + "metrics_present": False, + "breakdown_present": True, + "source_index": idx, + "cached_tokens": cached_tokens, + "estimated_new_tokens": estimated_new, + "actual_dispatch_s": actual_dispatch, + "actual_first_token_s": first_token, + "actual_finish_s": finish, + "actual_error_s": error_time, + "route_class": row.get("route_class"), + "policy": row.get("policy"), + "routed_to": row.get("routed_to"), + }) + return records + + +def merge_records( + trace_records: list[JsonDict], + metrics_records: list[JsonDict], + breakdown_records: list[JsonDict], +) -> JsonDict: + warnings: list[str] = [] + canonical = [dict(r) for r in (trace_records or metrics_records or breakdown_records)] + by_id: dict[str, JsonDict] = {str(r["request_id"]): r for r in canonical} + + unmatched_metrics: list[JsonDict] = [] + if trace_records: + for metric in metrics_records: + target = by_id.get(str(metric["request_id"])) + if target is None: + unmatched_metrics.append(metric) + continue + overlay_record(target, metric, "metrics") + else: + unmatched_metrics = [] + + unmatched_breakdown: list[JsonDict] = [] + if trace_records or metrics_records: + for item in breakdown_records: + target = by_id.get(str(item["request_id"])) + if target is None: + unmatched_breakdown.append(item) + continue + overlay_record(target, item, "breakdown") + else: + unmatched_breakdown = [] + + if trace_records and metrics_records and len(unmatched_metrics) == len(metrics_records): + warnings.append( + "no metrics request_id matched trace request_id; metrics were saved under raw/unmatched_metrics.jsonl" + ) + if (trace_records or metrics_records) and breakdown_records and len(unmatched_breakdown) == len(breakdown_records): + warnings.append( + "no breakdown request_id matched analyzed records; breakdown rows were saved under raw/unmatched_breakdown.jsonl" + ) + + return { + "records": canonical, + "unmatched_metrics": unmatched_metrics, + "unmatched_breakdown": unmatched_breakdown, + "warnings": warnings, + } + + +def overlay_record(target: JsonDict, source: JsonDict, source_name: str) -> None: + target[f"{source_name}_present"] = True + for key, value in source.items(): + if key in {"request_id", "source_index", "trace_present", "metrics_present", "breakdown_present"}: + continue + if value is None: + continue + if key == "hash_ids" and not value: + continue + if target.get(key) is None or key in { + "cached_tokens", + "latency_s", + "ttft_s", + "tpot_s", + "actual_output_tokens", + "requested_output_tokens", + "finish_reason", + "error", + "actual_dispatch_s", + "actual_first_token_s", + "actual_finish_s", + "actual_error_s", + "scheduled_finish_estimate_s", + "scheduled_first_token_estimate_s", + "estimated_new_tokens", + "route_class", + "policy", + "routed_to", + }: + target[key] = value + + +def analyze_batch0( + records: list[JsonDict], + metrics_summary: JsonDict, + config: JsonDict, + classification_override: str, +) -> JsonDict: + trace_profile = build_trace_profile(records, metrics_summary) + actual_concurrency = compute_session_concurrency( + records, + start_key="actual_dispatch_s", + finish_key="actual_finish_s", + error_key="actual_error_s", + ) + scheduled_estimate = compute_session_concurrency( + records, + start_key="scheduled_dispatch_s", + finish_key="scheduled_finish_estimate_s", + error_key=None, + ) + + session_concurrency = { + **actual_concurrency, + "timestamp_requirements": { + "dispatch_timestamp_fields": DISPATCH_FIELDS, + "finish_timestamp_fields": FINISH_FIELDS, + "error_timestamp_fields": ERROR_TIME_FIELDS, + "note": ( + "Actual max in-flight is only conclusive when dispatch and finish/error " + "timestamps exist per request. scheduled_estimate uses trace_timestamp_s " + "plus latency_s and is not proof of replay sequentiality." + ), + }, + "scheduled_estimate": scheduled_estimate, + } + + arrival_stats = compute_session_arrivals(records) + turn_interval_stats = compute_turn_intervals(records) + classification = classify_run( + override=classification_override, + session_concurrency=session_concurrency, + trace_profile=trace_profile, + config=config, + ) + return { + "session_concurrency": session_concurrency, + "session_arrival_stats": arrival_stats, + "turn_interval_stats": turn_interval_stats, + "trace_profile": trace_profile, + "classification": classification, + } + + +def build_trace_profile(records: list[JsonDict], metrics_summary: JsonDict) -> JsonDict: + records_with_session = [r for r in records if r.get("session_id") is not None] + sessions = group_by_session(records) + scheduled = sorted(clean_numbers(r.get("scheduled_dispatch_s") for r in records)) + input_tokens = clean_numbers(r.get("input_tokens") for r in records) + output_tokens = clean_numbers(r.get("output_tokens") for r in records) + request_types = collections.Counter(str(r.get("request_type", "")) for r in records) + errors = [r for r in records if r.get("error")] + metrics_present = [r for r in records if r.get("metrics_present")] + successes = [r for r in metrics_present if not r.get("error")] + timing_completed = [ + r for r in records + if to_float(r.get("actual_finish_s")) is not None + ] + timing_errors = [ + r for r in records + if to_float(r.get("actual_error_s")) is not None + ] + completed_count = len(successes) if metrics_present else ( + len(timing_completed) if timing_completed or timing_errors else None + ) + error_count = len(errors) if metrics_present else ( + len(timing_errors) if timing_completed or timing_errors else None + ) + wall_clock = to_float(metrics_summary.get("wall_clock_s")) + if wall_clock is None and scheduled: + wall_clock = scheduled[-1] - scheduled[0] + if wall_clock is None: + actual_starts = clean_numbers(r.get("actual_dispatch_s") for r in records) + actual_ends = clean_numbers( + first_present(r, ["actual_finish_s", "actual_error_s"]) for r in records + ) + if actual_starts and actual_ends: + wall_clock = max(actual_ends) - min(actual_starts) + latency_stats = stats( + [to_float(r.get("latency_s")) for r in successes if to_float(r.get("latency_s")) is not None] + ) + ttft_stats = stats( + [to_float(r.get("ttft_s")) for r in successes if to_float(r.get("ttft_s")) is not None] + ) + tpot_stats = stats( + [to_float(r.get("tpot_s")) for r in successes if to_float(r.get("tpot_s")) is not None] + ) + hash_counts = [len(r.get("hash_ids") or []) for r in records] + unique_hashes: set[int] = set() + hash_refcount: collections.Counter[int] = collections.Counter() + for r in records: + for hid in r.get("hash_ids") or []: + unique_hashes.add(int(hid)) + hash_refcount[int(hid)] += 1 + shared_hashes = sum(1 for count in hash_refcount.values() if count > 1) + session_turns = [len(v) for v in sessions.values()] + return { + "attempted_requests": len(records), + "completed_requests": completed_count, + "error_requests": error_count, + "metrics_available": bool(metrics_present), + "records_with_session": len(records_with_session), + "session_count": len(sessions), + "multi_turn_sessions": sum(1 for turn_count in session_turns if turn_count > 1), + "turns_per_session": stats(session_turns), + "trace_span_s": (scheduled[-1] - scheduled[0]) if len(scheduled) >= 2 else 0.0, + "request_rate_per_s": len(records) / (scheduled[-1] - scheduled[0]) if len(scheduled) >= 2 and scheduled[-1] > scheduled[0] else None, + "goodput_per_s": completed_count / wall_clock if completed_count is not None and wall_clock and wall_clock > 0 else None, + "latency_stats_s": latency_stats, + "ttft_stats_s": ttft_stats, + "tpot_stats_s": tpot_stats, + "latency_percentiles_over_successes_only": bool(metrics_present), + "input_tokens": stats(input_tokens), + "output_tokens": stats(output_tokens), + "request_type_counts": dict(request_types), + "hash_block_stats": { + "records_with_hash_ids": sum(1 for count in hash_counts if count > 0), + "unique_hash_blocks": len(unique_hashes), + "shared_hash_blocks": shared_hashes, + "shared_hash_block_fraction": shared_hashes / len(unique_hashes) if unique_hashes else None, + "hash_blocks_per_request": stats(hash_counts), + }, + "comparison_metric_availability": { + "per_worker_queue_metrics": { + "status": "unavailable", + "needed_input": "per-worker queue depth/delay metrics exported by proxy or engine", + }, + "per_worker_gpu_utilization": { + "status": "unavailable", + "needed_input": "gpu_util.csv or equivalent per-worker utilization artifact", + }, + "per_worker_kv_occupancy": { + "status": "unavailable", + "needed_input": "per-worker KV occupancy metrics from engine /metrics snapshots", + }, + "per_worker_apc_cache_hit": { + "status": ( + "aggregate_available" + if "prefix_cache_hit_ratio" in metrics_summary + else "unavailable" + ), + "needed_input": "per-worker APC/cache-hit counters; metrics.summary.json only provides aggregate APC when present", + }, + }, + "metrics_summary_fields": sorted(metrics_summary.keys()), + } + + +def compute_session_concurrency( + records: list[JsonDict], + *, + start_key: str, + finish_key: str, + error_key: str | None, +) -> JsonDict: + records_with_session = [r for r in records if r.get("session_id") is not None] + intervals: dict[str, list[tuple[float, float, str]]] = collections.defaultdict(list) + missing_start = 0 + missing_finish = 0 + for r in records_with_session: + start = to_float(r.get(start_key)) + finish = to_float(r.get(finish_key)) + if finish is None and error_key is not None: + finish = to_float(r.get(error_key)) + if start is None: + missing_start += 1 + continue + if finish is None: + missing_finish += 1 + continue + if finish < start: + finish = start + intervals[str(r["session_id"])].append((start, finish, str(r.get("request_id", "")))) + + per_session: list[JsonDict] = [] + max_seen: int | None = None + violations: list[JsonDict] = [] + overlap_examples: list[JsonDict] = [] + for sid, items in intervals.items(): + events: list[tuple[float, int]] = [] + for start, finish, _ in items: + events.append((start, 1)) + events.append((finish, -1)) + events.sort(key=lambda item: (item[0], item[1])) + cur = 0 + max_cur = 0 + for _, delta in events: + cur += delta + max_cur = max(max_cur, cur) + max_seen = max(max_seen or 0, max_cur) + sorted_items = sorted(items) + session_overlaps = 0 + prev_finish = None + prev_id = None + for start, finish, req_id in sorted_items: + if prev_finish is not None and start < prev_finish: + session_overlaps += 1 + if len(overlap_examples) < 20: + overlap_examples.append({ + "session_id": sid, + "previous_request_id": prev_id, + "request_id": req_id, + "previous_finish_s": prev_finish, + "request_start_s": start, + "overlap_s": prev_finish - start, + }) + if prev_finish is None or finish > prev_finish: + prev_finish = finish + prev_id = req_id + item = { + "session_id": sid, + "interval_count": len(items), + "max_inflight": max_cur, + "overlap_count": session_overlaps, + } + per_session.append(item) + if max_cur > 1: + violations.append(item) + + covered = sum(len(v) for v in intervals.values()) + if not records_with_session: + status = "unavailable" + elif covered == len(records_with_session): + status = "available" + elif covered > 0: + status = "partial" + else: + status = "unavailable" + + session_sequential = None + if status == "available": + session_sequential = bool((max_seen or 0) <= 1) + elif status == "partial" and max_seen and max_seen > 1: + session_sequential = False + + per_session.sort(key=lambda r: (-int(r["max_inflight"]), -int(r["interval_count"]), str(r["session_id"]))) + return { + "status": status, + "start_field": start_key, + "finish_field": finish_key, + "records_with_session": len(records_with_session), + "intervals_with_timestamps": covered, + "missing_dispatch_timestamps": missing_start, + "missing_finish_or_error_timestamps": missing_finish, + "session_sequential": session_sequential, + "max_inflight_per_session": max_seen if status in {"available", "partial"} else None, + "sessions_with_overlap": len(violations), + "per_session": per_session, + "overlap_examples": overlap_examples, + "unavailable_reason": ( + None if status == "available" + else "missing dispatch and/or finish/error timestamps for at least one request" + ), + } + + +def compute_session_arrivals(records: list[JsonDict]) -> JsonDict: + sessions = group_by_session(records) + starts: list[float] = [] + turns_per_session: list[int] = [] + for rows in sessions.values(): + times = clean_numbers(r.get("scheduled_dispatch_s") for r in rows) + if not times: + continue + starts.append(min(times)) + turns_per_session.append(len(rows)) + starts.sort() + if not starts: + return { + "status": "unavailable", + "reason": "missing session_id or scheduled_dispatch_s/timestamp", + } + base = starts[0] + offsets = [v - base for v in starts] + inter_arrivals = [starts[i + 1] - starts[i] for i in range(len(starts) - 1)] + return { + "status": "available", + "session_count": len(starts), + "session_start_offsets_s": offsets, + "session_start_offset_stats_s": stats(offsets), + "session_interarrival_stats_s": stats(inter_arrivals), + "arrival_span_s": starts[-1] - starts[0] if len(starts) > 1 else 0.0, + "mean_session_arrival_rate_per_s": len(starts) / (starts[-1] - starts[0]) if len(starts) > 1 and starts[-1] > starts[0] else None, + "turns_per_session": stats(turns_per_session), + } + + +def compute_turn_intervals(records: list[JsonDict]) -> JsonDict: + sessions = group_by_session(records) + scheduled_intervals: list[float] = [] + actual_gaps: list[float] = [] + missing_actual = 0 + for rows in sessions.values(): + rows = sorted( + rows, + key=lambda r: ( + value_or_inf(r.get("turn_id")), + value_or_inf(r.get("scheduled_dispatch_s")), + str(r.get("request_id", "")), + ), + ) + for prev, cur in zip(rows, rows[1:]): + prev_start = to_float(prev.get("scheduled_dispatch_s")) + cur_start = to_float(cur.get("scheduled_dispatch_s")) + if prev_start is not None and cur_start is not None: + scheduled_intervals.append(cur_start - prev_start) + prev_finish = to_float(prev.get("actual_finish_s")) + cur_actual_start = to_float(cur.get("actual_dispatch_s")) + if prev_finish is not None and cur_actual_start is not None: + actual_gaps.append(cur_actual_start - prev_finish) + else: + missing_actual += 1 + return { + "status": "available" if scheduled_intervals else "unavailable", + "scheduled_turn_intervals_s": stats(scheduled_intervals), + "scheduled_turn_interval_values_s": scheduled_intervals, + "actual_turn_gaps_status": "available" if actual_gaps and missing_actual == 0 else "partial" if actual_gaps else "unavailable", + "actual_turn_gaps_s": stats(actual_gaps), + "actual_negative_gap_count": sum(1 for v in actual_gaps if v < 0), + "missing_actual_gap_pairs": missing_actual, + "note": "scheduled intervals use trace timestamps; actual gaps require dispatch and finish timestamps.", + } + + +def classify_run( + *, + override: str, + session_concurrency: JsonDict, + trace_profile: JsonDict, + config: JsonDict, +) -> JsonDict: + if override != "auto": + return { + "label": override, + "source": "user_override", + "reason": "classification override was provided", + } + time_scale = to_float(config.get("time_scale")) + max_sessions = first_present(config, ["max_sessions", "max_inflight_sessions"]) + stress_indicators = [] + if time_scale is not None and not math.isclose(time_scale, 1.0): + stress_indicators.append(f"time_scale={time_scale}") + if max_sessions is not None: + stress_indicators.append(f"max_sessions={max_sessions}") + + sequential = session_concurrency.get("session_sequential") + status = session_concurrency.get("status") + if sequential is True and not stress_indicators: + label = "online_realistic" + reason = "actual per-session interval timestamps prove max_inflight_per_session <= 1" + elif sequential is False: + label = "invalid_for_online_claim" + reason = "per-session interval overlap was observed" + elif stress_indicators: + label = "burst_stress" + reason = "stress indicators present and sequentiality is not proven: " + ", ".join(stress_indicators) + elif trace_profile.get("attempted_requests", 0) <= 10: + label = "synthetic_microbench" + reason = "small run without enough timestamp data to support online workload claims" + elif status != "available": + label = "invalid_for_online_claim" + reason = "actual dispatch/finish timestamps are unavailable, so online sequentiality cannot be proven" + else: + label = "invalid_for_online_claim" + reason = "run does not satisfy online-realistic criteria" + return { + "label": label, + "source": "auto", + "reason": reason, + "stress_indicators": stress_indicators, + } + + +def analyze_batch1( + *, + records: list[JsonDict], + unmatched_breakdown: list[JsonDict], + kv_bytes_per_token: float, + block_size: int, + shared_prefix_min_sessions: int, + system_prefix_blocks: int, +) -> JsonDict: + input_tokens = [v for v in clean_numbers(r.get("input_tokens") for r in records)] + output_tokens = [v for v in clean_numbers(r.get("output_tokens") for r in records)] + ratios = [ + inp / out + for inp, out in zip( + [to_float(r.get("input_tokens")) for r in records], + [to_float(r.get("output_tokens")) for r in records], + ) + if inp is not None and out is not None and out > 0 + ] + workload_summary = { + "status": "available" if input_tokens or output_tokens else "unavailable", + "request_count": len(records), + "input_tokens": stats(input_tokens), + "output_tokens": stats(output_tokens), + "input_output_ratio": stats(ratios), + "total_input_tokens": sum(input_tokens), + "total_output_tokens": sum(output_tokens), + "long_input_thresholds": { + "requests_gt_8k": count_gt(input_tokens, 8_000), + "requests_gt_32k": count_gt(input_tokens, 32_000), + "requests_gt_64k": count_gt(input_tokens, 64_000), + }, + } + + kv_footprint_summary = build_kv_footprint_summary(input_tokens, kv_bytes_per_token) + append_delta_stats = build_append_delta_stats(records, unmatched_breakdown) + session_skew = build_session_skew(records) + reuse_decomposition = build_reuse_decomposition( + records=records, + block_size=block_size, + shared_prefix_min_sessions=shared_prefix_min_sessions, + system_prefix_blocks=system_prefix_blocks, + ) + return { + "workload_summary": workload_summary, + "kv_footprint_summary": kv_footprint_summary, + "reuse_decomposition": reuse_decomposition, + "session_skew": session_skew, + "append_delta_stats": append_delta_stats, + } + + +def build_kv_footprint_summary(input_tokens: list[float], kv_bytes_per_token: float) -> JsonDict: + if kv_bytes_per_token <= 0: + return { + "status": "unavailable", + "reason": "kv_bytes_per_token was not provided", + "needed_input": "--kv-bytes-per-token ", + "formula": "kv_bytes_per_request = input_tokens * kv_bytes_per_token", + } + footprints = [v * kv_bytes_per_token for v in input_tokens] + return { + "status": "available", + "kv_bytes_per_token": kv_bytes_per_token, + "formula": "kv_bytes_per_request = input_tokens * kv_bytes_per_token", + "kv_bytes_per_request": stats(footprints), + "kv_mib_per_request": stats([v / (1024 * 1024) for v in footprints]), + "total_kv_gib": sum(footprints) / (1024 ** 3), + } + + +def build_append_delta_stats(records: list[JsonDict], unmatched_breakdown: list[JsonDict]) -> JsonDict: + rows = [r for r in records if has_number(r.get("input_tokens")) and has_number(r.get("cached_tokens"))] + source = "analyzed_records" + if not rows: + rows = [ + r for r in unmatched_breakdown + if has_number(r.get("input_tokens")) and has_number(r.get("cached_tokens")) + ] + source = "unmatched_breakdown_without_session_join" + if not rows: + return { + "status": "unavailable", + "reason": "missing per-request cached token field", + "needed_fields": ["cached_tokens or cache_hit", "input_length/input_tokens"], + "formula": "uncached_tokens = input_tokens - cached_tokens", + } + uncached = [] + cached = [] + append_ratios = [] + inputs = [] + long_small_append = 0 + for r in rows: + inp = to_float(r.get("input_tokens")) + cache = to_float(r.get("cached_tokens")) + if inp is None or cache is None: + continue + unc = max(inp - cache, 0.0) + inputs.append(inp) + cached.append(cache) + uncached.append(unc) + if inp > 0: + ratio = unc / inp + append_ratios.append(ratio) + if inp >= 32_000 and ratio <= 0.2: + long_small_append += 1 + return { + "status": "available", + "source": source, + "request_count": len(rows), + "input_tokens": stats(inputs), + "cached_tokens": stats(cached), + "uncached_tokens": stats(uncached), + "uncached_to_input_ratio": stats(append_ratios), + "total_input_tokens": sum(inputs), + "total_cached_tokens": sum(cached), + "total_uncached_tokens": sum(uncached), + "overall_cached_fraction": sum(cached) / sum(inputs) if sum(inputs) > 0 else None, + "long_prompt_small_append_count": long_small_append, + "long_prompt_small_append_fraction": long_small_append / len(rows) if rows else None, + "formula": "uncached_tokens = max(input_tokens - cached_tokens, 0)", + } + + +def build_session_skew(records: list[JsonDict]) -> JsonDict: + sessions = group_by_session(records) + if not sessions: + return { + "status": "unavailable", + "reason": "missing session_id", + } + rows: list[JsonDict] = [] + uncached_available = True + for sid, items in sessions.items(): + input_sum = sum(to_float(r.get("input_tokens")) or 0.0 for r in items) + output_sum = sum(to_float(r.get("output_tokens")) or 0.0 for r in items) + cached_values = [to_float(r.get("cached_tokens")) for r in items] + if any(v is None for v in cached_values): + uncached_available = False + uncached_sum = None + else: + uncached_sum = sum( + max((to_float(r.get("input_tokens")) or 0.0) - (to_float(r.get("cached_tokens")) or 0.0), 0.0) + for r in items + ) + rows.append({ + "session_id": sid, + "turns": len(items), + "input_tokens": input_sum, + "output_tokens": output_sum, + "uncached_tokens": uncached_sum, + }) + input_values = [float(r["input_tokens"]) for r in rows] + output_values = [float(r["output_tokens"]) for r in rows] + uncached_values = [float(r["uncached_tokens"]) for r in rows if r["uncached_tokens"] is not None] + rows_by_input = sorted(rows, key=lambda r: (-float(r["input_tokens"]), str(r["session_id"]))) + return { + "status": "available", + "session_count": len(rows), + "turns_per_session": stats([float(r["turns"]) for r in rows]), + "cumulative_input_tokens_per_session": stats(input_values), + "cumulative_output_tokens_per_session": stats(output_values), + "cumulative_uncached_tokens_per_session": stats(uncached_values) if uncached_available else None, + "uncached_tokens_status": "available" if uncached_available else "unavailable_missing_cached_tokens", + "top_session_contribution": { + "input_tokens": top_contribution(input_values), + "output_tokens": top_contribution(output_values), + "uncached_tokens": top_contribution(uncached_values) if uncached_available else None, + }, + "top_sessions_by_input_tokens": rows_by_input[:20], + "lorenz_input_tokens": lorenz_points(input_values), + } + + +def build_reuse_decomposition( + *, + records: list[JsonDict], + block_size: int, + shared_prefix_min_sessions: int, + system_prefix_blocks: int, +) -> JsonDict: + with_cached = [r for r in records if has_number(r.get("cached_tokens"))] + with_hash = [r for r in records if r.get("hash_ids")] + potential = trace_potential_reuse(records) + if not with_cached: + return { + "status": "unavailable", + "reason": "missing per-request cached token field", + "needed_fields": ["cached_tokens or cache_hit"], + "trace_potential_reuse": potential, + } + if not with_hash: + return { + "status": "unavailable", + "reason": "missing hash_ids needed to classify cached tokens as intra-session or cross-session", + "needed_fields": ["hash_ids"], + "trace_potential_reuse": potential, + } + + sessions_by_hash: dict[int, set[str]] = collections.defaultdict(set) + for r in records: + sid = r.get("session_id") + if sid is None: + continue + for hid in r.get("hash_ids") or []: + sessions_by_hash[int(hid)].add(str(sid)) + + ordered = sorted( + records, + key=lambda r: ( + value_or_inf(r.get("scheduled_dispatch_s")), + value_or_inf(r.get("source_index")), + str(r.get("request_id", "")), + ), + ) + seen_sessions_by_hash: dict[int, set[str]] = collections.defaultdict(set) + buckets = { + "intra_session_tokens": 0.0, + "cross_session_tokens": 0.0, + "shared_system_prefix_tokens": 0.0, + "cached_but_unclassified_tokens": 0.0, + } + request_rows: list[JsonDict] = [] + for r in ordered: + sid = str(r.get("session_id")) if r.get("session_id") is not None else None + hash_ids = [int(h) for h in (r.get("hash_ids") or [])] + cached_remaining = max(to_float(r.get("cached_tokens")) or 0.0, 0.0) + local = {key: 0.0 for key in buckets} + for pos, hid in enumerate(hash_ids): + if cached_remaining <= 0: + break + weight = min(float(block_size), cached_remaining) + prior_sessions = seen_sessions_by_hash.get(hid, set()) + if ( + len(sessions_by_hash.get(hid, set())) >= shared_prefix_min_sessions + and pos < system_prefix_blocks + ): + key = "shared_system_prefix_tokens" + elif sid is not None and sid in prior_sessions: + key = "intra_session_tokens" + elif prior_sessions: + key = "cross_session_tokens" + else: + key = "cached_but_unclassified_tokens" + buckets[key] += weight + local[key] += weight + cached_remaining -= weight + if any(v > 0 for v in local.values()): + request_rows.append({ + "request_id": r.get("request_id"), + "session_id": sid, + **local, + }) + if sid is not None: + for hid in hash_ids: + seen_sessions_by_hash[int(hid)].add(sid) + + total = sum(buckets.values()) + fractions = { + key.replace("_tokens", "_fraction"): value / total if total > 0 else None + for key, value in buckets.items() + } + return { + "status": "available" if total > 0 else "available_no_cached_reuse_observed", + "block_size": block_size, + "shared_prefix_min_sessions": shared_prefix_min_sessions, + "system_prefix_blocks": system_prefix_blocks, + "method": ( + "Classify each cached hash block by earlier occurrences. Blocks in the first " + "system_prefix_blocks positions that appear in at least shared_prefix_min_sessions " + "sessions are counted as shared/system-prefix." + ), + **buckets, + **fractions, + "total_classified_cached_tokens": total, + "request_level_reuse_rows": request_rows[:200], + "trace_potential_reuse": potential, + } + + +def trace_potential_reuse(records: list[JsonDict]) -> JsonDict: + seen_sessions_by_hash: dict[int, set[str]] = collections.defaultdict(set) + counts = { + "first_seen_block_occurrences": 0, + "intra_session_reused_block_occurrences": 0, + "cross_session_reused_block_occurrences": 0, + } + ordered = sorted( + records, + key=lambda r: ( + value_or_inf(r.get("scheduled_dispatch_s")), + value_or_inf(r.get("source_index")), + str(r.get("request_id", "")), + ), + ) + for r in ordered: + sid = str(r.get("session_id")) if r.get("session_id") is not None else None + if sid is None: + continue + for hid in r.get("hash_ids") or []: + hid = int(hid) + prior_sessions = seen_sessions_by_hash.get(hid, set()) + if sid in prior_sessions: + counts["intra_session_reused_block_occurrences"] += 1 + elif prior_sessions: + counts["cross_session_reused_block_occurrences"] += 1 + else: + counts["first_seen_block_occurrences"] += 1 + seen_sessions_by_hash[hid].add(sid) + total_reused = ( + counts["intra_session_reused_block_occurrences"] + + counts["cross_session_reused_block_occurrences"] + ) + return { + **counts, + "intra_session_reuse_fraction": counts["intra_session_reused_block_occurrences"] / total_reused if total_reused else None, + "cross_session_reuse_fraction": counts["cross_session_reused_block_occurrences"] / total_reused if total_reused else None, + "note": "Potential reuse from trace hash_ids only; it is not actual cache-hit reuse.", + } + + +def generate_figures( + out_dir: Path, + records: list[JsonDict], + batch0: JsonDict, + batch1: JsonDict, + no_figures: bool, +) -> JsonDict: + if no_figures: + return {"status": "skipped", "reason": "--no-figures"} + try: + import matplotlib + matplotlib.use("Agg") + import matplotlib.pyplot as plt + except Exception as exc: # pragma: no cover - depends on optional dependency. + return { + "status": "skipped", + "reason": f"matplotlib unavailable: {exc!r}", + } + + figures_dir = out_dir / "figures" + created: list[str] = [] + skipped: dict[str, str] = {} + + def save_current(name: str) -> None: + for suffix in ["png", "pdf"]: + path = figures_dir / f"{name}.{suffix}" + plt.savefig(path, bbox_inches="tight") + created.append(str(path.relative_to(out_dir))) + plt.close() + + def cdf_plot(values: list[float], name: str, xlabel: str, label: str | None = None) -> None: + clean = sorted(v for v in values if v is not None and math.isfinite(v)) + if not clean: + skipped[name] = "no data" + return + y = [(i + 1) / len(clean) for i in range(len(clean))] + plt.figure(figsize=(6, 4)) + plt.plot(clean, y, label=label or xlabel) + if label: + plt.legend() + plt.xlabel(xlabel) + plt.ylabel("CDF") + plt.grid(True, alpha=0.3) + save_current(name) + + starts = batch0["session_arrival_stats"].get("session_start_offsets_s") or [] + cdf_plot(starts, "session_start_time_cdf", "session start offset (s)") + + actual_per_session = batch0["session_concurrency"].get("per_session") or [] + max_inflight = [float(r["max_inflight"]) for r in actual_per_session if "max_inflight" in r] + if max_inflight: + plt.figure(figsize=(6, 4)) + bins = range(1, int(max(max_inflight)) + 2) + plt.hist(max_inflight, bins=bins, align="left", rwidth=0.8) + plt.xlabel("per-session max in-flight") + plt.ylabel("sessions") + plt.grid(True, alpha=0.3) + save_current("per_session_max_inflight_hist") + else: + skipped["per_session_max_inflight_hist"] = "actual interval data unavailable" + + turns = [] + sessions = group_by_session(records) + for rows in sessions.values(): + turns.append(float(len(rows))) + cdf_plot(turns, "turns_per_session_cdf", "turns per session") + + turn_values = batch0["turn_interval_stats"].get("scheduled_turn_interval_values_s") or [] + cdf_plot(turn_values, "turn_interarrival_cdf", "scheduled turn inter-arrival (s)") + + input_values = clean_numbers(r.get("input_tokens") for r in records) + output_values = clean_numbers(r.get("output_tokens") for r in records) + if input_values or output_values: + plt.figure(figsize=(6, 4)) + if input_values: + xs = sorted(input_values) + ys = [(i + 1) / len(xs) for i in range(len(xs))] + plt.plot(xs, ys, label="input") + if output_values: + xs = sorted(output_values) + ys = [(i + 1) / len(xs) for i in range(len(xs))] + plt.plot(xs, ys, label="output") + plt.xlabel("tokens") + plt.ylabel("CDF") + plt.legend() + plt.grid(True, alpha=0.3) + save_current("input_output_token_cdf") + else: + skipped["input_output_token_cdf"] = "no token data" + + ratios = [] + for r in records: + inp = to_float(r.get("input_tokens")) + out = to_float(r.get("output_tokens")) + if inp is not None and out is not None and out > 0: + ratios.append(inp / out) + cdf_plot(ratios, "input_output_ratio_cdf", "input/output token ratio") + + kv_summary = batch1["kv_footprint_summary"] + if kv_summary.get("status") == "available": + kv_bpt = float(kv_summary["kv_bytes_per_token"]) + cdf_plot([v * kv_bpt / (1024 * 1024) for v in input_values], "kv_footprint_cdf", "KV footprint (MiB)") + else: + skipped["kv_footprint_cdf"] = kv_summary.get("reason", "unavailable") + + reuse = batch1["reuse_decomposition"] + if reuse.get("status", "").startswith("available"): + labels = [ + "intra", + "cross", + "shared/system", + "unclassified", + ] + values = [ + reuse.get("intra_session_tokens") or 0, + reuse.get("cross_session_tokens") or 0, + reuse.get("shared_system_prefix_tokens") or 0, + reuse.get("cached_but_unclassified_tokens") or 0, + ] + plt.figure(figsize=(6, 2.5)) + left = 0.0 + for label, value in zip(labels, values): + plt.barh(["cached reuse"], [value], left=left, label=label) + left += value + plt.xlabel("tokens") + plt.legend(loc="best", fontsize=8) + save_current("reuse_decomposition_stacked_bar") + else: + skipped["reuse_decomposition_stacked_bar"] = reuse.get("reason", "unavailable") + + skew = batch1["session_skew"] + lorenz = skew.get("lorenz_input_tokens") or [] + if lorenz: + plt.figure(figsize=(5, 5)) + plt.plot([p["session_fraction"] for p in lorenz], [p["token_fraction"] for p in lorenz]) + plt.plot([0, 1], [0, 1], linestyle="--", color="gray", linewidth=1) + plt.xlabel("fraction of sessions") + plt.ylabel("fraction of input tokens") + plt.grid(True, alpha=0.3) + save_current("per_session_token_mass_lorenz") + else: + skipped["per_session_token_mass_lorenz"] = "no session skew data" + + top_sessions = skew.get("top_sessions_by_input_tokens") or [] + if top_sessions: + top = top_sessions[:10] + plt.figure(figsize=(7, 4)) + plt.bar([str(r["session_id"]) for r in top], [float(r["input_tokens"]) for r in top]) + plt.xlabel("session id") + plt.ylabel("input tokens") + plt.xticks(rotation=45, ha="right", fontsize=8) + save_current("top_sessions_token_contribution") + else: + skipped["top_sessions_token_contribution"] = "no session skew data" + + append_rows = [r for r in records if has_number(r.get("input_tokens")) and has_number(r.get("cached_tokens"))] + if append_rows: + x = [float(r["input_tokens"]) for r in append_rows] + y = [max(float(r["input_tokens"]) - float(r["cached_tokens"]), 0.0) for r in append_rows] + plt.figure(figsize=(6, 4)) + plt.scatter(x, y, s=10, alpha=0.5) + plt.xlabel("input tokens") + plt.ylabel("uncached tokens") + plt.grid(True, alpha=0.3) + save_current("total_input_vs_uncached_scatter") + else: + skipped["total_input_vs_uncached_scatter"] = "missing cached_tokens/cache_hit" + + return { + "status": "available", + "created": created, + "skipped": skipped, + } + + +def build_manifest( + *, + args: argparse.Namespace, + config: JsonDict, + out_dir: Path, + started_at: dt.datetime, + finished_at: dt.datetime, + batch0: JsonDict, +) -> JsonDict: + time_scale = args.time_scale + if time_scale is None: + time_scale = to_float(config.get("time_scale")) + request_limit = args.request_limit + if request_limit is None: + request_limit = to_int(first_present(config, ["requests", "request_limit"])) + policy = args.policy or str(config.get("policy", "")) + trace_info = file_info(args.trace, hash_inputs=args.hash_inputs) + return { + "git_commit": git_commit(), + "host": platform.node(), + "gpu_type": args.gpu_type, + "gpu_count": args.gpu_count, + "canonical_trace_data_sources": { + "dash0_formatted_trace_dir": "~/ali-trace/trace-glm5.1-formatted/", + "dash0_raw_trace_dir": "~/ali-trace/trace-glm5.1/", + "usage_note": ( + "Full trace analysis can be run CPU-only on dash0, or the needed " + "JSONL files can be copied/rsynced from dash0 to this machine before " + "running this analyzer." + ), + }, + "input_requirements": { + "trace_jsonl": [ + "chat_id", + "parent_chat_id", + "timestamp", + "input_length", + "output_length", + "turn", + "hash_ids", + "optional session_id", + ], + "metrics_jsonl": [ + "request_id", + "session_id", + "trace_timestamp_s", + "input_length", + "output_length", + "latency_s", + "ttft_s", + "tpot_s", + "error", + "optional cached_tokens", + ], + "actual_sequentiality_proof": [ + "per-request dispatch timestamp", + "per-request finish or error/timeout timestamp", + "request_id join to trace/metrics when timing source is separate", + ], + "reuse_decomposition": [ + "cached_tokens or cache_hit", + "hash_ids", + "session_id", + ], + }, + "trace_path": str(args.trace) if args.trace else "", + "trace_sha256": trace_info.get("sha256", ""), + "trace_file_info": trace_info, + "policy": policy, + "launch_command": args.launch_command or " ".join(sys.argv), + "request_limit": request_limit, + "time_scale": time_scale, + "session_sampling_method": args.session_sampling_method, + "session_sequential": batch0["session_concurrency"].get("session_sequential"), + "start_time": started_at.isoformat(), + "end_time": finished_at.isoformat(), + "output_dir": str(out_dir), + } + + +def render_invalid_runs(batch0: JsonDict) -> str: + cls = batch0["classification"] + conc = batch0["session_concurrency"] + lines = [ + "# Invalid Runs", + "", + f"Classification: `{cls['label']}`", + "", + f"Reason: {cls['reason']}", + "", + "## Sequentiality", + "", + f"- Actual interval status: `{conc['status']}`", + f"- Session sequential: `{conc['session_sequential']}`", + f"- Max in-flight per session: `{conc['max_inflight_per_session']}`", + f"- Sessions with overlap: `{conc['sessions_with_overlap']}`", + "", + ] + if conc.get("overlap_examples"): + lines.append("## Overlap Examples") + lines.append("") + for item in conc["overlap_examples"][:10]: + lines.append( + f"- session `{item['session_id']}` request `{item['request_id']}` " + f"started {item['overlap_s']:.6g}s before `{item['previous_request_id']}` finished" + ) + lines.append("") + if cls["label"] == "invalid_for_online_claim": + lines.extend([ + "## Claims Still Supported", + "", + "- Workload shape and trace-level token/session distributions.", + "- Offline or stress-test observations that do not require online per-session sequentiality.", + "", + "## Claims Not Supported", + "", + "- Online-serving SRR/SLO claims that require at most one in-flight turn per session.", + "", + ]) + return "\n".join(lines) + + +def render_summary_md(summary: JsonDict, batch0: JsonDict, batch1: JsonDict) -> str: + prof = batch0["trace_profile"] + conc = batch0["session_concurrency"] + workload = batch1["workload_summary"] + kv = batch1["kv_footprint_summary"] + reuse = batch1["reuse_decomposition"] + append = batch1["append_delta_stats"] + lines = [ + "# Characterization Summary", + "", + f"- Classification: `{summary['classification']['label']}`", + f"- Analyzed records: {summary['analyzed_records']}", + f"- Sessions: {prof.get('session_count')}", + f"- Attempted/completed/errors: {prof.get('attempted_requests')} / {prof.get('completed_requests')} / {prof.get('error_requests')}", + f"- Goodput: {fmt(prof.get('goodput_per_s'))} req/s", + "", + "## Batch 0", + "", + f"- Actual interval status: `{conc['status']}`", + f"- Session sequential: `{conc['session_sequential']}`", + f"- Max in-flight per session: `{conc['max_inflight_per_session']}`", + "", + "## Batch 1", + "", + f"- Input tokens p50/p90/p99: {stats_line(workload.get('input_tokens'))}", + f"- Output tokens p50/p90/p99: {stats_line(workload.get('output_tokens'))}", + f"- KV footprint: `{kv.get('status')}`", + f"- Reuse decomposition: `{reuse.get('status')}`", + f"- Append/uncached stats: `{append.get('status')}`", + "", + "Raw data and figures are in this output directory. See `audit.md` for unavailable fields and claim checks.", + "", + ] + return "\n".join(lines) + + +def render_audit_md(batch0: JsonDict, batch1: JsonDict) -> str: + conc = batch0["session_concurrency"] + prof = batch0["trace_profile"] + workload = batch1["workload_summary"] + kv = batch1["kv_footprint_summary"] + reuse = batch1["reuse_decomposition"] + skew = batch1["session_skew"] + append = batch1["append_delta_stats"] + top = skew.get("top_session_contribution") or {} + input_top = top.get("input_tokens") or {} + lines = [ + "# Audit", + "", + "## Batch 0 Checks", + "", + f"1. Does the main trace satisfy `max_inflight_per_session == 1`? {answer_sequential(conc)}", + f"2. If not, is the run labeled stress or invalid? Classification is `{batch0['classification']['label']}`.", + f"3. Are attempted/completed/error counts included? Attempted={prof.get('attempted_requests')}, completed={prof.get('completed_requests')}, errors={prof.get('error_requests')}.", + f"4. Are latency percentiles over successes, with goodput reported? Success-only={prof.get('latency_percentiles_over_successes_only')}, goodput={fmt(prof.get('goodput_per_s'))} req/s.", + "", + "## Batch 1 Checks", + "", + f"1. Input p50/p90/p99: {stats_line(workload.get('input_tokens'))}.", + f"2. Output p50/p90/p99: {stats_line(workload.get('output_tokens'))}.", + f"3. Estimated KV footprint p50/p90/p99: {kv_line(kv)}.", + f"4. Fraction of reuse intra-session: {reuse_fraction_line(reuse, 'intra_session_fraction')}.", + f"5. Top 1% / 5% session token mass: {fmt(input_top.get('top_1pct_fraction'))} / {fmt(input_top.get('top_5pct_fraction'))}.", + f"6. Are long prompts often small appends after cache reuse? {append_line(append)}", + "", + "## Missing Inputs", + "", + ] + missing = collect_missing_inputs(conc, kv, reuse, append) + if missing: + lines.extend(f"- {item}" for item in missing) + else: + lines.append("- None detected by the analyzer.") + lines.append("") + return "\n".join(lines) + + +def answer_sequential(conc: JsonDict) -> str: + if conc.get("session_sequential") is True: + return "yes, actual interval data shows max in-flight <= 1." + if conc.get("session_sequential") is False: + return "no, overlap was observed." + return ( + "unavailable, because actual dispatch and finish/error timestamps " + "are missing or only partially available." + ) + + +def collect_missing_inputs(conc: JsonDict, kv: JsonDict, reuse: JsonDict, append: JsonDict) -> list[str]: + missing: list[str] = [] + if conc.get("status") != "available": + missing.append( + "Batch 0 actual sequentiality needs per-request dispatch timestamp plus finish/error timestamp." + ) + if kv.get("status") != "available": + missing.append(str(kv.get("needed_input", "KV bytes per token is required."))) + if reuse.get("status") == "unavailable": + missing.append("Reuse decomposition needs `cached_tokens` or `cache_hit` and `hash_ids`.") + if append.get("status") == "unavailable": + missing.append("Append stats need `cached_tokens` or `cache_hit` plus input length.") + return missing + + +def relative_output_list(out_dir: Path) -> list[str]: + return sorted( + str(path.relative_to(out_dir)) + for path in out_dir.rglob("*") + if path.is_file() + ) + + +def group_by_session(records: list[JsonDict]) -> dict[str, list[JsonDict]]: + sessions: dict[str, list[JsonDict]] = collections.defaultdict(list) + for r in records: + sid = r.get("session_id") + if sid is not None: + sessions[str(sid)].append(r) + return dict(sessions) + + +def stats(values: list[float] | list[int]) -> JsonDict | None: + clean = sorted(float(v) for v in values if v is not None and math.isfinite(float(v))) + if not clean: + return None + return { + "count": len(clean), + "mean": statistics.fmean(clean), + "min": clean[0], + "p50": percentile(clean, 0.50), + "p90": percentile(clean, 0.90), + "p95": percentile(clean, 0.95), + "p99": percentile(clean, 0.99), + "max": clean[-1], + } + + +def percentile(sorted_values: list[float], pct: float) -> float: + if not sorted_values: + raise ValueError("percentile requires at least one value") + if len(sorted_values) == 1: + return sorted_values[0] + rank = pct * (len(sorted_values) - 1) + lo = int(rank) + hi = min(lo + 1, len(sorted_values) - 1) + frac = rank - lo + return sorted_values[lo] * (1 - frac) + sorted_values[hi] * frac + + +def clean_numbers(values: Any) -> list[float]: + out: list[float] = [] + for value in values: + num = to_float(value) + if num is not None and math.isfinite(num): + out.append(num) + return out + + +def top_contribution(values: list[float]) -> JsonDict: + clean = sorted([v for v in values if v is not None and math.isfinite(v)], reverse=True) + total = sum(clean) + if not clean or total <= 0: + return { + "top_1pct_fraction": None, + "top_5pct_fraction": None, + "top_10pct_fraction": None, + "top_20pct_fraction": None, + } + def frac(pct: float) -> float: + k = max(1, math.ceil(len(clean) * pct)) + return sum(clean[:k]) / total + return { + "top_1pct_fraction": frac(0.01), + "top_5pct_fraction": frac(0.05), + "top_10pct_fraction": frac(0.10), + "top_20pct_fraction": frac(0.20), + } + + +def lorenz_points(values: list[float]) -> list[JsonDict]: + clean = sorted(v for v in values if v is not None and math.isfinite(v) and v >= 0) + total = sum(clean) + if not clean or total <= 0: + return [] + points = [{"session_fraction": 0.0, "token_fraction": 0.0}] + cumulative = 0.0 + for idx, value in enumerate(clean, start=1): + cumulative += value + points.append({ + "session_fraction": idx / len(clean), + "token_fraction": cumulative / total, + }) + return points + + +def count_gt(values: list[float], threshold: float) -> JsonDict: + if not values: + return {"count": 0, "fraction": None} + count = sum(1 for v in values if v > threshold) + return {"count": count, "fraction": count / len(values)} + + +def first_present(row: JsonDict, keys: list[str], default: Any = None) -> Any: + for key in keys: + if key in row and row[key] is not None: + return row[key] + return default + + +def to_float(value: Any, default: float | None = None) -> float | None: + if value is None: + return default + try: + return float(value) + except (TypeError, ValueError): + return default + + +def to_int(value: Any, default: int | None = None) -> int | None: + if value is None: + return default + try: + return int(value) + except (TypeError, ValueError): + return default + + +def has_number(value: Any) -> bool: + return to_float(value) is not None + + +def value_or_inf(value: Any) -> float: + num = to_float(value) + return num if num is not None else float("inf") + + +def fmt(value: Any) -> str: + num = to_float(value) + if num is None: + return "unavailable" + if abs(num) >= 1000: + return f"{num:,.3f}" + return f"{num:.6g}" + + +def stats_line(s: JsonDict | None) -> str: + if not s: + return "unavailable" + return f"{fmt(s.get('p50'))} / {fmt(s.get('p90'))} / {fmt(s.get('p99'))}" + + +def kv_line(kv: JsonDict) -> str: + if kv.get("status") != "available": + return f"unavailable ({kv.get('reason')})" + return stats_line(kv.get("kv_mib_per_request")) + " MiB" + + +def reuse_fraction_line(reuse: JsonDict, key: str) -> str: + if not reuse.get("status", "").startswith("available"): + return f"unavailable ({reuse.get('reason')})" + return fmt(reuse.get(key)) + + +def append_line(append: JsonDict) -> str: + if append.get("status") != "available": + return f"unavailable ({append.get('reason')})" + return ( + f"long_prompt_small_append_fraction={fmt(append.get('long_prompt_small_append_fraction'))}; " + f"uncached/input p50/p90/p99={stats_line(append.get('uncached_to_input_ratio'))}" + ) + + +def write_json(path: Path, data: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + +def write_jsonl(path: Path, rows: list[JsonDict]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, sort_keys=True) + "\n") + + +def sha256_file(path: Path | None) -> str: + if path is None or not path.exists(): + return "" + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1024 * 1024), b""): + digest.update(chunk) + return digest.hexdigest() + + +def file_info(path: Path | None, *, hash_inputs: bool) -> JsonDict: + if path is None: + return { + "path": "", + "exists": False, + "sha256_status": "unavailable_no_path", + } + info: JsonDict = { + "path": str(path), + "exists": path.exists(), + } + if not path.exists(): + info["sha256_status"] = "unavailable_missing_file" + return info + stat = path.stat() + info.update({ + "size_bytes": stat.st_size, + "mtime_s": stat.st_mtime, + }) + if hash_inputs: + info["sha256"] = sha256_file(path) + info["sha256_status"] = "available" + else: + info["sha256"] = "" + info["sha256_status"] = "skipped_use_--hash-inputs" + return info + + +def git_commit() -> str: + try: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + text=True, + ) + except Exception: + return "" + return result.stdout.strip() + + +if __name__ == "__main__": + main() diff --git a/analysis/characterization/current_results/all_figures_index.md b/analysis/characterization/current_results/all_figures_index.md new file mode 100644 index 0000000..16ca8e7 --- /dev/null +++ b/analysis/characterization/current_results/all_figures_index.md @@ -0,0 +1,10 @@ +# Figures Index + +No generated figures are committed by this script. Batch-specific figures should be generated from: + +- `analysis/characterization/analyze.py` for Batch 0/1 trace figures. +- future Batch 2 step-timeline artifacts for interference plots. +- future Batch 3 per-worker/session artifacts for hot-spot plots. +- future Batch 4 arrival-rate sweep artifacts for SRR curves. + +This file exists so the audit package has a stable placeholder until fresh figures are generated. diff --git a/analysis/characterization/current_results/characterization_claim_matrix.md b/analysis/characterization/current_results/characterization_claim_matrix.md new file mode 100644 index 0000000..cfe0d5b --- /dev/null +++ b/analysis/characterization/current_results/characterization_claim_matrix.md @@ -0,0 +1,11 @@ +# Characterization Claim Matrix + +| Claim | Status | Supporting Data | Needed Next | Reviewer Risk | +|---|---|---|---|---| +| Batch 0 substrate audit is only partially complete for existing runs. | `partially_supported` | metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts. | Add request dispatch and finish/error timestamps to future replayer/proxy metrics. | Cannot use these runs to prove online per-session sequentiality. | +| Batch 1 workload shape can be characterized from formatted traces and metrics. | `supported_for_trace_shape` | Full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass. | Add cache-hit joined records for actual reuse decomposition. | Actual cache reuse decomposition needs cached_tokens joined with hash_ids. | +| Static PD separation is worse than combined in existing 200-request GPU A/B. | `supported_by_existing_artifact` | outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json. | Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology. | Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy. | +| Elastic transfer-based migration does not improve high-contention 500-request run. | `supported_by_existing_artifact` | outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv. | Attribute whether failure is trigger quality, transfer overhead, or wrong load regime. | Existing metrics lack actual sequentiality proof and per-request transfer waterfall. | +| PD-colo prefill/decode interference is not yet directly proven by step-level data in this package. | `not_yet_supported` | No decode-step and prefill-overlap timestamp artifact found in summarized runs. | Run Batch 2 controlled same-worker/different-worker injection with step timestamps. | Cannot claim interference as causal without Batch 2. | +| Session hot-spot residual imbalance is suggested but not fully attributed. | `partially_supported` | gpu_util.csv shows per-GPU mean-util imbalance in existing runs. | Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker. | GPU util imbalance alone is not enough to prove session hot-spot. | +| SRR is not measured by existing fixed-request runs. | `not_yet_supported` | No arrival-rate sweep artifacts found. | Implement Batch 4 Poisson session-arrival SRR sweep. | Latency-at-one-load cannot support sustainable throughput claim. | diff --git a/analysis/characterization/current_results/claim_matrix.json b/analysis/characterization/current_results/claim_matrix.json new file mode 100644 index 0000000..3343afd --- /dev/null +++ b/analysis/characterization/current_results/claim_matrix.json @@ -0,0 +1,51 @@ +[ + { + "claim": "Batch 0 substrate audit is only partially complete for existing runs.", + "needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.", + "reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.", + "status": "partially_supported", + "supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts." + }, + { + "claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.", + "needed_next": "Add cache-hit joined records for actual reuse decomposition.", + "reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.", + "status": "supported_for_trace_shape", + "supporting_data": "Full compact trace CPU summary in full_trace_summary.json: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass." + }, + { + "claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.", + "needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.", + "reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.", + "status": "supported_by_existing_artifact", + "supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json." + }, + { + "claim": "Elastic transfer-based migration does not improve high-contention 500-request run.", + "needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.", + "reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.", + "status": "supported_by_existing_artifact", + "supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv." + }, + { + "claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.", + "needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.", + "reviewer_risk": "Cannot claim interference as causal without Batch 2.", + "status": "not_yet_supported", + "supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs." + }, + { + "claim": "Session hot-spot residual imbalance is suggested but not fully attributed.", + "needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.", + "reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.", + "status": "partially_supported", + "supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs." + }, + { + "claim": "SRR is not measured by existing fixed-request runs.", + "needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.", + "reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.", + "status": "not_yet_supported", + "supporting_data": "No arrival-rate sweep artifacts found." + } +] diff --git a/analysis/characterization/current_results/comparisons.json b/analysis/characterization/current_results/comparisons.json new file mode 100644 index 0000000..94e7068 --- /dev/null +++ b/analysis/characterization/current_results/comparisons.json @@ -0,0 +1,95 @@ +[ + { + "baseline": "outputs/gpu_ab_combined", + "e2e_p50_delta_pct": 40.870329127661, + "e2e_p90_delta_pct": 15.206416995091814, + "error_count": [ + 2, + 13 + ], + "gpu_imbalance_ratio": [ + 3.2445157838416265, + 11.149056603773586 + ], + "gpu_mean_util": [ + 30.541666666666664, + 12.367081447963802 + ], + "name": "combined_vs_pdsep_200", + "request_count": [ + 200, + 200 + ], + "success_count": [ + 198, + 187 + ], + "tpot_p90_delta_pct": 1.3481309269699875, + "ttft_p50_delta_pct": 98.06752892925572, + "ttft_p90_delta_pct": 44.79649177751278, + "variant": "outputs/gpu_ab_pdsep", + "wall_clock_delta_pct": 142.27736808267244 + }, + { + "baseline": "outputs/contention_16s_ts10", + "e2e_p50_delta_pct": 11.538788125232664, + "e2e_p90_delta_pct": -5.080083318118138, + "error_count": [ + 2, + 2 + ], + "gpu_imbalance_ratio": [ + 2.310775410408662, + 2.600767754318618 + ], + "gpu_mean_util": [ + 23.030492424242425, + 26.349561403508773 + ], + "name": "contention_baseline_vs_elastic_500", + "request_count": [ + 500, + 500 + ], + "success_count": [ + 498, + 498 + ], + "tpot_p90_delta_pct": 13.63098996823875, + "ttft_p50_delta_pct": 12.433589435386224, + "ttft_p90_delta_pct": 13.412576920999959, + "variant": "outputs/contention_16s_elastic", + "wall_clock_delta_pct": -0.5645626396767849 + }, + { + "baseline": "outputs/combined_1000req", + "e2e_p50_delta_pct": 202.85189980479385, + "e2e_p90_delta_pct": 128.274511020719, + "error_count": [ + 2, + 204 + ], + "gpu_imbalance_ratio": [ + null, + null + ], + "gpu_mean_util": [ + null, + null + ], + "name": "combined_1000_vs_pdsep_mooncake", + "request_count": [ + 1000, + 1000 + ], + "success_count": [ + 998, + 796 + ], + "tpot_p90_delta_pct": -34.83638659447109, + "ttft_p50_delta_pct": 781.9835547522864, + "ttft_p90_delta_pct": 1030.68607857992, + "variant": "outputs/exp3_pd_sep_tp1_mooncake", + "wall_clock_delta_pct": 119.18997774599991 + } +] diff --git a/analysis/characterization/current_results/current_results.md b/analysis/characterization/current_results/current_results.md new file mode 100644 index 0000000..cde7efd --- /dev/null +++ b/analysis/characterization/current_results/current_results.md @@ -0,0 +1,77 @@ +# Current Characterization Results + +Generated: 2026-05-25T06:52:18.096448+00:00 +Git commit: `21ffb3d4f77956d008b1815a3c0d46e0188ac390` + +## Canonical Full-Trace CPU Summary + +Source: `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl`. +This is CPU-only parsing of the compact formatted trace with session IDs +reconstructed from `parent_chat_id` chains. + +| Metric | Value | +|---|---:| +| Requests | 2,114,220 | +| Sessions | 1,307,276 | +| Trace span | 7,199.975 s | +| Input tokens p50/p90/p99 | 20,030 / 87,855 / 125,527 | +| Output tokens p50/p90/p99 | 80 / 811 / 6,615 | +| Input/output ratio p50/p90/p99 | 217.8 / 1,204.4 / 4,251.6 | +| Turns/session p50/p90/p99/max | 1 / 1 / 18 / 3,091 | +| Session input tokens p50/p90/p99/max | 12,486 / 72,676 / 974,934 / 156,756,974 | +| Top 1% / 5% / 10% sessions by input-token mass | 46.5% / 66.5% / 74.6% | + +Immediate reading: the full trace strongly supports long-input/short-output +and heavy-tailed session token mass. It does **not** by itself prove online +sequentiality or actual cache-hit reuse; those require runtime timestamps and +cache-hit fields. + +## Existing Run Summaries + +| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance | +|---|---:|---:|---:|---:|---:|---:| +| outputs/gpu_ab_combined | 198/200 | 1.01/9.36 | 5.05/30.2 | 0.0732 | 30.5 | 3.24 | +| outputs/gpu_ab_pdsep | 187/200 | 1.99/13.5 | 7.11/34.8 | 0.0742 | 12.4 | 11.1 | +| outputs/contention_16s_ts10 | 498/500 | 0.826/9.71 | 5.8/51 | 0.103 | 23 | 2.31 | +| outputs/contention_16s_elastic | 498/500 | 0.929/11 | 6.47/48.4 | 0.117 | 26.3 | 2.6 | +| outputs/combined_1000req | 998/1000 | 0.393/2.57 | 3.22/28 | 0.113 | n/a | n/a | +| outputs/exp3_pd_sep_tp1_mooncake | 796/1000 | 3.47/29 | 9.75/63.9 | 0.0739 | n/a | n/a | + +## Pairwise Comparisons + +| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ | +|---|---:|---:|---:|---:|---:|---:| +| combined_vs_pdsep_200 | +98.1% | +44.8% | +40.9% | +15.2% | +1.3% | +142.3% | +| contention_baseline_vs_elastic_500 | +12.4% | +13.4% | +11.5% | -5.1% | +13.6% | -0.6% | +| combined_1000_vs_pdsep_mooncake | +782.0% | +1030.7% | +202.9% | +128.3% | -34.8% | +119.2% | + +## What We Can Say Now + +- **partially_supported**: Batch 0 substrate audit is only partially complete for existing runs. + Supporting data: metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts. + Next: Add request dispatch and finish/error timestamps to future replayer/proxy metrics. +- **supported_for_trace_shape**: Batch 1 workload shape can be characterized from formatted traces and metrics. + Supporting data: full compact trace CPU summary in `full_trace_summary.json`: input p50/p90/p99 = 20k/87.9k/125.5k, output p50/p90/p99 = 80/811/6.6k, top 1% sessions hold 46.5% of input-token mass. + Next: add cache-hit joined records for actual reuse decomposition. +- **supported_by_existing_artifact**: Static PD separation is worse than combined in existing 200-request GPU A/B. + Supporting data: outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json. + Next: Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology. +- **supported_by_existing_artifact**: Elastic transfer-based migration does not improve high-contention 500-request run. + Supporting data: outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv. + Next: Attribute whether failure is trigger quality, transfer overhead, or wrong load regime. +- **not_yet_supported**: PD-colo prefill/decode interference is not yet directly proven by step-level data in this package. + Supporting data: No decode-step and prefill-overlap timestamp artifact found in summarized runs. + Next: Run Batch 2 controlled same-worker/different-worker injection with step timestamps. +- **partially_supported**: Session hot-spot residual imbalance is suggested but not fully attributed. + Supporting data: gpu_util.csv shows per-GPU mean-util imbalance in existing runs. + Next: Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker. +- **not_yet_supported**: SRR is not measured by existing fixed-request runs. + Supporting data: No arrival-rate sweep artifacts found. + Next: Implement Batch 4 Poisson session-arrival SRR sweep. + +## Main Reviewer Risks + +- **high**: Session sequentiality not proven - Add dispatch/finish timestamps and run Batch 0 before SRR claims. +- **medium**: Legacy PD-sep data may not match final methodology - Use fresh PD matrix for paper-grade claims. +- **medium**: GPU util is not a sufficient hot-spot proof - Add route-decision and per-worker queue logs for Batch 3. +- **medium**: Cache reuse decomposition is incomplete without joined hash/cache-hit data - Emit hash_ids/session_id/cached_tokens in the same per-request record. diff --git a/analysis/characterization/current_results/full_trace_summary.json b/analysis/characterization/current_results/full_trace_summary.json new file mode 100644 index 0000000..0c3cae4 --- /dev/null +++ b/analysis/characterization/current_results/full_trace_summary.json @@ -0,0 +1,56 @@ +{ + "input": { + "count": 2114220, + "max": 202371, + "mean": 33637.38370084476, + "p50": 20030.0, + "p90": 87855.1000000001, + "p95": 104738.0, + "p99": 125527.0 + }, + "input_output_ratio": { + "count": 2108130, + "max": 143664.0, + "mean": 534.3516074828406, + "p50": 217.8, + "p90": 1204.3769610389616, + "p95": 1814.3478327228322, + "p99": 4251.585499999998 + }, + "output": { + "count": 2114220, + "max": 132665, + "mean": 444.97059624826176, + "p50": 80.0, + "p90": 811.0, + "p95": 2213.0, + "p99": 6614.810000000056 + }, + "path": "/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl", + "records": 2114220, + "session_input_tokens": { + "count": 1307276, + "max": 156756974, + "mean": 54400.77639916896, + "p50": 12486.0, + "p90": 72676.0, + "p95": 108523.25, + "p99": 974933.75 + }, + "sessions": 1307276, + "top_session_input_fraction": { + "top10pct": 0.7464402483455778, + "top1pct": 0.46456810581415175, + "top5pct": 0.6651718740752172 + }, + "trace_span_s": 7199.975, + "turns_per_session": { + "count": 1307276, + "max": 3091, + "mean": 1.6172713336739908, + "p50": 1.0, + "p90": 1.0, + "p95": 2.0, + "p99": 18.0 + } +} diff --git a/analysis/characterization/current_results/main_claim_allowed_runs.md b/analysis/characterization/current_results/main_claim_allowed_runs.md new file mode 100644 index 0000000..56fa7fe --- /dev/null +++ b/analysis/characterization/current_results/main_claim_allowed_runs.md @@ -0,0 +1,66 @@ +# Main-Claim Allowed Runs + +Status: current audit gate +Date: 2026-05-25 + +## Allowed For Workload-Shape Claims + +These artifacts can support trace/workload characterization claims: + +- `dash0:/home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl` + - Compact formatted full trace. + - CPU summary recorded in `full_trace_summary.json`. + - Supports long-input/short-output and session token-mass skew claims. + - Does not prove runtime cache hits or online sequentiality. +- `traces/w600_r0.0015_st30.jsonl` + - Local sampled trace. + - Useful for local dry runs and figure generation. + - Not the canonical full-trace source. + +## Allowed For Legacy Baseline Sanity Claims + +These existing runs can support sanity-level comparisons, but not final +paper-grade SRR claims: + +- `outputs/gpu_ab_combined` +- `outputs/gpu_ab_pdsep` +- `outputs/contention_16s_ts10` +- `outputs/contention_16s_elastic` +- `outputs/combined_1000req` +- `outputs/exp3_pd_sep_tp1_mooncake` + +Allowed claims: + +- Static PD-sep was worse than combined in these existing fixed-request runs. +- Elastic transfer-based migration did not improve the summarized 500-request + high-contention run. +- GPU-util imbalance exists in these artifacts. + +Disallowed claims: + +- Online SRR. +- Per-session sequentiality. +- Causal attribution of prefill/decode interference. +- Causal attribution of session hot spots from GPU utilization alone. + +## Not Yet Allowed For Main Claims + +The following need fresh instrumentation or fresh runs: + +- Batch 2 prefill/decode interference. +- Batch 3 session hot-spot root cause. +- Batch 4 sustainable request rate. +- Batch 5 failure attribution near SRR boundary. + +## Required Upgrade Before Paper-Grade Claims + +Future main-claim runs must include: + +- per-request actual dispatch timestamp; +- per-request finish/error timestamp; +- route decision and selected worker; +- per-worker queue delay; +- per-worker KV occupancy; +- per-worker APC/cache-hit snapshot; +- attempted/completed/error/goodput counters; +- session-causal load generation. diff --git a/analysis/characterization/current_results/reproduction_commands.sh b/analysis/characterization/current_results/reproduction_commands.sh new file mode 100644 index 0000000..f427e6f --- /dev/null +++ b/analysis/characterization/current_results/reproduction_commands.sh @@ -0,0 +1,17 @@ +#!/usr/bin/env bash +set -euo pipefail + +# Rebuild this current-results audit package. +python3 analysis/characterization/summarize_runs.py --output-dir analysis/characterization/current_results --runs outputs/gpu_ab_combined outputs/gpu_ab_pdsep outputs/contention_16s_ts10 outputs/contention_16s_elastic outputs/combined_1000req outputs/exp3_pd_sep_tp1_mooncake + +# Example Batch 0/1 local trace analysis. +python3 analysis/characterization/analyze.py \ + --trace traces/w600_r0.0015_st30.jsonl \ + --kv-bytes-per-token 98304 \ + --task-name w600_local_full_trace \ + --overwrite + +# CPU-only full compact trace summary was computed on dash0 from: +# /home/admin/cpfs/wjh/ali-trace/trace-glm5.1-formatted/051315-051317.jsonl +# Recompute either by running analyze.py on dash0, or by copying that compact +# formatted JSONL locally. Do not use the 487G raw file directly. diff --git a/analysis/characterization/current_results/reviewer_risk_register.json b/analysis/characterization/current_results/reviewer_risk_register.json new file mode 100644 index 0000000..02ef73f --- /dev/null +++ b/analysis/characterization/current_results/reviewer_risk_register.json @@ -0,0 +1,26 @@ +[ + { + "evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.", + "mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.", + "risk": "Session sequentiality not proven", + "severity": "high" + }, + { + "evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.", + "mitigation": "Use fresh PD matrix for paper-grade claims.", + "risk": "Legacy PD-sep data may not match final methodology", + "severity": "medium" + }, + { + "evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.", + "mitigation": "Add route-decision and per-worker queue logs for Batch 3.", + "risk": "GPU util is not a sufficient hot-spot proof", + "severity": "medium" + }, + { + "evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.", + "mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.", + "risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data", + "severity": "medium" + } +] diff --git a/analysis/characterization/current_results/reviewer_risk_register.md b/analysis/characterization/current_results/reviewer_risk_register.md new file mode 100644 index 0000000..d89d693 --- /dev/null +++ b/analysis/characterization/current_results/reviewer_risk_register.md @@ -0,0 +1,8 @@ +# Reviewer Risk Register + +| Risk | Severity | Evidence | Mitigation | +|---|---|---|---| +| Session sequentiality not proven | `high` | Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps. | Add dispatch/finish timestamps and run Batch 0 before SRR claims. | +| Legacy PD-sep data may not match final methodology | `medium` | PD matrix scaffold exists separately; some old runs used earlier flags/methodology. | Use fresh PD matrix for paper-grade claims. | +| GPU util is not a sufficient hot-spot proof | `medium` | Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership. | Add route-decision and per-worker queue logs for Batch 3. | +| Cache reuse decomposition is incomplete without joined hash/cache-hit data | `medium` | Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts. | Emit hash_ids/session_id/cached_tokens in the same per-request record. | diff --git a/analysis/characterization/current_results/run_summaries.json b/analysis/characterization/current_results/run_summaries.json new file mode 100644 index 0000000..b02e281 --- /dev/null +++ b/analysis/characterization/current_results/run_summaries.json @@ -0,0 +1,720 @@ +[ + { + "apc_summary": { + "reason": "apc.txt missing", + "status": "unavailable" + }, + "artifact_availability": { + "apc_txt": false, + "breakdown_json": false, + "gpu_util_csv": true, + "metrics_jsonl": true, + "metrics_summary_json": true + }, + "breakdown_summary": { + "reason": "breakdown.json missing", + "status": "unavailable" + }, + "error_count": 2, + "exists": true, + "external_cache_hit_ratio": null, + "gpu_summary": { + "gpu_count": 8, + "max_mean_util_pct": 63.166666666666664, + "max_min_ratio": 3.2445157838416265, + "mean_util_pct": 30.541666666666664, + "min_mean_util_pct": 19.46875, + "per_gpu_mean_util_pct": { + "0": 29.145833333333332, + "1": 20.041666666666668, + "2": 25.0, + "3": 63.166666666666664, + "4": 21.927083333333332, + "5": 34.90625, + "6": 19.46875, + "7": 30.677083333333332 + }, + "status": "available", + "stddev_across_gpu_mean_util_pct": 13.337857305429534 + }, + "latency_stats_s": { + "count": 198.0, + "mean": 13.01780862723021, + "p50": 5.048548387829214, + "p90": 30.18109704903327, + "p99": 119.01174414204434 + }, + "metrics_jsonl_rows": 200, + "metrics_summary_available": true, + "prefix_cache_hit_ratio": 0.0, + "request_count": 200, + "run": "outputs/gpu_ab_combined", + "session_summary": { + "request_cached_tokens": { + "count": 200, + "max": 0.0, + "mean": 0.0, + "p50": 0.0, + "p90": 0.0, + "p95": 0.0, + "p99": 0.0 + }, + "request_input_tokens": { + "count": 200, + "max": 111927.0, + "mean": 29318.375, + "p50": 21376.0, + "p90": 81218.19999999998, + "p95": 87571.2, + "p99": 101619.27999999994 + }, + "request_output_tokens": { + "count": 200, + "max": 5083.0, + "mean": 257.675, + "p50": 72.0, + "p90": 664.3, + "p95": 1063.9999999999993, + "p99": 3300.1799999999994 + }, + "session_count": 145, + "session_input_tokens": { + "count": 145, + "max": 1567423.0, + "mean": 40439.137931034486, + "p50": 10879.0, + "p90": 72438.39999999997, + "p95": 106934.39999999988, + "p99": 374927.28 + }, + "status": "available", + "top_session_input_fraction": { + "top_10pct": 0.6588294883328288, + "top_1pct": 0.33276622595897626, + "top_5pct": 0.5534430199490934 + }, + "turns_per_session": { + "count": 145, + "max": 21.0, + "mean": 1.3793103448275863, + "p50": 1.0, + "p90": 1.0, + "p95": 2.0, + "p99": 10.560000000000002 + } + }, + "success_count": 198, + "tpot_stats_s": { + "count": 198.0, + "mean": 0.04929455188644857, + "p50": 0.03717198147904128, + "p90": 0.07317714408040046, + "p99": 0.10039294634234945 + }, + "ttft_stats_s": { + "count": 198.0, + "mean": 3.8987488178453984, + "p50": 1.0068706551101059, + "p90": 9.355209570843726, + "p99": 33.855437273858115 + }, + "wall_clock_s": 483.543720243033 + }, + { + "apc_summary": { + "reason": "apc.txt missing", + "status": "unavailable" + }, + "artifact_availability": { + "apc_txt": false, + "breakdown_json": false, + "gpu_util_csv": true, + "metrics_jsonl": true, + "metrics_summary_json": true + }, + "breakdown_summary": { + "reason": "breakdown.json missing", + "status": "unavailable" + }, + "error_count": 13, + "exists": true, + "external_cache_hit_ratio": null, + "gpu_summary": { + "gpu_count": 8, + "max_mean_util_pct": 26.737556561085974, + "max_min_ratio": 11.149056603773586, + "mean_util_pct": 12.367081447963802, + "min_mean_util_pct": 2.3981900452488687, + "per_gpu_mean_util_pct": { + "0": 26.737556561085974, + "1": 14.44343891402715, + "2": 19.036199095022624, + "3": 7.4389140271493215, + "4": 2.3981900452488687, + "5": 8.841628959276019, + "6": 11.963800904977376, + "7": 8.076923076923077 + }, + "status": "available", + "stddev_across_gpu_mean_util_pct": 7.15857427072345 + }, + "latency_stats_s": { + "count": 187.0, + "mean": 15.366858840781827, + "p50": 7.111906730104238, + "p90": 34.77056052000262, + "p99": 119.2064151619561 + }, + "metrics_jsonl_rows": 200, + "metrics_summary_available": true, + "prefix_cache_hit_ratio": 0.0, + "request_count": 200, + "run": "outputs/gpu_ab_pdsep", + "session_summary": { + "request_cached_tokens": { + "count": 200, + "max": 0.0, + "mean": 0.0, + "p50": 0.0, + "p90": 0.0, + "p95": 0.0, + "p99": 0.0 + }, + "request_input_tokens": { + "count": 200, + "max": 111927.0, + "mean": 29318.375, + "p50": 21376.0, + "p90": 81218.19999999998, + "p95": 87571.2, + "p99": 101619.27999999994 + }, + "request_output_tokens": { + "count": 200, + "max": 5097.0, + "mean": 290.18, + "p50": 75.5, + "p90": 685.8, + "p95": 1102.5999999999997, + "p99": 3433.6599999999844 + }, + "session_count": 145, + "session_input_tokens": { + "count": 145, + "max": 1567423.0, + "mean": 40439.137931034486, + "p50": 10879.0, + "p90": 72438.39999999997, + "p95": 106934.39999999988, + "p99": 374927.28 + }, + "status": "available", + "top_session_input_fraction": { + "top_10pct": 0.6588294883328288, + "top_1pct": 0.33276622595897626, + "top_5pct": 0.5534430199490934 + }, + "turns_per_session": { + "count": 145, + "max": 21.0, + "mean": 1.3793103448275863, + "p50": 1.0, + "p90": 1.0, + "p95": 2.0, + "p99": 10.560000000000002 + } + }, + "success_count": 187, + "tpot_stats_s": { + "count": 187.0, + "mean": 0.05175559044923285, + "p50": 0.04020531923068981, + "p90": 0.07416366779122173, + "p99": 0.10770365060609463 + }, + "ttft_stats_s": { + "count": 187.0, + "mean": 5.3933139261814524, + "p50": 1.9942838260903955, + "p90": 13.546015257015824, + "p99": 39.30951087013818 + }, + "wall_clock_s": 1171.516998933861 + }, + { + "apc_summary": { + "reason": "apc.txt missing", + "status": "unavailable" + }, + "artifact_availability": { + "apc_txt": false, + "breakdown_json": false, + "gpu_util_csv": true, + "metrics_jsonl": true, + "metrics_summary_json": true + }, + "breakdown_summary": { + "reason": "breakdown.json missing", + "status": "unavailable" + }, + "error_count": 2, + "exists": true, + "external_cache_hit_ratio": 0.0, + "gpu_summary": { + "gpu_count": 8, + "max_mean_util_pct": 40.095454545454544, + "max_min_ratio": 2.310775410408662, + "mean_util_pct": 23.030492424242425, + "min_mean_util_pct": 17.35151515151515, + "per_gpu_mean_util_pct": { + "0": 22.243939393939392, + "1": 22.798484848484847, + "2": 22.09090909090909, + "3": 40.095454545454544, + "4": 17.484848484848484, + "5": 20.225757575757576, + "6": 17.35151515151515, + "7": 21.953030303030303 + }, + "status": "available", + "stddev_across_gpu_mean_util_pct": 6.752783259358802 + }, + "latency_stats_s": { + "count": 498.0, + "mean": 21.908606036002105, + "p50": 5.798741589998826, + "p90": 51.00022796400299, + "p99": 201.6967467680006 + }, + "metrics_jsonl_rows": 500, + "metrics_summary_available": true, + "prefix_cache_hit_ratio": 0.0, + "request_count": 500, + "run": "outputs/contention_16s_ts10", + "session_summary": { + "request_cached_tokens": { + "count": 500, + "max": 0.0, + "mean": 0.0, + "p50": 0.0, + "p90": 0.0, + "p95": 0.0, + "p99": 0.0 + }, + "request_input_tokens": { + "count": 500, + "max": 134865.0, + "mean": 29190.97, + "p50": 17756.0, + "p90": 77365.70000000006, + "p95": 87790.89999999997, + "p99": 111360.44 + }, + "request_output_tokens": { + "count": 500, + "max": 16427.0, + "mean": 404.286, + "p50": 75.0, + "p90": 863.7000000000004, + "p95": 2384.45, + "p99": 4928.559999999999 + }, + "session_count": 347, + "session_input_tokens": { + "count": 347, + "max": 1929072.0, + "mean": 42061.916426512966, + "p50": 11342.0, + "p90": 68297.20000000004, + "p95": 116213.79999999993, + "p99": 521898.36000000057 + }, + "status": "available", + "top_session_input_fraction": { + "top_10pct": 0.6824335059780473, + "top_1pct": 0.32352388426969025, + "top_5pct": 0.579125805000656 + }, + "turns_per_session": { + "count": 347, + "max": 29.0, + "mean": 1.440922190201729, + "p50": 1.0, + "p90": 1.0, + "p95": 3.0, + "p99": 10.54000000000002 + } + }, + "success_count": 498, + "tpot_stats_s": { + "count": 498.0, + "mean": 0.07772806444638436, + "p50": 0.06774563665097065, + "p90": 0.10325221968416468, + "p99": 0.503443661631568 + }, + "ttft_stats_s": { + "count": 498.0, + "mean": 3.5127640336366928, + "p50": 0.8264882279981975, + "p90": 9.71474659699743, + "p99": 34.40133859900379 + }, + "wall_clock_s": 1472.1331836190002 + }, + { + "apc_summary": { + "line_count": 8, + "preview": "inst_0: prefix=45.7% ext=24.7%\ninst_1: prefix=33.1% ext=13.3%\ninst_2: prefix=14.2% ext=0.0%\ninst_3: prefix=50.7% ext=6.3%\ninst_4: prefix=33.6% ext=0.0%\ninst_5: prefix=66.6% ext=9.2%\ninst_6: prefix=42.1% ext=18.8%\ninst_7: prefix=36.7% ext=7.9%", + "status": "available" + }, + "artifact_availability": { + "apc_txt": true, + "breakdown_json": true, + "gpu_util_csv": true, + "metrics_jsonl": true, + "metrics_summary_json": true + }, + "breakdown_summary": { + "field_sample": [ + "cache_hit", + "estimated_new_tokens", + "input_length", + "policy", + "request_id", + "route_class", + "routed_to", + "t_done", + "t_first_token", + "t_proxy_recv" + ], + "mode_counts": { + "HEAVY_COLO": 111, + "HEAVY_OFFLOAD": 13, + "MEDIUM": 163, + "WARM": 213 + }, + "route_counts": { + "linear": 487 + }, + "row_count": 500, + "status": "available" + }, + "error_count": 2, + "exists": true, + "external_cache_hit_ratio": 0.0, + "gpu_summary": { + "gpu_count": 8, + "max_mean_util_pct": 47.54385964912281, + "max_min_ratio": 2.600767754318618, + "mean_util_pct": 26.349561403508773, + "min_mean_util_pct": 18.280701754385966, + "per_gpu_mean_util_pct": { + "0": 22.410526315789475, + "1": 26.57894736842105, + "2": 23.54035087719298, + "3": 47.54385964912281, + "4": 18.96842105263158, + "5": 26.403508771929825, + "6": 18.280701754385966, + "7": 27.07017543859649 + }, + "status": "available", + "stddev_across_gpu_mean_util_pct": 8.6079068381278 + }, + "latency_stats_s": { + "count": 498.0, + "mean": 21.780466573040204, + "p50": 6.467846095998539, + "p90": 48.40937389100145, + "p99": 196.93401125499804 + }, + "metrics_jsonl_rows": 500, + "metrics_summary_available": true, + "prefix_cache_hit_ratio": 0.0, + "request_count": 500, + "run": "outputs/contention_16s_elastic", + "session_summary": { + "request_cached_tokens": { + "count": 500, + "max": 0.0, + "mean": 0.0, + "p50": 0.0, + "p90": 0.0, + "p95": 0.0, + "p99": 0.0 + }, + "request_input_tokens": { + "count": 500, + "max": 134865.0, + "mean": 29190.97, + "p50": 17756.0, + "p90": 77365.70000000006, + "p95": 87790.89999999997, + "p99": 111360.44 + }, + "request_output_tokens": { + "count": 500, + "max": 16427.0, + "mean": 395.466, + "p50": 75.0, + "p90": 809.8000000000013, + "p95": 2264.2999999999943, + "p99": 4928.559999999999 + }, + "session_count": 347, + "session_input_tokens": { + "count": 347, + "max": 1929072.0, + "mean": 42061.916426512966, + "p50": 11342.0, + "p90": 68297.20000000004, + "p95": 116213.79999999993, + "p99": 521898.36000000057 + }, + "status": "available", + "top_session_input_fraction": { + "top_10pct": 0.6824335059780473, + "top_1pct": 0.32352388426969025, + "top_5pct": 0.579125805000656 + }, + "turns_per_session": { + "count": 347, + "max": 29.0, + "mean": 1.440922190201729, + "p50": 1.0, + "p90": 1.0, + "p95": 3.0, + "p99": 10.54000000000002 + } + }, + "success_count": 498, + "tpot_stats_s": { + "count": 498.0, + "mean": 0.08306021258238128, + "p50": 0.06973490711122092, + "p90": 0.117326519391297, + "p99": 0.5879216773072795 + }, + "ttft_stats_s": { + "count": 498.0, + "mean": 3.757858794331448, + "p50": 0.9292503809992922, + "p90": 11.017744456999935, + "p99": 34.20241540799907 + }, + "wall_clock_s": 1463.8220696580029 + }, + { + "apc_summary": { + "reason": "apc.txt missing", + "status": "unavailable" + }, + "artifact_availability": { + "apc_txt": false, + "breakdown_json": false, + "gpu_util_csv": false, + "metrics_jsonl": true, + "metrics_summary_json": true + }, + "breakdown_summary": { + "reason": "breakdown.json missing", + "status": "unavailable" + }, + "error_count": 2, + "exists": true, + "external_cache_hit_ratio": null, + "gpu_summary": { + "reason": "gpu_util.csv missing", + "status": "unavailable" + }, + "latency_stats_s": { + "count": 998.0, + "mean": 15.744418202954922, + "p50": 3.220371481962502, + "p90": 28.005282410187647, + "p99": 204.00479479599744 + }, + "metrics_jsonl_rows": 1000, + "metrics_summary_available": true, + "prefix_cache_hit_ratio": 0.5443149471989938, + "request_count": 1000, + "run": "outputs/combined_1000req", + "session_summary": { + "request_cached_tokens": { + "count": 1000, + "max": 0.0, + "mean": 0.0, + "p50": 0.0, + "p90": 0.0, + "p95": 0.0, + "p99": 0.0 + }, + "request_input_tokens": { + "count": 1000, + "max": 171427.0, + "mean": 31611.14, + "p50": 19798.0, + "p90": 82584.20000000001, + "p95": 93305.64999999997, + "p99": 111947.83999999998 + }, + "request_output_tokens": { + "count": 1000, + "max": 41233.0, + "mean": 392.579, + "p50": 70.0, + "p90": 685.8000000000002, + "p95": 1834.6999999999975, + "p99": 4584.459999999996 + }, + "session_count": 671, + "session_input_tokens": { + "count": 671, + "max": 2320064.0, + "mean": 47110.49180327869, + "p50": 12394.0, + "p90": 71875.0, + "p95": 112984.5, + "p99": 760591.4999999925 + }, + "status": "available", + "top_session_input_fraction": { + "top_10pct": 0.6915149849072194, + "top_1pct": 0.36223103627392117, + "top_5pct": 0.5945521736957288 + }, + "turns_per_session": { + "count": 671, + "max": 36.0, + "mean": 1.4903129657228018, + "p50": 1.0, + "p90": 1.0, + "p95": 3.0, + "p99": 12.599999999999909 + } + }, + "success_count": 998, + "tpot_stats_s": { + "count": 998.0, + "mean": 0.05681003092240363, + "p50": 0.04029440155459775, + "p90": 0.11343103184021618, + "p99": 0.2962149563411783 + }, + "ttft_stats_s": { + "count": 998.0, + "mean": 1.0351265607308877, + "p50": 0.39304836583323777, + "p90": 2.5655168020166457, + "p99": 8.210839052917436 + }, + "wall_clock_s": 1340.061015482992 + }, + { + "apc_summary": { + "reason": "apc.txt missing", + "status": "unavailable" + }, + "artifact_availability": { + "apc_txt": false, + "breakdown_json": false, + "gpu_util_csv": false, + "metrics_jsonl": true, + "metrics_summary_json": true + }, + "breakdown_summary": { + "reason": "breakdown.json missing", + "status": "unavailable" + }, + "error_count": 204, + "exists": true, + "external_cache_hit_ratio": null, + "gpu_summary": { + "reason": "gpu_util.csv missing", + "status": "unavailable" + }, + "latency_stats_s": { + "count": 796.0, + "mean": 23.561836449842698, + "p50": 9.752956213895231, + "p90": 63.928921481827274, + "p99": 168.2179710320197 + }, + "metrics_jsonl_rows": 1000, + "metrics_summary_available": true, + "prefix_cache_hit_ratio": 0.0, + "request_count": 1000, + "run": "outputs/exp3_pd_sep_tp1_mooncake", + "session_summary": { + "request_cached_tokens": { + "count": 1000, + "max": 0.0, + "mean": 0.0, + "p50": 0.0, + "p90": 0.0, + "p95": 0.0, + "p99": 0.0 + }, + "request_input_tokens": { + "count": 1000, + "max": 171427.0, + "mean": 31611.14, + "p50": 19798.0, + "p90": 82584.20000000001, + "p95": 93305.64999999997, + "p99": 111947.83999999998 + }, + "request_output_tokens": { + "count": 1000, + "max": 41233.0, + "mean": 412.176, + "p50": 74.0, + "p90": 759.7000000000008, + "p95": 1834.6999999999975, + "p99": 4928.559999999999 + }, + "session_count": 671, + "session_input_tokens": { + "count": 671, + "max": 2320064.0, + "mean": 47110.49180327869, + "p50": 12394.0, + "p90": 71875.0, + "p95": 112984.5, + "p99": 760591.4999999925 + }, + "status": "available", + "top_session_input_fraction": { + "top_10pct": 0.6915149849072194, + "top_1pct": 0.36223103627392117, + "top_5pct": 0.5945521736957288 + }, + "turns_per_session": { + "count": 671, + "max": 36.0, + "mean": 1.4903129657228018, + "p50": 1.0, + "p90": 1.0, + "p95": 3.0, + "p99": 12.599999999999909 + } + }, + "success_count": 796, + "tpot_stats_s": { + "count": 796.0, + "mean": 0.05211825330315642, + "p50": 0.06622354774330945, + "p90": 0.07391575907026088, + "p99": 0.10499966285609896 + }, + "ttft_stats_s": { + "count": 796.0, + "mean": 10.753033336598003, + "p50": 3.4666219488717616, + "p90": 29.00794132403098, + "p99": 81.7531874559354 + }, + "wall_clock_s": 2937.2794416199904 + } +] diff --git a/analysis/characterization/protocols.md b/analysis/characterization/protocols.md new file mode 100644 index 0000000..d1be69f --- /dev/null +++ b/analysis/characterization/protocols.md @@ -0,0 +1,264 @@ +# Characterization Protocols For Remaining Batches + +Status: implementation protocol and audit checklist +Date: 2026-05-25 + +This file completes the `analysis/characterization` scaffold for the TODO +list. It separates what is already implemented from what requires fresh GPU +runs or new engine/proxy instrumentation. + +## Implemented Now + +### Batch 0/1 Analyzer + +Use: + +```bash +python3 analysis/characterization/analyze.py \ + --trace traces/w600_r0.0015_st30.jsonl \ + --kv-bytes-per-token 98304 \ + --task-name w600_local_full_trace \ + --overwrite +``` + +The analyzer writes: + +- `manifest.json` +- `summary.json` +- `summary.md` +- `audit.md` +- `session_concurrency.json` +- `session_arrival_stats.json` +- `turn_interval_stats.json` +- `trace_profile.json` +- `workload_summary.json` +- `kv_footprint_summary.json` +- `reuse_decomposition.json` +- `session_skew.json` +- `append_delta_stats.json` + +Limitations: + +- Actual online sequentiality requires dispatch and finish/error timestamps. + Existing `metrics.jsonl` artifacts generally do not contain these fields. +- Actual reuse decomposition requires `cached_tokens`/`cache_hit`, `hash_ids`, + and `session_id` in the same joinable request record. + +### Existing-Run Audit + +Use: + +```bash +python3 analysis/characterization/summarize_runs.py +``` + +The script writes an audit package under: + +```text +analysis/characterization/current_results/ +``` + +It summarizes already completed runs and explicitly marks which claims are +supported, partially supported, or not yet supported. + +## Batch 2 Protocol: PD-Colo Prefill/Decode Interference + +Purpose: + +Prove whether same-worker prefill overlap increases decode TPOT/queue delay. + +Required new instrumentation: + +- per-request dispatch timestamp +- per-request finish/error timestamp +- per decode step timestamp +- decode step worker id +- prefill chunk start/end timestamp +- prefill worker id +- request/session id associated with each prefill chunk + +Required arms: + +1. decode-only steady load +2. decode + same-worker heavy prefill injection +3. decode + different-worker heavy prefill injection +4. trace replay with overlap labels + +Required sweep: + +```text +uncached_prefill_tokens in {2k, 8k, 16k, 32k, 64k} +chunked_prefill_size in available engine values +``` + +Required outputs: + +- `interference_microbench_summary.json` +- `decode_step_timeseries.csv` +- `prefill_overlap_events.jsonl` +- `interference_index.json` +- TPOT timeline figure with prefill overlays +- same-worker vs different-worker TPOT boxplot + +Pass condition: + +```text +TPOT_p90(overlap_same_worker) / TPOT_p90(no_overlap) > 1 +``` + +and the effect must be materially weaker in the different-worker control. + +## Batch 3 Protocol: Session Hot-Spot Residual Imbalance + +Purpose: + +Prove whether cache-aware/LMetric still leaves hot workers under +session-heavy skew. + +Required new instrumentation: + +- route decision per request +- chosen worker +- candidate worker scores +- cache hit / estimated uncached tokens per candidate +- per-worker request queue length/delay +- per-worker decode queue length/delay +- per-worker KV occupancy +- per-worker APC/cache-hit snapshot + +Required arms: + +1. corrected LMetric/cache-aware +2. load-only routing +3. hard sticky routing +4. current Unified hybrid +5. session-mass capped/equalized replay + +Required outputs: + +- `worker_balance_summary.json` +- `session_to_worker_map.json` +- `session_mass_summary.json` +- `routing_policy_comparison.json` +- `hotspot_index.json` +- per-worker queue delay bar +- APC vs queue delay scatter +- top-session contribution bar +- policy tradeoff plot: APC vs hot-spot index + +Pass condition: + +LMetric/cache-aware must show measurable residual worker skew, and that skew +must correlate with session token mass or locality. + +GPU utilization alone is not enough for this claim. + +## Batch 4 Protocol: Sustainable Request Rate + +Purpose: + +Measure: + +```text +SRR(SLO) = max arrival rate satisfying SLO in steady state +``` + +Required load generator behavior: + +- open-loop session arrivals, preferably Poisson +- session-internal sequentiality +- warmup window +- steady-state measurement window +- explicit attempted/completed/error counters + +Provisional SLO: + +```text +TTFT_p90 <= T_ttft +E2E_p90 <= T_e2e +TPOT_p90 <= T_tpot +error_rate <= epsilon +queue length stable +KV occupancy stable +``` + +Required arms: + +1. PD-colo corrected LMetric/cache-aware +2. static PD-disagg +3. current Unified hybrid +4. optional hard sticky +5. optional load-only + +Required outputs: + +- `srr_curve.json` +- `lambda_runs//summary.json` +- `slo_violation_reason.json` +- `goodput_vs_arrival_rate.json` +- SRR bar chart +- latency vs arrival rate curves +- goodput vs arrival rate +- queue/KV stability plot near failure point + +Pass condition: + +Each policy has a measured max sustainable lambda under the same SLO and +same session-causal arrival process. + +## Batch 5 Protocol: Failure Attribution Near SRR Boundary + +Purpose: + +Explain why each policy fails near SRR. + +Required rates: + +```text +lambda = 0.9 * SRR +lambda = 1.0 * SRR +lambda = 1.1 * SRR +``` + +Labels for each slow/SLO-violating request: + +- same-worker prefill overlap +- hot worker queue +- high KV occupancy +- cache miss / large uncached append +- transfer wait +- P queue wait +- D admission wait +- unknown + +Required outputs: + +- `slow_request_attribution.jsonl` +- `failure_breakdown.json` +- `case_studies.md` +- `worker_failure_windows.json` +- violation cause stacked bar +- slow request waterfall +- worker timeline near failure + +Pass condition: + +The analysis must explain whether PD-colo is limited by interference, +hot-spot, KV pressure, or a mixture, and whether Unified/PUSH underperforms +because of trigger quality, transfer cost, target admission, or load regime. + +## Batch 6 Protocol: Audit Package + +Implemented by `summarize_runs.py` for existing runs and extended by fresh +Batch 2-5 outputs later. + +Required files: + +- `characterization_claim_matrix.md` +- `all_figures_index.md` +- `reviewer_risk_register.md` +- `reproduction_commands.sh` +- `main_claim_allowed_runs.md` + +Current package intentionally marks Batch 2/4/5 claims as not yet supported +until fresh instrumented experiments exist. diff --git a/analysis/characterization/summarize_runs.py b/analysis/characterization/summarize_runs.py new file mode 100644 index 0000000..2edaa51 --- /dev/null +++ b/analysis/characterization/summarize_runs.py @@ -0,0 +1,666 @@ +#!/usr/bin/env python3 +"""Summarize existing benchmark artifacts for characterization review. + +This is a CPU-only companion to ``analyze.py``. It does not run benchmarks. +It reads completed output directories and produces an audit-oriented package +that helps decide which TODO claims are currently supported by existing data +and which still need fresh GPU runs or additional instrumentation. +""" + +from __future__ import annotations + +import argparse +import csv +import datetime as dt +import json +import math +import statistics +import subprocess +from pathlib import Path +from typing import Any + + +JsonDict = dict[str, Any] + + +DEFAULT_RUNS = [ + "outputs/gpu_ab_combined", + "outputs/gpu_ab_pdsep", + "outputs/contention_16s_ts10", + "outputs/contention_16s_elastic", + "outputs/combined_1000req", + "outputs/exp3_pd_sep_tp1_mooncake", +] + + +def main() -> None: + args = parse_args() + out_dir = args.output_dir + out_dir.mkdir(parents=True, exist_ok=True) + + run_dirs = [Path(p) for p in (args.runs or DEFAULT_RUNS)] + summaries = [summarize_run(path) for path in run_dirs] + comparisons = build_comparisons(summaries) + claim_matrix = build_claim_matrix(summaries, comparisons) + risk_register = build_risk_register(summaries) + + write_json(out_dir / "run_summaries.json", summaries) + write_json(out_dir / "comparisons.json", comparisons) + write_json(out_dir / "claim_matrix.json", claim_matrix) + write_json(out_dir / "reviewer_risk_register.json", risk_register) + (out_dir / "current_results.md").write_text( + render_current_results(summaries, comparisons, claim_matrix, risk_register), + encoding="utf-8", + ) + (out_dir / "characterization_claim_matrix.md").write_text( + render_claim_matrix(claim_matrix), + encoding="utf-8", + ) + (out_dir / "reviewer_risk_register.md").write_text( + render_risk_register(risk_register), + encoding="utf-8", + ) + (out_dir / "all_figures_index.md").write_text( + render_figures_index(summaries), + encoding="utf-8", + ) + (out_dir / "reproduction_commands.sh").write_text( + render_reproduction_commands(args, run_dirs), + encoding="utf-8", + ) + print(f"Wrote run summary package to {out_dir}") + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Summarize existing characterization-relevant output dirs.", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + parser.add_argument( + "--runs", + nargs="*", + default=None, + help="Output directories to summarize. Defaults to a small curated set.", + ) + parser.add_argument( + "--output-dir", + type=Path, + default=Path("analysis/characterization/current_results"), + help="Directory for generated review artifacts.", + ) + return parser.parse_args() + + +def summarize_run(path: Path) -> JsonDict: + metrics_summary = load_json(path / "metrics.summary.json") + metrics_rows = load_jsonl(path / "metrics.jsonl") + gpu_summary = summarize_gpu(path / "gpu_util.csv") + breakdown_summary = summarize_breakdown(path / "breakdown.json") + apc_summary = summarize_apc(path / "apc.txt") + return { + "run": str(path), + "exists": path.exists(), + "metrics_summary_available": bool(metrics_summary), + "metrics_jsonl_rows": len(metrics_rows), + "request_count": first_present(metrics_summary, ["request_count"]), + "success_count": first_present(metrics_summary, ["success_count"]), + "error_count": first_present(metrics_summary, ["error_count"]), + "wall_clock_s": first_present(metrics_summary, ["wall_clock_s"]), + "latency_stats_s": metrics_summary.get("latency_stats_s"), + "ttft_stats_s": metrics_summary.get("ttft_stats_s"), + "tpot_stats_s": metrics_summary.get("tpot_stats_s"), + "prefix_cache_hit_ratio": metrics_summary.get("prefix_cache_hit_ratio"), + "external_cache_hit_ratio": metrics_summary.get("external_cache_hit_ratio"), + "session_summary": summarize_sessions(metrics_rows), + "gpu_summary": gpu_summary, + "breakdown_summary": breakdown_summary, + "apc_summary": apc_summary, + "artifact_availability": { + "metrics_summary_json": (path / "metrics.summary.json").exists(), + "metrics_jsonl": (path / "metrics.jsonl").exists(), + "gpu_util_csv": (path / "gpu_util.csv").exists(), + "breakdown_json": (path / "breakdown.json").exists(), + "apc_txt": (path / "apc.txt").exists(), + }, + } + + +def summarize_sessions(rows: list[JsonDict]) -> JsonDict: + if not rows: + return { + "status": "unavailable", + "reason": "metrics.jsonl missing", + } + sessions: dict[str, JsonDict] = {} + input_values = [] + output_values = [] + cached_values = [] + for row in rows: + sid = str(row.get("session_id", "")) + item = sessions.setdefault( + sid, + { + "turns": 0, + "input_tokens": 0.0, + "output_tokens": 0.0, + "cached_tokens": 0.0, + }, + ) + inp = to_float(row.get("input_length")) or 0.0 + out = to_float(row.get("actual_output_tokens")) or to_float(row.get("output_length")) or 0.0 + cached = to_float(row.get("cached_tokens")) or 0.0 + item["turns"] += 1 + item["input_tokens"] += inp + item["output_tokens"] += out + item["cached_tokens"] += cached + input_values.append(inp) + output_values.append(out) + cached_values.append(cached) + per_session_input = [v["input_tokens"] for v in sessions.values()] + return { + "status": "available", + "request_input_tokens": stats(input_values), + "request_output_tokens": stats(output_values), + "request_cached_tokens": stats(cached_values), + "session_count": len(sessions), + "turns_per_session": stats([v["turns"] for v in sessions.values()]), + "session_input_tokens": stats(per_session_input), + "top_session_input_fraction": top_contribution(per_session_input), + } + + +def summarize_gpu(path: Path) -> JsonDict: + if not path.exists(): + return { + "status": "unavailable", + "reason": "gpu_util.csv missing", + } + values: dict[str, list[float]] = {} + with path.open() as handle: + reader = csv.DictReader(handle) + for row in reader: + gpu = str(row.get("gpu", "")) + util = to_float(row.get("util_pct")) + if gpu and util is not None: + values.setdefault(gpu, []).append(util) + means = {gpu: statistics.fmean(vals) for gpu, vals in values.items() if vals} + if not means: + return { + "status": "unavailable", + "reason": "gpu_util.csv had no util_pct rows", + } + mean_values = list(means.values()) + return { + "status": "available", + "gpu_count": len(means), + "per_gpu_mean_util_pct": means, + "mean_util_pct": statistics.fmean(mean_values), + "stddev_across_gpu_mean_util_pct": statistics.pstdev(mean_values), + "max_mean_util_pct": max(mean_values), + "min_mean_util_pct": min(mean_values), + "max_min_ratio": max(mean_values) / max(min(mean_values), 1e-9), + } + + +def summarize_breakdown(path: Path) -> JsonDict: + if not path.exists(): + return { + "status": "unavailable", + "reason": "breakdown.json missing", + } + try: + data = json.loads(path.read_text(encoding="utf-8")) + except Exception as exc: + return { + "status": "unavailable", + "reason": f"failed to parse breakdown: {exc}", + } + rows: list[JsonDict] + if isinstance(data, list): + rows = [r for r in data if isinstance(r, dict)] + elif isinstance(data, dict): + rows = data.get("records") if isinstance(data.get("records"), list) else [] + if not rows: + rows = [data] + else: + rows = [] + mode_counts: dict[str, int] = {} + route_counts: dict[str, int] = {} + for row in rows: + mode = row.get("mode") or row.get("execution_mode") or row.get("route_class") + route = row.get("route") or row.get("decision") or row.get("policy") + if mode is not None: + mode_counts[str(mode)] = mode_counts.get(str(mode), 0) + 1 + if route is not None: + route_counts[str(route)] = route_counts.get(str(route), 0) + 1 + return { + "status": "available", + "row_count": len(rows), + "mode_counts": mode_counts, + "route_counts": route_counts, + "field_sample": sorted(rows[0].keys()) if rows else [], + } + + +def summarize_apc(path: Path) -> JsonDict: + if not path.exists(): + return { + "status": "unavailable", + "reason": "apc.txt missing", + } + text = path.read_text(encoding="utf-8", errors="replace") + return { + "status": "available", + "line_count": len(text.splitlines()), + "preview": "\n".join(text.splitlines()[:20]), + } + + +def build_comparisons(summaries: list[JsonDict]) -> list[JsonDict]: + by_run = {s["run"]: s for s in summaries} + pairs = [ + ("combined_vs_pdsep_200", "outputs/gpu_ab_combined", "outputs/gpu_ab_pdsep"), + ("contention_baseline_vs_elastic_500", "outputs/contention_16s_ts10", "outputs/contention_16s_elastic"), + ("combined_1000_vs_pdsep_mooncake", "outputs/combined_1000req", "outputs/exp3_pd_sep_tp1_mooncake"), + ] + out = [] + for name, base, variant in pairs: + if base not in by_run or variant not in by_run: + continue + out.append(compare_pair(name, by_run[base], by_run[variant])) + return out + + +def compare_pair(name: str, base: JsonDict, variant: JsonDict) -> JsonDict: + return { + "name": name, + "baseline": base["run"], + "variant": variant["run"], + "request_count": [base.get("request_count"), variant.get("request_count")], + "success_count": [base.get("success_count"), variant.get("success_count")], + "error_count": [base.get("error_count"), variant.get("error_count")], + "ttft_p50_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p50"), stat_value(variant, "ttft_stats_s", "p50")), + "ttft_p90_delta_pct": pct_delta(stat_value(base, "ttft_stats_s", "p90"), stat_value(variant, "ttft_stats_s", "p90")), + "e2e_p50_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p50"), stat_value(variant, "latency_stats_s", "p50")), + "e2e_p90_delta_pct": pct_delta(stat_value(base, "latency_stats_s", "p90"), stat_value(variant, "latency_stats_s", "p90")), + "tpot_p90_delta_pct": pct_delta(stat_value(base, "tpot_stats_s", "p90"), stat_value(variant, "tpot_stats_s", "p90")), + "wall_clock_delta_pct": pct_delta(base.get("wall_clock_s"), variant.get("wall_clock_s")), + "gpu_mean_util": [ + nested(base, ["gpu_summary", "mean_util_pct"]), + nested(variant, ["gpu_summary", "mean_util_pct"]), + ], + "gpu_imbalance_ratio": [ + nested(base, ["gpu_summary", "max_min_ratio"]), + nested(variant, ["gpu_summary", "max_min_ratio"]), + ], + } + + +def build_claim_matrix(summaries: list[JsonDict], comparisons: list[JsonDict]) -> list[JsonDict]: + has_gpu = any((s.get("gpu_summary") or {}).get("status") == "available" for s in summaries) + has_metrics = any(s.get("metrics_summary_available") for s in summaries) + return [ + { + "claim": "Batch 0 substrate audit is only partially complete for existing runs.", + "status": "partially_supported", + "supporting_data": "metrics.jsonl lacks actual dispatch/finish timestamps in current artifacts.", + "needed_next": "Add request dispatch and finish/error timestamps to future replayer/proxy metrics.", + "reviewer_risk": "Cannot use these runs to prove online per-session sequentiality.", + }, + { + "claim": "Batch 1 workload shape can be characterized from formatted traces and metrics.", + "status": "supported_for_trace_shape", + "supporting_data": "analysis/characterization/analyze.py outputs workload_summary/session_skew/KV footprint when given trace and kv_bytes_per_token.", + "needed_next": "Run on dash0 compact formatted trace for canonical full-trace numbers.", + "reviewer_risk": "Actual cache reuse decomposition needs cached_tokens joined with hash_ids.", + }, + { + "claim": "Static PD separation is worse than combined in existing 200-request GPU A/B.", + "status": "supported_by_existing_artifact" if has_metrics else "unavailable", + "supporting_data": "outputs/gpu_ab_combined vs outputs/gpu_ab_pdsep metrics.summary.json.", + "needed_next": "Refresh with PD matrix, multiple seeds, cudagraph-enabled methodology.", + "reviewer_risk": "Legacy run has no per-stage TTFT breakdown and no step-level KV occupancy.", + }, + { + "claim": "Elastic transfer-based migration does not improve high-contention 500-request run.", + "status": "supported_by_existing_artifact" if has_metrics else "unavailable", + "supporting_data": "outputs/contention_16s_ts10 vs outputs/contention_16s_elastic metrics.summary.json and gpu_util.csv.", + "needed_next": "Attribute whether failure is trigger quality, transfer overhead, or wrong load regime.", + "reviewer_risk": "Existing metrics lack actual sequentiality proof and per-request transfer waterfall.", + }, + { + "claim": "PD-colo prefill/decode interference is not yet directly proven by step-level data in this package.", + "status": "not_yet_supported", + "supporting_data": "No decode-step and prefill-overlap timestamp artifact found in summarized runs.", + "needed_next": "Run Batch 2 controlled same-worker/different-worker injection with step timestamps.", + "reviewer_risk": "Cannot claim interference as causal without Batch 2.", + }, + { + "claim": "Session hot-spot residual imbalance is suggested but not fully attributed.", + "status": "partially_supported" if has_gpu else "unavailable", + "supporting_data": "gpu_util.csv shows per-GPU mean-util imbalance in existing runs.", + "needed_next": "Collect per-worker queue delay, session-to-worker map, and per-session token mass per worker.", + "reviewer_risk": "GPU util imbalance alone is not enough to prove session hot-spot.", + }, + { + "claim": "SRR is not measured by existing fixed-request runs.", + "status": "not_yet_supported", + "supporting_data": "No arrival-rate sweep artifacts found.", + "needed_next": "Implement Batch 4 Poisson session-arrival SRR sweep.", + "reviewer_risk": "Latency-at-one-load cannot support sustainable throughput claim.", + }, + ] + + +def build_risk_register(summaries: list[JsonDict]) -> list[JsonDict]: + return [ + { + "risk": "Session sequentiality not proven", + "severity": "high", + "evidence": "Current metrics include trace timestamp and latency but not actual dispatch/finish wall-clock timestamps.", + "mitigation": "Add dispatch/finish timestamps and run Batch 0 before SRR claims.", + }, + { + "risk": "Legacy PD-sep data may not match final methodology", + "severity": "medium", + "evidence": "PD matrix scaffold exists separately; some old runs used earlier flags/methodology.", + "mitigation": "Use fresh PD matrix for paper-grade claims.", + }, + { + "risk": "GPU util is not a sufficient hot-spot proof", + "severity": "medium", + "evidence": "Existing artifacts have gpu_util.csv but lack per-worker queue and session ownership.", + "mitigation": "Add route-decision and per-worker queue logs for Batch 3.", + }, + { + "risk": "Cache reuse decomposition is incomplete without joined hash/cache-hit data", + "severity": "medium", + "evidence": "Trace has hash_ids; metrics have cached_tokens; request IDs may not join across all artifacts.", + "mitigation": "Emit hash_ids/session_id/cached_tokens in the same per-request record.", + }, + ] + + +def render_current_results( + summaries: list[JsonDict], + comparisons: list[JsonDict], + claim_matrix: list[JsonDict], + risk_register: list[JsonDict], +) -> str: + lines = [ + "# Current Characterization Results", + "", + f"Generated: {dt.datetime.now(dt.timezone.utc).isoformat()}", + f"Git commit: `{git_commit()}`", + "", + "## Existing Run Summaries", + "", + "| Run | OK/Req | TTFT p50/p90 | E2E p50/p90 | TPOT p90 | GPU mean util | GPU imbalance |", + "|---|---:|---:|---:|---:|---:|---:|", + ] + for s in summaries: + lines.append( + "| {run} | {ok}/{req} | {ttft50}/{ttft90} | {e2e50}/{e2e90} | {tpot90} | {gpu_mean} | {gpu_imb} |".format( + run=s["run"], + ok=fmt(s.get("success_count")), + req=fmt(s.get("request_count")), + ttft50=fmt(stat_value(s, "ttft_stats_s", "p50")), + ttft90=fmt(stat_value(s, "ttft_stats_s", "p90")), + e2e50=fmt(stat_value(s, "latency_stats_s", "p50")), + e2e90=fmt(stat_value(s, "latency_stats_s", "p90")), + tpot90=fmt(stat_value(s, "tpot_stats_s", "p90")), + gpu_mean=fmt(nested(s, ["gpu_summary", "mean_util_pct"])), + gpu_imb=fmt(nested(s, ["gpu_summary", "max_min_ratio"])), + ) + ) + lines.extend([ + "", + "## Pairwise Comparisons", + "", + "| Comparison | TTFT p50 Δ | TTFT p90 Δ | E2E p50 Δ | E2E p90 Δ | TPOT p90 Δ | Wall-clock Δ |", + "|---|---:|---:|---:|---:|---:|---:|", + ]) + for c in comparisons: + lines.append( + "| {name} | {ttft50} | {ttft90} | {e2e50} | {e2e90} | {tpot90} | {wall} |".format( + name=c["name"], + ttft50=fmt_pct(c.get("ttft_p50_delta_pct")), + ttft90=fmt_pct(c.get("ttft_p90_delta_pct")), + e2e50=fmt_pct(c.get("e2e_p50_delta_pct")), + e2e90=fmt_pct(c.get("e2e_p90_delta_pct")), + tpot90=fmt_pct(c.get("tpot_p90_delta_pct")), + wall=fmt_pct(c.get("wall_clock_delta_pct")), + ) + ) + lines.extend([ + "", + "## What We Can Say Now", + "", + ]) + for item in claim_matrix: + lines.append(f"- **{item['status']}**: {item['claim']}") + lines.append(f" Supporting data: {item['supporting_data']}") + lines.append(f" Next: {item['needed_next']}") + lines.extend([ + "", + "## Main Reviewer Risks", + "", + ]) + for item in risk_register: + lines.append(f"- **{item['severity']}**: {item['risk']} - {item['mitigation']}") + lines.append("") + return "\n".join(lines) + + +def render_claim_matrix(items: list[JsonDict]) -> str: + lines = [ + "# Characterization Claim Matrix", + "", + "| Claim | Status | Supporting Data | Needed Next | Reviewer Risk |", + "|---|---|---|---|---|", + ] + for item in items: + lines.append( + f"| {item['claim']} | `{item['status']}` | {item['supporting_data']} | {item['needed_next']} | {item['reviewer_risk']} |" + ) + lines.append("") + return "\n".join(lines) + + +def render_risk_register(items: list[JsonDict]) -> str: + lines = [ + "# Reviewer Risk Register", + "", + "| Risk | Severity | Evidence | Mitigation |", + "|---|---|---|---|", + ] + for item in items: + lines.append( + f"| {item['risk']} | `{item['severity']}` | {item['evidence']} | {item['mitigation']} |" + ) + lines.append("") + return "\n".join(lines) + + +def render_figures_index(summaries: list[JsonDict]) -> str: + return "\n".join([ + "# Figures Index", + "", + "No generated figures are committed by this script. Batch-specific figures should be generated from:", + "", + "- `analysis/characterization/analyze.py` for Batch 0/1 trace figures.", + "- future Batch 2 step-timeline artifacts for interference plots.", + "- future Batch 3 per-worker/session artifacts for hot-spot plots.", + "- future Batch 4 arrival-rate sweep artifacts for SRR curves.", + "", + "This file exists so the audit package has a stable placeholder until fresh figures are generated.", + "", + ]) + + +def render_reproduction_commands(args: argparse.Namespace, run_dirs: list[Path]) -> str: + runs = " ".join(str(p) for p in run_dirs) + return "\n".join([ + "#!/usr/bin/env bash", + "set -euo pipefail", + "", + "# Rebuild this current-results audit package.", + f"python3 analysis/characterization/summarize_runs.py --output-dir {args.output_dir} --runs {runs}", + "", + "# Example Batch 0/1 local trace analysis.", + "python3 analysis/characterization/analyze.py \\", + " --trace traces/w600_r0.0015_st30.jsonl \\", + " --kv-bytes-per-token 98304 \\", + " --task-name w600_local_full_trace \\", + " --overwrite", + "", + ]) + + +def load_json(path: Path) -> JsonDict: + if not path.exists(): + return {} + try: + data = json.loads(path.read_text(encoding="utf-8")) + except Exception: + return {} + return data if isinstance(data, dict) else {} + + +def load_jsonl(path: Path) -> list[JsonDict]: + if not path.exists(): + return [] + rows = [] + with path.open(encoding="utf-8") as handle: + for line in handle: + if not line.strip(): + continue + try: + row = json.loads(line) + except Exception: + continue + if isinstance(row, dict): + rows.append(row) + return rows + + +def write_json(path: Path, data: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") + + +def first_present(row: JsonDict, keys: list[str]) -> Any: + for key in keys: + if key in row: + return row[key] + return None + + +def stat_value(run: JsonDict, stat_key: str, value_key: str) -> float | None: + stats_obj = run.get(stat_key) + if not isinstance(stats_obj, dict): + return None + return to_float(stats_obj.get(value_key)) + + +def nested(row: JsonDict, keys: list[str]) -> Any: + cur: Any = row + for key in keys: + if not isinstance(cur, dict): + return None + cur = cur.get(key) + return cur + + +def pct_delta(base: Any, variant: Any) -> float | None: + b = to_float(base) + v = to_float(variant) + if b is None or v is None or b == 0: + return None + return (v - b) / b * 100.0 + + +def to_float(value: Any) -> float | None: + if value is None: + return None + try: + out = float(value) + except (TypeError, ValueError): + return None + return out if math.isfinite(out) else None + + +def stats(values: list[float]) -> JsonDict | None: + clean = sorted(float(v) for v in values if math.isfinite(float(v))) + if not clean: + return None + return { + "count": len(clean), + "mean": statistics.fmean(clean), + "p50": percentile(clean, 0.50), + "p90": percentile(clean, 0.90), + "p95": percentile(clean, 0.95), + "p99": percentile(clean, 0.99), + "max": clean[-1], + } + + +def percentile(values: list[float], q: float) -> float: + if len(values) == 1: + return values[0] + rank = q * (len(values) - 1) + lo = int(rank) + hi = min(lo + 1, len(values) - 1) + frac = rank - lo + return values[lo] * (1 - frac) + values[hi] * frac + + +def top_contribution(values: list[float]) -> JsonDict: + clean = sorted([v for v in values if math.isfinite(v)], reverse=True) + total = sum(clean) + if not clean or total <= 0: + return {"top_1pct": None, "top_5pct": None, "top_10pct": None} + + def frac(pct: float) -> float: + k = max(1, math.ceil(len(clean) * pct)) + return sum(clean[:k]) / total + + return { + "top_1pct": frac(0.01), + "top_5pct": frac(0.05), + "top_10pct": frac(0.10), + } + + +def fmt(value: Any) -> str: + num = to_float(value) + if num is None: + return "n/a" + if abs(num - round(num)) < 1e-9 and abs(num) < 1_000_000: + return str(int(round(num))) + return f"{num:.3g}" + + +def fmt_pct(value: Any) -> str: + num = to_float(value) + if num is None: + return "n/a" + return f"{num:+.1f}%" + + +def git_commit() -> str: + try: + result = subprocess.run( + ["git", "rev-parse", "HEAD"], + check=True, + stdout=subprocess.PIPE, + stderr=subprocess.DEVNULL, + text=True, + ) + except Exception: + return "" + return result.stdout.strip() + + +if __name__ == "__main__": + main() diff --git a/analysis/characterization_todo_for_interns.md b/analysis/characterization_todo_for_interns.md new file mode 100644 index 0000000..a37c56e --- /dev/null +++ b/analysis/characterization_todo_for_interns.md @@ -0,0 +1,641 @@ +# Agentic Workload Characterization TODO + +Status: execution checklist for interns +Date: 2026-05-25 + +## 0. Purpose + +We are not starting from the assumption that Unified routing or PUSH +migration is already the answer. + +The first goal is to build a rigorous characterization package that proves: + +1. which dimensions make agentic serving different; +2. where static PD-disaggregation works poorly; +3. where PD-colocation/cache-aware routing still has residual failure modes; +4. how these failure modes reduce sustainable request rate under SLO. + +Only after these facts are established should we refine the positive system +design. + +Primary system goal: + +```text +maximize sustainable request rate under SLO +``` + +Prefill-decode interference and session hot-spot imbalance are mechanisms +that may reduce SRR. They are not the final metric by themselves. + +## 1. Global Delivery Rules + +Every task must produce data, figures, and an audit trail. A task is not +complete if it only produces a written conclusion. + +Use this output layout: + +```text +outputs/characterization/// +├── manifest.json +├── raw/ +├── summary.json +├── summary.md +├── figures/ +└── audit.md +``` + +Required fields in `manifest.json`: + +```json +{ + "git_commit": "", + "host": "", + "gpu_type": "", + "gpu_count": 0, + "trace_path": "", + "trace_sha256": "", + "policy": "", + "launch_command": "", + "request_limit": null, + "time_scale": null, + "session_sampling_method": "", + "session_sequential": true, + "start_time": "", + "end_time": "" +} +``` + +Every comparison must report: + +- attempted requests +- completed requests +- errors / timeouts +- goodput +- TTFT p50/p90/p99 +- E2E p50/p90/p99 +- TPOT p50/p90/p99 +- per-worker queue metrics +- per-worker GPU utilization +- per-worker KV occupancy if available +- per-worker APC / cache-hit metrics + +Every figure must be reproducible from raw data by a script committed or +saved alongside the artifact. + +## 2. Batch 0: Benchmark Substrate Audit + +### Goal + +Prove the load generator and trace replay are valid before trusting any +performance result. + +The most important invariant: + +```text +For online agentic serving, each session must have at most one in-flight turn. +Turn N+1 must not be sent before turn N completes. +``` + +### TODO + +1. Implement or run an analyzer that reconstructs per-session request + intervals: + - dispatch timestamp + - first-token timestamp + - finish timestamp + - error / timeout timestamp +2. Compute max concurrent in-flight turns per session. +3. Compute session start-time distribution. +4. Compute turn inter-arrival distribution. +5. Classify each existing run as one of: + - `online_realistic` + - `burst_stress` + - `synthetic_microbench` + - `invalid_for_online_claim` +6. For any run where session sequentiality is violated, write down exactly + which claim it can still support. + +### Data Artifacts + +- `session_concurrency.json` +- `session_arrival_stats.json` +- `turn_interval_stats.json` +- `trace_profile.json` +- `invalid_runs.md` + +### Figures + +- session start-time CDF +- per-session max in-flight histogram +- turns per session CDF +- turn inter-arrival CDF + +### Audit Checks + +The `audit.md` must answer: + +1. Does the main trace satisfy `max_inflight_per_session == 1`? +2. If not, is the run explicitly labeled as stress or invalid? +3. Are attempted/completed/error counts included? +4. Are latency percentiles computed only over successes, and if so, is + goodput also reported? + +### Pass Criteria + +- Main online-serving experiments must have `max_inflight_per_session == 1`. +- Any violation must be clearly labeled and excluded from SRR claims. + +## 3. Batch 1: Workload Characterization + +### Goal + +Establish agentic workload facts independent of any proposed system. + +Required facts: + +1. long input, short output; +2. large per-request KV footprint; +3. reuse is mostly intra-session; +4. session token mass is heavy-tailed; +5. total prompt length and effective uncached prefill work are different. + +### TODO + +1. Compute input token CDF. +2. Compute output token CDF. +3. Compute input/output ratio. +4. Estimate KV footprint per request: + + ```text + kv_bytes_per_request = input_tokens * kv_bytes_per_token + ``` + +5. Decompose reusable KV into: + - intra-session reuse + - cross-session reuse + - shared/system-prefix reuse +6. Compute session-level skew: + - turns per session + - cumulative input tokens per session + - cumulative output tokens per session + - cumulative uncached tokens per session + - top-k session contribution +7. Compute append / effective-prefill distribution: + + ```text + uncached_tokens = input_tokens - cached_tokens + ``` + +8. Compare total input length vs uncached tokens. + +### Data Artifacts + +- `workload_summary.json` +- `kv_footprint_summary.json` +- `reuse_decomposition.json` +- `session_skew.json` +- `append_delta_stats.json` + +### Figures + +- input/output token CDF +- input/output ratio CDF +- KV footprint CDF +- reuse decomposition stacked bar +- turns per session CDF +- per-session token mass Lorenz curve +- top-k sessions token contribution bar +- total input vs uncached tokens scatter + +### Audit Checks + +The `audit.md` must answer: + +1. What are input p50/p90/p99? +2. What are output p50/p90/p99? +3. What is the estimated KV footprint p50/p90/p99? +4. What fraction of reuse is intra-session? +5. What fraction of total token mass comes from top 1% / 5% sessions? +6. Are long prompts often small appends after cache reuse? + +### Pass Criteria + +The batch passes only if these facts can be stated numerically with raw data +links and plotted figures. + +## 4. Batch 2: PD-Colo Prefill-Decode Interference Proof + +### Goal + +Prove that PD-colocation can suffer from prefill-decode interference under +high load, and quantify how much this affects TPOT, decode queueing, and SLO. + +Hypothesis: + +```text +When heavy uncached prefill overlaps with active decode on the same worker, +decode TPOT and/or decode queue delay increases. +``` + +### TODO + +1. Run controlled microbenchmarks: + - decode-only steady load; + - decode load plus same-worker heavy prefill injection; + - decode load plus different-worker heavy prefill injection. +2. Sweep uncached prefill sizes: + - 2k + - 8k + - 16k + - 32k + - 64k +3. If supported, sweep chunked prefill size. +4. Log timestamps for: + - decode steps; + - prefill start/end; + - prefill chunks; + - queue admission; + - request completion. +5. In trace replay, label decode steps by whether they overlap with + same-worker prefill. +6. Compute: + + ```text + interference_index = + TPOT_p90(decode steps overlapping same-worker prefill) + / TPOT_p90(decode steps without same-worker prefill) + ``` + +7. Compare same-worker vs different-worker controls. + +### Data Artifacts + +- `interference_microbench_summary.json` +- `decode_step_timeseries.csv` +- `prefill_overlap_events.jsonl` +- `interference_index.json` +- `trace_overlap_summary.json` + +### Figures + +- TPOT time series with prefill overlap annotation +- interference index vs uncached prefill size +- same-worker vs different-worker TPOT boxplot +- chunk size vs TTFT/TPOT tradeoff +- trace replay overlap vs non-overlap TPOT comparison + +### Audit Checks + +The `audit.md` must answer: + +1. Is the interference observed on the same worker? +2. Is the different-worker control significantly weaker? +3. Does interference grow with uncached prefill size? +4. Does the phenomenon appear in real trace replay, not only microbench? +5. Could the result be explained by global load instead of local colocation? + +### Pass Criteria + +- Same-worker overlap must measurably increase TPOT or decode queue delay. +- The effect must be weaker or absent in the different-worker control. +- The effect must be visible in at least one trace replay setting. + +## 5. Batch 3: Session Hot-Spot Residual Imbalance Proof + +### Goal + +Prove that cache-aware/LMetric is a strong baseline but still leaves residual +hot-worker imbalance due to session skew and locality. + +Hypothesis: + +```text +Cache-aware routing preserves locality by attracting future turns to cached +workers. This is usually good, but heavy-tailed sessions can create hot +workers whose queue delay/SLO violations are much worse than the median +worker even when other workers still have headroom. +``` + +### TODO + +1. Run the same session-causal trace with: + - corrected LMetric/cache-aware; + - load-only routing; + - hard sticky routing; + - current Unified hybrid, if available. +2. For each worker, record: + - assigned session count; + - cumulative input tokens; + - cumulative uncached tokens; + - cumulative output tokens; + - request queue delay; + - decode queue delay; + - GPU utilization; + - KV occupancy; + - APC / cache-hit rate; + - SLO violations. +3. For each session, record: + - worker set used; + - primary worker; + - cumulative token mass; + - number of turns; + - latency contribution; + - whether it appears in slow-request set. +4. Create a session-mass capped or equalized replay: + - cap max session turns or token mass; + - rerun LMetric/cache-aware; + - compare hot-spot index. +5. Compute: + + ```text + hotspot_index = + max_worker_queue_delay_p90 / median_worker_queue_delay_p90 + ``` + +6. Compute locality/load tradeoff: + + ```text + locality_gain = APC(policy) - APC(load_only) + imbalance_cost = + max_worker_latency_p90(policy) - median_worker_latency_p90(policy) + ``` + +### Data Artifacts + +- `worker_balance_summary.json` +- `session_to_worker_map.json` +- `session_mass_summary.json` +- `routing_policy_comparison.json` +- `hotspot_index.json` +- `capped_session_replay_summary.json` + +### Figures + +- per-worker queue delay bar +- per-worker token mass bar +- GPU utilization timeline by worker +- KV occupancy timeline by worker +- APC vs queue delay scatter +- top sessions contribution bar +- policy tradeoff plot: APC vs hotspot_index +- original vs session-capped hot-spot comparison + +### Audit Checks + +The `audit.md` must answer: + +1. Does LMetric/cache-aware still show worker-level skew? +2. Are SLO violations concentrated on hot workers or hot sessions? +3. Does load-only routing improve balance but reduce APC/locality? +4. Does hard sticky improve locality but worsen hot-spot/HOL? +5. Does session-mass capping reduce hot spots? + +### Pass Criteria + +- LMetric/cache-aware must be shown as strong but imperfect. +- There must be measurable residual hot-worker imbalance. +- The imbalance must correlate with session token mass or locality. + +## 6. Batch 4: Sustainable Request Rate Sweep + +### Goal + +Connect interference and hot-spot mechanisms to the final metric: + +```text +SRR(SLO) = max arrival rate satisfying SLO in steady state +``` + +### TODO + +1. Define provisional SLO thresholds. Use configurable values, for example: + + ```text + TTFT_p90 <= T_ttft + E2E_p90 <= T_e2e + TPOT_p90 <= T_tpot + error_rate <= epsilon + queue length stable + KV occupancy stable + ``` + +2. Implement arrival-rate sweep: + - Poisson session arrivals; + - session-internal sequentiality; + - warmup window; + - steady-state measurement window. +3. For each arrival rate `lambda`, run: + - PD-colo cache-aware/LMetric; + - static PD-disagg; + - current Unified hybrid; + - optional hard sticky; + - optional load-only. +4. Find maximum sustainable lambda for each policy. +5. Report instability reasons: + - SLO violation; + - queue growth; + - KV occupancy growth; + - error/timeout growth. + +### Data Artifacts + +- `srr_curve.json` +- `lambda_runs//summary.json` +- `slo_violation_reason.json` +- `goodput_vs_arrival_rate.json` +- `stability_summary.json` + +### Figures + +- SRR bar chart +- TTFT p90 vs arrival rate +- E2E p90 vs arrival rate +- TPOT p90 vs arrival rate +- goodput vs arrival rate +- error rate vs arrival rate +- queue length over time near failure point +- KV occupancy over time near failure point + +### Audit Checks + +The `audit.md` must answer: + +1. Are session arrivals open-loop and Poisson? +2. Is session-internal sequentiality enforced? +3. How long are warmup and steady-state windows? +4. Is SRR failure persistent rather than transient? +5. Are completed/requested counts reported at every lambda? +6. Are policies compared on the same trace and same arrival process? + +### Pass Criteria + +- Each policy must have a measured SRR under the same SLO. +- Failure must be attributed to persistent SLO violation, queue growth, KV + growth, or error growth. +- Data must be session-causal. + +## 7. Batch 5: Failure Attribution Near SRR Boundary + +### Goal + +At and around the PD-colo/LMetric failure point, determine whether SLO +violations are caused by prefill-decode interference, session hot spots, KV +pressure, cache misses, or other mechanisms. + +### TODO + +1. Select three arrival rates: + + ```text + lambda = 0.9 * SRR + lambda = 1.0 * SRR + lambda = 1.1 * SRR + ``` + +2. For every slow or SLO-violating request, assign labels: + - same-worker prefill overlap; + - hot worker queue; + - high KV occupancy; + - cache miss / large uncached append; + - transfer wait; + - P queue wait; + - D admission wait; + - unknown. +3. Produce per-request waterfall for representative slow requests. +4. Produce per-worker timeline around failure windows. +5. Summarize cause distribution. + +### Data Artifacts + +- `slow_request_attribution.jsonl` +- `failure_breakdown.json` +- `case_studies.md` +- `worker_failure_windows.json` + +### Figures + +- SLO violation cause stacked bar +- slow request waterfall +- worker timeline near failure +- prefill/decode/KV/queue stacked breakdown +- failure cause vs arrival rate + +### Audit Checks + +The `audit.md` must answer: + +1. What fraction of slow requests overlap same-worker prefill? +2. What fraction are on hot workers? +3. What fraction happen under high KV occupancy? +4. What fraction are large uncached append requests? +5. For PD-disagg/Unified migration, how much time is transfer/P queue/D wait? +6. What remains unexplained? + +### Pass Criteria + +The batch must answer: + +1. Why PD-colo/LMetric hits its SRR limit. +2. Why static PD-disagg hits its SRR limit. +3. If Unified/PUSH underperforms, whether the cause is trigger quality, cost + model, transfer overhead, wrong load regime, or something else. + +## 8. Batch 6: Audit Package + +### Goal + +Make the whole characterization package reviewable by a strict systems +reviewer. + +### TODO + +1. Write a claim matrix: + + ```text + claim -> data artifact -> figure -> script -> caveat -> reviewer risk + ``` + +2. Write a figure index: + - figure filename; + - source data; + - generation command; + - intended claim. +3. Write a reviewer risk register: + - loadgen validity risks; + - trace representativeness risks; + - metric bias risks; + - implementation-specific risks; + - generalization risks. +4. Write a reproduction script or command list. +5. Mark experiments that cannot support main claims. + +### Final Artifacts + +- `characterization_claim_matrix.md` +- `all_figures_index.md` +- `reviewer_risk_register.md` +- `reproduction_commands.sh` +- `main_claim_allowed_runs.md` + +### Audit Checks + +The final package must satisfy: + +1. Every claim links to raw data. +2. Every figure can be regenerated. +3. Every experiment has a manifest. +4. Every caveat is explicit. +5. Invalid or stress-only runs are not used for online-serving claims. + +## 9. Priority Order + +### Priority 1 + +Do these first: + +1. Batch 0: Benchmark Substrate Audit +2. Batch 1: Workload Characterization +3. Batch 3: Session Hot-Spot Residual Imbalance Proof + +Reason: + +These define whether the trace and routing problem are real. Without them, +SRR sweeps and system experiments are not trustworthy. + +### Priority 2 + +Do these after the substrate and workload facts are stable: + +1. Batch 2: PD-Colo Prefill-Decode Interference Proof +2. Batch 5: Failure Attribution Near SRR Boundary + +Reason: + +These explain the mechanisms behind SLO/SRR failure and determine what the +positive system should actually fix. + +### Priority 3 + +Do these after instrumentation and attribution are ready: + +1. Batch 4: Sustainable Request Rate Sweep +2. Batch 6: Audit Package + +Reason: + +SRR sweeps are expensive. They should run only after trace validity, +logging, and attribution labels are ready. + +## 10. Non-Negotiable Reviewer Rules + +1. Do not use session-nonsequential loadgen for online-serving claims. +2. Do not compare latency percentiles without attempted/completed/error counts. +3. Do not use APC alone as a success metric. +4. Do not use average GPU utilization as proof of load balance. +5. Do not compare policies on different traces unless explicitly labeled. +6. Do not hide failed requests or timeouts. +7. Do not claim Unified/PUSH is the answer before failure attribution proves + the relevant bottleneck and cost budget. +8. Treat corrected LMetric/cache-aware PD-colo as the main baseline. +9. Treat static PD-disagg as an important baseline, not a strawman. +10. Every result must be reproducible from raw artifacts and commands.