gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(metrics): exclude aborted requests from latency/ttft/tpot stats

Browse Source 
The old filter `if row.latency_s is not None` accepted SGLang's fast
input-length-aborts (latency_s ~ 0.08s, finish_reason='abort/BadRequest')
as if they were successful zero-cost requests. This deflated mean/p50
of any run where the model rejected oversized inputs.

Impact on existing comparisons (ts=1 4-run validation + v2):
  KVC v2 has 40 aborts + 5 ReadTimeouts (was reported as just 5);
  DP 4w  has 67 aborts (was reported as 5).
Both runs have abort behavior; the asymmetry (40 vs 67) is purely from
SGLang's mem-fraction-derived max-input-len: KVC decode-only worker gets
~10 GB free GPU mem -> max-input=92098, DP fused worker gets ~9 GB ->
max-input=87811, because DP also needs chunked-prefill workspace.

The KVC-vs-DP latency-win direction holds and widens slightly under the
fixed filter (lat mean delta: -0.8% -> -1.4%); see V2_DEEP_ANALYSIS_ZH
§4.3 for the recomputed table.

Changes:
- metrics.py: new _is_failed_request(row) helper; latency/ttft/tpot
  stats now exclude both errors and aborts. New summary fields
  abort_count and failure_count expose the counts directly.
- scripts/analysis/recompute_summary.py: re-derives summary.json from
  existing metrics.jsonl using the fixed code, with optional --diff
  against the old buggy summary for inspection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kzlin
2026-05-11 17:29:18 +08:00
parent 0c25168cad
commit 5eac9b4f6b
2 changed files with 130 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
src/agentic_pd_hybrid/metrics.py
View File

@@ -114,6 +114,16 @@ def write_metrics_jsonl(path: Path, rows: list[RequestMetrics]) -> None:
handle.write(json.dumps(asdict(row), sort_keys=True) + "\n")
def _is_failed_request(row: RequestMetrics) -> bool:
if row.error is not None:
return True
if row.finish_reason is not None:
fr = str(row.finish_reason).lower()
if "abort" in fr or "badrequest" in fr:
return True
return False
def write_summary_json(
path: Path,
rows: list[RequestMetrics],
@@ -121,9 +131,10 @@ def write_summary_json(
trace_path: Path,
router_url: str | None,
) -> None:
latencies = [row.latency_s for row in rows if row.latency_s is not None]
ttfts = [row.ttft_s for row in rows if row.ttft_s is not None]
tpots = [row.tpot_s for row in rows if row.tpot_s is not None]
successful = [row for row in rows if not _is_failed_request(row)]
latencies = [row.latency_s for row in successful if row.latency_s is not None]
ttfts = [row.ttft_s for row in successful if row.ttft_s is not None]
tpots = [row.tpot_s for row in successful if row.tpot_s is not None]
per_decode_load = Counter(row.assigned_decode_node for row in rows)
per_prefill_load = Counter(row.assigned_prefill_node for row in rows)
prefill_priorities = Counter(
@@ -167,6 +178,17 @@ def write_summary_json(
str(key): value for key, value in sorted(decode_priorities.items())
},
"error_count": sum(1 for row in rows if row.error is not None),
"abort_count": sum(
1
for row in rows
if row.error is None
and row.finish_reason is not None
and (
"abort" in str(row.finish_reason).lower()
or "badrequest" in str(row.finish_reason).lower()
)
),
"failure_count": sum(1 for row in rows if _is_failed_request(row)),
"truncated_request_count": sum(
1
for row in rows