feat(analyzer): surface TTFT/TPOT as features and CDF figures
The gateway records three epoch-ms stamps per request (admitted, forwarded to backend, backend first token). They reached *-raw.jsonl but stopped there: features.csv had only total latency, and none of the figures covered time-to-first-token. Derive ttft_ms, ttft_e2e_ms and tpot_ms in one helper shared by the streaming path (preparation.py) and the in-memory path (features.py) so the two cannot drift, and add figures 14 (TTFT, backend vs end-to-end) and 15 (TPOT). The two TTFT series are plotted together because their offset is the gateway admission delay, which is then readable straight off the chart. Rows missing a backend stamp and single-token responses are recorded as 0 and filtered at plot time; left in, they would stack at the origin and pull every percentile down. Figure 15 is a single panel: TPOT is near-deterministic (p50 21.9, p90 23.0), so a p90 zoom renders a flat line and a vertical edge. Verified by recomputing ttft_ms straight from the original SLS rows for all 2190 requests of a sample: 0 mismatches. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -154,6 +154,38 @@ def series_stats(values):
|
||||
}
|
||||
|
||||
|
||||
def latency_breakdown(meta, output_tokens):
|
||||
"""Split one request's latency into time-to-first-token and time-per-output-token.
|
||||
|
||||
The gateway records three epoch-ms stamps per request: when it admitted the
|
||||
request, when it forwarded it to the backend, and when the backend's first
|
||||
token came back. So:
|
||||
|
||||
ttft_ms backend serving time to first token (prefill + backend queue)
|
||||
ttft_e2e_ms the same, plus the gateway admission delay ahead of it
|
||||
tpot_ms decode phase averaged over the tokens after the first
|
||||
|
||||
Rows missing either backend stamp -- about 0.5% of a production dump -- and
|
||||
single-token responses (no decode interval exists) yield 0. Callers must drop
|
||||
those rather than plot them as genuine zero-latency requests.
|
||||
"""
|
||||
first_request_ms = safe_int(getattr(meta, "backend_first_request_time_ms", 0))
|
||||
first_response_ms = safe_int(getattr(meta, "backend_first_response_time_ms", 0))
|
||||
ready_ms = safe_int(getattr(meta, "request_ready_time_ms", 0))
|
||||
end_ms = safe_int(getattr(meta, "request_end_time_ms", 0))
|
||||
output_tokens = safe_int(output_tokens)
|
||||
|
||||
ttft_ms = first_response_ms - first_request_ms if first_request_ms and first_response_ms else 0
|
||||
ttft_e2e_ms = first_response_ms - ready_ms if ready_ms and first_response_ms else 0
|
||||
decode_ms = end_ms - first_response_ms if end_ms and first_response_ms else 0
|
||||
tpot_ms = decode_ms / (output_tokens - 1) if decode_ms > 0 and output_tokens > 1 else 0.0
|
||||
return {
|
||||
"ttft_ms": ttft_ms if ttft_ms > 0 else 0,
|
||||
"ttft_e2e_ms": ttft_e2e_ms if ttft_e2e_ms > 0 else 0,
|
||||
"tpot_ms": round(tpot_ms, 3),
|
||||
}
|
||||
|
||||
|
||||
def safe_div(numerator, denominator):
|
||||
if not denominator:
|
||||
return 0.0
|
||||
|
||||
Reference in New Issue
Block a user