diff --git a/docs/simulator-claim-map-20260716.md b/docs/simulator-claim-map-20260716.md new file mode 100644 index 0000000..0914161 --- /dev/null +++ b/docs/simulator-claim-map-20260716.md @@ -0,0 +1,75 @@ +# Simulator-for-config-tuning related-work claim map + +日期：2026-07-16。目的：为「Frontier/Vidur-class simulator 能否低成本解决 config tuning」这条主线建立 related-work 边界。Vidur 与 LLMServingSim 的条目基于原文（PDF 全文核读）；SimAI 基于论文页与摘要口径。每项按 Context / Claim / Assumption / Mechanism / Evidence / Boundary / 与本 project 的关系提取。 + +## Vidur（MLSys 2024，arXiv:2405.05465） + +| 维度 | 内容 | +|---|---| +| Context | MSR India。首个面向 LLM inference 的大规模模拟器。Motivation 与我们一致：config search 复杂度 O(\|M\|·\|T\|)，且 optimal config 是 (model, trace) 的函数——Fig 1b 显示跨 trace misconfiguration 代价最高 2×。 | +| Claim | (a) request-level 预测误差 <9%：static trace 上 P95 normalized execution latency 误差 ≤3.33%（4 模型 × 3 trace），dynamic trace 在 **85% capacity** 负载下误差 <5%。(b) Vidur-Search 用约 1 小时 96-core CPU（$9.93/h）为 LLaMA2-70B 找到最优 config，对比 deployment-based exploration 估算 42K GPU-hours ≈ $218K。(c) what-if 全量探索 $125 模拟成本 vs 估算 $1.14M 真机成本。 | +| Assumption | operator runtime 可由单 GPU profiling + 小型 ML 估计器（random forest）插值；prefill attention 可用等效单序列 sqrt(Σp_i²) 近似；decode attention runtime 只依赖总 KV 读量而非 per-request context 分布；LLM 架构同质（小算子集合跨模型共享）。 | +| Mechanism | 声明式 model spec → 算子三分类（token-level / sequence-level / communication）→ 单 GPU CUPTI profiling → RF runtime estimator → event-driven simulator + 三层 hierarchical scheduler（支持 vLLM/Orca+/Sarathi-Serve/FasterTransformer/LightLLM 策略）→ Vidur-Search 对每个 config 二分搜索 max QPS（判据 P99 scheduling delay <5s），目标 QPS/dollar。 | +| Evidence | LLaMA2-7B/70B、InternLM-20B、Qwen-72B（全 dense）；Azure A100/H100 4-GPU pairwise-NVLink 节点；Chat-1M / Arxiv-4K / BWB-4K trace，总长截断到 4096 tokens。 | +| Boundary | **作者明示**：接近 capacity point 时小误差会因排队失控放大，故 fidelity 评测停在 85% capacity。**结构性**：无 MoE、无 FP8/量化、无 prefix-cache reuse（多轮对话按独立请求处理）、无 speculative decoding（列为 future work）、PP 仅同步、长上下文未覆盖（4K 截断）。metric 口径为 normalized execution latency（static 排除 scheduling delay）。**最关键**：sim 选出的 config 在真机 ground-truth surface 上的 selection regret 从未被验证；42K GPU-h/$218K 是反事实估算，分母是穷举式 exploration 而非 strong sequential tuner。 | +| 与本 project 的关系 | Frontier 即 Vidur-class（代码直接使用 vidur backend）+ 我们的 FP8/MoE/EP/decode-profile patches。我们的所有实验恰好工作在 Vidur 声明误差爆炸并回避的 regime（capacity point + SLO gate），补的正是它缺的 selection-regret ground truth。我们的 zero-shot 失败（25–30% regret）与其 <9% 不矛盾——不同 metric、不同 load regime、不同 stack alignment，论文必须主动写明这一点。其 Fig 1b 的 workload-conditioned 结论与我们 P4 sign-flip、P6 churn 互为独立佐证 → 支持 retune 频率 / amortization 论证（C3）。 | + +## LLMServingSim（IISWC 2024，arXiv:2408.05499） + +| 维度 | 内容 | +|---|---| +| Context | KAIST。scale-out LLM serving 的 HW/SW co-simulation，面向 NPU/PIM/异构加速器设计探索，基于 ASTRA-sim。 | +| Claim | 对真实 multi-GPU vLLM serving 平均误差 14.7% 且「趋势一致」；比 mNPUsim/GeneSys/NeuPIMs 快 34.7–491×（摘要口径 91.5×）。 | +| Assumption | iteration-level 模拟 + decoder-block 冗余复用（编译一个 block 复制展开、attention/非 attention 分离）可在可行时间内保持足够精度；硬件行为可由可插拔 accelerator compiler+simulator 栈表达（GeneSys 原型）。 | +| Mechanism | 逐 iteration：scheduler（iteration-level batching、KV paging、operator mapping）→ per-device 硬件模拟 → graph converter（Chakra）→ ASTRA-sim 网络级模拟 → 循环。 | +| Evidence | 与 multi-GPU vLLM 真机对照，变量为 LLM 架构、并行方案、NPU 数量、异构度；报告平均误差与趋势一致性。 | +| Boundary | 定位是硬件/系统设计空间探索，不是 engine-knob config tuning；validation 口径是 trend-following，无 SLO-gated capacity、无 selection-regret；14.7% 平均误差大于典型 config 间 capacity margin（我们 12-cell 面上 top-2 差距 0.76%），故该精度不足以支撑近邻 config 选择。 | +| 与本 project 的关系 | 说明「模拟保 trend」是社区通行 validation 标准；「trend ≠ selection」这一缺口对它同样成立。不构成直接 baseline，但在 related work 中界定我们评测口径（selection regret at capacity point）的必要性。 | + +## SimAI（NSDI 2025，Alibaba，aliyun/SimAI） + +| 维度 | 内容 | +|---|---| +| Context | 大规模 LLM **training** 的架构设计与参数调优模拟，生产背景（Alibaba Cloud）。 | +| Claim | 各测试场景平均 98.1% 与真实结果对齐；为 host 设计与参数设置提供生产可用 guidance。 | +| Assumption | training 过程可由 framework + kernel computation + collective communication 的选择性高保真集成复现。 | +| Mechanism | 高保真集成三层栈 + 多线程加速 + lock-free global context sharing。 | +| Evidence | 与生产 training 场景对齐（论文口径，未逐一核读实验细节）。 | +| Boundary | training-only。training iteration 均匀、batch 组成静态——恰是 Vidur 指出 inference 所缺的性质，因此 98.1% 不可外推到 serving capacity point。 | +| 与本 project 的关系 | 「simulator 指导 infra 决策」的工业先例与动机背书；不与 serving config tuning claim 竞争。引用价值在 motivation，不在 evaluation 对照。 | + +## Frontier（本 project 被测对象，非 related work） + +内部 Vidur-class 实现（vidur backend）+ 本 project 的 FP8/MoE tuning-key、Qwen MoE serving plan、TP/EP-aware cache key、critical-lane、decode/true-mixed profile 补丁。我们全部 fidelity 结论限定于该实现与已声明的 patch 集；见 `simulator-fidelity.md`。 + +## Consensus / disagreement / uncovered regime + +**Consensus（三方一致或与我们互证）：** + +1. operator/iteration profile + 调度复合的模拟器，在中低负载下能达到 5–15% latency 误差；模拟成本比真机低数个数量级。 +2. optimal config 是 (model, workload) 的函数，misconfiguration 代价可达 ~2×（Vidur Fig 1b；我们 P4 pattern sign-flip 与 P6 engine-churn 独立复证）。 + +**Disagreement：** 无直接冲突数字。我们的 zero-shot 失败与 Vidur <9% 处于不同 metric/regime，论文需主动解释，防止被误读为矛盾或重复。 + +**Uncovered regime（本 project 的空间）：** + +1. **capacity-point + SLO-gated selection regret 无人用真机 ground-truth surface 验证。** Vidur 自认该 regime 误差爆炸并把评测停在 85% load；LLMServingSim 只验 trend。而 config tuning 的决策恰好发生在 capacity point。 +2. MoE、FP8、prefix reuse、speculative decoding、EP topology、长上下文均在已发表 fidelity envelope 之外。 +3. **alignment/profiling 成本从不与真机 tuning 成本同表比较。** Vidur 的 $218K 对比用穷举做分母；正确分母是 strong sequential tuner（我们实测 0.27–0.45 H20h/task，`runs/tuning-cost/metrics.json`）。 +4. envelope 失效的低成本检测（workload/runtime/topology 变化后何时还能信 simulator）无人提出。 + +## 对本 project claim 的直接影响 + +- **C1 定位句**：不是「Vidur 错了」，而是「Vidur-class claim 停在 sub-capacity load 的 prediction fidelity；把它外推到 SLO-gated capacity selection 是社区的隐含用法，我们证明该外推在 zero-shot 下失败（25–30% regret），并给出恢复 ranking 所需的最小真机证据层级」。 +- **C2**：Vidur 没有 minimum-real-evidence 的概念（要么全模拟、要么全真机）；per-TP calibration / 同栈 profile + KV capacity + patches 的证据层级是新贡献面。 +- **C3**：省钱叙事必须从「数量级」修正为「仅在 amortization 下成立」，分母换成 strong tuner 实测值；Vidur Fig 1b + 我们 P6 churn 共同支撑 retune 频率前提。 + +## 待 triage 的相邻工作（未读原文，暂不写 claim） + +APEX（arXiv:2411.17651，并行执行计划模拟）、LLMServingSim 2.0（arXiv:2602.23036，异构+分离式）、Charon（arXiv:2605.17164，training+inference 统一）、inference-fleet-sim（arXiv:2603.16054，排队论容量规划）、AgentServeSim（arXiv:2606.09613，多轮 agent serving）。若审稿风险评估需要，按本表格式各补一行。 + +## Sources + +- Vidur: （全文核读版本：mlsys24 PDF） +- LLMServingSim: +- SimAI: 、 diff --git a/runs/frontier-fidelity-envelope-v1/run_qwen235_v020_smoke.sh b/runs/frontier-fidelity-envelope-v1/run_qwen235_v020_smoke.sh index f2affde..227579c 100644 --- a/runs/frontier-fidelity-envelope-v1/run_qwen235_v020_smoke.sh +++ b/runs/frontier-fidelity-envelope-v1/run_qwen235_v020_smoke.sh @@ -54,6 +54,11 @@ if [[ ! -x "${TRTLLM_DG_NVCC_COMPILER}" ]]; then echo "ERROR: TRTLLM_DG_NVCC_COMPILER is not executable: ${TRTLLM_DG_NVCC_COMPILER}" >&2 exit 1 fi +if ! "${VENV_ROOT}/bin/python" "${CLIENT}" --help 2>&1 \ + | grep -F --quiet -- '--model-path'; then + echo "ERROR: CLIENT must implement the fixed-shape client contract (--model-path, --rate, --input-tokens, --output-tokens): ${CLIENT}" >&2 + exit 1 +fi case "${DEEP_GEMM_JIT_DEBUG}" in true|false) ;; *) echo "ERROR: DEEP_GEMM_JIT_DEBUG must be true or false" >&2; exit 1 ;; diff --git a/runs/frontier-multicase-sufficiency-v0/audit_ground_truth.py b/runs/frontier-multicase-sufficiency-v0/audit_ground_truth.py new file mode 100644 index 0000000..84fd81d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/audit_ground_truth.py @@ -0,0 +1,280 @@ +#!/usr/bin/env python3 +"""Audit the Qwen235B real-machine surfaces before comparing Frontier. + +This script intentionally does not consume simulator output. It establishes +whether each real response surface is complete and discriminative enough to +support a later claim about simulator config selection. +""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import math +from pathlib import Path +from typing import Any + + +SCHEMA = "frontier-multicase-ground-truth-v0" +EXPECTED_PROBES = 6 + + +def load_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as source: + value = json.load(source) + if not isinstance(value, dict): + raise ValueError(f"expected JSON object: {path}") + return value + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def config_from_result(result: dict[str, Any]) -> dict[str, int]: + flags = result["config_patch"]["flag_patch"] + tp = int(flags["tensor-parallel-size"]) + dp = int(flags.get("data-parallel-size", 1)) + return { + "tp": tp, + "dp": dp, + "ep": int(flags.get("expert-parallel-size", 1)), + "mns": int(flags["max-num-seqs"]), + "mbt": int(flags["max-num-batched-tokens"]), + "gpu_count": tp * dp, + } + + +def cell_id(config: dict[str, int]) -> str: + topology = f"tp{config['tp']}" + if config["dp"] != 1 or config["ep"] != 1: + topology += f"_dp{config['dp']}_ep{config['ep']}" + return f"{topology}_mns{config['mns']}_mbt{config['mbt']}" + + +def trial_record(case: str, path: Path) -> dict[str, Any]: + result = load_json(path) + config = config_from_result(result) + score = float(result["best_request_rate"]) / config["gpu_count"] + probes = result.get("probes", []) + infeasible_above = [ + float(probe.get("payload", probe)["request_rate"]) / config["gpu_count"] + for probe in probes + if not probe["feasible"] + and float(probe.get("payload", probe)["request_rate"]) + / config["gpu_count"] + > score + ] + upper_bound = min(infeasible_above) if infeasible_above else None + probe_count = len(result.get("probes", [])) + primary_result = result.get("best_source") == "primary_search" + no_probe_failure = not bool(result.get("completed_with_probe_failure", False)) + fully_valid = ( + result.get("status") == "completed" + and probe_count == EXPECTED_PROBES + and primary_result + and no_probe_failure + ) + return { + "case": case, + "cell_id": cell_id(config), + **config, + "score_req_s_per_gpu": score, + "capacity_lower_bound_req_s_per_gpu": score, + "capacity_upper_bound_req_s_per_gpu": upper_bound, + "capacity_bracket_width_req_s_per_gpu": ( + upper_bound - score if upper_bound is not None else None + ), + "best_request_rate_req_s": float(result["best_request_rate"]), + "best_sampling_u": float(result["best_sampling_u"]), + "best_pass_rate": float(result["best_pass_rate"]), + "probe_count": probe_count, + "best_source": result.get("best_source"), + "completed_with_probe_failure": bool( + result.get("completed_with_probe_failure", False) + ), + "fully_valid": fully_valid, + "result_path": str(path), + "result_sha256": sha256_file(path), + } + + +def summarize_case(case: str, rows: list[dict[str, Any]]) -> dict[str, Any]: + if not rows: + raise ValueError(f"no rows for case: {case}") + ids = [row["cell_id"] for row in rows] + if len(ids) != len(set(ids)): + duplicates = sorted(cell for cell in set(ids) if ids.count(cell) > 1) + raise ValueError(f"duplicate cells for {case}: {duplicates}") + + best = max(row["score_req_s_per_gpu"] for row in rows) + tolerance = max(1e-12, best * 1e-9) + top = [ + row["cell_id"] + for row in rows + if math.isclose(row["score_req_s_per_gpu"], best, abs_tol=tolerance) + ] + distinct_scores = [] + for score in sorted({row["score_req_s_per_gpu"] for row in rows}, reverse=True): + if not any(math.isclose(score, seen, abs_tol=tolerance) for seen in distinct_scores): + distinct_scores.append(score) + + max_lower_bound = max(row["capacity_lower_bound_req_s_per_gpu"] for row in rows) + possibly_optimal = [ + row["cell_id"] + for row in rows + if row["capacity_upper_bound_req_s_per_gpu"] is None + or row["capacity_upper_bound_req_s_per_gpu"] + tolerance >= max_lower_bound + ] + + total_pairs = len(rows) * (len(rows) - 1) // 2 + tied_pairs = sum( + 1 + for left_index, left in enumerate(rows) + for right in rows[left_index + 1 :] + if math.isclose( + left["score_req_s_per_gpu"], + right["score_req_s_per_gpu"], + abs_tol=tolerance, + ) + ) + return { + "case": case, + "cell_count": len(rows), + "fully_valid_cell_count": sum(bool(row["fully_valid"]) for row in rows), + "invalid_cells": [row["cell_id"] for row in rows if not row["fully_valid"]], + "best_score_req_s_per_gpu": best, + "top_set": sorted(top), + "top_set_size": len(top), + "random_top_set_hit_rate": len(top) / len(rows), + "distinct_score_count": len(distinct_scores), + "distinct_scores_req_s_per_gpu": distinct_scores, + "possibly_optimal_set_from_search_brackets": sorted(possibly_optimal), + "possibly_optimal_set_size": len(possibly_optimal), + "pair_count": total_pairs, + "tied_pair_count": tied_pairs, + "informative_pair_count": total_pairs - tied_pairs, + "informative_pair_fraction": ( + (total_pairs - tied_pairs) / total_pairs if total_pairs else 0.0 + ), + } + + +def markdown_report(metrics: dict[str, Any], rows: list[dict[str, Any]]) -> str: + lines = [ + "# Qwen235B ground-truth audit", + "", + "Objective: maximum SLO-feasible offered request throughput per GPU.", + "This report contains real-machine data only; it makes no Frontier match claim.", + "", + "| case | valid cells | score levels | top-set size | random top-set hit | informative pairs |", + "|---|---:|---:|---:|---:|---:|", + ] + for case in metrics["cases"]: + lines.append( + f"| {case['case']} | {case['fully_valid_cell_count']}/{case['cell_count']} " + f"| {case['distinct_score_count']} | {case['top_set_size']}/{case['cell_count']} " + f"| {case['random_top_set_hit_rate']:.1%} " + f"| {case['informative_pair_count']}/{case['pair_count']} " + f"({case['informative_pair_fraction']:.1%}) |" + ) + + lines.extend(["", "## Cells", ""]) + for case in metrics["cases"]: + lines.extend( + [ + f"### {case['case']}", + "", + "| cell | capacity bracket (req/s/GPU) | valid | probes | source |", + "|---|---:|---:|---:|---|", + ] + ) + for row in sorted( + (row for row in rows if row["case"] == case["case"]), + key=lambda row: row["cell_id"], + ): + upper = row["capacity_upper_bound_req_s_per_gpu"] + bracket = ( + f"[{row['capacity_lower_bound_req_s_per_gpu']:.9f}, " + f"{upper:.9f})" + if upper is not None + else f"[{row['capacity_lower_bound_req_s_per_gpu']:.9f}, +inf)" + ) + lines.append( + f"| {row['cell_id']} | {bracket} " + f"| {'yes' if row['fully_valid'] else 'no'} | {row['probe_count']} " + f"| {row['best_source']} |" + ) + lines.extend( + [ + "", + f"Top set: `{', '.join(case['top_set'])}`.", + f"Possibly optimal under binary-search brackets: " + f"`{', '.join(case['possibly_optimal_set_from_search_brackets'])}`.", + "", + ] + ) + + lines.extend( + [ + "## Interpretation guardrail", + "", + "A Frontier top-set hit is insufficient by itself because the surfaces contain " + "large ties. The later comparison must report selected-config regret and " + "tie-aware pairwise ranking, and must keep invalid real cells visible.", + "", + ] + ) + return "\n".join(lines) + + +def write_csv(path: Path, rows: list[dict[str, Any]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8", newline="") as target: + writer = csv.DictWriter(target, fieldnames=list(rows[0])) + writer.writeheader() + writer.writerows(rows) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--prefill-root", action="append", type=Path, required=True) + parser.add_argument("--decode-root", action="append", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + cases = {"qwen235b_prefill_only": args.prefill_root, "qwen235b_decode_only": args.decode_root} + rows = [] + for case, roots in cases.items(): + for root in roots: + paths = sorted(root.glob("store/*/trials/trial-*/result.json")) + if not paths: + raise ValueError(f"no result files below {root}") + rows.extend(trial_record(case, path) for path in paths) + + summaries = [ + summarize_case(case, [row for row in rows if row["case"] == case]) + for case in cases + ] + metrics = {"schema": SCHEMA, "cases": summaries} + args.output_dir.mkdir(parents=True, exist_ok=True) + write_csv(args.output_dir / "cells.csv", rows) + (args.output_dir / "metrics.json").write_text( + json.dumps(metrics, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + (args.output_dir / "report.md").write_text( + markdown_report(metrics, rows), encoding="utf-8" + ) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v0/audit_qwen30_baseline.py b/runs/frontier-multicase-sufficiency-v0/audit_qwen30_baseline.py new file mode 100644 index 0000000..3613b30 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/audit_qwen30_baseline.py @@ -0,0 +1,110 @@ +#!/usr/bin/env python3 +"""Extract the predictive-versus-calibrated Frontier Qwen30B baseline.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path +from typing import Any + + +MODES = ("uncalibrated/SLO-gated", "frozen-calibrated/SLO-gated") + + +def load_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as source: + value = json.load(source) + if not isinstance(value, dict): + raise ValueError(f"expected JSON object: {path}") + return value + + +def extract(metrics: dict[str, Any], protocol: dict[str, Any]) -> dict[str, Any]: + analyses = metrics["analyses"] + rows = [] + for mode in MODES: + analysis = analyses[mode] + values = analysis["metrics"] + top1 = values["top1"] + confusion = analysis["false_feasibility"]["overall"] + rows.append( + { + "mode": mode, + "selected_cells": top1["candidate_cells"], + "optimistic_real_regret": top1["optimistic_regret"], + "worst_case_real_regret": top1["worst_case_regret"], + "kendall_tau_b": values["kendall_tau_b"]["tau_b"], + "pairwise_exact_sign_accuracy": values["pairwise_direction"][ + "exact_sign_accuracy" + ], + "false_feasible": confusion["false_feasible"], + "false_infeasible": confusion["false_infeasible"], + "agreement": confusion["agreement"], + } + ) + return { + "schema": "frontier-qwen30-calibration-audit-v0", + "rows": rows, + "calibration": { + "fitted_a_tp": protocol["fitted_a_tp"], + "fit_fixture": protocol["fit_fixture"], + "holdout_fixture": protocol["holdout_fixture"], + "loss": protocol["loss"], + "refit_on_holdout": protocol["refit_on_holdout"], + }, + } + + +def report(result: dict[str, Any]) -> str: + lines = [ + "# Qwen30B Frontier baseline audit", + "", + "| mode | selected cells | worst real regret | Kendall tau-b | pair sign accuracy | feasibility (agree/FP/FN) |", + "|---|---|---:|---:|---:|---:|", + ] + for row in result["rows"]: + lines.append( + f"| {row['mode']} | {', '.join(row['selected_cells'])} " + f"| {row['worst_case_real_regret']:.2%} | {row['kendall_tau_b']:.4f} " + f"| {row['pairwise_exact_sign_accuracy']:.2%} " + f"| {row['agreement']}/{row['false_feasible']}/{row['false_infeasible']} |" + ) + calibration = result["calibration"] + lines.extend( + [ + "", + "The calibrated mode applies a distinct end-to-end execution-time scale per TP: " + + ", ".join( + f"TP{tp}={value:.6f}" + for tp, value in sorted(calibration["fitted_a_tp"].items()) + ) + + ".", + "", + f"Those scales were fitted against real total throughput on " + f"`{calibration['fit_fixture']}` and checked without refitting on " + f"`{calibration['holdout_fixture']}`. This validates within-workload transfer of " + "the calibration, not zero-shot Frontier prediction across TP.", + "", + ] + ) + return "\n".join(lines) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--metrics", type=Path, required=True) + parser.add_argument("--calibration-protocol", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + + result = extract(load_json(args.metrics), load_json(args.calibration_protocol)) + args.output_dir.mkdir(parents=True, exist_ok=True) + (args.output_dir / "metrics.json").write_text( + json.dumps(result, indent=2, sort_keys=True) + "\n", encoding="utf-8" + ) + (args.output_dir / "report.md").write_text(report(result), encoding="utf-8") + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v0/community-qwen235b-protocol.md b/runs/frontier-multicase-sufficiency-v0/community-qwen235b-protocol.md new file mode 100644 index 0000000..bc4ed2f --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/community-qwen235b-protocol.md @@ -0,0 +1,166 @@ +# Community vLLM Qwen235B versus Frontier protocol v0 + +## Hypothesis + +I believe that collecting compute and communication profiles from the same +community vLLM execution stack used for serving removes the current +internal-runtime confounder. I will verify this by freezing Frontier's +profile-only predictions before collecting the corresponding serving response +surface, then measuring config-selection regret and rank agreement. + +This experiment tests Frontier inside a declared compatibility envelope. It +does not claim fidelity for the previous internal vLLM, EAGLE3, DeepEP, or +external-KV setup. + +## Frozen system boundary + +- Host: `dash0`, 8 NVIDIA H20 GPUs, driver `580.95.05`. +- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8`. +- Model config SHA256: + `702c46d431bb984db9035a1225186bbfdb52c0d19c82104df4a37cd005e0369e`. +- Model index SHA256: + `e03d4abec9611fca05844d2c8b5a08318ddc3c86acfb9481a0ef5692bf76e6d6`. +- Frontier: NetX-lab/Frontier commit + `d9cfeb6d8791fbf2f295dd9744c56a666171776e`. +- vLLM target: exact community release `0.10.2`, installed in a new isolated + environment with wheel/source hash recorded. Frontier declares + `vllm>=0.10,<0.11`; the model README declares `vllm>=0.8.5`. +- Transformers is pinned to `4.55.2` and tokenizers resolves to `0.21.4`. + vLLM 0.10.2 only declares a lower bound on Transformers, while current + Transformers 5.x removes a tokenizer API used by this vLLM release. +- Do not use the shared `/usr/local` vLLM. Its imported version and package + metadata disagree, so it is not a reproducible community baseline. + +The profiler and serving process must import the same vLLM installation. The +following execution choices are fixed for the first validation pass: + +- FlashInfer attention in both profiler and serving; +- eager execution, with CUDA graphs disabled; +- no speculative decoding; +- no external KV connector; +- no prefix reuse; +- chunked prefill enabled; +- BF16 KV cache (`auto` for this model), while the checkpoint's block-wise FP8 + weight quantization remains enabled; +- community vLLM default MoE implementation unless a backend is explicitly + frozen and supported on both sides. + +The SM90 path cannot use `CutlassBlockScaledGroupedGemm` (the vLLM 0.10.2 +implementation gates that path on SM100), so this system uses Triton block-FP8 +MoE. However, the two sides do not yet select the same Triton tuning config. +Serving resolves the checked-in H20 config named with +`dtype=fp8_w8a8`, while Frontier's standalone MoE profiler requests a filename +without that FP8 dtype component and falls back to a default config. Formal +comparison is gated on repairing this profiler/runtime tuning-config mismatch +and re-profiling; recording the same vLLM package version is not sufficient. + +These controls isolate operator composition and scheduling. CUDA graphs, +FlashAttention, speculative decoding, external KV, and optimized expert +communication become separate stress cases after the controlled pass. + +## Case P: prefill-only + +Reuse the original trace window and SLO because the community model's 40,960 +token limit covers the filtered input plus the one-token completion. + +- Window: `thinking_w20260327_1000`. +- Input filter: 0--32,768 tokens. +- Output override: 1 token. +- Replay scale: 1.0. +- SLO pass rate: at least 0.95. +- TTFT SLO: 1 s for input <=8,191; 2 s otherwise. +- Objective: maximum SLO-feasible offered requests/s/GPU. +- Candidate grid: TP `{4,8}` x MNS `{64,128}` x MBT `{8192,16384}`. +- DP=1 and expert parallel disabled. + +This case can test the TP4-versus-TP8 decision and batching effects. It must not +reuse performance values from the internal 256k model as ground truth. + +## Case D: decode-dominant + +Do not initially reproduce the previous strict decode-only case. It depends on +`DecodeBenchConnector`, EAGLE3, FP8 KV, DeepEP/NVSHMEM, and decode CUDA graphs, +which are outside the controlled Frontier profile contract. + +Construct a community-only decode-dominant case that both systems can express: + +- same trace window and timestamp/sampling fields; +- input filter: 1--512 tokens; +- output override: 512 tokens with EOS ignored; +- SLO pass rate: at least 0.95; +- TPOT SLO: 40 ms; +- objective: maximum SLO-feasible offered requests/s/GPU; +- topology grid: `(TP=4, DP=2, EP=8)` and `(TP=2, DP=4, EP=8)`; +- batching grid: MNS `{64,128}` x MBT `{256,384}`. + +This is deliberately named decode-dominant, not decode-only. A strict +decode-only claim requires an initial-KV state contract in Frontier. + +## Required profile closure + +Profile data are measurement inputs, not end-to-end calibration. No serving +throughput or latency from either case may scale the profiles. + +1. Linear/operator profiles for the TP degrees consumed by the two grids. +2. FlashInfer attention profiles covering the observed prefill/decode batch, + context-length, and chunk-size ranges. +3. FP8 MoE profiles for the actually consumed parallel pairs: + `(MoE TP=4, EP=1)`, `(MoE TP=8, EP=1)`, and `(MoE TP=1, EP=8)`. +4. H20 intra-node collective profiles for TP all-reduce at world sizes 2, 4, + and 8 and the EP8 all-to-all path. + +Frontier already provides an H20 device description but no checked-in H20 +network profiles. Its public collective profiler covers all-reduce and +send/recv, not all-to-all. Therefore Case P may proceed after H20 all-reduce +closure; Case D remains blocked until EP8 all-to-all is either measured and +consumed or the selected communication model is independently validated +against those measurements. + +## Blind run order + +1. Build the isolated environment and record package/binary hashes. +2. Run one-row compute and collective smokes, then one TP4 server-load/request + smoke. A smoke failure stops the campaign. +3. Collect profiles and validate CSV metadata/coverage. +4. Run Frontier for every candidate and offered-load anchor. +5. Freeze simulator outputs and their SHA256 checksums. +6. Only then collect community-vLLM serving ground truth. Randomize the first + trial order and reverse it for the second trial. +7. Refine only decision-relevant capacity intervals that still overlap. + +## Metrics and decision rule + +Report per case: + +- absolute simulated and real TTFT/TPOT/throughput values; +- anchor-level SLO feasibility confusion; +- selected-config real regret, including interval-robust regret; +- Kendall tau-b with ties preserved; +- informative-pair direction accuracy; +- top-set hit and random top-set hit probability; +- profile and real-GPU measurement cost. + +Frontier is sufficient as a config ranker for this controlled family only if +every completed case has: + +- worst selected-config real regret <=5%; +- Kendall tau-b >=0.8 on enough informative pairs; +- no unresolved ground-truth interval capable of reversing the decision; +- no per-case or per-action end-to-end calibration. + +If the controlled pass succeeds, enable one omitted mechanism at a time. If it +fails, use operator/communication/stage residuals to localize which composition +assumption reverses the ranking before proposing a new tuner mechanism. + +## Launch gates and initial cost cap + +The first authorized GPU action should be smoke-only: + +- isolated vLLM import and Qwen235B TP4 load; +- one request through the community server; +- one representative FP8 linear, attention, MoE, and all-reduce profile point; +- expected wall time: 20--40 minutes; +- hard GPU budget: 2 H20-GPU-hours. + +No full response-surface sweep is authorized by this protocol. Its cost and +anchor count must be resolved from the smoke timings and echoed separately. diff --git a/runs/frontier-multicase-sufficiency-v0/findings.md b/runs/frontier-multicase-sufficiency-v0/findings.md new file mode 100644 index 0000000..567e308 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/findings.md @@ -0,0 +1,114 @@ +# Frontier multi-case sufficiency: current finding + +## Bottom line + +The existing evidence does **not** show that Frontier already solves config +selection. The strongest Qwen30B match is an action-conditioned calibrated +result, while the zero-shot/profile-only result selects the wrong TP family. +The Qwen235B surfaces are useful follow-up cases, but decode ground truth and +Frontier's execution semantics must be repaired before a match claim is valid. + +## Qwen30B: calibration is decision-bearing + +Under the aligned maximum-SLO-feasible-throughput objective: + +| reading | Frontier selection | worst real regret | Kendall tau-b | +|---|---|---:|---:| +| profile-only | TP4/MNS32 or TP4/MNS64 | 25.63% | 0.0000 | +| per-TP calibrated | TP2/MNS32 or TP2/MNS64 | 0.76% | 0.9668 | + +The calibrated run multiplies all simulated execution times by a separately +fitted factor for each TP: TP1=0.723481, TP2=0.468089, TP4=0.352137. These +factors were fitted to real end-to-end throughput on the same model/workload +family. The large, TP-dependent residual is therefore evidence that the +profile-only simulator misses action-dependent execution behavior; it is not +evidence that the unmodified simulator predicts the action correctly. + +## Qwen235B real response surfaces + +### Prefill-only + +- 8/8 cells are complete primary searches. +- The point-estimate top set contains all four TP8 cells, so a random cell has a + 50% top-set hit probability. +- Capacity brackets still separate all TP8 cells from all TP4 cells. Thus this + case can test the **TP4 versus TP8** decision, but the current six probes do + not distinguish MNS/MBT within TP8. + +### Decode-only + +- Only 7/8 cells are fully valid; `TP2/DP4/EP8, MNS128, MBT384` is marked + `partial_probe_before_failure` after an engine restart/port failure. +- There are only two point-estimate score levels, and the top set contains 5/8 + cells (62.5% random hit probability). +- More importantly, all eight binary-search capacity brackets overlap the best + observed lower bound. The current data cannot rule out any cell as optimal. + +Consequently, a decode top-set hit on these results is not a match. After the +simulator selects a cell, the cheapest rigorous next step is to refine only +that cell and one competing topology/batching cell until their capacity +intervals separate or remain statistically indistinguishable. + +## Frontier semantic coverage for Qwen235B + +The topology itself is expressible: Frontier exposes attention TP/DP and MoE +TP/EP separately, so TP4/DP2/EP8 and TP2/DP4/EP8 need not be decomposed into +independent scheduling and execution problems. + +The current execution model is not yet aligned, however: + +- no checked-in H20 Qwen235B `linear_op.csv`, `attention.csv`, or `moe.csv`; +- checked-in Qwen235B config is BF16 with max position 40960, whereas the real + run uses FP8 weights, FP8 KV, and max model length 262144; +- real prefill uses FlashAttention and internal BLADNN kernels, while Frontier's + attention profiler exposes only FlashInfer and NO-OP backends; +- real decode starts with a dummy-filled external KV cache through + `DecodeBenchConnector`; Frontier's trace request generator requires positive + prefill tokens and has no equivalent initial-KV trace contract; +- real decode combines DeepEP/NVSHMEM, EAGLE3, and FULL_DECODE_ONLY CUDA graphs; + Frontier explicitly treats speculative decode plus decode CUDA graphs as a + conflicting/diagnostic combination. + +Running dummy profiles or silently substituting FlashInfer/eager decode would +produce a number, but it would not test whether Frontier matches this system. + +## Community-vLLM Qwen235B smoke + +The controlled community stack is feasible: vLLM 0.10.2 loaded the +Qwen3-235B-A22B-FP8 checkpoint on TP4, allocated a BF16 KV cache, and completed +a real request. Representative FP8 linear/MoE, FlashInfer attention, and TP4 +NCCL paths also executed successfully. + +The smoke nevertheless found a profiler/runtime mismatch before any scheduler +model was involved. Frontier's MoE wrapper calls `get_config_dtype_str` without +`use_fp8_w8a8=True`, so it misses vLLM's tuned H20 block-FP8 Triton config and +uses a default. At TP4/EP1 with 16 tokens, a paired five-routing-seed factorial +measured: + +| variant | grouped-GEMM mean | paired delta vs original | +|---|---:|---:| +| original default config + FP16 compute type | 0.3100 ms | 0.00% | +| FP8 config key only | 0.2508 ms | -19.12% | +| BF16 compute type only | 0.3088 ms | -0.40% | +| both aligned | 0.2512 ms | -18.99% | + +The config-key-only routing-seed 95% interval is [-21.26%, -16.98%]; the +compute-type-only interval crosses zero. This localizes the dominant error at +this point to kernel tuning-config selection. The original MoE CSV must not be +used for a formal Frontier ranking until this path and the related shuffling +block-size lookup are aligned and re-profiled. Full evidence and hashes are in +`results/community-qwen235b-smoke/`. + +## Research implication + +The most interesting observation is already visible: Frontier's error is not a +single global time bias. It changes strongly with the execution action (TP in +Qwen30B), and a per-action residual can reverse the selected config. Qwen235B +prefill and decode offer held-out tests of whether those residuals are explained +by measurable execution-state features (kernel family, communication mode, +graph mode, speculative width, and KV initial state) or require case-specific +end-to-end fitting. + +That is a systems question: **which execution-state transitions make operator +profile composition non-invariant across configurations, and what is the +minimum real evidence needed to recover the counterfactual ordering?** diff --git a/runs/frontier-multicase-sufficiency-v0/protocol.md b/runs/frontier-multicase-sufficiency-v0/protocol.md new file mode 100644 index 0000000..7c59622 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/protocol.md @@ -0,0 +1,79 @@ +# Frontier multi-case sufficiency protocol v0 + +## Claim under test + +Can Frontier select a low-regret configuration for the frozen candidate grid +under the objective **maximum SLO-feasible offered request throughput per GPU**? + +The allowed conclusion is scoped to the tested model/runtime/hardware/workload +family. Three matches cannot establish universal simulator fidelity. + +## Cases + +1. Qwen3-30B-A3B mixed chat, H20, TP x MNS grid (existing data). +2. Qwen3-235B-A22B prefill-only, H20, TP x MNS x MBT grid. +3. Qwen3-235B-A22B decode-only, H20, TP/DP/EP x MNS x MBT grid. + +For Qwen235B, keep the real trace window, request filtering, SLO, model build, +runtime flags, and candidate cells recorded by the original trials. Frontier is +evaluated at the same offered-load anchors; no capacity extrapolation beyond +the common anchors is allowed. + +## Two separate simulator readings + +- **Zero-shot/profile-only:** operator profiles and documented hardware/model + inputs are allowed; no end-to-end measurement from the evaluated workload is + used to scale Frontier. +- **Calibrated:** every real measurement used to fit a scalar or residual is + charged and reported. Calibration is fitted on a declared train fixture and + evaluated on a disjoint workload/config holdout. + +These readings must never be merged. A per-TP scale fitted from the same +workload does not count as zero-shot simulator accuracy. + +## Ground-truth gate + +Before a match decision: + +- every result must be a completed primary search without probe failure; +- capacity is an interval from the largest feasible anchor to the next + infeasible anchor, not merely the feasible lower bound; +- refine probes when the possibly-optimal set induced by those intervals is too + broad to distinguish the simulator's selected cell; +- record the random top-set hit rate and the number of informative (non-tied) + pairs. + +## Primary metrics and predeclared decision rule + +Per case, report: + +1. selected-config real regret (point estimate and interval-robust bound); +2. Kendall tau-b and exact pair-direction accuracy with real ties preserved; +3. top-set hit and its random-hit baseline; +4. anchor-level SLO feasibility confusion; +5. profile/calibration cost and all right-censored cells. + +The profile-only simulator is considered sufficient as a **config ranker for +the tested family** only if every case has: + +- worst selected-config real regret <= 5%; +- Kendall tau-b >= 0.8 on a response surface with enough informative pairs; +- no unresolved ground-truth interval that can reverse the selected decision; +- no per-case or per-action end-to-end calibration. + +SLO-oracle sufficiency is a stronger claim and additionally requires low false +feasible/false infeasible rates; ranker success alone does not establish it. + +## Current representational gaps to disclose + +- The checked-in Frontier Qwen235B model config is BF16 and has no H20 + Qwen235B operator profiles, while the real runtime uses FP8 weights and FP8 KV. +- Prefill uses internal vLLM/BLADNN paths and PIECEWISE CUDA graphs. +- Decode uses TP/DP with EP8, DeepEP/NVSHMEM, EAGLE3, DecodeBenchConnector, and + FULL_DECODE_ONLY CUDA graphs. +- Frontier currently rejects the faithful combination of speculative decoding + and decode CUDA-graph modeling unless using a diagnostic opt-in; its public + examples disable decode CUDA graphs for speculative decoding. + +Therefore an as-is Frontier run and an upgraded/semantically aligned run, if +implemented, must be reported separately. diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/model_config.frontier.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/model_config.frontier.json new file mode 100644 index 0000000..6bd9e50 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/model_config.frontier.json @@ -0,0 +1,333 @@ +{ + "architectures": [ + "Qwen3MoeForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 151643, + "decoder_sparse_step": 1, + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 4096, + "initializer_range": 0.02, + "intermediate_size": 12288, + "max_position_embeddings": 40960, + "max_window_layers": 94, + "mlp_only_layers": [], + "model_type": "qwen3_moe", + "moe_intermediate_size": 1536, + "norm_topk_prob": true, + "num_attention_heads": 64, + "num_experts": 128, + "num_experts_per_tok": 8, + "num_hidden_layers": 94, + "num_key_value_heads": 4, + "output_router_logits": false, + "rms_norm_eps": 0.000001, + "rope_scaling": null, + "rope_theta": 1000000.0, + "router_aux_loss_coef": 0.001, + "sliding_window": null, + "tie_word_embeddings": false, + "torch_dtype": "bfloat16", + "transformers_version": "4.51.0", + "use_cache": true, + "use_sliding_window": false, + "vocab_size": 151936, + "quantization_config": { + "activation_scheme": "dynamic", + "modules_to_not_convert": [ + "lm_head", + "model.layers.0.input_layernorm", + "model.layers.0.mlp.gate", + "model.layers.0.post_attention_layernorm", + "model.layers.1.input_layernorm", + "model.layers.1.mlp.gate", + "model.layers.1.post_attention_layernorm", + "model.layers.2.input_layernorm", + "model.layers.2.mlp.gate", + "model.layers.2.post_attention_layernorm", + "model.layers.3.input_layernorm", + "model.layers.3.mlp.gate", + "model.layers.3.post_attention_layernorm", + "model.layers.4.input_layernorm", + "model.layers.4.mlp.gate", + "model.layers.4.post_attention_layernorm", + "model.layers.5.input_layernorm", + "model.layers.5.mlp.gate", + "model.layers.5.post_attention_layernorm", + "model.layers.6.input_layernorm", + "model.layers.6.mlp.gate", + "model.layers.6.post_attention_layernorm", + "model.layers.7.input_layernorm", + "model.layers.7.mlp.gate", + "model.layers.7.post_attention_layernorm", + "model.layers.8.input_layernorm", + "model.layers.8.mlp.gate", + "model.layers.8.post_attention_layernorm", + "model.layers.9.input_layernorm", + "model.layers.9.mlp.gate", + "model.layers.9.post_attention_layernorm", + "model.layers.10.input_layernorm", + "model.layers.10.mlp.gate", + "model.layers.10.post_attention_layernorm", + "model.layers.11.input_layernorm", + "model.layers.11.mlp.gate", + "model.layers.11.post_attention_layernorm", + "model.layers.12.input_layernorm", + "model.layers.12.mlp.gate", + "model.layers.12.post_attention_layernorm", + "model.layers.13.input_layernorm", + "model.layers.13.mlp.gate", + "model.layers.13.post_attention_layernorm", + "model.layers.14.input_layernorm", + "model.layers.14.mlp.gate", + "model.layers.14.post_attention_layernorm", + "model.layers.15.input_layernorm", + "model.layers.15.mlp.gate", + "model.layers.15.post_attention_layernorm", + "model.layers.16.input_layernorm", + "model.layers.16.mlp.gate", + "model.layers.16.post_attention_layernorm", + "model.layers.17.input_layernorm", + "model.layers.17.mlp.gate", + "model.layers.17.post_attention_layernorm", + "model.layers.18.input_layernorm", + "model.layers.18.mlp.gate", + "model.layers.18.post_attention_layernorm", + "model.layers.19.input_layernorm", + "model.layers.19.mlp.gate", + "model.layers.19.post_attention_layernorm", + "model.layers.20.input_layernorm", + "model.layers.20.mlp.gate", + "model.layers.20.post_attention_layernorm", + "model.layers.21.input_layernorm", + "model.layers.21.mlp.gate", + "model.layers.21.post_attention_layernorm", + "model.layers.22.input_layernorm", + "model.layers.22.mlp.gate", + "model.layers.22.post_attention_layernorm", + "model.layers.23.input_layernorm", + "model.layers.23.mlp.gate", + "model.layers.23.post_attention_layernorm", + "model.layers.24.input_layernorm", + "model.layers.24.mlp.gate", + "model.layers.24.post_attention_layernorm", + "model.layers.25.input_layernorm", + "model.layers.25.mlp.gate", + "model.layers.25.post_attention_layernorm", + "model.layers.26.input_layernorm", + "model.layers.26.mlp.gate", + "model.layers.26.post_attention_layernorm", + "model.layers.27.input_layernorm", + "model.layers.27.mlp.gate", + "model.layers.27.post_attention_layernorm", + "model.layers.28.input_layernorm", + "model.layers.28.mlp.gate", + "model.layers.28.post_attention_layernorm", + "model.layers.29.input_layernorm", + "model.layers.29.mlp.gate", + "model.layers.29.post_attention_layernorm", + "model.layers.30.input_layernorm", + "model.layers.30.mlp.gate", + "model.layers.30.post_attention_layernorm", + "model.layers.31.input_layernorm", + "model.layers.31.mlp.gate", + "model.layers.31.post_attention_layernorm", + "model.layers.32.input_layernorm", + "model.layers.32.mlp.gate", + "model.layers.32.post_attention_layernorm", + "model.layers.33.input_layernorm", + "model.layers.33.mlp.gate", + "model.layers.33.post_attention_layernorm", + "model.layers.34.input_layernorm", + "model.layers.34.mlp.gate", + "model.layers.34.post_attention_layernorm", + "model.layers.35.input_layernorm", + "model.layers.35.mlp.gate", + "model.layers.35.post_attention_layernorm", + "model.layers.36.input_layernorm", + "model.layers.36.mlp.gate", + "model.layers.36.post_attention_layernorm", + "model.layers.37.input_layernorm", + "model.layers.37.mlp.gate", + "model.layers.37.post_attention_layernorm", + "model.layers.38.input_layernorm", + "model.layers.38.mlp.gate", + "model.layers.38.post_attention_layernorm", + "model.layers.39.input_layernorm", + "model.layers.39.mlp.gate", + "model.layers.39.post_attention_layernorm", + "model.layers.40.input_layernorm", + "model.layers.40.mlp.gate", + "model.layers.40.post_attention_layernorm", + "model.layers.41.input_layernorm", + "model.layers.41.mlp.gate", + "model.layers.41.post_attention_layernorm", + "model.layers.42.input_layernorm", + "model.layers.42.mlp.gate", + "model.layers.42.post_attention_layernorm", + "model.layers.43.input_layernorm", + "model.layers.43.mlp.gate", + "model.layers.43.post_attention_layernorm", + "model.layers.44.input_layernorm", + "model.layers.44.mlp.gate", + "model.layers.44.post_attention_layernorm", + "model.layers.45.input_layernorm", + "model.layers.45.mlp.gate", + "model.layers.45.post_attention_layernorm", + "model.layers.46.input_layernorm", + "model.layers.46.mlp.gate", + "model.layers.46.post_attention_layernorm", + "model.layers.47.input_layernorm", + "model.layers.47.mlp.gate", + "model.layers.47.post_attention_layernorm", + "model.layers.48.input_layernorm", + "model.layers.48.mlp.gate", + "model.layers.48.post_attention_layernorm", + "model.layers.49.input_layernorm", + "model.layers.49.mlp.gate", + "model.layers.49.post_attention_layernorm", + "model.layers.50.input_layernorm", + "model.layers.50.mlp.gate", + "model.layers.50.post_attention_layernorm", + "model.layers.51.input_layernorm", + "model.layers.51.mlp.gate", + "model.layers.51.post_attention_layernorm", + "model.layers.52.input_layernorm", + "model.layers.52.mlp.gate", + "model.layers.52.post_attention_layernorm", + "model.layers.53.input_layernorm", + "model.layers.53.mlp.gate", + "model.layers.53.post_attention_layernorm", + "model.layers.54.input_layernorm", + "model.layers.54.mlp.gate", + "model.layers.54.post_attention_layernorm", + "model.layers.55.input_layernorm", + "model.layers.55.mlp.gate", + "model.layers.55.post_attention_layernorm", + "model.layers.56.input_layernorm", + "model.layers.56.mlp.gate", + "model.layers.56.post_attention_layernorm", + "model.layers.57.input_layernorm", + "model.layers.57.mlp.gate", + "model.layers.57.post_attention_layernorm", + "model.layers.58.input_layernorm", + "model.layers.58.mlp.gate", + "model.layers.58.post_attention_layernorm", + "model.layers.59.input_layernorm", + "model.layers.59.mlp.gate", + "model.layers.59.post_attention_layernorm", + "model.layers.60.input_layernorm", + "model.layers.60.mlp.gate", + "model.layers.60.post_attention_layernorm", + "model.layers.61.input_layernorm", + "model.layers.61.mlp.gate", + "model.layers.61.post_attention_layernorm", + "model.layers.62.input_layernorm", + "model.layers.62.mlp.gate", + "model.layers.62.post_attention_layernorm", + "model.layers.63.input_layernorm", + "model.layers.63.mlp.gate", + "model.layers.63.post_attention_layernorm", + "model.layers.64.input_layernorm", + "model.layers.64.mlp.gate", + "model.layers.64.post_attention_layernorm", + "model.layers.65.input_layernorm", + "model.layers.65.mlp.gate", + "model.layers.65.post_attention_layernorm", + "model.layers.66.input_layernorm", + "model.layers.66.mlp.gate", + "model.layers.66.post_attention_layernorm", + "model.layers.67.input_layernorm", + "model.layers.67.mlp.gate", + "model.layers.67.post_attention_layernorm", + "model.layers.68.input_layernorm", + "model.layers.68.mlp.gate", + "model.layers.68.post_attention_layernorm", + "model.layers.69.input_layernorm", + "model.layers.69.mlp.gate", + "model.layers.69.post_attention_layernorm", + "model.layers.70.input_layernorm", + "model.layers.70.mlp.gate", + "model.layers.70.post_attention_layernorm", + "model.layers.71.input_layernorm", + "model.layers.71.mlp.gate", + "model.layers.71.post_attention_layernorm", + "model.layers.72.input_layernorm", + "model.layers.72.mlp.gate", + "model.layers.72.post_attention_layernorm", + "model.layers.73.input_layernorm", + "model.layers.73.mlp.gate", + "model.layers.73.post_attention_layernorm", + "model.layers.74.input_layernorm", + "model.layers.74.mlp.gate", + "model.layers.74.post_attention_layernorm", + "model.layers.75.input_layernorm", + "model.layers.75.mlp.gate", + "model.layers.75.post_attention_layernorm", + "model.layers.76.input_layernorm", + "model.layers.76.mlp.gate", + "model.layers.76.post_attention_layernorm", + "model.layers.77.input_layernorm", + "model.layers.77.mlp.gate", + "model.layers.77.post_attention_layernorm", + "model.layers.78.input_layernorm", + "model.layers.78.mlp.gate", + "model.layers.78.post_attention_layernorm", + "model.layers.79.input_layernorm", + "model.layers.79.mlp.gate", + "model.layers.79.post_attention_layernorm", + "model.layers.80.input_layernorm", + "model.layers.80.mlp.gate", + "model.layers.80.post_attention_layernorm", + "model.layers.81.input_layernorm", + "model.layers.81.mlp.gate", + "model.layers.81.post_attention_layernorm", + "model.layers.82.input_layernorm", + "model.layers.82.mlp.gate", + "model.layers.82.post_attention_layernorm", + "model.layers.83.input_layernorm", + "model.layers.83.mlp.gate", + "model.layers.83.post_attention_layernorm", + "model.layers.84.input_layernorm", + "model.layers.84.mlp.gate", + "model.layers.84.post_attention_layernorm", + "model.layers.85.input_layernorm", + "model.layers.85.mlp.gate", + "model.layers.85.post_attention_layernorm", + "model.layers.86.input_layernorm", + "model.layers.86.mlp.gate", + "model.layers.86.post_attention_layernorm", + "model.layers.87.input_layernorm", + "model.layers.87.mlp.gate", + "model.layers.87.post_attention_layernorm", + "model.layers.88.input_layernorm", + "model.layers.88.mlp.gate", + "model.layers.88.post_attention_layernorm", + "model.layers.89.input_layernorm", + "model.layers.89.mlp.gate", + "model.layers.89.post_attention_layernorm", + "model.layers.90.input_layernorm", + "model.layers.90.mlp.gate", + "model.layers.90.post_attention_layernorm", + "model.layers.91.input_layernorm", + "model.layers.91.mlp.gate", + "model.layers.91.post_attention_layernorm", + "model.layers.92.input_layernorm", + "model.layers.92.mlp.gate", + "model.layers.92.post_attention_layernorm", + "model.layers.93.input_layernorm", + "model.layers.93.mlp.gate", + "model.layers.93.post_attention_layernorm" + ], + "fmt": "e4m3", + "quant_method": "fp8", + "weight_block_size": [ + 128, + 128 + ], + "is_checkpoint_fp8_serialized": true + } +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/aligned/compute/h20/Qwen3-235B-A22B-FP8/moe.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/aligned/compute/h20/Qwen3-235B-A22B-FP8/moe.csv new file mode 100644 index 0000000..5aafbc6 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/aligned/compute/h20/Qwen3-235B-A22B-FP8/moe.csv @@ -0,0 +1,6 @@ +time_stats.moe_gating_linear.min,time_stats.moe_gating_linear.max,time_stats.moe_gating_linear.mean,time_stats.moe_gating_linear.median,time_stats.moe_gating_linear.std,time_stats.moe_gating_routing_topk.min,time_stats.moe_gating_routing_topk.max,time_stats.moe_gating_routing_topk.mean,time_stats.moe_gating_routing_topk.median,time_stats.moe_gating_routing_topk.std,time_stats.moe_shuffling.min,time_stats.moe_shuffling.max,time_stats.moe_shuffling.mean,time_stats.moe_shuffling.median,time_stats.moe_shuffling.std,time_stats.moe_grouped_gemm.min,time_stats.moe_grouped_gemm.max,time_stats.moe_grouped_gemm.mean,time_stats.moe_grouped_gemm.median,time_stats.moe_grouped_gemm.std,num_tokens,num_experts,num_experts_per_device,expert_parallel_size,routing_runtime_path,routing_assignment_policy,routing_weight_policy,routing_uses_router_logits,gating_runtime_context,gating_runtime_context_impl,router_topk,hidden_dim,expert_hidden_dim,use_gated,num_tensor_parallel_workers,total_routed_tokens,model_expansion_ratio,tokens_per_expert_avg,tokens_to_experts_ratio,expert_utilization,min_load_ratio,load_imbalance_cv,max_load_ratio,load_entropy,load_gini_coefficient,load_distribution,seed,moe_grouped_gemm_backend,measurement_type,profiling_precision,model_arch,quant_signature +0.03094400092959404,0.052000001072883606,0.03423200035467744,0.032816000282764435,0.004586225105504425,0.05225599929690361,0.08137600123882294,0.06316960025578737,0.0586559996008873,0.009046516570964667,0.025087999179959297,0.05766399949789047,0.030939200054854156,0.028768000192940235,0.006844666670168347,0.23388800024986267,0.31091201305389404,0.24544477462768555,0.23836800456047058,0.018666831776499748,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.609375,0.0,1.0231690964840563,4.0,6.122626857503489,0.5433349609375,uniform,0,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030368000268936157,0.05008000135421753,0.03488799966871738,0.03270399942994118,0.004964435488742052,0.05142400041222572,0.07401599735021591,0.05652640014886856,0.054847998544573784,0.005118024227402908,0.02534399926662445,0.0424639992415905,0.02885119989514351,0.028447999618947506,0.004027125677475938,0.24774399399757385,0.26895999908447266,0.25432640314102173,0.25200000405311584,0.00582256680354476,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.6171875,0.0,0.9682458365518543,4.0,6.171569533299451,0.521240234375,uniform,1,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.0307839997112751,0.04447999969124794,0.03474240032956004,0.03252799995243549,0.00435773849976965,0.051231998950242996,0.07468800246715546,0.057651200145483014,0.0561280008405447,0.006050027441177822,0.02579200081527233,0.033504001796245575,0.02867360021919012,0.02796800062060356,0.00238178652420847,0.2533760070800781,0.2958720028400421,0.2585648000240326,0.25679999589920044,0.009045161306858063,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.640625,0.0,0.9842509842514764,4.0,6.193092091810443,0.5164794921875,uniform,2,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030592000111937523,0.04108799993991852,0.03223840007558465,0.031279999762773514,0.002451635813407811,0.05177599936723709,0.077504001557827,0.05718399975448847,0.0533440001308918,0.00711033940595454,0.02457600086927414,0.051552001386880875,0.03192960014566779,0.028431999497115612,0.007824328074264588,0.2531839907169342,0.28329598903656006,0.26060959696769714,0.2584640085697174,0.006803486030548811,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.671875,0.0,0.9354143466934853,5.0,6.268369216857352,0.48486328125,uniform,3,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030527999624609947,0.03500799834728241,0.03184479987248778,0.03139200061559677,0.001123070738289876,0.05104000121355057,0.07103999704122543,0.05895199999213219,0.05676800012588501,0.0068221931474717916,0.024383999407291412,0.04211200028657913,0.02812959998846054,0.026959999464452267,0.003887262202895282,0.2314240038394928,0.2559039890766144,0.23696160316467285,0.23375999927520752,0.006377317477017641,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.59375,0.0,1.1319231422671772,6.0,6.039891775809659,0.5689697265625,uniform,4,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/aligned/compute/h20/moe_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/aligned/compute/h20/moe_config.yaml new file mode 100644 index 0000000..f01e8d4 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/aligned/compute/h20/moe_config.yaml @@ -0,0 +1,27 @@ +block_shape: null +device: h20 +disable_ray: true +disable_replicated: false +enable_load_imbalance: true +expert_parallel_sizes: +- 1 +extra_num_tokens: null +gating_runtime_context: prefill_hot +load_distributions: +- uniform +max_tokens: 16 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_samples_per_distribution: 5 +num_tensor_parallel_workers: +- 4 +num_tokens_list: +- 16 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/aligned +per_channel_quant: false +precision: null +profile_method: cuda_event +routing_runtime_path: standard_fused_topk +skip_confirmation: true +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/compute_type/compute/h20/Qwen3-235B-A22B-FP8/moe.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/compute_type/compute/h20/Qwen3-235B-A22B-FP8/moe.csv new file mode 100644 index 0000000..2b58135 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/compute_type/compute/h20/Qwen3-235B-A22B-FP8/moe.csv @@ -0,0 +1,6 @@ +time_stats.moe_gating_linear.min,time_stats.moe_gating_linear.max,time_stats.moe_gating_linear.mean,time_stats.moe_gating_linear.median,time_stats.moe_gating_linear.std,time_stats.moe_gating_routing_topk.min,time_stats.moe_gating_routing_topk.max,time_stats.moe_gating_routing_topk.mean,time_stats.moe_gating_routing_topk.median,time_stats.moe_gating_routing_topk.std,time_stats.moe_shuffling.min,time_stats.moe_shuffling.max,time_stats.moe_shuffling.mean,time_stats.moe_shuffling.median,time_stats.moe_shuffling.std,time_stats.moe_grouped_gemm.min,time_stats.moe_grouped_gemm.max,time_stats.moe_grouped_gemm.mean,time_stats.moe_grouped_gemm.median,time_stats.moe_grouped_gemm.std,num_tokens,num_experts,num_experts_per_device,expert_parallel_size,routing_runtime_path,routing_assignment_policy,routing_weight_policy,routing_uses_router_logits,gating_runtime_context,gating_runtime_context_impl,router_topk,hidden_dim,expert_hidden_dim,use_gated,num_tensor_parallel_workers,total_routed_tokens,model_expansion_ratio,tokens_per_expert_avg,tokens_to_experts_ratio,expert_utilization,min_load_ratio,load_imbalance_cv,max_load_ratio,load_entropy,load_gini_coefficient,load_distribution,seed,moe_grouped_gemm_backend,measurement_type,profiling_precision,model_arch,quant_signature +0.03232000023126602,0.07411199808120728,0.040055999718606475,0.03792000003159046,0.010027335937042725,0.05955199897289276,0.09151999652385712,0.06805919948965311,0.06542399898171425,0.009156033646696494,0.02735999971628189,0.04198399931192398,0.031628800183534624,0.03081599995493889,0.0038650835976484594,0.29440000653266907,0.37036800384521484,0.3045775890350342,0.3012160062789917,0.01622786745429039,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.609375,0.0,1.0231690964840563,4.0,6.122626857503489,0.5433349609375,uniform,0,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.03190400078892708,0.0639680027961731,0.036531200259923936,0.03387199901044369,0.007307958553439168,0.05407999828457832,0.08684799820184708,0.0631104001775384,0.059248000383377075,0.008437822166776614,0.02537599951028824,0.045343998819589615,0.029841599892824887,0.028095999732613564,0.004578510902843549,0.29868799448013306,0.3216319978237152,0.30579519271850586,0.30294400453567505,0.006667278707027435,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.6171875,0.0,0.9682458365518543,4.0,6.171569533299451,0.521240234375,uniform,1,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.031199999153614044,0.06745599955320358,0.03718719966709614,0.03391999937593937,0.008084069620200455,0.05353600159287453,0.06828799843788147,0.059427200257778166,0.05902400054037571,0.0036973349098869714,0.026688000187277794,0.03788800165057182,0.030459199845790864,0.02908799983561039,0.0032231049972104124,0.30588799715042114,0.3216319978237152,0.3109855651855469,0.31036800146102905,0.004134960938245058,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.640625,0.0,0.9842509842514764,4.0,6.193092091810443,0.5164794921875,uniform,2,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.031488001346588135,0.04790399968624115,0.03488959986716509,0.033615998923778534,0.003995903359891917,0.05503999814391136,0.08563199639320374,0.06150399968028068,0.057312000542879105,0.007950341155323642,0.026335999369621277,0.04598399996757507,0.030313600040972232,0.02792000025510788,0.005356499011224523,0.31516799330711365,0.32972800731658936,0.3209056258201599,0.3196159899234772,0.004338567610830069,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.671875,0.0,0.9354143466934853,5.0,6.268369216857352,0.48486328125,uniform,3,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030751999467611313,0.044895999133586884,0.03544640000909567,0.03302400000393391,0.00457756723742277,0.05331199988722801,0.07606399804353714,0.05961279980838299,0.05753600038588047,0.006503363955077464,0.02630399912595749,0.04368000105023384,0.03112160013988614,0.028815999627113342,0.005258638278682459,0.2905920147895813,0.3538239896297455,0.3015664219856262,0.29631999135017395,0.014866933226585388,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.59375,0.0,1.1319231422671772,6.0,6.039891775809659,0.5689697265625,uniform,4,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/compute_type/compute/h20/moe_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/compute_type/compute/h20/moe_config.yaml new file mode 100644 index 0000000..3584f34 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/compute_type/compute/h20/moe_config.yaml @@ -0,0 +1,27 @@ +block_shape: null +device: h20 +disable_ray: true +disable_replicated: false +enable_load_imbalance: true +expert_parallel_sizes: +- 1 +extra_num_tokens: null +gating_runtime_context: prefill_hot +load_distributions: +- uniform +max_tokens: 16 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_samples_per_distribution: 5 +num_tensor_parallel_workers: +- 4 +num_tokens_list: +- 16 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/compute_type +per_channel_quant: false +precision: null +profile_method: cuda_event +routing_runtime_path: standard_fused_topk +skip_confirmation: true +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/config_key/compute/h20/Qwen3-235B-A22B-FP8/moe.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/config_key/compute/h20/Qwen3-235B-A22B-FP8/moe.csv new file mode 100644 index 0000000..462a99c --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/config_key/compute/h20/Qwen3-235B-A22B-FP8/moe.csv @@ -0,0 +1,6 @@ +time_stats.moe_gating_linear.min,time_stats.moe_gating_linear.max,time_stats.moe_gating_linear.mean,time_stats.moe_gating_linear.median,time_stats.moe_gating_linear.std,time_stats.moe_gating_routing_topk.min,time_stats.moe_gating_routing_topk.max,time_stats.moe_gating_routing_topk.mean,time_stats.moe_gating_routing_topk.median,time_stats.moe_gating_routing_topk.std,time_stats.moe_shuffling.min,time_stats.moe_shuffling.max,time_stats.moe_shuffling.mean,time_stats.moe_shuffling.median,time_stats.moe_shuffling.std,time_stats.moe_grouped_gemm.min,time_stats.moe_grouped_gemm.max,time_stats.moe_grouped_gemm.mean,time_stats.moe_grouped_gemm.median,time_stats.moe_grouped_gemm.std,num_tokens,num_experts,num_experts_per_device,expert_parallel_size,routing_runtime_path,routing_assignment_policy,routing_weight_policy,routing_uses_router_logits,gating_runtime_context,gating_runtime_context_impl,router_topk,hidden_dim,expert_hidden_dim,use_gated,num_tensor_parallel_workers,total_routed_tokens,model_expansion_ratio,tokens_per_expert_avg,tokens_to_experts_ratio,expert_utilization,min_load_ratio,load_imbalance_cv,max_load_ratio,load_entropy,load_gini_coefficient,load_distribution,seed,moe_grouped_gemm_backend,measurement_type,profiling_precision,model_arch,quant_signature +0.029888000339269638,0.05558399856090546,0.036182400118559596,0.03598400019109249,0.00590375348379892,0.05135999992489815,0.07407999783754349,0.05939359981566668,0.05599999986588955,0.007274243962538696,0.024831999093294144,0.060575999319553375,0.03204159988090396,0.029280000366270542,0.008986441399241034,0.23401600122451782,0.2914240062236786,0.24253761768341064,0.23852799832820892,0.012656064704060555,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.609375,0.0,1.0231690964840563,4.0,6.122626857503489,0.5433349609375,uniform,0,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.0306560005992651,0.04927999898791313,0.0345616003498435,0.032096000388264656,0.005272357401818311,0.0514880008995533,0.08454400300979614,0.05950720049440861,0.055904000997543335,0.008599599958802731,0.024639999493956566,0.04726399853825569,0.029726399946957825,0.028032000176608562,0.005891950556036845,0.24633599817752838,0.276095986366272,0.2544096112251282,0.25091201066970825,0.006771203130483627,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.6171875,0.0,0.9682458365518543,4.0,6.171569533299451,0.521240234375,uniform,1,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030208000913262367,0.062144000083208084,0.03510080017149449,0.03203200176358223,0.007286512102806094,0.050592001527547836,0.07017599791288376,0.05696159955114126,0.054735999554395676,0.006055532934843585,0.0244159996509552,0.04364800080657005,0.02875520009547472,0.02700799982994795,0.00429941663275523,0.252703994512558,0.27452799677848816,0.25873440504074097,0.2567040026187897,0.006180537864565849,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.640625,0.0,0.9842509842514764,4.0,6.193092091810443,0.5164794921875,uniform,2,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.028863999992609024,0.04185599833726883,0.03244960019364953,0.03129600081592798,0.00309197601199972,0.05049600079655647,0.07241600006818771,0.05591519977897406,0.05273599922657013,0.006825017255220366,0.024000000208616257,0.04150399938225746,0.027609600126743315,0.02619200013577938,0.0041488046270812895,0.25491198897361755,0.27008000016212463,0.2593088150024414,0.25811201333999634,0.003932233899831772,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.671875,0.0,0.9354143466934853,5.0,6.268369216857352,0.48486328125,uniform,3,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030400000512599945,0.039135999977588654,0.03235360030084848,0.031888000667095184,0.002007792126883597,0.050944000482559204,0.06815999746322632,0.054641599953174594,0.053279999643564224,0.003969876178544029,0.024064000695943832,0.03145600110292435,0.027134399861097336,0.026559999212622643,0.0021875101737656014,0.23164799809455872,0.2622720003128052,0.2388928234577179,0.23686400055885315,0.007063580676913261,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.59375,0.0,1.1319231422671772,6.0,6.039891775809659,0.5689697265625,uniform,4,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/config_key/compute/h20/moe_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/config_key/compute/h20/moe_config.yaml new file mode 100644 index 0000000..8d60f31 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/config_key/compute/h20/moe_config.yaml @@ -0,0 +1,27 @@ +block_shape: null +device: h20 +disable_ray: true +disable_replicated: false +enable_load_imbalance: true +expert_parallel_sizes: +- 1 +extra_num_tokens: null +gating_runtime_context: prefill_hot +load_distributions: +- uniform +max_tokens: 16 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_samples_per_distribution: 5 +num_tensor_parallel_workers: +- 4 +num_tokens_list: +- 16 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/config_key +per_channel_quant: false +precision: null +profile_method: cuda_event +routing_runtime_path: standard_fused_topk +skip_confirmation: true +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/original/compute/h20/Qwen3-235B-A22B-FP8/moe.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/original/compute/h20/Qwen3-235B-A22B-FP8/moe.csv new file mode 100644 index 0000000..941211d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/original/compute/h20/Qwen3-235B-A22B-FP8/moe.csv @@ -0,0 +1,6 @@ +time_stats.moe_gating_linear.min,time_stats.moe_gating_linear.max,time_stats.moe_gating_linear.mean,time_stats.moe_gating_linear.median,time_stats.moe_gating_linear.std,time_stats.moe_gating_routing_topk.min,time_stats.moe_gating_routing_topk.max,time_stats.moe_gating_routing_topk.mean,time_stats.moe_gating_routing_topk.median,time_stats.moe_gating_routing_topk.std,time_stats.moe_shuffling.min,time_stats.moe_shuffling.max,time_stats.moe_shuffling.mean,time_stats.moe_shuffling.median,time_stats.moe_shuffling.std,time_stats.moe_grouped_gemm.min,time_stats.moe_grouped_gemm.max,time_stats.moe_grouped_gemm.mean,time_stats.moe_grouped_gemm.median,time_stats.moe_grouped_gemm.std,num_tokens,num_experts,num_experts_per_device,expert_parallel_size,routing_runtime_path,routing_assignment_policy,routing_weight_policy,routing_uses_router_logits,gating_runtime_context,gating_runtime_context_impl,router_topk,hidden_dim,expert_hidden_dim,use_gated,num_tensor_parallel_workers,total_routed_tokens,model_expansion_ratio,tokens_per_expert_avg,tokens_to_experts_ratio,expert_utilization,min_load_ratio,load_imbalance_cv,max_load_ratio,load_entropy,load_gini_coefficient,load_distribution,seed,moe_grouped_gemm_backend,measurement_type,profiling_precision,model_arch,quant_signature +0.03094400092959404,0.05427199974656105,0.03751839986070991,0.03566399961709976,0.0063355673062837495,0.052960000932216644,0.10150399804115295,0.06679840013384819,0.06265599839389324,0.012010412593353396,0.025087999179959297,0.0607680007815361,0.03164320001378655,0.028655999340116978,0.008879480144565048,0.29548799991607666,0.3161599934101105,0.30511200428009033,0.30371201038360596,0.006655826233327389,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.609375,0.0,1.0231690964840563,4.0,6.122626857503489,0.5433349609375,uniform,0,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030368000268936157,0.045184001326560974,0.03435200024396181,0.03299199976027012,0.003866832846889978,0.053247999399900436,0.10054399818181992,0.061161600053310394,0.05702400021255016,0.010951609505981603,0.025248000398278236,0.039872001856565475,0.029195200372487306,0.027520000003278255,0.003716525371772022,0.29631999135017395,0.3163839876651764,0.30516156554222107,0.30246400833129883,0.0062532913871109486,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.6171875,0.0,0.9682458365518543,4.0,6.171569533299451,0.521240234375,uniform,1,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030688000842928886,0.04044799879193306,0.03336800048127771,0.032368000596761703,0.002819613112867307,0.053568001836538315,0.06889600306749344,0.05800320040434599,0.055616000667214394,0.00500911399891226,0.025631999596953392,0.040991999208927155,0.028961599990725517,0.027951999567449093,0.003321893930324548,0.30828800797462463,0.3516480028629303,0.3159376084804535,0.3127039968967438,0.009800842963159084,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.640625,0.0,0.9842509842514764,4.0,6.193092091810443,0.5164794921875,uniform,2,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.031007999554276466,0.044319998472929,0.03418559962883592,0.03271999955177307,0.0038270996330861703,0.0541439987719059,0.07507199794054031,0.06345439981669188,0.06393599882721901,0.006346258200489106,0.025407999753952026,0.03494400158524513,0.028180800192058087,0.027375999838113785,0.0024874374625372497,0.3179520070552826,0.331712007522583,0.3228943943977356,0.32150399684906006,0.004389750771224499,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.671875,0.0,0.9354143466934853,5.0,6.268369216857352,0.48486328125,uniform,3,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 +0.030912000685930252,0.04745600000023842,0.03517600009217858,0.03243200108408928,0.005035846835878437,0.053279999643564224,0.09200000017881393,0.06018720027059317,0.05721599981188774,0.008845081624923161,0.025151999667286873,0.04851200059056282,0.0323488000780344,0.02820800058543682,0.0077821073314592575,0.2917119860649109,0.3189440071582794,0.301118403673172,0.29817599058151245,0.008174914866685867,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.59375,0.0,1.1319231422671772,6.0,6.039891775809659,0.5689697265625,uniform,4,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/original/compute/h20/moe_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/original/compute/h20/moe_config.yaml new file mode 100644 index 0000000..2d22b0b --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles-moe-factorial-r5/original/compute/h20/moe_config.yaml @@ -0,0 +1,27 @@ +block_shape: null +device: h20 +disable_ray: true +disable_replicated: false +enable_load_imbalance: true +expert_parallel_sizes: +- 1 +extra_num_tokens: null +gating_runtime_context: prefill_hot +load_distributions: +- uniform +max_tokens: 16 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_samples_per_distribution: 5 +num_tensor_parallel_workers: +- 4 +num_tokens_list: +- 16 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/original +per_channel_quant: false +precision: null +profile_method: cuda_event +routing_runtime_path: standard_fused_topk +skip_confirmation: true +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/attention.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/attention.csv new file mode 100644 index 0000000..4d75de3 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/attention.csv @@ -0,0 +1,5 @@ +time_stats.attn_input_reshape.min,time_stats.attn_input_reshape.max,time_stats.attn_input_reshape.mean,time_stats.attn_input_reshape.median,time_stats.attn_input_reshape.std,time_stats.attn_kv_cache_save.min,time_stats.attn_kv_cache_save.max,time_stats.attn_kv_cache_save.mean,time_stats.attn_kv_cache_save.median,time_stats.attn_kv_cache_save.std,time_stats.attn_prefill.min,time_stats.attn_prefill.max,time_stats.attn_prefill.mean,time_stats.attn_prefill.median,time_stats.attn_prefill.std,time_stats.attn_decode.min,time_stats.attn_decode.max,time_stats.attn_decode.mean,time_stats.attn_decode.median,time_stats.attn_decode.std,time_stats.attn_output_reshape.min,time_stats.attn_output_reshape.max,time_stats.attn_output_reshape.mean,time_stats.attn_output_reshape.median,time_stats.attn_output_reshape.std,n_embd,n_q_head,n_kv_head,block_size,num_tensor_parallel_workers,max_model_len,batch_size,prefill_chunk_size,kv_cache_size,is_prefill,attention_backend,is_mixed_batch,mode,seq_lens,total_tokens,max_seq_len,min_seq_len,avg_seq_len,equal_seq_len,seq_len_variance,seq_len_std,seq_len_cv,is_chunked_prefill_sample,chunk_start_token,chunk_end_token,total_prefill_tokens,profiling_precision,model_arch,quant_signature,measurement_type +0.010463999584317207,0.04681599885225296,0.018502399697899817,0.01104000024497509,0.014205381674858167,0.021247999742627144,0.033055998384952545,0.025439999625086786,0.024639999493956566,0.004202660707027564,0.05206400156021118,0.09040000289678574,0.0665344014763832,0.06451199948787689,0.013110543934264527,0.007327999919652939,0.011680000461637974,0.008480000123381615,0.0077760000713169575,0.0016149067494504018,0.009472000412642956,0.011103999800980091,0.010118400119245053,0.009600000455975533,0.0007061491557578713,4096,64,4,16,4,40960,1,128,0,True,FLASHINFER,False,even,[128],128,128,128,128.0,128,0.0,0.0,0.0,False,0,128,128,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT +0.010751999914646149,0.012896000407636166,0.011795200034976005,0.011359999887645245,0.0009191526000700493,0.020191999152302742,0.02755199931561947,0.02319999970495701,0.021247999742627144,0.002944486774412461,0.049215998500585556,0.06700800359249115,0.05813760012388229,0.058240000158548355,0.006895964777123649,0.007519999984651804,0.009727999567985535,0.008140799775719642,0.007807999849319458,0.0008127242879711131,0.009472000412642956,0.025248000398278236,0.01303040012717247,0.010015999898314476,0.0061233119538994085,4096,64,4,16,4,40960,1,32,0,True,FLASHINFER,False,even,[32],32,32,32,32.0,32,0.0,0.0,0.0,False,0,32,32,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT +0.010239999741315842,0.011008000001311302,0.010515199974179268,0.010432000271975994,0.0002635681936991548,0.02112000063061714,0.03843199834227562,0.025894399732351303,0.023231999948620796,0.006378760808481684,0.05417599901556969,0.06438399851322174,0.05709439888596535,0.055615998804569244,0.003697870830170538,0.007135999854654074,0.00774399982765317,0.007500800024718046,0.007552000228315592,0.0002009640969704077,0.009279999881982803,0.012736000120639801,0.01031040009111166,0.009568000212311745,0.0012940667684018582,4096,64,4,16,4,40960,1,64,0,True,FLASHINFER,False,even,[64],64,64,64,64.0,64,0.0,0.0,0.0,False,0,64,64,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT +0.010367999784648418,0.02332800067961216,0.013459200039505959,0.010847999714314938,0.004984552697426283,0.02054399996995926,0.0261439997702837,0.02255360037088394,0.022112000733613968,0.001928344367854761,0.04851200059056282,0.07968000322580338,0.058873600512743,0.056352000683546066,0.010813603870069636,0.00723200011998415,0.008352000266313553,0.007686400134116411,0.007648000027984381,0.00039524862182609345,0.008960000239312649,0.011103999800980091,0.009881599992513656,0.00940799992531538,0.0009090251198168286,4096,64,4,16,4,40960,1,96,0,True,FLASHINFER,False,even,[96],96,96,96,96.0,96,0.0,0.0,0.0,False,0,96,96,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/attention_combined.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/attention_combined.csv new file mode 100644 index 0000000..6fedbb1 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/attention_combined.csv @@ -0,0 +1,5 @@ +time_stats.attn_input_reshape.min,time_stats.attn_input_reshape.max,time_stats.attn_input_reshape.mean,time_stats.attn_input_reshape.median,time_stats.attn_input_reshape.std,time_stats.attn_kv_cache_save.min,time_stats.attn_kv_cache_save.max,time_stats.attn_kv_cache_save.mean,time_stats.attn_kv_cache_save.median,time_stats.attn_kv_cache_save.std,time_stats.attn_prefill.min,time_stats.attn_prefill.max,time_stats.attn_prefill.mean,time_stats.attn_prefill.median,time_stats.attn_prefill.std,time_stats.attn_decode.min,time_stats.attn_decode.max,time_stats.attn_decode.mean,time_stats.attn_decode.median,time_stats.attn_decode.std,time_stats.attn_output_reshape.min,time_stats.attn_output_reshape.max,time_stats.attn_output_reshape.mean,time_stats.attn_output_reshape.median,time_stats.attn_output_reshape.std,n_embd,n_q_head,n_kv_head,block_size,num_tensor_parallel_workers,max_model_len,batch_size,prefill_chunk_size,kv_cache_size,is_prefill,attention_backend,is_mixed_batch,mode,seq_lens,total_tokens,max_seq_len,min_seq_len,avg_seq_len,equal_seq_len,seq_len_variance,seq_len_std,seq_len_cv,is_chunked_prefill_sample,chunk_start_token,chunk_end_token,total_prefill_tokens,profiling_precision,model_arch,quant_signature,measurement_type,is_true_mixed_batch +0.010463999584317207,0.04681599885225296,0.018502399697899817,0.01104000024497509,0.014205381674858167,0.021247999742627144,0.033055998384952545,0.025439999625086786,0.024639999493956566,0.004202660707027564,0.05206400156021118,0.09040000289678574,0.0665344014763832,0.06451199948787689,0.013110543934264527,0.007327999919652939,0.011680000461637974,0.008480000123381615,0.0077760000713169575,0.0016149067494504018,0.009472000412642956,0.011103999800980091,0.010118400119245053,0.009600000455975533,0.0007061491557578713,4096,64,4,16,4,40960,1,128,0,True,FLASHINFER,False,even,[128],128,128,128,128.0,128,0.0,0.0,0.0,False,0,128,128,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT,False +0.010751999914646149,0.012896000407636166,0.011795200034976005,0.011359999887645245,0.0009191526000700493,0.020191999152302742,0.02755199931561947,0.02319999970495701,0.021247999742627144,0.002944486774412461,0.049215998500585556,0.06700800359249115,0.05813760012388229,0.058240000158548355,0.006895964777123649,0.007519999984651804,0.009727999567985535,0.008140799775719642,0.007807999849319458,0.0008127242879711131,0.009472000412642956,0.025248000398278236,0.01303040012717247,0.010015999898314476,0.0061233119538994085,4096,64,4,16,4,40960,1,32,0,True,FLASHINFER,False,even,[32],32,32,32,32.0,32,0.0,0.0,0.0,False,0,32,32,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT,False +0.010239999741315842,0.011008000001311302,0.010515199974179268,0.010432000271975994,0.0002635681936991548,0.02112000063061714,0.03843199834227562,0.025894399732351303,0.023231999948620796,0.006378760808481684,0.05417599901556969,0.06438399851322174,0.05709439888596535,0.055615998804569244,0.003697870830170538,0.007135999854654074,0.00774399982765317,0.007500800024718046,0.007552000228315592,0.0002009640969704077,0.009279999881982803,0.012736000120639801,0.01031040009111166,0.009568000212311745,0.0012940667684018582,4096,64,4,16,4,40960,1,64,0,True,FLASHINFER,False,even,[64],64,64,64,64.0,64,0.0,0.0,0.0,False,0,64,64,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT,False +0.010367999784648418,0.02332800067961216,0.013459200039505959,0.010847999714314938,0.004984552697426283,0.02054399996995926,0.0261439997702837,0.02255360037088394,0.022112000733613968,0.001928344367854761,0.04851200059056282,0.07968000322580338,0.058873600512743,0.056352000683546066,0.010813603870069636,0.00723200011998415,0.008352000266313553,0.007686400134116411,0.007648000027984381,0.00039524862182609345,0.008960000239312649,0.011103999800980091,0.009881599992513656,0.00940799992531538,0.0009090251198168286,4096,64,4,16,4,40960,1,96,0,True,FLASHINFER,False,even,[96],96,96,96,96.0,96,0.0,0.0,0.0,False,0,96,96,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128,CUDA_EVENT,False diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/linear_op.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/linear_op.csv new file mode 100644 index 0000000..ba5b628 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/linear_op.csv @@ -0,0 +1,3 @@ +time_stats.attn_pre_proj.min,time_stats.attn_pre_proj.max,time_stats.attn_pre_proj.mean,time_stats.attn_pre_proj.median,time_stats.attn_pre_proj.std,time_stats.attn_rope.min,time_stats.attn_rope.max,time_stats.attn_rope.mean,time_stats.attn_rope.median,time_stats.attn_rope.std,time_stats.attn_post_proj.min,time_stats.attn_post_proj.max,time_stats.attn_post_proj.mean,time_stats.attn_post_proj.median,time_stats.attn_post_proj.std,time_stats.emb.min,time_stats.emb.max,time_stats.emb.mean,time_stats.emb.median,time_stats.emb.std,time_stats.input_layernorm.min,time_stats.input_layernorm.max,time_stats.input_layernorm.mean,time_stats.input_layernorm.median,time_stats.input_layernorm.std,time_stats.post_attention_layernorm.min,time_stats.post_attention_layernorm.max,time_stats.post_attention_layernorm.mean,time_stats.post_attention_layernorm.median,time_stats.post_attention_layernorm.std,n_head,n_kv_head,n_embd,n_expanded_embd,vocab_size,use_gated_mlp,use_qk_norm,attn_output_gate,num_tokens,num_tensor_parallel_workers,padded_n_embd,padded_n_expanded_embd,model_arch,is_step2_mini,share_expert_dim,share_q_dim,measurement_type,profiling_precision,quant_signature +0.19225600361824036,0.26678401231765747,0.22318400144577027,0.22147200256586075,0.020290217906394733,0.024639999493956566,0.04320000112056732,0.028563199937343596,0.026335999369621277,0.006067654243700026,0.09071999788284302,0.12992000579833984,0.11053120009601117,0.11033599823713303,0.011489030217015594,,,,,,,,,,,,,,,,64,4,4096,1536,151936,True,True,False,16,4,4096,1536,generic,False,,,CUDA_EVENT,BF16,method=fp8|act=dynamic|serialized=True|block=128x128 +,,,,,,,,,,,,,,,0.05990400165319443,0.1345279961824417,0.08581680012866855,0.0899839997291565,0.020920650895404905,0.020160000771284103,0.0360959991812706,0.02369120018556714,0.021824000403285027,0.00439515404502141,0.01833599992096424,0.02425600029528141,0.020108799915760756,0.019407999701797962,0.001409577392533308,64,4,4096,1536,151936,True,True,False,16,1,4096,1536,generic,False,,,CUDA_EVENT,BF16,method=fp8|act=dynamic|serialized=True|block=128x128 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/moe.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/moe.csv new file mode 100644 index 0000000..169a754 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/Qwen3-235B-A22B-FP8/moe.csv @@ -0,0 +1,2 @@ +time_stats.moe_gating_linear.min,time_stats.moe_gating_linear.max,time_stats.moe_gating_linear.mean,time_stats.moe_gating_linear.median,time_stats.moe_gating_linear.std,time_stats.moe_gating_routing_topk.min,time_stats.moe_gating_routing_topk.max,time_stats.moe_gating_routing_topk.mean,time_stats.moe_gating_routing_topk.median,time_stats.moe_gating_routing_topk.std,time_stats.moe_shuffling.min,time_stats.moe_shuffling.max,time_stats.moe_shuffling.mean,time_stats.moe_shuffling.median,time_stats.moe_shuffling.std,time_stats.moe_grouped_gemm.min,time_stats.moe_grouped_gemm.max,time_stats.moe_grouped_gemm.mean,time_stats.moe_grouped_gemm.median,time_stats.moe_grouped_gemm.std,num_tokens,num_experts,num_experts_per_device,expert_parallel_size,routing_runtime_path,routing_assignment_policy,routing_weight_policy,routing_uses_router_logits,gating_runtime_context,gating_runtime_context_impl,router_topk,hidden_dim,expert_hidden_dim,use_gated,num_tensor_parallel_workers,total_routed_tokens,model_expansion_ratio,tokens_per_expert_avg,tokens_to_experts_ratio,expert_utilization,min_load_ratio,load_imbalance_cv,max_load_ratio,load_entropy,load_gini_coefficient,load_distribution,seed,moe_grouped_gemm_backend,measurement_type,profiling_precision,model_arch,quant_signature +0.03097599931061268,0.049056001007556915,0.03467839974910021,0.03254400007426739,0.005093522706269737,0.05193600058555603,0.08419200032949448,0.06054240055382252,0.05641600117087364,0.009051489911083033,0.025919999927282333,0.04064000025391579,0.030527999717742206,0.030608000233769417,0.0040231266205605675,0.29603201150894165,0.3494400084018707,0.3075023889541626,0.30137598514556885,0.014090820215642452,16,128,128,1,standard_fused_topk,logit_topk,softmax_renorm,True,prefill_hot,ffn_like_prefix_20x,8,4096,1536,True,4,128,0.375,1.0,1.0,0.609375,0.0,1.0231690964840563,4.0,6.122626857503489,0.5433349609375,uniform,0,vllm_fused,CUDA_EVENT,BF16,generic,method=fp8|act=dynamic|serialized=True|block=128x128 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/attention_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/attention_config.yaml new file mode 100644 index 0000000..bf8fcfa --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/attention_config.yaml @@ -0,0 +1,65 @@ +attention_backend: FLASHINFER +batch_size_list: +- 1 +block_shape: null +block_size: 16 +decode_kv_cache_size_list: null +device: h20 +disable_ray: true +disable_replicated: false +enable_chunked_prefill_grid_search: false +enable_mixed_prefill: false +enable_true_mixed: false +fixed_chunked_prefill_size: 128 +max_batch_size: 1 +max_mixed_batch_size: 8 +max_model_len: 40960 +max_pipeline_parallel_size: 1 +max_seq_len: 128 +min_batch_size: 1 +mixed_batch_size_list: null +mixed_batch_size_max: 32 +mixed_batch_size_min: 2 +mixed_kv_cache_size_list: +- 0 +mixed_mode: both +mixed_num_samples: 3 +mixed_profile_strategy: default +mixed_shapes_per_point: 2 +mixed_total_tokens_list: null +mixed_total_tokens_max: 1055 +mixed_total_tokens_min: 1025 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_tensor_parallel_workers: +- 4 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles +precision: null +profile_method: cuda_event +profile_only_decode: false +profile_only_prefill: true +skip_confirmation: true +true_mixed_decode_batch_sizes: +- 1 +- 2 +- 4 +- 8 +true_mixed_decode_kv_cache_sizes: +- 128 +- 256 +- 512 +- 1024 +- 2048 +true_mixed_prefill_batch_sizes: +- 1 +- 2 +- 4 +true_mixed_prefill_chunk_sizes: +- 64 +- 128 +- 256 +- 512 +- 1024 +true_mixed_prefill_kv_cache_size: 0 +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/linear_op_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/linear_op_config.yaml new file mode 100644 index 0000000..d33078c --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/linear_op_config.yaml @@ -0,0 +1,23 @@ +attn_tp: null +block_shape: null +device: h20 +disable_ray: true +disable_replicated: false +extra_num_tokens: null +ffn_tp: null +include_target_embedded_mtp: false +is_moe: true +max_tokens: 16 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_tensor_parallel_workers: +- 4 +num_tokens_list: +- 16 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles +precision: null +profile_method: cuda_event +ray_enable_datasets_serializers: false +skip_confirmation: true +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/moe_config.yaml b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/moe_config.yaml new file mode 100644 index 0000000..a21e879 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/profiles/compute/h20/moe_config.yaml @@ -0,0 +1,27 @@ +block_shape: null +device: h20 +disable_ray: true +disable_replicated: false +enable_load_imbalance: true +expert_parallel_sizes: +- 1 +extra_num_tokens: null +gating_runtime_context: prefill_hot +load_distributions: +- uniform +max_tokens: 16 +models: +- Qwen3-235B-A22B-FP8 +num_gpus: 1 +num_samples_per_distribution: 1 +num_tensor_parallel_workers: +- 4 +num_tokens_list: +- 16 +output_dir: /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles +per_channel_quant: false +precision: null +profile_method: cuda_event +routing_runtime_path: standard_fused_topk +skip_confirmation: true +use_fp8: null diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.json new file mode 100644 index 0000000..4ba0feb --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.json @@ -0,0 +1,12 @@ +{ + "flashinfer_metadata_version": "0.3.1.post1", + "flashinfer_path": "/tmp/wjh-frontier-vllm0102-smoke/.venv/lib/python3.12/site-packages/flashinfer/__init__.py", + "frontier_metadata_version": "0.1.0", + "frontier_path": "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6/frontier", + "python": "3.12.3", + "torch": "2.8.0+cu128", + "torch_cuda": "12.8", + "vllm_import_version": "0.10.2", + "vllm_metadata_version": "0.10.2", + "vllm_path": "/tmp/wjh-frontier-vllm0102-smoke/.venv/lib/python3.12/site-packages/vllm/__init__.py" +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.post_transformers_pin.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.post_transformers_pin.json new file mode 100644 index 0000000..b36217d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.post_transformers_pin.json @@ -0,0 +1,9 @@ +{ + "flashinfer-python": "0.3.1.post1", + "frontier-simulator": "0.1.0", + "python": "3.12.3", + "tokenizers": "0.21.4", + "torch": "2.8.0", + "transformers": "4.55.2", + "vllm": "0.10.2" +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.post_transformers_pin.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.post_transformers_pin.sha256 new file mode 100644 index 0000000..d52125e --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.post_transformers_pin.sha256 @@ -0,0 +1,2 @@ +5184cb3f458685616364a09db28eee1603af26d58894073428a67ab677d655f5 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/environment.post_transformers_pin.json +2d17f91ca22ed56dbffc4549972814be911901f321963f8f07a2b940df191340 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/requirements.freeze.txt diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.pre_transformers_pin.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.pre_transformers_pin.json new file mode 100644 index 0000000..4ba0feb --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.pre_transformers_pin.json @@ -0,0 +1,12 @@ +{ + "flashinfer_metadata_version": "0.3.1.post1", + "flashinfer_path": "/tmp/wjh-frontier-vllm0102-smoke/.venv/lib/python3.12/site-packages/flashinfer/__init__.py", + "frontier_metadata_version": "0.1.0", + "frontier_path": "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6/frontier", + "python": "3.12.3", + "torch": "2.8.0+cu128", + "torch_cuda": "12.8", + "vllm_import_version": "0.10.2", + "vllm_metadata_version": "0.10.2", + "vllm_path": "/tmp/wjh-frontier-vllm0102-smoke/.venv/lib/python3.12/site-packages/vllm/__init__.py" +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.sha256 new file mode 100644 index 0000000..d7b2b17 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/environment.sha256 @@ -0,0 +1,2 @@ +8564395327d10aff62c48e0e4fb74bdd050d17faed3d8d3948ff2e96e767d03a /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/environment.json +a931219d3038697371775d2e5d797321adefc4ca48da2ab5999e1206e17a3881 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/requirements.freeze.txt diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_config.original.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_config.original.json new file mode 100644 index 0000000..a812ce1 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_config.original.json @@ -0,0 +1,332 @@ +{ + "architectures": [ + "Qwen3MoeForCausalLM" + ], + "attention_bias": false, + "attention_dropout": 0.0, + "bos_token_id": 151643, + "decoder_sparse_step": 1, + "eos_token_id": 151645, + "head_dim": 128, + "hidden_act": "silu", + "hidden_size": 4096, + "initializer_range": 0.02, + "intermediate_size": 12288, + "max_position_embeddings": 40960, + "max_window_layers": 94, + "mlp_only_layers": [], + "model_type": "qwen3_moe", + "moe_intermediate_size": 1536, + "norm_topk_prob": true, + "num_attention_heads": 64, + "num_experts": 128, + "num_experts_per_tok": 8, + "num_hidden_layers": 94, + "num_key_value_heads": 4, + "output_router_logits": false, + "rms_norm_eps": 1e-06, + "rope_scaling": null, + "rope_theta": 1000000.0, + "router_aux_loss_coef": 0.001, + "sliding_window": null, + "tie_word_embeddings": false, + "torch_dtype": "bfloat16", + "transformers_version": "4.51.0", + "use_cache": true, + "use_sliding_window": false, + "vocab_size": 151936, + "quantization_config": { + "activation_scheme": "dynamic", + "modules_to_not_convert": [ + "lm_head", + "model.layers.0.input_layernorm", + "model.layers.0.mlp.gate", + "model.layers.0.post_attention_layernorm", + "model.layers.1.input_layernorm", + "model.layers.1.mlp.gate", + "model.layers.1.post_attention_layernorm", + "model.layers.2.input_layernorm", + "model.layers.2.mlp.gate", + "model.layers.2.post_attention_layernorm", + "model.layers.3.input_layernorm", + "model.layers.3.mlp.gate", + "model.layers.3.post_attention_layernorm", + "model.layers.4.input_layernorm", + "model.layers.4.mlp.gate", + "model.layers.4.post_attention_layernorm", + "model.layers.5.input_layernorm", + "model.layers.5.mlp.gate", + "model.layers.5.post_attention_layernorm", + "model.layers.6.input_layernorm", + "model.layers.6.mlp.gate", + "model.layers.6.post_attention_layernorm", + "model.layers.7.input_layernorm", + "model.layers.7.mlp.gate", + "model.layers.7.post_attention_layernorm", + "model.layers.8.input_layernorm", + "model.layers.8.mlp.gate", + "model.layers.8.post_attention_layernorm", + "model.layers.9.input_layernorm", + "model.layers.9.mlp.gate", + "model.layers.9.post_attention_layernorm", + "model.layers.10.input_layernorm", + "model.layers.10.mlp.gate", + "model.layers.10.post_attention_layernorm", + "model.layers.11.input_layernorm", + "model.layers.11.mlp.gate", + "model.layers.11.post_attention_layernorm", + "model.layers.12.input_layernorm", + "model.layers.12.mlp.gate", + "model.layers.12.post_attention_layernorm", + "model.layers.13.input_layernorm", + "model.layers.13.mlp.gate", + "model.layers.13.post_attention_layernorm", + "model.layers.14.input_layernorm", + "model.layers.14.mlp.gate", + "model.layers.14.post_attention_layernorm", + "model.layers.15.input_layernorm", + "model.layers.15.mlp.gate", + "model.layers.15.post_attention_layernorm", + "model.layers.16.input_layernorm", + "model.layers.16.mlp.gate", + "model.layers.16.post_attention_layernorm", + "model.layers.17.input_layernorm", + "model.layers.17.mlp.gate", + "model.layers.17.post_attention_layernorm", + "model.layers.18.input_layernorm", + "model.layers.18.mlp.gate", + "model.layers.18.post_attention_layernorm", + "model.layers.19.input_layernorm", + "model.layers.19.mlp.gate", + "model.layers.19.post_attention_layernorm", + "model.layers.20.input_layernorm", + "model.layers.20.mlp.gate", + "model.layers.20.post_attention_layernorm", + "model.layers.21.input_layernorm", + "model.layers.21.mlp.gate", + "model.layers.21.post_attention_layernorm", + "model.layers.22.input_layernorm", + "model.layers.22.mlp.gate", + "model.layers.22.post_attention_layernorm", + "model.layers.23.input_layernorm", + "model.layers.23.mlp.gate", + "model.layers.23.post_attention_layernorm", + "model.layers.24.input_layernorm", + "model.layers.24.mlp.gate", + "model.layers.24.post_attention_layernorm", + "model.layers.25.input_layernorm", + "model.layers.25.mlp.gate", + "model.layers.25.post_attention_layernorm", + "model.layers.26.input_layernorm", + "model.layers.26.mlp.gate", + "model.layers.26.post_attention_layernorm", + "model.layers.27.input_layernorm", + "model.layers.27.mlp.gate", + "model.layers.27.post_attention_layernorm", + "model.layers.28.input_layernorm", + "model.layers.28.mlp.gate", + "model.layers.28.post_attention_layernorm", + "model.layers.29.input_layernorm", + "model.layers.29.mlp.gate", + "model.layers.29.post_attention_layernorm", + "model.layers.30.input_layernorm", + "model.layers.30.mlp.gate", + "model.layers.30.post_attention_layernorm", + "model.layers.31.input_layernorm", + "model.layers.31.mlp.gate", + "model.layers.31.post_attention_layernorm", + "model.layers.32.input_layernorm", + "model.layers.32.mlp.gate", + "model.layers.32.post_attention_layernorm", + "model.layers.33.input_layernorm", + "model.layers.33.mlp.gate", + "model.layers.33.post_attention_layernorm", + "model.layers.34.input_layernorm", + "model.layers.34.mlp.gate", + "model.layers.34.post_attention_layernorm", + "model.layers.35.input_layernorm", + "model.layers.35.mlp.gate", + "model.layers.35.post_attention_layernorm", + "model.layers.36.input_layernorm", + "model.layers.36.mlp.gate", + "model.layers.36.post_attention_layernorm", + "model.layers.37.input_layernorm", + "model.layers.37.mlp.gate", + "model.layers.37.post_attention_layernorm", + "model.layers.38.input_layernorm", + "model.layers.38.mlp.gate", + "model.layers.38.post_attention_layernorm", + "model.layers.39.input_layernorm", + "model.layers.39.mlp.gate", + "model.layers.39.post_attention_layernorm", + "model.layers.40.input_layernorm", + "model.layers.40.mlp.gate", + "model.layers.40.post_attention_layernorm", + "model.layers.41.input_layernorm", + "model.layers.41.mlp.gate", + "model.layers.41.post_attention_layernorm", + "model.layers.42.input_layernorm", + "model.layers.42.mlp.gate", + "model.layers.42.post_attention_layernorm", + "model.layers.43.input_layernorm", + "model.layers.43.mlp.gate", + "model.layers.43.post_attention_layernorm", + "model.layers.44.input_layernorm", + "model.layers.44.mlp.gate", + "model.layers.44.post_attention_layernorm", + "model.layers.45.input_layernorm", + "model.layers.45.mlp.gate", + "model.layers.45.post_attention_layernorm", + "model.layers.46.input_layernorm", + "model.layers.46.mlp.gate", + "model.layers.46.post_attention_layernorm", + "model.layers.47.input_layernorm", + "model.layers.47.mlp.gate", + "model.layers.47.post_attention_layernorm", + "model.layers.48.input_layernorm", + "model.layers.48.mlp.gate", + "model.layers.48.post_attention_layernorm", + "model.layers.49.input_layernorm", + "model.layers.49.mlp.gate", + "model.layers.49.post_attention_layernorm", + "model.layers.50.input_layernorm", + "model.layers.50.mlp.gate", + "model.layers.50.post_attention_layernorm", + "model.layers.51.input_layernorm", + "model.layers.51.mlp.gate", + "model.layers.51.post_attention_layernorm", + "model.layers.52.input_layernorm", + "model.layers.52.mlp.gate", + "model.layers.52.post_attention_layernorm", + "model.layers.53.input_layernorm", + "model.layers.53.mlp.gate", + "model.layers.53.post_attention_layernorm", + "model.layers.54.input_layernorm", + "model.layers.54.mlp.gate", + "model.layers.54.post_attention_layernorm", + "model.layers.55.input_layernorm", + "model.layers.55.mlp.gate", + "model.layers.55.post_attention_layernorm", + "model.layers.56.input_layernorm", + "model.layers.56.mlp.gate", + "model.layers.56.post_attention_layernorm", + "model.layers.57.input_layernorm", + "model.layers.57.mlp.gate", + "model.layers.57.post_attention_layernorm", + "model.layers.58.input_layernorm", + "model.layers.58.mlp.gate", + "model.layers.58.post_attention_layernorm", + "model.layers.59.input_layernorm", + "model.layers.59.mlp.gate", + "model.layers.59.post_attention_layernorm", + "model.layers.60.input_layernorm", + "model.layers.60.mlp.gate", + "model.layers.60.post_attention_layernorm", + "model.layers.61.input_layernorm", + "model.layers.61.mlp.gate", + "model.layers.61.post_attention_layernorm", + "model.layers.62.input_layernorm", + "model.layers.62.mlp.gate", + "model.layers.62.post_attention_layernorm", + "model.layers.63.input_layernorm", + "model.layers.63.mlp.gate", + "model.layers.63.post_attention_layernorm", + "model.layers.64.input_layernorm", + "model.layers.64.mlp.gate", + "model.layers.64.post_attention_layernorm", + "model.layers.65.input_layernorm", + "model.layers.65.mlp.gate", + "model.layers.65.post_attention_layernorm", + "model.layers.66.input_layernorm", + "model.layers.66.mlp.gate", + "model.layers.66.post_attention_layernorm", + "model.layers.67.input_layernorm", + "model.layers.67.mlp.gate", + "model.layers.67.post_attention_layernorm", + "model.layers.68.input_layernorm", + "model.layers.68.mlp.gate", + "model.layers.68.post_attention_layernorm", + "model.layers.69.input_layernorm", + "model.layers.69.mlp.gate", + "model.layers.69.post_attention_layernorm", + "model.layers.70.input_layernorm", + "model.layers.70.mlp.gate", + "model.layers.70.post_attention_layernorm", + "model.layers.71.input_layernorm", + "model.layers.71.mlp.gate", + "model.layers.71.post_attention_layernorm", + "model.layers.72.input_layernorm", + "model.layers.72.mlp.gate", + "model.layers.72.post_attention_layernorm", + "model.layers.73.input_layernorm", + "model.layers.73.mlp.gate", + "model.layers.73.post_attention_layernorm", + "model.layers.74.input_layernorm", + "model.layers.74.mlp.gate", + "model.layers.74.post_attention_layernorm", + "model.layers.75.input_layernorm", + "model.layers.75.mlp.gate", + "model.layers.75.post_attention_layernorm", + "model.layers.76.input_layernorm", + "model.layers.76.mlp.gate", + "model.layers.76.post_attention_layernorm", + "model.layers.77.input_layernorm", + "model.layers.77.mlp.gate", + "model.layers.77.post_attention_layernorm", + "model.layers.78.input_layernorm", + "model.layers.78.mlp.gate", + "model.layers.78.post_attention_layernorm", + "model.layers.79.input_layernorm", + "model.layers.79.mlp.gate", + "model.layers.79.post_attention_layernorm", + "model.layers.80.input_layernorm", + "model.layers.80.mlp.gate", + "model.layers.80.post_attention_layernorm", + "model.layers.81.input_layernorm", + "model.layers.81.mlp.gate", + "model.layers.81.post_attention_layernorm", + "model.layers.82.input_layernorm", + "model.layers.82.mlp.gate", + "model.layers.82.post_attention_layernorm", + "model.layers.83.input_layernorm", + "model.layers.83.mlp.gate", + "model.layers.83.post_attention_layernorm", + "model.layers.84.input_layernorm", + "model.layers.84.mlp.gate", + "model.layers.84.post_attention_layernorm", + "model.layers.85.input_layernorm", + "model.layers.85.mlp.gate", + "model.layers.85.post_attention_layernorm", + "model.layers.86.input_layernorm", + "model.layers.86.mlp.gate", + "model.layers.86.post_attention_layernorm", + "model.layers.87.input_layernorm", + "model.layers.87.mlp.gate", + "model.layers.87.post_attention_layernorm", + "model.layers.88.input_layernorm", + "model.layers.88.mlp.gate", + "model.layers.88.post_attention_layernorm", + "model.layers.89.input_layernorm", + "model.layers.89.mlp.gate", + "model.layers.89.post_attention_layernorm", + "model.layers.90.input_layernorm", + "model.layers.90.mlp.gate", + "model.layers.90.post_attention_layernorm", + "model.layers.91.input_layernorm", + "model.layers.91.mlp.gate", + "model.layers.91.post_attention_layernorm", + "model.layers.92.input_layernorm", + "model.layers.92.mlp.gate", + "model.layers.92.post_attention_layernorm", + "model.layers.93.input_layernorm", + "model.layers.93.mlp.gate", + "model.layers.93.post_attention_layernorm" + ], + "fmt": "e4m3", + "quant_method": "fp8", + "weight_block_size": [ + 128, + 128 + ] + } +} \ No newline at end of file diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_config.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_config.sha256 new file mode 100644 index 0000000..d7fe21d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_config.sha256 @@ -0,0 +1,2 @@ +702c46d431bb984db9035a1225186bbfdb52c0d19c82104df4a37cd005e0369e /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/model_config.original.json +1d7389f77563bb26bdb8cad077f9ae81f94dbe0b7f2e51cc72f7f4ff0f550acb /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6/data/config/models/Qwen3-235B-A22B-FP8.json diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_parse.txt b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_parse.txt new file mode 100644 index 0000000..c7943ab --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/model_parse.txt @@ -0,0 +1,2 @@ +{'name': 'Qwen3-235B-A22B-FP8', 'num_layers': 94, 'num_q_heads': 64, 'num_kv_heads': 4, 'embedding_dim': 4096, 'mlp_hidden_dim': 1536, 'max_position_embeddings': 40960, 'use_gated_mlp': True, 'use_bias': False, 'use_qkv_bias': False, 'activation': 'silu', 'norm': 'rms_norm', 'post_attn_norm': True, 'vocab_size': 151936, 'is_neox_style': True, 'rope_theta': 1000000.0, 'rope_scaling': None, 'partial_rotary_factor': 1.0, 'no_tensor_parallel': False, 'is_moe': True, 'num_experts': 128, 'num_experts_per_tok': 8, 'moe_layers_enum': None, 'use_qk_norm': True, 'attn_output_gate': False, 'rms_norm_eps': 1e-06, 'dtype': 'BF16', 'model_type': 'qwen3_moe', 'fused_add_norm_capability': True, 'model_arch': 'generic', 'share_expert_dim': None, 'share_q_dim': None, 'head_dim': 128, 'quantization_config': {'quant_method': 'fp8', 'activation_scheme': 'dynamic', 'is_checkpoint_fp8_serialized': True, 'weight_block_size': (128, 128), 'ignored_layers': []}, 'tie_word_embeddings': False} +{'tp_size': 4, 'attn_enabled': True, 'ffn_enabled': True, 'attn_sharded_enabled': True, 'ffn_sharded_enabled': True, 'replicated_enabled': True, 'disable_replicated': False, 'enabled_ops': ['input_layernorm', 'post_attention_layernorm', 'add', 'emb', 'attn_pre_proj', 'attn_rope', 'attn_post_proj'], 'disabled_ops': [], 'replicated_ops': ['input_layernorm', 'post_attention_layernorm', 'add', 'emb'], 'padded_n_embd': 4096, 'padded_n_expanded_embd': 1536, 'skip_reasons': []} diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/platform.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/platform.sha256 new file mode 100644 index 0000000..565b08d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/platform.sha256 @@ -0,0 +1 @@ +4459cca6fe7f01c44477460ccca85ec1f2a0eacdc764170c68fa02a102f4104e /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/platform.txt diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/platform.txt b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/platform.txt new file mode 100644 index 0000000..e3a0afb --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/platform.txt @@ -0,0 +1,99 @@ +captured_utc=2026-07-15T09:34:36Z +ds-07429c65-1-6c5fd97778-9vhkr +Linux ds-07429c65-1-6c5fd97778-9vhkr 5.10.134-013.8.2.kangaroo.al8.x86_64 #1 SMP Thu Mar 12 10:20:37 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux +PRETTY_NAME="Ubuntu 24.04.2 LTS" +NAME="Ubuntu" +VERSION_ID="24.04" +VERSION="24.04.2 LTS (Noble Numbat)" +VERSION_CODENAME=noble +ID=ubuntu +ID_LIKE=debian +HOME_URL="https://www.ubuntu.com/" +SUPPORT_URL="https://help.ubuntu.com/" +BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" +PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" +UBUNTU_CODENAME=noble +LOGO=ubuntu-logo +Architecture: x86_64 +CPU op-mode(s): 32-bit, 64-bit +Address sizes: 52 bits physical, 57 bits virtual +Byte Order: Little Endian +CPU(s): 160 +On-line CPU(s) list: 0-159 +Vendor ID: GenuineIntel +Model name: Intel(R) Xeon(R) Processor +CPU family: 6 +Model: 143 +Thread(s) per core: 1 +Core(s) per socket: 80 +Socket(s): 2 +Stepping: 8 +BogoMIPS: 5200.00 +Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd avx512vbmi umip pku waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities +Hypervisor vendor: KVM +Virtualization type: full +L1d cache: 3.8 MiB (80 instances) +L1i cache: 2.5 MiB (80 instances) +L2 cache: 160 MiB (80 instances) +L3 cache: 195 MiB (2 instances) +NUMA node(s): 2 +NUMA node0 CPU(s): 0-79 +NUMA node1 CPU(s): 80-159 +Vulnerability Itlb multihit: Not affected +Vulnerability L1tf: Not affected +Vulnerability Mds: Not affected +Vulnerability Meltdown: Not affected +Vulnerability Mmio stale data: Not affected +Vulnerability Retbleed: Not affected +Vulnerability Spec rstack overflow: Not affected +Vulnerability Spec store bypass: Vulnerable +Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers +Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Vulnerable +Vulnerability Srbds: Not affected +Vulnerability Tsx async abort: Not affected +0, NVIDIA H20, GPU-ad3e049a-5bf0-44b7-e7f1-9af297b172af, 580.95.05, 97871 MiB, 9.0 +1, NVIDIA H20, GPU-8c088079-d0f5-ba23-8650-5e6b1436691f, 580.95.05, 97871 MiB, 9.0 +2, NVIDIA H20, GPU-a9f6fe67-324b-8bb2-19b3-c7f1b9cce96a, 580.95.05, 97871 MiB, 9.0 +3, NVIDIA H20, GPU-6bcd68b7-ffa7-26b5-df6a-b7eb3f65c901, 580.95.05, 97871 MiB, 9.0 +4, NVIDIA H20, GPU-b409f9c9-05b6-55ef-3f3b-12eaa7c6ebfe, 580.95.05, 97871 MiB, 9.0 +5, NVIDIA H20, GPU-56932433-efce-8215-6418-98166d8ab798, 580.95.05, 97871 MiB, 9.0 +6, NVIDIA H20, GPU-ddcd1b77-e38d-75f5-ac24-788c76e56c89, 580.95.05, 97871 MiB, 9.0 +7, NVIDIA H20, GPU-9b4a31bd-5e78-b5a7-55aa-d786ae5a3f21, 580.95.05, 97871 MiB, 9.0 + GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 CPU Affinity NUMA Affinity GPU NUMA ID +GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX PHB SYS SYS 0-79 0 N/A +GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 PXB PHB SYS SYS 0-79 0 N/A +GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 PHB PIX SYS SYS 0-79 0 N/A +GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 PHB PXB SYS SYS 0-79 0 N/A +GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS PIX PHB 80-159 1 N/A +GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS PXB PHB 80-159 1 N/A +GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS PHB PIX 80-159 1 N/A +GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS PHB PXB 80-159 1 N/A +NIC0 PIX PXB PHB PHB SYS SYS SYS SYS X PHB SYS SYS +NIC1 PHB PHB PIX PXB SYS SYS SYS SYS PHB X SYS SYS +NIC2 SYS SYS SYS SYS PIX PXB PHB PHB SYS SYS X PHB +NIC3 SYS SYS SYS SYS PHB PHB PIX PXB SYS SYS PHB X + +Legend: + + X = Self + SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) + NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node + PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) + PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) + PIX = Connection traversing at most a single PCIe bridge + NV# = Connection traversing a bonded set of # NVLinks + +NIC Legend: + + NIC0: mlx5_0 + NIC1: mlx5_1 + NIC2: mlx5_2 + NIC3: mlx5_3 + +nvcc: NVIDIA (R) Cuda compiler driver +Copyright (c) 2005-2025 NVIDIA Corporation +Built on Tue_May_27_02:21:03_PDT_2025 +Cuda compilation tools, release 12.9, V12.9.86 +Build cuda_12.9.r12.9/compiler.36037853_0 +gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 +ldd (Ubuntu GLIBC 2.39-0ubuntu8.4) 2.39 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.freeze.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.freeze.sha256 new file mode 100644 index 0000000..69ae395 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.freeze.sha256 @@ -0,0 +1 @@ +2d17f91ca22ed56dbffc4549972814be911901f321963f8f07a2b940df191340 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/provenance/requirements.freeze.txt diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.freeze.txt b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.freeze.txt new file mode 100644 index 0000000..1b8b6e4 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.freeze.txt @@ -0,0 +1,163 @@ +aiohappyeyeballs==2.7.1 +aiohttp==3.14.1 +aiosignal==1.4.0 +annotated-doc==0.0.4 +annotated-types==0.7.0 +anyio==4.14.2 +astor==0.8.1 +attrs==26.1.0 +blake3==1.0.9 +cachetools==7.1.4 +cbor2==6.1.3 +certifi==2026.6.17 +cffi==2.1.0 +charset-normalizer==3.4.9 +click==8.4.2 +cloudpickle==3.1.2 +compressed-tensors==0.11.0 +cuda-pathfinder==1.5.6 +cupy-cuda12x==14.1.1 +ddsketch==3.0.1 +depyf==0.19.0 +detect-installer==0.1.0 +dill==0.4.1 +diskcache==5.6.3 +distro==1.9.0 +dnspython==2.8.0 +einops==0.8.2 +email-validator==2.3.0 +fastapi==0.139.0 +fastapi-cli==0.0.29 +fastapi-cloud-cli==0.22.2 +fastar==0.11.0 +fasteners==0.20 +filelock==3.29.7 +flashinfer-python==0.3.1.post1 +-e file:///home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6 +frozendict==2.4.7 +frozenlist==1.8.0 +fsspec==2026.6.0 +gguf==0.19.0 +h11==0.16.0 +hf-xet==1.5.1 +httpcore==1.0.9 +httptools==0.8.0 +httpx==0.28.1 +huggingface-hub==0.36.2 +idna==3.18 +iniconfig==2.3.0 +interegular==0.3.3 +jinja2==3.1.6 +jiter==0.16.0 +joblib==1.5.3 +jsonschema==4.26.0 +jsonschema-specifications==2025.9.1 +lark==1.2.2 +llguidance==0.7.30 +llvmlite==0.44.0 +lm-format-enforcer==0.11.3 +markdown-it-py==4.2.0 +markupsafe==3.0.3 +mdurl==0.1.2 +mistral-common==1.11.5 +mpmath==1.3.0 +msgpack==1.2.1 +msgspec==0.21.1 +multidict==6.7.1 +narwhals==2.24.0 +networkx==3.6.1 +ninja==1.13.0 +numba==0.61.2 +numpy==2.2.6 +nvidia-cublas-cu12==12.8.4.1 +nvidia-cuda-cupti-cu12==12.8.90 +nvidia-cuda-nvrtc-cu12==12.8.93 +nvidia-cuda-runtime-cu12==12.8.90 +nvidia-cudnn-cu12==9.10.2.21 +nvidia-cudnn-frontend==1.26.0 +nvidia-cufft-cu12==11.3.3.83 +nvidia-cufile-cu12==1.13.1.3 +nvidia-curand-cu12==10.3.9.90 +nvidia-cusolver-cu12==11.7.3.90 +nvidia-cusparse-cu12==12.5.8.93 +nvidia-cusparselt-cu12==0.7.1 +nvidia-ml-py==13.610.43 +nvidia-nccl-cu12==2.27.3 +nvidia-nvjitlink-cu12==12.8.93 +nvidia-nvtx-cu12==12.8.90 +openai==2.45.0 +openai-harmony==0.0.8 +opencv-python-headless==5.0.0.93 +outlines-core==0.2.11 +packaging==26.2 +pandas==3.0.3 +partial-json-parser==0.2.1.1.post7 +pillow==12.3.0 +plotly==6.9.0 +pluggy==1.6.0 +prometheus-client==0.25.0 +prometheus-fastapi-instrumentator==8.0.2 +propcache==0.5.2 +protobuf==7.35.1 +psutil==7.2.2 +py-cpuinfo==9.0.0 +pybase64==1.4.3 +pycountry==26.2.16 +pycparser==3.0 +pydantic==2.13.4 +pydantic-core==2.46.4 +pydantic-extra-types==2.11.1 +pydantic-settings==2.14.2 +pygments==2.20.0 +pynvml==13.0.1 +pytest==9.1.1 +python-dateutil==2.9.0.post0 +python-dotenv==1.2.2 +python-json-logger==4.1.0 +python-multipart==0.0.32 +pyyaml==6.0.3 +pyzmq==27.1.0 +ray==2.56.0 +referencing==0.37.0 +regex==2026.7.10 +requests==2.34.2 +rich==15.0.0 +rich-toolkit==0.20.3 +rignore==0.7.6 +rpds-py==2026.6.3 +safetensors==0.8.0 +scikit-learn==1.9.0 +scipy==1.18.0 +sentencepiece==0.2.2 +sentry-sdk==2.65.0 +setproctitle==1.3.7 +setuptools==79.0.1 +shellingham==1.5.4 +six==1.17.0 +sniffio==1.3.1 +soundfile==0.14.0 +soxr==1.1.0 +starlette==1.3.1 +sympy==1.14.0 +tabulate==0.10.0 +threadpoolctl==3.6.0 +tiktoken==0.13.0 +tokenizers==0.21.4 +torch==2.8.0 +torchaudio==2.8.0 +torchvision==0.23.0 +tqdm==4.68.4 +transformers==4.55.2 +triton==3.4.0 +typer==0.26.8 +typing-extensions==4.16.0 +typing-inspection==0.4.2 +urllib3==2.7.0 +uvicorn==0.51.0 +uvloop==0.22.1 +vllm==0.10.2 +watchfiles==1.2.0 +websockets==16.1 +xformers==0.0.32.post1 +xgrammar==0.1.23 +yarl==1.24.2 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.pre_transformers_pin.freeze.txt b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.pre_transformers_pin.freeze.txt new file mode 100644 index 0000000..f058233 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/provenance/requirements.pre_transformers_pin.freeze.txt @@ -0,0 +1,163 @@ +aiohappyeyeballs==2.7.1 +aiohttp==3.14.1 +aiosignal==1.4.0 +annotated-doc==0.0.4 +annotated-types==0.7.0 +anyio==4.14.2 +astor==0.8.1 +attrs==26.1.0 +blake3==1.0.9 +cachetools==7.1.4 +cbor2==6.1.3 +certifi==2026.6.17 +cffi==2.1.0 +charset-normalizer==3.4.9 +click==8.4.2 +cloudpickle==3.1.2 +compressed-tensors==0.11.0 +cuda-pathfinder==1.5.6 +cupy-cuda12x==14.1.1 +ddsketch==3.0.1 +depyf==0.19.0 +detect-installer==0.1.0 +dill==0.4.1 +diskcache==5.6.3 +distro==1.9.0 +dnspython==2.8.0 +einops==0.8.2 +email-validator==2.3.0 +fastapi==0.139.0 +fastapi-cli==0.0.29 +fastapi-cloud-cli==0.22.2 +fastar==0.11.0 +fasteners==0.20 +filelock==3.29.7 +flashinfer-python==0.3.1.post1 +-e file:///home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6 +frozendict==2.4.7 +frozenlist==1.8.0 +fsspec==2026.6.0 +gguf==0.19.0 +h11==0.16.0 +hf-xet==1.5.1 +httpcore==1.0.9 +httptools==0.8.0 +httpx==0.28.1 +huggingface-hub==1.23.0 +idna==3.18 +iniconfig==2.3.0 +interegular==0.3.3 +jinja2==3.1.6 +jiter==0.16.0 +joblib==1.5.3 +jsonschema==4.26.0 +jsonschema-specifications==2025.9.1 +lark==1.2.2 +llguidance==0.7.30 +llvmlite==0.44.0 +lm-format-enforcer==0.11.3 +markdown-it-py==4.2.0 +markupsafe==3.0.3 +mdurl==0.1.2 +mistral-common==1.11.5 +mpmath==1.3.0 +msgpack==1.2.1 +msgspec==0.21.1 +multidict==6.7.1 +narwhals==2.24.0 +networkx==3.6.1 +ninja==1.13.0 +numba==0.61.2 +numpy==2.2.6 +nvidia-cublas-cu12==12.8.4.1 +nvidia-cuda-cupti-cu12==12.8.90 +nvidia-cuda-nvrtc-cu12==12.8.93 +nvidia-cuda-runtime-cu12==12.8.90 +nvidia-cudnn-cu12==9.10.2.21 +nvidia-cudnn-frontend==1.26.0 +nvidia-cufft-cu12==11.3.3.83 +nvidia-cufile-cu12==1.13.1.3 +nvidia-curand-cu12==10.3.9.90 +nvidia-cusolver-cu12==11.7.3.90 +nvidia-cusparse-cu12==12.5.8.93 +nvidia-cusparselt-cu12==0.7.1 +nvidia-ml-py==13.610.43 +nvidia-nccl-cu12==2.27.3 +nvidia-nvjitlink-cu12==12.8.93 +nvidia-nvtx-cu12==12.8.90 +openai==2.45.0 +openai-harmony==0.0.8 +opencv-python-headless==5.0.0.93 +outlines-core==0.2.11 +packaging==26.2 +pandas==3.0.3 +partial-json-parser==0.2.1.1.post7 +pillow==12.3.0 +plotly==6.9.0 +pluggy==1.6.0 +prometheus-client==0.25.0 +prometheus-fastapi-instrumentator==8.0.2 +propcache==0.5.2 +protobuf==7.35.1 +psutil==7.2.2 +py-cpuinfo==9.0.0 +pybase64==1.4.3 +pycountry==26.2.16 +pycparser==3.0 +pydantic==2.13.4 +pydantic-core==2.46.4 +pydantic-extra-types==2.11.1 +pydantic-settings==2.14.2 +pygments==2.20.0 +pynvml==13.0.1 +pytest==9.1.1 +python-dateutil==2.9.0.post0 +python-dotenv==1.2.2 +python-json-logger==4.1.0 +python-multipart==0.0.32 +pyyaml==6.0.3 +pyzmq==27.1.0 +ray==2.56.0 +referencing==0.37.0 +regex==2026.7.10 +requests==2.34.2 +rich==15.0.0 +rich-toolkit==0.20.3 +rignore==0.7.6 +rpds-py==2026.6.3 +safetensors==0.8.0 +scikit-learn==1.9.0 +scipy==1.18.0 +sentencepiece==0.2.2 +sentry-sdk==2.65.0 +setproctitle==1.3.7 +setuptools==79.0.1 +shellingham==1.5.4 +six==1.17.0 +sniffio==1.3.1 +soundfile==0.14.0 +soxr==1.1.0 +starlette==1.3.1 +sympy==1.14.0 +tabulate==0.10.0 +threadpoolctl==3.6.0 +tiktoken==0.13.0 +tokenizers==0.22.2 +torch==2.8.0 +torchaudio==2.8.0 +torchvision==0.23.0 +tqdm==4.68.4 +transformers==5.13.1 +triton==3.4.0 +typer==0.26.8 +typing-extensions==4.16.0 +typing-inspection==0.4.2 +urllib3==2.7.0 +uvicorn==0.51.0 +uvloop==0.22.1 +vllm==0.10.2 +watchfiles==1.2.0 +websockets==16.1 +xformers==0.0.32.post1 +xgrammar==0.1.23 +yarl==1.24.2 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/report.md b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/report.md new file mode 100644 index 0000000..c6491cd --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/report.md @@ -0,0 +1,209 @@ +# Community Qwen235B / Frontier smoke report + +## Bottom line + +The community-vLLM TP4 feasibility smoke passed, but the unmodified Frontier +profiles are not valid inputs for a formal simulator-versus-runtime ranking +test yet. The smoke exposed an execution-semantic mismatch inside the MoE +profiler: serving selects vLLM's tuned H20 block-FP8 Triton config, while the +standalone Frontier profiler omits the FP8 dtype key and silently falls back to +a default config. At the tested TP4/EP1/16-token point, fixing this lookup +reduces measured grouped-GEMM time by about 19%, with the direction consistent +across five paired routing seeds. + +This is precisely why merely installing the same vLLM release on both sides is +not a sufficient alignment contract. + +## Change + +- Built an isolated Python 3.12 environment around community vLLM 0.10.2 and + Frontier commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e`. +- Pinned Transformers 4.55.2 and tokenizers 0.21.4 after reproducing an + incompatibility with the unconstrained Transformers 5.13.1 resolver result. +- Added Frontier metadata for the local block-FP8 checkpoint without changing + the original Hugging Face config. +- Collected representative FP8 linear, FlashInfer prefill-attention, FP8 MoE, + and TP4 NCCL all-reduce measurements. +- Loaded the full Qwen3-235B-A22B-FP8 checkpoint on four H20s through community + vLLM and completed one real request. +- Ran a 2x2 MoE diagnostic that independently toggles the runtime FP8 tuning + key and BF16 compute type. + +## Expected effect + +The smoke was intended to determine whether community vLLM can provide a +shared, reproducible profiler/serving substrate before paying for the complete +response surface. It was not intended to establish Frontier ranking accuracy. + +Success required all representative operator and collective paths to execute, +the TP4 server to become ready, one request to return successfully, and all GPU +processes to be released. A discovered profiler/runtime semantic mismatch was +treated as a gate on the later simulator comparison rather than calibrated +away with end-to-end serving data. + +## Frozen environment + +- Host: `dash0`; 8 NVIDIA H20 GPUs. +- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8`. +- Original model-config SHA256: + `702c46d431bb984db9035a1225186bbfdb52c0d19c82104df4a37cd005e0369e`. +- Python 3.12.3, torch 2.8.0+cu128, vLLM 0.10.2, + flashinfer-python 0.3.1.post1, Transformers 4.55.2, tokenizers 0.21.4. +- Ubuntu 24.04.2, kernel 5.10.134, CUDA runtime 12.8, CUDA toolkit 12.9, + NVIDIA driver 580.95.05, GCC 13.3.0, glibc 2.39. +- Two 80-core Intel Xeon sockets (160 online CPUs), 8 H20 GPUs with + 97,871 MiB each and all-to-all NV18 links; GPU0--3 share NUMA node 0. +- Serving: TP4, FlashInfer, eager execution, custom all-reduce disabled, + FP8 weights, BF16 KV cache, MNS=64, MBT=8192, max model length 40,960, + prefix caching and speculative decoding disabled. + +The first environment resolved Transformers 5.13.1 because vLLM 0.10.2 only +declares `transformers>=4.55.2`. That environment failed before model loading +because vLLM accesses `all_special_tokens_extended`, an API absent from the +new tokenizer implementation. Pinning the declared minimum restored the API; +the original and corrected freezes are both retained. + +## Verification + +### End-to-end smoke + +| Stage | Result | +|---|---| +| FP8 linear, TP4, 16 tokens | Passed; QKV projection mean 0.2232 ms and output projection mean 0.1105 ms | +| FlashInfer prefill attention, TP4 | Passed for sequence lengths 32/64/96/128; attention mean 0.0571--0.0665 ms | +| FP8 MoE, TP4/EP1, 16 tokens | Passed; unmodified grouped-GEMM mean 0.3075 ms in the initial point | +| TP4 NCCL all-reduce, 128 KiB/rank | Passed; mean 0.03694 ms, p50 0.03138 ms, p95 0.05843 ms | +| Community-vLLM TP4 model load | Passed; 55.1328 GiB of weights/rank, 627 seconds | +| KV-cache sizing | 18.72 GiB/rank, 417,616 tokens; reported 10.20x concurrency at length 40,960 | +| API readiness and one-token completion | Passed; `/v1/models` and `/v1/completions` returned HTTP 200 | +| Cleanup | Passed; all eight GPUs returned to zero compute processes and zero MiB used | + +The successful serving-only run started at 09:09:52 UTC and completed at +09:22:29 UTC (12 minutes 37 seconds). The one-request response contains one +completion choice and one completion token. The copied artifacts match the +remote SHA256 manifests. + +### FP8 path audit + +The Frontier confirmation UI prints `FP8 Quantization: N/A` and labels the +operations BF16 because it displays the unset CLI override rather than the +model-config-derived operation precision. This is misleading, not evidence of +a BF16-weight profile: + +- the quantization manager configured `attn_pre_proj`, `attn_post_proj`, and + `moe_grouped_gemm` as FP8 from the model config; +- the linear implementation invokes `apply_w8a8_block_fp8_linear` with a + 128x128 weight block; +- the MoE implementation quantizes weights and activations and invokes vLLM's + fused kernel with `use_fp8_w8a8=True`; +- `FRONTIER_FP8_GEMM_SURROGATE` was not enabled; +- each CSV carries + `method=fp8|act=dynamic|serialized=True|block=128x128` separately from its + BF16 output/compute dtype metadata. + +### MoE runtime-alignment factorial + +Point: Qwen3-235B-A22B-FP8, TP4, EP1, 16 input tokens, uniform routing seeds +0--4. Values are the mean of each row's CUDA-event samples, then averaged +across the five paired routing seeds. + +| Variant | Grouped GEMM mean (ms) | Paired delta vs original | Routing-seed 95% CI | +|---|---:|---:|---:| +| Original: default config + FP16 compute type | 0.3100 | 0.00% | -- | +| FP8 config key only + FP16 compute type | 0.2508 | -19.12% | [-21.26%, -16.98%] | +| Default config + BF16 compute type only | 0.3088 | -0.40% | [-1.31%, 0.51%] | +| FP8 config key + BF16 compute type | 0.2512 | -18.99% | [-21.14%, -16.85%] | + +These intervals describe variation across the five routing seeds, not +independent process/server-run uncertainty. Even with that limitation, the +factorial localizes the dominant error at this point to kernel tuning-config +selection rather than FP16-versus-BF16 compute type. + +The concrete lookup difference is: + +- Frontier calls `get_config_dtype_str(base_dtype)`. For BF16 this returns + `None`, so it searches for a config without an FP8 dtype component and uses + the default when that file is absent. +- vLLM serving calls the same helper with `use_fp8_w8a8=True`, obtains + `fp8_w8a8`, and loads + `E=128,N=384,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[128,128].json`. + +Frontier also hard-codes `tl.float16` as the FP8 kernel compute type, while the +serving path uses BF16 for this model. That difference was not material at this +single point, but it should still be corrected to make the counterfactual +execution semantics identical. + +## Validity audit + +### Headline claims and benchmark surface + +| Claim | Verdict | Scope | +|---|---|---| +| Community vLLM 0.10.2 can load and serve this model on TP4 H20 | PASS | One frozen TP4 configuration and one real request | +| Unmodified Frontier chooses a different MoE tuning config, causing a material operator-time error at the tested point | PASS | TP4/EP1, 16 tokens, five paired routing seeds | +| Frontier is or is not sufficient as a Qwen235B config ranker | NEEDS EVIDENCE | No aligned full profiles, frozen simulator surface, or real config surface yet | + +The mechanism diagnostic uses the unmodified Frontier wrapper as baseline, +grouped-GEMM CUDA-event latency as its direct metric, and the same model, +hardware, vLLM binary, token count, TP/EP point, and routing seeds in all four +cells. Absolute times, paired relative differences, across-seed standard +deviations, and a paired routing-seed interval are retained. The data range is +deliberately narrow and cannot support an end-to-end or cross-config claim. + +| Benchmark issue | Verdict | Severity | Evidence / required action | +|---|---|---|---| +| Microbenchmark presented as end-to-end performance | PASS | -- | The 19% number is reported only as mechanism evidence; ranking remains blocked. | +| Simplified profiler matches the real execution path | FAIL | Blocking | Tuning-key and compute-type semantics differ. Apply the alignment patch and test exact config dictionaries before simulation. | +| Statistical significance / repeat protocol | NEEDS EVIDENCE | Major | Five paired routing seeds are not independent process repeats. Repeat in reverse/randomized variant order and at separated times. | +| Selective parameter range | NEEDS EVIDENCE | Major | Only TP4/EP1/16 tokens is measured. Cover the full decision-relevant token and TP range, including boundary values. | +| Calibration set equals evaluation set | PASS | -- | No serving result calibrated the profile; the protocol freezes simulator outputs before ground truth. | +| Proper diagnostic baseline | PASS | -- | Each ablation is compared with the exact unmodified Frontier commit on the same point. | +| Platform and absolute values missing | PASS | -- | OS, kernel, CPU, GPU topology, driver/toolkit/runtime versions, commands, absolute metrics, logs, and hashes are retained. | +| Full ranking baseline/SOTA comparison | N/A | -- | This smoke makes no ranking or tuner-superiority claim; it becomes required in the formal evaluation. | + +Overall audit decision: the feasibility smoke and localized mismatch result are +reportable, but any Frontier-sufficiency or config-ranking claim is **Block** +until the required reruns and end-to-end comparison are complete. + +## Result + +The shared community stack is feasible: profiles can be collected and the +235B checkpoint can serve on TP4 within memory. The stronger result, however, +is a falsification of the current alignment assumption. Same model, hardware, +vLLM package, quantization signature, and nominal backend did not imply the +same kernel configuration. A small wrapper-level omission produced a roughly +19% operator-time error before any scheduling approximation was involved. + +Therefore no Frontier config ranking should be generated from the original +MoE CSV and presented as a fair simulator-versus-real comparison. The correct +next boundary is: first make the profiler reproduce the serving kernel +selection, then freeze profile-only simulator outputs, and only then collect +the real response surface. + +## Remaining risk and next gate + +- The 19% result is one TP4/EP1/16-token point. It must be checked across the + decision-relevant token counts and TP4/TP8 before claiming its effect on + config ordering. +- Frontier's shuffling helper still performs a separate BF16/full-expert-width + config lookup and reports a missing `E=128,N=1536` H20 file. At 16 tokens its + default `BLOCK_SIZE_M` matches the tuned path, but this must be validated over + the full token grid. +- The derived Frontier model config adds + `is_checkpoint_fp8_serialized=true`, which Frontier requires to parse the + checkpoint metadata. This adaptation must remain explicit and hash-tracked. +- The representative attention and collective smokes do not provide the + profile coverage needed by the prefill candidate grid. Decode-dominant EP8 + remains blocked on all-to-all measurement/consumption. +- A TP4 cold start spends about 10.5 minutes reading weights. Using that time as + a lower-bound proxy, eight TP4/TP8 cells already cost roughly 8.4 + H20-GPU-hours in weight loading alone; a real one-pass response surface will + exceed 10 H20-GPU-hours after initialization and load probes. It requires a + separate launch approval after simulator outputs are frozen. + +Recommended next step: upstream the two semantic fixes with regression tests +that compare the exact vLLM config dictionary selected by profiler and serving +for each `(M, TP, EP, quantization)` point; align the shuffling block-size +lookup; regenerate TP4/TP8 profile closure; then run and hash the Frontier +prefill predictions. Do not start the eight-cell real serving sweep before +that gate passes. diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/models.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/models.json new file mode 100644 index 0000000..5a9ba6a --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/models.json @@ -0,0 +1 @@ +{"object":"list","data":[{"id":"qwen3-235b-community-smoke","object":"model","created":1784107344,"owned_by":"vllm","root":"/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8","parent":null,"max_model_len":40960,"permission":[{"id":"modelperm-1715f628151642daa743a9bc47bffc88","object":"model_permission","created":1784107344,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]} \ No newline at end of file diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5.sha256 new file mode 100644 index 0000000..bc8f7aa --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5.sha256 @@ -0,0 +1,6 @@ +b186a5da9809c3f2a3ac8e008837719af2f0dd65c583e022d44183b3bd7587d0 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/results/moe_factorial_r5_summary.json +ef9e8136bc6fc6d97a675cab95cb5b0425ac727d6d1e77598beadbf3d9cf4331 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/results/moe_factorial_r5_cells.csv +8d9a6aad4ee9fbb3ed054ef1798850042077df40f16c4cf3e490abe7d416bf6e /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/aligned/compute/h20/Qwen3-235B-A22B-FP8/moe.csv +bc2bf6fdb6987dc73d07c17034037f539ab2ba79901ee3ff6d33094c6beb2ffd /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/compute_type/compute/h20/Qwen3-235B-A22B-FP8/moe.csv +9769cd09cb35060ce6e3ed6610700485edf0ab00eb74c97199b200a0f90bf3bc /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/config_key/compute/h20/Qwen3-235B-A22B-FP8/moe.csv +0607dcb87a952ded5679f74ce151ca1624aa8f0484a40924920f0fd2ccb85634 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-moe-factorial-r5/original/compute/h20/Qwen3-235B-A22B-FP8/moe.csv diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5_cells.csv b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5_cells.csv new file mode 100644 index 0000000..b2691bd --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5_cells.csv @@ -0,0 +1,21 @@ +variant,seed,grouped_gemm_mean_ms,paired_delta_vs_original_pct +original,0,0.3051120042800903,0.0 +original,1,0.305161565542221,0.0 +original,2,0.3159376084804535,0.0 +original,3,0.3228943943977356,0.0 +original,4,0.301118403673172,0.0 +config_key,0,0.2425376176834106,-20.508660989699024 +config_key,1,0.2544096112251282,-16.631175104543427 +config_key,2,0.2587344050407409,-18.105854416901945 +config_key,3,0.2593088150024414,-19.692376361594743 +config_key,4,0.2388928234577179,-20.664821364752093 +compute_type,0,0.3045775890350342,-0.17515379190571334 +compute_type,1,0.3057951927185058,0.20763662526077642 +compute_type,2,0.3109855651855469,-1.567411780675354 +compute_type,3,0.3209056258201599,-0.6159192020924187 +compute_type,4,0.3015664219856262,0.14878476605517665 +aligned,0,0.2454447746276855,-19.55584467847773 +aligned,1,0.2543264031410217,-16.658441999690808 +aligned,2,0.2585648000240326,-18.159537489811207 +aligned,3,0.2606095969676971,-19.289525773964723 +aligned,4,0.2369616031646728,-21.306170504986376 diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5_summary.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5_summary.json new file mode 100644 index 0000000..f7f3d4f --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/moe_factorial_r5_summary.json @@ -0,0 +1,55 @@ +{ + "ci_scope": "paired t interval across routing seeds; not independent server-run uncertainty", + "metric": "time_stats.moe_grouped_gemm.mean", + "point": { + "ep": 1, + "model": "Qwen3-235B-A22B-FP8", + "routing_seeds": [ + 0, + 1, + 2, + 3, + 4 + ], + "tokens": 16, + "tp": 4 + }, + "variants": { + "aligned": { + "mean_ms": 0.25118143558502193, + "paired_delta_pct_95ci": [ + -21.135576186756534, + -16.8522319920158 + ], + "paired_delta_pct_mean": -18.993904089386167, + "sd_across_seeds_ms": 0.009854035105320383 + }, + "compute_type": { + "mean_ms": 0.3087660789489746, + "paired_delta_pct_95ci": [ + -1.3065750560434517, + 0.5057497027004384 + ], + "paired_delta_pct_mean": -0.4004126766715066, + "sd_across_seeds_ms": 0.007591103189515662 + }, + "config_key": { + "mean_ms": 0.2507766544818878, + "paired_delta_pct_95ci": [ + -21.259089864169376, + -16.98206543082712 + ], + "paired_delta_pct_mean": -19.120577647498248, + "sd_across_seeds_ms": 0.009466111912783425 + }, + "original": { + "mean_ms": 0.3100447952747345, + "paired_delta_pct_95ci": [ + 0.0, + 0.0 + ], + "paired_delta_pct_mean": 0.0, + "sd_across_seeds_ms": 0.009051191520196651 + } + } +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/one_request.json b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/one_request.json new file mode 100644 index 0000000..be119b8 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/one_request.json @@ -0,0 +1 @@ +{"id":"cmpl-78d1852cd51e445dab3372e1e2025d75","object":"text_completion","created":1784107344,"model":"qwen3-235b-community-smoke","choices":[{"index":0,"text":" ","logprobs":null,"finish_reason":"length","stop_reason":null,"token_ids":null,"prompt_logprobs":null,"prompt_token_ids":null}],"service_tier":null,"system_fingerprint":null,"usage":{"prompt_tokens":1,"total_tokens":2,"completion_tokens":1,"prompt_tokens_details":null},"kv_transfer_params":null} \ No newline at end of file diff --git a/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/results.sha256 b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/results.sha256 new file mode 100644 index 0000000..1685b77 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/community-qwen235b-smoke/results/results.sha256 @@ -0,0 +1,3 @@ +a44ff3d6a0998324b52df358baf778b375f98be3e03c24b07f903f9c51525beb /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/results/allreduce_tp4.jsonl +b6edf2c29065cdd303ec642bac85ceafcec30494dcce7ddf3d401c36de5162fe /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/results/models.json +921505259d86a9037f2241cf5d611942cdf2985352c7080d1af0d39750fab211 /home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/results/one_request.json diff --git a/runs/frontier-multicase-sufficiency-v0/results/ground-truth/cells.csv b/runs/frontier-multicase-sufficiency-v0/results/ground-truth/cells.csv new file mode 100644 index 0000000..d847f10 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/ground-truth/cells.csv @@ -0,0 +1,17 @@ +case,cell_id,tp,dp,ep,mns,mbt,gpu_count,score_req_s_per_gpu,capacity_lower_bound_req_s_per_gpu,capacity_upper_bound_req_s_per_gpu,capacity_bracket_width_req_s_per_gpu,best_request_rate_req_s,best_sampling_u,best_pass_rate,probe_count,best_source,completed_with_probe_failure,fully_valid,result_path,result_sha256 +qwen235b_prefill_only,tp4_mns64_mbt8192,4,1,1,64,8192,4,0.1175,0.1175,0.13333333333333333,0.015833333333333338,0.47,0.021484375,0.9609929078014184,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-tp-mns-mbt-dash1-d8899c5-20260702T025518Z/store/interaction-qwen235b-prefill-c2-tp-mns-mbt-dash1-d8899c5-20260702T025518Z/trials/trial-0001/result.json,cc9d8e3611c9719b5d8f8a14d376c24f08b348162cc49022889d9435816da13b +qwen235b_prefill_only,tp4_mns64_mbt16384,4,1,1,64,16384,4,0.10666666666666667,0.10666666666666667,0.1175,0.01083333333333332,0.4266666666666667,0.01953125,0.9609375,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-tp-mns-mbt-dash1-d8899c5-20260702T025518Z/store/interaction-qwen235b-prefill-c2-tp-mns-mbt-dash1-d8899c5-20260702T025518Z/trials/trial-0002/result.json,768360741f64ab39b3e7af497757c4fd92c66d5a0eff0125cc3d13f2814a0186 +qwen235b_prefill_only,tp4_mns128_mbt8192,4,1,1,128,8192,4,0.1175,0.1175,0.13333333333333333,0.015833333333333338,0.47,0.021484375,0.9609929078014184,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-tp-mns-mbt-dash1-d8899c5-20260702T025518Z/store/interaction-qwen235b-prefill-c2-tp-mns-mbt-dash1-d8899c5-20260702T025518Z/trials/trial-0003/result.json,6cb56fb8ad0f9dc6e68b30528ed8993207c3951ef152ce8705a04c7fcf45a02a +qwen235b_prefill_only,tp4_mns128_mbt16384,4,1,1,128,16384,4,0.10666666666666667,0.10666666666666667,0.1175,0.01083333333333332,0.4266666666666667,0.01953125,0.9609375,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/store/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/trials/trial-0001/result.json,3c485648244243d529df8d7f16b0fb3d782cef1beec695f67cdb1fd5ea02b440 +qwen235b_prefill_only,tp8_mns64_mbt8192,8,1,1,64,8192,8,0.17270833333333332,0.17270833333333332,0.17791666666666667,0.005208333333333343,1.3816666666666666,0.0546875,0.9638118214716526,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/store/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/trials/trial-0002/result.json,95c38d14ddc88f7f08730b7486ebb11538da6cec9ede11f5d16862b683290850 +qwen235b_prefill_only,tp8_mns64_mbt16384,8,1,1,64,16384,8,0.17270833333333332,0.17270833333333332,0.17791666666666667,0.005208333333333343,1.3816666666666666,0.0546875,0.9565741857659831,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/store/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/trials/trial-0003/result.json,97b96526c568a572f10de602075c59a0a14555114f65d7c8e928b51fcf1cb787 +qwen235b_prefill_only,tp8_mns128_mbt8192,8,1,1,128,8192,8,0.17270833333333332,0.17270833333333332,0.17791666666666667,0.005208333333333343,1.3816666666666666,0.0546875,0.9601930036188179,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/store/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/trials/trial-0004/result.json,f2695efe64c9c728f381e0d39ee42e7553b771928547f7a74a2cb2dafdc892f4 +qwen235b_prefill_only,tp8_mns128_mbt16384,8,1,1,128,16384,8,0.17270833333333332,0.17270833333333332,0.17791666666666667,0.005208333333333343,1.3816666666666666,0.0546875,0.9589867310012062,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/store/interaction-qwen235b-prefill-c2-remainder-dash1-d8899c5-20260702T163624Z/trials/trial-0005/result.json,1bc4d53f5ecbf4fbbd7c749a29ecf0f62e8382b84d899cc366d5bb485d78e261 +qwen235b_decode_only,tp4_dp2_ep8_mns64_mbt256,4,2,8,64,256,8,0.05354166666666667,0.05354166666666667,0.058958333333333335,0.005416666666666667,0.42833333333333334,0.01953125,1.0,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0001/result.json,f5a1c9ff3e8f98c37f8d3623e0376f34f67990f2e71e3b012beb8ac8ed447367 +qwen235b_decode_only,tp4_dp2_ep8_mns64_mbt384,4,2,8,64,384,8,0.05354166666666667,0.05354166666666667,0.058958333333333335,0.005416666666666667,0.42833333333333334,0.01953125,0.9922178988326849,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0002/result.json,adaafb060dc3ba642da34af7b69ee09ee0d04d857b8d4b5121a4b124373e1c57 +qwen235b_decode_only,tp4_dp2_ep8_mns128_mbt256,4,2,8,128,256,8,0.058958333333333335,0.058958333333333335,0.066875,0.007916666666666669,0.4716666666666667,0.021484375,0.9929328621908127,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0003/result.json,3555b3ecdef6337050c4e132e2b6f36a3cb98ea002a96f3efe118d301ad9d258 +qwen235b_decode_only,tp4_dp2_ep8_mns128_mbt384,4,2,8,128,384,8,0.058958333333333335,0.058958333333333335,0.066875,0.007916666666666669,0.4716666666666667,0.021484375,0.9929328621908127,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0004/result.json,3a010b17bc2cb785a57ef80a08646972109ad2dae28eeb38c153b35a1e3b6be2 +qwen235b_decode_only,tp2_dp4_ep8_mns64_mbt256,2,4,8,64,256,8,0.058958333333333335,0.058958333333333335,0.066875,0.007916666666666669,0.4716666666666667,0.021484375,0.9752650176678446,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0005/result.json,79649819c5599c8b81a115281a8e73aab5bc1a8188a1f611ace2329f9ae2f418 +qwen235b_decode_only,tp2_dp4_ep8_mns64_mbt384,2,4,8,64,384,8,0.05354166666666667,0.05354166666666667,0.058958333333333335,0.005416666666666667,0.42833333333333334,0.01953125,0.9961089494163424,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0006/result.json,6fdfeccfdbeb92bad8669251b26546966f46b42d667ae70eacdc544b44ad231d +qwen235b_decode_only,tp2_dp4_ep8_mns128_mbt256,2,4,8,128,256,8,0.058958333333333335,0.058958333333333335,0.066875,0.007916666666666669,0.4716666666666667,0.021484375,0.9787985865724381,6,primary_search,False,True,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0007/result.json,8bc1c359e338de1d62bc87c43b1fa270b15023d223450552bffe5d1b778458f3 +qwen235b_decode_only,tp2_dp4_ep8_mns128_mbt384,2,4,8,128,384,8,0.058958333333333335,0.058958333333333335,0.066875,0.007916666666666669,0.4716666666666667,0.021484375,0.9823321554770318,6,partial_probe_before_failure,True,False,recovered-stores/aituner-interaction-runs-dash1-20260710/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/store/interaction-qwen235b-decode-c3-topo-mns-mbt-fixed-dash1-d8899c5-20260703T022514Z/trials/trial-0008/result.json,c371c2dd38db875db827ab85d4f399d0219cad513dde3c3b4db8cc782ec4debb diff --git a/runs/frontier-multicase-sufficiency-v0/results/ground-truth/metrics.json b/runs/frontier-multicase-sufficiency-v0/results/ground-truth/metrics.json new file mode 100644 index 0000000..5164e3d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/ground-truth/metrics.json @@ -0,0 +1,75 @@ +{ + "cases": [ + { + "best_score_req_s_per_gpu": 0.17270833333333332, + "case": "qwen235b_prefill_only", + "cell_count": 8, + "distinct_score_count": 3, + "distinct_scores_req_s_per_gpu": [ + 0.17270833333333332, + 0.1175, + 0.10666666666666667 + ], + "fully_valid_cell_count": 8, + "informative_pair_count": 20, + "informative_pair_fraction": 0.7142857142857143, + "invalid_cells": [], + "pair_count": 28, + "possibly_optimal_set_from_search_brackets": [ + "tp8_mns128_mbt16384", + "tp8_mns128_mbt8192", + "tp8_mns64_mbt16384", + "tp8_mns64_mbt8192" + ], + "possibly_optimal_set_size": 4, + "random_top_set_hit_rate": 0.5, + "tied_pair_count": 8, + "top_set": [ + "tp8_mns128_mbt16384", + "tp8_mns128_mbt8192", + "tp8_mns64_mbt16384", + "tp8_mns64_mbt8192" + ], + "top_set_size": 4 + }, + { + "best_score_req_s_per_gpu": 0.058958333333333335, + "case": "qwen235b_decode_only", + "cell_count": 8, + "distinct_score_count": 2, + "distinct_scores_req_s_per_gpu": [ + 0.058958333333333335, + 0.05354166666666667 + ], + "fully_valid_cell_count": 7, + "informative_pair_count": 15, + "informative_pair_fraction": 0.5357142857142857, + "invalid_cells": [ + "tp2_dp4_ep8_mns128_mbt384" + ], + "pair_count": 28, + "possibly_optimal_set_from_search_brackets": [ + "tp2_dp4_ep8_mns128_mbt256", + "tp2_dp4_ep8_mns128_mbt384", + "tp2_dp4_ep8_mns64_mbt256", + "tp2_dp4_ep8_mns64_mbt384", + "tp4_dp2_ep8_mns128_mbt256", + "tp4_dp2_ep8_mns128_mbt384", + "tp4_dp2_ep8_mns64_mbt256", + "tp4_dp2_ep8_mns64_mbt384" + ], + "possibly_optimal_set_size": 8, + "random_top_set_hit_rate": 0.625, + "tied_pair_count": 13, + "top_set": [ + "tp2_dp4_ep8_mns128_mbt256", + "tp2_dp4_ep8_mns128_mbt384", + "tp2_dp4_ep8_mns64_mbt256", + "tp4_dp2_ep8_mns128_mbt256", + "tp4_dp2_ep8_mns128_mbt384" + ], + "top_set_size": 5 + } + ], + "schema": "frontier-multicase-ground-truth-v0" +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/ground-truth/report.md b/runs/frontier-multicase-sufficiency-v0/results/ground-truth/report.md new file mode 100644 index 0000000..8c75f8d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/ground-truth/report.md @@ -0,0 +1,47 @@ +# Qwen235B ground-truth audit + +Objective: maximum SLO-feasible offered request throughput per GPU. +This report contains real-machine data only; it makes no Frontier match claim. + +| case | valid cells | score levels | top-set size | random top-set hit | informative pairs | +|---|---:|---:|---:|---:|---:| +| qwen235b_prefill_only | 8/8 | 3 | 4/8 | 50.0% | 20/28 (71.4%) | +| qwen235b_decode_only | 7/8 | 2 | 5/8 | 62.5% | 15/28 (53.6%) | + +## Cells + +### qwen235b_prefill_only + +| cell | capacity bracket (req/s/GPU) | valid | probes | source | +|---|---:|---:|---:|---| +| tp4_mns128_mbt16384 | [0.106666667, 0.117500000) | yes | 6 | primary_search | +| tp4_mns128_mbt8192 | [0.117500000, 0.133333333) | yes | 6 | primary_search | +| tp4_mns64_mbt16384 | [0.106666667, 0.117500000) | yes | 6 | primary_search | +| tp4_mns64_mbt8192 | [0.117500000, 0.133333333) | yes | 6 | primary_search | +| tp8_mns128_mbt16384 | [0.172708333, 0.177916667) | yes | 6 | primary_search | +| tp8_mns128_mbt8192 | [0.172708333, 0.177916667) | yes | 6 | primary_search | +| tp8_mns64_mbt16384 | [0.172708333, 0.177916667) | yes | 6 | primary_search | +| tp8_mns64_mbt8192 | [0.172708333, 0.177916667) | yes | 6 | primary_search | + +Top set: `tp8_mns128_mbt16384, tp8_mns128_mbt8192, tp8_mns64_mbt16384, tp8_mns64_mbt8192`. +Possibly optimal under binary-search brackets: `tp8_mns128_mbt16384, tp8_mns128_mbt8192, tp8_mns64_mbt16384, tp8_mns64_mbt8192`. + +### qwen235b_decode_only + +| cell | capacity bracket (req/s/GPU) | valid | probes | source | +|---|---:|---:|---:|---| +| tp2_dp4_ep8_mns128_mbt256 | [0.058958333, 0.066875000) | yes | 6 | primary_search | +| tp2_dp4_ep8_mns128_mbt384 | [0.058958333, 0.066875000) | no | 6 | partial_probe_before_failure | +| tp2_dp4_ep8_mns64_mbt256 | [0.058958333, 0.066875000) | yes | 6 | primary_search | +| tp2_dp4_ep8_mns64_mbt384 | [0.053541667, 0.058958333) | yes | 6 | primary_search | +| tp4_dp2_ep8_mns128_mbt256 | [0.058958333, 0.066875000) | yes | 6 | primary_search | +| tp4_dp2_ep8_mns128_mbt384 | [0.058958333, 0.066875000) | yes | 6 | primary_search | +| tp4_dp2_ep8_mns64_mbt256 | [0.053541667, 0.058958333) | yes | 6 | primary_search | +| tp4_dp2_ep8_mns64_mbt384 | [0.053541667, 0.058958333) | yes | 6 | primary_search | + +Top set: `tp2_dp4_ep8_mns128_mbt256, tp2_dp4_ep8_mns128_mbt384, tp2_dp4_ep8_mns64_mbt256, tp4_dp2_ep8_mns128_mbt256, tp4_dp2_ep8_mns128_mbt384`. +Possibly optimal under binary-search brackets: `tp2_dp4_ep8_mns128_mbt256, tp2_dp4_ep8_mns128_mbt384, tp2_dp4_ep8_mns64_mbt256, tp2_dp4_ep8_mns64_mbt384, tp4_dp2_ep8_mns128_mbt256, tp4_dp2_ep8_mns128_mbt384, tp4_dp2_ep8_mns64_mbt256, tp4_dp2_ep8_mns64_mbt384`. + +## Interpretation guardrail + +A Frontier top-set hit is insufficient by itself because the surfaces contain large ties. The later comparison must report selected-config regret and tie-aware pairwise ranking, and must keep invalid real cells visible. diff --git a/runs/frontier-multicase-sufficiency-v0/results/qwen30-baseline/metrics.json b/runs/frontier-multicase-sufficiency-v0/results/qwen30-baseline/metrics.json new file mode 100644 index 0000000..6a52f69 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/qwen30-baseline/metrics.json @@ -0,0 +1,44 @@ +{ + "calibration": { + "fit_fixture": "coder_200_ts2", + "fitted_a_tp": { + "1": 0.7234810457606639, + "2": 0.4680889959260082, + "4": 0.3521372005220769 + }, + "holdout_fixture": "coder_200_ts3", + "loss": "[log(G_raw_rerun(tp,scale2;a)/F_raw(tp,scale2))]^2 per TP", + "refit_on_holdout": false + }, + "rows": [ + { + "agreement": 37, + "false_feasible": 0, + "false_infeasible": 55, + "kendall_tau_b": 0.0, + "mode": "uncalibrated/SLO-gated", + "optimistic_real_regret": 0.25634517766497456, + "pairwise_exact_sign_accuracy": 0.3787878787878788, + "selected_cells": [ + "tp4_mns32", + "tp4_mns64" + ], + "worst_case_real_regret": 0.25634517766497456 + }, + { + "agreement": 64, + "false_feasible": 21, + "false_infeasible": 7, + "kendall_tau_b": 0.9668009539030813, + "mode": "frozen-calibrated/SLO-gated", + "optimistic_real_regret": 0.0, + "pairwise_exact_sign_accuracy": 0.9393939393939394, + "selected_cells": [ + "tp2_mns32", + "tp2_mns64" + ], + "worst_case_real_regret": 0.0076142131979695165 + } + ], + "schema": "frontier-qwen30-calibration-audit-v0" +} diff --git a/runs/frontier-multicase-sufficiency-v0/results/qwen30-baseline/report.md b/runs/frontier-multicase-sufficiency-v0/results/qwen30-baseline/report.md new file mode 100644 index 0000000..f228ae1 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/results/qwen30-baseline/report.md @@ -0,0 +1,10 @@ +# Qwen30B Frontier baseline audit + +| mode | selected cells | worst real regret | Kendall tau-b | pair sign accuracy | feasibility (agree/FP/FN) | +|---|---|---:|---:|---:|---:| +| uncalibrated/SLO-gated | tp4_mns32, tp4_mns64 | 25.63% | 0.0000 | 37.88% | 37/0/55 | +| frozen-calibrated/SLO-gated | tp2_mns32, tp2_mns64 | 0.76% | 0.9668 | 93.94% | 64/21/7 | + +The calibrated mode applies a distinct end-to-end execution-time scale per TP: TP1=0.723481, TP2=0.468089, TP4=0.352137. + +Those scales were fitted against real total throughput on `coder_200_ts2` and checked without refitting on `coder_200_ts3`. This validates within-workload transfer of the calibration, not zero-shot Frontier prediction across TP. diff --git a/runs/frontier-multicase-sufficiency-v0/smoke/allreduce_smoke.py b/runs/frontier-multicase-sufficiency-v0/smoke/allreduce_smoke.py new file mode 100644 index 0000000..7a384de --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/smoke/allreduce_smoke.py @@ -0,0 +1,55 @@ +#!/usr/bin/env python3 + +import json +import os +import statistics + +import torch +import torch.distributed as dist + + +def main() -> None: + local_rank = int(os.environ["LOCAL_RANK"]) + torch.cuda.set_device(local_rank) + dist.init_process_group(backend="nccl") + + # 16 tokens x 4096 hidden values in BF16: 128 KiB per rank. + tensor = torch.ones((16, 4096), dtype=torch.bfloat16, device="cuda") + for _ in range(10): + dist.all_reduce(tensor) + torch.cuda.synchronize() + + samples_ms = [] + for _ in range(50): + start = torch.cuda.Event(enable_timing=True) + end = torch.cuda.Event(enable_timing=True) + start.record() + dist.all_reduce(tensor) + end.record() + end.synchronize() + samples_ms.append(float(start.elapsed_time(end))) + + if dist.get_rank() == 0: + ordered = sorted(samples_ms) + result = { + "backend": "nccl", + "collective": "all_reduce", + "dtype": "bfloat16", + "elements_per_rank": tensor.numel(), + "bytes_per_rank": tensor.numel() * tensor.element_size(), + "world_size": dist.get_world_size(), + "warmup_iterations": 10, + "measured_iterations": len(samples_ms), + "mean_ms": statistics.fmean(samples_ms), + "p50_ms": statistics.median(samples_ms), + "p95_ms": ordered[int(0.95 * (len(ordered) - 1))], + "min_ms": min(samples_ms), + "max_ms": max(samples_ms), + } + print(json.dumps(result, sort_keys=True)) + + dist.destroy_process_group() + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_compute_type_only.patch b/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_compute_type_only.patch new file mode 100644 index 0000000..ebd977d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_compute_type_only.patch @@ -0,0 +1,6 @@ +diff --git a/frontier/profiling/moe/moe_vllm_kernel.py b/frontier/profiling/moe/moe_vllm_kernel.py +--- a/frontier/profiling/moe/moe_vllm_kernel.py ++++ b/frontier/profiling/moe/moe_vllm_kernel.py +@@ -232 +232 @@ def _invoke_kernel( +- compute_type = tl.float16 # FP8 accumulates in FP16 ++ compute_type = tl.bfloat16 diff --git a/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_config_key_only.patch b/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_config_key_only.patch new file mode 100644 index 0000000..4deb15d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_config_key_only.patch @@ -0,0 +1,9 @@ +diff --git a/frontier/profiling/moe/moe_vllm_kernel.py b/frontier/profiling/moe/moe_vllm_kernel.py +--- a/frontier/profiling/moe/moe_vllm_kernel.py ++++ b/frontier/profiling/moe/moe_vllm_kernel.py +@@ -510 +510,4 @@ def profile_vllm_fused_moe( +- config_dtype = get_config_dtype_str(base_dtype) ++ config_dtype = get_config_dtype_str( ++ base_dtype, ++ use_fp8_w8a8=use_fp8, ++ ) diff --git a/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_runtime_alignment.patch b/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_runtime_alignment.patch new file mode 100644 index 0000000..27a0194 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/smoke/frontier_moe_runtime_alignment.patch @@ -0,0 +1,12 @@ +diff --git a/frontier/profiling/moe/moe_vllm_kernel.py b/frontier/profiling/moe/moe_vllm_kernel.py +--- a/frontier/profiling/moe/moe_vllm_kernel.py ++++ b/frontier/profiling/moe/moe_vllm_kernel.py +@@ -232 +232 @@ def _invoke_kernel( +- compute_type = tl.float16 # FP8 accumulates in FP16 ++ compute_type = tl.bfloat16 +@@ -510 +510,4 @@ def profile_vllm_fused_moe( +- config_dtype = get_config_dtype_str(base_dtype) ++ config_dtype = get_config_dtype_str( ++ base_dtype, ++ use_fp8_w8a8=use_fp8, ++ ) diff --git a/runs/frontier-multicase-sufficiency-v0/smoke/run_gpu_smoke.sh b/runs/frontier-multicase-sufficiency-v0/smoke/run_gpu_smoke.sh new file mode 100644 index 0000000..3af7d06 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/smoke/run_gpu_smoke.sh @@ -0,0 +1,189 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:-/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715}" +FRONTIER_ROOT="${FRONTIER_ROOT:-${OUTPUT_ROOT}/Frontier-d9cfeb6}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh-frontier-vllm0102-smoke/.venv}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8}" +PROFILE_ROOT="${OUTPUT_ROOT}/profiles" +LOG_DIR="${OUTPUT_ROOT}/logs" +RESULT_DIR="${OUTPUT_ROOT}/results" +SERVER_PORT="${SERVER_PORT:-18900}" +SKIP_LINEAR="${SKIP_LINEAR:-0}" +SERVING_ONLY="${SERVING_ONLY:-0}" +SERVED_MODEL="qwen3-235b-community-smoke" +SERVER_PID="" + +mkdir -p "${PROFILE_ROOT}" "${LOG_DIR}" "${RESULT_DIR}" +exec > >(tee -a "${LOG_DIR}/gpu_smoke.log") 2>&1 + +cleanup() { + if [[ -n "${SERVER_PID}" ]] && kill -0 "${SERVER_PID}" 2>/dev/null; then + kill -TERM -- "-${SERVER_PID}" 2>/dev/null || true + for _ in $(seq 1 30); do + if ! kill -0 "${SERVER_PID}" 2>/dev/null; then + break + fi + sleep 1 + done + kill -KILL -- "-${SERVER_PID}" 2>/dev/null || true + fi +} +trap cleanup EXIT INT TERM + +if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then + echo "ERROR: CUDA_VISIBLE_DEVICES must name exactly four allocated GPUs" >&2 + exit 1 +fi +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES}" +if [[ "${#GPU_IDS[@]}" -ne 4 ]]; then + echo "ERROR: expected four allocated GPUs, got CUDA_VISIBLE_DEVICES=${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "LAUNCH_ECHO host=$(hostname) gpus=${CUDA_VISIBLE_DEVICES} model=${MODEL_ROOT} frontier=d9cfeb6 vllm=community-0.10.2 transformers=4.55.2 backend=FLASHINFER execution=eager kv=BF16 spec=off tasks=representative-FP8-linear/attention/MoE+TP4-allreduce+TP4-model-load+one-request skip_linear=${SKIP_LINEAR} serving_only=${SERVING_ONLY} hard_wall_cap=1800s hard_gpu_cap=2_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader + +test -x "${VENV_ROOT}/bin/python" +test -f "${FRONTIER_ROOT}/pyproject.toml" +test -f "${MODEL_ROOT}/config.json" + +export PYTHONPATH="${FRONTIER_ROOT}${PYTHONPATH:+:${PYTHONPATH}}" +export TOKENIZERS_PARALLELISM=false +export VLLM_USE_V1=1 +export VLLM_ATTENTION_BACKEND=FLASHINFER +export TORCH_CUDA_ARCH_LIST=9.0 + +cd "${FRONTIER_ROOT}" + +if [[ "${SERVING_ONLY}" -eq 0 && "${SKIP_LINEAR}" -eq 0 ]]; then + echo "STAGE linear_op" + timeout 300 "${VENV_ROOT}/bin/python" -m frontier.profiling.linear_op.main \ + --disable_ray \ + --models Qwen3-235B-A22B-FP8 \ + --num_gpus 1 \ + --max_tokens 16 \ + --num_tokens_list 16 \ + --num_tensor_parallel_workers 4 \ + --profile_method cuda_event \ + --device h20 \ + --output_dir "${PROFILE_ROOT}" \ + --is_moe \ + --yes +elif [[ "${SERVING_ONLY}" -eq 0 ]]; then + echo "STAGE linear_op SKIPPED (existing artifact retained)" +fi + +if [[ "${SERVING_ONLY}" -eq 0 ]]; then + echo "STAGE attention" + timeout 300 "${VENV_ROOT}/bin/python" -m frontier.profiling.attention.main \ + --disable_ray \ + --models Qwen3-235B-A22B-FP8 \ + --num_gpus 1 \ + --max_model_len 40960 \ + --max_seq_len 128 \ + --min_batch_size 1 \ + --max_batch_size 1 \ + --batch_size_list 1 \ + --num_tensor_parallel_workers 4 \ + --max_pipeline_parallel_size 1 \ + --attention_backend FLASHINFER \ + --block_size 16 \ + --profile_only_prefill \ + --fixed_chunked_prefill_size 128 \ + --device h20 \ + --profile_method cuda_event \ + --output_dir "${PROFILE_ROOT}" \ + --yes + + echo "STAGE moe" + timeout 300 "${VENV_ROOT}/bin/python" -m frontier.profiling.moe.main \ + --disable_ray \ + --models Qwen3-235B-A22B-FP8 \ + --device h20 \ + --num_gpus 1 \ + --max_tokens 16 \ + --num_tokens_list 16 \ + --num_tensor_parallel_workers 4 \ + --expert_parallel_sizes 1 \ + --load_distributions uniform \ + --num_samples_per_distribution 1 \ + --routing_runtime_path standard_fused_topk \ + --gating_runtime_context prefill_hot \ + --profile_method cuda_event \ + --output_dir "${PROFILE_ROOT}" \ + --yes + + echo "STAGE allreduce" + timeout 180 "${VENV_ROOT}/bin/torchrun" \ + --standalone \ + --nnodes=1 \ + --nproc-per-node=4 \ + "${OUTPUT_ROOT}/scripts/allreduce_smoke.py" \ + | tee "${RESULT_DIR}/allreduce_tp4.jsonl" +else + echo "STAGES Frontier profiles and allreduce SKIPPED (existing artifacts retained)" +fi + +echo "STAGE serving" +setsid "${VENV_ROOT}/bin/vllm" serve "${MODEL_ROOT}" \ + --host 127.0.0.1 \ + --port "${SERVER_PORT}" \ + --served-model-name "${SERVED_MODEL}" \ + --tensor-parallel-size 4 \ + --disable-custom-all-reduce \ + --quantization fp8 \ + --gpu-memory-utilization 0.80 \ + --kv-cache-dtype auto \ + --max-model-len 40960 \ + --max-num-batched-tokens 8192 \ + --max-num-seqs 64 \ + --no-enable-prefix-caching \ + --enable-chunked-prefill \ + --enforce-eager \ + --disable-log-requests \ + > "${LOG_DIR}/server.log" 2>&1 & +SERVER_PID=$! + +READY=0 +for _ in $(seq 1 180); do + if curl -fsS --max-time 2 "http://127.0.0.1:${SERVER_PORT}/v1/models" \ + > "${RESULT_DIR}/models.json" 2>/dev/null; then + READY=1 + break + fi + if ! kill -0 "${SERVER_PID}" 2>/dev/null; then + echo "ERROR: vLLM server exited before readiness" >&2 + tail -200 "${LOG_DIR}/server.log" >&2 || true + exit 1 + fi + sleep 5 +done +if [[ "${READY}" -ne 1 ]]; then + echo "ERROR: vLLM server did not become ready within 900 seconds" >&2 + tail -200 "${LOG_DIR}/server.log" >&2 || true + exit 1 +fi + +curl -fsS --max-time 120 \ + -H 'Content-Type: application/json' \ + -d '{"model":"qwen3-235b-community-smoke","prompt":"Hello","max_tokens":1,"temperature":0}' \ + "http://127.0.0.1:${SERVER_PORT}/v1/completions" \ + | tee "${RESULT_DIR}/one_request.json" +echo +jq -e '.choices | length == 1' "${RESULT_DIR}/one_request.json" >/dev/null + +cleanup +SERVER_PID="" + +find "${PROFILE_ROOT}" -type f -maxdepth 5 -print -exec sha256sum {} \; +sha256sum \ + "${RESULT_DIR}/allreduce_tp4.jsonl" \ + "${RESULT_DIR}/models.json" \ + "${RESULT_DIR}/one_request.json" \ + > "${RESULT_DIR}/results.sha256" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "GPU_SMOKE_COMPLETE" diff --git a/runs/frontier-multicase-sufficiency-v0/smoke/setup_env.sh b/runs/frontier-multicase-sufficiency-v0/smoke/setup_env.sh new file mode 100755 index 0000000..389a717 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/smoke/setup_env.sh @@ -0,0 +1,66 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:-/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715}" +FRONTIER_ROOT="${FRONTIER_ROOT:-${OUTPUT_ROOT}/Frontier-d9cfeb6}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh-frontier-vllm0102-smoke/.venv}" +LOG_DIR="${OUTPUT_ROOT}/logs" + +export UV_HTTP_TIMEOUT="${UV_HTTP_TIMEOUT:-300}" + +mkdir -p "${LOG_DIR}" "${OUTPUT_ROOT}/provenance" +exec > >(tee -a "${LOG_DIR}/setup_env.log") 2>&1 + +echo "SETUP_ENV output=${OUTPUT_ROOT} frontier=${FRONTIER_ROOT} venv=${VENV_ROOT} python=/usr/local/bin/python3.12 vllm=0.10.2 transformers=4.55.2" + +if [[ ! -f "${FRONTIER_ROOT}/pyproject.toml" ]]; then + echo "ERROR: Frontier source is missing at ${FRONTIER_ROOT}" >&2 + exit 1 +fi + +uv venv --clear --python /usr/local/bin/python3.12 "${VENV_ROOT}" +uv pip install \ + --python "${VENV_ROOT}/bin/python" \ + "vllm==0.10.2" \ + "transformers==4.55.2" \ + "flashinfer-python>=0.3,<0.4" \ + -e "${FRONTIER_ROOT}[test]" + +"${VENV_ROOT}/bin/python" - <<'PY' | tee "${OUTPUT_ROOT}/provenance/environment.json" +import importlib.metadata as metadata +import json +import platform + +import flashinfer +import frontier +import torch +import vllm + +record = { + "python": platform.python_version(), + "torch": torch.__version__, + "torch_cuda": torch.version.cuda, + "vllm_import_version": vllm.__version__, + "vllm_metadata_version": metadata.version("vllm"), + "vllm_path": vllm.__file__, + "transformers_metadata_version": metadata.version("transformers"), + "tokenizers_metadata_version": metadata.version("tokenizers"), + "flashinfer_metadata_version": metadata.version("flashinfer-python"), + "flashinfer_path": flashinfer.__file__, + "frontier_metadata_version": metadata.version("frontier-simulator"), + "frontier_path": frontier.__path__[0], +} +print(json.dumps(record, indent=2, sort_keys=True)) +assert record["vllm_import_version"] == "0.10.2", record +assert record["vllm_metadata_version"] == "0.10.2", record +assert record["transformers_metadata_version"] == "4.55.2", record +PY + +uv pip freeze --python "${VENV_ROOT}/bin/python" > "${OUTPUT_ROOT}/provenance/requirements.freeze.txt" +sha256sum \ + "${OUTPUT_ROOT}/provenance/environment.json" \ + "${OUTPUT_ROOT}/provenance/requirements.freeze.txt" \ + > "${OUTPUT_ROOT}/provenance/environment.sha256" + +echo "SETUP_ENV_COMPLETE" diff --git a/runs/frontier-multicase-sufficiency-v0/test_audit_ground_truth.py b/runs/frontier-multicase-sufficiency-v0/test_audit_ground_truth.py new file mode 100644 index 0000000..dba4365 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v0/test_audit_ground_truth.py @@ -0,0 +1,71 @@ +import importlib.util +import unittest +from pathlib import Path + + +MODULE_PATH = Path(__file__).with_name("audit_ground_truth.py") +SPEC = importlib.util.spec_from_file_location("audit_ground_truth", MODULE_PATH) +audit = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +SPEC.loader.exec_module(audit) + + +class AuditGroundTruthTest(unittest.TestCase): + def test_case_summary_exposes_tied_top_set(self): + rows = [ + { + "cell_id": "a", + "score_req_s_per_gpu": 2.0, + "capacity_lower_bound_req_s_per_gpu": 2.0, + "capacity_upper_bound_req_s_per_gpu": 2.1, + "fully_valid": True, + }, + { + "cell_id": "b", + "score_req_s_per_gpu": 2.0, + "capacity_lower_bound_req_s_per_gpu": 2.0, + "capacity_upper_bound_req_s_per_gpu": 2.2, + "fully_valid": True, + }, + { + "cell_id": "c", + "score_req_s_per_gpu": 1.0, + "capacity_lower_bound_req_s_per_gpu": 1.0, + "capacity_upper_bound_req_s_per_gpu": 1.5, + "fully_valid": False, + }, + ] + + summary = audit.summarize_case("test", rows) + + self.assertEqual(summary["top_set"], ["a", "b"]) + self.assertEqual(summary["distinct_score_count"], 2) + self.assertEqual(summary["tied_pair_count"], 1) + self.assertEqual(summary["informative_pair_count"], 2) + self.assertAlmostEqual(summary["random_top_set_hit_rate"], 2 / 3) + self.assertEqual(summary["invalid_cells"], ["c"]) + self.assertEqual( + summary["possibly_optimal_set_from_search_brackets"], ["a", "b"] + ) + + def test_config_gpu_count_includes_data_parallelism(self): + result = { + "config_patch": { + "flag_patch": { + "tensor-parallel-size": 2, + "data-parallel-size": 4, + "expert-parallel-size": 8, + "max-num-seqs": 128, + "max-num-batched-tokens": 384, + } + } + } + + config = audit.config_from_result(result) + + self.assertEqual(config["gpu_count"], 8) + self.assertEqual(audit.cell_id(config), "tp2_dp4_ep8_mns128_mbt384") + + +if __name__ == "__main__": + unittest.main() diff --git a/runs/frontier-multicase-sufficiency-v1/analyze_t0_surface.py b/runs/frontier-multicase-sufficiency-v1/analyze_t0_surface.py new file mode 100644 index 0000000..b1c6b67 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/analyze_t0_surface.py @@ -0,0 +1,321 @@ +#!/usr/bin/env python3 +"""Validate and compare frozen T0 Frontier and two-round real surfaces.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import math +from datetime import datetime +from pathlib import Path +from typing import Any + + +SLOS = ("tpot_40ms", "tpot_120ms", "tpot_150ms", "tpot_180ms") +RATE_LATTICE = (0.10, 0.20, 0.40, 0.80, 1.20, 1.60, 2.40, 3.20) + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def write_json(path: Path, payload: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + + +def rate_key(rate: float) -> str: + return f"r{rate:.2f}".replace(".", "p") + + +def find_real_result(roots: list[Path], name: str, round_id: int, rate: float) -> Path: + relative = Path(name) / f"round{round_id}/results" / f"{rate_key(rate)}.json" + matches = [root / relative for root in roots if (root / relative).is_file()] + if len(matches) != 1: + raise ValueError(f"expected one real result for {relative}, got {matches}") + return matches[0] + + +def capacity(loads: list[dict[str, Any]], slo: str, field: str) -> float | None: + values = [float(load["rate"]) for load in loads if bool(load[field][slo]["feasible"])] + return max(values) if values else None + + +def real_boundary_status(loads: list[dict[str, Any]], slo: str) -> str: + labels = { + float(load["rate"]): bool(load["real_conservative"][slo]["feasible"]) + for load in loads + } + ordered = [(rate, labels[rate]) for rate in RATE_LATTICE if rate in labels] + if any(not left and right for (_, left), (_, right) in zip(ordered, ordered[1:])): + return "non_monotonic_requires_full_lattice" + if len(labels) == len(RATE_LATTICE): + return "complete_lattice" + for lower, upper in zip(RATE_LATTICE, RATE_LATTICE[1:]): + if labels.get(lower) is True and labels.get(upper) is False: + return "adjacent_transition_bracketed" + if labels.get(RATE_LATTICE[-1]) is True: + return "upper_lattice_reached" + if labels.get(RATE_LATTICE[0]) is False and not any(labels.values()): + return "lowest_anchor_infeasible" + return "unbracketed_requires_expansion" + + +def kendall_tau_b(left: list[float], right: list[float]) -> float | None: + concordant = discordant = left_ties = right_ties = 0 + for i in range(len(left)): + for j in range(i + 1, len(left)): + x = (left[i] > left[j]) - (left[i] < left[j]) + y = (right[i] > right[j]) - (right[i] < right[j]) + if x == 0 and y == 0: + continue + if x == 0: + left_ties += 1 + elif y == 0: + right_ties += 1 + elif x == y: + concordant += 1 + else: + discordant += 1 + denominator = math.sqrt( + (concordant + discordant + left_ties) + * (concordant + discordant + right_ties) + ) + return (concordant - discordant) / denominator if denominator else None + + +def sign(left: float, right: float) -> int: + return (left > right) - (left < right) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--frontier-freeze", type=Path, required=True) + parser.add_argument("--real-plan", type=Path, required=True) + parser.add_argument("--real-root", type=Path, action="append", required=True) + parser.add_argument("--output-root", type=Path, required=True) + args = parser.parse_args() + + freeze = json.loads(args.frontier_freeze.read_text()) + plan = json.loads(args.real_plan.read_text()) + real_roots = [root.resolve() for root in args.real_root] + if freeze.get("status") != "frozen_before_real_surface" or len(freeze.get("config_results") or []) != 8: + raise ValueError("Frontier freeze is incomplete") + if plan.get("frontier_freeze", {}).get("sha256") != sha256(args.frontier_freeze): + raise ValueError("real plan does not point to this Frontier freeze") + sim_by_name = {item["config"]["name"]: item for item in freeze["config_results"]} + + cells = [] + for cell in plan["cells"]: + config = cell["config"] + name = config["name"] + sim_loads = {float(load["offered_request_rate"]): load for load in sim_by_name[name]["loads"]} + loads = [] + for rate in cell["rates"]: + round_summaries = [] + files = [] + for round_id in (1, 2): + path = find_real_result(real_roots, name, round_id, float(rate)) + payload = json.loads(path.read_text()) + if payload.get("schema") != "qwen235b-t0-rate-anchor-v1": + raise ValueError(f"bad real result schema: {path}") + if payload["summary"]["completed"] != 64 or payload["summary"]["failed"] != 0: + raise ValueError(f"incomplete real anchor: {path}") + if float(payload["workload"]["offered_request_rate"]) != float(rate): + raise ValueError(f"offered-rate drift: {path}") + round_summaries.append(payload["summary"]["slos"]) + files.append({"path": str(path), "sha256": sha256(path)}) + conservative = { + slo: { + "feasible": all(summary[slo]["feasible"] for summary in round_summaries), + "round_pass_rates": [summary[slo]["pass_rate"] for summary in round_summaries], + } + for slo in SLOS + } + loads.append( + { + "rate": float(rate), + "real_conservative": conservative, + "sim": sim_loads[float(rate)]["slos"], + "real_files": files, + } + ) + cells.append({"config": config, "loads": loads}) + + comparisons = {} + for slo in SLOS: + records = [] + for cell in cells: + real = capacity(cell["loads"], slo, "real_conservative") + sim_values = [ + float(load["offered_request_rate"]) + for load in sim_by_name[cell["config"]["name"]]["loads"] + if bool(load["slos"][slo]["feasible"]) + ] + sim = max(sim_values) if sim_values else None + tp = int(cell["config"]["tp"]) + boundary = real_boundary_status(cell["loads"], slo) + records.append( + { + "config": cell["config"], + "real_capacity": real, + "sim_capacity": sim, + "real_capacity_per_gpu": real / tp if real is not None else None, + "sim_capacity_per_gpu": sim / tp if sim is not None else None, + "real_boundary_status": boundary, + "expansion_required": boundary in { + "non_monotonic_requires_full_lattice", + "unbracketed_requires_expansion", + }, + } + ) + rankable = [ + row + for row in records + if not row["expansion_required"] + and row["real_capacity_per_gpu"] is not None + and row["sim_capacity_per_gpu"] is not None + ] + tau = kendall_tau_b( + [row["real_capacity_per_gpu"] for row in rankable], + [row["sim_capacity_per_gpu"] for row in rankable], + ) + pairwise = { + "all_pairs": 0, + "exact_sign_matches": 0, + "real_non_tie_pairs": 0, + "real_non_tie_direction_matches": 0, + } + for i, left in enumerate(rankable): + for right in rankable[i + 1 :]: + real_sign = sign(left["real_capacity_per_gpu"], right["real_capacity_per_gpu"]) + sim_sign = sign(left["sim_capacity_per_gpu"], right["sim_capacity_per_gpu"]) + pairwise["all_pairs"] += 1 + pairwise["exact_sign_matches"] += real_sign == sim_sign + if real_sign: + pairwise["real_non_tie_pairs"] += 1 + pairwise["real_non_tie_direction_matches"] += real_sign == sim_sign + real_best = max((row["real_capacity_per_gpu"] for row in records if row["real_capacity_per_gpu"] is not None), default=None) + sim_best = max((row["sim_capacity_per_gpu"] for row in records if row["sim_capacity_per_gpu"] is not None), default=None) + sim_top = [row for row in records if sim_best is not None and row["sim_capacity_per_gpu"] == sim_best] + real_top = [row for row in records if real_best is not None and row["real_capacity_per_gpu"] == real_best] + optimistic_regret = worst_regret = None + if real_best is not None and sim_top and all(row["real_capacity_per_gpu"] is not None for row in sim_top): + regrets = [(real_best - row["real_capacity_per_gpu"]) / real_best for row in sim_top] + optimistic_regret = min(regrets) + worst_regret = max(regrets) + confusion = {"anchors": 0, "match": 0, "false_feasible": 0, "false_infeasible": 0} + for cell in cells: + for load in cell["loads"]: + real_feasible = bool(load["real_conservative"][slo]["feasible"]) + sim_feasible = bool(load["sim"][slo]["feasible"]) + confusion["anchors"] += 1 + confusion["match"] += real_feasible == sim_feasible + confusion["false_feasible"] += sim_feasible and not real_feasible + confusion["false_infeasible"] += real_feasible and not sim_feasible + comparisons[slo] = { + "records": records, + "kendall_tau_b": tau, + "pairwise": pairwise, + "anchor_confusion": confusion, + "real_top_set": [row["config"]["name"] for row in real_top], + "sim_top_set": [row["config"]["name"] for row in sim_top], + "top_set_intersection": sorted( + {row["config"]["name"] for row in real_top} + & {row["config"]["name"] for row in sim_top} + ), + "top_set_exact_match": { + row["config"]["name"] for row in real_top + } == {row["config"]["name"] for row in sim_top}, + "optimistic_regret": optimistic_regret, + "worst_tie_break_regret": worst_regret, + } + + run_costs = [] + for root in real_roots: + config_names = [child.name for child in root.iterdir() if child.is_dir() and child.name in sim_by_name] + if len(config_names) != 1: + raise ValueError(f"expected one config directory in real root {root}, got {config_names}") + config = sim_by_name[config_names[0]]["config"] + remote_run = root.parents[2] / "remote_run" + started = datetime.fromisoformat((remote_run / "started_at").read_text().strip()) + finished = datetime.fromisoformat((remote_run / "finished_at").read_text().strip()) + wall_seconds = (finished - started).total_seconds() + run_costs.append( + { + "run_id": root.parents[2].name, + "config": config_names[0], + "gpu_count": int(config["tp"]), + "wall_seconds": wall_seconds, + "h20_gpu_hours": wall_seconds * int(config["tp"]) / 3600, + "started_at": started.isoformat(), + "finished_at": finished.isoformat(), + } + ) + fresh_server_anchors = 2 * sum(len(cell["loads"]) for cell in cells) + real_execution_cost = { + "accepted_fleet_jobs": len(run_costs), + "fresh_server_anchors": fresh_server_anchors, + "measured_requests": fresh_server_anchors * 64, + "warmup_requests": 2 + * sum( + min(32, max(4, math.ceil(float(load["rate"]) * 20))) + for cell in cells + for load in cell["loads"] + ), + "accepted_h20_gpu_hours": sum(run["h20_gpu_hours"] for run in run_costs), + "campaign_wall_span_seconds": ( + max(datetime.fromisoformat(run["finished_at"]) for run in run_costs) + - min(datetime.fromisoformat(run["started_at"]) for run in run_costs) + ).total_seconds(), + "runs": run_costs, + } + + output = { + "schema": "qwen235b-t0-simulator-real-comparison-v1", + "frontier_freeze_sha256": sha256(args.frontier_freeze), + "real_plan_sha256": sha256(args.real_plan), + "real_execution_cost": real_execution_cost, + "cells": cells, + "comparisons": comparisons, + } + args.output_root.mkdir(parents=True, exist_ok=True) + write_json(args.output_root / "comparison.json", output) + with (args.output_root / "capacity.csv").open("w", newline="") as target: + writer = csv.DictWriter( + target, + fieldnames=[ + "slo", + "config", + "tp", + "mns", + "mbt", + "real_capacity_per_gpu", + "sim_capacity_per_gpu", + "real_boundary_status", + "expansion_required", + ], + ) + writer.writeheader() + for slo, comparison in comparisons.items(): + for row in comparison["records"]: + writer.writerow( + { + "slo": slo, + "config": row["config"]["name"], + "tp": row["config"]["tp"], + "mns": row["config"]["mns"], + "mbt": row["config"]["mbt"], + "real_capacity_per_gpu": row["real_capacity_per_gpu"], + "sim_capacity_per_gpu": row["sim_capacity_per_gpu"], + "real_boundary_status": row["real_boundary_status"], + "expansion_required": row["expansion_required"], + } + ) + print(args.output_root / "comparison.json") + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/artifacts/frontier-t0-surface-v1/frontier_surface_frozen.json b/runs/frontier-multicase-sufficiency-v1/artifacts/frontier-t0-surface-v1/frontier_surface_frozen.json new file mode 100644 index 0000000..eb0bb5f --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/artifacts/frontier-t0-surface-v1/frontier_surface_frozen.json @@ -0,0 +1,27855 @@ +{ + "config_results": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 190.62253308296204, + "offered_request_rate": 0.1, + "request_metrics_sha256": "b45cd3a473a98bee1127b2315d0c0c59ded580275f6e6d10a3e5a52d85d24de4", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 11656.879780530655, + "request_id": 0, + "tpot_ms": 89.67953789863164, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 11729.945143691864, + "request_id": 1, + "tpot_ms": 89.99959080602144, + "ttft_ms": 299.9971113271407 + }, + { + "e2e_ms": 11712.730146443742, + "request_id": 2, + "tpot_ms": 90.00056214193414, + "ttft_ms": 282.65875441810806 + }, + { + "e2e_ms": 11785.795509604945, + "request_id": 3, + "tpot_ms": 89.99959080602139, + "ttft_ms": 355.8474772402285 + }, + { + "e2e_ms": 11770.95360336775, + "request_id": 4, + "tpot_ms": 90.01924789792568, + "ttft_ms": 338.50912033118874 + }, + { + "e2e_ms": 11756.111697130556, + "request_id": 5, + "tpot_ms": 90.0192478979257, + "ttft_ms": 323.66721409399446 + }, + { + "e2e_ms": 11741.269790893028, + "request_id": 6, + "tpot_ms": 90.01924789792305, + "ttft_ms": 308.8253078568002 + }, + { + "e2e_ms": 11726.427884655508, + "request_id": 7, + "tpot_ms": 90.01924789792311, + "ttft_ms": 293.98340161927194 + }, + { + "e2e_ms": 11709.212887407048, + "request_id": 8, + "tpot_ms": 90.00056214193147, + "ttft_ms": 279.1414953817508 + }, + { + "e2e_ms": 11782.278250567928, + "request_id": 9, + "tpot_ms": 89.99959080601886, + "ttft_ms": 352.3302182035337 + }, + { + "e2e_ms": 11767.436344330406, + "request_id": 10, + "tpot_ms": 90.01924789792311, + "ttft_ms": 334.9918612941707 + }, + { + "e2e_ms": 11752.594438092885, + "request_id": 11, + "tpot_ms": 90.01924789792311, + "ttft_ms": 320.14995505664956 + }, + { + "e2e_ms": 11737.752531855705, + "request_id": 12, + "tpot_ms": 90.01924789792581, + "ttft_ms": 305.30804881912843 + }, + { + "e2e_ms": 11722.91062561817, + "request_id": 13, + "tpot_ms": 90.019247897923, + "ttft_ms": 290.46614258194836 + }, + { + "e2e_ms": 11705.69562836971, + "request_id": 14, + "tpot_ms": 90.00056214193148, + "ttft_ms": 275.624236344413 + }, + { + "e2e_ms": 11778.76099153056, + "request_id": 15, + "tpot_ms": 89.99959080601862, + "ttft_ms": 348.8129591661959 + }, + { + "e2e_ms": 11763.919085293026, + "request_id": 16, + "tpot_ms": 90.019247897923, + "ttft_ms": 331.4746022568045 + }, + { + "e2e_ms": 11749.07717905549, + "request_id": 17, + "tpot_ms": 90.019247897923, + "ttft_ms": 316.63269601926913 + }, + { + "e2e_ms": 11734.235272817954, + "request_id": 18, + "tpot_ms": 90.019247897923, + "ttft_ms": 301.7907897817338 + }, + { + "e2e_ms": 11719.39336658042, + "request_id": 19, + "tpot_ms": 90.019247897923, + "ttft_ms": 286.94888354419845 + }, + { + "e2e_ms": 11702.178369331961, + "request_id": 20, + "tpot_ms": 90.00056214193148, + "ttft_ms": 272.1069773066631 + }, + { + "e2e_ms": 11775.24373249281, + "request_id": 21, + "tpot_ms": 89.99959080601862, + "ttft_ms": 345.295700128446 + }, + { + "e2e_ms": 11760.401826255276, + "request_id": 22, + "tpot_ms": 90.01924789792301, + "ttft_ms": 327.95734321905456 + }, + { + "e2e_ms": 11745.55992001774, + "request_id": 23, + "tpot_ms": 90.019247897923, + "ttft_ms": 313.1154369815192 + }, + { + "e2e_ms": 11730.718013780204, + "request_id": 24, + "tpot_ms": 90.019247897923, + "ttft_ms": 298.2735307439839 + }, + { + "e2e_ms": 11713.50301653041, + "request_id": 25, + "tpot_ms": 90.00056214192095, + "ttft_ms": 283.43162450644854 + }, + { + "e2e_ms": 11786.568379689925, + "request_id": 26, + "tpot_ms": 89.9995908060081, + "ttft_ms": 356.6203473268956 + }, + { + "e2e_ms": 11771.726473451054, + "request_id": 27, + "tpot_ms": 90.01924789791249, + "ttft_ms": 339.28199041616836 + }, + { + "e2e_ms": 11756.884567212182, + "request_id": 28, + "tpot_ms": 90.01924789791249, + "ttft_ms": 324.4400841772972 + }, + { + "e2e_ms": 11742.042660973311, + "request_id": 29, + "tpot_ms": 90.01924789791249, + "ttft_ms": 309.59817793842603 + }, + { + "e2e_ms": 11727.20075473444, + "request_id": 30, + "tpot_ms": 90.01924789791249, + "ttft_ms": 294.7562716995549 + }, + { + "e2e_ms": 11709.985757484617, + "request_id": 31, + "tpot_ms": 90.00056214192074, + "ttft_ms": 279.9143654606837 + }, + { + "e2e_ms": 11783.051120644132, + "request_id": 32, + "tpot_ms": 89.9995908060081, + "ttft_ms": 353.1030882811024 + }, + { + "e2e_ms": 11768.20921440526, + "request_id": 33, + "tpot_ms": 90.01924789791249, + "ttft_ms": 335.7647313703751 + }, + { + "e2e_ms": 11753.36730816639, + "request_id": 34, + "tpot_ms": 90.01924789791249, + "ttft_ms": 320.92282513150394 + }, + { + "e2e_ms": 11738.525401927518, + "request_id": 35, + "tpot_ms": 90.01924789791249, + "ttft_ms": 306.0809188926328 + }, + { + "e2e_ms": 11723.683495688647, + "request_id": 36, + "tpot_ms": 90.01924789791249, + "ttft_ms": 291.2390126537616 + }, + { + "e2e_ms": 11706.468498438824, + "request_id": 37, + "tpot_ms": 90.00056214192074, + "ttft_ms": 276.39710641489046 + }, + { + "e2e_ms": 11779.533861598338, + "request_id": 38, + "tpot_ms": 89.9995908060081, + "ttft_ms": 349.5858292353091 + }, + { + "e2e_ms": 11764.691955359467, + "request_id": 39, + "tpot_ms": 90.01924789791249, + "ttft_ms": 332.24747232458185 + }, + { + "e2e_ms": 11749.850049120596, + "request_id": 40, + "tpot_ms": 90.01924789791249, + "ttft_ms": 317.4055660857107 + }, + { + "e2e_ms": 11735.008142881725, + "request_id": 41, + "tpot_ms": 90.01924789791249, + "ttft_ms": 302.5636598468395 + }, + { + "e2e_ms": 11720.166236642854, + "request_id": 42, + "tpot_ms": 90.01924789791249, + "ttft_ms": 287.72175360796837 + }, + { + "e2e_ms": 11702.95123939303, + "request_id": 43, + "tpot_ms": 90.00056214192074, + "ttft_ms": 272.8798473690972 + }, + { + "e2e_ms": 11776.016602552545, + "request_id": 44, + "tpot_ms": 89.9995908060081, + "ttft_ms": 346.06857018951587 + }, + { + "e2e_ms": 11761.174696313674, + "request_id": 45, + "tpot_ms": 90.01924789791249, + "ttft_ms": 328.7302132787886 + }, + { + "e2e_ms": 11746.332790074803, + "request_id": 46, + "tpot_ms": 90.01924789791249, + "ttft_ms": 313.88830703991744 + }, + { + "e2e_ms": 11731.490883835932, + "request_id": 47, + "tpot_ms": 90.01924789791249, + "ttft_ms": 299.0464008010463 + }, + { + "e2e_ms": 11716.64897759706, + "request_id": 48, + "tpot_ms": 90.01924789791249, + "ttft_ms": 284.2044945621751 + }, + { + "e2e_ms": 11699.433980347238, + "request_id": 49, + "tpot_ms": 90.00056214192074, + "ttft_ms": 269.36258832330395 + }, + { + "e2e_ms": 11772.499343506752, + "request_id": 50, + "tpot_ms": 89.9995908060081, + "ttft_ms": 342.5513111437226 + }, + { + "e2e_ms": 11757.657437270042, + "request_id": 51, + "tpot_ms": 90.0192478979295, + "ttft_ms": 325.21295423299534 + }, + { + "e2e_ms": 11742.815531033444, + "request_id": 52, + "tpot_ms": 90.01924789792636, + "ttft_ms": 310.3710479967958 + }, + { + "e2e_ms": 11727.973624796847, + "request_id": 53, + "tpot_ms": 90.01924789792636, + "ttft_ms": 295.5291417601984 + }, + { + "e2e_ms": 11710.758627549352, + "request_id": 54, + "tpot_ms": 90.00056214193505, + "ttft_ms": 280.687235523601 + }, + { + "e2e_ms": 11783.823990711198, + "request_id": 55, + "tpot_ms": 89.99959080602287, + "ttft_ms": 353.8759583462934 + }, + { + "e2e_ms": 11768.982084474601, + "request_id": 56, + "tpot_ms": 90.01924789792636, + "ttft_ms": 336.5376014379535 + }, + { + "e2e_ms": 11754.140178238004, + "request_id": 57, + "tpot_ms": 90.01924789792636, + "ttft_ms": 321.6956952013561 + }, + { + "e2e_ms": 11739.298272001406, + "request_id": 58, + "tpot_ms": 90.01924789792636, + "ttft_ms": 306.8537889647587 + }, + { + "e2e_ms": 11724.456365764809, + "request_id": 59, + "tpot_ms": 90.01924789792636, + "ttft_ms": 292.01188272816125 + }, + { + "e2e_ms": 11707.241368517316, + "request_id": 60, + "tpot_ms": 90.00056214193505, + "ttft_ms": 277.16997649156383 + }, + { + "e2e_ms": 11780.30673167916, + "request_id": 61, + "tpot_ms": 89.99959080602287, + "ttft_ms": 350.3586993142562 + }, + { + "e2e_ms": 11765.464825442563, + "request_id": 62, + "tpot_ms": 90.01924789792635, + "ttft_ms": 333.0203424059164 + }, + { + "e2e_ms": 11530.943387303068, + "request_id": 63, + "tpot_ms": 88.28948780420275, + "ttft_ms": 318.17843616931896 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 27.596914052963257, + "offered_request_rate": 0.2, + "request_metrics_sha256": "7930ffb53e440b9eefa724912c777842a9c33cefdb46d9014e1c5003223221d2", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 11975.67900227787, + "request_id": 0, + "tpot_ms": 92.18976799112939, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12131.199338988026, + "request_id": 1, + "tpot_ms": 93.2890801728389, + "ttft_ms": 283.4861570374869 + }, + { + "e2e_ms": 12196.411822684866, + "request_id": 2, + "tpot_ms": 93.31228634935276, + "ttft_ms": 345.75145631706584 + }, + { + "e2e_ms": 12171.220586311463, + "request_id": 3, + "tpot_ms": 93.31547315047263, + "ttft_ms": 320.15549620144054 + }, + { + "e2e_ms": 12145.933482881133, + "request_id": 4, + "tpot_ms": 93.31640326087744, + "ttft_ms": 294.75026874969853 + }, + { + "e2e_ms": 12120.646379450847, + "request_id": 5, + "tpot_ms": 93.31564840216181, + "ttft_ms": 269.5590323762964 + }, + { + "e2e_ms": 12185.858863147729, + "request_id": 6, + "tpot_ms": 93.31471829175732, + "ttft_ms": 334.88964009454844 + }, + { + "e2e_ms": 12160.571759717443, + "request_id": 7, + "tpot_ms": 93.31471829175764, + "ttft_ms": 309.6025366642223 + }, + { + "e2e_ms": 12135.284656287155, + "request_id": 8, + "tpot_ms": 93.31564840216245, + "ttft_ms": 284.19730921252295 + }, + { + "e2e_ms": 12200.497139984038, + "request_id": 9, + "tpot_ms": 93.31471829175764, + "ttft_ms": 349.52791693081764 + }, + { + "e2e_ms": 12175.305903610677, + "request_id": 10, + "tpot_ms": 93.3154731504736, + "ttft_ms": 324.2408135005306 + }, + { + "e2e_ms": 12150.018800180305, + "request_id": 11, + "tpot_ms": 93.31640326087775, + "ttft_ms": 298.8355860488312 + }, + { + "e2e_ms": 12124.731696749934, + "request_id": 12, + "tpot_ms": 93.31564840216113, + "ttft_ms": 273.6443496754717 + }, + { + "e2e_ms": 12189.94418044673, + "request_id": 13, + "tpot_ms": 93.31471829175568, + "ttft_ms": 338.97495739375927 + }, + { + "e2e_ms": 12164.657077016358, + "request_id": 14, + "tpot_ms": 93.31471829175568, + "ttft_ms": 313.68785396338694 + }, + { + "e2e_ms": 12139.369973585985, + "request_id": 15, + "tpot_ms": 93.31564840216049, + "ttft_ms": 288.2826265116023 + }, + { + "e2e_ms": 12204.582457282782, + "request_id": 16, + "tpot_ms": 93.31471829175568, + "ttft_ms": 353.6132342298117 + }, + { + "e2e_ms": 12179.391220909338, + "request_id": 17, + "tpot_ms": 93.31547315047163, + "ttft_ms": 328.3261307994394 + }, + { + "e2e_ms": 12154.104117478966, + "request_id": 18, + "tpot_ms": 93.31640326087647, + "ttft_ms": 302.92090334765476 + }, + { + "e2e_ms": 12128.817014048593, + "request_id": 19, + "tpot_ms": 93.31564840216049, + "ttft_ms": 277.72966697421 + }, + { + "e2e_ms": 12194.02949774539, + "request_id": 20, + "tpot_ms": 93.31471829175568, + "ttft_ms": 343.0602746924194 + }, + { + "e2e_ms": 12168.742394315017, + "request_id": 21, + "tpot_ms": 93.31471829175567, + "ttft_ms": 317.7731712620471 + }, + { + "e2e_ms": 12143.455290884645, + "request_id": 22, + "tpot_ms": 93.31564840216049, + "ttft_ms": 292.36794381026243 + }, + { + "e2e_ms": 12208.667774581443, + "request_id": 23, + "tpot_ms": 93.31471829175568, + "ttft_ms": 357.69855152847185 + }, + { + "e2e_ms": 12183.476538207999, + "request_id": 24, + "tpot_ms": 93.31547315047165, + "ttft_ms": 332.4114480980995 + }, + { + "e2e_ms": 12158.189434777625, + "request_id": 25, + "tpot_ms": 93.31640326087646, + "ttft_ms": 307.0062206463149 + }, + { + "e2e_ms": 12132.902331347253, + "request_id": 26, + "tpot_ms": 93.31564840216049, + "ttft_ms": 281.8149842728701 + }, + { + "e2e_ms": 12198.11481504405, + "request_id": 27, + "tpot_ms": 93.31471829175567, + "ttft_ms": 347.1455919910795 + }, + { + "e2e_ms": 12172.923578670605, + "request_id": 28, + "tpot_ms": 93.31547315047163, + "ttft_ms": 321.8584885607072 + }, + { + "e2e_ms": 12147.636475240233, + "request_id": 29, + "tpot_ms": 93.31640326087646, + "ttft_ms": 296.45326110892256 + }, + { + "e2e_ms": 12122.349371809862, + "request_id": 30, + "tpot_ms": 93.3156484021605, + "ttft_ms": 271.2620247354778 + }, + { + "e2e_ms": 12187.561855506658, + "request_id": 31, + "tpot_ms": 93.31471829175568, + "ttft_ms": 336.5926324536872 + }, + { + "e2e_ms": 12162.274752076286, + "request_id": 32, + "tpot_ms": 93.31471829175568, + "ttft_ms": 311.3055290233149 + }, + { + "e2e_ms": 12136.987648645914, + "request_id": 33, + "tpot_ms": 93.3156484021605, + "ttft_ms": 285.90030157153024 + }, + { + "e2e_ms": 12202.20013234271, + "request_id": 34, + "tpot_ms": 93.31471829175568, + "ttft_ms": 351.23090928973966 + }, + { + "e2e_ms": 12177.008895969266, + "request_id": 35, + "tpot_ms": 93.31547315047163, + "ttft_ms": 325.94380585936733 + }, + { + "e2e_ms": 12151.721792538894, + "request_id": 36, + "tpot_ms": 93.31640326087647, + "ttft_ms": 300.5385784075827 + }, + { + "e2e_ms": 12126.43468910852, + "request_id": 37, + "tpot_ms": 93.31564840216049, + "ttft_ms": 275.3473420341379 + }, + { + "e2e_ms": 12191.647172805318, + "request_id": 38, + "tpot_ms": 93.31471829175568, + "ttft_ms": 340.67794975234733 + }, + { + "e2e_ms": 12166.360069374947, + "request_id": 39, + "tpot_ms": 93.3147182917557, + "ttft_ms": 315.390846321975 + }, + { + "e2e_ms": 12141.072965944573, + "request_id": 40, + "tpot_ms": 93.31564840216049, + "ttft_ms": 289.98561887019036 + }, + { + "e2e_ms": 12206.285449641371, + "request_id": 41, + "tpot_ms": 93.31471829175568, + "ttft_ms": 355.3162265883998 + }, + { + "e2e_ms": 12181.094213267927, + "request_id": 42, + "tpot_ms": 93.31547315047165, + "ttft_ms": 330.02912315802746 + }, + { + "e2e_ms": 12155.807109837553, + "request_id": 43, + "tpot_ms": 93.31640326087646, + "ttft_ms": 304.6238957062428 + }, + { + "e2e_ms": 12130.520006407181, + "request_id": 44, + "tpot_ms": 93.31564840216049, + "ttft_ms": 279.43265933279804 + }, + { + "e2e_ms": 12195.73249010398, + "request_id": 45, + "tpot_ms": 93.3147182917557, + "ttft_ms": 344.76326705100746 + }, + { + "e2e_ms": 12170.445386673606, + "request_id": 46, + "tpot_ms": 93.31471829175568, + "ttft_ms": 319.47616362063513 + }, + { + "e2e_ms": 12145.158283243234, + "request_id": 47, + "tpot_ms": 93.31564840216049, + "ttft_ms": 294.0709361688505 + }, + { + "e2e_ms": 12210.37076694003, + "request_id": 48, + "tpot_ms": 93.31471829175568, + "ttft_ms": 359.4015438870599 + }, + { + "e2e_ms": 12185.179530566586, + "request_id": 49, + "tpot_ms": 93.31547315047163, + "ttft_ms": 334.1144404566876 + }, + { + "e2e_ms": 12159.892427136241, + "request_id": 50, + "tpot_ms": 93.31640326087668, + "ttft_ms": 308.70921300490295 + }, + { + "e2e_ms": 12134.605323705899, + "request_id": 51, + "tpot_ms": 93.31564840216095, + "ttft_ms": 283.51797663145817 + }, + { + "e2e_ms": 12199.817807402724, + "request_id": 52, + "tpot_ms": 93.31471829175612, + "ttft_ms": 348.848584349696 + }, + { + "e2e_ms": 12174.626571029308, + "request_id": 53, + "tpot_ms": 93.3154731504721, + "ttft_ms": 323.5614809193521 + }, + { + "e2e_ms": 12149.339467598964, + "request_id": 54, + "tpot_ms": 93.31640326087691, + "ttft_ms": 298.1562534675959 + }, + { + "e2e_ms": 12124.05236416862, + "request_id": 55, + "tpot_ms": 93.31564840216095, + "ttft_ms": 272.96501709417953 + }, + { + "e2e_ms": 12189.264847865445, + "request_id": 56, + "tpot_ms": 93.31471829175612, + "ttft_ms": 338.2956248124174 + }, + { + "e2e_ms": 12163.977744435102, + "request_id": 57, + "tpot_ms": 93.31471829175614, + "ttft_ms": 313.00852138207347 + }, + { + "e2e_ms": 12138.690641004758, + "request_id": 58, + "tpot_ms": 93.31564840216095, + "ttft_ms": 287.60329393031725 + }, + { + "e2e_ms": 12203.903124701583, + "request_id": 59, + "tpot_ms": 93.31471829175612, + "ttft_ms": 352.9339016485551 + }, + { + "e2e_ms": 12178.711888328166, + "request_id": 60, + "tpot_ms": 93.31547315047209, + "ttft_ms": 327.6467982182112 + }, + { + "e2e_ms": 12153.424784897823, + "request_id": 61, + "tpot_ms": 93.31640326087691, + "ttft_ms": 302.24157076645497 + }, + { + "e2e_ms": 11949.074413391485, + "request_id": 62, + "tpot_ms": 91.90570140943657, + "ttft_ms": 277.0503343930386 + }, + { + "e2e_ms": 11702.728382593137, + "request_id": 63, + "tpot_ms": 89.45155464946347, + "ttft_ms": 342.38094211127645 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 22.99045443534851, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e62567aa33ec429331fcd9f24053a0c79db61ef288f7994e6488ae0f0fa79b7e", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 12394.3405299904, + "request_id": 0, + "tpot_ms": 95.48631545343278, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12520.094438712134, + "request_id": 1, + "tpot_ms": 96.04733720338449, + "ttft_ms": 322.08261388230494 + }, + { + "e2e_ms": 12556.880443424032, + "request_id": 2, + "tpot_ms": 96.10807445129032, + "ttft_ms": 351.1549881101628 + }, + { + "e2e_ms": 12502.892311945501, + "request_id": 3, + "tpot_ms": 96.14618123456218, + "ttft_ms": 292.32729515610424 + }, + { + "e2e_ms": 12539.549613984442, + "request_id": 4, + "tpot_ms": 96.30977229626299, + "ttft_ms": 308.2085323590427 + }, + { + "e2e_ms": 12576.335618696337, + "request_id": 5, + "tpot_ms": 96.32399459689755, + "ttft_ms": 343.188304890349 + }, + { + "e2e_ms": 12522.347487217807, + "request_id": 6, + "tpot_ms": 96.32399459689753, + "ttft_ms": 289.200173411821 + }, + { + "e2e_ms": 12559.133491929704, + "request_id": 7, + "tpot_ms": 96.32399459689753, + "ttft_ms": 325.98617812371697 + }, + { + "e2e_ms": 12505.145360451173, + "request_id": 8, + "tpot_ms": 96.32399459689753, + "ttft_ms": 271.99804664518723 + }, + { + "e2e_ms": 12541.802662490163, + "request_id": 9, + "tpot_ms": 96.32399459689792, + "ttft_ms": 308.6553486841268 + }, + { + "e2e_ms": 12578.588667202104, + "request_id": 10, + "tpot_ms": 96.32399459689827, + "ttft_ms": 345.4413533960228 + }, + { + "e2e_ms": 12524.60053572362, + "request_id": 11, + "tpot_ms": 96.32399459689864, + "ttft_ms": 291.453221917493 + }, + { + "e2e_ms": 12561.386540435564, + "request_id": 12, + "tpot_ms": 96.32399459689903, + "ttft_ms": 328.239226629389 + }, + { + "e2e_ms": 12507.39840895708, + "request_id": 13, + "tpot_ms": 96.32399459689938, + "ttft_ms": 274.2510951508592 + }, + { + "e2e_ms": 12544.05571099607, + "request_id": 14, + "tpot_ms": 96.32399459689938, + "ttft_ms": 310.90839718984853 + }, + { + "e2e_ms": 12580.841715708011, + "request_id": 15, + "tpot_ms": 96.32399459689938, + "ttft_ms": 347.6944019017907 + }, + { + "e2e_ms": 12526.853584229528, + "request_id": 16, + "tpot_ms": 96.32399459689938, + "ttft_ms": 293.7062704233071 + }, + { + "e2e_ms": 12563.63958894147, + "request_id": 17, + "tpot_ms": 96.32399459689938, + "ttft_ms": 330.49227513524926 + }, + { + "e2e_ms": 12509.651457462987, + "request_id": 18, + "tpot_ms": 96.32399459689938, + "ttft_ms": 276.5041436567657 + }, + { + "e2e_ms": 12546.308759501975, + "request_id": 19, + "tpot_ms": 96.32399459689938, + "ttft_ms": 313.161445695755 + }, + { + "e2e_ms": 12583.094764213918, + "request_id": 20, + "tpot_ms": 96.32399459689938, + "ttft_ms": 349.94745040769715 + }, + { + "e2e_ms": 12529.106632735506, + "request_id": 21, + "tpot_ms": 96.32399459689994, + "ttft_ms": 295.95931892921357 + }, + { + "e2e_ms": 12565.892637447447, + "request_id": 22, + "tpot_ms": 96.32399459689992, + "ttft_ms": 332.7453236411557 + }, + { + "e2e_ms": 12511.904505968956, + "request_id": 23, + "tpot_ms": 96.32399459689988, + "ttft_ms": 278.75719216267214 + }, + { + "e2e_ms": 12548.561808007931, + "request_id": 24, + "tpot_ms": 96.32399459689977, + "ttft_ms": 315.41449420166146 + }, + { + "e2e_ms": 12585.347812719874, + "request_id": 25, + "tpot_ms": 96.32399459689977, + "ttft_ms": 352.2004989136036 + }, + { + "e2e_ms": 12531.359681241383, + "request_id": 26, + "tpot_ms": 96.3239945968991, + "ttft_ms": 298.2123674351982 + }, + { + "e2e_ms": 12568.145685953326, + "request_id": 27, + "tpot_ms": 96.32399459689911, + "ttft_ms": 334.99837214714034 + }, + { + "e2e_ms": 12514.157554474836, + "request_id": 28, + "tpot_ms": 96.32399459689911, + "ttft_ms": 281.01024066864966 + }, + { + "e2e_ms": 12550.81485651381, + "request_id": 29, + "tpot_ms": 96.32399459689911, + "ttft_ms": 317.66754270762476 + }, + { + "e2e_ms": 12587.600861225752, + "request_id": 30, + "tpot_ms": 96.3239945968991, + "ttft_ms": 354.4535474195669 + }, + { + "e2e_ms": 12533.612729747261, + "request_id": 31, + "tpot_ms": 96.3239945968991, + "ttft_ms": 300.46541594107623 + }, + { + "e2e_ms": 12570.398734459204, + "request_id": 32, + "tpot_ms": 96.32399459689911, + "ttft_ms": 337.2514206530184 + }, + { + "e2e_ms": 12516.410602980714, + "request_id": 33, + "tpot_ms": 96.32399459689911, + "ttft_ms": 283.2632891745277 + }, + { + "e2e_ms": 12553.067905019689, + "request_id": 34, + "tpot_ms": 96.32399459689911, + "ttft_ms": 319.9205912135028 + }, + { + "e2e_ms": 12589.85390973163, + "request_id": 35, + "tpot_ms": 96.3239945968991, + "ttft_ms": 356.70659592544496 + }, + { + "e2e_ms": 12535.865778253139, + "request_id": 36, + "tpot_ms": 96.3239945968991, + "ttft_ms": 302.7184644469543 + }, + { + "e2e_ms": 12572.651782965082, + "request_id": 37, + "tpot_ms": 96.3239945968991, + "ttft_ms": 339.5044691588964 + }, + { + "e2e_ms": 12518.663651486591, + "request_id": 38, + "tpot_ms": 96.3239945968991, + "ttft_ms": 285.51633768040574 + }, + { + "e2e_ms": 12555.320953525566, + "request_id": 39, + "tpot_ms": 96.3239945968991, + "ttft_ms": 322.17363971938084 + }, + { + "e2e_ms": 12770.519034100516, + "request_id": 40, + "tpot_ms": 97.72881409188341, + "ttft_ms": 358.959644431323 + }, + { + "e2e_ms": 12538.070601439798, + "request_id": 41, + "tpot_ms": 96.32361486997611, + "ttft_ms": 304.9715129528323 + }, + { + "e2e_ms": 12574.856606151741, + "request_id": 42, + "tpot_ms": 96.32361486997611, + "ttft_ms": 341.75751766477447 + }, + { + "e2e_ms": 12520.86847467325, + "request_id": 43, + "tpot_ms": 96.32361486997611, + "ttft_ms": 287.7693861862838 + }, + { + "e2e_ms": 12557.654479385192, + "request_id": 44, + "tpot_ms": 96.32462827684986, + "ttft_ms": 324.4266882252589 + }, + { + "e2e_ms": 12503.666347906701, + "request_id": 45, + "tpot_ms": 96.3239945968991, + "ttft_ms": 270.51903410051636 + }, + { + "e2e_ms": 12540.323649945676, + "request_id": 46, + "tpot_ms": 96.3239945968991, + "ttft_ms": 307.17633613949147 + }, + { + "e2e_ms": 12577.109654657448, + "request_id": 47, + "tpot_ms": 96.32399459689775, + "ttft_ms": 343.9623408514336 + }, + { + "e2e_ms": 12523.121523178786, + "request_id": 48, + "tpot_ms": 96.3239945968964, + "ttft_ms": 289.97420937294294 + }, + { + "e2e_ms": 12559.907527890544, + "request_id": 49, + "tpot_ms": 96.32399459689495, + "ttft_ms": 326.7602140848851 + }, + { + "e2e_ms": 12505.919396411884, + "request_id": 50, + "tpot_ms": 96.32399459689363, + "ttft_ms": 272.7720826063944 + }, + { + "e2e_ms": 12542.576698450688, + "request_id": 51, + "tpot_ms": 96.32399459689228, + "ttft_ms": 309.4293846453695 + }, + { + "e2e_ms": 12579.362703162446, + "request_id": 52, + "tpot_ms": 96.32399459689228, + "ttft_ms": 346.2153893571269 + }, + { + "e2e_ms": 12525.374571683784, + "request_id": 53, + "tpot_ms": 96.32399459689228, + "ttft_ms": 292.2272578784657 + }, + { + "e2e_ms": 12562.160576395541, + "request_id": 54, + "tpot_ms": 96.32399459689228, + "ttft_ms": 329.0132625902231 + }, + { + "e2e_ms": 12508.17244491688, + "request_id": 55, + "tpot_ms": 96.32399459689226, + "ttft_ms": 275.0251311115619 + }, + { + "e2e_ms": 12544.829746955686, + "request_id": 56, + "tpot_ms": 96.32399459689228, + "ttft_ms": 311.6824331503665 + }, + { + "e2e_ms": 12581.615751667443, + "request_id": 57, + "tpot_ms": 96.32399459689228, + "ttft_ms": 348.4684378621239 + }, + { + "e2e_ms": 12527.627620188781, + "request_id": 58, + "tpot_ms": 96.32399459689228, + "ttft_ms": 294.4803063834627 + }, + { + "e2e_ms": 12564.413624900539, + "request_id": 59, + "tpot_ms": 96.32399459689228, + "ttft_ms": 331.2663110952201 + }, + { + "e2e_ms": 12314.305135768365, + "request_id": 60, + "tpot_ms": 94.77973981221895, + "ttft_ms": 277.2781796165589 + }, + { + "e2e_ms": 12170.36562563149, + "request_id": 61, + "tpot_ms": 93.35771766910335, + "ttft_ms": 313.93548165536345 + }, + { + "e2e_ms": 12023.933572014499, + "request_id": 62, + "tpot_ms": 91.91505579249903, + "ttft_ms": 350.72148636712086 + }, + { + "e2e_ms": 11724.699298497455, + "request_id": 63, + "tpot_ms": 89.98398380794484, + "ttft_ms": 296.73335488845964 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 23.932271718978882, + "offered_request_rate": 0.8, + "request_metrics_sha256": "b39b2c6a1c6236e30b246303e0b4d66b80a2515899d8c815efc015ef3f70db54", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 14028.106647728775, + "request_id": 0, + "tpot_ms": 108.35061559310502, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 14267.352384844213, + "request_id": 1, + "tpot_ms": 109.65331923259451, + "ttft_ms": 341.3808423047111 + }, + { + "e2e_ms": 14406.16288598222, + "request_id": 2, + "tpot_ms": 110.64064370419715, + "ttft_ms": 354.8011355491818 + }, + { + "e2e_ms": 14444.508775243683, + "request_id": 3, + "tpot_ms": 111.53949237897281, + "ttft_ms": 278.99324311413756 + }, + { + "e2e_ms": 14585.312621821135, + "request_id": 4, + "tpot_ms": 112.58765687327464, + "ttft_ms": 286.6801989152554 + }, + { + "e2e_ms": 14727.157204966976, + "request_id": 5, + "tpot_ms": 113.57630070139629, + "ttft_ms": 302.9670158896467 + }, + { + "e2e_ms": 14871.045643448613, + "request_id": 6, + "tpot_ms": 114.43037546464966, + "ttft_ms": 338.38795943810675 + }, + { + "e2e_ms": 14913.487485283544, + "request_id": 7, + "tpot_ms": 115.12950641852738, + "ttft_ms": 292.0401701305657 + }, + { + "e2e_ms": 15059.457396901958, + "request_id": 8, + "tpot_ms": 115.80285438725795, + "ttft_ms": 352.4948897201998 + }, + { + "e2e_ms": 15271.481983406971, + "request_id": 9, + "tpot_ms": 117.58757541921905, + "ttft_ms": 337.85990516615175 + }, + { + "e2e_ms": 15316.138289012584, + "request_id": 10, + "tpot_ms": 117.88145073243837, + "ttft_ms": 345.1940459929119 + }, + { + "e2e_ms": 15090.835365768904, + "request_id": 11, + "tpot_ms": 116.63565919716638, + "ttft_ms": 278.1066477287748 + }, + { + "e2e_ms": 15302.859952273917, + "request_id": 12, + "tpot_ms": 118.0348305674148, + "ttft_ms": 312.43647021223705 + }, + { + "e2e_ms": 15347.54563377864, + "request_id": 13, + "tpot_ms": 118.10044582475892, + "ttft_ms": 348.7890140342564 + }, + { + "e2e_ms": 15289.743982068245, + "request_id": 14, + "tpot_ms": 118.15013414242924, + "ttft_ms": 284.6769459797329 + }, + { + "e2e_ms": 15334.400287673823, + "request_id": 15, + "tpot_ms": 118.19982246009941, + "ttft_ms": 323.022835241197 + }, + { + "e2e_ms": 15380.492763872971, + "request_id": 16, + "tpot_ms": 118.2528557378034, + "ttft_ms": 362.38008517193873 + }, + { + "e2e_ms": 15322.691112162565, + "request_id": 17, + "tpot_ms": 118.27848723653601, + "ttft_ms": 301.32323312249196 + }, + { + "e2e_ms": 15367.376793667254, + "request_id": 18, + "tpot_ms": 118.29615526543176, + "ttft_ms": 343.765074957421 + }, + { + "e2e_ms": 15309.575141956848, + "request_id": 19, + "tpot_ms": 118.30562851819849, + "ttft_ms": 284.7603201456401 + }, + { + "e2e_ms": 15354.260823461536, + "request_id": 20, + "tpot_ms": 118.30690699483614, + "ttft_ms": 329.2836351173456 + }, + { + "e2e_ms": 15296.45917175113, + "request_id": 21, + "tpot_ms": 118.3069069948359, + "ttft_ms": 271.4819834069715 + }, + { + "e2e_ms": 15341.115477356709, + "request_id": 22, + "tpot_ms": 118.30690699483563, + "ttft_ms": 316.138289012585 + }, + { + "e2e_ms": 15387.207953555859, + "request_id": 23, + "tpot_ms": 118.31926259505173, + "ttft_ms": 360.6616039842905 + }, + { + "e2e_ms": 15329.406301845453, + "request_id": 24, + "tpot_ms": 118.31926259505147, + "ttft_ms": 302.8599522739164 + }, + { + "e2e_ms": 15375.4987780446, + "request_id": 25, + "tpot_ms": 118.33033971862962, + "ttft_ms": 347.5456337786404 + }, + { + "e2e_ms": 15317.697126334195, + "request_id": 26, + "tpot_ms": 118.33033971862953, + "ttft_ms": 289.74398206824503 + }, + { + "e2e_ms": 15362.382807838883, + "request_id": 27, + "tpot_ms": 118.33057102492174, + "ttft_ms": 334.400287673823 + }, + { + "e2e_ms": 15304.581156128477, + "request_id": 28, + "tpot_ms": 118.33057102492174, + "ttft_ms": 276.59863596341694 + }, + { + "e2e_ms": 15349.237461734056, + "request_id": 29, + "tpot_ms": 118.3192625950511, + "ttft_ms": 322.6911121625662 + }, + { + "e2e_ms": 15395.329937933206, + "request_id": 30, + "tpot_ms": 118.33033971862955, + "ttft_ms": 367.37679366725473 + }, + { + "e2e_ms": 15337.5282862228, + "request_id": 31, + "tpot_ms": 118.33033971862955, + "ttft_ms": 309.57514195684865 + }, + { + "e2e_ms": 15383.620762421948, + "request_id": 32, + "tpot_ms": 118.34141684220796, + "ttft_ms": 354.2608234615372 + }, + { + "e2e_ms": 15325.819110711542, + "request_id": 33, + "tpot_ms": 118.34141684220796, + "ttft_ms": 296.4591717511311 + }, + { + "e2e_ms": 15370.50479221623, + "request_id": 34, + "tpot_ms": 118.34164814850017, + "ttft_ms": 341.1154773567091 + }, + { + "e2e_ms": 15312.703140505824, + "request_id": 35, + "tpot_ms": 118.34164814850017, + "ttft_ms": 283.31382564630303 + }, + { + "e2e_ms": 15357.388822010513, + "request_id": 36, + "tpot_ms": 118.33057102492174, + "ttft_ms": 329.4063018454523 + }, + { + "e2e_ms": 15299.587170300107, + "request_id": 37, + "tpot_ms": 118.33057102492174, + "ttft_ms": 271.6046501350462 + }, + { + "e2e_ms": 15344.243475905685, + "request_id": 38, + "tpot_ms": 118.3192625950511, + "ttft_ms": 317.69712633419545 + }, + { + "e2e_ms": 15390.335952104835, + "request_id": 39, + "tpot_ms": 118.33033971862955, + "ttft_ms": 362.382807838884 + }, + { + "e2e_ms": 15332.53430039447, + "request_id": 40, + "tpot_ms": 118.33033971862987, + "ttft_ms": 304.5811561284779 + }, + { + "e2e_ms": 15378.626776593663, + "request_id": 41, + "tpot_ms": 118.34164814850084, + "ttft_ms": 349.2374617340559 + }, + { + "e2e_ms": 15320.8251248833, + "request_id": 42, + "tpot_ms": 118.34164814850119, + "ttft_ms": 291.4358100236498 + }, + { + "e2e_ms": 15365.51080638803, + "request_id": 43, + "tpot_ms": 118.33057102492307, + "ttft_ms": 337.52828622279907 + }, + { + "e2e_ms": 15307.709154677666, + "request_id": 44, + "tpot_ms": 118.33057102492342, + "ttft_ms": 279.726634512393 + }, + { + "e2e_ms": 15352.365460283294, + "request_id": 45, + "tpot_ms": 118.31926259505316, + "ttft_ms": 325.81911071154224 + }, + { + "e2e_ms": 15398.457936482486, + "request_id": 46, + "tpot_ms": 118.33033971863193, + "ttft_ms": 370.5047922162308 + }, + { + "e2e_ms": 15340.656284772123, + "request_id": 47, + "tpot_ms": 118.33033971863227, + "ttft_ms": 312.7031405058247 + }, + { + "e2e_ms": 15386.748760971315, + "request_id": 48, + "tpot_ms": 118.34141684221105, + "ttft_ms": 357.38882201051325 + }, + { + "e2e_ms": 15328.947109260951, + "request_id": 49, + "tpot_ms": 118.34141684221137, + "ttft_ms": 299.5871703001072 + }, + { + "e2e_ms": 15375.039585460143, + "request_id": 50, + "tpot_ms": 118.35272527208234, + "ttft_ms": 344.24347590568516 + }, + { + "e2e_ms": 15317.23793374978, + "request_id": 51, + "tpot_ms": 118.3527252720827, + "ttft_ms": 286.4418241952791 + }, + { + "e2e_ms": 15194.4223437212, + "request_id": 52, + "tpot_ms": 117.02274049863567, + "ttft_ms": 332.53430039447096 + }, + { + "e2e_ms": 14959.152693280232, + "request_id": 53, + "tpot_ms": 115.62535468185925, + "ttft_ms": 274.7326486841075 + }, + { + "e2e_ms": 14796.032219342294, + "request_id": 54, + "tpot_ms": 113.97800861778735, + "ttft_ms": 320.8251248832994 + }, + { + "e2e_ms": 14608.764053703837, + "request_id": 55, + "tpot_ms": 112.1516003725654, + "ttft_ms": 365.5108063880306 + }, + { + "e2e_ms": 14305.495223453576, + "request_id": 56, + "tpot_ms": 110.21878794311739, + "ttft_ms": 307.70915467766713 + }, + { + "e2e_ms": 14083.450969580696, + "request_id": 57, + "tpot_ms": 108.11878353777482, + "ttft_ms": 352.36546028329485 + }, + { + "e2e_ms": 13758.528543574443, + "request_id": 58, + "tpot_ms": 106.01547035434261, + "ttft_ms": 294.5638085729314 + }, + { + "e2e_ms": 13507.044041669204, + "request_id": 59, + "tpot_ms": 103.67234454249669, + "ttft_ms": 340.6562847721233 + }, + { + "e2e_ms": 13157.052579489218, + "request_id": 60, + "tpot_ms": 101.37163737344456, + "ttft_ms": 282.85463306175984 + }, + { + "e2e_ms": 12903.847402123616, + "request_id": 61, + "tpot_ms": 99.01496293592649, + "ttft_ms": 328.9471092609517 + }, + { + "e2e_ms": 12559.065843040158, + "request_id": 62, + "tpot_ms": 96.75527862590212, + "ttft_ms": 271.1454575505883 + }, + { + "e2e_ms": 12277.402762692504, + "request_id": 63, + "tpot_ms": 94.17452621214743, + "ttft_ms": 317.23793374978015 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 27.697141647338867, + "offered_request_rate": 1.2, + "request_metrics_sha256": "3825f14be2103486ce094e0b335c86deef683645cae8f05ef3d4beb6fb24cf95", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 15728.871662360909, + "request_id": 0, + "tpot_ms": 121.74246610201946, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 15904.548256233129, + "request_id": 1, + "tpot_ms": 122.72470152955445, + "ttft_ms": 318.5111619797136 + }, + { + "e2e_ms": 16140.50586206045, + "request_id": 2, + "tpot_ms": 124.75532551591714, + "ttft_ms": 296.5795215389744 + }, + { + "e2e_ms": 16211.825024161684, + "request_id": 3, + "tpot_ms": 125.48331552046938, + "ttft_ms": 275.44395306207383 + }, + { + "e2e_ms": 16388.84099440166, + "request_id": 4, + "tpot_ms": 126.36635826198417, + "ttft_ms": 340.31349512967114 + }, + { + "e2e_ms": 16460.2590055204, + "request_id": 5, + "tpot_ms": 127.08834011289514, + "ttft_ms": 320.03981118271605 + }, + { + "e2e_ms": 16531.726441149392, + "request_id": 6, + "tpot_ms": 127.72809453706097, + "ttft_ms": 310.25843494264916 + }, + { + "e2e_ms": 16603.193876778383, + "request_id": 7, + "tpot_ms": 128.3170301134443, + "ttft_ms": 306.93105237095784 + }, + { + "e2e_ms": 16674.731660081485, + "request_id": 8, + "tpot_ms": 128.84921275092722, + "ttft_ms": 310.8816407137276 + }, + { + "e2e_ms": 16746.318867894835, + "request_id": 9, + "tpot_ms": 129.28898520065795, + "ttft_ms": 326.61774741127834 + }, + { + "e2e_ms": 16982.978169021266, + "request_id": 10, + "tpot_ms": 130.92468009104587, + "ttft_ms": 355.5437974584379 + }, + { + "e2e_ms": 16948.91870099869, + "request_id": 11, + "tpot_ms": 131.099598546656, + "ttft_ms": 299.26968557337295 + }, + { + "e2e_ms": 17020.55604111586, + "request_id": 12, + "tpot_ms": 131.2589874619586, + "ttft_ms": 350.6646334471153 + }, + { + "e2e_ms": 16986.496573094286, + "request_id": 13, + "tpot_ms": 131.335760528223, + "ttft_ms": 306.854986009963 + }, + { + "e2e_ms": 17059.03976122884, + "request_id": 14, + "tpot_ms": 131.4398579526774, + "ttft_ms": 366.1778012388126 + }, + { + "e2e_ms": 17024.98029320726, + "request_id": 15, + "tpot_ms": 131.50690676411472, + "ttft_ms": 323.6031341646921 + }, + { + "e2e_ms": 16990.920825185683, + "request_id": 16, + "tpot_ms": 131.56847300071718, + "ttft_ms": 281.7247540946006 + }, + { + "e2e_ms": 17063.464013320234, + "request_id": 17, + "tpot_ms": 131.63886355765484, + "ttft_ms": 345.32834149806746 + }, + { + "e2e_ms": 17029.40454529866, + "request_id": 18, + "tpot_ms": 131.67812927130495, + "ttft_ms": 306.28212784293396 + }, + { + "e2e_ms": 16995.3450772771, + "request_id": 19, + "tpot_ms": 131.6904025418972, + "ttft_ms": 270.66395445615575 + }, + { + "e2e_ms": 17067.888265411675, + "request_id": 20, + "tpot_ms": 131.7034495326512, + "ttft_ms": 341.5501747649721 + }, + { + "e2e_ms": 17033.828797390113, + "request_id": 21, + "tpot_ms": 131.70595487135952, + "ttft_ms": 307.17252872745246 + }, + { + "e2e_ms": 16999.769329368555, + "request_id": 22, + "tpot_ms": 131.7084602100679, + "ttft_ms": 272.79488268993293 + }, + { + "e2e_ms": 17072.312517503127, + "request_id": 23, + "tpot_ms": 131.71809821033827, + "ttft_ms": 344.1140447901674 + }, + { + "e2e_ms": 17038.25304948157, + "request_id": 24, + "tpot_ms": 131.71982521031822, + "ttft_ms": 309.83524777115434 + }, + { + "e2e_ms": 17004.193581460007, + "request_id": 25, + "tpot_ms": 131.72116304093396, + "ttft_ms": 275.60587526139454 + }, + { + "e2e_ms": 17076.736769594583, + "request_id": 26, + "tpot_ms": 131.72963353311178, + "ttft_ms": 347.07331088938884 + }, + { + "e2e_ms": 17042.67730157302, + "request_id": 27, + "tpot_ms": 131.7305821943633, + "ttft_ms": 312.8933628888824 + }, + { + "e2e_ms": 17008.617833551463, + "request_id": 28, + "tpot_ms": 131.7309769369132, + "ttft_ms": 278.783762563485 + }, + { + "e2e_ms": 17081.161021686035, + "request_id": 29, + "tpot_ms": 131.73850434102513, + "ttft_ms": 350.37097037584175 + }, + { + "e2e_ms": 17047.986478516996, + "request_id": 30, + "tpot_ms": 131.74547225324986, + "ttft_ms": 316.3115023542638 + }, + { + "e2e_ms": 17013.927010495438, + "request_id": 31, + "tpot_ms": 131.74547225325, + "ttft_ms": 282.25203433268575 + }, + { + "e2e_ms": 17086.47019863001, + "request_id": 32, + "tpot_ms": 131.75260491481217, + "ttft_ms": 353.8893744488618 + }, + { + "e2e_ms": 17053.29565546097, + "request_id": 33, + "tpot_ms": 131.7595728270369, + "ttft_ms": 319.82990642728385 + }, + { + "e2e_ms": 17019.236187439412, + "request_id": 34, + "tpot_ms": 131.75957282703703, + "ttft_ms": 285.7704384057058 + }, + { + "e2e_ms": 17091.779375573984, + "request_id": 35, + "tpot_ms": 131.7595728270372, + "ttft_ms": 358.31362654026225 + }, + { + "e2e_ms": 17058.604832404944, + "request_id": 36, + "tpot_ms": 131.7665407392619, + "ttft_ms": 324.2541585186842 + }, + { + "e2e_ms": 17024.545364383386, + "request_id": 37, + "tpot_ms": 131.76654073926207, + "ttft_ms": 290.1946904971062 + }, + { + "e2e_ms": 17097.08855251796, + "request_id": 38, + "tpot_ms": 131.76654073926218, + "ttft_ms": 362.7378786316626 + }, + { + "e2e_ms": 17063.934932514778, + "request_id": 39, + "tpot_ms": 131.7736734008242, + "ttft_ms": 328.6784106101024 + }, + { + "e2e_ms": 17029.875464493216, + "request_id": 40, + "tpot_ms": 131.7736734008242, + "ttft_ms": 294.61894258854215 + }, + { + "e2e_ms": 17102.41865262779, + "request_id": 41, + "tpot_ms": 131.7736734008242, + "ttft_ms": 367.1621307231163 + }, + { + "e2e_ms": 17069.265032624608, + "request_id": 42, + "tpot_ms": 131.78080606238623, + "ttft_ms": 333.1026627015561 + }, + { + "e2e_ms": 17035.205564603046, + "request_id": 43, + "tpot_ms": 131.78080606238623, + "ttft_ms": 299.04319467999585 + }, + { + "e2e_ms": 16941.812657737883, + "request_id": 44, + "tpot_ms": 130.47422263719145, + "ttft_ms": 371.58638281457 + }, + { + "e2e_ms": 16742.48916517699, + "request_id": 45, + "tpot_ms": 129.1729311053857, + "ttft_ms": 337.5269147930098 + }, + { + "e2e_ms": 16538.03643831958, + "request_id": 46, + "tpot_ms": 127.83125190195379, + "ttft_ms": 303.46744677144954 + }, + { + "e2e_ms": 16435.957848641778, + "request_id": 47, + "tpot_ms": 126.45627727350987, + "ttft_ms": 376.0106349060237 + }, + { + "e2e_ms": 16228.476901947985, + "request_id": 48, + "tpot_ms": 125.09075381939779, + "ttft_ms": 341.9511668844635 + }, + { + "e2e_ms": 16020.052127617348, + "request_id": 49, + "tpot_ms": 123.7177986516098, + "ttft_ms": 307.89169886290324 + }, + { + "e2e_ms": 15809.427558622445, + "request_id": 50, + "tpot_ms": 122.32055435377619, + "ttft_ms": 274.71715569286914 + }, + { + "e2e_ms": 15703.511289947706, + "request_id": 51, + "tpot_ms": 120.91536178046667, + "ttft_ms": 347.26034382843807 + }, + { + "e2e_ms": 15485.101955905271, + "request_id": 52, + "tpot_ms": 119.46378803227081, + "ttft_ms": 313.20087580687783 + }, + { + "e2e_ms": 15260.624457846197, + "request_id": 53, + "tpot_ms": 117.95746555282956, + "ttft_ms": 280.02633263684373 + }, + { + "e2e_ms": 15118.941732007286, + "request_id": 54, + "tpot_ms": 116.27064733255806, + "ttft_ms": 352.56952077241266 + }, + { + "e2e_ms": 14865.280308326035, + "request_id": 55, + "tpot_ms": 114.54149807539514, + "ttft_ms": 318.5100527508524 + }, + { + "e2e_ms": 14599.985676841925, + "request_id": 56, + "tpot_ms": 112.71378084457564, + "ttft_ms": 285.3355095808183 + }, + { + "e2e_ms": 14420.806000135271, + "request_id": 57, + "tpot_ms": 110.73171104266838, + "ttft_ms": 357.87869771638725 + }, + { + "e2e_ms": 14142.519211198518, + "request_id": 58, + "tpot_ms": 108.80866127168261, + "ttft_ms": 323.819229694827 + }, + { + "e2e_ms": 13853.830695007076, + "request_id": 59, + "tpot_ms": 106.79657547493251, + "ttft_ms": 290.6656096906488 + }, + { + "e2e_ms": 13650.503338148035, + "request_id": 60, + "tpot_ms": 104.62436645922692, + "ttft_ms": 363.20879782621773 + }, + { + "e2e_ms": 13360.876271706531, + "request_id": 61, + "tpot_ms": 102.61202316458169, + "ttft_ms": 329.1493298046575 + }, + { + "e2e_ms": 13070.674002922458, + "request_id": 62, + "tpot_ms": 100.58801805607857, + "ttft_ms": 295.9957098004793 + }, + { + "e2e_ms": 12853.555073004585, + "request_id": 63, + "tpot_ms": 98.30721397691761, + "ttft_ms": 368.5388979360482 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 28.44595432281494, + "offered_request_rate": 1.6, + "request_metrics_sha256": "b7d73a1cbef8ca659f753135a4944cb3023d3a2424a19a741ae2198f39c2d172", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 17513.570713343794, + "request_id": 0, + "tpot_ms": 135.79521453495556, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 17840.282304762335, + "request_id": 1, + "tpot_ms": 137.71064841138914, + "ttft_ms": 351.0299565159147 + }, + { + "e2e_ms": 17895.63618825912, + "request_id": 2, + "tpot_ms": 138.10245652748367, + "ttft_ms": 356.6242092686922 + }, + { + "e2e_ms": 17848.77057711112, + "request_id": 3, + "tpot_ms": 138.39719968518088, + "ttft_ms": 272.32621709314975 + }, + { + "e2e_ms": 17904.1244606079, + "request_id": 4, + "tpot_ms": 138.80479077491768, + "ttft_ms": 275.9160321933569 + }, + { + "e2e_ms": 17959.478344104686, + "request_id": 5, + "tpot_ms": 139.18735062645055, + "ttft_ms": 282.6848145454632 + }, + { + "e2e_ms": 18014.921946878916, + "request_id": 6, + "tpot_ms": 139.5152186600215, + "ttft_ms": 296.48917705618504 + }, + { + "e2e_ms": 18070.36554965315, + "request_id": 7, + "tpot_ms": 139.8086106441042, + "ttft_ms": 314.671997851919 + }, + { + "e2e_ms": 18125.898871704834, + "request_id": 8, + "tpot_ms": 140.0646556798527, + "ttft_ms": 337.68760036353916 + }, + { + "e2e_ms": 18079.30241838919, + "request_id": 9, + "tpot_ms": 140.21470125635153, + "ttft_ms": 272.0353588325457 + }, + { + "e2e_ms": 18134.83860521682, + "request_id": 10, + "tpot_ms": 140.33922237580643, + "ttft_ms": 311.75736348940484 + }, + { + "e2e_ms": 18190.4645113219, + "request_id": 11, + "tpot_ms": 140.37869703173442, + "ttft_ms": 362.36998829163 + }, + { + "e2e_ms": 18143.870922782207, + "request_id": 12, + "tpot_ms": 140.38698051363622, + "ttft_ms": 314.7243975504068 + }, + { + "e2e_ms": 18097.274469466563, + "request_id": 13, + "tpot_ms": 140.36074521133403, + "ttft_ms": 271.4598276271438 + }, + { + "e2e_ms": 18152.810656294194, + "request_id": 14, + "tpot_ms": 140.32455966134754, + "ttft_ms": 331.5915793030584 + }, + { + "e2e_ms": 18106.217067754496, + "request_id": 15, + "tpot_ms": 140.29231877139563, + "ttft_ms": 289.0925837872533 + }, + { + "e2e_ms": 18161.753254582127, + "request_id": 16, + "tpot_ms": 140.24009333090902, + "ttft_ms": 351.2614015566804 + }, + { + "e2e_ms": 18115.15966604243, + "request_id": 17, + "tpot_ms": 140.19912163029534, + "ttft_ms": 309.87121899492286 + }, + { + "e2e_ms": 18170.69585287006, + "request_id": 18, + "tpot_ms": 140.12856604535176, + "ttft_ms": 374.3679651103839 + }, + { + "e2e_ms": 18124.102264330366, + "request_id": 19, + "tpot_ms": 140.05569589908265, + "ttft_ms": 337.0288851468697 + }, + { + "e2e_ms": 18077.42933293013, + "request_id": 20, + "tpot_ms": 139.98073600581233, + "ttft_ms": 299.87586019196044 + }, + { + "e2e_ms": 18132.96551975776, + "request_id": 21, + "tpot_ms": 139.8495015438229, + "ttft_ms": 372.0788236922541 + }, + { + "e2e_ms": 18086.369066442116, + "request_id": 22, + "tpot_ms": 139.76199603449132, + "ttft_ms": 336.5955700617178 + }, + { + "e2e_ms": 18039.682893849018, + "request_id": 23, + "tpot_ms": 139.67091992877667, + "ttft_ms": 301.4760628943822 + }, + { + "e2e_ms": 18095.21908067664, + "request_id": 24, + "tpot_ms": 139.54200325642663, + "ttft_ms": 373.38466711045726 + }, + { + "e2e_ms": 18048.532908083544, + "request_id": 25, + "tpot_ms": 139.5517742330953, + "ttft_ms": 325.4575804804407 + }, + { + "e2e_ms": 18001.846735490446, + "request_id": 26, + "tpot_ms": 139.5610841122856, + "ttft_ms": 277.589053230173 + }, + { + "e2e_ms": 18057.290338264673, + "request_id": 27, + "tpot_ms": 139.56778701592805, + "ttft_ms": 332.18138724181046 + }, + { + "e2e_ms": 18010.60416567157, + "request_id": 28, + "tpot_ms": 139.57154350494935, + "ttft_ms": 285.01814054300564 + }, + { + "e2e_ms": 18066.06100963866, + "request_id": 29, + "tpot_ms": 139.5730606683175, + "ttft_ms": 340.2823047623365 + }, + { + "e2e_ms": 18019.37483704556, + "request_id": 30, + "tpot_ms": 139.57447357032456, + "ttft_ms": 293.41669361433986 + }, + { + "e2e_ms": 18074.831681012645, + "request_id": 31, + "tpot_ms": 139.57528428268915, + "ttft_ms": 348.7705771111216 + }, + { + "e2e_ms": 18028.145508419548, + "request_id": 32, + "tpot_ms": 139.57669718469626, + "ttft_ms": 301.904965963125 + }, + { + "e2e_ms": 18083.678830471228, + "request_id": 33, + "tpot_ms": 139.5781100867033, + "ttft_ms": 357.25884945990674 + }, + { + "e2e_ms": 18036.99265787813, + "request_id": 34, + "tpot_ms": 139.57881653770684, + "ttft_ms": 310.48295758936104 + }, + { + "e2e_ms": 18092.528844705754, + "request_id": 35, + "tpot_ms": 139.57954554600127, + "ttft_ms": 365.9265603635937 + }, + { + "e2e_ms": 17876.05367572695, + "request_id": 36, + "tpot_ms": 138.24262431461773, + "ttft_ms": 319.240387770499 + }, + { + "e2e_ms": 17659.21004164318, + "request_id": 37, + "tpot_ms": 136.90209533219152, + "ttft_ms": 272.64393445485524 + }, + { + "e2e_ms": 17543.75228976072, + "request_id": 38, + "tpot_ms": 135.55570892326125, + "ttft_ms": 328.1772565065388 + }, + { + "e2e_ms": 17326.367174433202, + "request_id": 39, + "tpot_ms": 134.21089375170362, + "ttft_ms": 281.5836679668422 + }, + { + "e2e_ms": 17233.832476815856, + "request_id": 40, + "tpot_ms": 133.04427482475538, + "ttft_ms": 337.20957407192384 + }, + { + "e2e_ms": 17034.265611203216, + "request_id": 41, + "tpot_ms": 131.83976083205505, + "ttft_ms": 290.6159855322272 + }, + { + "e2e_ms": 16940.502121615402, + "request_id": 42, + "tpot_ms": 130.66419538607474, + "ttft_ms": 346.14930758391085 + }, + { + "e2e_ms": 16741.97064984836, + "request_id": 43, + "tpot_ms": 129.46783410082003, + "ttft_ms": 299.5557190442142 + }, + { + "e2e_ms": 16645.559306371404, + "request_id": 44, + "tpot_ms": 128.27139685432724, + "ttft_ms": 355.0919058718449 + }, + { + "e2e_ms": 16443.232533552808, + "request_id": 45, + "tpot_ms": 127.04515130882409, + "ttft_ms": 308.4983173321483 + }, + { + "e2e_ms": 16342.299705615802, + "request_id": 46, + "tpot_ms": 125.81311182248837, + "ttft_ms": 364.034504159779 + }, + { + "e2e_ms": 16135.159559133626, + "request_id": 47, + "tpot_ms": 124.54896569695704, + "ttft_ms": 317.44091562008236 + }, + { + "e2e_ms": 16031.7032146663, + "request_id": 48, + "tpot_ms": 123.29695174036006, + "ttft_ms": 372.9903436405735 + }, + { + "e2e_ms": 15823.308920667878, + "request_id": 49, + "tpot_ms": 122.0236594458299, + "ttft_ms": 326.3041710474788 + }, + { + "e2e_ms": 15613.44809689394, + "request_id": 50, + "tpot_ms": 120.73809066445794, + "ttft_ms": 279.71058250778214 + }, + { + "e2e_ms": 15508.031714506984, + "request_id": 51, + "tpot_ms": 119.47077015706711, + "ttft_ms": 335.2439045594622 + }, + { + "e2e_ms": 15292.981047367362, + "request_id": 52, + "tpot_ms": 118.14506547559841, + "ttft_ms": 288.5577319663639 + }, + { + "e2e_ms": 15175.360910096466, + "request_id": 53, + "tpot_ms": 116.78162985277542, + "ttft_ms": 344.09391879398754 + }, + { + "e2e_ms": 14945.589828664517, + "request_id": 54, + "tpot_ms": 115.34001639735139, + "ttft_ms": 297.4077462008893 + }, + { + "e2e_ms": 14803.64975337431, + "request_id": 55, + "tpot_ms": 113.78581420786767, + "ttft_ms": 352.85134897511483 + }, + { + "e2e_ms": 14557.342221274233, + "request_id": 56, + "tpot_ms": 112.21399247946627, + "ttft_ms": 306.16517638201657 + }, + { + "e2e_ms": 14399.594833150197, + "request_id": 57, + "tpot_ms": 110.5352189984338, + "ttft_ms": 361.6220203491025 + }, + { + "e2e_ms": 14144.625862747695, + "request_id": 58, + "tpot_ms": 108.89519696843851, + "ttft_ms": 314.93584775600425 + }, + { + "e2e_ms": 13973.496543699825, + "request_id": 59, + "tpot_ms": 107.1110539525727, + "ttft_ms": 370.3926917230902 + }, + { + "e2e_ms": 13708.4999588278, + "request_id": 60, + "tpot_ms": 105.3920743283292, + "ttft_ms": 323.70651912999193 + }, + { + "e2e_ms": 13445.970803422128, + "request_id": 61, + "tpot_ms": 103.6925232825609, + "ttft_ms": 277.02034653689367 + }, + { + "e2e_ms": 13273.580023880399, + "request_id": 62, + "tpot_ms": 101.89784531725846, + "ttft_ms": 332.5536685885737 + }, + { + "e2e_ms": 13000.702540117643, + "request_id": 63, + "tpot_ms": 100.1168113710407, + "ttft_ms": 285.86749599547545 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 33.35526180267334, + "offered_request_rate": 2.4, + "request_metrics_sha256": "3b42135fbabbd54647284962ccc9fdf0b900fbc0b6ed0bdecf03477d5c409d18", + "request_rate_per_gpu": 0.6, + "requests": [ + { + "e2e_ms": 21942.173036302996, + "request_id": 0, + "tpot_ms": 170.66609896770518, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 22008.341136862466, + "request_id": 1, + "tpot_ms": 170.96613901739173, + "ttft_ms": 295.64148165371654 + }, + { + "e2e_ms": 22074.54102422534, + "request_id": 2, + "tpot_ms": 171.22660942224667, + "ttft_ms": 328.7616276000109 + }, + { + "e2e_ms": 22036.227907899276, + "request_id": 3, + "tpot_ms": 171.37344968894675, + "ttft_ms": 271.7997974030406 + }, + { + "e2e_ms": 22102.42779526115, + "request_id": 4, + "tpot_ms": 171.64131409153669, + "ttft_ms": 303.98090563598765 + }, + { + "e2e_ms": 22168.65946942643, + "request_id": 5, + "tpot_ms": 171.89659303259273, + "ttft_ms": 337.79215428715537 + }, + { + "e2e_ms": 22130.37813990278, + "request_id": 6, + "tpot_ms": 172.0281077294899, + "ttft_ms": 282.80845825756097 + }, + { + "e2e_ms": 22365.99994575454, + "request_id": 7, + "tpot_ms": 173.57085404199856, + "ttft_ms": 322.5014824207202 + }, + { + "e2e_ms": 22158.3592864644, + "request_id": 8, + "tpot_ms": 172.3496956452424, + "ttft_ms": 269.9479395186155 + }, + { + "e2e_ms": 22393.981092316164, + "request_id": 9, + "tpot_ms": 173.8419366523574, + "ttft_ms": 316.0551374667744 + }, + { + "e2e_ms": 22460.243568400445, + "request_id": 10, + "tpot_ms": 173.96590922572528, + "ttft_ms": 366.57309673333316 + }, + { + "e2e_ms": 22252.602909110316, + "request_id": 11, + "tpot_ms": 172.6883751510306, + "ttft_ms": 321.1792649294285 + }, + { + "e2e_ms": 22044.894176229613, + "request_id": 12, + "tpot_ms": 171.40112664047544, + "ttft_ms": 276.9510928892318 + }, + { + "e2e_ms": 21941.653543442477, + "request_id": 13, + "tpot_ms": 170.1065182287643, + "ttft_ms": 338.1257283894126 + }, + { + "e2e_ms": 21734.138079618475, + "request_id": 14, + "tpot_ms": 168.80971544527205, + "ttft_ms": 295.30421806892537 + }, + { + "e2e_ms": 21631.119179231846, + "request_id": 15, + "tpot_ms": 167.51267286797687, + "ttft_ms": 357.0097249987798 + }, + { + "e2e_ms": 21420.354658102242, + "request_id": 16, + "tpot_ms": 166.18662353135022, + "ttft_ms": 314.65346962076256 + }, + { + "e2e_ms": 21209.648295563282, + "request_id": 17, + "tpot_ms": 164.85776701341018, + "ttft_ms": 272.7118848601879 + }, + { + "e2e_ms": 21101.69867612793, + "request_id": 18, + "tpot_ms": 163.50974377121332, + "ttft_ms": 335.9612171838391 + }, + { + "e2e_ms": 20890.778369055995, + "request_id": 19, + "tpot_ms": 162.16628083276893, + "ttft_ms": 295.66070329433813 + }, + { + "e2e_ms": 20782.558056179907, + "request_id": 20, + "tpot_ms": 160.7985342727971, + "ttft_ms": 361.14420353467835 + }, + { + "e2e_ms": 20571.512469390585, + "request_id": 21, + "tpot_ms": 159.44598375979533, + "ttft_ms": 321.8725318965792 + }, + { + "e2e_ms": 20360.405278761264, + "request_id": 22, + "tpot_ms": 158.09565605925098, + "ttft_ms": 282.2569592363866 + }, + { + "e2e_ms": 20251.586130359967, + "request_id": 23, + "tpot_ms": 156.71227679254048, + "ttft_ms": 349.12697770732314 + }, + { + "e2e_ms": 20042.10448060375, + "request_id": 24, + "tpot_ms": 155.37142770924103, + "ttft_ms": 309.93316153013774 + }, + { + "e2e_ms": 19934.008109113514, + "request_id": 25, + "tpot_ms": 154.05960281316982, + "ttft_ms": 368.43855184094565 + }, + { + "e2e_ms": 19722.984661067087, + "request_id": 26, + "tpot_ms": 152.73931080974432, + "ttft_ms": 325.0921882295579 + }, + { + "e2e_ms": 19511.768043076878, + "request_id": 27, + "tpot_ms": 151.41621155599998, + "ttft_ms": 281.9091754648788 + }, + { + "e2e_ms": 19403.203575383635, + "request_id": 28, + "tpot_ms": 150.09414530000853, + "ttft_ms": 341.2471222825513 + }, + { + "e2e_ms": 19191.714653347688, + "request_id": 29, + "tpot_ms": 148.76487274343572, + "ttft_ms": 298.57581493135046 + }, + { + "e2e_ms": 19082.708845359608, + "request_id": 30, + "tpot_ms": 147.43434256455518, + "ttft_ms": 358.5473396611025 + }, + { + "e2e_ms": 18871.078709760895, + "request_id": 31, + "tpot_ms": 146.10277278695904, + "ttft_ms": 316.0265658170971 + }, + { + "e2e_ms": 18659.370447496967, + "request_id": 32, + "tpot_ms": 144.7695321688903, + "ttft_ms": 273.6398620479008 + }, + { + "e2e_ms": 18549.637648096734, + "request_id": 33, + "tpot_ms": 143.43132538741386, + "ttft_ms": 333.85932389517234 + }, + { + "e2e_ms": 18337.808235788358, + "request_id": 34, + "tpot_ms": 142.0958670025312, + "ttft_ms": 291.63312646689656 + }, + { + "e2e_ms": 18227.80864738452, + "request_id": 35, + "tpot_ms": 140.75352098043697, + "ttft_ms": 352.1114828690255 + }, + { + "e2e_ms": 18015.40726100407, + "request_id": 36, + "tpot_ms": 139.41149279948888, + "ttft_ms": 310.14767546898446 + }, + { + "e2e_ms": 17904.857868774245, + "request_id": 37, + "tpot_ms": 138.06211837292886, + "ttft_ms": 370.9688354122758 + }, + { + "e2e_ms": 17692.008101355255, + "request_id": 38, + "tpot_ms": 136.71510258553306, + "ttft_ms": 329.1900729925583 + }, + { + "e2e_ms": 17479.148877024498, + "request_id": 39, + "tpot_ms": 135.36749298254742, + "ttft_ms": 287.47726824097344 + }, + { + "e2e_ms": 17381.961391787096, + "request_id": 40, + "tpot_ms": 134.1089349239123, + "ttft_ms": 350.1266564502323 + }, + { + "e2e_ms": 17178.01129231477, + "request_id": 41, + "tpot_ms": 132.8327622110564, + "ttft_ms": 308.2504915106092 + }, + { + "e2e_ms": 17080.086531895085, + "request_id": 42, + "tpot_ms": 131.5805170059729, + "ttft_ms": 369.36087213652866 + }, + { + "e2e_ms": 16876.654129344566, + "request_id": 43, + "tpot_ms": 130.30640250229436, + "ttft_ms": 327.74101155318294 + }, + { + "e2e_ms": 16671.422925287778, + "request_id": 44, + "tpot_ms": 129.0178430005707, + "ttft_ms": 286.15686421529676 + }, + { + "e2e_ms": 16571.761179006833, + "request_id": 45, + "tpot_ms": 127.74972161297816, + "ttft_ms": 347.5465341586066 + }, + { + "e2e_ms": 16364.721381165033, + "request_id": 46, + "tpot_ms": 126.44572566423672, + "ttft_ms": 306.1142218069683 + }, + { + "e2e_ms": 16261.449604637399, + "request_id": 47, + "tpot_ms": 125.14700916940275, + "ttft_ms": 367.77944012325037 + }, + { + "e2e_ms": 16053.400400183471, + "request_id": 48, + "tpot_ms": 123.8340136000885, + "ttft_ms": 326.48067297223093 + }, + { + "e2e_ms": 15845.03658651726, + "request_id": 49, + "tpot_ms": 122.50141376236263, + "ttft_ms": 287.35703869720555 + }, + { + "e2e_ms": 15739.724302020804, + "request_id": 50, + "tpot_ms": 121.15600683324593, + "ttft_ms": 352.9114341985711 + }, + { + "e2e_ms": 15530.891082399023, + "request_id": 51, + "tpot_ms": 119.81946138926746, + "ttft_ms": 313.8194859620569 + }, + { + "e2e_ms": 15319.199082162215, + "request_id": 52, + "tpot_ms": 118.45427332697808, + "ttft_ms": 275.5063696359983 + }, + { + "e2e_ms": 15206.960333133673, + "request_id": 53, + "tpot_ms": 117.04949498375757, + "ttft_ms": 341.6744701964625 + }, + { + "e2e_ms": 14987.9081257507, + "request_id": 54, + "tpot_ms": 115.62635253449054, + "ttft_ms": 303.36135387040383 + }, + { + "e2e_ms": 14861.077413909577, + "request_id": 55, + "tpot_ms": 114.10642655651417, + "ttft_ms": 369.5612412322795 + }, + { + "e2e_ms": 14633.756981193535, + "request_id": 56, + "tpot_ms": 112.61817997075843, + "ttft_ms": 331.2481249072157 + }, + { + "e2e_ms": 14403.991359276923, + "request_id": 57, + "tpot_ms": 111.11042963695559, + "ttft_ms": 292.9667953835633 + }, + { + "e2e_ms": 14264.847964808048, + "request_id": 58, + "tpot_ms": 109.49330311228506, + "ttft_ms": 359.1984695478452 + }, + { + "e2e_ms": 14029.729570522895, + "request_id": 59, + "tpot_ms": 107.94340496454889, + "ttft_ms": 320.91714002518756 + }, + { + "e2e_ms": 13793.064611419886, + "request_id": 60, + "tpot_ms": 106.38108660628622, + "ttft_ms": 282.66661242153646 + }, + { + "e2e_ms": 13648.251078198633, + "request_id": 61, + "tpot_ms": 104.71931331978594, + "ttft_ms": 348.89828658581837 + }, + { + "e2e_ms": 13412.62809971594, + "request_id": 62, + "tpot_ms": 103.16519953332896, + "ttft_ms": 310.647758983162 + }, + { + "e2e_ms": 13260.053320226874, + "request_id": 63, + "tpot_ms": 101.44207153668842, + "ttft_ms": 376.9102350674451 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 34.759880781173706, + "offered_request_rate": 3.2, + "request_metrics_sha256": "bfba1ae71cfb6d0595614b14bc49ac47591bfad6792be0a3d98d2c98b16012e8", + "request_rate_per_gpu": 0.8, + "requests": [ + { + "e2e_ms": 24003.81066522886, + "request_id": 0, + "tpot_ms": 186.89946612460176, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 23901.058156177503, + "request_id": 1, + "tpot_ms": 185.74139588389912, + "ttft_ms": 311.9008789223171 + }, + { + "e2e_ms": 23798.190109844723, + "request_id": 2, + "tpot_ms": 184.56225305811301, + "ttft_ms": 358.7839714643697 + }, + { + "e2e_ms": 23590.44639829378, + "request_id": 3, + "tpot_ms": 183.2673714648551, + "ttft_ms": 315.49022225718517 + }, + { + "e2e_ms": 23382.70131446869, + "request_id": 4, + "tpot_ms": 181.9732705315891, + "ttft_ms": 272.09595695687216 + }, + { + "e2e_ms": 23279.64954297851, + "request_id": 5, + "tpot_ms": 180.78914255292864, + "ttft_ms": 319.4284387565705 + }, + { + "e2e_ms": 23071.973865900134, + "request_id": 6, + "tpot_ms": 179.49398625037213, + "ttft_ms": 276.2376121028707 + }, + { + "e2e_ms": 22969.022726489202, + "request_id": 7, + "tpot_ms": 178.2861180199918, + "ttft_ms": 326.6857379502426 + }, + { + "e2e_ms": 22761.29942646063, + "request_id": 8, + "tpot_ms": 176.98924979786668, + "ttft_ms": 283.66470213156344 + }, + { + "e2e_ms": 22658.264587165988, + "request_id": 9, + "tpot_ms": 175.7551174352214, + "ttft_ms": 337.36467289287384 + }, + { + "e2e_ms": 22450.304224673095, + "request_id": 10, + "tpot_ms": 174.4550575162989, + "ttft_ms": 294.51192010313275 + }, + { + "e2e_ms": 22346.821408551063, + "request_id": 11, + "tpot_ms": 173.1774539652428, + "ttft_ms": 353.28475496522805 + }, + { + "e2e_ms": 22138.79812545183, + "request_id": 12, + "tpot_ms": 171.87427427697472, + "ttft_ms": 310.7652922760442 + }, + { + "e2e_ms": 22035.248814705075, + "request_id": 13, + "tpot_ms": 170.57508935568035, + "ttft_ms": 372.21246653366967 + }, + { + "e2e_ms": 21827.324416126074, + "request_id": 14, + "tpot_ms": 169.27188182433355, + "ttft_ms": 329.79542443571484 + }, + { + "e2e_ms": 21619.373671552865, + "request_id": 15, + "tpot_ms": 167.96821838814947, + "ttft_ms": 287.40993625788303 + }, + { + "e2e_ms": 21512.77581709026, + "request_id": 16, + "tpot_ms": 166.64216905152279, + "ttft_ms": 349.2203475468658 + }, + { + "e2e_ms": 21303.255969153783, + "request_id": 17, + "tpot_ms": 165.32482604305434, + "ttft_ms": 307.003061685883 + }, + { + "e2e_ms": 21196.567333974876, + "request_id": 18, + "tpot_ms": 163.99164288866731, + "ttft_ms": 369.62868711412875 + }, + { + "e2e_ms": 20986.94051377241, + "request_id": 19, + "tpot_ms": 162.67002407488405, + "ttft_ms": 327.84745626213316 + }, + { + "e2e_ms": 20777.255965035714, + "request_id": 20, + "tpot_ms": 161.34765321373814, + "ttft_ms": 286.10400689097213 + }, + { + "e2e_ms": 20670.377044913395, + "request_id": 21, + "tpot_ms": 159.9951027007364, + "ttft_ms": 350.99900191987297 + }, + { + "e2e_ms": 20460.656782932234, + "request_id": 22, + "tpot_ms": 158.67196467076286, + "ttft_ms": 309.3172697453532 + }, + { + "e2e_ms": 20353.38846177537, + "request_id": 23, + "tpot_ms": 157.31793948495124, + "ttft_ms": 374.01014718656 + }, + { + "e2e_ms": 20144.48097023076, + "request_id": 24, + "tpot_ms": 155.99981541047092, + "ttft_ms": 332.5044131009545 + }, + { + "e2e_ms": 19934.83773528968, + "request_id": 25, + "tpot_ms": 154.6729368346084, + "ttft_ms": 291.37475729441144 + }, + { + "e2e_ms": 19827.980953909257, + "request_id": 26, + "tpot_ms": 153.35264483118294, + "ttft_ms": 352.1950603490236 + }, + { + "e2e_ms": 19618.20597824765, + "request_id": 27, + "tpot_ms": 152.02412197401114, + "ttft_ms": 311.1424875482367 + }, + { + "e2e_ms": 19511.107444230154, + "request_id": 28, + "tpot_ms": 150.6975599886974, + "ttft_ms": 372.5173256655836 + }, + { + "e2e_ms": 19301.196316545185, + "request_id": 29, + "tpot_ms": 149.36537894653213, + "ttft_ms": 331.7931903356044 + }, + { + "e2e_ms": 19091.249936104825, + "request_id": 30, + "tpot_ms": 148.03267778881258, + "ttft_ms": 291.0998569256247 + }, + { + "e2e_ms": 18983.78646717311, + "request_id": 31, + "tpot_ms": 146.7011080112165, + "ttft_ms": 352.7457497486193 + }, + { + "e2e_ms": 18773.765669374145, + "request_id": 32, + "tpot_ms": 145.36784483585674, + "ttft_ms": 312.0493752203348 + }, + { + "e2e_ms": 18665.888247551986, + "request_id": 33, + "tpot_ms": 144.03124523287858, + "ttft_ms": 373.9201029764097 + }, + { + "e2e_ms": 18455.8068747313, + "request_id": 34, + "tpot_ms": 142.6968275070543, + "ttft_ms": 333.30978133540157 + }, + { + "e2e_ms": 18245.529234152025, + "request_id": 35, + "tpot_ms": 141.3602386069844, + "ttft_ms": 292.7789310650048 + }, + { + "e2e_ms": 18137.294514438578, + "request_id": 36, + "tpot_ms": 140.01821042603632, + "ttft_ms": 354.98179033196385 + }, + { + "e2e_ms": 17926.83360591763, + "request_id": 37, + "tpot_ms": 138.67967407469433, + "ttft_ms": 314.5149984314486 + }, + { + "e2e_ms": 17716.242055541137, + "request_id": 38, + "tpot_ms": 137.3398665125213, + "ttft_ms": 274.0790084509328 + }, + { + "e2e_ms": 17607.549497877386, + "request_id": 39, + "tpot_ms": 135.99225690953574, + "ttft_ms": 336.53287036634794 + }, + { + "e2e_ms": 17401.54255835392, + "request_id": 40, + "tpot_ms": 134.6880568492608, + "ttft_ms": 296.1593384978016 + }, + { + "e2e_ms": 17301.7591255476, + "request_id": 41, + "tpot_ms": 133.41188413640484, + "ttft_ms": 358.4498402241838 + }, + { + "e2e_ms": 17095.506427630044, + "request_id": 42, + "tpot_ms": 132.10522228512968, + "ttft_ms": 318.1431974185731 + }, + { + "e2e_ms": 16889.623559688287, + "request_id": 43, + "tpot_ms": 130.8012299461049, + "ttft_ms": 277.86735653296194 + }, + { + "e2e_ms": 16788.559022297493, + "request_id": 44, + "tpot_ms": 129.5126704443812, + "ttft_ms": 340.449875861081 + }, + { + "e2e_ms": 16581.727329092846, + "request_id": 45, + "tpot_ms": 128.200721970688, + "ttft_ms": 300.23563881547057 + }, + { + "e2e_ms": 16478.854197918037, + "request_id": 46, + "tpot_ms": 126.8967260219465, + "ttft_ms": 362.9699931308306 + }, + { + "e2e_ms": 16270.819161297495, + "request_id": 47, + "tpot_ms": 125.57481733363996, + "ttft_ms": 322.8173599252209 + }, + { + "e2e_ms": 16062.627892404027, + "request_id": 48, + "tpot_ms": 124.25122687214136, + "ttft_ms": 282.72207964207576 + }, + { + "e2e_ms": 15958.430745404816, + "request_id": 49, + "tpot_ms": 122.91862703441548, + "ttft_ms": 347.7651120340504 + }, + { + "e2e_ms": 15749.715539461335, + "request_id": 50, + "tpot_ms": 121.58574387256766, + "ttft_ms": 308.3260676452415 + }, + { + "e2e_ms": 15645.048986506545, + "request_id": 51, + "tpot_ms": 120.24919842858912, + "ttft_ms": 373.40078607572735 + }, + { + "e2e_ms": 15435.036319721643, + "request_id": 52, + "tpot_ms": 118.8997102377205, + "ttft_ms": 334.77311953113895 + }, + { + "e2e_ms": 15224.012292267464, + "request_id": 53, + "tpot_ms": 117.54201604221191, + "ttft_ms": 296.1762549065519 + }, + { + "e2e_ms": 15109.12675155149, + "request_id": 54, + "tpot_ms": 116.11789378026353, + "ttft_ms": 362.1542414580219 + }, + { + "e2e_ms": 14893.238736493451, + "request_id": 55, + "tpot_ms": 114.72141540015762, + "ttft_ms": 323.6189806734337 + }, + { + "e2e_ms": 14675.411853468433, + "request_id": 56, + "tpot_ms": 113.30942780834319, + "ttft_ms": 285.1145218088469 + }, + { + "e2e_ms": 14549.812898218817, + "request_id": 57, + "tpot_ms": 111.79839094375423, + "ttft_ms": 351.4172483620292 + }, + { + "e2e_ms": 14329.820655618192, + "request_id": 58, + "tpot_ms": 110.36886820693505, + "ttft_ms": 312.97439333744137 + }, + { + "e2e_ms": 14108.094791808619, + "request_id": 59, + "tpot_ms": 108.92545237461232, + "ttft_ms": 274.56234023285475 + }, + { + "e2e_ms": 13975.596499372607, + "request_id": 60, + "tpot_ms": 107.35951607044248, + "ttft_ms": 340.93795842641225 + }, + { + "e2e_ms": 13753.714210521188, + "request_id": 61, + "tpot_ms": 105.91438347527057, + "ttft_ms": 302.58750916182464 + }, + { + "e2e_ms": 13622.257898704496, + "request_id": 62, + "tpot_ms": 104.35568588071192, + "ttft_ms": 369.08579185408286 + }, + { + "e2e_ms": 13397.788527763809, + "request_id": 63, + "tpot_ms": 102.88969749082138, + "ttft_ms": 330.7969464294942 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 31.752947092056274, + "offered_request_rate": 0.1, + "request_metrics_sha256": "b45cd3a473a98bee1127b2315d0c0c59ded580275f6e6d10a3e5a52d85d24de4", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 11656.879780530655, + "request_id": 0, + "tpot_ms": 89.67953789863164, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 11729.945143691864, + "request_id": 1, + "tpot_ms": 89.99959080602144, + "ttft_ms": 299.9971113271407 + }, + { + "e2e_ms": 11712.730146443742, + "request_id": 2, + "tpot_ms": 90.00056214193414, + "ttft_ms": 282.65875441810806 + }, + { + "e2e_ms": 11785.795509604945, + "request_id": 3, + "tpot_ms": 89.99959080602139, + "ttft_ms": 355.8474772402285 + }, + { + "e2e_ms": 11770.95360336775, + "request_id": 4, + "tpot_ms": 90.01924789792568, + "ttft_ms": 338.50912033118874 + }, + { + "e2e_ms": 11756.111697130556, + "request_id": 5, + "tpot_ms": 90.0192478979257, + "ttft_ms": 323.66721409399446 + }, + { + "e2e_ms": 11741.269790893028, + "request_id": 6, + "tpot_ms": 90.01924789792305, + "ttft_ms": 308.8253078568002 + }, + { + "e2e_ms": 11726.427884655508, + "request_id": 7, + "tpot_ms": 90.01924789792311, + "ttft_ms": 293.98340161927194 + }, + { + "e2e_ms": 11709.212887407048, + "request_id": 8, + "tpot_ms": 90.00056214193147, + "ttft_ms": 279.1414953817508 + }, + { + "e2e_ms": 11782.278250567928, + "request_id": 9, + "tpot_ms": 89.99959080601886, + "ttft_ms": 352.3302182035337 + }, + { + "e2e_ms": 11767.436344330406, + "request_id": 10, + "tpot_ms": 90.01924789792311, + "ttft_ms": 334.9918612941707 + }, + { + "e2e_ms": 11752.594438092885, + "request_id": 11, + "tpot_ms": 90.01924789792311, + "ttft_ms": 320.14995505664956 + }, + { + "e2e_ms": 11737.752531855705, + "request_id": 12, + "tpot_ms": 90.01924789792581, + "ttft_ms": 305.30804881912843 + }, + { + "e2e_ms": 11722.91062561817, + "request_id": 13, + "tpot_ms": 90.019247897923, + "ttft_ms": 290.46614258194836 + }, + { + "e2e_ms": 11705.69562836971, + "request_id": 14, + "tpot_ms": 90.00056214193148, + "ttft_ms": 275.624236344413 + }, + { + "e2e_ms": 11778.76099153056, + "request_id": 15, + "tpot_ms": 89.99959080601862, + "ttft_ms": 348.8129591661959 + }, + { + "e2e_ms": 11763.919085293026, + "request_id": 16, + "tpot_ms": 90.019247897923, + "ttft_ms": 331.4746022568045 + }, + { + "e2e_ms": 11749.07717905549, + "request_id": 17, + "tpot_ms": 90.019247897923, + "ttft_ms": 316.63269601926913 + }, + { + "e2e_ms": 11734.235272817954, + "request_id": 18, + "tpot_ms": 90.019247897923, + "ttft_ms": 301.7907897817338 + }, + { + "e2e_ms": 11719.39336658042, + "request_id": 19, + "tpot_ms": 90.019247897923, + "ttft_ms": 286.94888354419845 + }, + { + "e2e_ms": 11702.178369331961, + "request_id": 20, + "tpot_ms": 90.00056214193148, + "ttft_ms": 272.1069773066631 + }, + { + "e2e_ms": 11775.24373249281, + "request_id": 21, + "tpot_ms": 89.99959080601862, + "ttft_ms": 345.295700128446 + }, + { + "e2e_ms": 11760.401826255276, + "request_id": 22, + "tpot_ms": 90.01924789792301, + "ttft_ms": 327.95734321905456 + }, + { + "e2e_ms": 11745.55992001774, + "request_id": 23, + "tpot_ms": 90.019247897923, + "ttft_ms": 313.1154369815192 + }, + { + "e2e_ms": 11730.718013780204, + "request_id": 24, + "tpot_ms": 90.019247897923, + "ttft_ms": 298.2735307439839 + }, + { + "e2e_ms": 11713.50301653041, + "request_id": 25, + "tpot_ms": 90.00056214192095, + "ttft_ms": 283.43162450644854 + }, + { + "e2e_ms": 11786.568379689925, + "request_id": 26, + "tpot_ms": 89.9995908060081, + "ttft_ms": 356.6203473268956 + }, + { + "e2e_ms": 11771.726473451054, + "request_id": 27, + "tpot_ms": 90.01924789791249, + "ttft_ms": 339.28199041616836 + }, + { + "e2e_ms": 11756.884567212182, + "request_id": 28, + "tpot_ms": 90.01924789791249, + "ttft_ms": 324.4400841772972 + }, + { + "e2e_ms": 11742.042660973311, + "request_id": 29, + "tpot_ms": 90.01924789791249, + "ttft_ms": 309.59817793842603 + }, + { + "e2e_ms": 11727.20075473444, + "request_id": 30, + "tpot_ms": 90.01924789791249, + "ttft_ms": 294.7562716995549 + }, + { + "e2e_ms": 11709.985757484617, + "request_id": 31, + "tpot_ms": 90.00056214192074, + "ttft_ms": 279.9143654606837 + }, + { + "e2e_ms": 11783.051120644132, + "request_id": 32, + "tpot_ms": 89.9995908060081, + "ttft_ms": 353.1030882811024 + }, + { + "e2e_ms": 11768.20921440526, + "request_id": 33, + "tpot_ms": 90.01924789791249, + "ttft_ms": 335.7647313703751 + }, + { + "e2e_ms": 11753.36730816639, + "request_id": 34, + "tpot_ms": 90.01924789791249, + "ttft_ms": 320.92282513150394 + }, + { + "e2e_ms": 11738.525401927518, + "request_id": 35, + "tpot_ms": 90.01924789791249, + "ttft_ms": 306.0809188926328 + }, + { + "e2e_ms": 11723.683495688647, + "request_id": 36, + "tpot_ms": 90.01924789791249, + "ttft_ms": 291.2390126537616 + }, + { + "e2e_ms": 11706.468498438824, + "request_id": 37, + "tpot_ms": 90.00056214192074, + "ttft_ms": 276.39710641489046 + }, + { + "e2e_ms": 11779.533861598338, + "request_id": 38, + "tpot_ms": 89.9995908060081, + "ttft_ms": 349.5858292353091 + }, + { + "e2e_ms": 11764.691955359467, + "request_id": 39, + "tpot_ms": 90.01924789791249, + "ttft_ms": 332.24747232458185 + }, + { + "e2e_ms": 11749.850049120596, + "request_id": 40, + "tpot_ms": 90.01924789791249, + "ttft_ms": 317.4055660857107 + }, + { + "e2e_ms": 11735.008142881725, + "request_id": 41, + "tpot_ms": 90.01924789791249, + "ttft_ms": 302.5636598468395 + }, + { + "e2e_ms": 11720.166236642854, + "request_id": 42, + "tpot_ms": 90.01924789791249, + "ttft_ms": 287.72175360796837 + }, + { + "e2e_ms": 11702.95123939303, + "request_id": 43, + "tpot_ms": 90.00056214192074, + "ttft_ms": 272.8798473690972 + }, + { + "e2e_ms": 11776.016602552545, + "request_id": 44, + "tpot_ms": 89.9995908060081, + "ttft_ms": 346.06857018951587 + }, + { + "e2e_ms": 11761.174696313674, + "request_id": 45, + "tpot_ms": 90.01924789791249, + "ttft_ms": 328.7302132787886 + }, + { + "e2e_ms": 11746.332790074803, + "request_id": 46, + "tpot_ms": 90.01924789791249, + "ttft_ms": 313.88830703991744 + }, + { + "e2e_ms": 11731.490883835932, + "request_id": 47, + "tpot_ms": 90.01924789791249, + "ttft_ms": 299.0464008010463 + }, + { + "e2e_ms": 11716.64897759706, + "request_id": 48, + "tpot_ms": 90.01924789791249, + "ttft_ms": 284.2044945621751 + }, + { + "e2e_ms": 11699.433980347238, + "request_id": 49, + "tpot_ms": 90.00056214192074, + "ttft_ms": 269.36258832330395 + }, + { + "e2e_ms": 11772.499343506752, + "request_id": 50, + "tpot_ms": 89.9995908060081, + "ttft_ms": 342.5513111437226 + }, + { + "e2e_ms": 11757.657437270042, + "request_id": 51, + "tpot_ms": 90.0192478979295, + "ttft_ms": 325.21295423299534 + }, + { + "e2e_ms": 11742.815531033444, + "request_id": 52, + "tpot_ms": 90.01924789792636, + "ttft_ms": 310.3710479967958 + }, + { + "e2e_ms": 11727.973624796847, + "request_id": 53, + "tpot_ms": 90.01924789792636, + "ttft_ms": 295.5291417601984 + }, + { + "e2e_ms": 11710.758627549352, + "request_id": 54, + "tpot_ms": 90.00056214193505, + "ttft_ms": 280.687235523601 + }, + { + "e2e_ms": 11783.823990711198, + "request_id": 55, + "tpot_ms": 89.99959080602287, + "ttft_ms": 353.8759583462934 + }, + { + "e2e_ms": 11768.982084474601, + "request_id": 56, + "tpot_ms": 90.01924789792636, + "ttft_ms": 336.5376014379535 + }, + { + "e2e_ms": 11754.140178238004, + "request_id": 57, + "tpot_ms": 90.01924789792636, + "ttft_ms": 321.6956952013561 + }, + { + "e2e_ms": 11739.298272001406, + "request_id": 58, + "tpot_ms": 90.01924789792636, + "ttft_ms": 306.8537889647587 + }, + { + "e2e_ms": 11724.456365764809, + "request_id": 59, + "tpot_ms": 90.01924789792636, + "ttft_ms": 292.01188272816125 + }, + { + "e2e_ms": 11707.241368517316, + "request_id": 60, + "tpot_ms": 90.00056214193505, + "ttft_ms": 277.16997649156383 + }, + { + "e2e_ms": 11780.30673167916, + "request_id": 61, + "tpot_ms": 89.99959080602287, + "ttft_ms": 350.3586993142562 + }, + { + "e2e_ms": 11765.464825442563, + "request_id": 62, + "tpot_ms": 90.01924789792635, + "ttft_ms": 333.0203424059164 + }, + { + "e2e_ms": 11530.943387303068, + "request_id": 63, + "tpot_ms": 88.28948780420275, + "ttft_ms": 318.17843616931896 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 28.59816026687622, + "offered_request_rate": 0.2, + "request_metrics_sha256": "7930ffb53e440b9eefa724912c777842a9c33cefdb46d9014e1c5003223221d2", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 11975.67900227787, + "request_id": 0, + "tpot_ms": 92.18976799112939, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12131.199338988026, + "request_id": 1, + "tpot_ms": 93.2890801728389, + "ttft_ms": 283.4861570374869 + }, + { + "e2e_ms": 12196.411822684866, + "request_id": 2, + "tpot_ms": 93.31228634935276, + "ttft_ms": 345.75145631706584 + }, + { + "e2e_ms": 12171.220586311463, + "request_id": 3, + "tpot_ms": 93.31547315047263, + "ttft_ms": 320.15549620144054 + }, + { + "e2e_ms": 12145.933482881133, + "request_id": 4, + "tpot_ms": 93.31640326087744, + "ttft_ms": 294.75026874969853 + }, + { + "e2e_ms": 12120.646379450847, + "request_id": 5, + "tpot_ms": 93.31564840216181, + "ttft_ms": 269.5590323762964 + }, + { + "e2e_ms": 12185.858863147729, + "request_id": 6, + "tpot_ms": 93.31471829175732, + "ttft_ms": 334.88964009454844 + }, + { + "e2e_ms": 12160.571759717443, + "request_id": 7, + "tpot_ms": 93.31471829175764, + "ttft_ms": 309.6025366642223 + }, + { + "e2e_ms": 12135.284656287155, + "request_id": 8, + "tpot_ms": 93.31564840216245, + "ttft_ms": 284.19730921252295 + }, + { + "e2e_ms": 12200.497139984038, + "request_id": 9, + "tpot_ms": 93.31471829175764, + "ttft_ms": 349.52791693081764 + }, + { + "e2e_ms": 12175.305903610677, + "request_id": 10, + "tpot_ms": 93.3154731504736, + "ttft_ms": 324.2408135005306 + }, + { + "e2e_ms": 12150.018800180305, + "request_id": 11, + "tpot_ms": 93.31640326087775, + "ttft_ms": 298.8355860488312 + }, + { + "e2e_ms": 12124.731696749934, + "request_id": 12, + "tpot_ms": 93.31564840216113, + "ttft_ms": 273.6443496754717 + }, + { + "e2e_ms": 12189.94418044673, + "request_id": 13, + "tpot_ms": 93.31471829175568, + "ttft_ms": 338.97495739375927 + }, + { + "e2e_ms": 12164.657077016358, + "request_id": 14, + "tpot_ms": 93.31471829175568, + "ttft_ms": 313.68785396338694 + }, + { + "e2e_ms": 12139.369973585985, + "request_id": 15, + "tpot_ms": 93.31564840216049, + "ttft_ms": 288.2826265116023 + }, + { + "e2e_ms": 12204.582457282782, + "request_id": 16, + "tpot_ms": 93.31471829175568, + "ttft_ms": 353.6132342298117 + }, + { + "e2e_ms": 12179.391220909338, + "request_id": 17, + "tpot_ms": 93.31547315047163, + "ttft_ms": 328.3261307994394 + }, + { + "e2e_ms": 12154.104117478966, + "request_id": 18, + "tpot_ms": 93.31640326087647, + "ttft_ms": 302.92090334765476 + }, + { + "e2e_ms": 12128.817014048593, + "request_id": 19, + "tpot_ms": 93.31564840216049, + "ttft_ms": 277.72966697421 + }, + { + "e2e_ms": 12194.02949774539, + "request_id": 20, + "tpot_ms": 93.31471829175568, + "ttft_ms": 343.0602746924194 + }, + { + "e2e_ms": 12168.742394315017, + "request_id": 21, + "tpot_ms": 93.31471829175567, + "ttft_ms": 317.7731712620471 + }, + { + "e2e_ms": 12143.455290884645, + "request_id": 22, + "tpot_ms": 93.31564840216049, + "ttft_ms": 292.36794381026243 + }, + { + "e2e_ms": 12208.667774581443, + "request_id": 23, + "tpot_ms": 93.31471829175568, + "ttft_ms": 357.69855152847185 + }, + { + "e2e_ms": 12183.476538207999, + "request_id": 24, + "tpot_ms": 93.31547315047165, + "ttft_ms": 332.4114480980995 + }, + { + "e2e_ms": 12158.189434777625, + "request_id": 25, + "tpot_ms": 93.31640326087646, + "ttft_ms": 307.0062206463149 + }, + { + "e2e_ms": 12132.902331347253, + "request_id": 26, + "tpot_ms": 93.31564840216049, + "ttft_ms": 281.8149842728701 + }, + { + "e2e_ms": 12198.11481504405, + "request_id": 27, + "tpot_ms": 93.31471829175567, + "ttft_ms": 347.1455919910795 + }, + { + "e2e_ms": 12172.923578670605, + "request_id": 28, + "tpot_ms": 93.31547315047163, + "ttft_ms": 321.8584885607072 + }, + { + "e2e_ms": 12147.636475240233, + "request_id": 29, + "tpot_ms": 93.31640326087646, + "ttft_ms": 296.45326110892256 + }, + { + "e2e_ms": 12122.349371809862, + "request_id": 30, + "tpot_ms": 93.3156484021605, + "ttft_ms": 271.2620247354778 + }, + { + "e2e_ms": 12187.561855506658, + "request_id": 31, + "tpot_ms": 93.31471829175568, + "ttft_ms": 336.5926324536872 + }, + { + "e2e_ms": 12162.274752076286, + "request_id": 32, + "tpot_ms": 93.31471829175568, + "ttft_ms": 311.3055290233149 + }, + { + "e2e_ms": 12136.987648645914, + "request_id": 33, + "tpot_ms": 93.3156484021605, + "ttft_ms": 285.90030157153024 + }, + { + "e2e_ms": 12202.20013234271, + "request_id": 34, + "tpot_ms": 93.31471829175568, + "ttft_ms": 351.23090928973966 + }, + { + "e2e_ms": 12177.008895969266, + "request_id": 35, + "tpot_ms": 93.31547315047163, + "ttft_ms": 325.94380585936733 + }, + { + "e2e_ms": 12151.721792538894, + "request_id": 36, + "tpot_ms": 93.31640326087647, + "ttft_ms": 300.5385784075827 + }, + { + "e2e_ms": 12126.43468910852, + "request_id": 37, + "tpot_ms": 93.31564840216049, + "ttft_ms": 275.3473420341379 + }, + { + "e2e_ms": 12191.647172805318, + "request_id": 38, + "tpot_ms": 93.31471829175568, + "ttft_ms": 340.67794975234733 + }, + { + "e2e_ms": 12166.360069374947, + "request_id": 39, + "tpot_ms": 93.3147182917557, + "ttft_ms": 315.390846321975 + }, + { + "e2e_ms": 12141.072965944573, + "request_id": 40, + "tpot_ms": 93.31564840216049, + "ttft_ms": 289.98561887019036 + }, + { + "e2e_ms": 12206.285449641371, + "request_id": 41, + "tpot_ms": 93.31471829175568, + "ttft_ms": 355.3162265883998 + }, + { + "e2e_ms": 12181.094213267927, + "request_id": 42, + "tpot_ms": 93.31547315047165, + "ttft_ms": 330.02912315802746 + }, + { + "e2e_ms": 12155.807109837553, + "request_id": 43, + "tpot_ms": 93.31640326087646, + "ttft_ms": 304.6238957062428 + }, + { + "e2e_ms": 12130.520006407181, + "request_id": 44, + "tpot_ms": 93.31564840216049, + "ttft_ms": 279.43265933279804 + }, + { + "e2e_ms": 12195.73249010398, + "request_id": 45, + "tpot_ms": 93.3147182917557, + "ttft_ms": 344.76326705100746 + }, + { + "e2e_ms": 12170.445386673606, + "request_id": 46, + "tpot_ms": 93.31471829175568, + "ttft_ms": 319.47616362063513 + }, + { + "e2e_ms": 12145.158283243234, + "request_id": 47, + "tpot_ms": 93.31564840216049, + "ttft_ms": 294.0709361688505 + }, + { + "e2e_ms": 12210.37076694003, + "request_id": 48, + "tpot_ms": 93.31471829175568, + "ttft_ms": 359.4015438870599 + }, + { + "e2e_ms": 12185.179530566586, + "request_id": 49, + "tpot_ms": 93.31547315047163, + "ttft_ms": 334.1144404566876 + }, + { + "e2e_ms": 12159.892427136241, + "request_id": 50, + "tpot_ms": 93.31640326087668, + "ttft_ms": 308.70921300490295 + }, + { + "e2e_ms": 12134.605323705899, + "request_id": 51, + "tpot_ms": 93.31564840216095, + "ttft_ms": 283.51797663145817 + }, + { + "e2e_ms": 12199.817807402724, + "request_id": 52, + "tpot_ms": 93.31471829175612, + "ttft_ms": 348.848584349696 + }, + { + "e2e_ms": 12174.626571029308, + "request_id": 53, + "tpot_ms": 93.3154731504721, + "ttft_ms": 323.5614809193521 + }, + { + "e2e_ms": 12149.339467598964, + "request_id": 54, + "tpot_ms": 93.31640326087691, + "ttft_ms": 298.1562534675959 + }, + { + "e2e_ms": 12124.05236416862, + "request_id": 55, + "tpot_ms": 93.31564840216095, + "ttft_ms": 272.96501709417953 + }, + { + "e2e_ms": 12189.264847865445, + "request_id": 56, + "tpot_ms": 93.31471829175612, + "ttft_ms": 338.2956248124174 + }, + { + "e2e_ms": 12163.977744435102, + "request_id": 57, + "tpot_ms": 93.31471829175614, + "ttft_ms": 313.00852138207347 + }, + { + "e2e_ms": 12138.690641004758, + "request_id": 58, + "tpot_ms": 93.31564840216095, + "ttft_ms": 287.60329393031725 + }, + { + "e2e_ms": 12203.903124701583, + "request_id": 59, + "tpot_ms": 93.31471829175612, + "ttft_ms": 352.9339016485551 + }, + { + "e2e_ms": 12178.711888328166, + "request_id": 60, + "tpot_ms": 93.31547315047209, + "ttft_ms": 327.6467982182112 + }, + { + "e2e_ms": 12153.424784897823, + "request_id": 61, + "tpot_ms": 93.31640326087691, + "ttft_ms": 302.24157076645497 + }, + { + "e2e_ms": 11949.074413391485, + "request_id": 62, + "tpot_ms": 91.90570140943657, + "ttft_ms": 277.0503343930386 + }, + { + "e2e_ms": 11702.728382593137, + "request_id": 63, + "tpot_ms": 89.45155464946347, + "ttft_ms": 342.38094211127645 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 24.987496376037598, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e62567aa33ec429331fcd9f24053a0c79db61ef288f7994e6488ae0f0fa79b7e", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 12394.3405299904, + "request_id": 0, + "tpot_ms": 95.48631545343278, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12520.094438712134, + "request_id": 1, + "tpot_ms": 96.04733720338449, + "ttft_ms": 322.08261388230494 + }, + { + "e2e_ms": 12556.880443424032, + "request_id": 2, + "tpot_ms": 96.10807445129032, + "ttft_ms": 351.1549881101628 + }, + { + "e2e_ms": 12502.892311945501, + "request_id": 3, + "tpot_ms": 96.14618123456218, + "ttft_ms": 292.32729515610424 + }, + { + "e2e_ms": 12539.549613984442, + "request_id": 4, + "tpot_ms": 96.30977229626299, + "ttft_ms": 308.2085323590427 + }, + { + "e2e_ms": 12576.335618696337, + "request_id": 5, + "tpot_ms": 96.32399459689755, + "ttft_ms": 343.188304890349 + }, + { + "e2e_ms": 12522.347487217807, + "request_id": 6, + "tpot_ms": 96.32399459689753, + "ttft_ms": 289.200173411821 + }, + { + "e2e_ms": 12559.133491929704, + "request_id": 7, + "tpot_ms": 96.32399459689753, + "ttft_ms": 325.98617812371697 + }, + { + "e2e_ms": 12505.145360451173, + "request_id": 8, + "tpot_ms": 96.32399459689753, + "ttft_ms": 271.99804664518723 + }, + { + "e2e_ms": 12541.802662490163, + "request_id": 9, + "tpot_ms": 96.32399459689792, + "ttft_ms": 308.6553486841268 + }, + { + "e2e_ms": 12578.588667202104, + "request_id": 10, + "tpot_ms": 96.32399459689827, + "ttft_ms": 345.4413533960228 + }, + { + "e2e_ms": 12524.60053572362, + "request_id": 11, + "tpot_ms": 96.32399459689864, + "ttft_ms": 291.453221917493 + }, + { + "e2e_ms": 12561.386540435564, + "request_id": 12, + "tpot_ms": 96.32399459689903, + "ttft_ms": 328.239226629389 + }, + { + "e2e_ms": 12507.39840895708, + "request_id": 13, + "tpot_ms": 96.32399459689938, + "ttft_ms": 274.2510951508592 + }, + { + "e2e_ms": 12544.05571099607, + "request_id": 14, + "tpot_ms": 96.32399459689938, + "ttft_ms": 310.90839718984853 + }, + { + "e2e_ms": 12580.841715708011, + "request_id": 15, + "tpot_ms": 96.32399459689938, + "ttft_ms": 347.6944019017907 + }, + { + "e2e_ms": 12526.853584229528, + "request_id": 16, + "tpot_ms": 96.32399459689938, + "ttft_ms": 293.7062704233071 + }, + { + "e2e_ms": 12563.63958894147, + "request_id": 17, + "tpot_ms": 96.32399459689938, + "ttft_ms": 330.49227513524926 + }, + { + "e2e_ms": 12509.651457462987, + "request_id": 18, + "tpot_ms": 96.32399459689938, + "ttft_ms": 276.5041436567657 + }, + { + "e2e_ms": 12546.308759501975, + "request_id": 19, + "tpot_ms": 96.32399459689938, + "ttft_ms": 313.161445695755 + }, + { + "e2e_ms": 12583.094764213918, + "request_id": 20, + "tpot_ms": 96.32399459689938, + "ttft_ms": 349.94745040769715 + }, + { + "e2e_ms": 12529.106632735506, + "request_id": 21, + "tpot_ms": 96.32399459689994, + "ttft_ms": 295.95931892921357 + }, + { + "e2e_ms": 12565.892637447447, + "request_id": 22, + "tpot_ms": 96.32399459689992, + "ttft_ms": 332.7453236411557 + }, + { + "e2e_ms": 12511.904505968956, + "request_id": 23, + "tpot_ms": 96.32399459689988, + "ttft_ms": 278.75719216267214 + }, + { + "e2e_ms": 12548.561808007931, + "request_id": 24, + "tpot_ms": 96.32399459689977, + "ttft_ms": 315.41449420166146 + }, + { + "e2e_ms": 12585.347812719874, + "request_id": 25, + "tpot_ms": 96.32399459689977, + "ttft_ms": 352.2004989136036 + }, + { + "e2e_ms": 12531.359681241383, + "request_id": 26, + "tpot_ms": 96.3239945968991, + "ttft_ms": 298.2123674351982 + }, + { + "e2e_ms": 12568.145685953326, + "request_id": 27, + "tpot_ms": 96.32399459689911, + "ttft_ms": 334.99837214714034 + }, + { + "e2e_ms": 12514.157554474836, + "request_id": 28, + "tpot_ms": 96.32399459689911, + "ttft_ms": 281.01024066864966 + }, + { + "e2e_ms": 12550.81485651381, + "request_id": 29, + "tpot_ms": 96.32399459689911, + "ttft_ms": 317.66754270762476 + }, + { + "e2e_ms": 12587.600861225752, + "request_id": 30, + "tpot_ms": 96.3239945968991, + "ttft_ms": 354.4535474195669 + }, + { + "e2e_ms": 12533.612729747261, + "request_id": 31, + "tpot_ms": 96.3239945968991, + "ttft_ms": 300.46541594107623 + }, + { + "e2e_ms": 12570.398734459204, + "request_id": 32, + "tpot_ms": 96.32399459689911, + "ttft_ms": 337.2514206530184 + }, + { + "e2e_ms": 12516.410602980714, + "request_id": 33, + "tpot_ms": 96.32399459689911, + "ttft_ms": 283.2632891745277 + }, + { + "e2e_ms": 12553.067905019689, + "request_id": 34, + "tpot_ms": 96.32399459689911, + "ttft_ms": 319.9205912135028 + }, + { + "e2e_ms": 12589.85390973163, + "request_id": 35, + "tpot_ms": 96.3239945968991, + "ttft_ms": 356.70659592544496 + }, + { + "e2e_ms": 12535.865778253139, + "request_id": 36, + "tpot_ms": 96.3239945968991, + "ttft_ms": 302.7184644469543 + }, + { + "e2e_ms": 12572.651782965082, + "request_id": 37, + "tpot_ms": 96.3239945968991, + "ttft_ms": 339.5044691588964 + }, + { + "e2e_ms": 12518.663651486591, + "request_id": 38, + "tpot_ms": 96.3239945968991, + "ttft_ms": 285.51633768040574 + }, + { + "e2e_ms": 12555.320953525566, + "request_id": 39, + "tpot_ms": 96.3239945968991, + "ttft_ms": 322.17363971938084 + }, + { + "e2e_ms": 12770.519034100516, + "request_id": 40, + "tpot_ms": 97.72881409188341, + "ttft_ms": 358.959644431323 + }, + { + "e2e_ms": 12538.070601439798, + "request_id": 41, + "tpot_ms": 96.32361486997611, + "ttft_ms": 304.9715129528323 + }, + { + "e2e_ms": 12574.856606151741, + "request_id": 42, + "tpot_ms": 96.32361486997611, + "ttft_ms": 341.75751766477447 + }, + { + "e2e_ms": 12520.86847467325, + "request_id": 43, + "tpot_ms": 96.32361486997611, + "ttft_ms": 287.7693861862838 + }, + { + "e2e_ms": 12557.654479385192, + "request_id": 44, + "tpot_ms": 96.32462827684986, + "ttft_ms": 324.4266882252589 + }, + { + "e2e_ms": 12503.666347906701, + "request_id": 45, + "tpot_ms": 96.3239945968991, + "ttft_ms": 270.51903410051636 + }, + { + "e2e_ms": 12540.323649945676, + "request_id": 46, + "tpot_ms": 96.3239945968991, + "ttft_ms": 307.17633613949147 + }, + { + "e2e_ms": 12577.109654657448, + "request_id": 47, + "tpot_ms": 96.32399459689775, + "ttft_ms": 343.9623408514336 + }, + { + "e2e_ms": 12523.121523178786, + "request_id": 48, + "tpot_ms": 96.3239945968964, + "ttft_ms": 289.97420937294294 + }, + { + "e2e_ms": 12559.907527890544, + "request_id": 49, + "tpot_ms": 96.32399459689495, + "ttft_ms": 326.7602140848851 + }, + { + "e2e_ms": 12505.919396411884, + "request_id": 50, + "tpot_ms": 96.32399459689363, + "ttft_ms": 272.7720826063944 + }, + { + "e2e_ms": 12542.576698450688, + "request_id": 51, + "tpot_ms": 96.32399459689228, + "ttft_ms": 309.4293846453695 + }, + { + "e2e_ms": 12579.362703162446, + "request_id": 52, + "tpot_ms": 96.32399459689228, + "ttft_ms": 346.2153893571269 + }, + { + "e2e_ms": 12525.374571683784, + "request_id": 53, + "tpot_ms": 96.32399459689228, + "ttft_ms": 292.2272578784657 + }, + { + "e2e_ms": 12562.160576395541, + "request_id": 54, + "tpot_ms": 96.32399459689228, + "ttft_ms": 329.0132625902231 + }, + { + "e2e_ms": 12508.17244491688, + "request_id": 55, + "tpot_ms": 96.32399459689226, + "ttft_ms": 275.0251311115619 + }, + { + "e2e_ms": 12544.829746955686, + "request_id": 56, + "tpot_ms": 96.32399459689228, + "ttft_ms": 311.6824331503665 + }, + { + "e2e_ms": 12581.615751667443, + "request_id": 57, + "tpot_ms": 96.32399459689228, + "ttft_ms": 348.4684378621239 + }, + { + "e2e_ms": 12527.627620188781, + "request_id": 58, + "tpot_ms": 96.32399459689228, + "ttft_ms": 294.4803063834627 + }, + { + "e2e_ms": 12564.413624900539, + "request_id": 59, + "tpot_ms": 96.32399459689228, + "ttft_ms": 331.2663110952201 + }, + { + "e2e_ms": 12314.305135768365, + "request_id": 60, + "tpot_ms": 94.77973981221895, + "ttft_ms": 277.2781796165589 + }, + { + "e2e_ms": 12170.36562563149, + "request_id": 61, + "tpot_ms": 93.35771766910335, + "ttft_ms": 313.93548165536345 + }, + { + "e2e_ms": 12023.933572014499, + "request_id": 62, + "tpot_ms": 91.91505579249903, + "ttft_ms": 350.72148636712086 + }, + { + "e2e_ms": 11724.699298497455, + "request_id": 63, + "tpot_ms": 89.98398380794484, + "ttft_ms": 296.73335488845964 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 25.64221477508545, + "offered_request_rate": 0.8, + "request_metrics_sha256": "b39b2c6a1c6236e30b246303e0b4d66b80a2515899d8c815efc015ef3f70db54", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 14028.106647728775, + "request_id": 0, + "tpot_ms": 108.35061559310502, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 14267.352384844213, + "request_id": 1, + "tpot_ms": 109.65331923259451, + "ttft_ms": 341.3808423047111 + }, + { + "e2e_ms": 14406.16288598222, + "request_id": 2, + "tpot_ms": 110.64064370419715, + "ttft_ms": 354.8011355491818 + }, + { + "e2e_ms": 14444.508775243683, + "request_id": 3, + "tpot_ms": 111.53949237897281, + "ttft_ms": 278.99324311413756 + }, + { + "e2e_ms": 14585.312621821135, + "request_id": 4, + "tpot_ms": 112.58765687327464, + "ttft_ms": 286.6801989152554 + }, + { + "e2e_ms": 14727.157204966976, + "request_id": 5, + "tpot_ms": 113.57630070139629, + "ttft_ms": 302.9670158896467 + }, + { + "e2e_ms": 14871.045643448613, + "request_id": 6, + "tpot_ms": 114.43037546464966, + "ttft_ms": 338.38795943810675 + }, + { + "e2e_ms": 14913.487485283544, + "request_id": 7, + "tpot_ms": 115.12950641852738, + "ttft_ms": 292.0401701305657 + }, + { + "e2e_ms": 15059.457396901958, + "request_id": 8, + "tpot_ms": 115.80285438725795, + "ttft_ms": 352.4948897201998 + }, + { + "e2e_ms": 15271.481983406971, + "request_id": 9, + "tpot_ms": 117.58757541921905, + "ttft_ms": 337.85990516615175 + }, + { + "e2e_ms": 15316.138289012584, + "request_id": 10, + "tpot_ms": 117.88145073243837, + "ttft_ms": 345.1940459929119 + }, + { + "e2e_ms": 15090.835365768904, + "request_id": 11, + "tpot_ms": 116.63565919716638, + "ttft_ms": 278.1066477287748 + }, + { + "e2e_ms": 15302.859952273917, + "request_id": 12, + "tpot_ms": 118.0348305674148, + "ttft_ms": 312.43647021223705 + }, + { + "e2e_ms": 15347.54563377864, + "request_id": 13, + "tpot_ms": 118.10044582475892, + "ttft_ms": 348.7890140342564 + }, + { + "e2e_ms": 15289.743982068245, + "request_id": 14, + "tpot_ms": 118.15013414242924, + "ttft_ms": 284.6769459797329 + }, + { + "e2e_ms": 15334.400287673823, + "request_id": 15, + "tpot_ms": 118.19982246009941, + "ttft_ms": 323.022835241197 + }, + { + "e2e_ms": 15380.492763872971, + "request_id": 16, + "tpot_ms": 118.2528557378034, + "ttft_ms": 362.38008517193873 + }, + { + "e2e_ms": 15322.691112162565, + "request_id": 17, + "tpot_ms": 118.27848723653601, + "ttft_ms": 301.32323312249196 + }, + { + "e2e_ms": 15367.376793667254, + "request_id": 18, + "tpot_ms": 118.29615526543176, + "ttft_ms": 343.765074957421 + }, + { + "e2e_ms": 15309.575141956848, + "request_id": 19, + "tpot_ms": 118.30562851819849, + "ttft_ms": 284.7603201456401 + }, + { + "e2e_ms": 15354.260823461536, + "request_id": 20, + "tpot_ms": 118.30690699483614, + "ttft_ms": 329.2836351173456 + }, + { + "e2e_ms": 15296.45917175113, + "request_id": 21, + "tpot_ms": 118.3069069948359, + "ttft_ms": 271.4819834069715 + }, + { + "e2e_ms": 15341.115477356709, + "request_id": 22, + "tpot_ms": 118.30690699483563, + "ttft_ms": 316.138289012585 + }, + { + "e2e_ms": 15387.207953555859, + "request_id": 23, + "tpot_ms": 118.31926259505173, + "ttft_ms": 360.6616039842905 + }, + { + "e2e_ms": 15329.406301845453, + "request_id": 24, + "tpot_ms": 118.31926259505147, + "ttft_ms": 302.8599522739164 + }, + { + "e2e_ms": 15375.4987780446, + "request_id": 25, + "tpot_ms": 118.33033971862962, + "ttft_ms": 347.5456337786404 + }, + { + "e2e_ms": 15317.697126334195, + "request_id": 26, + "tpot_ms": 118.33033971862953, + "ttft_ms": 289.74398206824503 + }, + { + "e2e_ms": 15362.382807838883, + "request_id": 27, + "tpot_ms": 118.33057102492174, + "ttft_ms": 334.400287673823 + }, + { + "e2e_ms": 15304.581156128477, + "request_id": 28, + "tpot_ms": 118.33057102492174, + "ttft_ms": 276.59863596341694 + }, + { + "e2e_ms": 15349.237461734056, + "request_id": 29, + "tpot_ms": 118.3192625950511, + "ttft_ms": 322.6911121625662 + }, + { + "e2e_ms": 15395.329937933206, + "request_id": 30, + "tpot_ms": 118.33033971862955, + "ttft_ms": 367.37679366725473 + }, + { + "e2e_ms": 15337.5282862228, + "request_id": 31, + "tpot_ms": 118.33033971862955, + "ttft_ms": 309.57514195684865 + }, + { + "e2e_ms": 15383.620762421948, + "request_id": 32, + "tpot_ms": 118.34141684220796, + "ttft_ms": 354.2608234615372 + }, + { + "e2e_ms": 15325.819110711542, + "request_id": 33, + "tpot_ms": 118.34141684220796, + "ttft_ms": 296.4591717511311 + }, + { + "e2e_ms": 15370.50479221623, + "request_id": 34, + "tpot_ms": 118.34164814850017, + "ttft_ms": 341.1154773567091 + }, + { + "e2e_ms": 15312.703140505824, + "request_id": 35, + "tpot_ms": 118.34164814850017, + "ttft_ms": 283.31382564630303 + }, + { + "e2e_ms": 15357.388822010513, + "request_id": 36, + "tpot_ms": 118.33057102492174, + "ttft_ms": 329.4063018454523 + }, + { + "e2e_ms": 15299.587170300107, + "request_id": 37, + "tpot_ms": 118.33057102492174, + "ttft_ms": 271.6046501350462 + }, + { + "e2e_ms": 15344.243475905685, + "request_id": 38, + "tpot_ms": 118.3192625950511, + "ttft_ms": 317.69712633419545 + }, + { + "e2e_ms": 15390.335952104835, + "request_id": 39, + "tpot_ms": 118.33033971862955, + "ttft_ms": 362.382807838884 + }, + { + "e2e_ms": 15332.53430039447, + "request_id": 40, + "tpot_ms": 118.33033971862987, + "ttft_ms": 304.5811561284779 + }, + { + "e2e_ms": 15378.626776593663, + "request_id": 41, + "tpot_ms": 118.34164814850084, + "ttft_ms": 349.2374617340559 + }, + { + "e2e_ms": 15320.8251248833, + "request_id": 42, + "tpot_ms": 118.34164814850119, + "ttft_ms": 291.4358100236498 + }, + { + "e2e_ms": 15365.51080638803, + "request_id": 43, + "tpot_ms": 118.33057102492307, + "ttft_ms": 337.52828622279907 + }, + { + "e2e_ms": 15307.709154677666, + "request_id": 44, + "tpot_ms": 118.33057102492342, + "ttft_ms": 279.726634512393 + }, + { + "e2e_ms": 15352.365460283294, + "request_id": 45, + "tpot_ms": 118.31926259505316, + "ttft_ms": 325.81911071154224 + }, + { + "e2e_ms": 15398.457936482486, + "request_id": 46, + "tpot_ms": 118.33033971863193, + "ttft_ms": 370.5047922162308 + }, + { + "e2e_ms": 15340.656284772123, + "request_id": 47, + "tpot_ms": 118.33033971863227, + "ttft_ms": 312.7031405058247 + }, + { + "e2e_ms": 15386.748760971315, + "request_id": 48, + "tpot_ms": 118.34141684221105, + "ttft_ms": 357.38882201051325 + }, + { + "e2e_ms": 15328.947109260951, + "request_id": 49, + "tpot_ms": 118.34141684221137, + "ttft_ms": 299.5871703001072 + }, + { + "e2e_ms": 15375.039585460143, + "request_id": 50, + "tpot_ms": 118.35272527208234, + "ttft_ms": 344.24347590568516 + }, + { + "e2e_ms": 15317.23793374978, + "request_id": 51, + "tpot_ms": 118.3527252720827, + "ttft_ms": 286.4418241952791 + }, + { + "e2e_ms": 15194.4223437212, + "request_id": 52, + "tpot_ms": 117.02274049863567, + "ttft_ms": 332.53430039447096 + }, + { + "e2e_ms": 14959.152693280232, + "request_id": 53, + "tpot_ms": 115.62535468185925, + "ttft_ms": 274.7326486841075 + }, + { + "e2e_ms": 14796.032219342294, + "request_id": 54, + "tpot_ms": 113.97800861778735, + "ttft_ms": 320.8251248832994 + }, + { + "e2e_ms": 14608.764053703837, + "request_id": 55, + "tpot_ms": 112.1516003725654, + "ttft_ms": 365.5108063880306 + }, + { + "e2e_ms": 14305.495223453576, + "request_id": 56, + "tpot_ms": 110.21878794311739, + "ttft_ms": 307.70915467766713 + }, + { + "e2e_ms": 14083.450969580696, + "request_id": 57, + "tpot_ms": 108.11878353777482, + "ttft_ms": 352.36546028329485 + }, + { + "e2e_ms": 13758.528543574443, + "request_id": 58, + "tpot_ms": 106.01547035434261, + "ttft_ms": 294.5638085729314 + }, + { + "e2e_ms": 13507.044041669204, + "request_id": 59, + "tpot_ms": 103.67234454249669, + "ttft_ms": 340.6562847721233 + }, + { + "e2e_ms": 13157.052579489218, + "request_id": 60, + "tpot_ms": 101.37163737344456, + "ttft_ms": 282.85463306175984 + }, + { + "e2e_ms": 12903.847402123616, + "request_id": 61, + "tpot_ms": 99.01496293592649, + "ttft_ms": 328.9471092609517 + }, + { + "e2e_ms": 12559.065843040158, + "request_id": 62, + "tpot_ms": 96.75527862590212, + "ttft_ms": 271.1454575505883 + }, + { + "e2e_ms": 12277.402762692504, + "request_id": 63, + "tpot_ms": 94.17452621214743, + "ttft_ms": 317.23793374978015 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 25.590887784957886, + "offered_request_rate": 1.2, + "request_metrics_sha256": "3825f14be2103486ce094e0b335c86deef683645cae8f05ef3d4beb6fb24cf95", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 15728.871662360909, + "request_id": 0, + "tpot_ms": 121.74246610201946, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 15904.548256233129, + "request_id": 1, + "tpot_ms": 122.72470152955445, + "ttft_ms": 318.5111619797136 + }, + { + "e2e_ms": 16140.50586206045, + "request_id": 2, + "tpot_ms": 124.75532551591714, + "ttft_ms": 296.5795215389744 + }, + { + "e2e_ms": 16211.825024161684, + "request_id": 3, + "tpot_ms": 125.48331552046938, + "ttft_ms": 275.44395306207383 + }, + { + "e2e_ms": 16388.84099440166, + "request_id": 4, + "tpot_ms": 126.36635826198417, + "ttft_ms": 340.31349512967114 + }, + { + "e2e_ms": 16460.2590055204, + "request_id": 5, + "tpot_ms": 127.08834011289514, + "ttft_ms": 320.03981118271605 + }, + { + "e2e_ms": 16531.726441149392, + "request_id": 6, + "tpot_ms": 127.72809453706097, + "ttft_ms": 310.25843494264916 + }, + { + "e2e_ms": 16603.193876778383, + "request_id": 7, + "tpot_ms": 128.3170301134443, + "ttft_ms": 306.93105237095784 + }, + { + "e2e_ms": 16674.731660081485, + "request_id": 8, + "tpot_ms": 128.84921275092722, + "ttft_ms": 310.8816407137276 + }, + { + "e2e_ms": 16746.318867894835, + "request_id": 9, + "tpot_ms": 129.28898520065795, + "ttft_ms": 326.61774741127834 + }, + { + "e2e_ms": 16982.978169021266, + "request_id": 10, + "tpot_ms": 130.92468009104587, + "ttft_ms": 355.5437974584379 + }, + { + "e2e_ms": 16948.91870099869, + "request_id": 11, + "tpot_ms": 131.099598546656, + "ttft_ms": 299.26968557337295 + }, + { + "e2e_ms": 17020.55604111586, + "request_id": 12, + "tpot_ms": 131.2589874619586, + "ttft_ms": 350.6646334471153 + }, + { + "e2e_ms": 16986.496573094286, + "request_id": 13, + "tpot_ms": 131.335760528223, + "ttft_ms": 306.854986009963 + }, + { + "e2e_ms": 17059.03976122884, + "request_id": 14, + "tpot_ms": 131.4398579526774, + "ttft_ms": 366.1778012388126 + }, + { + "e2e_ms": 17024.98029320726, + "request_id": 15, + "tpot_ms": 131.50690676411472, + "ttft_ms": 323.6031341646921 + }, + { + "e2e_ms": 16990.920825185683, + "request_id": 16, + "tpot_ms": 131.56847300071718, + "ttft_ms": 281.7247540946006 + }, + { + "e2e_ms": 17063.464013320234, + "request_id": 17, + "tpot_ms": 131.63886355765484, + "ttft_ms": 345.32834149806746 + }, + { + "e2e_ms": 17029.40454529866, + "request_id": 18, + "tpot_ms": 131.67812927130495, + "ttft_ms": 306.28212784293396 + }, + { + "e2e_ms": 16995.3450772771, + "request_id": 19, + "tpot_ms": 131.6904025418972, + "ttft_ms": 270.66395445615575 + }, + { + "e2e_ms": 17067.888265411675, + "request_id": 20, + "tpot_ms": 131.7034495326512, + "ttft_ms": 341.5501747649721 + }, + { + "e2e_ms": 17033.828797390113, + "request_id": 21, + "tpot_ms": 131.70595487135952, + "ttft_ms": 307.17252872745246 + }, + { + "e2e_ms": 16999.769329368555, + "request_id": 22, + "tpot_ms": 131.7084602100679, + "ttft_ms": 272.79488268993293 + }, + { + "e2e_ms": 17072.312517503127, + "request_id": 23, + "tpot_ms": 131.71809821033827, + "ttft_ms": 344.1140447901674 + }, + { + "e2e_ms": 17038.25304948157, + "request_id": 24, + "tpot_ms": 131.71982521031822, + "ttft_ms": 309.83524777115434 + }, + { + "e2e_ms": 17004.193581460007, + "request_id": 25, + "tpot_ms": 131.72116304093396, + "ttft_ms": 275.60587526139454 + }, + { + "e2e_ms": 17076.736769594583, + "request_id": 26, + "tpot_ms": 131.72963353311178, + "ttft_ms": 347.07331088938884 + }, + { + "e2e_ms": 17042.67730157302, + "request_id": 27, + "tpot_ms": 131.7305821943633, + "ttft_ms": 312.8933628888824 + }, + { + "e2e_ms": 17008.617833551463, + "request_id": 28, + "tpot_ms": 131.7309769369132, + "ttft_ms": 278.783762563485 + }, + { + "e2e_ms": 17081.161021686035, + "request_id": 29, + "tpot_ms": 131.73850434102513, + "ttft_ms": 350.37097037584175 + }, + { + "e2e_ms": 17047.986478516996, + "request_id": 30, + "tpot_ms": 131.74547225324986, + "ttft_ms": 316.3115023542638 + }, + { + "e2e_ms": 17013.927010495438, + "request_id": 31, + "tpot_ms": 131.74547225325, + "ttft_ms": 282.25203433268575 + }, + { + "e2e_ms": 17086.47019863001, + "request_id": 32, + "tpot_ms": 131.75260491481217, + "ttft_ms": 353.8893744488618 + }, + { + "e2e_ms": 17053.29565546097, + "request_id": 33, + "tpot_ms": 131.7595728270369, + "ttft_ms": 319.82990642728385 + }, + { + "e2e_ms": 17019.236187439412, + "request_id": 34, + "tpot_ms": 131.75957282703703, + "ttft_ms": 285.7704384057058 + }, + { + "e2e_ms": 17091.779375573984, + "request_id": 35, + "tpot_ms": 131.7595728270372, + "ttft_ms": 358.31362654026225 + }, + { + "e2e_ms": 17058.604832404944, + "request_id": 36, + "tpot_ms": 131.7665407392619, + "ttft_ms": 324.2541585186842 + }, + { + "e2e_ms": 17024.545364383386, + "request_id": 37, + "tpot_ms": 131.76654073926207, + "ttft_ms": 290.1946904971062 + }, + { + "e2e_ms": 17097.08855251796, + "request_id": 38, + "tpot_ms": 131.76654073926218, + "ttft_ms": 362.7378786316626 + }, + { + "e2e_ms": 17063.934932514778, + "request_id": 39, + "tpot_ms": 131.7736734008242, + "ttft_ms": 328.6784106101024 + }, + { + "e2e_ms": 17029.875464493216, + "request_id": 40, + "tpot_ms": 131.7736734008242, + "ttft_ms": 294.61894258854215 + }, + { + "e2e_ms": 17102.41865262779, + "request_id": 41, + "tpot_ms": 131.7736734008242, + "ttft_ms": 367.1621307231163 + }, + { + "e2e_ms": 17069.265032624608, + "request_id": 42, + "tpot_ms": 131.78080606238623, + "ttft_ms": 333.1026627015561 + }, + { + "e2e_ms": 17035.205564603046, + "request_id": 43, + "tpot_ms": 131.78080606238623, + "ttft_ms": 299.04319467999585 + }, + { + "e2e_ms": 16941.812657737883, + "request_id": 44, + "tpot_ms": 130.47422263719145, + "ttft_ms": 371.58638281457 + }, + { + "e2e_ms": 16742.48916517699, + "request_id": 45, + "tpot_ms": 129.1729311053857, + "ttft_ms": 337.5269147930098 + }, + { + "e2e_ms": 16538.03643831958, + "request_id": 46, + "tpot_ms": 127.83125190195379, + "ttft_ms": 303.46744677144954 + }, + { + "e2e_ms": 16435.957848641778, + "request_id": 47, + "tpot_ms": 126.45627727350987, + "ttft_ms": 376.0106349060237 + }, + { + "e2e_ms": 16228.476901947985, + "request_id": 48, + "tpot_ms": 125.09075381939779, + "ttft_ms": 341.9511668844635 + }, + { + "e2e_ms": 16020.052127617348, + "request_id": 49, + "tpot_ms": 123.7177986516098, + "ttft_ms": 307.89169886290324 + }, + { + "e2e_ms": 15809.427558622445, + "request_id": 50, + "tpot_ms": 122.32055435377619, + "ttft_ms": 274.71715569286914 + }, + { + "e2e_ms": 15703.511289947706, + "request_id": 51, + "tpot_ms": 120.91536178046667, + "ttft_ms": 347.26034382843807 + }, + { + "e2e_ms": 15485.101955905271, + "request_id": 52, + "tpot_ms": 119.46378803227081, + "ttft_ms": 313.20087580687783 + }, + { + "e2e_ms": 15260.624457846197, + "request_id": 53, + "tpot_ms": 117.95746555282956, + "ttft_ms": 280.02633263684373 + }, + { + "e2e_ms": 15118.941732007286, + "request_id": 54, + "tpot_ms": 116.27064733255806, + "ttft_ms": 352.56952077241266 + }, + { + "e2e_ms": 14865.280308326035, + "request_id": 55, + "tpot_ms": 114.54149807539514, + "ttft_ms": 318.5100527508524 + }, + { + "e2e_ms": 14599.985676841925, + "request_id": 56, + "tpot_ms": 112.71378084457564, + "ttft_ms": 285.3355095808183 + }, + { + "e2e_ms": 14420.806000135271, + "request_id": 57, + "tpot_ms": 110.73171104266838, + "ttft_ms": 357.87869771638725 + }, + { + "e2e_ms": 14142.519211198518, + "request_id": 58, + "tpot_ms": 108.80866127168261, + "ttft_ms": 323.819229694827 + }, + { + "e2e_ms": 13853.830695007076, + "request_id": 59, + "tpot_ms": 106.79657547493251, + "ttft_ms": 290.6656096906488 + }, + { + "e2e_ms": 13650.503338148035, + "request_id": 60, + "tpot_ms": 104.62436645922692, + "ttft_ms": 363.20879782621773 + }, + { + "e2e_ms": 13360.876271706531, + "request_id": 61, + "tpot_ms": 102.61202316458169, + "ttft_ms": 329.1493298046575 + }, + { + "e2e_ms": 13070.674002922458, + "request_id": 62, + "tpot_ms": 100.58801805607857, + "ttft_ms": 295.9957098004793 + }, + { + "e2e_ms": 12853.555073004585, + "request_id": 63, + "tpot_ms": 98.30721397691761, + "ttft_ms": 368.5388979360482 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 28.949723482131958, + "offered_request_rate": 1.6, + "request_metrics_sha256": "b7d73a1cbef8ca659f753135a4944cb3023d3a2424a19a741ae2198f39c2d172", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 17513.570713343794, + "request_id": 0, + "tpot_ms": 135.79521453495556, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 17840.282304762335, + "request_id": 1, + "tpot_ms": 137.71064841138914, + "ttft_ms": 351.0299565159147 + }, + { + "e2e_ms": 17895.63618825912, + "request_id": 2, + "tpot_ms": 138.10245652748367, + "ttft_ms": 356.6242092686922 + }, + { + "e2e_ms": 17848.77057711112, + "request_id": 3, + "tpot_ms": 138.39719968518088, + "ttft_ms": 272.32621709314975 + }, + { + "e2e_ms": 17904.1244606079, + "request_id": 4, + "tpot_ms": 138.80479077491768, + "ttft_ms": 275.9160321933569 + }, + { + "e2e_ms": 17959.478344104686, + "request_id": 5, + "tpot_ms": 139.18735062645055, + "ttft_ms": 282.6848145454632 + }, + { + "e2e_ms": 18014.921946878916, + "request_id": 6, + "tpot_ms": 139.5152186600215, + "ttft_ms": 296.48917705618504 + }, + { + "e2e_ms": 18070.36554965315, + "request_id": 7, + "tpot_ms": 139.8086106441042, + "ttft_ms": 314.671997851919 + }, + { + "e2e_ms": 18125.898871704834, + "request_id": 8, + "tpot_ms": 140.0646556798527, + "ttft_ms": 337.68760036353916 + }, + { + "e2e_ms": 18079.30241838919, + "request_id": 9, + "tpot_ms": 140.21470125635153, + "ttft_ms": 272.0353588325457 + }, + { + "e2e_ms": 18134.83860521682, + "request_id": 10, + "tpot_ms": 140.33922237580643, + "ttft_ms": 311.75736348940484 + }, + { + "e2e_ms": 18190.4645113219, + "request_id": 11, + "tpot_ms": 140.37869703173442, + "ttft_ms": 362.36998829163 + }, + { + "e2e_ms": 18143.870922782207, + "request_id": 12, + "tpot_ms": 140.38698051363622, + "ttft_ms": 314.7243975504068 + }, + { + "e2e_ms": 18097.274469466563, + "request_id": 13, + "tpot_ms": 140.36074521133403, + "ttft_ms": 271.4598276271438 + }, + { + "e2e_ms": 18152.810656294194, + "request_id": 14, + "tpot_ms": 140.32455966134754, + "ttft_ms": 331.5915793030584 + }, + { + "e2e_ms": 18106.217067754496, + "request_id": 15, + "tpot_ms": 140.29231877139563, + "ttft_ms": 289.0925837872533 + }, + { + "e2e_ms": 18161.753254582127, + "request_id": 16, + "tpot_ms": 140.24009333090902, + "ttft_ms": 351.2614015566804 + }, + { + "e2e_ms": 18115.15966604243, + "request_id": 17, + "tpot_ms": 140.19912163029534, + "ttft_ms": 309.87121899492286 + }, + { + "e2e_ms": 18170.69585287006, + "request_id": 18, + "tpot_ms": 140.12856604535176, + "ttft_ms": 374.3679651103839 + }, + { + "e2e_ms": 18124.102264330366, + "request_id": 19, + "tpot_ms": 140.05569589908265, + "ttft_ms": 337.0288851468697 + }, + { + "e2e_ms": 18077.42933293013, + "request_id": 20, + "tpot_ms": 139.98073600581233, + "ttft_ms": 299.87586019196044 + }, + { + "e2e_ms": 18132.96551975776, + "request_id": 21, + "tpot_ms": 139.8495015438229, + "ttft_ms": 372.0788236922541 + }, + { + "e2e_ms": 18086.369066442116, + "request_id": 22, + "tpot_ms": 139.76199603449132, + "ttft_ms": 336.5955700617178 + }, + { + "e2e_ms": 18039.682893849018, + "request_id": 23, + "tpot_ms": 139.67091992877667, + "ttft_ms": 301.4760628943822 + }, + { + "e2e_ms": 18095.21908067664, + "request_id": 24, + "tpot_ms": 139.54200325642663, + "ttft_ms": 373.38466711045726 + }, + { + "e2e_ms": 18048.532908083544, + "request_id": 25, + "tpot_ms": 139.5517742330953, + "ttft_ms": 325.4575804804407 + }, + { + "e2e_ms": 18001.846735490446, + "request_id": 26, + "tpot_ms": 139.5610841122856, + "ttft_ms": 277.589053230173 + }, + { + "e2e_ms": 18057.290338264673, + "request_id": 27, + "tpot_ms": 139.56778701592805, + "ttft_ms": 332.18138724181046 + }, + { + "e2e_ms": 18010.60416567157, + "request_id": 28, + "tpot_ms": 139.57154350494935, + "ttft_ms": 285.01814054300564 + }, + { + "e2e_ms": 18066.06100963866, + "request_id": 29, + "tpot_ms": 139.5730606683175, + "ttft_ms": 340.2823047623365 + }, + { + "e2e_ms": 18019.37483704556, + "request_id": 30, + "tpot_ms": 139.57447357032456, + "ttft_ms": 293.41669361433986 + }, + { + "e2e_ms": 18074.831681012645, + "request_id": 31, + "tpot_ms": 139.57528428268915, + "ttft_ms": 348.7705771111216 + }, + { + "e2e_ms": 18028.145508419548, + "request_id": 32, + "tpot_ms": 139.57669718469626, + "ttft_ms": 301.904965963125 + }, + { + "e2e_ms": 18083.678830471228, + "request_id": 33, + "tpot_ms": 139.5781100867033, + "ttft_ms": 357.25884945990674 + }, + { + "e2e_ms": 18036.99265787813, + "request_id": 34, + "tpot_ms": 139.57881653770684, + "ttft_ms": 310.48295758936104 + }, + { + "e2e_ms": 18092.528844705754, + "request_id": 35, + "tpot_ms": 139.57954554600127, + "ttft_ms": 365.9265603635937 + }, + { + "e2e_ms": 17876.05367572695, + "request_id": 36, + "tpot_ms": 138.24262431461773, + "ttft_ms": 319.240387770499 + }, + { + "e2e_ms": 17659.21004164318, + "request_id": 37, + "tpot_ms": 136.90209533219152, + "ttft_ms": 272.64393445485524 + }, + { + "e2e_ms": 17543.75228976072, + "request_id": 38, + "tpot_ms": 135.55570892326125, + "ttft_ms": 328.1772565065388 + }, + { + "e2e_ms": 17326.367174433202, + "request_id": 39, + "tpot_ms": 134.21089375170362, + "ttft_ms": 281.5836679668422 + }, + { + "e2e_ms": 17233.832476815856, + "request_id": 40, + "tpot_ms": 133.04427482475538, + "ttft_ms": 337.20957407192384 + }, + { + "e2e_ms": 17034.265611203216, + "request_id": 41, + "tpot_ms": 131.83976083205505, + "ttft_ms": 290.6159855322272 + }, + { + "e2e_ms": 16940.502121615402, + "request_id": 42, + "tpot_ms": 130.66419538607474, + "ttft_ms": 346.14930758391085 + }, + { + "e2e_ms": 16741.97064984836, + "request_id": 43, + "tpot_ms": 129.46783410082003, + "ttft_ms": 299.5557190442142 + }, + { + "e2e_ms": 16645.559306371404, + "request_id": 44, + "tpot_ms": 128.27139685432724, + "ttft_ms": 355.0919058718449 + }, + { + "e2e_ms": 16443.232533552808, + "request_id": 45, + "tpot_ms": 127.04515130882409, + "ttft_ms": 308.4983173321483 + }, + { + "e2e_ms": 16342.299705615802, + "request_id": 46, + "tpot_ms": 125.81311182248837, + "ttft_ms": 364.034504159779 + }, + { + "e2e_ms": 16135.159559133626, + "request_id": 47, + "tpot_ms": 124.54896569695704, + "ttft_ms": 317.44091562008236 + }, + { + "e2e_ms": 16031.7032146663, + "request_id": 48, + "tpot_ms": 123.29695174036006, + "ttft_ms": 372.9903436405735 + }, + { + "e2e_ms": 15823.308920667878, + "request_id": 49, + "tpot_ms": 122.0236594458299, + "ttft_ms": 326.3041710474788 + }, + { + "e2e_ms": 15613.44809689394, + "request_id": 50, + "tpot_ms": 120.73809066445794, + "ttft_ms": 279.71058250778214 + }, + { + "e2e_ms": 15508.031714506984, + "request_id": 51, + "tpot_ms": 119.47077015706711, + "ttft_ms": 335.2439045594622 + }, + { + "e2e_ms": 15292.981047367362, + "request_id": 52, + "tpot_ms": 118.14506547559841, + "ttft_ms": 288.5577319663639 + }, + { + "e2e_ms": 15175.360910096466, + "request_id": 53, + "tpot_ms": 116.78162985277542, + "ttft_ms": 344.09391879398754 + }, + { + "e2e_ms": 14945.589828664517, + "request_id": 54, + "tpot_ms": 115.34001639735139, + "ttft_ms": 297.4077462008893 + }, + { + "e2e_ms": 14803.64975337431, + "request_id": 55, + "tpot_ms": 113.78581420786767, + "ttft_ms": 352.85134897511483 + }, + { + "e2e_ms": 14557.342221274233, + "request_id": 56, + "tpot_ms": 112.21399247946627, + "ttft_ms": 306.16517638201657 + }, + { + "e2e_ms": 14399.594833150197, + "request_id": 57, + "tpot_ms": 110.5352189984338, + "ttft_ms": 361.6220203491025 + }, + { + "e2e_ms": 14144.625862747695, + "request_id": 58, + "tpot_ms": 108.89519696843851, + "ttft_ms": 314.93584775600425 + }, + { + "e2e_ms": 13973.496543699825, + "request_id": 59, + "tpot_ms": 107.1110539525727, + "ttft_ms": 370.3926917230902 + }, + { + "e2e_ms": 13708.4999588278, + "request_id": 60, + "tpot_ms": 105.3920743283292, + "ttft_ms": 323.70651912999193 + }, + { + "e2e_ms": 13445.970803422128, + "request_id": 61, + "tpot_ms": 103.6925232825609, + "ttft_ms": 277.02034653689367 + }, + { + "e2e_ms": 13273.580023880399, + "request_id": 62, + "tpot_ms": 101.89784531725846, + "ttft_ms": 332.5536685885737 + }, + { + "e2e_ms": 13000.702540117643, + "request_id": 63, + "tpot_ms": 100.1168113710407, + "ttft_ms": 285.86749599547545 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 41.7739622592926, + "offered_request_rate": 2.4, + "request_metrics_sha256": "3b42135fbabbd54647284962ccc9fdf0b900fbc0b6ed0bdecf03477d5c409d18", + "request_rate_per_gpu": 0.6, + "requests": [ + { + "e2e_ms": 21942.173036302996, + "request_id": 0, + "tpot_ms": 170.66609896770518, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 22008.341136862466, + "request_id": 1, + "tpot_ms": 170.96613901739173, + "ttft_ms": 295.64148165371654 + }, + { + "e2e_ms": 22074.54102422534, + "request_id": 2, + "tpot_ms": 171.22660942224667, + "ttft_ms": 328.7616276000109 + }, + { + "e2e_ms": 22036.227907899276, + "request_id": 3, + "tpot_ms": 171.37344968894675, + "ttft_ms": 271.7997974030406 + }, + { + "e2e_ms": 22102.42779526115, + "request_id": 4, + "tpot_ms": 171.64131409153669, + "ttft_ms": 303.98090563598765 + }, + { + "e2e_ms": 22168.65946942643, + "request_id": 5, + "tpot_ms": 171.89659303259273, + "ttft_ms": 337.79215428715537 + }, + { + "e2e_ms": 22130.37813990278, + "request_id": 6, + "tpot_ms": 172.0281077294899, + "ttft_ms": 282.80845825756097 + }, + { + "e2e_ms": 22365.99994575454, + "request_id": 7, + "tpot_ms": 173.57085404199856, + "ttft_ms": 322.5014824207202 + }, + { + "e2e_ms": 22158.3592864644, + "request_id": 8, + "tpot_ms": 172.3496956452424, + "ttft_ms": 269.9479395186155 + }, + { + "e2e_ms": 22393.981092316164, + "request_id": 9, + "tpot_ms": 173.8419366523574, + "ttft_ms": 316.0551374667744 + }, + { + "e2e_ms": 22460.243568400445, + "request_id": 10, + "tpot_ms": 173.96590922572528, + "ttft_ms": 366.57309673333316 + }, + { + "e2e_ms": 22252.602909110316, + "request_id": 11, + "tpot_ms": 172.6883751510306, + "ttft_ms": 321.1792649294285 + }, + { + "e2e_ms": 22044.894176229613, + "request_id": 12, + "tpot_ms": 171.40112664047544, + "ttft_ms": 276.9510928892318 + }, + { + "e2e_ms": 21941.653543442477, + "request_id": 13, + "tpot_ms": 170.1065182287643, + "ttft_ms": 338.1257283894126 + }, + { + "e2e_ms": 21734.138079618475, + "request_id": 14, + "tpot_ms": 168.80971544527205, + "ttft_ms": 295.30421806892537 + }, + { + "e2e_ms": 21631.119179231846, + "request_id": 15, + "tpot_ms": 167.51267286797687, + "ttft_ms": 357.0097249987798 + }, + { + "e2e_ms": 21420.354658102242, + "request_id": 16, + "tpot_ms": 166.18662353135022, + "ttft_ms": 314.65346962076256 + }, + { + "e2e_ms": 21209.648295563282, + "request_id": 17, + "tpot_ms": 164.85776701341018, + "ttft_ms": 272.7118848601879 + }, + { + "e2e_ms": 21101.69867612793, + "request_id": 18, + "tpot_ms": 163.50974377121332, + "ttft_ms": 335.9612171838391 + }, + { + "e2e_ms": 20890.778369055995, + "request_id": 19, + "tpot_ms": 162.16628083276893, + "ttft_ms": 295.66070329433813 + }, + { + "e2e_ms": 20782.558056179907, + "request_id": 20, + "tpot_ms": 160.7985342727971, + "ttft_ms": 361.14420353467835 + }, + { + "e2e_ms": 20571.512469390585, + "request_id": 21, + "tpot_ms": 159.44598375979533, + "ttft_ms": 321.8725318965792 + }, + { + "e2e_ms": 20360.405278761264, + "request_id": 22, + "tpot_ms": 158.09565605925098, + "ttft_ms": 282.2569592363866 + }, + { + "e2e_ms": 20251.586130359967, + "request_id": 23, + "tpot_ms": 156.71227679254048, + "ttft_ms": 349.12697770732314 + }, + { + "e2e_ms": 20042.10448060375, + "request_id": 24, + "tpot_ms": 155.37142770924103, + "ttft_ms": 309.93316153013774 + }, + { + "e2e_ms": 19934.008109113514, + "request_id": 25, + "tpot_ms": 154.05960281316982, + "ttft_ms": 368.43855184094565 + }, + { + "e2e_ms": 19722.984661067087, + "request_id": 26, + "tpot_ms": 152.73931080974432, + "ttft_ms": 325.0921882295579 + }, + { + "e2e_ms": 19511.768043076878, + "request_id": 27, + "tpot_ms": 151.41621155599998, + "ttft_ms": 281.9091754648788 + }, + { + "e2e_ms": 19403.203575383635, + "request_id": 28, + "tpot_ms": 150.09414530000853, + "ttft_ms": 341.2471222825513 + }, + { + "e2e_ms": 19191.714653347688, + "request_id": 29, + "tpot_ms": 148.76487274343572, + "ttft_ms": 298.57581493135046 + }, + { + "e2e_ms": 19082.708845359608, + "request_id": 30, + "tpot_ms": 147.43434256455518, + "ttft_ms": 358.5473396611025 + }, + { + "e2e_ms": 18871.078709760895, + "request_id": 31, + "tpot_ms": 146.10277278695904, + "ttft_ms": 316.0265658170971 + }, + { + "e2e_ms": 18659.370447496967, + "request_id": 32, + "tpot_ms": 144.7695321688903, + "ttft_ms": 273.6398620479008 + }, + { + "e2e_ms": 18549.637648096734, + "request_id": 33, + "tpot_ms": 143.43132538741386, + "ttft_ms": 333.85932389517234 + }, + { + "e2e_ms": 18337.808235788358, + "request_id": 34, + "tpot_ms": 142.0958670025312, + "ttft_ms": 291.63312646689656 + }, + { + "e2e_ms": 18227.80864738452, + "request_id": 35, + "tpot_ms": 140.75352098043697, + "ttft_ms": 352.1114828690255 + }, + { + "e2e_ms": 18015.40726100407, + "request_id": 36, + "tpot_ms": 139.41149279948888, + "ttft_ms": 310.14767546898446 + }, + { + "e2e_ms": 17904.857868774245, + "request_id": 37, + "tpot_ms": 138.06211837292886, + "ttft_ms": 370.9688354122758 + }, + { + "e2e_ms": 17692.008101355255, + "request_id": 38, + "tpot_ms": 136.71510258553306, + "ttft_ms": 329.1900729925583 + }, + { + "e2e_ms": 17479.148877024498, + "request_id": 39, + "tpot_ms": 135.36749298254742, + "ttft_ms": 287.47726824097344 + }, + { + "e2e_ms": 17381.961391787096, + "request_id": 40, + "tpot_ms": 134.1089349239123, + "ttft_ms": 350.1266564502323 + }, + { + "e2e_ms": 17178.01129231477, + "request_id": 41, + "tpot_ms": 132.8327622110564, + "ttft_ms": 308.2504915106092 + }, + { + "e2e_ms": 17080.086531895085, + "request_id": 42, + "tpot_ms": 131.5805170059729, + "ttft_ms": 369.36087213652866 + }, + { + "e2e_ms": 16876.654129344566, + "request_id": 43, + "tpot_ms": 130.30640250229436, + "ttft_ms": 327.74101155318294 + }, + { + "e2e_ms": 16671.422925287778, + "request_id": 44, + "tpot_ms": 129.0178430005707, + "ttft_ms": 286.15686421529676 + }, + { + "e2e_ms": 16571.761179006833, + "request_id": 45, + "tpot_ms": 127.74972161297816, + "ttft_ms": 347.5465341586066 + }, + { + "e2e_ms": 16364.721381165033, + "request_id": 46, + "tpot_ms": 126.44572566423672, + "ttft_ms": 306.1142218069683 + }, + { + "e2e_ms": 16261.449604637399, + "request_id": 47, + "tpot_ms": 125.14700916940275, + "ttft_ms": 367.77944012325037 + }, + { + "e2e_ms": 16053.400400183471, + "request_id": 48, + "tpot_ms": 123.8340136000885, + "ttft_ms": 326.48067297223093 + }, + { + "e2e_ms": 15845.03658651726, + "request_id": 49, + "tpot_ms": 122.50141376236263, + "ttft_ms": 287.35703869720555 + }, + { + "e2e_ms": 15739.724302020804, + "request_id": 50, + "tpot_ms": 121.15600683324593, + "ttft_ms": 352.9114341985711 + }, + { + "e2e_ms": 15530.891082399023, + "request_id": 51, + "tpot_ms": 119.81946138926746, + "ttft_ms": 313.8194859620569 + }, + { + "e2e_ms": 15319.199082162215, + "request_id": 52, + "tpot_ms": 118.45427332697808, + "ttft_ms": 275.5063696359983 + }, + { + "e2e_ms": 15206.960333133673, + "request_id": 53, + "tpot_ms": 117.04949498375757, + "ttft_ms": 341.6744701964625 + }, + { + "e2e_ms": 14987.9081257507, + "request_id": 54, + "tpot_ms": 115.62635253449054, + "ttft_ms": 303.36135387040383 + }, + { + "e2e_ms": 14861.077413909577, + "request_id": 55, + "tpot_ms": 114.10642655651417, + "ttft_ms": 369.5612412322795 + }, + { + "e2e_ms": 14633.756981193535, + "request_id": 56, + "tpot_ms": 112.61817997075843, + "ttft_ms": 331.2481249072157 + }, + { + "e2e_ms": 14403.991359276923, + "request_id": 57, + "tpot_ms": 111.11042963695559, + "ttft_ms": 292.9667953835633 + }, + { + "e2e_ms": 14264.847964808048, + "request_id": 58, + "tpot_ms": 109.49330311228506, + "ttft_ms": 359.1984695478452 + }, + { + "e2e_ms": 14029.729570522895, + "request_id": 59, + "tpot_ms": 107.94340496454889, + "ttft_ms": 320.91714002518756 + }, + { + "e2e_ms": 13793.064611419886, + "request_id": 60, + "tpot_ms": 106.38108660628622, + "ttft_ms": 282.66661242153646 + }, + { + "e2e_ms": 13648.251078198633, + "request_id": 61, + "tpot_ms": 104.71931331978594, + "ttft_ms": 348.89828658581837 + }, + { + "e2e_ms": 13412.62809971594, + "request_id": 62, + "tpot_ms": 103.16519953332896, + "ttft_ms": 310.647758983162 + }, + { + "e2e_ms": 13260.053320226874, + "request_id": 63, + "tpot_ms": 101.44207153668842, + "ttft_ms": 376.9102350674451 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 41.42983865737915, + "offered_request_rate": 3.2, + "request_metrics_sha256": "bfba1ae71cfb6d0595614b14bc49ac47591bfad6792be0a3d98d2c98b16012e8", + "request_rate_per_gpu": 0.8, + "requests": [ + { + "e2e_ms": 24003.81066522886, + "request_id": 0, + "tpot_ms": 186.89946612460176, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 23901.058156177503, + "request_id": 1, + "tpot_ms": 185.74139588389912, + "ttft_ms": 311.9008789223171 + }, + { + "e2e_ms": 23798.190109844723, + "request_id": 2, + "tpot_ms": 184.56225305811301, + "ttft_ms": 358.7839714643697 + }, + { + "e2e_ms": 23590.44639829378, + "request_id": 3, + "tpot_ms": 183.2673714648551, + "ttft_ms": 315.49022225718517 + }, + { + "e2e_ms": 23382.70131446869, + "request_id": 4, + "tpot_ms": 181.9732705315891, + "ttft_ms": 272.09595695687216 + }, + { + "e2e_ms": 23279.64954297851, + "request_id": 5, + "tpot_ms": 180.78914255292864, + "ttft_ms": 319.4284387565705 + }, + { + "e2e_ms": 23071.973865900134, + "request_id": 6, + "tpot_ms": 179.49398625037213, + "ttft_ms": 276.2376121028707 + }, + { + "e2e_ms": 22969.022726489202, + "request_id": 7, + "tpot_ms": 178.2861180199918, + "ttft_ms": 326.6857379502426 + }, + { + "e2e_ms": 22761.29942646063, + "request_id": 8, + "tpot_ms": 176.98924979786668, + "ttft_ms": 283.66470213156344 + }, + { + "e2e_ms": 22658.264587165988, + "request_id": 9, + "tpot_ms": 175.7551174352214, + "ttft_ms": 337.36467289287384 + }, + { + "e2e_ms": 22450.304224673095, + "request_id": 10, + "tpot_ms": 174.4550575162989, + "ttft_ms": 294.51192010313275 + }, + { + "e2e_ms": 22346.821408551063, + "request_id": 11, + "tpot_ms": 173.1774539652428, + "ttft_ms": 353.28475496522805 + }, + { + "e2e_ms": 22138.79812545183, + "request_id": 12, + "tpot_ms": 171.87427427697472, + "ttft_ms": 310.7652922760442 + }, + { + "e2e_ms": 22035.248814705075, + "request_id": 13, + "tpot_ms": 170.57508935568035, + "ttft_ms": 372.21246653366967 + }, + { + "e2e_ms": 21827.324416126074, + "request_id": 14, + "tpot_ms": 169.27188182433355, + "ttft_ms": 329.79542443571484 + }, + { + "e2e_ms": 21619.373671552865, + "request_id": 15, + "tpot_ms": 167.96821838814947, + "ttft_ms": 287.40993625788303 + }, + { + "e2e_ms": 21512.77581709026, + "request_id": 16, + "tpot_ms": 166.64216905152279, + "ttft_ms": 349.2203475468658 + }, + { + "e2e_ms": 21303.255969153783, + "request_id": 17, + "tpot_ms": 165.32482604305434, + "ttft_ms": 307.003061685883 + }, + { + "e2e_ms": 21196.567333974876, + "request_id": 18, + "tpot_ms": 163.99164288866731, + "ttft_ms": 369.62868711412875 + }, + { + "e2e_ms": 20986.94051377241, + "request_id": 19, + "tpot_ms": 162.67002407488405, + "ttft_ms": 327.84745626213316 + }, + { + "e2e_ms": 20777.255965035714, + "request_id": 20, + "tpot_ms": 161.34765321373814, + "ttft_ms": 286.10400689097213 + }, + { + "e2e_ms": 20670.377044913395, + "request_id": 21, + "tpot_ms": 159.9951027007364, + "ttft_ms": 350.99900191987297 + }, + { + "e2e_ms": 20460.656782932234, + "request_id": 22, + "tpot_ms": 158.67196467076286, + "ttft_ms": 309.3172697453532 + }, + { + "e2e_ms": 20353.38846177537, + "request_id": 23, + "tpot_ms": 157.31793948495124, + "ttft_ms": 374.01014718656 + }, + { + "e2e_ms": 20144.48097023076, + "request_id": 24, + "tpot_ms": 155.99981541047092, + "ttft_ms": 332.5044131009545 + }, + { + "e2e_ms": 19934.83773528968, + "request_id": 25, + "tpot_ms": 154.6729368346084, + "ttft_ms": 291.37475729441144 + }, + { + "e2e_ms": 19827.980953909257, + "request_id": 26, + "tpot_ms": 153.35264483118294, + "ttft_ms": 352.1950603490236 + }, + { + "e2e_ms": 19618.20597824765, + "request_id": 27, + "tpot_ms": 152.02412197401114, + "ttft_ms": 311.1424875482367 + }, + { + "e2e_ms": 19511.107444230154, + "request_id": 28, + "tpot_ms": 150.6975599886974, + "ttft_ms": 372.5173256655836 + }, + { + "e2e_ms": 19301.196316545185, + "request_id": 29, + "tpot_ms": 149.36537894653213, + "ttft_ms": 331.7931903356044 + }, + { + "e2e_ms": 19091.249936104825, + "request_id": 30, + "tpot_ms": 148.03267778881258, + "ttft_ms": 291.0998569256247 + }, + { + "e2e_ms": 18983.78646717311, + "request_id": 31, + "tpot_ms": 146.7011080112165, + "ttft_ms": 352.7457497486193 + }, + { + "e2e_ms": 18773.765669374145, + "request_id": 32, + "tpot_ms": 145.36784483585674, + "ttft_ms": 312.0493752203348 + }, + { + "e2e_ms": 18665.888247551986, + "request_id": 33, + "tpot_ms": 144.03124523287858, + "ttft_ms": 373.9201029764097 + }, + { + "e2e_ms": 18455.8068747313, + "request_id": 34, + "tpot_ms": 142.6968275070543, + "ttft_ms": 333.30978133540157 + }, + { + "e2e_ms": 18245.529234152025, + "request_id": 35, + "tpot_ms": 141.3602386069844, + "ttft_ms": 292.7789310650048 + }, + { + "e2e_ms": 18137.294514438578, + "request_id": 36, + "tpot_ms": 140.01821042603632, + "ttft_ms": 354.98179033196385 + }, + { + "e2e_ms": 17926.83360591763, + "request_id": 37, + "tpot_ms": 138.67967407469433, + "ttft_ms": 314.5149984314486 + }, + { + "e2e_ms": 17716.242055541137, + "request_id": 38, + "tpot_ms": 137.3398665125213, + "ttft_ms": 274.0790084509328 + }, + { + "e2e_ms": 17607.549497877386, + "request_id": 39, + "tpot_ms": 135.99225690953574, + "ttft_ms": 336.53287036634794 + }, + { + "e2e_ms": 17401.54255835392, + "request_id": 40, + "tpot_ms": 134.6880568492608, + "ttft_ms": 296.1593384978016 + }, + { + "e2e_ms": 17301.7591255476, + "request_id": 41, + "tpot_ms": 133.41188413640484, + "ttft_ms": 358.4498402241838 + }, + { + "e2e_ms": 17095.506427630044, + "request_id": 42, + "tpot_ms": 132.10522228512968, + "ttft_ms": 318.1431974185731 + }, + { + "e2e_ms": 16889.623559688287, + "request_id": 43, + "tpot_ms": 130.8012299461049, + "ttft_ms": 277.86735653296194 + }, + { + "e2e_ms": 16788.559022297493, + "request_id": 44, + "tpot_ms": 129.5126704443812, + "ttft_ms": 340.449875861081 + }, + { + "e2e_ms": 16581.727329092846, + "request_id": 45, + "tpot_ms": 128.200721970688, + "ttft_ms": 300.23563881547057 + }, + { + "e2e_ms": 16478.854197918037, + "request_id": 46, + "tpot_ms": 126.8967260219465, + "ttft_ms": 362.9699931308306 + }, + { + "e2e_ms": 16270.819161297495, + "request_id": 47, + "tpot_ms": 125.57481733363996, + "ttft_ms": 322.8173599252209 + }, + { + "e2e_ms": 16062.627892404027, + "request_id": 48, + "tpot_ms": 124.25122687214136, + "ttft_ms": 282.72207964207576 + }, + { + "e2e_ms": 15958.430745404816, + "request_id": 49, + "tpot_ms": 122.91862703441548, + "ttft_ms": 347.7651120340504 + }, + { + "e2e_ms": 15749.715539461335, + "request_id": 50, + "tpot_ms": 121.58574387256766, + "ttft_ms": 308.3260676452415 + }, + { + "e2e_ms": 15645.048986506545, + "request_id": 51, + "tpot_ms": 120.24919842858912, + "ttft_ms": 373.40078607572735 + }, + { + "e2e_ms": 15435.036319721643, + "request_id": 52, + "tpot_ms": 118.8997102377205, + "ttft_ms": 334.77311953113895 + }, + { + "e2e_ms": 15224.012292267464, + "request_id": 53, + "tpot_ms": 117.54201604221191, + "ttft_ms": 296.1762549065519 + }, + { + "e2e_ms": 15109.12675155149, + "request_id": 54, + "tpot_ms": 116.11789378026353, + "ttft_ms": 362.1542414580219 + }, + { + "e2e_ms": 14893.238736493451, + "request_id": 55, + "tpot_ms": 114.72141540015762, + "ttft_ms": 323.6189806734337 + }, + { + "e2e_ms": 14675.411853468433, + "request_id": 56, + "tpot_ms": 113.30942780834319, + "ttft_ms": 285.1145218088469 + }, + { + "e2e_ms": 14549.812898218817, + "request_id": 57, + "tpot_ms": 111.79839094375423, + "ttft_ms": 351.4172483620292 + }, + { + "e2e_ms": 14329.820655618192, + "request_id": 58, + "tpot_ms": 110.36886820693505, + "ttft_ms": 312.97439333744137 + }, + { + "e2e_ms": 14108.094791808619, + "request_id": 59, + "tpot_ms": 108.92545237461232, + "ttft_ms": 274.56234023285475 + }, + { + "e2e_ms": 13975.596499372607, + "request_id": 60, + "tpot_ms": 107.35951607044248, + "ttft_ms": 340.93795842641225 + }, + { + "e2e_ms": 13753.714210521188, + "request_id": 61, + "tpot_ms": 105.91438347527057, + "ttft_ms": 302.58750916182464 + }, + { + "e2e_ms": 13622.257898704496, + "request_id": 62, + "tpot_ms": 104.35568588071192, + "ttft_ms": 369.08579185408286 + }, + { + "e2e_ms": 13397.788527763809, + "request_id": 63, + "tpot_ms": 102.88969749082138, + "ttft_ms": 330.7969464294942 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 34.255025148391724, + "offered_request_rate": 0.1, + "request_metrics_sha256": "b45cd3a473a98bee1127b2315d0c0c59ded580275f6e6d10a3e5a52d85d24de4", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 11656.879780530655, + "request_id": 0, + "tpot_ms": 89.67953789863164, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 11729.945143691864, + "request_id": 1, + "tpot_ms": 89.99959080602144, + "ttft_ms": 299.9971113271407 + }, + { + "e2e_ms": 11712.730146443742, + "request_id": 2, + "tpot_ms": 90.00056214193414, + "ttft_ms": 282.65875441810806 + }, + { + "e2e_ms": 11785.795509604945, + "request_id": 3, + "tpot_ms": 89.99959080602139, + "ttft_ms": 355.8474772402285 + }, + { + "e2e_ms": 11770.95360336775, + "request_id": 4, + "tpot_ms": 90.01924789792568, + "ttft_ms": 338.50912033118874 + }, + { + "e2e_ms": 11756.111697130556, + "request_id": 5, + "tpot_ms": 90.0192478979257, + "ttft_ms": 323.66721409399446 + }, + { + "e2e_ms": 11741.269790893028, + "request_id": 6, + "tpot_ms": 90.01924789792305, + "ttft_ms": 308.8253078568002 + }, + { + "e2e_ms": 11726.427884655508, + "request_id": 7, + "tpot_ms": 90.01924789792311, + "ttft_ms": 293.98340161927194 + }, + { + "e2e_ms": 11709.212887407048, + "request_id": 8, + "tpot_ms": 90.00056214193147, + "ttft_ms": 279.1414953817508 + }, + { + "e2e_ms": 11782.278250567928, + "request_id": 9, + "tpot_ms": 89.99959080601886, + "ttft_ms": 352.3302182035337 + }, + { + "e2e_ms": 11767.436344330406, + "request_id": 10, + "tpot_ms": 90.01924789792311, + "ttft_ms": 334.9918612941707 + }, + { + "e2e_ms": 11752.594438092885, + "request_id": 11, + "tpot_ms": 90.01924789792311, + "ttft_ms": 320.14995505664956 + }, + { + "e2e_ms": 11737.752531855705, + "request_id": 12, + "tpot_ms": 90.01924789792581, + "ttft_ms": 305.30804881912843 + }, + { + "e2e_ms": 11722.91062561817, + "request_id": 13, + "tpot_ms": 90.019247897923, + "ttft_ms": 290.46614258194836 + }, + { + "e2e_ms": 11705.69562836971, + "request_id": 14, + "tpot_ms": 90.00056214193148, + "ttft_ms": 275.624236344413 + }, + { + "e2e_ms": 11778.76099153056, + "request_id": 15, + "tpot_ms": 89.99959080601862, + "ttft_ms": 348.8129591661959 + }, + { + "e2e_ms": 11763.919085293026, + "request_id": 16, + "tpot_ms": 90.019247897923, + "ttft_ms": 331.4746022568045 + }, + { + "e2e_ms": 11749.07717905549, + "request_id": 17, + "tpot_ms": 90.019247897923, + "ttft_ms": 316.63269601926913 + }, + { + "e2e_ms": 11734.235272817954, + "request_id": 18, + "tpot_ms": 90.019247897923, + "ttft_ms": 301.7907897817338 + }, + { + "e2e_ms": 11719.39336658042, + "request_id": 19, + "tpot_ms": 90.019247897923, + "ttft_ms": 286.94888354419845 + }, + { + "e2e_ms": 11702.178369331961, + "request_id": 20, + "tpot_ms": 90.00056214193148, + "ttft_ms": 272.1069773066631 + }, + { + "e2e_ms": 11775.24373249281, + "request_id": 21, + "tpot_ms": 89.99959080601862, + "ttft_ms": 345.295700128446 + }, + { + "e2e_ms": 11760.401826255276, + "request_id": 22, + "tpot_ms": 90.01924789792301, + "ttft_ms": 327.95734321905456 + }, + { + "e2e_ms": 11745.55992001774, + "request_id": 23, + "tpot_ms": 90.019247897923, + "ttft_ms": 313.1154369815192 + }, + { + "e2e_ms": 11730.718013780204, + "request_id": 24, + "tpot_ms": 90.019247897923, + "ttft_ms": 298.2735307439839 + }, + { + "e2e_ms": 11713.50301653041, + "request_id": 25, + "tpot_ms": 90.00056214192095, + "ttft_ms": 283.43162450644854 + }, + { + "e2e_ms": 11786.568379689925, + "request_id": 26, + "tpot_ms": 89.9995908060081, + "ttft_ms": 356.6203473268956 + }, + { + "e2e_ms": 11771.726473451054, + "request_id": 27, + "tpot_ms": 90.01924789791249, + "ttft_ms": 339.28199041616836 + }, + { + "e2e_ms": 11756.884567212182, + "request_id": 28, + "tpot_ms": 90.01924789791249, + "ttft_ms": 324.4400841772972 + }, + { + "e2e_ms": 11742.042660973311, + "request_id": 29, + "tpot_ms": 90.01924789791249, + "ttft_ms": 309.59817793842603 + }, + { + "e2e_ms": 11727.20075473444, + "request_id": 30, + "tpot_ms": 90.01924789791249, + "ttft_ms": 294.7562716995549 + }, + { + "e2e_ms": 11709.985757484617, + "request_id": 31, + "tpot_ms": 90.00056214192074, + "ttft_ms": 279.9143654606837 + }, + { + "e2e_ms": 11783.051120644132, + "request_id": 32, + "tpot_ms": 89.9995908060081, + "ttft_ms": 353.1030882811024 + }, + { + "e2e_ms": 11768.20921440526, + "request_id": 33, + "tpot_ms": 90.01924789791249, + "ttft_ms": 335.7647313703751 + }, + { + "e2e_ms": 11753.36730816639, + "request_id": 34, + "tpot_ms": 90.01924789791249, + "ttft_ms": 320.92282513150394 + }, + { + "e2e_ms": 11738.525401927518, + "request_id": 35, + "tpot_ms": 90.01924789791249, + "ttft_ms": 306.0809188926328 + }, + { + "e2e_ms": 11723.683495688647, + "request_id": 36, + "tpot_ms": 90.01924789791249, + "ttft_ms": 291.2390126537616 + }, + { + "e2e_ms": 11706.468498438824, + "request_id": 37, + "tpot_ms": 90.00056214192074, + "ttft_ms": 276.39710641489046 + }, + { + "e2e_ms": 11779.533861598338, + "request_id": 38, + "tpot_ms": 89.9995908060081, + "ttft_ms": 349.5858292353091 + }, + { + "e2e_ms": 11764.691955359467, + "request_id": 39, + "tpot_ms": 90.01924789791249, + "ttft_ms": 332.24747232458185 + }, + { + "e2e_ms": 11749.850049120596, + "request_id": 40, + "tpot_ms": 90.01924789791249, + "ttft_ms": 317.4055660857107 + }, + { + "e2e_ms": 11735.008142881725, + "request_id": 41, + "tpot_ms": 90.01924789791249, + "ttft_ms": 302.5636598468395 + }, + { + "e2e_ms": 11720.166236642854, + "request_id": 42, + "tpot_ms": 90.01924789791249, + "ttft_ms": 287.72175360796837 + }, + { + "e2e_ms": 11702.95123939303, + "request_id": 43, + "tpot_ms": 90.00056214192074, + "ttft_ms": 272.8798473690972 + }, + { + "e2e_ms": 11776.016602552545, + "request_id": 44, + "tpot_ms": 89.9995908060081, + "ttft_ms": 346.06857018951587 + }, + { + "e2e_ms": 11761.174696313674, + "request_id": 45, + "tpot_ms": 90.01924789791249, + "ttft_ms": 328.7302132787886 + }, + { + "e2e_ms": 11746.332790074803, + "request_id": 46, + "tpot_ms": 90.01924789791249, + "ttft_ms": 313.88830703991744 + }, + { + "e2e_ms": 11731.490883835932, + "request_id": 47, + "tpot_ms": 90.01924789791249, + "ttft_ms": 299.0464008010463 + }, + { + "e2e_ms": 11716.64897759706, + "request_id": 48, + "tpot_ms": 90.01924789791249, + "ttft_ms": 284.2044945621751 + }, + { + "e2e_ms": 11699.433980347238, + "request_id": 49, + "tpot_ms": 90.00056214192074, + "ttft_ms": 269.36258832330395 + }, + { + "e2e_ms": 11772.499343506752, + "request_id": 50, + "tpot_ms": 89.9995908060081, + "ttft_ms": 342.5513111437226 + }, + { + "e2e_ms": 11757.657437270042, + "request_id": 51, + "tpot_ms": 90.0192478979295, + "ttft_ms": 325.21295423299534 + }, + { + "e2e_ms": 11742.815531033444, + "request_id": 52, + "tpot_ms": 90.01924789792636, + "ttft_ms": 310.3710479967958 + }, + { + "e2e_ms": 11727.973624796847, + "request_id": 53, + "tpot_ms": 90.01924789792636, + "ttft_ms": 295.5291417601984 + }, + { + "e2e_ms": 11710.758627549352, + "request_id": 54, + "tpot_ms": 90.00056214193505, + "ttft_ms": 280.687235523601 + }, + { + "e2e_ms": 11783.823990711198, + "request_id": 55, + "tpot_ms": 89.99959080602287, + "ttft_ms": 353.8759583462934 + }, + { + "e2e_ms": 11768.982084474601, + "request_id": 56, + "tpot_ms": 90.01924789792636, + "ttft_ms": 336.5376014379535 + }, + { + "e2e_ms": 11754.140178238004, + "request_id": 57, + "tpot_ms": 90.01924789792636, + "ttft_ms": 321.6956952013561 + }, + { + "e2e_ms": 11739.298272001406, + "request_id": 58, + "tpot_ms": 90.01924789792636, + "ttft_ms": 306.8537889647587 + }, + { + "e2e_ms": 11724.456365764809, + "request_id": 59, + "tpot_ms": 90.01924789792636, + "ttft_ms": 292.01188272816125 + }, + { + "e2e_ms": 11707.241368517316, + "request_id": 60, + "tpot_ms": 90.00056214193505, + "ttft_ms": 277.16997649156383 + }, + { + "e2e_ms": 11780.30673167916, + "request_id": 61, + "tpot_ms": 89.99959080602287, + "ttft_ms": 350.3586993142562 + }, + { + "e2e_ms": 11765.464825442563, + "request_id": 62, + "tpot_ms": 90.01924789792635, + "ttft_ms": 333.0203424059164 + }, + { + "e2e_ms": 11530.943387303068, + "request_id": 63, + "tpot_ms": 88.28948780420275, + "ttft_ms": 318.17843616931896 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 26.24324607849121, + "offered_request_rate": 0.2, + "request_metrics_sha256": "7930ffb53e440b9eefa724912c777842a9c33cefdb46d9014e1c5003223221d2", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 11975.67900227787, + "request_id": 0, + "tpot_ms": 92.18976799112939, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12131.199338988026, + "request_id": 1, + "tpot_ms": 93.2890801728389, + "ttft_ms": 283.4861570374869 + }, + { + "e2e_ms": 12196.411822684866, + "request_id": 2, + "tpot_ms": 93.31228634935276, + "ttft_ms": 345.75145631706584 + }, + { + "e2e_ms": 12171.220586311463, + "request_id": 3, + "tpot_ms": 93.31547315047263, + "ttft_ms": 320.15549620144054 + }, + { + "e2e_ms": 12145.933482881133, + "request_id": 4, + "tpot_ms": 93.31640326087744, + "ttft_ms": 294.75026874969853 + }, + { + "e2e_ms": 12120.646379450847, + "request_id": 5, + "tpot_ms": 93.31564840216181, + "ttft_ms": 269.5590323762964 + }, + { + "e2e_ms": 12185.858863147729, + "request_id": 6, + "tpot_ms": 93.31471829175732, + "ttft_ms": 334.88964009454844 + }, + { + "e2e_ms": 12160.571759717443, + "request_id": 7, + "tpot_ms": 93.31471829175764, + "ttft_ms": 309.6025366642223 + }, + { + "e2e_ms": 12135.284656287155, + "request_id": 8, + "tpot_ms": 93.31564840216245, + "ttft_ms": 284.19730921252295 + }, + { + "e2e_ms": 12200.497139984038, + "request_id": 9, + "tpot_ms": 93.31471829175764, + "ttft_ms": 349.52791693081764 + }, + { + "e2e_ms": 12175.305903610677, + "request_id": 10, + "tpot_ms": 93.3154731504736, + "ttft_ms": 324.2408135005306 + }, + { + "e2e_ms": 12150.018800180305, + "request_id": 11, + "tpot_ms": 93.31640326087775, + "ttft_ms": 298.8355860488312 + }, + { + "e2e_ms": 12124.731696749934, + "request_id": 12, + "tpot_ms": 93.31564840216113, + "ttft_ms": 273.6443496754717 + }, + { + "e2e_ms": 12189.94418044673, + "request_id": 13, + "tpot_ms": 93.31471829175568, + "ttft_ms": 338.97495739375927 + }, + { + "e2e_ms": 12164.657077016358, + "request_id": 14, + "tpot_ms": 93.31471829175568, + "ttft_ms": 313.68785396338694 + }, + { + "e2e_ms": 12139.369973585985, + "request_id": 15, + "tpot_ms": 93.31564840216049, + "ttft_ms": 288.2826265116023 + }, + { + "e2e_ms": 12204.582457282782, + "request_id": 16, + "tpot_ms": 93.31471829175568, + "ttft_ms": 353.6132342298117 + }, + { + "e2e_ms": 12179.391220909338, + "request_id": 17, + "tpot_ms": 93.31547315047163, + "ttft_ms": 328.3261307994394 + }, + { + "e2e_ms": 12154.104117478966, + "request_id": 18, + "tpot_ms": 93.31640326087647, + "ttft_ms": 302.92090334765476 + }, + { + "e2e_ms": 12128.817014048593, + "request_id": 19, + "tpot_ms": 93.31564840216049, + "ttft_ms": 277.72966697421 + }, + { + "e2e_ms": 12194.02949774539, + "request_id": 20, + "tpot_ms": 93.31471829175568, + "ttft_ms": 343.0602746924194 + }, + { + "e2e_ms": 12168.742394315017, + "request_id": 21, + "tpot_ms": 93.31471829175567, + "ttft_ms": 317.7731712620471 + }, + { + "e2e_ms": 12143.455290884645, + "request_id": 22, + "tpot_ms": 93.31564840216049, + "ttft_ms": 292.36794381026243 + }, + { + "e2e_ms": 12208.667774581443, + "request_id": 23, + "tpot_ms": 93.31471829175568, + "ttft_ms": 357.69855152847185 + }, + { + "e2e_ms": 12183.476538207999, + "request_id": 24, + "tpot_ms": 93.31547315047165, + "ttft_ms": 332.4114480980995 + }, + { + "e2e_ms": 12158.189434777625, + "request_id": 25, + "tpot_ms": 93.31640326087646, + "ttft_ms": 307.0062206463149 + }, + { + "e2e_ms": 12132.902331347253, + "request_id": 26, + "tpot_ms": 93.31564840216049, + "ttft_ms": 281.8149842728701 + }, + { + "e2e_ms": 12198.11481504405, + "request_id": 27, + "tpot_ms": 93.31471829175567, + "ttft_ms": 347.1455919910795 + }, + { + "e2e_ms": 12172.923578670605, + "request_id": 28, + "tpot_ms": 93.31547315047163, + "ttft_ms": 321.8584885607072 + }, + { + "e2e_ms": 12147.636475240233, + "request_id": 29, + "tpot_ms": 93.31640326087646, + "ttft_ms": 296.45326110892256 + }, + { + "e2e_ms": 12122.349371809862, + "request_id": 30, + "tpot_ms": 93.3156484021605, + "ttft_ms": 271.2620247354778 + }, + { + "e2e_ms": 12187.561855506658, + "request_id": 31, + "tpot_ms": 93.31471829175568, + "ttft_ms": 336.5926324536872 + }, + { + "e2e_ms": 12162.274752076286, + "request_id": 32, + "tpot_ms": 93.31471829175568, + "ttft_ms": 311.3055290233149 + }, + { + "e2e_ms": 12136.987648645914, + "request_id": 33, + "tpot_ms": 93.3156484021605, + "ttft_ms": 285.90030157153024 + }, + { + "e2e_ms": 12202.20013234271, + "request_id": 34, + "tpot_ms": 93.31471829175568, + "ttft_ms": 351.23090928973966 + }, + { + "e2e_ms": 12177.008895969266, + "request_id": 35, + "tpot_ms": 93.31547315047163, + "ttft_ms": 325.94380585936733 + }, + { + "e2e_ms": 12151.721792538894, + "request_id": 36, + "tpot_ms": 93.31640326087647, + "ttft_ms": 300.5385784075827 + }, + { + "e2e_ms": 12126.43468910852, + "request_id": 37, + "tpot_ms": 93.31564840216049, + "ttft_ms": 275.3473420341379 + }, + { + "e2e_ms": 12191.647172805318, + "request_id": 38, + "tpot_ms": 93.31471829175568, + "ttft_ms": 340.67794975234733 + }, + { + "e2e_ms": 12166.360069374947, + "request_id": 39, + "tpot_ms": 93.3147182917557, + "ttft_ms": 315.390846321975 + }, + { + "e2e_ms": 12141.072965944573, + "request_id": 40, + "tpot_ms": 93.31564840216049, + "ttft_ms": 289.98561887019036 + }, + { + "e2e_ms": 12206.285449641371, + "request_id": 41, + "tpot_ms": 93.31471829175568, + "ttft_ms": 355.3162265883998 + }, + { + "e2e_ms": 12181.094213267927, + "request_id": 42, + "tpot_ms": 93.31547315047165, + "ttft_ms": 330.02912315802746 + }, + { + "e2e_ms": 12155.807109837553, + "request_id": 43, + "tpot_ms": 93.31640326087646, + "ttft_ms": 304.6238957062428 + }, + { + "e2e_ms": 12130.520006407181, + "request_id": 44, + "tpot_ms": 93.31564840216049, + "ttft_ms": 279.43265933279804 + }, + { + "e2e_ms": 12195.73249010398, + "request_id": 45, + "tpot_ms": 93.3147182917557, + "ttft_ms": 344.76326705100746 + }, + { + "e2e_ms": 12170.445386673606, + "request_id": 46, + "tpot_ms": 93.31471829175568, + "ttft_ms": 319.47616362063513 + }, + { + "e2e_ms": 12145.158283243234, + "request_id": 47, + "tpot_ms": 93.31564840216049, + "ttft_ms": 294.0709361688505 + }, + { + "e2e_ms": 12210.37076694003, + "request_id": 48, + "tpot_ms": 93.31471829175568, + "ttft_ms": 359.4015438870599 + }, + { + "e2e_ms": 12185.179530566586, + "request_id": 49, + "tpot_ms": 93.31547315047163, + "ttft_ms": 334.1144404566876 + }, + { + "e2e_ms": 12159.892427136241, + "request_id": 50, + "tpot_ms": 93.31640326087668, + "ttft_ms": 308.70921300490295 + }, + { + "e2e_ms": 12134.605323705899, + "request_id": 51, + "tpot_ms": 93.31564840216095, + "ttft_ms": 283.51797663145817 + }, + { + "e2e_ms": 12199.817807402724, + "request_id": 52, + "tpot_ms": 93.31471829175612, + "ttft_ms": 348.848584349696 + }, + { + "e2e_ms": 12174.626571029308, + "request_id": 53, + "tpot_ms": 93.3154731504721, + "ttft_ms": 323.5614809193521 + }, + { + "e2e_ms": 12149.339467598964, + "request_id": 54, + "tpot_ms": 93.31640326087691, + "ttft_ms": 298.1562534675959 + }, + { + "e2e_ms": 12124.05236416862, + "request_id": 55, + "tpot_ms": 93.31564840216095, + "ttft_ms": 272.96501709417953 + }, + { + "e2e_ms": 12189.264847865445, + "request_id": 56, + "tpot_ms": 93.31471829175612, + "ttft_ms": 338.2956248124174 + }, + { + "e2e_ms": 12163.977744435102, + "request_id": 57, + "tpot_ms": 93.31471829175614, + "ttft_ms": 313.00852138207347 + }, + { + "e2e_ms": 12138.690641004758, + "request_id": 58, + "tpot_ms": 93.31564840216095, + "ttft_ms": 287.60329393031725 + }, + { + "e2e_ms": 12203.903124701583, + "request_id": 59, + "tpot_ms": 93.31471829175612, + "ttft_ms": 352.9339016485551 + }, + { + "e2e_ms": 12178.711888328166, + "request_id": 60, + "tpot_ms": 93.31547315047209, + "ttft_ms": 327.6467982182112 + }, + { + "e2e_ms": 12153.424784897823, + "request_id": 61, + "tpot_ms": 93.31640326087691, + "ttft_ms": 302.24157076645497 + }, + { + "e2e_ms": 11949.074413391485, + "request_id": 62, + "tpot_ms": 91.90570140943657, + "ttft_ms": 277.0503343930386 + }, + { + "e2e_ms": 11702.728382593137, + "request_id": 63, + "tpot_ms": 89.45155464946347, + "ttft_ms": 342.38094211127645 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 23.83781385421753, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e62567aa33ec429331fcd9f24053a0c79db61ef288f7994e6488ae0f0fa79b7e", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 12394.3405299904, + "request_id": 0, + "tpot_ms": 95.48631545343278, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12520.094438712134, + "request_id": 1, + "tpot_ms": 96.04733720338449, + "ttft_ms": 322.08261388230494 + }, + { + "e2e_ms": 12556.880443424032, + "request_id": 2, + "tpot_ms": 96.10807445129032, + "ttft_ms": 351.1549881101628 + }, + { + "e2e_ms": 12502.892311945501, + "request_id": 3, + "tpot_ms": 96.14618123456218, + "ttft_ms": 292.32729515610424 + }, + { + "e2e_ms": 12539.549613984442, + "request_id": 4, + "tpot_ms": 96.30977229626299, + "ttft_ms": 308.2085323590427 + }, + { + "e2e_ms": 12576.335618696337, + "request_id": 5, + "tpot_ms": 96.32399459689755, + "ttft_ms": 343.188304890349 + }, + { + "e2e_ms": 12522.347487217807, + "request_id": 6, + "tpot_ms": 96.32399459689753, + "ttft_ms": 289.200173411821 + }, + { + "e2e_ms": 12559.133491929704, + "request_id": 7, + "tpot_ms": 96.32399459689753, + "ttft_ms": 325.98617812371697 + }, + { + "e2e_ms": 12505.145360451173, + "request_id": 8, + "tpot_ms": 96.32399459689753, + "ttft_ms": 271.99804664518723 + }, + { + "e2e_ms": 12541.802662490163, + "request_id": 9, + "tpot_ms": 96.32399459689792, + "ttft_ms": 308.6553486841268 + }, + { + "e2e_ms": 12578.588667202104, + "request_id": 10, + "tpot_ms": 96.32399459689827, + "ttft_ms": 345.4413533960228 + }, + { + "e2e_ms": 12524.60053572362, + "request_id": 11, + "tpot_ms": 96.32399459689864, + "ttft_ms": 291.453221917493 + }, + { + "e2e_ms": 12561.386540435564, + "request_id": 12, + "tpot_ms": 96.32399459689903, + "ttft_ms": 328.239226629389 + }, + { + "e2e_ms": 12507.39840895708, + "request_id": 13, + "tpot_ms": 96.32399459689938, + "ttft_ms": 274.2510951508592 + }, + { + "e2e_ms": 12544.05571099607, + "request_id": 14, + "tpot_ms": 96.32399459689938, + "ttft_ms": 310.90839718984853 + }, + { + "e2e_ms": 12580.841715708011, + "request_id": 15, + "tpot_ms": 96.32399459689938, + "ttft_ms": 347.6944019017907 + }, + { + "e2e_ms": 12526.853584229528, + "request_id": 16, + "tpot_ms": 96.32399459689938, + "ttft_ms": 293.7062704233071 + }, + { + "e2e_ms": 12563.63958894147, + "request_id": 17, + "tpot_ms": 96.32399459689938, + "ttft_ms": 330.49227513524926 + }, + { + "e2e_ms": 12509.651457462987, + "request_id": 18, + "tpot_ms": 96.32399459689938, + "ttft_ms": 276.5041436567657 + }, + { + "e2e_ms": 12546.308759501975, + "request_id": 19, + "tpot_ms": 96.32399459689938, + "ttft_ms": 313.161445695755 + }, + { + "e2e_ms": 12583.094764213918, + "request_id": 20, + "tpot_ms": 96.32399459689938, + "ttft_ms": 349.94745040769715 + }, + { + "e2e_ms": 12529.106632735506, + "request_id": 21, + "tpot_ms": 96.32399459689994, + "ttft_ms": 295.95931892921357 + }, + { + "e2e_ms": 12565.892637447447, + "request_id": 22, + "tpot_ms": 96.32399459689992, + "ttft_ms": 332.7453236411557 + }, + { + "e2e_ms": 12511.904505968956, + "request_id": 23, + "tpot_ms": 96.32399459689988, + "ttft_ms": 278.75719216267214 + }, + { + "e2e_ms": 12548.561808007931, + "request_id": 24, + "tpot_ms": 96.32399459689977, + "ttft_ms": 315.41449420166146 + }, + { + "e2e_ms": 12585.347812719874, + "request_id": 25, + "tpot_ms": 96.32399459689977, + "ttft_ms": 352.2004989136036 + }, + { + "e2e_ms": 12531.359681241383, + "request_id": 26, + "tpot_ms": 96.3239945968991, + "ttft_ms": 298.2123674351982 + }, + { + "e2e_ms": 12568.145685953326, + "request_id": 27, + "tpot_ms": 96.32399459689911, + "ttft_ms": 334.99837214714034 + }, + { + "e2e_ms": 12514.157554474836, + "request_id": 28, + "tpot_ms": 96.32399459689911, + "ttft_ms": 281.01024066864966 + }, + { + "e2e_ms": 12550.81485651381, + "request_id": 29, + "tpot_ms": 96.32399459689911, + "ttft_ms": 317.66754270762476 + }, + { + "e2e_ms": 12587.600861225752, + "request_id": 30, + "tpot_ms": 96.3239945968991, + "ttft_ms": 354.4535474195669 + }, + { + "e2e_ms": 12533.612729747261, + "request_id": 31, + "tpot_ms": 96.3239945968991, + "ttft_ms": 300.46541594107623 + }, + { + "e2e_ms": 12570.398734459204, + "request_id": 32, + "tpot_ms": 96.32399459689911, + "ttft_ms": 337.2514206530184 + }, + { + "e2e_ms": 12516.410602980714, + "request_id": 33, + "tpot_ms": 96.32399459689911, + "ttft_ms": 283.2632891745277 + }, + { + "e2e_ms": 12553.067905019689, + "request_id": 34, + "tpot_ms": 96.32399459689911, + "ttft_ms": 319.9205912135028 + }, + { + "e2e_ms": 12589.85390973163, + "request_id": 35, + "tpot_ms": 96.3239945968991, + "ttft_ms": 356.70659592544496 + }, + { + "e2e_ms": 12535.865778253139, + "request_id": 36, + "tpot_ms": 96.3239945968991, + "ttft_ms": 302.7184644469543 + }, + { + "e2e_ms": 12572.651782965082, + "request_id": 37, + "tpot_ms": 96.3239945968991, + "ttft_ms": 339.5044691588964 + }, + { + "e2e_ms": 12518.663651486591, + "request_id": 38, + "tpot_ms": 96.3239945968991, + "ttft_ms": 285.51633768040574 + }, + { + "e2e_ms": 12555.320953525566, + "request_id": 39, + "tpot_ms": 96.3239945968991, + "ttft_ms": 322.17363971938084 + }, + { + "e2e_ms": 12770.519034100516, + "request_id": 40, + "tpot_ms": 97.72881409188341, + "ttft_ms": 358.959644431323 + }, + { + "e2e_ms": 12538.070601439798, + "request_id": 41, + "tpot_ms": 96.32361486997611, + "ttft_ms": 304.9715129528323 + }, + { + "e2e_ms": 12574.856606151741, + "request_id": 42, + "tpot_ms": 96.32361486997611, + "ttft_ms": 341.75751766477447 + }, + { + "e2e_ms": 12520.86847467325, + "request_id": 43, + "tpot_ms": 96.32361486997611, + "ttft_ms": 287.7693861862838 + }, + { + "e2e_ms": 12557.654479385192, + "request_id": 44, + "tpot_ms": 96.32462827684986, + "ttft_ms": 324.4266882252589 + }, + { + "e2e_ms": 12503.666347906701, + "request_id": 45, + "tpot_ms": 96.3239945968991, + "ttft_ms": 270.51903410051636 + }, + { + "e2e_ms": 12540.323649945676, + "request_id": 46, + "tpot_ms": 96.3239945968991, + "ttft_ms": 307.17633613949147 + }, + { + "e2e_ms": 12577.109654657448, + "request_id": 47, + "tpot_ms": 96.32399459689775, + "ttft_ms": 343.9623408514336 + }, + { + "e2e_ms": 12523.121523178786, + "request_id": 48, + "tpot_ms": 96.3239945968964, + "ttft_ms": 289.97420937294294 + }, + { + "e2e_ms": 12559.907527890544, + "request_id": 49, + "tpot_ms": 96.32399459689495, + "ttft_ms": 326.7602140848851 + }, + { + "e2e_ms": 12505.919396411884, + "request_id": 50, + "tpot_ms": 96.32399459689363, + "ttft_ms": 272.7720826063944 + }, + { + "e2e_ms": 12542.576698450688, + "request_id": 51, + "tpot_ms": 96.32399459689228, + "ttft_ms": 309.4293846453695 + }, + { + "e2e_ms": 12579.362703162446, + "request_id": 52, + "tpot_ms": 96.32399459689228, + "ttft_ms": 346.2153893571269 + }, + { + "e2e_ms": 12525.374571683784, + "request_id": 53, + "tpot_ms": 96.32399459689228, + "ttft_ms": 292.2272578784657 + }, + { + "e2e_ms": 12562.160576395541, + "request_id": 54, + "tpot_ms": 96.32399459689228, + "ttft_ms": 329.0132625902231 + }, + { + "e2e_ms": 12508.17244491688, + "request_id": 55, + "tpot_ms": 96.32399459689226, + "ttft_ms": 275.0251311115619 + }, + { + "e2e_ms": 12544.829746955686, + "request_id": 56, + "tpot_ms": 96.32399459689228, + "ttft_ms": 311.6824331503665 + }, + { + "e2e_ms": 12581.615751667443, + "request_id": 57, + "tpot_ms": 96.32399459689228, + "ttft_ms": 348.4684378621239 + }, + { + "e2e_ms": 12527.627620188781, + "request_id": 58, + "tpot_ms": 96.32399459689228, + "ttft_ms": 294.4803063834627 + }, + { + "e2e_ms": 12564.413624900539, + "request_id": 59, + "tpot_ms": 96.32399459689228, + "ttft_ms": 331.2663110952201 + }, + { + "e2e_ms": 12314.305135768365, + "request_id": 60, + "tpot_ms": 94.77973981221895, + "ttft_ms": 277.2781796165589 + }, + { + "e2e_ms": 12170.36562563149, + "request_id": 61, + "tpot_ms": 93.35771766910335, + "ttft_ms": 313.93548165536345 + }, + { + "e2e_ms": 12023.933572014499, + "request_id": 62, + "tpot_ms": 91.91505579249903, + "ttft_ms": 350.72148636712086 + }, + { + "e2e_ms": 11724.699298497455, + "request_id": 63, + "tpot_ms": 89.98398380794484, + "ttft_ms": 296.73335488845964 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 26.0877902507782, + "offered_request_rate": 0.8, + "request_metrics_sha256": "b39b2c6a1c6236e30b246303e0b4d66b80a2515899d8c815efc015ef3f70db54", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 14028.106647728775, + "request_id": 0, + "tpot_ms": 108.35061559310502, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 14267.352384844213, + "request_id": 1, + "tpot_ms": 109.65331923259451, + "ttft_ms": 341.3808423047111 + }, + { + "e2e_ms": 14406.16288598222, + "request_id": 2, + "tpot_ms": 110.64064370419715, + "ttft_ms": 354.8011355491818 + }, + { + "e2e_ms": 14444.508775243683, + "request_id": 3, + "tpot_ms": 111.53949237897281, + "ttft_ms": 278.99324311413756 + }, + { + "e2e_ms": 14585.312621821135, + "request_id": 4, + "tpot_ms": 112.58765687327464, + "ttft_ms": 286.6801989152554 + }, + { + "e2e_ms": 14727.157204966976, + "request_id": 5, + "tpot_ms": 113.57630070139629, + "ttft_ms": 302.9670158896467 + }, + { + "e2e_ms": 14871.045643448613, + "request_id": 6, + "tpot_ms": 114.43037546464966, + "ttft_ms": 338.38795943810675 + }, + { + "e2e_ms": 14913.487485283544, + "request_id": 7, + "tpot_ms": 115.12950641852738, + "ttft_ms": 292.0401701305657 + }, + { + "e2e_ms": 15059.457396901958, + "request_id": 8, + "tpot_ms": 115.80285438725795, + "ttft_ms": 352.4948897201998 + }, + { + "e2e_ms": 15271.481983406971, + "request_id": 9, + "tpot_ms": 117.58757541921905, + "ttft_ms": 337.85990516615175 + }, + { + "e2e_ms": 15316.138289012584, + "request_id": 10, + "tpot_ms": 117.88145073243837, + "ttft_ms": 345.1940459929119 + }, + { + "e2e_ms": 15090.835365768904, + "request_id": 11, + "tpot_ms": 116.63565919716638, + "ttft_ms": 278.1066477287748 + }, + { + "e2e_ms": 15302.859952273917, + "request_id": 12, + "tpot_ms": 118.0348305674148, + "ttft_ms": 312.43647021223705 + }, + { + "e2e_ms": 15347.54563377864, + "request_id": 13, + "tpot_ms": 118.10044582475892, + "ttft_ms": 348.7890140342564 + }, + { + "e2e_ms": 15289.743982068245, + "request_id": 14, + "tpot_ms": 118.15013414242924, + "ttft_ms": 284.6769459797329 + }, + { + "e2e_ms": 15334.400287673823, + "request_id": 15, + "tpot_ms": 118.19982246009941, + "ttft_ms": 323.022835241197 + }, + { + "e2e_ms": 15380.492763872971, + "request_id": 16, + "tpot_ms": 118.2528557378034, + "ttft_ms": 362.38008517193873 + }, + { + "e2e_ms": 15322.691112162565, + "request_id": 17, + "tpot_ms": 118.27848723653601, + "ttft_ms": 301.32323312249196 + }, + { + "e2e_ms": 15367.376793667254, + "request_id": 18, + "tpot_ms": 118.29615526543176, + "ttft_ms": 343.765074957421 + }, + { + "e2e_ms": 15309.575141956848, + "request_id": 19, + "tpot_ms": 118.30562851819849, + "ttft_ms": 284.7603201456401 + }, + { + "e2e_ms": 15354.260823461536, + "request_id": 20, + "tpot_ms": 118.30690699483614, + "ttft_ms": 329.2836351173456 + }, + { + "e2e_ms": 15296.45917175113, + "request_id": 21, + "tpot_ms": 118.3069069948359, + "ttft_ms": 271.4819834069715 + }, + { + "e2e_ms": 15341.115477356709, + "request_id": 22, + "tpot_ms": 118.30690699483563, + "ttft_ms": 316.138289012585 + }, + { + "e2e_ms": 15387.207953555859, + "request_id": 23, + "tpot_ms": 118.31926259505173, + "ttft_ms": 360.6616039842905 + }, + { + "e2e_ms": 15329.406301845453, + "request_id": 24, + "tpot_ms": 118.31926259505147, + "ttft_ms": 302.8599522739164 + }, + { + "e2e_ms": 15375.4987780446, + "request_id": 25, + "tpot_ms": 118.33033971862962, + "ttft_ms": 347.5456337786404 + }, + { + "e2e_ms": 15317.697126334195, + "request_id": 26, + "tpot_ms": 118.33033971862953, + "ttft_ms": 289.74398206824503 + }, + { + "e2e_ms": 15362.382807838883, + "request_id": 27, + "tpot_ms": 118.33057102492174, + "ttft_ms": 334.400287673823 + }, + { + "e2e_ms": 15304.581156128477, + "request_id": 28, + "tpot_ms": 118.33057102492174, + "ttft_ms": 276.59863596341694 + }, + { + "e2e_ms": 15349.237461734056, + "request_id": 29, + "tpot_ms": 118.3192625950511, + "ttft_ms": 322.6911121625662 + }, + { + "e2e_ms": 15395.329937933206, + "request_id": 30, + "tpot_ms": 118.33033971862955, + "ttft_ms": 367.37679366725473 + }, + { + "e2e_ms": 15337.5282862228, + "request_id": 31, + "tpot_ms": 118.33033971862955, + "ttft_ms": 309.57514195684865 + }, + { + "e2e_ms": 15383.620762421948, + "request_id": 32, + "tpot_ms": 118.34141684220796, + "ttft_ms": 354.2608234615372 + }, + { + "e2e_ms": 15325.819110711542, + "request_id": 33, + "tpot_ms": 118.34141684220796, + "ttft_ms": 296.4591717511311 + }, + { + "e2e_ms": 15370.50479221623, + "request_id": 34, + "tpot_ms": 118.34164814850017, + "ttft_ms": 341.1154773567091 + }, + { + "e2e_ms": 15312.703140505824, + "request_id": 35, + "tpot_ms": 118.34164814850017, + "ttft_ms": 283.31382564630303 + }, + { + "e2e_ms": 15357.388822010513, + "request_id": 36, + "tpot_ms": 118.33057102492174, + "ttft_ms": 329.4063018454523 + }, + { + "e2e_ms": 15299.587170300107, + "request_id": 37, + "tpot_ms": 118.33057102492174, + "ttft_ms": 271.6046501350462 + }, + { + "e2e_ms": 15344.243475905685, + "request_id": 38, + "tpot_ms": 118.3192625950511, + "ttft_ms": 317.69712633419545 + }, + { + "e2e_ms": 15390.335952104835, + "request_id": 39, + "tpot_ms": 118.33033971862955, + "ttft_ms": 362.382807838884 + }, + { + "e2e_ms": 15332.53430039447, + "request_id": 40, + "tpot_ms": 118.33033971862987, + "ttft_ms": 304.5811561284779 + }, + { + "e2e_ms": 15378.626776593663, + "request_id": 41, + "tpot_ms": 118.34164814850084, + "ttft_ms": 349.2374617340559 + }, + { + "e2e_ms": 15320.8251248833, + "request_id": 42, + "tpot_ms": 118.34164814850119, + "ttft_ms": 291.4358100236498 + }, + { + "e2e_ms": 15365.51080638803, + "request_id": 43, + "tpot_ms": 118.33057102492307, + "ttft_ms": 337.52828622279907 + }, + { + "e2e_ms": 15307.709154677666, + "request_id": 44, + "tpot_ms": 118.33057102492342, + "ttft_ms": 279.726634512393 + }, + { + "e2e_ms": 15352.365460283294, + "request_id": 45, + "tpot_ms": 118.31926259505316, + "ttft_ms": 325.81911071154224 + }, + { + "e2e_ms": 15398.457936482486, + "request_id": 46, + "tpot_ms": 118.33033971863193, + "ttft_ms": 370.5047922162308 + }, + { + "e2e_ms": 15340.656284772123, + "request_id": 47, + "tpot_ms": 118.33033971863227, + "ttft_ms": 312.7031405058247 + }, + { + "e2e_ms": 15386.748760971315, + "request_id": 48, + "tpot_ms": 118.34141684221105, + "ttft_ms": 357.38882201051325 + }, + { + "e2e_ms": 15328.947109260951, + "request_id": 49, + "tpot_ms": 118.34141684221137, + "ttft_ms": 299.5871703001072 + }, + { + "e2e_ms": 15375.039585460143, + "request_id": 50, + "tpot_ms": 118.35272527208234, + "ttft_ms": 344.24347590568516 + }, + { + "e2e_ms": 15317.23793374978, + "request_id": 51, + "tpot_ms": 118.3527252720827, + "ttft_ms": 286.4418241952791 + }, + { + "e2e_ms": 15194.4223437212, + "request_id": 52, + "tpot_ms": 117.02274049863567, + "ttft_ms": 332.53430039447096 + }, + { + "e2e_ms": 14959.152693280232, + "request_id": 53, + "tpot_ms": 115.62535468185925, + "ttft_ms": 274.7326486841075 + }, + { + "e2e_ms": 14796.032219342294, + "request_id": 54, + "tpot_ms": 113.97800861778735, + "ttft_ms": 320.8251248832994 + }, + { + "e2e_ms": 14608.764053703837, + "request_id": 55, + "tpot_ms": 112.1516003725654, + "ttft_ms": 365.5108063880306 + }, + { + "e2e_ms": 14305.495223453576, + "request_id": 56, + "tpot_ms": 110.21878794311739, + "ttft_ms": 307.70915467766713 + }, + { + "e2e_ms": 14083.450969580696, + "request_id": 57, + "tpot_ms": 108.11878353777482, + "ttft_ms": 352.36546028329485 + }, + { + "e2e_ms": 13758.528543574443, + "request_id": 58, + "tpot_ms": 106.01547035434261, + "ttft_ms": 294.5638085729314 + }, + { + "e2e_ms": 13507.044041669204, + "request_id": 59, + "tpot_ms": 103.67234454249669, + "ttft_ms": 340.6562847721233 + }, + { + "e2e_ms": 13157.052579489218, + "request_id": 60, + "tpot_ms": 101.37163737344456, + "ttft_ms": 282.85463306175984 + }, + { + "e2e_ms": 12903.847402123616, + "request_id": 61, + "tpot_ms": 99.01496293592649, + "ttft_ms": 328.9471092609517 + }, + { + "e2e_ms": 12559.065843040158, + "request_id": 62, + "tpot_ms": 96.75527862590212, + "ttft_ms": 271.1454575505883 + }, + { + "e2e_ms": 12277.402762692504, + "request_id": 63, + "tpot_ms": 94.17452621214743, + "ttft_ms": 317.23793374978015 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 25.24283003807068, + "offered_request_rate": 1.2, + "request_metrics_sha256": "3825f14be2103486ce094e0b335c86deef683645cae8f05ef3d4beb6fb24cf95", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 15728.871662360909, + "request_id": 0, + "tpot_ms": 121.74246610201946, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 15904.548256233129, + "request_id": 1, + "tpot_ms": 122.72470152955445, + "ttft_ms": 318.5111619797136 + }, + { + "e2e_ms": 16140.50586206045, + "request_id": 2, + "tpot_ms": 124.75532551591714, + "ttft_ms": 296.5795215389744 + }, + { + "e2e_ms": 16211.825024161684, + "request_id": 3, + "tpot_ms": 125.48331552046938, + "ttft_ms": 275.44395306207383 + }, + { + "e2e_ms": 16388.84099440166, + "request_id": 4, + "tpot_ms": 126.36635826198417, + "ttft_ms": 340.31349512967114 + }, + { + "e2e_ms": 16460.2590055204, + "request_id": 5, + "tpot_ms": 127.08834011289514, + "ttft_ms": 320.03981118271605 + }, + { + "e2e_ms": 16531.726441149392, + "request_id": 6, + "tpot_ms": 127.72809453706097, + "ttft_ms": 310.25843494264916 + }, + { + "e2e_ms": 16603.193876778383, + "request_id": 7, + "tpot_ms": 128.3170301134443, + "ttft_ms": 306.93105237095784 + }, + { + "e2e_ms": 16674.731660081485, + "request_id": 8, + "tpot_ms": 128.84921275092722, + "ttft_ms": 310.8816407137276 + }, + { + "e2e_ms": 16746.318867894835, + "request_id": 9, + "tpot_ms": 129.28898520065795, + "ttft_ms": 326.61774741127834 + }, + { + "e2e_ms": 16982.978169021266, + "request_id": 10, + "tpot_ms": 130.92468009104587, + "ttft_ms": 355.5437974584379 + }, + { + "e2e_ms": 16948.91870099869, + "request_id": 11, + "tpot_ms": 131.099598546656, + "ttft_ms": 299.26968557337295 + }, + { + "e2e_ms": 17020.55604111586, + "request_id": 12, + "tpot_ms": 131.2589874619586, + "ttft_ms": 350.6646334471153 + }, + { + "e2e_ms": 16986.496573094286, + "request_id": 13, + "tpot_ms": 131.335760528223, + "ttft_ms": 306.854986009963 + }, + { + "e2e_ms": 17059.03976122884, + "request_id": 14, + "tpot_ms": 131.4398579526774, + "ttft_ms": 366.1778012388126 + }, + { + "e2e_ms": 17024.98029320726, + "request_id": 15, + "tpot_ms": 131.50690676411472, + "ttft_ms": 323.6031341646921 + }, + { + "e2e_ms": 16990.920825185683, + "request_id": 16, + "tpot_ms": 131.56847300071718, + "ttft_ms": 281.7247540946006 + }, + { + "e2e_ms": 17063.464013320234, + "request_id": 17, + "tpot_ms": 131.63886355765484, + "ttft_ms": 345.32834149806746 + }, + { + "e2e_ms": 17029.40454529866, + "request_id": 18, + "tpot_ms": 131.67812927130495, + "ttft_ms": 306.28212784293396 + }, + { + "e2e_ms": 16995.3450772771, + "request_id": 19, + "tpot_ms": 131.6904025418972, + "ttft_ms": 270.66395445615575 + }, + { + "e2e_ms": 17067.888265411675, + "request_id": 20, + "tpot_ms": 131.7034495326512, + "ttft_ms": 341.5501747649721 + }, + { + "e2e_ms": 17033.828797390113, + "request_id": 21, + "tpot_ms": 131.70595487135952, + "ttft_ms": 307.17252872745246 + }, + { + "e2e_ms": 16999.769329368555, + "request_id": 22, + "tpot_ms": 131.7084602100679, + "ttft_ms": 272.79488268993293 + }, + { + "e2e_ms": 17072.312517503127, + "request_id": 23, + "tpot_ms": 131.71809821033827, + "ttft_ms": 344.1140447901674 + }, + { + "e2e_ms": 17038.25304948157, + "request_id": 24, + "tpot_ms": 131.71982521031822, + "ttft_ms": 309.83524777115434 + }, + { + "e2e_ms": 17004.193581460007, + "request_id": 25, + "tpot_ms": 131.72116304093396, + "ttft_ms": 275.60587526139454 + }, + { + "e2e_ms": 17076.736769594583, + "request_id": 26, + "tpot_ms": 131.72963353311178, + "ttft_ms": 347.07331088938884 + }, + { + "e2e_ms": 17042.67730157302, + "request_id": 27, + "tpot_ms": 131.7305821943633, + "ttft_ms": 312.8933628888824 + }, + { + "e2e_ms": 17008.617833551463, + "request_id": 28, + "tpot_ms": 131.7309769369132, + "ttft_ms": 278.783762563485 + }, + { + "e2e_ms": 17081.161021686035, + "request_id": 29, + "tpot_ms": 131.73850434102513, + "ttft_ms": 350.37097037584175 + }, + { + "e2e_ms": 17047.986478516996, + "request_id": 30, + "tpot_ms": 131.74547225324986, + "ttft_ms": 316.3115023542638 + }, + { + "e2e_ms": 17013.927010495438, + "request_id": 31, + "tpot_ms": 131.74547225325, + "ttft_ms": 282.25203433268575 + }, + { + "e2e_ms": 17086.47019863001, + "request_id": 32, + "tpot_ms": 131.75260491481217, + "ttft_ms": 353.8893744488618 + }, + { + "e2e_ms": 17053.29565546097, + "request_id": 33, + "tpot_ms": 131.7595728270369, + "ttft_ms": 319.82990642728385 + }, + { + "e2e_ms": 17019.236187439412, + "request_id": 34, + "tpot_ms": 131.75957282703703, + "ttft_ms": 285.7704384057058 + }, + { + "e2e_ms": 17091.779375573984, + "request_id": 35, + "tpot_ms": 131.7595728270372, + "ttft_ms": 358.31362654026225 + }, + { + "e2e_ms": 17058.604832404944, + "request_id": 36, + "tpot_ms": 131.7665407392619, + "ttft_ms": 324.2541585186842 + }, + { + "e2e_ms": 17024.545364383386, + "request_id": 37, + "tpot_ms": 131.76654073926207, + "ttft_ms": 290.1946904971062 + }, + { + "e2e_ms": 17097.08855251796, + "request_id": 38, + "tpot_ms": 131.76654073926218, + "ttft_ms": 362.7378786316626 + }, + { + "e2e_ms": 17063.934932514778, + "request_id": 39, + "tpot_ms": 131.7736734008242, + "ttft_ms": 328.6784106101024 + }, + { + "e2e_ms": 17029.875464493216, + "request_id": 40, + "tpot_ms": 131.7736734008242, + "ttft_ms": 294.61894258854215 + }, + { + "e2e_ms": 17102.41865262779, + "request_id": 41, + "tpot_ms": 131.7736734008242, + "ttft_ms": 367.1621307231163 + }, + { + "e2e_ms": 17069.265032624608, + "request_id": 42, + "tpot_ms": 131.78080606238623, + "ttft_ms": 333.1026627015561 + }, + { + "e2e_ms": 17035.205564603046, + "request_id": 43, + "tpot_ms": 131.78080606238623, + "ttft_ms": 299.04319467999585 + }, + { + "e2e_ms": 16941.812657737883, + "request_id": 44, + "tpot_ms": 130.47422263719145, + "ttft_ms": 371.58638281457 + }, + { + "e2e_ms": 16742.48916517699, + "request_id": 45, + "tpot_ms": 129.1729311053857, + "ttft_ms": 337.5269147930098 + }, + { + "e2e_ms": 16538.03643831958, + "request_id": 46, + "tpot_ms": 127.83125190195379, + "ttft_ms": 303.46744677144954 + }, + { + "e2e_ms": 16435.957848641778, + "request_id": 47, + "tpot_ms": 126.45627727350987, + "ttft_ms": 376.0106349060237 + }, + { + "e2e_ms": 16228.476901947985, + "request_id": 48, + "tpot_ms": 125.09075381939779, + "ttft_ms": 341.9511668844635 + }, + { + "e2e_ms": 16020.052127617348, + "request_id": 49, + "tpot_ms": 123.7177986516098, + "ttft_ms": 307.89169886290324 + }, + { + "e2e_ms": 15809.427558622445, + "request_id": 50, + "tpot_ms": 122.32055435377619, + "ttft_ms": 274.71715569286914 + }, + { + "e2e_ms": 15703.511289947706, + "request_id": 51, + "tpot_ms": 120.91536178046667, + "ttft_ms": 347.26034382843807 + }, + { + "e2e_ms": 15485.101955905271, + "request_id": 52, + "tpot_ms": 119.46378803227081, + "ttft_ms": 313.20087580687783 + }, + { + "e2e_ms": 15260.624457846197, + "request_id": 53, + "tpot_ms": 117.95746555282956, + "ttft_ms": 280.02633263684373 + }, + { + "e2e_ms": 15118.941732007286, + "request_id": 54, + "tpot_ms": 116.27064733255806, + "ttft_ms": 352.56952077241266 + }, + { + "e2e_ms": 14865.280308326035, + "request_id": 55, + "tpot_ms": 114.54149807539514, + "ttft_ms": 318.5100527508524 + }, + { + "e2e_ms": 14599.985676841925, + "request_id": 56, + "tpot_ms": 112.71378084457564, + "ttft_ms": 285.3355095808183 + }, + { + "e2e_ms": 14420.806000135271, + "request_id": 57, + "tpot_ms": 110.73171104266838, + "ttft_ms": 357.87869771638725 + }, + { + "e2e_ms": 14142.519211198518, + "request_id": 58, + "tpot_ms": 108.80866127168261, + "ttft_ms": 323.819229694827 + }, + { + "e2e_ms": 13853.830695007076, + "request_id": 59, + "tpot_ms": 106.79657547493251, + "ttft_ms": 290.6656096906488 + }, + { + "e2e_ms": 13650.503338148035, + "request_id": 60, + "tpot_ms": 104.62436645922692, + "ttft_ms": 363.20879782621773 + }, + { + "e2e_ms": 13360.876271706531, + "request_id": 61, + "tpot_ms": 102.61202316458169, + "ttft_ms": 329.1493298046575 + }, + { + "e2e_ms": 13070.674002922458, + "request_id": 62, + "tpot_ms": 100.58801805607857, + "ttft_ms": 295.9957098004793 + }, + { + "e2e_ms": 12853.555073004585, + "request_id": 63, + "tpot_ms": 98.30721397691761, + "ttft_ms": 368.5388979360482 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 29.54408025741577, + "offered_request_rate": 1.6, + "request_metrics_sha256": "b7d73a1cbef8ca659f753135a4944cb3023d3a2424a19a741ae2198f39c2d172", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 17513.570713343794, + "request_id": 0, + "tpot_ms": 135.79521453495556, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 17840.282304762335, + "request_id": 1, + "tpot_ms": 137.71064841138914, + "ttft_ms": 351.0299565159147 + }, + { + "e2e_ms": 17895.63618825912, + "request_id": 2, + "tpot_ms": 138.10245652748367, + "ttft_ms": 356.6242092686922 + }, + { + "e2e_ms": 17848.77057711112, + "request_id": 3, + "tpot_ms": 138.39719968518088, + "ttft_ms": 272.32621709314975 + }, + { + "e2e_ms": 17904.1244606079, + "request_id": 4, + "tpot_ms": 138.80479077491768, + "ttft_ms": 275.9160321933569 + }, + { + "e2e_ms": 17959.478344104686, + "request_id": 5, + "tpot_ms": 139.18735062645055, + "ttft_ms": 282.6848145454632 + }, + { + "e2e_ms": 18014.921946878916, + "request_id": 6, + "tpot_ms": 139.5152186600215, + "ttft_ms": 296.48917705618504 + }, + { + "e2e_ms": 18070.36554965315, + "request_id": 7, + "tpot_ms": 139.8086106441042, + "ttft_ms": 314.671997851919 + }, + { + "e2e_ms": 18125.898871704834, + "request_id": 8, + "tpot_ms": 140.0646556798527, + "ttft_ms": 337.68760036353916 + }, + { + "e2e_ms": 18079.30241838919, + "request_id": 9, + "tpot_ms": 140.21470125635153, + "ttft_ms": 272.0353588325457 + }, + { + "e2e_ms": 18134.83860521682, + "request_id": 10, + "tpot_ms": 140.33922237580643, + "ttft_ms": 311.75736348940484 + }, + { + "e2e_ms": 18190.4645113219, + "request_id": 11, + "tpot_ms": 140.37869703173442, + "ttft_ms": 362.36998829163 + }, + { + "e2e_ms": 18143.870922782207, + "request_id": 12, + "tpot_ms": 140.38698051363622, + "ttft_ms": 314.7243975504068 + }, + { + "e2e_ms": 18097.274469466563, + "request_id": 13, + "tpot_ms": 140.36074521133403, + "ttft_ms": 271.4598276271438 + }, + { + "e2e_ms": 18152.810656294194, + "request_id": 14, + "tpot_ms": 140.32455966134754, + "ttft_ms": 331.5915793030584 + }, + { + "e2e_ms": 18106.217067754496, + "request_id": 15, + "tpot_ms": 140.29231877139563, + "ttft_ms": 289.0925837872533 + }, + { + "e2e_ms": 18161.753254582127, + "request_id": 16, + "tpot_ms": 140.24009333090902, + "ttft_ms": 351.2614015566804 + }, + { + "e2e_ms": 18115.15966604243, + "request_id": 17, + "tpot_ms": 140.19912163029534, + "ttft_ms": 309.87121899492286 + }, + { + "e2e_ms": 18170.69585287006, + "request_id": 18, + "tpot_ms": 140.12856604535176, + "ttft_ms": 374.3679651103839 + }, + { + "e2e_ms": 18124.102264330366, + "request_id": 19, + "tpot_ms": 140.05569589908265, + "ttft_ms": 337.0288851468697 + }, + { + "e2e_ms": 18077.42933293013, + "request_id": 20, + "tpot_ms": 139.98073600581233, + "ttft_ms": 299.87586019196044 + }, + { + "e2e_ms": 18132.96551975776, + "request_id": 21, + "tpot_ms": 139.8495015438229, + "ttft_ms": 372.0788236922541 + }, + { + "e2e_ms": 18086.369066442116, + "request_id": 22, + "tpot_ms": 139.76199603449132, + "ttft_ms": 336.5955700617178 + }, + { + "e2e_ms": 18039.682893849018, + "request_id": 23, + "tpot_ms": 139.67091992877667, + "ttft_ms": 301.4760628943822 + }, + { + "e2e_ms": 18095.21908067664, + "request_id": 24, + "tpot_ms": 139.54200325642663, + "ttft_ms": 373.38466711045726 + }, + { + "e2e_ms": 18048.532908083544, + "request_id": 25, + "tpot_ms": 139.5517742330953, + "ttft_ms": 325.4575804804407 + }, + { + "e2e_ms": 18001.846735490446, + "request_id": 26, + "tpot_ms": 139.5610841122856, + "ttft_ms": 277.589053230173 + }, + { + "e2e_ms": 18057.290338264673, + "request_id": 27, + "tpot_ms": 139.56778701592805, + "ttft_ms": 332.18138724181046 + }, + { + "e2e_ms": 18010.60416567157, + "request_id": 28, + "tpot_ms": 139.57154350494935, + "ttft_ms": 285.01814054300564 + }, + { + "e2e_ms": 18066.06100963866, + "request_id": 29, + "tpot_ms": 139.5730606683175, + "ttft_ms": 340.2823047623365 + }, + { + "e2e_ms": 18019.37483704556, + "request_id": 30, + "tpot_ms": 139.57447357032456, + "ttft_ms": 293.41669361433986 + }, + { + "e2e_ms": 18074.831681012645, + "request_id": 31, + "tpot_ms": 139.57528428268915, + "ttft_ms": 348.7705771111216 + }, + { + "e2e_ms": 18028.145508419548, + "request_id": 32, + "tpot_ms": 139.57669718469626, + "ttft_ms": 301.904965963125 + }, + { + "e2e_ms": 18083.678830471228, + "request_id": 33, + "tpot_ms": 139.5781100867033, + "ttft_ms": 357.25884945990674 + }, + { + "e2e_ms": 18036.99265787813, + "request_id": 34, + "tpot_ms": 139.57881653770684, + "ttft_ms": 310.48295758936104 + }, + { + "e2e_ms": 18092.528844705754, + "request_id": 35, + "tpot_ms": 139.57954554600127, + "ttft_ms": 365.9265603635937 + }, + { + "e2e_ms": 17876.05367572695, + "request_id": 36, + "tpot_ms": 138.24262431461773, + "ttft_ms": 319.240387770499 + }, + { + "e2e_ms": 17659.21004164318, + "request_id": 37, + "tpot_ms": 136.90209533219152, + "ttft_ms": 272.64393445485524 + }, + { + "e2e_ms": 17543.75228976072, + "request_id": 38, + "tpot_ms": 135.55570892326125, + "ttft_ms": 328.1772565065388 + }, + { + "e2e_ms": 17326.367174433202, + "request_id": 39, + "tpot_ms": 134.21089375170362, + "ttft_ms": 281.5836679668422 + }, + { + "e2e_ms": 17233.832476815856, + "request_id": 40, + "tpot_ms": 133.04427482475538, + "ttft_ms": 337.20957407192384 + }, + { + "e2e_ms": 17034.265611203216, + "request_id": 41, + "tpot_ms": 131.83976083205505, + "ttft_ms": 290.6159855322272 + }, + { + "e2e_ms": 16940.502121615402, + "request_id": 42, + "tpot_ms": 130.66419538607474, + "ttft_ms": 346.14930758391085 + }, + { + "e2e_ms": 16741.97064984836, + "request_id": 43, + "tpot_ms": 129.46783410082003, + "ttft_ms": 299.5557190442142 + }, + { + "e2e_ms": 16645.559306371404, + "request_id": 44, + "tpot_ms": 128.27139685432724, + "ttft_ms": 355.0919058718449 + }, + { + "e2e_ms": 16443.232533552808, + "request_id": 45, + "tpot_ms": 127.04515130882409, + "ttft_ms": 308.4983173321483 + }, + { + "e2e_ms": 16342.299705615802, + "request_id": 46, + "tpot_ms": 125.81311182248837, + "ttft_ms": 364.034504159779 + }, + { + "e2e_ms": 16135.159559133626, + "request_id": 47, + "tpot_ms": 124.54896569695704, + "ttft_ms": 317.44091562008236 + }, + { + "e2e_ms": 16031.7032146663, + "request_id": 48, + "tpot_ms": 123.29695174036006, + "ttft_ms": 372.9903436405735 + }, + { + "e2e_ms": 15823.308920667878, + "request_id": 49, + "tpot_ms": 122.0236594458299, + "ttft_ms": 326.3041710474788 + }, + { + "e2e_ms": 15613.44809689394, + "request_id": 50, + "tpot_ms": 120.73809066445794, + "ttft_ms": 279.71058250778214 + }, + { + "e2e_ms": 15508.031714506984, + "request_id": 51, + "tpot_ms": 119.47077015706711, + "ttft_ms": 335.2439045594622 + }, + { + "e2e_ms": 15292.981047367362, + "request_id": 52, + "tpot_ms": 118.14506547559841, + "ttft_ms": 288.5577319663639 + }, + { + "e2e_ms": 15175.360910096466, + "request_id": 53, + "tpot_ms": 116.78162985277542, + "ttft_ms": 344.09391879398754 + }, + { + "e2e_ms": 14945.589828664517, + "request_id": 54, + "tpot_ms": 115.34001639735139, + "ttft_ms": 297.4077462008893 + }, + { + "e2e_ms": 14803.64975337431, + "request_id": 55, + "tpot_ms": 113.78581420786767, + "ttft_ms": 352.85134897511483 + }, + { + "e2e_ms": 14557.342221274233, + "request_id": 56, + "tpot_ms": 112.21399247946627, + "ttft_ms": 306.16517638201657 + }, + { + "e2e_ms": 14399.594833150197, + "request_id": 57, + "tpot_ms": 110.5352189984338, + "ttft_ms": 361.6220203491025 + }, + { + "e2e_ms": 14144.625862747695, + "request_id": 58, + "tpot_ms": 108.89519696843851, + "ttft_ms": 314.93584775600425 + }, + { + "e2e_ms": 13973.496543699825, + "request_id": 59, + "tpot_ms": 107.1110539525727, + "ttft_ms": 370.3926917230902 + }, + { + "e2e_ms": 13708.4999588278, + "request_id": 60, + "tpot_ms": 105.3920743283292, + "ttft_ms": 323.70651912999193 + }, + { + "e2e_ms": 13445.970803422128, + "request_id": 61, + "tpot_ms": 103.6925232825609, + "ttft_ms": 277.02034653689367 + }, + { + "e2e_ms": 13273.580023880399, + "request_id": 62, + "tpot_ms": 101.89784531725846, + "ttft_ms": 332.5536685885737 + }, + { + "e2e_ms": 13000.702540117643, + "request_id": 63, + "tpot_ms": 100.1168113710407, + "ttft_ms": 285.86749599547545 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 33.14970922470093, + "offered_request_rate": 2.4, + "request_metrics_sha256": "3b42135fbabbd54647284962ccc9fdf0b900fbc0b6ed0bdecf03477d5c409d18", + "request_rate_per_gpu": 0.6, + "requests": [ + { + "e2e_ms": 21942.173036302996, + "request_id": 0, + "tpot_ms": 170.66609896770518, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 22008.341136862466, + "request_id": 1, + "tpot_ms": 170.96613901739173, + "ttft_ms": 295.64148165371654 + }, + { + "e2e_ms": 22074.54102422534, + "request_id": 2, + "tpot_ms": 171.22660942224667, + "ttft_ms": 328.7616276000109 + }, + { + "e2e_ms": 22036.227907899276, + "request_id": 3, + "tpot_ms": 171.37344968894675, + "ttft_ms": 271.7997974030406 + }, + { + "e2e_ms": 22102.42779526115, + "request_id": 4, + "tpot_ms": 171.64131409153669, + "ttft_ms": 303.98090563598765 + }, + { + "e2e_ms": 22168.65946942643, + "request_id": 5, + "tpot_ms": 171.89659303259273, + "ttft_ms": 337.79215428715537 + }, + { + "e2e_ms": 22130.37813990278, + "request_id": 6, + "tpot_ms": 172.0281077294899, + "ttft_ms": 282.80845825756097 + }, + { + "e2e_ms": 22365.99994575454, + "request_id": 7, + "tpot_ms": 173.57085404199856, + "ttft_ms": 322.5014824207202 + }, + { + "e2e_ms": 22158.3592864644, + "request_id": 8, + "tpot_ms": 172.3496956452424, + "ttft_ms": 269.9479395186155 + }, + { + "e2e_ms": 22393.981092316164, + "request_id": 9, + "tpot_ms": 173.8419366523574, + "ttft_ms": 316.0551374667744 + }, + { + "e2e_ms": 22460.243568400445, + "request_id": 10, + "tpot_ms": 173.96590922572528, + "ttft_ms": 366.57309673333316 + }, + { + "e2e_ms": 22252.602909110316, + "request_id": 11, + "tpot_ms": 172.6883751510306, + "ttft_ms": 321.1792649294285 + }, + { + "e2e_ms": 22044.894176229613, + "request_id": 12, + "tpot_ms": 171.40112664047544, + "ttft_ms": 276.9510928892318 + }, + { + "e2e_ms": 21941.653543442477, + "request_id": 13, + "tpot_ms": 170.1065182287643, + "ttft_ms": 338.1257283894126 + }, + { + "e2e_ms": 21734.138079618475, + "request_id": 14, + "tpot_ms": 168.80971544527205, + "ttft_ms": 295.30421806892537 + }, + { + "e2e_ms": 21631.119179231846, + "request_id": 15, + "tpot_ms": 167.51267286797687, + "ttft_ms": 357.0097249987798 + }, + { + "e2e_ms": 21420.354658102242, + "request_id": 16, + "tpot_ms": 166.18662353135022, + "ttft_ms": 314.65346962076256 + }, + { + "e2e_ms": 21209.648295563282, + "request_id": 17, + "tpot_ms": 164.85776701341018, + "ttft_ms": 272.7118848601879 + }, + { + "e2e_ms": 21101.69867612793, + "request_id": 18, + "tpot_ms": 163.50974377121332, + "ttft_ms": 335.9612171838391 + }, + { + "e2e_ms": 20890.778369055995, + "request_id": 19, + "tpot_ms": 162.16628083276893, + "ttft_ms": 295.66070329433813 + }, + { + "e2e_ms": 20782.558056179907, + "request_id": 20, + "tpot_ms": 160.7985342727971, + "ttft_ms": 361.14420353467835 + }, + { + "e2e_ms": 20571.512469390585, + "request_id": 21, + "tpot_ms": 159.44598375979533, + "ttft_ms": 321.8725318965792 + }, + { + "e2e_ms": 20360.405278761264, + "request_id": 22, + "tpot_ms": 158.09565605925098, + "ttft_ms": 282.2569592363866 + }, + { + "e2e_ms": 20251.586130359967, + "request_id": 23, + "tpot_ms": 156.71227679254048, + "ttft_ms": 349.12697770732314 + }, + { + "e2e_ms": 20042.10448060375, + "request_id": 24, + "tpot_ms": 155.37142770924103, + "ttft_ms": 309.93316153013774 + }, + { + "e2e_ms": 19934.008109113514, + "request_id": 25, + "tpot_ms": 154.05960281316982, + "ttft_ms": 368.43855184094565 + }, + { + "e2e_ms": 19722.984661067087, + "request_id": 26, + "tpot_ms": 152.73931080974432, + "ttft_ms": 325.0921882295579 + }, + { + "e2e_ms": 19511.768043076878, + "request_id": 27, + "tpot_ms": 151.41621155599998, + "ttft_ms": 281.9091754648788 + }, + { + "e2e_ms": 19403.203575383635, + "request_id": 28, + "tpot_ms": 150.09414530000853, + "ttft_ms": 341.2471222825513 + }, + { + "e2e_ms": 19191.714653347688, + "request_id": 29, + "tpot_ms": 148.76487274343572, + "ttft_ms": 298.57581493135046 + }, + { + "e2e_ms": 19082.708845359608, + "request_id": 30, + "tpot_ms": 147.43434256455518, + "ttft_ms": 358.5473396611025 + }, + { + "e2e_ms": 18871.078709760895, + "request_id": 31, + "tpot_ms": 146.10277278695904, + "ttft_ms": 316.0265658170971 + }, + { + "e2e_ms": 18659.370447496967, + "request_id": 32, + "tpot_ms": 144.7695321688903, + "ttft_ms": 273.6398620479008 + }, + { + "e2e_ms": 18549.637648096734, + "request_id": 33, + "tpot_ms": 143.43132538741386, + "ttft_ms": 333.85932389517234 + }, + { + "e2e_ms": 18337.808235788358, + "request_id": 34, + "tpot_ms": 142.0958670025312, + "ttft_ms": 291.63312646689656 + }, + { + "e2e_ms": 18227.80864738452, + "request_id": 35, + "tpot_ms": 140.75352098043697, + "ttft_ms": 352.1114828690255 + }, + { + "e2e_ms": 18015.40726100407, + "request_id": 36, + "tpot_ms": 139.41149279948888, + "ttft_ms": 310.14767546898446 + }, + { + "e2e_ms": 17904.857868774245, + "request_id": 37, + "tpot_ms": 138.06211837292886, + "ttft_ms": 370.9688354122758 + }, + { + "e2e_ms": 17692.008101355255, + "request_id": 38, + "tpot_ms": 136.71510258553306, + "ttft_ms": 329.1900729925583 + }, + { + "e2e_ms": 17479.148877024498, + "request_id": 39, + "tpot_ms": 135.36749298254742, + "ttft_ms": 287.47726824097344 + }, + { + "e2e_ms": 17381.961391787096, + "request_id": 40, + "tpot_ms": 134.1089349239123, + "ttft_ms": 350.1266564502323 + }, + { + "e2e_ms": 17178.01129231477, + "request_id": 41, + "tpot_ms": 132.8327622110564, + "ttft_ms": 308.2504915106092 + }, + { + "e2e_ms": 17080.086531895085, + "request_id": 42, + "tpot_ms": 131.5805170059729, + "ttft_ms": 369.36087213652866 + }, + { + "e2e_ms": 16876.654129344566, + "request_id": 43, + "tpot_ms": 130.30640250229436, + "ttft_ms": 327.74101155318294 + }, + { + "e2e_ms": 16671.422925287778, + "request_id": 44, + "tpot_ms": 129.0178430005707, + "ttft_ms": 286.15686421529676 + }, + { + "e2e_ms": 16571.761179006833, + "request_id": 45, + "tpot_ms": 127.74972161297816, + "ttft_ms": 347.5465341586066 + }, + { + "e2e_ms": 16364.721381165033, + "request_id": 46, + "tpot_ms": 126.44572566423672, + "ttft_ms": 306.1142218069683 + }, + { + "e2e_ms": 16261.449604637399, + "request_id": 47, + "tpot_ms": 125.14700916940275, + "ttft_ms": 367.77944012325037 + }, + { + "e2e_ms": 16053.400400183471, + "request_id": 48, + "tpot_ms": 123.8340136000885, + "ttft_ms": 326.48067297223093 + }, + { + "e2e_ms": 15845.03658651726, + "request_id": 49, + "tpot_ms": 122.50141376236263, + "ttft_ms": 287.35703869720555 + }, + { + "e2e_ms": 15739.724302020804, + "request_id": 50, + "tpot_ms": 121.15600683324593, + "ttft_ms": 352.9114341985711 + }, + { + "e2e_ms": 15530.891082399023, + "request_id": 51, + "tpot_ms": 119.81946138926746, + "ttft_ms": 313.8194859620569 + }, + { + "e2e_ms": 15319.199082162215, + "request_id": 52, + "tpot_ms": 118.45427332697808, + "ttft_ms": 275.5063696359983 + }, + { + "e2e_ms": 15206.960333133673, + "request_id": 53, + "tpot_ms": 117.04949498375757, + "ttft_ms": 341.6744701964625 + }, + { + "e2e_ms": 14987.9081257507, + "request_id": 54, + "tpot_ms": 115.62635253449054, + "ttft_ms": 303.36135387040383 + }, + { + "e2e_ms": 14861.077413909577, + "request_id": 55, + "tpot_ms": 114.10642655651417, + "ttft_ms": 369.5612412322795 + }, + { + "e2e_ms": 14633.756981193535, + "request_id": 56, + "tpot_ms": 112.61817997075843, + "ttft_ms": 331.2481249072157 + }, + { + "e2e_ms": 14403.991359276923, + "request_id": 57, + "tpot_ms": 111.11042963695559, + "ttft_ms": 292.9667953835633 + }, + { + "e2e_ms": 14264.847964808048, + "request_id": 58, + "tpot_ms": 109.49330311228506, + "ttft_ms": 359.1984695478452 + }, + { + "e2e_ms": 14029.729570522895, + "request_id": 59, + "tpot_ms": 107.94340496454889, + "ttft_ms": 320.91714002518756 + }, + { + "e2e_ms": 13793.064611419886, + "request_id": 60, + "tpot_ms": 106.38108660628622, + "ttft_ms": 282.66661242153646 + }, + { + "e2e_ms": 13648.251078198633, + "request_id": 61, + "tpot_ms": 104.71931331978594, + "ttft_ms": 348.89828658581837 + }, + { + "e2e_ms": 13412.62809971594, + "request_id": 62, + "tpot_ms": 103.16519953332896, + "ttft_ms": 310.647758983162 + }, + { + "e2e_ms": 13260.053320226874, + "request_id": 63, + "tpot_ms": 101.44207153668842, + "ttft_ms": 376.9102350674451 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 36.70769214630127, + "offered_request_rate": 3.2, + "request_metrics_sha256": "bfba1ae71cfb6d0595614b14bc49ac47591bfad6792be0a3d98d2c98b16012e8", + "request_rate_per_gpu": 0.8, + "requests": [ + { + "e2e_ms": 24003.81066522886, + "request_id": 0, + "tpot_ms": 186.89946612460176, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 23901.058156177503, + "request_id": 1, + "tpot_ms": 185.74139588389912, + "ttft_ms": 311.9008789223171 + }, + { + "e2e_ms": 23798.190109844723, + "request_id": 2, + "tpot_ms": 184.56225305811301, + "ttft_ms": 358.7839714643697 + }, + { + "e2e_ms": 23590.44639829378, + "request_id": 3, + "tpot_ms": 183.2673714648551, + "ttft_ms": 315.49022225718517 + }, + { + "e2e_ms": 23382.70131446869, + "request_id": 4, + "tpot_ms": 181.9732705315891, + "ttft_ms": 272.09595695687216 + }, + { + "e2e_ms": 23279.64954297851, + "request_id": 5, + "tpot_ms": 180.78914255292864, + "ttft_ms": 319.4284387565705 + }, + { + "e2e_ms": 23071.973865900134, + "request_id": 6, + "tpot_ms": 179.49398625037213, + "ttft_ms": 276.2376121028707 + }, + { + "e2e_ms": 22969.022726489202, + "request_id": 7, + "tpot_ms": 178.2861180199918, + "ttft_ms": 326.6857379502426 + }, + { + "e2e_ms": 22761.29942646063, + "request_id": 8, + "tpot_ms": 176.98924979786668, + "ttft_ms": 283.66470213156344 + }, + { + "e2e_ms": 22658.264587165988, + "request_id": 9, + "tpot_ms": 175.7551174352214, + "ttft_ms": 337.36467289287384 + }, + { + "e2e_ms": 22450.304224673095, + "request_id": 10, + "tpot_ms": 174.4550575162989, + "ttft_ms": 294.51192010313275 + }, + { + "e2e_ms": 22346.821408551063, + "request_id": 11, + "tpot_ms": 173.1774539652428, + "ttft_ms": 353.28475496522805 + }, + { + "e2e_ms": 22138.79812545183, + "request_id": 12, + "tpot_ms": 171.87427427697472, + "ttft_ms": 310.7652922760442 + }, + { + "e2e_ms": 22035.248814705075, + "request_id": 13, + "tpot_ms": 170.57508935568035, + "ttft_ms": 372.21246653366967 + }, + { + "e2e_ms": 21827.324416126074, + "request_id": 14, + "tpot_ms": 169.27188182433355, + "ttft_ms": 329.79542443571484 + }, + { + "e2e_ms": 21619.373671552865, + "request_id": 15, + "tpot_ms": 167.96821838814947, + "ttft_ms": 287.40993625788303 + }, + { + "e2e_ms": 21512.77581709026, + "request_id": 16, + "tpot_ms": 166.64216905152279, + "ttft_ms": 349.2203475468658 + }, + { + "e2e_ms": 21303.255969153783, + "request_id": 17, + "tpot_ms": 165.32482604305434, + "ttft_ms": 307.003061685883 + }, + { + "e2e_ms": 21196.567333974876, + "request_id": 18, + "tpot_ms": 163.99164288866731, + "ttft_ms": 369.62868711412875 + }, + { + "e2e_ms": 20986.94051377241, + "request_id": 19, + "tpot_ms": 162.67002407488405, + "ttft_ms": 327.84745626213316 + }, + { + "e2e_ms": 20777.255965035714, + "request_id": 20, + "tpot_ms": 161.34765321373814, + "ttft_ms": 286.10400689097213 + }, + { + "e2e_ms": 20670.377044913395, + "request_id": 21, + "tpot_ms": 159.9951027007364, + "ttft_ms": 350.99900191987297 + }, + { + "e2e_ms": 20460.656782932234, + "request_id": 22, + "tpot_ms": 158.67196467076286, + "ttft_ms": 309.3172697453532 + }, + { + "e2e_ms": 20353.38846177537, + "request_id": 23, + "tpot_ms": 157.31793948495124, + "ttft_ms": 374.01014718656 + }, + { + "e2e_ms": 20144.48097023076, + "request_id": 24, + "tpot_ms": 155.99981541047092, + "ttft_ms": 332.5044131009545 + }, + { + "e2e_ms": 19934.83773528968, + "request_id": 25, + "tpot_ms": 154.6729368346084, + "ttft_ms": 291.37475729441144 + }, + { + "e2e_ms": 19827.980953909257, + "request_id": 26, + "tpot_ms": 153.35264483118294, + "ttft_ms": 352.1950603490236 + }, + { + "e2e_ms": 19618.20597824765, + "request_id": 27, + "tpot_ms": 152.02412197401114, + "ttft_ms": 311.1424875482367 + }, + { + "e2e_ms": 19511.107444230154, + "request_id": 28, + "tpot_ms": 150.6975599886974, + "ttft_ms": 372.5173256655836 + }, + { + "e2e_ms": 19301.196316545185, + "request_id": 29, + "tpot_ms": 149.36537894653213, + "ttft_ms": 331.7931903356044 + }, + { + "e2e_ms": 19091.249936104825, + "request_id": 30, + "tpot_ms": 148.03267778881258, + "ttft_ms": 291.0998569256247 + }, + { + "e2e_ms": 18983.78646717311, + "request_id": 31, + "tpot_ms": 146.7011080112165, + "ttft_ms": 352.7457497486193 + }, + { + "e2e_ms": 18773.765669374145, + "request_id": 32, + "tpot_ms": 145.36784483585674, + "ttft_ms": 312.0493752203348 + }, + { + "e2e_ms": 18665.888247551986, + "request_id": 33, + "tpot_ms": 144.03124523287858, + "ttft_ms": 373.9201029764097 + }, + { + "e2e_ms": 18455.8068747313, + "request_id": 34, + "tpot_ms": 142.6968275070543, + "ttft_ms": 333.30978133540157 + }, + { + "e2e_ms": 18245.529234152025, + "request_id": 35, + "tpot_ms": 141.3602386069844, + "ttft_ms": 292.7789310650048 + }, + { + "e2e_ms": 18137.294514438578, + "request_id": 36, + "tpot_ms": 140.01821042603632, + "ttft_ms": 354.98179033196385 + }, + { + "e2e_ms": 17926.83360591763, + "request_id": 37, + "tpot_ms": 138.67967407469433, + "ttft_ms": 314.5149984314486 + }, + { + "e2e_ms": 17716.242055541137, + "request_id": 38, + "tpot_ms": 137.3398665125213, + "ttft_ms": 274.0790084509328 + }, + { + "e2e_ms": 17607.549497877386, + "request_id": 39, + "tpot_ms": 135.99225690953574, + "ttft_ms": 336.53287036634794 + }, + { + "e2e_ms": 17401.54255835392, + "request_id": 40, + "tpot_ms": 134.6880568492608, + "ttft_ms": 296.1593384978016 + }, + { + "e2e_ms": 17301.7591255476, + "request_id": 41, + "tpot_ms": 133.41188413640484, + "ttft_ms": 358.4498402241838 + }, + { + "e2e_ms": 17095.506427630044, + "request_id": 42, + "tpot_ms": 132.10522228512968, + "ttft_ms": 318.1431974185731 + }, + { + "e2e_ms": 16889.623559688287, + "request_id": 43, + "tpot_ms": 130.8012299461049, + "ttft_ms": 277.86735653296194 + }, + { + "e2e_ms": 16788.559022297493, + "request_id": 44, + "tpot_ms": 129.5126704443812, + "ttft_ms": 340.449875861081 + }, + { + "e2e_ms": 16581.727329092846, + "request_id": 45, + "tpot_ms": 128.200721970688, + "ttft_ms": 300.23563881547057 + }, + { + "e2e_ms": 16478.854197918037, + "request_id": 46, + "tpot_ms": 126.8967260219465, + "ttft_ms": 362.9699931308306 + }, + { + "e2e_ms": 16270.819161297495, + "request_id": 47, + "tpot_ms": 125.57481733363996, + "ttft_ms": 322.8173599252209 + }, + { + "e2e_ms": 16062.627892404027, + "request_id": 48, + "tpot_ms": 124.25122687214136, + "ttft_ms": 282.72207964207576 + }, + { + "e2e_ms": 15958.430745404816, + "request_id": 49, + "tpot_ms": 122.91862703441548, + "ttft_ms": 347.7651120340504 + }, + { + "e2e_ms": 15749.715539461335, + "request_id": 50, + "tpot_ms": 121.58574387256766, + "ttft_ms": 308.3260676452415 + }, + { + "e2e_ms": 15645.048986506545, + "request_id": 51, + "tpot_ms": 120.24919842858912, + "ttft_ms": 373.40078607572735 + }, + { + "e2e_ms": 15435.036319721643, + "request_id": 52, + "tpot_ms": 118.8997102377205, + "ttft_ms": 334.77311953113895 + }, + { + "e2e_ms": 15224.012292267464, + "request_id": 53, + "tpot_ms": 117.54201604221191, + "ttft_ms": 296.1762549065519 + }, + { + "e2e_ms": 15109.12675155149, + "request_id": 54, + "tpot_ms": 116.11789378026353, + "ttft_ms": 362.1542414580219 + }, + { + "e2e_ms": 14893.238736493451, + "request_id": 55, + "tpot_ms": 114.72141540015762, + "ttft_ms": 323.6189806734337 + }, + { + "e2e_ms": 14675.411853468433, + "request_id": 56, + "tpot_ms": 113.30942780834319, + "ttft_ms": 285.1145218088469 + }, + { + "e2e_ms": 14549.812898218817, + "request_id": 57, + "tpot_ms": 111.79839094375423, + "ttft_ms": 351.4172483620292 + }, + { + "e2e_ms": 14329.820655618192, + "request_id": 58, + "tpot_ms": 110.36886820693505, + "ttft_ms": 312.97439333744137 + }, + { + "e2e_ms": 14108.094791808619, + "request_id": 59, + "tpot_ms": 108.92545237461232, + "ttft_ms": 274.56234023285475 + }, + { + "e2e_ms": 13975.596499372607, + "request_id": 60, + "tpot_ms": 107.35951607044248, + "ttft_ms": 340.93795842641225 + }, + { + "e2e_ms": 13753.714210521188, + "request_id": 61, + "tpot_ms": 105.91438347527057, + "ttft_ms": 302.58750916182464 + }, + { + "e2e_ms": 13622.257898704496, + "request_id": 62, + "tpot_ms": 104.35568588071192, + "ttft_ms": 369.08579185408286 + }, + { + "e2e_ms": 13397.788527763809, + "request_id": 63, + "tpot_ms": 102.88969749082138, + "ttft_ms": 330.7969464294942 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 32.150320291519165, + "offered_request_rate": 0.1, + "request_metrics_sha256": "b45cd3a473a98bee1127b2315d0c0c59ded580275f6e6d10a3e5a52d85d24de4", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 11656.879780530655, + "request_id": 0, + "tpot_ms": 89.67953789863164, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 11729.945143691864, + "request_id": 1, + "tpot_ms": 89.99959080602144, + "ttft_ms": 299.9971113271407 + }, + { + "e2e_ms": 11712.730146443742, + "request_id": 2, + "tpot_ms": 90.00056214193414, + "ttft_ms": 282.65875441810806 + }, + { + "e2e_ms": 11785.795509604945, + "request_id": 3, + "tpot_ms": 89.99959080602139, + "ttft_ms": 355.8474772402285 + }, + { + "e2e_ms": 11770.95360336775, + "request_id": 4, + "tpot_ms": 90.01924789792568, + "ttft_ms": 338.50912033118874 + }, + { + "e2e_ms": 11756.111697130556, + "request_id": 5, + "tpot_ms": 90.0192478979257, + "ttft_ms": 323.66721409399446 + }, + { + "e2e_ms": 11741.269790893028, + "request_id": 6, + "tpot_ms": 90.01924789792305, + "ttft_ms": 308.8253078568002 + }, + { + "e2e_ms": 11726.427884655508, + "request_id": 7, + "tpot_ms": 90.01924789792311, + "ttft_ms": 293.98340161927194 + }, + { + "e2e_ms": 11709.212887407048, + "request_id": 8, + "tpot_ms": 90.00056214193147, + "ttft_ms": 279.1414953817508 + }, + { + "e2e_ms": 11782.278250567928, + "request_id": 9, + "tpot_ms": 89.99959080601886, + "ttft_ms": 352.3302182035337 + }, + { + "e2e_ms": 11767.436344330406, + "request_id": 10, + "tpot_ms": 90.01924789792311, + "ttft_ms": 334.9918612941707 + }, + { + "e2e_ms": 11752.594438092885, + "request_id": 11, + "tpot_ms": 90.01924789792311, + "ttft_ms": 320.14995505664956 + }, + { + "e2e_ms": 11737.752531855705, + "request_id": 12, + "tpot_ms": 90.01924789792581, + "ttft_ms": 305.30804881912843 + }, + { + "e2e_ms": 11722.91062561817, + "request_id": 13, + "tpot_ms": 90.019247897923, + "ttft_ms": 290.46614258194836 + }, + { + "e2e_ms": 11705.69562836971, + "request_id": 14, + "tpot_ms": 90.00056214193148, + "ttft_ms": 275.624236344413 + }, + { + "e2e_ms": 11778.76099153056, + "request_id": 15, + "tpot_ms": 89.99959080601862, + "ttft_ms": 348.8129591661959 + }, + { + "e2e_ms": 11763.919085293026, + "request_id": 16, + "tpot_ms": 90.019247897923, + "ttft_ms": 331.4746022568045 + }, + { + "e2e_ms": 11749.07717905549, + "request_id": 17, + "tpot_ms": 90.019247897923, + "ttft_ms": 316.63269601926913 + }, + { + "e2e_ms": 11734.235272817954, + "request_id": 18, + "tpot_ms": 90.019247897923, + "ttft_ms": 301.7907897817338 + }, + { + "e2e_ms": 11719.39336658042, + "request_id": 19, + "tpot_ms": 90.019247897923, + "ttft_ms": 286.94888354419845 + }, + { + "e2e_ms": 11702.178369331961, + "request_id": 20, + "tpot_ms": 90.00056214193148, + "ttft_ms": 272.1069773066631 + }, + { + "e2e_ms": 11775.24373249281, + "request_id": 21, + "tpot_ms": 89.99959080601862, + "ttft_ms": 345.295700128446 + }, + { + "e2e_ms": 11760.401826255276, + "request_id": 22, + "tpot_ms": 90.01924789792301, + "ttft_ms": 327.95734321905456 + }, + { + "e2e_ms": 11745.55992001774, + "request_id": 23, + "tpot_ms": 90.019247897923, + "ttft_ms": 313.1154369815192 + }, + { + "e2e_ms": 11730.718013780204, + "request_id": 24, + "tpot_ms": 90.019247897923, + "ttft_ms": 298.2735307439839 + }, + { + "e2e_ms": 11713.50301653041, + "request_id": 25, + "tpot_ms": 90.00056214192095, + "ttft_ms": 283.43162450644854 + }, + { + "e2e_ms": 11786.568379689925, + "request_id": 26, + "tpot_ms": 89.9995908060081, + "ttft_ms": 356.6203473268956 + }, + { + "e2e_ms": 11771.726473451054, + "request_id": 27, + "tpot_ms": 90.01924789791249, + "ttft_ms": 339.28199041616836 + }, + { + "e2e_ms": 11756.884567212182, + "request_id": 28, + "tpot_ms": 90.01924789791249, + "ttft_ms": 324.4400841772972 + }, + { + "e2e_ms": 11742.042660973311, + "request_id": 29, + "tpot_ms": 90.01924789791249, + "ttft_ms": 309.59817793842603 + }, + { + "e2e_ms": 11727.20075473444, + "request_id": 30, + "tpot_ms": 90.01924789791249, + "ttft_ms": 294.7562716995549 + }, + { + "e2e_ms": 11709.985757484617, + "request_id": 31, + "tpot_ms": 90.00056214192074, + "ttft_ms": 279.9143654606837 + }, + { + "e2e_ms": 11783.051120644132, + "request_id": 32, + "tpot_ms": 89.9995908060081, + "ttft_ms": 353.1030882811024 + }, + { + "e2e_ms": 11768.20921440526, + "request_id": 33, + "tpot_ms": 90.01924789791249, + "ttft_ms": 335.7647313703751 + }, + { + "e2e_ms": 11753.36730816639, + "request_id": 34, + "tpot_ms": 90.01924789791249, + "ttft_ms": 320.92282513150394 + }, + { + "e2e_ms": 11738.525401927518, + "request_id": 35, + "tpot_ms": 90.01924789791249, + "ttft_ms": 306.0809188926328 + }, + { + "e2e_ms": 11723.683495688647, + "request_id": 36, + "tpot_ms": 90.01924789791249, + "ttft_ms": 291.2390126537616 + }, + { + "e2e_ms": 11706.468498438824, + "request_id": 37, + "tpot_ms": 90.00056214192074, + "ttft_ms": 276.39710641489046 + }, + { + "e2e_ms": 11779.533861598338, + "request_id": 38, + "tpot_ms": 89.9995908060081, + "ttft_ms": 349.5858292353091 + }, + { + "e2e_ms": 11764.691955359467, + "request_id": 39, + "tpot_ms": 90.01924789791249, + "ttft_ms": 332.24747232458185 + }, + { + "e2e_ms": 11749.850049120596, + "request_id": 40, + "tpot_ms": 90.01924789791249, + "ttft_ms": 317.4055660857107 + }, + { + "e2e_ms": 11735.008142881725, + "request_id": 41, + "tpot_ms": 90.01924789791249, + "ttft_ms": 302.5636598468395 + }, + { + "e2e_ms": 11720.166236642854, + "request_id": 42, + "tpot_ms": 90.01924789791249, + "ttft_ms": 287.72175360796837 + }, + { + "e2e_ms": 11702.95123939303, + "request_id": 43, + "tpot_ms": 90.00056214192074, + "ttft_ms": 272.8798473690972 + }, + { + "e2e_ms": 11776.016602552545, + "request_id": 44, + "tpot_ms": 89.9995908060081, + "ttft_ms": 346.06857018951587 + }, + { + "e2e_ms": 11761.174696313674, + "request_id": 45, + "tpot_ms": 90.01924789791249, + "ttft_ms": 328.7302132787886 + }, + { + "e2e_ms": 11746.332790074803, + "request_id": 46, + "tpot_ms": 90.01924789791249, + "ttft_ms": 313.88830703991744 + }, + { + "e2e_ms": 11731.490883835932, + "request_id": 47, + "tpot_ms": 90.01924789791249, + "ttft_ms": 299.0464008010463 + }, + { + "e2e_ms": 11716.64897759706, + "request_id": 48, + "tpot_ms": 90.01924789791249, + "ttft_ms": 284.2044945621751 + }, + { + "e2e_ms": 11699.433980347238, + "request_id": 49, + "tpot_ms": 90.00056214192074, + "ttft_ms": 269.36258832330395 + }, + { + "e2e_ms": 11772.499343506752, + "request_id": 50, + "tpot_ms": 89.9995908060081, + "ttft_ms": 342.5513111437226 + }, + { + "e2e_ms": 11757.657437270042, + "request_id": 51, + "tpot_ms": 90.0192478979295, + "ttft_ms": 325.21295423299534 + }, + { + "e2e_ms": 11742.815531033444, + "request_id": 52, + "tpot_ms": 90.01924789792636, + "ttft_ms": 310.3710479967958 + }, + { + "e2e_ms": 11727.973624796847, + "request_id": 53, + "tpot_ms": 90.01924789792636, + "ttft_ms": 295.5291417601984 + }, + { + "e2e_ms": 11710.758627549352, + "request_id": 54, + "tpot_ms": 90.00056214193505, + "ttft_ms": 280.687235523601 + }, + { + "e2e_ms": 11783.823990711198, + "request_id": 55, + "tpot_ms": 89.99959080602287, + "ttft_ms": 353.8759583462934 + }, + { + "e2e_ms": 11768.982084474601, + "request_id": 56, + "tpot_ms": 90.01924789792636, + "ttft_ms": 336.5376014379535 + }, + { + "e2e_ms": 11754.140178238004, + "request_id": 57, + "tpot_ms": 90.01924789792636, + "ttft_ms": 321.6956952013561 + }, + { + "e2e_ms": 11739.298272001406, + "request_id": 58, + "tpot_ms": 90.01924789792636, + "ttft_ms": 306.8537889647587 + }, + { + "e2e_ms": 11724.456365764809, + "request_id": 59, + "tpot_ms": 90.01924789792636, + "ttft_ms": 292.01188272816125 + }, + { + "e2e_ms": 11707.241368517316, + "request_id": 60, + "tpot_ms": 90.00056214193505, + "ttft_ms": 277.16997649156383 + }, + { + "e2e_ms": 11780.30673167916, + "request_id": 61, + "tpot_ms": 89.99959080602287, + "ttft_ms": 350.3586993142562 + }, + { + "e2e_ms": 11765.464825442563, + "request_id": 62, + "tpot_ms": 90.01924789792635, + "ttft_ms": 333.0203424059164 + }, + { + "e2e_ms": 11530.943387303068, + "request_id": 63, + "tpot_ms": 88.28948780420275, + "ttft_ms": 318.17843616931896 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 28.99924373626709, + "offered_request_rate": 0.2, + "request_metrics_sha256": "7930ffb53e440b9eefa724912c777842a9c33cefdb46d9014e1c5003223221d2", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 11975.67900227787, + "request_id": 0, + "tpot_ms": 92.18976799112939, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12131.199338988026, + "request_id": 1, + "tpot_ms": 93.2890801728389, + "ttft_ms": 283.4861570374869 + }, + { + "e2e_ms": 12196.411822684866, + "request_id": 2, + "tpot_ms": 93.31228634935276, + "ttft_ms": 345.75145631706584 + }, + { + "e2e_ms": 12171.220586311463, + "request_id": 3, + "tpot_ms": 93.31547315047263, + "ttft_ms": 320.15549620144054 + }, + { + "e2e_ms": 12145.933482881133, + "request_id": 4, + "tpot_ms": 93.31640326087744, + "ttft_ms": 294.75026874969853 + }, + { + "e2e_ms": 12120.646379450847, + "request_id": 5, + "tpot_ms": 93.31564840216181, + "ttft_ms": 269.5590323762964 + }, + { + "e2e_ms": 12185.858863147729, + "request_id": 6, + "tpot_ms": 93.31471829175732, + "ttft_ms": 334.88964009454844 + }, + { + "e2e_ms": 12160.571759717443, + "request_id": 7, + "tpot_ms": 93.31471829175764, + "ttft_ms": 309.6025366642223 + }, + { + "e2e_ms": 12135.284656287155, + "request_id": 8, + "tpot_ms": 93.31564840216245, + "ttft_ms": 284.19730921252295 + }, + { + "e2e_ms": 12200.497139984038, + "request_id": 9, + "tpot_ms": 93.31471829175764, + "ttft_ms": 349.52791693081764 + }, + { + "e2e_ms": 12175.305903610677, + "request_id": 10, + "tpot_ms": 93.3154731504736, + "ttft_ms": 324.2408135005306 + }, + { + "e2e_ms": 12150.018800180305, + "request_id": 11, + "tpot_ms": 93.31640326087775, + "ttft_ms": 298.8355860488312 + }, + { + "e2e_ms": 12124.731696749934, + "request_id": 12, + "tpot_ms": 93.31564840216113, + "ttft_ms": 273.6443496754717 + }, + { + "e2e_ms": 12189.94418044673, + "request_id": 13, + "tpot_ms": 93.31471829175568, + "ttft_ms": 338.97495739375927 + }, + { + "e2e_ms": 12164.657077016358, + "request_id": 14, + "tpot_ms": 93.31471829175568, + "ttft_ms": 313.68785396338694 + }, + { + "e2e_ms": 12139.369973585985, + "request_id": 15, + "tpot_ms": 93.31564840216049, + "ttft_ms": 288.2826265116023 + }, + { + "e2e_ms": 12204.582457282782, + "request_id": 16, + "tpot_ms": 93.31471829175568, + "ttft_ms": 353.6132342298117 + }, + { + "e2e_ms": 12179.391220909338, + "request_id": 17, + "tpot_ms": 93.31547315047163, + "ttft_ms": 328.3261307994394 + }, + { + "e2e_ms": 12154.104117478966, + "request_id": 18, + "tpot_ms": 93.31640326087647, + "ttft_ms": 302.92090334765476 + }, + { + "e2e_ms": 12128.817014048593, + "request_id": 19, + "tpot_ms": 93.31564840216049, + "ttft_ms": 277.72966697421 + }, + { + "e2e_ms": 12194.02949774539, + "request_id": 20, + "tpot_ms": 93.31471829175568, + "ttft_ms": 343.0602746924194 + }, + { + "e2e_ms": 12168.742394315017, + "request_id": 21, + "tpot_ms": 93.31471829175567, + "ttft_ms": 317.7731712620471 + }, + { + "e2e_ms": 12143.455290884645, + "request_id": 22, + "tpot_ms": 93.31564840216049, + "ttft_ms": 292.36794381026243 + }, + { + "e2e_ms": 12208.667774581443, + "request_id": 23, + "tpot_ms": 93.31471829175568, + "ttft_ms": 357.69855152847185 + }, + { + "e2e_ms": 12183.476538207999, + "request_id": 24, + "tpot_ms": 93.31547315047165, + "ttft_ms": 332.4114480980995 + }, + { + "e2e_ms": 12158.189434777625, + "request_id": 25, + "tpot_ms": 93.31640326087646, + "ttft_ms": 307.0062206463149 + }, + { + "e2e_ms": 12132.902331347253, + "request_id": 26, + "tpot_ms": 93.31564840216049, + "ttft_ms": 281.8149842728701 + }, + { + "e2e_ms": 12198.11481504405, + "request_id": 27, + "tpot_ms": 93.31471829175567, + "ttft_ms": 347.1455919910795 + }, + { + "e2e_ms": 12172.923578670605, + "request_id": 28, + "tpot_ms": 93.31547315047163, + "ttft_ms": 321.8584885607072 + }, + { + "e2e_ms": 12147.636475240233, + "request_id": 29, + "tpot_ms": 93.31640326087646, + "ttft_ms": 296.45326110892256 + }, + { + "e2e_ms": 12122.349371809862, + "request_id": 30, + "tpot_ms": 93.3156484021605, + "ttft_ms": 271.2620247354778 + }, + { + "e2e_ms": 12187.561855506658, + "request_id": 31, + "tpot_ms": 93.31471829175568, + "ttft_ms": 336.5926324536872 + }, + { + "e2e_ms": 12162.274752076286, + "request_id": 32, + "tpot_ms": 93.31471829175568, + "ttft_ms": 311.3055290233149 + }, + { + "e2e_ms": 12136.987648645914, + "request_id": 33, + "tpot_ms": 93.3156484021605, + "ttft_ms": 285.90030157153024 + }, + { + "e2e_ms": 12202.20013234271, + "request_id": 34, + "tpot_ms": 93.31471829175568, + "ttft_ms": 351.23090928973966 + }, + { + "e2e_ms": 12177.008895969266, + "request_id": 35, + "tpot_ms": 93.31547315047163, + "ttft_ms": 325.94380585936733 + }, + { + "e2e_ms": 12151.721792538894, + "request_id": 36, + "tpot_ms": 93.31640326087647, + "ttft_ms": 300.5385784075827 + }, + { + "e2e_ms": 12126.43468910852, + "request_id": 37, + "tpot_ms": 93.31564840216049, + "ttft_ms": 275.3473420341379 + }, + { + "e2e_ms": 12191.647172805318, + "request_id": 38, + "tpot_ms": 93.31471829175568, + "ttft_ms": 340.67794975234733 + }, + { + "e2e_ms": 12166.360069374947, + "request_id": 39, + "tpot_ms": 93.3147182917557, + "ttft_ms": 315.390846321975 + }, + { + "e2e_ms": 12141.072965944573, + "request_id": 40, + "tpot_ms": 93.31564840216049, + "ttft_ms": 289.98561887019036 + }, + { + "e2e_ms": 12206.285449641371, + "request_id": 41, + "tpot_ms": 93.31471829175568, + "ttft_ms": 355.3162265883998 + }, + { + "e2e_ms": 12181.094213267927, + "request_id": 42, + "tpot_ms": 93.31547315047165, + "ttft_ms": 330.02912315802746 + }, + { + "e2e_ms": 12155.807109837553, + "request_id": 43, + "tpot_ms": 93.31640326087646, + "ttft_ms": 304.6238957062428 + }, + { + "e2e_ms": 12130.520006407181, + "request_id": 44, + "tpot_ms": 93.31564840216049, + "ttft_ms": 279.43265933279804 + }, + { + "e2e_ms": 12195.73249010398, + "request_id": 45, + "tpot_ms": 93.3147182917557, + "ttft_ms": 344.76326705100746 + }, + { + "e2e_ms": 12170.445386673606, + "request_id": 46, + "tpot_ms": 93.31471829175568, + "ttft_ms": 319.47616362063513 + }, + { + "e2e_ms": 12145.158283243234, + "request_id": 47, + "tpot_ms": 93.31564840216049, + "ttft_ms": 294.0709361688505 + }, + { + "e2e_ms": 12210.37076694003, + "request_id": 48, + "tpot_ms": 93.31471829175568, + "ttft_ms": 359.4015438870599 + }, + { + "e2e_ms": 12185.179530566586, + "request_id": 49, + "tpot_ms": 93.31547315047163, + "ttft_ms": 334.1144404566876 + }, + { + "e2e_ms": 12159.892427136241, + "request_id": 50, + "tpot_ms": 93.31640326087668, + "ttft_ms": 308.70921300490295 + }, + { + "e2e_ms": 12134.605323705899, + "request_id": 51, + "tpot_ms": 93.31564840216095, + "ttft_ms": 283.51797663145817 + }, + { + "e2e_ms": 12199.817807402724, + "request_id": 52, + "tpot_ms": 93.31471829175612, + "ttft_ms": 348.848584349696 + }, + { + "e2e_ms": 12174.626571029308, + "request_id": 53, + "tpot_ms": 93.3154731504721, + "ttft_ms": 323.5614809193521 + }, + { + "e2e_ms": 12149.339467598964, + "request_id": 54, + "tpot_ms": 93.31640326087691, + "ttft_ms": 298.1562534675959 + }, + { + "e2e_ms": 12124.05236416862, + "request_id": 55, + "tpot_ms": 93.31564840216095, + "ttft_ms": 272.96501709417953 + }, + { + "e2e_ms": 12189.264847865445, + "request_id": 56, + "tpot_ms": 93.31471829175612, + "ttft_ms": 338.2956248124174 + }, + { + "e2e_ms": 12163.977744435102, + "request_id": 57, + "tpot_ms": 93.31471829175614, + "ttft_ms": 313.00852138207347 + }, + { + "e2e_ms": 12138.690641004758, + "request_id": 58, + "tpot_ms": 93.31564840216095, + "ttft_ms": 287.60329393031725 + }, + { + "e2e_ms": 12203.903124701583, + "request_id": 59, + "tpot_ms": 93.31471829175612, + "ttft_ms": 352.9339016485551 + }, + { + "e2e_ms": 12178.711888328166, + "request_id": 60, + "tpot_ms": 93.31547315047209, + "ttft_ms": 327.6467982182112 + }, + { + "e2e_ms": 12153.424784897823, + "request_id": 61, + "tpot_ms": 93.31640326087691, + "ttft_ms": 302.24157076645497 + }, + { + "e2e_ms": 11949.074413391485, + "request_id": 62, + "tpot_ms": 91.90570140943657, + "ttft_ms": 277.0503343930386 + }, + { + "e2e_ms": 11702.728382593137, + "request_id": 63, + "tpot_ms": 89.45155464946347, + "ttft_ms": 342.38094211127645 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 24.38782286643982, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e62567aa33ec429331fcd9f24053a0c79db61ef288f7994e6488ae0f0fa79b7e", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 12394.3405299904, + "request_id": 0, + "tpot_ms": 95.48631545343278, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 12520.094438712134, + "request_id": 1, + "tpot_ms": 96.04733720338449, + "ttft_ms": 322.08261388230494 + }, + { + "e2e_ms": 12556.880443424032, + "request_id": 2, + "tpot_ms": 96.10807445129032, + "ttft_ms": 351.1549881101628 + }, + { + "e2e_ms": 12502.892311945501, + "request_id": 3, + "tpot_ms": 96.14618123456218, + "ttft_ms": 292.32729515610424 + }, + { + "e2e_ms": 12539.549613984442, + "request_id": 4, + "tpot_ms": 96.30977229626299, + "ttft_ms": 308.2085323590427 + }, + { + "e2e_ms": 12576.335618696337, + "request_id": 5, + "tpot_ms": 96.32399459689755, + "ttft_ms": 343.188304890349 + }, + { + "e2e_ms": 12522.347487217807, + "request_id": 6, + "tpot_ms": 96.32399459689753, + "ttft_ms": 289.200173411821 + }, + { + "e2e_ms": 12559.133491929704, + "request_id": 7, + "tpot_ms": 96.32399459689753, + "ttft_ms": 325.98617812371697 + }, + { + "e2e_ms": 12505.145360451173, + "request_id": 8, + "tpot_ms": 96.32399459689753, + "ttft_ms": 271.99804664518723 + }, + { + "e2e_ms": 12541.802662490163, + "request_id": 9, + "tpot_ms": 96.32399459689792, + "ttft_ms": 308.6553486841268 + }, + { + "e2e_ms": 12578.588667202104, + "request_id": 10, + "tpot_ms": 96.32399459689827, + "ttft_ms": 345.4413533960228 + }, + { + "e2e_ms": 12524.60053572362, + "request_id": 11, + "tpot_ms": 96.32399459689864, + "ttft_ms": 291.453221917493 + }, + { + "e2e_ms": 12561.386540435564, + "request_id": 12, + "tpot_ms": 96.32399459689903, + "ttft_ms": 328.239226629389 + }, + { + "e2e_ms": 12507.39840895708, + "request_id": 13, + "tpot_ms": 96.32399459689938, + "ttft_ms": 274.2510951508592 + }, + { + "e2e_ms": 12544.05571099607, + "request_id": 14, + "tpot_ms": 96.32399459689938, + "ttft_ms": 310.90839718984853 + }, + { + "e2e_ms": 12580.841715708011, + "request_id": 15, + "tpot_ms": 96.32399459689938, + "ttft_ms": 347.6944019017907 + }, + { + "e2e_ms": 12526.853584229528, + "request_id": 16, + "tpot_ms": 96.32399459689938, + "ttft_ms": 293.7062704233071 + }, + { + "e2e_ms": 12563.63958894147, + "request_id": 17, + "tpot_ms": 96.32399459689938, + "ttft_ms": 330.49227513524926 + }, + { + "e2e_ms": 12509.651457462987, + "request_id": 18, + "tpot_ms": 96.32399459689938, + "ttft_ms": 276.5041436567657 + }, + { + "e2e_ms": 12546.308759501975, + "request_id": 19, + "tpot_ms": 96.32399459689938, + "ttft_ms": 313.161445695755 + }, + { + "e2e_ms": 12583.094764213918, + "request_id": 20, + "tpot_ms": 96.32399459689938, + "ttft_ms": 349.94745040769715 + }, + { + "e2e_ms": 12529.106632735506, + "request_id": 21, + "tpot_ms": 96.32399459689994, + "ttft_ms": 295.95931892921357 + }, + { + "e2e_ms": 12565.892637447447, + "request_id": 22, + "tpot_ms": 96.32399459689992, + "ttft_ms": 332.7453236411557 + }, + { + "e2e_ms": 12511.904505968956, + "request_id": 23, + "tpot_ms": 96.32399459689988, + "ttft_ms": 278.75719216267214 + }, + { + "e2e_ms": 12548.561808007931, + "request_id": 24, + "tpot_ms": 96.32399459689977, + "ttft_ms": 315.41449420166146 + }, + { + "e2e_ms": 12585.347812719874, + "request_id": 25, + "tpot_ms": 96.32399459689977, + "ttft_ms": 352.2004989136036 + }, + { + "e2e_ms": 12531.359681241383, + "request_id": 26, + "tpot_ms": 96.3239945968991, + "ttft_ms": 298.2123674351982 + }, + { + "e2e_ms": 12568.145685953326, + "request_id": 27, + "tpot_ms": 96.32399459689911, + "ttft_ms": 334.99837214714034 + }, + { + "e2e_ms": 12514.157554474836, + "request_id": 28, + "tpot_ms": 96.32399459689911, + "ttft_ms": 281.01024066864966 + }, + { + "e2e_ms": 12550.81485651381, + "request_id": 29, + "tpot_ms": 96.32399459689911, + "ttft_ms": 317.66754270762476 + }, + { + "e2e_ms": 12587.600861225752, + "request_id": 30, + "tpot_ms": 96.3239945968991, + "ttft_ms": 354.4535474195669 + }, + { + "e2e_ms": 12533.612729747261, + "request_id": 31, + "tpot_ms": 96.3239945968991, + "ttft_ms": 300.46541594107623 + }, + { + "e2e_ms": 12570.398734459204, + "request_id": 32, + "tpot_ms": 96.32399459689911, + "ttft_ms": 337.2514206530184 + }, + { + "e2e_ms": 12516.410602980714, + "request_id": 33, + "tpot_ms": 96.32399459689911, + "ttft_ms": 283.2632891745277 + }, + { + "e2e_ms": 12553.067905019689, + "request_id": 34, + "tpot_ms": 96.32399459689911, + "ttft_ms": 319.9205912135028 + }, + { + "e2e_ms": 12589.85390973163, + "request_id": 35, + "tpot_ms": 96.3239945968991, + "ttft_ms": 356.70659592544496 + }, + { + "e2e_ms": 12535.865778253139, + "request_id": 36, + "tpot_ms": 96.3239945968991, + "ttft_ms": 302.7184644469543 + }, + { + "e2e_ms": 12572.651782965082, + "request_id": 37, + "tpot_ms": 96.3239945968991, + "ttft_ms": 339.5044691588964 + }, + { + "e2e_ms": 12518.663651486591, + "request_id": 38, + "tpot_ms": 96.3239945968991, + "ttft_ms": 285.51633768040574 + }, + { + "e2e_ms": 12555.320953525566, + "request_id": 39, + "tpot_ms": 96.3239945968991, + "ttft_ms": 322.17363971938084 + }, + { + "e2e_ms": 12770.519034100516, + "request_id": 40, + "tpot_ms": 97.72881409188341, + "ttft_ms": 358.959644431323 + }, + { + "e2e_ms": 12538.070601439798, + "request_id": 41, + "tpot_ms": 96.32361486997611, + "ttft_ms": 304.9715129528323 + }, + { + "e2e_ms": 12574.856606151741, + "request_id": 42, + "tpot_ms": 96.32361486997611, + "ttft_ms": 341.75751766477447 + }, + { + "e2e_ms": 12520.86847467325, + "request_id": 43, + "tpot_ms": 96.32361486997611, + "ttft_ms": 287.7693861862838 + }, + { + "e2e_ms": 12557.654479385192, + "request_id": 44, + "tpot_ms": 96.32462827684986, + "ttft_ms": 324.4266882252589 + }, + { + "e2e_ms": 12503.666347906701, + "request_id": 45, + "tpot_ms": 96.3239945968991, + "ttft_ms": 270.51903410051636 + }, + { + "e2e_ms": 12540.323649945676, + "request_id": 46, + "tpot_ms": 96.3239945968991, + "ttft_ms": 307.17633613949147 + }, + { + "e2e_ms": 12577.109654657448, + "request_id": 47, + "tpot_ms": 96.32399459689775, + "ttft_ms": 343.9623408514336 + }, + { + "e2e_ms": 12523.121523178786, + "request_id": 48, + "tpot_ms": 96.3239945968964, + "ttft_ms": 289.97420937294294 + }, + { + "e2e_ms": 12559.907527890544, + "request_id": 49, + "tpot_ms": 96.32399459689495, + "ttft_ms": 326.7602140848851 + }, + { + "e2e_ms": 12505.919396411884, + "request_id": 50, + "tpot_ms": 96.32399459689363, + "ttft_ms": 272.7720826063944 + }, + { + "e2e_ms": 12542.576698450688, + "request_id": 51, + "tpot_ms": 96.32399459689228, + "ttft_ms": 309.4293846453695 + }, + { + "e2e_ms": 12579.362703162446, + "request_id": 52, + "tpot_ms": 96.32399459689228, + "ttft_ms": 346.2153893571269 + }, + { + "e2e_ms": 12525.374571683784, + "request_id": 53, + "tpot_ms": 96.32399459689228, + "ttft_ms": 292.2272578784657 + }, + { + "e2e_ms": 12562.160576395541, + "request_id": 54, + "tpot_ms": 96.32399459689228, + "ttft_ms": 329.0132625902231 + }, + { + "e2e_ms": 12508.17244491688, + "request_id": 55, + "tpot_ms": 96.32399459689226, + "ttft_ms": 275.0251311115619 + }, + { + "e2e_ms": 12544.829746955686, + "request_id": 56, + "tpot_ms": 96.32399459689228, + "ttft_ms": 311.6824331503665 + }, + { + "e2e_ms": 12581.615751667443, + "request_id": 57, + "tpot_ms": 96.32399459689228, + "ttft_ms": 348.4684378621239 + }, + { + "e2e_ms": 12527.627620188781, + "request_id": 58, + "tpot_ms": 96.32399459689228, + "ttft_ms": 294.4803063834627 + }, + { + "e2e_ms": 12564.413624900539, + "request_id": 59, + "tpot_ms": 96.32399459689228, + "ttft_ms": 331.2663110952201 + }, + { + "e2e_ms": 12314.305135768365, + "request_id": 60, + "tpot_ms": 94.77973981221895, + "ttft_ms": 277.2781796165589 + }, + { + "e2e_ms": 12170.36562563149, + "request_id": 61, + "tpot_ms": 93.35771766910335, + "ttft_ms": 313.93548165536345 + }, + { + "e2e_ms": 12023.933572014499, + "request_id": 62, + "tpot_ms": 91.91505579249903, + "ttft_ms": 350.72148636712086 + }, + { + "e2e_ms": 11724.699298497455, + "request_id": 63, + "tpot_ms": 89.98398380794484, + "ttft_ms": 296.73335488845964 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 22.886351823806763, + "offered_request_rate": 0.8, + "request_metrics_sha256": "b39b2c6a1c6236e30b246303e0b4d66b80a2515899d8c815efc015ef3f70db54", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 14028.106647728775, + "request_id": 0, + "tpot_ms": 108.35061559310502, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 14267.352384844213, + "request_id": 1, + "tpot_ms": 109.65331923259451, + "ttft_ms": 341.3808423047111 + }, + { + "e2e_ms": 14406.16288598222, + "request_id": 2, + "tpot_ms": 110.64064370419715, + "ttft_ms": 354.8011355491818 + }, + { + "e2e_ms": 14444.508775243683, + "request_id": 3, + "tpot_ms": 111.53949237897281, + "ttft_ms": 278.99324311413756 + }, + { + "e2e_ms": 14585.312621821135, + "request_id": 4, + "tpot_ms": 112.58765687327464, + "ttft_ms": 286.6801989152554 + }, + { + "e2e_ms": 14727.157204966976, + "request_id": 5, + "tpot_ms": 113.57630070139629, + "ttft_ms": 302.9670158896467 + }, + { + "e2e_ms": 14871.045643448613, + "request_id": 6, + "tpot_ms": 114.43037546464966, + "ttft_ms": 338.38795943810675 + }, + { + "e2e_ms": 14913.487485283544, + "request_id": 7, + "tpot_ms": 115.12950641852738, + "ttft_ms": 292.0401701305657 + }, + { + "e2e_ms": 15059.457396901958, + "request_id": 8, + "tpot_ms": 115.80285438725795, + "ttft_ms": 352.4948897201998 + }, + { + "e2e_ms": 15271.481983406971, + "request_id": 9, + "tpot_ms": 117.58757541921905, + "ttft_ms": 337.85990516615175 + }, + { + "e2e_ms": 15316.138289012584, + "request_id": 10, + "tpot_ms": 117.88145073243837, + "ttft_ms": 345.1940459929119 + }, + { + "e2e_ms": 15090.835365768904, + "request_id": 11, + "tpot_ms": 116.63565919716638, + "ttft_ms": 278.1066477287748 + }, + { + "e2e_ms": 15302.859952273917, + "request_id": 12, + "tpot_ms": 118.0348305674148, + "ttft_ms": 312.43647021223705 + }, + { + "e2e_ms": 15347.54563377864, + "request_id": 13, + "tpot_ms": 118.10044582475892, + "ttft_ms": 348.7890140342564 + }, + { + "e2e_ms": 15289.743982068245, + "request_id": 14, + "tpot_ms": 118.15013414242924, + "ttft_ms": 284.6769459797329 + }, + { + "e2e_ms": 15334.400287673823, + "request_id": 15, + "tpot_ms": 118.19982246009941, + "ttft_ms": 323.022835241197 + }, + { + "e2e_ms": 15380.492763872971, + "request_id": 16, + "tpot_ms": 118.2528557378034, + "ttft_ms": 362.38008517193873 + }, + { + "e2e_ms": 15322.691112162565, + "request_id": 17, + "tpot_ms": 118.27848723653601, + "ttft_ms": 301.32323312249196 + }, + { + "e2e_ms": 15367.376793667254, + "request_id": 18, + "tpot_ms": 118.29615526543176, + "ttft_ms": 343.765074957421 + }, + { + "e2e_ms": 15309.575141956848, + "request_id": 19, + "tpot_ms": 118.30562851819849, + "ttft_ms": 284.7603201456401 + }, + { + "e2e_ms": 15354.260823461536, + "request_id": 20, + "tpot_ms": 118.30690699483614, + "ttft_ms": 329.2836351173456 + }, + { + "e2e_ms": 15296.45917175113, + "request_id": 21, + "tpot_ms": 118.3069069948359, + "ttft_ms": 271.4819834069715 + }, + { + "e2e_ms": 15341.115477356709, + "request_id": 22, + "tpot_ms": 118.30690699483563, + "ttft_ms": 316.138289012585 + }, + { + "e2e_ms": 15387.207953555859, + "request_id": 23, + "tpot_ms": 118.31926259505173, + "ttft_ms": 360.6616039842905 + }, + { + "e2e_ms": 15329.406301845453, + "request_id": 24, + "tpot_ms": 118.31926259505147, + "ttft_ms": 302.8599522739164 + }, + { + "e2e_ms": 15375.4987780446, + "request_id": 25, + "tpot_ms": 118.33033971862962, + "ttft_ms": 347.5456337786404 + }, + { + "e2e_ms": 15317.697126334195, + "request_id": 26, + "tpot_ms": 118.33033971862953, + "ttft_ms": 289.74398206824503 + }, + { + "e2e_ms": 15362.382807838883, + "request_id": 27, + "tpot_ms": 118.33057102492174, + "ttft_ms": 334.400287673823 + }, + { + "e2e_ms": 15304.581156128477, + "request_id": 28, + "tpot_ms": 118.33057102492174, + "ttft_ms": 276.59863596341694 + }, + { + "e2e_ms": 15349.237461734056, + "request_id": 29, + "tpot_ms": 118.3192625950511, + "ttft_ms": 322.6911121625662 + }, + { + "e2e_ms": 15395.329937933206, + "request_id": 30, + "tpot_ms": 118.33033971862955, + "ttft_ms": 367.37679366725473 + }, + { + "e2e_ms": 15337.5282862228, + "request_id": 31, + "tpot_ms": 118.33033971862955, + "ttft_ms": 309.57514195684865 + }, + { + "e2e_ms": 15383.620762421948, + "request_id": 32, + "tpot_ms": 118.34141684220796, + "ttft_ms": 354.2608234615372 + }, + { + "e2e_ms": 15325.819110711542, + "request_id": 33, + "tpot_ms": 118.34141684220796, + "ttft_ms": 296.4591717511311 + }, + { + "e2e_ms": 15370.50479221623, + "request_id": 34, + "tpot_ms": 118.34164814850017, + "ttft_ms": 341.1154773567091 + }, + { + "e2e_ms": 15312.703140505824, + "request_id": 35, + "tpot_ms": 118.34164814850017, + "ttft_ms": 283.31382564630303 + }, + { + "e2e_ms": 15357.388822010513, + "request_id": 36, + "tpot_ms": 118.33057102492174, + "ttft_ms": 329.4063018454523 + }, + { + "e2e_ms": 15299.587170300107, + "request_id": 37, + "tpot_ms": 118.33057102492174, + "ttft_ms": 271.6046501350462 + }, + { + "e2e_ms": 15344.243475905685, + "request_id": 38, + "tpot_ms": 118.3192625950511, + "ttft_ms": 317.69712633419545 + }, + { + "e2e_ms": 15390.335952104835, + "request_id": 39, + "tpot_ms": 118.33033971862955, + "ttft_ms": 362.382807838884 + }, + { + "e2e_ms": 15332.53430039447, + "request_id": 40, + "tpot_ms": 118.33033971862987, + "ttft_ms": 304.5811561284779 + }, + { + "e2e_ms": 15378.626776593663, + "request_id": 41, + "tpot_ms": 118.34164814850084, + "ttft_ms": 349.2374617340559 + }, + { + "e2e_ms": 15320.8251248833, + "request_id": 42, + "tpot_ms": 118.34164814850119, + "ttft_ms": 291.4358100236498 + }, + { + "e2e_ms": 15365.51080638803, + "request_id": 43, + "tpot_ms": 118.33057102492307, + "ttft_ms": 337.52828622279907 + }, + { + "e2e_ms": 15307.709154677666, + "request_id": 44, + "tpot_ms": 118.33057102492342, + "ttft_ms": 279.726634512393 + }, + { + "e2e_ms": 15352.365460283294, + "request_id": 45, + "tpot_ms": 118.31926259505316, + "ttft_ms": 325.81911071154224 + }, + { + "e2e_ms": 15398.457936482486, + "request_id": 46, + "tpot_ms": 118.33033971863193, + "ttft_ms": 370.5047922162308 + }, + { + "e2e_ms": 15340.656284772123, + "request_id": 47, + "tpot_ms": 118.33033971863227, + "ttft_ms": 312.7031405058247 + }, + { + "e2e_ms": 15386.748760971315, + "request_id": 48, + "tpot_ms": 118.34141684221105, + "ttft_ms": 357.38882201051325 + }, + { + "e2e_ms": 15328.947109260951, + "request_id": 49, + "tpot_ms": 118.34141684221137, + "ttft_ms": 299.5871703001072 + }, + { + "e2e_ms": 15375.039585460143, + "request_id": 50, + "tpot_ms": 118.35272527208234, + "ttft_ms": 344.24347590568516 + }, + { + "e2e_ms": 15317.23793374978, + "request_id": 51, + "tpot_ms": 118.3527252720827, + "ttft_ms": 286.4418241952791 + }, + { + "e2e_ms": 15194.4223437212, + "request_id": 52, + "tpot_ms": 117.02274049863567, + "ttft_ms": 332.53430039447096 + }, + { + "e2e_ms": 14959.152693280232, + "request_id": 53, + "tpot_ms": 115.62535468185925, + "ttft_ms": 274.7326486841075 + }, + { + "e2e_ms": 14796.032219342294, + "request_id": 54, + "tpot_ms": 113.97800861778735, + "ttft_ms": 320.8251248832994 + }, + { + "e2e_ms": 14608.764053703837, + "request_id": 55, + "tpot_ms": 112.1516003725654, + "ttft_ms": 365.5108063880306 + }, + { + "e2e_ms": 14305.495223453576, + "request_id": 56, + "tpot_ms": 110.21878794311739, + "ttft_ms": 307.70915467766713 + }, + { + "e2e_ms": 14083.450969580696, + "request_id": 57, + "tpot_ms": 108.11878353777482, + "ttft_ms": 352.36546028329485 + }, + { + "e2e_ms": 13758.528543574443, + "request_id": 58, + "tpot_ms": 106.01547035434261, + "ttft_ms": 294.5638085729314 + }, + { + "e2e_ms": 13507.044041669204, + "request_id": 59, + "tpot_ms": 103.67234454249669, + "ttft_ms": 340.6562847721233 + }, + { + "e2e_ms": 13157.052579489218, + "request_id": 60, + "tpot_ms": 101.37163737344456, + "ttft_ms": 282.85463306175984 + }, + { + "e2e_ms": 12903.847402123616, + "request_id": 61, + "tpot_ms": 99.01496293592649, + "ttft_ms": 328.9471092609517 + }, + { + "e2e_ms": 12559.065843040158, + "request_id": 62, + "tpot_ms": 96.75527862590212, + "ttft_ms": 271.1454575505883 + }, + { + "e2e_ms": 12277.402762692504, + "request_id": 63, + "tpot_ms": 94.17452621214743, + "ttft_ms": 317.23793374978015 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 27.091737508773804, + "offered_request_rate": 1.2, + "request_metrics_sha256": "3825f14be2103486ce094e0b335c86deef683645cae8f05ef3d4beb6fb24cf95", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 15728.871662360909, + "request_id": 0, + "tpot_ms": 121.74246610201946, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 15904.548256233129, + "request_id": 1, + "tpot_ms": 122.72470152955445, + "ttft_ms": 318.5111619797136 + }, + { + "e2e_ms": 16140.50586206045, + "request_id": 2, + "tpot_ms": 124.75532551591714, + "ttft_ms": 296.5795215389744 + }, + { + "e2e_ms": 16211.825024161684, + "request_id": 3, + "tpot_ms": 125.48331552046938, + "ttft_ms": 275.44395306207383 + }, + { + "e2e_ms": 16388.84099440166, + "request_id": 4, + "tpot_ms": 126.36635826198417, + "ttft_ms": 340.31349512967114 + }, + { + "e2e_ms": 16460.2590055204, + "request_id": 5, + "tpot_ms": 127.08834011289514, + "ttft_ms": 320.03981118271605 + }, + { + "e2e_ms": 16531.726441149392, + "request_id": 6, + "tpot_ms": 127.72809453706097, + "ttft_ms": 310.25843494264916 + }, + { + "e2e_ms": 16603.193876778383, + "request_id": 7, + "tpot_ms": 128.3170301134443, + "ttft_ms": 306.93105237095784 + }, + { + "e2e_ms": 16674.731660081485, + "request_id": 8, + "tpot_ms": 128.84921275092722, + "ttft_ms": 310.8816407137276 + }, + { + "e2e_ms": 16746.318867894835, + "request_id": 9, + "tpot_ms": 129.28898520065795, + "ttft_ms": 326.61774741127834 + }, + { + "e2e_ms": 16982.978169021266, + "request_id": 10, + "tpot_ms": 130.92468009104587, + "ttft_ms": 355.5437974584379 + }, + { + "e2e_ms": 16948.91870099869, + "request_id": 11, + "tpot_ms": 131.099598546656, + "ttft_ms": 299.26968557337295 + }, + { + "e2e_ms": 17020.55604111586, + "request_id": 12, + "tpot_ms": 131.2589874619586, + "ttft_ms": 350.6646334471153 + }, + { + "e2e_ms": 16986.496573094286, + "request_id": 13, + "tpot_ms": 131.335760528223, + "ttft_ms": 306.854986009963 + }, + { + "e2e_ms": 17059.03976122884, + "request_id": 14, + "tpot_ms": 131.4398579526774, + "ttft_ms": 366.1778012388126 + }, + { + "e2e_ms": 17024.98029320726, + "request_id": 15, + "tpot_ms": 131.50690676411472, + "ttft_ms": 323.6031341646921 + }, + { + "e2e_ms": 16990.920825185683, + "request_id": 16, + "tpot_ms": 131.56847300071718, + "ttft_ms": 281.7247540946006 + }, + { + "e2e_ms": 17063.464013320234, + "request_id": 17, + "tpot_ms": 131.63886355765484, + "ttft_ms": 345.32834149806746 + }, + { + "e2e_ms": 17029.40454529866, + "request_id": 18, + "tpot_ms": 131.67812927130495, + "ttft_ms": 306.28212784293396 + }, + { + "e2e_ms": 16995.3450772771, + "request_id": 19, + "tpot_ms": 131.6904025418972, + "ttft_ms": 270.66395445615575 + }, + { + "e2e_ms": 17067.888265411675, + "request_id": 20, + "tpot_ms": 131.7034495326512, + "ttft_ms": 341.5501747649721 + }, + { + "e2e_ms": 17033.828797390113, + "request_id": 21, + "tpot_ms": 131.70595487135952, + "ttft_ms": 307.17252872745246 + }, + { + "e2e_ms": 16999.769329368555, + "request_id": 22, + "tpot_ms": 131.7084602100679, + "ttft_ms": 272.79488268993293 + }, + { + "e2e_ms": 17072.312517503127, + "request_id": 23, + "tpot_ms": 131.71809821033827, + "ttft_ms": 344.1140447901674 + }, + { + "e2e_ms": 17038.25304948157, + "request_id": 24, + "tpot_ms": 131.71982521031822, + "ttft_ms": 309.83524777115434 + }, + { + "e2e_ms": 17004.193581460007, + "request_id": 25, + "tpot_ms": 131.72116304093396, + "ttft_ms": 275.60587526139454 + }, + { + "e2e_ms": 17076.736769594583, + "request_id": 26, + "tpot_ms": 131.72963353311178, + "ttft_ms": 347.07331088938884 + }, + { + "e2e_ms": 17042.67730157302, + "request_id": 27, + "tpot_ms": 131.7305821943633, + "ttft_ms": 312.8933628888824 + }, + { + "e2e_ms": 17008.617833551463, + "request_id": 28, + "tpot_ms": 131.7309769369132, + "ttft_ms": 278.783762563485 + }, + { + "e2e_ms": 17081.161021686035, + "request_id": 29, + "tpot_ms": 131.73850434102513, + "ttft_ms": 350.37097037584175 + }, + { + "e2e_ms": 17047.986478516996, + "request_id": 30, + "tpot_ms": 131.74547225324986, + "ttft_ms": 316.3115023542638 + }, + { + "e2e_ms": 17013.927010495438, + "request_id": 31, + "tpot_ms": 131.74547225325, + "ttft_ms": 282.25203433268575 + }, + { + "e2e_ms": 17086.47019863001, + "request_id": 32, + "tpot_ms": 131.75260491481217, + "ttft_ms": 353.8893744488618 + }, + { + "e2e_ms": 17053.29565546097, + "request_id": 33, + "tpot_ms": 131.7595728270369, + "ttft_ms": 319.82990642728385 + }, + { + "e2e_ms": 17019.236187439412, + "request_id": 34, + "tpot_ms": 131.75957282703703, + "ttft_ms": 285.7704384057058 + }, + { + "e2e_ms": 17091.779375573984, + "request_id": 35, + "tpot_ms": 131.7595728270372, + "ttft_ms": 358.31362654026225 + }, + { + "e2e_ms": 17058.604832404944, + "request_id": 36, + "tpot_ms": 131.7665407392619, + "ttft_ms": 324.2541585186842 + }, + { + "e2e_ms": 17024.545364383386, + "request_id": 37, + "tpot_ms": 131.76654073926207, + "ttft_ms": 290.1946904971062 + }, + { + "e2e_ms": 17097.08855251796, + "request_id": 38, + "tpot_ms": 131.76654073926218, + "ttft_ms": 362.7378786316626 + }, + { + "e2e_ms": 17063.934932514778, + "request_id": 39, + "tpot_ms": 131.7736734008242, + "ttft_ms": 328.6784106101024 + }, + { + "e2e_ms": 17029.875464493216, + "request_id": 40, + "tpot_ms": 131.7736734008242, + "ttft_ms": 294.61894258854215 + }, + { + "e2e_ms": 17102.41865262779, + "request_id": 41, + "tpot_ms": 131.7736734008242, + "ttft_ms": 367.1621307231163 + }, + { + "e2e_ms": 17069.265032624608, + "request_id": 42, + "tpot_ms": 131.78080606238623, + "ttft_ms": 333.1026627015561 + }, + { + "e2e_ms": 17035.205564603046, + "request_id": 43, + "tpot_ms": 131.78080606238623, + "ttft_ms": 299.04319467999585 + }, + { + "e2e_ms": 16941.812657737883, + "request_id": 44, + "tpot_ms": 130.47422263719145, + "ttft_ms": 371.58638281457 + }, + { + "e2e_ms": 16742.48916517699, + "request_id": 45, + "tpot_ms": 129.1729311053857, + "ttft_ms": 337.5269147930098 + }, + { + "e2e_ms": 16538.03643831958, + "request_id": 46, + "tpot_ms": 127.83125190195379, + "ttft_ms": 303.46744677144954 + }, + { + "e2e_ms": 16435.957848641778, + "request_id": 47, + "tpot_ms": 126.45627727350987, + "ttft_ms": 376.0106349060237 + }, + { + "e2e_ms": 16228.476901947985, + "request_id": 48, + "tpot_ms": 125.09075381939779, + "ttft_ms": 341.9511668844635 + }, + { + "e2e_ms": 16020.052127617348, + "request_id": 49, + "tpot_ms": 123.7177986516098, + "ttft_ms": 307.89169886290324 + }, + { + "e2e_ms": 15809.427558622445, + "request_id": 50, + "tpot_ms": 122.32055435377619, + "ttft_ms": 274.71715569286914 + }, + { + "e2e_ms": 15703.511289947706, + "request_id": 51, + "tpot_ms": 120.91536178046667, + "ttft_ms": 347.26034382843807 + }, + { + "e2e_ms": 15485.101955905271, + "request_id": 52, + "tpot_ms": 119.46378803227081, + "ttft_ms": 313.20087580687783 + }, + { + "e2e_ms": 15260.624457846197, + "request_id": 53, + "tpot_ms": 117.95746555282956, + "ttft_ms": 280.02633263684373 + }, + { + "e2e_ms": 15118.941732007286, + "request_id": 54, + "tpot_ms": 116.27064733255806, + "ttft_ms": 352.56952077241266 + }, + { + "e2e_ms": 14865.280308326035, + "request_id": 55, + "tpot_ms": 114.54149807539514, + "ttft_ms": 318.5100527508524 + }, + { + "e2e_ms": 14599.985676841925, + "request_id": 56, + "tpot_ms": 112.71378084457564, + "ttft_ms": 285.3355095808183 + }, + { + "e2e_ms": 14420.806000135271, + "request_id": 57, + "tpot_ms": 110.73171104266838, + "ttft_ms": 357.87869771638725 + }, + { + "e2e_ms": 14142.519211198518, + "request_id": 58, + "tpot_ms": 108.80866127168261, + "ttft_ms": 323.819229694827 + }, + { + "e2e_ms": 13853.830695007076, + "request_id": 59, + "tpot_ms": 106.79657547493251, + "ttft_ms": 290.6656096906488 + }, + { + "e2e_ms": 13650.503338148035, + "request_id": 60, + "tpot_ms": 104.62436645922692, + "ttft_ms": 363.20879782621773 + }, + { + "e2e_ms": 13360.876271706531, + "request_id": 61, + "tpot_ms": 102.61202316458169, + "ttft_ms": 329.1493298046575 + }, + { + "e2e_ms": 13070.674002922458, + "request_id": 62, + "tpot_ms": 100.58801805607857, + "ttft_ms": 295.9957098004793 + }, + { + "e2e_ms": 12853.555073004585, + "request_id": 63, + "tpot_ms": 98.30721397691761, + "ttft_ms": 368.5388979360482 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 27.491333484649658, + "offered_request_rate": 1.6, + "request_metrics_sha256": "b7d73a1cbef8ca659f753135a4944cb3023d3a2424a19a741ae2198f39c2d172", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 17513.570713343794, + "request_id": 0, + "tpot_ms": 135.79521453495556, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 17840.282304762335, + "request_id": 1, + "tpot_ms": 137.71064841138914, + "ttft_ms": 351.0299565159147 + }, + { + "e2e_ms": 17895.63618825912, + "request_id": 2, + "tpot_ms": 138.10245652748367, + "ttft_ms": 356.6242092686922 + }, + { + "e2e_ms": 17848.77057711112, + "request_id": 3, + "tpot_ms": 138.39719968518088, + "ttft_ms": 272.32621709314975 + }, + { + "e2e_ms": 17904.1244606079, + "request_id": 4, + "tpot_ms": 138.80479077491768, + "ttft_ms": 275.9160321933569 + }, + { + "e2e_ms": 17959.478344104686, + "request_id": 5, + "tpot_ms": 139.18735062645055, + "ttft_ms": 282.6848145454632 + }, + { + "e2e_ms": 18014.921946878916, + "request_id": 6, + "tpot_ms": 139.5152186600215, + "ttft_ms": 296.48917705618504 + }, + { + "e2e_ms": 18070.36554965315, + "request_id": 7, + "tpot_ms": 139.8086106441042, + "ttft_ms": 314.671997851919 + }, + { + "e2e_ms": 18125.898871704834, + "request_id": 8, + "tpot_ms": 140.0646556798527, + "ttft_ms": 337.68760036353916 + }, + { + "e2e_ms": 18079.30241838919, + "request_id": 9, + "tpot_ms": 140.21470125635153, + "ttft_ms": 272.0353588325457 + }, + { + "e2e_ms": 18134.83860521682, + "request_id": 10, + "tpot_ms": 140.33922237580643, + "ttft_ms": 311.75736348940484 + }, + { + "e2e_ms": 18190.4645113219, + "request_id": 11, + "tpot_ms": 140.37869703173442, + "ttft_ms": 362.36998829163 + }, + { + "e2e_ms": 18143.870922782207, + "request_id": 12, + "tpot_ms": 140.38698051363622, + "ttft_ms": 314.7243975504068 + }, + { + "e2e_ms": 18097.274469466563, + "request_id": 13, + "tpot_ms": 140.36074521133403, + "ttft_ms": 271.4598276271438 + }, + { + "e2e_ms": 18152.810656294194, + "request_id": 14, + "tpot_ms": 140.32455966134754, + "ttft_ms": 331.5915793030584 + }, + { + "e2e_ms": 18106.217067754496, + "request_id": 15, + "tpot_ms": 140.29231877139563, + "ttft_ms": 289.0925837872533 + }, + { + "e2e_ms": 18161.753254582127, + "request_id": 16, + "tpot_ms": 140.24009333090902, + "ttft_ms": 351.2614015566804 + }, + { + "e2e_ms": 18115.15966604243, + "request_id": 17, + "tpot_ms": 140.19912163029534, + "ttft_ms": 309.87121899492286 + }, + { + "e2e_ms": 18170.69585287006, + "request_id": 18, + "tpot_ms": 140.12856604535176, + "ttft_ms": 374.3679651103839 + }, + { + "e2e_ms": 18124.102264330366, + "request_id": 19, + "tpot_ms": 140.05569589908265, + "ttft_ms": 337.0288851468697 + }, + { + "e2e_ms": 18077.42933293013, + "request_id": 20, + "tpot_ms": 139.98073600581233, + "ttft_ms": 299.87586019196044 + }, + { + "e2e_ms": 18132.96551975776, + "request_id": 21, + "tpot_ms": 139.8495015438229, + "ttft_ms": 372.0788236922541 + }, + { + "e2e_ms": 18086.369066442116, + "request_id": 22, + "tpot_ms": 139.76199603449132, + "ttft_ms": 336.5955700617178 + }, + { + "e2e_ms": 18039.682893849018, + "request_id": 23, + "tpot_ms": 139.67091992877667, + "ttft_ms": 301.4760628943822 + }, + { + "e2e_ms": 18095.21908067664, + "request_id": 24, + "tpot_ms": 139.54200325642663, + "ttft_ms": 373.38466711045726 + }, + { + "e2e_ms": 18048.532908083544, + "request_id": 25, + "tpot_ms": 139.5517742330953, + "ttft_ms": 325.4575804804407 + }, + { + "e2e_ms": 18001.846735490446, + "request_id": 26, + "tpot_ms": 139.5610841122856, + "ttft_ms": 277.589053230173 + }, + { + "e2e_ms": 18057.290338264673, + "request_id": 27, + "tpot_ms": 139.56778701592805, + "ttft_ms": 332.18138724181046 + }, + { + "e2e_ms": 18010.60416567157, + "request_id": 28, + "tpot_ms": 139.57154350494935, + "ttft_ms": 285.01814054300564 + }, + { + "e2e_ms": 18066.06100963866, + "request_id": 29, + "tpot_ms": 139.5730606683175, + "ttft_ms": 340.2823047623365 + }, + { + "e2e_ms": 18019.37483704556, + "request_id": 30, + "tpot_ms": 139.57447357032456, + "ttft_ms": 293.41669361433986 + }, + { + "e2e_ms": 18074.831681012645, + "request_id": 31, + "tpot_ms": 139.57528428268915, + "ttft_ms": 348.7705771111216 + }, + { + "e2e_ms": 18028.145508419548, + "request_id": 32, + "tpot_ms": 139.57669718469626, + "ttft_ms": 301.904965963125 + }, + { + "e2e_ms": 18083.678830471228, + "request_id": 33, + "tpot_ms": 139.5781100867033, + "ttft_ms": 357.25884945990674 + }, + { + "e2e_ms": 18036.99265787813, + "request_id": 34, + "tpot_ms": 139.57881653770684, + "ttft_ms": 310.48295758936104 + }, + { + "e2e_ms": 18092.528844705754, + "request_id": 35, + "tpot_ms": 139.57954554600127, + "ttft_ms": 365.9265603635937 + }, + { + "e2e_ms": 17876.05367572695, + "request_id": 36, + "tpot_ms": 138.24262431461773, + "ttft_ms": 319.240387770499 + }, + { + "e2e_ms": 17659.21004164318, + "request_id": 37, + "tpot_ms": 136.90209533219152, + "ttft_ms": 272.64393445485524 + }, + { + "e2e_ms": 17543.75228976072, + "request_id": 38, + "tpot_ms": 135.55570892326125, + "ttft_ms": 328.1772565065388 + }, + { + "e2e_ms": 17326.367174433202, + "request_id": 39, + "tpot_ms": 134.21089375170362, + "ttft_ms": 281.5836679668422 + }, + { + "e2e_ms": 17233.832476815856, + "request_id": 40, + "tpot_ms": 133.04427482475538, + "ttft_ms": 337.20957407192384 + }, + { + "e2e_ms": 17034.265611203216, + "request_id": 41, + "tpot_ms": 131.83976083205505, + "ttft_ms": 290.6159855322272 + }, + { + "e2e_ms": 16940.502121615402, + "request_id": 42, + "tpot_ms": 130.66419538607474, + "ttft_ms": 346.14930758391085 + }, + { + "e2e_ms": 16741.97064984836, + "request_id": 43, + "tpot_ms": 129.46783410082003, + "ttft_ms": 299.5557190442142 + }, + { + "e2e_ms": 16645.559306371404, + "request_id": 44, + "tpot_ms": 128.27139685432724, + "ttft_ms": 355.0919058718449 + }, + { + "e2e_ms": 16443.232533552808, + "request_id": 45, + "tpot_ms": 127.04515130882409, + "ttft_ms": 308.4983173321483 + }, + { + "e2e_ms": 16342.299705615802, + "request_id": 46, + "tpot_ms": 125.81311182248837, + "ttft_ms": 364.034504159779 + }, + { + "e2e_ms": 16135.159559133626, + "request_id": 47, + "tpot_ms": 124.54896569695704, + "ttft_ms": 317.44091562008236 + }, + { + "e2e_ms": 16031.7032146663, + "request_id": 48, + "tpot_ms": 123.29695174036006, + "ttft_ms": 372.9903436405735 + }, + { + "e2e_ms": 15823.308920667878, + "request_id": 49, + "tpot_ms": 122.0236594458299, + "ttft_ms": 326.3041710474788 + }, + { + "e2e_ms": 15613.44809689394, + "request_id": 50, + "tpot_ms": 120.73809066445794, + "ttft_ms": 279.71058250778214 + }, + { + "e2e_ms": 15508.031714506984, + "request_id": 51, + "tpot_ms": 119.47077015706711, + "ttft_ms": 335.2439045594622 + }, + { + "e2e_ms": 15292.981047367362, + "request_id": 52, + "tpot_ms": 118.14506547559841, + "ttft_ms": 288.5577319663639 + }, + { + "e2e_ms": 15175.360910096466, + "request_id": 53, + "tpot_ms": 116.78162985277542, + "ttft_ms": 344.09391879398754 + }, + { + "e2e_ms": 14945.589828664517, + "request_id": 54, + "tpot_ms": 115.34001639735139, + "ttft_ms": 297.4077462008893 + }, + { + "e2e_ms": 14803.64975337431, + "request_id": 55, + "tpot_ms": 113.78581420786767, + "ttft_ms": 352.85134897511483 + }, + { + "e2e_ms": 14557.342221274233, + "request_id": 56, + "tpot_ms": 112.21399247946627, + "ttft_ms": 306.16517638201657 + }, + { + "e2e_ms": 14399.594833150197, + "request_id": 57, + "tpot_ms": 110.5352189984338, + "ttft_ms": 361.6220203491025 + }, + { + "e2e_ms": 14144.625862747695, + "request_id": 58, + "tpot_ms": 108.89519696843851, + "ttft_ms": 314.93584775600425 + }, + { + "e2e_ms": 13973.496543699825, + "request_id": 59, + "tpot_ms": 107.1110539525727, + "ttft_ms": 370.3926917230902 + }, + { + "e2e_ms": 13708.4999588278, + "request_id": 60, + "tpot_ms": 105.3920743283292, + "ttft_ms": 323.70651912999193 + }, + { + "e2e_ms": 13445.970803422128, + "request_id": 61, + "tpot_ms": 103.6925232825609, + "ttft_ms": 277.02034653689367 + }, + { + "e2e_ms": 13273.580023880399, + "request_id": 62, + "tpot_ms": 101.89784531725846, + "ttft_ms": 332.5536685885737 + }, + { + "e2e_ms": 13000.702540117643, + "request_id": 63, + "tpot_ms": 100.1168113710407, + "ttft_ms": 285.86749599547545 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 35.55747365951538, + "offered_request_rate": 2.4, + "request_metrics_sha256": "3b42135fbabbd54647284962ccc9fdf0b900fbc0b6ed0bdecf03477d5c409d18", + "request_rate_per_gpu": 0.6, + "requests": [ + { + "e2e_ms": 21942.173036302996, + "request_id": 0, + "tpot_ms": 170.66609896770518, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 22008.341136862466, + "request_id": 1, + "tpot_ms": 170.96613901739173, + "ttft_ms": 295.64148165371654 + }, + { + "e2e_ms": 22074.54102422534, + "request_id": 2, + "tpot_ms": 171.22660942224667, + "ttft_ms": 328.7616276000109 + }, + { + "e2e_ms": 22036.227907899276, + "request_id": 3, + "tpot_ms": 171.37344968894675, + "ttft_ms": 271.7997974030406 + }, + { + "e2e_ms": 22102.42779526115, + "request_id": 4, + "tpot_ms": 171.64131409153669, + "ttft_ms": 303.98090563598765 + }, + { + "e2e_ms": 22168.65946942643, + "request_id": 5, + "tpot_ms": 171.89659303259273, + "ttft_ms": 337.79215428715537 + }, + { + "e2e_ms": 22130.37813990278, + "request_id": 6, + "tpot_ms": 172.0281077294899, + "ttft_ms": 282.80845825756097 + }, + { + "e2e_ms": 22365.99994575454, + "request_id": 7, + "tpot_ms": 173.57085404199856, + "ttft_ms": 322.5014824207202 + }, + { + "e2e_ms": 22158.3592864644, + "request_id": 8, + "tpot_ms": 172.3496956452424, + "ttft_ms": 269.9479395186155 + }, + { + "e2e_ms": 22393.981092316164, + "request_id": 9, + "tpot_ms": 173.8419366523574, + "ttft_ms": 316.0551374667744 + }, + { + "e2e_ms": 22460.243568400445, + "request_id": 10, + "tpot_ms": 173.96590922572528, + "ttft_ms": 366.57309673333316 + }, + { + "e2e_ms": 22252.602909110316, + "request_id": 11, + "tpot_ms": 172.6883751510306, + "ttft_ms": 321.1792649294285 + }, + { + "e2e_ms": 22044.894176229613, + "request_id": 12, + "tpot_ms": 171.40112664047544, + "ttft_ms": 276.9510928892318 + }, + { + "e2e_ms": 21941.653543442477, + "request_id": 13, + "tpot_ms": 170.1065182287643, + "ttft_ms": 338.1257283894126 + }, + { + "e2e_ms": 21734.138079618475, + "request_id": 14, + "tpot_ms": 168.80971544527205, + "ttft_ms": 295.30421806892537 + }, + { + "e2e_ms": 21631.119179231846, + "request_id": 15, + "tpot_ms": 167.51267286797687, + "ttft_ms": 357.0097249987798 + }, + { + "e2e_ms": 21420.354658102242, + "request_id": 16, + "tpot_ms": 166.18662353135022, + "ttft_ms": 314.65346962076256 + }, + { + "e2e_ms": 21209.648295563282, + "request_id": 17, + "tpot_ms": 164.85776701341018, + "ttft_ms": 272.7118848601879 + }, + { + "e2e_ms": 21101.69867612793, + "request_id": 18, + "tpot_ms": 163.50974377121332, + "ttft_ms": 335.9612171838391 + }, + { + "e2e_ms": 20890.778369055995, + "request_id": 19, + "tpot_ms": 162.16628083276893, + "ttft_ms": 295.66070329433813 + }, + { + "e2e_ms": 20782.558056179907, + "request_id": 20, + "tpot_ms": 160.7985342727971, + "ttft_ms": 361.14420353467835 + }, + { + "e2e_ms": 20571.512469390585, + "request_id": 21, + "tpot_ms": 159.44598375979533, + "ttft_ms": 321.8725318965792 + }, + { + "e2e_ms": 20360.405278761264, + "request_id": 22, + "tpot_ms": 158.09565605925098, + "ttft_ms": 282.2569592363866 + }, + { + "e2e_ms": 20251.586130359967, + "request_id": 23, + "tpot_ms": 156.71227679254048, + "ttft_ms": 349.12697770732314 + }, + { + "e2e_ms": 20042.10448060375, + "request_id": 24, + "tpot_ms": 155.37142770924103, + "ttft_ms": 309.93316153013774 + }, + { + "e2e_ms": 19934.008109113514, + "request_id": 25, + "tpot_ms": 154.05960281316982, + "ttft_ms": 368.43855184094565 + }, + { + "e2e_ms": 19722.984661067087, + "request_id": 26, + "tpot_ms": 152.73931080974432, + "ttft_ms": 325.0921882295579 + }, + { + "e2e_ms": 19511.768043076878, + "request_id": 27, + "tpot_ms": 151.41621155599998, + "ttft_ms": 281.9091754648788 + }, + { + "e2e_ms": 19403.203575383635, + "request_id": 28, + "tpot_ms": 150.09414530000853, + "ttft_ms": 341.2471222825513 + }, + { + "e2e_ms": 19191.714653347688, + "request_id": 29, + "tpot_ms": 148.76487274343572, + "ttft_ms": 298.57581493135046 + }, + { + "e2e_ms": 19082.708845359608, + "request_id": 30, + "tpot_ms": 147.43434256455518, + "ttft_ms": 358.5473396611025 + }, + { + "e2e_ms": 18871.078709760895, + "request_id": 31, + "tpot_ms": 146.10277278695904, + "ttft_ms": 316.0265658170971 + }, + { + "e2e_ms": 18659.370447496967, + "request_id": 32, + "tpot_ms": 144.7695321688903, + "ttft_ms": 273.6398620479008 + }, + { + "e2e_ms": 18549.637648096734, + "request_id": 33, + "tpot_ms": 143.43132538741386, + "ttft_ms": 333.85932389517234 + }, + { + "e2e_ms": 18337.808235788358, + "request_id": 34, + "tpot_ms": 142.0958670025312, + "ttft_ms": 291.63312646689656 + }, + { + "e2e_ms": 18227.80864738452, + "request_id": 35, + "tpot_ms": 140.75352098043697, + "ttft_ms": 352.1114828690255 + }, + { + "e2e_ms": 18015.40726100407, + "request_id": 36, + "tpot_ms": 139.41149279948888, + "ttft_ms": 310.14767546898446 + }, + { + "e2e_ms": 17904.857868774245, + "request_id": 37, + "tpot_ms": 138.06211837292886, + "ttft_ms": 370.9688354122758 + }, + { + "e2e_ms": 17692.008101355255, + "request_id": 38, + "tpot_ms": 136.71510258553306, + "ttft_ms": 329.1900729925583 + }, + { + "e2e_ms": 17479.148877024498, + "request_id": 39, + "tpot_ms": 135.36749298254742, + "ttft_ms": 287.47726824097344 + }, + { + "e2e_ms": 17381.961391787096, + "request_id": 40, + "tpot_ms": 134.1089349239123, + "ttft_ms": 350.1266564502323 + }, + { + "e2e_ms": 17178.01129231477, + "request_id": 41, + "tpot_ms": 132.8327622110564, + "ttft_ms": 308.2504915106092 + }, + { + "e2e_ms": 17080.086531895085, + "request_id": 42, + "tpot_ms": 131.5805170059729, + "ttft_ms": 369.36087213652866 + }, + { + "e2e_ms": 16876.654129344566, + "request_id": 43, + "tpot_ms": 130.30640250229436, + "ttft_ms": 327.74101155318294 + }, + { + "e2e_ms": 16671.422925287778, + "request_id": 44, + "tpot_ms": 129.0178430005707, + "ttft_ms": 286.15686421529676 + }, + { + "e2e_ms": 16571.761179006833, + "request_id": 45, + "tpot_ms": 127.74972161297816, + "ttft_ms": 347.5465341586066 + }, + { + "e2e_ms": 16364.721381165033, + "request_id": 46, + "tpot_ms": 126.44572566423672, + "ttft_ms": 306.1142218069683 + }, + { + "e2e_ms": 16261.449604637399, + "request_id": 47, + "tpot_ms": 125.14700916940275, + "ttft_ms": 367.77944012325037 + }, + { + "e2e_ms": 16053.400400183471, + "request_id": 48, + "tpot_ms": 123.8340136000885, + "ttft_ms": 326.48067297223093 + }, + { + "e2e_ms": 15845.03658651726, + "request_id": 49, + "tpot_ms": 122.50141376236263, + "ttft_ms": 287.35703869720555 + }, + { + "e2e_ms": 15739.724302020804, + "request_id": 50, + "tpot_ms": 121.15600683324593, + "ttft_ms": 352.9114341985711 + }, + { + "e2e_ms": 15530.891082399023, + "request_id": 51, + "tpot_ms": 119.81946138926746, + "ttft_ms": 313.8194859620569 + }, + { + "e2e_ms": 15319.199082162215, + "request_id": 52, + "tpot_ms": 118.45427332697808, + "ttft_ms": 275.5063696359983 + }, + { + "e2e_ms": 15206.960333133673, + "request_id": 53, + "tpot_ms": 117.04949498375757, + "ttft_ms": 341.6744701964625 + }, + { + "e2e_ms": 14987.9081257507, + "request_id": 54, + "tpot_ms": 115.62635253449054, + "ttft_ms": 303.36135387040383 + }, + { + "e2e_ms": 14861.077413909577, + "request_id": 55, + "tpot_ms": 114.10642655651417, + "ttft_ms": 369.5612412322795 + }, + { + "e2e_ms": 14633.756981193535, + "request_id": 56, + "tpot_ms": 112.61817997075843, + "ttft_ms": 331.2481249072157 + }, + { + "e2e_ms": 14403.991359276923, + "request_id": 57, + "tpot_ms": 111.11042963695559, + "ttft_ms": 292.9667953835633 + }, + { + "e2e_ms": 14264.847964808048, + "request_id": 58, + "tpot_ms": 109.49330311228506, + "ttft_ms": 359.1984695478452 + }, + { + "e2e_ms": 14029.729570522895, + "request_id": 59, + "tpot_ms": 107.94340496454889, + "ttft_ms": 320.91714002518756 + }, + { + "e2e_ms": 13793.064611419886, + "request_id": 60, + "tpot_ms": 106.38108660628622, + "ttft_ms": 282.66661242153646 + }, + { + "e2e_ms": 13648.251078198633, + "request_id": 61, + "tpot_ms": 104.71931331978594, + "ttft_ms": 348.89828658581837 + }, + { + "e2e_ms": 13412.62809971594, + "request_id": 62, + "tpot_ms": 103.16519953332896, + "ttft_ms": 310.647758983162 + }, + { + "e2e_ms": 13260.053320226874, + "request_id": 63, + "tpot_ms": 101.44207153668842, + "ttft_ms": 376.9102350674451 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "elapsed_seconds": 33.80778503417969, + "offered_request_rate": 3.2, + "request_metrics_sha256": "bfba1ae71cfb6d0595614b14bc49ac47591bfad6792be0a3d98d2c98b16012e8", + "request_rate_per_gpu": 0.8, + "requests": [ + { + "e2e_ms": 24003.81066522886, + "request_id": 0, + "tpot_ms": 186.89946612460176, + "ttft_ms": 267.578467404437 + }, + { + "e2e_ms": 23901.058156177503, + "request_id": 1, + "tpot_ms": 185.74139588389912, + "ttft_ms": 311.9008789223171 + }, + { + "e2e_ms": 23798.190109844723, + "request_id": 2, + "tpot_ms": 184.56225305811301, + "ttft_ms": 358.7839714643697 + }, + { + "e2e_ms": 23590.44639829378, + "request_id": 3, + "tpot_ms": 183.2673714648551, + "ttft_ms": 315.49022225718517 + }, + { + "e2e_ms": 23382.70131446869, + "request_id": 4, + "tpot_ms": 181.9732705315891, + "ttft_ms": 272.09595695687216 + }, + { + "e2e_ms": 23279.64954297851, + "request_id": 5, + "tpot_ms": 180.78914255292864, + "ttft_ms": 319.4284387565705 + }, + { + "e2e_ms": 23071.973865900134, + "request_id": 6, + "tpot_ms": 179.49398625037213, + "ttft_ms": 276.2376121028707 + }, + { + "e2e_ms": 22969.022726489202, + "request_id": 7, + "tpot_ms": 178.2861180199918, + "ttft_ms": 326.6857379502426 + }, + { + "e2e_ms": 22761.29942646063, + "request_id": 8, + "tpot_ms": 176.98924979786668, + "ttft_ms": 283.66470213156344 + }, + { + "e2e_ms": 22658.264587165988, + "request_id": 9, + "tpot_ms": 175.7551174352214, + "ttft_ms": 337.36467289287384 + }, + { + "e2e_ms": 22450.304224673095, + "request_id": 10, + "tpot_ms": 174.4550575162989, + "ttft_ms": 294.51192010313275 + }, + { + "e2e_ms": 22346.821408551063, + "request_id": 11, + "tpot_ms": 173.1774539652428, + "ttft_ms": 353.28475496522805 + }, + { + "e2e_ms": 22138.79812545183, + "request_id": 12, + "tpot_ms": 171.87427427697472, + "ttft_ms": 310.7652922760442 + }, + { + "e2e_ms": 22035.248814705075, + "request_id": 13, + "tpot_ms": 170.57508935568035, + "ttft_ms": 372.21246653366967 + }, + { + "e2e_ms": 21827.324416126074, + "request_id": 14, + "tpot_ms": 169.27188182433355, + "ttft_ms": 329.79542443571484 + }, + { + "e2e_ms": 21619.373671552865, + "request_id": 15, + "tpot_ms": 167.96821838814947, + "ttft_ms": 287.40993625788303 + }, + { + "e2e_ms": 21512.77581709026, + "request_id": 16, + "tpot_ms": 166.64216905152279, + "ttft_ms": 349.2203475468658 + }, + { + "e2e_ms": 21303.255969153783, + "request_id": 17, + "tpot_ms": 165.32482604305434, + "ttft_ms": 307.003061685883 + }, + { + "e2e_ms": 21196.567333974876, + "request_id": 18, + "tpot_ms": 163.99164288866731, + "ttft_ms": 369.62868711412875 + }, + { + "e2e_ms": 20986.94051377241, + "request_id": 19, + "tpot_ms": 162.67002407488405, + "ttft_ms": 327.84745626213316 + }, + { + "e2e_ms": 20777.255965035714, + "request_id": 20, + "tpot_ms": 161.34765321373814, + "ttft_ms": 286.10400689097213 + }, + { + "e2e_ms": 20670.377044913395, + "request_id": 21, + "tpot_ms": 159.9951027007364, + "ttft_ms": 350.99900191987297 + }, + { + "e2e_ms": 20460.656782932234, + "request_id": 22, + "tpot_ms": 158.67196467076286, + "ttft_ms": 309.3172697453532 + }, + { + "e2e_ms": 20353.38846177537, + "request_id": 23, + "tpot_ms": 157.31793948495124, + "ttft_ms": 374.01014718656 + }, + { + "e2e_ms": 20144.48097023076, + "request_id": 24, + "tpot_ms": 155.99981541047092, + "ttft_ms": 332.5044131009545 + }, + { + "e2e_ms": 19934.83773528968, + "request_id": 25, + "tpot_ms": 154.6729368346084, + "ttft_ms": 291.37475729441144 + }, + { + "e2e_ms": 19827.980953909257, + "request_id": 26, + "tpot_ms": 153.35264483118294, + "ttft_ms": 352.1950603490236 + }, + { + "e2e_ms": 19618.20597824765, + "request_id": 27, + "tpot_ms": 152.02412197401114, + "ttft_ms": 311.1424875482367 + }, + { + "e2e_ms": 19511.107444230154, + "request_id": 28, + "tpot_ms": 150.6975599886974, + "ttft_ms": 372.5173256655836 + }, + { + "e2e_ms": 19301.196316545185, + "request_id": 29, + "tpot_ms": 149.36537894653213, + "ttft_ms": 331.7931903356044 + }, + { + "e2e_ms": 19091.249936104825, + "request_id": 30, + "tpot_ms": 148.03267778881258, + "ttft_ms": 291.0998569256247 + }, + { + "e2e_ms": 18983.78646717311, + "request_id": 31, + "tpot_ms": 146.7011080112165, + "ttft_ms": 352.7457497486193 + }, + { + "e2e_ms": 18773.765669374145, + "request_id": 32, + "tpot_ms": 145.36784483585674, + "ttft_ms": 312.0493752203348 + }, + { + "e2e_ms": 18665.888247551986, + "request_id": 33, + "tpot_ms": 144.03124523287858, + "ttft_ms": 373.9201029764097 + }, + { + "e2e_ms": 18455.8068747313, + "request_id": 34, + "tpot_ms": 142.6968275070543, + "ttft_ms": 333.30978133540157 + }, + { + "e2e_ms": 18245.529234152025, + "request_id": 35, + "tpot_ms": 141.3602386069844, + "ttft_ms": 292.7789310650048 + }, + { + "e2e_ms": 18137.294514438578, + "request_id": 36, + "tpot_ms": 140.01821042603632, + "ttft_ms": 354.98179033196385 + }, + { + "e2e_ms": 17926.83360591763, + "request_id": 37, + "tpot_ms": 138.67967407469433, + "ttft_ms": 314.5149984314486 + }, + { + "e2e_ms": 17716.242055541137, + "request_id": 38, + "tpot_ms": 137.3398665125213, + "ttft_ms": 274.0790084509328 + }, + { + "e2e_ms": 17607.549497877386, + "request_id": 39, + "tpot_ms": 135.99225690953574, + "ttft_ms": 336.53287036634794 + }, + { + "e2e_ms": 17401.54255835392, + "request_id": 40, + "tpot_ms": 134.6880568492608, + "ttft_ms": 296.1593384978016 + }, + { + "e2e_ms": 17301.7591255476, + "request_id": 41, + "tpot_ms": 133.41188413640484, + "ttft_ms": 358.4498402241838 + }, + { + "e2e_ms": 17095.506427630044, + "request_id": 42, + "tpot_ms": 132.10522228512968, + "ttft_ms": 318.1431974185731 + }, + { + "e2e_ms": 16889.623559688287, + "request_id": 43, + "tpot_ms": 130.8012299461049, + "ttft_ms": 277.86735653296194 + }, + { + "e2e_ms": 16788.559022297493, + "request_id": 44, + "tpot_ms": 129.5126704443812, + "ttft_ms": 340.449875861081 + }, + { + "e2e_ms": 16581.727329092846, + "request_id": 45, + "tpot_ms": 128.200721970688, + "ttft_ms": 300.23563881547057 + }, + { + "e2e_ms": 16478.854197918037, + "request_id": 46, + "tpot_ms": 126.8967260219465, + "ttft_ms": 362.9699931308306 + }, + { + "e2e_ms": 16270.819161297495, + "request_id": 47, + "tpot_ms": 125.57481733363996, + "ttft_ms": 322.8173599252209 + }, + { + "e2e_ms": 16062.627892404027, + "request_id": 48, + "tpot_ms": 124.25122687214136, + "ttft_ms": 282.72207964207576 + }, + { + "e2e_ms": 15958.430745404816, + "request_id": 49, + "tpot_ms": 122.91862703441548, + "ttft_ms": 347.7651120340504 + }, + { + "e2e_ms": 15749.715539461335, + "request_id": 50, + "tpot_ms": 121.58574387256766, + "ttft_ms": 308.3260676452415 + }, + { + "e2e_ms": 15645.048986506545, + "request_id": 51, + "tpot_ms": 120.24919842858912, + "ttft_ms": 373.40078607572735 + }, + { + "e2e_ms": 15435.036319721643, + "request_id": 52, + "tpot_ms": 118.8997102377205, + "ttft_ms": 334.77311953113895 + }, + { + "e2e_ms": 15224.012292267464, + "request_id": 53, + "tpot_ms": 117.54201604221191, + "ttft_ms": 296.1762549065519 + }, + { + "e2e_ms": 15109.12675155149, + "request_id": 54, + "tpot_ms": 116.11789378026353, + "ttft_ms": 362.1542414580219 + }, + { + "e2e_ms": 14893.238736493451, + "request_id": 55, + "tpot_ms": 114.72141540015762, + "ttft_ms": 323.6189806734337 + }, + { + "e2e_ms": 14675.411853468433, + "request_id": 56, + "tpot_ms": 113.30942780834319, + "ttft_ms": 285.1145218088469 + }, + { + "e2e_ms": 14549.812898218817, + "request_id": 57, + "tpot_ms": 111.79839094375423, + "ttft_ms": 351.4172483620292 + }, + { + "e2e_ms": 14329.820655618192, + "request_id": 58, + "tpot_ms": 110.36886820693505, + "ttft_ms": 312.97439333744137 + }, + { + "e2e_ms": 14108.094791808619, + "request_id": 59, + "tpot_ms": 108.92545237461232, + "ttft_ms": 274.56234023285475 + }, + { + "e2e_ms": 13975.596499372607, + "request_id": 60, + "tpot_ms": 107.35951607044248, + "ttft_ms": 340.93795842641225 + }, + { + "e2e_ms": 13753.714210521188, + "request_id": 61, + "tpot_ms": 105.91438347527057, + "ttft_ms": 302.58750916182464 + }, + { + "e2e_ms": 13622.257898704496, + "request_id": 62, + "tpot_ms": 104.35568588071192, + "ttft_ms": 369.08579185408286 + }, + { + "e2e_ms": 13397.788527763809, + "request_id": 63, + "tpot_ms": 102.88969749082138, + "ttft_ms": 330.7969464294942 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 778.7386267185211, + "offered_request_rate": 0.1, + "request_metrics_sha256": "1146954f1d73891536473385a3ecb02b14b7eaeee231291f2cbe6f9d4d222414", + "request_rate_per_gpu": 0.0125, + "requests": [ + { + "e2e_ms": 12579.084570749343, + "request_id": 0, + "tpot_ms": 96.75132177962064, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12682.66705927438, + "request_id": 1, + "tpot_ms": 96.90664947750408, + "ttft_ms": 375.5225756313614 + }, + { + "e2e_ms": 12692.014437909165, + "request_id": 2, + "tpot_ms": 96.91832475611541, + "ttft_ms": 383.3871938825091 + }, + { + "e2e_ms": 12605.650216029226, + "request_id": 3, + "tpot_ms": 96.9198422551915, + "ttft_ms": 296.8302496199051 + }, + { + "e2e_ms": 12614.259349375281, + "request_id": 4, + "tpot_ms": 96.9125118011697, + "ttft_ms": 306.3703506267288 + }, + { + "e2e_ms": 12622.868482721336, + "request_id": 5, + "tpot_ms": 96.9125118011697, + "ttft_ms": 314.97948397278464 + }, + { + "e2e_ms": 12631.47761606615, + "request_id": 6, + "tpot_ms": 96.91251180115991, + "ttft_ms": 323.5886173188405 + }, + { + "e2e_ms": 12640.086749411268, + "request_id": 7, + "tpot_ms": 96.91251180113692, + "ttft_ms": 332.19775066687873 + }, + { + "e2e_ms": 12648.695882756385, + "request_id": 8, + "tpot_ms": 96.91251180113692, + "ttft_ms": 340.80688401199666 + }, + { + "e2e_ms": 12657.305016101503, + "request_id": 9, + "tpot_ms": 96.91251180113692, + "ttft_ms": 349.4160173571146 + }, + { + "e2e_ms": 12665.91414944662, + "request_id": 10, + "tpot_ms": 96.9125118011369, + "ttft_ms": 358.0251507022325 + }, + { + "e2e_ms": 12674.523282791739, + "request_id": 11, + "tpot_ms": 96.91251180113692, + "ttft_ms": 366.63428404735043 + }, + { + "e2e_ms": 12683.132416062932, + "request_id": 12, + "tpot_ms": 96.91251180055482, + "ttft_ms": 375.24341739246836 + }, + { + "e2e_ms": 12692.479794575775, + "request_id": 13, + "tpot_ms": 96.91832475507798, + "ttft_ms": 383.85255068087076 + }, + { + "e2e_ms": 12606.11557253071, + "request_id": 14, + "tpot_ms": 96.91984225378269, + "ttft_ms": 297.2956063003096 + }, + { + "e2e_ms": 12614.724705710416, + "request_id": 15, + "tpot_ms": 96.91251179969895, + "ttft_ms": 306.8357071486503 + }, + { + "e2e_ms": 12623.33383889012, + "request_id": 16, + "tpot_ms": 96.91251179969895, + "ttft_ms": 315.44484032835385 + }, + { + "e2e_ms": 12631.942972069823, + "request_id": 17, + "tpot_ms": 96.91251179969893, + "ttft_ms": 324.0539735080574 + }, + { + "e2e_ms": 12640.552105249526, + "request_id": 18, + "tpot_ms": 96.91251179969893, + "ttft_ms": 332.663106687761 + }, + { + "e2e_ms": 12649.16123842923, + "request_id": 19, + "tpot_ms": 96.91251179969893, + "ttft_ms": 341.2722398674646 + }, + { + "e2e_ms": 12657.770371608933, + "request_id": 20, + "tpot_ms": 96.91251179969893, + "ttft_ms": 349.88137304716815 + }, + { + "e2e_ms": 12666.379504788636, + "request_id": 21, + "tpot_ms": 96.91251179969893, + "ttft_ms": 358.49050622687173 + }, + { + "e2e_ms": 12674.988637968341, + "request_id": 22, + "tpot_ms": 96.91251179969895, + "ttft_ms": 367.0996394065753 + }, + { + "e2e_ms": 12683.597771148045, + "request_id": 23, + "tpot_ms": 96.91251179969895, + "ttft_ms": 375.7087725862789 + }, + { + "e2e_ms": 12692.945149660885, + "request_id": 24, + "tpot_ms": 96.91832475507798, + "ttft_ms": 384.31790576598246 + }, + { + "e2e_ms": 12606.580927732637, + "request_id": 25, + "tpot_ms": 96.91984225470249, + "ttft_ms": 297.7609613854213 + }, + { + "e2e_ms": 12615.190061213469, + "request_id": 26, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.3010623465393 + }, + { + "e2e_ms": 12623.7991946943, + "request_id": 27, + "tpot_ms": 96.9125118021018, + "ttft_ms": 315.9101958273709 + }, + { + "e2e_ms": 12632.408328175132, + "request_id": 28, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.5193293082025 + }, + { + "e2e_ms": 12641.017461655963, + "request_id": 29, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.1284627890341 + }, + { + "e2e_ms": 12649.626595136795, + "request_id": 30, + "tpot_ms": 96.9125118021018, + "ttft_ms": 341.73759626986566 + }, + { + "e2e_ms": 12658.235728617627, + "request_id": 31, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.34672975069725 + }, + { + "e2e_ms": 12666.844862098458, + "request_id": 32, + "tpot_ms": 96.9125118021018, + "ttft_ms": 358.95586323152884 + }, + { + "e2e_ms": 12675.45399557929, + "request_id": 33, + "tpot_ms": 96.9125118021018, + "ttft_ms": 367.5649967123604 + }, + { + "e2e_ms": 12684.063129060121, + "request_id": 34, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.174130193192 + }, + { + "e2e_ms": 12693.41050787034, + "request_id": 35, + "tpot_ms": 96.9183247574513, + "ttft_ms": 384.7832636740236 + }, + { + "e2e_ms": 12607.04628612126, + "request_id": 36, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.22631959211776 + }, + { + "e2e_ms": 12615.655419602092, + "request_id": 37, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.76642073516314 + }, + { + "e2e_ms": 12624.264553082923, + "request_id": 38, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.3755542159947 + }, + { + "e2e_ms": 12632.873686563755, + "request_id": 39, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.9846876968263 + }, + { + "e2e_ms": 12641.482820044586, + "request_id": 40, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.5938211776579 + }, + { + "e2e_ms": 12650.091953525418, + "request_id": 41, + "tpot_ms": 96.9125118021018, + "ttft_ms": 342.2029546584895 + }, + { + "e2e_ms": 12658.70108700625, + "request_id": 42, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.8120881393211 + }, + { + "e2e_ms": 12667.310220487081, + "request_id": 43, + "tpot_ms": 96.9125118021018, + "ttft_ms": 359.42122162015266 + }, + { + "e2e_ms": 12675.919353967913, + "request_id": 44, + "tpot_ms": 96.9125118021018, + "ttft_ms": 368.03035510098425 + }, + { + "e2e_ms": 12684.528487448744, + "request_id": 45, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.63948858181584 + }, + { + "e2e_ms": 12693.875866258964, + "request_id": 46, + "tpot_ms": 96.91832475745132, + "ttft_ms": 385.2486220626474 + }, + { + "e2e_ms": 12607.511644509885, + "request_id": 47, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.6916779807416 + }, + { + "e2e_ms": 12616.120777990716, + "request_id": 48, + "tpot_ms": 96.9125118021018, + "ttft_ms": 308.23177912378696 + }, + { + "e2e_ms": 12624.729911471548, + "request_id": 49, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.84091260461855 + }, + { + "e2e_ms": 12633.339044941862, + "request_id": 50, + "tpot_ms": 96.91251180201898, + "ttft_ms": 325.45004608545014 + }, + { + "e2e_ms": 12641.94817818975, + "request_id": 51, + "tpot_ms": 96.91251180018479, + "ttft_ms": 334.0591795662817 + }, + { + "e2e_ms": 12650.557311437638, + "request_id": 52, + "tpot_ms": 96.91251179996682, + "ttft_ms": 342.66831284185173 + }, + { + "e2e_ms": 12659.166444685525, + "request_id": 53, + "tpot_ms": 96.91251179996682, + "ttft_ms": 351.277446089739 + }, + { + "e2e_ms": 12667.775577933411, + "request_id": 54, + "tpot_ms": 96.91251179996681, + "ttft_ms": 359.88657933762624 + }, + { + "e2e_ms": 12676.3847111813, + "request_id": 55, + "tpot_ms": 96.91251179996682, + "ttft_ms": 368.4957125855135 + }, + { + "e2e_ms": 12684.993844429187, + "request_id": 56, + "tpot_ms": 96.91251179996682, + "ttft_ms": 377.10484583340076 + }, + { + "e2e_ms": 12694.3412230047, + "request_id": 57, + "tpot_ms": 96.91832475530246, + "ttft_ms": 385.713979081288 + }, + { + "e2e_ms": 12607.977001027848, + "request_id": 58, + "tpot_ms": 96.9198422540282, + "ttft_ms": 299.1570347662673 + }, + { + "e2e_ms": 12616.586134275734, + "request_id": 59, + "tpot_ms": 96.91251179996681, + "ttft_ms": 308.6971356799495 + }, + { + "e2e_ms": 12625.195267523623, + "request_id": 60, + "tpot_ms": 96.91251179996682, + "ttft_ms": 317.30626892783675 + }, + { + "e2e_ms": 12633.80440077151, + "request_id": 61, + "tpot_ms": 96.91251179996682, + "ttft_ms": 325.915402175724 + }, + { + "e2e_ms": 12642.413534019397, + "request_id": 62, + "tpot_ms": 96.91251179996682, + "ttft_ms": 334.52453542361127 + }, + { + "e2e_ms": 12436.81062947212, + "request_id": 63, + "tpot_ms": 95.22580284094978, + "ttft_ms": 343.1336686714985 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 408.0712172985077, + "offered_request_rate": 0.2, + "request_metrics_sha256": "cf8a095cdf380d42316d1ad169ce9b2b45ae7d64348cd38271a332663de349ad", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 12823.625057543633, + "request_id": 0, + "tpot_ms": 98.67683742367018, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12920.533566322107, + "request_id": 1, + "tpot_ms": 99.10315669196875, + "ttft_ms": 334.4326664420745 + }, + { + "e2e_ms": 12921.724204826443, + "request_id": 2, + "tpot_ms": 99.22365133274592, + "ttft_ms": 320.32048556771286 + }, + { + "e2e_ms": 12922.91484333078, + "request_id": 3, + "tpot_ms": 99.22887081733072, + "ttft_ms": 320.84824952977976 + }, + { + "e2e_ms": 12924.105481838353, + "request_id": 4, + "tpot_ms": 99.22887081736695, + "ttft_ms": 322.0388880327505 + }, + { + "e2e_ms": 12925.296120352506, + "request_id": 5, + "tpot_ms": 99.22887081744425, + "ttft_ms": 323.2295265370873 + }, + { + "e2e_ms": 12926.486758866658, + "request_id": 6, + "tpot_ms": 99.22887081752154, + "ttft_ms": 324.420165041424 + }, + { + "e2e_ms": 12927.677397380812, + "request_id": 7, + "tpot_ms": 99.22887081749418, + "ttft_ms": 325.6108035590515 + }, + { + "e2e_ms": 12928.868035894964, + "request_id": 8, + "tpot_ms": 99.22887081749417, + "ttft_ms": 326.8014420732044 + }, + { + "e2e_ms": 12930.058674409118, + "request_id": 9, + "tpot_ms": 99.22887081749418, + "ttft_ms": 327.9920805873573 + }, + { + "e2e_ms": 12931.24931292327, + "request_id": 10, + "tpot_ms": 99.22887081749417, + "ttft_ms": 329.1827191015102 + }, + { + "e2e_ms": 12932.439951441467, + "request_id": 11, + "tpot_ms": 99.22887081752602, + "ttft_ms": 330.3733576156631 + }, + { + "e2e_ms": 12933.63058995827, + "request_id": 12, + "tpot_ms": 99.22887081754688, + "ttft_ms": 331.563996129816 + }, + { + "e2e_ms": 12934.821228475073, + "request_id": 13, + "tpot_ms": 99.22887081758537, + "ttft_ms": 332.7546346417307 + }, + { + "e2e_ms": 12936.011866991876, + "request_id": 14, + "tpot_ms": 99.22887081758537, + "ttft_ms": 333.94527315853395 + }, + { + "e2e_ms": 12937.20250550868, + "request_id": 15, + "tpot_ms": 99.22887081758537, + "ttft_ms": 335.1359116753372 + }, + { + "e2e_ms": 12938.393144025482, + "request_id": 16, + "tpot_ms": 99.22887081758537, + "ttft_ms": 336.3265501921404 + }, + { + "e2e_ms": 12939.583782542286, + "request_id": 17, + "tpot_ms": 99.22887081758537, + "ttft_ms": 337.51718870894365 + }, + { + "e2e_ms": 12940.77442105909, + "request_id": 18, + "tpot_ms": 99.22887081758537, + "ttft_ms": 338.7078272257469 + }, + { + "e2e_ms": 12941.965059575892, + "request_id": 19, + "tpot_ms": 99.22887081758537, + "ttft_ms": 339.8984657425501 + }, + { + "e2e_ms": 12943.155698092696, + "request_id": 20, + "tpot_ms": 99.22887081758537, + "ttft_ms": 341.08910425935335 + }, + { + "e2e_ms": 12944.346336609498, + "request_id": 21, + "tpot_ms": 99.22887081758537, + "ttft_ms": 342.2797427761566 + }, + { + "e2e_ms": 12945.536975126302, + "request_id": 22, + "tpot_ms": 99.22887081758537, + "ttft_ms": 343.4703812929598 + }, + { + "e2e_ms": 12946.727613643105, + "request_id": 23, + "tpot_ms": 99.22887081758537, + "ttft_ms": 344.66101980976305 + }, + { + "e2e_ms": 12947.918252142103, + "request_id": 24, + "tpot_ms": 99.22887081744517, + "ttft_ms": 345.8516583265663 + }, + { + "e2e_ms": 12949.10889064036, + "request_id": 25, + "tpot_ms": 99.22887081729912, + "ttft_ms": 347.0422968433695 + }, + { + "e2e_ms": 12950.299529138618, + "request_id": 26, + "tpot_ms": 99.2288708172622, + "ttft_ms": 348.23293534631716 + }, + { + "e2e_ms": 12951.490167636877, + "request_id": 27, + "tpot_ms": 99.22887081726222, + "ttft_ms": 349.42357384457523 + }, + { + "e2e_ms": 12952.680806135135, + "request_id": 28, + "tpot_ms": 99.22887081726222, + "ttft_ms": 350.6142123428333 + }, + { + "e2e_ms": 12953.871444633392, + "request_id": 29, + "tpot_ms": 99.2288708172622, + "ttft_ms": 351.80485084109137 + }, + { + "e2e_ms": 12955.06208313165, + "request_id": 30, + "tpot_ms": 99.2288708172622, + "ttft_ms": 352.99548933934943 + }, + { + "e2e_ms": 12956.25272162991, + "request_id": 31, + "tpot_ms": 99.22887081726222, + "ttft_ms": 354.1861278376075 + }, + { + "e2e_ms": 12957.443360128167, + "request_id": 32, + "tpot_ms": 99.22887081726222, + "ttft_ms": 355.37676633586557 + }, + { + "e2e_ms": 12958.633998626425, + "request_id": 33, + "tpot_ms": 99.2288708172622, + "ttft_ms": 356.56740483412364 + }, + { + "e2e_ms": 12959.824637124682, + "request_id": 34, + "tpot_ms": 99.2288708172622, + "ttft_ms": 357.7580433323817 + }, + { + "e2e_ms": 12961.015275622942, + "request_id": 35, + "tpot_ms": 99.22887081726222, + "ttft_ms": 358.94868183063977 + }, + { + "e2e_ms": 12962.2059141212, + "request_id": 36, + "tpot_ms": 99.22887081726222, + "ttft_ms": 360.13932032889784 + }, + { + "e2e_ms": 12963.396552619457, + "request_id": 37, + "tpot_ms": 99.2288708172622, + "ttft_ms": 361.3299588271559 + }, + { + "e2e_ms": 12964.587191117715, + "request_id": 38, + "tpot_ms": 99.2288708172622, + "ttft_ms": 362.520597325414 + }, + { + "e2e_ms": 12965.777829615974, + "request_id": 39, + "tpot_ms": 99.22887081726222, + "ttft_ms": 363.71123582367204 + }, + { + "e2e_ms": 12966.968468114232, + "request_id": 40, + "tpot_ms": 99.22887081726222, + "ttft_ms": 364.9018743219301 + }, + { + "e2e_ms": 12968.15910661249, + "request_id": 41, + "tpot_ms": 99.2288708172622, + "ttft_ms": 366.0925128201882 + }, + { + "e2e_ms": 12969.349745110747, + "request_id": 42, + "tpot_ms": 99.2288708172622, + "ttft_ms": 367.28315131844624 + }, + { + "e2e_ms": 12970.540383609006, + "request_id": 43, + "tpot_ms": 99.22887081726222, + "ttft_ms": 368.4737898167043 + }, + { + "e2e_ms": 12971.731022107264, + "request_id": 44, + "tpot_ms": 99.22887081726222, + "ttft_ms": 369.6644283149624 + }, + { + "e2e_ms": 12972.921660605522, + "request_id": 45, + "tpot_ms": 99.2288708172622, + "ttft_ms": 370.85506681322045 + }, + { + "e2e_ms": 12974.11229910378, + "request_id": 46, + "tpot_ms": 99.2288708172622, + "ttft_ms": 372.0457053114785 + }, + { + "e2e_ms": 12975.302937602039, + "request_id": 47, + "tpot_ms": 99.22887081726222, + "ttft_ms": 373.2363438097366 + }, + { + "e2e_ms": 12976.493576100296, + "request_id": 48, + "tpot_ms": 99.22887081726222, + "ttft_ms": 374.42698230799465 + }, + { + "e2e_ms": 12977.684214615067, + "request_id": 49, + "tpot_ms": 99.22887081739223, + "ttft_ms": 375.6176208062527 + }, + { + "e2e_ms": 12978.874853139188, + "request_id": 50, + "tpot_ms": 99.22887081759589, + "ttft_ms": 376.8082593045108 + }, + { + "e2e_ms": 12980.06549166331, + "request_id": 51, + "tpot_ms": 99.22887081779955, + "ttft_ms": 377.99889780276885 + }, + { + "e2e_ms": 12981.256130187432, + "request_id": 52, + "tpot_ms": 99.22887081784184, + "ttft_ms": 379.189536321519 + }, + { + "e2e_ms": 12982.446768711554, + "request_id": 53, + "tpot_ms": 99.22887081784184, + "ttft_ms": 380.3801748456408 + }, + { + "e2e_ms": 12983.637407235676, + "request_id": 54, + "tpot_ms": 99.22887081784184, + "ttft_ms": 381.5708133697626 + }, + { + "e2e_ms": 12984.828045759798, + "request_id": 55, + "tpot_ms": 99.22887081784184, + "ttft_ms": 382.76145189388444 + }, + { + "e2e_ms": 12986.01868428392, + "request_id": 56, + "tpot_ms": 99.22887081784184, + "ttft_ms": 383.95209041800626 + }, + { + "e2e_ms": 12987.209322808041, + "request_id": 57, + "tpot_ms": 99.22887081784184, + "ttft_ms": 385.1427289421281 + }, + { + "e2e_ms": 12988.399961332163, + "request_id": 58, + "tpot_ms": 99.22887081784184, + "ttft_ms": 386.3333674662499 + }, + { + "e2e_ms": 12989.881679406835, + "request_id": 59, + "tpot_ms": 99.2311627828068, + "ttft_ms": 387.52400599037173 + }, + { + "e2e_ms": 12991.363397481507, + "request_id": 60, + "tpot_ms": 99.23345474777176, + "ttft_ms": 388.71464451449356 + }, + { + "e2e_ms": 12896.836165730803, + "request_id": 61, + "tpot_ms": 99.23492287861664, + "ttft_ms": 294.0009601464908 + }, + { + "e2e_ms": 12689.81487892944, + "request_id": 62, + "tpot_ms": 97.59316693471084, + "ttft_ms": 295.4826782211626 + }, + { + "e2e_ms": 12448.118757066311, + "request_id": 63, + "tpot_ms": 95.67691266262345, + "ttft_ms": 297.1508489131338 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 220.7558696269989, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e09d047b2039786ee02cd4c59c081ca95e5197ae65723b6170e43f6c0924921a", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 13556.392075191316, + "request_id": 0, + "tpot_ms": 104.4466564602661, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 13715.056219396885, + "request_id": 1, + "tpot_ms": 105.11756526634255, + "ttft_ms": 365.12543057138026 + }, + { + "e2e_ms": 13775.965021971426, + "request_id": 2, + "tpot_ms": 105.6336255570719, + "ttft_ms": 360.4945762232949 + }, + { + "e2e_ms": 13838.18765138338, + "request_id": 3, + "tpot_ms": 106.07242606904647, + "ttft_ms": 366.98954061447785 + }, + { + "e2e_ms": 13802.282961907104, + "request_id": 4, + "tpot_ms": 106.21712281117094, + "ttft_ms": 312.70836488839393 + }, + { + "e2e_ms": 13864.505591319055, + "request_id": 5, + "tpot_ms": 106.32495967758555, + "ttft_ms": 361.2357122656906 + }, + { + "e2e_ms": 13828.78708091747, + "request_id": 6, + "tpot_ms": 106.35892091285005, + "ttft_ms": 321.20412498551156 + }, + { + "e2e_ms": 13892.323156276025, + "request_id": 7, + "tpot_ms": 106.3810710779423, + "ttft_ms": 381.9271293773525 + }, + { + "e2e_ms": 13856.60464587857, + "request_id": 8, + "tpot_ms": 106.3914161711828, + "ttft_ms": 344.89479213835494 + }, + { + "e2e_ms": 13820.886135481267, + "request_id": 9, + "tpot_ms": 106.39288214818258, + "ttft_ms": 308.9901026620794 + }, + { + "e2e_ms": 13883.108764897528, + "request_id": 10, + "tpot_ms": 106.39288214821651, + "ttft_ms": 371.21273207403146 + }, + { + "e2e_ms": 13847.390254500226, + "request_id": 11, + "tpot_ms": 106.39288214825024, + "ttft_ms": 335.4942216724446 + }, + { + "e2e_ms": 13811.485565028235, + "request_id": 12, + "tpot_ms": 106.39288214828399, + "ttft_ms": 299.589532196169 + }, + { + "e2e_ms": 13873.708194444496, + "request_id": 13, + "tpot_ms": 106.38107407728566, + "ttft_ms": 363.31178662921815 + }, + { + "e2e_ms": 13837.989684047194, + "request_id": 14, + "tpot_ms": 106.38107407728566, + "ttft_ms": 327.5932762319158 + }, + { + "e2e_ms": 13901.52575941027, + "request_id": 15, + "tpot_ms": 106.39141617135505, + "ttft_ms": 389.81590564817736 + }, + { + "e2e_ms": 13865.807249012967, + "request_id": 16, + "tpot_ms": 106.39141617135505, + "ttft_ms": 354.097395250875 + }, + { + "e2e_ms": 13830.088738615665, + "request_id": 17, + "tpot_ms": 106.3928821483211, + "ttft_ms": 318.19270577888403 + }, + { + "e2e_ms": 13893.62481397874, + "request_id": 18, + "tpot_ms": 106.40322424239051, + "ttft_ms": 380.4153351951456 + }, + { + "e2e_ms": 13857.906303581438, + "request_id": 19, + "tpot_ms": 106.40322424239051, + "ttft_ms": 344.69682479784325 + }, + { + "e2e_ms": 13822.187793184135, + "request_id": 20, + "tpot_ms": 106.40469021935657, + "ttft_ms": 308.79213532585226 + }, + { + "e2e_ms": 13884.410422593844, + "request_id": 21, + "tpot_ms": 106.39288214826952, + "ttft_ms": 372.51438976361584 + }, + { + "e2e_ms": 13848.691912190276, + "request_id": 22, + "tpot_ms": 106.39288214822018, + "ttft_ms": 336.7958793663135 + }, + { + "e2e_ms": 13812.78722271334, + "request_id": 23, + "tpot_ms": 106.39288214818123, + "ttft_ms": 300.8911898943225 + }, + { + "e2e_ms": 13875.009852123696, + "request_id": 24, + "tpot_ms": 106.38107407709929, + "ttft_ms": 364.6134443320861 + }, + { + "e2e_ms": 13839.291341720127, + "request_id": 25, + "tpot_ms": 106.38107407704995, + "ttft_ms": 328.89493393478375 + }, + { + "e2e_ms": 13902.827417078257, + "request_id": 26, + "tpot_ms": 106.39141617110043, + "ttft_ms": 391.11756334850156 + }, + { + "e2e_ms": 13867.108906674688, + "request_id": 27, + "tpot_ms": 106.39141617110043, + "ttft_ms": 355.39905294493224 + }, + { + "e2e_ms": 13831.390396271117, + "request_id": 28, + "tpot_ms": 106.39288214805607, + "ttft_ms": 319.49436346799587 + }, + { + "e2e_ms": 13894.926471629247, + "request_id": 29, + "tpot_ms": 106.40322424213302, + "ttft_ms": 381.7169928783528 + }, + { + "e2e_ms": 13859.207961225678, + "request_id": 30, + "tpot_ms": 106.40322424213302, + "ttft_ms": 345.9984824747835 + }, + { + "e2e_ms": 13823.48945082211, + "request_id": 31, + "tpot_ms": 106.40469021908869, + "ttft_ms": 310.0937929978471 + }, + { + "e2e_ms": 13885.712080232466, + "request_id": 32, + "tpot_ms": 106.39288214805609, + "ttft_ms": 373.8160474293437 + }, + { + "e2e_ms": 13849.993569828897, + "request_id": 33, + "tpot_ms": 106.39288214805609, + "ttft_ms": 338.0975370257744 + }, + { + "e2e_ms": 13814.08888035196, + "request_id": 34, + "tpot_ms": 106.39288214805607, + "ttft_ms": 302.192847548838 + }, + { + "e2e_ms": 13876.311509762318, + "request_id": 35, + "tpot_ms": 106.3810740770235, + "ttft_ms": 365.9151019803346 + }, + { + "e2e_ms": 13840.592999358747, + "request_id": 36, + "tpot_ms": 106.38107407702348, + "ttft_ms": 330.1965915767653 + }, + { + "e2e_ms": 13904.129074716877, + "request_id": 37, + "tpot_ms": 106.39141617110043, + "ttft_ms": 392.41922098712223 + }, + { + "e2e_ms": 13868.410564313308, + "request_id": 38, + "tpot_ms": 106.39141617110043, + "ttft_ms": 356.7007105835529 + }, + { + "e2e_ms": 13832.69205390974, + "request_id": 39, + "tpot_ms": 106.39288214805609, + "ttft_ms": 320.79602110661654 + }, + { + "e2e_ms": 13896.228129267869, + "request_id": 40, + "tpot_ms": 106.40322424213304, + "ttft_ms": 383.0186505169735 + }, + { + "e2e_ms": 13860.509618864298, + "request_id": 41, + "tpot_ms": 106.40322424213302, + "ttft_ms": 347.30014011340415 + }, + { + "e2e_ms": 13824.79110846073, + "request_id": 42, + "tpot_ms": 106.40469021908868, + "ttft_ms": 311.3954506364678 + }, + { + "e2e_ms": 13888.32718381886, + "request_id": 43, + "tpot_ms": 106.40322424213302, + "ttft_ms": 375.1177050679644 + }, + { + "e2e_ms": 13852.60867341529, + "request_id": 44, + "tpot_ms": 106.40322424213304, + "ttft_ms": 339.39919466439505 + }, + { + "e2e_ms": 13816.89016301172, + "request_id": 45, + "tpot_ms": 106.40469021908868, + "ttft_ms": 303.4945051874587 + }, + { + "e2e_ms": 13879.11279242067, + "request_id": 46, + "tpot_ms": 106.392882148045, + "ttft_ms": 367.2167596189553 + }, + { + "e2e_ms": 13843.394282013549, + "request_id": 47, + "tpot_ms": 106.39288214801702, + "ttft_ms": 331.49824921538595 + }, + { + "e2e_ms": 13807.489592530403, + "request_id": 48, + "tpot_ms": 106.39288214796812, + "ttft_ms": 295.5935597384496 + }, + { + "e2e_ms": 13869.712221935884, + "request_id": 49, + "tpot_ms": 106.38107407689715, + "ttft_ms": 359.3158141699462 + }, + { + "e2e_ms": 13833.993711528763, + "request_id": 50, + "tpot_ms": 106.38107407686918, + "ttft_ms": 323.59730376637685 + }, + { + "e2e_ms": 13897.5297868798, + "request_id": 51, + "tpot_ms": 106.3914161708903, + "ttft_ms": 385.8199331767338 + }, + { + "e2e_ms": 13861.81127647268, + "request_id": 52, + "tpot_ms": 106.39141617089378, + "ttft_ms": 350.1014227691712 + }, + { + "e2e_ms": 13826.092766065556, + "request_id": 53, + "tpot_ms": 106.39288214787034, + "ttft_ms": 314.1967332860247 + }, + { + "e2e_ms": 13889.628841416596, + "request_id": 54, + "tpot_ms": 106.40322424192983, + "ttft_ms": 376.41936269150733 + }, + { + "e2e_ms": 13853.910331009472, + "request_id": 55, + "tpot_ms": 106.4032242419298, + "ttft_ms": 340.7008522843853 + }, + { + "e2e_ms": 13818.191820602351, + "request_id": 56, + "tpot_ms": 106.4046902189064, + "ttft_ms": 304.7961628012388 + }, + { + "e2e_ms": 13880.414450007833, + "request_id": 57, + "tpot_ms": 106.39288214787034, + "ttft_ms": 368.5184172283016 + }, + { + "e2e_ms": 13844.695939600711, + "request_id": 58, + "tpot_ms": 106.39288214787034, + "ttft_ms": 332.79990682117955 + }, + { + "e2e_ms": 13600.134841430987, + "request_id": 59, + "tpot_ms": 104.74991829994453, + "ttft_ms": 296.89521733803304 + }, + { + "e2e_ms": 13451.749583806417, + "request_id": 60, + "tpot_ms": 103.07978040977419, + "ttft_ms": 360.61747176509584 + }, + { + "e2e_ms": 13168.506714577745, + "request_id": 61, + "tpot_ms": 101.13076971039189, + "ttft_ms": 324.8989613579738 + }, + { + "e2e_ms": 12970.21046401943, + "request_id": 62, + "tpot_ms": 99.07943994689744, + "ttft_ms": 387.1215907634564 + }, + { + "e2e_ms": 12659.030247905519, + "request_id": 63, + "tpot_ms": 96.91045013818255, + "ttft_ms": 351.4030803563344 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 142.3273642063141, + "offered_request_rate": 0.8, + "request_metrics_sha256": "6b246d56546536e46b4e8adfb23076b3cbf053da4b0af0288346df10122d6022", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 15423.90313583537, + "request_id": 0, + "tpot_ms": 119.15146796140039, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 15680.782114004433, + "request_id": 1, + "tpot_ms": 120.47488426274414, + "ttft_ms": 380.47181263592745 + }, + { + "e2e_ms": 15824.99160915063, + "request_id": 2, + "tpot_ms": 121.60222806639713, + "ttft_ms": 381.5086447181937 + }, + { + "e2e_ms": 15964.925902693089, + "request_id": 3, + "tpot_ms": 122.6571852387084, + "ttft_ms": 387.4633773771219 + }, + { + "e2e_ms": 15994.3969757696, + "request_id": 4, + "tpot_ms": 123.48492728290344, + "ttft_ms": 311.8112108408626 + }, + { + "e2e_ms": 16132.273068839513, + "request_id": 5, + "tpot_ms": 124.38017511477935, + "ttft_ms": 335.9908292625358 + }, + { + "e2e_ms": 16268.09096143688, + "request_id": 6, + "tpot_ms": 125.15372351796954, + "ttft_ms": 373.5680746547478 + }, + { + "e2e_ms": 16291.239371010352, + "request_id": 7, + "tpot_ms": 125.67763083904141, + "ttft_ms": 330.18025445209173 + }, + { + "e2e_ms": 16612.993923996273, + "request_id": 8, + "tpot_ms": 127.71634872045071, + "ttft_ms": 393.0176364990334 + }, + { + "e2e_ms": 16636.356874836954, + "request_id": 9, + "tpot_ms": 128.11558123152213, + "ttft_ms": 365.67805843364454 + }, + { + "e2e_ms": 16659.624174048608, + "request_id": 10, + "tpot_ms": 128.3236060799542, + "ttft_ms": 362.52620189442376 + }, + { + "e2e_ms": 16683.03953534565, + "request_id": 11, + "tpot_ms": 128.34365422438844, + "ttft_ms": 383.3954488483169 + }, + { + "e2e_ms": 16598.14552827833, + "request_id": 12, + "tpot_ms": 128.20865630197483, + "ttft_ms": 315.6461779275297 + }, + { + "e2e_ms": 16621.508479120537, + "request_id": 13, + "tpot_ms": 128.0747814081899, + "ttft_ms": 356.0112402804201 + }, + { + "e2e_ms": 16644.871429963147, + "request_id": 14, + "tpot_ms": 127.97565080664988, + "ttft_ms": 391.9637775186118 + }, + { + "e2e_ms": 16559.977422897646, + "request_id": 15, + "tpot_ms": 127.91134905305496, + "ttft_ms": 315.2360931596654 + }, + { + "e2e_ms": 16583.340373740255, + "request_id": 16, + "tpot_ms": 127.86325360239432, + "ttft_ms": 344.7071662361765 + }, + { + "e2e_ms": 16606.703324582868, + "request_id": 17, + "tpot_ms": 127.81515815173371, + "ttft_ms": 374.17823931268754 + }, + { + "e2e_ms": 16630.066275425477, + "request_id": 18, + "tpot_ms": 127.7832690040065, + "ttft_ms": 401.59111191665176 + }, + { + "e2e_ms": 16354.971144941388, + "request_id": 19, + "tpot_ms": 126.2873116642462, + "ttft_ms": 316.48256358212024 + }, + { + "e2e_ms": 16568.46874002116, + "request_id": 20, + "tpot_ms": 127.78612414854776, + "ttft_ms": 339.6309731555931 + }, + { + "e2e_ms": 16591.83169086377, + "request_id": 21, + "tpot_ms": 127.78612414856296, + "ttft_ms": 362.9939239962745 + }, + { + "e2e_ms": 16615.19464170638, + "request_id": 22, + "tpot_ms": 127.78612414857815, + "ttft_ms": 386.3568748369559 + }, + { + "e2e_ms": 16339.951449136564, + "request_id": 23, + "tpot_ms": 126.28847750749865, + "ttft_ms": 301.3148056842354 + }, + { + "e2e_ms": 16553.449044216337, + "request_id": 24, + "tpot_ms": 127.78518800972483, + "ttft_ms": 324.73016698128276 + }, + { + "e2e_ms": 16576.811995058946, + "request_id": 25, + "tpot_ms": 127.78477532898123, + "ttft_ms": 348.1455282783301 + }, + { + "e2e_ms": 16600.174945901555, + "request_id": 26, + "tpot_ms": 127.78477532898441, + "ttft_ms": 371.50847912053564 + }, + { + "e2e_ms": 16623.53789674417, + "request_id": 27, + "tpot_ms": 127.78477532898442, + "ttft_ms": 394.87142996314617 + }, + { + "e2e_ms": 16348.29470417435, + "request_id": 28, + "tpot_ms": 126.28596284469846, + "ttft_ms": 309.97742289764574 + }, + { + "e2e_ms": 16561.79229925412, + "request_id": 29, + "tpot_ms": 127.78308602766822, + "ttft_ms": 333.34037374025627 + }, + { + "e2e_ms": 16585.155250096734, + "request_id": 30, + "tpot_ms": 127.78308602766825, + "ttft_ms": 356.7033245828668 + }, + { + "e2e_ms": 16608.518200939343, + "request_id": 31, + "tpot_ms": 127.78308602766823, + "ttft_ms": 380.06627542547733 + }, + { + "e2e_ms": 16631.881151781956, + "request_id": 32, + "tpot_ms": 127.78360948578973, + "ttft_ms": 403.36274708666053 + }, + { + "e2e_ms": 16356.786021297865, + "request_id": 33, + "tpot_ms": 126.28596284469846, + "ttft_ms": 318.4687400211601 + }, + { + "e2e_ms": 16570.283616377637, + "request_id": 34, + "tpot_ms": 127.78308602766825, + "ttft_ms": 341.83169086377063 + }, + { + "e2e_ms": 16593.646567220247, + "request_id": 35, + "tpot_ms": 127.78308602766823, + "ttft_ms": 365.19464170638116 + }, + { + "e2e_ms": 16617.00951806286, + "request_id": 36, + "tpot_ms": 127.78477532898444, + "ttft_ms": 388.34305128183644 + }, + { + "e2e_ms": 16341.766325493041, + "request_id": 37, + "tpot_ms": 126.28596284469846, + "ttft_ms": 303.449044216336 + }, + { + "e2e_ms": 16555.263920572714, + "request_id": 38, + "tpot_ms": 127.78308602766747, + "ttft_ms": 326.81199505894654 + }, + { + "e2e_ms": 16578.62687141207, + "request_id": 39, + "tpot_ms": 127.78308602764184, + "ttft_ms": 350.17494590155707 + }, + { + "e2e_ms": 16601.989822251424, + "request_id": 40, + "tpot_ms": 127.78308602761618, + "ttft_ms": 373.5378967441676 + }, + { + "e2e_ms": 16625.35277309078, + "request_id": 41, + "tpot_ms": 127.78477532890675, + "ttft_ms": 396.6863063196229 + }, + { + "e2e_ms": 16350.109580518321, + "request_id": 42, + "tpot_ms": 126.2859628446, + "ttft_ms": 311.79229925412244 + }, + { + "e2e_ms": 16563.607175594883, + "request_id": 43, + "tpot_ms": 127.78308602754448, + "ttft_ms": 335.15525009673297 + }, + { + "e2e_ms": 16586.970126434237, + "request_id": 44, + "tpot_ms": 127.78308602751885, + "ttft_ms": 358.5182009393435 + }, + { + "e2e_ms": 16610.333077273594, + "request_id": 45, + "tpot_ms": 127.78308602749323, + "ttft_ms": 381.88115178195403 + }, + { + "e2e_ms": 16633.69602811295, + "request_id": 46, + "tpot_ms": 127.78360948558907, + "ttft_ms": 405.17762344313724 + }, + { + "e2e_ms": 16548.802021044266, + "request_id": 47, + "tpot_ms": 127.78360948556401, + "ttft_ms": 320.2836163776368 + }, + { + "e2e_ms": 16572.164971883623, + "request_id": 48, + "tpot_ms": 127.7836094855384, + "ttft_ms": 343.64656722024733 + }, + { + "e2e_ms": 16595.52792272298, + "request_id": 49, + "tpot_ms": 127.78360948551277, + "ttft_ms": 367.00951806285786 + }, + { + "e2e_ms": 16618.890873562337, + "request_id": 50, + "tpot_ms": 127.78529878680334, + "ttft_ms": 390.15792763831314 + }, + { + "e2e_ms": 16343.647680989874, + "request_id": 51, + "tpot_ms": 126.28648630249732, + "ttft_ms": 305.26392057271323 + }, + { + "e2e_ms": 16199.563507361632, + "request_id": 52, + "tpot_ms": 124.96800500747688, + "ttft_ms": 328.6268714120695 + }, + { + "e2e_ms": 16032.332947106512, + "request_id": 53, + "tpot_ms": 123.46726869964634, + "ttft_ms": 351.9898222514257 + }, + { + "e2e_ms": 15838.885472588829, + "request_id": 54, + "tpot_ms": 121.7600999960476, + "ttft_ms": 375.35277309078197 + }, + { + "e2e_ms": 15619.209002299955, + "request_id": 55, + "tpot_ms": 119.84809306800307, + "ttft_ms": 398.5011826635656 + }, + { + "e2e_ms": 15285.473398871076, + "request_id": 56, + "tpot_ms": 117.8887104194976, + "ttft_ms": 313.60717559488194 + }, + { + "e2e_ms": 15049.070462905931, + "request_id": 57, + "tpot_ms": 115.84330973599758, + "ttft_ms": 336.9701264342382 + }, + { + "e2e_ms": 14793.47811125659, + "request_id": 58, + "tpot_ms": 113.6468112912047, + "ttft_ms": 360.3330772735944 + }, + { + "e2e_ms": 14524.751300129523, + "request_id": 59, + "tpot_ms": 111.34689190564231, + "ttft_ms": 383.69602811295067 + }, + { + "e2e_ms": 14156.606828520311, + "request_id": 60, + "tpot_ms": 109.11657328721294, + "ttft_ms": 298.802021044267 + }, + { + "e2e_ms": 13870.414276689999, + "request_id": 61, + "tpot_ms": 106.67912838430216, + "ttft_ms": 322.16497188362325 + }, + { + "e2e_ms": 13579.457505623892, + "request_id": 62, + "tpot_ms": 104.20416994410166, + "ttft_ms": 345.5279227229795 + }, + { + "e2e_ms": 13281.118281252404, + "request_id": 63, + "tpot_ms": 101.6710819503155, + "ttft_ms": 368.89087356233574 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.15625, + "passed": 10 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 138.39458417892456, + "offered_request_rate": 1.2, + "request_metrics_sha256": "dfafc592cb8f7c26725458265ca04cb89642135218d5f80993dfbd6e83fcea9c", + "request_rate_per_gpu": 0.15, + "requests": [ + { + "e2e_ms": 17349.6420929535, + "request_id": 0, + "tpot_ms": 134.3147668363463, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 17472.38469413562, + "request_id": 1, + "tpot_ms": 135.04025977135157, + "ttft_ms": 322.27170317397037 + }, + { + "e2e_ms": 17594.570506858923, + "request_id": 2, + "tpot_ms": 135.72808492940857, + "ttft_ms": 357.1037208240346 + }, + { + "e2e_ms": 17607.2367009099, + "request_id": 3, + "tpot_ms": 136.28731175562888, + "ttft_ms": 298.7481079450336 + }, + { + "e2e_ms": 17921.914449676293, + "request_id": 4, + "tpot_ms": 138.39011926907332, + "ttft_ms": 346.36930250398024 + }, + { + "e2e_ms": 17935.406822135243, + "request_id": 5, + "tpot_ms": 138.88107821741167, + "ttft_ms": 297.50988852396 + }, + { + "e2e_ms": 18057.925904932094, + "request_id": 6, + "tpot_ms": 139.39521236971183, + "ttft_ms": 354.73393397869125 + }, + { + "e2e_ms": 18071.482157513437, + "request_id": 7, + "tpot_ms": 139.75067177492934, + "ttft_ms": 323.14684209741085 + }, + { + "e2e_ms": 18194.065120430685, + "request_id": 8, + "tpot_ms": 140.142283173617, + "ttft_ms": 395.99515738132584 + }, + { + "e2e_ms": 18207.834601589766, + "request_id": 9, + "tpot_ms": 140.4257392143267, + "ttft_ms": 373.76572137027256 + }, + { + "e2e_ms": 18221.607066695313, + "request_id": 10, + "tpot_ms": 140.60137526077196, + "ttft_ms": 365.232408577274 + }, + { + "e2e_ms": 18235.379531799863, + "request_id": 11, + "tpot_ms": 140.670460527253, + "ttft_ms": 370.23104483873226 + }, + { + "e2e_ms": 18249.21289308034, + "request_id": 12, + "tpot_ms": 140.64257855984852, + "ttft_ms": 387.6054159795803 + }, + { + "e2e_ms": 18263.046254360816, + "request_id": 13, + "tpot_ms": 140.69995603545655, + "ttft_ms": 394.1518378578337 + }, + { + "e2e_ms": 18276.879615640297, + "request_id": 14, + "tpot_ms": 140.75508408264292, + "ttft_ms": 400.98393714464817 + }, + { + "e2e_ms": 18181.536918127527, + "request_id": 15, + "tpot_ms": 140.80327015717634, + "ttft_ms": 299.52160816613116 + }, + { + "e2e_ms": 18195.15705083027, + "request_id": 16, + "tpot_ms": 140.84624138271937, + "ttft_ms": 307.6843952249071 + }, + { + "e2e_ms": 18208.77718353195, + "request_id": 17, + "tpot_ms": 140.87800086343316, + "ttft_ms": 317.2710738759417 + }, + { + "e2e_ms": 18222.54666469151, + "request_id": 18, + "tpot_ms": 140.90607860065492, + "ttft_ms": 327.4746824083348 + }, + { + "e2e_ms": 18236.319129797193, + "request_id": 19, + "tpot_ms": 140.91088449209656, + "ttft_ms": 340.6367993009294 + }, + { + "e2e_ms": 18250.152491077173, + "request_id": 20, + "tpot_ms": 140.8981539798204, + "ttft_ms": 356.0869356399827 + }, + { + "e2e_ms": 18263.985852358146, + "request_id": 21, + "tpot_ms": 140.89857595079965, + "ttft_ms": 369.8667066065901 + }, + { + "e2e_ms": 18277.81921363912, + "request_id": 22, + "tpot_ms": 140.90338208286406, + "ttft_ms": 383.0896891153834 + }, + { + "e2e_ms": 18291.6525749191, + "request_id": 23, + "tpot_ms": 140.91257237601366, + "ttft_ms": 395.7558831653678 + }, + { + "e2e_ms": 18305.48593620007, + "request_id": 24, + "tpot_ms": 140.9217626691632, + "ttft_ms": 408.4220772163469 + }, + { + "e2e_ms": 18210.143238687502, + "request_id": 25, + "tpot_ms": 140.92327164841032, + "ttft_ms": 312.8877393393914 + }, + { + "e2e_ms": 18223.912719846063, + "request_id": 26, + "tpot_ms": 140.92545360667498, + "ttft_ms": 326.38011179834336 + }, + { + "e2e_ms": 18237.68518495175, + "request_id": 27, + "tpot_ms": 140.92765906057843, + "ttft_ms": 339.8724842582901 + }, + { + "e2e_ms": 18251.518546232724, + "request_id": 28, + "tpot_ms": 140.92984101884323, + "ttft_ms": 353.42873683963205 + }, + { + "e2e_ms": 18265.3519075127, + "request_id": 29, + "tpot_ms": 140.93202297710803, + "ttft_ms": 366.98498941997923 + }, + { + "e2e_ms": 18279.185268793677, + "request_id": 30, + "tpot_ms": 140.93370194229104, + "ttft_ms": 380.60512212271647 + }, + { + "e2e_ms": 18293.01863007465, + "request_id": 31, + "tpot_ms": 140.9341814397353, + "ttft_ms": 394.3775872282629 + }, + { + "e2e_ms": 18306.851991354626, + "request_id": 32, + "tpot_ms": 140.93466093717961, + "ttft_ms": 408.15005233281454 + }, + { + "e2e_ms": 18211.509293842057, + "request_id": 33, + "tpot_ms": 140.933484965084, + "ttft_ms": 312.956703276388 + }, + { + "e2e_ms": 18225.278775001614, + "request_id": 34, + "tpot_ms": 140.9329819720059, + "ttft_ms": 326.79006455686374 + }, + { + "e2e_ms": 18239.11213628159, + "request_id": 35, + "tpot_ms": 140.93298197200983, + "ttft_ms": 340.6234258363447 + }, + { + "e2e_ms": 18252.945497562563, + "request_id": 36, + "tpot_ms": 140.93298197201372, + "ttft_ms": 354.45678711682046 + }, + { + "e2e_ms": 18266.778858843536, + "request_id": 37, + "tpot_ms": 140.93466093719667, + "ttft_ms": 368.0769198195577 + }, + { + "e2e_ms": 18280.612220123516, + "request_id": 38, + "tpot_ms": 140.93633990237933, + "ttft_ms": 381.69705252133923 + }, + { + "e2e_ms": 18294.44558140449, + "request_id": 39, + "tpot_ms": 140.93801886756114, + "ttft_ms": 395.3171852242221 + }, + { + "e2e_ms": 18308.27894268546, + "request_id": 40, + "tpot_ms": 140.93849836500434, + "ttft_ms": 409.0896503299106 + }, + { + "e2e_ms": 18212.936245171895, + "request_id": 41, + "tpot_ms": 140.93732239290452, + "ttft_ms": 313.8963012730187 + }, + { + "e2e_ms": 18226.708710277584, + "request_id": 42, + "tpot_ms": 140.93684289546135, + "ttft_ms": 327.72966255399183 + }, + { + "e2e_ms": 18047.749752064, + "request_id": 43, + "tpot_ms": 139.41879313566167, + "ttft_ms": 341.56302383496495 + }, + { + "e2e_ms": 17872.42465139999, + "request_id": 44, + "tpot_ms": 137.92935642744132, + "ttft_ms": 355.3963851149433 + }, + { + "e2e_ms": 17695.178674250878, + "request_id": 45, + "tpot_ms": 136.4247947075194, + "ttft_ms": 369.22974639591644 + }, + { + "e2e_ms": 17514.86814366097, + "request_id": 46, + "tpot_ms": 134.89610264554392, + "ttft_ms": 383.06310767688956 + }, + { + "e2e_ms": 17334.24324324006, + "request_id": 47, + "tpot_ms": 133.36661419575813, + "ttft_ms": 396.68324037877767 + }, + { + "e2e_ms": 17152.122611965817, + "request_id": 48, + "tpot_ms": 131.8241488699319, + "ttft_ms": 410.4557054844662 + }, + { + "e2e_ms": 16860.906258226023, + "request_id": 49, + "tpot_ms": 130.28066064407446, + "ttft_ms": 315.26235642856903 + }, + { + "e2e_ms": 16678.317507592936, + "request_id": 50, + "tpot_ms": 128.73402984160936, + "ttft_ms": 329.0957177085474 + }, + { + "e2e_ms": 16495.414294222966, + "request_id": 51, + "tpot_ms": 127.18492295459406, + "ttft_ms": 342.9290789895205 + }, + { + "e2e_ms": 16325.716031493954, + "request_id": 52, + "tpot_ms": 125.73979205687765, + "ttft_ms": 356.76244027049364 + }, + { + "e2e_ms": 16142.044362006885, + "request_id": 53, + "tpot_ms": 124.18463433430247, + "ttft_ms": 370.595801550472 + }, + { + "e2e_ms": 15942.642543961227, + "request_id": 54, + "tpot_ms": 122.50561717425025, + "ttft_ms": 384.4291628314451 + }, + { + "e2e_ms": 15727.503328455541, + "request_id": 55, + "tpot_ms": 120.70436246394655, + "ttft_ms": 398.049295534328 + }, + { + "e2e_ms": 15403.205770995846, + "request_id": 56, + "tpot_ms": 118.90039231904261, + "ttft_ms": 302.8559464774361 + }, + { + "e2e_ms": 15178.030676088234, + "request_id": 57, + "tpot_ms": 117.01843597110097, + "ttft_ms": 316.6893077584092 + }, + { + "e2e_ms": 14941.341931765628, + "request_id": 58, + "tpot_ms": 115.04582096634839, + "ttft_ms": 330.5226690393823 + }, + { + "e2e_ms": 14696.772511761146, + "request_id": 59, + "tpot_ms": 113.01115339717941, + "ttft_ms": 344.3560303193607 + }, + { + "e2e_ms": 14451.34286402164, + "request_id": 60, + "tpot_ms": 110.96971238127013, + "ttft_ms": 358.1893916003338 + }, + { + "e2e_ms": 14196.293999590453, + "request_id": 61, + "tpot_ms": 108.85252950164681, + "ttft_ms": 372.02275288130693 + }, + { + "e2e_ms": 13938.386603616791, + "request_id": 62, + "tpot_ms": 106.71283849964966, + "ttft_ms": 385.8561141612853 + }, + { + "e2e_ms": 13676.049735660896, + "request_id": 63, + "tpot_ms": 104.53992523504405, + "ttft_ms": 399.4792308103001 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 164.71713185310364, + "offered_request_rate": 1.6, + "request_metrics_sha256": "e85085ea5517c1afa9f850cb838a638360d45db695cde8d3c9fef6baf6af44c4", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 19447.058061600903, + "request_id": 0, + "tpot_ms": 150.8298532036487, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 19750.640261300512, + "request_id": 1, + "tpot_ms": 152.83450342712686, + "ttft_ms": 340.65832605540146 + }, + { + "e2e_ms": 19750.02534103151, + "request_id": 2, + "tpot_ms": 153.17602578595117, + "ttft_ms": 296.67006621570977 + }, + { + "e2e_ms": 19855.93358266601, + "request_id": 3, + "tpot_ms": 153.5864661294274, + "ttft_ms": 350.4523842287295 + }, + { + "e2e_ms": 19855.232098385564, + "request_id": 4, + "tpot_ms": 153.87573615619343, + "ttft_ms": 313.01360654900145 + }, + { + "e2e_ms": 19961.05377600862, + "request_id": 5, + "tpot_ms": 154.2244676371592, + "ttft_ms": 374.54638608940184 + }, + { + "e2e_ms": 19960.265727716724, + "request_id": 6, + "tpot_ms": 154.4718650613629, + "ttft_ms": 342.33886492363473 + }, + { + "e2e_ms": 19959.47767942483, + "request_id": 7, + "tpot_ms": 154.6679176552935, + "ttft_ms": 316.6521372025519 + }, + { + "e2e_ms": 20065.255011109733, + "request_id": 8, + "tpot_ms": 154.88778812601134, + "ttft_ms": 394.50591910629294 + }, + { + "e2e_ms": 20064.42261687969, + "request_id": 9, + "tpot_ms": 155.03615010045385, + "ttft_ms": 374.8315541220517 + }, + { + "e2e_ms": 20063.590222649644, + "request_id": 10, + "tpot_ms": 155.117974090142, + "ttft_ms": 363.6075132016101 + }, + { + "e2e_ms": 20062.7578284196, + "request_id": 11, + "tpot_ms": 155.13454318347627, + "ttft_ms": 360.67084411811567 + }, + { + "e2e_ms": 20061.925434189554, + "request_id": 12, + "tpot_ms": 155.09034115377113, + "ttft_ms": 365.45210766062206 + }, + { + "e2e_ms": 20061.09303995951, + "request_id": 13, + "tpot_ms": 155.09665427482798, + "ttft_ms": 363.8179470563543 + }, + { + "e2e_ms": 20060.260645729464, + "request_id": 14, + "tpot_ms": 155.10154332898202, + "ttft_ms": 362.3646429487479 + }, + { + "e2e_ms": 20059.42825149942, + "request_id": 15, + "tpot_ms": 155.10604675613317, + "ttft_ms": 360.96031347050683 + }, + { + "e2e_ms": 20058.595857269374, + "request_id": 16, + "tpot_ms": 155.10168661355542, + "ttft_ms": 360.6816573478344 + }, + { + "e2e_ms": 20057.76346303933, + "request_id": 17, + "tpot_ms": 155.0900405289204, + "ttft_ms": 361.3283158664373 + }, + { + "e2e_ms": 20056.931068809285, + "request_id": 18, + "tpot_ms": 155.07436184636043, + "ttft_ms": 362.4871143215085 + }, + { + "e2e_ms": 20056.09867457924, + "request_id": 19, + "tpot_ms": 155.0434256316313, + "ttft_ms": 365.58361936206654 + }, + { + "e2e_ms": 20055.26628034948, + "request_id": 20, + "tpot_ms": 154.9995151863024, + "ttft_ms": 370.32785168907446 + }, + { + "e2e_ms": 20054.433886119732, + "request_id": 21, + "tpot_ms": 154.96275487551375, + "ttft_ms": 374.16401692948574 + }, + { + "e2e_ms": 20053.601491889985, + "request_id": 22, + "tpot_ms": 154.92303029896564, + "ttft_ms": 378.3766439213494 + }, + { + "e2e_ms": 20052.76909766024, + "request_id": 23, + "tpot_ms": 154.88711594543284, + "ttft_ms": 382.105372590269 + }, + { + "e2e_ms": 20051.936703430492, + "request_id": 24, + "tpot_ms": 154.8482609975464, + "ttft_ms": 386.20755674210193 + }, + { + "e2e_ms": 20051.104309200746, + "request_id": 25, + "tpot_ms": 154.85607608193587, + "ttft_ms": 384.3826467948901 + }, + { + "e2e_ms": 20050.271914971, + "request_id": 26, + "tpot_ms": 154.86595665257255, + "ttft_ms": 382.2954200942838 + }, + { + "e2e_ms": 20049.439520741253, + "request_id": 27, + "tpot_ms": 154.87441181850545, + "ttft_ms": 380.3892197910592 + }, + { + "e2e_ms": 20048.607126511506, + "request_id": 28, + "tpot_ms": 154.87585298283847, + "ttft_ms": 379.37379769102006 + }, + { + "e2e_ms": 20047.77473228176, + "request_id": 29, + "tpot_ms": 154.8790680483812, + "ttft_ms": 378.13309013734295 + }, + { + "e2e_ms": 20046.942338052013, + "request_id": 30, + "tpot_ms": 154.879633497606, + "ttft_ms": 377.228883856052 + }, + { + "e2e_ms": 20046.109943822266, + "request_id": 31, + "tpot_ms": 154.8807461594705, + "ttft_ms": 376.2551815695119 + }, + { + "e2e_ms": 20045.27754959252, + "request_id": 32, + "tpot_ms": 154.8790337660788, + "ttft_ms": 375.64026130051076 + }, + { + "e2e_ms": 19846.84310747305, + "request_id": 33, + "tpot_ms": 153.32139973576017, + "ttft_ms": 375.02534103150964 + }, + { + "e2e_ms": 19648.543544766126, + "request_id": 34, + "tpot_ms": 151.76482774805996, + "ttft_ms": 374.4104207625085 + }, + { + "e2e_ms": 19450.677394587543, + "request_id": 35, + "tpot_ms": 150.2123500638227, + "ttft_ms": 373.7089364820605 + }, + { + "e2e_ms": 19253.237256186894, + "request_id": 36, + "tpot_ms": 148.66322680303372, + "ttft_ms": 373.00745220161247 + }, + { + "e2e_ms": 19054.80482620875, + "request_id": 37, + "tpot_ms": 147.10697182912625, + "ttft_ms": 372.2194039097175 + }, + { + "e2e_ms": 18856.422732154002, + "request_id": 38, + "tpot_ms": 145.55111320107227, + "ttft_ms": 371.4313556178226 + }, + { + "e2e_ms": 18658.616312962055, + "request_id": 39, + "tpot_ms": 143.99978744595376, + "ttft_ms": 370.64330732592765 + }, + { + "e2e_ms": 18469.244245876795, + "request_id": 40, + "tpot_ms": 142.51522309276308, + "ttft_ms": 369.81091309588265 + }, + { + "e2e_ms": 18279.907702313452, + "request_id": 41, + "tpot_ms": 141.0309384523434, + "ttft_ms": 368.97851886583766 + }, + { + "e2e_ms": 18091.61193748763, + "request_id": 42, + "tpot_ms": 139.55484892009318, + "ttft_ms": 368.14612463579266 + }, + { + "e2e_ms": 17903.060652176137, + "request_id": 43, + "tpot_ms": 138.07674741551486, + "ttft_ms": 367.31373040574766 + }, + { + "e2e_ms": 17716.73280728946, + "request_id": 44, + "tpot_ms": 136.61615331585634, + "ttft_ms": 366.48133617570267 + }, + { + "e2e_ms": 17529.12437807872, + "request_id": 45, + "tpot_ms": 135.14547587506345, + "ttft_ms": 365.64894194565767 + }, + { + "e2e_ms": 17339.472913240785, + "request_id": 46, + "tpot_ms": 133.65871153956826, + "ttft_ms": 364.81654771561267 + }, + { + "e2e_ms": 17149.611868516175, + "request_id": 47, + "tpot_ms": 132.17029696874496, + "ttft_ms": 363.9841534855677 + }, + { + "e2e_ms": 16958.75366988868, + "request_id": 48, + "tpot_ms": 130.67403079238707, + "ttft_ms": 363.1517592555227 + }, + { + "e2e_ms": 16767.44725356325, + "request_id": 49, + "tpot_ms": 129.1742353428171, + "ttft_ms": 362.3193650254777 + }, + { + "e2e_ms": 16576.276975363853, + "request_id": 50, + "tpot_ms": 127.67551184699543, + "ttft_ms": 361.4869707954327 + }, + { + "e2e_ms": 16384.89705533921, + "request_id": 51, + "tpot_ms": 126.1751376281405, + "ttft_ms": 360.65457656536637 + }, + { + "e2e_ms": 16202.320435741867, + "request_id": 52, + "tpot_ms": 124.74408073548226, + "ttft_ms": 359.8221823356198 + }, + { + "e2e_ms": 16010.428211639819, + "request_id": 53, + "tpot_ms": 123.23967262625155, + "ttft_ms": 358.98978810587323 + }, + { + "e2e_ms": 15808.049221831383, + "request_id": 54, + "tpot_ms": 121.6526915587028, + "ttft_ms": 358.15739387612666 + }, + { + "e2e_ms": 15595.17863371626, + "request_id": 55, + "tpot_ms": 119.98309948086519, + "ttft_ms": 357.3249996463801 + }, + { + "e2e_ms": 15377.4072544157, + "request_id": 56, + "tpot_ms": 118.27491849605565, + "ttft_ms": 356.4926054166335 + }, + { + "e2e_ms": 15157.846080032627, + "request_id": 57, + "tpot_ms": 116.55264463658062, + "ttft_ms": 355.66021118688695 + }, + { + "e2e_ms": 14930.609139372891, + "request_id": 58, + "tpot_ms": 114.76993167256497, + "ttft_ms": 354.8278169571404 + }, + { + "e2e_ms": 14698.1184149259, + "request_id": 59, + "tpot_ms": 112.9458503322717, + "ttft_ms": 353.9954227273938 + }, + { + "e2e_ms": 14465.054205320868, + "request_id": 60, + "tpot_ms": 111.11725336081277, + "ttft_ms": 353.16302849764725 + }, + { + "e2e_ms": 14225.577184589198, + "request_id": 61, + "tpot_ms": 109.23816181355352, + "ttft_ms": 352.3306342679007 + }, + { + "e2e_ms": 13984.194476163324, + "request_id": 62, + "tpot_ms": 107.34406485137929, + "ttft_ms": 351.4982400381541 + }, + { + "e2e_ms": 13739.858786414643, + "request_id": 63, + "tpot_ms": 105.42671606776564, + "ttft_ms": 350.66584580840754 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 391.55592226982117, + "offered_request_rate": 2.4, + "request_metrics_sha256": "c4ae9e9bb2050189ca506883c0c12efd5faecce0ec6249ad641d32f3b9a7f130", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 25856.75271176486, + "request_id": 0, + "tpot_ms": 201.29988981911293, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26179.284600266838, + "request_id": 1, + "tpot_ms": 203.30897363251975, + "ttft_ms": 359.0449489368311 + }, + { + "e2e_ms": 25981.87281483019, + "request_id": 2, + "tpot_ms": 201.96786246415923, + "ttft_ms": 331.9542818819672 + }, + { + "e2e_ms": 25784.35394549511, + "request_id": 3, + "tpot_ms": 200.62102535110301, + "ttft_ms": 305.48372590502584 + }, + { + "e2e_ms": 25696.083985341404, + "request_id": 4, + "tpot_ms": 199.35042104178692, + "ttft_ms": 378.5805130344673 + }, + { + "e2e_ms": 25498.25455731707, + "request_id": 5, + "tpot_ms": 197.981349219281, + "ttft_ms": 354.62320646838384 + }, + { + "e2e_ms": 25300.295013812065, + "request_id": 6, + "tpot_ms": 196.59886557857163, + "ttft_ms": 332.2390853334691 + }, + { + "e2e_ms": 25102.100957670827, + "request_id": 7, + "tpot_ms": 195.19408059762435, + "ttft_ms": 312.45272177253236 + }, + { + "e2e_ms": 25012.968951433395, + "request_id": 8, + "tpot_ms": 193.83981874911316, + "ttft_ms": 395.3119702960262 + }, + { + "e2e_ms": 24814.29543067174, + "request_id": 9, + "tpot_ms": 192.41025790813484, + "ttft_ms": 378.1926763386156 + }, + { + "e2e_ms": 24615.47335754875, + "request_id": 10, + "tpot_ms": 190.95429488392014, + "ttft_ms": 364.2779072908935 + }, + { + "e2e_ms": 24416.46039907787, + "request_id": 11, + "tpot_ms": 189.47286981272285, + "ttft_ms": 353.405932862068 + }, + { + "e2e_ms": 24216.993764693514, + "request_id": 12, + "tpot_ms": 187.96377697549212, + "ttft_ms": 345.59408880601427 + }, + { + "e2e_ms": 24017.241128749425, + "request_id": 13, + "tpot_ms": 186.4682030159872, + "ttft_ms": 335.7793457190477 + }, + { + "e2e_ms": 23816.999491057053, + "request_id": 14, + "tpot_ms": 184.96817994362138, + "ttft_ms": 326.0406382171395 + }, + { + "e2e_ms": 23616.430479744246, + "request_id": 15, + "tpot_ms": 183.46465752234423, + "ttft_ms": 316.41897440652934 + }, + { + "e2e_ms": 23412.66959473249, + "request_id": 16, + "tpot_ms": 181.92890330417333, + "ttft_ms": 307.69887510247963 + }, + { + "e2e_ms": 23315.084360363784, + "request_id": 17, + "tpot_ms": 180.37041078851854, + "ttft_ms": 408.042190221928 + }, + { + "e2e_ms": 23111.095991700124, + "request_id": 18, + "tpot_ms": 178.82629774094855, + "ttft_ms": 400.1561785996568 + }, + { + "e2e_ms": 22907.001651879975, + "request_id": 19, + "tpot_ms": 177.27413055190453, + "ttft_ms": 393.18707178810007 + }, + { + "e2e_ms": 22702.9918523004, + "request_id": 20, + "tpot_ms": 175.71469647400258, + "ttft_ms": 387.22540010207496 + }, + { + "e2e_ms": 22499.024594644914, + "request_id": 21, + "tpot_ms": 174.1539791842547, + "ttft_ms": 381.46923824456815 + }, + { + "e2e_ms": 22294.98921706886, + "request_id": 22, + "tpot_ms": 172.59076723758398, + "ttft_ms": 375.96177789569475 + }, + { + "e2e_ms": 22091.055170736552, + "request_id": 23, + "tpot_ms": 171.02806584364455, + "ttft_ms": 370.49080859369354 + }, + { + "e2e_ms": 21886.829371806663, + "request_id": 24, + "tpot_ms": 169.4599867344259, + "ttft_ms": 365.4110565345743 + }, + { + "e2e_ms": 21682.517380669786, + "request_id": 25, + "tpot_ms": 167.90637996068475, + "ttft_ms": 358.40712566282207 + }, + { + "e2e_ms": 21478.40806232245, + "request_id": 26, + "tpot_ms": 166.35416808544093, + "ttft_ms": 351.4287154714495 + }, + { + "e2e_ms": 21274.47246393178, + "request_id": 27, + "tpot_ms": 164.80170050625136, + "ttft_ms": 344.6564996378552 + }, + { + "e2e_ms": 21070.655727841604, + "request_id": 28, + "tpot_ms": 163.24706150884583, + "ttft_ms": 338.278916218183 + }, + { + "e2e_ms": 20866.83848786135, + "request_id": 29, + "tpot_ms": 161.69251523328387, + "ttft_ms": 331.88905323430083 + }, + { + "e2e_ms": 20663.178771213603, + "request_id": 30, + "tpot_ms": 160.1348533340772, + "ttft_ms": 326.0523977857979 + }, + { + "e2e_ms": 20459.48769489552, + "request_id": 31, + "tpot_ms": 158.57430185728822, + "ttft_ms": 320.55135901991605 + }, + { + "e2e_ms": 20255.951788183756, + "request_id": 32, + "tpot_ms": 157.01541837295076, + "ttft_ms": 314.9936548190073 + }, + { + "e2e_ms": 20052.589009493837, + "request_id": 33, + "tpot_ms": 155.45687267778135, + "ttft_ms": 309.5661794156026 + }, + { + "e2e_ms": 19955.947670796646, + "request_id": 34, + "tpot_ms": 153.8983040696114, + "ttft_ms": 410.8630539559979 + }, + { + "e2e_ms": 19752.847929041352, + "request_id": 35, + "tpot_ms": 152.3343481355919, + "ttft_ms": 406.3857158211803 + }, + { + "e2e_ms": 19549.96119317403, + "request_id": 36, + "tpot_ms": 150.77184485827368, + "ttft_ms": 401.93689617327345 + }, + { + "e2e_ms": 19346.57831151796, + "request_id": 37, + "tpot_ms": 149.20507336304368, + "ttft_ms": 397.53399441140937 + }, + { + "e2e_ms": 19143.22059782458, + "request_id": 38, + "tpot_ms": 147.63442381272193, + "ttft_ms": 393.6487736088932 + }, + { + "e2e_ms": 18940.150721561607, + "request_id": 39, + "tpot_ms": 146.06310547094216, + "ttft_ms": 390.13632675195356 + }, + { + "e2e_ms": 18741.29802135198, + "request_id": 40, + "tpot_ms": 144.5231784918296, + "ttft_ms": 386.85435288961753 + }, + { + "e2e_ms": 18542.463082904305, + "request_id": 41, + "tpot_ms": 142.98167897793616, + "ttft_ms": 383.7898527064141 + }, + { + "e2e_ms": 18344.148533824395, + "request_id": 42, + "tpot_ms": 141.4401875845621, + "ttft_ms": 381.2447105850083 + }, + { + "e2e_ms": 18145.706224501653, + "request_id": 43, + "tpot_ms": 139.89417135289136, + "ttft_ms": 379.1464626844494 + }, + { + "e2e_ms": 17948.37563539231, + "request_id": 44, + "tpot_ms": 138.3533077037345, + "ttft_ms": 377.5055570180292 + }, + { + "e2e_ms": 17750.40475411994, + "request_id": 45, + "tpot_ms": 136.80479670989413, + "ttft_ms": 376.19557196338246 + }, + { + "e2e_ms": 17551.412355033975, + "request_id": 46, + "tpot_ms": 135.2471347218224, + "ttft_ms": 375.0262453625339 + }, + { + "e2e_ms": 17352.31516600567, + "request_id": 47, + "tpot_ms": 133.6870504253856, + "ttft_ms": 374.0597619816981 + }, + { + "e2e_ms": 17152.719400024922, + "request_id": 48, + "tpot_ms": 132.11931086353468, + "ttft_ms": 373.56692035601566 + }, + { + "e2e_ms": 16952.899525195207, + "request_id": 49, + "tpot_ms": 130.53228068835782, + "ttft_ms": 375.29987777376664 + }, + { + "e2e_ms": 16753.147719429508, + "request_id": 50, + "tpot_ms": 128.94354832701373, + "ttft_ms": 377.31708189876304 + }, + { + "e2e_ms": 16553.291092750187, + "request_id": 51, + "tpot_ms": 127.34853489391125, + "ttft_ms": 380.0271612234596 + }, + { + "e2e_ms": 16357.836116284518, + "request_id": 52, + "tpot_ms": 125.77762361641257, + "ttft_ms": 384.07791700011984 + }, + { + "e2e_ms": 16157.72333756749, + "request_id": 53, + "tpot_ms": 124.16553823707912, + "ttft_ms": 388.6999814584407 + }, + { + "e2e_ms": 15952.367175996273, + "request_id": 54, + "tpot_ms": 122.51027887411597, + "ttft_ms": 393.5617589835445 + }, + { + "e2e_ms": 15741.765215271713, + "request_id": 55, + "tpot_ms": 120.81149800475544, + "ttft_ms": 398.70496866777216 + }, + { + "e2e_ms": 15528.71285895543, + "request_id": 56, + "tpot_ms": 119.08582914503211, + "ttft_ms": 404.8125575363528 + }, + { + "e2e_ms": 15314.765605096894, + "request_id": 57, + "tpot_ms": 117.34894161957355, + "ttft_ms": 411.4500194110526 + }, + { + "e2e_ms": 15096.980468100028, + "request_id": 58, + "tpot_ms": 115.58065073324867, + "ttft_ms": 418.2378249774459 + }, + { + "e2e_ms": 14778.441120322277, + "request_id": 59, + "tpot_ms": 113.87954551567164, + "ttft_ms": 315.7388398319796 + }, + { + "e2e_ms": 14557.742348852762, + "request_id": 60, + "tpot_ms": 112.08473611057838, + "ttft_ms": 322.9808628093096 + }, + { + "e2e_ms": 14333.83717181993, + "request_id": 61, + "tpot_ms": 110.26282336485625, + "ttft_ms": 330.4586044831872 + }, + { + "e2e_ms": 14108.97915094099, + "request_id": 62, + "tpot_ms": 108.43298632152924, + "ttft_ms": 337.989888106776 + }, + { + "e2e_ms": 13882.64463939965, + "request_id": 63, + "tpot_ms": 106.58813679106942, + "ttft_ms": 345.9512669338345 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 401.55159425735474, + "offered_request_rate": 3.2, + "request_metrics_sha256": "48ff1d03e0aeb45bb4395dca86e089cdbe1c7de16b8557fbe8f25e8e65c77def", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 26482.398681960134, + "request_id": 0, + "tpot_ms": 206.22623604112295, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26389.27725660841, + "request_id": 1, + "tpot_ms": 204.89007083645959, + "ttft_ms": 368.23826037804076 + }, + { + "e2e_ms": 26186.398050214455, + "request_id": 2, + "tpot_ms": 203.43938315354237, + "ttft_ms": 349.5963897145746 + }, + { + "e2e_ms": 25983.46530187178, + "request_id": 3, + "tpot_ms": 201.98715121187504, + "ttft_ms": 331.0970979636474 + }, + { + "e2e_ms": 25780.42643937068, + "request_id": 4, + "tpot_ms": 200.53276712329495, + "ttft_ms": 312.76501471222184 + }, + { + "e2e_ms": 25577.34505869151, + "request_id": 5, + "tpot_ms": 199.0733214464005, + "ttft_ms": 295.0332349986446 + }, + { + "e2e_ms": 25483.55218185351, + "request_id": 6, + "tpot_ms": 197.69083780569105, + "ttft_ms": 376.81578053074594 + }, + { + "e2e_ms": 25280.28848711639, + "request_id": 7, + "tpot_ms": 196.22290463044078, + "ttft_ms": 359.97959905041466 + }, + { + "e2e_ms": 25076.966707259246, + "request_id": 8, + "tpot_ms": 194.75030755695653, + "ttft_ms": 343.6776475257637 + }, + { + "e2e_ms": 24873.46328021192, + "request_id": 9, + "tpot_ms": 193.27265662306831, + "ttft_ms": 327.83588908224147 + }, + { + "e2e_ms": 24669.88557698392, + "request_id": 10, + "tpot_ms": 191.78984111033463, + "ttft_ms": 312.57575597142215 + }, + { + "e2e_ms": 24466.212431081483, + "request_id": 11, + "tpot_ms": 190.30296300954802, + "ttft_ms": 297.7361288688827 + }, + { + "e2e_ms": 24370.912463364126, + "request_id": 12, + "tpot_ms": 188.79387017231738, + "ttft_ms": 394.09095147981964 + }, + { + "e2e_ms": 24166.86947872558, + "request_id": 13, + "tpot_ms": 187.2967209572732, + "ttft_ms": 380.1859171518851 + }, + { + "e2e_ms": 23962.581993212396, + "request_id": 14, + "tpot_ms": 185.79746051386414, + "ttft_ms": 366.30450795165007 + }, + { + "e2e_ms": 23758.13082088949, + "request_id": 15, + "tpot_ms": 184.29572160562404, + "ttft_ms": 352.57417697523687 + }, + { + "e2e_ms": 23552.083711717114, + "request_id": 16, + "tpot_ms": 182.7751996506789, + "ttft_ms": 339.633356080892 + }, + { + "e2e_ms": 23345.944189149544, + "request_id": 17, + "tpot_ms": 181.25255279330477, + "ttft_ms": 326.8699843998375 + }, + { + "e2e_ms": 23139.783338151214, + "request_id": 18, + "tpot_ms": 179.72694307410856, + "ttft_ms": 314.461567739424 + }, + { + "e2e_ms": 22933.56950157464, + "request_id": 19, + "tpot_ms": 178.1977153273412, + "ttft_ms": 302.45965500230557 + }, + { + "e2e_ms": 22833.726368661068, + "request_id": 20, + "tpot_ms": 176.63828124943853, + "ttft_ms": 400.6646499823745 + }, + { + "e2e_ms": 22627.576073166823, + "request_id": 21, + "tpot_ms": 175.09813975268048, + "ttft_ms": 390.11232457640244 + }, + { + "e2e_ms": 22421.391717712297, + "request_id": 22, + "tpot_ms": 173.55627478074584, + "ttft_ms": 379.7448205575735 + }, + { + "e2e_ms": 22215.258027879143, + "request_id": 23, + "tpot_ms": 172.01247264259936, + "ttft_ms": 369.67400226902566 + }, + { + "e2e_ms": 22008.978461747698, + "request_id": 24, + "tpot_ms": 170.46437149324944, + "ttft_ms": 360.0032821050201 + }, + { + "e2e_ms": 21802.65579951293, + "request_id": 25, + "tpot_ms": 168.91532250977505, + "ttft_ms": 350.4098407714995 + }, + { + "e2e_ms": 21596.434473671965, + "request_id": 26, + "tpot_ms": 167.36573728219415, + "ttft_ms": 340.98583883331025 + }, + { + "e2e_ms": 21390.300007810118, + "request_id": 27, + "tpot_ms": 165.81511332736412, + "ttft_ms": 331.7806152348748 + }, + { + "e2e_ms": 21184.224973098888, + "request_id": 28, + "tpot_ms": 164.26380332238384, + "ttft_ms": 322.72195115614187 + }, + { + "e2e_ms": 20978.149686441437, + "request_id": 29, + "tpot_ms": 162.71174941720716, + "ttft_ms": 313.7575104561279 + }, + { + "e2e_ms": 20772.153161450715, + "request_id": 30, + "tpot_ms": 161.15510654351164, + "ttft_ms": 305.4546304247392 + }, + { + "e2e_ms": 20672.62875180034, + "request_id": 31, + "tpot_ms": 159.59455506672822, + "ttft_ms": 404.12025832585744 + }, + { + "e2e_ms": 20466.69413177714, + "request_id": 32, + "tpot_ms": 158.03533915661183, + "ttft_ms": 396.2060588874419 + }, + { + "e2e_ms": 20260.84607576572, + "request_id": 33, + "tpot_ms": 156.475168525323, + "ttft_ms": 388.49967304969545 + }, + { + "e2e_ms": 20055.02118508905, + "request_id": 34, + "tpot_ms": 154.9132321630232, + "ttft_ms": 381.04070038510326 + }, + { + "e2e_ms": 19849.304647544788, + "request_id": 35, + "tpot_ms": 153.3461190070677, + "ttft_ms": 374.34753364718887 + }, + { + "e2e_ms": 19643.694612944317, + "request_id": 36, + "tpot_ms": 151.77962188577303, + "ttft_ms": 367.6826334511443 + }, + { + "e2e_ms": 19437.836505449406, + "request_id": 37, + "tpot_ms": 150.210341914627, + "ttft_ms": 361.1230822917744 + }, + { + "e2e_ms": 19231.99098193562, + "request_id": 38, + "tpot_ms": 148.63640086508497, + "ttft_ms": 355.16807206983003 + }, + { + "e2e_ms": 19026.289377137648, + "request_id": 39, + "tpot_ms": 147.05985988148976, + "ttft_ms": 349.6871721884496 + }, + { + "e2e_ms": 18822.696360365982, + "request_id": 40, + "tpot_ms": 145.49844654462956, + "ttft_ms": 344.3936491980288 + }, + { + "e2e_ms": 18619.112224474804, + "request_id": 41, + "tpot_ms": 143.93654065290042, + "ttft_ms": 339.17156155644966 + }, + { + "e2e_ms": 18415.78828326805, + "request_id": 42, + "tpot_ms": 142.37219516274803, + "ttft_ms": 334.51949759905375 + }, + { + "e2e_ms": 18212.400461939902, + "request_id": 43, + "tpot_ms": 140.8040960160447, + "ttft_ms": 330.280267902225 + }, + { + "e2e_ms": 18009.568500717836, + "request_id": 44, + "tpot_ms": 139.2366051127279, + "ttft_ms": 326.5196514013962 + }, + { + "e2e_ms": 17806.416393414805, + "request_id": 45, + "tpot_ms": 137.66365486442547, + "ttft_ms": 323.1322256327704 + }, + { + "e2e_ms": 17602.75352720573, + "request_id": 46, + "tpot_ms": 136.08599255484802, + "ttft_ms": 319.83247274003014 + }, + { + "e2e_ms": 17399.03826602459, + "request_id": 47, + "tpot_ms": 134.50648843613666, + "ttft_ms": 316.71423463523496 + }, + { + "e2e_ms": 17195.073716367715, + "request_id": 48, + "tpot_ms": 132.92201961792392, + "ttft_ms": 313.9772248913761 + }, + { + "e2e_ms": 16990.997112286357, + "request_id": 49, + "tpot_ms": 131.3320936883647, + "ttft_ms": 311.8212138640413 + }, + { + "e2e_ms": 16895.41197318666, + "request_id": 50, + "tpot_ms": 129.74336132702078, + "ttft_ms": 418.0050846550216 + }, + { + "e2e_ms": 16691.3169931805, + "request_id": 51, + "tpot_ms": 128.14881090811454, + "ttft_ms": 416.418007849952 + }, + { + "e2e_ms": 16489.422838281163, + "request_id": 52, + "tpot_ms": 126.56215930365238, + "ttft_ms": 416.0286067173118 + }, + { + "e2e_ms": 16285.199782255653, + "request_id": 53, + "tpot_ms": 124.95445752714085, + "ttft_ms": 415.9836763087661 + }, + { + "e2e_ms": 16078.355034803544, + "request_id": 54, + "tpot_ms": 123.3249765236981, + "ttft_ms": 416.08301629388575 + }, + { + "e2e_ms": 15868.887387774763, + "request_id": 55, + "tpot_ms": 121.67321161824907, + "ttft_ms": 416.3895122571297 + }, + { + "e2e_ms": 15658.194542949623, + "request_id": 56, + "tpot_ms": 120.00563624201372, + "ttft_ms": 417.4787402138804 + }, + { + "e2e_ms": 15447.054249353854, + "request_id": 57, + "tpot_ms": 118.33082151865976, + "ttft_ms": 419.0399164840635 + }, + { + "e2e_ms": 15233.99501418892, + "request_id": 58, + "tpot_ms": 116.64063648866745, + "ttft_ms": 420.6341801281539 + }, + { + "e2e_ms": 15019.622333077174, + "request_id": 59, + "tpot_ms": 114.93953127109042, + "ttft_ms": 422.3018616486911 + }, + { + "e2e_ms": 14805.106280675915, + "request_id": 60, + "tpot_ms": 113.23514577866216, + "ttft_ms": 424.2427667858202 + }, + { + "e2e_ms": 14588.987025492997, + "request_id": 61, + "tpot_ms": 111.5171161341123, + "ttft_ms": 426.31327646073555 + }, + { + "e2e_ms": 14372.391348386529, + "request_id": 62, + "tpot_ms": 109.79194855942059, + "ttft_ms": 428.8138813401155 + }, + { + "e2e_ms": 14155.057425949359, + "request_id": 63, + "tpot_ms": 108.05983074662959, + "ttft_ms": 431.45892112740114 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.109375, + "passed": 7 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.40625, + "passed": 26 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 775.6971135139465, + "offered_request_rate": 0.1, + "request_metrics_sha256": "1146954f1d73891536473385a3ecb02b14b7eaeee231291f2cbe6f9d4d222414", + "request_rate_per_gpu": 0.0125, + "requests": [ + { + "e2e_ms": 12579.084570749343, + "request_id": 0, + "tpot_ms": 96.75132177962064, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12682.66705927438, + "request_id": 1, + "tpot_ms": 96.90664947750408, + "ttft_ms": 375.5225756313614 + }, + { + "e2e_ms": 12692.014437909165, + "request_id": 2, + "tpot_ms": 96.91832475611541, + "ttft_ms": 383.3871938825091 + }, + { + "e2e_ms": 12605.650216029226, + "request_id": 3, + "tpot_ms": 96.9198422551915, + "ttft_ms": 296.8302496199051 + }, + { + "e2e_ms": 12614.259349375281, + "request_id": 4, + "tpot_ms": 96.9125118011697, + "ttft_ms": 306.3703506267288 + }, + { + "e2e_ms": 12622.868482721336, + "request_id": 5, + "tpot_ms": 96.9125118011697, + "ttft_ms": 314.97948397278464 + }, + { + "e2e_ms": 12631.47761606615, + "request_id": 6, + "tpot_ms": 96.91251180115991, + "ttft_ms": 323.5886173188405 + }, + { + "e2e_ms": 12640.086749411268, + "request_id": 7, + "tpot_ms": 96.91251180113692, + "ttft_ms": 332.19775066687873 + }, + { + "e2e_ms": 12648.695882756385, + "request_id": 8, + "tpot_ms": 96.91251180113692, + "ttft_ms": 340.80688401199666 + }, + { + "e2e_ms": 12657.305016101503, + "request_id": 9, + "tpot_ms": 96.91251180113692, + "ttft_ms": 349.4160173571146 + }, + { + "e2e_ms": 12665.91414944662, + "request_id": 10, + "tpot_ms": 96.9125118011369, + "ttft_ms": 358.0251507022325 + }, + { + "e2e_ms": 12674.523282791739, + "request_id": 11, + "tpot_ms": 96.91251180113692, + "ttft_ms": 366.63428404735043 + }, + { + "e2e_ms": 12683.132416062932, + "request_id": 12, + "tpot_ms": 96.91251180055482, + "ttft_ms": 375.24341739246836 + }, + { + "e2e_ms": 12692.479794575775, + "request_id": 13, + "tpot_ms": 96.91832475507798, + "ttft_ms": 383.85255068087076 + }, + { + "e2e_ms": 12606.11557253071, + "request_id": 14, + "tpot_ms": 96.91984225378269, + "ttft_ms": 297.2956063003096 + }, + { + "e2e_ms": 12614.724705710416, + "request_id": 15, + "tpot_ms": 96.91251179969895, + "ttft_ms": 306.8357071486503 + }, + { + "e2e_ms": 12623.33383889012, + "request_id": 16, + "tpot_ms": 96.91251179969895, + "ttft_ms": 315.44484032835385 + }, + { + "e2e_ms": 12631.942972069823, + "request_id": 17, + "tpot_ms": 96.91251179969893, + "ttft_ms": 324.0539735080574 + }, + { + "e2e_ms": 12640.552105249526, + "request_id": 18, + "tpot_ms": 96.91251179969893, + "ttft_ms": 332.663106687761 + }, + { + "e2e_ms": 12649.16123842923, + "request_id": 19, + "tpot_ms": 96.91251179969893, + "ttft_ms": 341.2722398674646 + }, + { + "e2e_ms": 12657.770371608933, + "request_id": 20, + "tpot_ms": 96.91251179969893, + "ttft_ms": 349.88137304716815 + }, + { + "e2e_ms": 12666.379504788636, + "request_id": 21, + "tpot_ms": 96.91251179969893, + "ttft_ms": 358.49050622687173 + }, + { + "e2e_ms": 12674.988637968341, + "request_id": 22, + "tpot_ms": 96.91251179969895, + "ttft_ms": 367.0996394065753 + }, + { + "e2e_ms": 12683.597771148045, + "request_id": 23, + "tpot_ms": 96.91251179969895, + "ttft_ms": 375.7087725862789 + }, + { + "e2e_ms": 12692.945149660885, + "request_id": 24, + "tpot_ms": 96.91832475507798, + "ttft_ms": 384.31790576598246 + }, + { + "e2e_ms": 12606.580927732637, + "request_id": 25, + "tpot_ms": 96.91984225470249, + "ttft_ms": 297.7609613854213 + }, + { + "e2e_ms": 12615.190061213469, + "request_id": 26, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.3010623465393 + }, + { + "e2e_ms": 12623.7991946943, + "request_id": 27, + "tpot_ms": 96.9125118021018, + "ttft_ms": 315.9101958273709 + }, + { + "e2e_ms": 12632.408328175132, + "request_id": 28, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.5193293082025 + }, + { + "e2e_ms": 12641.017461655963, + "request_id": 29, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.1284627890341 + }, + { + "e2e_ms": 12649.626595136795, + "request_id": 30, + "tpot_ms": 96.9125118021018, + "ttft_ms": 341.73759626986566 + }, + { + "e2e_ms": 12658.235728617627, + "request_id": 31, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.34672975069725 + }, + { + "e2e_ms": 12666.844862098458, + "request_id": 32, + "tpot_ms": 96.9125118021018, + "ttft_ms": 358.95586323152884 + }, + { + "e2e_ms": 12675.45399557929, + "request_id": 33, + "tpot_ms": 96.9125118021018, + "ttft_ms": 367.5649967123604 + }, + { + "e2e_ms": 12684.063129060121, + "request_id": 34, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.174130193192 + }, + { + "e2e_ms": 12693.41050787034, + "request_id": 35, + "tpot_ms": 96.9183247574513, + "ttft_ms": 384.7832636740236 + }, + { + "e2e_ms": 12607.04628612126, + "request_id": 36, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.22631959211776 + }, + { + "e2e_ms": 12615.655419602092, + "request_id": 37, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.76642073516314 + }, + { + "e2e_ms": 12624.264553082923, + "request_id": 38, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.3755542159947 + }, + { + "e2e_ms": 12632.873686563755, + "request_id": 39, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.9846876968263 + }, + { + "e2e_ms": 12641.482820044586, + "request_id": 40, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.5938211776579 + }, + { + "e2e_ms": 12650.091953525418, + "request_id": 41, + "tpot_ms": 96.9125118021018, + "ttft_ms": 342.2029546584895 + }, + { + "e2e_ms": 12658.70108700625, + "request_id": 42, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.8120881393211 + }, + { + "e2e_ms": 12667.310220487081, + "request_id": 43, + "tpot_ms": 96.9125118021018, + "ttft_ms": 359.42122162015266 + }, + { + "e2e_ms": 12675.919353967913, + "request_id": 44, + "tpot_ms": 96.9125118021018, + "ttft_ms": 368.03035510098425 + }, + { + "e2e_ms": 12684.528487448744, + "request_id": 45, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.63948858181584 + }, + { + "e2e_ms": 12693.875866258964, + "request_id": 46, + "tpot_ms": 96.91832475745132, + "ttft_ms": 385.2486220626474 + }, + { + "e2e_ms": 12607.511644509885, + "request_id": 47, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.6916779807416 + }, + { + "e2e_ms": 12616.120777990716, + "request_id": 48, + "tpot_ms": 96.9125118021018, + "ttft_ms": 308.23177912378696 + }, + { + "e2e_ms": 12624.729911471548, + "request_id": 49, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.84091260461855 + }, + { + "e2e_ms": 12633.339044941862, + "request_id": 50, + "tpot_ms": 96.91251180201898, + "ttft_ms": 325.45004608545014 + }, + { + "e2e_ms": 12641.94817818975, + "request_id": 51, + "tpot_ms": 96.91251180018479, + "ttft_ms": 334.0591795662817 + }, + { + "e2e_ms": 12650.557311437638, + "request_id": 52, + "tpot_ms": 96.91251179996682, + "ttft_ms": 342.66831284185173 + }, + { + "e2e_ms": 12659.166444685525, + "request_id": 53, + "tpot_ms": 96.91251179996682, + "ttft_ms": 351.277446089739 + }, + { + "e2e_ms": 12667.775577933411, + "request_id": 54, + "tpot_ms": 96.91251179996681, + "ttft_ms": 359.88657933762624 + }, + { + "e2e_ms": 12676.3847111813, + "request_id": 55, + "tpot_ms": 96.91251179996682, + "ttft_ms": 368.4957125855135 + }, + { + "e2e_ms": 12684.993844429187, + "request_id": 56, + "tpot_ms": 96.91251179996682, + "ttft_ms": 377.10484583340076 + }, + { + "e2e_ms": 12694.3412230047, + "request_id": 57, + "tpot_ms": 96.91832475530246, + "ttft_ms": 385.713979081288 + }, + { + "e2e_ms": 12607.977001027848, + "request_id": 58, + "tpot_ms": 96.9198422540282, + "ttft_ms": 299.1570347662673 + }, + { + "e2e_ms": 12616.586134275734, + "request_id": 59, + "tpot_ms": 96.91251179996681, + "ttft_ms": 308.6971356799495 + }, + { + "e2e_ms": 12625.195267523623, + "request_id": 60, + "tpot_ms": 96.91251179996682, + "ttft_ms": 317.30626892783675 + }, + { + "e2e_ms": 12633.80440077151, + "request_id": 61, + "tpot_ms": 96.91251179996682, + "ttft_ms": 325.915402175724 + }, + { + "e2e_ms": 12642.413534019397, + "request_id": 62, + "tpot_ms": 96.91251179996682, + "ttft_ms": 334.52453542361127 + }, + { + "e2e_ms": 12436.81062947212, + "request_id": 63, + "tpot_ms": 95.22580284094978, + "ttft_ms": 343.1336686714985 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 404.65002274513245, + "offered_request_rate": 0.2, + "request_metrics_sha256": "cf8a095cdf380d42316d1ad169ce9b2b45ae7d64348cd38271a332663de349ad", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 12823.625057543633, + "request_id": 0, + "tpot_ms": 98.67683742367018, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12920.533566322107, + "request_id": 1, + "tpot_ms": 99.10315669196875, + "ttft_ms": 334.4326664420745 + }, + { + "e2e_ms": 12921.724204826443, + "request_id": 2, + "tpot_ms": 99.22365133274592, + "ttft_ms": 320.32048556771286 + }, + { + "e2e_ms": 12922.91484333078, + "request_id": 3, + "tpot_ms": 99.22887081733072, + "ttft_ms": 320.84824952977976 + }, + { + "e2e_ms": 12924.105481838353, + "request_id": 4, + "tpot_ms": 99.22887081736695, + "ttft_ms": 322.0388880327505 + }, + { + "e2e_ms": 12925.296120352506, + "request_id": 5, + "tpot_ms": 99.22887081744425, + "ttft_ms": 323.2295265370873 + }, + { + "e2e_ms": 12926.486758866658, + "request_id": 6, + "tpot_ms": 99.22887081752154, + "ttft_ms": 324.420165041424 + }, + { + "e2e_ms": 12927.677397380812, + "request_id": 7, + "tpot_ms": 99.22887081749418, + "ttft_ms": 325.6108035590515 + }, + { + "e2e_ms": 12928.868035894964, + "request_id": 8, + "tpot_ms": 99.22887081749417, + "ttft_ms": 326.8014420732044 + }, + { + "e2e_ms": 12930.058674409118, + "request_id": 9, + "tpot_ms": 99.22887081749418, + "ttft_ms": 327.9920805873573 + }, + { + "e2e_ms": 12931.24931292327, + "request_id": 10, + "tpot_ms": 99.22887081749417, + "ttft_ms": 329.1827191015102 + }, + { + "e2e_ms": 12932.439951441467, + "request_id": 11, + "tpot_ms": 99.22887081752602, + "ttft_ms": 330.3733576156631 + }, + { + "e2e_ms": 12933.63058995827, + "request_id": 12, + "tpot_ms": 99.22887081754688, + "ttft_ms": 331.563996129816 + }, + { + "e2e_ms": 12934.821228475073, + "request_id": 13, + "tpot_ms": 99.22887081758537, + "ttft_ms": 332.7546346417307 + }, + { + "e2e_ms": 12936.011866991876, + "request_id": 14, + "tpot_ms": 99.22887081758537, + "ttft_ms": 333.94527315853395 + }, + { + "e2e_ms": 12937.20250550868, + "request_id": 15, + "tpot_ms": 99.22887081758537, + "ttft_ms": 335.1359116753372 + }, + { + "e2e_ms": 12938.393144025482, + "request_id": 16, + "tpot_ms": 99.22887081758537, + "ttft_ms": 336.3265501921404 + }, + { + "e2e_ms": 12939.583782542286, + "request_id": 17, + "tpot_ms": 99.22887081758537, + "ttft_ms": 337.51718870894365 + }, + { + "e2e_ms": 12940.77442105909, + "request_id": 18, + "tpot_ms": 99.22887081758537, + "ttft_ms": 338.7078272257469 + }, + { + "e2e_ms": 12941.965059575892, + "request_id": 19, + "tpot_ms": 99.22887081758537, + "ttft_ms": 339.8984657425501 + }, + { + "e2e_ms": 12943.155698092696, + "request_id": 20, + "tpot_ms": 99.22887081758537, + "ttft_ms": 341.08910425935335 + }, + { + "e2e_ms": 12944.346336609498, + "request_id": 21, + "tpot_ms": 99.22887081758537, + "ttft_ms": 342.2797427761566 + }, + { + "e2e_ms": 12945.536975126302, + "request_id": 22, + "tpot_ms": 99.22887081758537, + "ttft_ms": 343.4703812929598 + }, + { + "e2e_ms": 12946.727613643105, + "request_id": 23, + "tpot_ms": 99.22887081758537, + "ttft_ms": 344.66101980976305 + }, + { + "e2e_ms": 12947.918252142103, + "request_id": 24, + "tpot_ms": 99.22887081744517, + "ttft_ms": 345.8516583265663 + }, + { + "e2e_ms": 12949.10889064036, + "request_id": 25, + "tpot_ms": 99.22887081729912, + "ttft_ms": 347.0422968433695 + }, + { + "e2e_ms": 12950.299529138618, + "request_id": 26, + "tpot_ms": 99.2288708172622, + "ttft_ms": 348.23293534631716 + }, + { + "e2e_ms": 12951.490167636877, + "request_id": 27, + "tpot_ms": 99.22887081726222, + "ttft_ms": 349.42357384457523 + }, + { + "e2e_ms": 12952.680806135135, + "request_id": 28, + "tpot_ms": 99.22887081726222, + "ttft_ms": 350.6142123428333 + }, + { + "e2e_ms": 12953.871444633392, + "request_id": 29, + "tpot_ms": 99.2288708172622, + "ttft_ms": 351.80485084109137 + }, + { + "e2e_ms": 12955.06208313165, + "request_id": 30, + "tpot_ms": 99.2288708172622, + "ttft_ms": 352.99548933934943 + }, + { + "e2e_ms": 12956.25272162991, + "request_id": 31, + "tpot_ms": 99.22887081726222, + "ttft_ms": 354.1861278376075 + }, + { + "e2e_ms": 12957.443360128167, + "request_id": 32, + "tpot_ms": 99.22887081726222, + "ttft_ms": 355.37676633586557 + }, + { + "e2e_ms": 12958.633998626425, + "request_id": 33, + "tpot_ms": 99.2288708172622, + "ttft_ms": 356.56740483412364 + }, + { + "e2e_ms": 12959.824637124682, + "request_id": 34, + "tpot_ms": 99.2288708172622, + "ttft_ms": 357.7580433323817 + }, + { + "e2e_ms": 12961.015275622942, + "request_id": 35, + "tpot_ms": 99.22887081726222, + "ttft_ms": 358.94868183063977 + }, + { + "e2e_ms": 12962.2059141212, + "request_id": 36, + "tpot_ms": 99.22887081726222, + "ttft_ms": 360.13932032889784 + }, + { + "e2e_ms": 12963.396552619457, + "request_id": 37, + "tpot_ms": 99.2288708172622, + "ttft_ms": 361.3299588271559 + }, + { + "e2e_ms": 12964.587191117715, + "request_id": 38, + "tpot_ms": 99.2288708172622, + "ttft_ms": 362.520597325414 + }, + { + "e2e_ms": 12965.777829615974, + "request_id": 39, + "tpot_ms": 99.22887081726222, + "ttft_ms": 363.71123582367204 + }, + { + "e2e_ms": 12966.968468114232, + "request_id": 40, + "tpot_ms": 99.22887081726222, + "ttft_ms": 364.9018743219301 + }, + { + "e2e_ms": 12968.15910661249, + "request_id": 41, + "tpot_ms": 99.2288708172622, + "ttft_ms": 366.0925128201882 + }, + { + "e2e_ms": 12969.349745110747, + "request_id": 42, + "tpot_ms": 99.2288708172622, + "ttft_ms": 367.28315131844624 + }, + { + "e2e_ms": 12970.540383609006, + "request_id": 43, + "tpot_ms": 99.22887081726222, + "ttft_ms": 368.4737898167043 + }, + { + "e2e_ms": 12971.731022107264, + "request_id": 44, + "tpot_ms": 99.22887081726222, + "ttft_ms": 369.6644283149624 + }, + { + "e2e_ms": 12972.921660605522, + "request_id": 45, + "tpot_ms": 99.2288708172622, + "ttft_ms": 370.85506681322045 + }, + { + "e2e_ms": 12974.11229910378, + "request_id": 46, + "tpot_ms": 99.2288708172622, + "ttft_ms": 372.0457053114785 + }, + { + "e2e_ms": 12975.302937602039, + "request_id": 47, + "tpot_ms": 99.22887081726222, + "ttft_ms": 373.2363438097366 + }, + { + "e2e_ms": 12976.493576100296, + "request_id": 48, + "tpot_ms": 99.22887081726222, + "ttft_ms": 374.42698230799465 + }, + { + "e2e_ms": 12977.684214615067, + "request_id": 49, + "tpot_ms": 99.22887081739223, + "ttft_ms": 375.6176208062527 + }, + { + "e2e_ms": 12978.874853139188, + "request_id": 50, + "tpot_ms": 99.22887081759589, + "ttft_ms": 376.8082593045108 + }, + { + "e2e_ms": 12980.06549166331, + "request_id": 51, + "tpot_ms": 99.22887081779955, + "ttft_ms": 377.99889780276885 + }, + { + "e2e_ms": 12981.256130187432, + "request_id": 52, + "tpot_ms": 99.22887081784184, + "ttft_ms": 379.189536321519 + }, + { + "e2e_ms": 12982.446768711554, + "request_id": 53, + "tpot_ms": 99.22887081784184, + "ttft_ms": 380.3801748456408 + }, + { + "e2e_ms": 12983.637407235676, + "request_id": 54, + "tpot_ms": 99.22887081784184, + "ttft_ms": 381.5708133697626 + }, + { + "e2e_ms": 12984.828045759798, + "request_id": 55, + "tpot_ms": 99.22887081784184, + "ttft_ms": 382.76145189388444 + }, + { + "e2e_ms": 12986.01868428392, + "request_id": 56, + "tpot_ms": 99.22887081784184, + "ttft_ms": 383.95209041800626 + }, + { + "e2e_ms": 12987.209322808041, + "request_id": 57, + "tpot_ms": 99.22887081784184, + "ttft_ms": 385.1427289421281 + }, + { + "e2e_ms": 12988.399961332163, + "request_id": 58, + "tpot_ms": 99.22887081784184, + "ttft_ms": 386.3333674662499 + }, + { + "e2e_ms": 12989.881679406835, + "request_id": 59, + "tpot_ms": 99.2311627828068, + "ttft_ms": 387.52400599037173 + }, + { + "e2e_ms": 12991.363397481507, + "request_id": 60, + "tpot_ms": 99.23345474777176, + "ttft_ms": 388.71464451449356 + }, + { + "e2e_ms": 12896.836165730803, + "request_id": 61, + "tpot_ms": 99.23492287861664, + "ttft_ms": 294.0009601464908 + }, + { + "e2e_ms": 12689.81487892944, + "request_id": 62, + "tpot_ms": 97.59316693471084, + "ttft_ms": 295.4826782211626 + }, + { + "e2e_ms": 12448.118757066311, + "request_id": 63, + "tpot_ms": 95.67691266262345, + "ttft_ms": 297.1508489131338 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 223.00242853164673, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e09d047b2039786ee02cd4c59c081ca95e5197ae65723b6170e43f6c0924921a", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 13556.392075191316, + "request_id": 0, + "tpot_ms": 104.4466564602661, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 13715.056219396885, + "request_id": 1, + "tpot_ms": 105.11756526634255, + "ttft_ms": 365.12543057138026 + }, + { + "e2e_ms": 13775.965021971426, + "request_id": 2, + "tpot_ms": 105.6336255570719, + "ttft_ms": 360.4945762232949 + }, + { + "e2e_ms": 13838.18765138338, + "request_id": 3, + "tpot_ms": 106.07242606904647, + "ttft_ms": 366.98954061447785 + }, + { + "e2e_ms": 13802.282961907104, + "request_id": 4, + "tpot_ms": 106.21712281117094, + "ttft_ms": 312.70836488839393 + }, + { + "e2e_ms": 13864.505591319055, + "request_id": 5, + "tpot_ms": 106.32495967758555, + "ttft_ms": 361.2357122656906 + }, + { + "e2e_ms": 13828.78708091747, + "request_id": 6, + "tpot_ms": 106.35892091285005, + "ttft_ms": 321.20412498551156 + }, + { + "e2e_ms": 13892.323156276025, + "request_id": 7, + "tpot_ms": 106.3810710779423, + "ttft_ms": 381.9271293773525 + }, + { + "e2e_ms": 13856.60464587857, + "request_id": 8, + "tpot_ms": 106.3914161711828, + "ttft_ms": 344.89479213835494 + }, + { + "e2e_ms": 13820.886135481267, + "request_id": 9, + "tpot_ms": 106.39288214818258, + "ttft_ms": 308.9901026620794 + }, + { + "e2e_ms": 13883.108764897528, + "request_id": 10, + "tpot_ms": 106.39288214821651, + "ttft_ms": 371.21273207403146 + }, + { + "e2e_ms": 13847.390254500226, + "request_id": 11, + "tpot_ms": 106.39288214825024, + "ttft_ms": 335.4942216724446 + }, + { + "e2e_ms": 13811.485565028235, + "request_id": 12, + "tpot_ms": 106.39288214828399, + "ttft_ms": 299.589532196169 + }, + { + "e2e_ms": 13873.708194444496, + "request_id": 13, + "tpot_ms": 106.38107407728566, + "ttft_ms": 363.31178662921815 + }, + { + "e2e_ms": 13837.989684047194, + "request_id": 14, + "tpot_ms": 106.38107407728566, + "ttft_ms": 327.5932762319158 + }, + { + "e2e_ms": 13901.52575941027, + "request_id": 15, + "tpot_ms": 106.39141617135505, + "ttft_ms": 389.81590564817736 + }, + { + "e2e_ms": 13865.807249012967, + "request_id": 16, + "tpot_ms": 106.39141617135505, + "ttft_ms": 354.097395250875 + }, + { + "e2e_ms": 13830.088738615665, + "request_id": 17, + "tpot_ms": 106.3928821483211, + "ttft_ms": 318.19270577888403 + }, + { + "e2e_ms": 13893.62481397874, + "request_id": 18, + "tpot_ms": 106.40322424239051, + "ttft_ms": 380.4153351951456 + }, + { + "e2e_ms": 13857.906303581438, + "request_id": 19, + "tpot_ms": 106.40322424239051, + "ttft_ms": 344.69682479784325 + }, + { + "e2e_ms": 13822.187793184135, + "request_id": 20, + "tpot_ms": 106.40469021935657, + "ttft_ms": 308.79213532585226 + }, + { + "e2e_ms": 13884.410422593844, + "request_id": 21, + "tpot_ms": 106.39288214826952, + "ttft_ms": 372.51438976361584 + }, + { + "e2e_ms": 13848.691912190276, + "request_id": 22, + "tpot_ms": 106.39288214822018, + "ttft_ms": 336.7958793663135 + }, + { + "e2e_ms": 13812.78722271334, + "request_id": 23, + "tpot_ms": 106.39288214818123, + "ttft_ms": 300.8911898943225 + }, + { + "e2e_ms": 13875.009852123696, + "request_id": 24, + "tpot_ms": 106.38107407709929, + "ttft_ms": 364.6134443320861 + }, + { + "e2e_ms": 13839.291341720127, + "request_id": 25, + "tpot_ms": 106.38107407704995, + "ttft_ms": 328.89493393478375 + }, + { + "e2e_ms": 13902.827417078257, + "request_id": 26, + "tpot_ms": 106.39141617110043, + "ttft_ms": 391.11756334850156 + }, + { + "e2e_ms": 13867.108906674688, + "request_id": 27, + "tpot_ms": 106.39141617110043, + "ttft_ms": 355.39905294493224 + }, + { + "e2e_ms": 13831.390396271117, + "request_id": 28, + "tpot_ms": 106.39288214805607, + "ttft_ms": 319.49436346799587 + }, + { + "e2e_ms": 13894.926471629247, + "request_id": 29, + "tpot_ms": 106.40322424213302, + "ttft_ms": 381.7169928783528 + }, + { + "e2e_ms": 13859.207961225678, + "request_id": 30, + "tpot_ms": 106.40322424213302, + "ttft_ms": 345.9984824747835 + }, + { + "e2e_ms": 13823.48945082211, + "request_id": 31, + "tpot_ms": 106.40469021908869, + "ttft_ms": 310.0937929978471 + }, + { + "e2e_ms": 13885.712080232466, + "request_id": 32, + "tpot_ms": 106.39288214805609, + "ttft_ms": 373.8160474293437 + }, + { + "e2e_ms": 13849.993569828897, + "request_id": 33, + "tpot_ms": 106.39288214805609, + "ttft_ms": 338.0975370257744 + }, + { + "e2e_ms": 13814.08888035196, + "request_id": 34, + "tpot_ms": 106.39288214805607, + "ttft_ms": 302.192847548838 + }, + { + "e2e_ms": 13876.311509762318, + "request_id": 35, + "tpot_ms": 106.3810740770235, + "ttft_ms": 365.9151019803346 + }, + { + "e2e_ms": 13840.592999358747, + "request_id": 36, + "tpot_ms": 106.38107407702348, + "ttft_ms": 330.1965915767653 + }, + { + "e2e_ms": 13904.129074716877, + "request_id": 37, + "tpot_ms": 106.39141617110043, + "ttft_ms": 392.41922098712223 + }, + { + "e2e_ms": 13868.410564313308, + "request_id": 38, + "tpot_ms": 106.39141617110043, + "ttft_ms": 356.7007105835529 + }, + { + "e2e_ms": 13832.69205390974, + "request_id": 39, + "tpot_ms": 106.39288214805609, + "ttft_ms": 320.79602110661654 + }, + { + "e2e_ms": 13896.228129267869, + "request_id": 40, + "tpot_ms": 106.40322424213304, + "ttft_ms": 383.0186505169735 + }, + { + "e2e_ms": 13860.509618864298, + "request_id": 41, + "tpot_ms": 106.40322424213302, + "ttft_ms": 347.30014011340415 + }, + { + "e2e_ms": 13824.79110846073, + "request_id": 42, + "tpot_ms": 106.40469021908868, + "ttft_ms": 311.3954506364678 + }, + { + "e2e_ms": 13888.32718381886, + "request_id": 43, + "tpot_ms": 106.40322424213302, + "ttft_ms": 375.1177050679644 + }, + { + "e2e_ms": 13852.60867341529, + "request_id": 44, + "tpot_ms": 106.40322424213304, + "ttft_ms": 339.39919466439505 + }, + { + "e2e_ms": 13816.89016301172, + "request_id": 45, + "tpot_ms": 106.40469021908868, + "ttft_ms": 303.4945051874587 + }, + { + "e2e_ms": 13879.11279242067, + "request_id": 46, + "tpot_ms": 106.392882148045, + "ttft_ms": 367.2167596189553 + }, + { + "e2e_ms": 13843.394282013549, + "request_id": 47, + "tpot_ms": 106.39288214801702, + "ttft_ms": 331.49824921538595 + }, + { + "e2e_ms": 13807.489592530403, + "request_id": 48, + "tpot_ms": 106.39288214796812, + "ttft_ms": 295.5935597384496 + }, + { + "e2e_ms": 13869.712221935884, + "request_id": 49, + "tpot_ms": 106.38107407689715, + "ttft_ms": 359.3158141699462 + }, + { + "e2e_ms": 13833.993711528763, + "request_id": 50, + "tpot_ms": 106.38107407686918, + "ttft_ms": 323.59730376637685 + }, + { + "e2e_ms": 13897.5297868798, + "request_id": 51, + "tpot_ms": 106.3914161708903, + "ttft_ms": 385.8199331767338 + }, + { + "e2e_ms": 13861.81127647268, + "request_id": 52, + "tpot_ms": 106.39141617089378, + "ttft_ms": 350.1014227691712 + }, + { + "e2e_ms": 13826.092766065556, + "request_id": 53, + "tpot_ms": 106.39288214787034, + "ttft_ms": 314.1967332860247 + }, + { + "e2e_ms": 13889.628841416596, + "request_id": 54, + "tpot_ms": 106.40322424192983, + "ttft_ms": 376.41936269150733 + }, + { + "e2e_ms": 13853.910331009472, + "request_id": 55, + "tpot_ms": 106.4032242419298, + "ttft_ms": 340.7008522843853 + }, + { + "e2e_ms": 13818.191820602351, + "request_id": 56, + "tpot_ms": 106.4046902189064, + "ttft_ms": 304.7961628012388 + }, + { + "e2e_ms": 13880.414450007833, + "request_id": 57, + "tpot_ms": 106.39288214787034, + "ttft_ms": 368.5184172283016 + }, + { + "e2e_ms": 13844.695939600711, + "request_id": 58, + "tpot_ms": 106.39288214787034, + "ttft_ms": 332.79990682117955 + }, + { + "e2e_ms": 13600.134841430987, + "request_id": 59, + "tpot_ms": 104.74991829994453, + "ttft_ms": 296.89521733803304 + }, + { + "e2e_ms": 13451.749583806417, + "request_id": 60, + "tpot_ms": 103.07978040977419, + "ttft_ms": 360.61747176509584 + }, + { + "e2e_ms": 13168.506714577745, + "request_id": 61, + "tpot_ms": 101.13076971039189, + "ttft_ms": 324.8989613579738 + }, + { + "e2e_ms": 12970.21046401943, + "request_id": 62, + "tpot_ms": 99.07943994689744, + "ttft_ms": 387.1215907634564 + }, + { + "e2e_ms": 12659.030247905519, + "request_id": 63, + "tpot_ms": 96.91045013818255, + "ttft_ms": 351.4030803563344 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 142.2287290096283, + "offered_request_rate": 0.8, + "request_metrics_sha256": "6b246d56546536e46b4e8adfb23076b3cbf053da4b0af0288346df10122d6022", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 15423.90313583537, + "request_id": 0, + "tpot_ms": 119.15146796140039, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 15680.782114004433, + "request_id": 1, + "tpot_ms": 120.47488426274414, + "ttft_ms": 380.47181263592745 + }, + { + "e2e_ms": 15824.99160915063, + "request_id": 2, + "tpot_ms": 121.60222806639713, + "ttft_ms": 381.5086447181937 + }, + { + "e2e_ms": 15964.925902693089, + "request_id": 3, + "tpot_ms": 122.6571852387084, + "ttft_ms": 387.4633773771219 + }, + { + "e2e_ms": 15994.3969757696, + "request_id": 4, + "tpot_ms": 123.48492728290344, + "ttft_ms": 311.8112108408626 + }, + { + "e2e_ms": 16132.273068839513, + "request_id": 5, + "tpot_ms": 124.38017511477935, + "ttft_ms": 335.9908292625358 + }, + { + "e2e_ms": 16268.09096143688, + "request_id": 6, + "tpot_ms": 125.15372351796954, + "ttft_ms": 373.5680746547478 + }, + { + "e2e_ms": 16291.239371010352, + "request_id": 7, + "tpot_ms": 125.67763083904141, + "ttft_ms": 330.18025445209173 + }, + { + "e2e_ms": 16612.993923996273, + "request_id": 8, + "tpot_ms": 127.71634872045071, + "ttft_ms": 393.0176364990334 + }, + { + "e2e_ms": 16636.356874836954, + "request_id": 9, + "tpot_ms": 128.11558123152213, + "ttft_ms": 365.67805843364454 + }, + { + "e2e_ms": 16659.624174048608, + "request_id": 10, + "tpot_ms": 128.3236060799542, + "ttft_ms": 362.52620189442376 + }, + { + "e2e_ms": 16683.03953534565, + "request_id": 11, + "tpot_ms": 128.34365422438844, + "ttft_ms": 383.3954488483169 + }, + { + "e2e_ms": 16598.14552827833, + "request_id": 12, + "tpot_ms": 128.20865630197483, + "ttft_ms": 315.6461779275297 + }, + { + "e2e_ms": 16621.508479120537, + "request_id": 13, + "tpot_ms": 128.0747814081899, + "ttft_ms": 356.0112402804201 + }, + { + "e2e_ms": 16644.871429963147, + "request_id": 14, + "tpot_ms": 127.97565080664988, + "ttft_ms": 391.9637775186118 + }, + { + "e2e_ms": 16559.977422897646, + "request_id": 15, + "tpot_ms": 127.91134905305496, + "ttft_ms": 315.2360931596654 + }, + { + "e2e_ms": 16583.340373740255, + "request_id": 16, + "tpot_ms": 127.86325360239432, + "ttft_ms": 344.7071662361765 + }, + { + "e2e_ms": 16606.703324582868, + "request_id": 17, + "tpot_ms": 127.81515815173371, + "ttft_ms": 374.17823931268754 + }, + { + "e2e_ms": 16630.066275425477, + "request_id": 18, + "tpot_ms": 127.7832690040065, + "ttft_ms": 401.59111191665176 + }, + { + "e2e_ms": 16354.971144941388, + "request_id": 19, + "tpot_ms": 126.2873116642462, + "ttft_ms": 316.48256358212024 + }, + { + "e2e_ms": 16568.46874002116, + "request_id": 20, + "tpot_ms": 127.78612414854776, + "ttft_ms": 339.6309731555931 + }, + { + "e2e_ms": 16591.83169086377, + "request_id": 21, + "tpot_ms": 127.78612414856296, + "ttft_ms": 362.9939239962745 + }, + { + "e2e_ms": 16615.19464170638, + "request_id": 22, + "tpot_ms": 127.78612414857815, + "ttft_ms": 386.3568748369559 + }, + { + "e2e_ms": 16339.951449136564, + "request_id": 23, + "tpot_ms": 126.28847750749865, + "ttft_ms": 301.3148056842354 + }, + { + "e2e_ms": 16553.449044216337, + "request_id": 24, + "tpot_ms": 127.78518800972483, + "ttft_ms": 324.73016698128276 + }, + { + "e2e_ms": 16576.811995058946, + "request_id": 25, + "tpot_ms": 127.78477532898123, + "ttft_ms": 348.1455282783301 + }, + { + "e2e_ms": 16600.174945901555, + "request_id": 26, + "tpot_ms": 127.78477532898441, + "ttft_ms": 371.50847912053564 + }, + { + "e2e_ms": 16623.53789674417, + "request_id": 27, + "tpot_ms": 127.78477532898442, + "ttft_ms": 394.87142996314617 + }, + { + "e2e_ms": 16348.29470417435, + "request_id": 28, + "tpot_ms": 126.28596284469846, + "ttft_ms": 309.97742289764574 + }, + { + "e2e_ms": 16561.79229925412, + "request_id": 29, + "tpot_ms": 127.78308602766822, + "ttft_ms": 333.34037374025627 + }, + { + "e2e_ms": 16585.155250096734, + "request_id": 30, + "tpot_ms": 127.78308602766825, + "ttft_ms": 356.7033245828668 + }, + { + "e2e_ms": 16608.518200939343, + "request_id": 31, + "tpot_ms": 127.78308602766823, + "ttft_ms": 380.06627542547733 + }, + { + "e2e_ms": 16631.881151781956, + "request_id": 32, + "tpot_ms": 127.78360948578973, + "ttft_ms": 403.36274708666053 + }, + { + "e2e_ms": 16356.786021297865, + "request_id": 33, + "tpot_ms": 126.28596284469846, + "ttft_ms": 318.4687400211601 + }, + { + "e2e_ms": 16570.283616377637, + "request_id": 34, + "tpot_ms": 127.78308602766825, + "ttft_ms": 341.83169086377063 + }, + { + "e2e_ms": 16593.646567220247, + "request_id": 35, + "tpot_ms": 127.78308602766823, + "ttft_ms": 365.19464170638116 + }, + { + "e2e_ms": 16617.00951806286, + "request_id": 36, + "tpot_ms": 127.78477532898444, + "ttft_ms": 388.34305128183644 + }, + { + "e2e_ms": 16341.766325493041, + "request_id": 37, + "tpot_ms": 126.28596284469846, + "ttft_ms": 303.449044216336 + }, + { + "e2e_ms": 16555.263920572714, + "request_id": 38, + "tpot_ms": 127.78308602766747, + "ttft_ms": 326.81199505894654 + }, + { + "e2e_ms": 16578.62687141207, + "request_id": 39, + "tpot_ms": 127.78308602764184, + "ttft_ms": 350.17494590155707 + }, + { + "e2e_ms": 16601.989822251424, + "request_id": 40, + "tpot_ms": 127.78308602761618, + "ttft_ms": 373.5378967441676 + }, + { + "e2e_ms": 16625.35277309078, + "request_id": 41, + "tpot_ms": 127.78477532890675, + "ttft_ms": 396.6863063196229 + }, + { + "e2e_ms": 16350.109580518321, + "request_id": 42, + "tpot_ms": 126.2859628446, + "ttft_ms": 311.79229925412244 + }, + { + "e2e_ms": 16563.607175594883, + "request_id": 43, + "tpot_ms": 127.78308602754448, + "ttft_ms": 335.15525009673297 + }, + { + "e2e_ms": 16586.970126434237, + "request_id": 44, + "tpot_ms": 127.78308602751885, + "ttft_ms": 358.5182009393435 + }, + { + "e2e_ms": 16610.333077273594, + "request_id": 45, + "tpot_ms": 127.78308602749323, + "ttft_ms": 381.88115178195403 + }, + { + "e2e_ms": 16633.69602811295, + "request_id": 46, + "tpot_ms": 127.78360948558907, + "ttft_ms": 405.17762344313724 + }, + { + "e2e_ms": 16548.802021044266, + "request_id": 47, + "tpot_ms": 127.78360948556401, + "ttft_ms": 320.2836163776368 + }, + { + "e2e_ms": 16572.164971883623, + "request_id": 48, + "tpot_ms": 127.7836094855384, + "ttft_ms": 343.64656722024733 + }, + { + "e2e_ms": 16595.52792272298, + "request_id": 49, + "tpot_ms": 127.78360948551277, + "ttft_ms": 367.00951806285786 + }, + { + "e2e_ms": 16618.890873562337, + "request_id": 50, + "tpot_ms": 127.78529878680334, + "ttft_ms": 390.15792763831314 + }, + { + "e2e_ms": 16343.647680989874, + "request_id": 51, + "tpot_ms": 126.28648630249732, + "ttft_ms": 305.26392057271323 + }, + { + "e2e_ms": 16199.563507361632, + "request_id": 52, + "tpot_ms": 124.96800500747688, + "ttft_ms": 328.6268714120695 + }, + { + "e2e_ms": 16032.332947106512, + "request_id": 53, + "tpot_ms": 123.46726869964634, + "ttft_ms": 351.9898222514257 + }, + { + "e2e_ms": 15838.885472588829, + "request_id": 54, + "tpot_ms": 121.7600999960476, + "ttft_ms": 375.35277309078197 + }, + { + "e2e_ms": 15619.209002299955, + "request_id": 55, + "tpot_ms": 119.84809306800307, + "ttft_ms": 398.5011826635656 + }, + { + "e2e_ms": 15285.473398871076, + "request_id": 56, + "tpot_ms": 117.8887104194976, + "ttft_ms": 313.60717559488194 + }, + { + "e2e_ms": 15049.070462905931, + "request_id": 57, + "tpot_ms": 115.84330973599758, + "ttft_ms": 336.9701264342382 + }, + { + "e2e_ms": 14793.47811125659, + "request_id": 58, + "tpot_ms": 113.6468112912047, + "ttft_ms": 360.3330772735944 + }, + { + "e2e_ms": 14524.751300129523, + "request_id": 59, + "tpot_ms": 111.34689190564231, + "ttft_ms": 383.69602811295067 + }, + { + "e2e_ms": 14156.606828520311, + "request_id": 60, + "tpot_ms": 109.11657328721294, + "ttft_ms": 298.802021044267 + }, + { + "e2e_ms": 13870.414276689999, + "request_id": 61, + "tpot_ms": 106.67912838430216, + "ttft_ms": 322.16497188362325 + }, + { + "e2e_ms": 13579.457505623892, + "request_id": 62, + "tpot_ms": 104.20416994410166, + "ttft_ms": 345.5279227229795 + }, + { + "e2e_ms": 13281.118281252404, + "request_id": 63, + "tpot_ms": 101.6710819503155, + "ttft_ms": 368.89087356233574 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.15625, + "passed": 10 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 139.56721377372742, + "offered_request_rate": 1.2, + "request_metrics_sha256": "dfafc592cb8f7c26725458265ca04cb89642135218d5f80993dfbd6e83fcea9c", + "request_rate_per_gpu": 0.15, + "requests": [ + { + "e2e_ms": 17349.6420929535, + "request_id": 0, + "tpot_ms": 134.3147668363463, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 17472.38469413562, + "request_id": 1, + "tpot_ms": 135.04025977135157, + "ttft_ms": 322.27170317397037 + }, + { + "e2e_ms": 17594.570506858923, + "request_id": 2, + "tpot_ms": 135.72808492940857, + "ttft_ms": 357.1037208240346 + }, + { + "e2e_ms": 17607.2367009099, + "request_id": 3, + "tpot_ms": 136.28731175562888, + "ttft_ms": 298.7481079450336 + }, + { + "e2e_ms": 17921.914449676293, + "request_id": 4, + "tpot_ms": 138.39011926907332, + "ttft_ms": 346.36930250398024 + }, + { + "e2e_ms": 17935.406822135243, + "request_id": 5, + "tpot_ms": 138.88107821741167, + "ttft_ms": 297.50988852396 + }, + { + "e2e_ms": 18057.925904932094, + "request_id": 6, + "tpot_ms": 139.39521236971183, + "ttft_ms": 354.73393397869125 + }, + { + "e2e_ms": 18071.482157513437, + "request_id": 7, + "tpot_ms": 139.75067177492934, + "ttft_ms": 323.14684209741085 + }, + { + "e2e_ms": 18194.065120430685, + "request_id": 8, + "tpot_ms": 140.142283173617, + "ttft_ms": 395.99515738132584 + }, + { + "e2e_ms": 18207.834601589766, + "request_id": 9, + "tpot_ms": 140.4257392143267, + "ttft_ms": 373.76572137027256 + }, + { + "e2e_ms": 18221.607066695313, + "request_id": 10, + "tpot_ms": 140.60137526077196, + "ttft_ms": 365.232408577274 + }, + { + "e2e_ms": 18235.379531799863, + "request_id": 11, + "tpot_ms": 140.670460527253, + "ttft_ms": 370.23104483873226 + }, + { + "e2e_ms": 18249.21289308034, + "request_id": 12, + "tpot_ms": 140.64257855984852, + "ttft_ms": 387.6054159795803 + }, + { + "e2e_ms": 18263.046254360816, + "request_id": 13, + "tpot_ms": 140.69995603545655, + "ttft_ms": 394.1518378578337 + }, + { + "e2e_ms": 18276.879615640297, + "request_id": 14, + "tpot_ms": 140.75508408264292, + "ttft_ms": 400.98393714464817 + }, + { + "e2e_ms": 18181.536918127527, + "request_id": 15, + "tpot_ms": 140.80327015717634, + "ttft_ms": 299.52160816613116 + }, + { + "e2e_ms": 18195.15705083027, + "request_id": 16, + "tpot_ms": 140.84624138271937, + "ttft_ms": 307.6843952249071 + }, + { + "e2e_ms": 18208.77718353195, + "request_id": 17, + "tpot_ms": 140.87800086343316, + "ttft_ms": 317.2710738759417 + }, + { + "e2e_ms": 18222.54666469151, + "request_id": 18, + "tpot_ms": 140.90607860065492, + "ttft_ms": 327.4746824083348 + }, + { + "e2e_ms": 18236.319129797193, + "request_id": 19, + "tpot_ms": 140.91088449209656, + "ttft_ms": 340.6367993009294 + }, + { + "e2e_ms": 18250.152491077173, + "request_id": 20, + "tpot_ms": 140.8981539798204, + "ttft_ms": 356.0869356399827 + }, + { + "e2e_ms": 18263.985852358146, + "request_id": 21, + "tpot_ms": 140.89857595079965, + "ttft_ms": 369.8667066065901 + }, + { + "e2e_ms": 18277.81921363912, + "request_id": 22, + "tpot_ms": 140.90338208286406, + "ttft_ms": 383.0896891153834 + }, + { + "e2e_ms": 18291.6525749191, + "request_id": 23, + "tpot_ms": 140.91257237601366, + "ttft_ms": 395.7558831653678 + }, + { + "e2e_ms": 18305.48593620007, + "request_id": 24, + "tpot_ms": 140.9217626691632, + "ttft_ms": 408.4220772163469 + }, + { + "e2e_ms": 18210.143238687502, + "request_id": 25, + "tpot_ms": 140.92327164841032, + "ttft_ms": 312.8877393393914 + }, + { + "e2e_ms": 18223.912719846063, + "request_id": 26, + "tpot_ms": 140.92545360667498, + "ttft_ms": 326.38011179834336 + }, + { + "e2e_ms": 18237.68518495175, + "request_id": 27, + "tpot_ms": 140.92765906057843, + "ttft_ms": 339.8724842582901 + }, + { + "e2e_ms": 18251.518546232724, + "request_id": 28, + "tpot_ms": 140.92984101884323, + "ttft_ms": 353.42873683963205 + }, + { + "e2e_ms": 18265.3519075127, + "request_id": 29, + "tpot_ms": 140.93202297710803, + "ttft_ms": 366.98498941997923 + }, + { + "e2e_ms": 18279.185268793677, + "request_id": 30, + "tpot_ms": 140.93370194229104, + "ttft_ms": 380.60512212271647 + }, + { + "e2e_ms": 18293.01863007465, + "request_id": 31, + "tpot_ms": 140.9341814397353, + "ttft_ms": 394.3775872282629 + }, + { + "e2e_ms": 18306.851991354626, + "request_id": 32, + "tpot_ms": 140.93466093717961, + "ttft_ms": 408.15005233281454 + }, + { + "e2e_ms": 18211.509293842057, + "request_id": 33, + "tpot_ms": 140.933484965084, + "ttft_ms": 312.956703276388 + }, + { + "e2e_ms": 18225.278775001614, + "request_id": 34, + "tpot_ms": 140.9329819720059, + "ttft_ms": 326.79006455686374 + }, + { + "e2e_ms": 18239.11213628159, + "request_id": 35, + "tpot_ms": 140.93298197200983, + "ttft_ms": 340.6234258363447 + }, + { + "e2e_ms": 18252.945497562563, + "request_id": 36, + "tpot_ms": 140.93298197201372, + "ttft_ms": 354.45678711682046 + }, + { + "e2e_ms": 18266.778858843536, + "request_id": 37, + "tpot_ms": 140.93466093719667, + "ttft_ms": 368.0769198195577 + }, + { + "e2e_ms": 18280.612220123516, + "request_id": 38, + "tpot_ms": 140.93633990237933, + "ttft_ms": 381.69705252133923 + }, + { + "e2e_ms": 18294.44558140449, + "request_id": 39, + "tpot_ms": 140.93801886756114, + "ttft_ms": 395.3171852242221 + }, + { + "e2e_ms": 18308.27894268546, + "request_id": 40, + "tpot_ms": 140.93849836500434, + "ttft_ms": 409.0896503299106 + }, + { + "e2e_ms": 18212.936245171895, + "request_id": 41, + "tpot_ms": 140.93732239290452, + "ttft_ms": 313.8963012730187 + }, + { + "e2e_ms": 18226.708710277584, + "request_id": 42, + "tpot_ms": 140.93684289546135, + "ttft_ms": 327.72966255399183 + }, + { + "e2e_ms": 18047.749752064, + "request_id": 43, + "tpot_ms": 139.41879313566167, + "ttft_ms": 341.56302383496495 + }, + { + "e2e_ms": 17872.42465139999, + "request_id": 44, + "tpot_ms": 137.92935642744132, + "ttft_ms": 355.3963851149433 + }, + { + "e2e_ms": 17695.178674250878, + "request_id": 45, + "tpot_ms": 136.4247947075194, + "ttft_ms": 369.22974639591644 + }, + { + "e2e_ms": 17514.86814366097, + "request_id": 46, + "tpot_ms": 134.89610264554392, + "ttft_ms": 383.06310767688956 + }, + { + "e2e_ms": 17334.24324324006, + "request_id": 47, + "tpot_ms": 133.36661419575813, + "ttft_ms": 396.68324037877767 + }, + { + "e2e_ms": 17152.122611965817, + "request_id": 48, + "tpot_ms": 131.8241488699319, + "ttft_ms": 410.4557054844662 + }, + { + "e2e_ms": 16860.906258226023, + "request_id": 49, + "tpot_ms": 130.28066064407446, + "ttft_ms": 315.26235642856903 + }, + { + "e2e_ms": 16678.317507592936, + "request_id": 50, + "tpot_ms": 128.73402984160936, + "ttft_ms": 329.0957177085474 + }, + { + "e2e_ms": 16495.414294222966, + "request_id": 51, + "tpot_ms": 127.18492295459406, + "ttft_ms": 342.9290789895205 + }, + { + "e2e_ms": 16325.716031493954, + "request_id": 52, + "tpot_ms": 125.73979205687765, + "ttft_ms": 356.76244027049364 + }, + { + "e2e_ms": 16142.044362006885, + "request_id": 53, + "tpot_ms": 124.18463433430247, + "ttft_ms": 370.595801550472 + }, + { + "e2e_ms": 15942.642543961227, + "request_id": 54, + "tpot_ms": 122.50561717425025, + "ttft_ms": 384.4291628314451 + }, + { + "e2e_ms": 15727.503328455541, + "request_id": 55, + "tpot_ms": 120.70436246394655, + "ttft_ms": 398.049295534328 + }, + { + "e2e_ms": 15403.205770995846, + "request_id": 56, + "tpot_ms": 118.90039231904261, + "ttft_ms": 302.8559464774361 + }, + { + "e2e_ms": 15178.030676088234, + "request_id": 57, + "tpot_ms": 117.01843597110097, + "ttft_ms": 316.6893077584092 + }, + { + "e2e_ms": 14941.341931765628, + "request_id": 58, + "tpot_ms": 115.04582096634839, + "ttft_ms": 330.5226690393823 + }, + { + "e2e_ms": 14696.772511761146, + "request_id": 59, + "tpot_ms": 113.01115339717941, + "ttft_ms": 344.3560303193607 + }, + { + "e2e_ms": 14451.34286402164, + "request_id": 60, + "tpot_ms": 110.96971238127013, + "ttft_ms": 358.1893916003338 + }, + { + "e2e_ms": 14196.293999590453, + "request_id": 61, + "tpot_ms": 108.85252950164681, + "ttft_ms": 372.02275288130693 + }, + { + "e2e_ms": 13938.386603616791, + "request_id": 62, + "tpot_ms": 106.71283849964966, + "ttft_ms": 385.8561141612853 + }, + { + "e2e_ms": 13676.049735660896, + "request_id": 63, + "tpot_ms": 104.53992523504405, + "ttft_ms": 399.4792308103001 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 167.9100992679596, + "offered_request_rate": 1.6, + "request_metrics_sha256": "e85085ea5517c1afa9f850cb838a638360d45db695cde8d3c9fef6baf6af44c4", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 19447.058061600903, + "request_id": 0, + "tpot_ms": 150.8298532036487, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 19750.640261300512, + "request_id": 1, + "tpot_ms": 152.83450342712686, + "ttft_ms": 340.65832605540146 + }, + { + "e2e_ms": 19750.02534103151, + "request_id": 2, + "tpot_ms": 153.17602578595117, + "ttft_ms": 296.67006621570977 + }, + { + "e2e_ms": 19855.93358266601, + "request_id": 3, + "tpot_ms": 153.5864661294274, + "ttft_ms": 350.4523842287295 + }, + { + "e2e_ms": 19855.232098385564, + "request_id": 4, + "tpot_ms": 153.87573615619343, + "ttft_ms": 313.01360654900145 + }, + { + "e2e_ms": 19961.05377600862, + "request_id": 5, + "tpot_ms": 154.2244676371592, + "ttft_ms": 374.54638608940184 + }, + { + "e2e_ms": 19960.265727716724, + "request_id": 6, + "tpot_ms": 154.4718650613629, + "ttft_ms": 342.33886492363473 + }, + { + "e2e_ms": 19959.47767942483, + "request_id": 7, + "tpot_ms": 154.6679176552935, + "ttft_ms": 316.6521372025519 + }, + { + "e2e_ms": 20065.255011109733, + "request_id": 8, + "tpot_ms": 154.88778812601134, + "ttft_ms": 394.50591910629294 + }, + { + "e2e_ms": 20064.42261687969, + "request_id": 9, + "tpot_ms": 155.03615010045385, + "ttft_ms": 374.8315541220517 + }, + { + "e2e_ms": 20063.590222649644, + "request_id": 10, + "tpot_ms": 155.117974090142, + "ttft_ms": 363.6075132016101 + }, + { + "e2e_ms": 20062.7578284196, + "request_id": 11, + "tpot_ms": 155.13454318347627, + "ttft_ms": 360.67084411811567 + }, + { + "e2e_ms": 20061.925434189554, + "request_id": 12, + "tpot_ms": 155.09034115377113, + "ttft_ms": 365.45210766062206 + }, + { + "e2e_ms": 20061.09303995951, + "request_id": 13, + "tpot_ms": 155.09665427482798, + "ttft_ms": 363.8179470563543 + }, + { + "e2e_ms": 20060.260645729464, + "request_id": 14, + "tpot_ms": 155.10154332898202, + "ttft_ms": 362.3646429487479 + }, + { + "e2e_ms": 20059.42825149942, + "request_id": 15, + "tpot_ms": 155.10604675613317, + "ttft_ms": 360.96031347050683 + }, + { + "e2e_ms": 20058.595857269374, + "request_id": 16, + "tpot_ms": 155.10168661355542, + "ttft_ms": 360.6816573478344 + }, + { + "e2e_ms": 20057.76346303933, + "request_id": 17, + "tpot_ms": 155.0900405289204, + "ttft_ms": 361.3283158664373 + }, + { + "e2e_ms": 20056.931068809285, + "request_id": 18, + "tpot_ms": 155.07436184636043, + "ttft_ms": 362.4871143215085 + }, + { + "e2e_ms": 20056.09867457924, + "request_id": 19, + "tpot_ms": 155.0434256316313, + "ttft_ms": 365.58361936206654 + }, + { + "e2e_ms": 20055.26628034948, + "request_id": 20, + "tpot_ms": 154.9995151863024, + "ttft_ms": 370.32785168907446 + }, + { + "e2e_ms": 20054.433886119732, + "request_id": 21, + "tpot_ms": 154.96275487551375, + "ttft_ms": 374.16401692948574 + }, + { + "e2e_ms": 20053.601491889985, + "request_id": 22, + "tpot_ms": 154.92303029896564, + "ttft_ms": 378.3766439213494 + }, + { + "e2e_ms": 20052.76909766024, + "request_id": 23, + "tpot_ms": 154.88711594543284, + "ttft_ms": 382.105372590269 + }, + { + "e2e_ms": 20051.936703430492, + "request_id": 24, + "tpot_ms": 154.8482609975464, + "ttft_ms": 386.20755674210193 + }, + { + "e2e_ms": 20051.104309200746, + "request_id": 25, + "tpot_ms": 154.85607608193587, + "ttft_ms": 384.3826467948901 + }, + { + "e2e_ms": 20050.271914971, + "request_id": 26, + "tpot_ms": 154.86595665257255, + "ttft_ms": 382.2954200942838 + }, + { + "e2e_ms": 20049.439520741253, + "request_id": 27, + "tpot_ms": 154.87441181850545, + "ttft_ms": 380.3892197910592 + }, + { + "e2e_ms": 20048.607126511506, + "request_id": 28, + "tpot_ms": 154.87585298283847, + "ttft_ms": 379.37379769102006 + }, + { + "e2e_ms": 20047.77473228176, + "request_id": 29, + "tpot_ms": 154.8790680483812, + "ttft_ms": 378.13309013734295 + }, + { + "e2e_ms": 20046.942338052013, + "request_id": 30, + "tpot_ms": 154.879633497606, + "ttft_ms": 377.228883856052 + }, + { + "e2e_ms": 20046.109943822266, + "request_id": 31, + "tpot_ms": 154.8807461594705, + "ttft_ms": 376.2551815695119 + }, + { + "e2e_ms": 20045.27754959252, + "request_id": 32, + "tpot_ms": 154.8790337660788, + "ttft_ms": 375.64026130051076 + }, + { + "e2e_ms": 19846.84310747305, + "request_id": 33, + "tpot_ms": 153.32139973576017, + "ttft_ms": 375.02534103150964 + }, + { + "e2e_ms": 19648.543544766126, + "request_id": 34, + "tpot_ms": 151.76482774805996, + "ttft_ms": 374.4104207625085 + }, + { + "e2e_ms": 19450.677394587543, + "request_id": 35, + "tpot_ms": 150.2123500638227, + "ttft_ms": 373.7089364820605 + }, + { + "e2e_ms": 19253.237256186894, + "request_id": 36, + "tpot_ms": 148.66322680303372, + "ttft_ms": 373.00745220161247 + }, + { + "e2e_ms": 19054.80482620875, + "request_id": 37, + "tpot_ms": 147.10697182912625, + "ttft_ms": 372.2194039097175 + }, + { + "e2e_ms": 18856.422732154002, + "request_id": 38, + "tpot_ms": 145.55111320107227, + "ttft_ms": 371.4313556178226 + }, + { + "e2e_ms": 18658.616312962055, + "request_id": 39, + "tpot_ms": 143.99978744595376, + "ttft_ms": 370.64330732592765 + }, + { + "e2e_ms": 18469.244245876795, + "request_id": 40, + "tpot_ms": 142.51522309276308, + "ttft_ms": 369.81091309588265 + }, + { + "e2e_ms": 18279.907702313452, + "request_id": 41, + "tpot_ms": 141.0309384523434, + "ttft_ms": 368.97851886583766 + }, + { + "e2e_ms": 18091.61193748763, + "request_id": 42, + "tpot_ms": 139.55484892009318, + "ttft_ms": 368.14612463579266 + }, + { + "e2e_ms": 17903.060652176137, + "request_id": 43, + "tpot_ms": 138.07674741551486, + "ttft_ms": 367.31373040574766 + }, + { + "e2e_ms": 17716.73280728946, + "request_id": 44, + "tpot_ms": 136.61615331585634, + "ttft_ms": 366.48133617570267 + }, + { + "e2e_ms": 17529.12437807872, + "request_id": 45, + "tpot_ms": 135.14547587506345, + "ttft_ms": 365.64894194565767 + }, + { + "e2e_ms": 17339.472913240785, + "request_id": 46, + "tpot_ms": 133.65871153956826, + "ttft_ms": 364.81654771561267 + }, + { + "e2e_ms": 17149.611868516175, + "request_id": 47, + "tpot_ms": 132.17029696874496, + "ttft_ms": 363.9841534855677 + }, + { + "e2e_ms": 16958.75366988868, + "request_id": 48, + "tpot_ms": 130.67403079238707, + "ttft_ms": 363.1517592555227 + }, + { + "e2e_ms": 16767.44725356325, + "request_id": 49, + "tpot_ms": 129.1742353428171, + "ttft_ms": 362.3193650254777 + }, + { + "e2e_ms": 16576.276975363853, + "request_id": 50, + "tpot_ms": 127.67551184699543, + "ttft_ms": 361.4869707954327 + }, + { + "e2e_ms": 16384.89705533921, + "request_id": 51, + "tpot_ms": 126.1751376281405, + "ttft_ms": 360.65457656536637 + }, + { + "e2e_ms": 16202.320435741867, + "request_id": 52, + "tpot_ms": 124.74408073548226, + "ttft_ms": 359.8221823356198 + }, + { + "e2e_ms": 16010.428211639819, + "request_id": 53, + "tpot_ms": 123.23967262625155, + "ttft_ms": 358.98978810587323 + }, + { + "e2e_ms": 15808.049221831383, + "request_id": 54, + "tpot_ms": 121.6526915587028, + "ttft_ms": 358.15739387612666 + }, + { + "e2e_ms": 15595.17863371626, + "request_id": 55, + "tpot_ms": 119.98309948086519, + "ttft_ms": 357.3249996463801 + }, + { + "e2e_ms": 15377.4072544157, + "request_id": 56, + "tpot_ms": 118.27491849605565, + "ttft_ms": 356.4926054166335 + }, + { + "e2e_ms": 15157.846080032627, + "request_id": 57, + "tpot_ms": 116.55264463658062, + "ttft_ms": 355.66021118688695 + }, + { + "e2e_ms": 14930.609139372891, + "request_id": 58, + "tpot_ms": 114.76993167256497, + "ttft_ms": 354.8278169571404 + }, + { + "e2e_ms": 14698.1184149259, + "request_id": 59, + "tpot_ms": 112.9458503322717, + "ttft_ms": 353.9954227273938 + }, + { + "e2e_ms": 14465.054205320868, + "request_id": 60, + "tpot_ms": 111.11725336081277, + "ttft_ms": 353.16302849764725 + }, + { + "e2e_ms": 14225.577184589198, + "request_id": 61, + "tpot_ms": 109.23816181355352, + "ttft_ms": 352.3306342679007 + }, + { + "e2e_ms": 13984.194476163324, + "request_id": 62, + "tpot_ms": 107.34406485137929, + "ttft_ms": 351.4982400381541 + }, + { + "e2e_ms": 13739.858786414643, + "request_id": 63, + "tpot_ms": 105.42671606776564, + "ttft_ms": 350.66584580840754 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 390.6987729072571, + "offered_request_rate": 2.4, + "request_metrics_sha256": "c4ae9e9bb2050189ca506883c0c12efd5faecce0ec6249ad641d32f3b9a7f130", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 25856.75271176486, + "request_id": 0, + "tpot_ms": 201.29988981911293, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26179.284600266838, + "request_id": 1, + "tpot_ms": 203.30897363251975, + "ttft_ms": 359.0449489368311 + }, + { + "e2e_ms": 25981.87281483019, + "request_id": 2, + "tpot_ms": 201.96786246415923, + "ttft_ms": 331.9542818819672 + }, + { + "e2e_ms": 25784.35394549511, + "request_id": 3, + "tpot_ms": 200.62102535110301, + "ttft_ms": 305.48372590502584 + }, + { + "e2e_ms": 25696.083985341404, + "request_id": 4, + "tpot_ms": 199.35042104178692, + "ttft_ms": 378.5805130344673 + }, + { + "e2e_ms": 25498.25455731707, + "request_id": 5, + "tpot_ms": 197.981349219281, + "ttft_ms": 354.62320646838384 + }, + { + "e2e_ms": 25300.295013812065, + "request_id": 6, + "tpot_ms": 196.59886557857163, + "ttft_ms": 332.2390853334691 + }, + { + "e2e_ms": 25102.100957670827, + "request_id": 7, + "tpot_ms": 195.19408059762435, + "ttft_ms": 312.45272177253236 + }, + { + "e2e_ms": 25012.968951433395, + "request_id": 8, + "tpot_ms": 193.83981874911316, + "ttft_ms": 395.3119702960262 + }, + { + "e2e_ms": 24814.29543067174, + "request_id": 9, + "tpot_ms": 192.41025790813484, + "ttft_ms": 378.1926763386156 + }, + { + "e2e_ms": 24615.47335754875, + "request_id": 10, + "tpot_ms": 190.95429488392014, + "ttft_ms": 364.2779072908935 + }, + { + "e2e_ms": 24416.46039907787, + "request_id": 11, + "tpot_ms": 189.47286981272285, + "ttft_ms": 353.405932862068 + }, + { + "e2e_ms": 24216.993764693514, + "request_id": 12, + "tpot_ms": 187.96377697549212, + "ttft_ms": 345.59408880601427 + }, + { + "e2e_ms": 24017.241128749425, + "request_id": 13, + "tpot_ms": 186.4682030159872, + "ttft_ms": 335.7793457190477 + }, + { + "e2e_ms": 23816.999491057053, + "request_id": 14, + "tpot_ms": 184.96817994362138, + "ttft_ms": 326.0406382171395 + }, + { + "e2e_ms": 23616.430479744246, + "request_id": 15, + "tpot_ms": 183.46465752234423, + "ttft_ms": 316.41897440652934 + }, + { + "e2e_ms": 23412.66959473249, + "request_id": 16, + "tpot_ms": 181.92890330417333, + "ttft_ms": 307.69887510247963 + }, + { + "e2e_ms": 23315.084360363784, + "request_id": 17, + "tpot_ms": 180.37041078851854, + "ttft_ms": 408.042190221928 + }, + { + "e2e_ms": 23111.095991700124, + "request_id": 18, + "tpot_ms": 178.82629774094855, + "ttft_ms": 400.1561785996568 + }, + { + "e2e_ms": 22907.001651879975, + "request_id": 19, + "tpot_ms": 177.27413055190453, + "ttft_ms": 393.18707178810007 + }, + { + "e2e_ms": 22702.9918523004, + "request_id": 20, + "tpot_ms": 175.71469647400258, + "ttft_ms": 387.22540010207496 + }, + { + "e2e_ms": 22499.024594644914, + "request_id": 21, + "tpot_ms": 174.1539791842547, + "ttft_ms": 381.46923824456815 + }, + { + "e2e_ms": 22294.98921706886, + "request_id": 22, + "tpot_ms": 172.59076723758398, + "ttft_ms": 375.96177789569475 + }, + { + "e2e_ms": 22091.055170736552, + "request_id": 23, + "tpot_ms": 171.02806584364455, + "ttft_ms": 370.49080859369354 + }, + { + "e2e_ms": 21886.829371806663, + "request_id": 24, + "tpot_ms": 169.4599867344259, + "ttft_ms": 365.4110565345743 + }, + { + "e2e_ms": 21682.517380669786, + "request_id": 25, + "tpot_ms": 167.90637996068475, + "ttft_ms": 358.40712566282207 + }, + { + "e2e_ms": 21478.40806232245, + "request_id": 26, + "tpot_ms": 166.35416808544093, + "ttft_ms": 351.4287154714495 + }, + { + "e2e_ms": 21274.47246393178, + "request_id": 27, + "tpot_ms": 164.80170050625136, + "ttft_ms": 344.6564996378552 + }, + { + "e2e_ms": 21070.655727841604, + "request_id": 28, + "tpot_ms": 163.24706150884583, + "ttft_ms": 338.278916218183 + }, + { + "e2e_ms": 20866.83848786135, + "request_id": 29, + "tpot_ms": 161.69251523328387, + "ttft_ms": 331.88905323430083 + }, + { + "e2e_ms": 20663.178771213603, + "request_id": 30, + "tpot_ms": 160.1348533340772, + "ttft_ms": 326.0523977857979 + }, + { + "e2e_ms": 20459.48769489552, + "request_id": 31, + "tpot_ms": 158.57430185728822, + "ttft_ms": 320.55135901991605 + }, + { + "e2e_ms": 20255.951788183756, + "request_id": 32, + "tpot_ms": 157.01541837295076, + "ttft_ms": 314.9936548190073 + }, + { + "e2e_ms": 20052.589009493837, + "request_id": 33, + "tpot_ms": 155.45687267778135, + "ttft_ms": 309.5661794156026 + }, + { + "e2e_ms": 19955.947670796646, + "request_id": 34, + "tpot_ms": 153.8983040696114, + "ttft_ms": 410.8630539559979 + }, + { + "e2e_ms": 19752.847929041352, + "request_id": 35, + "tpot_ms": 152.3343481355919, + "ttft_ms": 406.3857158211803 + }, + { + "e2e_ms": 19549.96119317403, + "request_id": 36, + "tpot_ms": 150.77184485827368, + "ttft_ms": 401.93689617327345 + }, + { + "e2e_ms": 19346.57831151796, + "request_id": 37, + "tpot_ms": 149.20507336304368, + "ttft_ms": 397.53399441140937 + }, + { + "e2e_ms": 19143.22059782458, + "request_id": 38, + "tpot_ms": 147.63442381272193, + "ttft_ms": 393.6487736088932 + }, + { + "e2e_ms": 18940.150721561607, + "request_id": 39, + "tpot_ms": 146.06310547094216, + "ttft_ms": 390.13632675195356 + }, + { + "e2e_ms": 18741.29802135198, + "request_id": 40, + "tpot_ms": 144.5231784918296, + "ttft_ms": 386.85435288961753 + }, + { + "e2e_ms": 18542.463082904305, + "request_id": 41, + "tpot_ms": 142.98167897793616, + "ttft_ms": 383.7898527064141 + }, + { + "e2e_ms": 18344.148533824395, + "request_id": 42, + "tpot_ms": 141.4401875845621, + "ttft_ms": 381.2447105850083 + }, + { + "e2e_ms": 18145.706224501653, + "request_id": 43, + "tpot_ms": 139.89417135289136, + "ttft_ms": 379.1464626844494 + }, + { + "e2e_ms": 17948.37563539231, + "request_id": 44, + "tpot_ms": 138.3533077037345, + "ttft_ms": 377.5055570180292 + }, + { + "e2e_ms": 17750.40475411994, + "request_id": 45, + "tpot_ms": 136.80479670989413, + "ttft_ms": 376.19557196338246 + }, + { + "e2e_ms": 17551.412355033975, + "request_id": 46, + "tpot_ms": 135.2471347218224, + "ttft_ms": 375.0262453625339 + }, + { + "e2e_ms": 17352.31516600567, + "request_id": 47, + "tpot_ms": 133.6870504253856, + "ttft_ms": 374.0597619816981 + }, + { + "e2e_ms": 17152.719400024922, + "request_id": 48, + "tpot_ms": 132.11931086353468, + "ttft_ms": 373.56692035601566 + }, + { + "e2e_ms": 16952.899525195207, + "request_id": 49, + "tpot_ms": 130.53228068835782, + "ttft_ms": 375.29987777376664 + }, + { + "e2e_ms": 16753.147719429508, + "request_id": 50, + "tpot_ms": 128.94354832701373, + "ttft_ms": 377.31708189876304 + }, + { + "e2e_ms": 16553.291092750187, + "request_id": 51, + "tpot_ms": 127.34853489391125, + "ttft_ms": 380.0271612234596 + }, + { + "e2e_ms": 16357.836116284518, + "request_id": 52, + "tpot_ms": 125.77762361641257, + "ttft_ms": 384.07791700011984 + }, + { + "e2e_ms": 16157.72333756749, + "request_id": 53, + "tpot_ms": 124.16553823707912, + "ttft_ms": 388.6999814584407 + }, + { + "e2e_ms": 15952.367175996273, + "request_id": 54, + "tpot_ms": 122.51027887411597, + "ttft_ms": 393.5617589835445 + }, + { + "e2e_ms": 15741.765215271713, + "request_id": 55, + "tpot_ms": 120.81149800475544, + "ttft_ms": 398.70496866777216 + }, + { + "e2e_ms": 15528.71285895543, + "request_id": 56, + "tpot_ms": 119.08582914503211, + "ttft_ms": 404.8125575363528 + }, + { + "e2e_ms": 15314.765605096894, + "request_id": 57, + "tpot_ms": 117.34894161957355, + "ttft_ms": 411.4500194110526 + }, + { + "e2e_ms": 15096.980468100028, + "request_id": 58, + "tpot_ms": 115.58065073324867, + "ttft_ms": 418.2378249774459 + }, + { + "e2e_ms": 14778.441120322277, + "request_id": 59, + "tpot_ms": 113.87954551567164, + "ttft_ms": 315.7388398319796 + }, + { + "e2e_ms": 14557.742348852762, + "request_id": 60, + "tpot_ms": 112.08473611057838, + "ttft_ms": 322.9808628093096 + }, + { + "e2e_ms": 14333.83717181993, + "request_id": 61, + "tpot_ms": 110.26282336485625, + "ttft_ms": 330.4586044831872 + }, + { + "e2e_ms": 14108.97915094099, + "request_id": 62, + "tpot_ms": 108.43298632152924, + "ttft_ms": 337.989888106776 + }, + { + "e2e_ms": 13882.64463939965, + "request_id": 63, + "tpot_ms": 106.58813679106942, + "ttft_ms": 345.9512669338345 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 402.1317756175995, + "offered_request_rate": 3.2, + "request_metrics_sha256": "48ff1d03e0aeb45bb4395dca86e089cdbe1c7de16b8557fbe8f25e8e65c77def", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 26482.398681960134, + "request_id": 0, + "tpot_ms": 206.22623604112295, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26389.27725660841, + "request_id": 1, + "tpot_ms": 204.89007083645959, + "ttft_ms": 368.23826037804076 + }, + { + "e2e_ms": 26186.398050214455, + "request_id": 2, + "tpot_ms": 203.43938315354237, + "ttft_ms": 349.5963897145746 + }, + { + "e2e_ms": 25983.46530187178, + "request_id": 3, + "tpot_ms": 201.98715121187504, + "ttft_ms": 331.0970979636474 + }, + { + "e2e_ms": 25780.42643937068, + "request_id": 4, + "tpot_ms": 200.53276712329495, + "ttft_ms": 312.76501471222184 + }, + { + "e2e_ms": 25577.34505869151, + "request_id": 5, + "tpot_ms": 199.0733214464005, + "ttft_ms": 295.0332349986446 + }, + { + "e2e_ms": 25483.55218185351, + "request_id": 6, + "tpot_ms": 197.69083780569105, + "ttft_ms": 376.81578053074594 + }, + { + "e2e_ms": 25280.28848711639, + "request_id": 7, + "tpot_ms": 196.22290463044078, + "ttft_ms": 359.97959905041466 + }, + { + "e2e_ms": 25076.966707259246, + "request_id": 8, + "tpot_ms": 194.75030755695653, + "ttft_ms": 343.6776475257637 + }, + { + "e2e_ms": 24873.46328021192, + "request_id": 9, + "tpot_ms": 193.27265662306831, + "ttft_ms": 327.83588908224147 + }, + { + "e2e_ms": 24669.88557698392, + "request_id": 10, + "tpot_ms": 191.78984111033463, + "ttft_ms": 312.57575597142215 + }, + { + "e2e_ms": 24466.212431081483, + "request_id": 11, + "tpot_ms": 190.30296300954802, + "ttft_ms": 297.7361288688827 + }, + { + "e2e_ms": 24370.912463364126, + "request_id": 12, + "tpot_ms": 188.79387017231738, + "ttft_ms": 394.09095147981964 + }, + { + "e2e_ms": 24166.86947872558, + "request_id": 13, + "tpot_ms": 187.2967209572732, + "ttft_ms": 380.1859171518851 + }, + { + "e2e_ms": 23962.581993212396, + "request_id": 14, + "tpot_ms": 185.79746051386414, + "ttft_ms": 366.30450795165007 + }, + { + "e2e_ms": 23758.13082088949, + "request_id": 15, + "tpot_ms": 184.29572160562404, + "ttft_ms": 352.57417697523687 + }, + { + "e2e_ms": 23552.083711717114, + "request_id": 16, + "tpot_ms": 182.7751996506789, + "ttft_ms": 339.633356080892 + }, + { + "e2e_ms": 23345.944189149544, + "request_id": 17, + "tpot_ms": 181.25255279330477, + "ttft_ms": 326.8699843998375 + }, + { + "e2e_ms": 23139.783338151214, + "request_id": 18, + "tpot_ms": 179.72694307410856, + "ttft_ms": 314.461567739424 + }, + { + "e2e_ms": 22933.56950157464, + "request_id": 19, + "tpot_ms": 178.1977153273412, + "ttft_ms": 302.45965500230557 + }, + { + "e2e_ms": 22833.726368661068, + "request_id": 20, + "tpot_ms": 176.63828124943853, + "ttft_ms": 400.6646499823745 + }, + { + "e2e_ms": 22627.576073166823, + "request_id": 21, + "tpot_ms": 175.09813975268048, + "ttft_ms": 390.11232457640244 + }, + { + "e2e_ms": 22421.391717712297, + "request_id": 22, + "tpot_ms": 173.55627478074584, + "ttft_ms": 379.7448205575735 + }, + { + "e2e_ms": 22215.258027879143, + "request_id": 23, + "tpot_ms": 172.01247264259936, + "ttft_ms": 369.67400226902566 + }, + { + "e2e_ms": 22008.978461747698, + "request_id": 24, + "tpot_ms": 170.46437149324944, + "ttft_ms": 360.0032821050201 + }, + { + "e2e_ms": 21802.65579951293, + "request_id": 25, + "tpot_ms": 168.91532250977505, + "ttft_ms": 350.4098407714995 + }, + { + "e2e_ms": 21596.434473671965, + "request_id": 26, + "tpot_ms": 167.36573728219415, + "ttft_ms": 340.98583883331025 + }, + { + "e2e_ms": 21390.300007810118, + "request_id": 27, + "tpot_ms": 165.81511332736412, + "ttft_ms": 331.7806152348748 + }, + { + "e2e_ms": 21184.224973098888, + "request_id": 28, + "tpot_ms": 164.26380332238384, + "ttft_ms": 322.72195115614187 + }, + { + "e2e_ms": 20978.149686441437, + "request_id": 29, + "tpot_ms": 162.71174941720716, + "ttft_ms": 313.7575104561279 + }, + { + "e2e_ms": 20772.153161450715, + "request_id": 30, + "tpot_ms": 161.15510654351164, + "ttft_ms": 305.4546304247392 + }, + { + "e2e_ms": 20672.62875180034, + "request_id": 31, + "tpot_ms": 159.59455506672822, + "ttft_ms": 404.12025832585744 + }, + { + "e2e_ms": 20466.69413177714, + "request_id": 32, + "tpot_ms": 158.03533915661183, + "ttft_ms": 396.2060588874419 + }, + { + "e2e_ms": 20260.84607576572, + "request_id": 33, + "tpot_ms": 156.475168525323, + "ttft_ms": 388.49967304969545 + }, + { + "e2e_ms": 20055.02118508905, + "request_id": 34, + "tpot_ms": 154.9132321630232, + "ttft_ms": 381.04070038510326 + }, + { + "e2e_ms": 19849.304647544788, + "request_id": 35, + "tpot_ms": 153.3461190070677, + "ttft_ms": 374.34753364718887 + }, + { + "e2e_ms": 19643.694612944317, + "request_id": 36, + "tpot_ms": 151.77962188577303, + "ttft_ms": 367.6826334511443 + }, + { + "e2e_ms": 19437.836505449406, + "request_id": 37, + "tpot_ms": 150.210341914627, + "ttft_ms": 361.1230822917744 + }, + { + "e2e_ms": 19231.99098193562, + "request_id": 38, + "tpot_ms": 148.63640086508497, + "ttft_ms": 355.16807206983003 + }, + { + "e2e_ms": 19026.289377137648, + "request_id": 39, + "tpot_ms": 147.05985988148976, + "ttft_ms": 349.6871721884496 + }, + { + "e2e_ms": 18822.696360365982, + "request_id": 40, + "tpot_ms": 145.49844654462956, + "ttft_ms": 344.3936491980288 + }, + { + "e2e_ms": 18619.112224474804, + "request_id": 41, + "tpot_ms": 143.93654065290042, + "ttft_ms": 339.17156155644966 + }, + { + "e2e_ms": 18415.78828326805, + "request_id": 42, + "tpot_ms": 142.37219516274803, + "ttft_ms": 334.51949759905375 + }, + { + "e2e_ms": 18212.400461939902, + "request_id": 43, + "tpot_ms": 140.8040960160447, + "ttft_ms": 330.280267902225 + }, + { + "e2e_ms": 18009.568500717836, + "request_id": 44, + "tpot_ms": 139.2366051127279, + "ttft_ms": 326.5196514013962 + }, + { + "e2e_ms": 17806.416393414805, + "request_id": 45, + "tpot_ms": 137.66365486442547, + "ttft_ms": 323.1322256327704 + }, + { + "e2e_ms": 17602.75352720573, + "request_id": 46, + "tpot_ms": 136.08599255484802, + "ttft_ms": 319.83247274003014 + }, + { + "e2e_ms": 17399.03826602459, + "request_id": 47, + "tpot_ms": 134.50648843613666, + "ttft_ms": 316.71423463523496 + }, + { + "e2e_ms": 17195.073716367715, + "request_id": 48, + "tpot_ms": 132.92201961792392, + "ttft_ms": 313.9772248913761 + }, + { + "e2e_ms": 16990.997112286357, + "request_id": 49, + "tpot_ms": 131.3320936883647, + "ttft_ms": 311.8212138640413 + }, + { + "e2e_ms": 16895.41197318666, + "request_id": 50, + "tpot_ms": 129.74336132702078, + "ttft_ms": 418.0050846550216 + }, + { + "e2e_ms": 16691.3169931805, + "request_id": 51, + "tpot_ms": 128.14881090811454, + "ttft_ms": 416.418007849952 + }, + { + "e2e_ms": 16489.422838281163, + "request_id": 52, + "tpot_ms": 126.56215930365238, + "ttft_ms": 416.0286067173118 + }, + { + "e2e_ms": 16285.199782255653, + "request_id": 53, + "tpot_ms": 124.95445752714085, + "ttft_ms": 415.9836763087661 + }, + { + "e2e_ms": 16078.355034803544, + "request_id": 54, + "tpot_ms": 123.3249765236981, + "ttft_ms": 416.08301629388575 + }, + { + "e2e_ms": 15868.887387774763, + "request_id": 55, + "tpot_ms": 121.67321161824907, + "ttft_ms": 416.3895122571297 + }, + { + "e2e_ms": 15658.194542949623, + "request_id": 56, + "tpot_ms": 120.00563624201372, + "ttft_ms": 417.4787402138804 + }, + { + "e2e_ms": 15447.054249353854, + "request_id": 57, + "tpot_ms": 118.33082151865976, + "ttft_ms": 419.0399164840635 + }, + { + "e2e_ms": 15233.99501418892, + "request_id": 58, + "tpot_ms": 116.64063648866745, + "ttft_ms": 420.6341801281539 + }, + { + "e2e_ms": 15019.622333077174, + "request_id": 59, + "tpot_ms": 114.93953127109042, + "ttft_ms": 422.3018616486911 + }, + { + "e2e_ms": 14805.106280675915, + "request_id": 60, + "tpot_ms": 113.23514577866216, + "ttft_ms": 424.2427667858202 + }, + { + "e2e_ms": 14588.987025492997, + "request_id": 61, + "tpot_ms": 111.5171161341123, + "ttft_ms": 426.31327646073555 + }, + { + "e2e_ms": 14372.391348386529, + "request_id": 62, + "tpot_ms": 109.79194855942059, + "ttft_ms": 428.8138813401155 + }, + { + "e2e_ms": 14155.057425949359, + "request_id": 63, + "tpot_ms": 108.05983074662959, + "ttft_ms": 431.45892112740114 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.109375, + "passed": 7 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.40625, + "passed": 26 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 781.3052506446838, + "offered_request_rate": 0.1, + "request_metrics_sha256": "1146954f1d73891536473385a3ecb02b14b7eaeee231291f2cbe6f9d4d222414", + "request_rate_per_gpu": 0.0125, + "requests": [ + { + "e2e_ms": 12579.084570749343, + "request_id": 0, + "tpot_ms": 96.75132177962064, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12682.66705927438, + "request_id": 1, + "tpot_ms": 96.90664947750408, + "ttft_ms": 375.5225756313614 + }, + { + "e2e_ms": 12692.014437909165, + "request_id": 2, + "tpot_ms": 96.91832475611541, + "ttft_ms": 383.3871938825091 + }, + { + "e2e_ms": 12605.650216029226, + "request_id": 3, + "tpot_ms": 96.9198422551915, + "ttft_ms": 296.8302496199051 + }, + { + "e2e_ms": 12614.259349375281, + "request_id": 4, + "tpot_ms": 96.9125118011697, + "ttft_ms": 306.3703506267288 + }, + { + "e2e_ms": 12622.868482721336, + "request_id": 5, + "tpot_ms": 96.9125118011697, + "ttft_ms": 314.97948397278464 + }, + { + "e2e_ms": 12631.47761606615, + "request_id": 6, + "tpot_ms": 96.91251180115991, + "ttft_ms": 323.5886173188405 + }, + { + "e2e_ms": 12640.086749411268, + "request_id": 7, + "tpot_ms": 96.91251180113692, + "ttft_ms": 332.19775066687873 + }, + { + "e2e_ms": 12648.695882756385, + "request_id": 8, + "tpot_ms": 96.91251180113692, + "ttft_ms": 340.80688401199666 + }, + { + "e2e_ms": 12657.305016101503, + "request_id": 9, + "tpot_ms": 96.91251180113692, + "ttft_ms": 349.4160173571146 + }, + { + "e2e_ms": 12665.91414944662, + "request_id": 10, + "tpot_ms": 96.9125118011369, + "ttft_ms": 358.0251507022325 + }, + { + "e2e_ms": 12674.523282791739, + "request_id": 11, + "tpot_ms": 96.91251180113692, + "ttft_ms": 366.63428404735043 + }, + { + "e2e_ms": 12683.132416062932, + "request_id": 12, + "tpot_ms": 96.91251180055482, + "ttft_ms": 375.24341739246836 + }, + { + "e2e_ms": 12692.479794575775, + "request_id": 13, + "tpot_ms": 96.91832475507798, + "ttft_ms": 383.85255068087076 + }, + { + "e2e_ms": 12606.11557253071, + "request_id": 14, + "tpot_ms": 96.91984225378269, + "ttft_ms": 297.2956063003096 + }, + { + "e2e_ms": 12614.724705710416, + "request_id": 15, + "tpot_ms": 96.91251179969895, + "ttft_ms": 306.8357071486503 + }, + { + "e2e_ms": 12623.33383889012, + "request_id": 16, + "tpot_ms": 96.91251179969895, + "ttft_ms": 315.44484032835385 + }, + { + "e2e_ms": 12631.942972069823, + "request_id": 17, + "tpot_ms": 96.91251179969893, + "ttft_ms": 324.0539735080574 + }, + { + "e2e_ms": 12640.552105249526, + "request_id": 18, + "tpot_ms": 96.91251179969893, + "ttft_ms": 332.663106687761 + }, + { + "e2e_ms": 12649.16123842923, + "request_id": 19, + "tpot_ms": 96.91251179969893, + "ttft_ms": 341.2722398674646 + }, + { + "e2e_ms": 12657.770371608933, + "request_id": 20, + "tpot_ms": 96.91251179969893, + "ttft_ms": 349.88137304716815 + }, + { + "e2e_ms": 12666.379504788636, + "request_id": 21, + "tpot_ms": 96.91251179969893, + "ttft_ms": 358.49050622687173 + }, + { + "e2e_ms": 12674.988637968341, + "request_id": 22, + "tpot_ms": 96.91251179969895, + "ttft_ms": 367.0996394065753 + }, + { + "e2e_ms": 12683.597771148045, + "request_id": 23, + "tpot_ms": 96.91251179969895, + "ttft_ms": 375.7087725862789 + }, + { + "e2e_ms": 12692.945149660885, + "request_id": 24, + "tpot_ms": 96.91832475507798, + "ttft_ms": 384.31790576598246 + }, + { + "e2e_ms": 12606.580927732637, + "request_id": 25, + "tpot_ms": 96.91984225470249, + "ttft_ms": 297.7609613854213 + }, + { + "e2e_ms": 12615.190061213469, + "request_id": 26, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.3010623465393 + }, + { + "e2e_ms": 12623.7991946943, + "request_id": 27, + "tpot_ms": 96.9125118021018, + "ttft_ms": 315.9101958273709 + }, + { + "e2e_ms": 12632.408328175132, + "request_id": 28, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.5193293082025 + }, + { + "e2e_ms": 12641.017461655963, + "request_id": 29, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.1284627890341 + }, + { + "e2e_ms": 12649.626595136795, + "request_id": 30, + "tpot_ms": 96.9125118021018, + "ttft_ms": 341.73759626986566 + }, + { + "e2e_ms": 12658.235728617627, + "request_id": 31, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.34672975069725 + }, + { + "e2e_ms": 12666.844862098458, + "request_id": 32, + "tpot_ms": 96.9125118021018, + "ttft_ms": 358.95586323152884 + }, + { + "e2e_ms": 12675.45399557929, + "request_id": 33, + "tpot_ms": 96.9125118021018, + "ttft_ms": 367.5649967123604 + }, + { + "e2e_ms": 12684.063129060121, + "request_id": 34, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.174130193192 + }, + { + "e2e_ms": 12693.41050787034, + "request_id": 35, + "tpot_ms": 96.9183247574513, + "ttft_ms": 384.7832636740236 + }, + { + "e2e_ms": 12607.04628612126, + "request_id": 36, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.22631959211776 + }, + { + "e2e_ms": 12615.655419602092, + "request_id": 37, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.76642073516314 + }, + { + "e2e_ms": 12624.264553082923, + "request_id": 38, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.3755542159947 + }, + { + "e2e_ms": 12632.873686563755, + "request_id": 39, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.9846876968263 + }, + { + "e2e_ms": 12641.482820044586, + "request_id": 40, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.5938211776579 + }, + { + "e2e_ms": 12650.091953525418, + "request_id": 41, + "tpot_ms": 96.9125118021018, + "ttft_ms": 342.2029546584895 + }, + { + "e2e_ms": 12658.70108700625, + "request_id": 42, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.8120881393211 + }, + { + "e2e_ms": 12667.310220487081, + "request_id": 43, + "tpot_ms": 96.9125118021018, + "ttft_ms": 359.42122162015266 + }, + { + "e2e_ms": 12675.919353967913, + "request_id": 44, + "tpot_ms": 96.9125118021018, + "ttft_ms": 368.03035510098425 + }, + { + "e2e_ms": 12684.528487448744, + "request_id": 45, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.63948858181584 + }, + { + "e2e_ms": 12693.875866258964, + "request_id": 46, + "tpot_ms": 96.91832475745132, + "ttft_ms": 385.2486220626474 + }, + { + "e2e_ms": 12607.511644509885, + "request_id": 47, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.6916779807416 + }, + { + "e2e_ms": 12616.120777990716, + "request_id": 48, + "tpot_ms": 96.9125118021018, + "ttft_ms": 308.23177912378696 + }, + { + "e2e_ms": 12624.729911471548, + "request_id": 49, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.84091260461855 + }, + { + "e2e_ms": 12633.339044941862, + "request_id": 50, + "tpot_ms": 96.91251180201898, + "ttft_ms": 325.45004608545014 + }, + { + "e2e_ms": 12641.94817818975, + "request_id": 51, + "tpot_ms": 96.91251180018479, + "ttft_ms": 334.0591795662817 + }, + { + "e2e_ms": 12650.557311437638, + "request_id": 52, + "tpot_ms": 96.91251179996682, + "ttft_ms": 342.66831284185173 + }, + { + "e2e_ms": 12659.166444685525, + "request_id": 53, + "tpot_ms": 96.91251179996682, + "ttft_ms": 351.277446089739 + }, + { + "e2e_ms": 12667.775577933411, + "request_id": 54, + "tpot_ms": 96.91251179996681, + "ttft_ms": 359.88657933762624 + }, + { + "e2e_ms": 12676.3847111813, + "request_id": 55, + "tpot_ms": 96.91251179996682, + "ttft_ms": 368.4957125855135 + }, + { + "e2e_ms": 12684.993844429187, + "request_id": 56, + "tpot_ms": 96.91251179996682, + "ttft_ms": 377.10484583340076 + }, + { + "e2e_ms": 12694.3412230047, + "request_id": 57, + "tpot_ms": 96.91832475530246, + "ttft_ms": 385.713979081288 + }, + { + "e2e_ms": 12607.977001027848, + "request_id": 58, + "tpot_ms": 96.9198422540282, + "ttft_ms": 299.1570347662673 + }, + { + "e2e_ms": 12616.586134275734, + "request_id": 59, + "tpot_ms": 96.91251179996681, + "ttft_ms": 308.6971356799495 + }, + { + "e2e_ms": 12625.195267523623, + "request_id": 60, + "tpot_ms": 96.91251179996682, + "ttft_ms": 317.30626892783675 + }, + { + "e2e_ms": 12633.80440077151, + "request_id": 61, + "tpot_ms": 96.91251179996682, + "ttft_ms": 325.915402175724 + }, + { + "e2e_ms": 12642.413534019397, + "request_id": 62, + "tpot_ms": 96.91251179996682, + "ttft_ms": 334.52453542361127 + }, + { + "e2e_ms": 12436.81062947212, + "request_id": 63, + "tpot_ms": 95.22580284094978, + "ttft_ms": 343.1336686714985 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 401.1116807460785, + "offered_request_rate": 0.2, + "request_metrics_sha256": "cf8a095cdf380d42316d1ad169ce9b2b45ae7d64348cd38271a332663de349ad", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 12823.625057543633, + "request_id": 0, + "tpot_ms": 98.67683742367018, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12920.533566322107, + "request_id": 1, + "tpot_ms": 99.10315669196875, + "ttft_ms": 334.4326664420745 + }, + { + "e2e_ms": 12921.724204826443, + "request_id": 2, + "tpot_ms": 99.22365133274592, + "ttft_ms": 320.32048556771286 + }, + { + "e2e_ms": 12922.91484333078, + "request_id": 3, + "tpot_ms": 99.22887081733072, + "ttft_ms": 320.84824952977976 + }, + { + "e2e_ms": 12924.105481838353, + "request_id": 4, + "tpot_ms": 99.22887081736695, + "ttft_ms": 322.0388880327505 + }, + { + "e2e_ms": 12925.296120352506, + "request_id": 5, + "tpot_ms": 99.22887081744425, + "ttft_ms": 323.2295265370873 + }, + { + "e2e_ms": 12926.486758866658, + "request_id": 6, + "tpot_ms": 99.22887081752154, + "ttft_ms": 324.420165041424 + }, + { + "e2e_ms": 12927.677397380812, + "request_id": 7, + "tpot_ms": 99.22887081749418, + "ttft_ms": 325.6108035590515 + }, + { + "e2e_ms": 12928.868035894964, + "request_id": 8, + "tpot_ms": 99.22887081749417, + "ttft_ms": 326.8014420732044 + }, + { + "e2e_ms": 12930.058674409118, + "request_id": 9, + "tpot_ms": 99.22887081749418, + "ttft_ms": 327.9920805873573 + }, + { + "e2e_ms": 12931.24931292327, + "request_id": 10, + "tpot_ms": 99.22887081749417, + "ttft_ms": 329.1827191015102 + }, + { + "e2e_ms": 12932.439951441467, + "request_id": 11, + "tpot_ms": 99.22887081752602, + "ttft_ms": 330.3733576156631 + }, + { + "e2e_ms": 12933.63058995827, + "request_id": 12, + "tpot_ms": 99.22887081754688, + "ttft_ms": 331.563996129816 + }, + { + "e2e_ms": 12934.821228475073, + "request_id": 13, + "tpot_ms": 99.22887081758537, + "ttft_ms": 332.7546346417307 + }, + { + "e2e_ms": 12936.011866991876, + "request_id": 14, + "tpot_ms": 99.22887081758537, + "ttft_ms": 333.94527315853395 + }, + { + "e2e_ms": 12937.20250550868, + "request_id": 15, + "tpot_ms": 99.22887081758537, + "ttft_ms": 335.1359116753372 + }, + { + "e2e_ms": 12938.393144025482, + "request_id": 16, + "tpot_ms": 99.22887081758537, + "ttft_ms": 336.3265501921404 + }, + { + "e2e_ms": 12939.583782542286, + "request_id": 17, + "tpot_ms": 99.22887081758537, + "ttft_ms": 337.51718870894365 + }, + { + "e2e_ms": 12940.77442105909, + "request_id": 18, + "tpot_ms": 99.22887081758537, + "ttft_ms": 338.7078272257469 + }, + { + "e2e_ms": 12941.965059575892, + "request_id": 19, + "tpot_ms": 99.22887081758537, + "ttft_ms": 339.8984657425501 + }, + { + "e2e_ms": 12943.155698092696, + "request_id": 20, + "tpot_ms": 99.22887081758537, + "ttft_ms": 341.08910425935335 + }, + { + "e2e_ms": 12944.346336609498, + "request_id": 21, + "tpot_ms": 99.22887081758537, + "ttft_ms": 342.2797427761566 + }, + { + "e2e_ms": 12945.536975126302, + "request_id": 22, + "tpot_ms": 99.22887081758537, + "ttft_ms": 343.4703812929598 + }, + { + "e2e_ms": 12946.727613643105, + "request_id": 23, + "tpot_ms": 99.22887081758537, + "ttft_ms": 344.66101980976305 + }, + { + "e2e_ms": 12947.918252142103, + "request_id": 24, + "tpot_ms": 99.22887081744517, + "ttft_ms": 345.8516583265663 + }, + { + "e2e_ms": 12949.10889064036, + "request_id": 25, + "tpot_ms": 99.22887081729912, + "ttft_ms": 347.0422968433695 + }, + { + "e2e_ms": 12950.299529138618, + "request_id": 26, + "tpot_ms": 99.2288708172622, + "ttft_ms": 348.23293534631716 + }, + { + "e2e_ms": 12951.490167636877, + "request_id": 27, + "tpot_ms": 99.22887081726222, + "ttft_ms": 349.42357384457523 + }, + { + "e2e_ms": 12952.680806135135, + "request_id": 28, + "tpot_ms": 99.22887081726222, + "ttft_ms": 350.6142123428333 + }, + { + "e2e_ms": 12953.871444633392, + "request_id": 29, + "tpot_ms": 99.2288708172622, + "ttft_ms": 351.80485084109137 + }, + { + "e2e_ms": 12955.06208313165, + "request_id": 30, + "tpot_ms": 99.2288708172622, + "ttft_ms": 352.99548933934943 + }, + { + "e2e_ms": 12956.25272162991, + "request_id": 31, + "tpot_ms": 99.22887081726222, + "ttft_ms": 354.1861278376075 + }, + { + "e2e_ms": 12957.443360128167, + "request_id": 32, + "tpot_ms": 99.22887081726222, + "ttft_ms": 355.37676633586557 + }, + { + "e2e_ms": 12958.633998626425, + "request_id": 33, + "tpot_ms": 99.2288708172622, + "ttft_ms": 356.56740483412364 + }, + { + "e2e_ms": 12959.824637124682, + "request_id": 34, + "tpot_ms": 99.2288708172622, + "ttft_ms": 357.7580433323817 + }, + { + "e2e_ms": 12961.015275622942, + "request_id": 35, + "tpot_ms": 99.22887081726222, + "ttft_ms": 358.94868183063977 + }, + { + "e2e_ms": 12962.2059141212, + "request_id": 36, + "tpot_ms": 99.22887081726222, + "ttft_ms": 360.13932032889784 + }, + { + "e2e_ms": 12963.396552619457, + "request_id": 37, + "tpot_ms": 99.2288708172622, + "ttft_ms": 361.3299588271559 + }, + { + "e2e_ms": 12964.587191117715, + "request_id": 38, + "tpot_ms": 99.2288708172622, + "ttft_ms": 362.520597325414 + }, + { + "e2e_ms": 12965.777829615974, + "request_id": 39, + "tpot_ms": 99.22887081726222, + "ttft_ms": 363.71123582367204 + }, + { + "e2e_ms": 12966.968468114232, + "request_id": 40, + "tpot_ms": 99.22887081726222, + "ttft_ms": 364.9018743219301 + }, + { + "e2e_ms": 12968.15910661249, + "request_id": 41, + "tpot_ms": 99.2288708172622, + "ttft_ms": 366.0925128201882 + }, + { + "e2e_ms": 12969.349745110747, + "request_id": 42, + "tpot_ms": 99.2288708172622, + "ttft_ms": 367.28315131844624 + }, + { + "e2e_ms": 12970.540383609006, + "request_id": 43, + "tpot_ms": 99.22887081726222, + "ttft_ms": 368.4737898167043 + }, + { + "e2e_ms": 12971.731022107264, + "request_id": 44, + "tpot_ms": 99.22887081726222, + "ttft_ms": 369.6644283149624 + }, + { + "e2e_ms": 12972.921660605522, + "request_id": 45, + "tpot_ms": 99.2288708172622, + "ttft_ms": 370.85506681322045 + }, + { + "e2e_ms": 12974.11229910378, + "request_id": 46, + "tpot_ms": 99.2288708172622, + "ttft_ms": 372.0457053114785 + }, + { + "e2e_ms": 12975.302937602039, + "request_id": 47, + "tpot_ms": 99.22887081726222, + "ttft_ms": 373.2363438097366 + }, + { + "e2e_ms": 12976.493576100296, + "request_id": 48, + "tpot_ms": 99.22887081726222, + "ttft_ms": 374.42698230799465 + }, + { + "e2e_ms": 12977.684214615067, + "request_id": 49, + "tpot_ms": 99.22887081739223, + "ttft_ms": 375.6176208062527 + }, + { + "e2e_ms": 12978.874853139188, + "request_id": 50, + "tpot_ms": 99.22887081759589, + "ttft_ms": 376.8082593045108 + }, + { + "e2e_ms": 12980.06549166331, + "request_id": 51, + "tpot_ms": 99.22887081779955, + "ttft_ms": 377.99889780276885 + }, + { + "e2e_ms": 12981.256130187432, + "request_id": 52, + "tpot_ms": 99.22887081784184, + "ttft_ms": 379.189536321519 + }, + { + "e2e_ms": 12982.446768711554, + "request_id": 53, + "tpot_ms": 99.22887081784184, + "ttft_ms": 380.3801748456408 + }, + { + "e2e_ms": 12983.637407235676, + "request_id": 54, + "tpot_ms": 99.22887081784184, + "ttft_ms": 381.5708133697626 + }, + { + "e2e_ms": 12984.828045759798, + "request_id": 55, + "tpot_ms": 99.22887081784184, + "ttft_ms": 382.76145189388444 + }, + { + "e2e_ms": 12986.01868428392, + "request_id": 56, + "tpot_ms": 99.22887081784184, + "ttft_ms": 383.95209041800626 + }, + { + "e2e_ms": 12987.209322808041, + "request_id": 57, + "tpot_ms": 99.22887081784184, + "ttft_ms": 385.1427289421281 + }, + { + "e2e_ms": 12988.399961332163, + "request_id": 58, + "tpot_ms": 99.22887081784184, + "ttft_ms": 386.3333674662499 + }, + { + "e2e_ms": 12989.881679406835, + "request_id": 59, + "tpot_ms": 99.2311627828068, + "ttft_ms": 387.52400599037173 + }, + { + "e2e_ms": 12991.363397481507, + "request_id": 60, + "tpot_ms": 99.23345474777176, + "ttft_ms": 388.71464451449356 + }, + { + "e2e_ms": 12896.836165730803, + "request_id": 61, + "tpot_ms": 99.23492287861664, + "ttft_ms": 294.0009601464908 + }, + { + "e2e_ms": 12689.81487892944, + "request_id": 62, + "tpot_ms": 97.59316693471084, + "ttft_ms": 295.4826782211626 + }, + { + "e2e_ms": 12448.118757066311, + "request_id": 63, + "tpot_ms": 95.67691266262345, + "ttft_ms": 297.1508489131338 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 218.6981554031372, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e09d047b2039786ee02cd4c59c081ca95e5197ae65723b6170e43f6c0924921a", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 13556.392075191316, + "request_id": 0, + "tpot_ms": 104.4466564602661, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 13715.056219396885, + "request_id": 1, + "tpot_ms": 105.11756526634255, + "ttft_ms": 365.12543057138026 + }, + { + "e2e_ms": 13775.965021971426, + "request_id": 2, + "tpot_ms": 105.6336255570719, + "ttft_ms": 360.4945762232949 + }, + { + "e2e_ms": 13838.18765138338, + "request_id": 3, + "tpot_ms": 106.07242606904647, + "ttft_ms": 366.98954061447785 + }, + { + "e2e_ms": 13802.282961907104, + "request_id": 4, + "tpot_ms": 106.21712281117094, + "ttft_ms": 312.70836488839393 + }, + { + "e2e_ms": 13864.505591319055, + "request_id": 5, + "tpot_ms": 106.32495967758555, + "ttft_ms": 361.2357122656906 + }, + { + "e2e_ms": 13828.78708091747, + "request_id": 6, + "tpot_ms": 106.35892091285005, + "ttft_ms": 321.20412498551156 + }, + { + "e2e_ms": 13892.323156276025, + "request_id": 7, + "tpot_ms": 106.3810710779423, + "ttft_ms": 381.9271293773525 + }, + { + "e2e_ms": 13856.60464587857, + "request_id": 8, + "tpot_ms": 106.3914161711828, + "ttft_ms": 344.89479213835494 + }, + { + "e2e_ms": 13820.886135481267, + "request_id": 9, + "tpot_ms": 106.39288214818258, + "ttft_ms": 308.9901026620794 + }, + { + "e2e_ms": 13883.108764897528, + "request_id": 10, + "tpot_ms": 106.39288214821651, + "ttft_ms": 371.21273207403146 + }, + { + "e2e_ms": 13847.390254500226, + "request_id": 11, + "tpot_ms": 106.39288214825024, + "ttft_ms": 335.4942216724446 + }, + { + "e2e_ms": 13811.485565028235, + "request_id": 12, + "tpot_ms": 106.39288214828399, + "ttft_ms": 299.589532196169 + }, + { + "e2e_ms": 13873.708194444496, + "request_id": 13, + "tpot_ms": 106.38107407728566, + "ttft_ms": 363.31178662921815 + }, + { + "e2e_ms": 13837.989684047194, + "request_id": 14, + "tpot_ms": 106.38107407728566, + "ttft_ms": 327.5932762319158 + }, + { + "e2e_ms": 13901.52575941027, + "request_id": 15, + "tpot_ms": 106.39141617135505, + "ttft_ms": 389.81590564817736 + }, + { + "e2e_ms": 13865.807249012967, + "request_id": 16, + "tpot_ms": 106.39141617135505, + "ttft_ms": 354.097395250875 + }, + { + "e2e_ms": 13830.088738615665, + "request_id": 17, + "tpot_ms": 106.3928821483211, + "ttft_ms": 318.19270577888403 + }, + { + "e2e_ms": 13893.62481397874, + "request_id": 18, + "tpot_ms": 106.40322424239051, + "ttft_ms": 380.4153351951456 + }, + { + "e2e_ms": 13857.906303581438, + "request_id": 19, + "tpot_ms": 106.40322424239051, + "ttft_ms": 344.69682479784325 + }, + { + "e2e_ms": 13822.187793184135, + "request_id": 20, + "tpot_ms": 106.40469021935657, + "ttft_ms": 308.79213532585226 + }, + { + "e2e_ms": 13884.410422593844, + "request_id": 21, + "tpot_ms": 106.39288214826952, + "ttft_ms": 372.51438976361584 + }, + { + "e2e_ms": 13848.691912190276, + "request_id": 22, + "tpot_ms": 106.39288214822018, + "ttft_ms": 336.7958793663135 + }, + { + "e2e_ms": 13812.78722271334, + "request_id": 23, + "tpot_ms": 106.39288214818123, + "ttft_ms": 300.8911898943225 + }, + { + "e2e_ms": 13875.009852123696, + "request_id": 24, + "tpot_ms": 106.38107407709929, + "ttft_ms": 364.6134443320861 + }, + { + "e2e_ms": 13839.291341720127, + "request_id": 25, + "tpot_ms": 106.38107407704995, + "ttft_ms": 328.89493393478375 + }, + { + "e2e_ms": 13902.827417078257, + "request_id": 26, + "tpot_ms": 106.39141617110043, + "ttft_ms": 391.11756334850156 + }, + { + "e2e_ms": 13867.108906674688, + "request_id": 27, + "tpot_ms": 106.39141617110043, + "ttft_ms": 355.39905294493224 + }, + { + "e2e_ms": 13831.390396271117, + "request_id": 28, + "tpot_ms": 106.39288214805607, + "ttft_ms": 319.49436346799587 + }, + { + "e2e_ms": 13894.926471629247, + "request_id": 29, + "tpot_ms": 106.40322424213302, + "ttft_ms": 381.7169928783528 + }, + { + "e2e_ms": 13859.207961225678, + "request_id": 30, + "tpot_ms": 106.40322424213302, + "ttft_ms": 345.9984824747835 + }, + { + "e2e_ms": 13823.48945082211, + "request_id": 31, + "tpot_ms": 106.40469021908869, + "ttft_ms": 310.0937929978471 + }, + { + "e2e_ms": 13885.712080232466, + "request_id": 32, + "tpot_ms": 106.39288214805609, + "ttft_ms": 373.8160474293437 + }, + { + "e2e_ms": 13849.993569828897, + "request_id": 33, + "tpot_ms": 106.39288214805609, + "ttft_ms": 338.0975370257744 + }, + { + "e2e_ms": 13814.08888035196, + "request_id": 34, + "tpot_ms": 106.39288214805607, + "ttft_ms": 302.192847548838 + }, + { + "e2e_ms": 13876.311509762318, + "request_id": 35, + "tpot_ms": 106.3810740770235, + "ttft_ms": 365.9151019803346 + }, + { + "e2e_ms": 13840.592999358747, + "request_id": 36, + "tpot_ms": 106.38107407702348, + "ttft_ms": 330.1965915767653 + }, + { + "e2e_ms": 13904.129074716877, + "request_id": 37, + "tpot_ms": 106.39141617110043, + "ttft_ms": 392.41922098712223 + }, + { + "e2e_ms": 13868.410564313308, + "request_id": 38, + "tpot_ms": 106.39141617110043, + "ttft_ms": 356.7007105835529 + }, + { + "e2e_ms": 13832.69205390974, + "request_id": 39, + "tpot_ms": 106.39288214805609, + "ttft_ms": 320.79602110661654 + }, + { + "e2e_ms": 13896.228129267869, + "request_id": 40, + "tpot_ms": 106.40322424213304, + "ttft_ms": 383.0186505169735 + }, + { + "e2e_ms": 13860.509618864298, + "request_id": 41, + "tpot_ms": 106.40322424213302, + "ttft_ms": 347.30014011340415 + }, + { + "e2e_ms": 13824.79110846073, + "request_id": 42, + "tpot_ms": 106.40469021908868, + "ttft_ms": 311.3954506364678 + }, + { + "e2e_ms": 13888.32718381886, + "request_id": 43, + "tpot_ms": 106.40322424213302, + "ttft_ms": 375.1177050679644 + }, + { + "e2e_ms": 13852.60867341529, + "request_id": 44, + "tpot_ms": 106.40322424213304, + "ttft_ms": 339.39919466439505 + }, + { + "e2e_ms": 13816.89016301172, + "request_id": 45, + "tpot_ms": 106.40469021908868, + "ttft_ms": 303.4945051874587 + }, + { + "e2e_ms": 13879.11279242067, + "request_id": 46, + "tpot_ms": 106.392882148045, + "ttft_ms": 367.2167596189553 + }, + { + "e2e_ms": 13843.394282013549, + "request_id": 47, + "tpot_ms": 106.39288214801702, + "ttft_ms": 331.49824921538595 + }, + { + "e2e_ms": 13807.489592530403, + "request_id": 48, + "tpot_ms": 106.39288214796812, + "ttft_ms": 295.5935597384496 + }, + { + "e2e_ms": 13869.712221935884, + "request_id": 49, + "tpot_ms": 106.38107407689715, + "ttft_ms": 359.3158141699462 + }, + { + "e2e_ms": 13833.993711528763, + "request_id": 50, + "tpot_ms": 106.38107407686918, + "ttft_ms": 323.59730376637685 + }, + { + "e2e_ms": 13897.5297868798, + "request_id": 51, + "tpot_ms": 106.3914161708903, + "ttft_ms": 385.8199331767338 + }, + { + "e2e_ms": 13861.81127647268, + "request_id": 52, + "tpot_ms": 106.39141617089378, + "ttft_ms": 350.1014227691712 + }, + { + "e2e_ms": 13826.092766065556, + "request_id": 53, + "tpot_ms": 106.39288214787034, + "ttft_ms": 314.1967332860247 + }, + { + "e2e_ms": 13889.628841416596, + "request_id": 54, + "tpot_ms": 106.40322424192983, + "ttft_ms": 376.41936269150733 + }, + { + "e2e_ms": 13853.910331009472, + "request_id": 55, + "tpot_ms": 106.4032242419298, + "ttft_ms": 340.7008522843853 + }, + { + "e2e_ms": 13818.191820602351, + "request_id": 56, + "tpot_ms": 106.4046902189064, + "ttft_ms": 304.7961628012388 + }, + { + "e2e_ms": 13880.414450007833, + "request_id": 57, + "tpot_ms": 106.39288214787034, + "ttft_ms": 368.5184172283016 + }, + { + "e2e_ms": 13844.695939600711, + "request_id": 58, + "tpot_ms": 106.39288214787034, + "ttft_ms": 332.79990682117955 + }, + { + "e2e_ms": 13600.134841430987, + "request_id": 59, + "tpot_ms": 104.74991829994453, + "ttft_ms": 296.89521733803304 + }, + { + "e2e_ms": 13451.749583806417, + "request_id": 60, + "tpot_ms": 103.07978040977419, + "ttft_ms": 360.61747176509584 + }, + { + "e2e_ms": 13168.506714577745, + "request_id": 61, + "tpot_ms": 101.13076971039189, + "ttft_ms": 324.8989613579738 + }, + { + "e2e_ms": 12970.21046401943, + "request_id": 62, + "tpot_ms": 99.07943994689744, + "ttft_ms": 387.1215907634564 + }, + { + "e2e_ms": 12659.030247905519, + "request_id": 63, + "tpot_ms": 96.91045013818255, + "ttft_ms": 351.4030803563344 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 144.67649912834167, + "offered_request_rate": 0.8, + "request_metrics_sha256": "6b246d56546536e46b4e8adfb23076b3cbf053da4b0af0288346df10122d6022", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 15423.90313583537, + "request_id": 0, + "tpot_ms": 119.15146796140039, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 15680.782114004433, + "request_id": 1, + "tpot_ms": 120.47488426274414, + "ttft_ms": 380.47181263592745 + }, + { + "e2e_ms": 15824.99160915063, + "request_id": 2, + "tpot_ms": 121.60222806639713, + "ttft_ms": 381.5086447181937 + }, + { + "e2e_ms": 15964.925902693089, + "request_id": 3, + "tpot_ms": 122.6571852387084, + "ttft_ms": 387.4633773771219 + }, + { + "e2e_ms": 15994.3969757696, + "request_id": 4, + "tpot_ms": 123.48492728290344, + "ttft_ms": 311.8112108408626 + }, + { + "e2e_ms": 16132.273068839513, + "request_id": 5, + "tpot_ms": 124.38017511477935, + "ttft_ms": 335.9908292625358 + }, + { + "e2e_ms": 16268.09096143688, + "request_id": 6, + "tpot_ms": 125.15372351796954, + "ttft_ms": 373.5680746547478 + }, + { + "e2e_ms": 16291.239371010352, + "request_id": 7, + "tpot_ms": 125.67763083904141, + "ttft_ms": 330.18025445209173 + }, + { + "e2e_ms": 16612.993923996273, + "request_id": 8, + "tpot_ms": 127.71634872045071, + "ttft_ms": 393.0176364990334 + }, + { + "e2e_ms": 16636.356874836954, + "request_id": 9, + "tpot_ms": 128.11558123152213, + "ttft_ms": 365.67805843364454 + }, + { + "e2e_ms": 16659.624174048608, + "request_id": 10, + "tpot_ms": 128.3236060799542, + "ttft_ms": 362.52620189442376 + }, + { + "e2e_ms": 16683.03953534565, + "request_id": 11, + "tpot_ms": 128.34365422438844, + "ttft_ms": 383.3954488483169 + }, + { + "e2e_ms": 16598.14552827833, + "request_id": 12, + "tpot_ms": 128.20865630197483, + "ttft_ms": 315.6461779275297 + }, + { + "e2e_ms": 16621.508479120537, + "request_id": 13, + "tpot_ms": 128.0747814081899, + "ttft_ms": 356.0112402804201 + }, + { + "e2e_ms": 16644.871429963147, + "request_id": 14, + "tpot_ms": 127.97565080664988, + "ttft_ms": 391.9637775186118 + }, + { + "e2e_ms": 16559.977422897646, + "request_id": 15, + "tpot_ms": 127.91134905305496, + "ttft_ms": 315.2360931596654 + }, + { + "e2e_ms": 16583.340373740255, + "request_id": 16, + "tpot_ms": 127.86325360239432, + "ttft_ms": 344.7071662361765 + }, + { + "e2e_ms": 16606.703324582868, + "request_id": 17, + "tpot_ms": 127.81515815173371, + "ttft_ms": 374.17823931268754 + }, + { + "e2e_ms": 16630.066275425477, + "request_id": 18, + "tpot_ms": 127.7832690040065, + "ttft_ms": 401.59111191665176 + }, + { + "e2e_ms": 16354.971144941388, + "request_id": 19, + "tpot_ms": 126.2873116642462, + "ttft_ms": 316.48256358212024 + }, + { + "e2e_ms": 16568.46874002116, + "request_id": 20, + "tpot_ms": 127.78612414854776, + "ttft_ms": 339.6309731555931 + }, + { + "e2e_ms": 16591.83169086377, + "request_id": 21, + "tpot_ms": 127.78612414856296, + "ttft_ms": 362.9939239962745 + }, + { + "e2e_ms": 16615.19464170638, + "request_id": 22, + "tpot_ms": 127.78612414857815, + "ttft_ms": 386.3568748369559 + }, + { + "e2e_ms": 16339.951449136564, + "request_id": 23, + "tpot_ms": 126.28847750749865, + "ttft_ms": 301.3148056842354 + }, + { + "e2e_ms": 16553.449044216337, + "request_id": 24, + "tpot_ms": 127.78518800972483, + "ttft_ms": 324.73016698128276 + }, + { + "e2e_ms": 16576.811995058946, + "request_id": 25, + "tpot_ms": 127.78477532898123, + "ttft_ms": 348.1455282783301 + }, + { + "e2e_ms": 16600.174945901555, + "request_id": 26, + "tpot_ms": 127.78477532898441, + "ttft_ms": 371.50847912053564 + }, + { + "e2e_ms": 16623.53789674417, + "request_id": 27, + "tpot_ms": 127.78477532898442, + "ttft_ms": 394.87142996314617 + }, + { + "e2e_ms": 16348.29470417435, + "request_id": 28, + "tpot_ms": 126.28596284469846, + "ttft_ms": 309.97742289764574 + }, + { + "e2e_ms": 16561.79229925412, + "request_id": 29, + "tpot_ms": 127.78308602766822, + "ttft_ms": 333.34037374025627 + }, + { + "e2e_ms": 16585.155250096734, + "request_id": 30, + "tpot_ms": 127.78308602766825, + "ttft_ms": 356.7033245828668 + }, + { + "e2e_ms": 16608.518200939343, + "request_id": 31, + "tpot_ms": 127.78308602766823, + "ttft_ms": 380.06627542547733 + }, + { + "e2e_ms": 16631.881151781956, + "request_id": 32, + "tpot_ms": 127.78360948578973, + "ttft_ms": 403.36274708666053 + }, + { + "e2e_ms": 16356.786021297865, + "request_id": 33, + "tpot_ms": 126.28596284469846, + "ttft_ms": 318.4687400211601 + }, + { + "e2e_ms": 16570.283616377637, + "request_id": 34, + "tpot_ms": 127.78308602766825, + "ttft_ms": 341.83169086377063 + }, + { + "e2e_ms": 16593.646567220247, + "request_id": 35, + "tpot_ms": 127.78308602766823, + "ttft_ms": 365.19464170638116 + }, + { + "e2e_ms": 16617.00951806286, + "request_id": 36, + "tpot_ms": 127.78477532898444, + "ttft_ms": 388.34305128183644 + }, + { + "e2e_ms": 16341.766325493041, + "request_id": 37, + "tpot_ms": 126.28596284469846, + "ttft_ms": 303.449044216336 + }, + { + "e2e_ms": 16555.263920572714, + "request_id": 38, + "tpot_ms": 127.78308602766747, + "ttft_ms": 326.81199505894654 + }, + { + "e2e_ms": 16578.62687141207, + "request_id": 39, + "tpot_ms": 127.78308602764184, + "ttft_ms": 350.17494590155707 + }, + { + "e2e_ms": 16601.989822251424, + "request_id": 40, + "tpot_ms": 127.78308602761618, + "ttft_ms": 373.5378967441676 + }, + { + "e2e_ms": 16625.35277309078, + "request_id": 41, + "tpot_ms": 127.78477532890675, + "ttft_ms": 396.6863063196229 + }, + { + "e2e_ms": 16350.109580518321, + "request_id": 42, + "tpot_ms": 126.2859628446, + "ttft_ms": 311.79229925412244 + }, + { + "e2e_ms": 16563.607175594883, + "request_id": 43, + "tpot_ms": 127.78308602754448, + "ttft_ms": 335.15525009673297 + }, + { + "e2e_ms": 16586.970126434237, + "request_id": 44, + "tpot_ms": 127.78308602751885, + "ttft_ms": 358.5182009393435 + }, + { + "e2e_ms": 16610.333077273594, + "request_id": 45, + "tpot_ms": 127.78308602749323, + "ttft_ms": 381.88115178195403 + }, + { + "e2e_ms": 16633.69602811295, + "request_id": 46, + "tpot_ms": 127.78360948558907, + "ttft_ms": 405.17762344313724 + }, + { + "e2e_ms": 16548.802021044266, + "request_id": 47, + "tpot_ms": 127.78360948556401, + "ttft_ms": 320.2836163776368 + }, + { + "e2e_ms": 16572.164971883623, + "request_id": 48, + "tpot_ms": 127.7836094855384, + "ttft_ms": 343.64656722024733 + }, + { + "e2e_ms": 16595.52792272298, + "request_id": 49, + "tpot_ms": 127.78360948551277, + "ttft_ms": 367.00951806285786 + }, + { + "e2e_ms": 16618.890873562337, + "request_id": 50, + "tpot_ms": 127.78529878680334, + "ttft_ms": 390.15792763831314 + }, + { + "e2e_ms": 16343.647680989874, + "request_id": 51, + "tpot_ms": 126.28648630249732, + "ttft_ms": 305.26392057271323 + }, + { + "e2e_ms": 16199.563507361632, + "request_id": 52, + "tpot_ms": 124.96800500747688, + "ttft_ms": 328.6268714120695 + }, + { + "e2e_ms": 16032.332947106512, + "request_id": 53, + "tpot_ms": 123.46726869964634, + "ttft_ms": 351.9898222514257 + }, + { + "e2e_ms": 15838.885472588829, + "request_id": 54, + "tpot_ms": 121.7600999960476, + "ttft_ms": 375.35277309078197 + }, + { + "e2e_ms": 15619.209002299955, + "request_id": 55, + "tpot_ms": 119.84809306800307, + "ttft_ms": 398.5011826635656 + }, + { + "e2e_ms": 15285.473398871076, + "request_id": 56, + "tpot_ms": 117.8887104194976, + "ttft_ms": 313.60717559488194 + }, + { + "e2e_ms": 15049.070462905931, + "request_id": 57, + "tpot_ms": 115.84330973599758, + "ttft_ms": 336.9701264342382 + }, + { + "e2e_ms": 14793.47811125659, + "request_id": 58, + "tpot_ms": 113.6468112912047, + "ttft_ms": 360.3330772735944 + }, + { + "e2e_ms": 14524.751300129523, + "request_id": 59, + "tpot_ms": 111.34689190564231, + "ttft_ms": 383.69602811295067 + }, + { + "e2e_ms": 14156.606828520311, + "request_id": 60, + "tpot_ms": 109.11657328721294, + "ttft_ms": 298.802021044267 + }, + { + "e2e_ms": 13870.414276689999, + "request_id": 61, + "tpot_ms": 106.67912838430216, + "ttft_ms": 322.16497188362325 + }, + { + "e2e_ms": 13579.457505623892, + "request_id": 62, + "tpot_ms": 104.20416994410166, + "ttft_ms": 345.5279227229795 + }, + { + "e2e_ms": 13281.118281252404, + "request_id": 63, + "tpot_ms": 101.6710819503155, + "ttft_ms": 368.89087356233574 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.15625, + "passed": 10 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 138.42228722572327, + "offered_request_rate": 1.2, + "request_metrics_sha256": "dfafc592cb8f7c26725458265ca04cb89642135218d5f80993dfbd6e83fcea9c", + "request_rate_per_gpu": 0.15, + "requests": [ + { + "e2e_ms": 17349.6420929535, + "request_id": 0, + "tpot_ms": 134.3147668363463, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 17472.38469413562, + "request_id": 1, + "tpot_ms": 135.04025977135157, + "ttft_ms": 322.27170317397037 + }, + { + "e2e_ms": 17594.570506858923, + "request_id": 2, + "tpot_ms": 135.72808492940857, + "ttft_ms": 357.1037208240346 + }, + { + "e2e_ms": 17607.2367009099, + "request_id": 3, + "tpot_ms": 136.28731175562888, + "ttft_ms": 298.7481079450336 + }, + { + "e2e_ms": 17921.914449676293, + "request_id": 4, + "tpot_ms": 138.39011926907332, + "ttft_ms": 346.36930250398024 + }, + { + "e2e_ms": 17935.406822135243, + "request_id": 5, + "tpot_ms": 138.88107821741167, + "ttft_ms": 297.50988852396 + }, + { + "e2e_ms": 18057.925904932094, + "request_id": 6, + "tpot_ms": 139.39521236971183, + "ttft_ms": 354.73393397869125 + }, + { + "e2e_ms": 18071.482157513437, + "request_id": 7, + "tpot_ms": 139.75067177492934, + "ttft_ms": 323.14684209741085 + }, + { + "e2e_ms": 18194.065120430685, + "request_id": 8, + "tpot_ms": 140.142283173617, + "ttft_ms": 395.99515738132584 + }, + { + "e2e_ms": 18207.834601589766, + "request_id": 9, + "tpot_ms": 140.4257392143267, + "ttft_ms": 373.76572137027256 + }, + { + "e2e_ms": 18221.607066695313, + "request_id": 10, + "tpot_ms": 140.60137526077196, + "ttft_ms": 365.232408577274 + }, + { + "e2e_ms": 18235.379531799863, + "request_id": 11, + "tpot_ms": 140.670460527253, + "ttft_ms": 370.23104483873226 + }, + { + "e2e_ms": 18249.21289308034, + "request_id": 12, + "tpot_ms": 140.64257855984852, + "ttft_ms": 387.6054159795803 + }, + { + "e2e_ms": 18263.046254360816, + "request_id": 13, + "tpot_ms": 140.69995603545655, + "ttft_ms": 394.1518378578337 + }, + { + "e2e_ms": 18276.879615640297, + "request_id": 14, + "tpot_ms": 140.75508408264292, + "ttft_ms": 400.98393714464817 + }, + { + "e2e_ms": 18181.536918127527, + "request_id": 15, + "tpot_ms": 140.80327015717634, + "ttft_ms": 299.52160816613116 + }, + { + "e2e_ms": 18195.15705083027, + "request_id": 16, + "tpot_ms": 140.84624138271937, + "ttft_ms": 307.6843952249071 + }, + { + "e2e_ms": 18208.77718353195, + "request_id": 17, + "tpot_ms": 140.87800086343316, + "ttft_ms": 317.2710738759417 + }, + { + "e2e_ms": 18222.54666469151, + "request_id": 18, + "tpot_ms": 140.90607860065492, + "ttft_ms": 327.4746824083348 + }, + { + "e2e_ms": 18236.319129797193, + "request_id": 19, + "tpot_ms": 140.91088449209656, + "ttft_ms": 340.6367993009294 + }, + { + "e2e_ms": 18250.152491077173, + "request_id": 20, + "tpot_ms": 140.8981539798204, + "ttft_ms": 356.0869356399827 + }, + { + "e2e_ms": 18263.985852358146, + "request_id": 21, + "tpot_ms": 140.89857595079965, + "ttft_ms": 369.8667066065901 + }, + { + "e2e_ms": 18277.81921363912, + "request_id": 22, + "tpot_ms": 140.90338208286406, + "ttft_ms": 383.0896891153834 + }, + { + "e2e_ms": 18291.6525749191, + "request_id": 23, + "tpot_ms": 140.91257237601366, + "ttft_ms": 395.7558831653678 + }, + { + "e2e_ms": 18305.48593620007, + "request_id": 24, + "tpot_ms": 140.9217626691632, + "ttft_ms": 408.4220772163469 + }, + { + "e2e_ms": 18210.143238687502, + "request_id": 25, + "tpot_ms": 140.92327164841032, + "ttft_ms": 312.8877393393914 + }, + { + "e2e_ms": 18223.912719846063, + "request_id": 26, + "tpot_ms": 140.92545360667498, + "ttft_ms": 326.38011179834336 + }, + { + "e2e_ms": 18237.68518495175, + "request_id": 27, + "tpot_ms": 140.92765906057843, + "ttft_ms": 339.8724842582901 + }, + { + "e2e_ms": 18251.518546232724, + "request_id": 28, + "tpot_ms": 140.92984101884323, + "ttft_ms": 353.42873683963205 + }, + { + "e2e_ms": 18265.3519075127, + "request_id": 29, + "tpot_ms": 140.93202297710803, + "ttft_ms": 366.98498941997923 + }, + { + "e2e_ms": 18279.185268793677, + "request_id": 30, + "tpot_ms": 140.93370194229104, + "ttft_ms": 380.60512212271647 + }, + { + "e2e_ms": 18293.01863007465, + "request_id": 31, + "tpot_ms": 140.9341814397353, + "ttft_ms": 394.3775872282629 + }, + { + "e2e_ms": 18306.851991354626, + "request_id": 32, + "tpot_ms": 140.93466093717961, + "ttft_ms": 408.15005233281454 + }, + { + "e2e_ms": 18211.509293842057, + "request_id": 33, + "tpot_ms": 140.933484965084, + "ttft_ms": 312.956703276388 + }, + { + "e2e_ms": 18225.278775001614, + "request_id": 34, + "tpot_ms": 140.9329819720059, + "ttft_ms": 326.79006455686374 + }, + { + "e2e_ms": 18239.11213628159, + "request_id": 35, + "tpot_ms": 140.93298197200983, + "ttft_ms": 340.6234258363447 + }, + { + "e2e_ms": 18252.945497562563, + "request_id": 36, + "tpot_ms": 140.93298197201372, + "ttft_ms": 354.45678711682046 + }, + { + "e2e_ms": 18266.778858843536, + "request_id": 37, + "tpot_ms": 140.93466093719667, + "ttft_ms": 368.0769198195577 + }, + { + "e2e_ms": 18280.612220123516, + "request_id": 38, + "tpot_ms": 140.93633990237933, + "ttft_ms": 381.69705252133923 + }, + { + "e2e_ms": 18294.44558140449, + "request_id": 39, + "tpot_ms": 140.93801886756114, + "ttft_ms": 395.3171852242221 + }, + { + "e2e_ms": 18308.27894268546, + "request_id": 40, + "tpot_ms": 140.93849836500434, + "ttft_ms": 409.0896503299106 + }, + { + "e2e_ms": 18212.936245171895, + "request_id": 41, + "tpot_ms": 140.93732239290452, + "ttft_ms": 313.8963012730187 + }, + { + "e2e_ms": 18226.708710277584, + "request_id": 42, + "tpot_ms": 140.93684289546135, + "ttft_ms": 327.72966255399183 + }, + { + "e2e_ms": 18047.749752064, + "request_id": 43, + "tpot_ms": 139.41879313566167, + "ttft_ms": 341.56302383496495 + }, + { + "e2e_ms": 17872.42465139999, + "request_id": 44, + "tpot_ms": 137.92935642744132, + "ttft_ms": 355.3963851149433 + }, + { + "e2e_ms": 17695.178674250878, + "request_id": 45, + "tpot_ms": 136.4247947075194, + "ttft_ms": 369.22974639591644 + }, + { + "e2e_ms": 17514.86814366097, + "request_id": 46, + "tpot_ms": 134.89610264554392, + "ttft_ms": 383.06310767688956 + }, + { + "e2e_ms": 17334.24324324006, + "request_id": 47, + "tpot_ms": 133.36661419575813, + "ttft_ms": 396.68324037877767 + }, + { + "e2e_ms": 17152.122611965817, + "request_id": 48, + "tpot_ms": 131.8241488699319, + "ttft_ms": 410.4557054844662 + }, + { + "e2e_ms": 16860.906258226023, + "request_id": 49, + "tpot_ms": 130.28066064407446, + "ttft_ms": 315.26235642856903 + }, + { + "e2e_ms": 16678.317507592936, + "request_id": 50, + "tpot_ms": 128.73402984160936, + "ttft_ms": 329.0957177085474 + }, + { + "e2e_ms": 16495.414294222966, + "request_id": 51, + "tpot_ms": 127.18492295459406, + "ttft_ms": 342.9290789895205 + }, + { + "e2e_ms": 16325.716031493954, + "request_id": 52, + "tpot_ms": 125.73979205687765, + "ttft_ms": 356.76244027049364 + }, + { + "e2e_ms": 16142.044362006885, + "request_id": 53, + "tpot_ms": 124.18463433430247, + "ttft_ms": 370.595801550472 + }, + { + "e2e_ms": 15942.642543961227, + "request_id": 54, + "tpot_ms": 122.50561717425025, + "ttft_ms": 384.4291628314451 + }, + { + "e2e_ms": 15727.503328455541, + "request_id": 55, + "tpot_ms": 120.70436246394655, + "ttft_ms": 398.049295534328 + }, + { + "e2e_ms": 15403.205770995846, + "request_id": 56, + "tpot_ms": 118.90039231904261, + "ttft_ms": 302.8559464774361 + }, + { + "e2e_ms": 15178.030676088234, + "request_id": 57, + "tpot_ms": 117.01843597110097, + "ttft_ms": 316.6893077584092 + }, + { + "e2e_ms": 14941.341931765628, + "request_id": 58, + "tpot_ms": 115.04582096634839, + "ttft_ms": 330.5226690393823 + }, + { + "e2e_ms": 14696.772511761146, + "request_id": 59, + "tpot_ms": 113.01115339717941, + "ttft_ms": 344.3560303193607 + }, + { + "e2e_ms": 14451.34286402164, + "request_id": 60, + "tpot_ms": 110.96971238127013, + "ttft_ms": 358.1893916003338 + }, + { + "e2e_ms": 14196.293999590453, + "request_id": 61, + "tpot_ms": 108.85252950164681, + "ttft_ms": 372.02275288130693 + }, + { + "e2e_ms": 13938.386603616791, + "request_id": 62, + "tpot_ms": 106.71283849964966, + "ttft_ms": 385.8561141612853 + }, + { + "e2e_ms": 13676.049735660896, + "request_id": 63, + "tpot_ms": 104.53992523504405, + "ttft_ms": 399.4792308103001 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 163.37065601348877, + "offered_request_rate": 1.6, + "request_metrics_sha256": "e85085ea5517c1afa9f850cb838a638360d45db695cde8d3c9fef6baf6af44c4", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 19447.058061600903, + "request_id": 0, + "tpot_ms": 150.8298532036487, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 19750.640261300512, + "request_id": 1, + "tpot_ms": 152.83450342712686, + "ttft_ms": 340.65832605540146 + }, + { + "e2e_ms": 19750.02534103151, + "request_id": 2, + "tpot_ms": 153.17602578595117, + "ttft_ms": 296.67006621570977 + }, + { + "e2e_ms": 19855.93358266601, + "request_id": 3, + "tpot_ms": 153.5864661294274, + "ttft_ms": 350.4523842287295 + }, + { + "e2e_ms": 19855.232098385564, + "request_id": 4, + "tpot_ms": 153.87573615619343, + "ttft_ms": 313.01360654900145 + }, + { + "e2e_ms": 19961.05377600862, + "request_id": 5, + "tpot_ms": 154.2244676371592, + "ttft_ms": 374.54638608940184 + }, + { + "e2e_ms": 19960.265727716724, + "request_id": 6, + "tpot_ms": 154.4718650613629, + "ttft_ms": 342.33886492363473 + }, + { + "e2e_ms": 19959.47767942483, + "request_id": 7, + "tpot_ms": 154.6679176552935, + "ttft_ms": 316.6521372025519 + }, + { + "e2e_ms": 20065.255011109733, + "request_id": 8, + "tpot_ms": 154.88778812601134, + "ttft_ms": 394.50591910629294 + }, + { + "e2e_ms": 20064.42261687969, + "request_id": 9, + "tpot_ms": 155.03615010045385, + "ttft_ms": 374.8315541220517 + }, + { + "e2e_ms": 20063.590222649644, + "request_id": 10, + "tpot_ms": 155.117974090142, + "ttft_ms": 363.6075132016101 + }, + { + "e2e_ms": 20062.7578284196, + "request_id": 11, + "tpot_ms": 155.13454318347627, + "ttft_ms": 360.67084411811567 + }, + { + "e2e_ms": 20061.925434189554, + "request_id": 12, + "tpot_ms": 155.09034115377113, + "ttft_ms": 365.45210766062206 + }, + { + "e2e_ms": 20061.09303995951, + "request_id": 13, + "tpot_ms": 155.09665427482798, + "ttft_ms": 363.8179470563543 + }, + { + "e2e_ms": 20060.260645729464, + "request_id": 14, + "tpot_ms": 155.10154332898202, + "ttft_ms": 362.3646429487479 + }, + { + "e2e_ms": 20059.42825149942, + "request_id": 15, + "tpot_ms": 155.10604675613317, + "ttft_ms": 360.96031347050683 + }, + { + "e2e_ms": 20058.595857269374, + "request_id": 16, + "tpot_ms": 155.10168661355542, + "ttft_ms": 360.6816573478344 + }, + { + "e2e_ms": 20057.76346303933, + "request_id": 17, + "tpot_ms": 155.0900405289204, + "ttft_ms": 361.3283158664373 + }, + { + "e2e_ms": 20056.931068809285, + "request_id": 18, + "tpot_ms": 155.07436184636043, + "ttft_ms": 362.4871143215085 + }, + { + "e2e_ms": 20056.09867457924, + "request_id": 19, + "tpot_ms": 155.0434256316313, + "ttft_ms": 365.58361936206654 + }, + { + "e2e_ms": 20055.26628034948, + "request_id": 20, + "tpot_ms": 154.9995151863024, + "ttft_ms": 370.32785168907446 + }, + { + "e2e_ms": 20054.433886119732, + "request_id": 21, + "tpot_ms": 154.96275487551375, + "ttft_ms": 374.16401692948574 + }, + { + "e2e_ms": 20053.601491889985, + "request_id": 22, + "tpot_ms": 154.92303029896564, + "ttft_ms": 378.3766439213494 + }, + { + "e2e_ms": 20052.76909766024, + "request_id": 23, + "tpot_ms": 154.88711594543284, + "ttft_ms": 382.105372590269 + }, + { + "e2e_ms": 20051.936703430492, + "request_id": 24, + "tpot_ms": 154.8482609975464, + "ttft_ms": 386.20755674210193 + }, + { + "e2e_ms": 20051.104309200746, + "request_id": 25, + "tpot_ms": 154.85607608193587, + "ttft_ms": 384.3826467948901 + }, + { + "e2e_ms": 20050.271914971, + "request_id": 26, + "tpot_ms": 154.86595665257255, + "ttft_ms": 382.2954200942838 + }, + { + "e2e_ms": 20049.439520741253, + "request_id": 27, + "tpot_ms": 154.87441181850545, + "ttft_ms": 380.3892197910592 + }, + { + "e2e_ms": 20048.607126511506, + "request_id": 28, + "tpot_ms": 154.87585298283847, + "ttft_ms": 379.37379769102006 + }, + { + "e2e_ms": 20047.77473228176, + "request_id": 29, + "tpot_ms": 154.8790680483812, + "ttft_ms": 378.13309013734295 + }, + { + "e2e_ms": 20046.942338052013, + "request_id": 30, + "tpot_ms": 154.879633497606, + "ttft_ms": 377.228883856052 + }, + { + "e2e_ms": 20046.109943822266, + "request_id": 31, + "tpot_ms": 154.8807461594705, + "ttft_ms": 376.2551815695119 + }, + { + "e2e_ms": 20045.27754959252, + "request_id": 32, + "tpot_ms": 154.8790337660788, + "ttft_ms": 375.64026130051076 + }, + { + "e2e_ms": 19846.84310747305, + "request_id": 33, + "tpot_ms": 153.32139973576017, + "ttft_ms": 375.02534103150964 + }, + { + "e2e_ms": 19648.543544766126, + "request_id": 34, + "tpot_ms": 151.76482774805996, + "ttft_ms": 374.4104207625085 + }, + { + "e2e_ms": 19450.677394587543, + "request_id": 35, + "tpot_ms": 150.2123500638227, + "ttft_ms": 373.7089364820605 + }, + { + "e2e_ms": 19253.237256186894, + "request_id": 36, + "tpot_ms": 148.66322680303372, + "ttft_ms": 373.00745220161247 + }, + { + "e2e_ms": 19054.80482620875, + "request_id": 37, + "tpot_ms": 147.10697182912625, + "ttft_ms": 372.2194039097175 + }, + { + "e2e_ms": 18856.422732154002, + "request_id": 38, + "tpot_ms": 145.55111320107227, + "ttft_ms": 371.4313556178226 + }, + { + "e2e_ms": 18658.616312962055, + "request_id": 39, + "tpot_ms": 143.99978744595376, + "ttft_ms": 370.64330732592765 + }, + { + "e2e_ms": 18469.244245876795, + "request_id": 40, + "tpot_ms": 142.51522309276308, + "ttft_ms": 369.81091309588265 + }, + { + "e2e_ms": 18279.907702313452, + "request_id": 41, + "tpot_ms": 141.0309384523434, + "ttft_ms": 368.97851886583766 + }, + { + "e2e_ms": 18091.61193748763, + "request_id": 42, + "tpot_ms": 139.55484892009318, + "ttft_ms": 368.14612463579266 + }, + { + "e2e_ms": 17903.060652176137, + "request_id": 43, + "tpot_ms": 138.07674741551486, + "ttft_ms": 367.31373040574766 + }, + { + "e2e_ms": 17716.73280728946, + "request_id": 44, + "tpot_ms": 136.61615331585634, + "ttft_ms": 366.48133617570267 + }, + { + "e2e_ms": 17529.12437807872, + "request_id": 45, + "tpot_ms": 135.14547587506345, + "ttft_ms": 365.64894194565767 + }, + { + "e2e_ms": 17339.472913240785, + "request_id": 46, + "tpot_ms": 133.65871153956826, + "ttft_ms": 364.81654771561267 + }, + { + "e2e_ms": 17149.611868516175, + "request_id": 47, + "tpot_ms": 132.17029696874496, + "ttft_ms": 363.9841534855677 + }, + { + "e2e_ms": 16958.75366988868, + "request_id": 48, + "tpot_ms": 130.67403079238707, + "ttft_ms": 363.1517592555227 + }, + { + "e2e_ms": 16767.44725356325, + "request_id": 49, + "tpot_ms": 129.1742353428171, + "ttft_ms": 362.3193650254777 + }, + { + "e2e_ms": 16576.276975363853, + "request_id": 50, + "tpot_ms": 127.67551184699543, + "ttft_ms": 361.4869707954327 + }, + { + "e2e_ms": 16384.89705533921, + "request_id": 51, + "tpot_ms": 126.1751376281405, + "ttft_ms": 360.65457656536637 + }, + { + "e2e_ms": 16202.320435741867, + "request_id": 52, + "tpot_ms": 124.74408073548226, + "ttft_ms": 359.8221823356198 + }, + { + "e2e_ms": 16010.428211639819, + "request_id": 53, + "tpot_ms": 123.23967262625155, + "ttft_ms": 358.98978810587323 + }, + { + "e2e_ms": 15808.049221831383, + "request_id": 54, + "tpot_ms": 121.6526915587028, + "ttft_ms": 358.15739387612666 + }, + { + "e2e_ms": 15595.17863371626, + "request_id": 55, + "tpot_ms": 119.98309948086519, + "ttft_ms": 357.3249996463801 + }, + { + "e2e_ms": 15377.4072544157, + "request_id": 56, + "tpot_ms": 118.27491849605565, + "ttft_ms": 356.4926054166335 + }, + { + "e2e_ms": 15157.846080032627, + "request_id": 57, + "tpot_ms": 116.55264463658062, + "ttft_ms": 355.66021118688695 + }, + { + "e2e_ms": 14930.609139372891, + "request_id": 58, + "tpot_ms": 114.76993167256497, + "ttft_ms": 354.8278169571404 + }, + { + "e2e_ms": 14698.1184149259, + "request_id": 59, + "tpot_ms": 112.9458503322717, + "ttft_ms": 353.9954227273938 + }, + { + "e2e_ms": 14465.054205320868, + "request_id": 60, + "tpot_ms": 111.11725336081277, + "ttft_ms": 353.16302849764725 + }, + { + "e2e_ms": 14225.577184589198, + "request_id": 61, + "tpot_ms": 109.23816181355352, + "ttft_ms": 352.3306342679007 + }, + { + "e2e_ms": 13984.194476163324, + "request_id": 62, + "tpot_ms": 107.34406485137929, + "ttft_ms": 351.4982400381541 + }, + { + "e2e_ms": 13739.858786414643, + "request_id": 63, + "tpot_ms": 105.42671606776564, + "ttft_ms": 350.66584580840754 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 392.34758615493774, + "offered_request_rate": 2.4, + "request_metrics_sha256": "c4ae9e9bb2050189ca506883c0c12efd5faecce0ec6249ad641d32f3b9a7f130", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 25856.75271176486, + "request_id": 0, + "tpot_ms": 201.29988981911293, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26179.284600266838, + "request_id": 1, + "tpot_ms": 203.30897363251975, + "ttft_ms": 359.0449489368311 + }, + { + "e2e_ms": 25981.87281483019, + "request_id": 2, + "tpot_ms": 201.96786246415923, + "ttft_ms": 331.9542818819672 + }, + { + "e2e_ms": 25784.35394549511, + "request_id": 3, + "tpot_ms": 200.62102535110301, + "ttft_ms": 305.48372590502584 + }, + { + "e2e_ms": 25696.083985341404, + "request_id": 4, + "tpot_ms": 199.35042104178692, + "ttft_ms": 378.5805130344673 + }, + { + "e2e_ms": 25498.25455731707, + "request_id": 5, + "tpot_ms": 197.981349219281, + "ttft_ms": 354.62320646838384 + }, + { + "e2e_ms": 25300.295013812065, + "request_id": 6, + "tpot_ms": 196.59886557857163, + "ttft_ms": 332.2390853334691 + }, + { + "e2e_ms": 25102.100957670827, + "request_id": 7, + "tpot_ms": 195.19408059762435, + "ttft_ms": 312.45272177253236 + }, + { + "e2e_ms": 25012.968951433395, + "request_id": 8, + "tpot_ms": 193.83981874911316, + "ttft_ms": 395.3119702960262 + }, + { + "e2e_ms": 24814.29543067174, + "request_id": 9, + "tpot_ms": 192.41025790813484, + "ttft_ms": 378.1926763386156 + }, + { + "e2e_ms": 24615.47335754875, + "request_id": 10, + "tpot_ms": 190.95429488392014, + "ttft_ms": 364.2779072908935 + }, + { + "e2e_ms": 24416.46039907787, + "request_id": 11, + "tpot_ms": 189.47286981272285, + "ttft_ms": 353.405932862068 + }, + { + "e2e_ms": 24216.993764693514, + "request_id": 12, + "tpot_ms": 187.96377697549212, + "ttft_ms": 345.59408880601427 + }, + { + "e2e_ms": 24017.241128749425, + "request_id": 13, + "tpot_ms": 186.4682030159872, + "ttft_ms": 335.7793457190477 + }, + { + "e2e_ms": 23816.999491057053, + "request_id": 14, + "tpot_ms": 184.96817994362138, + "ttft_ms": 326.0406382171395 + }, + { + "e2e_ms": 23616.430479744246, + "request_id": 15, + "tpot_ms": 183.46465752234423, + "ttft_ms": 316.41897440652934 + }, + { + "e2e_ms": 23412.66959473249, + "request_id": 16, + "tpot_ms": 181.92890330417333, + "ttft_ms": 307.69887510247963 + }, + { + "e2e_ms": 23315.084360363784, + "request_id": 17, + "tpot_ms": 180.37041078851854, + "ttft_ms": 408.042190221928 + }, + { + "e2e_ms": 23111.095991700124, + "request_id": 18, + "tpot_ms": 178.82629774094855, + "ttft_ms": 400.1561785996568 + }, + { + "e2e_ms": 22907.001651879975, + "request_id": 19, + "tpot_ms": 177.27413055190453, + "ttft_ms": 393.18707178810007 + }, + { + "e2e_ms": 22702.9918523004, + "request_id": 20, + "tpot_ms": 175.71469647400258, + "ttft_ms": 387.22540010207496 + }, + { + "e2e_ms": 22499.024594644914, + "request_id": 21, + "tpot_ms": 174.1539791842547, + "ttft_ms": 381.46923824456815 + }, + { + "e2e_ms": 22294.98921706886, + "request_id": 22, + "tpot_ms": 172.59076723758398, + "ttft_ms": 375.96177789569475 + }, + { + "e2e_ms": 22091.055170736552, + "request_id": 23, + "tpot_ms": 171.02806584364455, + "ttft_ms": 370.49080859369354 + }, + { + "e2e_ms": 21886.829371806663, + "request_id": 24, + "tpot_ms": 169.4599867344259, + "ttft_ms": 365.4110565345743 + }, + { + "e2e_ms": 21682.517380669786, + "request_id": 25, + "tpot_ms": 167.90637996068475, + "ttft_ms": 358.40712566282207 + }, + { + "e2e_ms": 21478.40806232245, + "request_id": 26, + "tpot_ms": 166.35416808544093, + "ttft_ms": 351.4287154714495 + }, + { + "e2e_ms": 21274.47246393178, + "request_id": 27, + "tpot_ms": 164.80170050625136, + "ttft_ms": 344.6564996378552 + }, + { + "e2e_ms": 21070.655727841604, + "request_id": 28, + "tpot_ms": 163.24706150884583, + "ttft_ms": 338.278916218183 + }, + { + "e2e_ms": 20866.83848786135, + "request_id": 29, + "tpot_ms": 161.69251523328387, + "ttft_ms": 331.88905323430083 + }, + { + "e2e_ms": 20663.178771213603, + "request_id": 30, + "tpot_ms": 160.1348533340772, + "ttft_ms": 326.0523977857979 + }, + { + "e2e_ms": 20459.48769489552, + "request_id": 31, + "tpot_ms": 158.57430185728822, + "ttft_ms": 320.55135901991605 + }, + { + "e2e_ms": 20255.951788183756, + "request_id": 32, + "tpot_ms": 157.01541837295076, + "ttft_ms": 314.9936548190073 + }, + { + "e2e_ms": 20052.589009493837, + "request_id": 33, + "tpot_ms": 155.45687267778135, + "ttft_ms": 309.5661794156026 + }, + { + "e2e_ms": 19955.947670796646, + "request_id": 34, + "tpot_ms": 153.8983040696114, + "ttft_ms": 410.8630539559979 + }, + { + "e2e_ms": 19752.847929041352, + "request_id": 35, + "tpot_ms": 152.3343481355919, + "ttft_ms": 406.3857158211803 + }, + { + "e2e_ms": 19549.96119317403, + "request_id": 36, + "tpot_ms": 150.77184485827368, + "ttft_ms": 401.93689617327345 + }, + { + "e2e_ms": 19346.57831151796, + "request_id": 37, + "tpot_ms": 149.20507336304368, + "ttft_ms": 397.53399441140937 + }, + { + "e2e_ms": 19143.22059782458, + "request_id": 38, + "tpot_ms": 147.63442381272193, + "ttft_ms": 393.6487736088932 + }, + { + "e2e_ms": 18940.150721561607, + "request_id": 39, + "tpot_ms": 146.06310547094216, + "ttft_ms": 390.13632675195356 + }, + { + "e2e_ms": 18741.29802135198, + "request_id": 40, + "tpot_ms": 144.5231784918296, + "ttft_ms": 386.85435288961753 + }, + { + "e2e_ms": 18542.463082904305, + "request_id": 41, + "tpot_ms": 142.98167897793616, + "ttft_ms": 383.7898527064141 + }, + { + "e2e_ms": 18344.148533824395, + "request_id": 42, + "tpot_ms": 141.4401875845621, + "ttft_ms": 381.2447105850083 + }, + { + "e2e_ms": 18145.706224501653, + "request_id": 43, + "tpot_ms": 139.89417135289136, + "ttft_ms": 379.1464626844494 + }, + { + "e2e_ms": 17948.37563539231, + "request_id": 44, + "tpot_ms": 138.3533077037345, + "ttft_ms": 377.5055570180292 + }, + { + "e2e_ms": 17750.40475411994, + "request_id": 45, + "tpot_ms": 136.80479670989413, + "ttft_ms": 376.19557196338246 + }, + { + "e2e_ms": 17551.412355033975, + "request_id": 46, + "tpot_ms": 135.2471347218224, + "ttft_ms": 375.0262453625339 + }, + { + "e2e_ms": 17352.31516600567, + "request_id": 47, + "tpot_ms": 133.6870504253856, + "ttft_ms": 374.0597619816981 + }, + { + "e2e_ms": 17152.719400024922, + "request_id": 48, + "tpot_ms": 132.11931086353468, + "ttft_ms": 373.56692035601566 + }, + { + "e2e_ms": 16952.899525195207, + "request_id": 49, + "tpot_ms": 130.53228068835782, + "ttft_ms": 375.29987777376664 + }, + { + "e2e_ms": 16753.147719429508, + "request_id": 50, + "tpot_ms": 128.94354832701373, + "ttft_ms": 377.31708189876304 + }, + { + "e2e_ms": 16553.291092750187, + "request_id": 51, + "tpot_ms": 127.34853489391125, + "ttft_ms": 380.0271612234596 + }, + { + "e2e_ms": 16357.836116284518, + "request_id": 52, + "tpot_ms": 125.77762361641257, + "ttft_ms": 384.07791700011984 + }, + { + "e2e_ms": 16157.72333756749, + "request_id": 53, + "tpot_ms": 124.16553823707912, + "ttft_ms": 388.6999814584407 + }, + { + "e2e_ms": 15952.367175996273, + "request_id": 54, + "tpot_ms": 122.51027887411597, + "ttft_ms": 393.5617589835445 + }, + { + "e2e_ms": 15741.765215271713, + "request_id": 55, + "tpot_ms": 120.81149800475544, + "ttft_ms": 398.70496866777216 + }, + { + "e2e_ms": 15528.71285895543, + "request_id": 56, + "tpot_ms": 119.08582914503211, + "ttft_ms": 404.8125575363528 + }, + { + "e2e_ms": 15314.765605096894, + "request_id": 57, + "tpot_ms": 117.34894161957355, + "ttft_ms": 411.4500194110526 + }, + { + "e2e_ms": 15096.980468100028, + "request_id": 58, + "tpot_ms": 115.58065073324867, + "ttft_ms": 418.2378249774459 + }, + { + "e2e_ms": 14778.441120322277, + "request_id": 59, + "tpot_ms": 113.87954551567164, + "ttft_ms": 315.7388398319796 + }, + { + "e2e_ms": 14557.742348852762, + "request_id": 60, + "tpot_ms": 112.08473611057838, + "ttft_ms": 322.9808628093096 + }, + { + "e2e_ms": 14333.83717181993, + "request_id": 61, + "tpot_ms": 110.26282336485625, + "ttft_ms": 330.4586044831872 + }, + { + "e2e_ms": 14108.97915094099, + "request_id": 62, + "tpot_ms": 108.43298632152924, + "ttft_ms": 337.989888106776 + }, + { + "e2e_ms": 13882.64463939965, + "request_id": 63, + "tpot_ms": 106.58813679106942, + "ttft_ms": 345.9512669338345 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 402.0613024234772, + "offered_request_rate": 3.2, + "request_metrics_sha256": "48ff1d03e0aeb45bb4395dca86e089cdbe1c7de16b8557fbe8f25e8e65c77def", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 26482.398681960134, + "request_id": 0, + "tpot_ms": 206.22623604112295, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26389.27725660841, + "request_id": 1, + "tpot_ms": 204.89007083645959, + "ttft_ms": 368.23826037804076 + }, + { + "e2e_ms": 26186.398050214455, + "request_id": 2, + "tpot_ms": 203.43938315354237, + "ttft_ms": 349.5963897145746 + }, + { + "e2e_ms": 25983.46530187178, + "request_id": 3, + "tpot_ms": 201.98715121187504, + "ttft_ms": 331.0970979636474 + }, + { + "e2e_ms": 25780.42643937068, + "request_id": 4, + "tpot_ms": 200.53276712329495, + "ttft_ms": 312.76501471222184 + }, + { + "e2e_ms": 25577.34505869151, + "request_id": 5, + "tpot_ms": 199.0733214464005, + "ttft_ms": 295.0332349986446 + }, + { + "e2e_ms": 25483.55218185351, + "request_id": 6, + "tpot_ms": 197.69083780569105, + "ttft_ms": 376.81578053074594 + }, + { + "e2e_ms": 25280.28848711639, + "request_id": 7, + "tpot_ms": 196.22290463044078, + "ttft_ms": 359.97959905041466 + }, + { + "e2e_ms": 25076.966707259246, + "request_id": 8, + "tpot_ms": 194.75030755695653, + "ttft_ms": 343.6776475257637 + }, + { + "e2e_ms": 24873.46328021192, + "request_id": 9, + "tpot_ms": 193.27265662306831, + "ttft_ms": 327.83588908224147 + }, + { + "e2e_ms": 24669.88557698392, + "request_id": 10, + "tpot_ms": 191.78984111033463, + "ttft_ms": 312.57575597142215 + }, + { + "e2e_ms": 24466.212431081483, + "request_id": 11, + "tpot_ms": 190.30296300954802, + "ttft_ms": 297.7361288688827 + }, + { + "e2e_ms": 24370.912463364126, + "request_id": 12, + "tpot_ms": 188.79387017231738, + "ttft_ms": 394.09095147981964 + }, + { + "e2e_ms": 24166.86947872558, + "request_id": 13, + "tpot_ms": 187.2967209572732, + "ttft_ms": 380.1859171518851 + }, + { + "e2e_ms": 23962.581993212396, + "request_id": 14, + "tpot_ms": 185.79746051386414, + "ttft_ms": 366.30450795165007 + }, + { + "e2e_ms": 23758.13082088949, + "request_id": 15, + "tpot_ms": 184.29572160562404, + "ttft_ms": 352.57417697523687 + }, + { + "e2e_ms": 23552.083711717114, + "request_id": 16, + "tpot_ms": 182.7751996506789, + "ttft_ms": 339.633356080892 + }, + { + "e2e_ms": 23345.944189149544, + "request_id": 17, + "tpot_ms": 181.25255279330477, + "ttft_ms": 326.8699843998375 + }, + { + "e2e_ms": 23139.783338151214, + "request_id": 18, + "tpot_ms": 179.72694307410856, + "ttft_ms": 314.461567739424 + }, + { + "e2e_ms": 22933.56950157464, + "request_id": 19, + "tpot_ms": 178.1977153273412, + "ttft_ms": 302.45965500230557 + }, + { + "e2e_ms": 22833.726368661068, + "request_id": 20, + "tpot_ms": 176.63828124943853, + "ttft_ms": 400.6646499823745 + }, + { + "e2e_ms": 22627.576073166823, + "request_id": 21, + "tpot_ms": 175.09813975268048, + "ttft_ms": 390.11232457640244 + }, + { + "e2e_ms": 22421.391717712297, + "request_id": 22, + "tpot_ms": 173.55627478074584, + "ttft_ms": 379.7448205575735 + }, + { + "e2e_ms": 22215.258027879143, + "request_id": 23, + "tpot_ms": 172.01247264259936, + "ttft_ms": 369.67400226902566 + }, + { + "e2e_ms": 22008.978461747698, + "request_id": 24, + "tpot_ms": 170.46437149324944, + "ttft_ms": 360.0032821050201 + }, + { + "e2e_ms": 21802.65579951293, + "request_id": 25, + "tpot_ms": 168.91532250977505, + "ttft_ms": 350.4098407714995 + }, + { + "e2e_ms": 21596.434473671965, + "request_id": 26, + "tpot_ms": 167.36573728219415, + "ttft_ms": 340.98583883331025 + }, + { + "e2e_ms": 21390.300007810118, + "request_id": 27, + "tpot_ms": 165.81511332736412, + "ttft_ms": 331.7806152348748 + }, + { + "e2e_ms": 21184.224973098888, + "request_id": 28, + "tpot_ms": 164.26380332238384, + "ttft_ms": 322.72195115614187 + }, + { + "e2e_ms": 20978.149686441437, + "request_id": 29, + "tpot_ms": 162.71174941720716, + "ttft_ms": 313.7575104561279 + }, + { + "e2e_ms": 20772.153161450715, + "request_id": 30, + "tpot_ms": 161.15510654351164, + "ttft_ms": 305.4546304247392 + }, + { + "e2e_ms": 20672.62875180034, + "request_id": 31, + "tpot_ms": 159.59455506672822, + "ttft_ms": 404.12025832585744 + }, + { + "e2e_ms": 20466.69413177714, + "request_id": 32, + "tpot_ms": 158.03533915661183, + "ttft_ms": 396.2060588874419 + }, + { + "e2e_ms": 20260.84607576572, + "request_id": 33, + "tpot_ms": 156.475168525323, + "ttft_ms": 388.49967304969545 + }, + { + "e2e_ms": 20055.02118508905, + "request_id": 34, + "tpot_ms": 154.9132321630232, + "ttft_ms": 381.04070038510326 + }, + { + "e2e_ms": 19849.304647544788, + "request_id": 35, + "tpot_ms": 153.3461190070677, + "ttft_ms": 374.34753364718887 + }, + { + "e2e_ms": 19643.694612944317, + "request_id": 36, + "tpot_ms": 151.77962188577303, + "ttft_ms": 367.6826334511443 + }, + { + "e2e_ms": 19437.836505449406, + "request_id": 37, + "tpot_ms": 150.210341914627, + "ttft_ms": 361.1230822917744 + }, + { + "e2e_ms": 19231.99098193562, + "request_id": 38, + "tpot_ms": 148.63640086508497, + "ttft_ms": 355.16807206983003 + }, + { + "e2e_ms": 19026.289377137648, + "request_id": 39, + "tpot_ms": 147.05985988148976, + "ttft_ms": 349.6871721884496 + }, + { + "e2e_ms": 18822.696360365982, + "request_id": 40, + "tpot_ms": 145.49844654462956, + "ttft_ms": 344.3936491980288 + }, + { + "e2e_ms": 18619.112224474804, + "request_id": 41, + "tpot_ms": 143.93654065290042, + "ttft_ms": 339.17156155644966 + }, + { + "e2e_ms": 18415.78828326805, + "request_id": 42, + "tpot_ms": 142.37219516274803, + "ttft_ms": 334.51949759905375 + }, + { + "e2e_ms": 18212.400461939902, + "request_id": 43, + "tpot_ms": 140.8040960160447, + "ttft_ms": 330.280267902225 + }, + { + "e2e_ms": 18009.568500717836, + "request_id": 44, + "tpot_ms": 139.2366051127279, + "ttft_ms": 326.5196514013962 + }, + { + "e2e_ms": 17806.416393414805, + "request_id": 45, + "tpot_ms": 137.66365486442547, + "ttft_ms": 323.1322256327704 + }, + { + "e2e_ms": 17602.75352720573, + "request_id": 46, + "tpot_ms": 136.08599255484802, + "ttft_ms": 319.83247274003014 + }, + { + "e2e_ms": 17399.03826602459, + "request_id": 47, + "tpot_ms": 134.50648843613666, + "ttft_ms": 316.71423463523496 + }, + { + "e2e_ms": 17195.073716367715, + "request_id": 48, + "tpot_ms": 132.92201961792392, + "ttft_ms": 313.9772248913761 + }, + { + "e2e_ms": 16990.997112286357, + "request_id": 49, + "tpot_ms": 131.3320936883647, + "ttft_ms": 311.8212138640413 + }, + { + "e2e_ms": 16895.41197318666, + "request_id": 50, + "tpot_ms": 129.74336132702078, + "ttft_ms": 418.0050846550216 + }, + { + "e2e_ms": 16691.3169931805, + "request_id": 51, + "tpot_ms": 128.14881090811454, + "ttft_ms": 416.418007849952 + }, + { + "e2e_ms": 16489.422838281163, + "request_id": 52, + "tpot_ms": 126.56215930365238, + "ttft_ms": 416.0286067173118 + }, + { + "e2e_ms": 16285.199782255653, + "request_id": 53, + "tpot_ms": 124.95445752714085, + "ttft_ms": 415.9836763087661 + }, + { + "e2e_ms": 16078.355034803544, + "request_id": 54, + "tpot_ms": 123.3249765236981, + "ttft_ms": 416.08301629388575 + }, + { + "e2e_ms": 15868.887387774763, + "request_id": 55, + "tpot_ms": 121.67321161824907, + "ttft_ms": 416.3895122571297 + }, + { + "e2e_ms": 15658.194542949623, + "request_id": 56, + "tpot_ms": 120.00563624201372, + "ttft_ms": 417.4787402138804 + }, + { + "e2e_ms": 15447.054249353854, + "request_id": 57, + "tpot_ms": 118.33082151865976, + "ttft_ms": 419.0399164840635 + }, + { + "e2e_ms": 15233.99501418892, + "request_id": 58, + "tpot_ms": 116.64063648866745, + "ttft_ms": 420.6341801281539 + }, + { + "e2e_ms": 15019.622333077174, + "request_id": 59, + "tpot_ms": 114.93953127109042, + "ttft_ms": 422.3018616486911 + }, + { + "e2e_ms": 14805.106280675915, + "request_id": 60, + "tpot_ms": 113.23514577866216, + "ttft_ms": 424.2427667858202 + }, + { + "e2e_ms": 14588.987025492997, + "request_id": 61, + "tpot_ms": 111.5171161341123, + "ttft_ms": 426.31327646073555 + }, + { + "e2e_ms": 14372.391348386529, + "request_id": 62, + "tpot_ms": 109.79194855942059, + "ttft_ms": 428.8138813401155 + }, + { + "e2e_ms": 14155.057425949359, + "request_id": 63, + "tpot_ms": 108.05983074662959, + "ttft_ms": 431.45892112740114 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.109375, + "passed": 7 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.40625, + "passed": 26 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 774.4871437549591, + "offered_request_rate": 0.1, + "request_metrics_sha256": "1146954f1d73891536473385a3ecb02b14b7eaeee231291f2cbe6f9d4d222414", + "request_rate_per_gpu": 0.0125, + "requests": [ + { + "e2e_ms": 12579.084570749343, + "request_id": 0, + "tpot_ms": 96.75132177962064, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12682.66705927438, + "request_id": 1, + "tpot_ms": 96.90664947750408, + "ttft_ms": 375.5225756313614 + }, + { + "e2e_ms": 12692.014437909165, + "request_id": 2, + "tpot_ms": 96.91832475611541, + "ttft_ms": 383.3871938825091 + }, + { + "e2e_ms": 12605.650216029226, + "request_id": 3, + "tpot_ms": 96.9198422551915, + "ttft_ms": 296.8302496199051 + }, + { + "e2e_ms": 12614.259349375281, + "request_id": 4, + "tpot_ms": 96.9125118011697, + "ttft_ms": 306.3703506267288 + }, + { + "e2e_ms": 12622.868482721336, + "request_id": 5, + "tpot_ms": 96.9125118011697, + "ttft_ms": 314.97948397278464 + }, + { + "e2e_ms": 12631.47761606615, + "request_id": 6, + "tpot_ms": 96.91251180115991, + "ttft_ms": 323.5886173188405 + }, + { + "e2e_ms": 12640.086749411268, + "request_id": 7, + "tpot_ms": 96.91251180113692, + "ttft_ms": 332.19775066687873 + }, + { + "e2e_ms": 12648.695882756385, + "request_id": 8, + "tpot_ms": 96.91251180113692, + "ttft_ms": 340.80688401199666 + }, + { + "e2e_ms": 12657.305016101503, + "request_id": 9, + "tpot_ms": 96.91251180113692, + "ttft_ms": 349.4160173571146 + }, + { + "e2e_ms": 12665.91414944662, + "request_id": 10, + "tpot_ms": 96.9125118011369, + "ttft_ms": 358.0251507022325 + }, + { + "e2e_ms": 12674.523282791739, + "request_id": 11, + "tpot_ms": 96.91251180113692, + "ttft_ms": 366.63428404735043 + }, + { + "e2e_ms": 12683.132416062932, + "request_id": 12, + "tpot_ms": 96.91251180055482, + "ttft_ms": 375.24341739246836 + }, + { + "e2e_ms": 12692.479794575775, + "request_id": 13, + "tpot_ms": 96.91832475507798, + "ttft_ms": 383.85255068087076 + }, + { + "e2e_ms": 12606.11557253071, + "request_id": 14, + "tpot_ms": 96.91984225378269, + "ttft_ms": 297.2956063003096 + }, + { + "e2e_ms": 12614.724705710416, + "request_id": 15, + "tpot_ms": 96.91251179969895, + "ttft_ms": 306.8357071486503 + }, + { + "e2e_ms": 12623.33383889012, + "request_id": 16, + "tpot_ms": 96.91251179969895, + "ttft_ms": 315.44484032835385 + }, + { + "e2e_ms": 12631.942972069823, + "request_id": 17, + "tpot_ms": 96.91251179969893, + "ttft_ms": 324.0539735080574 + }, + { + "e2e_ms": 12640.552105249526, + "request_id": 18, + "tpot_ms": 96.91251179969893, + "ttft_ms": 332.663106687761 + }, + { + "e2e_ms": 12649.16123842923, + "request_id": 19, + "tpot_ms": 96.91251179969893, + "ttft_ms": 341.2722398674646 + }, + { + "e2e_ms": 12657.770371608933, + "request_id": 20, + "tpot_ms": 96.91251179969893, + "ttft_ms": 349.88137304716815 + }, + { + "e2e_ms": 12666.379504788636, + "request_id": 21, + "tpot_ms": 96.91251179969893, + "ttft_ms": 358.49050622687173 + }, + { + "e2e_ms": 12674.988637968341, + "request_id": 22, + "tpot_ms": 96.91251179969895, + "ttft_ms": 367.0996394065753 + }, + { + "e2e_ms": 12683.597771148045, + "request_id": 23, + "tpot_ms": 96.91251179969895, + "ttft_ms": 375.7087725862789 + }, + { + "e2e_ms": 12692.945149660885, + "request_id": 24, + "tpot_ms": 96.91832475507798, + "ttft_ms": 384.31790576598246 + }, + { + "e2e_ms": 12606.580927732637, + "request_id": 25, + "tpot_ms": 96.91984225470249, + "ttft_ms": 297.7609613854213 + }, + { + "e2e_ms": 12615.190061213469, + "request_id": 26, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.3010623465393 + }, + { + "e2e_ms": 12623.7991946943, + "request_id": 27, + "tpot_ms": 96.9125118021018, + "ttft_ms": 315.9101958273709 + }, + { + "e2e_ms": 12632.408328175132, + "request_id": 28, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.5193293082025 + }, + { + "e2e_ms": 12641.017461655963, + "request_id": 29, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.1284627890341 + }, + { + "e2e_ms": 12649.626595136795, + "request_id": 30, + "tpot_ms": 96.9125118021018, + "ttft_ms": 341.73759626986566 + }, + { + "e2e_ms": 12658.235728617627, + "request_id": 31, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.34672975069725 + }, + { + "e2e_ms": 12666.844862098458, + "request_id": 32, + "tpot_ms": 96.9125118021018, + "ttft_ms": 358.95586323152884 + }, + { + "e2e_ms": 12675.45399557929, + "request_id": 33, + "tpot_ms": 96.9125118021018, + "ttft_ms": 367.5649967123604 + }, + { + "e2e_ms": 12684.063129060121, + "request_id": 34, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.174130193192 + }, + { + "e2e_ms": 12693.41050787034, + "request_id": 35, + "tpot_ms": 96.9183247574513, + "ttft_ms": 384.7832636740236 + }, + { + "e2e_ms": 12607.04628612126, + "request_id": 36, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.22631959211776 + }, + { + "e2e_ms": 12615.655419602092, + "request_id": 37, + "tpot_ms": 96.9125118021018, + "ttft_ms": 307.76642073516314 + }, + { + "e2e_ms": 12624.264553082923, + "request_id": 38, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.3755542159947 + }, + { + "e2e_ms": 12632.873686563755, + "request_id": 39, + "tpot_ms": 96.9125118021018, + "ttft_ms": 324.9846876968263 + }, + { + "e2e_ms": 12641.482820044586, + "request_id": 40, + "tpot_ms": 96.9125118021018, + "ttft_ms": 333.5938211776579 + }, + { + "e2e_ms": 12650.091953525418, + "request_id": 41, + "tpot_ms": 96.9125118021018, + "ttft_ms": 342.2029546584895 + }, + { + "e2e_ms": 12658.70108700625, + "request_id": 42, + "tpot_ms": 96.9125118021018, + "ttft_ms": 350.8120881393211 + }, + { + "e2e_ms": 12667.310220487081, + "request_id": 43, + "tpot_ms": 96.9125118021018, + "ttft_ms": 359.42122162015266 + }, + { + "e2e_ms": 12675.919353967913, + "request_id": 44, + "tpot_ms": 96.9125118021018, + "ttft_ms": 368.03035510098425 + }, + { + "e2e_ms": 12684.528487448744, + "request_id": 45, + "tpot_ms": 96.9125118021018, + "ttft_ms": 376.63948858181584 + }, + { + "e2e_ms": 12693.875866258964, + "request_id": 46, + "tpot_ms": 96.91832475745132, + "ttft_ms": 385.2486220626474 + }, + { + "e2e_ms": 12607.511644509885, + "request_id": 47, + "tpot_ms": 96.91984225613498, + "ttft_ms": 298.6916779807416 + }, + { + "e2e_ms": 12616.120777990716, + "request_id": 48, + "tpot_ms": 96.9125118021018, + "ttft_ms": 308.23177912378696 + }, + { + "e2e_ms": 12624.729911471548, + "request_id": 49, + "tpot_ms": 96.9125118021018, + "ttft_ms": 316.84091260461855 + }, + { + "e2e_ms": 12633.339044941862, + "request_id": 50, + "tpot_ms": 96.91251180201898, + "ttft_ms": 325.45004608545014 + }, + { + "e2e_ms": 12641.94817818975, + "request_id": 51, + "tpot_ms": 96.91251180018479, + "ttft_ms": 334.0591795662817 + }, + { + "e2e_ms": 12650.557311437638, + "request_id": 52, + "tpot_ms": 96.91251179996682, + "ttft_ms": 342.66831284185173 + }, + { + "e2e_ms": 12659.166444685525, + "request_id": 53, + "tpot_ms": 96.91251179996682, + "ttft_ms": 351.277446089739 + }, + { + "e2e_ms": 12667.775577933411, + "request_id": 54, + "tpot_ms": 96.91251179996681, + "ttft_ms": 359.88657933762624 + }, + { + "e2e_ms": 12676.3847111813, + "request_id": 55, + "tpot_ms": 96.91251179996682, + "ttft_ms": 368.4957125855135 + }, + { + "e2e_ms": 12684.993844429187, + "request_id": 56, + "tpot_ms": 96.91251179996682, + "ttft_ms": 377.10484583340076 + }, + { + "e2e_ms": 12694.3412230047, + "request_id": 57, + "tpot_ms": 96.91832475530246, + "ttft_ms": 385.713979081288 + }, + { + "e2e_ms": 12607.977001027848, + "request_id": 58, + "tpot_ms": 96.9198422540282, + "ttft_ms": 299.1570347662673 + }, + { + "e2e_ms": 12616.586134275734, + "request_id": 59, + "tpot_ms": 96.91251179996681, + "ttft_ms": 308.6971356799495 + }, + { + "e2e_ms": 12625.195267523623, + "request_id": 60, + "tpot_ms": 96.91251179996682, + "ttft_ms": 317.30626892783675 + }, + { + "e2e_ms": 12633.80440077151, + "request_id": 61, + "tpot_ms": 96.91251179996682, + "ttft_ms": 325.915402175724 + }, + { + "e2e_ms": 12642.413534019397, + "request_id": 62, + "tpot_ms": 96.91251179996682, + "ttft_ms": 334.52453542361127 + }, + { + "e2e_ms": 12436.81062947212, + "request_id": 63, + "tpot_ms": 95.22580284094978, + "ttft_ms": 343.1336686714985 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "37e166453683700c9df7abffa278431119205a29c83e92211ffa69d90e1dca0e" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 403.8260774612427, + "offered_request_rate": 0.2, + "request_metrics_sha256": "cf8a095cdf380d42316d1ad169ce9b2b45ae7d64348cd38271a332663de349ad", + "request_rate_per_gpu": 0.025, + "requests": [ + { + "e2e_ms": 12823.625057543633, + "request_id": 0, + "tpot_ms": 98.67683742367018, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 12920.533566322107, + "request_id": 1, + "tpot_ms": 99.10315669196875, + "ttft_ms": 334.4326664420745 + }, + { + "e2e_ms": 12921.724204826443, + "request_id": 2, + "tpot_ms": 99.22365133274592, + "ttft_ms": 320.32048556771286 + }, + { + "e2e_ms": 12922.91484333078, + "request_id": 3, + "tpot_ms": 99.22887081733072, + "ttft_ms": 320.84824952977976 + }, + { + "e2e_ms": 12924.105481838353, + "request_id": 4, + "tpot_ms": 99.22887081736695, + "ttft_ms": 322.0388880327505 + }, + { + "e2e_ms": 12925.296120352506, + "request_id": 5, + "tpot_ms": 99.22887081744425, + "ttft_ms": 323.2295265370873 + }, + { + "e2e_ms": 12926.486758866658, + "request_id": 6, + "tpot_ms": 99.22887081752154, + "ttft_ms": 324.420165041424 + }, + { + "e2e_ms": 12927.677397380812, + "request_id": 7, + "tpot_ms": 99.22887081749418, + "ttft_ms": 325.6108035590515 + }, + { + "e2e_ms": 12928.868035894964, + "request_id": 8, + "tpot_ms": 99.22887081749417, + "ttft_ms": 326.8014420732044 + }, + { + "e2e_ms": 12930.058674409118, + "request_id": 9, + "tpot_ms": 99.22887081749418, + "ttft_ms": 327.9920805873573 + }, + { + "e2e_ms": 12931.24931292327, + "request_id": 10, + "tpot_ms": 99.22887081749417, + "ttft_ms": 329.1827191015102 + }, + { + "e2e_ms": 12932.439951441467, + "request_id": 11, + "tpot_ms": 99.22887081752602, + "ttft_ms": 330.3733576156631 + }, + { + "e2e_ms": 12933.63058995827, + "request_id": 12, + "tpot_ms": 99.22887081754688, + "ttft_ms": 331.563996129816 + }, + { + "e2e_ms": 12934.821228475073, + "request_id": 13, + "tpot_ms": 99.22887081758537, + "ttft_ms": 332.7546346417307 + }, + { + "e2e_ms": 12936.011866991876, + "request_id": 14, + "tpot_ms": 99.22887081758537, + "ttft_ms": 333.94527315853395 + }, + { + "e2e_ms": 12937.20250550868, + "request_id": 15, + "tpot_ms": 99.22887081758537, + "ttft_ms": 335.1359116753372 + }, + { + "e2e_ms": 12938.393144025482, + "request_id": 16, + "tpot_ms": 99.22887081758537, + "ttft_ms": 336.3265501921404 + }, + { + "e2e_ms": 12939.583782542286, + "request_id": 17, + "tpot_ms": 99.22887081758537, + "ttft_ms": 337.51718870894365 + }, + { + "e2e_ms": 12940.77442105909, + "request_id": 18, + "tpot_ms": 99.22887081758537, + "ttft_ms": 338.7078272257469 + }, + { + "e2e_ms": 12941.965059575892, + "request_id": 19, + "tpot_ms": 99.22887081758537, + "ttft_ms": 339.8984657425501 + }, + { + "e2e_ms": 12943.155698092696, + "request_id": 20, + "tpot_ms": 99.22887081758537, + "ttft_ms": 341.08910425935335 + }, + { + "e2e_ms": 12944.346336609498, + "request_id": 21, + "tpot_ms": 99.22887081758537, + "ttft_ms": 342.2797427761566 + }, + { + "e2e_ms": 12945.536975126302, + "request_id": 22, + "tpot_ms": 99.22887081758537, + "ttft_ms": 343.4703812929598 + }, + { + "e2e_ms": 12946.727613643105, + "request_id": 23, + "tpot_ms": 99.22887081758537, + "ttft_ms": 344.66101980976305 + }, + { + "e2e_ms": 12947.918252142103, + "request_id": 24, + "tpot_ms": 99.22887081744517, + "ttft_ms": 345.8516583265663 + }, + { + "e2e_ms": 12949.10889064036, + "request_id": 25, + "tpot_ms": 99.22887081729912, + "ttft_ms": 347.0422968433695 + }, + { + "e2e_ms": 12950.299529138618, + "request_id": 26, + "tpot_ms": 99.2288708172622, + "ttft_ms": 348.23293534631716 + }, + { + "e2e_ms": 12951.490167636877, + "request_id": 27, + "tpot_ms": 99.22887081726222, + "ttft_ms": 349.42357384457523 + }, + { + "e2e_ms": 12952.680806135135, + "request_id": 28, + "tpot_ms": 99.22887081726222, + "ttft_ms": 350.6142123428333 + }, + { + "e2e_ms": 12953.871444633392, + "request_id": 29, + "tpot_ms": 99.2288708172622, + "ttft_ms": 351.80485084109137 + }, + { + "e2e_ms": 12955.06208313165, + "request_id": 30, + "tpot_ms": 99.2288708172622, + "ttft_ms": 352.99548933934943 + }, + { + "e2e_ms": 12956.25272162991, + "request_id": 31, + "tpot_ms": 99.22887081726222, + "ttft_ms": 354.1861278376075 + }, + { + "e2e_ms": 12957.443360128167, + "request_id": 32, + "tpot_ms": 99.22887081726222, + "ttft_ms": 355.37676633586557 + }, + { + "e2e_ms": 12958.633998626425, + "request_id": 33, + "tpot_ms": 99.2288708172622, + "ttft_ms": 356.56740483412364 + }, + { + "e2e_ms": 12959.824637124682, + "request_id": 34, + "tpot_ms": 99.2288708172622, + "ttft_ms": 357.7580433323817 + }, + { + "e2e_ms": 12961.015275622942, + "request_id": 35, + "tpot_ms": 99.22887081726222, + "ttft_ms": 358.94868183063977 + }, + { + "e2e_ms": 12962.2059141212, + "request_id": 36, + "tpot_ms": 99.22887081726222, + "ttft_ms": 360.13932032889784 + }, + { + "e2e_ms": 12963.396552619457, + "request_id": 37, + "tpot_ms": 99.2288708172622, + "ttft_ms": 361.3299588271559 + }, + { + "e2e_ms": 12964.587191117715, + "request_id": 38, + "tpot_ms": 99.2288708172622, + "ttft_ms": 362.520597325414 + }, + { + "e2e_ms": 12965.777829615974, + "request_id": 39, + "tpot_ms": 99.22887081726222, + "ttft_ms": 363.71123582367204 + }, + { + "e2e_ms": 12966.968468114232, + "request_id": 40, + "tpot_ms": 99.22887081726222, + "ttft_ms": 364.9018743219301 + }, + { + "e2e_ms": 12968.15910661249, + "request_id": 41, + "tpot_ms": 99.2288708172622, + "ttft_ms": 366.0925128201882 + }, + { + "e2e_ms": 12969.349745110747, + "request_id": 42, + "tpot_ms": 99.2288708172622, + "ttft_ms": 367.28315131844624 + }, + { + "e2e_ms": 12970.540383609006, + "request_id": 43, + "tpot_ms": 99.22887081726222, + "ttft_ms": 368.4737898167043 + }, + { + "e2e_ms": 12971.731022107264, + "request_id": 44, + "tpot_ms": 99.22887081726222, + "ttft_ms": 369.6644283149624 + }, + { + "e2e_ms": 12972.921660605522, + "request_id": 45, + "tpot_ms": 99.2288708172622, + "ttft_ms": 370.85506681322045 + }, + { + "e2e_ms": 12974.11229910378, + "request_id": 46, + "tpot_ms": 99.2288708172622, + "ttft_ms": 372.0457053114785 + }, + { + "e2e_ms": 12975.302937602039, + "request_id": 47, + "tpot_ms": 99.22887081726222, + "ttft_ms": 373.2363438097366 + }, + { + "e2e_ms": 12976.493576100296, + "request_id": 48, + "tpot_ms": 99.22887081726222, + "ttft_ms": 374.42698230799465 + }, + { + "e2e_ms": 12977.684214615067, + "request_id": 49, + "tpot_ms": 99.22887081739223, + "ttft_ms": 375.6176208062527 + }, + { + "e2e_ms": 12978.874853139188, + "request_id": 50, + "tpot_ms": 99.22887081759589, + "ttft_ms": 376.8082593045108 + }, + { + "e2e_ms": 12980.06549166331, + "request_id": 51, + "tpot_ms": 99.22887081779955, + "ttft_ms": 377.99889780276885 + }, + { + "e2e_ms": 12981.256130187432, + "request_id": 52, + "tpot_ms": 99.22887081784184, + "ttft_ms": 379.189536321519 + }, + { + "e2e_ms": 12982.446768711554, + "request_id": 53, + "tpot_ms": 99.22887081784184, + "ttft_ms": 380.3801748456408 + }, + { + "e2e_ms": 12983.637407235676, + "request_id": 54, + "tpot_ms": 99.22887081784184, + "ttft_ms": 381.5708133697626 + }, + { + "e2e_ms": 12984.828045759798, + "request_id": 55, + "tpot_ms": 99.22887081784184, + "ttft_ms": 382.76145189388444 + }, + { + "e2e_ms": 12986.01868428392, + "request_id": 56, + "tpot_ms": 99.22887081784184, + "ttft_ms": 383.95209041800626 + }, + { + "e2e_ms": 12987.209322808041, + "request_id": 57, + "tpot_ms": 99.22887081784184, + "ttft_ms": 385.1427289421281 + }, + { + "e2e_ms": 12988.399961332163, + "request_id": 58, + "tpot_ms": 99.22887081784184, + "ttft_ms": 386.3333674662499 + }, + { + "e2e_ms": 12989.881679406835, + "request_id": 59, + "tpot_ms": 99.2311627828068, + "ttft_ms": 387.52400599037173 + }, + { + "e2e_ms": 12991.363397481507, + "request_id": 60, + "tpot_ms": 99.23345474777176, + "ttft_ms": 388.71464451449356 + }, + { + "e2e_ms": 12896.836165730803, + "request_id": 61, + "tpot_ms": 99.23492287861664, + "ttft_ms": 294.0009601464908 + }, + { + "e2e_ms": 12689.81487892944, + "request_id": 62, + "tpot_ms": 97.59316693471084, + "ttft_ms": 295.4826782211626 + }, + { + "e2e_ms": 12448.118757066311, + "request_id": 63, + "tpot_ms": 95.67691266262345, + "ttft_ms": 297.1508489131338 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "e6cebfc858981b2ae4632d2c75f5e8c1f83280e7306351e15c7cd611d2664d80" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 222.21414136886597, + "offered_request_rate": 0.4, + "request_metrics_sha256": "e09d047b2039786ee02cd4c59c081ca95e5197ae65723b6170e43f6c0924921a", + "request_rate_per_gpu": 0.05, + "requests": [ + { + "e2e_ms": 13556.392075191316, + "request_id": 0, + "tpot_ms": 104.4466564602661, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 13715.056219396885, + "request_id": 1, + "tpot_ms": 105.11756526634255, + "ttft_ms": 365.12543057138026 + }, + { + "e2e_ms": 13775.965021971426, + "request_id": 2, + "tpot_ms": 105.6336255570719, + "ttft_ms": 360.4945762232949 + }, + { + "e2e_ms": 13838.18765138338, + "request_id": 3, + "tpot_ms": 106.07242606904647, + "ttft_ms": 366.98954061447785 + }, + { + "e2e_ms": 13802.282961907104, + "request_id": 4, + "tpot_ms": 106.21712281117094, + "ttft_ms": 312.70836488839393 + }, + { + "e2e_ms": 13864.505591319055, + "request_id": 5, + "tpot_ms": 106.32495967758555, + "ttft_ms": 361.2357122656906 + }, + { + "e2e_ms": 13828.78708091747, + "request_id": 6, + "tpot_ms": 106.35892091285005, + "ttft_ms": 321.20412498551156 + }, + { + "e2e_ms": 13892.323156276025, + "request_id": 7, + "tpot_ms": 106.3810710779423, + "ttft_ms": 381.9271293773525 + }, + { + "e2e_ms": 13856.60464587857, + "request_id": 8, + "tpot_ms": 106.3914161711828, + "ttft_ms": 344.89479213835494 + }, + { + "e2e_ms": 13820.886135481267, + "request_id": 9, + "tpot_ms": 106.39288214818258, + "ttft_ms": 308.9901026620794 + }, + { + "e2e_ms": 13883.108764897528, + "request_id": 10, + "tpot_ms": 106.39288214821651, + "ttft_ms": 371.21273207403146 + }, + { + "e2e_ms": 13847.390254500226, + "request_id": 11, + "tpot_ms": 106.39288214825024, + "ttft_ms": 335.4942216724446 + }, + { + "e2e_ms": 13811.485565028235, + "request_id": 12, + "tpot_ms": 106.39288214828399, + "ttft_ms": 299.589532196169 + }, + { + "e2e_ms": 13873.708194444496, + "request_id": 13, + "tpot_ms": 106.38107407728566, + "ttft_ms": 363.31178662921815 + }, + { + "e2e_ms": 13837.989684047194, + "request_id": 14, + "tpot_ms": 106.38107407728566, + "ttft_ms": 327.5932762319158 + }, + { + "e2e_ms": 13901.52575941027, + "request_id": 15, + "tpot_ms": 106.39141617135505, + "ttft_ms": 389.81590564817736 + }, + { + "e2e_ms": 13865.807249012967, + "request_id": 16, + "tpot_ms": 106.39141617135505, + "ttft_ms": 354.097395250875 + }, + { + "e2e_ms": 13830.088738615665, + "request_id": 17, + "tpot_ms": 106.3928821483211, + "ttft_ms": 318.19270577888403 + }, + { + "e2e_ms": 13893.62481397874, + "request_id": 18, + "tpot_ms": 106.40322424239051, + "ttft_ms": 380.4153351951456 + }, + { + "e2e_ms": 13857.906303581438, + "request_id": 19, + "tpot_ms": 106.40322424239051, + "ttft_ms": 344.69682479784325 + }, + { + "e2e_ms": 13822.187793184135, + "request_id": 20, + "tpot_ms": 106.40469021935657, + "ttft_ms": 308.79213532585226 + }, + { + "e2e_ms": 13884.410422593844, + "request_id": 21, + "tpot_ms": 106.39288214826952, + "ttft_ms": 372.51438976361584 + }, + { + "e2e_ms": 13848.691912190276, + "request_id": 22, + "tpot_ms": 106.39288214822018, + "ttft_ms": 336.7958793663135 + }, + { + "e2e_ms": 13812.78722271334, + "request_id": 23, + "tpot_ms": 106.39288214818123, + "ttft_ms": 300.8911898943225 + }, + { + "e2e_ms": 13875.009852123696, + "request_id": 24, + "tpot_ms": 106.38107407709929, + "ttft_ms": 364.6134443320861 + }, + { + "e2e_ms": 13839.291341720127, + "request_id": 25, + "tpot_ms": 106.38107407704995, + "ttft_ms": 328.89493393478375 + }, + { + "e2e_ms": 13902.827417078257, + "request_id": 26, + "tpot_ms": 106.39141617110043, + "ttft_ms": 391.11756334850156 + }, + { + "e2e_ms": 13867.108906674688, + "request_id": 27, + "tpot_ms": 106.39141617110043, + "ttft_ms": 355.39905294493224 + }, + { + "e2e_ms": 13831.390396271117, + "request_id": 28, + "tpot_ms": 106.39288214805607, + "ttft_ms": 319.49436346799587 + }, + { + "e2e_ms": 13894.926471629247, + "request_id": 29, + "tpot_ms": 106.40322424213302, + "ttft_ms": 381.7169928783528 + }, + { + "e2e_ms": 13859.207961225678, + "request_id": 30, + "tpot_ms": 106.40322424213302, + "ttft_ms": 345.9984824747835 + }, + { + "e2e_ms": 13823.48945082211, + "request_id": 31, + "tpot_ms": 106.40469021908869, + "ttft_ms": 310.0937929978471 + }, + { + "e2e_ms": 13885.712080232466, + "request_id": 32, + "tpot_ms": 106.39288214805609, + "ttft_ms": 373.8160474293437 + }, + { + "e2e_ms": 13849.993569828897, + "request_id": 33, + "tpot_ms": 106.39288214805609, + "ttft_ms": 338.0975370257744 + }, + { + "e2e_ms": 13814.08888035196, + "request_id": 34, + "tpot_ms": 106.39288214805607, + "ttft_ms": 302.192847548838 + }, + { + "e2e_ms": 13876.311509762318, + "request_id": 35, + "tpot_ms": 106.3810740770235, + "ttft_ms": 365.9151019803346 + }, + { + "e2e_ms": 13840.592999358747, + "request_id": 36, + "tpot_ms": 106.38107407702348, + "ttft_ms": 330.1965915767653 + }, + { + "e2e_ms": 13904.129074716877, + "request_id": 37, + "tpot_ms": 106.39141617110043, + "ttft_ms": 392.41922098712223 + }, + { + "e2e_ms": 13868.410564313308, + "request_id": 38, + "tpot_ms": 106.39141617110043, + "ttft_ms": 356.7007105835529 + }, + { + "e2e_ms": 13832.69205390974, + "request_id": 39, + "tpot_ms": 106.39288214805609, + "ttft_ms": 320.79602110661654 + }, + { + "e2e_ms": 13896.228129267869, + "request_id": 40, + "tpot_ms": 106.40322424213304, + "ttft_ms": 383.0186505169735 + }, + { + "e2e_ms": 13860.509618864298, + "request_id": 41, + "tpot_ms": 106.40322424213302, + "ttft_ms": 347.30014011340415 + }, + { + "e2e_ms": 13824.79110846073, + "request_id": 42, + "tpot_ms": 106.40469021908868, + "ttft_ms": 311.3954506364678 + }, + { + "e2e_ms": 13888.32718381886, + "request_id": 43, + "tpot_ms": 106.40322424213302, + "ttft_ms": 375.1177050679644 + }, + { + "e2e_ms": 13852.60867341529, + "request_id": 44, + "tpot_ms": 106.40322424213304, + "ttft_ms": 339.39919466439505 + }, + { + "e2e_ms": 13816.89016301172, + "request_id": 45, + "tpot_ms": 106.40469021908868, + "ttft_ms": 303.4945051874587 + }, + { + "e2e_ms": 13879.11279242067, + "request_id": 46, + "tpot_ms": 106.392882148045, + "ttft_ms": 367.2167596189553 + }, + { + "e2e_ms": 13843.394282013549, + "request_id": 47, + "tpot_ms": 106.39288214801702, + "ttft_ms": 331.49824921538595 + }, + { + "e2e_ms": 13807.489592530403, + "request_id": 48, + "tpot_ms": 106.39288214796812, + "ttft_ms": 295.5935597384496 + }, + { + "e2e_ms": 13869.712221935884, + "request_id": 49, + "tpot_ms": 106.38107407689715, + "ttft_ms": 359.3158141699462 + }, + { + "e2e_ms": 13833.993711528763, + "request_id": 50, + "tpot_ms": 106.38107407686918, + "ttft_ms": 323.59730376637685 + }, + { + "e2e_ms": 13897.5297868798, + "request_id": 51, + "tpot_ms": 106.3914161708903, + "ttft_ms": 385.8199331767338 + }, + { + "e2e_ms": 13861.81127647268, + "request_id": 52, + "tpot_ms": 106.39141617089378, + "ttft_ms": 350.1014227691712 + }, + { + "e2e_ms": 13826.092766065556, + "request_id": 53, + "tpot_ms": 106.39288214787034, + "ttft_ms": 314.1967332860247 + }, + { + "e2e_ms": 13889.628841416596, + "request_id": 54, + "tpot_ms": 106.40322424192983, + "ttft_ms": 376.41936269150733 + }, + { + "e2e_ms": 13853.910331009472, + "request_id": 55, + "tpot_ms": 106.4032242419298, + "ttft_ms": 340.7008522843853 + }, + { + "e2e_ms": 13818.191820602351, + "request_id": 56, + "tpot_ms": 106.4046902189064, + "ttft_ms": 304.7961628012388 + }, + { + "e2e_ms": 13880.414450007833, + "request_id": 57, + "tpot_ms": 106.39288214787034, + "ttft_ms": 368.5184172283016 + }, + { + "e2e_ms": 13844.695939600711, + "request_id": 58, + "tpot_ms": 106.39288214787034, + "ttft_ms": 332.79990682117955 + }, + { + "e2e_ms": 13600.134841430987, + "request_id": 59, + "tpot_ms": 104.74991829994453, + "ttft_ms": 296.89521733803304 + }, + { + "e2e_ms": 13451.749583806417, + "request_id": 60, + "tpot_ms": 103.07978040977419, + "ttft_ms": 360.61747176509584 + }, + { + "e2e_ms": 13168.506714577745, + "request_id": 61, + "tpot_ms": 101.13076971039189, + "ttft_ms": 324.8989613579738 + }, + { + "e2e_ms": 12970.21046401943, + "request_id": 62, + "tpot_ms": 99.07943994689744, + "ttft_ms": 387.1215907634564 + }, + { + "e2e_ms": 12659.030247905519, + "request_id": 63, + "tpot_ms": 96.91045013818255, + "ttft_ms": 351.4030803563344 + } + ], + "slos": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "021442528b6c4067d63827496b66b6f530adbe9f4b29732b1aa7aab2112ced9f" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 145.18082785606384, + "offered_request_rate": 0.8, + "request_metrics_sha256": "6b246d56546536e46b4e8adfb23076b3cbf053da4b0af0288346df10122d6022", + "request_rate_per_gpu": 0.1, + "requests": [ + { + "e2e_ms": 15423.90313583537, + "request_id": 0, + "tpot_ms": 119.15146796140039, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 15680.782114004433, + "request_id": 1, + "tpot_ms": 120.47488426274414, + "ttft_ms": 380.47181263592745 + }, + { + "e2e_ms": 15824.99160915063, + "request_id": 2, + "tpot_ms": 121.60222806639713, + "ttft_ms": 381.5086447181937 + }, + { + "e2e_ms": 15964.925902693089, + "request_id": 3, + "tpot_ms": 122.6571852387084, + "ttft_ms": 387.4633773771219 + }, + { + "e2e_ms": 15994.3969757696, + "request_id": 4, + "tpot_ms": 123.48492728290344, + "ttft_ms": 311.8112108408626 + }, + { + "e2e_ms": 16132.273068839513, + "request_id": 5, + "tpot_ms": 124.38017511477935, + "ttft_ms": 335.9908292625358 + }, + { + "e2e_ms": 16268.09096143688, + "request_id": 6, + "tpot_ms": 125.15372351796954, + "ttft_ms": 373.5680746547478 + }, + { + "e2e_ms": 16291.239371010352, + "request_id": 7, + "tpot_ms": 125.67763083904141, + "ttft_ms": 330.18025445209173 + }, + { + "e2e_ms": 16612.993923996273, + "request_id": 8, + "tpot_ms": 127.71634872045071, + "ttft_ms": 393.0176364990334 + }, + { + "e2e_ms": 16636.356874836954, + "request_id": 9, + "tpot_ms": 128.11558123152213, + "ttft_ms": 365.67805843364454 + }, + { + "e2e_ms": 16659.624174048608, + "request_id": 10, + "tpot_ms": 128.3236060799542, + "ttft_ms": 362.52620189442376 + }, + { + "e2e_ms": 16683.03953534565, + "request_id": 11, + "tpot_ms": 128.34365422438844, + "ttft_ms": 383.3954488483169 + }, + { + "e2e_ms": 16598.14552827833, + "request_id": 12, + "tpot_ms": 128.20865630197483, + "ttft_ms": 315.6461779275297 + }, + { + "e2e_ms": 16621.508479120537, + "request_id": 13, + "tpot_ms": 128.0747814081899, + "ttft_ms": 356.0112402804201 + }, + { + "e2e_ms": 16644.871429963147, + "request_id": 14, + "tpot_ms": 127.97565080664988, + "ttft_ms": 391.9637775186118 + }, + { + "e2e_ms": 16559.977422897646, + "request_id": 15, + "tpot_ms": 127.91134905305496, + "ttft_ms": 315.2360931596654 + }, + { + "e2e_ms": 16583.340373740255, + "request_id": 16, + "tpot_ms": 127.86325360239432, + "ttft_ms": 344.7071662361765 + }, + { + "e2e_ms": 16606.703324582868, + "request_id": 17, + "tpot_ms": 127.81515815173371, + "ttft_ms": 374.17823931268754 + }, + { + "e2e_ms": 16630.066275425477, + "request_id": 18, + "tpot_ms": 127.7832690040065, + "ttft_ms": 401.59111191665176 + }, + { + "e2e_ms": 16354.971144941388, + "request_id": 19, + "tpot_ms": 126.2873116642462, + "ttft_ms": 316.48256358212024 + }, + { + "e2e_ms": 16568.46874002116, + "request_id": 20, + "tpot_ms": 127.78612414854776, + "ttft_ms": 339.6309731555931 + }, + { + "e2e_ms": 16591.83169086377, + "request_id": 21, + "tpot_ms": 127.78612414856296, + "ttft_ms": 362.9939239962745 + }, + { + "e2e_ms": 16615.19464170638, + "request_id": 22, + "tpot_ms": 127.78612414857815, + "ttft_ms": 386.3568748369559 + }, + { + "e2e_ms": 16339.951449136564, + "request_id": 23, + "tpot_ms": 126.28847750749865, + "ttft_ms": 301.3148056842354 + }, + { + "e2e_ms": 16553.449044216337, + "request_id": 24, + "tpot_ms": 127.78518800972483, + "ttft_ms": 324.73016698128276 + }, + { + "e2e_ms": 16576.811995058946, + "request_id": 25, + "tpot_ms": 127.78477532898123, + "ttft_ms": 348.1455282783301 + }, + { + "e2e_ms": 16600.174945901555, + "request_id": 26, + "tpot_ms": 127.78477532898441, + "ttft_ms": 371.50847912053564 + }, + { + "e2e_ms": 16623.53789674417, + "request_id": 27, + "tpot_ms": 127.78477532898442, + "ttft_ms": 394.87142996314617 + }, + { + "e2e_ms": 16348.29470417435, + "request_id": 28, + "tpot_ms": 126.28596284469846, + "ttft_ms": 309.97742289764574 + }, + { + "e2e_ms": 16561.79229925412, + "request_id": 29, + "tpot_ms": 127.78308602766822, + "ttft_ms": 333.34037374025627 + }, + { + "e2e_ms": 16585.155250096734, + "request_id": 30, + "tpot_ms": 127.78308602766825, + "ttft_ms": 356.7033245828668 + }, + { + "e2e_ms": 16608.518200939343, + "request_id": 31, + "tpot_ms": 127.78308602766823, + "ttft_ms": 380.06627542547733 + }, + { + "e2e_ms": 16631.881151781956, + "request_id": 32, + "tpot_ms": 127.78360948578973, + "ttft_ms": 403.36274708666053 + }, + { + "e2e_ms": 16356.786021297865, + "request_id": 33, + "tpot_ms": 126.28596284469846, + "ttft_ms": 318.4687400211601 + }, + { + "e2e_ms": 16570.283616377637, + "request_id": 34, + "tpot_ms": 127.78308602766825, + "ttft_ms": 341.83169086377063 + }, + { + "e2e_ms": 16593.646567220247, + "request_id": 35, + "tpot_ms": 127.78308602766823, + "ttft_ms": 365.19464170638116 + }, + { + "e2e_ms": 16617.00951806286, + "request_id": 36, + "tpot_ms": 127.78477532898444, + "ttft_ms": 388.34305128183644 + }, + { + "e2e_ms": 16341.766325493041, + "request_id": 37, + "tpot_ms": 126.28596284469846, + "ttft_ms": 303.449044216336 + }, + { + "e2e_ms": 16555.263920572714, + "request_id": 38, + "tpot_ms": 127.78308602766747, + "ttft_ms": 326.81199505894654 + }, + { + "e2e_ms": 16578.62687141207, + "request_id": 39, + "tpot_ms": 127.78308602764184, + "ttft_ms": 350.17494590155707 + }, + { + "e2e_ms": 16601.989822251424, + "request_id": 40, + "tpot_ms": 127.78308602761618, + "ttft_ms": 373.5378967441676 + }, + { + "e2e_ms": 16625.35277309078, + "request_id": 41, + "tpot_ms": 127.78477532890675, + "ttft_ms": 396.6863063196229 + }, + { + "e2e_ms": 16350.109580518321, + "request_id": 42, + "tpot_ms": 126.2859628446, + "ttft_ms": 311.79229925412244 + }, + { + "e2e_ms": 16563.607175594883, + "request_id": 43, + "tpot_ms": 127.78308602754448, + "ttft_ms": 335.15525009673297 + }, + { + "e2e_ms": 16586.970126434237, + "request_id": 44, + "tpot_ms": 127.78308602751885, + "ttft_ms": 358.5182009393435 + }, + { + "e2e_ms": 16610.333077273594, + "request_id": 45, + "tpot_ms": 127.78308602749323, + "ttft_ms": 381.88115178195403 + }, + { + "e2e_ms": 16633.69602811295, + "request_id": 46, + "tpot_ms": 127.78360948558907, + "ttft_ms": 405.17762344313724 + }, + { + "e2e_ms": 16548.802021044266, + "request_id": 47, + "tpot_ms": 127.78360948556401, + "ttft_ms": 320.2836163776368 + }, + { + "e2e_ms": 16572.164971883623, + "request_id": 48, + "tpot_ms": 127.7836094855384, + "ttft_ms": 343.64656722024733 + }, + { + "e2e_ms": 16595.52792272298, + "request_id": 49, + "tpot_ms": 127.78360948551277, + "ttft_ms": 367.00951806285786 + }, + { + "e2e_ms": 16618.890873562337, + "request_id": 50, + "tpot_ms": 127.78529878680334, + "ttft_ms": 390.15792763831314 + }, + { + "e2e_ms": 16343.647680989874, + "request_id": 51, + "tpot_ms": 126.28648630249732, + "ttft_ms": 305.26392057271323 + }, + { + "e2e_ms": 16199.563507361632, + "request_id": 52, + "tpot_ms": 124.96800500747688, + "ttft_ms": 328.6268714120695 + }, + { + "e2e_ms": 16032.332947106512, + "request_id": 53, + "tpot_ms": 123.46726869964634, + "ttft_ms": 351.9898222514257 + }, + { + "e2e_ms": 15838.885472588829, + "request_id": 54, + "tpot_ms": 121.7600999960476, + "ttft_ms": 375.35277309078197 + }, + { + "e2e_ms": 15619.209002299955, + "request_id": 55, + "tpot_ms": 119.84809306800307, + "ttft_ms": 398.5011826635656 + }, + { + "e2e_ms": 15285.473398871076, + "request_id": 56, + "tpot_ms": 117.8887104194976, + "ttft_ms": 313.60717559488194 + }, + { + "e2e_ms": 15049.070462905931, + "request_id": 57, + "tpot_ms": 115.84330973599758, + "ttft_ms": 336.9701264342382 + }, + { + "e2e_ms": 14793.47811125659, + "request_id": 58, + "tpot_ms": 113.6468112912047, + "ttft_ms": 360.3330772735944 + }, + { + "e2e_ms": 14524.751300129523, + "request_id": 59, + "tpot_ms": 111.34689190564231, + "ttft_ms": 383.69602811295067 + }, + { + "e2e_ms": 14156.606828520311, + "request_id": 60, + "tpot_ms": 109.11657328721294, + "ttft_ms": 298.802021044267 + }, + { + "e2e_ms": 13870.414276689999, + "request_id": 61, + "tpot_ms": 106.67912838430216, + "ttft_ms": 322.16497188362325 + }, + { + "e2e_ms": 13579.457505623892, + "request_id": 62, + "tpot_ms": 104.20416994410166, + "ttft_ms": 345.5279227229795 + }, + { + "e2e_ms": 13281.118281252404, + "request_id": 63, + "tpot_ms": 101.6710819503155, + "ttft_ms": 368.89087356233574 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.15625, + "passed": 10 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "9ed2bf69d3193253248833822a357db5130e6ca29282e93e04cc92961dba468d" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 139.77111339569092, + "offered_request_rate": 1.2, + "request_metrics_sha256": "dfafc592cb8f7c26725458265ca04cb89642135218d5f80993dfbd6e83fcea9c", + "request_rate_per_gpu": 0.15, + "requests": [ + { + "e2e_ms": 17349.6420929535, + "request_id": 0, + "tpot_ms": 134.3147668363463, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 17472.38469413562, + "request_id": 1, + "tpot_ms": 135.04025977135157, + "ttft_ms": 322.27170317397037 + }, + { + "e2e_ms": 17594.570506858923, + "request_id": 2, + "tpot_ms": 135.72808492940857, + "ttft_ms": 357.1037208240346 + }, + { + "e2e_ms": 17607.2367009099, + "request_id": 3, + "tpot_ms": 136.28731175562888, + "ttft_ms": 298.7481079450336 + }, + { + "e2e_ms": 17921.914449676293, + "request_id": 4, + "tpot_ms": 138.39011926907332, + "ttft_ms": 346.36930250398024 + }, + { + "e2e_ms": 17935.406822135243, + "request_id": 5, + "tpot_ms": 138.88107821741167, + "ttft_ms": 297.50988852396 + }, + { + "e2e_ms": 18057.925904932094, + "request_id": 6, + "tpot_ms": 139.39521236971183, + "ttft_ms": 354.73393397869125 + }, + { + "e2e_ms": 18071.482157513437, + "request_id": 7, + "tpot_ms": 139.75067177492934, + "ttft_ms": 323.14684209741085 + }, + { + "e2e_ms": 18194.065120430685, + "request_id": 8, + "tpot_ms": 140.142283173617, + "ttft_ms": 395.99515738132584 + }, + { + "e2e_ms": 18207.834601589766, + "request_id": 9, + "tpot_ms": 140.4257392143267, + "ttft_ms": 373.76572137027256 + }, + { + "e2e_ms": 18221.607066695313, + "request_id": 10, + "tpot_ms": 140.60137526077196, + "ttft_ms": 365.232408577274 + }, + { + "e2e_ms": 18235.379531799863, + "request_id": 11, + "tpot_ms": 140.670460527253, + "ttft_ms": 370.23104483873226 + }, + { + "e2e_ms": 18249.21289308034, + "request_id": 12, + "tpot_ms": 140.64257855984852, + "ttft_ms": 387.6054159795803 + }, + { + "e2e_ms": 18263.046254360816, + "request_id": 13, + "tpot_ms": 140.69995603545655, + "ttft_ms": 394.1518378578337 + }, + { + "e2e_ms": 18276.879615640297, + "request_id": 14, + "tpot_ms": 140.75508408264292, + "ttft_ms": 400.98393714464817 + }, + { + "e2e_ms": 18181.536918127527, + "request_id": 15, + "tpot_ms": 140.80327015717634, + "ttft_ms": 299.52160816613116 + }, + { + "e2e_ms": 18195.15705083027, + "request_id": 16, + "tpot_ms": 140.84624138271937, + "ttft_ms": 307.6843952249071 + }, + { + "e2e_ms": 18208.77718353195, + "request_id": 17, + "tpot_ms": 140.87800086343316, + "ttft_ms": 317.2710738759417 + }, + { + "e2e_ms": 18222.54666469151, + "request_id": 18, + "tpot_ms": 140.90607860065492, + "ttft_ms": 327.4746824083348 + }, + { + "e2e_ms": 18236.319129797193, + "request_id": 19, + "tpot_ms": 140.91088449209656, + "ttft_ms": 340.6367993009294 + }, + { + "e2e_ms": 18250.152491077173, + "request_id": 20, + "tpot_ms": 140.8981539798204, + "ttft_ms": 356.0869356399827 + }, + { + "e2e_ms": 18263.985852358146, + "request_id": 21, + "tpot_ms": 140.89857595079965, + "ttft_ms": 369.8667066065901 + }, + { + "e2e_ms": 18277.81921363912, + "request_id": 22, + "tpot_ms": 140.90338208286406, + "ttft_ms": 383.0896891153834 + }, + { + "e2e_ms": 18291.6525749191, + "request_id": 23, + "tpot_ms": 140.91257237601366, + "ttft_ms": 395.7558831653678 + }, + { + "e2e_ms": 18305.48593620007, + "request_id": 24, + "tpot_ms": 140.9217626691632, + "ttft_ms": 408.4220772163469 + }, + { + "e2e_ms": 18210.143238687502, + "request_id": 25, + "tpot_ms": 140.92327164841032, + "ttft_ms": 312.8877393393914 + }, + { + "e2e_ms": 18223.912719846063, + "request_id": 26, + "tpot_ms": 140.92545360667498, + "ttft_ms": 326.38011179834336 + }, + { + "e2e_ms": 18237.68518495175, + "request_id": 27, + "tpot_ms": 140.92765906057843, + "ttft_ms": 339.8724842582901 + }, + { + "e2e_ms": 18251.518546232724, + "request_id": 28, + "tpot_ms": 140.92984101884323, + "ttft_ms": 353.42873683963205 + }, + { + "e2e_ms": 18265.3519075127, + "request_id": 29, + "tpot_ms": 140.93202297710803, + "ttft_ms": 366.98498941997923 + }, + { + "e2e_ms": 18279.185268793677, + "request_id": 30, + "tpot_ms": 140.93370194229104, + "ttft_ms": 380.60512212271647 + }, + { + "e2e_ms": 18293.01863007465, + "request_id": 31, + "tpot_ms": 140.9341814397353, + "ttft_ms": 394.3775872282629 + }, + { + "e2e_ms": 18306.851991354626, + "request_id": 32, + "tpot_ms": 140.93466093717961, + "ttft_ms": 408.15005233281454 + }, + { + "e2e_ms": 18211.509293842057, + "request_id": 33, + "tpot_ms": 140.933484965084, + "ttft_ms": 312.956703276388 + }, + { + "e2e_ms": 18225.278775001614, + "request_id": 34, + "tpot_ms": 140.9329819720059, + "ttft_ms": 326.79006455686374 + }, + { + "e2e_ms": 18239.11213628159, + "request_id": 35, + "tpot_ms": 140.93298197200983, + "ttft_ms": 340.6234258363447 + }, + { + "e2e_ms": 18252.945497562563, + "request_id": 36, + "tpot_ms": 140.93298197201372, + "ttft_ms": 354.45678711682046 + }, + { + "e2e_ms": 18266.778858843536, + "request_id": 37, + "tpot_ms": 140.93466093719667, + "ttft_ms": 368.0769198195577 + }, + { + "e2e_ms": 18280.612220123516, + "request_id": 38, + "tpot_ms": 140.93633990237933, + "ttft_ms": 381.69705252133923 + }, + { + "e2e_ms": 18294.44558140449, + "request_id": 39, + "tpot_ms": 140.93801886756114, + "ttft_ms": 395.3171852242221 + }, + { + "e2e_ms": 18308.27894268546, + "request_id": 40, + "tpot_ms": 140.93849836500434, + "ttft_ms": 409.0896503299106 + }, + { + "e2e_ms": 18212.936245171895, + "request_id": 41, + "tpot_ms": 140.93732239290452, + "ttft_ms": 313.8963012730187 + }, + { + "e2e_ms": 18226.708710277584, + "request_id": 42, + "tpot_ms": 140.93684289546135, + "ttft_ms": 327.72966255399183 + }, + { + "e2e_ms": 18047.749752064, + "request_id": 43, + "tpot_ms": 139.41879313566167, + "ttft_ms": 341.56302383496495 + }, + { + "e2e_ms": 17872.42465139999, + "request_id": 44, + "tpot_ms": 137.92935642744132, + "ttft_ms": 355.3963851149433 + }, + { + "e2e_ms": 17695.178674250878, + "request_id": 45, + "tpot_ms": 136.4247947075194, + "ttft_ms": 369.22974639591644 + }, + { + "e2e_ms": 17514.86814366097, + "request_id": 46, + "tpot_ms": 134.89610264554392, + "ttft_ms": 383.06310767688956 + }, + { + "e2e_ms": 17334.24324324006, + "request_id": 47, + "tpot_ms": 133.36661419575813, + "ttft_ms": 396.68324037877767 + }, + { + "e2e_ms": 17152.122611965817, + "request_id": 48, + "tpot_ms": 131.8241488699319, + "ttft_ms": 410.4557054844662 + }, + { + "e2e_ms": 16860.906258226023, + "request_id": 49, + "tpot_ms": 130.28066064407446, + "ttft_ms": 315.26235642856903 + }, + { + "e2e_ms": 16678.317507592936, + "request_id": 50, + "tpot_ms": 128.73402984160936, + "ttft_ms": 329.0957177085474 + }, + { + "e2e_ms": 16495.414294222966, + "request_id": 51, + "tpot_ms": 127.18492295459406, + "ttft_ms": 342.9290789895205 + }, + { + "e2e_ms": 16325.716031493954, + "request_id": 52, + "tpot_ms": 125.73979205687765, + "ttft_ms": 356.76244027049364 + }, + { + "e2e_ms": 16142.044362006885, + "request_id": 53, + "tpot_ms": 124.18463433430247, + "ttft_ms": 370.595801550472 + }, + { + "e2e_ms": 15942.642543961227, + "request_id": 54, + "tpot_ms": 122.50561717425025, + "ttft_ms": 384.4291628314451 + }, + { + "e2e_ms": 15727.503328455541, + "request_id": 55, + "tpot_ms": 120.70436246394655, + "ttft_ms": 398.049295534328 + }, + { + "e2e_ms": 15403.205770995846, + "request_id": 56, + "tpot_ms": 118.90039231904261, + "ttft_ms": 302.8559464774361 + }, + { + "e2e_ms": 15178.030676088234, + "request_id": 57, + "tpot_ms": 117.01843597110097, + "ttft_ms": 316.6893077584092 + }, + { + "e2e_ms": 14941.341931765628, + "request_id": 58, + "tpot_ms": 115.04582096634839, + "ttft_ms": 330.5226690393823 + }, + { + "e2e_ms": 14696.772511761146, + "request_id": 59, + "tpot_ms": 113.01115339717941, + "ttft_ms": 344.3560303193607 + }, + { + "e2e_ms": 14451.34286402164, + "request_id": 60, + "tpot_ms": 110.96971238127013, + "ttft_ms": 358.1893916003338 + }, + { + "e2e_ms": 14196.293999590453, + "request_id": 61, + "tpot_ms": 108.85252950164681, + "ttft_ms": 372.02275288130693 + }, + { + "e2e_ms": 13938.386603616791, + "request_id": 62, + "tpot_ms": 106.71283849964966, + "ttft_ms": 385.8561141612853 + }, + { + "e2e_ms": 13676.049735660896, + "request_id": 63, + "tpot_ms": 104.53992523504405, + "ttft_ms": 399.4792308103001 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "3e07b28ac20428d11b8d83064d68561f03804f8adf37111de43398fdd8d94138" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 163.70222067832947, + "offered_request_rate": 1.6, + "request_metrics_sha256": "e85085ea5517c1afa9f850cb838a638360d45db695cde8d3c9fef6baf6af44c4", + "request_rate_per_gpu": 0.2, + "requests": [ + { + "e2e_ms": 19447.058061600903, + "request_id": 0, + "tpot_ms": 150.8298532036487, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 19750.640261300512, + "request_id": 1, + "tpot_ms": 152.83450342712686, + "ttft_ms": 340.65832605540146 + }, + { + "e2e_ms": 19750.02534103151, + "request_id": 2, + "tpot_ms": 153.17602578595117, + "ttft_ms": 296.67006621570977 + }, + { + "e2e_ms": 19855.93358266601, + "request_id": 3, + "tpot_ms": 153.5864661294274, + "ttft_ms": 350.4523842287295 + }, + { + "e2e_ms": 19855.232098385564, + "request_id": 4, + "tpot_ms": 153.87573615619343, + "ttft_ms": 313.01360654900145 + }, + { + "e2e_ms": 19961.05377600862, + "request_id": 5, + "tpot_ms": 154.2244676371592, + "ttft_ms": 374.54638608940184 + }, + { + "e2e_ms": 19960.265727716724, + "request_id": 6, + "tpot_ms": 154.4718650613629, + "ttft_ms": 342.33886492363473 + }, + { + "e2e_ms": 19959.47767942483, + "request_id": 7, + "tpot_ms": 154.6679176552935, + "ttft_ms": 316.6521372025519 + }, + { + "e2e_ms": 20065.255011109733, + "request_id": 8, + "tpot_ms": 154.88778812601134, + "ttft_ms": 394.50591910629294 + }, + { + "e2e_ms": 20064.42261687969, + "request_id": 9, + "tpot_ms": 155.03615010045385, + "ttft_ms": 374.8315541220517 + }, + { + "e2e_ms": 20063.590222649644, + "request_id": 10, + "tpot_ms": 155.117974090142, + "ttft_ms": 363.6075132016101 + }, + { + "e2e_ms": 20062.7578284196, + "request_id": 11, + "tpot_ms": 155.13454318347627, + "ttft_ms": 360.67084411811567 + }, + { + "e2e_ms": 20061.925434189554, + "request_id": 12, + "tpot_ms": 155.09034115377113, + "ttft_ms": 365.45210766062206 + }, + { + "e2e_ms": 20061.09303995951, + "request_id": 13, + "tpot_ms": 155.09665427482798, + "ttft_ms": 363.8179470563543 + }, + { + "e2e_ms": 20060.260645729464, + "request_id": 14, + "tpot_ms": 155.10154332898202, + "ttft_ms": 362.3646429487479 + }, + { + "e2e_ms": 20059.42825149942, + "request_id": 15, + "tpot_ms": 155.10604675613317, + "ttft_ms": 360.96031347050683 + }, + { + "e2e_ms": 20058.595857269374, + "request_id": 16, + "tpot_ms": 155.10168661355542, + "ttft_ms": 360.6816573478344 + }, + { + "e2e_ms": 20057.76346303933, + "request_id": 17, + "tpot_ms": 155.0900405289204, + "ttft_ms": 361.3283158664373 + }, + { + "e2e_ms": 20056.931068809285, + "request_id": 18, + "tpot_ms": 155.07436184636043, + "ttft_ms": 362.4871143215085 + }, + { + "e2e_ms": 20056.09867457924, + "request_id": 19, + "tpot_ms": 155.0434256316313, + "ttft_ms": 365.58361936206654 + }, + { + "e2e_ms": 20055.26628034948, + "request_id": 20, + "tpot_ms": 154.9995151863024, + "ttft_ms": 370.32785168907446 + }, + { + "e2e_ms": 20054.433886119732, + "request_id": 21, + "tpot_ms": 154.96275487551375, + "ttft_ms": 374.16401692948574 + }, + { + "e2e_ms": 20053.601491889985, + "request_id": 22, + "tpot_ms": 154.92303029896564, + "ttft_ms": 378.3766439213494 + }, + { + "e2e_ms": 20052.76909766024, + "request_id": 23, + "tpot_ms": 154.88711594543284, + "ttft_ms": 382.105372590269 + }, + { + "e2e_ms": 20051.936703430492, + "request_id": 24, + "tpot_ms": 154.8482609975464, + "ttft_ms": 386.20755674210193 + }, + { + "e2e_ms": 20051.104309200746, + "request_id": 25, + "tpot_ms": 154.85607608193587, + "ttft_ms": 384.3826467948901 + }, + { + "e2e_ms": 20050.271914971, + "request_id": 26, + "tpot_ms": 154.86595665257255, + "ttft_ms": 382.2954200942838 + }, + { + "e2e_ms": 20049.439520741253, + "request_id": 27, + "tpot_ms": 154.87441181850545, + "ttft_ms": 380.3892197910592 + }, + { + "e2e_ms": 20048.607126511506, + "request_id": 28, + "tpot_ms": 154.87585298283847, + "ttft_ms": 379.37379769102006 + }, + { + "e2e_ms": 20047.77473228176, + "request_id": 29, + "tpot_ms": 154.8790680483812, + "ttft_ms": 378.13309013734295 + }, + { + "e2e_ms": 20046.942338052013, + "request_id": 30, + "tpot_ms": 154.879633497606, + "ttft_ms": 377.228883856052 + }, + { + "e2e_ms": 20046.109943822266, + "request_id": 31, + "tpot_ms": 154.8807461594705, + "ttft_ms": 376.2551815695119 + }, + { + "e2e_ms": 20045.27754959252, + "request_id": 32, + "tpot_ms": 154.8790337660788, + "ttft_ms": 375.64026130051076 + }, + { + "e2e_ms": 19846.84310747305, + "request_id": 33, + "tpot_ms": 153.32139973576017, + "ttft_ms": 375.02534103150964 + }, + { + "e2e_ms": 19648.543544766126, + "request_id": 34, + "tpot_ms": 151.76482774805996, + "ttft_ms": 374.4104207625085 + }, + { + "e2e_ms": 19450.677394587543, + "request_id": 35, + "tpot_ms": 150.2123500638227, + "ttft_ms": 373.7089364820605 + }, + { + "e2e_ms": 19253.237256186894, + "request_id": 36, + "tpot_ms": 148.66322680303372, + "ttft_ms": 373.00745220161247 + }, + { + "e2e_ms": 19054.80482620875, + "request_id": 37, + "tpot_ms": 147.10697182912625, + "ttft_ms": 372.2194039097175 + }, + { + "e2e_ms": 18856.422732154002, + "request_id": 38, + "tpot_ms": 145.55111320107227, + "ttft_ms": 371.4313556178226 + }, + { + "e2e_ms": 18658.616312962055, + "request_id": 39, + "tpot_ms": 143.99978744595376, + "ttft_ms": 370.64330732592765 + }, + { + "e2e_ms": 18469.244245876795, + "request_id": 40, + "tpot_ms": 142.51522309276308, + "ttft_ms": 369.81091309588265 + }, + { + "e2e_ms": 18279.907702313452, + "request_id": 41, + "tpot_ms": 141.0309384523434, + "ttft_ms": 368.97851886583766 + }, + { + "e2e_ms": 18091.61193748763, + "request_id": 42, + "tpot_ms": 139.55484892009318, + "ttft_ms": 368.14612463579266 + }, + { + "e2e_ms": 17903.060652176137, + "request_id": 43, + "tpot_ms": 138.07674741551486, + "ttft_ms": 367.31373040574766 + }, + { + "e2e_ms": 17716.73280728946, + "request_id": 44, + "tpot_ms": 136.61615331585634, + "ttft_ms": 366.48133617570267 + }, + { + "e2e_ms": 17529.12437807872, + "request_id": 45, + "tpot_ms": 135.14547587506345, + "ttft_ms": 365.64894194565767 + }, + { + "e2e_ms": 17339.472913240785, + "request_id": 46, + "tpot_ms": 133.65871153956826, + "ttft_ms": 364.81654771561267 + }, + { + "e2e_ms": 17149.611868516175, + "request_id": 47, + "tpot_ms": 132.17029696874496, + "ttft_ms": 363.9841534855677 + }, + { + "e2e_ms": 16958.75366988868, + "request_id": 48, + "tpot_ms": 130.67403079238707, + "ttft_ms": 363.1517592555227 + }, + { + "e2e_ms": 16767.44725356325, + "request_id": 49, + "tpot_ms": 129.1742353428171, + "ttft_ms": 362.3193650254777 + }, + { + "e2e_ms": 16576.276975363853, + "request_id": 50, + "tpot_ms": 127.67551184699543, + "ttft_ms": 361.4869707954327 + }, + { + "e2e_ms": 16384.89705533921, + "request_id": 51, + "tpot_ms": 126.1751376281405, + "ttft_ms": 360.65457656536637 + }, + { + "e2e_ms": 16202.320435741867, + "request_id": 52, + "tpot_ms": 124.74408073548226, + "ttft_ms": 359.8221823356198 + }, + { + "e2e_ms": 16010.428211639819, + "request_id": 53, + "tpot_ms": 123.23967262625155, + "ttft_ms": 358.98978810587323 + }, + { + "e2e_ms": 15808.049221831383, + "request_id": 54, + "tpot_ms": 121.6526915587028, + "ttft_ms": 358.15739387612666 + }, + { + "e2e_ms": 15595.17863371626, + "request_id": 55, + "tpot_ms": 119.98309948086519, + "ttft_ms": 357.3249996463801 + }, + { + "e2e_ms": 15377.4072544157, + "request_id": 56, + "tpot_ms": 118.27491849605565, + "ttft_ms": 356.4926054166335 + }, + { + "e2e_ms": 15157.846080032627, + "request_id": 57, + "tpot_ms": 116.55264463658062, + "ttft_ms": 355.66021118688695 + }, + { + "e2e_ms": 14930.609139372891, + "request_id": 58, + "tpot_ms": 114.76993167256497, + "ttft_ms": 354.8278169571404 + }, + { + "e2e_ms": 14698.1184149259, + "request_id": 59, + "tpot_ms": 112.9458503322717, + "ttft_ms": 353.9954227273938 + }, + { + "e2e_ms": 14465.054205320868, + "request_id": 60, + "tpot_ms": 111.11725336081277, + "ttft_ms": 353.16302849764725 + }, + { + "e2e_ms": 14225.577184589198, + "request_id": 61, + "tpot_ms": 109.23816181355352, + "ttft_ms": 352.3306342679007 + }, + { + "e2e_ms": 13984.194476163324, + "request_id": 62, + "tpot_ms": 107.34406485137929, + "ttft_ms": 351.4982400381541 + }, + { + "e2e_ms": 13739.858786414643, + "request_id": 63, + "tpot_ms": 105.42671606776564, + "ttft_ms": 350.66584580840754 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "850087527f8ec5b333ae4bb00c5837c85f7dafca77de3d8c80820e071da8dfbb" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 386.0896592140198, + "offered_request_rate": 2.4, + "request_metrics_sha256": "c4ae9e9bb2050189ca506883c0c12efd5faecce0ec6249ad641d32f3b9a7f130", + "request_rate_per_gpu": 0.3, + "requests": [ + { + "e2e_ms": 25856.75271176486, + "request_id": 0, + "tpot_ms": 201.29988981911293, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26179.284600266838, + "request_id": 1, + "tpot_ms": 203.30897363251975, + "ttft_ms": 359.0449489368311 + }, + { + "e2e_ms": 25981.87281483019, + "request_id": 2, + "tpot_ms": 201.96786246415923, + "ttft_ms": 331.9542818819672 + }, + { + "e2e_ms": 25784.35394549511, + "request_id": 3, + "tpot_ms": 200.62102535110301, + "ttft_ms": 305.48372590502584 + }, + { + "e2e_ms": 25696.083985341404, + "request_id": 4, + "tpot_ms": 199.35042104178692, + "ttft_ms": 378.5805130344673 + }, + { + "e2e_ms": 25498.25455731707, + "request_id": 5, + "tpot_ms": 197.981349219281, + "ttft_ms": 354.62320646838384 + }, + { + "e2e_ms": 25300.295013812065, + "request_id": 6, + "tpot_ms": 196.59886557857163, + "ttft_ms": 332.2390853334691 + }, + { + "e2e_ms": 25102.100957670827, + "request_id": 7, + "tpot_ms": 195.19408059762435, + "ttft_ms": 312.45272177253236 + }, + { + "e2e_ms": 25012.968951433395, + "request_id": 8, + "tpot_ms": 193.83981874911316, + "ttft_ms": 395.3119702960262 + }, + { + "e2e_ms": 24814.29543067174, + "request_id": 9, + "tpot_ms": 192.41025790813484, + "ttft_ms": 378.1926763386156 + }, + { + "e2e_ms": 24615.47335754875, + "request_id": 10, + "tpot_ms": 190.95429488392014, + "ttft_ms": 364.2779072908935 + }, + { + "e2e_ms": 24416.46039907787, + "request_id": 11, + "tpot_ms": 189.47286981272285, + "ttft_ms": 353.405932862068 + }, + { + "e2e_ms": 24216.993764693514, + "request_id": 12, + "tpot_ms": 187.96377697549212, + "ttft_ms": 345.59408880601427 + }, + { + "e2e_ms": 24017.241128749425, + "request_id": 13, + "tpot_ms": 186.4682030159872, + "ttft_ms": 335.7793457190477 + }, + { + "e2e_ms": 23816.999491057053, + "request_id": 14, + "tpot_ms": 184.96817994362138, + "ttft_ms": 326.0406382171395 + }, + { + "e2e_ms": 23616.430479744246, + "request_id": 15, + "tpot_ms": 183.46465752234423, + "ttft_ms": 316.41897440652934 + }, + { + "e2e_ms": 23412.66959473249, + "request_id": 16, + "tpot_ms": 181.92890330417333, + "ttft_ms": 307.69887510247963 + }, + { + "e2e_ms": 23315.084360363784, + "request_id": 17, + "tpot_ms": 180.37041078851854, + "ttft_ms": 408.042190221928 + }, + { + "e2e_ms": 23111.095991700124, + "request_id": 18, + "tpot_ms": 178.82629774094855, + "ttft_ms": 400.1561785996568 + }, + { + "e2e_ms": 22907.001651879975, + "request_id": 19, + "tpot_ms": 177.27413055190453, + "ttft_ms": 393.18707178810007 + }, + { + "e2e_ms": 22702.9918523004, + "request_id": 20, + "tpot_ms": 175.71469647400258, + "ttft_ms": 387.22540010207496 + }, + { + "e2e_ms": 22499.024594644914, + "request_id": 21, + "tpot_ms": 174.1539791842547, + "ttft_ms": 381.46923824456815 + }, + { + "e2e_ms": 22294.98921706886, + "request_id": 22, + "tpot_ms": 172.59076723758398, + "ttft_ms": 375.96177789569475 + }, + { + "e2e_ms": 22091.055170736552, + "request_id": 23, + "tpot_ms": 171.02806584364455, + "ttft_ms": 370.49080859369354 + }, + { + "e2e_ms": 21886.829371806663, + "request_id": 24, + "tpot_ms": 169.4599867344259, + "ttft_ms": 365.4110565345743 + }, + { + "e2e_ms": 21682.517380669786, + "request_id": 25, + "tpot_ms": 167.90637996068475, + "ttft_ms": 358.40712566282207 + }, + { + "e2e_ms": 21478.40806232245, + "request_id": 26, + "tpot_ms": 166.35416808544093, + "ttft_ms": 351.4287154714495 + }, + { + "e2e_ms": 21274.47246393178, + "request_id": 27, + "tpot_ms": 164.80170050625136, + "ttft_ms": 344.6564996378552 + }, + { + "e2e_ms": 21070.655727841604, + "request_id": 28, + "tpot_ms": 163.24706150884583, + "ttft_ms": 338.278916218183 + }, + { + "e2e_ms": 20866.83848786135, + "request_id": 29, + "tpot_ms": 161.69251523328387, + "ttft_ms": 331.88905323430083 + }, + { + "e2e_ms": 20663.178771213603, + "request_id": 30, + "tpot_ms": 160.1348533340772, + "ttft_ms": 326.0523977857979 + }, + { + "e2e_ms": 20459.48769489552, + "request_id": 31, + "tpot_ms": 158.57430185728822, + "ttft_ms": 320.55135901991605 + }, + { + "e2e_ms": 20255.951788183756, + "request_id": 32, + "tpot_ms": 157.01541837295076, + "ttft_ms": 314.9936548190073 + }, + { + "e2e_ms": 20052.589009493837, + "request_id": 33, + "tpot_ms": 155.45687267778135, + "ttft_ms": 309.5661794156026 + }, + { + "e2e_ms": 19955.947670796646, + "request_id": 34, + "tpot_ms": 153.8983040696114, + "ttft_ms": 410.8630539559979 + }, + { + "e2e_ms": 19752.847929041352, + "request_id": 35, + "tpot_ms": 152.3343481355919, + "ttft_ms": 406.3857158211803 + }, + { + "e2e_ms": 19549.96119317403, + "request_id": 36, + "tpot_ms": 150.77184485827368, + "ttft_ms": 401.93689617327345 + }, + { + "e2e_ms": 19346.57831151796, + "request_id": 37, + "tpot_ms": 149.20507336304368, + "ttft_ms": 397.53399441140937 + }, + { + "e2e_ms": 19143.22059782458, + "request_id": 38, + "tpot_ms": 147.63442381272193, + "ttft_ms": 393.6487736088932 + }, + { + "e2e_ms": 18940.150721561607, + "request_id": 39, + "tpot_ms": 146.06310547094216, + "ttft_ms": 390.13632675195356 + }, + { + "e2e_ms": 18741.29802135198, + "request_id": 40, + "tpot_ms": 144.5231784918296, + "ttft_ms": 386.85435288961753 + }, + { + "e2e_ms": 18542.463082904305, + "request_id": 41, + "tpot_ms": 142.98167897793616, + "ttft_ms": 383.7898527064141 + }, + { + "e2e_ms": 18344.148533824395, + "request_id": 42, + "tpot_ms": 141.4401875845621, + "ttft_ms": 381.2447105850083 + }, + { + "e2e_ms": 18145.706224501653, + "request_id": 43, + "tpot_ms": 139.89417135289136, + "ttft_ms": 379.1464626844494 + }, + { + "e2e_ms": 17948.37563539231, + "request_id": 44, + "tpot_ms": 138.3533077037345, + "ttft_ms": 377.5055570180292 + }, + { + "e2e_ms": 17750.40475411994, + "request_id": 45, + "tpot_ms": 136.80479670989413, + "ttft_ms": 376.19557196338246 + }, + { + "e2e_ms": 17551.412355033975, + "request_id": 46, + "tpot_ms": 135.2471347218224, + "ttft_ms": 375.0262453625339 + }, + { + "e2e_ms": 17352.31516600567, + "request_id": 47, + "tpot_ms": 133.6870504253856, + "ttft_ms": 374.0597619816981 + }, + { + "e2e_ms": 17152.719400024922, + "request_id": 48, + "tpot_ms": 132.11931086353468, + "ttft_ms": 373.56692035601566 + }, + { + "e2e_ms": 16952.899525195207, + "request_id": 49, + "tpot_ms": 130.53228068835782, + "ttft_ms": 375.29987777376664 + }, + { + "e2e_ms": 16753.147719429508, + "request_id": 50, + "tpot_ms": 128.94354832701373, + "ttft_ms": 377.31708189876304 + }, + { + "e2e_ms": 16553.291092750187, + "request_id": 51, + "tpot_ms": 127.34853489391125, + "ttft_ms": 380.0271612234596 + }, + { + "e2e_ms": 16357.836116284518, + "request_id": 52, + "tpot_ms": 125.77762361641257, + "ttft_ms": 384.07791700011984 + }, + { + "e2e_ms": 16157.72333756749, + "request_id": 53, + "tpot_ms": 124.16553823707912, + "ttft_ms": 388.6999814584407 + }, + { + "e2e_ms": 15952.367175996273, + "request_id": 54, + "tpot_ms": 122.51027887411597, + "ttft_ms": 393.5617589835445 + }, + { + "e2e_ms": 15741.765215271713, + "request_id": 55, + "tpot_ms": 120.81149800475544, + "ttft_ms": 398.70496866777216 + }, + { + "e2e_ms": 15528.71285895543, + "request_id": 56, + "tpot_ms": 119.08582914503211, + "ttft_ms": 404.8125575363528 + }, + { + "e2e_ms": 15314.765605096894, + "request_id": 57, + "tpot_ms": 117.34894161957355, + "ttft_ms": 411.4500194110526 + }, + { + "e2e_ms": 15096.980468100028, + "request_id": 58, + "tpot_ms": 115.58065073324867, + "ttft_ms": 418.2378249774459 + }, + { + "e2e_ms": 14778.441120322277, + "request_id": 59, + "tpot_ms": 113.87954551567164, + "ttft_ms": 315.7388398319796 + }, + { + "e2e_ms": 14557.742348852762, + "request_id": 60, + "tpot_ms": 112.08473611057838, + "ttft_ms": 322.9808628093096 + }, + { + "e2e_ms": 14333.83717181993, + "request_id": 61, + "tpot_ms": 110.26282336485625, + "ttft_ms": 330.4586044831872 + }, + { + "e2e_ms": 14108.97915094099, + "request_id": 62, + "tpot_ms": 108.43298632152924, + "ttft_ms": 337.989888106776 + }, + { + "e2e_ms": 13882.64463939965, + "request_id": 63, + "tpot_ms": 106.58813679106942, + "ttft_ms": 345.9512669338345 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "0b9f30d883de83b4af69dd114bd5d2bb974b7fd1f01d45f913bf5049e4b72a86" + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "elapsed_seconds": 400.40727281570435, + "offered_request_rate": 3.2, + "request_metrics_sha256": "48ff1d03e0aeb45bb4395dca86e089cdbe1c7de16b8557fbe8f25e8e65c77def", + "request_rate_per_gpu": 0.4, + "requests": [ + { + "e2e_ms": 26482.398681960134, + "request_id": 0, + "tpot_ms": 206.22623604112295, + "ttft_ms": 291.66670473752106 + }, + { + "e2e_ms": 26389.27725660841, + "request_id": 1, + "tpot_ms": 204.89007083645959, + "ttft_ms": 368.23826037804076 + }, + { + "e2e_ms": 26186.398050214455, + "request_id": 2, + "tpot_ms": 203.43938315354237, + "ttft_ms": 349.5963897145746 + }, + { + "e2e_ms": 25983.46530187178, + "request_id": 3, + "tpot_ms": 201.98715121187504, + "ttft_ms": 331.0970979636474 + }, + { + "e2e_ms": 25780.42643937068, + "request_id": 4, + "tpot_ms": 200.53276712329495, + "ttft_ms": 312.76501471222184 + }, + { + "e2e_ms": 25577.34505869151, + "request_id": 5, + "tpot_ms": 199.0733214464005, + "ttft_ms": 295.0332349986446 + }, + { + "e2e_ms": 25483.55218185351, + "request_id": 6, + "tpot_ms": 197.69083780569105, + "ttft_ms": 376.81578053074594 + }, + { + "e2e_ms": 25280.28848711639, + "request_id": 7, + "tpot_ms": 196.22290463044078, + "ttft_ms": 359.97959905041466 + }, + { + "e2e_ms": 25076.966707259246, + "request_id": 8, + "tpot_ms": 194.75030755695653, + "ttft_ms": 343.6776475257637 + }, + { + "e2e_ms": 24873.46328021192, + "request_id": 9, + "tpot_ms": 193.27265662306831, + "ttft_ms": 327.83588908224147 + }, + { + "e2e_ms": 24669.88557698392, + "request_id": 10, + "tpot_ms": 191.78984111033463, + "ttft_ms": 312.57575597142215 + }, + { + "e2e_ms": 24466.212431081483, + "request_id": 11, + "tpot_ms": 190.30296300954802, + "ttft_ms": 297.7361288688827 + }, + { + "e2e_ms": 24370.912463364126, + "request_id": 12, + "tpot_ms": 188.79387017231738, + "ttft_ms": 394.09095147981964 + }, + { + "e2e_ms": 24166.86947872558, + "request_id": 13, + "tpot_ms": 187.2967209572732, + "ttft_ms": 380.1859171518851 + }, + { + "e2e_ms": 23962.581993212396, + "request_id": 14, + "tpot_ms": 185.79746051386414, + "ttft_ms": 366.30450795165007 + }, + { + "e2e_ms": 23758.13082088949, + "request_id": 15, + "tpot_ms": 184.29572160562404, + "ttft_ms": 352.57417697523687 + }, + { + "e2e_ms": 23552.083711717114, + "request_id": 16, + "tpot_ms": 182.7751996506789, + "ttft_ms": 339.633356080892 + }, + { + "e2e_ms": 23345.944189149544, + "request_id": 17, + "tpot_ms": 181.25255279330477, + "ttft_ms": 326.8699843998375 + }, + { + "e2e_ms": 23139.783338151214, + "request_id": 18, + "tpot_ms": 179.72694307410856, + "ttft_ms": 314.461567739424 + }, + { + "e2e_ms": 22933.56950157464, + "request_id": 19, + "tpot_ms": 178.1977153273412, + "ttft_ms": 302.45965500230557 + }, + { + "e2e_ms": 22833.726368661068, + "request_id": 20, + "tpot_ms": 176.63828124943853, + "ttft_ms": 400.6646499823745 + }, + { + "e2e_ms": 22627.576073166823, + "request_id": 21, + "tpot_ms": 175.09813975268048, + "ttft_ms": 390.11232457640244 + }, + { + "e2e_ms": 22421.391717712297, + "request_id": 22, + "tpot_ms": 173.55627478074584, + "ttft_ms": 379.7448205575735 + }, + { + "e2e_ms": 22215.258027879143, + "request_id": 23, + "tpot_ms": 172.01247264259936, + "ttft_ms": 369.67400226902566 + }, + { + "e2e_ms": 22008.978461747698, + "request_id": 24, + "tpot_ms": 170.46437149324944, + "ttft_ms": 360.0032821050201 + }, + { + "e2e_ms": 21802.65579951293, + "request_id": 25, + "tpot_ms": 168.91532250977505, + "ttft_ms": 350.4098407714995 + }, + { + "e2e_ms": 21596.434473671965, + "request_id": 26, + "tpot_ms": 167.36573728219415, + "ttft_ms": 340.98583883331025 + }, + { + "e2e_ms": 21390.300007810118, + "request_id": 27, + "tpot_ms": 165.81511332736412, + "ttft_ms": 331.7806152348748 + }, + { + "e2e_ms": 21184.224973098888, + "request_id": 28, + "tpot_ms": 164.26380332238384, + "ttft_ms": 322.72195115614187 + }, + { + "e2e_ms": 20978.149686441437, + "request_id": 29, + "tpot_ms": 162.71174941720716, + "ttft_ms": 313.7575104561279 + }, + { + "e2e_ms": 20772.153161450715, + "request_id": 30, + "tpot_ms": 161.15510654351164, + "ttft_ms": 305.4546304247392 + }, + { + "e2e_ms": 20672.62875180034, + "request_id": 31, + "tpot_ms": 159.59455506672822, + "ttft_ms": 404.12025832585744 + }, + { + "e2e_ms": 20466.69413177714, + "request_id": 32, + "tpot_ms": 158.03533915661183, + "ttft_ms": 396.2060588874419 + }, + { + "e2e_ms": 20260.84607576572, + "request_id": 33, + "tpot_ms": 156.475168525323, + "ttft_ms": 388.49967304969545 + }, + { + "e2e_ms": 20055.02118508905, + "request_id": 34, + "tpot_ms": 154.9132321630232, + "ttft_ms": 381.04070038510326 + }, + { + "e2e_ms": 19849.304647544788, + "request_id": 35, + "tpot_ms": 153.3461190070677, + "ttft_ms": 374.34753364718887 + }, + { + "e2e_ms": 19643.694612944317, + "request_id": 36, + "tpot_ms": 151.77962188577303, + "ttft_ms": 367.6826334511443 + }, + { + "e2e_ms": 19437.836505449406, + "request_id": 37, + "tpot_ms": 150.210341914627, + "ttft_ms": 361.1230822917744 + }, + { + "e2e_ms": 19231.99098193562, + "request_id": 38, + "tpot_ms": 148.63640086508497, + "ttft_ms": 355.16807206983003 + }, + { + "e2e_ms": 19026.289377137648, + "request_id": 39, + "tpot_ms": 147.05985988148976, + "ttft_ms": 349.6871721884496 + }, + { + "e2e_ms": 18822.696360365982, + "request_id": 40, + "tpot_ms": 145.49844654462956, + "ttft_ms": 344.3936491980288 + }, + { + "e2e_ms": 18619.112224474804, + "request_id": 41, + "tpot_ms": 143.93654065290042, + "ttft_ms": 339.17156155644966 + }, + { + "e2e_ms": 18415.78828326805, + "request_id": 42, + "tpot_ms": 142.37219516274803, + "ttft_ms": 334.51949759905375 + }, + { + "e2e_ms": 18212.400461939902, + "request_id": 43, + "tpot_ms": 140.8040960160447, + "ttft_ms": 330.280267902225 + }, + { + "e2e_ms": 18009.568500717836, + "request_id": 44, + "tpot_ms": 139.2366051127279, + "ttft_ms": 326.5196514013962 + }, + { + "e2e_ms": 17806.416393414805, + "request_id": 45, + "tpot_ms": 137.66365486442547, + "ttft_ms": 323.1322256327704 + }, + { + "e2e_ms": 17602.75352720573, + "request_id": 46, + "tpot_ms": 136.08599255484802, + "ttft_ms": 319.83247274003014 + }, + { + "e2e_ms": 17399.03826602459, + "request_id": 47, + "tpot_ms": 134.50648843613666, + "ttft_ms": 316.71423463523496 + }, + { + "e2e_ms": 17195.073716367715, + "request_id": 48, + "tpot_ms": 132.92201961792392, + "ttft_ms": 313.9772248913761 + }, + { + "e2e_ms": 16990.997112286357, + "request_id": 49, + "tpot_ms": 131.3320936883647, + "ttft_ms": 311.8212138640413 + }, + { + "e2e_ms": 16895.41197318666, + "request_id": 50, + "tpot_ms": 129.74336132702078, + "ttft_ms": 418.0050846550216 + }, + { + "e2e_ms": 16691.3169931805, + "request_id": 51, + "tpot_ms": 128.14881090811454, + "ttft_ms": 416.418007849952 + }, + { + "e2e_ms": 16489.422838281163, + "request_id": 52, + "tpot_ms": 126.56215930365238, + "ttft_ms": 416.0286067173118 + }, + { + "e2e_ms": 16285.199782255653, + "request_id": 53, + "tpot_ms": 124.95445752714085, + "ttft_ms": 415.9836763087661 + }, + { + "e2e_ms": 16078.355034803544, + "request_id": 54, + "tpot_ms": 123.3249765236981, + "ttft_ms": 416.08301629388575 + }, + { + "e2e_ms": 15868.887387774763, + "request_id": 55, + "tpot_ms": 121.67321161824907, + "ttft_ms": 416.3895122571297 + }, + { + "e2e_ms": 15658.194542949623, + "request_id": 56, + "tpot_ms": 120.00563624201372, + "ttft_ms": 417.4787402138804 + }, + { + "e2e_ms": 15447.054249353854, + "request_id": 57, + "tpot_ms": 118.33082151865976, + "ttft_ms": 419.0399164840635 + }, + { + "e2e_ms": 15233.99501418892, + "request_id": 58, + "tpot_ms": 116.64063648866745, + "ttft_ms": 420.6341801281539 + }, + { + "e2e_ms": 15019.622333077174, + "request_id": 59, + "tpot_ms": 114.93953127109042, + "ttft_ms": 422.3018616486911 + }, + { + "e2e_ms": 14805.106280675915, + "request_id": 60, + "tpot_ms": 113.23514577866216, + "ttft_ms": 424.2427667858202 + }, + { + "e2e_ms": 14588.987025492997, + "request_id": 61, + "tpot_ms": 111.5171161341123, + "ttft_ms": 426.31327646073555 + }, + { + "e2e_ms": 14372.391348386529, + "request_id": 62, + "tpot_ms": 109.79194855942059, + "ttft_ms": 428.8138813401155 + }, + { + "e2e_ms": 14155.057425949359, + "request_id": 63, + "tpot_ms": 108.05983074662959, + "ttft_ms": 431.45892112740114 + } + ], + "slos": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.109375, + "passed": 7 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.40625, + "passed": 26 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + }, + "status": "completed", + "trace_sha256": "96c4e04e8f133e66449a5567fcde568f27af44bd9f5d1cadeade66b771a8f0f3" + } + ] + } + ], + "contract": { + "input_tokens": 2048, + "output_tokens": 128, + "prefix_caching": false, + "rates": [ + 0.1, + 0.2, + 0.4, + 0.8, + 1.2, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "target_pass_rate": 0.95, + "tpot_slos_ms": [ + 40.0, + 120.0, + 150.0, + 180.0 + ], + "ttft_slo_ms": 1256.0 + }, + "frontier": { + "declared_upstream_commit": "d9cfeb6d8791fbf2f295dd9744c56a666171776e", + "fingerprint_source": "prefill-grid-v3 frozen run manifest for the same immutable source snapshot", + "python_and_config_tree_sha256": "172fc7ae19c40e67030208ff488d0d5d90764888ed76a7a543be6037ba62dc11", + "source": "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3" + }, + "profiles": { + "coverage": { + "rows": 1104, + "standard_decode_by_tp": { + "4": 81, + "8": 81 + }, + "true_mixed_by_tp": { + "4": 108, + "8": 108 + } + }, + "files_sha256": { + "all_reduce": "8812393e3397d4a0380eae123bbad716ef8242cd2db1fab45e13066f595852b1", + "attention": "3cc6c78171a85adce6f5c9945ba21be76b9912e580f99e3f2063a15d7c21e1cd", + "closure_manifest": "38b982b265c5a71b91dbe876d4daaec3b0b15785624eecc042f26836014df1f3", + "linear": "60006d61a84e2976fe81895a4a1800aa722ffe43ea5d6f2b33a8c40ca6084f33", + "moe": "58b2b117f332d37a7d2550cce46cf9312de7ccdedaccb2b5680f485e02a06f55" + }, + "root": "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/profiles-t0-full-closed-v1" + }, + "rankings": { + "tpot_120ms": [ + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.8, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.8, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.8, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.8, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.05, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.05, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.05, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 0.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.05, + "upper_censored": false + } + ], + "tpot_150ms": [ + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.4, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.4, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.4, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.4, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.2, + "maximum_tested_feasible_request_rate_per_gpu": 0.15, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.2, + "maximum_tested_feasible_request_rate_per_gpu": 0.15, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.2, + "maximum_tested_feasible_request_rate_per_gpu": 0.15, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.2, + "maximum_tested_feasible_request_rate_per_gpu": 0.15, + "upper_censored": false + } + ], + "tpot_180ms": [ + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 2.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.6, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 2.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.6, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 2.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.6, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 2.4, + "maximum_tested_feasible_request_rate_per_gpu": 0.6, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": false, + "maximum_tested_feasible_request_rate": 1.6, + "maximum_tested_feasible_request_rate_per_gpu": 0.2, + "upper_censored": false + } + ], + "tpot_40ms": [ + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "lower_censored": true, + "maximum_tested_feasible_request_rate": null, + "maximum_tested_feasible_request_rate_per_gpu": null, + "upper_censored": false + } + ] + }, + "runner": { + "path": "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/run_frontier_t0_surface.py", + "sha256": "cd69c9b8321361c90ca368a86c551b6ddd6bbe7c0a5cbfbb617affa9e783d9df" + }, + "schema": "frontier-qwen235b-t0-surface-v1", + "status": "frozen_before_real_surface" +} diff --git a/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/exclusions.json b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/exclusions.json new file mode 100644 index 0000000..b15f445 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/exclusions.json @@ -0,0 +1,61 @@ +{ + "schema": "qwen235b-t0-real-exclusions-v1", + "policy": "Exclude the complete output directory whenever another process shares any allocated GPU or benchmark endpoint during warmup or a measured anchor.", + "excluded_attempts": [ + { + "reason": "Fleet monitor re-probed during model load and oversubscribed both TP4 allocations; the later MNS128 attempts overlapped the MNS64 warmup and start of r0p10.", + "remote_quarantine": "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/contaminated-queue-race-20260716T045500Z", + "run_ids": [ + "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T045338281933Z", + "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T045339503446Z", + "qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T045458846499Z", + "qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T045500253803Z" + ] + }, + { + "reason": "A dispatch command still copying the source was mistakenly resubmitted. The duplicate controllers reused the first pair's open ports and issued concurrent warmup requests, so both the original and duplicate outputs were rejected.", + "remote_quarantine": "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/contaminated-duplicate-dispatch-20260716T0501Z", + "run_ids": [ + "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T050017477963Z", + "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T050018748070Z", + "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T050123485776Z", + "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T050124800483Z" + ] + }, + { + "reason": "Clean diagnostic run exposed cross-anchor execution-state leakage: TPOT at 1.60 req/s failed after a sustained 0.10 anchor but passed after a 2.40 anchor. The multi-rate-per-server contract does not match Frontier's independent rate runs, so the complete attempt is diagnostic-only.", + "remote_quarantine": "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/diagnostic-cross-anchor-state-20260716T0504Z", + "run_ids": [ + "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T050405703312Z", + "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T050407104293Z" + ] + } + ], + "accepted_attempts": [ + "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z", + "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z", + "qwen235b-t0-real-tp4_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T062904775741Z", + "qwen235b-t0-real-tp4_mns64_mbt16384-expansion-r3p20-20260716-v1-20260716T062906244819Z", + "qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z", + "qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z", + "qwen235b-t0-real-tp4_mns128_mbt8192-expansion-r3p20-20260716-v1-20260716T072121119478Z", + "qwen235b-t0-real-tp4_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T072122501851Z", + "qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z", + "qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r2p40-20260716-v1-20260716T081536141326Z", + "qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T082603837807Z", + "qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z", + "qwen235b-t0-real-tp8_mns64_mbt16384-expansion-r2p40-20260716-v1-20260716T092040851202Z", + "qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z", + "qwen235b-t0-real-tp8_mns128_mbt8192-expansion-r2p40-20260716-v1-20260716T101734683241Z", + "qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z", + "qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r2p40-20260716-v1-20260716T111437770615Z", + "qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T112608807037Z" + ], + "accepted_warnings": [ + { + "run_id": "qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z", + "anchor": "round1/r1p20", + "reason": "One rank-7 TCPStore heartbeat warning occurred 1.1 s after the complete 64-request result was written, while all eight workers were terminating after the runner intentionally stopped the fresh server. The application then shut down normally; no request, engine, OOM, or in-window NCCL failure occurred." + } + ] +} diff --git a/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan-executed.json b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan-executed.json new file mode 100644 index 0000000..cce7ddd --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan-executed.json @@ -0,0 +1,299 @@ +{ + "cells": [ + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "preregistered_rates": [ + 0.1, + 1.6, + 2.4 + ], + "rates": [ + 0.1, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "preregistered_rates": [ + 0.1, + 1.6, + 2.4 + ], + "rates": [ + 0.1, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "preregistered_rates": [ + 0.1, + 1.6, + 2.4 + ], + "rates": [ + 0.1, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "preregistered_rates": [ + 0.1, + 1.6, + 2.4 + ], + "rates": [ + 0.1, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "preregistered_rates": [ + 0.1, + 1.2, + 1.6 + ], + "rates": [ + 0.1, + 1.2, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "preregistered_rates": [ + 0.1, + 1.2, + 1.6 + ], + "rates": [ + 0.1, + 1.2, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "preregistered_rates": [ + 0.1, + 1.2, + 1.6 + ], + "rates": [ + 0.1, + 1.2, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "preregistered_rates": [ + 0.1, + 1.2, + 1.6 + ], + "rates": [ + 0.1, + 1.2, + 1.6, + 2.4, + 3.2 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + } + ], + "execution_protocol_amendment": { + "contract": "fresh server and target-rate warmup for every config-rate-round", + "reason": "observed cross-anchor GPU/kernel/batch warm-state leakage", + "timing": "after excluded multi-rate-per-server diagnostic and before any accepted real surface cell" + }, + "expansion_artifact_roots": [ + "artifacts/t0-real-expansion-r2p40-v1", + "artifacts/t0-real-expansion-r3p20-v1" + ], + "expected_total_h20_gpu_hours": 39.98888888888888, + "frontier_freeze": { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/artifacts/frontier-t0-surface-v1/frontier_surface_frozen.json", + "sha256": "801aa36451c8647f71cc87011144622d2203786e82f189ed1375d964399b106a" + }, + "hard_timeout_hours_per_cell": 2.0, + "post_pilot_sensitivities": [ + "tpot_120ms", + "tpot_150ms", + "tpot_180ms" + ], + "preregistered_plan": { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan.json", + "sha256": "f7acb45d1183a6026dba4bbf9bc2d7578452b815b8de0035ca732512f3d9d47a" + }, + "schema": "qwen235b-t0-real-executed-plan-v1", + "selection_slo": "tpot_150ms", + "selection_timing": "rate anchors frozen after complete simulator surface and before any accepted real surface cell", + "status": "complete", + "strict_preregistered_slo": "tpot_40ms" +} diff --git a/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan-pre-anchor-isolation-amendment.json b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan-pre-anchor-isolation-amendment.json new file mode 100644 index 0000000..9ab031d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan-pre-anchor-isolation-amendment.json @@ -0,0 +1,187 @@ +{ + "cells": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 2.2125, + "expected_wall_seconds": 1991.25, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 2.2125, + "expected_wall_seconds": 1991.25, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 2.2125, + "expected_wall_seconds": 1991.25, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 2.2125, + "expected_wall_seconds": 1991.25, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 4.541666666666667, + "expected_wall_seconds": 2043.75, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 4.541666666666667, + "expected_wall_seconds": 2043.75, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 4.541666666666667, + "expected_wall_seconds": 2043.75, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 4.541666666666667, + "expected_wall_seconds": 2043.75, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2 + } + ], + "expected_total_h20_gpu_hours": 27.016666666666666, + "frontier_freeze": { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/artifacts/frontier-t0-surface-v1/frontier_surface_frozen.json", + "sha256": "801aa36451c8647f71cc87011144622d2203786e82f189ed1375d964399b106a" + }, + "hard_timeout_hours_per_cell": 2.0, + "post_pilot_sensitivities": [ + "tpot_120ms", + "tpot_150ms", + "tpot_180ms" + ], + "schema": "qwen235b-t0-real-plan-v1", + "selection_slo": "tpot_150ms", + "selection_timing": "after_complete_simulator_freeze_before_any_real_surface_cell", + "strict_preregistered_slo": "tpot_40ms" +} diff --git a/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan.json b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan.json new file mode 100644 index 0000000..924555d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/artifacts/t0-real-surface-v1/real-plan.json @@ -0,0 +1,240 @@ +{ + "cells": [ + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 3.284259259259259, + "expected_wall_seconds": 2955.833333333333, + "rates": [ + 0.1, + 1.6, + 2.4 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.60": 32, + "2.40": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + }, + { + "anchor_isolation": "fresh_server_per_rate_per_round", + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + "expected_h20_gpu_hours": 6.712962962962962, + "expected_wall_seconds": 3020.833333333333, + "rates": [ + 0.1, + 1.2, + 1.6 + ], + "requests_per_anchor": 64, + "rounds": 2, + "target_rate_warmup_requests": { + "0.10": 4, + "1.20": 24, + "1.60": 32 + } + } + ], + "execution_protocol_amendment": { + "contract": "fresh server and target-rate warmup for every config-rate-round", + "reason": "observed cross-anchor GPU/kernel/batch warm-state leakage", + "timing": "after excluded multi-rate-per-server diagnostic and before any accepted real surface cell" + }, + "expected_total_h20_gpu_hours": 39.98888888888888, + "frontier_freeze": { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/artifacts/frontier-t0-surface-v1/frontier_surface_frozen.json", + "sha256": "801aa36451c8647f71cc87011144622d2203786e82f189ed1375d964399b106a" + }, + "hard_timeout_hours_per_cell": 2.0, + "post_pilot_sensitivities": [ + "tpot_120ms", + "tpot_150ms", + "tpot_180ms" + ], + "schema": "qwen235b-t0-real-plan-v1", + "selection_slo": "tpot_150ms", + "selection_timing": "rate anchors frozen after complete simulator surface and before any accepted real surface cell", + "strict_preregistered_slo": "tpot_40ms" +} diff --git a/runs/frontier-multicase-sufficiency-v1/assemble_frontier_profile_root.py b/runs/frontier-multicase-sufficiency-v1/assemble_frontier_profile_root.py new file mode 100644 index 0000000..555fd2f --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/assemble_frontier_profile_root.py @@ -0,0 +1,186 @@ +#!/usr/bin/env python3 +"""Create an immutable Frontier profile root with measured decode rows.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import shutil +import tempfile +from pathlib import Path +from typing import Any + + +MODEL = "Qwen3-235B-A22B-FP8" +ATTENTION_RELATIVE_PATH = Path("compute/h20") / MODEL / "attention.csv" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--base-profile-root", type=Path, required=True) + parser.add_argument("--decode-attention-csv", type=Path, required=True) + parser.add_argument("--true-mixed-attention-csv", type=Path) + parser.add_argument("--output-root", type=Path, required=True) + return parser.parse_args() + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def read_csv(path: Path) -> tuple[list[str], list[dict[str, str]]]: + with path.open(newline="") as source: + reader = csv.DictReader(source) + if reader.fieldnames is None: + raise ValueError(f"missing CSV header: {path}") + return reader.fieldnames, list(reader) + + +def is_true(value: str) -> bool: + return value.strip().lower() == "true" + + +def profile_hashes(root: Path) -> dict[str, str]: + return { + str(path.relative_to(root)): sha256(path) + for path in sorted(root.rglob("*")) + if path.is_file() and path.name != "profile_closure_manifest.json" + } + + +def write_json(path: Path, payload: Any) -> None: + path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + + +def main() -> None: + args = parse_args() + base_root = args.base_profile_root.resolve() + decode_csv = args.decode_attention_csv.resolve() + output_root = args.output_root.resolve() + base_attention = base_root / ATTENTION_RELATIVE_PATH + + if output_root.exists(): + raise FileExistsError(f"refusing to overwrite profile root: {output_root}") + for path in (base_attention, decode_csv): + if not path.is_file(): + raise FileNotFoundError(path) + + base_fields, base_rows = read_csv(base_attention) + decode_fields, decode_source_rows = read_csv(decode_csv) + if base_fields != decode_fields: + raise ValueError("base and decode attention CSV schemas differ") + if not base_rows or any(not is_true(row["is_prefill"]) for row in base_rows): + raise ValueError("base attention profile must contain only prefill rows") + decode_rows = [row for row in decode_source_rows if not is_true(row["is_prefill"])] + if not decode_rows: + raise ValueError("decode attention profile contains no decode rows") + if any(not row["time_stats.attn_decode.median"] for row in decode_rows): + raise ValueError("decode attention profile has an empty median") + + dimensions = { + ( + int(row["num_tensor_parallel_workers"]), + int(row["batch_size"]), + int(row["kv_cache_size"]), + row["attention_backend"], + ) + for row in decode_rows + } + if len(dimensions) != len(decode_rows): + raise ValueError("decode attention profile has duplicate coverage coordinates") + if any(dimension[-1] != "FLASHINFER" for dimension in dimensions): + raise ValueError("decode attention profile is not entirely FlashInfer") + + true_mixed_csv = ( + args.true_mixed_attention_csv.resolve() + if args.true_mixed_attention_csv is not None + else None + ) + true_mixed_rows: list[dict[str, str]] = [] + output_fields = list(base_fields) + if true_mixed_csv is not None: + if not true_mixed_csv.is_file(): + raise FileNotFoundError(true_mixed_csv) + true_mixed_fields, true_mixed_rows = read_csv(true_mixed_csv) + required = { + "is_true_mixed_batch", + "decode_batch_size", + "decode_avg_kv_cache_size", + "num_prefill_seqs", + "time_stats.attn_decode.median", + "time_stats.attn_prefill.median", + } + missing = required - set(true_mixed_fields) + if missing: + raise ValueError(f"true-mixed attention CSV lacks columns: {sorted(missing)}") + if not true_mixed_rows or any( + not is_true(row["is_true_mixed_batch"]) for row in true_mixed_rows + ): + raise ValueError("true-mixed attention CSV has non-mixed rows") + if any( + not row["time_stats.attn_decode.median"] + or not row["time_stats.attn_prefill.median"] + for row in true_mixed_rows + ): + raise ValueError("true-mixed attention profile has an empty median") + if { + int(row["num_tensor_parallel_workers"]) for row in true_mixed_rows + } != {4, 8}: + raise ValueError("true-mixed attention profile must cover TP4 and TP8") + output_fields.extend( + field for field in true_mixed_fields if field not in output_fields + ) + + for row in [*base_rows, *decode_rows]: + if "is_true_mixed_batch" in output_fields: + row["is_true_mixed_batch"] = "False" + + output_root.parent.mkdir(parents=True, exist_ok=True) + with tempfile.TemporaryDirectory( + prefix=f".{output_root.name}.", dir=output_root.parent + ) as temporary: + temporary_root = Path(temporary) / output_root.name + shutil.copytree(base_root, temporary_root) + merged_attention = temporary_root / ATTENTION_RELATIVE_PATH + with merged_attention.open("w", newline="") as output: + writer = csv.DictWriter(output, fieldnames=output_fields, lineterminator="\n") + writer.writeheader() + writer.writerows([*base_rows, *decode_rows, *true_mixed_rows]) + + payload = { + "schema": "frontier-profile-closure-v1", + "model": MODEL, + "base_profile_root": str(base_root), + "base_attention_sha256": sha256(base_attention), + "decode_attention_csv": str(decode_csv), + "decode_attention_sha256": sha256(decode_csv), + "base_prefill_rows": len(base_rows), + "decode_source_rows": len(decode_source_rows), + "added_decode_rows": len(decode_rows), + "true_mixed_attention_csv": ( + str(true_mixed_csv) if true_mixed_csv is not None else None + ), + "true_mixed_attention_sha256": ( + sha256(true_mixed_csv) if true_mixed_csv is not None else None + ), + "added_true_mixed_rows": len(true_mixed_rows), + "merged_attention_rows": ( + len(base_rows) + len(decode_rows) + len(true_mixed_rows) + ), + "decode_dimensions": [list(values) for values in sorted(dimensions)], + "output_files_sha256": profile_hashes(temporary_root), + } + write_json(temporary_root / "profile_closure_manifest.json", payload) + temporary_root.rename(output_root) + + print(json.dumps(payload, indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/audit_trace_contract.py b/runs/frontier-multicase-sufficiency-v1/audit_trace_contract.py new file mode 100644 index 0000000..22e6e96 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/audit_trace_contract.py @@ -0,0 +1,183 @@ +#!/usr/bin/env python3 +"""Audit Qwen235B trace token lengths and source prefix-hash identities.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +import platform +import socket +import time +from pathlib import Path +from typing import Any + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--trace", type=Path, required=True) + parser.add_argument("--model", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--max-model-len", type=int, default=40960) + parser.add_argument("--source-block-size", type=int, default=64) + parser.add_argument("--batch-size", type=int, default=16) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + if args.max_model_len <= 0 or args.source_block_size <= 0 or args.batch_size <= 0: + raise ValueError("length and batch-size arguments must be positive") + + import transformers + from transformers import AutoTokenizer + + rows = [json.loads(line) for line in args.trace.open() if line.strip()] + context_exceeded = [ + row + for row in rows + if int(row["input_length"]) + int(row["output_length"]) > args.max_model_len + ] + zero_output = [row for row in rows if int(row["output_length"]) == 0] + eligible = [ + row + for row in rows + if int(row["input_length"]) + int(row["output_length"]) <= args.max_model_len + and int(row["output_length"]) > 0 + ] + + tokenizer = AutoTokenizer.from_pretrained(args.model, trust_remote_code=True) + length_mismatch_count = 0 + hash_count_mismatch_count = 0 + hash_to_key: dict[str, bytes] = {} + key_to_hash: dict[bytes, str] = {} + hash_to_key_conflict_count = 0 + key_to_hash_conflict_count = 0 + total_tokens = 0 + full_blocks = 0 + partial_blocks = 0 + length_digest = hashlib.sha256() + token_digest = hashlib.sha256() + started = time.time() + + for start in range(0, len(eligible), args.batch_size): + batch = eligible[start : start + args.batch_size] + encoded = tokenizer( + [row["prompt"] for row in batch], + add_special_tokens=False, + padding=False, + truncation=False, + )["input_ids"] + for offset, (row, token_ids) in enumerate(zip(batch, encoded, strict=True)): + row_index = start + offset + actual_length = len(token_ids) + expected_length = int(row["input_length"]) + total_tokens += actual_length + length_digest.update(f"{row_index}:{actual_length}\n".encode()) + if actual_length != expected_length: + length_mismatch_count += 1 + + source_hashes = row["hash_ids"] + expected_hashes = math.ceil(actual_length / args.source_block_size) + if len(source_hashes) != expected_hashes: + hash_count_mismatch_count += 1 + continue + + request_token_digest = hashlib.sha256() + parent = b"ROOT" + for block_index, source_hash in enumerate(source_hashes): + begin = block_index * args.source_block_size + chunk = token_ids[begin : begin + args.source_block_size] + token_payload = b"".join( + int(token_id).to_bytes(4, "little", signed=False) + for token_id in chunk + ) + request_token_digest.update(token_payload) + chunk_digest = hashlib.blake2b(token_payload, digest_size=16).digest() + key_digest = hashlib.blake2b( + parent + b"\0" + chunk_digest, digest_size=16 + ).digest() + if len(chunk) == args.source_block_size: + full_blocks += 1 + else: + partial_blocks += 1 + + hash_id = str(source_hash) + previous_key = hash_to_key.setdefault(hash_id, key_digest) + if previous_key != key_digest: + hash_to_key_conflict_count += 1 + previous_hash = key_to_hash.setdefault(key_digest, hash_id) + if previous_hash != hash_id: + key_to_hash_conflict_count += 1 + parent = hash_id.encode() + + token_digest.update(row_index.to_bytes(4, "little")) + token_digest.update(request_token_digest.digest()) + + payload: dict[str, Any] = { + "schema": "qwen235b-trace-contract-audit-v1", + "status": "pass_offline_source_contract" + if not any( + ( + length_mismatch_count, + hash_count_mismatch_count, + hash_to_key_conflict_count, + key_to_hash_conflict_count, + ) + ) + else "fail", + "execution": { + "host": socket.gethostname(), + "device": "cpu_only", + "elapsed_seconds": round(time.time() - started, 3), + "python_version": platform.python_version(), + "tokenizer_class": type(tokenizer).__name__, + "transformers_version": transformers.__version__, + "model_path": str(args.model.resolve()), + }, + "trace": { + "path": str(args.trace.resolve()), + "sha256": sha256_file(args.trace), + "source_request_count": len(rows), + "context_exceeded_count": len(context_exceeded), + "zero_output_count": len(zero_output), + "exclusion_overlap_count": sum(row in zero_output for row in context_exceeded), + "eligible_request_count": len(eligible), + }, + "tokenization": { + "total_token_count": total_tokens, + "input_length_mismatch_count": length_mismatch_count, + "length_order_sha256": length_digest.hexdigest(), + "per_request_token_digest_sha256": token_digest.hexdigest(), + }, + "source_hash_contract": { + "source_block_size_tokens": args.source_block_size, + "hash_count_mismatch_count": hash_count_mismatch_count, + "full_block_count": full_blocks, + "partial_block_count": partial_blocks, + "unique_hash_id_count": len(hash_to_key), + "unique_parent_chunk_key_count": len(key_to_hash), + "hash_id_to_parent_chunk_conflict_count": hash_to_key_conflict_count, + "parent_chunk_to_hash_id_conflict_count": key_to_hash_conflict_count, + "key_definition": ( + "(parent source hash id, BLAKE2b-128 of the tokenizer token-id chunk)" + ), + }, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + if payload["status"] != "pass_offline_source_contract": + raise SystemExit(1) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/fleet.toml b/runs/frontier-multicase-sufficiency-v1/fleet.toml new file mode 100644 index 0000000..2539913 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/fleet.toml @@ -0,0 +1,24 @@ +version = 1 + +[paths] +state_dir = "runs/frontier-multicase-sufficiency-v1/fleet-state" +artifacts_dir = "runs/frontier-multicase-sufficiency-v1/fleet-artifacts" + +[ssh] +connect_timeout_sec = 10 + +[scheduler] +gpu_free_memory_mb = 1024 +gpu_free_utilization_pct = 10 +prefer_pack = true + +[sync] +mode = "scp" +local_path = "runs/frontier-multicase-sufficiency-v1" + +[[hosts]] +name = "dash0" +ssh_alias = "dash0" +enabled = true +sync_remote_path = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1" +fleet_root = "/home/admin/cpfs/wjh/aituner/gpu-fleet-fidelity-v1" diff --git a/runs/frontier-multicase-sufficiency-v1/generate_t0_real_jobs.py b/runs/frontier-multicase-sufficiency-v1/generate_t0_real_jobs.py new file mode 100644 index 0000000..b463daa --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/generate_t0_real_jobs.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python3 +"""Render the fleet queue for a frozen T0 real-surface plan.""" + +from __future__ import annotations + +import argparse +import json +from pathlib import Path + + +def quoted(value: object) -> str: + return json.dumps(str(value)) + + +def main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--plan", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--config", action="append", dest="configs") + parser.add_argument("--rates", nargs="+", type=float) + parser.add_argument("--artifact-root", default="artifacts/t0-real-surface-v1") + parser.add_argument("--name-suffix", default="") + args = parser.parse_args() + plan = json.loads(args.plan.read_text()) + if plan.get("schema") != "qwen235b-t0-real-plan-v1" or len(plan.get("cells") or []) != 8: + raise ValueError("invalid or incomplete T0 real plan") + lattice = {0.10, 0.20, 0.40, 0.80, 1.20, 1.60, 2.40, 3.20} + if args.rates and any(rate not in lattice for rate in args.rates): + raise ValueError("rate override must stay on the frozen T0 lattice") + + indexed_cells = list(enumerate(plan["cells"])) + if args.configs: + requested = set(args.configs) + known = {cell["config"]["name"] for _, cell in indexed_cells} + if not requested <= known: + raise ValueError(f"unknown configs: {sorted(requested - known)}") + indexed_cells = [ + (index, cell) + for index, cell in indexed_cells + if cell["config"]["name"] in requested + ] + + lines = [ + "# Generated from the frozen T0 real plan; do not edit rates in place.", + "version = 1", + "", + ] + for index, cell in indexed_cells: + config = cell["config"] + suffix = f"-{args.name_suffix}" if args.name_suffix else "" + name = f"qwen235b-t0-real-{config['name']}{suffix}-20260716-v1" + artifact = f"{args.artifact_root.rstrip('/')}/{config['name']}" + rates = args.rates or cell["rates"] + lines.extend( + [ + "[[jobs]]", + f"name = {quoted(name)}", + f"gpus = {int(config['tp'])}", + 'gpu_model = "H20"', + 'hosts = ["dash0"]', + 'command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh"', + f"artifacts = [{quoted(artifact)}]", + "", + "[jobs.env]", + f"OUTPUT_ROOT = {quoted('/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/' + artifact)}", + f"TP = {quoted(config['tp'])}", + f"MNS = {quoted(config['mns'])}", + f"MBT = {quoted(config['mbt'])}", + f"RATES = {quoted(' '.join(f'{rate:.2f}' for rate in rates))}", + f"SERVER_PORT = {quoted(18920 + index)}", + 'VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv"', + 'MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8"', + "", + ] + ) + args.output.write_text("\n".join(lines)) + print(args.output) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/jobs.toml b/runs/frontier-multicase-sufficiency-v1/jobs.toml new file mode 100644 index 0000000..7175078 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs.toml @@ -0,0 +1,43 @@ +# Append-only queue for the simulator-fidelity v1 campaign. +version = 1 + +[[jobs]] +name = "qwen235b-t0-tp4-smoke-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 1800 bash run_t0_smoke.sh" +artifacts = ["artifacts/t0-smoke-20260716"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-smoke-20260716" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" +FRONTIER_ROOT = "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6" +SERVER_PORT = "18910" + +[[jobs]] +name = "qwen235b-decode-attention-profile-20260716-v1" +gpus = 1 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=30s 900 bash run_decode_attention_profile.sh" +artifacts = ["artifacts/decode-attention-profile-20260716"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/decode-attention-profile-20260716" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +FRONTIER_ROOT = "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3" + +[[jobs]] +name = "qwen235b-t0-full-attention-profile-20260716-v1" +gpus = 1 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=30s 900 bash run_t0_full_attention_profile.sh" +artifacts = ["artifacts/t0-full-attention-profile-20260716"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-full-attention-profile-20260716" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +FRONTIER_ROOT = "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_decode_profile.toml b/runs/frontier-multicase-sufficiency-v1/jobs_decode_profile.toml new file mode 100644 index 0000000..aae957a --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_decode_profile.toml @@ -0,0 +1,16 @@ +# One-shot queue view for the next authorized profile job. The canonical +# append-only campaign queue remains jobs.toml. +version = 1 + +[[jobs]] +name = "qwen235b-decode-attention-profile-20260716-v1" +gpus = 1 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=30s 900 bash run_decode_attention_profile.sh" +artifacts = ["artifacts/decode-attention-profile-20260716"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/decode-attention-profile-20260716" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +FRONTIER_ROOT = "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_anchor_isolation_smoke.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_anchor_isolation_smoke.toml new file mode 100644 index 0000000..c361715 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_anchor_isolation_smoke.toml @@ -0,0 +1,37 @@ +version = 1 + +[[jobs]] +name = "qwen235b-t0-anchor-isolation-smoke-tp4-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 1800 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-anchor-isolation-smoke-v1/tp4_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-anchor-isolation-smoke-v1/tp4_mns64_mbt8192" +TP = "4" +MNS = "64" +MBT = "8192" +RATES = "1.60" +SERVER_PORT = "18930" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-anchor-isolation-smoke-tp4-mbt16k-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 1800 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-anchor-isolation-smoke-v1/tp4_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-anchor-isolation-smoke-v1/tp4_mns64_mbt16384" +TP = "4" +MNS = "64" +MBT = "16384" +RATES = "1.60" +SERVER_PORT = "18931" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_full_profile.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_full_profile.toml new file mode 100644 index 0000000..e93e33e --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_full_profile.toml @@ -0,0 +1,15 @@ +# One-shot queue view for the T0 full-coverage attention profile. +version = 1 + +[[jobs]] +name = "qwen235b-t0-full-attention-profile-20260716-v1" +gpus = 1 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=30s 900 bash run_t0_full_attention_profile.sh" +artifacts = ["artifacts/t0-full-attention-profile-20260716"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-full-attention-profile-20260716" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +FRONTIER_ROOT = "/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp4_mns128.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp4_mns128.toml new file mode 100644 index 0000000..e48a9bd --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp4_mns128.toml @@ -0,0 +1,38 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns128_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192" +TP = "4" +MNS = "128" +MBT = "8192" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18922" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384" +TP = "4" +MNS = "128" +MBT = "16384" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18923" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp4_mns64.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp4_mns64.toml new file mode 100644 index 0000000..0afa514 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp4_mns64.toml @@ -0,0 +1,38 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192" +TP = "4" +MNS = "64" +MBT = "8192" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18920" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384" +TP = "4" +MNS = "64" +MBT = "16384" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18921" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns128_mbt16384.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns128_mbt16384.toml new file mode 100644 index 0000000..7965b2e --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns128_mbt16384.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384" +TP = "8" +MNS = "128" +MBT = "16384" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18927" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns128_mbt8192.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns128_mbt8192.toml new file mode 100644 index 0000000..014a918 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns128_mbt8192.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns128_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192" +TP = "8" +MNS = "128" +MBT = "8192" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18926" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns64_mbt16384.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns64_mbt16384.toml new file mode 100644 index 0000000..2eca2c0 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns64_mbt16384.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384" +TP = "8" +MNS = "64" +MBT = "16384" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18925" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns64_mbt8192.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns64_mbt8192.toml new file mode 100644 index 0000000..c1f64bf --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_batch_tp8_mns64_mbt8192.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192" +TP = "8" +MNS = "64" +MBT = "8192" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18924" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp4_mns128_r3p20.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp4_mns128_r3p20.toml new file mode 100644 index 0000000..a7f887d --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp4_mns128_r3p20.toml @@ -0,0 +1,38 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns128_mbt8192-expansion-r3p20-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt8192" +TP = "4" +MNS = "128" +MBT = "8192" +RATES = "3.20" +SERVER_PORT = "18922" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns128_mbt16384-expansion-r3p20-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt16384" +TP = "4" +MNS = "128" +MBT = "16384" +RATES = "3.20" +SERVER_PORT = "18923" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp4_mns64_r3p20.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp4_mns64_r3p20.toml new file mode 100644 index 0000000..38d76eb --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp4_mns64_r3p20.toml @@ -0,0 +1,38 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns64_mbt8192-expansion-r3p20-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt8192" +TP = "4" +MNS = "64" +MBT = "8192" +RATES = "3.20" +SERVER_PORT = "18920" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns64_mbt16384-expansion-r3p20-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt16384" +TP = "4" +MNS = "64" +MBT = "16384" +RATES = "3.20" +SERVER_PORT = "18921" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt16384_r2p40.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt16384_r2p40.toml new file mode 100644 index 0000000..eeec4ef --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt16384_r2p40.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r2p40-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt16384" +TP = "8" +MNS = "128" +MBT = "16384" +RATES = "2.40" +SERVER_PORT = "18927" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt16384_r3p20.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt16384_r3p20.toml new file mode 100644 index 0000000..b1774ee --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt16384_r3p20.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r3p20-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r3p20-v1/tp8_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r3p20-v1/tp8_mns128_mbt16384" +TP = "8" +MNS = "128" +MBT = "16384" +RATES = "3.20" +SERVER_PORT = "18927" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt8192_r2p40.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt8192_r2p40.toml new file mode 100644 index 0000000..b419760 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns128_mbt8192_r2p40.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt8192-expansion-r2p40-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt8192" +TP = "8" +MNS = "128" +MBT = "8192" +RATES = "2.40" +SERVER_PORT = "18926" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt16384_r2p40.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt16384_r2p40.toml new file mode 100644 index 0000000..4ecd869 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt16384_r2p40.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt16384-expansion-r2p40-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt16384" +TP = "8" +MNS = "64" +MBT = "16384" +RATES = "2.40" +SERVER_PORT = "18925" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt8192_r2p40.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt8192_r2p40.toml new file mode 100644 index 0000000..cf3ea94 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt8192_r2p40.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r2p40-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt8192" +TP = "8" +MNS = "64" +MBT = "8192" +RATES = "2.40" +SERVER_PORT = "18924" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt8192_r3p20.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt8192_r3p20.toml new file mode 100644 index 0000000..8ba63c0 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_expansion_tp8_mns64_mbt8192_r3p20.toml @@ -0,0 +1,20 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r3p20-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-expansion-r3p20-v1/tp8_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-expansion-r3p20-v1/tp8_mns64_mbt8192" +TP = "8" +MNS = "64" +MBT = "8192" +RATES = "3.20" +SERVER_PORT = "18924" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_surface.toml b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_surface.toml new file mode 100644 index 0000000..91880b6 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/jobs_t0_real_surface.toml @@ -0,0 +1,146 @@ +# Generated from the frozen T0 real plan; do not edit rates in place. +version = 1 + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192" +TP = "4" +MNS = "64" +MBT = "8192" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18920" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384" +TP = "4" +MNS = "64" +MBT = "16384" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18921" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns128_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192" +TP = "4" +MNS = "128" +MBT = "8192" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18922" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1" +gpus = 4 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp4_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384" +TP = "4" +MNS = "128" +MBT = "16384" +RATES = "0.10 1.60 2.40" +SERVER_PORT = "18923" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns64_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192" +TP = "8" +MNS = "64" +MBT = "8192" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18924" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns64_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384" +TP = "8" +MNS = "64" +MBT = "16384" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18925" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns128_mbt8192"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192" +TP = "8" +MNS = "128" +MBT = "8192" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18926" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + +[[jobs]] +name = "qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1" +gpus = 8 +gpu_model = "H20" +hosts = ["dash0"] +command = "timeout --signal=TERM --kill-after=60s 7200 bash run_t0_real_config.sh" +artifacts = ["artifacts/t0-real-surface-v1/tp8_mns128_mbt16384"] + +[jobs.env] +OUTPUT_ROOT = "/home/admin/cpfs/wjh/aituner/aituner-fidelity-v1/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384" +TP = "8" +MNS = "128" +MBT = "16384" +RATES = "0.10 1.20 1.60" +SERVER_PORT = "18927" +VENV_ROOT = "/tmp/wjh-frontier-vllm0102-smoke/.venv" +MODEL_ROOT = "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" diff --git a/runs/frontier-multicase-sufficiency-v1/prepare_t0_real_plan.py b/runs/frontier-multicase-sufficiency-v1/prepare_t0_real_plan.py new file mode 100644 index 0000000..09b3f57 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/prepare_t0_real_plan.py @@ -0,0 +1,128 @@ +#!/usr/bin/env python3 +"""Select predeclared simulator-lattice anchors for blind real confirmation.""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import math +from pathlib import Path +from typing import Any + + +RATES = (0.10, 0.20, 0.40, 0.80, 1.20, 1.60, 2.40, 3.20) +CONFIG_NAMES = { + f"tp{tp}_mns{mns}_mbt{mbt}" + for tp in (4, 8) + for mns in (64, 128) + for mbt in (8192, 16384) +} + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--frontier-freeze", type=Path, required=True) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--selection-slo", default="tpot_150ms") + return parser.parse_args() + + +def sha256(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + +def select_rates(loads: list[dict[str, Any]], slo: str) -> list[float]: + ordered = sorted(loads, key=lambda item: float(item["offered_request_rate"])) + rates = [float(item["offered_request_rate"]) for item in ordered] + feasible = [bool(item["slos"][slo]["feasible"]) for item in ordered] + selected = {rates[0]} + for index in range(len(rates) - 1): + if feasible[index] != feasible[index + 1]: + selected.update((rates[index], rates[index + 1])) + if len(selected) == 1: + if all(feasible): + selected.update(rates[-2:]) + elif not any(feasible): + selected.update(rates[:2]) + return sorted(selected) + + +def warmup_requests(rate: float) -> int: + return min(32, max(4, math.ceil(rate * 20.0))) + + +def main() -> None: + args = parse_args() + freeze_path = args.frontier_freeze.resolve() + freeze = json.loads(freeze_path.read_text()) + if freeze.get("schema") != "frontier-qwen235b-t0-surface-v1": + raise ValueError("unexpected Frontier freeze schema") + if freeze.get("status") != "frozen_before_real_surface": + raise ValueError("Frontier surface is not frozen") + results = freeze.get("config_results") or [] + if len(results) != 8 or any(len(item.get("loads") or []) != 8 for item in results): + raise ValueError("Frontier surface is incomplete") + names = {item.get("config", {}).get("name") for item in results} + if names != CONFIG_NAMES: + raise ValueError(f"Frontier config set mismatch: {names}") + for item in results: + rates = tuple(sorted(float(load["offered_request_rate"]) for load in item["loads"])) + if rates != RATES: + raise ValueError(f"Frontier rate lattice mismatch for {item['config']['name']}: {rates}") + + cells = [] + total_expected_seconds = 0.0 + for item in results: + config = item["config"] + rates = select_rates(item["loads"], args.selection_slo) + # Every anchor gets an independent server in both rounds. The estimate + # includes server startup, target-rate warmup and conservative drain + # allowances for both the discarded and measured request streams. + expected_seconds = 2 * sum( + 120.0 + + (warmup_requests(rate) - 1) / rate + + 60.0 + + 63.0 / rate + + 60.0 + for rate in rates + ) + total_expected_seconds += expected_seconds * int(config["tp"]) + cells.append( + { + "config": config, + "rates": rates, + "rounds": 2, + "requests_per_anchor": 64, + "anchor_isolation": "fresh_server_per_rate_per_round", + "target_rate_warmup_requests": { + f"{rate:.2f}": warmup_requests(rate) for rate in rates + }, + "expected_wall_seconds": expected_seconds, + "expected_h20_gpu_hours": expected_seconds * int(config["tp"]) / 3600.0, + "expansion_rule": "if real labels do not bracket a transition, expand to the next frozen lattice anchor and repeat both directions", + } + ) + payload = { + "schema": "qwen235b-t0-real-plan-v1", + "frontier_freeze": {"path": str(freeze_path), "sha256": sha256(freeze_path)}, + "selection_slo": args.selection_slo, + "selection_timing": "rate anchors frozen after complete simulator surface and before any accepted real surface cell", + "execution_protocol_amendment": { + "timing": "after excluded multi-rate-per-server diagnostic and before any accepted real surface cell", + "reason": "observed cross-anchor GPU/kernel/batch warm-state leakage", + "contract": "fresh server and target-rate warmup for every config-rate-round", + }, + "strict_preregistered_slo": "tpot_40ms", + "post_pilot_sensitivities": ["tpot_120ms", "tpot_150ms", "tpot_180ms"], + "cells": cells, + "expected_total_h20_gpu_hours": total_expected_seconds / 3600.0, + "hard_timeout_hours_per_cell": 2.0, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + print(json.dumps({"cells": len(cells), "expected_total_h20_gpu_hours": payload["expected_total_h20_gpu_hours"]}, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/protocol.md b/runs/frontier-multicase-sufficiency-v1/protocol.md new file mode 100644 index 0000000..7e84a80 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/protocol.md @@ -0,0 +1,123 @@ +# Qwen235B simulator fidelity 预注册协议 v1 + +状态：`IN PROGRESS`。更新日期：2026-07-16。T0 的 simulator 与 real 8-config surface 已完成；T1 和 T2 尚未运行。pilot 单独记录在 [t0-smoke-report.md](t0-smoke-report.md)，最终 T0 rank evaluation 记录在 [comparison.json](results/t0-final/comparison.json) 和根目录 [simulator-fidelity.md](../../simulator-fidelity.md)。 + +## Research question 与成功标准 + +对同一 workload、SLO 和候选 config surface，Frontier 是否能找到真机上低 regret 的 config，而不要求绝对 latency 或 capacity 完全一致？ + +主目标统一为： + +```text +capacity(c) = max { 被真实测试的 offered req/s | 至少 95% requests 满足全部 SLO } +score(c) = capacity(c) / config 实际占用的 GPU 数 +``` + +每个 case 都报告 real/sim capacity、top set、worst tie-break regret、Kendall tau-b、informative-pair direction、anchor-level SLO confusion，以及为达到该 fidelity 使用的 profile、patch 和真机 calibration 成本。只有同时满足下列条件，才能说 Frontier 对该 compatibility envelope 是足够的 config ranker： + +- worst selected-config regret 不超过 5%； +- tie-aware rank correlation 不低于 0.8，且有足够的 non-tied pairs； +- ground-truth capacity bracket 不足以反转最优决策； +- 不使用被评测 workload/config 的端到端测量做 per-action calibration。 + +## 冻结的平台与软件边界 + +| 项目 | 设置 | +|---|---| +| machine | 仅 `dash0`，8×NVIDIA H20；不得调度到 `dash1` | +| remote repo | `/home/admin/cpfs/wjh/aituner/aituner`；实验使用独立 clean worktree/clone，不修改当前 dirty checkout | +| model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8` | +| serving | community vLLM `0.10.2` isolated environment，eager，BF16 KV；包版本与二进制 hash 入 manifest | +| simulator | Frontier upstream commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e` + 明示、hash 后的 FP8/MoE/trace patches | +| trace | `thinking_w20260327_1000.jsonl`，SHA256 `f878e9af18f94dcfaced94a8e1e6b20a2f7d97d64aa862448025660dbbd965b2` | +| source window | 600 s，15,479 requests，natural offered rate 25.798 req/s | + +每个 real trial 使用 fresh server。第一轮 config 顺序由固定 seed 随机化，第二轮反序；同一 config 的 offered-load anchors 从低到高和从高到低各跑一次。任何 OOM、engine crash、timeout 和右删失 cell 都作为结果保留，不能静默丢弃。 + +## Trace fidelity contract + +source row 同时包含 exact prompt、arrival timestamp、`input_length`、`output_length`、session/turn/parent、`sampling_u`，以及 block-size=64 的 `hash_ids`。主 trace case 不设置 output override，也不按 input/output length 选择“好跑”的 cohort。 + +community model 的最大 context 是 40,960 tokens。预先冻结的 eligible universe 如下： + +| universe | requests | input mean/p50/p95 | output mean/p50/p95 | total mean/p50/p95 | +|---|---:|---:|---:|---:| +| source | 15,479 | 3,660.0 / 1,491 / 19,610.6 | 3,924.6 / 3,435 / 8,945.1 | 7,584.6 / 5,317 / 25,102.1 | +| eligible | 15,401 | 3,575.0 / 1,490 / 18,887 | 3,823.3 / 3,417 / 8,768 | 7,398.2 / 5,305 / 23,697 | + +唯一排除项是 72 个 `input_length + output_length > 40960` rows，以及 6 个 `output_length = 0` rows；二者无重叠。eligible 比例为 99.50%。 + +capacity search 使用 `sampling_u <= u` 调整 offered rate。`sampling_u` 在同一 session 内共享，因此这是与长度无关的 session-coherent thinning，而不是 workload filtering。对每个 anchor，real 与 sim 必须具有相同的 source row IDs、arrival/order hash、input/output vector hash、session vector hash 和 prefix vector hash。每个被选 request 仍使用原始 arrival、prompt、input、output 和 hash。 + +第一个 correctness gate 已完成：在 `dash0` 上使用 community model 的 `Qwen2TokenizerFast` 全量处理 15,401 个 prompts、55,057,919 tokens，实际 token length 与 trace `input_length` 的 mismatch 为 0；在 852,407 个完整和 15,131 个末尾 partial source blocks 上，`hash_id ↔ (parent_hash_id, 64-token chunk)` 均无冲突。结果和 digest 记录在 [trace-contract-audit.json](trace-contract-audit.json)。这说明 exact prompt/source hash 可以表达原 trace 的 prefix-equivalence relation。 + +在启动 GPU 前还需完成第二个 correctness gate： + +1. real 与 Frontier 的 KV block size 都冻结为 16。source `hash_ids` 的 block size 是 64，因此应从 exact prompt token IDs 生成 block-16 content/parent identities，并验证每四个完整 block-16 对应的 64-token source equality/reuse relation 没有改变。两侧逐 request 比较 computed/hit/allocated block counts。 + +## T0：fixed-shape sanity case + +T0 是最简单的机制隔离基线，不声称代表 production trace。 + +| 项目 | 设置 | +|---|---| +| requests | deterministic 生成；ISL=2,048、OSL=128 | +| arrivals | uniform QPS；相同 request IDs/order；每个 anchor 至少 64 个 completed requests | +| prefix | off；不同 token content，保证没有共享完整 KV blocks | +| SLO | TTFT `<= 1000 ms + 1000×input_tokens/8000`（本 case 为 1,256 ms），TPOT `<= 40 ms`，joint pass rate ≥0.95 | +| configs | `TP∈{4,8} × MNS∈{64,128} × MBT∈{8192,16384}`；沿用已闭合的 TP4/TP8 MoE mapping | + +先在 simulator 中用宽 anchor lattice 找到每个 config 的边界，再冻结 simulator output 和 hash；真机只运行相同 anchors。T0 回答的是：在没有 length variance、prefix reuse 和 burstiness 时，profile composition 与 scheduler 是否已经能保持 config rank。它不能用于回答真实 trace fidelity。 + +首次真机 smoke 之后、完整 surface 之前预先记录如下 amendment：TP4 的 zero-queue TPOT 已经是 136.2 ms，因此 40 ms 主 SLO 对至少 TP4 没有非零 capacity。40 ms 仍保留为 preregistered primary，不因结果不可行而删除；同时固定报告 TPOT 120/150/180 ms sensitivity。150 ms 是 post-pilot 的 decision-bearing sensitivity，因为它略高于已观测 TP4 idle floor，又仍可能在 batch/queue 增大时产生 capacity knee；它不能冒充 blind primary result。 + +完整 simulator lattice 固定为 system offered rate `{0.10,0.20,0.40,0.80,1.20,1.60,2.40,3.20}` req/s，每个点 64 requests。所有 8×8 simulator cells 必须在任何 real surface cell 之前冻结。真机对每个 config 先测 simulator 在 150 ms sensitivity 下的相邻 transition anchors，并始终加入最低 0.10 req/s anchor；若没有 transition，则测边界方向的两个 anchors。若这些 real labels 没有 bracket，按冻结 lattice 向外扩展，且 simulator 结果不得重算。每个被测 anchor 做两个 fresh-server rounds，rate 顺序正向/反向，所有 strict/sensitivity SLO 均在同一 raw requests 上离线打分。真机 anchor 只有在两个 round 都达到 95% joint pass rate 时才保守地记为 feasible；两个 round 标签不一致时同时报告原始 pass rates，并把该点记为重复性不确定而不是择优取样。 + +第一次多-rate 真机 run 后、接受任何 real surface cell 之前增加一项 execution amendment。原 runner 每个 round 只启动一次 server，正向 `0.10→1.60→2.40` 时 1.60 的前 24 个 requests 出现 114--234 ms 的递减 TPOT transient；反向先运行 2.40 后，同一 1.60 anchor 从第一个 request 起稳定在 113--128 ms。两个独立 TP4 config 同时复现，说明前一 rate 的 GPU/kernel/batch warm state 泄漏到后一 rate，而 Frontier 的各 rate 是独立 simulator run。该诊断 run 整体排除，不作为 ground truth。修正后的每个 `(config, rate, round)` 使用独立 fresh server，并在测量前按同一 target rate 丢弃 `min(32, max(4, ceil(rate×20)))` 个 exact-shape warmup requests；随后从空 scheduler queue 开始记录 64 requests。正/逆序只用于复测和控制机器时间漂移，不再共享 execution state。这个 amendment 是观察 diagnostic transient 后作出的，必须在结果中明示，不能写成原始 blind protocol。 + +steady-QPS profile closure 不能只含 pure decode。完整 profile 预先覆盖 TP `{4,8}`、decode batch `{1,2,4,8,16,32,64,96,128}`、KV 2,048--2,175，并加入 prefill-chunk=2,048 的 true mixed prefill+decode rows。最终 attention root 含 726 个既有 prefill rows、162 个 standard decode rows 和 216 个 true-mixed rows；没有使用 T0 端到端 latency 做 calibration。 + +## T1：trace-faithful mixed case + +T1 直接使用上述 15,401-row eligible universe：不设置 completion override，不改变入选 request 的 input/output，不做 length-stratified sampling，prefix caching 在 real 与 sim 两侧同时打开。 + +| 项目 | 设置 | +|---|---| +| selection | source `sampling_u <= u`；同 session 一起入选；real/sim 共用 frozen anchor files | +| arrival | 原 timestamp、600 s window、原 burst/order；`replay_time_scale=1.0` | +| output | `min_tokens=max_tokens=output_length`；记录实际 usage，必须与 trace 相等 | +| prefix | exact prompt token blocks；两侧相同 block size、capacity、cache policy 与 sticky session routing | +| SLO | 与 T0 相同的 TTFT 规则和 TPOT 40 ms；另报告 TPOT 20/50 ms sensitivity，不用 sensitivity 改选主结论 | +| configs | 与 T0 相同的 8 cells，先隔离 workload state，再扩展 DP/EP topology | + +由于 natural QPS 远高于该模型 capacity，直接全量同时到达只会让所有 config 都 infeasible，不能产生排序信息。这里 `sampling_u` 是 trace 原生的负载抽样维度；它只改变入选 session 数量，不改变入选请求的 joint distribution 和字段。每个边界至少用两个预先冻结的 session-hash folds 或两个独立 trace windows 重复，避免一个低-u 小样本偶然决定 ranking；若暂时只有一个 window，结论标为 single-window evidence。 + +## T2：Qwen235B strict decode-only case + +T2 不用“短 prefill + 长 output”冒充 decode-only。它要求一个可检查的 initial-KV contract：request 到达 scheduler 时，与 input token IDs 对应的 KV blocks 已经 resident，request 状态为 prefill complete，首次被调度的 token 是第一个 decode token。 + +两侧 contract 必须共同记录：initial KV token/block count、block identities、replica placement、填充/传输开始结束时间、首次 scheduler admission time。若真机 connector 在 GPU 上同步填充 dummy KV，这项带宽和同步成本会干扰并发 decode；Frontier 必须显式建模这段 admission cost，或真机把它移到计时区间之前。不能只忽略 TTFT，就假设 connector 没有影响 TPOT。 + +T2 分两步，避免把 execution mechanisms 混成一个不可解释的 gap： + +| 层次 | Workload / mechanisms | Config surface | 目的 | +|---|---|---|---| +| T2a controlled decode | fixed ISL=2,048、OSL=512；BF16 KV；eager；prefix/speculation/CUDA graph off | `TP∈{4,8} × MNS∈{64,128} × MBT∈{256,384}`，DP=1 | 验证 initial-KV、decode attention/batching 与 TP rank | +| T2b topology stress | trace exact input/output；EP8；仍先关闭 EAGLE3 与 decode graph | `(TP4,DP2,EP8)` 与 `(TP2,DP4,EP8)` × MNS `{64,128}` × MBT `{256,384}` | 激活 replica count、expert communication、KV residency 的耦合 | + +只有 T2a/T2b 对齐后，才按单变量顺序加入 FP8 KV、EAGLE3、DeepEP/NVSHMEM 和 `FULL_DECODE_ONLY` CUDA graph；每加入一项都重新冻结 simulator ranking 并测真机。现有 internal-vLLM decode surface 同时打开了这些机制，而且 8 个 config 的 capacity brackets 全部可能包含最优值，只作为 historical real-only evidence，不作为本协议 ground truth。 + +## Run order 与停止条件 + +1. 完成 tokenizer、row vector、prefix-block correctness audit；不使用 GPU。tokenizer/source-hash 离线部分已完成，block-16 runtime-counter parity 待完成。 +2. 在 dash0 做一次 community vLLM TP4 model-load + T0 one-request smoke。已完成：server 与 exact 2,048/128-token requests 成功，但最低负载 TPOT 为 136.2 ms，40 ms primary SLO 不可行；对应 Frontier run 暴露 decode attention profile 未闭合。 +3. 完成 T0 的 simulator 8-cell surface并冻结 SHA256；随后运行 T0 real anchors。已完成：64/64 simulator cells 冻结，8/8 real config boundaries 在 150 ms decision-bearing SLO 下闭合。 +4. 实现并单测 trace block translation，完成 T1 simulator freeze，再运行 T1 real anchors。 +5. 实现 community-vLLM 0.10.2 与 Frontier 的 initial-KV contract；一请求和并发请求状态机 smoke 通过后，运行 T2a。 +6. 只有 profile closure 包含 EP8 all-to-all 且 T2a 可解释时，运行 T2b。 + +每层遇到 mismatch，先按 request/stage 分解 queue wait、prefill、decode step、collective、MoE 和 KV admission residual。只有证据定位到某一 composition invariant 失效，才修改 simulator；不通过端到端 scalar 把 ranking 调到正确。 + +## 首次 GPU launch gate + +首次 GPU smoke、full attention closure 与完整 T0 sweep 均已于 2026-07-16 在 `dash0` 完成，GPU 已释放。接受的 T0 ground truth 为 68 个 fresh-server anchors，消耗 36.26 H20-GPU-hours；污染和 cross-anchor-state diagnostic attempts 已隔离。任何 T1/T2 GPU launch 都必须重新 echo resolved workload、artifact paths、GPU 预算与预计时长。 diff --git a/runs/frontier-multicase-sufficiency-v1/results/t0-final/capacity.csv b/runs/frontier-multicase-sufficiency-v1/results/t0-final/capacity.csv new file mode 100644 index 0000000..50f53c5 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/results/t0-final/capacity.csv @@ -0,0 +1,33 @@ +slo,config,tp,mns,mbt,real_capacity_per_gpu,sim_capacity_per_gpu,real_boundary_status,expansion_required +tpot_40ms,tp4_mns64_mbt8192,4,64,8192,,,lowest_anchor_infeasible,False +tpot_40ms,tp4_mns64_mbt16384,4,64,16384,,,lowest_anchor_infeasible,False +tpot_40ms,tp4_mns128_mbt8192,4,128,8192,,,lowest_anchor_infeasible,False +tpot_40ms,tp4_mns128_mbt16384,4,128,16384,,,lowest_anchor_infeasible,False +tpot_40ms,tp8_mns64_mbt8192,8,64,8192,,,lowest_anchor_infeasible,False +tpot_40ms,tp8_mns64_mbt16384,8,64,16384,,,lowest_anchor_infeasible,False +tpot_40ms,tp8_mns128_mbt8192,8,128,8192,,,lowest_anchor_infeasible,False +tpot_40ms,tp8_mns128_mbt16384,8,128,16384,,,lowest_anchor_infeasible,False +tpot_120ms,tp4_mns64_mbt8192,4,64,8192,0.025,0.2,unbracketed_requires_expansion,True +tpot_120ms,tp4_mns64_mbt16384,4,64,16384,0.025,0.2,unbracketed_requires_expansion,True +tpot_120ms,tp4_mns128_mbt8192,4,128,8192,0.025,0.2,unbracketed_requires_expansion,True +tpot_120ms,tp4_mns128_mbt16384,4,128,16384,0.025,0.2,unbracketed_requires_expansion,True +tpot_120ms,tp8_mns64_mbt8192,8,64,8192,0.0125,0.05,unbracketed_requires_expansion,True +tpot_120ms,tp8_mns64_mbt16384,8,64,16384,0.0125,0.05,unbracketed_requires_expansion,True +tpot_120ms,tp8_mns128_mbt8192,8,128,8192,0.0125,0.05,unbracketed_requires_expansion,True +tpot_120ms,tp8_mns128_mbt16384,8,128,16384,0.0125,0.05,unbracketed_requires_expansion,True +tpot_150ms,tp4_mns64_mbt8192,4,64,8192,0.6,0.4,adjacent_transition_bracketed,False +tpot_150ms,tp4_mns64_mbt16384,4,64,16384,0.6,0.4,adjacent_transition_bracketed,False +tpot_150ms,tp4_mns128_mbt8192,4,128,8192,0.6,0.4,adjacent_transition_bracketed,False +tpot_150ms,tp4_mns128_mbt16384,4,128,16384,0.6,0.4,adjacent_transition_bracketed,False +tpot_150ms,tp8_mns64_mbt8192,8,64,8192,0.3,0.15,adjacent_transition_bracketed,False +tpot_150ms,tp8_mns64_mbt16384,8,64,16384,0.2,0.15,adjacent_transition_bracketed,False +tpot_150ms,tp8_mns128_mbt8192,8,128,8192,0.2,0.15,adjacent_transition_bracketed,False +tpot_150ms,tp8_mns128_mbt16384,8,128,16384,0.3,0.15,adjacent_transition_bracketed,False +tpot_180ms,tp4_mns64_mbt8192,4,64,8192,0.8,0.6,upper_lattice_reached,False +tpot_180ms,tp4_mns64_mbt16384,4,64,16384,0.8,0.6,upper_lattice_reached,False +tpot_180ms,tp4_mns128_mbt8192,4,128,8192,0.8,0.6,upper_lattice_reached,False +tpot_180ms,tp4_mns128_mbt16384,4,128,16384,0.8,0.6,upper_lattice_reached,False +tpot_180ms,tp8_mns64_mbt8192,8,64,8192,0.4,0.2,upper_lattice_reached,False +tpot_180ms,tp8_mns64_mbt16384,8,64,16384,0.3,0.2,unbracketed_requires_expansion,True +tpot_180ms,tp8_mns128_mbt8192,8,128,8192,0.3,0.2,unbracketed_requires_expansion,True +tpot_180ms,tp8_mns128_mbt16384,8,128,16384,0.4,0.2,upper_lattice_reached,False diff --git a/runs/frontier-multicase-sufficiency-v1/results/t0-final/comparison.json b/runs/frontier-multicase-sufficiency-v1/results/t0-final/comparison.json new file mode 100644 index 0000000..608a2ae --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/results/t0-final/comparison.json @@ -0,0 +1,3176 @@ +{ + "cells": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192/round1/results/r0p10.json", + "sha256": "70e6f6320f7652a4866a971e64d8cb60c3b2be21627da4853f1ee6a76fe3e2d8" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192/round2/results/r0p10.json", + "sha256": "b3aa9826f6c83272a24fd510a74518cdd7870c656e880fb75e56f1365652cf96" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.25, + 0.265625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192/round1/results/r1p60.json", + "sha256": "442eba957f112eee10971040e3ba8d602f7b03287989e205165c9c23ce09f291" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192/round2/results/r1p60.json", + "sha256": "3e86ea9f8d298d966944349fda1d18db2545a0e3571865a4d44f1334e0b90fbb" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.1875, + 0.046875 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192/round1/results/r2p40.json", + "sha256": "aa749f0d9d8876c75c631d21b0f6b052db27391e4fdc913afddfe015f42c7fe6" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt8192/round2/results/r2p40.json", + "sha256": "d03f64e8a73cddfcfcbca43a16fbb647101343c5c757c774d7cc68ddb78f9078" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 3.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.140625, + 0.203125 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.890625, + 0.921875 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T062904775741Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt8192/round1/results/r3p20.json", + "sha256": "d20854a4bc6c460d1c63ec3bcbcaa404b1512fdcd80a373d7c34b1d124d60104" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T062904775741Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt8192/round2/results/r3p20.json", + "sha256": "6807859835655dd31cac26b4e67a8582f40b4f2f8bf16152b2718cdf881922e8" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384/round1/results/r0p10.json", + "sha256": "c00eecbff86947b50e7aceaca0fad7cc3b08cda67299ae62c2c79d3bca854a7b" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384/round2/results/r0p10.json", + "sha256": "4e935891030a4f83edf859085ea80d0f9cafb66612cea21305b786835d16180d" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.234375, + 0.203125 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384/round1/results/r1p60.json", + "sha256": "a30ecfba9fd05f84cf66cefcd200210f29cc02695f93ada537788b15d98200ba" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384/round2/results/r1p60.json", + "sha256": "56eb4c4f112eda060d6dd31390d625ef132043798f751f90c4d176d0f5e3d707" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.21875, + 0.078125 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384/round1/results/r2p40.json", + "sha256": "a161abc47c3f44a124859fe56bc48a212e1c405df3ffbca08bb1f26c33db5af5" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns64_mbt16384/round2/results/r2p40.json", + "sha256": "51dd5c65eb3067d82626050fb838e697c14adaefec16514f43754a9529364ab2" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 3.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.203125, + 0.1875 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.96875, + 0.890625 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-expansion-r3p20-20260716-v1-20260716T062906244819Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt16384/round1/results/r3p20.json", + "sha256": "1c6de1cdfcf5768ef85a8a76de046884c592964fec6bc591a6889a2a9ecbd567" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns64_mbt16384-expansion-r3p20-20260716-v1-20260716T062906244819Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns64_mbt16384/round2/results/r3p20.json", + "sha256": "dc68d3d18fbb6c4a2aaf4be2eaf50b22270cbd6ce05308f8fe265b73e1b54b21" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192/round1/results/r0p10.json", + "sha256": "abd74b9bd505d92b29973ddd80fdadf89534bbd6deb55d975668cacb4adcf648" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192/round2/results/r0p10.json", + "sha256": "92bfac8b35371124b7402d5ec99f271f7426b1a56ccbad37d81378db41a00e02" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.28125, + 0.234375 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192/round1/results/r1p60.json", + "sha256": "5a32ec8b0173a908d7cd9da76ea2455b973ce843fd14d55c7431b5c4eb4ce64c" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192/round2/results/r1p60.json", + "sha256": "fe07a391a0e959189926baf70028a028a1ba4f7888d048d2f6d4d31028e1e2da" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.171875, + 0.15625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192/round1/results/r2p40.json", + "sha256": "cd83d672696138f7190bbff86bcc1ac5bf4d74d8286be00497d6ba95dba31ea0" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt8192/round2/results/r2p40.json", + "sha256": "319601805cc2c8ae4d992a79bb796134e907b0ab3fc52f300d0b6c38b4b279dd" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 3.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.140625, + 0.15625 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.828125, + 0.859375 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-expansion-r3p20-20260716-v1-20260716T072121119478Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt8192/round1/results/r3p20.json", + "sha256": "ea9f52c5adaefa7a55805e546d666fb3f93f7fa53b339d3a92314f8e66a634d0" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt8192-expansion-r3p20-20260716-v1-20260716T072121119478Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt8192/round2/results/r3p20.json", + "sha256": "4e6ec7b56d6a16d83a6082b49f00ea25a7086d2bad00c677f9ed567df7e4aaed" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384/round1/results/r0p10.json", + "sha256": "b90f4d0e04452c8eb3e0a3eea7ffce237ed6b9739cac253c14d1d7242f76a6ea" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384/round2/results/r0p10.json", + "sha256": "d6d9759bb93bbab24a24dcbfe7d9c26dde16266b4a755fbae4332606940ebe80" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.234375, + 0.25 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384/round1/results/r1p60.json", + "sha256": "93f797d5a5abbd8136380ef52e05089c7b54fe99e263f3431894b741b15b5f37" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384/round2/results/r1p60.json", + "sha256": "561bff3e5376fa5901f463409df72fa0a548cb93d8e4db01d790e9c25361321d" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.171875, + 0.1875 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384/round1/results/r2p40.json", + "sha256": "b5b50f24898bfbfd8661b9b8b80b9ae0af1c0821de957c43c748994c6ba3c9d2" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z/artifacts/artifacts/t0-real-surface-v1/tp4_mns128_mbt16384/round2/results/r2p40.json", + "sha256": "193026996ccb07ba2b0bcb696530271ab694774c6082df6f8748e369e7275c75" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.203125, + "passed": 13 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 3.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.109375, + 0.140625 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.8125, + 0.84375 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T072122501851Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt16384/round1/results/r3p20.json", + "sha256": "b8b4b470c00d28a78c545852ca8ffd3ed333ff15abe3ddcc822e1922d07857f7" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp4_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T072122501851Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp4_mns128_mbt16384/round2/results/r3p20.json", + "sha256": "18aa4c93236cf01939dcaf2999d2fbaf0b07871b47151cc2f30edaa03a66604e" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.1875, + "passed": 12 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.546875, + "passed": 35 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.90625, + "passed": 58 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192/round1/results/r0p10.json", + "sha256": "735ab074c822115f2aa7a783d5129c76e8b987967fdfd428b56ae197c7bbd2cf" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192/round2/results/r0p10.json", + "sha256": "72337cfe69cb1a42a4c916515fa2b3ac627fe2897a5a01260b5a293d76a0974f" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.140625, + 0.140625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192/round1/results/r1p20.json", + "sha256": "9892c9994e313caf46851213beddc9fc2f492a13ec39311780187fea96edab16" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192/round2/results/r1p20.json", + "sha256": "b4b4ac7ecd028619e322364287f316271ba4e1b3cf780813276755e48469fcf9" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.109375, + 0.171875 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192/round1/results/r1p60.json", + "sha256": "98e44d02a725d74e5fa79219e54706191c43d57f246e0008e2191731b7f1dc1f" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt8192/round2/results/r1p60.json", + "sha256": "bf601143bb6192d8c99136d6d2514ee02d5d12d96b666775096f6c4ae7ddf98c" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.078125, + 0.15625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r2p40-20260716-v1-20260716T081536141326Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt8192/round1/results/r2p40.json", + "sha256": "d53dc0fd284cb0086f4684ec3faf5f0095a14b0aa0d292775d16785eb14ca165" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r2p40-20260716-v1-20260716T081536141326Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt8192/round2/results/r2p40.json", + "sha256": "36ed4ce070c637670755a7f5749e154ea9494acb28dd2efa9295baff446ba83a" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 3.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.03125 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.71875, + 0.75 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T082603837807Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp8_mns64_mbt8192/round1/results/r3p20.json", + "sha256": "fb2e55eb858fbcd7e8bd62d37d60742ba27cf350a76b005f0a62548b79199dd8" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T082603837807Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp8_mns64_mbt8192/round2/results/r3p20.json", + "sha256": "6952b9f2f2352b0bc51540fe0eca7afe163546e6433230ab537539ce6af63eb9" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.109375, + "passed": 7 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.40625, + "passed": 26 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 0.984375, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384/round1/results/r0p10.json", + "sha256": "24fda44413d2651f478d2ad0f2065de09e823d7b5c25187c0efa434df0b56ef2" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384/round2/results/r0p10.json", + "sha256": "fb60a85474a20bdc2b130c9812f5fb90a9eb899b1a04a32c906ea92421c4fc57" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.109375, + 0.140625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384/round1/results/r1p20.json", + "sha256": "862f4d0cb399d9e694dadf48b059d4a5fa384db257d38ef62b90c31ef85ac418" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384/round2/results/r1p20.json", + "sha256": "4f658d1d4392a340c8da4c0938fa8698fd0081da22915061a851e987ada24817" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.015625, + 0.125 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384/round1/results/r1p60.json", + "sha256": "81b83eddaa2c4a57c10dc3d49971c70a4d47801cda4578a04d48f34a41889b00" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns64_mbt16384/round2/results/r1p60.json", + "sha256": "af75c496cc43439e9c2ed34ae5fa61a43838fbce1fb3b0427c01a46c0643ffe2" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.078125, + 0.046875 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.703125, + 0.828125 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-expansion-r2p40-20260716-v1-20260716T092040851202Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt16384/round1/results/r2p40.json", + "sha256": "238e20ba03824bb8746d21fa7529b7b6e9662cfba4de039201a0a8d5b41c6360" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns64_mbt16384-expansion-r2p40-20260716-v1-20260716T092040851202Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns64_mbt16384/round2/results/r2p40.json", + "sha256": "bb32dd95a562be19cd49eb5aa915e3c58a9ef08b5868ba1f5835cb9263b9aeb8" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192/round1/results/r0p10.json", + "sha256": "a98554e8c0f0f16b24823eac1d7b4c36c5e13b94c4b6816f7d18708f436e33cb" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192/round2/results/r0p10.json", + "sha256": "8cc19335b5b4ea28f0cf297f8058fa53542e8439e69d19cbcb88c298a391eec2" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.078125, + 0.15625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192/round1/results/r1p20.json", + "sha256": "9e6b2e3c7f35f3befcd6a96148b991f277d5d2dc971451e485cadb14a54b035e" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192/round2/results/r1p20.json", + "sha256": "f69456dde5884f3c1080bfa1d0fd832ce288eacd0f771f443c62a83ad13ffb88" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.15625, + 0.09375 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192/round1/results/r1p60.json", + "sha256": "b1218de54c19589f233e3128c36e5dbc02f5e3cdb0b5611be3d4eb397659be7f" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt8192/round2/results/r1p60.json", + "sha256": "580f97a4950870a3429b7975392c40daddfdfabe5440134b8adfc393dd02c56f" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.0625, + 0.125 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.625, + 0.875 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-expansion-r2p40-20260716-v1-20260716T101734683241Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt8192/round1/results/r2p40.json", + "sha256": "4059c6ef758bb7363ae904ec09750792f21417a439d5ab1b0e1bcf38b0f836de" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt8192-expansion-r2p40-20260716-v1-20260716T101734683241Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt8192/round2/results/r2p40.json", + "sha256": "94e91a76a79f2123f66b19cb1f3023c93ac54f8a87b6ae65ad116116b03d38e1" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "loads": [ + { + "rate": 0.1, + "real_conservative": { + "tpot_120ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384/round1/results/r0p10.json", + "sha256": "d41528e5078098be6a3c38c1f25711c156edf3b38aaca995ffec3f6d65181427" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384/round2/results/r0p10.json", + "sha256": "3a302927e19f81e64e6925400212b059763e57c0545e1666c7ede7a5ccc788e9" + } + ], + "sim": { + "tpot_120ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.109375, + 0.09375 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384/round1/results/r1p20.json", + "sha256": "b317303377202e22f76cf3092436ff91e323d0a39e375e4bf6c091c7a88e1a61" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384/round2/results/r1p20.json", + "sha256": "0b0b704ea663719e8f0fe22347d23324a33da89f5e617caacc439f10bb22d4a0" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 1.6, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.046875, + 0.09375 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384/round1/results/r1p60.json", + "sha256": "785525671e6bfbff47a7d6c134ab33ce20b4736100bd6600ad871adcd20ebcdb" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z/artifacts/artifacts/t0-real-surface-v1/tp8_mns128_mbt16384/round2/results/r1p60.json", + "sha256": "68dcd748fe15978528738685b5e8798fb54abfb9dccfb08d4848839969419155" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.140625, + "passed": 9 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.4375, + "passed": 28 + }, + "tpot_180ms": { + "feasible": true, + "pass_rate": 1.0, + "passed": 64 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 2.4, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0625 + ] + }, + "tpot_150ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r2p40-20260716-v1-20260716T111437770615Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt16384/round1/results/r2p40.json", + "sha256": "34e4ae97f3137bd0e84fd6cf96da2435d3dd27af1459a38ae066e2e811257d85" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r2p40-20260716-v1-20260716T111437770615Z/artifacts/artifacts/t0-real-expansion-r2p40-v1/tp8_mns128_mbt16384/round2/results/r2p40.json", + "sha256": "4e4e7756011330c7c6adcbc788e3e82605551bbdd1d04d679914d492271f77b0" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.125, + "passed": 8 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.421875, + "passed": 27 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + }, + { + "rate": 3.2, + "real_conservative": { + "tpot_120ms": { + "feasible": false, + "round_pass_rates": [ + 0.078125, + 0.046875 + ] + }, + "tpot_150ms": { + "feasible": false, + "round_pass_rates": [ + 0.78125, + 0.71875 + ] + }, + "tpot_180ms": { + "feasible": true, + "round_pass_rates": [ + 1.0, + 1.0 + ] + }, + "tpot_40ms": { + "feasible": false, + "round_pass_rates": [ + 0.0, + 0.0 + ] + } + }, + "real_files": [ + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T112608807037Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp8_mns128_mbt16384/round1/results/r3p20.json", + "sha256": "3222b3d3e561c029d595951d90d99513abd2cf63c787dab924341aca3defc179" + }, + { + "path": "/home/gahow/phd/aituner/runs/frontier-multicase-sufficiency-v1/fleet-artifacts/qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T112608807037Z/artifacts/artifacts/t0-real-expansion-r3p20-v1/tp8_mns128_mbt16384/round2/results/r3p20.json", + "sha256": "e7914e696ac6eaa46d9cc83e76ac2f7e8cd195e76f48445cc98cddb905c3c213" + } + ], + "sim": { + "tpot_120ms": { + "feasible": false, + "pass_rate": 0.109375, + "passed": 7 + }, + "tpot_150ms": { + "feasible": false, + "pass_rate": 0.40625, + "passed": 26 + }, + "tpot_180ms": { + "feasible": false, + "pass_rate": 0.71875, + "passed": 46 + }, + "tpot_40ms": { + "feasible": false, + "pass_rate": 0.0, + "passed": 0 + } + } + } + ] + } + ], + "comparisons": { + "tpot_120ms": { + "anchor_confusion": { + "anchors": 34, + "false_feasible": 0, + "false_infeasible": 0, + "match": 34 + }, + "kendall_tau_b": null, + "optimistic_regret": 0.0, + "pairwise": { + "all_pairs": 0, + "exact_sign_matches": 0, + "real_non_tie_direction_matches": 0, + "real_non_tie_pairs": 0 + }, + "real_top_set": [ + "tp4_mns64_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns128_mbt16384" + ], + "records": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.025, + "sim_capacity": 0.8, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.025, + "sim_capacity": 0.8, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.025, + "sim_capacity": 0.8, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.025, + "sim_capacity": 0.8, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.0125, + "sim_capacity": 0.4, + "sim_capacity_per_gpu": 0.05 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.0125, + "sim_capacity": 0.4, + "sim_capacity_per_gpu": 0.05 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.0125, + "sim_capacity": 0.4, + "sim_capacity_per_gpu": 0.05 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 0.1, + "real_capacity_per_gpu": 0.0125, + "sim_capacity": 0.4, + "sim_capacity_per_gpu": 0.05 + } + ], + "sim_top_set": [ + "tp4_mns64_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns128_mbt16384" + ], + "top_set_exact_match": true, + "top_set_intersection": [ + "tp4_mns128_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns64_mbt8192" + ], + "worst_tie_break_regret": 0.0 + }, + "tpot_150ms": { + "anchor_confusion": { + "anchors": 34, + "false_feasible": 0, + "false_infeasible": 10, + "match": 24 + }, + "kendall_tau_b": 0.8944271909999159, + "optimistic_regret": 0.0, + "pairwise": { + "all_pairs": 28, + "exact_sign_matches": 24, + "real_non_tie_direction_matches": 16, + "real_non_tie_pairs": 20 + }, + "real_top_set": [ + "tp4_mns64_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns128_mbt16384" + ], + "records": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.6, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.4 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.6, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.4 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.6, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.4 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.6, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.4 + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.3, + "sim_capacity": 1.2, + "sim_capacity_per_gpu": 0.15 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 1.6, + "real_capacity_per_gpu": 0.2, + "sim_capacity": 1.2, + "sim_capacity_per_gpu": 0.15 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 1.6, + "real_capacity_per_gpu": 0.2, + "sim_capacity": 1.2, + "sim_capacity_per_gpu": 0.15 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "adjacent_transition_bracketed", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.3, + "sim_capacity": 1.2, + "sim_capacity_per_gpu": 0.15 + } + ], + "sim_top_set": [ + "tp4_mns64_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns128_mbt16384" + ], + "top_set_exact_match": true, + "top_set_intersection": [ + "tp4_mns128_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns64_mbt8192" + ], + "worst_tie_break_regret": 0.0 + }, + "tpot_180ms": { + "anchor_confusion": { + "anchors": 34, + "false_feasible": 0, + "false_infeasible": 10, + "match": 24 + }, + "kendall_tau_b": 1.0, + "optimistic_regret": 0.0, + "pairwise": { + "all_pairs": 15, + "exact_sign_matches": 15, + "real_non_tie_direction_matches": 8, + "real_non_tie_pairs": 8 + }, + "real_top_set": [ + "tp4_mns64_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns128_mbt16384" + ], + "records": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "upper_lattice_reached", + "real_capacity": 3.2, + "real_capacity_per_gpu": 0.8, + "sim_capacity": 2.4, + "sim_capacity_per_gpu": 0.6 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "upper_lattice_reached", + "real_capacity": 3.2, + "real_capacity_per_gpu": 0.8, + "sim_capacity": 2.4, + "sim_capacity_per_gpu": 0.6 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "upper_lattice_reached", + "real_capacity": 3.2, + "real_capacity_per_gpu": 0.8, + "sim_capacity": 2.4, + "sim_capacity_per_gpu": 0.6 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "upper_lattice_reached", + "real_capacity": 3.2, + "real_capacity_per_gpu": 0.8, + "sim_capacity": 2.4, + "sim_capacity_per_gpu": 0.6 + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "upper_lattice_reached", + "real_capacity": 3.2, + "real_capacity_per_gpu": 0.4, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.3, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": true, + "real_boundary_status": "unbracketed_requires_expansion", + "real_capacity": 2.4, + "real_capacity_per_gpu": 0.3, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.2 + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "upper_lattice_reached", + "real_capacity": 3.2, + "real_capacity_per_gpu": 0.4, + "sim_capacity": 1.6, + "sim_capacity_per_gpu": 0.2 + } + ], + "sim_top_set": [ + "tp4_mns64_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns128_mbt16384" + ], + "top_set_exact_match": true, + "top_set_intersection": [ + "tp4_mns128_mbt16384", + "tp4_mns128_mbt8192", + "tp4_mns64_mbt16384", + "tp4_mns64_mbt8192" + ], + "worst_tie_break_regret": 0.0 + }, + "tpot_40ms": { + "anchor_confusion": { + "anchors": 34, + "false_feasible": 0, + "false_infeasible": 0, + "match": 34 + }, + "kendall_tau_b": null, + "optimistic_regret": null, + "pairwise": { + "all_pairs": 0, + "exact_sign_matches": 0, + "real_non_tie_direction_matches": 0, + "real_non_tie_pairs": 0 + }, + "real_top_set": [], + "records": [ + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns64_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt8192", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 1, + "moe_tp": 4, + "name": "tp4_mns128_mbt16384", + "num_gpu_blocks": 26101, + "tp": 4 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 8192, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 16384, + "mns": 64, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns64_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 8192, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt8192", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + }, + { + "config": { + "mbt": 16384, + "mns": 128, + "moe_ep": 8, + "moe_tp": 1, + "name": "tp8_mns128_mbt16384", + "num_gpu_blocks": 62351, + "tp": 8 + }, + "expansion_required": false, + "real_boundary_status": "lowest_anchor_infeasible", + "real_capacity": null, + "real_capacity_per_gpu": null, + "sim_capacity": null, + "sim_capacity_per_gpu": null + } + ], + "sim_top_set": [], + "top_set_exact_match": true, + "top_set_intersection": [], + "worst_tie_break_regret": null + } + }, + "frontier_freeze_sha256": "801aa36451c8647f71cc87011144622d2203786e82f189ed1375d964399b106a", + "real_execution_cost": { + "accepted_fleet_jobs": 18, + "accepted_h20_gpu_hours": 36.25666666666667, + "campaign_wall_span_seconds": 20670.0, + "fresh_server_anchors": 68, + "measured_requests": 4352, + "runs": [ + { + "config": "tp4_mns64_mbt8192", + "finished_at": "2026-07-16T06:25:28+00:00", + "gpu_count": 4, + "h20_gpu_hours": 2.5844444444444443, + "run_id": "qwen235b-t0-real-tp4_mns64_mbt8192-20260716-v1-20260716T054640909739Z", + "started_at": "2026-07-16T05:46:42+00:00", + "wall_seconds": 2326.0 + }, + { + "config": "tp4_mns64_mbt16384", + "finished_at": "2026-07-16T06:25:30+00:00", + "gpu_count": 4, + "h20_gpu_hours": 2.5855555555555556, + "run_id": "qwen235b-t0-real-tp4_mns64_mbt16384-20260716-v1-20260716T054642115787Z", + "started_at": "2026-07-16T05:46:43+00:00", + "wall_seconds": 2327.0 + }, + { + "config": "tp4_mns64_mbt8192", + "finished_at": "2026-07-16T06:34:32+00:00", + "gpu_count": 4, + "h20_gpu_hours": 0.3622222222222222, + "run_id": "qwen235b-t0-real-tp4_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T062904775741Z", + "started_at": "2026-07-16T06:29:06+00:00", + "wall_seconds": 326.0 + }, + { + "config": "tp4_mns64_mbt16384", + "finished_at": "2026-07-16T06:34:32+00:00", + "gpu_count": 4, + "h20_gpu_hours": 0.3611111111111111, + "run_id": "qwen235b-t0-real-tp4_mns64_mbt16384-expansion-r3p20-20260716-v1-20260716T062906244819Z", + "started_at": "2026-07-16T06:29:07+00:00", + "wall_seconds": 325.0 + }, + { + "config": "tp4_mns128_mbt8192", + "finished_at": "2026-07-16T07:16:55+00:00", + "gpu_count": 4, + "h20_gpu_hours": 2.5644444444444443, + "run_id": "qwen235b-t0-real-tp4_mns128_mbt8192-20260716-v1-20260716T063826749477Z", + "started_at": "2026-07-16T06:38:27+00:00", + "wall_seconds": 2308.0 + }, + { + "config": "tp4_mns128_mbt16384", + "finished_at": "2026-07-16T07:17:04+00:00", + "gpu_count": 4, + "h20_gpu_hours": 2.5733333333333333, + "run_id": "qwen235b-t0-real-tp4_mns128_mbt16384-20260716-v1-20260716T063827805737Z", + "started_at": "2026-07-16T06:38:28+00:00", + "wall_seconds": 2316.0 + }, + { + "config": "tp4_mns128_mbt8192", + "finished_at": "2026-07-16T07:26:47+00:00", + "gpu_count": 4, + "h20_gpu_hours": 0.3611111111111111, + "run_id": "qwen235b-t0-real-tp4_mns128_mbt8192-expansion-r3p20-20260716-v1-20260716T072121119478Z", + "started_at": "2026-07-16T07:21:22+00:00", + "wall_seconds": 325.0 + }, + { + "config": "tp4_mns128_mbt16384", + "finished_at": "2026-07-16T07:26:49+00:00", + "gpu_count": 4, + "h20_gpu_hours": 0.3622222222222222, + "run_id": "qwen235b-t0-real-tp4_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T072122501851Z", + "started_at": "2026-07-16T07:21:23+00:00", + "wall_seconds": 326.0 + }, + { + "config": "tp8_mns64_mbt8192", + "finished_at": "2026-07-16T08:09:49+00:00", + "gpu_count": 8, + "h20_gpu_hours": 5.055555555555555, + "run_id": "qwen235b-t0-real-tp8_mns64_mbt8192-20260716-v1-20260716T073151443129Z", + "started_at": "2026-07-16T07:31:54+00:00", + "wall_seconds": 2275.0 + }, + { + "config": "tp8_mns64_mbt8192", + "finished_at": "2026-07-16T08:20:48+00:00", + "gpu_count": 8, + "h20_gpu_hours": 0.6911111111111111, + "run_id": "qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r2p40-20260716-v1-20260716T081536141326Z", + "started_at": "2026-07-16T08:15:37+00:00", + "wall_seconds": 311.0 + }, + { + "config": "tp8_mns64_mbt8192", + "finished_at": "2026-07-16T08:31:03+00:00", + "gpu_count": 8, + "h20_gpu_hours": 0.6622222222222223, + "run_id": "qwen235b-t0-real-tp8_mns64_mbt8192-expansion-r3p20-20260716-v1-20260716T082603837807Z", + "started_at": "2026-07-16T08:26:05+00:00", + "wall_seconds": 298.0 + }, + { + "config": "tp8_mns64_mbt16384", + "finished_at": "2026-07-16T09:14:50+00:00", + "gpu_count": 8, + "h20_gpu_hours": 5.104444444444445, + "run_id": "qwen235b-t0-real-tp8_mns64_mbt16384-20260716-v1-20260716T083632620015Z", + "started_at": "2026-07-16T08:36:33+00:00", + "wall_seconds": 2297.0 + }, + { + "config": "tp8_mns64_mbt16384", + "finished_at": "2026-07-16T09:26:03+00:00", + "gpu_count": 8, + "h20_gpu_hours": 0.7133333333333334, + "run_id": "qwen235b-t0-real-tp8_mns64_mbt16384-expansion-r2p40-20260716-v1-20260716T092040851202Z", + "started_at": "2026-07-16T09:20:42+00:00", + "wall_seconds": 321.0 + }, + { + "config": "tp8_mns128_mbt8192", + "finished_at": "2026-07-16T10:10:53+00:00", + "gpu_count": 8, + "h20_gpu_hours": 5.08, + "run_id": "qwen235b-t0-real-tp8_mns128_mbt8192-20260716-v1-20260716T093245831612Z", + "started_at": "2026-07-16T09:32:47+00:00", + "wall_seconds": 2286.0 + }, + { + "config": "tp8_mns128_mbt8192", + "finished_at": "2026-07-16T10:22:54+00:00", + "gpu_count": 8, + "h20_gpu_hours": 0.7088888888888889, + "run_id": "qwen235b-t0-real-tp8_mns128_mbt8192-expansion-r2p40-20260716-v1-20260716T101734683241Z", + "started_at": "2026-07-16T10:17:35+00:00", + "wall_seconds": 319.0 + }, + { + "config": "tp8_mns128_mbt16384", + "finished_at": "2026-07-16T11:07:16+00:00", + "gpu_count": 8, + "h20_gpu_hours": 5.108888888888889, + "run_id": "qwen235b-t0-real-tp8_mns128_mbt16384-20260716-v1-20260716T102855604708Z", + "started_at": "2026-07-16T10:28:57+00:00", + "wall_seconds": 2299.0 + }, + { + "config": "tp8_mns128_mbt16384", + "finished_at": "2026-07-16T11:19:57+00:00", + "gpu_count": 8, + "h20_gpu_hours": 0.7066666666666667, + "run_id": "qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r2p40-20260716-v1-20260716T111437770615Z", + "started_at": "2026-07-16T11:14:39+00:00", + "wall_seconds": 318.0 + }, + { + "config": "tp8_mns128_mbt16384", + "finished_at": "2026-07-16T11:31:12+00:00", + "gpu_count": 8, + "h20_gpu_hours": 0.6711111111111111, + "run_id": "qwen235b-t0-real-tp8_mns128_mbt16384-expansion-r3p20-20260716-v1-20260716T112608807037Z", + "started_at": "2026-07-16T11:26:10+00:00", + "wall_seconds": 302.0 + } + ], + "warmup_requests": 1664 + }, + "real_plan_sha256": "c934d42d66443e98884c136017057fdd636c4ef8845f96b4f49f364a5d8da1c2", + "schema": "qwen235b-t0-simulator-real-comparison-v1" +} diff --git a/runs/frontier-multicase-sufficiency-v1/run_decode_attention_profile.sh b/runs/frontier-multicase-sufficiency-v1/run_decode_attention_profile.sh new file mode 100644 index 0000000..cf56871 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/run_decode_attention_profile.sh @@ -0,0 +1,95 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:-$(pwd)/artifacts/decode-attention-profile-20260716}" +FRONTIER_ROOT="${FRONTIER_ROOT:-/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh-frontier-vllm0102-smoke/.venv}" +PROFILE_ROOT="${OUTPUT_ROOT}/profiles" +LOG_DIR="${OUTPUT_ROOT}/logs" +PROVENANCE_DIR="${OUTPUT_ROOT}/provenance" +MODEL="Qwen3-235B-A22B-FP8" + +mkdir -p "${PROFILE_ROOT}" "${LOG_DIR}" "${PROVENANCE_DIR}" +exec > >(tee -a "${LOG_DIR}/profile.log") 2>&1 + +if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then + echo "ERROR: CUDA_VISIBLE_DEVICES must contain the fleet-allocated GPU" >&2 + exit 1 +fi +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES}" +if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then + echo "ERROR: expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} model=${MODEL} operator=FlashInfer_attention phase=decode TP_workers=4 batch_sizes=1,2 kv_sizes=2048,2176 block=16 measurement=CUDA_EVENT output=${OUTPUT_ROOT} expected_wall=5-10m hard_wall=900s hard_gpu_cap=0.25_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader + +test -x "${VENV_ROOT}/bin/python" +test -f "${FRONTIER_ROOT}/pyproject.toml" +test -f "${FRONTIER_ROOT}/data/config/models/${MODEL}.json" +sha256sum run_decode_attention_profile.sh > "${PROVENANCE_DIR}/source.sha256" + +export PYTHONPATH="${FRONTIER_ROOT}" +export TOKENIZERS_PARALLELISM=false +export TORCH_CUDA_ARCH_LIST=9.0 + +cd "${FRONTIER_ROOT}" +timeout --signal=TERM --kill-after=30s 600 \ + "${VENV_ROOT}/bin/python" -m frontier.profiling.attention.main \ + --disable_ray \ + --models "${MODEL}" \ + --num_gpus 1 \ + --max_model_len 40960 \ + --max_seq_len 2176 \ + --min_batch_size 1 \ + --max_batch_size 2 \ + --batch_size_list 1 2 \ + --decode_kv_cache_size_list 2048 2176 \ + --num_tensor_parallel_workers 4 \ + --max_pipeline_parallel_size 1 \ + --attention_backend FLASHINFER \ + --block_size 16 \ + --profile_only_decode \ + --device h20 \ + --profile_method cuda_event \ + --output_dir "${PROFILE_ROOT}" \ + --yes + +ATTENTION_CSV="${PROFILE_ROOT}/compute/h20/${MODEL}/attention.csv" +test -s "${ATTENTION_CSV}" +"${VENV_ROOT}/bin/python" - "${ATTENTION_CSV}" \ + > "${PROVENANCE_DIR}/coverage.json" <<'PY' +import json +import sys + +import pandas as pd + +path = sys.argv[1] +frame = pd.read_csv(path) +decode = frame[frame["is_prefill"] == False] # noqa: E712 +payload = { + "path": path, + "row_count": len(frame), + "decode_row_count": len(decode), + "batch_sizes": sorted(int(value) for value in decode["batch_size"].unique()), + "kv_cache_sizes": sorted(int(value) for value in decode["kv_cache_size"].unique()), + "attn_decode_median_non_null": int( + decode["time_stats.attn_decode.median"].notna().sum() + ), +} +print(json.dumps(payload, indent=2, sort_keys=True)) +if payload["decode_row_count"] < 4 or payload["attn_decode_median_non_null"] < 4: + raise SystemExit(1) +PY + +sha256sum \ + "${ATTENTION_CSV}" \ + "${PROVENANCE_DIR}/coverage.json" \ + "${PROVENANCE_DIR}/source.sha256" \ + > "${PROVENANCE_DIR}/artifacts.sha256" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "DECODE_ATTENTION_PROFILE_COMPLETE" diff --git a/runs/frontier-multicase-sufficiency-v1/run_frontier_t0_smoke.py b/runs/frontier-multicase-sufficiency-v1/run_frontier_t0_smoke.py new file mode 100644 index 0000000..84ec8bd --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/run_frontier_t0_smoke.py @@ -0,0 +1,298 @@ +#!/usr/bin/env python3 +"""Run Frontier on the exact single/concurrency-2 T0 smoke workload.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import os +import subprocess +import time +from pathlib import Path +from typing import Any + + +MODEL = "Qwen3-235B-A22B-FP8" + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--frontier-source", type=Path, required=True) + parser.add_argument("--profile-root", type=Path, required=True) + parser.add_argument("--python", type=Path, default=Path("/usr/bin/python3.12")) + parser.add_argument("--output-root", type=Path, required=True) + return parser.parse_args() + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def write_json(path: Path, payload: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + + +def write_trace(path: Path, request_count: int) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", newline="") as output: + writer = csv.DictWriter( + output, + fieldnames=[ + "arrived_at", + "num_prefill_tokens", + "num_decode_tokens", + "slo_ttft_ms", + ], + ) + writer.writeheader() + for _ in range(request_count): + writer.writerow( + { + "arrived_at": 0.0, + "num_prefill_tokens": 2048, + "num_decode_tokens": 128, + "slo_ttft_ms": 1256.0, + } + ) + + +def build_command( + *, + args: argparse.Namespace, + trace: Path, + run_dir: Path, + run_id: str, +) -> list[str]: + compute_root = args.profile_root / "compute/h20" / MODEL + network = args.profile_root / "network/h20_nccl/all_reduce.csv" + return [ + str(args.python), + "-m", + "frontier.main", + "--simulation_mode", + "offline", + "--offline_use_generated_request_arrivals", + "--sys_arch", + "co-location", + "--cluster_config_num_replicas", + "1", + "--replica_config_model_name", + MODEL, + "--replica_config_attn_tensor_parallel_size", + "4", + "--replica_config_attn_data_parallel_size", + "1", + "--replica_config_moe_tensor_parallel_size", + "4", + "--replica_config_moe_expert_parallel_size", + "1", + "--replica_config_total_expert_num", + "128", + "--replica_config_router_topk", + "8", + "--replica_config_moe_routing_mode", + "simulation", + "--replica_config_moe_routing_seed", + "42", + "--replica_config_num_pipeline_stages", + "1", + "--replica_config_device", + "h20", + "--replica_config_network_device", + "h20_dgx", + "--cc_backend_config_type", + "vidur", + "--vidur_cc_backend_config_profiling_data_dir", + str(args.profile_root), + "--vidur_cc_backend_config_cache_dir", + str(run_dir / "cache/collectives"), + "--vidur_cc_backend_config_all_reduce_input_file", + str(network), + "--replica_scheduler_config_type", + "vllm_v1", + "--decode_cuda_graph_mode", + "none", + "--vllm_v1_scheduler_config_batch_size_cap", + "64", + "--vllm_v1_scheduler_config_block_size", + "16", + "--vllm_v1_scheduler_config_num_blocks", + "26101", + "--vllm_v1_scheduler_config_num_blocks_mode", + "explicit", + "--vllm_v1_scheduler_config_max_tokens_in_batch", + "8192", + "--vllm_v1_scheduler_config_enable_chunked_prefill", + "--no-vllm_v1_scheduler_config_enable_prefix_caching", + "--request_generator_config_type", + "trace_replay", + "--trace_request_generator_config_trace_file", + str(trace), + "--trace_request_generator_config_time_scale_factor", + "1", + "--trace_request_generator_config_prefill_scale_factor", + "1", + "--trace_request_generator_config_decode_scale_factor", + "1", + "--trace_request_generator_config_max_tokens", + "40960", + "--no-random_forrest_execution_time_predictor_config_enable_dummy_mode", + "--random_forrest_execution_time_predictor_config_linear_op_input_file", + str(compute_root / "linear_op.csv"), + "--random_forrest_execution_time_predictor_config_atten_input_file", + str(compute_root / "attention.csv"), + "--random_forrest_execution_time_predictor_config_moe_input_file", + str(compute_root / "moe.csv"), + "--random_forrest_execution_time_predictor_config_all_reduce_input_file", + str(network), + "--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size", + "16384", + "--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request", + "40960", + "--random_forrest_execution_time_predictor_config_prediction_max_batch_size", + "128", + "--random_forrest_execution_time_predictor_config_skip_cpu_overhead_modeling", + "--metrics_config_cache_dir", + str(run_dir / "cache/execution"), + "--metrics_config_output_dir", + str(run_dir / "metrics"), + "--metrics_config_run_id", + run_id, + "--metrics_config_write_metrics", + "--metrics_config_store_request_metrics", + "--no-metrics_config_store_plots", + "--no-metrics_config_enable_chrome_trace", + "--no-metrics_config_write_json_trace", + ] + + +def find_request_metrics(run_dir: Path) -> Path: + matches = list((run_dir / "metrics").rglob("request_metrics.csv")) + if len(matches) != 1: + raise RuntimeError(f"expected one request_metrics.csv, found {matches}") + return matches[0] + + +def score(metrics_path: Path) -> dict[str, Any]: + with metrics_path.open(newline="") as source: + rows = list(csv.DictReader(source)) + requests = [] + for row in rows: + ttft_ms = float(row["ttft"]) + e2e_ms = float(row["request_e2e_time"]) + decode_tokens = int(float(row["request_num_decode_tokens"])) + aligned_tpot_ms = ( + (e2e_ms - ttft_ms) / (decode_tokens - 1) if decode_tokens > 1 else 0.0 + ) + requests.append( + { + "request_id": int(row["Request Id"]), + "prompt_tokens": int(float(row["request_num_prefill_tokens"])), + "completion_tokens": decode_tokens, + "ttft_ms": ttft_ms, + "e2e_ms": e2e_ms, + "tpot_ms_aligned": aligned_tpot_ms, + "frontier_decode_e2e_time_per_token_ms": float( + row["decode_e2e_time_per_token"] + ), + "joint_slo_pass": ttft_ms <= 1256.0 and aligned_tpot_ms <= 40.0, + } + ) + return { + "request_count": len(requests), + "joint_slo_pass_count": sum(row["joint_slo_pass"] for row in requests), + "requests": requests, + } + + +def main() -> None: + args = parse_args() + source = args.frontier_source.resolve() + profiles = args.profile_root.resolve() + output = args.output_root.resolve() + required = [ + source / "pyproject.toml", + profiles / f"compute/h20/{MODEL}/linear_op.csv", + profiles / f"compute/h20/{MODEL}/attention.csv", + profiles / f"compute/h20/{MODEL}/moe.csv", + profiles / "network/h20_nccl/all_reduce.csv", + ] + missing = [str(path) for path in required if not path.is_file()] + if missing: + raise FileNotFoundError(missing) + + results: dict[str, Any] = {} + for label, count in (("single", 1), ("concurrency2", 2)): + run_dir = output / label + trace = run_dir / "trace.csv" + write_trace(trace, count) + command = build_command( + args=args, + trace=trace, + run_dir=run_dir, + run_id=f"t0_{label}", + ) + write_json(run_dir / "command.json", command) + env = dict(os.environ) + env["PYTHONPATH"] = str(source) + started = time.time() + with (run_dir / "stdout.log").open("w") as stdout: + completed = subprocess.run( + command, + cwd=source, + env=env, + stdout=stdout, + stderr=subprocess.STDOUT, + check=False, + text=True, + timeout=300, + ) + if completed.returncode != 0: + raise RuntimeError( + f"Frontier {label} failed with {completed.returncode}; " + f"see {run_dir / 'stdout.log'}" + ) + metrics = find_request_metrics(run_dir) + results[label] = { + "elapsed_seconds": time.time() - started, + "trace_sha256": sha256(trace), + "request_metrics_path": str(metrics), + "request_metrics_sha256": sha256(metrics), + **score(metrics), + } + + profile_files = required[1:] + payload = { + "schema": "frontier-qwen235b-t0-smoke-v1", + "contract": { + "topology": "TP4/DP1/MoE-TP4/EP1", + "mns": 64, + "mbt": 8192, + "block_size": 16, + "num_gpu_blocks": 26101, + "prefix_caching": False, + "input_tokens": 2048, + "output_tokens": 128, + "arrivals": "all at t=0", + }, + "frontier": { + "source": str(source), + "declared_upstream_commit": "d9cfeb6d8791fbf2f295dd9744c56a666171776e", + "python_and_config_tree_sha256": "172fc7ae19c40e67030208ff488d0d5d90764888ed76a7a543be6037ba62dc11", + }, + "profiles": {str(path): sha256(path) for path in profile_files}, + "results": results, + } + write_json(output / "summary.json", payload) + print(json.dumps(payload["results"], indent=2, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/run_frontier_t0_surface.py b/runs/frontier-multicase-sufficiency-v1/run_frontier_t0_surface.py new file mode 100644 index 0000000..41c3a54 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/run_frontier_t0_surface.py @@ -0,0 +1,387 @@ +#!/usr/bin/env python3 +"""Freeze the full fixed-shape T0 Frontier response surface.""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import math +import os +import subprocess +import time +from dataclasses import asdict, dataclass +from pathlib import Path +from typing import Any + + +MODEL = "Qwen3-235B-A22B-FP8" +FRONTIER_DECLARED_UPSTREAM_COMMIT = "d9cfeb6d8791fbf2f295dd9744c56a666171776e" +FRONTIER_PYTHON_CONFIG_TREE_SHA256 = "172fc7ae19c40e67030208ff488d0d5d90764888ed76a7a543be6037ba62dc11" +RATES = (0.10, 0.20, 0.40, 0.80, 1.20, 1.60, 2.40, 3.20) +TPOT_SLOS_MS = (40.0, 120.0, 150.0, 180.0) +TTFT_SLO_MS = 1256.0 +TARGET_PASS_RATE = 0.95 + + +@dataclass(frozen=True) +class Config: + tp: int + mns: int + mbt: int + moe_tp: int + moe_ep: int + num_gpu_blocks: int + + @property + def name(self) -> str: + return f"tp{self.tp}_mns{self.mns}_mbt{self.mbt}" + + +GRID = tuple( + Config( + tp=tp, + mns=mns, + mbt=mbt, + moe_tp=4 if tp == 4 else 1, + moe_ep=1 if tp == 4 else 8, + num_gpu_blocks=26101 if tp == 4 else 62351, + ) + for tp in (4, 8) + for mns in (64, 128) + for mbt in (8192, 16384) +) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--frontier-source", type=Path, required=True) + parser.add_argument("--profile-root", type=Path, required=True) + parser.add_argument("--python", type=Path, required=True) + parser.add_argument("--output-root", type=Path, required=True) + parser.add_argument("--requests", type=int, default=64) + parser.add_argument("--rate", type=float, action="append") + parser.add_argument("--config", action="append") + parser.add_argument("--subprocess-timeout-seconds", type=int, default=1800) + return parser.parse_args() + + +def sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def write_json(path: Path, payload: Any) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + temporary = path.with_suffix(path.suffix + ".tmp") + temporary.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + os.replace(temporary, path) + + +def rate_key(rate: float) -> str: + return f"r{rate:.2f}".replace(".", "p") + + +def write_trace(path: Path, *, request_count: int, rate: float) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", newline="") as output: + writer = csv.DictWriter( + output, + fieldnames=[ + "arrived_at", + "num_prefill_tokens", + "num_decode_tokens", + "slo_ttft_ms", + ], + ) + writer.writeheader() + for request_id in range(request_count): + writer.writerow( + { + "arrived_at": f"{request_id / rate:.12f}", + "num_prefill_tokens": 2048, + "num_decode_tokens": 128, + "slo_ttft_ms": TTFT_SLO_MS, + } + ) + + +def profile_paths(root: Path) -> dict[str, Path]: + compute = root / "compute/h20" / MODEL + paths = { + "linear": compute / "linear_op.csv", + "attention": compute / "attention.csv", + "moe": compute / "moe.csv", + "all_reduce": root / "network/h20_nccl/all_reduce.csv", + "closure_manifest": root / "profile_closure_manifest.json", + } + missing = [str(path) for path in paths.values() if not path.is_file()] + if missing: + raise FileNotFoundError(missing) + return paths + + +def validate_attention_coverage(path: Path) -> dict[str, Any]: + with path.open(newline="") as source: + rows = list(csv.DictReader(source)) + standard_decode = [ + row + for row in rows + if row["is_prefill"].lower() == "false" + and row.get("is_true_mixed_batch", "").lower() != "true" + ] + true_mixed = [ + row for row in rows if row.get("is_true_mixed_batch", "").lower() == "true" + ] + standard_by_tp = { + str(tp): sum(int(row["num_tensor_parallel_workers"]) == tp for row in standard_decode) + for tp in (4, 8) + } + mixed_by_tp = { + str(tp): sum(int(row["num_tensor_parallel_workers"]) == tp for row in true_mixed) + for tp in (4, 8) + } + if standard_by_tp != {"4": 81, "8": 81}: + raise ValueError(f"standard decode coverage mismatch: {standard_by_tp}") + if mixed_by_tp != {"4": 108, "8": 108}: + raise ValueError(f"true-mixed coverage mismatch: {mixed_by_tp}") + return { + "rows": len(rows), + "standard_decode_by_tp": standard_by_tp, + "true_mixed_by_tp": mixed_by_tp, + } + + +def build_command( + args: argparse.Namespace, + paths: dict[str, Path], + config: Config, + trace: Path, + run_dir: Path, +) -> list[str]: + cache = args.output_root / "cache" / f"tp{config.tp}" + return [ + str(args.python), "-m", "frontier.main", + "--simulation_mode", "offline", + "--offline_use_generated_request_arrivals", + "--sys_arch", "co-location", + "--cluster_config_num_replicas", "1", + "--replica_config_model_name", MODEL, + "--replica_config_attn_tensor_parallel_size", str(config.tp), + "--replica_config_attn_data_parallel_size", "1", + "--replica_config_moe_tensor_parallel_size", str(config.moe_tp), + "--replica_config_moe_expert_parallel_size", str(config.moe_ep), + "--replica_config_total_expert_num", "128", + "--replica_config_router_topk", "8", + "--replica_config_moe_routing_mode", "simulation", + "--replica_config_moe_routing_seed", "42", + "--replica_config_num_pipeline_stages", "1", + "--replica_config_device", "h20", + "--replica_config_network_device", "h20_dgx", + "--cc_backend_config_type", "vidur", + "--vidur_cc_backend_config_profiling_data_dir", str(args.profile_root), + "--vidur_cc_backend_config_cache_dir", str(cache / "collectives"), + "--vidur_cc_backend_config_all_reduce_input_file", str(paths["all_reduce"]), + "--replica_scheduler_config_type", "vllm_v1", + "--decode_cuda_graph_mode", "none", + "--vllm_v1_scheduler_config_batch_size_cap", str(config.mns), + "--vllm_v1_scheduler_config_block_size", "16", + "--vllm_v1_scheduler_config_num_blocks", str(config.num_gpu_blocks), + "--vllm_v1_scheduler_config_num_blocks_mode", "explicit", + "--vllm_v1_scheduler_config_max_tokens_in_batch", str(config.mbt), + "--vllm_v1_scheduler_config_enable_chunked_prefill", + "--no-vllm_v1_scheduler_config_enable_prefix_caching", + "--request_generator_config_type", "trace_replay", + "--trace_request_generator_config_trace_file", str(trace), + "--trace_request_generator_config_time_scale_factor", "1", + "--trace_request_generator_config_prefill_scale_factor", "1", + "--trace_request_generator_config_decode_scale_factor", "1", + "--trace_request_generator_config_max_tokens", "40960", + "--no-random_forrest_execution_time_predictor_config_enable_dummy_mode", + "--random_forrest_execution_time_predictor_config_linear_op_input_file", str(paths["linear"]), + "--random_forrest_execution_time_predictor_config_atten_input_file", str(paths["attention"]), + "--random_forrest_execution_time_predictor_config_moe_input_file", str(paths["moe"]), + "--random_forrest_execution_time_predictor_config_all_reduce_input_file", str(paths["all_reduce"]), + "--random_forrest_execution_time_predictor_config_prediction_max_prefill_chunk_size", "16384", + "--random_forrest_execution_time_predictor_config_prediction_max_tokens_per_request", "40960", + "--random_forrest_execution_time_predictor_config_prediction_max_batch_size", "128", + "--random_forrest_execution_time_predictor_config_skip_cpu_overhead_modeling", + "--metrics_config_cache_dir", str(cache / "execution"), + "--metrics_config_output_dir", str(run_dir / "metrics"), + "--metrics_config_run_id", f"{config.name}_{run_dir.name}", + "--metrics_config_write_metrics", + "--metrics_config_store_request_metrics", + "--no-metrics_config_store_plots", + "--no-metrics_config_enable_chrome_trace", + "--no-metrics_config_write_json_trace", + ] + + +def find_metrics(run_dir: Path) -> Path: + matches = list((run_dir / "metrics").rglob("request_metrics.csv")) + if len(matches) != 1: + raise RuntimeError(f"expected one request_metrics.csv, got {matches}") + return matches[0] + + +def score(metrics: Path, expected_requests: int) -> dict[str, Any]: + with metrics.open(newline="") as source: + rows = list(csv.DictReader(source)) + if len(rows) != expected_requests: + raise ValueError(f"request count mismatch: {len(rows)} != {expected_requests}") + requests = [] + for row in rows: + prompt = int(float(row["request_num_prefill_tokens"])) + completion = int(float(row["request_num_decode_tokens"])) + ttft = float(row["ttft"]) + e2e = float(row["request_e2e_time"]) + tpot = (e2e - ttft) / (completion - 1) + if prompt != 2048 or completion != 128: + raise ValueError("request shape drift") + if not all(math.isfinite(value) and value >= 0 for value in (ttft, e2e, tpot)): + raise ValueError("non-finite or negative latency") + requests.append({"request_id": int(row["Request Id"]), "ttft_ms": ttft, "tpot_ms": tpot, "e2e_ms": e2e}) + slos = {} + for limit in TPOT_SLOS_MS: + passed = sum(row["ttft_ms"] <= TTFT_SLO_MS and row["tpot_ms"] <= limit for row in requests) + pass_rate = passed / len(requests) + slos[f"tpot_{int(limit)}ms"] = { + "passed": passed, + "pass_rate": pass_rate, + "feasible": pass_rate >= TARGET_PASS_RATE, + } + return {"requests": requests, "slos": slos} + + +def main() -> None: + args = parse_args() + args.frontier_source = args.frontier_source.resolve() + args.profile_root = args.profile_root.resolve() + # A venv interpreter is commonly a symlink to the system executable. + # Keep the venv path so Python discovers that environment's site-packages. + args.python = args.python.absolute() + args.output_root = args.output_root.resolve() + if args.requests < 2: + raise ValueError("requests must be at least two") + rates = tuple(args.rate or RATES) + if any(rate <= 0 for rate in rates) or len(set(rates)) != len(rates): + raise ValueError("rates must be positive and unique") + selected = list(GRID) + if args.config: + names = set(args.config) + selected = [config for config in GRID if config.name in names] + if {config.name for config in selected} != names: + raise ValueError(f"unknown configs: {names - {config.name for config in selected}}") + + paths = profile_paths(args.profile_root) + coverage = validate_attention_coverage(paths["attention"]) + args.output_root.mkdir(parents=True, exist_ok=True) + traces = {} + for rate in rates: + path = args.output_root / "traces" / f"{rate_key(rate)}.csv" + write_trace(path, request_count=args.requests, rate=rate) + traces[rate] = path + + config_results = [] + environment = dict(os.environ) + environment.update( + { + "PYTHONPATH": str(args.frontier_source), + "WANDB_DISABLED": "true", + "VIDUR_DISABLE_WANDB": "1", + # The best-effort source emits a per-layer OP-TRACE at INFO. It is + # diagnostic only and can produce hundreds of MiB per T0 cell. + "FRONTIER_LOG_LEVEL": "WARNING", + } + ) + for config in selected: + loads = [] + for rate in rates: + run_dir = args.output_root / "runs" / config.name / rate_key(rate) + result_path = run_dir / "result.json" + if result_path.is_file(): + result = json.loads(result_path.read_text()) + if result.get("status") == "completed": + loads.append(result) + continue + run_dir.mkdir(parents=True, exist_ok=True) + command = build_command(args, paths, config, traces[rate], run_dir) + write_json(run_dir / "command.json", command) + started = time.time() + with (run_dir / "stdout.log").open("w") as output: + completed = subprocess.run( + command, + cwd=args.frontier_source, + env=environment, + stdout=output, + stderr=subprocess.STDOUT, + timeout=args.subprocess_timeout_seconds, + check=False, + text=True, + ) + if completed.returncode != 0: + raise RuntimeError(f"Frontier failed: {config.name} rate={rate}, rc={completed.returncode}") + metrics = find_metrics(run_dir) + result = { + "status": "completed", + "config": asdict(config) | {"name": config.name}, + "offered_request_rate": rate, + "request_rate_per_gpu": rate / config.tp, + "elapsed_seconds": time.time() - started, + "trace_sha256": sha256(traces[rate]), + "request_metrics_sha256": sha256(metrics), + **score(metrics, args.requests), + } + write_json(result_path, result) + loads.append(result) + print(json.dumps({"config": config.name, "rate": rate, "elapsed_seconds": result["elapsed_seconds"], "slos": result["slos"]}, sort_keys=True), flush=True) + config_results.append({"config": asdict(config) | {"name": config.name}, "loads": loads}) + + rankings = {} + for slo in (f"tpot_{int(value)}ms" for value in TPOT_SLOS_MS): + records = [] + for item in config_results: + feasible = [load["offered_request_rate"] for load in item["loads"] if load["slos"][slo]["feasible"]] + capacity = max(feasible) if feasible else None + records.append({ + "config": item["config"], + "maximum_tested_feasible_request_rate": capacity, + "maximum_tested_feasible_request_rate_per_gpu": capacity / item["config"]["tp"] if capacity is not None else None, + "lower_censored": capacity is None, + "upper_censored": capacity == max(rates) if capacity is not None else False, + }) + records.sort(key=lambda row: (-(row["maximum_tested_feasible_request_rate_per_gpu"] if row["maximum_tested_feasible_request_rate_per_gpu"] is not None else -1), row["config"]["name"])) + rankings[slo] = records + + is_complete_preregistered_surface = ( + selected == list(GRID) and rates == RATES and args.requests == 64 + ) + manifest = { + "schema": "frontier-qwen235b-t0-surface-v1", + "status": ( + "frozen_before_real_surface" + if is_complete_preregistered_surface + else "partial_surface_not_decision_bearing" + ), + "contract": {"requests_per_anchor": args.requests, "rates": rates, "input_tokens": 2048, "output_tokens": 128, "ttft_slo_ms": TTFT_SLO_MS, "tpot_slos_ms": TPOT_SLOS_MS, "target_pass_rate": TARGET_PASS_RATE, "prefix_caching": False}, + "frontier": { + "source": str(args.frontier_source), + "declared_upstream_commit": FRONTIER_DECLARED_UPSTREAM_COMMIT, + "python_and_config_tree_sha256": FRONTIER_PYTHON_CONFIG_TREE_SHA256, + "fingerprint_source": "prefill-grid-v3 frozen run manifest for the same immutable source snapshot", + }, + "runner": {"path": str(Path(__file__).resolve()), "sha256": sha256(Path(__file__).resolve())}, + "profiles": {"root": str(args.profile_root), "coverage": coverage, "files_sha256": {name: sha256(path) for name, path in paths.items()}}, + "config_results": config_results, + "rankings": rankings, + } + write_json(args.output_root / "frontier_surface_frozen.json", manifest) + print(args.output_root / "frontier_surface_frozen.json") + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/run_t0_full_attention_profile.sh b/runs/frontier-multicase-sufficiency-v1/run_t0_full_attention_profile.sh new file mode 100644 index 0000000..a5eff18 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/run_t0_full_attention_profile.sh @@ -0,0 +1,143 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:-$(pwd)/artifacts/t0-full-attention-profile-20260716}" +FRONTIER_ROOT="${FRONTIER_ROOT:-/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6-best-effort-v6-batched-lanes-r3}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh-frontier-vllm0102-smoke/.venv}" +PROFILE_ROOT="${OUTPUT_ROOT}/profiles" +LOG_DIR="${OUTPUT_ROOT}/logs" +PROVENANCE_DIR="${OUTPUT_ROOT}/provenance" +MODEL="Qwen3-235B-A22B-FP8" + +mkdir -p "${PROFILE_ROOT}" "${LOG_DIR}" "${PROVENANCE_DIR}" +exec > >(tee -a "${LOG_DIR}/profile.log") 2>&1 + +if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then + echo "ERROR: CUDA_VISIBLE_DEVICES must contain the fleet-allocated GPU" >&2 + exit 1 +fi +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES}" +if [[ "${#GPU_IDS[@]}" -ne 1 ]]; then + echo "ERROR: expected exactly one GPU, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "FULL_PROFILE_LAUNCH_ECHO host=$(hostname) gpu=${CUDA_VISIBLE_DEVICES} model=${MODEL} operator=FlashInfer_attention phases=standard_decode,true_mixed TP_workers=4,8 batch_sizes=1,2,4,8,16,32,64,96,128 kv_sizes=2048:2175 true_mixed_prefill_chunk=2048 block=16 measurement=CUDA_EVENT output=${OUTPUT_ROOT} expected_wall=5-10m hard_wall=900s hard_gpu_cap=0.25_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader + +test -x "${VENV_ROOT}/bin/python" +test -f "${FRONTIER_ROOT}/pyproject.toml" +test -f "${FRONTIER_ROOT}/data/config/models/${MODEL}.json" +sha256sum run_t0_full_attention_profile.sh > "${PROVENANCE_DIR}/source.sha256" + +export PYTHONPATH="${FRONTIER_ROOT}" +export TOKENIZERS_PARALLELISM=false +export TORCH_CUDA_ARCH_LIST=9.0 + +cd "${FRONTIER_ROOT}" +timeout --signal=TERM --kill-after=30s 780 \ + "${VENV_ROOT}/bin/python" -m frontier.profiling.attention.main \ + --disable_ray \ + --models "${MODEL}" \ + --num_gpus 1 \ + --max_model_len 40960 \ + --max_seq_len 2176 \ + --min_batch_size 1 \ + --max_batch_size 128 \ + --batch_size_list 1 2 4 8 16 32 64 96 128 \ + --decode_kv_cache_size_list 2048 2064 2080 2096 2112 2128 2144 2160 2175 \ + --num_tensor_parallel_workers 4 8 \ + --max_pipeline_parallel_size 1 \ + --attention_backend FLASHINFER \ + --block_size 16 \ + --enable_true_mixed \ + --true_mixed_prefill_batch_sizes 1 2 4 7 \ + --true_mixed_prefill_chunk_sizes 2048 \ + --true_mixed_decode_batch_sizes 1 2 4 8 16 32 64 96 124 127 \ + --true_mixed_decode_kv_cache_sizes 2048 2112 2175 \ + --true_mixed_prefill_kv_cache_size 0 \ + --device h20 \ + --profile_method cuda_event \ + --output_dir "${PROFILE_ROOT}" \ + --yes + +MODEL_PROFILE_DIR="${PROFILE_ROOT}/compute/h20/${MODEL}" +STANDARD_CSV="${MODEL_PROFILE_DIR}/attention.csv" +TRUE_MIXED_CSV="${MODEL_PROFILE_DIR}/attention_true_mixed.csv" +COMBINED_CSV="${MODEL_PROFILE_DIR}/attention_combined.csv" +test -s "${STANDARD_CSV}" +test -s "${TRUE_MIXED_CSV}" +test -s "${COMBINED_CSV}" + +"${VENV_ROOT}/bin/python" - "${STANDARD_CSV}" "${TRUE_MIXED_CSV}" \ + > "${PROVENANCE_DIR}/coverage.json" <<'PY' +import json +import sys + +import pandas as pd + +standard_path, true_mixed_path = sys.argv[1:] +standard = pd.read_csv(standard_path) +true_mixed = pd.read_csv(true_mixed_path) +decode = standard[standard["is_prefill"] == False] # noqa: E712 +payload = { + "standard_path": standard_path, + "standard_rows": len(standard), + "decode_rows": len(decode), + "decode_rows_by_tp": { + str(int(key)): int(value) + for key, value in decode.groupby("num_tensor_parallel_workers").size().items() + }, + "decode_batch_sizes": sorted(int(value) for value in decode["batch_size"].unique()), + "decode_kv_cache_sizes": sorted( + int(value) for value in decode["kv_cache_size"].unique() + ), + "decode_median_non_null": int( + decode["time_stats.attn_decode.median"].notna().sum() + ), + "true_mixed_path": true_mixed_path, + "true_mixed_rows": len(true_mixed), + "true_mixed_rows_by_tp": { + str(int(key)): int(value) + for key, value in true_mixed.groupby("num_tensor_parallel_workers").size().items() + }, + "true_mixed_decode_median_non_null": int( + true_mixed["time_stats.attn_decode.median"].notna().sum() + ), + "true_mixed_prefill_median_non_null": int( + true_mixed["time_stats.attn_prefill.median"].notna().sum() + ), +} +print(json.dumps(payload, indent=2, sort_keys=True)) +expected_batch_sizes = [1, 2, 4, 8, 16, 32, 64, 96, 128] +expected_kv_sizes = [2048, 2064, 2080, 2096, 2112, 2128, 2144, 2160, 2175] +if payload["decode_rows_by_tp"] != {"4": 81, "8": 81}: + raise SystemExit(1) +if payload["decode_batch_sizes"] != expected_batch_sizes: + raise SystemExit(1) +if payload["decode_kv_cache_sizes"] != expected_kv_sizes: + raise SystemExit(1) +if payload["decode_median_non_null"] != 162: + raise SystemExit(1) +if set(payload["true_mixed_rows_by_tp"]) != {"4", "8"}: + raise SystemExit(1) +if payload["true_mixed_rows"] < 100: + raise SystemExit(1) +if payload["true_mixed_decode_median_non_null"] != payload["true_mixed_rows"]: + raise SystemExit(1) +if payload["true_mixed_prefill_median_non_null"] != payload["true_mixed_rows"]: + raise SystemExit(1) +PY + +sha256sum \ + "${STANDARD_CSV}" \ + "${TRUE_MIXED_CSV}" \ + "${COMBINED_CSV}" \ + "${PROVENANCE_DIR}/coverage.json" \ + "${PROVENANCE_DIR}/source.sha256" \ + > "${PROVENANCE_DIR}/artifacts.sha256" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "T0_FULL_ATTENTION_PROFILE_COMPLETE" diff --git a/runs/frontier-multicase-sufficiency-v1/run_t0_real_config.sh b/runs/frontier-multicase-sufficiency-v1/run_t0_real_config.sh new file mode 100644 index 0000000..80aaedb --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/run_t0_real_config.sh @@ -0,0 +1,137 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT is required}" +TP="${TP:?TP is required}" +MNS="${MNS:?MNS is required}" +MBT="${MBT:?MBT is required}" +RATES="${RATES:?RATES is required}" +SERVER_PORT="${SERVER_PORT:?SERVER_PORT is required}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh-frontier-vllm0102-smoke/.venv}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8}" +SERVED_MODEL="qwen3-235b-t0-surface" +SERVER_PID="" + +mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" +exec > >(tee -a "${OUTPUT_ROOT}/logs/controller.log") 2>&1 + +cleanup() { + if [[ -n "${SERVER_PID}" ]] && kill -0 "${SERVER_PID}" 2>/dev/null; then + kill -TERM -- "-${SERVER_PID}" 2>/dev/null || true + for _ in $(seq 1 30); do + kill -0 "${SERVER_PID}" 2>/dev/null || break + sleep 1 + done + kill -KILL -- "-${SERVER_PID}" 2>/dev/null || true + fi + SERVER_PID="" +} +trap cleanup EXIT INT TERM + +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:-}" +if [[ "${#GPU_IDS[@]}" -ne "${TP}" ]]; then + echo "ERROR: expected ${TP} allocated GPUs, got ${CUDA_VISIBLE_DEVICES:-unset}" >&2 + exit 1 +fi +read -r -a RATE_ARRAY <<< "${RATES}" +if [[ "${#RATE_ARRAY[@]}" -lt 1 ]]; then + echo "ERROR: at least one frozen rate is required" >&2 + exit 1 +fi + +echo "REAL_CONFIG_LAUNCH_ECHO host=$(hostname) gpus=${CUDA_VISIBLE_DEVICES} model=${MODEL_ROOT} config=TP${TP}_MNS${MNS}_MBT${MBT} rates=${RATES// /,} repeats=2 requests_per_anchor=64 isolation=fresh_server_per_anchor target_warmup=min32_max4_ceil_rate_x20 trace=fixed_ISL2048_OSL128 prefix=off runtime=community_vllm_0.10.2 execution=eager kv=BF16 output=${OUTPUT_ROOT}" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +sha256sum run_t0_real_config.sh t0_rate_client.py > "${OUTPUT_ROOT}/provenance/source.sha256" +"${VENV_ROOT}/bin/python" - "${TP}" "${MNS}" "${MBT}" "${RATES}" \ + > "${OUTPUT_ROOT}/provenance/contract.json" <<'PY' +import importlib.metadata as metadata +import json +import platform +import sys + +tp, mns, mbt, rates = sys.argv[1:] +print(json.dumps({ + "python": platform.python_version(), + "torch": metadata.version("torch"), + "transformers": metadata.version("transformers"), + "vllm": metadata.version("vllm"), + "config": {"tp": int(tp), "mns": int(mns), "mbt": int(mbt)}, + "rates": [float(value) for value in rates.split()], + "rounds": 2, + "requests_per_anchor": 64, + "anchor_isolation": "fresh_server_per_rate_per_round", + "target_rate_warmup_requests": "min(32, max(4, ceil(rate * 20)))", + "ttft_slo_ms": 1256.0, + "tpot_slos_ms": [40.0, 120.0, 150.0, 180.0], +}, indent=2, sort_keys=True)) +PY +nvidia-smi --query-gpu=index,name,uuid,driver_version --format=csv,noheader \ + > "${OUTPUT_ROOT}/provenance/gpus.csv" +sha256sum "${MODEL_ROOT}/config.json" > "${OUTPUT_ROOT}/provenance/model.sha256" + +export TOKENIZERS_PARALLELISM=false +export VLLM_USE_V1=1 +export VLLM_ATTENTION_BACKEND=FLASHINFER +export TORCH_CUDA_ARCH_LIST=9.0 + +EXTRA_FLAGS=() +NUM_BLOCKS=26101 +if [[ "${TP}" -eq 8 ]]; then + EXTRA_FLAGS+=(--enable-expert-parallel) + NUM_BLOCKS=62351 +fi + +for ROUND in 1 2; do + ROUND_ROOT="${OUTPUT_ROOT}/round${ROUND}" + mkdir -p "${ROUND_ROOT}/logs" "${ROUND_ROOT}/results" + ORDERED_RATES=("${RATE_ARRAY[@]}") + if [[ "${ROUND}" -eq 2 ]]; then + ORDERED_RATES=() + for ((index=${#RATE_ARRAY[@]}-1; index>=0; index--)); do ORDERED_RATES+=("${RATE_ARRAY[index]}"); done + fi + for RATE in "${ORDERED_RATES[@]}"; do + KEY="$(printf 'r%.2f' "${RATE}" | tr '.' 'p')" + SERVER_LOG="${ROUND_ROOT}/logs/server_${KEY}.log" + setsid "${VENV_ROOT}/bin/vllm" serve "${MODEL_ROOT}" \ + --host 127.0.0.1 --port "${SERVER_PORT}" --served-model-name "${SERVED_MODEL}" \ + --tensor-parallel-size "${TP}" --disable-custom-all-reduce --quantization fp8 \ + --gpu-memory-utilization 0.80 --num-gpu-blocks-override "${NUM_BLOCKS}" \ + --kv-cache-dtype auto --max-model-len 40960 --max-num-batched-tokens "${MBT}" \ + --max-num-seqs "${MNS}" --no-enable-prefix-caching --enable-chunked-prefill \ + --enforce-eager --disable-log-requests "${EXTRA_FLAGS[@]}" \ + > "${SERVER_LOG}" 2>&1 & + SERVER_PID=$! + READY=0 + for _ in $(seq 1 180); do + if curl -fsS --max-time 2 "http://127.0.0.1:${SERVER_PORT}/v1/models" \ + > "${ROUND_ROOT}/results/models_${KEY}.json" 2>/dev/null; then + READY=1 + break + fi + if ! kill -0 "${SERVER_PID}" 2>/dev/null; then tail -200 "${SERVER_LOG}"; exit 1; fi + sleep 5 + done + if [[ "${READY}" -ne 1 ]]; then tail -200 "${SERVER_LOG}"; exit 1; fi + + WARMUP_REQUESTS="$("${VENV_ROOT}/bin/python" - "${RATE}" <<'PY' +import math +import sys +print(min(32, max(4, math.ceil(float(sys.argv[1]) * 20.0)))) +PY +)" + "${VENV_ROOT}/bin/python" t0_rate_client.py --port "${SERVER_PORT}" \ + --served-model "${SERVED_MODEL}" --model-path "${MODEL_ROOT}" --rate "${RATE}" \ + --requests "${WARMUP_REQUESTS}" \ + --output "${ROUND_ROOT}/results/warmup_${KEY}.json" + "${VENV_ROOT}/bin/python" t0_rate_client.py --port "${SERVER_PORT}" \ + --served-model "${SERVED_MODEL}" --model-path "${MODEL_ROOT}" --rate "${RATE}" \ + --requests 64 --output "${ROUND_ROOT}/results/${KEY}.json" + cleanup + done +done + +find "${OUTPUT_ROOT}" -type f ! -path '*/provenance/artifacts.sha256' -print0 \ + | sort -z | xargs -0 sha256sum > "${OUTPUT_ROOT}/provenance/artifacts.sha256" +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "T0_REAL_CONFIG_COMPLETE" diff --git a/runs/frontier-multicase-sufficiency-v1/run_t0_smoke.sh b/runs/frontier-multicase-sufficiency-v1/run_t0_smoke.sh new file mode 100644 index 0000000..710728c --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/run_t0_smoke.sh @@ -0,0 +1,146 @@ +#!/usr/bin/env bash + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:-$(pwd)/artifacts/t0-smoke-20260716}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh-frontier-vllm0102-smoke/.venv}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8}" +FRONTIER_ROOT="${FRONTIER_ROOT:-/home/admin/cpfs/wjh/frontier-community-qwen235-smoke-20260715/Frontier-d9cfeb6}" +SERVER_PORT="${SERVER_PORT:-18910}" +SERVED_MODEL="qwen3-235b-t0-smoke" +LOG_DIR="${OUTPUT_ROOT}/logs" +RESULT_DIR="${OUTPUT_ROOT}/results" +PROVENANCE_DIR="${OUTPUT_ROOT}/provenance" +SERVER_PID="" + +mkdir -p "${LOG_DIR}" "${RESULT_DIR}" "${PROVENANCE_DIR}" +exec > >(tee -a "${LOG_DIR}/smoke.log") 2>&1 + +cleanup() { + if [[ -n "${SERVER_PID}" ]] && kill -0 "${SERVER_PID}" 2>/dev/null; then + kill -TERM -- "-${SERVER_PID}" 2>/dev/null || true + for _ in $(seq 1 30); do + if ! kill -0 "${SERVER_PID}" 2>/dev/null; then + break + fi + sleep 1 + done + kill -KILL -- "-${SERVER_PID}" 2>/dev/null || true + fi +} +trap cleanup EXIT INT TERM + +if [[ -z "${CUDA_VISIBLE_DEVICES:-}" ]]; then + echo "ERROR: CUDA_VISIBLE_DEVICES must contain the four fleet-allocated GPUs" >&2 + exit 1 +fi +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES}" +if [[ "${#GPU_IDS[@]}" -ne 4 ]]; then + echo "ERROR: expected exactly four GPUs, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +echo "LAUNCH_ECHO host=$(hostname) gpus=${CUDA_VISIBLE_DEVICES} model=${MODEL_ROOT} trace=fixed_ISL2048_OSL128 prefix=off qps=single_then_concurrency2 runtime=community_vllm_0.10.2 topology=TP4_DP1 execution=eager kv=BF16 spec=off cuda_graph=off output=${OUTPUT_ROOT} expected_wall=20-30m hard_wall=1800s hard_gpu_cap=2_H20h" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader + +test -x "${VENV_ROOT}/bin/vllm" +test -x "${VENV_ROOT}/bin/python" +test -f "${MODEL_ROOT}/config.json" +test -f "${FRONTIER_ROOT}/pyproject.toml" +test -f "$(pwd)/t0_smoke_client.py" + +sha256sum run_t0_smoke.sh t0_smoke_client.py > "${PROVENANCE_DIR}/source.sha256" +"${VENV_ROOT}/bin/python" - <<'PY' > "${PROVENANCE_DIR}/environment.json" +import importlib.metadata as metadata +import json +import platform + +print(json.dumps({ + "python": platform.python_version(), + "torch": metadata.version("torch"), + "transformers": metadata.version("transformers"), + "vllm": metadata.version("vllm"), +}, indent=2, sort_keys=True)) +PY + +export TOKENIZERS_PARALLELISM=false +export VLLM_USE_V1=1 +export VLLM_ATTENTION_BACKEND=FLASHINFER +export TORCH_CUDA_ARCH_LIST=9.0 + +echo "STAGE server_start" +setsid "${VENV_ROOT}/bin/vllm" serve "${MODEL_ROOT}" \ + --host 127.0.0.1 \ + --port "${SERVER_PORT}" \ + --served-model-name "${SERVED_MODEL}" \ + --tensor-parallel-size 4 \ + --disable-custom-all-reduce \ + --quantization fp8 \ + --gpu-memory-utilization 0.80 \ + --kv-cache-dtype auto \ + --max-model-len 40960 \ + --max-num-batched-tokens 8192 \ + --max-num-seqs 64 \ + --no-enable-prefix-caching \ + --enable-chunked-prefill \ + --enforce-eager \ + --disable-log-requests \ + > "${LOG_DIR}/server.log" 2>&1 & +SERVER_PID=$! + +READY=0 +for _ in $(seq 1 180); do + if curl -fsS --max-time 2 "http://127.0.0.1:${SERVER_PORT}/v1/models" \ + > "${RESULT_DIR}/models.json" 2>/dev/null; then + READY=1 + break + fi + if ! kill -0 "${SERVER_PID}" 2>/dev/null; then + echo "ERROR: vLLM server exited before readiness" >&2 + tail -200 "${LOG_DIR}/server.log" >&2 || true + exit 1 + fi + sleep 5 +done +if [[ "${READY}" -ne 1 ]]; then + echo "ERROR: vLLM server did not become ready within 900 seconds" >&2 + tail -200 "${LOG_DIR}/server.log" >&2 || true + exit 1 +fi + +echo "STAGE fixed_shape_single" +"${VENV_ROOT}/bin/python" t0_smoke_client.py \ + --port "${SERVER_PORT}" \ + --served-model "${SERVED_MODEL}" \ + --model-path "${MODEL_ROOT}" \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 1 \ + --requests 1 \ + --output "${RESULT_DIR}/single.json" + +echo "STAGE fixed_shape_concurrency2" +"${VENV_ROOT}/bin/python" t0_smoke_client.py \ + --port "${SERVER_PORT}" \ + --served-model "${SERVED_MODEL}" \ + --model-path "${MODEL_ROOT}" \ + --input-tokens 2048 \ + --output-tokens 128 \ + --concurrency 2 \ + --requests 2 \ + --output "${RESULT_DIR}/concurrency2.json" + +cleanup +SERVER_PID="" + +sha256sum \ + "${PROVENANCE_DIR}/environment.json" \ + "${PROVENANCE_DIR}/source.sha256" \ + "${RESULT_DIR}/models.json" \ + "${RESULT_DIR}/single.json" \ + "${RESULT_DIR}/concurrency2.json" \ + > "${PROVENANCE_DIR}/artifacts.sha256" +nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "T0_SMOKE_COMPLETE" diff --git a/runs/frontier-multicase-sufficiency-v1/t0-smoke-report.md b/runs/frontier-multicase-sufficiency-v1/t0-smoke-report.md new file mode 100644 index 0000000..8b7914c --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/t0-smoke-report.md @@ -0,0 +1,89 @@ +# Qwen235B T0 fixed-shape smoke report + +日期：2026-07-16。状态：real 与 profile-closed Frontier smoke complete；不是 config-rank evaluation。 + +## Setup + +| 项目 | 设置 | +|---|---| +| host | `dash0`，仅 GPU 0--3；GPU 4--7 始终空闲 | +| model | `Qwen/Qwen3-235B-A22B-FP8` | +| runtime | community vLLM 0.10.2、torch 2.8.0、Transformers 4.55.2 | +| config | TP4/DP1、MNS64、MBT8192、block16、26,101 KV blocks/GPU | +| execution | FlashInfer、eager、BF16 KV、prefix/spec/CUDA graph off | +| workload | exact prompt token IDs；ISL=2,048、OSL=128；single 和两个同时到达 requests | +| SLO | TTFT ≤1,256 ms、TPOT ≤40 ms，joint pass | +| GPU cost | 02:33:48--02:36:01 UTC，133 s × 4 H20 = 0.148 H20-hours | + +vLLM 报告每卡 model weights 约55.13 GiB、KV capacity 417,616 tokens，以及 `num_gpu_blocks=26101`。这与此前输入 Frontier 的 TP4 KV capacity 一致。 + +## Real result + +| load | request | usage | TTFT | TPOT | E2E | joint pass | +|---|---:|---|---:|---:|---:|---:| +| single | 0 | 2,048+128 | 586.23 ms | 136.18 ms | 17,881.30 ms | no | +| concurrency=2 | 0 | 2,048+128 | 186.09 ms | 124.87 ms | 16,045.24 ms | no | +| concurrency=2 | 1 | 2,048+128 | 434.29 ms | 123.78 ms | 16,154.03 ms | no | + +所有 response 的 usage completion tokens 和逐 chunk token IDs 都是128；不是 EOS 提前退出或 client length-hint 误差。artifact SHA256 全部通过 harvest 后复验。 + +单请求已经违反40 ms TPOT，说明该 SLO 对 community/eager controlled stack 在最低负载下不可行。concurrency=2 的 batching 将平均 TPOT 改善到124.33 ms，但仍不足以通过。这是 pilot finding，不应用于事后挑选一个恰好产生所需 ranking 的阈值。 + +## Frontier result + +Frontier 使用冻结的 best-effort source： + +```text +upstream commit: d9cfeb6d8791fbf2f295dd9744c56a666171776e +source/config tree SHA256: 172fc7ae19c40e67030208ff488d0d5d90764888ed76a7a543be6037ba62dc11 +profile root: profiles-best-effort-final-v2 +TP4/DP1/MoE-TP4/EP1, MNS64, MBT8192, block16, 26101 blocks +``` + +simulator 正确加载1个请求，执行2,048-token prefill，并提交第一个 decode token。下一轮成为 pure-decode batch 时失败： + +```text +Skipping eager attn_decode training: no standard decode rows +ValueError: attention decode prediction cache not found for cluster monolithic +``` + +profile audit 确认 final attention CSV 有726 rows，但 `is_prefill` 只有 `True`。另外两个已有 Qwen235 attention files 也只有 prefill rows。由于没有产生完整 request metrics，本文不报告 simulator TTFT/TPOT，也不把 crash 当成 SLO failure。使用 dummy/fallback latency 会把缺失 measurement coverage 隐藏成虚假的 simulator prediction。 + +## Profile closure 与 rerun + +在 `dash0` 单张 H20 上，用同一 vLLM 0.10.2/FlashInfer stack 测量 TP4、batch `{1,2}`、KV `{2048,2176}` 的4个 CUDA-event decode-attention points；4/4 rows 的 `attn_decode.median` 非空。它们与原726个 prefill rows 按完全相同的55-column schema 合并到新的 immutable profile root，未覆盖原 profile，也未加入 dummy 或 E2E calibration。 + +同一 Frontier smoke 随后精确完成1/2个请求： + +| load | request | TTFT | TPOT | E2E | +|---|---:|---:|---:|---:| +| single | 0 | 267.67 ms | 88.27 ms | 11,478.45 ms | +| concurrency=2 | 0 | 470.66 ms | 90.54 ms | 11,969.73 ms | +| concurrency=2 | 1 | 470.66 ms | 90.54 ms | 11,969.73 ms | + +请求数、ISL/OSL 与 real contract 完全一致，所有 latency 均有限且非负，representation gate 因而通过。absolute TPOT error 并非固定比例：single 低估约35%，concurrency=2 低估约27%。这组数据只能证明 simulator 现在能表达该 state path，不能证明 config rank 正确。 + +steady-QPS 下还会出现 prefill+decode true-mixed batches，因此 full surface 没有沿用4-row smoke profile。后续完整 closure 覆盖 TP4/TP8 各81个 standard-decode 与108个 true-mixed points；合并后的 attention root 共1,104 rows，并已确认 `attn_decode_in_mixed` 从每个 TP 的108个真实 samples 训练。 + +## Interpretation + +Change：首次把原 prefill-only compatibility envelope用于有真实 decode tokens 的最简单 fixed-shape case。 + +Expected effect：如果 profiles 和 execution model 已闭合，Frontier 应至少产生同一 config 的 TTFT/TPOT，之后才能讨论绝对 gap 或 rank。 + +Verification：real exact-token streaming run；Frontier 同 config/blocks/profile run；profile CSV 与完整 traceback 审计。 + +Result：real serving 成功；原 Frontier compatibility envelope 在 pure-decode attention 处断裂；补齐最小 measurement coverage 后 smoke 通过，但出现27%--35%的 TPOT absolute error。因此此前的 prefill rank-match 仍不能支持“Frontier 已足够解决 Qwen235 mixed config selection”的外推，必须看完整 config response surface。 + +Remaining risk：完整 profile 已加入 mixed-attention coverage，但 MoE decode routing context、EP8 communication、batch-dependent TPOT 和 simulator/real action ranking 仍需由 full surface 检验；smoke 只关闭第一个必要缺口，并未证明它是唯一缺口。 + +## Evidence + +- Real single request：[single.json](fleet-artifacts/qwen235b-t0-tp4-smoke-20260716-v1-20260716T023346844896Z/artifacts/artifacts/t0-smoke-20260716/results/single.json) +- Real concurrency=2：[concurrency2.json](fleet-artifacts/qwen235b-t0-tp4-smoke-20260716-v1-20260716T023346844896Z/artifacts/artifacts/t0-smoke-20260716/results/concurrency2.json) +- Server log：[server.log](fleet-artifacts/qwen235b-t0-tp4-smoke-20260716-v1-20260716T023346844896Z/artifacts/artifacts/t0-smoke-20260716/logs/server.log) +- Frontier command：[command.json](frontier-smoke-failure/single/command.json) +- Frontier traceback：[stdout.log](frontier-smoke-failure/single/stdout.log) +- Frontier trace：[trace.csv](frontier-smoke-failure/single/trace.csv) +- Minimal decode profile coverage：[coverage.json](fleet-artifacts/qwen235b-decode-attention-profile-20260716-v1-20260716T030929269383Z/artifacts/artifacts/decode-attention-profile-20260716/provenance/coverage.json) +- Full attention profile coverage：[coverage.json](fleet-artifacts/qwen235b-t0-full-attention-profile-20260716-v1-20260716T032536186518Z/artifacts/artifacts/t0-full-attention-profile-20260716/provenance/coverage.json) diff --git a/runs/frontier-multicase-sufficiency-v1/t0_rate_client.py b/runs/frontier-multicase-sufficiency-v1/t0_rate_client.py new file mode 100644 index 0000000..2520deb --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/t0_rate_client.py @@ -0,0 +1,223 @@ +#!/usr/bin/env python3 +"""Open-loop fixed-shape completion workload for one T0 offered-load anchor.""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import hashlib +import http.client +import json +import math +import statistics +import time +from pathlib import Path +from typing import Any + + +TPOT_SLOS_MS = (40.0, 120.0, 150.0, 180.0) +TTFT_SLO_MS = 1256.0 + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, required=True) + parser.add_argument("--served-model", required=True) + parser.add_argument("--model-path", type=Path, required=True) + parser.add_argument("--rate", type=float, required=True) + parser.add_argument("--requests", type=int, default=64) + parser.add_argument("--input-tokens", type=int, default=2048) + parser.add_argument("--output-tokens", type=int, default=128) + parser.add_argument("--timeout-seconds", type=float, default=900.0) + parser.add_argument("--output", type=Path, required=True) + return parser.parse_args() + + +def percentile(values: list[float], fraction: float) -> float | None: + if not values: + return None + ordered = sorted(values) + index = min(len(ordered) - 1, max(0, math.ceil(fraction * len(ordered)) - 1)) + return ordered[index] + + +def run_request( + *, + request_index: int, + scheduled_at: float, + benchmark_start: float, + args: argparse.Namespace, + prompt_ids: list[int], +) -> dict[str, Any]: + delay = scheduled_at - time.perf_counter() + if delay > 0: + time.sleep(delay) + admitted = time.perf_counter() + record: dict[str, Any] = { + "request_index": request_index, + "scheduled_s": scheduled_at - benchmark_start, + "admitted_s": admitted - benchmark_start, + "admission_lag_ms": (admitted - scheduled_at) * 1000.0, + "success": False, + } + connection = http.client.HTTPConnection(args.host, args.port, timeout=args.timeout_seconds) + body = { + "model": args.served_model, + "prompt": prompt_ids, + "min_tokens": args.output_tokens, + "max_tokens": args.output_tokens, + "ignore_eos": True, + "temperature": 0, + "stream": True, + "stream_options": {"include_usage": True}, + "return_token_ids": True, + } + try: + started = time.perf_counter() + connection.request( + "POST", + "/v1/completions", + body=json.dumps(body, separators=(",", ":")).encode(), + headers={"Content-Type": "application/json"}, + ) + response = connection.getresponse() + if response.status != 200: + raise RuntimeError(f"HTTP {response.status}: {response.read().decode(errors='replace')}") + first_token_at = None + last_token_at = None + streamed_tokens = 0 + usage = None + while True: + raw = response.readline() + if not raw: + break + line = raw.decode(errors="replace").strip() + if not line.startswith("data:"): + continue + data = line[5:].strip() + if data == "[DONE]": + break + payload = json.loads(data) + if payload.get("usage"): + usage = payload["usage"] + emitted = 0 + for choice in payload.get("choices") or []: + token_ids = choice.get("token_ids") or [] + emitted += len(token_ids) if token_ids else int(bool(choice.get("text"))) + if emitted: + now = time.perf_counter() + first_token_at = first_token_at or now + last_token_at = now + streamed_tokens += emitted + finished = time.perf_counter() + if first_token_at is None or last_token_at is None or usage is None: + raise RuntimeError("missing streaming tokens or usage") + prompt_tokens = int(usage["prompt_tokens"]) + completion_tokens = int(usage["completion_tokens"]) + if prompt_tokens != args.input_tokens or completion_tokens != args.output_tokens: + raise RuntimeError(f"usage mismatch: {prompt_tokens}+{completion_tokens}") + ttft = (first_token_at - started) * 1000.0 + tpot = (last_token_at - first_token_at) * 1000.0 / (completion_tokens - 1) + record.update( + { + "success": True, + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "streamed_token_count": streamed_tokens, + "ttft_ms": ttft, + "tpot_ms": tpot, + "e2e_ms": (finished - started) * 1000.0, + } + ) + except Exception as error: # Preserve failed requests as SLO failures. + record["error"] = f"{type(error).__name__}: {error}" + finally: + connection.close() + return record + + +def main() -> None: + args = parse_args() + if args.rate <= 0 or args.requests <= 0: + raise ValueError("rate and requests must be positive") + from transformers import AutoTokenizer + + tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True) + excluded = set(tokenizer.all_special_ids) + candidates = [token_id for token_id in range(tokenizer.vocab_size) if token_id not in excluded] + if len(candidates) < args.requests + 1: + raise RuntimeError("tokenizer has too few non-special token IDs") + base_id = candidates[0] + prompts = [ + [candidates[index + 1], *([base_id] * (args.input_tokens - 1))] + for index in range(args.requests) + ] + prompt_hash = hashlib.sha256( + "\n".join(",".join(map(str, prompt)) for prompt in prompts).encode() + ).hexdigest() + + benchmark_start = time.perf_counter() + 2.0 + with concurrent.futures.ThreadPoolExecutor(max_workers=args.requests) as pool: + futures = [ + pool.submit( + run_request, + request_index=index, + scheduled_at=benchmark_start + index / args.rate, + benchmark_start=benchmark_start, + args=args, + prompt_ids=prompts[index], + ) + for index in range(args.requests) + ] + requests = [future.result() for future in futures] + requests.sort(key=lambda row: int(row["request_index"])) + completed = [row for row in requests if row["success"]] + slos = {} + for limit in TPOT_SLOS_MS: + passed = sum( + row["success"] + and float(row["ttft_ms"]) <= TTFT_SLO_MS + and float(row["tpot_ms"]) <= limit + for row in requests + ) + slos[f"tpot_{int(limit)}ms"] = { + "passed": passed, + "pass_rate": passed / len(requests), + "feasible": passed / len(requests) >= 0.95, + } + ttfts = [float(row["ttft_ms"]) for row in completed] + tpots = [float(row["tpot_ms"]) for row in completed] + payload = { + "schema": "qwen235b-t0-rate-anchor-v1", + "workload": { + "offered_request_rate": args.rate, + "request_count": args.requests, + "input_tokens": args.input_tokens, + "output_tokens": args.output_tokens, + "prefix_caching": False, + "arrival": "open_loop_uniform", + "last_scheduled_arrival_s": (args.requests - 1) / args.rate, + "prompt_vector_sha256": prompt_hash, + }, + "summary": { + "completed": len(completed), + "failed": len(requests) - len(completed), + "ttft_p50_ms": percentile(ttfts, 0.50), + "ttft_p95_ms": percentile(ttfts, 0.95), + "tpot_p50_ms": percentile(tpots, 0.50), + "tpot_p95_ms": percentile(tpots, 0.95), + "admission_lag_max_ms": max(float(row["admission_lag_ms"]) for row in requests), + "slos": slos, + }, + "requests": requests, + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + print(json.dumps(payload["summary"], sort_keys=True), flush=True) + if len(completed) != args.requests: + raise SystemExit(2) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/t0_smoke_client.py b/runs/frontier-multicase-sufficiency-v1/t0_smoke_client.py new file mode 100644 index 0000000..4efccf4 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/t0_smoke_client.py @@ -0,0 +1,200 @@ +#!/usr/bin/env python3 +"""Issue exact fixed-shape completion requests and record streaming latency.""" + +from __future__ import annotations + +import argparse +import concurrent.futures +import http.client +import json +import statistics +import threading +import time +from pathlib import Path +from typing import Any + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, required=True) + parser.add_argument("--served-model", required=True) + parser.add_argument("--model-path", type=Path, required=True) + parser.add_argument("--input-tokens", type=int, default=2048) + parser.add_argument("--output-tokens", type=int, default=128) + parser.add_argument("--concurrency", type=int, required=True) + parser.add_argument("--requests", type=int, required=True) + parser.add_argument("--timeout-seconds", type=float, default=600.0) + parser.add_argument("--output", type=Path, required=True) + return parser.parse_args() + + +def run_request( + *, + request_index: int, + args: argparse.Namespace, + prompt_token_id: int, + start_barrier: threading.Barrier, +) -> dict[str, Any]: + body = { + "model": args.served_model, + "prompt": [prompt_token_id] * args.input_tokens, + "min_tokens": args.output_tokens, + "max_tokens": args.output_tokens, + "ignore_eos": True, + "temperature": 0, + "stream": True, + "stream_options": {"include_usage": True}, + "return_token_ids": True, + } + encoded = json.dumps(body, separators=(",", ":")).encode() + connection = http.client.HTTPConnection( + args.host, args.port, timeout=args.timeout_seconds + ) + start_barrier.wait() + started = time.perf_counter() + connection.request( + "POST", + "/v1/completions", + body=encoded, + headers={"Content-Type": "application/json"}, + ) + response = connection.getresponse() + if response.status != 200: + detail = response.read().decode(errors="replace") + raise RuntimeError(f"request {request_index} failed: HTTP {response.status}: {detail}") + + first_token_at: float | None = None + last_token_at: float | None = None + streamed_token_count = 0 + usage: dict[str, Any] | None = None + while True: + raw = response.readline() + if not raw: + break + line = raw.decode(errors="replace").strip() + if not line.startswith("data:"): + continue + data = line[5:].strip() + if data == "[DONE]": + break + payload = json.loads(data) + if payload.get("usage"): + usage = payload["usage"] + emitted = 0 + for choice in payload.get("choices") or []: + token_ids = choice.get("token_ids") or [] + if token_ids: + emitted += len(token_ids) + elif choice.get("text"): + emitted += 1 + if emitted: + now = time.perf_counter() + if first_token_at is None: + first_token_at = now + last_token_at = now + streamed_token_count += emitted + finished = time.perf_counter() + connection.close() + + if first_token_at is None or last_token_at is None or usage is None: + raise RuntimeError( + f"request {request_index} missing streaming token or usage metadata" + ) + prompt_tokens = int(usage["prompt_tokens"]) + completion_tokens = int(usage["completion_tokens"]) + if prompt_tokens != args.input_tokens or completion_tokens != args.output_tokens: + raise RuntimeError( + f"request {request_index} usage mismatch: prompt={prompt_tokens}, " + f"completion={completion_tokens}" + ) + + ttft_ms = (first_token_at - started) * 1000.0 + tpot_ms = ( + (last_token_at - first_token_at) * 1000.0 / (completion_tokens - 1) + if completion_tokens > 1 + else 0.0 + ) + return { + "request_index": request_index, + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "streamed_token_count": streamed_token_count, + "ttft_ms": ttft_ms, + "tpot_ms": tpot_ms, + "e2e_ms": (finished - started) * 1000.0, + "ttft_slo_ms": 1000.0 + args.input_tokens / 8.0, + "tpot_slo_ms": 40.0, + "joint_slo_pass": ttft_ms <= 1000.0 + args.input_tokens / 8.0 + and tpot_ms <= 40.0, + } + + +def percentile(values: list[float], fraction: float) -> float: + ordered = sorted(values) + index = min(len(ordered) - 1, max(0, round((len(ordered) - 1) * fraction))) + return ordered[index] + + +def main() -> None: + args = parse_args() + if args.concurrency <= 0 or args.requests < args.concurrency: + raise ValueError("requests must be at least concurrency, and both must be positive") + if args.input_tokens <= 0 or args.output_tokens <= 0: + raise ValueError("token lengths must be positive") + + from transformers import AutoTokenizer + + tokenizer = AutoTokenizer.from_pretrained(args.model_path, trust_remote_code=True) + candidate_ids = tokenizer.encode(" hello", add_special_tokens=False) + if not candidate_ids: + raise RuntimeError("tokenizer returned no prompt token id") + prompt_token_id = int(candidate_ids[0]) + + results: list[dict[str, Any]] = [] + for batch_start in range(0, args.requests, args.concurrency): + batch_count = min(args.concurrency, args.requests - batch_start) + barrier = threading.Barrier(batch_count) + with concurrent.futures.ThreadPoolExecutor(max_workers=batch_count) as pool: + futures = [ + pool.submit( + run_request, + request_index=batch_start + offset, + args=args, + prompt_token_id=prompt_token_id, + start_barrier=barrier, + ) + for offset in range(batch_count) + ] + results.extend(future.result() for future in futures) + + ttfts = [float(row["ttft_ms"]) for row in results] + tpots = [float(row["tpot_ms"]) for row in results] + payload = { + "schema": "qwen235b-t0-smoke-v1", + "workload": { + "input_tokens": args.input_tokens, + "output_tokens": args.output_tokens, + "uniform_qps": None, + "prefix_caching": False, + "concurrency": args.concurrency, + "request_count": args.requests, + "prompt_token_id": prompt_token_id, + }, + "summary": { + "completed_requests": len(results), + "joint_slo_pass_count": sum(bool(row["joint_slo_pass"]) for row in results), + "ttft_mean_ms": statistics.fmean(ttfts), + "ttft_p95_ms": percentile(ttfts, 0.95), + "tpot_mean_ms": statistics.fmean(tpots), + "tpot_p95_ms": percentile(tpots, 0.95), + }, + "requests": sorted(results, key=lambda row: int(row["request_index"])), + } + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + print(json.dumps(payload["summary"], sort_keys=True), flush=True) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-multicase-sufficiency-v1/trace-contract-audit.json b/runs/frontier-multicase-sufficiency-v1/trace-contract-audit.json new file mode 100644 index 0000000..6f5df00 --- /dev/null +++ b/runs/frontier-multicase-sufficiency-v1/trace-contract-audit.json @@ -0,0 +1,41 @@ +{ + "schema": "qwen235b-trace-contract-audit-v1", + "status": "pass_offline_source_contract", + "recorded_at": "2026-07-16", + "execution": { + "host": "dash0", + "device": "cpu_only", + "elapsed_seconds": 45.643, + "tokenizer_class": "Qwen2TokenizerFast", + "transformers_version": "4.55.2", + "model_path": "/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8" + }, + "trace": { + "path": "/home/admin/cpfs/wjh/aituner/aituner/trace_windows/traces/thinking_w20260327_1000.jsonl", + "sha256": "f878e9af18f94dcfaced94a8e1e6b20a2f7d97d64aa862448025660dbbd965b2", + "source_request_count": 15479, + "eligible_request_count": 15401, + "eligibility": "input_length + output_length <= 40960 and output_length > 0" + }, + "tokenization": { + "eligible_request_count": 15401, + "total_token_count": 55057919, + "input_length_mismatch_count": 0, + "length_order_sha256": "8bb1c9b7278b261fe1be695ff16dd0c4735d63bd0b1830b612953961fefc370a", + "per_request_token_digest_sha256": "b15b01c00042813c76701ffcc75fc0facde3c01806e167afdc69b5b5861e2381" + }, + "source_hash_contract": { + "source_block_size_tokens": 64, + "full_block_count": 852407, + "partial_block_count": 15131, + "unique_hash_id_count": 509437, + "unique_parent_chunk_key_count": 509437, + "hash_id_to_parent_chunk_conflict_count": 0, + "parent_chunk_to_hash_id_conflict_count": 0, + "key_definition": "(parent source hash id, BLAKE2b-128 of the tokenizer token-id chunk)" + }, + "interpretation": { + "established": "The exact prompts reproduce input_length, and source hash ids preserve the tokenizer-visible parent/chunk prefix-equivalence relation over the eligible universe.", + "not_yet_established": "Runtime parity of vLLM and Frontier block-size-16 computed, hit, allocated, eviction, and placement counters." + } +} diff --git a/runs/frontier-slo-alignment-v0/analyze.py b/runs/frontier-slo-alignment-v0/analyze.py new file mode 100644 index 0000000..934ce42 --- /dev/null +++ b/runs/frontier-slo-alignment-v0/analyze.py @@ -0,0 +1,653 @@ +#!/usr/bin/env python3 +"""Re-evaluate Frontier and real replay with one SLO-feasible throughput objective. + +The analysis is deliberately paired: a cell's capacity is the largest offered +load that satisfies the same request-level SLO among the anchors observed by +both systems. It does not extrapolate beyond the common anchor grid. +""" + +from __future__ import annotations + +import argparse +import csv +import hashlib +import json +import math +import subprocess +import sys +from collections import Counter +from datetime import datetime, timezone +from pathlib import Path +from typing import Any, Iterable + + +SCHEMA = "frontier-slo-alignment-v0" +MODE = "frozen-calibrated" +READING = "paired-grid-slo-feasible-max-offered-throughput" +EXPECTED_CELLS = { + f"tp{tp}_mns{mns}" for tp in (1, 2, 4) for mns in (8, 16, 32, 64) +} + + +def load_json(path: Path) -> dict[str, Any]: + with path.open(encoding="utf-8") as source: + value = json.load(source) + if not isinstance(value, dict): + raise ValueError(f"expected JSON object: {path}") + return value + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as source: + for chunk in iter(lambda: source.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def git_revision(path: Path) -> dict[str, str]: + def run(*arguments: str) -> str: + return subprocess.run( + ["git", *arguments], + cwd=path, + check=True, + capture_output=True, + text=True, + ).stdout.strip() + + return {"head": run("rev-parse", "HEAD"), "status_short": run("status", "--short")} + + +def load_real_cells(ground_truth_path: Path) -> dict[str, dict[str, Any]]: + ground_truth = load_json(ground_truth_path) + cells = {} + for cell in ground_truth["cells"]: + cell_id = str(cell["cell_id"]) + if cell_id in cells: + raise ValueError(f"duplicate real cell: {cell_id}") + probes = {int(probe["probe_index"]): probe for probe in cell["probe_history"]} + if len(probes) != len(cell["probe_history"]): + raise ValueError(f"duplicate real probe index: {cell_id}") + cells[cell_id] = {**cell, "probes": probes} + if set(cells) != EXPECTED_CELLS: + raise ValueError(f"unexpected real cells: {sorted(cells)}") + return cells + + +def load_frontier_runs(results_dir: Path, mode: str = MODE) -> dict[tuple[str, int], dict[str, Any]]: + runs: dict[tuple[str, int], dict[str, Any]] = {} + manifests = sorted((results_dir / "raw").glob("*/trial-*/run_manifest.json")) + for manifest_path in manifests: + manifest = load_json(manifest_path) + run = manifest["run"] + if run["mode"] != mode: + continue + status_path = manifest_path.with_name("trial_status.json") + if not status_path.exists(): + raise ValueError(f"missing trial status: {status_path}") + status = load_json(status_path) + if status["status"] != "pass": + continue + scorer_path = manifest_path.with_name("scorer_output.json") + if not scorer_path.exists(): + raise ValueError(f"missing scorer output: {scorer_path}") + scorer = load_json(scorer_path) + key = (str(run["cell_id"]), int(run["probe_index"])) + if key in runs: + raise ValueError(f"duplicate Frontier run: {key}") + if int(scorer["total_requests"]) != int(run["request_count"]): + raise ValueError(f"request count mismatch: {key}") + runs[key] = { + "manifest_path": str(manifest_path), + "manifest": manifest, + "scorer_path": str(scorer_path), + "scorer": scorer, + } + return runs + + +def build_anchor_records( + real_cells: dict[str, dict[str, Any]], + frontier_runs: dict[tuple[str, int], dict[str, Any]], +) -> list[dict[str, Any]]: + expected = { + (cell_id, probe_index) + for cell_id, cell in real_cells.items() + for probe_index in cell["probes"] + } + if set(frontier_runs) != expected: + missing = sorted(expected - set(frontier_runs)) + extra = sorted(set(frontier_runs) - expected) + raise ValueError(f"Frontier/real anchor mismatch; missing={missing}, extra={extra}") + + anchors = [] + for cell_id, probe_index in sorted(expected): + cell = real_cells[cell_id] + probe = cell["probes"][probe_index] + frontier = frontier_runs[(cell_id, probe_index)] + run = frontier["manifest"]["run"] + scorer = frontier["scorer"] + if int(run["request_count"]) != int(probe["request_count"]): + raise ValueError(f"paired request count mismatch: {(cell_id, probe_index)}") + if not math.isclose( + float(run["sampling_u"]), float(probe["sampling_u"]), rel_tol=0.0, abs_tol=1e-15 + ): + raise ValueError(f"paired sampling_u mismatch: {(cell_id, probe_index)}") + rate = float(probe["request_rate_per_gpu_req_s_gpu"]) + anchors.append( + { + "cell_id": cell_id, + "tp": int(cell["tensor_parallel_size"]), + "mns": int(cell["max_num_seqs"]), + "probe_index": probe_index, + "sampling_u": float(probe["sampling_u"]), + "request_count": int(probe["request_count"]), + "offered_req_s_per_gpu": rate, + "real_feasible": bool(probe["feasible"]), + "real_pass_rate": float(probe["pass_rate"]), + "frontier_feasible": bool(scorer["slo"]["feasible"]), + "frontier_pass_rate": float(scorer["slo"]["pass_rate"]), + "frontier_completed_req_s_per_gpu": float( + scorer["throughput_requests_per_second_per_gpu"] + ), + } + ) + return anchors + + +def monotonic_violations(rows: list[dict[str, Any]], field: str) -> list[dict[str, Any]]: + ordered = sorted(rows, key=lambda row: (row["offered_req_s_per_gpu"], row["probe_index"])) + violations = [] + for lower_index, lower in enumerate(ordered): + for upper in ordered[lower_index + 1 :]: + if ( + lower["offered_req_s_per_gpu"] < upper["offered_req_s_per_gpu"] + and not lower[field] + and upper[field] + ): + violations.append( + { + "lower_probe": lower["probe_index"], + "lower_rate": lower["offered_req_s_per_gpu"], + "upper_probe": upper["probe_index"], + "upper_rate": upper["offered_req_s_per_gpu"], + } + ) + return violations + + +def selected_anchor(rows: list[dict[str, Any]], field: str) -> dict[str, Any] | None: + feasible = [row for row in rows if row[field]] + if not feasible: + return None + return max(feasible, key=lambda row: (row["offered_req_s_per_gpu"], -row["probe_index"])) + + +def summarize_cells(anchors: list[dict[str, Any]]) -> dict[str, dict[str, Any]]: + grouped = {cell: [] for cell in sorted(EXPECTED_CELLS)} + for row in anchors: + grouped[row["cell_id"]].append(row) + + summaries = {} + for cell_id, rows in grouped.items(): + real_selected = selected_anchor(rows, "real_feasible") + frontier_selected = selected_anchor(rows, "frontier_feasible") + real_score = real_selected["offered_req_s_per_gpu"] if real_selected else 0.0 + frontier_score = frontier_selected["offered_req_s_per_gpu"] if frontier_selected else 0.0 + max_rate = max(row["offered_req_s_per_gpu"] for row in rows) + real_violations = monotonic_violations(rows, "real_feasible") + frontier_violations = monotonic_violations(rows, "frontier_feasible") + summaries[cell_id] = { + "tp": rows[0]["tp"], + "mns": rows[0]["mns"], + "anchor_count": len(rows), + "max_common_anchor_req_s_per_gpu": max_rate, + "real_feasible_anchor_count": sum(bool(row["real_feasible"]) for row in rows), + "frontier_feasible_anchor_count": sum( + bool(row["frontier_feasible"]) for row in rows + ), + "real_score": real_score, + "frontier_score": frontier_score, + "absolute_error": frontier_score - real_score, + "relative_error": (frontier_score - real_score) / real_score if real_score else None, + "real_selected_probe": real_selected["probe_index"] if real_selected else None, + "frontier_selected_probe": frontier_selected["probe_index"] if frontier_selected else None, + "frontier_right_censored": bool( + frontier_selected and math.isclose(frontier_score, max_rate, abs_tol=1e-15) + ), + "frontier_boundary_bracketed": bool( + frontier_selected + and any( + not row["frontier_feasible"] + and row["offered_req_s_per_gpu"] > frontier_score + for row in rows + ) + ), + "real_monotonic_violation_count": len(real_violations), + "frontier_monotonic_violation_count": len(frontier_violations), + "real_monotonic_violations": real_violations, + "frontier_monotonic_violations": frontier_violations, + } + return summaries + + +def score_buckets(scores: dict[str, float]) -> tuple[float, dict[str, int]]: + tolerance = max(1e-9, 1e-6 * max(abs(value) for value in scores.values())) + return tolerance, {cell: math.floor(value / tolerance) for cell, value in scores.items()} + + +def sign(value: int) -> int: + return (value > 0) - (value < 0) + + +def ranking_metrics( + real_scores: dict[str, float], frontier_scores: dict[str, float] +) -> dict[str, Any]: + if set(real_scores) != set(frontier_scores): + raise ValueError("ranking score cells differ") + real_tolerance, real_buckets = score_buckets(real_scores) + frontier_tolerance, frontier_buckets = score_buckets(frontier_scores) + cells = sorted(real_scores) + counts = Counter() + exact = 0 + for left_index, left in enumerate(cells): + for right in cells[left_index + 1 :]: + real_sign = sign(real_buckets[left] - real_buckets[right]) + frontier_sign = sign(frontier_buckets[left] - frontier_buckets[right]) + exact += int(real_sign == frontier_sign) + if real_sign == 0 and frontier_sign == 0: + counts["both_tied"] += 1 + elif real_sign == 0: + counts["real_only_tied"] += 1 + elif frontier_sign == 0: + counts["frontier_only_tied"] += 1 + elif real_sign == frontier_sign: + counts["concordant"] += 1 + else: + counts["discordant"] += 1 + numerator = counts["concordant"] - counts["discordant"] + denominator = math.sqrt( + (counts["concordant"] + counts["discordant"] + counts["real_only_tied"]) + * ( + counts["concordant"] + + counts["discordant"] + + counts["frontier_only_tied"] + ) + ) + tau = numerator / denominator if denominator else 0.0 + top_bucket = max(frontier_buckets.values()) + candidates = sorted(cell for cell, bucket in frontier_buckets.items() if bucket == top_bucket) + real_best = max(real_scores.values()) + regrets = [(real_best - real_scores[cell]) / real_best for cell in candidates] + return { + "real_tolerance": real_tolerance, + "frontier_tolerance": frontier_tolerance, + "top1_candidate_cells": candidates, + "top1_effective_k": len(candidates), + "top1_optimistic_regret": min(regrets), + "top1_worst_case_regret": max(regrets), + "pair_count": len(cells) * (len(cells) - 1) // 2, + "pairwise_exact_sign_accuracy": exact / (len(cells) * (len(cells) - 1) // 2), + "kendall_tau_b": tau, + "kendall_counts": dict(counts), + } + + +def confusion_metrics(anchors: list[dict[str, Any]]) -> dict[str, Any]: + counts = Counter() + for row in anchors: + real = bool(row["real_feasible"]) + frontier = bool(row["frontier_feasible"]) + label = ( + "true_feasible" + if real and frontier + else "true_infeasible" + if not real and not frontier + else "false_feasible" + if not real and frontier + else "false_infeasible" + ) + counts[label] += 1 + total = len(anchors) + positive = counts["true_feasible"] + counts["false_infeasible"] + negative = counts["true_infeasible"] + counts["false_feasible"] + return { + **{name: counts[name] for name in ( + "true_feasible", + "true_infeasible", + "false_feasible", + "false_infeasible", + )}, + "total": total, + "accuracy": (counts["true_feasible"] + counts["true_infeasible"]) / total, + "feasible_recall": counts["true_feasible"] / positive if positive else None, + "infeasible_recall": counts["true_infeasible"] / negative if negative else None, + "false_feasible_rate": counts["false_feasible"] / negative if negative else None, + "false_infeasible_rate": counts["false_infeasible"] / positive if positive else None, + } + + +def error_metrics(cell_summaries: dict[str, dict[str, Any]]) -> dict[str, float]: + errors = [row["absolute_error"] for row in cell_summaries.values()] + relative = [abs(row["relative_error"]) for row in cell_summaries.values()] + return { + "mean_absolute_error_req_s_per_gpu": sum(abs(value) for value in errors) / len(errors), + "root_mean_square_error_req_s_per_gpu": math.sqrt( + sum(value * value for value in errors) / len(errors) + ), + "mean_absolute_percentage_error": sum(relative) / len(relative), + "right_censored_cell_count": sum( + bool(row["frontier_right_censored"]) for row in cell_summaries.values() + ), + "bracketed_cell_count": sum( + bool(row["frontier_boundary_bracketed"]) for row in cell_summaries.values() + ), + } + + +def grouped_error_metrics( + cell_summaries: dict[str, dict[str, Any]], field: str +) -> dict[str, dict[str, float]]: + groups: dict[int, list[dict[str, Any]]] = {} + for row in cell_summaries.values(): + groups.setdefault(int(row[field]), []).append(row) + output = {} + for key, rows in sorted(groups.items()): + output[str(key)] = { + "cell_count": len(rows), + "mean_signed_error_req_s_per_gpu": sum(row["absolute_error"] for row in rows) + / len(rows), + "mean_absolute_error_req_s_per_gpu": sum( + abs(row["absolute_error"]) for row in rows + ) + / len(rows), + "right_censored_cell_count": sum(bool(row["frontier_right_censored"]) for row in rows), + } + return output + + +def old_proxy_summary(previous_metrics_path: Path) -> dict[str, Any]: + previous = load_json(previous_metrics_path) + analysis = previous["analyses"]["frozen-calibrated/throughput-proxy"] + metrics = analysis["metrics"] + return { + "reading": "completed-throughput proxy (old, not SLO gated)", + "top1_candidate_cells": metrics["top1"]["candidate_cells"], + "top1_worst_case_regret": metrics["top1"]["worst_case_regret"], + "kendall_tau_b": metrics["kendall_tau_b"]["tau_b"], + "pairwise_exact_sign_accuracy": metrics["pairwise_direction"]["exact_sign_accuracy"], + } + + +def validate_previous_slo_reading( + previous_metrics_path: Path, + real_scores: dict[str, float], + frontier_scores: dict[str, float], +) -> dict[str, Any]: + previous = load_json(previous_metrics_path) + previous_real = {cell: float(value) for cell, value in previous["real_scores"].items()} + previous_frontier = { + cell: float(value) + for cell, value in previous["analyses"]["frozen-calibrated/SLO-gated"][ + "simulated_scores" + ].items() + } + real_difference = max(abs(real_scores[cell] - previous_real[cell]) for cell in real_scores) + frontier_difference = max( + abs(frontier_scores[cell] - previous_frontier[cell]) for cell in frontier_scores + ) + if real_difference > 1e-12 or frontier_difference > 1e-12: + raise ValueError( + "independent aligned reconstruction disagrees with previous secondary reading: " + f"real={real_difference}, Frontier={frontier_difference}" + ) + return { + "status": "PASS", + "previous_reading": "frozen-calibrated/SLO-gated", + "maximum_real_score_absolute_difference": real_difference, + "maximum_frontier_score_absolute_difference": frontier_difference, + } + + +def write_csv(path: Path, fieldnames: list[str], rows: Iterable[dict[str, Any]]) -> None: + with path.open("w", encoding="utf-8", newline="") as output: + writer = csv.DictWriter(output, fieldnames=fieldnames) + writer.writeheader() + writer.writerows({field: row.get(field) for field in fieldnames} for row in rows) + + +def render_report(metrics: dict[str, Any]) -> str: + rank = metrics["aligned_ranking"] + confusion = metrics["anchor_feasibility"] + error = metrics["score_error"] + grouped = metrics["grouped_score_error"] + old = metrics["objective_comparison"]["old_proxy"] + cells = metrics["cells"] + lines = [ + "# Frontier SLO-aligned retrospective experiment", + "", + "## Result", + "", + ( + "After aligning both systems to the same paired-grid objective—maximum offered " + "req/s/GPU with request-level SLO pass rate >= 0.95—the old 30.46% top-1 " + "regret does not reproduce. Frontier nominates `" + + "`, `".join(rank["top1_candidate_cells"]) + + f"` (tie), with real-evaluated regret {100 * rank['top1_optimistic_regret']:.3f}%" + + f"--{100 * rank['top1_worst_case_regret']:.3f}%." + ), + "", + ( + f"Ranking agreement is high on this fixed candidate grid: Kendall tau-b " + f"{rank['kendall_tau_b']:.4f}, pairwise exact-sign accuracy " + f"{100 * rank['pairwise_exact_sign_accuracy']:.2f}%. This is materially different " + f"from the old completed-throughput proxy (tau-b {old['kendall_tau_b']:.4f}, " + f"top-1 regret {100 * old['top1_worst_case_regret']:.2f}%)." + ), + "", + "## Per-cell paired-grid capacity", + "", + "| Cell | Real | Frontier | Error | Frontier boundary | Real monotonic violations |", + "|---|---:|---:|---:|---|---:|", + ] + for cell_id in sorted(cells, key=lambda cell: (cells[cell]["tp"], cells[cell]["mns"])): + row = cells[cell_id] + boundary = "right-censored" if row["frontier_right_censored"] else ( + "bracketed" if row["frontier_boundary_bracketed"] else "unresolved" + ) + lines.append( + f"| `{cell_id}` | {row['real_score']:.6f} | {row['frontier_score']:.6f} | " + f"{row['absolute_error']:+.6f} | {boundary} | " + f"{row['real_monotonic_violation_count']} |" + ) + lines.extend( + [ + "", + "The paired-grid score MAE is " + f"{error['mean_absolute_error_req_s_per_gpu']:.4f} req/s/GPU and MAPE is " + f"{100 * error['mean_absolute_percentage_error']:.2f}%. These error aggregates " + "must be read as grid-clipped because " + f"{int(error['right_censored_cell_count'])}/12 Frontier cells remain feasible at " + "their highest common anchor.", + "", + "The error is configuration-dependent rather than a uniform scale offset. Mean " + f"signed error is {grouped['by_mns']['8']['mean_signed_error_req_s_per_gpu']:+.4f} " + "req/s/GPU for MNS=8, but rises to " + f"{grouped['by_mns']['32']['mean_signed_error_req_s_per_gpu']:+.4f} and " + f"{grouped['by_mns']['64']['mean_signed_error_req_s_per_gpu']:+.4f} for MNS=32/64. " + "The aligned score therefore still exposes a missing or distorted MNS response.", + "", + "## Anchor-level feasibility", + "", + f"Across {confusion['total']} paired anchors: true-feasible=" + f"{confusion['true_feasible']}, true-infeasible={confusion['true_infeasible']}, " + f"false-feasible={confusion['false_feasible']}, false-infeasible=" + f"{confusion['false_infeasible']}. Accuracy is {100 * confusion['accuracy']:.2f}%, " + f"but the false-feasible rate among real-infeasible anchors is " + f"{100 * confusion['false_feasible_rate']:.2f}%.", + "", + "Therefore the aligned experiment supports a narrower conclusion: Frontier can " + "recover the top configuration family on this frozen, ragged candidate grid, but " + "it is not yet a reliable SLO feasibility oracle. Good top-1 ranking is compatible " + "with many wrong boundary labels.", + "", + "## Validity limits", + "", + "- This is a retrospective single-run analysis; no confidence interval is valid.", + "- Real probes within a cell reused one vLLM process and prefix-cache history; " + "Frontier anchors were independent runs. History-dependent real pass-rate values " + "are retained rather than monotonicized; binary feasibility happens to remain " + "monotone on the observed per-cell grids.", + "- Right-censored cells expose only a lower bound on Frontier's own capacity. " + "They are valid for paired-grid selection, not for claiming an exact simulator knee.", + "- The frozen per-TP calibration and evaluation surface are not fully independent " + "across workload families, so this is not a generalization result.", + "", + "## Required prospective experiment", + "", + "Repeat a small set of boundary cells with one engine restart per anchor, explicit " + "cold-cache state, identical offered-load grid, and repeated trials. Extend the grid " + "until both real and Frontier have at least one feasible and one infeasible point. " + "Only that experiment can estimate boundary displacement and its uncertainty.", + "", + ] + ) + return "\n".join(lines) + + +def analyze( + ground_truth_path: Path, + results_dir: Path, + previous_metrics_path: Path, + repository_root: Path, +) -> tuple[dict[str, Any], list[dict[str, Any]]]: + real_cells = load_real_cells(ground_truth_path) + frontier_runs = load_frontier_runs(results_dir) + anchors = build_anchor_records(real_cells, frontier_runs) + cells = summarize_cells(anchors) + real_scores = {cell: row["real_score"] for cell, row in cells.items()} + frontier_scores = {cell: row["frontier_score"] for cell, row in cells.items()} + frontier_revisions = { + run["manifest"]["frontier"]["git_head"] for run in frontier_runs.values() + } + if len(frontier_revisions) != 1: + raise ValueError(f"mixed Frontier revisions: {sorted(frontier_revisions)}") + metrics = { + "schema_version": SCHEMA, + "generated_utc": datetime.now(timezone.utc).isoformat(), + "status": "VALID_RETROSPECTIVE_ALIGNED_ANALYSIS", + "claim": ( + "On the frozen common anchor grid, rank configurations by maximum offered " + "req/s/GPU whose request-level SLO pass rate is at least 0.95 in each system." + ), + "estimand": READING, + "inputs": { + "ground_truth": { + "path": str(ground_truth_path), + "sha256": sha256_file(ground_truth_path), + }, + "frontier_results": str(results_dir), + "previous_metrics": { + "path": str(previous_metrics_path), + "sha256": sha256_file(previous_metrics_path), + }, + }, + "provenance": { + "analysis_repository": git_revision(repository_root), + "frontier_git_head_recorded_by_runs": next(iter(frontier_revisions)), + "frontier_mode": MODE, + "real_engine_version": "vLLM 0.20.0", + "anchor_count": len(anchors), + "cell_count": len(cells), + }, + "objective_comparison": {"old_proxy": old_proxy_summary(previous_metrics_path)}, + "aligned_ranking": ranking_metrics(real_scores, frontier_scores), + "anchor_feasibility": confusion_metrics(anchors), + "score_error": error_metrics(cells), + "grouped_score_error": { + "by_tp": grouped_error_metrics(cells, "tp"), + "by_mns": grouped_error_metrics(cells, "mns"), + }, + "independent_reconstruction_crosscheck": validate_previous_slo_reading( + previous_metrics_path, real_scores, frontier_scores + ), + "cells": cells, + "statistics": { + "real_trials_per_anchor": 1, + "frontier_trials_per_anchor": 1, + "confidence_intervals": None, + "reason": "single retrospective observation per paired anchor", + }, + "known_limits": [ + "real probes reused a process and prefix-cache history within each cell", + "Frontier anchors were independent simulator runs", + "right-censored cells do not identify Frontier's exact capacity knee", + "the anchor grid is ragged across cells", + ], + } + return metrics, anchors + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--ground-truth", type=Path, required=True) + parser.add_argument("--frontier-results", type=Path, required=True) + parser.add_argument("--previous-metrics", type=Path, required=True) + parser.add_argument("--output-dir", type=Path, required=True) + return parser.parse_args() + + +def main() -> None: + args = parse_args() + repository_root = Path(__file__).resolve().parents[2] + metrics, anchors = analyze( + args.ground_truth.resolve(), + args.frontier_results.resolve(), + args.previous_metrics.resolve(), + repository_root, + ) + args.output_dir.mkdir(parents=True, exist_ok=True) + metrics_path = args.output_dir / "metrics.json" + metrics_path.write_text(json.dumps(metrics, indent=2, sort_keys=True) + "\n", encoding="utf-8") + write_csv( + args.output_dir / "cell_scores.csv", + [ + "cell_id", + "tp", + "mns", + "anchor_count", + "real_score", + "frontier_score", + "absolute_error", + "relative_error", + "frontier_right_censored", + "frontier_boundary_bracketed", + "real_monotonic_violation_count", + "frontier_monotonic_violation_count", + ], + ({"cell_id": cell_id, **row} for cell_id, row in sorted(metrics["cells"].items())), + ) + write_csv( + args.output_dir / "anchor_labels.csv", + [ + "cell_id", + "tp", + "mns", + "probe_index", + "sampling_u", + "request_count", + "offered_req_s_per_gpu", + "real_feasible", + "real_pass_rate", + "frontier_feasible", + "frontier_pass_rate", + "frontier_completed_req_s_per_gpu", + ], + anchors, + ) + (args.output_dir / "report.md").write_text(render_report(metrics), encoding="utf-8") + print(json.dumps({"status": metrics["status"], "metrics": str(metrics_path)}, sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/frontier-slo-alignment-v0/results/anchor_labels.csv b/runs/frontier-slo-alignment-v0/results/anchor_labels.csv new file mode 100644 index 0000000..c63657f --- /dev/null +++ b/runs/frontier-slo-alignment-v0/results/anchor_labels.csv @@ -0,0 +1,93 @@ +cell_id,tp,mns,probe_index,sampling_u,request_count,offered_req_s_per_gpu,real_feasible,real_pass_rate,frontier_feasible,frontier_pass_rate,frontier_completed_req_s_per_gpu +tp1_mns16,1,16,0,0.5,276,4.6,False,0.09057971014492754,False,0.14855072463768115,3.2422237998507493 +tp1_mns16,1,16,1,0.25,143,2.3833333333333333,False,0.8461538461538461,True,1.0,2.2847815885743166 +tp1_mns16,1,16,2,0.125,66,1.1,True,1.0,True,1.0,1.0641771799103616 +tp1_mns16,1,16,3,0.1875,103,1.7166666666666666,True,1.0,True,1.0,1.6601410596627497 +tp1_mns16,1,16,4,0.21875,121,2.0166666666666666,True,1.0,True,1.0,1.9360040008720856 +tp1_mns16,1,16,5,0.234375,132,2.2,True,0.9924242424242424,True,1.0,2.1118607283809685 +tp1_mns16,1,16,6,0.2421875,137,2.283333333333333,True,0.9927007299270073,True,1.0,2.1914280693644232 +tp1_mns16,1,16,7,0.24609375,141,2.35,True,0.9574468085106383,True,1.0,2.2549713872507646 +tp1_mns32,1,32,0,0.5,276,4.6,False,0.014492753623188406,False,0.6666666666666666,4.297017390853387 +tp1_mns32,1,32,1,0.25,143,2.3833333333333333,False,0.7132867132867133,True,1.0,2.284832740403262 +tp1_mns32,1,32,2,0.125,66,1.1,True,1.0,True,1.0,1.0641771799103616 +tp1_mns32,1,32,3,0.1875,103,1.7166666666666666,True,1.0,True,1.0,1.6601410596627497 +tp1_mns32,1,32,4,0.21875,121,2.0166666666666666,True,1.0,True,1.0,1.9360040008720856 +tp1_mns32,1,32,5,0.234375,132,2.2,True,0.9621212121212122,True,1.0,2.11158903487612 +tp1_mns32,1,32,6,0.2421875,137,2.283333333333333,True,0.9635036496350365,True,1.0,2.1911815099759533 +tp1_mns32,1,32,7,0.24609375,141,2.35,False,0.723404255319149,True,1.0,2.2550534049233777 +tp1_mns64,1,64,0,0.5,276,4.6,False,0.057971014492753624,False,0.6666666666666666,4.356763578770651 +tp1_mns64,1,64,1,0.25,143,2.3833333333333333,False,0.7132867132867133,True,1.0,2.284832740403262 +tp1_mns64,1,64,2,0.125,66,1.1,True,1.0,True,1.0,1.0641771799103616 +tp1_mns64,1,64,3,0.1875,103,1.7166666666666666,True,1.0,True,1.0,1.6601410596627497 +tp1_mns64,1,64,4,0.21875,121,2.0166666666666666,True,1.0,True,1.0,1.9360040008720856 +tp1_mns64,1,64,5,0.234375,132,2.2,True,0.9621212121212122,True,1.0,2.11158903487612 +tp1_mns64,1,64,6,0.2421875,137,2.283333333333333,True,0.9635036496350365,True,1.0,2.1911815099759533 +tp1_mns64,1,64,7,0.24609375,141,2.35,False,0.723404255319149,True,1.0,2.2550534049233777 +tp1_mns8,1,8,0,0.5,276,4.6,False,0.06521739130434782,False,0.06884057971014493,2.1725065543636846 +tp1_mns8,1,8,1,0.25,143,2.3833333333333333,False,0.6993006993006993,False,0.6153846153846154,2.0743292183634 +tp1_mns8,1,8,2,0.125,66,1.1,True,1.0,True,1.0,1.0641771799103616 +tp1_mns8,1,8,3,0.1875,103,1.7166666666666666,True,1.0,True,1.0,1.660069637502782 +tp1_mns8,1,8,4,0.21875,121,2.0166666666666666,True,1.0,False,0.8677685950413223,1.8977642969225545 +tp1_mns8,1,8,5,0.234375,132,2.2,False,0.7121212121212122,False,0.7575757575757576,2.0012415074420837 +tp1_mns8,1,8,6,0.2265625,126,2.1,True,1.0,False,0.7936507936507936,1.942525444038986 +tp1_mns8,1,8,7,0.23046875,130,2.1666666666666665,False,0.8846153846153846,False,0.7846153846153846,1.9778490149228842 +tp2_mns16,2,16,0,0.5,276,2.3,False,0.9492753623188406,True,1.0,2.2439374417654148 +tp2_mns16,2,16,1,0.25,143,1.1916666666666667,True,1.0,True,1.0,1.1652366234427673 +tp2_mns16,2,16,2,0.375,209,1.7416666666666667,True,1.0,True,1.0,1.7024011425654029 +tp2_mns16,2,16,3,0.4375,243,2.025,True,1.0,True,1.0,1.978602236135322 +tp2_mns16,2,16,4,0.46875,256,2.1333333333333333,True,1.0,True,1.0,2.083276142822733 +tp2_mns16,2,16,5,0.484375,265,2.2083333333333335,True,1.0,True,1.0,2.15467356806669 +tp2_mns16,2,16,6,0.4921875,269,2.2416666666666667,True,1.0,True,1.0,2.187119822377907 +tp2_mns16,2,16,7,0.49609375,273,2.275,True,1.0,True,1.0,2.219545738132131 +tp2_mns32,2,32,0,0.5,276,2.3,True,0.9855072463768116,True,1.0,2.2439374417654148 +tp2_mns32,2,32,1,0.75,391,3.2583333333333333,True,1.0,True,1.0,3.171147514698164 +tp2_mns32,2,32,2,0.875,450,3.75,False,0.42,True,1.0,3.6496440319651886 +tp2_mns32,2,32,3,0.8125,417,3.475,False,0.7745803357314148,True,1.0,3.381833218203857 +tp2_mns32,2,32,4,0.78125,407,3.3916666666666666,False,0.20147420147420148,True,1.0,3.3006981169615908 +tp2_mns32,2,32,5,0.765625,400,3.3333333333333335,False,0.695,True,1.0,3.243915834454301 +tp2_mns32,2,32,6,0.7578125,396,3.3,False,0.8535353535353535,True,1.0,3.2118868660091686 +tp2_mns32,2,32,7,0.75390625,394,3.283333333333333,True,1.0,True,1.0,3.195580748135602 +tp2_mns64,2,64,0,0.5,276,2.3,True,0.9855072463768116,True,1.0,2.2439374417654148 +tp2_mns64,2,64,1,0.75,391,3.2583333333333333,True,1.0,True,1.0,3.171147514698164 +tp2_mns64,2,64,2,0.875,450,3.75,False,0.34,True,1.0,3.6496440319651886 +tp2_mns64,2,64,3,0.8125,417,3.475,False,0.6282973621103117,True,1.0,3.381833218203857 +tp2_mns64,2,64,4,0.78125,407,3.3916666666666666,False,0.8427518427518428,True,1.0,3.3006981169615908 +tp2_mns64,2,64,5,0.765625,400,3.3333333333333335,False,0.0775,True,1.0,3.243915834454301 +tp2_mns64,2,64,6,0.7578125,396,3.3,False,0.29292929292929293,True,1.0,3.2118868660091686 +tp2_mns64,2,64,7,0.75390625,394,3.283333333333333,False,0.6040609137055838,True,1.0,3.195580748135602 +tp2_mns8,2,8,0,0.5,276,2.3,False,0.2028985507246377,False,0.39492753623188404,2.037309149499862 +tp2_mns8,2,8,1,0.25,143,1.1916666666666667,True,1.0,True,1.0,1.1652509326668725 +tp2_mns8,2,8,2,0.375,209,1.7416666666666667,True,1.0,True,1.0,1.6993884308462768 +tp2_mns8,2,8,3,0.4375,243,2.025,True,1.0,False,0.9465020576131687,1.9636944417416549 +tp2_mns8,2,8,4,0.46875,256,2.1333333333333333,True,1.0,False,0.890625,2.0353351328293683 +tp2_mns8,2,8,5,0.484375,265,2.2083333333333335,True,1.0,False,0.6943396226415094,2.0340077558764986 +tp2_mns8,2,8,6,0.4921875,269,2.2416666666666667,True,1.0,False,0.5204460966542751,2.0394007508896705 +tp2_mns8,2,8,7,0.49609375,273,2.275,True,1.0,False,0.4358974358974359,2.0358394808879736 +tp4_mns16,4,16,0,0.034252608017,600,2.5,False,0.16,True,1.0,2.4366476386646814 +tp4_mns16,4,16,1,0.017126304009,317,1.3208333333333333,True,1.0,True,1.0,1.3046624956163406 +tp4_mns16,4,16,2,0.025689456013,453,1.8875,True,1.0,True,1.0,1.8463926715458043 +tp4_mns16,4,16,3,0.029971032015,516,2.15,True,1.0,True,1.0,2.0988716639818685 +tp4_mns16,4,16,4,0.032111820016,552,2.3,True,1.0,True,1.0,2.2421056884527815 +tp4_mns16,4,16,5,0.033182214016,575,2.3958333333333335,True,1.0,True,1.0,2.335243044645089 +tp4_mns16,4,16,6,0.033717411016,586,2.441666666666667,True,1.0,True,1.0,2.3799430022241697 +tp4_mns32,4,32,0,0.034252608017,600,2.5,False,0.3466666666666667,True,1.0,2.462327006841985 +tp4_mns32,4,32,1,0.017126304009,317,1.3208333333333333,True,1.0,True,1.0,1.3046624956163406 +tp4_mns32,4,32,2,0.025689456013,453,1.8875,True,1.0,True,1.0,1.859491597548273 +tp4_mns32,4,32,3,0.029971032015,516,2.15,True,1.0,True,1.0,2.117703981656738 +tp4_mns32,4,32,4,0.032111820016,552,2.3,True,1.0,True,1.0,2.2653938418034003 +tp4_mns32,4,32,5,0.033182214016,575,2.3958333333333335,True,1.0,True,1.0,2.359733397875389 +tp4_mns32,4,32,6,0.033717411016,586,2.441666666666667,True,1.0,True,1.0,2.4049469442687075 +tp4_mns64,4,64,0,0.034252608017,600,2.5,False,0.3466666666666667,True,1.0,2.462327006841985 +tp4_mns64,4,64,1,0.017126304009,317,1.3208333333333333,True,1.0,True,1.0,1.3046624956163406 +tp4_mns64,4,64,2,0.025689456013,453,1.8875,True,1.0,True,1.0,1.859491597548273 +tp4_mns64,4,64,3,0.029971032015,516,2.15,True,1.0,True,1.0,2.117703981656738 +tp4_mns64,4,64,4,0.032111820016,552,2.3,True,1.0,True,1.0,2.2653938418034003 +tp4_mns64,4,64,5,0.033182214016,575,2.3958333333333335,True,1.0,True,1.0,2.359733397875389 +tp4_mns64,4,64,6,0.033717411016,586,2.441666666666667,True,1.0,True,1.0,2.4049469442687075 +tp4_mns8,4,8,0,0.034252608017,600,2.5,False,0.056666666666666664,False,0.095,1.5449814460277083 +tp4_mns8,4,8,1,0.017126304009,317,1.3208333333333333,False,0.8517350157728707,True,1.0,1.277194040226129 +tp4_mns8,4,8,2,0.008563152005,159,0.6625,True,1.0,True,1.0,0.6553587315711599 +tp4_mns8,4,8,3,0.012844728007,243,1.0125,True,1.0,True,1.0,0.9892987097876506 +tp4_mns8,4,8,4,0.014985516008,274,1.1416666666666666,True,1.0,True,1.0,1.1153521862441735 +tp4_mns8,4,8,5,0.016055910008,301,1.2541666666666667,True,1.0,True,1.0,1.2252281710185793 +tp4_mns8,4,8,6,0.016591107009,308,1.2833333333333334,True,1.0,True,1.0,1.2459143726426762 diff --git a/runs/frontier-slo-alignment-v0/results/cell_scores.csv b/runs/frontier-slo-alignment-v0/results/cell_scores.csv new file mode 100644 index 0000000..2b0bd85 --- /dev/null +++ b/runs/frontier-slo-alignment-v0/results/cell_scores.csv @@ -0,0 +1,13 @@ +cell_id,tp,mns,anchor_count,real_score,frontier_score,absolute_error,relative_error,frontier_right_censored,frontier_boundary_bracketed,real_monotonic_violation_count,frontier_monotonic_violation_count +tp1_mns16,1,16,8,2.35,2.3833333333333333,0.033333333333333215,0.014184397163120517,False,True,0,0 +tp1_mns32,1,32,8,2.283333333333333,2.3833333333333333,0.10000000000000009,0.04379562043795625,False,True,0,0 +tp1_mns64,1,64,8,2.283333333333333,2.3833333333333333,0.10000000000000009,0.04379562043795625,False,True,0,0 +tp1_mns8,1,8,8,2.1,1.7166666666666666,-0.3833333333333335,-0.1825396825396826,False,True,0,0 +tp2_mns16,2,16,8,2.275,2.3,0.02499999999999991,0.01098901098901095,True,False,0,0 +tp2_mns32,2,32,8,3.283333333333333,3.75,0.4666666666666668,0.1421319796954315,True,False,0,0 +tp2_mns64,2,64,8,3.2583333333333333,3.75,0.4916666666666667,0.15089514066496165,True,False,0,0 +tp2_mns8,2,8,8,2.275,1.7416666666666667,-0.5333333333333332,-0.23443223443223438,False,True,0,0 +tp4_mns16,4,16,7,2.441666666666667,2.5,0.058333333333333126,0.02389078498293507,True,False,0,0 +tp4_mns32,4,32,7,2.441666666666667,2.5,0.058333333333333126,0.02389078498293507,True,False,0,0 +tp4_mns64,4,64,7,2.441666666666667,2.5,0.058333333333333126,0.02389078498293507,True,False,0,0 +tp4_mns8,4,8,7,1.2833333333333334,1.3208333333333333,0.03749999999999987,0.029220779220779116,False,True,0,0 diff --git a/runs/frontier-slo-alignment-v0/results/metrics.json b/runs/frontier-slo-alignment-v0/results/metrics.json new file mode 100644 index 0000000..24f9a86 --- /dev/null +++ b/runs/frontier-slo-alignment-v0/results/metrics.json @@ -0,0 +1,387 @@ +{ + "aligned_ranking": { + "frontier_tolerance": 3.7499999999999997e-06, + "kendall_counts": { + "both_tied": 4, + "concordant": 58, + "frontier_only_tied": 3, + "real_only_tied": 1 + }, + "kendall_tau_b": 0.9668009539030813, + "pair_count": 66, + "pairwise_exact_sign_accuracy": 0.9393939393939394, + "real_tolerance": 3.283333333333333e-06, + "top1_candidate_cells": [ + "tp2_mns32", + "tp2_mns64" + ], + "top1_effective_k": 2, + "top1_optimistic_regret": 0.0, + "top1_worst_case_regret": 0.0076142131979695165 + }, + "anchor_feasibility": { + "accuracy": 0.6956521739130435, + "false_feasible": 21, + "false_feasible_rate": 0.7, + "false_infeasible": 7, + "false_infeasible_rate": 0.11290322580645161, + "feasible_recall": 0.8870967741935484, + "infeasible_recall": 0.3, + "total": 92, + "true_feasible": 55, + "true_infeasible": 9 + }, + "cells": { + "tp1_mns16": { + "absolute_error": 0.033333333333333215, + "anchor_count": 8, + "frontier_boundary_bracketed": true, + "frontier_feasible_anchor_count": 7, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": false, + "frontier_score": 2.3833333333333333, + "frontier_selected_probe": 1, + "max_common_anchor_req_s_per_gpu": 4.6, + "mns": 16, + "real_feasible_anchor_count": 6, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.35, + "real_selected_probe": 7, + "relative_error": 0.014184397163120517, + "tp": 1 + }, + "tp1_mns32": { + "absolute_error": 0.10000000000000009, + "anchor_count": 8, + "frontier_boundary_bracketed": true, + "frontier_feasible_anchor_count": 7, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": false, + "frontier_score": 2.3833333333333333, + "frontier_selected_probe": 1, + "max_common_anchor_req_s_per_gpu": 4.6, + "mns": 32, + "real_feasible_anchor_count": 5, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.283333333333333, + "real_selected_probe": 6, + "relative_error": 0.04379562043795625, + "tp": 1 + }, + "tp1_mns64": { + "absolute_error": 0.10000000000000009, + "anchor_count": 8, + "frontier_boundary_bracketed": true, + "frontier_feasible_anchor_count": 7, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": false, + "frontier_score": 2.3833333333333333, + "frontier_selected_probe": 1, + "max_common_anchor_req_s_per_gpu": 4.6, + "mns": 64, + "real_feasible_anchor_count": 5, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.283333333333333, + "real_selected_probe": 6, + "relative_error": 0.04379562043795625, + "tp": 1 + }, + "tp1_mns8": { + "absolute_error": -0.3833333333333335, + "anchor_count": 8, + "frontier_boundary_bracketed": true, + "frontier_feasible_anchor_count": 2, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": false, + "frontier_score": 1.7166666666666666, + "frontier_selected_probe": 3, + "max_common_anchor_req_s_per_gpu": 4.6, + "mns": 8, + "real_feasible_anchor_count": 4, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.1, + "real_selected_probe": 6, + "relative_error": -0.1825396825396826, + "tp": 1 + }, + "tp2_mns16": { + "absolute_error": 0.02499999999999991, + "anchor_count": 8, + "frontier_boundary_bracketed": false, + "frontier_feasible_anchor_count": 8, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": true, + "frontier_score": 2.3, + "frontier_selected_probe": 0, + "max_common_anchor_req_s_per_gpu": 2.3, + "mns": 16, + "real_feasible_anchor_count": 7, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.275, + "real_selected_probe": 7, + "relative_error": 0.01098901098901095, + "tp": 2 + }, + "tp2_mns32": { + "absolute_error": 0.4666666666666668, + "anchor_count": 8, + "frontier_boundary_bracketed": false, + "frontier_feasible_anchor_count": 8, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": true, + "frontier_score": 3.75, + "frontier_selected_probe": 2, + "max_common_anchor_req_s_per_gpu": 3.75, + "mns": 32, + "real_feasible_anchor_count": 3, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 3.283333333333333, + "real_selected_probe": 7, + "relative_error": 0.1421319796954315, + "tp": 2 + }, + "tp2_mns64": { + "absolute_error": 0.4916666666666667, + "anchor_count": 8, + "frontier_boundary_bracketed": false, + "frontier_feasible_anchor_count": 8, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": true, + "frontier_score": 3.75, + "frontier_selected_probe": 2, + "max_common_anchor_req_s_per_gpu": 3.75, + "mns": 64, + "real_feasible_anchor_count": 2, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 3.2583333333333333, + "real_selected_probe": 1, + "relative_error": 0.15089514066496165, + "tp": 2 + }, + "tp2_mns8": { + "absolute_error": -0.5333333333333332, + "anchor_count": 8, + "frontier_boundary_bracketed": true, + "frontier_feasible_anchor_count": 2, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": false, + "frontier_score": 1.7416666666666667, + "frontier_selected_probe": 2, + "max_common_anchor_req_s_per_gpu": 2.3, + "mns": 8, + "real_feasible_anchor_count": 7, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.275, + "real_selected_probe": 7, + "relative_error": -0.23443223443223438, + "tp": 2 + }, + "tp4_mns16": { + "absolute_error": 0.058333333333333126, + "anchor_count": 7, + "frontier_boundary_bracketed": false, + "frontier_feasible_anchor_count": 7, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": true, + "frontier_score": 2.5, + "frontier_selected_probe": 0, + "max_common_anchor_req_s_per_gpu": 2.5, + "mns": 16, + "real_feasible_anchor_count": 6, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.441666666666667, + "real_selected_probe": 6, + "relative_error": 0.02389078498293507, + "tp": 4 + }, + "tp4_mns32": { + "absolute_error": 0.058333333333333126, + "anchor_count": 7, + "frontier_boundary_bracketed": false, + "frontier_feasible_anchor_count": 7, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": true, + "frontier_score": 2.5, + "frontier_selected_probe": 0, + "max_common_anchor_req_s_per_gpu": 2.5, + "mns": 32, + "real_feasible_anchor_count": 6, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.441666666666667, + "real_selected_probe": 6, + "relative_error": 0.02389078498293507, + "tp": 4 + }, + "tp4_mns64": { + "absolute_error": 0.058333333333333126, + "anchor_count": 7, + "frontier_boundary_bracketed": false, + "frontier_feasible_anchor_count": 7, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": true, + "frontier_score": 2.5, + "frontier_selected_probe": 0, + "max_common_anchor_req_s_per_gpu": 2.5, + "mns": 64, + "real_feasible_anchor_count": 6, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 2.441666666666667, + "real_selected_probe": 6, + "relative_error": 0.02389078498293507, + "tp": 4 + }, + "tp4_mns8": { + "absolute_error": 0.03749999999999987, + "anchor_count": 7, + "frontier_boundary_bracketed": true, + "frontier_feasible_anchor_count": 6, + "frontier_monotonic_violation_count": 0, + "frontier_monotonic_violations": [], + "frontier_right_censored": false, + "frontier_score": 1.3208333333333333, + "frontier_selected_probe": 1, + "max_common_anchor_req_s_per_gpu": 2.5, + "mns": 8, + "real_feasible_anchor_count": 5, + "real_monotonic_violation_count": 0, + "real_monotonic_violations": [], + "real_score": 1.2833333333333334, + "real_selected_probe": 6, + "relative_error": 0.029220779220779116, + "tp": 4 + } + }, + "claim": "On the frozen common anchor grid, rank configurations by maximum offered req/s/GPU whose request-level SLO pass rate is at least 0.95 in each system.", + "estimand": "paired-grid-slo-feasible-max-offered-throughput", + "generated_utc": "2026-07-15T07:27:18.082402+00:00", + "grouped_score_error": { + "by_mns": { + "16": { + "cell_count": 3, + "mean_absolute_error_req_s_per_gpu": 0.03888888888888875, + "mean_signed_error_req_s_per_gpu": 0.03888888888888875, + "right_censored_cell_count": 2 + }, + "32": { + "cell_count": 3, + "mean_absolute_error_req_s_per_gpu": 0.20833333333333334, + "mean_signed_error_req_s_per_gpu": 0.20833333333333334, + "right_censored_cell_count": 2 + }, + "64": { + "cell_count": 3, + "mean_absolute_error_req_s_per_gpu": 0.21666666666666665, + "mean_signed_error_req_s_per_gpu": 0.21666666666666665, + "right_censored_cell_count": 2 + }, + "8": { + "cell_count": 3, + "mean_absolute_error_req_s_per_gpu": 0.31805555555555554, + "mean_signed_error_req_s_per_gpu": -0.2930555555555556, + "right_censored_cell_count": 0 + } + }, + "by_tp": { + "1": { + "cell_count": 4, + "mean_absolute_error_req_s_per_gpu": 0.15416666666666673, + "mean_signed_error_req_s_per_gpu": -0.03750000000000003, + "right_censored_cell_count": 0 + }, + "2": { + "cell_count": 4, + "mean_absolute_error_req_s_per_gpu": 0.37916666666666665, + "mean_signed_error_req_s_per_gpu": 0.11250000000000004, + "right_censored_cell_count": 3 + }, + "4": { + "cell_count": 4, + "mean_absolute_error_req_s_per_gpu": 0.05312499999999981, + "mean_signed_error_req_s_per_gpu": 0.05312499999999981, + "right_censored_cell_count": 3 + } + } + }, + "independent_reconstruction_crosscheck": { + "maximum_frontier_score_absolute_difference": 0.0, + "maximum_real_score_absolute_difference": 0.0, + "previous_reading": "frozen-calibrated/SLO-gated", + "status": "PASS" + }, + "inputs": { + "frontier_results": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results", + "ground_truth": { + "path": "/home/gahow/phd/replayserve/docs/assets/simfid_s2r/ground_truth.json", + "sha256": "23ff3e6f6b88df8632bf37d37c0ff87bfef9d1676db81592948c08e898f7a670" + }, + "previous_metrics": { + "path": "/home/gahow/phd/replayserve/runs/simfid_s2rb/results/metrics.json", + "sha256": "55edb37d5692e979ab6f6dc6c65913a9db0aa0a836c350e4c05d9c38eee78206" + } + }, + "known_limits": [ + "real probes reused a process and prefix-cache history within each cell", + "Frontier anchors were independent simulator runs", + "right-censored cells do not identify Frontier's exact capacity knee", + "the anchor grid is ragged across cells" + ], + "objective_comparison": { + "old_proxy": { + "kendall_tau_b": 0.44812907976513594, + "pairwise_exact_sign_accuracy": 0.6818181818181818, + "reading": "completed-throughput proxy (old, not SLO gated)", + "top1_candidate_cells": [ + "tp1_mns64" + ], + "top1_worst_case_regret": 0.30456852791878175 + } + }, + "provenance": { + "analysis_repository": { + "head": "9c8570f36b1753d1b77c7cf1b0d16ecc360cad33", + "status_short": "?? \"AITuner\\347\\263\\273\\347\\273\\237\\344\\274\\230\\345\\214\\226\\344\\270\\216\\346\\214\\221\\346\\210\\230.pdf\"\n?? runs/frontier-slo-alignment-v0/" + }, + "anchor_count": 92, + "cell_count": 12, + "frontier_git_head_recorded_by_runs": "d9cfeb6d8791fbf2f295dd9744c56a666171776e", + "frontier_mode": "frozen-calibrated", + "real_engine_version": "vLLM 0.20.0" + }, + "schema_version": "frontier-slo-alignment-v0", + "score_error": { + "bracketed_cell_count": 6, + "mean_absolute_error_req_s_per_gpu": 0.19548611111111105, + "mean_absolute_percentage_error": 0.07697140171082821, + "right_censored_cell_count": 6, + "root_mean_square_error_req_s_per_gpu": 0.2775267963371919 + }, + "statistics": { + "confidence_intervals": null, + "frontier_trials_per_anchor": 1, + "real_trials_per_anchor": 1, + "reason": "single retrospective observation per paired anchor" + }, + "status": "VALID_RETROSPECTIVE_ALIGNED_ANALYSIS" +} diff --git a/runs/frontier-slo-alignment-v0/results/report.md b/runs/frontier-slo-alignment-v0/results/report.md new file mode 100644 index 0000000..66f5539 --- /dev/null +++ b/runs/frontier-slo-alignment-v0/results/report.md @@ -0,0 +1,45 @@ +# Frontier SLO-aligned retrospective experiment + +## Result + +After aligning both systems to the same paired-grid objective—maximum offered req/s/GPU with request-level SLO pass rate >= 0.95—the old 30.46% top-1 regret does not reproduce. Frontier nominates `tp2_mns32`, `tp2_mns64` (tie), with real-evaluated regret 0.000%--0.761%. + +Ranking agreement is high on this fixed candidate grid: Kendall tau-b 0.9668, pairwise exact-sign accuracy 93.94%. This is materially different from the old completed-throughput proxy (tau-b 0.4481, top-1 regret 30.46%). + +## Per-cell paired-grid capacity + +| Cell | Real | Frontier | Error | Frontier boundary | Real monotonic violations | +|---|---:|---:|---:|---|---:| +| `tp1_mns8` | 2.100000 | 1.716667 | -0.383333 | bracketed | 0 | +| `tp1_mns16` | 2.350000 | 2.383333 | +0.033333 | bracketed | 0 | +| `tp1_mns32` | 2.283333 | 2.383333 | +0.100000 | bracketed | 0 | +| `tp1_mns64` | 2.283333 | 2.383333 | +0.100000 | bracketed | 0 | +| `tp2_mns8` | 2.275000 | 1.741667 | -0.533333 | bracketed | 0 | +| `tp2_mns16` | 2.275000 | 2.300000 | +0.025000 | right-censored | 0 | +| `tp2_mns32` | 3.283333 | 3.750000 | +0.466667 | right-censored | 0 | +| `tp2_mns64` | 3.258333 | 3.750000 | +0.491667 | right-censored | 0 | +| `tp4_mns8` | 1.283333 | 1.320833 | +0.037500 | bracketed | 0 | +| `tp4_mns16` | 2.441667 | 2.500000 | +0.058333 | right-censored | 0 | +| `tp4_mns32` | 2.441667 | 2.500000 | +0.058333 | right-censored | 0 | +| `tp4_mns64` | 2.441667 | 2.500000 | +0.058333 | right-censored | 0 | + +The paired-grid score MAE is 0.1955 req/s/GPU and MAPE is 7.70%. These error aggregates must be read as grid-clipped because 6/12 Frontier cells remain feasible at their highest common anchor. + +The error is configuration-dependent rather than a uniform scale offset. Mean signed error is -0.2931 req/s/GPU for MNS=8, but rises to +0.2083 and +0.2167 for MNS=32/64. The aligned score therefore still exposes a missing or distorted MNS response. + +## Anchor-level feasibility + +Across 92 paired anchors: true-feasible=55, true-infeasible=9, false-feasible=21, false-infeasible=7. Accuracy is 69.57%, but the false-feasible rate among real-infeasible anchors is 70.00%. + +Therefore the aligned experiment supports a narrower conclusion: Frontier can recover the top configuration family on this frozen, ragged candidate grid, but it is not yet a reliable SLO feasibility oracle. Good top-1 ranking is compatible with many wrong boundary labels. + +## Validity limits + +- This is a retrospective single-run analysis; no confidence interval is valid. +- Real probes within a cell reused one vLLM process and prefix-cache history; Frontier anchors were independent runs. History-dependent real pass-rate values are retained rather than monotonicized; binary feasibility happens to remain monotone on the observed per-cell grids. +- Right-censored cells expose only a lower bound on Frontier's own capacity. They are valid for paired-grid selection, not for claiming an exact simulator knee. +- The frozen per-TP calibration and evaluation surface are not fully independent across workload families, so this is not a generalization result. + +## Required prospective experiment + +Repeat a small set of boundary cells with one engine restart per anchor, explicit cold-cache state, identical offered-load grid, and repeated trials. Extend the grid until both real and Frontier have at least one feasible and one infeasible point. Only that experiment can estimate boundary displacement and its uncertainty. diff --git a/runs/frontier-slo-alignment-v0/test_analysis.py b/runs/frontier-slo-alignment-v0/test_analysis.py new file mode 100644 index 0000000..f963ae8 --- /dev/null +++ b/runs/frontier-slo-alignment-v0/test_analysis.py @@ -0,0 +1,64 @@ +#!/usr/bin/env python3 +from __future__ import annotations + +import importlib.util +import math +import sys +from pathlib import Path + + +HERE = Path(__file__).resolve().parent + + +def load_analysis(): + spec = importlib.util.spec_from_file_location("frontier_slo_alignment", HERE / "analyze.py") + module = importlib.util.module_from_spec(spec) + assert spec.loader is not None + sys.modules[spec.name] = module + spec.loader.exec_module(module) + return module + + +def anchor(rate: float, real: bool, frontier: bool, probe: int) -> dict[str, object]: + return { + "cell_id": "tp1_mns8", + "tp": 1, + "mns": 8, + "probe_index": probe, + "offered_req_s_per_gpu": rate, + "real_feasible": real, + "frontier_feasible": frontier, + } + + +def main() -> None: + analysis = load_analysis() + rows = [ + anchor(1.0, True, True, 0), + anchor(2.0, False, True, 1), + anchor(3.0, True, False, 2), + ] + real_violations = analysis.monotonic_violations(rows, "real_feasible") + assert real_violations == [ + {"lower_probe": 1, "lower_rate": 2.0, "upper_probe": 2, "upper_rate": 3.0} + ] + assert analysis.selected_anchor(rows, "real_feasible")["probe_index"] == 2 + confusion = analysis.confusion_metrics(rows) + assert confusion["true_feasible"] == 1 + assert confusion["false_feasible"] == 1 + assert confusion["false_infeasible"] == 1 + assert math.isclose(confusion["accuracy"], 1.0 / 3.0) + + rank = analysis.ranking_metrics( + {"a": 3.0, "b": 2.0, "c": 1.0}, + {"a": 2.0, "b": 3.0, "c": 3.0}, + ) + assert rank["top1_candidate_cells"] == ["b", "c"] + assert math.isclose(rank["top1_optimistic_regret"], 1.0 / 3.0) + assert math.isclose(rank["top1_worst_case_regret"], 2.0 / 3.0) + assert rank["pair_count"] == 3 + print("frontier SLO alignment analysis: PASS") + + +if __name__ == "__main__": + main() diff --git a/runs/simulator-tuning-latency-matrix-v0/analyze_qwen30_tp_normalized_real.py b/runs/simulator-tuning-latency-matrix-v0/analyze_qwen30_tp_normalized_real.py new file mode 100644 index 0000000..4b87030 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/analyze_qwen30_tp_normalized_real.py @@ -0,0 +1,266 @@ +#!/usr/bin/env python3 +"""Audit and summarize the TP-normalized Qwen30 real-serving surface. + +The script reads only public trace manifests and prompt-free client result +records. It refuses to score an incomplete or contract-drifting trial. +""" + +from __future__ import annotations + +import argparse +import json +import math +import statistics +from collections.abc import Iterable +from pathlib import Path +from typing import Any + + +METRICS = ("ttft_ms", "tpot_ms", "e2e_ms") +CONFIGS = tuple((tp, mns) for tp in (1, 2, 4) for mns in (8, 16, 32, 64)) +TRIALS = (1, 2, 3) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--output-root", type=Path, required=True) + parser.add_argument("--json-output", type=Path, required=True) + parser.add_argument("--markdown-output", type=Path, required=True) + return parser.parse_args() + + +def nearest_rank(values: Iterable[float], percentile: float) -> float: + ordered = sorted(values) + if not ordered: + raise ValueError("cannot calculate percentile of empty values") + return ordered[math.ceil(len(ordered) * percentile) - 1] + + +def number(value: Any, field: str) -> float: + if not isinstance(value, (int, float)) or isinstance(value, bool): + raise ValueError(f"{field} is not numeric: {value!r}") + value = float(value) + if not math.isfinite(value) or value < 0: + raise ValueError(f"{field} is invalid: {value!r}") + return value + + +def load_manifest(root: Path, tp: int) -> dict[str, Any]: + path = root / "traces" / f"tp{tp}" / "public" / "manifest.json" + manifest = json.loads(path.read_text()) + if manifest.get("schema") != "qwen30-tp-normalized-trace-v1": + raise ValueError(f"unexpected trace schema in {path}") + if manifest.get("tensor_parallel_size") != tp: + raise ValueError(f"TP mismatch in {path}") + if manifest.get("requests") != 129: + raise ValueError(f"unexpected request count in {path}") + return manifest + + +def validate_trial( + result_path: Path, manifest: dict[str, Any], tp: int, mns: int, trial: int +) -> tuple[dict[str, Any], dict[str, list[float]]]: + payload = json.loads(result_path.read_text()) + if payload.get("schema") != "qwen30-exact-trace-anchor-v1": + raise ValueError(f"unexpected result schema: {result_path}") + + contract = payload.get("contract") + summary = payload.get("summary") + records = payload.get("requests") + if not isinstance(contract, dict) or not isinstance(summary, dict) or not isinstance(records, list): + raise ValueError(f"malformed result payload: {result_path}") + expected_requests = int(manifest["requests"]) + checks = { + "requests": (contract.get("requests"), expected_requests), + "requests_file_sha256": ( + contract.get("requests_file_sha256"), + manifest["private_jsonl_sha256"], + ), + "row_vector_sha256": ( + contract.get("row_vector_sha256"), + manifest["normalized_row_vector_sha256"], + ), + "first_arrival_s": ( + contract.get("first_arrival_s"), manifest["normalized_first_arrival_s"], + ), + "last_arrival_s": ( + contract.get("last_arrival_s"), manifest["normalized_last_arrival_s"], + ), + "served_model_alias": ( + contract.get("served_model_alias"), "qwen3-30b-exact-trace" + ), + } + for field, (actual, expected) in checks.items(): + if isinstance(expected, float): + if not isinstance(actual, (int, float)) or not math.isclose( + float(actual), expected, abs_tol=1e-9 + ): + raise ValueError(f"{result_path}: contract {field} drift") + elif actual != expected: + raise ValueError(f"{result_path}: contract {field} drift") + + if len(records) != expected_requests: + raise ValueError(f"{result_path}: expected {expected_requests} records") + if summary.get("completed") != expected_requests or summary.get("failed") != 0: + raise ValueError(f"{result_path}: incomplete replay summary") + + source_indices: set[int] = set() + values: dict[str, list[float]] = {metric: [] for metric in METRICS} + for record in records: + if record.get("success") is not True: + raise ValueError(f"{result_path}: failed request record") + index = record.get("source_index") + if not isinstance(index, int) or index in source_indices: + raise ValueError(f"{result_path}: invalid source index") + source_indices.add(index) + if record.get("actual_input_tokens") != record.get("input_tokens"): + raise ValueError(f"{result_path}: input usage mismatch") + if record.get("actual_output_tokens") != record.get("requested_output_tokens"): + raise ValueError(f"{result_path}: output usage mismatch") + for metric in METRICS: + if metric == "tpot_ms" and record.get(metric) is None: + # OSL=1 prefill-only traces intentionally have no TPOT samples. + continue + values[metric].append(number(record.get(metric), metric)) + + if len(source_indices) != expected_requests: + raise ValueError(f"{result_path}: missing source index") + for metric, samples in values.items(): + if not samples: + raise ValueError(f"{result_path}: no {metric} samples") + + stats = { + "tp": tp, + "mns": mns, + "trial": trial, + "result_path": str(result_path), + "requests": expected_requests, + "metrics": { + metric: { + "samples": len(samples), + "mean_ms": statistics.fmean(samples), + "p90_ms": nearest_rank(samples, 0.90), + } + for metric, samples in values.items() + }, + } + return stats, values + + +def aggregate(config_trials: list[dict[str, Any]], pooled: dict[str, list[float]]) -> dict[str, Any]: + if len(config_trials) != len(TRIALS): + raise ValueError("aggregate requires exactly three trials") + metrics: dict[str, Any] = {} + for metric in METRICS: + trial_means = [row["metrics"][metric]["mean_ms"] for row in config_trials] + trial_p90s = [row["metrics"][metric]["p90_ms"] for row in config_trials] + values = pooled[metric] + metrics[metric] = { + "pooled_samples": len(values), + "pooled_mean_ms": statistics.fmean(values), + "pooled_p90_ms": nearest_rank(values, 0.90), + "trial_mean_of_means_ms": statistics.fmean(trial_means), + "trial_stddev_of_means_ms": statistics.stdev(trial_means), + "trial_mean_of_p90s_ms": statistics.fmean(trial_p90s), + } + return {"trials": config_trials, "metrics": metrics} + + +def winners(configs: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]: + result: dict[str, dict[str, Any]] = {} + for metric in METRICS: + for statistic in ("pooled_mean_ms", "pooled_p90_ms"): + ranked = sorted( + ( + (summary["metrics"][metric][statistic], key) + for key, summary in configs.items() + ), + key=lambda item: (item[0], item[1]), + ) + result[f"{metric}:{statistic}"] = { + "winner": ranked[0][1], + "winner_value_ms": ranked[0][0], + "ranking": [key for _, key in ranked], + } + return result + + +def format_ms(value: float) -> str: + return f"{value:.1f}" + + +def render_markdown(payload: dict[str, Any]) -> str: + lines = [ + "# Qwen3-30B-A3B TP-normalized Trace-PD: real vLLM audit", + "", + "All 36 fresh-server trials passed the trace contract: 129/129 exact-usage requests per trial. " + "The smoke run is excluded. Values below pool the three trials (387 requests/config); " + "p90 uses nearest-rank order statistics.", + "", + "| Config | TTFT mean/p90 (ms) | TPOT mean/p90 (ms) | E2E mean/p90 (ms) |", + "|---|---:|---:|---:|", + ] + for key, summary in payload["configs"].items(): + metrics = summary["metrics"] + lines.append( + "| {key} | {ttft_mean}/{ttft_p90} | {tpot_mean}/{tpot_p90} | {e2e_mean}/{e2e_p90} |".format( + key=key, + ttft_mean=format_ms(metrics["ttft_ms"]["pooled_mean_ms"]), + ttft_p90=format_ms(metrics["ttft_ms"]["pooled_p90_ms"]), + tpot_mean=format_ms(metrics["tpot_ms"]["pooled_mean_ms"]), + tpot_p90=format_ms(metrics["tpot_ms"]["pooled_p90_ms"]), + e2e_mean=format_ms(metrics["e2e_ms"]["pooled_mean_ms"]), + e2e_p90=format_ms(metrics["e2e_ms"]["pooled_p90_ms"]), + ) + ) + lines += ["", "## Per-metric winners", ""] + for target, winner in payload["winners"].items(): + lines.append( + f"- `{target}`: `{winner['winner']}` ({format_ms(winner['winner_value_ms'])} ms)" + ) + lines.append("") + return "\n".join(lines) + + +def main() -> None: + args = parse_args() + manifests = {tp: load_manifest(args.output_root, tp) for tp in (1, 2, 4)} + configs: dict[str, dict[str, Any]] = {} + for tp, mns in CONFIGS: + key = f"tp{tp}_mns{mns}" + trial_rows: list[dict[str, Any]] = [] + pooled = {metric: [] for metric in METRICS} + for trial in TRIALS: + path = args.output_root / "real" / key / f"trial{trial}" / "results" / "result.json" + trial_stats, values = validate_trial(path, manifests[tp], tp, mns, trial) + trial_rows.append(trial_stats) + for metric in METRICS: + pooled[metric].extend(values[metric]) + configs[key] = aggregate(trial_rows, pooled) + payload = { + "schema": "qwen30-tp-normalized-real-surface-audit-v1", + "trace_manifests": { + f"tp{tp}": { + field: manifests[tp][field] + for field in ( + "requests", + "private_jsonl_sha256", + "normalized_row_vector_sha256", + "global_offered_request_rate", + "per_gpu_offered_request_rate", + ) + } + for tp in manifests + }, + "configs": configs, + "winners": winners(configs), + } + args.json_output.parent.mkdir(parents=True, exist_ok=True) + args.markdown_output.parent.mkdir(parents=True, exist_ok=True) + args.json_output.write_text(json.dumps(payload, indent=2, sort_keys=True) + "\n") + args.markdown_output.write_text(render_markdown(payload)) + print(json.dumps(payload["winners"], sort_keys=True)) + + +if __name__ == "__main__": + main() diff --git a/runs/simulator-tuning-latency-matrix-v0/dense-preflight.md b/runs/simulator-tuning-latency-matrix-v0/dense-preflight.md new file mode 100644 index 0000000..df7e6bb --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/dense-preflight.md @@ -0,0 +1,136 @@ +# Qwen3.6-27B non-MoE/hybrid preflight and topology-smoke record + +> **Status (2026-07-17): `TOPOLOGY-SMOKE PASS; FRONTIER COVERAGE BLOCKED`.** +> The exact snapshot and official community-vLLM release are accepted through +> a fresh-server TP1/TP2/TP4 GPU smoke. No Frontier Qwen3.6 contract/profile, +> trace replay, or latency-matrix cell exists. The smoke is compatibility +> evidence only, not simulator-fidelity or latency-selection evidence. + +## Decision enabled by this preflight + +The latency-matrix experiment may add its dense branch only after the exact +model snapshot, a runnable open-source vLLM environment, and a Qwen3.6 +Frontier contract/profile root exist on the same stack. Until then, all dense +matrix cells are **`NOT RUN`**. They are neither simulator failures nor +evidence for dense+MoE generalization. + +## Executed staging record (after explicit approval) + +| Step | Evidence | Result | +|---|---|---| +| Immutable snapshot staging | ModelScope `Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469` was downloaded to `/home/admin/cpfs/wjh/models/Qwen/Qwen3.6-27B`. The downloader recorded `download_start` and `download_complete` at `2026-07-17T07:42:00Z` and `2026-07-17T07:44:45Z`, respectively, in `/home/admin/cpfs/wjh/models/.staging/qwen36-27b-20260717/download.log`. | **PASS**; 29 snapshot files, approximately 52 GiB. | +| Snapshot integrity | Sorted per-file SHA-256 inventory: `/home/admin/cpfs/wjh/models/.staging/qwen36-27b-20260717/files.sha256`; inventory digest `031cea103726cbf27732b6971e898e210aee77f0c32e4bc703c947e0372349c4`. It includes `config.json`, tokenizer files, and all 15 `model-*.safetensors` shards; no `*.incomplete`, `*.lock`, or `*.tmp` files were found. | **PASS**. | +| CPU config/tokenizer parser | The preinstalled system `transformers==4.57.5` rejects `model_type=qwen3_5`; this environment is excluded. A dedicated CPU-only environment with `transformers==5.5.3` successfully parsed the snapshot with `CUDA_VISIBLE_DEVICES=""`: `Qwen3_5ForConditionalGeneration`, text model type `qwen3_5_text`, 64 layers, hidden size 5120, `mamba_ssm_dtype=float32`, tokenizer length 248077. | **PASS**, but only for checkpoint/tokenizer parsing; it is not the serving runtime. | +| Community vLLM release runtime | The initial source-build attempt was stopped before acceptance and is excluded. Official PyPI wheel `vllm==0.20.2` was staged at `wheel-probe-proxy/vllm-0.20.2-cp38-abi3-manylinux_2_35_x86_64.whl`, SHA-256 `22a7dd06eb03371298e13d6100f3dedbf307352342aaf08e87c929c60aae9b4d`, then installed into the clean environment `/home/admin/cpfs/wjh/venvs/qwen36-vllm-0.20.2`. `pip check` reports no broken requirements. | **PASS (CPU-only)**: vLLM `0.20.2`, Torch `2.11.0`, Transformers `5.14.1`; with `CUDA_VISIBLE_DEVICES=""`, `vllm.transformers_utils.config.get_config(..., trust_remote_code=False)` parses the exact snapshot as `Qwen3_5ForConditionalGeneration` / `qwen3_5`. | +| Fresh GPU topology smoke | Fresh text-only servers on H20 TP1, TP2, TP4 used the exact release runtime with requested `MNS=8`, `MBT=8192`, BF16, chunked prefill on, prefix cache off, and one `2048 → 1` request per topology. Every response reported exactly 2048 prompt and 1 completion token; all four used GPUs were zero-memory after cleanup. Evidence root: `artifacts/qwen36/smoke-v4-20260717/`; artifact inventory SHA-256 `5b09c94e61801bfb7085257cfeea88049dfa81001f50810d85b9f87468088be0`. | **PASS (GPU compatibility)**. It establishes that TP1/2/4 can serve this model on the intended release, not that any topology has lower latency. | + +## Confirmed model identity and architecture + +| Item | Confirmed fact | Consequence for the experiment | +|---|---|---| +| Canonical source | ModelScope [`Qwen/Qwen3.6-27B`](https://www.modelscope.cn/models/Qwen/Qwen3.6-27B) | Use this repository, not an inferred Hugging Face mirror or an old Qwen3.5 checkpoint. | +| Immutable revision | Read-only `git ls-remote --symref` on 2026-07-17 resolved `master`/`HEAD` to `cea40373b9214dd387123e68841890af30dcd469`. | Pin this commit hash in the download record. Do **not** record only the mutable name `master`. Re-resolve it immediately before an approved download; a changed hash requires a new review. | +| Model metadata | ModelScope API reports `Architectures=["Qwen3_5ForConditionalGeneration"]` and `ModelType=["qwen3_5"]`. The official card says the artifact is a causal language model with a vision encoder. | This is dense with respect to MoE routing, but it is **not** a conventional all-attention dense transformer. The text benchmark must send no media; the model runner must still be the architecture selected from the exact checkpoint config. | +| Text architecture (official card) | 27B parameters; hidden size 5120; padded token embedding / LM output 248320; 64 layers; `16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))`; DeltaNet has V=48/QK=16 heads of 128; gated attention has Q=24/KV=4 heads of 256 and RoPE dimension 64; FFN intermediate size 17408; native context 262144. | A Frontier profile must cover both DeltaNet/linear-state and gated-attention paths, in prefill, decode, and true mixed execution. An all-attention Qwen3.5 or Qwen3-30B/MoE profile is not a valid substitute. | +| Tokenizer | The staged snapshot contains the tokenizer artifacts in the immutable file inventory. The earlier dedicated CPU parser reported tokenizer length `248077`; `248320` above is still only the padded embedding/output size, not the tokenizer contract. | Request input/output lengths must be tokenized with this staged tokenizer. Before trace materialization, freeze a tokenizer/config subset digest and do not borrow counts from a Qwen3.5 tokenizer. | + +The official ModelScope card states that these artifacts are compatible with +Transformers, vLLM, SGLang, and KTransformers. That is useful compatibility +evidence, not a successful run on the selected `dash0` stack. + +## Runtime provenance and excluded alternatives + +| Evidence | Read-only observation | Verdict | +|---|---|---| +| System vLLM | `/usr/local/bin/vllm` reports `0.13.0rc2.dev2111+gb44b43f43.d20260309`, with Torch `2.8.0a0+5228986c39.nv25.6` and Transformers `4.57.5`. Its installed source contains `qwen3_5.py`, `qwen3_5_mtp.py`, registry entries for `Qwen3_5ForConditionalGeneration`, and branches for `qwen3_5_text`. | Architecture-family support is present, but this development build is not yet the frozen community experiment runtime. | +| Uninstalled local source | `/home/admin/cpfs/wjh/agentic-kv/third_party/vllm_v20_build` is `v0.20.0` source with build by-products; `/home/admin/cpfs/wjh/venvs/vllm-0.20.0` does not contain a runnable vLLM/Transformers pair. | **Excluded.** It is not the experiment runtime and must not be cited as an alternative same-stack result. | +| Qwen-specific state/page rule | The v0.20 Qwen3.5 implementation obtains Mamba/DeltaNet state settings from the checkpoint when cache dtype is auto; the actual v0.20.2 GPU smoke then resolved its attention page to 784 tokens. | The state/page rule is a model-state contract, not an optional benchmark flag. The profile and simulator must consume the observed physical page setting. | + +Therefore the defensible statement now is: **the exact Qwen3.6-27B snapshot +is CPU-parsed and GPU-served by the frozen official community vLLM release on +`dash0` at TP1/2/4. It is a non-MoE hybrid model, not a conventional +all-attention dense-transformer evidence point.** + +## GPU topology-smoke observations (not a latency experiment) + +The runner is a one-request topology/usage check, with a fresh server for +each TP. Its E2E values are retained only to audit that a request completed; +they are not steady-state samples, are not TTFT/TPOT measurements, and must +not be ranked. + +| TP | Exact usage | Smoke E2E (audit only) | Resolved attention page/block size | GPU KV cache at `max_model_len=4096` | +|---|---|---:|---:|---:| +| 1 | 2048 prompt, 1 completion | 4581.977 ms | 784 tokens | 306,289 tokens | +| 2 | 2048 prompt, 1 completion | 5606.756 ms | 784 tokens | 1,068,600 tokens | +| 4 | 2048 prompt, 1 completion | 6252.274 ms | 784 tokens | 2,630,087 tokens | + +vLLM accepted the requested `--block-size 16` but then logged, at every TP, +`Setting attention block size to 784 tokens to ensure that attention page size +is >= mamba page size.` Thus **block size 16 is not the actual engine state +for this model**. A later real/simulator comparison must freeze and expose the +resolved 784-token page/state contract (or deliberately select a different +legal engine mode before profiling); it may not label this stack as a +block-16 comparison. First TP1 startup also compiled FlashInfer GDN kernels +and performed a cold profile/warmup; these one-time costs remain outside all +request-latency metrics. + +## Initial read-only `dash0` inventory (historical; superseded where noted) + +All paths below were inspected without mutation on 2026-07-17. + +| Area | Observation | Consequence | +|---|---|---| +| Disk | `/home/admin` root filesystem: 245 GiB available. Shared CPFS mounted at `/home/admin/cpfs`: 1.1 TiB available, already 95% used. Existing `/home/admin/cpfs/wjh/models` is 1.2 TiB and `/home/admin/resource/model` is 1.7 TiB. | Stage only after an explicit capacity check; do not rely on the root filesystem or an unconstrained default cache. | +| Checked model/cache locations | Before staging, the usual locations were absent. | **Superseded:** the immutable snapshot is now at `/home/admin/cpfs/wjh/models/Qwen/Qwen3.6-27B` and is bound by the inventory above. | +| Main checkout | `/home/admin/cpfs/wjh/aituner/aituner` had tracked and untracked user changes. | The smoke wrote only new artifacts and did not modify its source. A future profile implementation must use an explicitly versioned source/patch root rather than treating this dirty checkout as code provenance. | +| Frontier | Historical Frontier source exists at `/home/admin/cpfs/wjh/frontier-qwen30-vllm020-profile-v1/Frontier`, commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e` (`pre-release-v0.2`), with a clean tracked tree. A source search found no `Qwen3.6`, `Qwen3_5`, or `qwen3_5` model-contract hit. | It is not an existing Qwen3.6 profile/model contract. A new dense adapter/spec must be versioned and accepted before it may be used. | +| Existing profiler contract | The old Qwen3-30B profile launcher hard-codes `model=qwen3-a3b-30b-moe`, `--is_moe`, vLLM 0.20 API adapters, and Frontier commit `d9cf…`. | It cannot be repurposed by only changing a model path; its MoE assumptions and compatibility adapter are excluded from the dense baseline. | +| Qwen3.6 artifacts | A read-only search of the checkout's `runs` subtree found no Qwen3.6-named run/profile/model path. | No dense Frontier profile or real/simulator output is admissible today. | + +## Remaining gates before any latency-matrix cell + +1. **Trace/tokenizer lock:** materialize the Qwen3.6 tokenized Fixed/Trace-P/ + Trace-PD vectors and hash the exact tokenizer/config subset. Trace-PD is + not trace-faithful without the input, output, arrival and prefix-state + vector. +2. **New Frontier contract:** introduce a versioned Qwen3.6 **non-MoE hybrid** model spec + that maps the exact `config.json` to simulator operators. It must represent + Gated DeltaNet/linear-state, gated attention, FFN, TP collectives and KV/ + state-cache capacity, including the resolved 784-token page rule. It may + not import the Qwen3-30B MoE routing profile, an all-attention Qwen3.5 + profile, or an unrecorded additive timing scale. Any compatibility adapter + is a separate, hashed intervention. +3. **Profile manifest:** collect and freeze per-TP (1/2/4) profile rows for + the above stages over shapes covering Fixed-P, Fixed-PD, Trace-P and + Trace-PD. Include prefill, decode and genuine mixed batches, plus TP2/TP4 + all-reduce. The manifest must bind model revision/config hash, tokenizer + hash, vLLM/environment hash, Frontier commit/adapter hash, hardware, + precision, profiler command, CSV hashes, and shape coverage. +4. **CPU simulator acceptance:** reject the profile root unless every required profile + family has finite rows for all legal TP values and requested shapes; all + identity hashes match the real-server lock; profile/model manifests name + Qwen3.6; and the simulator emits request-level TTFT/E2E/TPOT where defined + for a no-GPU fixture. A missing simulator operator is a **coverage + blocker**, not a license to borrow a Qwen30 profile. + +Only after all four gates may a separately approved real/simulator 600-s +latency surface begin. The GPU smoke closed the runtime/topology gate only; it +does not close any of these simulator or evaluation gates. + +## Evidence locations + +- This record: `runs/simulator-tuning-latency-matrix-v0/dense-preflight.md`. +- Matrix decision/protocol: `runs/simulator-tuning-latency-matrix-v0/experiment-card.md`. +- Existing (excluded) MoE-v0.20 profile launcher and its hard-coded contract: + `runs/frontier-qwen30-vllm020-profile-v1/run_frontier_linear_smoke.sh` and + `runs/frontier-qwen30-vllm020-profile-v1/frontier_vllm020_compat.py`. +- Existing Frontier surface runner shows the required external source, + ReplayServe builder, profile root, request-level metric, and hash contract: + `runs/frontier-phase-factorial-v0/run_frontier_qwen30_prefill_surface.py`. + +## Not completed + +- No Qwen3.6 Frontier model spec or profile exists. +- No Qwen3.6 fixed/trace workload vector is frozen. +- No real or simulator latency measurement exists, so this record cannot + change the simulator-selection verdict. diff --git a/runs/simulator-tuning-latency-matrix-v0/existing-evidence-audit.md b/runs/simulator-tuning-latency-matrix-v0/existing-evidence-audit.md new file mode 100644 index 0000000..1e8b3e8 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/existing-evidence-audit.md @@ -0,0 +1,144 @@ +# Existing simulator evidence audit for the no-SLO latency matrix + +> **Scope:** local-artifact audit on 2026-07-17. This document does not run a +> simulator or a server. It separates (1) historical evidence about +> **SLO-feasible capacity selection** from (2) the new matrix's claim about +> **same-trace request-level mean/p90 TTFT, TPOT, and E2E latency selection**. +> These are different objectives; a historical top-set match is not silently +> promoted into a no-SLO latency-tuning result. + +## Executive verdict + +The checked-in evidence already rules out the broad claim that Frontier (or a +Vidur-class simulator) has *generally* solved serving-config tuning: a fully +covered Qwen3-30B prefill-only surface has no real/simulator top-set overlap, +with 12.5% worst regret and `τ-b=-1.0`. Conversely, two Qwen3-235B surfaces +show that an extensively aligned, patched profile can select the capacity +optimum in a particular envelope. Neither observation answers the new paper +question by itself, because every completed selection result optimizes +SLO-feasible offered throughput/GPU rather than a fixed trace's request-level +mean/p90 latency. + +The appropriate new claim before the latency matrix is therefore: + +> Existing results establish both a capability envelope and a counterexample +> for **SLO/capacity tuning**. They motivate, but do not replace, the planned +> non-MoE-hybrid/MoE × Fixed-P/Fixed-PD/Trace-P/Trace-PD latency-selection +> matrix. + +## Objective boundary (non-negotiable) + +All completed comparisons below rank a config by + +```text +maximum tested offered request rate / allocated GPU +subject to a joint request SLO pass-rate threshold (normally >= 95%). +``` + +The new matrix instead holds one trace fixed for every candidate and independently +ranks `mean`/`p90` TTFT, TPOT (when `OSL>1`), and E2E. It has no SLO feasibility +gate and no capacity search. The consequences are material: + +- A capacity top-set hit can arise from a large topology margin even when + request-level latency residuals and within-family ordering are wrong. +- A simulator that makes every config SLO-infeasible (or ties every config) has + not selected a config, even if its set mechanically contains the real best + config. +- Historical runs may contain per-request latency observations, but their + anchors differ by config because each is a capacity search. They cannot be + re-labelled as a same-trace latency ranking without a new fixed-trace + analysis contract. + +Thus **all historical regret and correlation numbers in this document are +capacity/SLO diagnostics only**. They must not appear in a result table as +mean/p90 TTFT, TPOT, or E2E selection evidence. + +## Completed selection comparisons + +`Coverage` distinguishes complete config-level rankings from request/anchor +coverage. A completed simulator run is not semantic coverage when it has no +finite request metrics or no discriminative ordering. + +| Case / condition | Model, runtime, workload | Real / simulator coverage | Historical selection result | What it can establish | What it cannot establish for the new matrix | +|---|---|---|---|---|---| +| **Q30-mixed, old profile-only** | Qwen3-30B-A3B; community vLLM 0.20.0/CUDA 12.9; `chat_w20260311_1000` replay, time scale 0.1, input 0–8192, output overridden to 128, max concurrency 64; `TP{1,2,4}×MNS{8,16,32,64}`. | 12/12 real configs; 92 real anchors. Frontier replayed the same frozen cohort for all 92 anchors. | Real best `TP2/MNS32`; sim top `{TP4/MNS32,TP4/MNS64}`; miss; worst regret **25.63%**; `τ-b=0.0000`; 37/0/55 SLO agree/false-feasible/false-infeasible. | A profile-only simulator can select the wrong TP family under a mixed serving capacity objective. | It is not trace-faithful under the new definition: time is scaled, outputs are overridden, and the score is SLO capacity, not latency. It proves no mean/p90 latency claim. | +| **Q30-mixed, vLLM-0.20 same-stack profile-only** | Same Qwen30/mixed 12-cell surface; BF16 H20 profiles re-collected on vLLM source `88d34…`; no serving E2E scale. | All **92/92** simulator probes completed (two CPU shards, no crash), but all 12 configs are SLO-infeasible. | No actionable selection: all 12 tie; worst tie-break regret **60.91%**; `τ-b=0.0000`; exact pair sign 7.58%. | Same-stack isolated operator provenance alone did not recover the historical capacity ordering. | “All tied” is not a top-set success and says nothing directly about fixed-trace latency ranking. The model/workload is also MoE mixed, not the dense branch. | +| **Q30-mixed, frozen per-TP E2E calibration** | Same Qwen30/mixed surface; per-TP execution-time scales fitted on separate `coder_200` serving workloads then frozen. | 12/12 configs and 92 anchors; request-level SLO labels still have 21 false-feasible and 7 false-infeasible anchors. | Sim top `{TP2/MNS32,TP2/MNS64}` intersects real best `{TP2/MNS32}`; worst regret **0.76%**; `τ-b=0.9668`. | An action-conditioned real-serving calibration can make this *capacity* surface near-optimal. | The per-TP E2E scale is prohibited by the new matrix's information boundary. It is an upper-bound/diagnostic, not zero-shot simulator evidence and not latency evidence. | +| **Q30 Fixed-P-like prefill-only, base Frontier** | Qwen3-30B-A3B; community vLLM 0.20.0+cu129 BF16/FA3/default CUDA graph; fixed ISL=2048, OSL=1, uniform QPS, prefix off; 12 configs `TP{1,2,4}×MNS{8,16,32,64}`, MBT=8192. | 12/12 valid real configs; 96 real/sim SLO anchor decisions, with all requested request metrics present in the accepted comparison. | Real top is all TP4 configs (8 req/s/GPU); sim top is all TP1/TP2 configs (8 vs real 7); **no overlap**; worst regret **12.50%**; `τ-b=-1.0000`; 0/32 real non-tied directions correct. | A strong counterexample: removing decode, true-mixed batch, prefix reuse, and initial-KV state is not sufficient for capacity-ranking fidelity. | It is closest only to the planned MoE Fixed-P case. It still ranks SLO capacity, not mean/p90 TTFT/E2E on one 600-s trace, so it cannot fill that matrix cell. | +| **Q30 Fixed-P-like, A2 measured-collective fix** | Same Q30 prefill workload/configs; measured Vidur collective estimator replaces the `>100k`-element analytical fallback. | 12/12 config scores exist. Anchor grid: 60 shared real/sim labels, 36 real-only, 0 sim-only; all simulator top scores tie. | Sim top includes all 12 configs; best tie-break regret 0 but **worst=12.50%**; `τ-b=N/A` (32 simulator-only ties); 0/32 non-tied directions correct. | The collective profile is consumed after the patch; its small change does not make the capacity choice usable. | Set overlap is accidental/non-discriminative. This is a simulator-only ablation against historical SLO ground truth, not a new latency result. | +| **Q30 Fixed-P-like, A3 batch-composition rows** | Same as A2, plus pure-prefill attention rows for MBT=8192-reachable compositions. | Same as A2: 12/12 config scores; 60 shared + 36 real-only anchor labels; no missing simulator config score. | Same non-selection as A2: all 12 tie; worst regret **12.50%**; `τ-b=N/A`; 0/32 non-tied directions correct. | Adding this static batch-profile closure did not resolve the historical capacity ranking. | It does not prove scheduler-state root cause, and cannot substitute for no-SLO request-latency ranking. | +| **Q235 prefill-only, best-effort aligned profile** | Qwen3-235B-A22B-FP8; community vLLM 0.10.2, eager, FP8 weights/BF16 KV; length-stratified fixed 64-request cohort from `thinking_w20260327_1000`, OSL=1, prefix/speculative off; `TP{4,8}×MNS{64,128}×MBT{8192,16384}`. | 8/8 rankable real configs; refined comparisons report 33 config-load labels with 6 false-infeasible labels, not a missing simulator surface. | Exact top-set match: TP4/MBT16K/MNS64 or 128; worst regret **0**; Spearman `ρ=0.9487`; all 20 comparable real non-tie pairs correct. | With same-stack FP8/MoE profiles, measured KV capacity and explicit compatibility patches, Frontier can select one prefill-only capacity surface. | The workload is a length-stratified cohort, not Fixed-P or Trace-P; it is a different MoE model/runtime/config surface and SLO objective. It does not generalize to dense, decode, prefix reuse, or latency metrics. | +| **Q235 Fixed-PD-like mixed (T0)** | Qwen3-235B-A22B-FP8; community vLLM 0.10.2 eager, FP8/BF16-KV; fixed ISL=2048, OSL=128, uniform QPS, prefix/spec/CUDA graph off; same 8 config surface. | **8/8** real capacity boundaries closed (68 fresh-server anchors); simulator **64/64** config×rate cells have 64 finite, non-negative, shape-exact request records; 34 measured labels with 10 false-infeasible, 0 false-feasible. | Exact TP4 top-set match; worst regret **0**; `τ-b=0.8944`; 16/20 real non-tied directions correct. | The most complete historical profile-closed mixed-capacity success. It demonstrates that a large TP4-vs-TP8 margin can tolerate sizeable absolute and within-TP residuals. | The primary 150-ms TPOT SLO was a disclosed post-pilot sensitivity (40-ms primary had no feasible capacity). It is not a fixed-trace mean/p90 TTFT/TPOT/E2E selection result, and it misses the TP8 MNS×MBT interaction. | + +### The capability envelope is not a global solution + +The two Q235 successes are real positive evidence, but must be reported with +their alignment cost and failure modes: + +- Q235 prefill used FP8/MoE serving-plan, TP/EP-aware cache-key and + critical-lane patches, same-stack operator/collective profiles, and real KV + capacity. It was not stock Frontier. +- Q235 mixed required an attention profile closure (prefill + standard decode + + true mixed) with 1,104 rows. The accepted real ground truth alone cost + 36.26 H20-GPU-hours. The simulator still flattened the TP8 MNS×MBT + checkerboard: 10/34 labels were false-infeasible even though its global top + set was correct. +- The Q30 prefill counterexample used the same 12-config type of surface as + the planned MoE Fixed-P branch and reverses the entire topology order. A + successful Q235 capacity ranking therefore cannot be offered as evidence + that a simulator already solves this new tuning problem. + +## Cases that are diagnostics or plans, not valid selection evidence + +| Artifact/case | Status and observed coverage | Why it is useful | Why it cannot enter a new-matrix verdict | +|---|---|---|---| +| Historical internal-runtime Q235 prefill | 8/8 real cells, but different serving/runtime contract from the aligned community-vLLM Frontier profile. | Records a real response surface and demonstrates stack sensitivity. | No fair real/simulator contract; excluded by `simulator-fidelity.md`. | +| Historical Q235 decode-only | Only 7/8 valid real cells; capacity brackets overlap such that all eight may be optimal. Frontier lacks equivalent initial-KV/EP8 execution semantics. | Identifies initial-KV, EAGLE3, DeepEP/NVSHMEM and decode-graph state as contract gaps. | No identifiable real best and no semantically aligned simulator; no hit/regret/`τ-b` is valid. | +| Q235 T0 pre-profile-closure smoke | Real TP4 test completed; original Frontier crash had no standard decode rows. After minimal closure, only a one/two-request representation smoke completed, with 27–35% TPOT absolute error. | Demonstrates that a simulator crash must be treated as coverage failure, not an SLO failure; motivated the full T0 profile closure. | A single-config smoke cannot select across configs and cannot support any ranking claim. | +| Envelope F1 short-prefill pilot | Fixed ISL=512/OSL=1, 64 requests. The 512-QPS pilot covers only 0.123 s and is explicitly non-decision-bearing. | Reveals the need for a steady-arrival duration contract. | No sustained real capacity surface and no paired selection metrics. | +| Envelope F1 steady, F2 fixed mixed, T1 exact trace | Protocol/frozen simulator artifacts exist; no paired valid real/simulator selection comparison is recorded. In particular, T1's simulator artifact is not a completed real trace evaluation. | The contracts are a useful starting point for fixed/trace workload materialization. | `NOT RUN` for the purpose of a simulator-selection result. Do not cite them as trace-faithful success/failure. | +| Q235 protocol T1/T2 | The protocol explicitly marks T1 trace-faithful mixed and T2 strict decode-only as not run; only T0 is complete. | T1 preserves source request fields; T2 correctly requires an explicit initial-KV contract. | No real/sim coverage, hit, regret, or `τ-b` exists. | + +## What can be reused safely + +| Reusable input | Allowed use in the new matrix | Prohibited inference | +|---|---|---| +| Q30 prefill accepted artifacts | Harness conventions: fresh server, two rounds, request shape/usage validation, request-level record layout, and a known failure case for coverage gates. | Treating the old TP4/TP1/2 capacity ordering or 12.5% regret as mean/p90 TTFT/E2E selection evidence. | +| Q235 T0 artifacts | Fixed-PD measurement practice: exact 2048/128 token accounting, separate warmup, finite request metrics, profile identity/coverage checks, and state-leakage exclusion. | Reusing its Q235 FP8/vLLM0.10 profile or TP4/TP8 conclusion for Qwen3.6 non-MoE hybrid or Qwen3-30B MoE. | +| Q235 trace audit | Trace manifest structure: exact prompt/token lengths, arrival/session order, source hashes, tokenizer audit, and explicit source-to-runtime block-size translation. | Calling a trace-faithful latency cell complete before runtime block/cache counter parity and paired real/simulator results exist. | +| A2/A3 ablations | A negative control for the hypothesis that better static collective or pure-prefill batch rows alone solve the Q30 residual. | Claiming the remaining error is uniquely caused by scheduler state; routing, graph/fusion, and collective composition remain alternatives. | + +## Required interpretation in the paper and the next experiment + +1. Say **“not solved in general”**, supported by the Q30 no-overlap + counterexample; do not say “all simulators fail,” because the Q235 envelope + has valid capacity-selection successes. +2. Report the positive cases as *conditional capacity results* and disclose + patches, profiles, real KV capacity, profile cost, and—where used—per-TP + E2E calibration. Do not call them zero-cost tuning. +3. Do not average these cases into an aggregate score. Model, precision, + runtime, topology, objective, and workload differ; the average would hide + the decision-bearing Q30 reversal. +4. Run the new matrix exactly as specified in + [experiment-card.md](experiment-card.md): for every legal config replay the + same fixed or trace-derived request vector, report mean/p90 TTFT/TPOT/E2E, + require simulator request-metric coverage, and calculate top-set hit, + worst tie-break regret, and `τ-b` separately for each latency objective. + `OSL=1` cases report TPOT as `N/A`, never zero. + +## Primary evidence paths + +- Overall completed-case synthesis and exclusions: + [simulator-fidelity.md](../../simulator-fidelity.md). +- Q30 prefill counterexample: [experiment card](../frontier-phase-factorial-v0/experiment-card.md), + [accepted comparison](../frontier-phase-factorial-v0/results/final/comparison.json). +- Q30 measured-collective/batch ablations: + [envelope card](../frontier-fidelity-envelope-v1/experiment-card.md), + [A2 comparison](../frontier-fidelity-envelope-v1/results/a2/comparison.json), + [A3 comparison](../frontier-fidelity-envelope-v1/results/a3/comparison.json). +- Q235 prefill alignment and exclusions: + [findings](../frontier-multicase-sufficiency-v0/findings.md), + [refined comparison](../frontier-multicase-sufficiency-v0/best_effort/fixed_cohort_evidence/v2_refined_comparison.json). +- Q235 fixed mixed T0: [protocol](../frontier-multicase-sufficiency-v1/protocol.md), + [smoke report](../frontier-multicase-sufficiency-v1/t0-smoke-report.md), + [final comparison](../frontier-multicase-sufficiency-v1/results/t0-final/comparison.json). diff --git a/runs/simulator-tuning-latency-matrix-v0/frontier-qwen36-coverage-audit.md b/runs/simulator-tuning-latency-matrix-v0/frontier-qwen36-coverage-audit.md new file mode 100644 index 0000000..0047f86 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/frontier-qwen36-coverage-audit.md @@ -0,0 +1,84 @@ +# Frontier coverage audit for Qwen3.6-27B + +> **Verdict (2026-07-17): `FAIL: original Frontier coverage`.** This is a +> static capability result, not a latency-fidelity result. It says that the +> examined existing Frontier cannot represent the real Qwen3.6 execution +> contract; it does not say that a future extended simulator will be wrong. + +## Question + +Before spending profile or replay GPU-hours, can the checked Frontier baseline +simulate the same model/engine state that the real server executes? + +## Compared contracts + +| Side | Observed contract | +|---|---| +| Real server | ModelScope `Qwen/Qwen3.6-27B@cea40373b9214dd387123e68841890af30dcd469`; vLLM 0.20.2, BF16; `Qwen3_5ForConditionalGeneration`; 64-layer non-MoE hybrid with three Gated DeltaNet/linear-state layers followed by one gated-attention layer per block. The release vLLM run used a FlashInfer GDN prefill kernel and changed requested `--block-size 16` to a 784-token attention/state page at TP1/2/4. | +| Frontier baseline | `/home/admin/cpfs/wjh/frontier-qwen30-vllm020-profile-v1/Frontier`, commit `d9cfeb6d8791fbf2f295dd9744c56a666171776e`. Its model-spec directory has Qwen2/Qwen3-MoE and all-attention dense examples, but no Qwen3.6 or Qwen3_5 spec. Source search across `*.py`, `*.json`, `*.md` returned no hit for `qwen3_5`, `qwen3.6`, `GatedDeltaNet`, `DeltaNet`, `mamba`, `linear attention`, `state cache`, or `gdn`. | + +## Frontier source model-spec inventory + +This is the complete `frontend/configs/model_configs/*.json` inventory at the +checked commit. It establishes that the simulator can instantiate these +*structural specifications*; it is **not** evidence that a matching vLLM +profile, runtime contract, or validated tuning result exists for each model. + +| Class | Source specs | +|---|---| +| All-attention, non-MoE | `Llama-3.1-405B-Instruct-FP8`, `Llama-3.2-1B-Instruct`, `llama2_7b_dense_example`, `llama3.1-405b`, `llama3.1-8b`, `llama3.3-70b`, `qwen2_dense_test` | +| MoE | `Phi-tiny-MoE-instruct`, `Qwen3-235B-A22B`, `Qwen3-235B-A22B__layers96`, `Qwen3-30B-A3B-tiny`, `deepseek-v3`, `mixtral_8x7b_moe`, `qwen2_moe_example`, `qwen3-a3b-30b-moe`, `qwen3-next-80b-a3b-instruct-reduced-l2`, `qwen3-next-80b-a3b-instruct-reduced-l20`, `Step2Mini-tiny`, `step-moe-noquant-small`, `step-moe-noquant`, `step-moe` | + +The exact JSON architecture labels are `llama`, `qwen2`, `phimoe`, +`qwen3_moe`, `deepseek_v3`, `mixtral`, `qwen2_moe`, `qwen3_next`, +`step2_mini`, and `step3_text`. The separately held experimental artifacts +in this project establish profile-and-surface evidence only for +`Qwen3-30B-A3B` and `Qwen3-235B-A22B`; the remaining names are not +automatically runnable or validated on H20/vLLM merely because a JSON exists. + +## Why the missing model file is semantic, not administrative + +The Frontier execution-time taxonomy separates `attention_prefill_execution_time` +and `attention_decode_execution_time`; its operator context and parameter +accounting are built from attention Q/KV heads, attention head dimension, FFN, +and optional MoE routing. No Gated-DeltaNet/linear-state operator, state-cache +capacity rule, or its mixed prefill/decode scheduling state was found. + +Adding a JSON whose layer count merely matches Qwen3.6 would therefore map +three quarters of its text layers onto an all-attention approximation. Reusing +Qwen30/MoE or Qwen3.5 profiles has the same defect. Neither is a valid +execution of the existing baseline. + +## Decision + +1. Record Frontier as **coverage failure** for the Qwen3.6 non-MoE-hybrid + branch. Do not rank configs, fabricate high latency, or spend GPU-hours on + an all-attention profile substitute. +2. A new Gated-DeltaNet/state-cache model and profiling implementation would + be an **extended Frontier** intervention. It must be evaluated separately + from the capability of the original simulator and cannot support the claim + that existing Frontier already solves tuning. +3. The Qwen3.6 result cannot be presented as evidence for a conventional pure + dense-transformer class. It is a useful non-MoE/hybrid boundary case. +4. No runnable local checkout/version of Vidur or APEX was found; only a + historical Vidur patch exists. Their coverage remains **unassessed**, not + passed or failed, until their exact released artifacts are frozen and + checked against this model/runtime contract. + +## Next critical experiment + +Do not extend Frontier before establishing the requested no-SLO metric on a +model it already represents. The next direct test is the existing +Qwen3-30B-A3B MoE branch under one same-trace Fixed-P replay surface, scored +by mean/p90 TTFT and E2E (TPOT `N/A`), with simulator-only coverage checked +before any real 600-s sweep. This separates a known architectural coverage +failure from a fidelity failure on an in-scope model. + +## Evidence locations + +- Real topology smoke and resolved page state: + `artifacts/qwen36/smoke-v4-20260717/` in this run directory. +- Real model/runtime audit: + [dense-preflight.md](dense-preflight.md). +- No-SLO latency protocol and selection gates: + [experiment-card.md](experiment-card.md). diff --git a/runs/simulator-tuning-latency-matrix-v0/latency-selection-matrix-schematic.svg b/runs/simulator-tuning-latency-matrix-v0/latency-selection-matrix-schematic.svg new file mode 100644 index 0000000..32d0410 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/latency-selection-matrix-schematic.svg @@ -0,0 +1,82 @@ + + Schematic latency-selection evaluation matrix + Schematic, not measured data: dense and MoE rows, four P-only and P-plus-decode workloads, then a real-versus-simulator config-selection protocol. + + + + SCHEMATIC — NOT MEASURED DATA + Can a simulator select the same low-latency config as the real engine? + Each model × workload × metric is one tuning task. Lower latency is better; SLO and capacity are not selection objectives. + + Frozen evaluation matrix + Fixed-P + ISL=2048 · OSL=1 · QPS=4 + Fixed-PD + ISL=2048 · OSL=128 · QPS=4 + Trace-P + exact input/arrival/prefix · OSL=1 + Trace-PD + exact input/output/arrival/prefix + + Dense + Qwen/Qwen3.6-27B + community vLLM; profile pending + + + + + + P-onlyTTFT, E2E · TPOT N/A + Prefill + decodeTTFT, TPOT, E2E + P-node trace replayoutput normalized to 1 + Joint trace replayprefix/KV state retained + + New Qwen3.6/community-vLLM profile contract; old internal-vLLM artifacts excluded. + + MoE + Qwen3-30B-A3B + community vLLM 0.20 + + + + + + historical raw P pilotre-run as 600-s matrix + trace materializer existsreal surface pending + derive held-out traceforce OSL=1 only + T1 simulator has stallsreal surface pending + + + + Per-objective selection + + Real engine + 3 fresh-server trials → bootstrap top set + + + + Simulator + complete request metrics → exact top set + + + + coverage · hit · regret · τ-b + report all four; never collapse a missing cell into “slow” + + + Decision gates + 1. Valid trace/usage and finite applicable metrics. + 2. Simulator covers every real-valid config; crash or scheduler stall is a coverage failure, not a latency value. + 3. For every mean/p90 objective: top-set hit, worst tie-break regret ≤5%, and τ-b reported. + + Information boundary + Profiles/calibration are frozen before evaluation and disjoint from held-out trace sessions. + No per-config E2E scale, post-hoc patch, or simulator-guided real subsetting. + Protocol status only: this figure does not contain experiment measurements. See experiment-card.md for trace contracts, candidate surface, provenance, and cost gate. + diff --git a/runs/simulator-tuning-latency-matrix-v0/launch_qwen30_tp_normalized_frontier.sh b/runs/simulator-tuning-latency-matrix-v0/launch_qwen30_tp_normalized_frontier.sh new file mode 100644 index 0000000..a2cba62 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/launch_qwen30_tp_normalized_frontier.sh @@ -0,0 +1,77 @@ +#!/usr/bin/env bash + +# CPU-only Frontier surface for the per-GPU-normalized Qwen3-30B Trace-PD case. +# This script is copied to the remote experiment output directory, never into +# the shared/dirty AITuner checkout. + +set -euo pipefail + +OUT="${1:?output root is required}" +RUNNER="/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-fidelity-envelope-v1/run_frontier_qwen30_exact_trace_surface.py" +FRONTIER="/home/admin/cpfs/wjh/aituner/frontier-t1-dash0-deadc4a" +REPLAYSERVE="/home/admin/cpfs/wjh/replayserve" +PYTHON_DEPS="/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1/lib/python3.12/site-packages" +PROFILE="/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-fidelity-envelope-v1/profiles/profile-v4-trace-final" +ALLREDUCE="/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-fidelity-envelope-v1/profiles/measured-allreduce.csv" +BASE_PUBLIC="/home/admin/cpfs/wjh/aituner/fidelity-envelope-private/trace-exact-v1/public/u0p01/frontier.csv" +BASE_PRIVATE="/home/admin/cpfs/wjh/aituner/fidelity-envelope-private/trace-exact-v1/private/u0p01/real_requests.jsonl" +MATERIALIZER="${OUT}/bin/materialize_qwen30_tp_normalized_trace.py" + +printf '%s\n' "SIMULATOR_LAUNCH_ECHO host=dash0; gpu_allocation=0; model=Qwen3-30B-A3B; trace=trace-exact-v1/u0p01; requests=129; transform=t_prime=t/TP; TP={1,2,4}; per_gpu_offered_rate=0.215 req/s; surface=TP{1,2,4}xMNS{8,16,32,64}; Frontier=deadc4a321f0baaa534c6ebd17f974123733cdc2; profiles=profile-v4-trace-final; expected_wall=10-45m; output=${OUT}" +date -u +START_UTC=%Y-%m-%dT%H:%M:%SZ + +mkdir -p "${OUT}/provenance" +sha256sum "${MATERIALIZER}" "${BASH_SOURCE[0]}" "${RUNNER}" "${BASE_PUBLIC}" \ + "${BASE_PRIVATE}" "${PROFILE}/manifest.json" "${ALLREDUCE}" \ + > "${OUT}/provenance/input.sha256" +git -C "${FRONTIER}" rev-parse HEAD > "${OUT}/provenance/frontier.commit" +git -C /home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1 rev-parse HEAD \ + > "${OUT}/provenance/aituner.commit" +printf '%s\n' "${PYTHON_DEPS}" > "${OUT}/provenance/python_deps_path" +/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1/bin/python -c \ + 'import importlib.metadata as m; [print(f"{name}=={m.version(name)}") for name in ("plotly", "numpy", "pandas", "scikit-learn", "PyYAML")]' \ + > "${OUT}/provenance/python_deps_versions.txt" + +for tp in 1 2 4; do + /usr/bin/python3 "${MATERIALIZER}" \ + --base-public-csv "${BASE_PUBLIC}" \ + --base-private-jsonl "${BASE_PRIVATE}" \ + --tp "${tp}" \ + --output-root "${OUT}/traces/tp${tp}" \ + > "${OUT}/traces-tp${tp}.log" +done + +for tp in 1 2 4; do + for mns in 8 16 32 64; do + cfg="tp${tp}_mns${mns}" + cfgout="${OUT}/simulator-retry-deps/${cfg}" + mkdir -p "${cfgout}" + ( + set +e + timeout --signal=TERM --kill-after=30s 2400 /usr/bin/python3 "${RUNNER}" \ + --frontier-source "${FRONTIER}" \ + --replayserve-root "${REPLAYSERVE}" \ + --profile-root "${PROFILE}" \ + --python-deps "${PYTHON_DEPS}" \ + --output-root "${cfgout}" \ + --trace "tp${tp}=${OUT}/traces/tp${tp}/public/frontier.csv" \ + --config "${cfg}" \ + --rate-contract trace-window \ + --prefix-caching \ + --cc-backend vidur \ + --allreduce-csv "${ALLREDUCE}" \ + --timeout-seconds 1800 \ + --predictor-training-job-threads 12 \ + --continue-on-failure \ + > "${cfgout}/launcher.stdout.log" 2> "${cfgout}/launcher.stderr.log" + printf '%s\n' "$?" > "${cfgout}/launcher.exit_code" + exit 0 + ) & + done +done +wait + +find "${OUT}" -type f ! -path '*/provenance/artifacts.sha256' -print0 \ + | sort -z | xargs -0 sha256sum > "${OUT}/provenance/artifacts.sha256" +date -u +END_UTC=%Y-%m-%dT%H:%M:%SZ +printf '%s\n' 'SIMULATOR_SURFACE_COMPLETE' diff --git a/runs/simulator-tuning-latency-matrix-v0/launch_qwen30_tp_normalized_real_surface.sh b/runs/simulator-tuning-latency-matrix-v0/launch_qwen30_tp_normalized_real_surface.sh new file mode 100644 index 0000000..b2db9fe --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/launch_qwen30_tp_normalized_real_surface.sh @@ -0,0 +1,173 @@ +#!/usr/bin/env bash + +# Three fresh-server real vLLM trials for the per-GPU-normalized Qwen30 +# Trace-PD surface. This script and its JIT-tolerant harness are copied into +# the remote output root, not the shared/dirty AITuner checkout. + +set -euo pipefail + +OUT="${1:?output root is required}" +RUNNER_DIR="/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-fidelity-envelope-v1" +RUNNER="${OUT}/bin/run_qwen30_exact_trace_real_anchor_jit_tolerant.sh" +CLIENT="${OUT}/bin/qwen30_exact_trace_client.py" +PREFILL_CLIENT="/home/admin/cpfs/wjh/aituner/aituner-qwen30-vllm020-profile-v1/runs/frontier-phase-factorial-v0/qwen30_prefill_client.py" +VENV_ROOT="/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1" +MODEL_ROOT="/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B" +TRACE_ROOT="${OUT}/traces" +# FlashInfer's JIT uses per-operation file locks. Reuse the smoke-populated +# cache across fresh servers so the 36 measurement runs do not each rebuild +# the same H20 MoE/all-reduce kernels. Server processes and metrics remain +# isolated per run under ${OUT}/real. +FLASHINFER_SHARED_WORKSPACE="${OUT}/flashinfer-shared-workspace" +PORT=8200 + +declare -a WAVE_PIDS=() + +wait_for_wave() { + local failed=0 pid + for pid in "${WAVE_PIDS[@]}"; do + if ! wait "${pid}"; then + failed=1 + fi + done + WAVE_PIDS=() + if [[ "${failed}" -ne 0 ]]; then + printf '%s\n' 'ERROR one or more real-serving runs failed; aborting surface.' \ + | tee -a "${OUT}/controller.log" >&2 + return 1 + fi +} + +preflight_gpus() { + nvidia-smi --query-gpu=index,name,memory.used,utilization.gpu --format=csv,noheader \ + | tee -a "${OUT}/controller.log" + if nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits \ + | awk '$1 != 0 {exit 1}'; then + return 0 + fi + printf '%s\n' 'ERROR non-idle GPU before a real-serving wave; refusing to start.' \ + | tee -a "${OUT}/controller.log" >&2 + return 1 +} + +assert_no_our_server() { + if pgrep -fa 'qwen3-30b-exact-trace' > "${OUT}/provenance/unexpected_server_processes.txt"; then + printf '%s\n' 'ERROR a prior experiment server remains; refusing the next wave.' \ + | tee -a "${OUT}/controller.log" >&2 + return 1 + fi +} + +launch_config() { + local trial="$1" tp="$2" mns="$3" gpus="$4" + local config="tp${tp}_mns${mns}" + local run_out="${OUT}/real/${config}/trial${trial}" + local requests="${TRACE_ROOT}/tp${tp}/private/real_requests.jsonl" + local port="${PORT}" + PORT=$((PORT + 1)) + mkdir -p "${run_out}" + ( + cd "${RUNNER_DIR}" + set +e + env \ + HOME=/tmp/wjh \ + XDG_CACHE_HOME=/tmp/wjh/.cache \ + VLLM_CACHE_ROOT=/tmp/wjh/.cache/vllm \ + CUDA_VISIBLE_DEVICES="${gpus}" \ + TP="${tp}" \ + MNS="${mns}" \ + TRACE_LABEL="tp${tp}-normalized-u0p01" \ + SERVER_PORT="${port}" \ + OUTPUT_ROOT="${run_out}" \ + REQUESTS_FILE="${requests}" \ + VENV_ROOT="${VENV_ROOT}" \ + MODEL_ROOT="${MODEL_ROOT}" \ + FLASHINFER_WORKSPACE_BASE="${FLASHINFER_SHARED_WORKSPACE}" \ + EXACT_TRACE_CLIENT="${CLIENT}" \ + timeout --signal=TERM --kill-after=60s 4200 bash "${RUNNER}" \ + > "${run_out}/launcher.stdout.log" 2> "${run_out}/launcher.stderr.log" + local_status="$?" + printf '%s\n' "${local_status}" > "${run_out}/launcher.exit_code" + exit "${local_status}" + ) & + WAVE_PIDS+=("$!") +} + +run_tp_wave() { + local trial="$1" tp="$2" order="$3" + IFS=',' read -r -a mnss <<< "${order}" + WAVE_PIDS=() + preflight_gpus + assert_no_our_server + printf 'WAVE_START trial=%s tp=%s mns=%s\n' "${trial}" "${tp}" "${order}" \ + | tee -a "${OUT}/controller.log" + case "${tp}" in + 4) + launch_config "${trial}" 4 "${mnss[0]}" '0,1,2,3' + launch_config "${trial}" 4 "${mnss[1]}" '4,5,6,7' + wait_for_wave + preflight_gpus + assert_no_our_server + launch_config "${trial}" 4 "${mnss[2]}" '0,1,2,3' + launch_config "${trial}" 4 "${mnss[3]}" '4,5,6,7' + ;; + 2) + launch_config "${trial}" 2 "${mnss[0]}" '0,1' + launch_config "${trial}" 2 "${mnss[1]}" '2,3' + launch_config "${trial}" 2 "${mnss[2]}" '4,5' + launch_config "${trial}" 2 "${mnss[3]}" '6,7' + ;; + 1) + launch_config "${trial}" 1 "${mnss[0]}" '0' + launch_config "${trial}" 1 "${mnss[1]}" '1' + launch_config "${trial}" 1 "${mnss[2]}" '2' + launch_config "${trial}" 1 "${mnss[3]}" '3' + ;; + *) + printf 'ERROR unsupported TP=%s\n' "${tp}" >&2 + return 1 + ;; + esac + wait_for_wave + preflight_gpus + assert_no_our_server + printf 'WAVE_COMPLETE trial=%s tp=%s\n' "${trial}" "${tp}" \ + | tee -a "${OUT}/controller.log" +} + +{ + printf '%s\n' "REAL_LAUNCH_ECHO host=dash0; model=Qwen3-30B-A3B; engine=vLLM-0.20.0+cu129; dtype=BF16; trace=trace-exact-v1/u0p01; requests=129; transform=t_prime=t/TP; per_gpu_offered_rate=0.215 req/s; surface=TP{1,2,4}xMNS{8,16,32,64}; trials=3; fresh_server=true; metrics=mean,p90(TTFT,TPOT,E2E); SLO=not_scored; flashinfer_cache=${FLASHINFER_SHARED_WORKSPACE}; expected_cost=13_H20-GPUh_nominal__41_H20-GPUh_max; output=${OUT}/real" + date -u +START_UTC=%Y-%m-%dT%H:%M:%SZ + mkdir -p "${OUT}/provenance" + mkdir -p "${FLASHINFER_SHARED_WORKSPACE}" + sha256sum "${BASH_SOURCE[0]}" "${RUNNER}" "${CLIENT}" "${PREFILL_CLIENT}" \ + "${MODEL_ROOT}/config.json" > "${OUT}/provenance/real-input.sha256" + for tp in 1 2 4; do + sha256sum "${TRACE_ROOT}/tp${tp}/public/manifest.json" \ + "${TRACE_ROOT}/tp${tp}/private/real_requests.jsonl" \ + >> "${OUT}/provenance/real-input.sha256" + done + "${VENV_ROOT}/bin/vllm" --version > "${OUT}/provenance/vllm.version" + "${VENV_ROOT}/bin/python" -c 'import torch, transformers, vllm; print(f"torch={torch.__version__}"); print(f"transformers={transformers.__version__}"); print(f"vllm={vllm.__version__}")' \ + > "${OUT}/provenance/runtime.versions" + nvidia-smi --query-gpu=index,name,uuid,driver_version,memory.total --format=csv,noheader \ + > "${OUT}/provenance/gpus.before.csv" + + # Rotate topology and MNS order across trials to avoid a fixed temporal order. + run_tp_wave 1 4 '8,16,32,64' + run_tp_wave 1 2 '8,16,32,64' + run_tp_wave 1 1 '8,16,32,64' + run_tp_wave 2 1 '64,32,16,8' + run_tp_wave 2 2 '64,32,16,8' + run_tp_wave 2 4 '64,32,16,8' + run_tp_wave 3 2 '16,32,64,8' + run_tp_wave 3 4 '16,32,64,8' + run_tp_wave 3 1 '16,32,64,8' + + nvidia-smi --query-gpu=index,name,uuid,driver_version,memory.total --format=csv,noheader \ + > "${OUT}/provenance/gpus.after.csv" + find "${OUT}/real" -type f ! -path '*/provenance/artifacts.sha256' -print0 \ + | sort -z | xargs -0 sha256sum > "${OUT}/provenance/real-artifacts.sha256" + date -u +END_UTC=%Y-%m-%dT%H:%M:%SZ + printf '%s\n' 'REAL_SURFACE_COMPLETE' +} >> "${OUT}/controller.log" 2>&1 diff --git a/runs/simulator-tuning-latency-matrix-v0/qwen30-tracepd-u0p01-real-audit.json b/runs/simulator-tuning-latency-matrix-v0/qwen30-tracepd-u0p01-real-audit.json new file mode 100644 index 0000000..a997586 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/qwen30-tracepd-u0p01-real-audit.json @@ -0,0 +1,1362 @@ +{ + "configs": { + "tp1_mns16": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 87355.16047904849, + "pooled_p90_ms": 156177.17167094816, + "pooled_samples": 387, + "trial_mean_of_means_ms": 87355.1604790485, + "trial_mean_of_p90s_ms": 156479.69853299824, + "trial_stddev_of_means_ms": 411.9489877870239 + }, + "tpot_ms": { + "pooled_mean_ms": 20.58492915882586, + "pooled_p90_ms": 25.316827059621797, + "pooled_samples": 387, + "trial_mean_of_means_ms": 20.58492915882586, + "trial_mean_of_p90s_ms": 25.430729349242615, + "trial_stddev_of_means_ms": 0.07031617503255957 + }, + "ttft_ms": { + "pooled_mean_ms": 17378.17525871318, + "pooled_p90_ms": 42739.809082006104, + "pooled_samples": 387, + "trial_mean_of_means_ms": 17378.17525871318, + "trial_mean_of_p90s_ms": 43264.360885717906, + "trial_stddev_of_means_ms": 247.95865334755402 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 87177.4532565533, + "p90_ms": 156177.17167094816, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 20.657075003239317, + "p90_ms": 25.51118326097509, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 17192.74773496445, + "p90_ms": 43084.54651210923, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns16/trial1/results/result.json", + "tp": 1, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 87061.89228269603, + "p90_ms": 156085.63073002733, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 20.58111452736139, + "p90_ms": 25.316827059621797, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 17281.95525217321, + "p90_ms": 42669.12407299969, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns16/trial2/results/result.json", + "tp": 1, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 87826.13589789615, + "p90_ms": 157176.29319801927, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 20.51659794587688, + "p90_ms": 25.464177727130956, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 17659.822789001886, + "p90_ms": 44039.4120720448, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns16/trial3/results/result.json", + "tp": 1, + "trial": 3 + } + ] + }, + "tp1_mns32": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 81051.19884917597, + "pooled_p90_ms": 162908.61000702716, + "pooled_samples": 387, + "trial_mean_of_means_ms": 81051.19884917597, + "trial_mean_of_p90s_ms": 163725.26491332488, + "trial_stddev_of_means_ms": 957.6364127854422 + }, + "tpot_ms": { + "pooled_mean_ms": 23.75712112165005, + "pooled_p90_ms": 29.99736783063116, + "pooled_samples": 387, + "trial_mean_of_means_ms": 23.757121121650055, + "trial_mean_of_p90s_ms": 29.953513044613036, + "trial_stddev_of_means_ms": 0.28945138797341263 + }, + "ttft_ms": { + "pooled_mean_ms": 620.4562074374239, + "pooled_p90_ms": 1931.474626995623, + "pooled_samples": 387, + "trial_mean_of_means_ms": 620.4562074374239, + "trial_mean_of_p90s_ms": 1952.5051703676581, + "trial_stddev_of_means_ms": 4.034284336592585 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 82155.01919551753, + "p90_ms": 165321.14125497174, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 24.088935365685685, + "p90_ms": 30.620627052829327, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 624.0473438989255, + "p90_ms": 1953.5144560504705, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns32/trial1/results/result.json", + "tp": 1, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 80442.25376262421, + "p90_ms": 162710.11894091498, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 23.556478044326088, + "p90_ms": 29.607802025608333, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 616.0909594027614, + "p90_ms": 1931.474626995623, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns32/trial2/results/result.json", + "tp": 1, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 80556.32358938617, + "p90_ms": 163144.53454408795, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 23.62594995493838, + "p90_ms": 29.632110055401448, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 621.2303190105847, + "p90_ms": 1972.5264280568808, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns32/trial3/results/result.json", + "tp": 1, + "trial": 3 + } + ] + }, + "tp1_mns64": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 82253.7312629709, + "pooled_p90_ms": 165355.2623600699, + "pooled_samples": 387, + "trial_mean_of_means_ms": 82253.73126297089, + "trial_mean_of_p90s_ms": 166168.38832168528, + "trial_stddev_of_means_ms": 276.04080210780324 + }, + "tpot_ms": { + "pooled_mean_ms": 24.114592194742286, + "pooled_p90_ms": 30.355145227612027, + "pooled_samples": 387, + "trial_mean_of_means_ms": 24.114592194742286, + "trial_mean_of_p90s_ms": 30.441765467132914, + "trial_stddev_of_means_ms": 0.09945812871448045 + }, + "ttft_ms": { + "pooled_mean_ms": 619.7960944975623, + "pooled_p90_ms": 1956.7296199966222, + "pooled_samples": 387, + "trial_mean_of_means_ms": 619.7960944975623, + "trial_mean_of_p90s_ms": 1975.1493813625227, + "trial_stddev_of_means_ms": 3.103486103420667 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 82496.79665562836, + "p90_ms": 166090.5934929615, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 24.210340237338897, + "p90_ms": 30.310964708207926, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 616.759402438269, + "p90_ms": 1956.7296199966222, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns64/trial1/results/result.json", + "tp": 1, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 81953.6266326841, + "p90_ms": 165355.2623600699, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 24.011798653948865, + "p90_ms": 30.622402682139015, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 622.9623185557335, + "p90_ms": 1988.6804190464318, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns64/trial2/results/result.json", + "tp": 1, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 82310.77050060022, + "p90_ms": 167059.3091120245, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 24.121637692939093, + "p90_ms": 30.391929011051804, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 619.6665624986844, + "p90_ms": 1980.038105044514, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns64/trial3/results/result.json", + "tp": 1, + "trial": 3 + } + ] + }, + "tp1_mns8": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 137996.3827215202, + "pooled_p90_ms": 222136.04307791684, + "pooled_samples": 387, + "trial_mean_of_means_ms": 137996.38272152017, + "trial_mean_of_p90s_ms": 223625.49974532644, + "trial_stddev_of_means_ms": 2140.2325979814923 + }, + "tpot_ms": { + "pooled_mean_ms": 14.282724020345517, + "pooled_p90_ms": 16.331995667962982, + "pooled_samples": 387, + "trial_mean_of_means_ms": 14.282724020345519, + "trial_mean_of_p90s_ms": 16.30076252284316, + "trial_stddev_of_means_ms": 0.06621858834469496 + }, + "ttft_ms": { + "pooled_mean_ms": 88479.94003801016, + "pooled_p90_ms": 168185.5451229494, + "pooled_samples": 387, + "trial_mean_of_means_ms": 88479.94003801014, + "trial_mean_of_p90s_ms": 168658.08321364844, + "trial_stddev_of_means_ms": 1884.4627766230617 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 140225.59553917762, + "p90_ms": 227556.57476198394, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 14.352062305808436, + "p90_ms": 16.440255243105753, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 90447.42801240031, + "p90_ms": 172230.12473701965, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns8/trial1/results/result.json", + "tp": 1, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 135957.89686511378, + "p90_ms": 219851.20729799382, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 14.220143355456331, + "p90_ms": 16.233834194804228, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 86691.24682294385, + "p90_ms": 165558.5797809763, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns8/trial2/results/result.json", + "tp": 1, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 137805.65576026912, + "p90_ms": 223468.71717600152, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 14.27596639977179, + "p90_ms": 16.22819813061951, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 88301.14527868626, + "p90_ms": 168185.5451229494, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp1_mns8/trial3/results/result.json", + "tp": 1, + "trial": 3 + } + ] + }, + "tp2_mns16": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 82936.70731938194, + "pooled_p90_ms": 137726.37015290093, + "pooled_samples": 387, + "trial_mean_of_means_ms": 82936.70731938194, + "trial_mean_of_p90s_ms": 138201.4721223386, + "trial_stddev_of_means_ms": 300.9695124191068 + }, + "tpot_ms": { + "pooled_mean_ms": 14.09701977962943, + "pooled_p90_ms": 16.149621762221777, + "pooled_samples": 387, + "trial_mean_of_means_ms": 14.097019779629429, + "trial_mean_of_p90s_ms": 16.09430993926139, + "trial_stddev_of_means_ms": 0.009955627205404972 + }, + "ttft_ms": { + "pooled_mean_ms": 34442.970551674625, + "pooled_p90_ms": 77922.3588249879, + "pooled_samples": 387, + "trial_mean_of_means_ms": 34442.970551674625, + "trial_mean_of_p90s_ms": 77968.50966336206, + "trial_stddev_of_means_ms": 245.29483636828064 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 82589.82040462908, + "p90_ms": 137726.37015290093, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 14.086598798707497, + "p90_ms": 16.11459195926328, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 34161.05435777441, + "p90_ms": 77612.42442007642, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns16/trial1/results/result.json", + "tp": 2, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 83128.44691530518, + "p90_ms": 138288.39251911268, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 14.106433468934316, + "p90_ms": 16.016419614614367, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 34560.222078616505, + "p90_ms": 78136.86232501641, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns16/trial2/results/result.json", + "tp": 2, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 83091.85463821157, + "p90_ms": 138589.6536950022, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 14.098027071246479, + "p90_ms": 16.151918243906525, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 34607.63521863295, + "p90_ms": 78156.24224499334, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns16/trial3/results/result.json", + "tp": 2, + "trial": 3 + } + ] + }, + "tp2_mns32": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 62267.50376182025, + "pooled_p90_ms": 117239.33770309668, + "pooled_samples": 387, + "trial_mean_of_means_ms": 62267.503761820255, + "trial_mean_of_p90s_ms": 117291.74644201218, + "trial_stddev_of_means_ms": 112.21230246579601 + }, + "tpot_ms": { + "pooled_mean_ms": 17.99748422275332, + "pooled_p90_ms": 21.280696114935413, + "pooled_samples": 387, + "trial_mean_of_means_ms": 17.997484222753318, + "trial_mean_of_p90s_ms": 21.350839715170533, + "trial_stddev_of_means_ms": 0.024026728370897792 + }, + "ttft_ms": { + "pooled_mean_ms": 1050.6337350342683, + "pooled_p90_ms": 2521.7670870479196, + "pooled_samples": 387, + "trial_mean_of_means_ms": 1050.6337350342683, + "trial_mean_of_p90s_ms": 2558.191083294029, + "trial_stddev_of_means_ms": 3.4732818152181575 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 62313.896939208076, + "p90_ms": 117369.6845669765, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 18.008634396299446, + "p90_ms": 21.280696114935413, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 1047.8459898673013, + "p90_ms": 2537.152646924369, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns32/trial1/results/result.json", + "tp": 2, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 62139.53429665742, + "p90_ms": 117239.33770309668, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 17.969908251938307, + "p90_ms": 21.518877847159697, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 1054.5246146724003, + "p90_ms": 2507.6513229869306, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns32/trial2/results/result.json", + "tp": 2, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 62349.08004959526, + "p90_ms": 117266.21705596335, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 18.01391002002221, + "p90_ms": 21.252945183416482, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 1049.5306005631032, + "p90_ms": 2629.7692799707875, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns32/trial3/results/result.json", + "tp": 2, + "trial": 3 + } + ] + }, + "tp2_mns64": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 62353.63015175185, + "pooled_p90_ms": 118392.33276096638, + "pooled_samples": 387, + "trial_mean_of_means_ms": 62353.63015175186, + "trial_mean_of_p90s_ms": 118379.07191404763, + "trial_stddev_of_means_ms": 167.93391905527702 + }, + "tpot_ms": { + "pooled_mean_ms": 18.239052597893746, + "pooled_p90_ms": 21.560091553416488, + "pooled_samples": 387, + "trial_mean_of_means_ms": 18.239052597893746, + "trial_mean_of_p90s_ms": 21.583819967748195, + "trial_stddev_of_means_ms": 0.049751307212647984 + }, + "ttft_ms": { + "pooled_mean_ms": 375.75464864569483, + "pooled_p90_ms": 1148.593939957209, + "pooled_samples": 387, + "trial_mean_of_means_ms": 375.7546486456948, + "trial_mean_of_p90s_ms": 1156.997084307174, + "trial_stddev_of_means_ms": 0.8716131740409625 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 62355.466284980466, + "p90_ms": 118664.41726102494, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 18.244751771577278, + "p90_ms": 21.63474199378721, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 376.58390670729614, + "p90_ms": 1167.6474569831043, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns64/trial1/results/result.json", + "tp": 2, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 62520.63847564941, + "p90_ms": 118612.22067801282, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 18.285708890842777, + "p90_ms": 21.623841600560585, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 374.8460987451046, + "p90_ms": 1154.7498559812084, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns64/trial2/results/result.json", + "tp": 2, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 62184.7856946257, + "p90_ms": 117860.57780310512, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 18.18669713126118, + "p90_ms": 21.492876308896783, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 375.83394048468375, + "p90_ms": 1148.593939957209, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns64/trial3/results/result.json", + "tp": 2, + "trial": 3 + } + ] + }, + "tp2_mns8": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 128296.94038681783, + "pooled_p90_ms": 215848.26136403717, + "pooled_samples": 387, + "trial_mean_of_means_ms": 128296.94038681783, + "trial_mean_of_p90s_ms": 216133.82774300408, + "trial_stddev_of_means_ms": 253.41927200868048 + }, + "tpot_ms": { + "pooled_mean_ms": 9.244370614307101, + "pooled_p90_ms": 10.190386958999568, + "pooled_samples": 387, + "trial_mean_of_means_ms": 9.244370614307103, + "trial_mean_of_p90s_ms": 10.191423497770423, + "trial_stddev_of_means_ms": 0.011292566107855785 + }, + "ttft_ms": { + "pooled_mean_ms": 95920.08463817694, + "pooled_p90_ms": 183878.83744493593, + "pooled_samples": 387, + "trial_mean_of_means_ms": 95920.08463817695, + "trial_mean_of_p90s_ms": 184091.12584633598, + "trial_stddev_of_means_ms": 246.8869690217293 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 128460.99048956431, + "p90_ms": 216078.09481292497, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 9.23733635271789, + "p90_ms": 10.187162502093368, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 96106.82740485722, + "p90_ms": 184006.2059359625, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns8/trial1/results/result.json", + "tp": 2, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 128005.065388418, + "p90_ms": 215848.26136403717, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 9.238379252990907, + "p90_ms": 10.190386958999568, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 95640.16961438723, + "p90_ms": 183862.3900989769, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns8/trial2/results/result.json", + "tp": 2, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 128424.76528247118, + "p90_ms": 216475.12705205008, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 9.257396237212511, + "p90_ms": 10.19672103221833, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 96013.2568952864, + "p90_ms": 184404.78150406852, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp2_mns8/trial3/results/result.json", + "tp": 2, + "trial": 3 + } + ] + }, + "tp4_mns16": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 65779.08377038215, + "pooled_p90_ms": 110374.3328540586, + "pooled_samples": 387, + "trial_mean_of_means_ms": 65779.08377038215, + "trial_mean_of_p90s_ms": 110593.98958502182, + "trial_stddev_of_means_ms": 79.17219724389233 + }, + "tpot_ms": { + "pooled_mean_ms": 8.806983272811275, + "pooled_p90_ms": 9.626243685920313, + "pooled_samples": 387, + "trial_mean_of_means_ms": 8.806983272811275, + "trial_mean_of_p90s_ms": 9.623114289857485, + "trial_stddev_of_means_ms": 0.00993955000910551 + }, + "ttft_ms": { + "pooled_mean_ms": 35042.61012925094, + "pooled_p90_ms": 74281.28085809294, + "pooled_samples": 387, + "trial_mean_of_means_ms": 35042.610129250934, + "trial_mean_of_p90s_ms": 74402.09861169569, + "trial_stddev_of_means_ms": 73.61057537965252 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 65867.343166196, + "p90_ms": 110630.9638969833, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 8.809118914012139, + "p90_ms": 9.634016542784963, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 35125.26244937818, + "p90_ms": 74573.83236999158, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns16/trial1/results/result.json", + "tp": 4, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 65755.59253424818, + "p90_ms": 110776.67200402357, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 8.815681410017397, + "p90_ms": 9.632762871791096, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 34984.109502939085, + "p90_ms": 74351.18260700256, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns16/trial2/results/result.json", + "tp": 4, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 65714.31561070224, + "p90_ms": 110374.3328540586, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 8.79614949440429, + "p90_ms": 9.602563454996394, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 35018.45843543555, + "p90_ms": 74281.28085809294, + "samples": 129 + } + }, + "mns": 16, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns16/trial3/results/result.json", + "tp": 4, + "trial": 3 + } + ] + }, + "tp4_mns32": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 47944.52887711636, + "pooled_p90_ms": 83825.7212981116, + "pooled_samples": 387, + "trial_mean_of_means_ms": 47944.52887711636, + "trial_mean_of_p90s_ms": 84201.27462669431, + "trial_stddev_of_means_ms": 314.84689084624006 + }, + "tpot_ms": { + "pooled_mean_ms": 12.181116414313188, + "pooled_p90_ms": 13.75045588274547, + "pooled_samples": 387, + "trial_mean_of_means_ms": 12.181116414313186, + "trial_mean_of_p90s_ms": 13.764242049706512, + "trial_stddev_of_means_ms": 0.05312117190893007 + }, + "ttft_ms": { + "pooled_mean_ms": 6296.228585964807, + "pooled_p90_ms": 19159.870272967964, + "pooled_samples": 387, + "trial_mean_of_means_ms": 6296.228585964807, + "trial_mean_of_p90s_ms": 19163.547096347127, + "trial_stddev_of_means_ms": 141.30091703005428 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 48268.361499181105, + "p90_ms": 84625.69403997622, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 12.233328917928599, + "p90_ms": 13.809786885754699, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 6448.833544885689, + "p90_ms": 19575.953547027893, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns32/trial1/results/result.json", + "tp": 4, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 47639.51159005465, + "p90_ms": 83825.7212981116, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 12.127130963187064, + "p90_ms": 13.718962375832668, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 6169.928666188873, + "p90_ms": 18779.106684960425, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns32/trial2/results/result.json", + "tp": 4, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 47925.713542113335, + "p90_ms": 84152.4085419951, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 12.182889361823895, + "p90_ms": 13.763976887532172, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 6269.923546819861, + "p90_ms": 19135.58105705306, + "samples": 129 + } + }, + "mns": 32, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns32/trial3/results/result.json", + "tp": 4, + "trial": 3 + } + ] + }, + "tp4_mns64": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 44985.13828221082, + "pooled_p90_ms": 83763.61756504048, + "pooled_samples": 387, + "trial_mean_of_means_ms": 44985.13828221082, + "trial_mean_of_p90s_ms": 83934.7783580112, + "trial_stddev_of_means_ms": 124.87733418991748 + }, + "tpot_ms": { + "pooled_mean_ms": 13.178025610291787, + "pooled_p90_ms": 15.389331985205848, + "pooled_samples": 387, + "trial_mean_of_means_ms": 13.178025610291789, + "trial_mean_of_p90s_ms": 15.426827041851524, + "trial_stddev_of_means_ms": 0.04180975339667582 + }, + "ttft_ms": { + "pooled_mean_ms": 245.9527667526406, + "pooled_p90_ms": 685.5076659703627, + "pooled_samples": 387, + "trial_mean_of_means_ms": 245.9527667526406, + "trial_mean_of_p90s_ms": 693.9916636717195, + "trial_stddev_of_means_ms": 2.5068538619550047 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 45089.82795085072, + "p90_ms": 84275.81048000138, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 13.210321995984648, + "p90_ms": 15.465764498700636, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 245.75881352882774, + "p90_ms": 685.5076659703627, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns64/trial1/results/result.json", + "tp": 4, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 44846.91910041485, + "p90_ms": 83676.52938805986, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 13.130800688491455, + "p90_ms": 15.375762174125736, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 243.54852308327867, + "p90_ms": 689.600904006511, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns64/trial2/results/result.json", + "tp": 4, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 45018.66779536688, + "p90_ms": 83851.99520597234, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 13.192954146399261, + "p90_ms": 15.438954452728199, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 248.5509636458154, + "p90_ms": 706.8664210382849, + "samples": 129 + } + }, + "mns": 64, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns64/trial3/results/result.json", + "tp": 4, + "trial": 3 + } + ] + }, + "tp4_mns8": { + "metrics": { + "e2e_ms": { + "pooled_mean_ms": 126405.41922729714, + "pooled_p90_ms": 209092.4072249327, + "pooled_samples": 387, + "trial_mean_of_means_ms": 126405.41922729714, + "trial_mean_of_p90s_ms": 209266.24333831327, + "trial_stddev_of_means_ms": 13.636256279124204 + }, + "tpot_ms": { + "pooled_mean_ms": 6.757398869596114, + "pooled_p90_ms": 7.242299378971089, + "pooled_samples": 387, + "trial_mean_of_means_ms": 6.757398869596114, + "trial_mean_of_p90s_ms": 7.256281134648607, + "trial_stddev_of_means_ms": 0.0021221970115212385 + }, + "ttft_ms": { + "pooled_mean_ms": 102605.87212059843, + "pooled_p90_ms": 193359.7398570273, + "pooled_samples": 387, + "trial_mean_of_means_ms": 102605.87212059843, + "trial_mean_of_p90s_ms": 193580.62508733323, + "trial_stddev_of_means_ms": 31.928988273539215 + } + }, + "trials": [ + { + "metrics": { + "e2e_ms": { + "mean_ms": 126396.78182327477, + "p90_ms": 209295.79258395825, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 6.757000882714381, + "p90_ms": 7.275805782008363, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 102581.32991039865, + "p90_ms": 193756.58583000768, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns8/trial1/results/result.json", + "tp": 4, + "trial": 1 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 126421.1394171115, + "p90_ms": 209410.53020604886, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 6.759691884285006, + "p90_ms": 7.268115562828778, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 102641.97010612892, + "p90_ms": 193625.54957496468, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns8/trial2/results/result.json", + "tp": 4, + "trial": 2 + }, + { + "metrics": { + "e2e_ms": { + "mean_ms": 126398.33644150518, + "p90_ms": 209092.4072249327, + "samples": 129 + }, + "tpot_ms": { + "mean_ms": 6.755503841788953, + "p90_ms": 7.224922059108682, + "samples": 129 + }, + "ttft_ms": { + "mean_ms": 102594.31634526773, + "p90_ms": 193359.7398570273, + "samples": 129 + } + }, + "mns": 8, + "requests": 129, + "result_path": "/home/admin/cpfs/wjh/aituner/simulator-tuning-latency-q30-tp-normalized-u0p01-20260717/real/tp4_mns8/trial3/results/result.json", + "tp": 4, + "trial": 3 + } + ] + } + }, + "schema": "qwen30-tp-normalized-real-surface-audit-v1", + "trace_manifests": { + "tp1": { + "global_offered_request_rate": 0.215, + "normalized_row_vector_sha256": "e0c0f9b327473dd2b34149eb478a13b464c26df2201ec8e43c6493c207b08786", + "per_gpu_offered_request_rate": 0.215, + "private_jsonl_sha256": "ba61259d2650b225eb3b3b2b6b9dec29503f9f616b4c573869f3b69881cc6e13", + "requests": 129 + }, + "tp2": { + "global_offered_request_rate": 0.43, + "normalized_row_vector_sha256": "b435481838bee09b18d0b51c23901488a4fe2f08aee4e9b5a46d2f9c160717fc", + "per_gpu_offered_request_rate": 0.215, + "private_jsonl_sha256": "b30ffdb42ee52e17f099d11e47d9f841673e18a05defccc64bf03ce3802d3576", + "requests": 129 + }, + "tp4": { + "global_offered_request_rate": 0.86, + "normalized_row_vector_sha256": "0eefd2d58674c741a7680b90554ed89cc1655478a709c49670a93d09f156ff2c", + "per_gpu_offered_request_rate": 0.215, + "private_jsonl_sha256": "a8c1d9641107978e960ecc78cd3d43fdce925e3b23d7635ccf0633fcadc3d812", + "requests": 129 + } + }, + "winners": { + "e2e_ms:pooled_mean_ms": { + "ranking": [ + "tp4_mns64", + "tp4_mns32", + "tp2_mns32", + "tp2_mns64", + "tp4_mns16", + "tp1_mns32", + "tp1_mns64", + "tp2_mns16", + "tp1_mns16", + "tp4_mns8", + "tp2_mns8", + "tp1_mns8" + ], + "winner": "tp4_mns64", + "winner_value_ms": 44985.13828221082 + }, + "e2e_ms:pooled_p90_ms": { + "ranking": [ + "tp4_mns64", + "tp4_mns32", + "tp4_mns16", + "tp2_mns32", + "tp2_mns64", + "tp2_mns16", + "tp1_mns16", + "tp1_mns32", + "tp1_mns64", + "tp4_mns8", + "tp2_mns8", + "tp1_mns8" + ], + "winner": "tp4_mns64", + "winner_value_ms": 83763.61756504048 + }, + "tpot_ms:pooled_mean_ms": { + "ranking": [ + "tp4_mns8", + "tp4_mns16", + "tp2_mns8", + "tp4_mns32", + "tp4_mns64", + "tp2_mns16", + "tp1_mns8", + "tp2_mns32", + "tp2_mns64", + "tp1_mns16", + "tp1_mns32", + "tp1_mns64" + ], + "winner": "tp4_mns8", + "winner_value_ms": 6.757398869596114 + }, + "tpot_ms:pooled_p90_ms": { + "ranking": [ + "tp4_mns8", + "tp4_mns16", + "tp2_mns8", + "tp4_mns32", + "tp4_mns64", + "tp2_mns16", + "tp1_mns8", + "tp2_mns32", + "tp2_mns64", + "tp1_mns16", + "tp1_mns32", + "tp1_mns64" + ], + "winner": "tp4_mns8", + "winner_value_ms": 7.242299378971089 + }, + "ttft_ms:pooled_mean_ms": { + "ranking": [ + "tp4_mns64", + "tp2_mns64", + "tp1_mns64", + "tp1_mns32", + "tp2_mns32", + "tp4_mns32", + "tp1_mns16", + "tp2_mns16", + "tp4_mns16", + "tp1_mns8", + "tp2_mns8", + "tp4_mns8" + ], + "winner": "tp4_mns64", + "winner_value_ms": 245.9527667526406 + }, + "ttft_ms:pooled_p90_ms": { + "ranking": [ + "tp4_mns64", + "tp2_mns64", + "tp1_mns32", + "tp1_mns64", + "tp2_mns32", + "tp4_mns32", + "tp1_mns16", + "tp4_mns16", + "tp2_mns16", + "tp1_mns8", + "tp2_mns8", + "tp4_mns8" + ], + "winner": "tp4_mns64", + "winner_value_ms": 685.5076659703627 + } + } +} diff --git a/runs/simulator-tuning-latency-matrix-v0/qwen30-tracepd-u0p01-real-audit.md b/runs/simulator-tuning-latency-matrix-v0/qwen30-tracepd-u0p01-real-audit.md new file mode 100644 index 0000000..935447a --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/qwen30-tracepd-u0p01-real-audit.md @@ -0,0 +1,27 @@ +# Qwen3-30B-A3B TP-normalized Trace-PD: real vLLM audit + +All 36 fresh-server trials passed the trace contract: 129/129 exact-usage requests per trial. The smoke run is excluded. Values below pool the three trials (387 requests/config); p90 uses nearest-rank order statistics. + +| Config | TTFT mean/p90 (ms) | TPOT mean/p90 (ms) | E2E mean/p90 (ms) | +|---|---:|---:|---:| +| tp1_mns8 | 88479.9/168185.5 | 14.3/16.3 | 137996.4/222136.0 | +| tp1_mns16 | 17378.2/42739.8 | 20.6/25.3 | 87355.2/156177.2 | +| tp1_mns32 | 620.5/1931.5 | 23.8/30.0 | 81051.2/162908.6 | +| tp1_mns64 | 619.8/1956.7 | 24.1/30.4 | 82253.7/165355.3 | +| tp2_mns8 | 95920.1/183878.8 | 9.2/10.2 | 128296.9/215848.3 | +| tp2_mns16 | 34443.0/77922.4 | 14.1/16.1 | 82936.7/137726.4 | +| tp2_mns32 | 1050.6/2521.8 | 18.0/21.3 | 62267.5/117239.3 | +| tp2_mns64 | 375.8/1148.6 | 18.2/21.6 | 62353.6/118392.3 | +| tp4_mns8 | 102605.9/193359.7 | 6.8/7.2 | 126405.4/209092.4 | +| tp4_mns16 | 35042.6/74281.3 | 8.8/9.6 | 65779.1/110374.3 | +| tp4_mns32 | 6296.2/19159.9 | 12.2/13.8 | 47944.5/83825.7 | +| tp4_mns64 | 246.0/685.5 | 13.2/15.4 | 44985.1/83763.6 | + +## Per-metric winners + +- `ttft_ms:pooled_mean_ms`: `tp4_mns64` (246.0 ms) +- `ttft_ms:pooled_p90_ms`: `tp4_mns64` (685.5 ms) +- `tpot_ms:pooled_mean_ms`: `tp4_mns8` (6.8 ms) +- `tpot_ms:pooled_p90_ms`: `tp4_mns8` (7.2 ms) +- `e2e_ms:pooled_mean_ms`: `tp4_mns64` (44985.1 ms) +- `e2e_ms:pooled_p90_ms`: `tp4_mns64` (83763.6 ms) diff --git a/runs/simulator-tuning-latency-matrix-v0/run_qwen30_exact_trace_real_anchor_jit_tolerant.sh b/runs/simulator-tuning-latency-matrix-v0/run_qwen30_exact_trace_real_anchor_jit_tolerant.sh new file mode 100644 index 0000000..8ba7c21 --- /dev/null +++ b/runs/simulator-tuning-latency-matrix-v0/run_qwen30_exact_trace_real_anchor_jit_tolerant.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash + +# Isolated copy of the real-anchor harness. It extends the startup budget for +# an uncached H20 FlashInfer MoE JIT and uses an artifact-pinned trace client +# that routes requests to the explicit server alias. It is run from the clean +# source directory so its warm-up client remains snapshot-pinned. + +set -euo pipefail + +OUTPUT_ROOT="${OUTPUT_ROOT:?OUTPUT_ROOT is required}" +REQUESTS_FILE="${REQUESTS_FILE:?REQUESTS_FILE is required}" +TP="${TP:?TP is required}" +MNS="${MNS:?MNS is required}" +TRACE_LABEL="${TRACE_LABEL:?TRACE_LABEL is required}" +SERVER_PORT="${SERVER_PORT:?SERVER_PORT is required}" +VENV_ROOT="${VENV_ROOT:-/tmp/wjh/venvs/vllm-0.20.0-cu129-profiler-v1}" +MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}" +FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE:-/tmp/wjh/flashinfer-workspace-vllm020-profiler-v1}" +SERVER_READY_ATTEMPTS="${SERVER_READY_ATTEMPTS:-300}" +EXACT_TRACE_CLIENT="${EXACT_TRACE_CLIENT:-qwen30_exact_trace_client.py}" +SERVED_MODEL="qwen3-30b-exact-trace" +SERVER_PID="" + +mkdir -p "${OUTPUT_ROOT}/logs" "${OUTPUT_ROOT}/provenance" "${OUTPUT_ROOT}/results" \ + "${FLASHINFER_WORKSPACE_BASE}" +exec > >(tee -a "${OUTPUT_ROOT}/logs/controller.log") 2>&1 + +cleanup() { + if [[ -n "${SERVER_PID}" ]] && kill -0 "${SERVER_PID}" 2>/dev/null; then + kill -TERM -- "-${SERVER_PID}" 2>/dev/null || true + for _ in $(seq 1 30); do + kill -0 "${SERVER_PID}" 2>/dev/null || break + sleep 1 + done + kill -KILL -- "-${SERVER_PID}" 2>/dev/null || true + fi + SERVER_PID="" +} +trap cleanup EXIT HUP INT TERM + +IFS=',' read -r -a GPU_IDS <<< "${CUDA_VISIBLE_DEVICES:?fleet GPU allocation is required}" +if [[ "${#GPU_IDS[@]}" -ne "${TP}" ]]; then + echo "ERROR: expected ${TP} GPUs, got ${CUDA_VISIBLE_DEVICES}" >&2 + exit 1 +fi + +REQUEST_COUNT="$(wc -l < "${REQUESTS_FILE}")" +echo "QWEN30_EXACT_TRACE_REAL_LAUNCH_ECHO host=$(hostname) gpus=${CUDA_VISIBLE_DEVICES} model=${MODEL_ROOT} runtime=vLLM-0.20.0+cu129 dtype=BF16 config=TP${TP}_MNS${MNS}_MBT8192 trace=${TRACE_LABEL} requests=${REQUEST_COUNT} source=${REQUESTS_FILE} arrivals=TP-normalized input_output_prompt=exact prefix=on block=16 metrics=mean,p90(TTFT,TPOT,E2E) SLO=not-scored flashinfer_workspace=${FLASHINFER_WORKSPACE_BASE} output=${OUTPUT_ROOT} ready_budget_s=$((SERVER_READY_ATTEMPTS * 3)) hard_wall=4200s" +date -u +"START_UTC=%Y-%m-%dT%H:%M:%SZ" +sha256sum "${BASH_SOURCE[0]}" "${EXACT_TRACE_CLIENT}" \ + ../frontier-phase-factorial-v0/qwen30_prefill_client.py \ + > "${OUTPUT_ROOT}/provenance/source.sha256" +sha256sum "${REQUESTS_FILE}" > "${OUTPUT_ROOT}/provenance/requests.sha256" +sha256sum "${MODEL_ROOT}/config.json" > "${OUTPUT_ROOT}/provenance/model.sha256" +nvidia-smi --query-gpu=index,name,uuid,driver_version --format=csv,noheader \ + > "${OUTPUT_ROOT}/provenance/gpus.csv" + +export TOKENIZERS_PARALLELISM=false +export VLLM_USE_V1=1 +export TORCH_CUDA_ARCH_LIST=9.0 +export HF_HUB_OFFLINE=1 +export TRANSFORMERS_OFFLINE=1 +export FLASHINFER_WORKSPACE_BASE +ulimit -n 65536 + +setsid "${VENV_ROOT}/bin/vllm" serve "${MODEL_ROOT}" \ + --host 127.0.0.1 --port "${SERVER_PORT}" --served-model-name "${SERVED_MODEL}" \ + --tensor-parallel-size "${TP}" --gpu-memory-utilization 0.92 \ + --max-model-len 40960 --max-num-batched-tokens 8192 --max-num-seqs "${MNS}" \ + --enable-prefix-caching --enable-chunked-prefill --no-enable-log-requests \ + > "${OUTPUT_ROOT}/logs/server.log" 2>&1 & +SERVER_PID=$! +READY=0 +for _ in $(seq 1 "${SERVER_READY_ATTEMPTS}"); do + if curl -fsS --max-time 2 "http://127.0.0.1:${SERVER_PORT}/v1/models" \ + > "${OUTPUT_ROOT}/results/models.json" 2>/dev/null; then + READY=1 + break + fi + if ! kill -0 "${SERVER_PID}" 2>/dev/null; then + tail -200 "${OUTPUT_ROOT}/logs/server.log" + exit 1 + fi + sleep 3 +done +if [[ "${READY}" -ne 1 ]]; then + echo "ERROR server did not become ready within $((SERVER_READY_ATTEMPTS * 3)) seconds" >&2 + tail -200 "${OUTPUT_ROOT}/logs/server.log" + exit 1 +fi + +"${VENV_ROOT}/bin/python" ../frontier-phase-factorial-v0/qwen30_prefill_client.py \ + --port "${SERVER_PORT}" --served-model "${SERVED_MODEL}" \ + --model-path "${MODEL_ROOT}" --rate 1 --requests 4 --input-tokens 512 \ + --output "${OUTPUT_ROOT}/results/warmup.json" + +"${VENV_ROOT}/bin/python" "${EXACT_TRACE_CLIENT}" \ + --port "${SERVER_PORT}" --requests-file "${REQUESTS_FILE}" \ + --output "${OUTPUT_ROOT}/results/result.json" --served-model "${SERVED_MODEL}" \ + --tpot-slo-ms 150 \ + --timeout-seconds 1800 + +cleanup +find "${OUTPUT_ROOT}" -type f ! -path '*/provenance/artifacts.sha256' -print0 \ + | sort -z | xargs -0 sha256sum > "${OUTPUT_ROOT}/provenance/artifacts.sha256" +date -u +"END_UTC=%Y-%m-%dT%H:%M:%SZ" +echo "QWEN30_EXACT_TRACE_REAL_ANCHOR_COMPLETE" diff --git a/scripts/plot_simulator_fidelity.py b/scripts/plot_simulator_fidelity.py new file mode 100644 index 0000000..88c0ac0 --- /dev/null +++ b/scripts/plot_simulator_fidelity.py @@ -0,0 +1,256 @@ +#!/usr/bin/env python3 +"""Render the per-config figures embedded by simulator-fidelity.md.""" + +from __future__ import annotations + +import json +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + + +ROOT = Path(__file__).resolve().parents[1] +ASSET_DIR = ROOT / "docs" / "assets" / "simulator-fidelity" +DATA_PATH = ASSET_DIR / "data.json" +T0_COMPARISON_PATH = ( + ROOT / "runs" / "frontier-multicase-sufficiency-v1" / "results" / "t0-final" / "comparison.json" +) + +REAL_COLOR = "#3D3D3D" +PROFILE_COLOR = "#E15759" +CALIBRATED_COLOR = "#4E79A7" + + +def load_data() -> dict: + with DATA_PATH.open(encoding="utf-8") as handle: + return json.load(handle) + + +def style_axis(ax: plt.Axes) -> None: + ax.grid(axis="y", alpha=0.23, linewidth=0.8) + ax.set_axisbelow(True) + ax.spines["top"].set_visible(False) + ax.spines["right"].set_visible(False) + + +def save(fig: plt.Figure, name: str) -> None: + ASSET_DIR.mkdir(parents=True, exist_ok=True) + fig.savefig(ASSET_DIR / f"{name}.png", dpi=220, bbox_inches="tight", facecolor="white") + fig.savefig(ASSET_DIR / f"{name}.svg", bbox_inches="tight", facecolor="white") + + +def mark_top(ax: plt.Axes, x: float, y: float, text: str, color: str) -> None: + ax.annotate( + text, + xy=(x, y), + xytext=(0, 7), + textcoords="offset points", + ha="center", + va="bottom", + color=color, + fontsize=8.5, + weight="bold", + ) + + +def plot_qwen30(data: dict) -> None: + case = data["qwen30_mixed"] + rows = sorted(case["configs"], key=lambda row: (-row["real"], row["name"])) + labels = [f"TP{row['tp']}\nMNS{row['mns']}" for row in rows] + real = np.array([row["real"] for row in rows]) + profile = np.array([row["frontier_profile_only"] for row in rows]) + calibrated = np.array([row["frontier_calibrated"] for row in rows]) + x = np.arange(len(rows)) + width = 0.38 + + fig, axes = plt.subplots(2, 1, figsize=(13.8, 8.7), sharex=True, sharey=True) + panels = [ + ( + axes[0], + profile, + PROFILE_COLOR, + "Frontier: operator profiles only", + "top-set miss · Kendall τ-b = 0.000 · worst-case regret = 25.63%", + set(case["profile_only_metrics"]["simulator_top_set"]), + ), + ( + axes[1], + calibrated, + CALIBRATED_COLOR, + "Frontier: frozen per-TP calibration", + "top-set hit · Kendall τ-b = 0.967 · worst-case regret = 0.76%", + set(case["calibrated_metrics"]["simulator_top_set"]), + ), + ] + real_top = set(case["calibrated_metrics"]["real_top_set"]) + + for ax, simulated, color, sim_label, subtitle, simulator_top in panels: + real_bars = ax.bar(x - width / 2, real, width, color=REAL_COLOR, label="Real vLLM") + sim_bars = ax.bar(x + width / 2, simulated, width, color=color, label=sim_label) + style_axis(ax) + ax.set_ylabel("SLO-feasible req/s/GPU") + ax.set_title(subtitle, loc="left", fontsize=11, weight="bold") + ax.set_ylim(0, 4.35) + ax.legend(frameon=False, ncol=2, loc="upper right") + for idx, row in enumerate(rows): + if row["name"] in real_top: + real_bars[idx].set_edgecolor("#000000") + real_bars[idx].set_linewidth(2.0) + mark_top(ax, idx - width / 2, real[idx], "real top", REAL_COLOR) + if row["name"] in simulator_top: + sim_bars[idx].set_edgecolor(color) + sim_bars[idx].set_linewidth(2.0) + sim_bars[idx].set_hatch("///") + mark_top(ax, idx + width / 2, simulated[idx], "sim top", color) + + axes[1].set_xticks(x, labels=labels) + axes[1].set_xlabel("Configuration (ordered by measured real capacity)") + fig.suptitle( + "Qwen3-30B mixed serving: calibration changes the config-selection result", + fontsize=14, + weight="bold", + y=0.995, + ) + fig.tight_layout(rect=(0, 0, 1, 0.97)) + save(fig, "qwen30-mixed-config-ranking") + plt.close(fig) + + +def plot_qwen235(data: dict) -> None: + case = data["qwen235_prefill"] + rows = sorted(case["configs"], key=lambda row: (-row["real"], row["name"])) + labels = [f"TP{row['tp']} MNS{row['mns']}\nMBT{row['mbt'] // 1024}K" for row in rows] + real = np.array([row["real"] for row in rows]) + frontier = np.array([row["frontier"] for row in rows]) + x = np.arange(len(rows)) + width = 0.38 + metrics = case["metrics"] + real_top = set(metrics["real_top_set"]) + simulator_top = set(metrics["simulator_top_set"]) + + fig, ax = plt.subplots(figsize=(13.4, 5.4)) + real_bars = ax.bar(x - width / 2, real, width, color=REAL_COLOR, label="Real community vLLM") + sim_bars = ax.bar(x + width / 2, frontier, width, color=CALIBRATED_COLOR, label="Frontier best-effort") + style_axis(ax) + ax.set_ylabel("SLO-feasible req/s/GPU") + ax.set_xticks(x, labels=labels) + ax.set_xlabel("Configuration (ordered by measured real capacity)") + ax.set_ylim(0, 0.088) + ax.legend(frameon=False, ncol=2, loc="upper right") + ax.set_title( + "Top set matches · Spearman ρ = 0.949 · 20/20 comparable pairs agree · regret = 0", + loc="left", + fontsize=11, + weight="bold", + ) + for idx, row in enumerate(rows): + if row["name"] in real_top: + real_bars[idx].set_edgecolor("#000000") + real_bars[idx].set_linewidth(2.0) + mark_top(ax, idx - width / 2, real[idx], "real top", REAL_COLOR) + if row["name"] in simulator_top: + sim_bars[idx].set_edgecolor(CALIBRATED_COLOR) + sim_bars[idx].set_linewidth(2.0) + sim_bars[idx].set_hatch("///") + mark_top(ax, idx + width / 2, frontier[idx], "sim top", CALIBRATED_COLOR) + fig.suptitle( + "Qwen3-235B-A22B-FP8 prefill-only: Frontier preserves the useful ordering", + fontsize=14, + weight="bold", + y=1.01, + ) + fig.tight_layout() + save(fig, "qwen235-prefill-config-ranking") + plt.close(fig) + + +def plot_qwen235_t0() -> None: + with T0_COMPARISON_PATH.open(encoding="utf-8") as handle: + comparison = json.load(handle)["comparisons"]["tpot_150ms"] + rows = sorted( + comparison["records"], + key=lambda row: (row["config"]["tp"], row["config"]["mns"], row["config"]["mbt"]), + ) + labels = [ + f"TP{row['config']['tp']} MNS{row['config']['mns']}\nMBT{row['config']['mbt'] // 1024}K" + for row in rows + ] + real = np.array([row["real_capacity_per_gpu"] for row in rows]) + frontier = np.array([row["sim_capacity_per_gpu"] for row in rows]) + x = np.arange(len(rows)) + width = 0.38 + + fig = plt.figure(figsize=(16.2, 5.7), constrained_layout=True) + grid = fig.add_gridspec(1, 3, width_ratios=[2.6, 0.72, 0.72]) + ax = fig.add_subplot(grid[0, 0]) + real_bars = ax.bar(x - width / 2, real, width, color=REAL_COLOR, label="Real community vLLM") + sim_bars = ax.bar(x + width / 2, frontier, width, color=CALIBRATED_COLOR, label="Frozen Frontier") + style_axis(ax) + ax.axvline(3.5, color="#999999", linewidth=1.0, linestyle="--") + ax.set_ylabel("SLO-feasible req/s/GPU") + ax.set_xticks(x, labels=labels) + ax.set_ylim(0, 0.71) + ax.legend(frameon=False, ncol=2, loc="upper right") + ax.set_title( + "Exact top-set match · Kendall τ-b = 0.894 · worst tie-break regret = 0", + loc="left", + fontsize=10.5, + weight="bold", + ) + real_top = set(comparison["real_top_set"]) + simulator_top = set(comparison["sim_top_set"]) + for idx, row in enumerate(rows): + name = row["config"]["name"] + if name in real_top: + real_bars[idx].set_edgecolor("#000000") + real_bars[idx].set_linewidth(1.8) + if name in simulator_top: + sim_bars[idx].set_edgecolor(CALIBRATED_COLOR) + sim_bars[idx].set_linewidth(1.8) + sim_bars[idx].set_hatch("///") + + matrices = [] + for source in ("real_capacity_per_gpu", "sim_capacity_per_gpu"): + matrix = np.empty((2, 2)) + for row in rows: + config = row["config"] + if config["tp"] != 8: + continue + matrix[[64, 128].index(config["mns"]), [8192, 16384].index(config["mbt"])] = row[source] + matrices.append(matrix) + heat_axes = [fig.add_subplot(grid[0, 1]), fig.add_subplot(grid[0, 2])] + titles = ["Real TP8", "Frontier TP8"] + images = [] + for heat_ax, matrix, title in zip(heat_axes, matrices, titles): + images.append(heat_ax.imshow(matrix, cmap="YlOrRd", vmin=0.15, vmax=0.30, aspect="equal")) + heat_ax.set_title(title, fontsize=11, weight="bold") + heat_ax.set_xticks([0, 1], labels=["8K", "16K"]) + heat_ax.set_yticks([0, 1], labels=["64", "128"]) + heat_ax.set_xlabel("MBT") + heat_ax.set_ylabel("MNS") + for i in range(2): + for j in range(2): + heat_ax.text(j, i, f"{matrix[i, j]:.2f}", ha="center", va="center", weight="bold") + fig.colorbar(images[-1], ax=heat_axes, location="bottom", shrink=0.75, pad=0.14, label="req/s/GPU") + fig.suptitle( + "Qwen3-235B fixed-shape mixed: correct deployment choice, incorrect TP8 control interaction", + fontsize=14, + weight="bold", + ) + save(fig, "qwen235-t0-fixed-shape-ranking") + plt.close(fig) + + +def main() -> None: + data = load_data() + plot_qwen30(data) + plot_qwen235(data) + plot_qwen235_t0() + + +if __name__ == "__main__": + main()