Document PD baseline comparison

.gitignore

@@ -12,6 +12,7 @@ src/*.egg-info
 .deps/
 outputs/
 
-# Local heavyweight checkouts and generated experiment artifacts
-third_party/
+# Vendored dependencies. Track only the maintained SGLang fork/snapshot.
+third_party/*
+!third_party/sglang/
 *.log

AGENTS.md

@@ -0,0 +1,94 @@
+# AGENTS.md
+
+## Environment
+
+Use `uv` to manage all python environment. `uv add` to manage deps so that we can `uv sync` to get exactly same runnable environment each time.
+
+## Goal
+
+Build a minimal prototype on top of **SGLang xPyD** to test whether **session-aware / KV-cache-aware P/D routing** can improve **end-to-end latency** for agentic coding workloads.
+
+Current setup:
+- SGLang: `v0.5.10`
+- Model: `Qwen3-Coder-30B-A3B-Instruct` (`~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct`)
+- xPyD runs on this single 8-GPU node, so the current constraint is **$x + y \le 8$**
+- Even in local experiments, the implementation should preserve the **P -> D RDMA-style data path** semantics as much as possible; local runs should treat this as a loopback-based stand-in rather than collapsing P/D into a special in-process shortcut
+- Traces:
+  - Ali coding agent (`~/ali-trace/trace-qwen3-coder-formatted/041715-041717.jsonl`)
+
+---
+
+## MVP Scope
+
+We only do the following:
+
+1. Run **SGLang xPyD** correctly on one machine
+2. Add a **baseline router**
+   - `turn1`: default routing
+   - `turn2+`: prefer previous `D` node for the same session
+3. Add a **KV-cache-aware routing** policy
+4. Replay traces and compare policies with the same evaluation pipeline
+
+Out of scope for now:
+- autoscaling
+- fault tolerance
+- large-scale cluster scheduler
+- production hardening
+- general multi-tenant serving
+
+---
+
+## What matters
+
+Primary metric:
+- **E2E latency**
+
+Secondary metrics:
+- TTFT
+- TPOT
+- KV transfer volume
+- cache hit / reuse
+- re-prefill count
+- per-node load
+
+Do not optimize TTFT alone if E2E does not improve.
+
+---
+
+## Development Order
+
+Implement in this order:
+
+1. **Bring up xPyD**
+2. **Add trace replay + metrics logging**
+3. **Implement sticky-to-D baseline**
+4. **Implement KV-cache-aware routing**
+5. **Analyze gains and failure cases**
+
+Do not skip step 2.
+
+---
+
+## Core Rules
+
+### 1. Keep policy separate from mechanism
+- mechanism = how requests / KV / xPyD work
+- policy = how we choose `P` and `D`
+
+Do not mix them unless necessary.
+
+### 2. Prefer simple, debuggable logic
+Start with simple heuristics before complex scoring.
+
+### 3. Log everything needed to explain results
+Each request should log:
+- request id
+- session id
+- turn id
+- assigned P node
+- assigned D node
+- latency
+- whether reuse was expected / observed
+
+### 4. Small interfaces only
+Avoid over-abstraction.

README.md

@@ -0,0 +1,101 @@
+# Agentic PD Hybrid
+
+这个项目是在 SGLang xPyD 上做一个最小实验框架，用来判断：
+
+**面向 agentic coding workload 的 session-aware / KV-cache-aware P/D routing，能不能降低端到端延迟。**
+
+更完整但仍然简洁的说明见 [docs/PROJECT_OVERVIEW.md](docs/PROJECT_OVERVIEW.md)。
+
+## 当前做了什么
+
+- 启动单机 SGLang P/D 栈。
+- 回放 Ali coding agent trace，并记录 request-level metrics。
+- 支持 `default`、`sticky`、`kv-aware` 路由策略。
+- 支持 `pd-disaggregation`、`kvcache-centric`、`pd-colo` 对比。
+- 支持小 append、多轮 session 的 micro-benchmark trace。
+- 维护了基于 SGLang `v0.5.10` 的本地 patch，放在 `third_party/sglang`。
+
+## 环境
+
+统一使用 `uv`：
+
+```bash
+uv sync
+```
+
+默认模型路径：
+
+```text
+~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct
+```
+
+当前主要测试环境是单机 8 GPU，约束是 `prefill + decode <= 8`。
+
+## 常用命令
+
+生成小 append trace：
+
+```bash
+uv run agentic-pd-hybrid make-small-append-trace \
+  --output outputs/smoke-hotcap-30k-1k-256.jsonl \
+  --session-count 4 \
+  --turns-per-session 3 \
+  --initial-input-length 30000 \
+  --append-input-length 1000 \
+  --output-length 256
+```
+
+跑 live benchmark：
+
+```bash
+uv run agentic-pd-hybrid benchmark-live \
+  --trace outputs/micro-serveable-varturn-30k-1k-256-20260424T0756Z.jsonl \
+  --output-root outputs/live-serveable-varturn-30k-1k-256-hotcap \
+  --mechanism kvcache-centric \
+  --policy kv-aware \
+  --kvcache-admission-mode worker \
+  --prefill-workers 1 \
+  --decode-workers 1 \
+  --prefill-gpu-ids 0 \
+  --decode-gpu-ids 1 \
+  --transfer-backend mooncake \
+  --target-duration-s 2000 \
+  --session-sample-rate 1.0 \
+  --min-turns 2 \
+  --time-scale 1 \
+  --concurrency-limit 1000
+```
+
+只回放并写 metrics：
+
+```bash
+uv run agentic-pd-hybrid replay \
+  --trace path/to/trace.jsonl \
+  --policy kv-aware \
+  --mechanism pd-disaggregation \
+  --router-url http://127.0.0.1:8000 \
+  --output outputs/replay.jsonl
+```
+
+## 输出
+
+每次 replay/benchmark 会写：
+
+- request metrics：`request-metrics.jsonl`
+- 汇总结果：`request-metrics.jsonl.summary.json`
+
+重点看：
+
+- E2E latency
+- TTFT / TPOT
+- execution mode
+- cached tokens
+- KV transfer blocks
+- error
+
+## 维护约定
+
+- 项目代码改动：`feat:` / `fix:` / `docs:`。
+- SGLang 改动：`feat(sglang): ...` / `fix(sglang): ...`。
+- `third_party/sglang` 的基线是 clean SGLang `v0.5.10` snapshot。
+- 不提交 `outputs/`、日志、`__pycache__`、虚拟环境。

docs/KVCACHE_CENTRIC_PROGRESS_ZH.md

@@ -0,0 +1,263 @@
+# KV-cache centric P/D routing 当前进展
+
+本文记录当前原型在 SGLang xPyD 上围绕 session-aware / KV-cache-aware P/D routing 的实现、实验结果和阶段性结论。实验日期为 2026-04-24 至 2026-04-25。
+
+## 目标和核心假设
+
+目标是在单机 8 GPU xPyD 环境中验证：针对 agentic coding workload，session-aware / KV-cache-aware P/D routing 是否能提升端到端延迟。
+
+当前重点假设：
+
+1. 在 PD-disaggregation 下，P 节点和 D 节点可能同时保留同一个 session 的 prefix KV，形成 P/D duplicate。
+2. 如果预测某个 session 后续会 direct-to-D，那么 P 侧在 radix/prefix cache eviction 时可以优先淘汰这部分 prefix cache。
+3. 这样可以给 P 节点释放 cache 空间，提高 P 侧 prefix cache reuse。
+4. 如果 P 侧 reuse 提升后 D 侧开始成为瓶颈，可以通过增加 xPyD 中 D 的配比，也就是增加 y，缓解 decode 侧压力。
+
+实验结果表明：第 2、3 点在 P cache 高压 workload 下成立；第 4 点只部分成立，因为瓶颈会从 decode prealloc 转移到 P->D transfer/bootstrap pipeline。
+
+## 已实现机制
+
+### 1. Trace profile 和 paired comparison
+
+新增 `agentic_pd_hybrid profile` 子命令和 `src/agentic_pd_hybrid/profile.py`。
+
+能力：
+
+- 统计 trace 的 session 数、turn2+ 数、append tokens、overlap ratio、direct-to-D eligible turn。
+- 对比 baseline/candidate metrics，输出 paired E2E latency delta。
+- 用于解释 micro-benchmark 与 Ali filtered workload 的差异。
+
+### 2. P 侧 priority eviction 支持
+
+修改 SGLang server args，允许：
+
+```bash
+--radix-eviction-policy priority
+```
+
+router/replay 支持内部字段：
+
+- `smg_prefill_priority`
+- `smg_decode_priority`
+
+router 会将内部字段剥离，只把标准 `priority` 分别传给 P/D backend。这样可以让 direct-to-D predicted session 在 P 侧使用更低优先级，例如 `-100`，普通请求使用 `100`。
+
+### 3. Kvcache seed/direct admission 控制
+
+新增多种 seed/reseed 过滤：
+
+- `kvcache_seed_max_resident_tokens`
+- `kvcache_seed_max_output_tokens`
+- `kvcache_seed_min_turn_id`
+- `kvcache_seed_only_multiturn_sessions`
+- `kvcache_seed_max_inflight_decode`
+
+新增 P streaming session backup 策略：
+
+- `release-after-transfer`：P->D transfer 后释放 P 侧 session backup。
+- `capacity-backup`：容量允许时保留 P 侧 backup。
+
+当前主实验使用 `release-after-transfer`。
+
+### 4. 稳定性修复
+
+之前 worker-admission 实验会在 replay 尾部卡住，不写出 metrics。原因是 benchmark 的长 `timeout_s=3600` 同时用于：
+
+- SGLang stack 启动等待
+- replay client 单请求
+- router 到 P/D backend 的单请求
+
+修复后新增：
+
+- `BenchmarkConfig.request_timeout_s`
+- CLI `benchmark-live --request-timeout-s`
+- launch plan `router_request_timeout_s`
+
+当前做法：
+
+- `timeout_s=3600` 继续用于 SGLang 启动和整体 stack 等待。
+- `request_timeout_s=180` 用于 replay client 和 router 到 P/D backend 的单请求。
+- control-plane probe/open/close session 使用 2s timeout，fail closed。
+
+修复效果：worker-admission 从“尾部卡死不落盘”变为“卡住请求记录为 ReadTimeout，整轮实验完成并写 metrics”。
+
+## 关键实验结果
+
+### P cache pressure 下 priority eviction 是否提升 P 侧 reuse
+
+配置：2P2D，P `--max-total-tokens 90000`，micro workload 316 requests / 58 sessions。
+
+| 配置 | ok/total | mean E2E | p99 | request cached tokens | P log cached tokens | P new-token total |
+|---|---:|---:|---:|---:|---:|---:|
+| LRU | 314/316 | 28.171s | 43.409s | 8.204M | 7.783M | 2.236M |
+| Priority | 314/316 | 28.165s | 41.935s | 8.401M | 7.981M | 2.039M |
+
+结论：
+
+- 在 P cache eviction 高频触发时，priority eviction 确实提高 P 侧 prefix reuse。
+- cached tokens 增加约 197k，new prefill tokens 减少约 197k。
+- 但 mean E2E 基本不变，说明性能瓶颈转移到 D decode/transfer。
+
+### 增加 D 配比前的 D 侧瓶颈证据
+
+2P2D priority pressure 下：
+
+- decode `#queue-req`：max/mean/p90 = 0/0/0
+- decode token usage：max 0.98，mean 0.842，p90 0.96
+- decode `#transfer-req`：max 7，mean 4.61，p90 7
+- decode `#prealloc-req`：max 21，mean 11.6，p90 21
+
+解释：
+
+- D 侧不是普通 waiting queue 堆积。
+- 真正压力在 token usage、transfer queue 和 prealloc queue。
+
+### D scaling：router admission 旧结果
+
+| 配置 | ok/total | mean | p50 | p90 | p99 | error |
+|---|---:|---:|---:|---:|---:|---:|
+| 2P2D | 314/316 | 28.165s | 30.576s | 38.267s | 41.935s | 2 |
+| 2P3D | 290/316 | 29.915s | 31.428s | 40.856s | 45.964s | 26 |
+| 2P4D | 285/316 | 30.566s | 32.823s | 40.566s | 44.838s | 31 |
+
+结论：
+
+- 直接增加 D 不稳定。
+- 2P3D/2P4D 的错误主要来自 `kvcache-centric` seed/direct 路径。
+- 日志显示 decode 侧出现 `WaitingForInput` timeout 和 `KVTransferError`。
+
+### D scaling：worker admission + request timeout 修复后
+
+配置：P `--max-total-tokens 90000`，priority eviction，worker admission，`request_timeout_s=180`。
+
+| 配置 | ok/total | mean | p50 | p90 | p99 | error |
+|---|---:|---:|---:|---:|---:|---:|
+| 2P2D | 313/316 | 29.838s | 30.742s | 39.641s | 52.506s | 3 |
+| 2P3D | 299/316 | 29.349s | 30.569s | 42.161s | 46.113s | 17 |
+| 2P4D | 312/316 | 26.442s | 27.759s | 38.197s | 47.970s | 4 |
+
+对应 decode log 摘要：
+
+| 配置 | decode usage mean | decode transfer mean | decode prealloc mean |
+|---|---:|---:|---:|
+| 2P2D | 0.859 | 4.86 | 11.3 |
+| 2P3D | 0.877 | 5.70 | 6.92 |
+| 2P4D | 0.809 | 5.31 | 3.46 |
+
+结论：
+
+- 2P4D + worker admission + request timeout 是当前最好的 D scaling 配置。
+- 相比旧 2P4D，成功率从 285/316 提升到 312/316，mean 从 30.566s 降到 26.442s。
+- 但 p99 仍未稳定改善，tail 仍由 P->D transfer/bootstrap timeout 主导。
+- 2P3D 不稳定，错误 17 个，不适合作为当前推荐配置。
+
+### 与默认 PD-disaggregation 的同配置对比
+
+为了避免和旧 2P2D/no-pressure 基线混比，补跑了同一 workload、同一 2P4D、同一 P `--max-total-tokens 90000`、同一 `time_scale=50`、同一 `request_timeout_s=180` 的默认 PD-disaggregation 基线。
+
+| 策略 | ok/total | mean | p50 | p90 | p99 | cached total | direct-to-D |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| 默认 PD-disaggregation 2P4D | 316/316 | 29.210s | 28.940s | 47.434s | 52.605s | 8.165M | 0 |
+| KVC 2P4D latency-best | 312/316 | 26.442s | 27.759s | 38.197s | 47.970s | 7.882M | 11 |
+| KVC 2P4D seed-min2 | 316/316 | 26.729s | 25.840s | 43.589s | 50.426s | 8.337M | 3 |
+
+结论：
+
+- 如果只看成功请求延迟，KVC 2P4D latency-best 相比默认 PD：
+  - mean 改善约 9.5%；
+  - p50 改善约 4.1%；
+  - p90 改善约 19.5%；
+  - p99 改善约 8.8%。
+- 但 latency-best 有 4 个 `ReadTimeout`，全部来自 turn1 大 seed 的 P->D transfer/bootstrap timeout。
+- `kvcache_seed_min_turn_id=2` 可以消除这些错误，达到 316/316 成功，同时 mean/p50/p90/p99 仍然优于默认 PD。
+- 因此当前推荐分成两档：
+  - 追求最低 mean/p90：使用 KVC 2P4D latency-best。
+  - 追求稳定性：使用 KVC 2P4D seed-min2。
+
+### 进一步优化尝试
+
+在当前最佳配置基础上尝试了三个优化方向：
+
+| 策略 | ok/total | mean | p50 | p90 | p99 | 结论 |
+|---|---:|---:|---:|---:|---:|---|
+| KVC 2P4D latency-best | 312/316 | 26.442s | 27.759s | 38.197s | 47.970s | 延迟最优，但有 4 个 turn1 seed timeout |
+| transfer-cap4 | 304/316 | 27.961s | 29.785s | 38.730s | 45.192s | 不可取，实时 transfer queue snapshot 滞后，错误更多 |
+| disable-failed-session | 285/316 | 29.179s | 30.878s | 42.794s | 46.411s | 不可取，失败 session 降级会放大后端异常状态 |
+| seed-min2 | 316/316 | 26.729s | 25.840s | 43.589s | 50.426s | 稳定性最优 |
+| inflight0 | 316/316 | 29.497s | 31.186s | 43.498s | 48.022s | 太保守，几乎关闭 KVC 收益 |
+
+从错误明细看，latency-best 的 4 个错误全部是 turn1 seed timeout，每个都是约 1250 KV blocks 的大 seed。`seed-min2` 跳过 turn1 seed 后，错误全部消失，说明当前主要稳定性问题是启动阶段 seed 风暴，而不是后续 direct append 本身。
+
+transfer-cap4 没有效果，说明仅依赖 D worker 的实时 `decode_transfer_queue_reqs` 不够；并发请求可能同时读取到尚可的 snapshot，然后一起进入 P->D transfer，导致 backlog 在 admission 后形成。
+
+## Ali filtered 当前状态
+
+Ali filtered small-append trace：
+
+- 81 requests
+- 28 sessions
+- 53 turn2+
+- max input 18901
+- max output 1925
+- span 5414s
+
+PD baseline：
+
+- ok 81/81
+- mean 9.072s
+- p50 7.086s
+- p90 21.761s
+- p99 26.813s
+
+Kvcache-centric 在 Ali filtered 上曾出现 58-67 个 router 200 后挂住、无 metrics 的问题。当前 request timeout 和 control-plane timeout 修复后，应重新跑 Ali filtered；在未重跑前，不把 Ali filtered 纳入最终性能结论。
+
+## 当前结论
+
+1. kvcache centric 可以提高 KV reuse，但需要满足 workload 条件：
+   - session 有多 turn；
+   - turn2+ append 较小；
+   - prefix overlap 高；
+   - P 侧 cache 有 eviction pressure；
+   - D 侧 seed/direct admission 不把 transfer pipeline 打爆。
+
+2. 不适合的 workload：
+   - 单 turn 或 session 间隔过长；
+   - turn2+ append 很大，direct-to-D 不能省掉多少 prefill；
+   - prefix overlap 低；
+   - P cache 没有 eviction pressure；
+   - D transfer/prealloc 已经高压。
+
+3. 用户关于 P/D duplicate 的假设部分成立：
+   - 如果 D session 已经 resident，P 侧对应 streaming session backup 可以视为 duplicate。
+   - `release-after-transfer` 可以避免长期保留 P/D 两份 session KV。
+   - priority eviction 进一步让 P 在必须 eviction 时优先淘汰 direct-to-D predicted session prefix。
+
+4. 但当前机制还没有完全解决性能问题：
+   - P 侧 reuse 提升后，E2E 不一定改善。
+   - 主要原因是 D 侧 transfer/bootstrap pipeline 成为瓶颈。
+   - 增加 D 可以降低 prealloc，但不能自动降低 transfer backlog。
+   - 当前同配置下 KVC 已经可以优于默认 PD，但必须在 latency-best 和 stable 两种策略之间取舍。
+
+## 下一步优化方向
+
+1. transfer-aware admission：
+   - seed/direct 不只看 D token capacity，也要看 `decode_transfer_queue_reqs`、`decode_prealloc_queue_reqs`、`decode_retracted_queue_reqs`。
+   - 当 transfer queue 高时，应该主动走 PD fallback。
+   - 当前实验显示，单纯使用实时 transfer queue threshold 不够，需要配合本地 reservation。
+
+2. per-D transfer budget：
+   - 对每个 D 设置 seed/reseed 并发上限。
+   - 不能只按 session residency 或 token headroom 判断。
+   - 特别要限制 turn1 大 seed 的并发，避免启动阶段 seed 风暴。
+
+3. P/D ratio 联合调度：
+   - 2P4D 当前最好，但 P queue 也随 D 增加而上升。
+   - 后续需要测试 3P3D、3P4D、4P4D 等组合，确认 P transfer source 是否成为瓶颈。
+
+4. Ali filtered 重跑：
+   - 使用 request timeout 修复后的版本重新跑 Ali filtered。
+   - 如果仍然没有收益，需要按 session gap、append size、overlap ratio 分桶分析。
+
+5. 更严格的成功率指标：
+   - 当前不能只看成功请求的 mean/p90。
+   - 必须同时报告 ok/total、timeout/error 类型和 tail latency。

docs/PROJECT_OVERVIEW.md

@@ -0,0 +1,98 @@
+# 项目概览
+
+这个项目验证一个问题：
+
+**agentic coding workload 里，如果 router 更懂 session 和 KV cache，P/D serving 的端到端延迟能不能更低。**
+
+当前基于：
+
+- SGLang `v0.5.10`
+- Qwen3-Coder-30B-A3B-Instruct
+- 单机 8 GPU
+- Mooncake loopback 模拟 P -> D 传输
+
+## 设计
+
+代码按两层分开：
+
+- **机制**：启动 SGLang、发送请求、管理 session、收集 metrics。
+- **策略**：决定请求去哪个 P node、哪个 D node。
+
+这样后续可以单独改 routing policy，不把它和 SGLang/xPyD 机制混在一起。
+
+## 已实现
+
+- 单机 P/D stack 启动和关闭。
+- 本地 Python PD router。
+- Ali trace 加载、session 级采样、synthetic prompt 生成。
+- 按 trace 原始到达时间 replay，不用固定 concurrency 强行压流量。
+- request-level metrics 和 summary。
+- 路由策略：
+  - `default`
+  - `sticky`
+  - `kv-aware`
+- serving 机制：
+  - `pd-disaggregation`
+  - `kvcache-centric`
+  - `pd-colo`
+- micro-benchmark trace 生成。
+- worker-managed / router-managed KV admission 对比。
+- worker-managed 下的 D session soft-cap，避免所有 session 都挤进 D KV。
+- SGLang patch：
+  - decode worker 支持 PD mode 下 local append-prefill；
+  - 暴露 streaming session cache 状态；
+  - 支持按 session 粒度 evict idle streaming session；
+  - 支持 direct append admission 查询。
+
+## 当前结论
+
+micro-benchmark 上，`kvcache-centric` 可以比 `pd-disaggregation` 好。
+
+原因很简单：session 少，D KV 放得下，turn2+ 可以直接走 D session，省掉一部分 P/D 路径开销。
+
+但在 300+ request、58 session 的测试上，情况不同：
+
+- D KV 放不下全部 session working set。
+- naive worker-managed 会频繁 evict/reseed 整个 session。
+- reseed 和 transfer 压力会抵消 KV reuse 收益。
+- aggressive P-backup 会增加尾延迟风险。
+
+当前 soft-cap 优化后：
+
+- worker-managed 比旧版本更稳；
+- TTFT 明显下降；
+- 没有再出现 600s transfer hang 被当成成功响应的问题；
+- 但 sampled Ali trace 上，`pd-disaggregation` 仍然略好。
+
+当前判断：
+
+**KV-cache-centric 只应该保留真正 hot 的 session。不是所有 session 都值得占 D KV。**
+
+下一步最有价值的是：
+
+- inter-turn-gap-aware admission；
+- session aging；
+- 更精确地预测哪些 session 会很快复用 KV。
+
+## SGLang 维护方式
+
+`third_party/sglang` 已纳入主仓库。
+
+历史结构：
+
+- `chore: vendor sglang v0.5.10 snapshot`：干净上游基线。
+- `feat(sglang): ...` / `fix(sglang): ...`：我们的 SGLang patch。
+
+后续改 SGLang 时：
+
+- 只改 `third_party/sglang` 下相关文件；
+- 单独提交；
+- commit message 带 `(sglang)`；
+- 不把 benchmark 输出、pyc、日志混进提交。
+
+## 已知限制
+
+- 这是实验原型，不是生产 router。
+- 当前主要验证单机 8 GPU。
+- Ali trace 没有原始 prompt，只能用 `hash_ids` 合成 prompt。
+- 当前 routing 还缺少真正的 hot-session 预测。

pyproject.toml

@@ -0,0 +1,24 @@
+[project]
+name = "agentic-pd-hybrid"
+version = "0.1.0"
+description = "Prototype for session-aware and KV-cache-aware PD routing on SGLang xPyD"
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "httpx>=0.28.1",
+    "mooncake-transfer-engine",
+    "sglang==0.5.10",
+]
+
+[project.scripts]
+agentic-pd-hybrid = "agentic_pd_hybrid.cli:main"
+
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.uv]
+prerelease = "allow"

src/agentic_pd_hybrid/__init__.py

@@ -0,0 +1,12 @@
+"""Agentic PD hybrid prototype."""
+
+__all__ = [
+    "cli",
+    "launcher",
+    "metrics",
+    "microbench",
+    "policies",
+    "replay",
+    "topology",
+    "trace",
+]

src/agentic_pd_hybrid/benchmark.py

@@ -0,0 +1,297 @@
+from __future__ import annotations
+
+import asyncio
+import json
+import signal
+from dataclasses import asdict, dataclass, replace
+from datetime import UTC, datetime
+from pathlib import Path
+
+from agentic_pd_hybrid.replay import ReplayConfig, replay_trace
+from agentic_pd_hybrid.sampling import SessionSampleConfig, sample_trace_sessions
+from agentic_pd_hybrid.stack import ManagedPdStack, launch_pd_stack
+from agentic_pd_hybrid.topology import SingleNodeTopology
+
+
+@dataclass(frozen=True)
+class BenchmarkConfig:
+    trace_path: Path
+    output_root: Path
+    topology: SingleNodeTopology
+    policy_name: str
+    mechanism_name: str = "pd-disaggregation"
+    target_duration_s: float = 600.0
+    start_time_s: float = 0.0
+    session_sample_rate: float = 0.01
+    min_turns: int = 1
+    time_scale: float = 1.0
+    concurrency_limit: int = 32
+    timeout_s: float = 1200.0
+    request_timeout_s: float | None = None
+    stream: bool = True
+    stream_idle_timeout_s: float | None = 900.0
+    kvcache_direct_max_uncached_tokens: int = 2048
+    kvcache_admission_mode: str = "router"
+    kvcache_seed_max_resident_tokens: int | None = None
+    kvcache_seed_max_output_tokens: int | None = None
+    kvcache_seed_min_turn_id: int = 1
+    kvcache_seed_only_multiturn_sessions: bool = False
+    kvcache_prefill_backup_policy: str = "release-after-transfer"
+    kvcache_seed_max_inflight_decode: int | None = 3
+    kvcache_seed_max_decode_transfer_queue_reqs: int | None = None
+    kvcache_direct_max_decode_transfer_queue_reqs: int | None = None
+    kvcache_prefill_priority_eviction: bool = False
+    kvcache_prefill_direct_priority: int = -100
+    kvcache_prefill_normal_priority: int = 100
+    sample_profile: str = "default"
+    min_initial_input_tokens: int | None = None
+    max_initial_input_tokens: int | None = None
+    max_append_input_tokens: int | None = None
+    max_output_tokens: int | None = None
+    min_overlap_ratio: float | None = None
+    launch_stack: bool = True
+
+
+@dataclass(frozen=True)
+class BenchmarkArtifacts:
+    run_dir: Path
+    sampled_trace_path: Path
+    metrics_path: Path
+    summary_path: Path
+    benchmark_config_path: Path
+
+
+def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
+    run_id = datetime.now(UTC).strftime("%Y%m%dT%H%M%SZ")
+    run_label = f"{config.mechanism_name}-{config.policy_name}"
+    if config.mechanism_name == "kvcache-centric":
+        run_label = f"{run_label}-{config.kvcache_admission_mode}-admission"
+    run_dir = config.output_root / f"{run_label}-{run_id}"
+    run_dir.mkdir(parents=True, exist_ok=True)
+
+    topology = config.topology
+    if config.mechanism_name == "kvcache-centric":
+        prefill_extra_server_args = topology.prefill_extra_server_args + (
+            "--enable-streaming-session",
+        )
+        if config.kvcache_prefill_priority_eviction:
+            prefill_extra_server_args = prefill_extra_server_args + (
+                "--radix-eviction-policy",
+                "priority",
+            )
+        topology = replace(
+            topology,
+            prefill_extra_server_args=prefill_extra_server_args,
+            decode_extra_server_args=topology.decode_extra_server_args
+            + (
+                "--enable-streaming-session",
+                "--disaggregation-decode-allow-local-prefill",
+            ),
+        )
+
+    sampled_trace_path = run_dir / "sampled-trace.jsonl"
+    sample_summary = sample_trace_sessions(
+        SessionSampleConfig(
+            trace_path=config.trace_path,
+            output_path=sampled_trace_path,
+            target_duration_s=config.target_duration_s,
+            start_time_s=config.start_time_s,
+            session_sample_rate=config.session_sample_rate,
+            min_turns=config.min_turns,
+            profile=config.sample_profile,  # type: ignore[arg-type]
+            min_initial_input_tokens=config.min_initial_input_tokens,
+            max_initial_input_tokens=config.max_initial_input_tokens,
+            max_append_input_tokens=config.max_append_input_tokens,
+            max_output_tokens=config.max_output_tokens,
+            min_overlap_ratio=config.min_overlap_ratio,
+        )
+    )
+
+    stack: ManagedPdStack | None = None
+    previous_sigint = signal.getsignal(signal.SIGINT)
+    previous_sigterm = signal.getsignal(signal.SIGTERM)
+
+    def _handle_termination(signum, _frame) -> None:
+        if stack is not None:
+            stack.stop()
+        raise SystemExit(128 + signum)
+
+    try:
+        signal.signal(signal.SIGINT, _handle_termination)
+        signal.signal(signal.SIGTERM, _handle_termination)
+        if config.launch_stack:
+            stack = launch_pd_stack(
+                topology=topology,
+                run_dir=run_dir,
+                prefill_policy="round_robin",
+                decode_policy=_decode_policy_for(config.policy_name),
+                timeout_s=config.timeout_s,
+                router_request_timeout_s=(
+                    config.request_timeout_s
+                    if config.request_timeout_s is not None
+                    else config.timeout_s
+                ),
+                include_router=(
+                    config.mechanism_name in {"pd-disaggregation", "kvcache-centric"}
+                ),
+            )
+            router_url = (
+                stack.router_url
+                if config.mechanism_name in {"pd-disaggregation", "kvcache-centric"}
+                else None
+            )
+        else:
+            router_url = (
+                topology.router_url
+                if config.mechanism_name in {"pd-disaggregation", "kvcache-centric"}
+                else None
+            )
+
+        metrics_path = run_dir / "request-metrics.jsonl"
+        replay_config = ReplayConfig(
+            trace_path=sampled_trace_path,
+            output_path=metrics_path,
+            policy_name=config.policy_name,
+            mechanism_name=config.mechanism_name,
+            topology=topology,
+            router_url=router_url,
+            model_name=topology.model_name,
+            pace=True,
+            time_scale=config.time_scale,
+            request_limit=None,
+            concurrency_limit=config.concurrency_limit,
+            header_mode=_header_mode_for(config.policy_name),
+            timeout_s=config.timeout_s,
+            stream=config.stream,
+            stream_idle_timeout_s=config.stream_idle_timeout_s,
+            kvcache_direct_max_uncached_tokens=config.kvcache_direct_max_uncached_tokens,
+            kvcache_admission_mode=config.kvcache_admission_mode,  # type: ignore[arg-type]
+            kvcache_seed_max_resident_tokens=config.kvcache_seed_max_resident_tokens,
+            kvcache_seed_max_output_tokens=config.kvcache_seed_max_output_tokens,
+            kvcache_seed_min_turn_id=config.kvcache_seed_min_turn_id,
+            kvcache_seed_only_multiturn_sessions=(
+                config.kvcache_seed_only_multiturn_sessions
+            ),
+            kvcache_prefill_backup_policy=config.kvcache_prefill_backup_policy,  # type: ignore[arg-type]
+            kvcache_seed_max_inflight_decode=(
+                config.kvcache_seed_max_inflight_decode
+            ),
+            kvcache_seed_max_decode_transfer_queue_reqs=(
+                config.kvcache_seed_max_decode_transfer_queue_reqs
+            ),
+            kvcache_direct_max_decode_transfer_queue_reqs=(
+                config.kvcache_direct_max_decode_transfer_queue_reqs
+            ),
+            kvcache_prefill_priority_eviction=(
+                config.kvcache_prefill_priority_eviction
+            ),
+            kvcache_prefill_direct_priority=config.kvcache_prefill_direct_priority,
+            kvcache_prefill_normal_priority=config.kvcache_prefill_normal_priority,
+        )
+        if config.request_timeout_s is not None:
+            replay_config = replace(
+                replay_config,
+                timeout_s=config.request_timeout_s,
+            )
+        asyncio.run(replay_trace(replay_config))
+    finally:
+        signal.signal(signal.SIGINT, previous_sigint)
+        signal.signal(signal.SIGTERM, previous_sigterm)
+        if stack is not None:
+            stack.stop()
+
+    benchmark_config_path = run_dir / "benchmark-config.json"
+    with benchmark_config_path.open("w", encoding="utf-8") as handle:
+        json.dump(
+            {
+                "policy_name": config.policy_name,
+                "mechanism_name": config.mechanism_name,
+                "target_duration_s": config.target_duration_s,
+                "start_time_s": config.start_time_s,
+                "session_sample_rate": config.session_sample_rate,
+                "min_turns": config.min_turns,
+                "time_scale": config.time_scale,
+                "concurrency_limit": config.concurrency_limit,
+                "timeout_s": config.timeout_s,
+                "request_timeout_s": config.request_timeout_s,
+                "stream": config.stream,
+                "stream_idle_timeout_s": config.stream_idle_timeout_s,
+                "kvcache_direct_max_uncached_tokens": config.kvcache_direct_max_uncached_tokens,
+                "kvcache_admission_mode": config.kvcache_admission_mode,
+                "kvcache_seed_max_resident_tokens": config.kvcache_seed_max_resident_tokens,
+                "kvcache_seed_max_output_tokens": config.kvcache_seed_max_output_tokens,
+                "kvcache_seed_min_turn_id": config.kvcache_seed_min_turn_id,
+                "kvcache_seed_only_multiturn_sessions": (
+                    config.kvcache_seed_only_multiturn_sessions
+                ),
+                "kvcache_prefill_backup_policy": config.kvcache_prefill_backup_policy,
+                "kvcache_seed_max_inflight_decode": (
+                    config.kvcache_seed_max_inflight_decode
+                ),
+                "kvcache_seed_max_decode_transfer_queue_reqs": (
+                    config.kvcache_seed_max_decode_transfer_queue_reqs
+                ),
+                "kvcache_direct_max_decode_transfer_queue_reqs": (
+                    config.kvcache_direct_max_decode_transfer_queue_reqs
+                ),
+                "kvcache_prefill_priority_eviction": (
+                    config.kvcache_prefill_priority_eviction
+                ),
+                "kvcache_prefill_direct_priority": (
+                    config.kvcache_prefill_direct_priority
+                ),
+                "kvcache_prefill_normal_priority": (
+                    config.kvcache_prefill_normal_priority
+                ),
+                "sample_profile": config.sample_profile,
+                "min_initial_input_tokens": config.min_initial_input_tokens,
+                "max_initial_input_tokens": config.max_initial_input_tokens,
+                "max_append_input_tokens": config.max_append_input_tokens,
+                "max_output_tokens": config.max_output_tokens,
+                "min_overlap_ratio": config.min_overlap_ratio,
+                "sample_summary": asdict(sample_summary),
+                "topology": {
+                    "model_path": config.topology.model_path,
+                    "router_url": topology.router_url,
+                    "transfer_backend": topology.transfer_backend,
+                    "force_rdma": topology.force_rdma,
+                    "ib_device": topology.ib_device,
+                    "prefill_workers": [
+                        worker.worker_id for worker in topology.prefill_workers
+                    ],
+                    "decode_workers": [
+                        worker.worker_id for worker in topology.decode_workers
+                    ],
+                    "direct_workers": [
+                        worker.worker_id for worker in topology.direct_workers
+                    ],
+                },
+            },
+            handle,
+            indent=2,
+            sort_keys=True,
+        )
+
+    return BenchmarkArtifacts(
+        run_dir=run_dir,
+        sampled_trace_path=sampled_trace_path,
+        metrics_path=run_dir / "request-metrics.jsonl",
+        summary_path=run_dir / "request-metrics.jsonl.summary.json",
+        benchmark_config_path=benchmark_config_path,
+    )
+
+
+def _decode_policy_for(policy_name: str) -> str:
+    if policy_name == "sticky":
+        return "manual"
+    if policy_name == "kv-aware":
+        return "consistent_hashing"
+    return "round_robin"
+
+
+def _header_mode_for(policy_name: str) -> str:
+    if policy_name == "sticky":
+        return "routing-key"
+    if policy_name == "kv-aware":
+        return "target-worker"
+    return "none"

src/agentic_pd_hybrid/cli.py

@@ -0,0 +1,757 @@
+from __future__ import annotations
+
+import argparse
+import asyncio
+from pathlib import Path
+
+from agentic_pd_hybrid.benchmark import BenchmarkConfig, run_live_benchmark
+from agentic_pd_hybrid.launcher import build_launch_plan
+from agentic_pd_hybrid.microbench import SmallAppendTraceConfig, write_small_append_trace
+from agentic_pd_hybrid.profile import ProfileConfig, print_profile_summary, write_profile
+from agentic_pd_hybrid.replay import ReplayConfig, replay_trace
+from agentic_pd_hybrid.sampling import SessionSampleConfig, sample_trace_sessions
+from agentic_pd_hybrid.trace_profiles import (
+    NormalizeTraceLengthsConfig,
+    normalize_trace_lengths,
+)
+from agentic_pd_hybrid.topology import build_single_node_topology
+
+
+def _normalize_mechanism_name(name: str) -> str:
+    normalized = name.strip().lower()
+    aliases = {
+        "pd-disagg": "pd-disaggregation",
+        "pd-disaggregation": "pd-disaggregation",
+        "pd-hybrid": "pd-disaggregation",
+        "baseline-pd-disagg": "pd-disaggregation",
+        "pd-colo": "pd-colo",
+        "direct-d": "pd-colo",
+        "colocation": "pd-colo",
+        "kvcache-centric": "kvcache-centric",
+        "turn2+-direct-to-d": "kvcache-centric",
+        "pd-with-d-append": "kvcache-centric",
+    }
+    if normalized not in aliases:
+        raise ValueError(f"Unsupported mechanism: {name}")
+    return aliases[normalized]
+
+
+def _parse_gpu_id_list(value: str | None) -> tuple[int, ...] | None:
+    if value is None:
+        return None
+    items = [item.strip() for item in value.split(",") if item.strip()]
+    if not items:
+        return tuple()
+    return tuple(int(item) for item in items)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Agentic PD hybrid prototype")
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    print_launch = subparsers.add_parser(
+        "print-launch",
+        help="Print one-node SGLang PD launch commands",
+    )
+    _add_topology_arguments(print_launch)
+    print_launch.add_argument("--prefill-policy", default="round_robin")
+    print_launch.add_argument("--decode-policy", default="manual")
+
+    replay = subparsers.add_parser(
+        "replay",
+        help="Replay trace and log request-level metrics",
+    )
+    _add_topology_arguments(replay)
+    replay.add_argument("--trace", type=Path, required=True)
+    replay.add_argument("--output", type=Path, required=True)
+    replay.add_argument(
+        "--policy",
+        choices=["default", "sticky", "kv-aware"],
+        default="sticky",
+    )
+    replay.add_argument(
+        "--mechanism",
+        choices=[
+            "pd-disaggregation",
+            "pd-hybrid",
+            "pd-disagg",
+            "pd-colo",
+            "direct-d",
+            "kvcache-centric",
+            "turn2+-direct-to-d",
+            "pd-with-d-append",
+        ],
+        default="pd-disaggregation",
+    )
+    replay.add_argument("--router-url")
+    replay.add_argument("--model")
+    replay.add_argument(
+        "--header-mode",
+        choices=["auto", "none", "routing-key", "target-worker"],
+        default="auto",
+    )
+    replay.add_argument(
+        "--request-limit",
+        type=int,
+        default=None,
+        help="Replay at most this many requests",
+    )
+    replay.add_argument(
+        "--no-pace",
+        action="store_true",
+        help="Disable wall-clock pacing from trace timestamps",
+    )
+    replay.add_argument(
+        "--time-scale",
+        type=float,
+        default=1.0,
+        help="Scale trace timing by this factor when pacing is enabled",
+    )
+    replay.add_argument(
+        "--concurrency-limit",
+        type=int,
+        default=32,
+    )
+    replay.add_argument(
+        "--timeout-s",
+        type=float,
+        default=600.0,
+    )
+    replay.add_argument(
+        "--no-stream",
+        action="store_true",
+        help="Use non-streaming OpenAI responses for more robust E2E-only replay.",
+    )
+    replay.add_argument(
+        "--stream-idle-timeout-s",
+        type=float,
+        default=900.0,
+        help="Abort a streaming request if no SSE line arrives within this many seconds.",
+    )
+    replay.add_argument(
+        "--kvcache-direct-max-uncached-tokens",
+        type=int,
+        default=2048,
+        help="For kvcache-centric routing, bypass P when the uncached suffix is at most this many tokens.",
+    )
+    replay.add_argument(
+        "--kvcache-admission-mode",
+        choices=["router", "worker"],
+        default="router",
+        help=(
+            "For kvcache-centric routing, use router shadow-state admission "
+            "or query the decode worker on the critical path."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-seed-max-resident-tokens",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric routing, do not seed/reseed a decode session "
+            "when input+output tokens exceed this value."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-seed-max-output-tokens",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric routing, do not seed/reseed a decode session "
+            "when output tokens exceed this value."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-seed-min-turn-id",
+        type=int,
+        default=1,
+        help=(
+            "For kvcache-centric routing, do not seed/reseed a decode session "
+            "before this turn id."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-seed-only-multiturn-sessions",
+        action="store_true",
+        help=(
+            "Oracle ablation for kvcache-centric routing: only seed sessions "
+            "that have more than one turn in the replay trace."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-prefill-backup-policy",
+        choices=["release-after-transfer", "capacity-backup"],
+        default="release-after-transfer",
+        help=(
+            "For kvcache-centric seed/reseed, release the P-side session after "
+            "P->D transfer or keep a capacity-limited P-side backup."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-seed-max-inflight-decode",
+        type=int,
+        default=3,
+        help=(
+            "For kvcache-centric routing, skip seed/reseed when the router "
+            "shadow inflight decode load assigned to the target D exceeds this value. "
+            "Use a negative value to disable this filter."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-seed-max-decode-transfer-queue-reqs",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric worker admission, skip seed/reseed when the "
+            "target D reports more transfer-queue requests than this value. "
+            "Use a negative value to disable this filter."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-direct-max-decode-transfer-queue-reqs",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric worker admission, skip direct-to-D append when "
+            "the target D reports more transfer-queue requests than this value. "
+            "Use a negative value to disable this filter."
+        ),
+    )
+    replay.add_argument(
+        "--kvcache-prefill-priority-eviction",
+        action="store_true",
+        help=(
+            "For kvcache-centric routing, mark P-side prefixes predicted to move "
+            "direct-to-D as lower priority. Requires P workers to use "
+            "--radix-eviction-policy priority."
+        ),
+    )
+    replay.add_argument("--kvcache-prefill-direct-priority", type=int, default=-100)
+    replay.add_argument("--kvcache-prefill-normal-priority", type=int, default=100)
+
+    sample = subparsers.add_parser(
+        "sample-sessions",
+        help="Sample a session-granularity trace shard for live benchmarking",
+    )
+    sample.add_argument("--trace", type=Path, required=True)
+    sample.add_argument("--output", type=Path, required=True)
+    sample.add_argument("--target-duration-s", type=float, default=600.0)
+    sample.add_argument("--start-time-s", type=float, default=0.0)
+    sample.add_argument("--session-sample-rate", type=float, default=0.01)
+    sample.add_argument("--min-turns", type=int, default=1)
+    sample.add_argument("--max-requests", type=int, default=None)
+    sample.add_argument(
+        "--profile",
+        choices=["default", "small-append"],
+        default="default",
+        help="Optional workload-shape filter for live benchmarks.",
+    )
+    sample.add_argument("--min-initial-input-tokens", type=int, default=None)
+    sample.add_argument("--max-initial-input-tokens", type=int, default=None)
+    sample.add_argument("--max-append-input-tokens", type=int, default=None)
+    sample.add_argument("--max-output-tokens", type=int, default=None)
+    sample.add_argument("--min-overlap-ratio", type=float, default=None)
+
+    normalize = subparsers.add_parser(
+        "normalize-trace-lengths",
+        help="Rewrite a trace to a fixed turn1/append/output length profile",
+    )
+    normalize.add_argument("--trace", type=Path, required=True)
+    normalize.add_argument("--output", type=Path, required=True)
+    normalize.add_argument("--initial-input-length", type=int, default=10_000)
+    normalize.add_argument("--append-input-length", type=int, default=1_000)
+    normalize.add_argument("--output-length", type=int, default=1_000)
+    normalize.add_argument("--max-requests", type=int, default=None)
+
+    profile = subparsers.add_parser(
+        "profile",
+        help="Profile a trace and optional request metrics for routing analysis",
+    )
+    profile.add_argument("--trace", type=Path, required=True)
+    profile.add_argument("--output", type=Path, default=None)
+    profile.add_argument("--metrics", type=Path, default=None)
+    profile.add_argument("--baseline-metrics", type=Path, default=None)
+    profile.add_argument("--candidate-metrics", type=Path, default=None)
+    profile.add_argument("--direct-max-uncached-tokens", type=int, default=2048)
+
+    micro = subparsers.add_parser(
+        "make-small-append-trace",
+        help="Generate a synthetic multi-turn trace with small turn2+ appends",
+    )
+    micro.add_argument("--output", type=Path, required=True)
+    micro.add_argument("--session-count", type=int, default=8)
+    micro.add_argument("--turns-per-session", type=int, default=3)
+    micro.add_argument("--initial-input-length", type=int, default=10_000)
+    micro.add_argument("--append-input-length", type=int, default=1_000)
+    micro.add_argument("--output-length", type=int, default=1_000)
+    micro.add_argument("--inter-turn-gap-s", type=float, default=1.0)
+    micro.add_argument("--session-stagger-s", type=float, default=0.1)
+
+    benchmark = subparsers.add_parser(
+        "benchmark-live",
+        help="Launch a real PD stack, sample sessions, and collect live E2E numbers",
+    )
+    _add_topology_arguments(benchmark)
+    benchmark.add_argument("--trace", type=Path, required=True)
+    benchmark.add_argument(
+        "--policy",
+        choices=["default", "sticky", "kv-aware"],
+        default="sticky",
+    )
+    benchmark.add_argument(
+        "--mechanism",
+        choices=[
+            "pd-disaggregation",
+            "pd-hybrid",
+            "pd-disagg",
+            "pd-colo",
+            "direct-d",
+            "kvcache-centric",
+            "turn2+-direct-to-d",
+            "pd-with-d-append",
+        ],
+        default="pd-disaggregation",
+    )
+    benchmark.add_argument("--output-root", type=Path, default=Path("outputs/live"))
+    benchmark.add_argument("--target-duration-s", type=float, default=600.0)
+    benchmark.add_argument("--start-time-s", type=float, default=0.0)
+    benchmark.add_argument("--session-sample-rate", type=float, default=0.01)
+    benchmark.add_argument("--min-turns", type=int, default=1)
+    benchmark.add_argument("--time-scale", type=float, default=1.0)
+    benchmark.add_argument("--concurrency-limit", type=int, default=32)
+    benchmark.add_argument("--timeout-s", type=float, default=1200.0)
+    benchmark.add_argument(
+        "--request-timeout-s",
+        type=float,
+        default=None,
+        help=(
+            "Per-request replay/router timeout. If unset, --timeout-s is used. "
+            "--timeout-s still controls stack startup."
+        ),
+    )
+    benchmark.add_argument(
+        "--no-stream",
+        action="store_true",
+        help="Use non-streaming OpenAI responses for E2E-only live benchmarking.",
+    )
+    benchmark.add_argument(
+        "--stream-idle-timeout-s",
+        type=float,
+        default=900.0,
+        help="Abort a streaming request if no SSE line arrives within this many seconds.",
+    )
+    benchmark.add_argument(
+        "--kvcache-direct-max-uncached-tokens",
+        type=int,
+        default=2048,
+        help="For kvcache-centric routing, bypass P when the uncached suffix is at most this many tokens.",
+    )
+    benchmark.add_argument(
+        "--kvcache-admission-mode",
+        choices=["router", "worker"],
+        default="router",
+        help=(
+            "For kvcache-centric routing, use router shadow-state admission "
+            "or query the decode worker on the critical path."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-seed-max-resident-tokens",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric routing, do not seed/reseed a decode session "
+            "when input+output tokens exceed this value."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-seed-max-output-tokens",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric routing, do not seed/reseed a decode session "
+            "when output tokens exceed this value."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-seed-min-turn-id",
+        type=int,
+        default=1,
+        help=(
+            "For kvcache-centric routing, do not seed/reseed a decode session "
+            "before this turn id."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-seed-only-multiturn-sessions",
+        action="store_true",
+        help=(
+            "Oracle ablation for kvcache-centric routing: only seed sessions "
+            "that have more than one turn in the replay trace."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-prefill-backup-policy",
+        choices=["release-after-transfer", "capacity-backup"],
+        default="release-after-transfer",
+        help=(
+            "For kvcache-centric seed/reseed, release the P-side session after "
+            "P->D transfer or keep a capacity-limited P-side backup."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-seed-max-inflight-decode",
+        type=int,
+        default=3,
+        help=(
+            "For kvcache-centric routing, skip seed/reseed when the router "
+            "shadow inflight decode load assigned to the target D exceeds this value. "
+            "Use a negative value to disable this filter."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-seed-max-decode-transfer-queue-reqs",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric worker admission, skip seed/reseed when the "
+            "target D reports more transfer-queue requests than this value. "
+            "Use a negative value to disable this filter."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-direct-max-decode-transfer-queue-reqs",
+        type=int,
+        default=None,
+        help=(
+            "For kvcache-centric worker admission, skip direct-to-D append when "
+            "the target D reports more transfer-queue requests than this value. "
+            "Use a negative value to disable this filter."
+        ),
+    )
+    benchmark.add_argument(
+        "--kvcache-prefill-priority-eviction",
+        action="store_true",
+        help=(
+            "For kvcache-centric benchmark-live, launch P workers with priority "
+            "radix eviction and mark direct-to-D predicted prefixes lower priority."
+        ),
+    )
+    benchmark.add_argument("--kvcache-prefill-direct-priority", type=int, default=-100)
+    benchmark.add_argument("--kvcache-prefill-normal-priority", type=int, default=100)
+    benchmark.add_argument(
+        "--sample-profile",
+        choices=["default", "small-append"],
+        default="default",
+        help="Optional session-shape filter applied before live replay.",
+    )
+    benchmark.add_argument("--min-initial-input-tokens", type=int, default=None)
+    benchmark.add_argument("--max-initial-input-tokens", type=int, default=None)
+    benchmark.add_argument("--max-append-input-tokens", type=int, default=None)
+    benchmark.add_argument("--max-output-tokens", type=int, default=None)
+    benchmark.add_argument("--min-overlap-ratio", type=float, default=None)
+
+    args = parser.parse_args()
+
+    if args.command == "print-launch":
+        topology = _topology_from_args(args)
+        plan = build_launch_plan(
+            topology,
+            prefill_policy=args.prefill_policy,
+            decode_policy=args.decode_policy,
+            include_router=bool(topology.prefill_workers and topology.decode_workers),
+        )
+        print(plan.render())
+        return
+
+    if args.command == "replay":
+        topology = _topology_from_args(args)
+        config = ReplayConfig(
+            trace_path=args.trace,
+            output_path=args.output,
+            policy_name=args.policy,
+            mechanism_name=_normalize_mechanism_name(args.mechanism),
+            topology=topology,
+            router_url=args.router_url,
+            model_name=args.model,
+            pace=not args.no_pace,
+            time_scale=args.time_scale,
+            request_limit=args.request_limit,
+            concurrency_limit=args.concurrency_limit,
+            header_mode=args.header_mode,
+            timeout_s=args.timeout_s,
+            stream=not args.no_stream,
+            stream_idle_timeout_s=args.stream_idle_timeout_s,
+            kvcache_direct_max_uncached_tokens=args.kvcache_direct_max_uncached_tokens,
+            kvcache_admission_mode=args.kvcache_admission_mode,
+            kvcache_seed_max_resident_tokens=args.kvcache_seed_max_resident_tokens,
+            kvcache_seed_max_output_tokens=args.kvcache_seed_max_output_tokens,
+            kvcache_seed_min_turn_id=args.kvcache_seed_min_turn_id,
+            kvcache_seed_only_multiturn_sessions=(
+                args.kvcache_seed_only_multiturn_sessions
+            ),
+            kvcache_prefill_backup_policy=args.kvcache_prefill_backup_policy,
+            kvcache_seed_max_inflight_decode=(
+                None
+                if args.kvcache_seed_max_inflight_decode < 0
+                else args.kvcache_seed_max_inflight_decode
+            ),
+            kvcache_seed_max_decode_transfer_queue_reqs=(
+                None
+                if args.kvcache_seed_max_decode_transfer_queue_reqs is None
+                or args.kvcache_seed_max_decode_transfer_queue_reqs < 0
+                else args.kvcache_seed_max_decode_transfer_queue_reqs
+            ),
+            kvcache_direct_max_decode_transfer_queue_reqs=(
+                None
+                if args.kvcache_direct_max_decode_transfer_queue_reqs is None
+                or args.kvcache_direct_max_decode_transfer_queue_reqs < 0
+                else args.kvcache_direct_max_decode_transfer_queue_reqs
+            ),
+            kvcache_prefill_priority_eviction=(
+                args.kvcache_prefill_priority_eviction
+            ),
+            kvcache_prefill_direct_priority=args.kvcache_prefill_direct_priority,
+            kvcache_prefill_normal_priority=args.kvcache_prefill_normal_priority,
+        )
+        results = asyncio.run(replay_trace(config))
+        print(
+            f"wrote {len(results)} request records to {args.output} and "
+            f"{args.output}{'.summary.json'}"
+        )
+        return
+
+    if args.command == "profile":
+        if (args.baseline_metrics is None) != (args.candidate_metrics is None):
+            raise ValueError(
+                "--baseline-metrics and --candidate-metrics must be provided together"
+            )
+        report = write_profile(
+            ProfileConfig(
+                trace_path=args.trace,
+                output_path=args.output,
+                metrics_path=args.metrics,
+                baseline_metrics_path=args.baseline_metrics,
+                candidate_metrics_path=args.candidate_metrics,
+                direct_max_uncached_tokens=args.direct_max_uncached_tokens,
+            )
+        )
+        print_profile_summary(report)
+        if args.output is not None:
+            print(f"wrote profile to {args.output}")
+        return
+
+    if args.command == "sample-sessions":
+        summary = sample_trace_sessions(
+            SessionSampleConfig(
+                trace_path=args.trace,
+                output_path=args.output,
+                target_duration_s=args.target_duration_s,
+                start_time_s=args.start_time_s,
+                session_sample_rate=args.session_sample_rate,
+                min_turns=args.min_turns,
+                max_requests=args.max_requests,
+                profile=args.profile,
+                min_initial_input_tokens=args.min_initial_input_tokens,
+                max_initial_input_tokens=args.max_initial_input_tokens,
+                max_append_input_tokens=args.max_append_input_tokens,
+                max_output_tokens=args.max_output_tokens,
+                min_overlap_ratio=args.min_overlap_ratio,
+            )
+        )
+        print(
+            f"wrote {summary.request_count} requests from {summary.session_count} sessions "
+            f"covering {summary.sampled_duration_s:.3f}s to {args.output}"
+        )
+        return
+
+    if args.command == "normalize-trace-lengths":
+        summary = normalize_trace_lengths(
+            NormalizeTraceLengthsConfig(
+                trace_path=args.trace,
+                output_path=args.output,
+                initial_input_length=args.initial_input_length,
+                append_input_length=args.append_input_length,
+                output_length=args.output_length,
+                max_requests=args.max_requests,
+            )
+        )
+        print(
+            f"wrote {summary.request_count} normalized requests from "
+            f"{summary.session_count} sessions to {args.output}"
+        )
+        return
+
+    if args.command == "make-small-append-trace":
+        summary = write_small_append_trace(
+            SmallAppendTraceConfig(
+                output_path=args.output,
+                session_count=args.session_count,
+                turns_per_session=args.turns_per_session,
+                initial_input_length=args.initial_input_length,
+                append_input_length=args.append_input_length,
+                output_length=args.output_length,
+                inter_turn_gap_s=args.inter_turn_gap_s,
+                session_stagger_s=args.session_stagger_s,
+            )
+        )
+        print(
+            f"wrote {summary.request_count} requests across {summary.session_count} sessions "
+            f"to {args.output}"
+        )
+        return
+
+    if args.command == "benchmark-live":
+        topology = _topology_from_args(args)
+        artifacts = run_live_benchmark(
+            BenchmarkConfig(
+                trace_path=args.trace,
+                output_root=args.output_root,
+                topology=topology,
+                policy_name=args.policy,
+                mechanism_name=_normalize_mechanism_name(args.mechanism),
+                target_duration_s=args.target_duration_s,
+                start_time_s=args.start_time_s,
+                session_sample_rate=args.session_sample_rate,
+                min_turns=args.min_turns,
+                time_scale=args.time_scale,
+                concurrency_limit=args.concurrency_limit,
+                timeout_s=args.timeout_s,
+                request_timeout_s=args.request_timeout_s,
+                stream=not args.no_stream,
+                stream_idle_timeout_s=args.stream_idle_timeout_s,
+                kvcache_direct_max_uncached_tokens=args.kvcache_direct_max_uncached_tokens,
+                kvcache_admission_mode=args.kvcache_admission_mode,
+                kvcache_seed_max_resident_tokens=args.kvcache_seed_max_resident_tokens,
+                kvcache_seed_max_output_tokens=args.kvcache_seed_max_output_tokens,
+                kvcache_seed_min_turn_id=args.kvcache_seed_min_turn_id,
+                kvcache_seed_only_multiturn_sessions=(
+                    args.kvcache_seed_only_multiturn_sessions
+                ),
+                kvcache_prefill_backup_policy=args.kvcache_prefill_backup_policy,
+                kvcache_seed_max_inflight_decode=(
+                    None
+                    if args.kvcache_seed_max_inflight_decode < 0
+                    else args.kvcache_seed_max_inflight_decode
+                ),
+                kvcache_seed_max_decode_transfer_queue_reqs=(
+                    None
+                    if args.kvcache_seed_max_decode_transfer_queue_reqs is None
+                    or args.kvcache_seed_max_decode_transfer_queue_reqs < 0
+                    else args.kvcache_seed_max_decode_transfer_queue_reqs
+                ),
+                kvcache_direct_max_decode_transfer_queue_reqs=(
+                    None
+                    if args.kvcache_direct_max_decode_transfer_queue_reqs is None
+                    or args.kvcache_direct_max_decode_transfer_queue_reqs < 0
+                    else args.kvcache_direct_max_decode_transfer_queue_reqs
+                ),
+                kvcache_prefill_priority_eviction=(
+                    args.kvcache_prefill_priority_eviction
+                ),
+                kvcache_prefill_direct_priority=(
+                    args.kvcache_prefill_direct_priority
+                ),
+                kvcache_prefill_normal_priority=(
+                    args.kvcache_prefill_normal_priority
+                ),
+                sample_profile=args.sample_profile,
+                min_initial_input_tokens=args.min_initial_input_tokens,
+                max_initial_input_tokens=args.max_initial_input_tokens,
+                max_append_input_tokens=args.max_append_input_tokens,
+                max_output_tokens=args.max_output_tokens,
+                min_overlap_ratio=args.min_overlap_ratio,
+                launch_stack=True,
+            )
+        )
+        print(
+            f"benchmark artifacts written under {artifacts.run_dir}; "
+            f"metrics={artifacts.metrics_path} summary={artifacts.summary_path}"
+        )
+        return
+
+    raise AssertionError(f"Unhandled command: {args.command}")
+
+
+def _add_topology_arguments(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--model-path",
+        default="~/models/Qwen/Qwen3-Coder-30B-A3B-Instruct",
+    )
+    parser.add_argument("--prefill-workers", type=int, default=1)
+    parser.add_argument("--decode-workers", type=int, default=1)
+    parser.add_argument("--direct-workers", type=int, default=0)
+    parser.add_argument("--prefill-tp-size", type=int, default=1)
+    parser.add_argument("--decode-tp-size", type=int, default=1)
+    parser.add_argument("--direct-tp-size", type=int, default=1)
+    parser.add_argument("--gpu-budget", type=int, default=8)
+    parser.add_argument(
+        "--prefill-gpu-ids",
+        default=None,
+        help="Comma-separated GPU IDs for prefill workers, e.g. 3,4",
+    )
+    parser.add_argument(
+        "--decode-gpu-ids",
+        default=None,
+        help="Comma-separated GPU IDs for decode workers, e.g. 5",
+    )
+    parser.add_argument(
+        "--direct-gpu-ids",
+        default=None,
+        help="Comma-separated GPU IDs for direct workers, e.g. 6",
+    )
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--router-port", type=int, default=8000)
+    parser.add_argument("--prefill-port-base", type=int, default=30000)
+    parser.add_argument("--decode-port-base", type=int, default=31000)
+    parser.add_argument("--direct-port-base", type=int, default=32000)
+    parser.add_argument("--bootstrap-port-base", type=int, default=8998)
+    parser.add_argument("--transfer-backend", default="nixl")
+    parser.add_argument(
+        "--force-rdma",
+        action="store_true",
+        help=(
+            "Force real RDMA transport for PD KV transfer. "
+            "Currently this requires Mooncake plus --ib-device."
+        ),
+    )
+    parser.add_argument("--ib-device", default=None)
+    parser.add_argument(
+        "--no-trust-remote-code",
+        action="store_true",
+    )
+
+
+def _topology_from_args(args: argparse.Namespace):
+    transfer_backend = args.transfer_backend
+    if args.force_rdma:
+        transfer_backend = "mooncake"
+
+    return build_single_node_topology(
+        model_path=str(Path(args.model_path).expanduser()),
+        prefill_worker_count=args.prefill_workers,
+        decode_worker_count=args.decode_workers,
+        direct_worker_count=args.direct_workers,
+        prefill_tp_size=args.prefill_tp_size,
+        decode_tp_size=args.decode_tp_size,
+        direct_tp_size=args.direct_tp_size,
+        prefill_gpu_ids=_parse_gpu_id_list(args.prefill_gpu_ids),
+        decode_gpu_ids=_parse_gpu_id_list(args.decode_gpu_ids),
+        direct_gpu_ids=_parse_gpu_id_list(args.direct_gpu_ids),
+        total_gpu_budget=args.gpu_budget,
+        host=args.host,
+        router_port=args.router_port,
+        prefill_port_base=args.prefill_port_base,
+        decode_port_base=args.decode_port_base,
+        direct_port_base=args.direct_port_base,
+        bootstrap_port_base=args.bootstrap_port_base,
+        transfer_backend=transfer_backend,
+        force_rdma=args.force_rdma,
+        trust_remote_code=not args.no_trust_remote_code,
+        ib_device=args.ib_device,
+        direct_extra_server_args=("--enable-streaming-session",),
+    )
+
+
+if __name__ == "__main__":
+    main()

src/agentic_pd_hybrid/launcher.py

@@ -0,0 +1,145 @@
+from __future__ import annotations
+
+import shlex
+import sys
+from dataclasses import dataclass
+
+from agentic_pd_hybrid.topology import SingleNodeTopology, WorkerSpec
+
+
+@dataclass(frozen=True)
+class LaunchPlan:
+    prefill_commands: tuple[tuple[str, ...], ...]
+    decode_commands: tuple[tuple[str, ...], ...]
+    direct_commands: tuple[tuple[str, ...], ...]
+    router_command: tuple[str, ...] | None
+
+    def render(self) -> str:
+        sections: list[str] = []
+        for idx, command in enumerate(self.prefill_commands):
+            sections.append(_render_named_command(f"prefill-{idx}", command))
+        for idx, command in enumerate(self.decode_commands):
+            sections.append(_render_named_command(f"decode-{idx}", command))
+        for idx, command in enumerate(self.direct_commands):
+            sections.append(_render_named_command(f"direct-{idx}", command))
+        if self.router_command is not None:
+            sections.append(_render_named_command("router", self.router_command))
+        return "\n\n".join(sections)
+
+
+def build_launch_plan(
+    topology: SingleNodeTopology,
+    *,
+    prefill_policy: str = "round_robin",
+    decode_policy: str = "manual",
+    include_router: bool = True,
+    router_request_timeout_s: float | None = None,
+) -> LaunchPlan:
+    return LaunchPlan(
+        prefill_commands=tuple(
+            _build_server_command(topology, worker) for worker in topology.prefill_workers
+        ),
+        decode_commands=tuple(
+            _build_server_command(topology, worker) for worker in topology.decode_workers
+        ),
+        direct_commands=tuple(
+            _build_server_command(topology, worker) for worker in topology.direct_workers
+        ),
+        router_command=(
+            _build_router_command(
+                topology,
+                prefill_policy=prefill_policy,
+                decode_policy=decode_policy,
+                request_timeout_s=router_request_timeout_s,
+            )
+            if include_router and topology.prefill_workers and topology.decode_workers
+            else None
+        ),
+    )
+
+
+def _build_server_command(
+    topology: SingleNodeTopology,
+    worker: WorkerSpec,
+) -> tuple[str, ...]:
+    command = [
+        sys.executable,
+        "-B",
+        "-u",
+        "-m",
+        "sglang.launch_server",
+        "--model-path",
+        topology.model_path,
+        "--host",
+        worker.host,
+        "--port",
+        str(worker.port),
+        "--base-gpu-id",
+        str(worker.gpu_id),
+        "--disaggregation-mode",
+        _disaggregation_mode_for(worker),
+        "--disaggregation-transfer-backend",
+        topology.transfer_backend,
+    ]
+    if worker.tp_size > 1:
+        command.extend(["--tp-size", str(worker.tp_size)])
+    if topology.trust_remote_code:
+        command.append("--trust-remote-code")
+    command.append("--enable-cache-report")
+    if worker.bootstrap_port is not None:
+        command.extend(
+            ["--disaggregation-bootstrap-port", str(worker.bootstrap_port)]
+        )
+    if topology.ib_device:
+        command.extend(["--disaggregation-ib-device", topology.ib_device])
+    command.extend(topology.extra_server_args)
+    if worker.role == "prefill":
+        command.extend(topology.prefill_extra_server_args)
+    elif worker.role == "decode":
+        command.extend(topology.decode_extra_server_args)
+    else:
+        command.extend(topology.direct_extra_server_args)
+    return tuple(command)
+
+
+def _build_router_command(
+    topology: SingleNodeTopology,
+    *,
+    prefill_policy: str,
+    decode_policy: str,
+    request_timeout_s: float | None,
+) -> tuple[str, ...]:
+    command: list[str] = [
+        sys.executable,
+        "-B",
+        "-u",
+        "-m",
+        "agentic_pd_hybrid.pd_router",
+        "--host",
+        topology.router_host,
+        "--port",
+        str(topology.router_port),
+        "--prefill-policy",
+        prefill_policy,
+        "--decode-policy",
+        decode_policy,
+    ]
+    if request_timeout_s is not None:
+        command.extend(["--request-timeout-s", str(request_timeout_s)])
+    for worker in topology.prefill_workers:
+        command.extend(
+            ["--prefill", worker.url, str(worker.bootstrap_port or topology.router_port)]
+        )
+    for worker in topology.decode_workers:
+        command.extend(["--decode", worker.url])
+    return tuple(command)
+
+
+def _render_named_command(name: str, command: tuple[str, ...]) -> str:
+    return f"# {name}\n" + " ".join(shlex.quote(part) for part in command)
+
+
+def _disaggregation_mode_for(worker: WorkerSpec) -> str:
+    if worker.role == "direct":
+        return "null"
+    return worker.role

src/agentic_pd_hybrid/metrics.py

@@ -0,0 +1,187 @@
+from __future__ import annotations
+
+import json
+import statistics
+from collections import Counter
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any
+
+from agentic_pd_hybrid.policies import RoutingDecision
+from agentic_pd_hybrid.trace import TraceRequest
+
+
+@dataclass(frozen=True)
+class RequestMetrics:
+    request_id: str
+    session_id: str
+    turn_id: int
+    mechanism_name: str
+    execution_mode: str
+    trace_timestamp_s: float
+    input_length: int
+    output_length: int
+    request_type: str
+    policy_name: str
+    assigned_prefill_node: str
+    assigned_decode_node: str
+    assigned_decode_index: int
+    inflight_decode_load_at_assignment: int
+    reuse_expected: bool
+    reuse_observed: bool
+    observed_overlap_blocks: int
+    kv_transfer_blocks: int
+    actual_kv_transfer_blocks: int
+    cached_tokens: int
+    prefill_request_priority: int | None
+    decode_request_priority: int | None
+    re_prefill_required: bool
+    effective_input_length: int | None
+    session_reused: bool
+    session_reset: bool
+    latency_s: float | None
+    ttft_s: float | None
+    tpot_s: float | None
+    error: str | None = None
+
+    @classmethod
+    def from_decision(
+        cls,
+        request: TraceRequest,
+        decision: RoutingDecision,
+        *,
+        mechanism_name: str,
+        execution_mode: str,
+        actual_kv_transfer_blocks: int,
+        effective_input_length: int | None,
+        cached_tokens: int,
+        session_reused: bool,
+        session_reset: bool,
+        latency_s: float | None,
+        ttft_s: float | None,
+        tpot_s: float | None,
+        prefill_request_priority: int | None = None,
+        decode_request_priority: int | None = None,
+        error: str | None = None,
+    ) -> "RequestMetrics":
+        return cls(
+            request_id=request.request_id,
+            session_id=request.session_id,
+            turn_id=request.turn_id,
+            mechanism_name=mechanism_name,
+            execution_mode=execution_mode,
+            trace_timestamp_s=request.timestamp_s,
+            input_length=request.input_length,
+            output_length=request.output_length,
+            request_type=request.request_type,
+            policy_name=decision.policy_name,
+            assigned_prefill_node=decision.prefill_worker_id,
+            assigned_decode_node=decision.decode_worker_id,
+            assigned_decode_index=decision.decode_worker_index,
+            inflight_decode_load_at_assignment=decision.inflight_decode_load,
+            reuse_expected=decision.reuse_expected,
+            reuse_observed=decision.observed_reuse,
+            observed_overlap_blocks=decision.observed_overlap_blocks,
+            kv_transfer_blocks=decision.kv_transfer_blocks,
+            actual_kv_transfer_blocks=actual_kv_transfer_blocks,
+            cached_tokens=cached_tokens,
+            prefill_request_priority=prefill_request_priority,
+            decode_request_priority=decode_request_priority,
+            re_prefill_required=decision.re_prefill_required,
+            effective_input_length=effective_input_length,
+            session_reused=session_reused,
+            session_reset=session_reset,
+            latency_s=latency_s,
+            ttft_s=ttft_s,
+            tpot_s=tpot_s,
+            error=error,
+        )
+
+
+def write_metrics_jsonl(path: Path, rows: list[RequestMetrics]) -> None:
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("w", encoding="utf-8") as handle:
+        for row in rows:
+            handle.write(json.dumps(asdict(row), sort_keys=True) + "\n")
+
+
+def write_summary_json(
+    path: Path,
+    rows: list[RequestMetrics],
+    *,
+    trace_path: Path,
+    router_url: str | None,
+) -> None:
+    latencies = [row.latency_s for row in rows if row.latency_s is not None]
+    ttfts = [row.ttft_s for row in rows if row.ttft_s is not None]
+    tpots = [row.tpot_s for row in rows if row.tpot_s is not None]
+    per_decode_load = Counter(row.assigned_decode_node for row in rows)
+    per_prefill_load = Counter(row.assigned_prefill_node for row in rows)
+    prefill_priorities = Counter(
+        row.prefill_request_priority
+        for row in rows
+        if row.prefill_request_priority is not None
+    )
+    decode_priorities = Counter(
+        row.decode_request_priority
+        for row in rows
+        if row.decode_request_priority is not None
+    )
+
+    summary: dict[str, Any] = {
+        "trace_path": str(trace_path),
+        "router_url": router_url,
+        "request_count": len(rows),
+        "mechanisms": dict(sorted(Counter(row.mechanism_name for row in rows).items())),
+        "execution_modes": dict(sorted(Counter(row.execution_mode for row in rows).items())),
+        "latency_stats_s": _stats(latencies),
+        "ttft_stats_s": _stats(ttfts),
+        "tpot_stats_s": _stats(tpots),
+        "reuse_expected_count": sum(1 for row in rows if row.reuse_expected),
+        "reuse_observed_count": sum(1 for row in rows if row.reuse_observed),
+        "re_prefill_count": sum(1 for row in rows if row.re_prefill_required),
+        "cache_hit_request_count": sum(1 for row in rows if row.cached_tokens > 0),
+        "total_cached_tokens": sum(row.cached_tokens for row in rows),
+        "cached_tokens_stats": _stats([float(row.cached_tokens) for row in rows]),
+        "session_reused_count": sum(1 for row in rows if row.session_reused),
+        "session_reset_count": sum(1 for row in rows if row.session_reset),
+        "total_kv_transfer_blocks": sum(row.kv_transfer_blocks for row in rows),
+        "total_actual_kv_transfer_blocks": sum(
+            row.actual_kv_transfer_blocks for row in rows
+        ),
+        "per_decode_load": dict(sorted(per_decode_load.items())),
+        "per_prefill_load": dict(sorted(per_prefill_load.items())),
+        "prefill_request_priorities": {
+            str(key): value for key, value in sorted(prefill_priorities.items())
+        },
+        "decode_request_priorities": {
+            str(key): value for key, value in sorted(decode_priorities.items())
+        },
+        "error_count": sum(1 for row in rows if row.error is not None),
+    }
+    path.parent.mkdir(parents=True, exist_ok=True)
+    with path.open("w", encoding="utf-8") as handle:
+        json.dump(summary, handle, indent=2, sort_keys=True)
+
+
+def _stats(values: list[float | None]) -> dict[str, float] | None:
+    clean = [value for value in values if value is not None]
+    if not clean:
+        return None
+    clean.sort()
+    return {
+        "count": float(len(clean)),
+        "mean": statistics.fmean(clean),
+        "p50": _percentile(clean, 0.50),
+        "p90": _percentile(clean, 0.90),
+        "p99": _percentile(clean, 0.99),
+    }
+
+
+def _percentile(sorted_values: list[float], percentile: float) -> float:
+    if not sorted_values:
+        raise ValueError("sorted_values must not be empty")
+    if len(sorted_values) == 1:
+        return sorted_values[0]
+    index = round((len(sorted_values) - 1) * percentile)
+    return sorted_values[index]

src/agentic_pd_hybrid/microbench.py

@@ -0,0 +1,123 @@
+from __future__ import annotations
+
+import json
+from dataclasses import asdict, dataclass
+from math import ceil
+from pathlib import Path
+
+
+BLOCK_TOKEN_BUDGET = 24
+
+
+@dataclass(frozen=True)
+class SmallAppendTraceConfig:
+    output_path: Path
+    session_count: int = 8
+    turns_per_session: int = 3
+    initial_input_length: int = 10_000
+    append_input_length: int = 1_000
+    output_length: int = 1_000
+    inter_turn_gap_s: float = 1.0
+    session_stagger_s: float = 0.1
+    request_type: str = "coder"
+
+
+@dataclass(frozen=True)
+class SmallAppendTraceSummary:
+    output_path: str
+    session_count: int
+    turns_per_session: int
+    request_count: int
+    initial_input_length: int
+    append_input_length: int
+    output_length: int
+    inter_turn_gap_s: float
+    session_stagger_s: float
+
+
+def write_small_append_trace(config: SmallAppendTraceConfig) -> SmallAppendTraceSummary:
+    if config.session_count <= 0:
+        raise ValueError("session_count must be > 0")
+    if config.turns_per_session <= 0:
+        raise ValueError("turns_per_session must be > 0")
+    if config.initial_input_length < 0:
+        raise ValueError("initial_input_length must be >= 0")
+    if config.append_input_length < 0:
+        raise ValueError("append_input_length must be >= 0")
+    if config.output_length < 0:
+        raise ValueError("output_length must be >= 0")
+
+    config.output_path.parent.mkdir(parents=True, exist_ok=True)
+    records: list[dict[str, object]] = []
+    next_chat_id = 1_000_000
+
+    for session_idx in range(config.session_count):
+        root_chat_id = next_chat_id
+        previous_chat_id = -1
+        session_base_time = session_idx * config.session_stagger_s
+        base_block_count = ceil(config.initial_input_length / BLOCK_TOKEN_BUDGET)
+        base_hash_ids = [
+            _hash_id_for(session_idx=session_idx, block_idx=block_idx)
+            for block_idx in range(base_block_count)
+        ]
+
+        for turn_idx in range(config.turns_per_session):
+            chat_id = root_chat_id if turn_idx == 0 else next_chat_id
+            if turn_idx > 0:
+                next_chat_id += 1
+
+            input_length = config.initial_input_length + turn_idx * (
+                config.append_input_length + config.output_length
+            )
+            total_block_count = ceil(input_length / BLOCK_TOKEN_BUDGET)
+            hash_ids = base_hash_ids + [
+                _hash_id_for(
+                    session_idx=session_idx,
+                    block_idx=base_block_count + append_block_idx,
+                )
+                for append_block_idx in range(max(0, total_block_count - base_block_count))
+            ]
+
+            records.append(
+                {
+                    "chat_id": chat_id,
+                    "parent_chat_id": previous_chat_id,
+                    "timestamp": session_base_time
+                    + turn_idx * config.inter_turn_gap_s,
+                    "input_length": input_length,
+                    "output_length": config.output_length,
+                    "type": config.request_type,
+                    "turn": turn_idx + 1,
+                    "hash_ids": hash_ids,
+                }
+            )
+            previous_chat_id = chat_id
+
+        next_chat_id += 1
+
+    records.sort(key=lambda item: float(item["timestamp"]))
+    with config.output_path.open("w", encoding="utf-8") as handle:
+        for record in records:
+            handle.write(json.dumps(record, sort_keys=True) + "\n")
+
+    summary = SmallAppendTraceSummary(
+        output_path=str(config.output_path),
+        session_count=config.session_count,
+        turns_per_session=config.turns_per_session,
+        request_count=len(records),
+        initial_input_length=config.initial_input_length,
+        append_input_length=config.append_input_length,
+        output_length=config.output_length,
+        inter_turn_gap_s=config.inter_turn_gap_s,
+        session_stagger_s=config.session_stagger_s,
+    )
+    summary_path = config.output_path.with_suffix(
+        config.output_path.suffix + ".summary.json"
+    )
+    with summary_path.open("w", encoding="utf-8") as handle:
+        json.dump(asdict(summary), handle, indent=2, sort_keys=True)
+    return summary
+
+
+def _hash_id_for(*, session_idx: int, block_idx: int) -> int:
+    return session_idx * 1_000_000 + block_idx

src/agentic_pd_hybrid/pd_router.py

@@ -0,0 +1,301 @@
+from __future__ import annotations
+
+import argparse
+import asyncio
+import random
+import urllib.parse
+from dataclasses import dataclass
+from http import HTTPStatus
+from itertools import chain
+from typing import AsyncIterator
+
+import aiohttp
+import orjson
+import uvicorn
+from fastapi import FastAPI, HTTPException, Request
+from fastapi.responses import ORJSONResponse, Response, StreamingResponse
+
+_STREAM_CHUNK_SIZE = 1024 * 64
+
+
+@dataclass
+class RouterConfig:
+    host: str
+    port: int
+    prefill_urls: list[tuple[str, int]]
+    decode_urls: list[str]
+    prefill_policy: str = "round_robin"
+    decode_policy: str = "manual"
+    request_timeout_s: float = 1800.0
+
+
+class RouterState:
+    def __init__(self, config: RouterConfig):
+        if not config.prefill_urls:
+            raise ValueError("At least one prefill worker is required")
+        if not config.decode_urls:
+            raise ValueError("At least one decode worker is required")
+        self.config = config
+        self.prefill_cursor = 0
+        self.decode_cursor = 0
+        self.sticky_decode_map: dict[str, int] = {}
+
+    def select_pair(self, headers: dict[str, str]) -> tuple[str, int, str]:
+        prefill_url, bootstrap_port = self.config.prefill_urls[
+            self.prefill_cursor % len(self.config.prefill_urls)
+        ]
+        self.prefill_cursor += 1
+        decode_index = self._select_decode_index(headers)
+        return prefill_url, bootstrap_port, self.config.decode_urls[decode_index]
+
+    def _select_decode_index(self, headers: dict[str, str]) -> int:
+        target_worker = headers.get("x-smg-target-worker")
+        routing_key = headers.get("x-smg-routing-key")
+
+        if (
+            self.config.decode_policy == "consistent_hashing"
+            and target_worker is not None
+        ):
+            idx = int(target_worker)
+            if 0 <= idx < len(self.config.decode_urls):
+                return idx
+
+        if self.config.decode_policy == "manual" and routing_key:
+            cached = self.sticky_decode_map.get(routing_key)
+            if cached is not None:
+                return cached
+            idx = self.decode_cursor % len(self.config.decode_urls)
+            self.decode_cursor += 1
+            self.sticky_decode_map[routing_key] = idx
+            return idx
+
+        idx = self.decode_cursor % len(self.config.decode_urls)
+        self.decode_cursor += 1
+        return idx
+
+
+app = FastAPI()
+router_state: RouterState | None = None
+
+
+@app.get("/health")
+async def health() -> Response:
+    return Response(status_code=200)
+
+
+@app.get("/health_generate")
+async def health_generate() -> Response:
+    state = _require_state()
+    async with aiohttp.ClientSession() as session:
+        tasks = []
+        for server in chain(
+            (url for url, _ in state.config.prefill_urls),
+            state.config.decode_urls,
+        ):
+            tasks.append(session.get(f"{server}/health_generate"))
+        for response in asyncio.as_completed(tasks):
+            async with await response:
+                pass
+    return Response(status_code=200)
+
+
+@app.get("/v1/models")
+async def models() -> ORJSONResponse:
+    state = _require_state()
+    async with aiohttp.ClientSession() as session:
+        async with session.get(f"{state.config.prefill_urls[0][0]}/v1/models") as response:
+            payload = await response.json()
+            return ORJSONResponse(payload, status_code=response.status)
+
+
+@app.post("/v1/chat/completions")
+async def chat_completions(request: Request) -> Response:
+    request_data = await request.json()
+    headers = {key.lower(): value for key, value in request.headers.items()}
+    return await _forward_to_backend(
+        request_data=request_data,
+        headers=headers,
+        endpoint_name="v1/chat/completions",
+    )
+
+
+@app.post("/v1/completions")
+async def completions(request: Request) -> Response:
+    request_data = await request.json()
+    headers = {key.lower(): value for key, value in request.headers.items()}
+    return await _forward_to_backend(
+        request_data=request_data,
+        headers=headers,
+        endpoint_name="v1/completions",
+    )
+
+
+@app.post("/generate")
+async def generate(request: Request) -> Response:
+    request_data = await request.json()
+    headers = {key.lower(): value for key, value in request.headers.items()}
+    return await _forward_to_backend(
+        request_data=request_data,
+        headers=headers,
+        endpoint_name="generate",
+    )
+
+
+async def _forward_to_backend(
+    *,
+    request_data: dict,
+    headers: dict[str, str],
+    endpoint_name: str,
+) -> Response:
+    state = _require_state()
+    prefill_server, bootstrap_port, decode_server = state.select_pair(headers)
+    prefill_request, decode_request = _build_backend_requests(
+        request_data=request_data,
+        prefill_server=prefill_server,
+        bootstrap_port=bootstrap_port,
+    )
+
+    if request_data.get("stream", False):
+        return StreamingResponse(
+            _stream_generate(
+                prefill_request=prefill_request,
+                decode_request=decode_request,
+                prefill_server=prefill_server,
+                decode_server=decode_server,
+                endpoint_name=endpoint_name,
+                timeout_s=state.config.request_timeout_s,
+            ),
+            media_type="text/event-stream",
+        )
+
+    async with aiohttp.ClientSession(
+        timeout=aiohttp.ClientTimeout(total=state.config.request_timeout_s)
+    ) as session:
+        prefill_response, decode_response = await asyncio.gather(
+            session.post(f"{prefill_server}/{endpoint_name}", json=prefill_request),
+            session.post(f"{decode_server}/{endpoint_name}", json=decode_request),
+        )
+        async with prefill_response:
+            await prefill_response.read()
+        async with decode_response:
+            body = await decode_response.read()
+            return Response(
+                content=body,
+                status_code=decode_response.status,
+                media_type=decode_response.content_type,
+            )
+
+
+async def _stream_generate(
+    *,
+    prefill_request: dict,
+    decode_request: dict,
+    prefill_server: str,
+    decode_server: str,
+    endpoint_name: str,
+    timeout_s: float,
+) -> AsyncIterator[bytes]:
+    async with aiohttp.ClientSession(
+        timeout=aiohttp.ClientTimeout(total=timeout_s)
+    ) as session:
+        prefill_response, decode_response = await asyncio.gather(
+            session.post(f"{prefill_server}/{endpoint_name}", json=prefill_request),
+            session.post(f"{decode_server}/{endpoint_name}", json=decode_request),
+        )
+        async with prefill_response, decode_response:
+            if decode_response.status != HTTPStatus.OK:
+                payload = await decode_response.read()
+                yield payload
+                return
+            async for chunk in decode_response.content.iter_chunked(_STREAM_CHUNK_SIZE):
+                yield chunk
+
+
+def _build_bootstrap_payload(prefill_server: str, bootstrap_port: int) -> dict[str, object]:
+    parsed_url = urllib.parse.urlparse(prefill_server)
+    hostname = parsed_url.hostname
+    if hostname is None:
+        raise HTTPException(
+            status_code=500,
+            detail=f"Unable to parse prefill hostname from {prefill_server}",
+        )
+    return {
+        "bootstrap_host": hostname,
+        "bootstrap_port": bootstrap_port,
+        "bootstrap_room": random.randint(0, 2**63 - 1),
+    }
+
+
+def _build_backend_requests(
+    *,
+    request_data: dict,
+    prefill_server: str,
+    bootstrap_port: int,
+) -> tuple[dict, dict]:
+    prefill_priority = request_data.get("smg_prefill_priority")
+    decode_priority = request_data.get("smg_decode_priority")
+
+    prefill_request = _strip_internal_fields(request_data)
+    decode_request = _strip_internal_fields(request_data)
+    bootstrap_payload = _build_bootstrap_payload(prefill_server, bootstrap_port)
+    prefill_request.update(bootstrap_payload)
+    decode_request.update(bootstrap_payload)
+
+    if prefill_priority is not None:
+        prefill_request["priority"] = int(prefill_priority)
+    if decode_priority is not None:
+        decode_request["priority"] = int(decode_priority)
+    return prefill_request, decode_request
+
+
+def _strip_internal_fields(request_data: dict) -> dict:
+    cleaned = request_data.copy()
+    cleaned.pop("smg_prefill_priority", None)
+    cleaned.pop("smg_decode_priority", None)
+    return cleaned
+
+
+def _require_state() -> RouterState:
+    if router_state is None:
+        raise HTTPException(status_code=500, detail="router not initialized")
+    return router_state
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Minimal local PD router")
+    parser.add_argument("--host", default="127.0.0.1")
+    parser.add_argument("--port", type=int, default=8000)
+    parser.add_argument(
+        "--prefill",
+        nargs=2,
+        metavar=("URL", "BOOTSTRAP_PORT"),
+        action="append",
+        required=True,
+    )
+    parser.add_argument(
+        "--decode",
+        action="append",
+        required=True,
+    )
+    parser.add_argument("--prefill-policy", default="round_robin")
+    parser.add_argument("--decode-policy", default="manual")
+    parser.add_argument("--request-timeout-s", type=float, default=1800.0)
+    args = parser.parse_args()
+
+    global router_state
+    router_state = RouterState(
+        RouterConfig(
+            host=args.host,
+            port=args.port,
+            prefill_urls=[(url, int(port)) for url, port in args.prefill],
+            decode_urls=list(args.decode),
+            prefill_policy=args.prefill_policy,
+            decode_policy=args.decode_policy,
+            request_timeout_s=args.request_timeout_s,
+        )
+    )
+    uvicorn.run(app, host=args.host, port=args.port, log_level="info")
+
+
+if __name__ == "__main__":
+    main()

src/agentic_pd_hybrid/policies.py

@@ -0,0 +1,235 @@
+from __future__ import annotations
+
+from collections import Counter
+from dataclasses import dataclass, field
+from typing import Protocol
+
+from agentic_pd_hybrid.topology import SingleNodeTopology
+from agentic_pd_hybrid.trace import TraceRequest
+
+
+@dataclass
+class SessionRouteState:
+    last_decode_worker: str | None = None
+
+
+@dataclass
+class RoutingDecision:
+    policy_name: str
+    prefill_worker_id: str
+    decode_worker_id: str
+    decode_worker_index: int
+    reuse_expected: bool
+    observed_overlap_blocks: int
+    kv_transfer_blocks: int
+    inflight_decode_load: int
+    session_id: str
+    request_id: str
+    turn_id: int
+
+    @property
+    def observed_reuse(self) -> bool:
+        return self.observed_overlap_blocks > 0
+
+    @property
+    def re_prefill_required(self) -> bool:
+        return self.turn_id > 1 and self.observed_overlap_blocks == 0
+
+
+@dataclass
+class RoutingState:
+    prefill_cursor: int = 0
+    decode_cursor: int = 0
+    session_state: dict[str, SessionRouteState] = field(default_factory=dict)
+    inflight_decode: Counter[str] = field(default_factory=Counter)
+    decode_assignment_counts: Counter[str] = field(default_factory=Counter)
+    decode_resident_blocks: dict[str, set[int]] = field(default_factory=dict)
+
+    @classmethod
+    def create(cls, topology: SingleNodeTopology) -> "RoutingState":
+        return cls(
+            decode_resident_blocks={
+                worker.worker_id: set() for worker in topology.route_workers
+            }
+        )
+
+    def next_prefill_worker_id(self, topology: SingleNodeTopology) -> str:
+        if not topology.prefill_workers:
+            return "none"
+        worker = topology.prefill_workers[self.prefill_cursor % len(topology.prefill_workers)]
+        self.prefill_cursor += 1
+        return worker.worker_id
+
+    def next_decode_worker_id(self, topology: SingleNodeTopology) -> str:
+        route_workers = topology.route_workers
+        worker = route_workers[self.decode_cursor % len(route_workers)]
+        self.decode_cursor += 1
+        return worker.worker_id
+
+    def finish(self, request: TraceRequest, decision: RoutingDecision) -> None:
+        session = self.session_state.setdefault(request.session_id, SessionRouteState())
+        session.last_decode_worker = decision.decode_worker_id
+        self.decode_resident_blocks[decision.decode_worker_id].update(request.hash_ids)
+        self.inflight_decode[decision.decode_worker_id] -= 1
+        if self.inflight_decode[decision.decode_worker_id] <= 0:
+            del self.inflight_decode[decision.decode_worker_id]
+
+
+class RoutingPolicy(Protocol):
+    name: str
+
+    def select(
+        self,
+        request: TraceRequest,
+        *,
+        topology: SingleNodeTopology,
+        state: RoutingState,
+    ) -> RoutingDecision:
+        ...
+
+
+@dataclass(frozen=True)
+class DefaultPolicy:
+    name: str = "default"
+
+    def select(
+        self,
+        request: TraceRequest,
+        *,
+        topology: SingleNodeTopology,
+        state: RoutingState,
+    ) -> RoutingDecision:
+        prefill_worker_id = state.next_prefill_worker_id(topology)
+        decode_worker_id = state.next_decode_worker_id(topology)
+        return _build_decision(
+            policy_name=self.name,
+            request=request,
+            topology=topology,
+            state=state,
+            prefill_worker_id=prefill_worker_id,
+            decode_worker_id=decode_worker_id,
+            reuse_expected=False,
+        )
+
+
+@dataclass(frozen=True)
+class StickyDecodePolicy:
+    name: str = "sticky"
+
+    def select(
+        self,
+        request: TraceRequest,
+        *,
+        topology: SingleNodeTopology,
+        state: RoutingState,
+    ) -> RoutingDecision:
+        session = state.session_state.get(request.session_id)
+        prefill_worker_id = state.next_prefill_worker_id(topology)
+        if request.turn_id > 1 and session and session.last_decode_worker is not None:
+            decode_worker_id = session.last_decode_worker
+            reuse_expected = True
+        else:
+            decode_worker_id = state.next_decode_worker_id(topology)
+            reuse_expected = False
+        return _build_decision(
+            policy_name=self.name,
+            request=request,
+            topology=topology,
+            state=state,
+            prefill_worker_id=prefill_worker_id,
+            decode_worker_id=decode_worker_id,
+            reuse_expected=reuse_expected,
+        )
+
+
+@dataclass(frozen=True)
+class KvAwarePolicy:
+    name: str = "kv-aware"
+    sticky_bonus: int = 1
+
+    def select(
+        self,
+        request: TraceRequest,
+        *,
+        topology: SingleNodeTopology,
+        state: RoutingState,
+    ) -> RoutingDecision:
+        prefill_worker_id = state.next_prefill_worker_id(topology)
+        session = state.session_state.get(request.session_id)
+
+        best_decode_worker_id: str | None = None
+        best_score: tuple[int, int, int] | None = None
+        for worker in topology.route_workers:
+            overlap = _overlap_blocks(request, state, worker.worker_id)
+            sticky = int(session is not None and session.last_decode_worker == worker.worker_id)
+            inflight_penalty = -state.inflight_decode.get(worker.worker_id, 0)
+            assignment_penalty = -state.decode_assignment_counts.get(worker.worker_id, 0)
+            score = (
+                overlap + sticky * self.sticky_bonus,
+                sticky,
+                inflight_penalty,
+                assignment_penalty,
+            )
+            if best_score is None or score > best_score:
+                best_score = score
+                best_decode_worker_id = worker.worker_id
+
+        assert best_decode_worker_id is not None
+        reuse_expected = bool(best_score and best_score[0] > 0)
+        return _build_decision(
+            policy_name=self.name,
+            request=request,
+            topology=topology,
+            state=state,
+            prefill_worker_id=prefill_worker_id,
+            decode_worker_id=best_decode_worker_id,
+            reuse_expected=reuse_expected,
+        )
+
+
+def create_policy(name: str) -> RoutingPolicy:
+    normalized = name.strip().lower()
+    if normalized == "default":
+        return DefaultPolicy()
+    if normalized == "sticky":
+        return StickyDecodePolicy()
+    if normalized in {"kv-aware", "kv_aware", "kv"}:
+        return KvAwarePolicy()
+    raise ValueError(f"Unsupported policy: {name}")
+
+
+def _build_decision(
+    *,
+    policy_name: str,
+    request: TraceRequest,
+    topology: SingleNodeTopology,
+    state: RoutingState,
+    prefill_worker_id: str,
+    decode_worker_id: str,
+    reuse_expected: bool,
+) -> RoutingDecision:
+    overlap = _overlap_blocks(request, state, decode_worker_id)
+    state.inflight_decode[decode_worker_id] += 1
+    state.decode_assignment_counts[decode_worker_id] += 1
+    return RoutingDecision(
+        policy_name=policy_name,
+        prefill_worker_id=prefill_worker_id,
+        decode_worker_id=decode_worker_id,
+        decode_worker_index=topology.route_index(decode_worker_id),
+        reuse_expected=reuse_expected,
+        observed_overlap_blocks=overlap,
+        kv_transfer_blocks=max(0, len(request.hash_ids) - overlap),
+        inflight_decode_load=state.inflight_decode[decode_worker_id],
+        session_id=request.session_id,
+        request_id=request.request_id,
+        turn_id=request.turn_id,
+    )
+
+
+def _overlap_blocks(
+    request: TraceRequest,
+    state: RoutingState,
+    decode_worker_id: str,
+) -> int:
+    resident = state.decode_resident_blocks.get(decode_worker_id, set())
+    return sum(1 for block in request.hash_ids if block in resident)

src/agentic_pd_hybrid/profile.py

@@ -0,0 +1,511 @@
+from __future__ import annotations
+
+import json
+import statistics
+from collections import Counter, defaultdict
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Iterable
+
+from agentic_pd_hybrid.trace import TraceRequest, load_trace
+
+
+@dataclass(frozen=True)
+class ProfileConfig:
+    trace_path: Path
+    output_path: Path | None = None
+    metrics_path: Path | None = None
+    baseline_metrics_path: Path | None = None
+    candidate_metrics_path: Path | None = None
+    direct_max_uncached_tokens: int = 2048
+
+
+def write_profile(config: ProfileConfig) -> dict[str, Any]:
+    report = build_profile(config)
+    if config.output_path is not None:
+        config.output_path.parent.mkdir(parents=True, exist_ok=True)
+        with config.output_path.open("w", encoding="utf-8") as handle:
+            json.dump(report, handle, indent=2, sort_keys=True)
+    return report
+
+
+def build_profile(config: ProfileConfig) -> dict[str, Any]:
+    requests = load_trace(config.trace_path)
+    features = _build_trace_features(
+        requests,
+        direct_max_uncached_tokens=config.direct_max_uncached_tokens,
+    )
+    report: dict[str, Any] = {
+        "trace_path": str(config.trace_path),
+        "direct_max_uncached_tokens": config.direct_max_uncached_tokens,
+        "trace_profile": _trace_profile(requests, features),
+    }
+
+    if config.metrics_path is not None:
+        metrics = _load_jsonl(config.metrics_path)
+        report["metrics_path"] = str(config.metrics_path)
+        report["metrics_profile"] = _metrics_profile(metrics, features)
+
+    if (
+        config.baseline_metrics_path is not None
+        and config.candidate_metrics_path is not None
+    ):
+        baseline = _load_jsonl(config.baseline_metrics_path)
+        candidate = _load_jsonl(config.candidate_metrics_path)
+        report["baseline_metrics_path"] = str(config.baseline_metrics_path)
+        report["candidate_metrics_path"] = str(config.candidate_metrics_path)
+        report["baseline_profile"] = _metrics_profile(baseline, features)
+        report["candidate_profile"] = _metrics_profile(candidate, features)
+        report["paired_comparison"] = _paired_comparison(
+            baseline=baseline,
+            candidate=candidate,
+            features=features,
+        )
+
+    return report
+
+
+def print_profile_summary(report: dict[str, Any]) -> None:
+    trace = report["trace_profile"]
+    print(
+        "trace: "
+        f"{trace['request_count']} requests, "
+        f"{trace['session_count']} sessions, "
+        f"{trace['multi_turn_session_count']} multi-turn sessions"
+    )
+    print(
+        "direct-eligible turns: "
+        f"{trace['direct_eligible_turn2plus_count']}/"
+        f"{trace['turn2plus_count']} "
+        f"({trace['direct_eligible_turn2plus_ratio']:.3f})"
+    )
+    append_stats = trace.get("append_input_tokens_stats")
+    output_stats = trace.get("output_tokens_stats")
+    if append_stats is not None:
+        print(
+            "append tokens: "
+            f"mean={append_stats['mean']:.1f} "
+            f"p50={append_stats['p50']:.1f} "
+            f"p90={append_stats['p90']:.1f} "
+            f"p99={append_stats['p99']:.1f}"
+        )
+    if output_stats is not None:
+        print(
+            "output tokens: "
+            f"mean={output_stats['mean']:.1f} "
+            f"p50={output_stats['p50']:.1f} "
+            f"p90={output_stats['p90']:.1f} "
+            f"p99={output_stats['p99']:.1f}"
+        )
+
+    comparison = report.get("paired_comparison")
+    if isinstance(comparison, dict):
+        overall = comparison.get("overall", {})
+        delta = overall.get("latency_delta_s_stats")
+        if delta is not None:
+            print(
+                "candidate - baseline E2E: "
+                f"mean={delta['mean']:.3f}s "
+                f"p50={delta['p50']:.3f}s "
+                f"p90={delta['p90']:.3f}s"
+            )
+        print(
+            "paired wins/losses: "
+            f"{overall.get('candidate_faster_count', 0)} faster, "
+            f"{overall.get('candidate_slower_count', 0)} slower, "
+            f"{overall.get('paired_count', 0)} paired"
+        )
+
+
+@dataclass(frozen=True)
+class _TraceFeature:
+    request_id: str
+    session_id: str
+    turn_id: int
+    input_length: int
+    output_length: int
+    resident_tokens: int
+    append_input_tokens: int | None
+    inter_turn_gap_s: float | None
+    overlap_blocks_with_previous: int | None
+    overlap_ratio_with_previous: float | None
+    direct_eligible: bool
+    turn_class: str
+    append_bin: str
+    input_bin: str
+    output_bin: str
+    resident_bin: str
+
+
+def _build_trace_features(
+    requests: list[TraceRequest],
+    *,
+    direct_max_uncached_tokens: int,
+) -> dict[str, _TraceFeature]:
+    ordered_by_session: dict[str, list[TraceRequest]] = defaultdict(list)
+    for request in requests:
+        ordered_by_session[request.session_id].append(request)
+
+    previous_by_request_id: dict[str, TraceRequest | None] = {}
+    for session_requests in ordered_by_session.values():
+        ordered = sorted(
+            session_requests,
+            key=lambda request: (request.timestamp_s, request.turn_id, request.chat_id),
+        )
+        previous: TraceRequest | None = None
+        for request in ordered:
+            previous_by_request_id[request.request_id] = previous
+            previous = request
+
+    features: dict[str, _TraceFeature] = {}
+    for request in requests:
+        previous = previous_by_request_id.get(request.request_id)
+        append_input_tokens: int | None = None
+        inter_turn_gap_s: float | None = None
+        overlap_blocks: int | None = None
+        overlap_ratio: float | None = None
+        direct_eligible = False
+
+        if previous is not None:
+            append_input_tokens = request.input_length - (
+                previous.input_length + previous.output_length
+            )
+            inter_turn_gap_s = request.timestamp_s - previous.timestamp_s
+            previous_blocks = set(previous.hash_ids)
+            overlap_blocks = sum(1 for block in request.hash_ids if block in previous_blocks)
+            overlap_ratio = (
+                overlap_blocks / len(request.hash_ids) if request.hash_ids else 0.0
+            )
+            direct_eligible = (
+                append_input_tokens > 0
+                and append_input_tokens <= direct_max_uncached_tokens
+                and overlap_blocks > 0
+            )
+
+        features[request.request_id] = _TraceFeature(
+            request_id=request.request_id,
+            session_id=request.session_id,
+            turn_id=request.turn_id,
+            input_length=request.input_length,
+            output_length=request.output_length,
+            resident_tokens=request.input_length + request.output_length,
+            append_input_tokens=append_input_tokens,
+            inter_turn_gap_s=inter_turn_gap_s,
+            overlap_blocks_with_previous=overlap_blocks,
+            overlap_ratio_with_previous=overlap_ratio,
+            direct_eligible=direct_eligible,
+            turn_class="turn1" if request.turn_id <= 1 else "turn2plus",
+            append_bin=_token_bin(append_input_tokens),
+            input_bin=_token_bin(request.input_length),
+            output_bin=_token_bin(request.output_length),
+            resident_bin=_token_bin(request.input_length + request.output_length),
+        )
+    return features
+
+
+def _trace_profile(
+    requests: list[TraceRequest],
+    features: dict[str, _TraceFeature],
+) -> dict[str, Any]:
+    session_turns = Counter(request.session_id for request in requests)
+    turn2plus = [feature for feature in features.values() if feature.turn_id > 1]
+    direct_eligible = [feature for feature in turn2plus if feature.direct_eligible]
+    append_values = [
+        feature.append_input_tokens
+        for feature in turn2plus
+        if feature.append_input_tokens is not None
+    ]
+    positive_append_values = [
+        value for value in append_values if value is not None and value > 0
+    ]
+    overlap_ratios = [
+        feature.overlap_ratio_with_previous
+        for feature in turn2plus
+        if feature.overlap_ratio_with_previous is not None
+    ]
+    gaps = [
+        feature.inter_turn_gap_s
+        for feature in turn2plus
+        if feature.inter_turn_gap_s is not None
+    ]
+    return {
+        "request_count": len(requests),
+        "session_count": len(session_turns),
+        "multi_turn_session_count": sum(1 for turns in session_turns.values() if turns > 1),
+        "turn2plus_count": len(turn2plus),
+        "direct_eligible_turn2plus_count": len(direct_eligible),
+        "direct_eligible_turn2plus_ratio": (
+            len(direct_eligible) / len(turn2plus) if turn2plus else 0.0
+        ),
+        "turn_count_distribution": dict(sorted(Counter(session_turns.values()).items())),
+        "request_type_distribution": dict(
+            sorted(Counter(request.request_type for request in requests).items())
+        ),
+        "turn_id_distribution": dict(
+            sorted(Counter(request.turn_id for request in requests).items())
+        ),
+        "append_bin_distribution": dict(
+            sorted(Counter(feature.append_bin for feature in turn2plus).items())
+        ),
+        "input_bin_distribution": dict(
+            sorted(Counter(feature.input_bin for feature in features.values()).items())
+        ),
+        "output_bin_distribution": dict(
+            sorted(Counter(feature.output_bin for feature in features.values()).items())
+        ),
+        "resident_bin_distribution": dict(
+            sorted(Counter(feature.resident_bin for feature in features.values()).items())
+        ),
+        "input_tokens_stats": _stats(
+            [float(request.input_length) for request in requests]
+        ),
+        "output_tokens_stats": _stats(
+            [float(request.output_length) for request in requests]
+        ),
+        "resident_tokens_stats": _stats(
+            [float(feature.resident_tokens) for feature in features.values()]
+        ),
+        "append_input_tokens_stats": _stats(
+            [float(value) for value in append_values if value is not None]
+        ),
+        "positive_append_input_tokens_stats": _stats(
+            [float(value) for value in positive_append_values]
+        ),
+        "inter_turn_gap_s_stats": _stats([float(value) for value in gaps]),
+        "overlap_ratio_stats": _stats([float(value) for value in overlap_ratios]),
+        "non_positive_append_count": sum(
+            1 for value in append_values if value is not None and value <= 0
+        ),
+    }
+
+
+def _metrics_profile(
+    rows: list[dict[str, Any]],
+    features: dict[str, _TraceFeature],
+) -> dict[str, Any]:
+    return {
+        "request_count": len(rows),
+        "mechanism_distribution": dict(
+            sorted(Counter(str(row.get("mechanism_name")) for row in rows).items())
+        ),
+        "execution_mode_distribution": dict(
+            sorted(Counter(str(row.get("execution_mode")) for row in rows).items())
+        ),
+        "latency_s_stats": _stats(_numeric_values(rows, "latency_s")),
+        "ttft_s_stats": _stats(_numeric_values(rows, "ttft_s")),
+        "tpot_s_stats": _stats(_numeric_values(rows, "tpot_s")),
+        "cached_tokens_stats": _stats(_numeric_values(rows, "cached_tokens")),
+        "actual_kv_transfer_blocks_stats": _stats(
+            _numeric_values(rows, "actual_kv_transfer_blocks")
+        ),
+        "session_reused_count": sum(1 for row in rows if row.get("session_reused")),
+        "session_reset_count": sum(1 for row in rows if row.get("session_reset")),
+        "error_count": sum(1 for row in rows if row.get("error") is not None),
+        "by_turn_class": _group_metrics(rows, features, lambda feature, _row: feature.turn_class),
+        "by_direct_eligible": _group_metrics(
+            rows,
+            features,
+            lambda feature, _row: "eligible" if feature.direct_eligible else "not_eligible",
+        ),
+        "by_append_bin": _group_metrics(rows, features, lambda feature, _row: feature.append_bin),
+        "by_resident_bin": _group_metrics(
+            rows,
+            features,
+            lambda feature, _row: feature.resident_bin,
+        ),
+        "by_execution_mode": _group_metrics(
+            rows,
+            features,
+            lambda _feature, row: str(row.get("execution_mode")),
+        ),
+    }
+
+
+def _paired_comparison(
+    *,
+    baseline: list[dict[str, Any]],
+    candidate: list[dict[str, Any]],
+    features: dict[str, _TraceFeature],
+) -> dict[str, Any]:
+    baseline_by_id = {
+        str(row.get("request_id")): row
+        for row in baseline
+        if row.get("latency_s") is not None
+    }
+    candidate_by_id = {
+        str(row.get("request_id")): row
+        for row in candidate
+        if row.get("latency_s") is not None
+    }
+    paired_ids = sorted(set(baseline_by_id) & set(candidate_by_id))
+    pairs = [
+        (baseline_by_id[request_id], candidate_by_id[request_id], features.get(request_id))
+        for request_id in paired_ids
+    ]
+    pairs = [pair for pair in pairs if pair[2] is not None]
+
+    return {
+        "overall": _delta_summary(pairs),
+        "by_turn_class": _group_deltas(
+            pairs,
+            lambda feature, _base, _cand: feature.turn_class,
+        ),
+        "by_direct_eligible": _group_deltas(
+            pairs,
+            lambda feature, _base, _cand: (
+                "eligible" if feature.direct_eligible else "not_eligible"
+            ),
+        ),
+        "by_append_bin": _group_deltas(
+            pairs,
+            lambda feature, _base, _cand: feature.append_bin,
+        ),
+        "by_resident_bin": _group_deltas(
+            pairs,
+            lambda feature, _base, _cand: feature.resident_bin,
+        ),
+        "by_candidate_execution_mode": _group_deltas(
+            pairs,
+            lambda _feature, _base, cand: str(cand.get("execution_mode")),
+        ),
+    }
+
+
+def _group_metrics(
+    rows: list[dict[str, Any]],
+    features: dict[str, _TraceFeature],
+    key_fn,
+) -> dict[str, Any]:
+    grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
+    for row in rows:
+        feature = features.get(str(row.get("request_id")))
+        if feature is None:
+            continue
+        grouped[str(key_fn(feature, row))].append(row)
+    return {
+        key: {
+            "count": len(group_rows),
+            "latency_s_stats": _stats(_numeric_values(group_rows, "latency_s")),
+            "ttft_s_stats": _stats(_numeric_values(group_rows, "ttft_s")),
+            "tpot_s_stats": _stats(_numeric_values(group_rows, "tpot_s")),
+            "session_reused_count": sum(
+                1 for row in group_rows if row.get("session_reused")
+            ),
+            "error_count": sum(1 for row in group_rows if row.get("error") is not None),
+        }
+        for key, group_rows in sorted(grouped.items())
+    }
+
+
+def _group_deltas(
+    pairs: list[tuple[dict[str, Any], dict[str, Any], _TraceFeature | None]],
+    key_fn,
+) -> dict[str, Any]:
+    grouped: dict[str, list[tuple[dict[str, Any], dict[str, Any], _TraceFeature | None]]] = (
+        defaultdict(list)
+    )
+    for base, cand, feature in pairs:
+        if feature is None:
+            continue
+        grouped[str(key_fn(feature, base, cand))].append((base, cand, feature))
+    return {key: _delta_summary(group_pairs) for key, group_pairs in sorted(grouped.items())}
+
+
+def _delta_summary(
+    pairs: list[tuple[dict[str, Any], dict[str, Any], _TraceFeature | None]],
+) -> dict[str, Any]:
+    latency_deltas = [
+        float(cand["latency_s"]) - float(base["latency_s"])
+        for base, cand, _feature in pairs
+        if base.get("latency_s") is not None and cand.get("latency_s") is not None
+    ]
+    ttft_deltas = [
+        float(cand["ttft_s"]) - float(base["ttft_s"])
+        for base, cand, _feature in pairs
+        if base.get("ttft_s") is not None and cand.get("ttft_s") is not None
+    ]
+    return {
+        "paired_count": len(latency_deltas),
+        "candidate_faster_count": sum(1 for delta in latency_deltas if delta < 0),
+        "candidate_slower_count": sum(1 for delta in latency_deltas if delta > 0),
+        "latency_delta_s_stats": _stats(latency_deltas),
+        "ttft_delta_s_stats": _stats(ttft_deltas),
+        "total_latency_delta_s": sum(latency_deltas),
+        "mean_baseline_latency_s": _mean(
+            [
+                float(base["latency_s"])
+                for base, cand, _feature in pairs
+                if base.get("latency_s") is not None and cand.get("latency_s") is not None
+            ]
+        ),
+        "mean_candidate_latency_s": _mean(
+            [
+                float(cand["latency_s"])
+                for base, cand, _feature in pairs
+                if base.get("latency_s") is not None and cand.get("latency_s") is not None
+            ]
+        ),
+    }
+
+
+def _load_jsonl(path: Path) -> list[dict[str, Any]]:
+    rows: list[dict[str, Any]] = []
+    with path.open("r", encoding="utf-8") as handle:
+        for line in handle:
+            if line.strip():
+                rows.append(json.loads(line))
+    return rows
+
+
+def _numeric_values(rows: Iterable[dict[str, Any]], key: str) -> list[float]:
+    values: list[float] = []
+    for row in rows:
+        value = row.get(key)
+        if value is not None:
+            values.append(float(value))
+    return values
+
+
+def _stats(values: list[float]) -> dict[str, float] | None:
+    clean = [value for value in values if value is not None]
+    if not clean:
+        return None
+    clean.sort()
+    return {
+        "count": float(len(clean)),
+        "mean": statistics.fmean(clean),
+        "p50": _percentile(clean, 0.50),
+        "p90": _percentile(clean, 0.90),
+        "p99": _percentile(clean, 0.99),
+        "min": clean[0],
+        "max": clean[-1],
+    }
+
+
+def _percentile(sorted_values: list[float], percentile: float) -> float:
+    if len(sorted_values) == 1:
+        return sorted_values[0]
+    index = round((len(sorted_values) - 1) * percentile)
+    return sorted_values[index]
+
+
+def _mean(values: list[float]) -> float | None:
+    if not values:
+        return None
+    return statistics.fmean(values)
+
+
+def _token_bin(value: int | None) -> str:
+    if value is None:
+        return "none"
+    if value <= 0:
+        return "<=0"
+    if value <= 512:
+        return "1-512"
+    if value <= 2048:
+        return "513-2048"
+    if value <= 8192:
+        return "2049-8192"
+    if value <= 32768:
+        return "8193-32768"
+    return ">32768"

src/agentic_pd_hybrid/sampling.py

@@ -0,0 +1,295 @@
+from __future__ import annotations
+
+import hashlib
+import json
+from collections import defaultdict
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Literal
+
+from agentic_pd_hybrid.trace import TraceRequest, load_trace
+
+
+SampleProfile = Literal["default", "small-append"]
+
+
+@dataclass(frozen=True)
+class SessionSampleConfig:
+    trace_path: Path
+    output_path: Path
+    target_duration_s: float = 600.0
+    start_time_s: float = 0.0
+    session_sample_rate: float = 1.0
+    min_turns: int = 1
+    max_requests: int | None = None
+    profile: SampleProfile = "default"
+    min_initial_input_tokens: int | None = None
+    max_initial_input_tokens: int | None = None
+    max_append_input_tokens: int | None = None
+    max_output_tokens: int | None = None
+    min_overlap_ratio: float | None = None
+
+
+@dataclass(frozen=True)
+class SessionSampleSummary:
+    input_trace_path: str
+    output_trace_path: str
+    request_count: int
+    session_count: int
+    multi_turn_session_count: int
+    start_time_s: float
+    end_time_s: float
+    sampled_duration_s: float
+    session_sample_rate: float
+    min_turns: int
+    profile: str
+    min_initial_input_tokens: int | None
+    max_initial_input_tokens: int | None
+    max_append_input_tokens: int | None
+    max_output_tokens: int | None
+    min_overlap_ratio: float | None
+    mean_append_input_tokens: float | None
+    mean_turn_overlap_ratio: float | None
+
+
+def sample_trace_sessions(config: SessionSampleConfig) -> SessionSampleSummary:
+    requests = load_trace(config.trace_path)
+    sessions: dict[str, list[TraceRequest]] = defaultdict(list)
+    for request in requests:
+        sessions[request.session_id].append(request)
+
+    filters = _resolve_filters(config)
+    eligible_sessions = {
+        session_id: session_requests
+        for session_id, session_requests in sessions.items()
+        if len(session_requests) >= filters.min_turns
+        and _session_matches_filters(session_requests, filters)
+        and _keep_session(session_id, config.session_sample_rate)
+    }
+    ordered_sessions = sorted(
+        eligible_sessions.values(),
+        key=lambda session_requests: session_requests[0].timestamp_s,
+    )
+
+    selected_requests: list[TraceRequest] = []
+    sampled_start: float | None = None
+    sampled_end: float | None = None
+    for session_requests in ordered_sessions:
+        session_first = session_requests[0].timestamp_s
+        if session_first < config.start_time_s:
+            continue
+
+        if sampled_start is None:
+            sampled_start = session_first
+
+        selected_requests.extend(session_requests)
+        sampled_end = max(request.timestamp_s for request in session_requests)
+
+        if config.max_requests is not None and len(selected_requests) >= config.max_requests:
+            break
+        if sampled_end - sampled_start >= config.target_duration_s:
+            break
+
+    selected_requests.sort(key=lambda request: request.timestamp_s)
+    if config.max_requests is not None:
+        selected_requests = selected_requests[: config.max_requests]
+
+    if not selected_requests:
+        raise ValueError("Sampling produced no requests; adjust the sampling arguments")
+
+    config.output_path.parent.mkdir(parents=True, exist_ok=True)
+    with config.output_path.open("w", encoding="utf-8") as handle:
+        for request in selected_requests:
+            payload = {
+                "request_id": request.request_id,
+                "session_id": request.session_id,
+                "chat_id": request.chat_id,
+                "parent_chat_id": request.parent_chat_id,
+                "timestamp": request.timestamp_s,
+                "input_length": request.input_length,
+                "output_length": request.output_length,
+                "type": request.request_type,
+                "turn": request.turn_id,
+                "hash_ids": list(request.hash_ids),
+            }
+            handle.write(json.dumps(payload, sort_keys=True) + "\n")
+
+    selected_session_ids = {request.session_id for request in selected_requests}
+    selected_session_requests = [
+        eligible_sessions[session_id] for session_id in selected_session_ids
+    ]
+    append_lengths = [
+        length
+        for session_requests in selected_session_requests
+        for length in _turn_append_lengths(session_requests)
+    ]
+    overlap_ratios = [
+        ratio
+        for session_requests in selected_session_requests
+        for ratio in _turn_overlap_ratios(session_requests)
+    ]
+    summary = SessionSampleSummary(
+        input_trace_path=str(config.trace_path),
+        output_trace_path=str(config.output_path),
+        request_count=len(selected_requests),
+        session_count=len(selected_session_ids),
+        multi_turn_session_count=sum(
+            1
+            for session_id in selected_session_ids
+            if len(eligible_sessions[session_id]) > 1
+        ),
+        start_time_s=selected_requests[0].timestamp_s,
+        end_time_s=selected_requests[-1].timestamp_s,
+        sampled_duration_s=selected_requests[-1].timestamp_s
+        - selected_requests[0].timestamp_s,
+        session_sample_rate=config.session_sample_rate,
+        min_turns=filters.min_turns,
+        profile=config.profile,
+        min_initial_input_tokens=filters.min_initial_input_tokens,
+        max_initial_input_tokens=filters.max_initial_input_tokens,
+        max_append_input_tokens=filters.max_append_input_tokens,
+        max_output_tokens=filters.max_output_tokens,
+        min_overlap_ratio=filters.min_overlap_ratio,
+        mean_append_input_tokens=_mean(append_lengths),
+        mean_turn_overlap_ratio=_mean(overlap_ratios),
+    )
+    summary_path = config.output_path.with_suffix(config.output_path.suffix + ".summary.json")
+    with summary_path.open("w", encoding="utf-8") as handle:
+        json.dump(asdict(summary), handle, indent=2, sort_keys=True)
+    return summary
+
+
+@dataclass(frozen=True)
+class _ResolvedFilters:
+    min_turns: int
+    min_initial_input_tokens: int | None
+    max_initial_input_tokens: int | None
+    max_append_input_tokens: int | None
+    max_output_tokens: int | None
+    min_overlap_ratio: float | None
+
+
+def _resolve_filters(config: SessionSampleConfig) -> _ResolvedFilters:
+    if config.profile == "default":
+        return _ResolvedFilters(
+            min_turns=config.min_turns,
+            min_initial_input_tokens=config.min_initial_input_tokens,
+            max_initial_input_tokens=config.max_initial_input_tokens,
+            max_append_input_tokens=config.max_append_input_tokens,
+            max_output_tokens=config.max_output_tokens,
+            min_overlap_ratio=config.min_overlap_ratio,
+        )
+
+    if config.profile != "small-append":
+        raise ValueError(f"Unsupported sample profile: {config.profile}")
+
+    return _ResolvedFilters(
+        min_turns=max(config.min_turns, 2),
+        min_initial_input_tokens=(
+            2048
+            if config.min_initial_input_tokens is None
+            else config.min_initial_input_tokens
+        ),
+        max_initial_input_tokens=(
+            16000
+            if config.max_initial_input_tokens is None
+            else config.max_initial_input_tokens
+        ),
+        max_append_input_tokens=(
+            2048
+            if config.max_append_input_tokens is None
+            else config.max_append_input_tokens
+        ),
+        max_output_tokens=(
+            2048 if config.max_output_tokens is None else config.max_output_tokens
+        ),
+        min_overlap_ratio=(
+            0.75 if config.min_overlap_ratio is None else config.min_overlap_ratio
+        ),
+    )
+
+
+def _session_matches_filters(
+    session_requests: list[TraceRequest],
+    filters: _ResolvedFilters,
+) -> bool:
+    ordered = sorted(
+        session_requests,
+        key=lambda request: (request.timestamp_s, request.turn_id, request.chat_id),
+    )
+    if not ordered:
+        return False
+
+    initial = ordered[0]
+    if (
+        filters.min_initial_input_tokens is not None
+        and initial.input_length < filters.min_initial_input_tokens
+    ):
+        return False
+    if (
+        filters.max_initial_input_tokens is not None
+        and initial.input_length > filters.max_initial_input_tokens
+    ):
+        return False
+    if filters.max_output_tokens is not None and any(
+        request.output_length > filters.max_output_tokens for request in ordered
+    ):
+        return False
+
+    append_lengths = _turn_append_lengths(ordered)
+    if filters.max_append_input_tokens is not None and any(
+        append_length <= 0 or append_length > filters.max_append_input_tokens
+        for append_length in append_lengths
+    ):
+        return False
+
+    overlap_ratios = _turn_overlap_ratios(ordered)
+    if filters.min_overlap_ratio is not None and any(
+        overlap_ratio < filters.min_overlap_ratio for overlap_ratio in overlap_ratios
+    ):
+        return False
+
+    return True
+
+
+def _turn_append_lengths(session_requests: list[TraceRequest]) -> list[int]:
+    ordered = sorted(
+        session_requests,
+        key=lambda request: (request.timestamp_s, request.turn_id, request.chat_id),
+    )
+    return [
+        current.input_length - (previous.input_length + previous.output_length)
+        for previous, current in zip(ordered, ordered[1:], strict=False)
+    ]
+
+
+def _turn_overlap_ratios(session_requests: list[TraceRequest]) -> list[float]:
+    ordered = sorted(
+        session_requests,
+        key=lambda request: (request.timestamp_s, request.turn_id, request.chat_id),
+    )
+    ratios: list[float] = []
+    for previous, current in zip(ordered, ordered[1:], strict=False):
+        if not current.hash_ids:
+            ratios.append(0.0)
+            continue
+        previous_blocks = set(previous.hash_ids)
+        overlap = sum(1 for block in current.hash_ids if block in previous_blocks)
+        ratios.append(overlap / len(current.hash_ids))
+    return ratios
+
+
+def _mean(values: list[int] | list[float]) -> float | None:
+    if not values:
+        return None
+    return sum(values) / len(values)
+
+
+def _keep_session(session_id: str, sample_rate: float) -> bool:
+    if sample_rate >= 1.0:
+        return True
+    if sample_rate <= 0.0:
+        return False
+    digest = hashlib.blake2b(session_id.encode("utf-8"), digest_size=8).digest()
+    bucket = int.from_bytes(digest, byteorder="big", signed=False) / 2**64
+    return bucket < sample_rate

src/agentic_pd_hybrid/stack.py

@@ -0,0 +1,224 @@
+from __future__ import annotations
+
+import os
+import signal
+import subprocess
+import time
+from dataclasses import dataclass
+from pathlib import Path
+
+import httpx
+
+from agentic_pd_hybrid.launcher import build_launch_plan
+from agentic_pd_hybrid.topology import SingleNodeTopology
+
+
+@dataclass
+class ManagedProcess:
+    name: str
+    command: tuple[str, ...]
+    process: subprocess.Popen[bytes]
+    log_path: Path
+
+
+@dataclass
+class ManagedPdStack:
+    topology: SingleNodeTopology
+    run_dir: Path
+    prefill_processes: list[ManagedProcess]
+    decode_processes: list[ManagedProcess]
+    direct_processes: list[ManagedProcess]
+    router_process: ManagedProcess | None
+
+    @property
+    def router_url(self) -> str:
+        return self.topology.router_url
+
+    def stop(self) -> None:
+        processes = (
+            ([self.router_process] if self.router_process is not None else [])
+            + self.direct_processes
+            + self.decode_processes
+            + self.prefill_processes
+        )
+        for managed in processes:
+            if managed.process.poll() is None:
+                os.killpg(os.getpgid(managed.process.pid), signal.SIGTERM)
+        deadline = time.time() + 20
+        for managed in processes:
+            if managed.process.poll() is not None:
+                continue
+            remaining = max(0.0, deadline - time.time())
+            try:
+                managed.process.wait(timeout=remaining)
+            except subprocess.TimeoutExpired:
+                if managed.process.poll() is None:
+                    os.killpg(os.getpgid(managed.process.pid), signal.SIGKILL)
+                    managed.process.wait(timeout=5)
+
+
+def launch_pd_stack(
+    *,
+    topology: SingleNodeTopology,
+    run_dir: Path,
+    prefill_policy: str,
+    decode_policy: str,
+    timeout_s: float = 1200.0,
+    router_request_timeout_s: float | None = None,
+    include_router: bool = True,
+) -> ManagedPdStack:
+    run_dir.mkdir(parents=True, exist_ok=True)
+    logs_dir = run_dir / "logs"
+    logs_dir.mkdir(parents=True, exist_ok=True)
+
+    plan = build_launch_plan(
+        topology,
+        prefill_policy=prefill_policy,
+        decode_policy=decode_policy,
+        include_router=include_router,
+        router_request_timeout_s=router_request_timeout_s,
+    )
+
+    prefill_processes = [
+        _spawn_process(
+            name=f"prefill-{idx}",
+            command=command,
+            log_path=logs_dir / f"prefill-{idx}.log",
+            topology=topology,
+        )
+        for idx, command in enumerate(plan.prefill_commands)
+    ]
+    decode_processes = [
+        _spawn_process(
+            name=f"decode-{idx}",
+            command=command,
+            log_path=logs_dir / f"decode-{idx}.log",
+            topology=topology,
+        )
+        for idx, command in enumerate(plan.decode_commands)
+    ]
+    direct_processes = [
+        _spawn_process(
+            name=f"direct-{idx}",
+            command=command,
+            log_path=logs_dir / f"direct-{idx}.log",
+            topology=topology,
+        )
+        for idx, command in enumerate(plan.direct_commands)
+    ]
+
+    router_process: ManagedProcess | None = None
+    try:
+        for worker in topology.prefill_workers:
+            _wait_for_ready_endpoint(f"{worker.url}/v1/models", timeout_s=timeout_s)
+        for worker in topology.decode_workers:
+            _wait_for_ready_endpoint(f"{worker.url}/v1/models", timeout_s=timeout_s)
+        for worker in topology.direct_workers:
+            _wait_for_ready_endpoint(f"{worker.url}/v1/models", timeout_s=timeout_s)
+
+        if plan.router_command is not None:
+            router_process = _spawn_process(
+                name="router",
+                command=plan.router_command,
+                log_path=logs_dir / "router.log",
+                topology=topology,
+            )
+            _wait_for_ready_endpoint(f"{topology.router_url}/health", timeout_s=timeout_s)
+    except Exception:
+        stack = ManagedPdStack(
+            topology=topology,
+            run_dir=run_dir,
+            prefill_processes=prefill_processes,
+            decode_processes=decode_processes,
+            direct_processes=direct_processes,
+            router_process=router_process,
+        )
+        stack.stop()
+        raise
+
+    return ManagedPdStack(
+        topology=topology,
+        run_dir=run_dir,
+        prefill_processes=prefill_processes,
+        decode_processes=decode_processes,
+        direct_processes=direct_processes,
+        router_process=router_process,
+    )
+
+
+def _spawn_process(
+    *,
+    name: str,
+    command: tuple[str, ...],
+    log_path: Path,
+    topology: SingleNodeTopology,
+) -> ManagedProcess:
+    log_handle = log_path.open("wb")
+    env = _build_process_env(topology)
+    process = subprocess.Popen(
+        command,
+        stdout=log_handle,
+        stderr=subprocess.STDOUT,
+        env=env,
+        preexec_fn=os.setsid,
+    )
+    return ManagedProcess(
+        name=name,
+        command=command,
+        process=process,
+        log_path=log_path,
+    )
+
+
+def _build_process_env(topology: SingleNodeTopology) -> dict[str, str]:
+    env = os.environ.copy()
+    env["PYTHONDONTWRITEBYTECODE"] = "1"
+    env["PYTHONUNBUFFERED"] = "1"
+
+    # SGLang's PD bootstrap path uses `requests`; force localhost traffic to stay local.
+    for key in (
+        "http_proxy",
+        "https_proxy",
+        "all_proxy",
+        "HTTP_PROXY",
+        "HTTPS_PROXY",
+        "ALL_PROXY",
+    ):
+        env.pop(key, None)
+    env["NO_PROXY"] = "*"
+    env["no_proxy"] = "*"
+    env.setdefault("SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT", "600")
+    env.setdefault("SGLANG_DISAGGREGATION_WAITING_TIMEOUT", "600")
+    env.setdefault("FLASHINFER_DISABLE_VERSION_CHECK", "1")
+    if topology.force_rdma:
+        env["MOONCAKE_PROTOCOL"] = "rdma"
+        env["MC_MS_AUTO_DISC"] = "0"
+        if topology.ib_device:
+            env["MOONCAKE_DEVICE"] = topology.ib_device
+
+    repo_root = Path(__file__).resolve().parents[2]
+    python_paths = [
+        str(repo_root / "src"),
+        str(repo_root / "third_party" / "sglang" / "python"),
+    ]
+    existing_pythonpath = env.get("PYTHONPATH")
+    if existing_pythonpath:
+        python_paths.append(existing_pythonpath)
+    env["PYTHONPATH"] = os.pathsep.join(python_paths)
+    return env
+
+
+def _wait_for_ready_endpoint(url: str, *, timeout_s: float) -> None:
+    start = time.perf_counter()
+    last_error: str | None = None
+    with httpx.Client(timeout=5.0, trust_env=False) as client:
+        while time.perf_counter() - start < timeout_s:
+            try:
+                response = client.get(url)
+                if response.status_code == 200:
+                    return
+                last_error = f"status={response.status_code}"
+            except Exception as exc:  # pragma: no cover
+                last_error = f"{type(exc).__name__}: {exc}"
+            time.sleep(1.0)
+    raise TimeoutError(f"Timed out waiting for {url} ({last_error})")

src/agentic_pd_hybrid/topology.py

@@ -0,0 +1,245 @@
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Literal
+
+
+WorkerRole = Literal["prefill", "decode", "direct"]
+
+
+@dataclass(frozen=True)
+class WorkerSpec:
+    role: WorkerRole
+    ordinal: int
+    gpu_ids: tuple[int, ...]
+    host: str
+    port: int
+    bootstrap_port: int | None = None
+
+    @property
+    def worker_id(self) -> str:
+        return f"{self.role}-{self.ordinal}"
+
+    @property
+    def url(self) -> str:
+        return f"http://{self.host}:{self.port}"
+
+    @property
+    def gpu_id(self) -> int:
+        return self.gpu_ids[0]
+
+    @property
+    def tp_size(self) -> int:
+        return len(self.gpu_ids)
+
+
+@dataclass(frozen=True)
+class SingleNodeTopology:
+    model_path: str
+    prefill_workers: tuple[WorkerSpec, ...]
+    decode_workers: tuple[WorkerSpec, ...]
+    direct_workers: tuple[WorkerSpec, ...]
+    router_host: str
+    router_port: int
+    transfer_backend: str
+    trust_remote_code: bool
+    force_rdma: bool = False
+    ib_device: str | None = None
+    extra_server_args: tuple[str, ...] = ()
+    prefill_extra_server_args: tuple[str, ...] = ()
+    decode_extra_server_args: tuple[str, ...] = ()
+    direct_extra_server_args: tuple[str, ...] = ()
+
+    @property
+    def model_name(self) -> str:
+        return Path(self.model_path).name
+
+    @property
+    def router_url(self) -> str:
+        return f"http://{self.router_host}:{self.router_port}"
+
+    @property
+    def route_workers(self) -> tuple[WorkerSpec, ...]:
+        if self.decode_workers:
+            return self.decode_workers
+        return self.direct_workers
+
+    def route_index(self, worker_id: str) -> int:
+        for idx, worker in enumerate(self.route_workers):
+            if worker.worker_id == worker_id:
+                return idx
+        raise KeyError(f"Unknown route worker: {worker_id}")
+
+
+def build_single_node_topology(
+    *,
+    model_path: str,
+    prefill_worker_count: int,
+    decode_worker_count: int,
+    direct_worker_count: int = 0,
+    prefill_tp_size: int = 1,
+    decode_tp_size: int = 1,
+    direct_tp_size: int = 1,
+    prefill_gpu_ids: tuple[int, ...] | None = None,
+    decode_gpu_ids: tuple[int, ...] | None = None,
+    direct_gpu_ids: tuple[int, ...] | None = None,
+    total_gpu_budget: int = 8,
+    host: str = "127.0.0.1",
+    router_port: int = 8000,
+    prefill_port_base: int = 30000,
+    decode_port_base: int = 31000,
+    direct_port_base: int = 32000,
+    bootstrap_port_base: int = 8998,
+    transfer_backend: str = "nixl",
+    force_rdma: bool = False,
+    trust_remote_code: bool = True,
+    ib_device: str | None = None,
+    extra_server_args: tuple[str, ...] = (),
+    prefill_extra_server_args: tuple[str, ...] = (),
+    decode_extra_server_args: tuple[str, ...] = (),
+    direct_extra_server_args: tuple[str, ...] = (),
+) -> SingleNodeTopology:
+    if prefill_worker_count < 0:
+        raise ValueError("prefill_worker_count must be >= 0")
+    if decode_worker_count < 0:
+        raise ValueError("decode_worker_count must be >= 0")
+    if direct_worker_count < 0:
+        raise ValueError("direct_worker_count must be >= 0")
+    if (
+        prefill_worker_count == 0
+        and decode_worker_count == 0
+        and direct_worker_count == 0
+    ):
+        raise ValueError("At least one worker must be configured")
+    if prefill_tp_size <= 0:
+        raise ValueError("prefill_tp_size must be >= 1")
+    if decode_tp_size <= 0:
+        raise ValueError("decode_tp_size must be >= 1")
+    if direct_tp_size <= 0:
+        raise ValueError("direct_tp_size must be >= 1")
+    if force_rdma and not ib_device:
+        raise ValueError("force_rdma requires --ib-device to be set")
+    if force_rdma and transfer_backend != "mooncake":
+        raise ValueError(
+            "force_rdma currently requires transfer_backend='mooncake' "
+            "to guarantee an RDMA path"
+        )
+
+    total_gpus_required = (
+        prefill_worker_count * prefill_tp_size
+        + decode_worker_count * decode_tp_size
+        + direct_worker_count * direct_tp_size
+    )
+    if total_gpus_required > total_gpu_budget:
+        raise ValueError(
+            "Single-node GPU budget exceeded: "
+            f"{prefill_worker_count} prefill x tp={prefill_tp_size} + "
+            f"{decode_worker_count} decode x tp={decode_tp_size} + "
+            f"{direct_worker_count} direct x tp={direct_tp_size} > "
+            f"{total_gpu_budget} GPUs"
+        )
+
+    if prefill_gpu_ids is None:
+        prefill_gpu_ids = tuple(range(prefill_worker_count * prefill_tp_size))
+    if decode_gpu_ids is None:
+        decode_gpu_ids = tuple(
+            range(
+                len(prefill_gpu_ids),
+                len(prefill_gpu_ids) + decode_worker_count * decode_tp_size,
+            )
+        )
+    if direct_gpu_ids is None:
+        direct_gpu_ids = tuple(
+            range(
+                len(prefill_gpu_ids) + len(decode_gpu_ids),
+                len(prefill_gpu_ids)
+                + len(decode_gpu_ids)
+                + direct_worker_count * direct_tp_size,
+            )
+        )
+
+    if len(prefill_gpu_ids) != prefill_worker_count * prefill_tp_size:
+        raise ValueError(
+            "prefill_gpu_ids length must equal prefill_worker_count * prefill_tp_size: "
+            f"{len(prefill_gpu_ids)} != {prefill_worker_count * prefill_tp_size}"
+        )
+    if len(decode_gpu_ids) != decode_worker_count * decode_tp_size:
+        raise ValueError(
+            "decode_gpu_ids length must equal decode_worker_count * decode_tp_size: "
+            f"{len(decode_gpu_ids)} != {decode_worker_count * decode_tp_size}"
+        )
+    if len(direct_gpu_ids) != direct_worker_count * direct_tp_size:
+        raise ValueError(
+            "direct_gpu_ids length must equal direct_worker_count * direct_tp_size: "
+            f"{len(direct_gpu_ids)} != {direct_worker_count * direct_tp_size}"
+        )
+    assigned_gpu_ids = prefill_gpu_ids + decode_gpu_ids + direct_gpu_ids
+    if len(set(assigned_gpu_ids)) != len(assigned_gpu_ids):
+        raise ValueError("prefill/decode/direct GPU IDs must be unique")
+    if any(gpu_id < 0 or gpu_id >= total_gpu_budget for gpu_id in assigned_gpu_ids):
+        raise ValueError(
+            "GPU IDs must fall within the single-node budget range "
+            f"[0, {total_gpu_budget - 1}]"
+        )
+
+    prefill_workers = tuple(
+        WorkerSpec(
+            role="prefill",
+            ordinal=idx,
+            gpu_ids=tuple(
+                prefill_gpu_ids[
+                    idx * prefill_tp_size : (idx + 1) * prefill_tp_size
+                ]
+            ),
+            host=host,
+            port=prefill_port_base + idx,
+            bootstrap_port=bootstrap_port_base + idx,
+        )
+        for idx in range(prefill_worker_count)
+    )
+    decode_workers = tuple(
+        WorkerSpec(
+            role="decode",
+            ordinal=idx,
+            gpu_ids=tuple(
+                decode_gpu_ids[
+                    idx * decode_tp_size : (idx + 1) * decode_tp_size
+                ]
+            ),
+            host=host,
+            port=decode_port_base + idx,
+        )
+        for idx in range(decode_worker_count)
+    )
+    direct_workers = tuple(
+        WorkerSpec(
+            role="direct",
+            ordinal=idx,
+            gpu_ids=tuple(
+                direct_gpu_ids[
+                    idx * direct_tp_size : (idx + 1) * direct_tp_size
+                ]
+            ),
+            host=host,
+            port=direct_port_base + idx,
+        )
+        for idx in range(direct_worker_count)
+    )
+
+    return SingleNodeTopology(
+        model_path=model_path,
+        prefill_workers=prefill_workers,
+        decode_workers=decode_workers,
+        direct_workers=direct_workers,
+        router_host=host,
+        router_port=router_port,
+        transfer_backend=transfer_backend,
+        trust_remote_code=trust_remote_code,
+        force_rdma=force_rdma,
+        ib_device=ib_device,
+        extra_server_args=extra_server_args,
+        prefill_extra_server_args=prefill_extra_server_args,
+        decode_extra_server_args=decode_extra_server_args,
+        direct_extra_server_args=direct_extra_server_args,
+    )

src/agentic_pd_hybrid/trace.py

@@ -0,0 +1,106 @@
+from __future__ import annotations
+
+import json
+from dataclasses import dataclass
+from pathlib import Path
+
+
+@dataclass(frozen=True)
+class TraceRequest:
+    request_id: str
+    session_id: str
+    chat_id: int
+    parent_chat_id: int
+    timestamp_s: float
+    input_length: int
+    output_length: int
+    request_type: str
+    turn_id: int
+    hash_ids: tuple[int, ...]
+
+
+def load_trace(path: Path, *, request_limit: int | None = None) -> list[TraceRequest]:
+    chat_to_session: dict[int, str] = {}
+    requests: list[TraceRequest] = []
+
+    with path.open("r", encoding="utf-8") as handle:
+        for index, line in enumerate(handle):
+            if request_limit is not None and len(requests) >= request_limit:
+                break
+
+            payload = json.loads(line)
+            chat_id = int(payload["chat_id"])
+            parent_chat_id = int(payload["parent_chat_id"])
+            session_id = _resolve_session_id(
+                chat_id=chat_id,
+                parent_chat_id=parent_chat_id,
+                chat_to_session=chat_to_session,
+            )
+            turn_id = int(payload["turn"])
+            request_id = f"{session_id}:{turn_id}:{chat_id}:{index}"
+            requests.append(
+                TraceRequest(
+                    request_id=request_id,
+                    session_id=session_id,
+                    chat_id=chat_id,
+                    parent_chat_id=parent_chat_id,
+                    timestamp_s=float(payload["timestamp"]),
+                    input_length=int(payload["input_length"]),
+                    output_length=int(payload["output_length"]),
+                    request_type=str(payload["type"]),
+                    turn_id=turn_id,
+                    hash_ids=tuple(int(item) for item in payload.get("hash_ids", [])),
+                )
+            )
+
+    return requests
+
+
+def build_synthetic_prompt(
+    request: TraceRequest,
+    *,
+    block_token_budget: int = 24,
+) -> str:
+    return " ".join(build_synthetic_prompt_tokens(request, block_token_budget=block_token_budget))
+
+
+def build_synthetic_prompt_tokens(
+    request: TraceRequest,
+    *,
+    block_token_budget: int = 24,
+) -> list[str]:
+    tokens: list[str] = []
+    for hash_id in request.hash_ids:
+        for offset in range(block_token_budget):
+            tokens.append(f"blk{hash_id}_{offset}")
+
+    while len(tokens) < request.input_length:
+        tokens.append(f"fill_{len(tokens) % 64}")
+
+    return tokens[: request.input_length]
+
+
+def build_synthetic_append_chunk(
+    request: TraceRequest,
+    append_length: int,
+) -> str:
+    if append_length <= 0:
+        return ""
+    return " ".join(
+        f"turn{request.turn_id}_append_{request.chat_id}_{offset}"
+        for offset in range(append_length)
+    )
+
+
+def _resolve_session_id(
+    *,
+    chat_id: int,
+    parent_chat_id: int,
+    chat_to_session: dict[int, str],
+) -> str:
+    if parent_chat_id < 0:
+        session_id = str(chat_id)
+    else:
+        session_id = chat_to_session.get(parent_chat_id, str(parent_chat_id))
+    chat_to_session[chat_id] = session_id
+    return session_id

src/agentic_pd_hybrid/trace_profiles.py

@@ -0,0 +1,127 @@
+from __future__ import annotations
+
+import json
+from collections import defaultdict
+from dataclasses import asdict, dataclass
+from math import ceil
+from pathlib import Path
+
+from agentic_pd_hybrid.trace import TraceRequest, load_trace
+
+
+BLOCK_TOKEN_BUDGET = 24
+
+
+@dataclass(frozen=True)
+class NormalizeTraceLengthsConfig:
+    trace_path: Path
+    output_path: Path
+    initial_input_length: int = 10_000
+    append_input_length: int = 1_000
+    output_length: int = 1_000
+    max_requests: int | None = None
+
+
+@dataclass(frozen=True)
+class NormalizeTraceLengthsSummary:
+    input_trace_path: str
+    output_trace_path: str
+    request_count: int
+    session_count: int
+    multi_turn_session_count: int
+    initial_input_length: int
+    append_input_length: int
+    output_length: int
+    max_turns_per_session: int
+    max_input_length: int
+
+
+def normalize_trace_lengths(
+    config: NormalizeTraceLengthsConfig,
+) -> NormalizeTraceLengthsSummary:
+    if config.initial_input_length < 0:
+        raise ValueError("initial_input_length must be >= 0")
+    if config.append_input_length < 0:
+        raise ValueError("append_input_length must be >= 0")
+    if config.output_length < 0:
+        raise ValueError("output_length must be >= 0")
+
+    requests = load_trace(config.trace_path, request_limit=config.max_requests)
+    sessions: dict[str, list[TraceRequest]] = defaultdict(list)
+    for request in requests:
+        sessions[request.session_id].append(request)
+
+    normalized_records: list[dict[str, object]] = []
+    max_turns_per_session = 0
+    max_input_length = 0
+
+    for session_idx, session_id in enumerate(sorted(sessions, key=_session_sort_key)):
+        session_requests = sorted(
+            sessions[session_id],
+            key=lambda request: (request.timestamp_s, request.turn_id, request.chat_id),
+        )
+        max_turns_per_session = max(max_turns_per_session, len(session_requests))
+        base_block_count = ceil(config.initial_input_length / BLOCK_TOKEN_BUDGET)
+        base_hash_ids = [
+            _hash_id_for(session_idx=session_idx, block_idx=block_idx)
+            for block_idx in range(base_block_count)
+        ]
+
+        for turn_idx, request in enumerate(session_requests):
+            input_length = config.initial_input_length + turn_idx * (
+                config.append_input_length + config.output_length
+            )
+            total_block_count = ceil(input_length / BLOCK_TOKEN_BUDGET)
+            hash_ids = base_hash_ids + [
+                _hash_id_for(
+                    session_idx=session_idx,
+                    block_idx=base_block_count + append_block_idx,
+                )
+                for append_block_idx in range(max(0, total_block_count - base_block_count))
+            ]
+            max_input_length = max(max_input_length, input_length)
+            normalized_records.append(
+                {
+                    "chat_id": request.chat_id,
+                    "parent_chat_id": request.parent_chat_id,
+                    "timestamp": request.timestamp_s,
+                    "input_length": input_length,
+                    "output_length": config.output_length,
+                    "type": request.request_type,
+                    "turn": request.turn_id,
+                    "hash_ids": hash_ids,
+                }
+            )
+
+    normalized_records.sort(key=lambda item: float(item["timestamp"]))
+    config.output_path.parent.mkdir(parents=True, exist_ok=True)
+    with config.output_path.open("w", encoding="utf-8") as handle:
+        for record in normalized_records:
+            handle.write(json.dumps(record, sort_keys=True) + "\n")
+
+    summary = NormalizeTraceLengthsSummary(
+        input_trace_path=str(config.trace_path),
+        output_trace_path=str(config.output_path),
+        request_count=len(normalized_records),
+        session_count=len(sessions),
+        multi_turn_session_count=sum(
+            1 for session_requests in sessions.values() if len(session_requests) > 1
+        ),
+        initial_input_length=config.initial_input_length,
+        append_input_length=config.append_input_length,
+        output_length=config.output_length,
+        max_turns_per_session=max_turns_per_session,
+        max_input_length=max_input_length,
+    )
+    summary_path = config.output_path.with_suffix(config.output_path.suffix + ".summary.json")
+    with summary_path.open("w", encoding="utf-8") as handle:
+        json.dump(asdict(summary), handle, indent=2, sort_keys=True)
+    return summary
+
+
+def _hash_id_for(*, session_idx: int, block_idx: int) -> int:
+    return session_idx * 1_000_000 + block_idx
+
+
+def _session_sort_key(session_id: str) -> tuple[int, str]:
+    return (0, session_id) if session_id.isdigit() else (1, session_id)

third_party/sglang/.claude/skills/add-jit-kernel/SKILL.md

@@ -0,0 +1,607 @@
+---
+name: add-jit-kernel
+description: Step-by-step tutorial for adding a new lightweight JIT CUDA kernel to sglang's jit_kernel module
+---
+
+# Tutorial: Adding a New JIT Kernel to SGLang
+
+This tutorial walks through adding a simple element-wise scale operation as a JIT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
+
+## Goal
+
+Add a new operation that scales each element of a tensor by a scalar factor:
+
+- Input: tensor `x` (CUDA) and scalar `factor` (float, passed at runtime)
+- Output: `x * factor` (element-wise), allocated internally
+- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
+
+## When to use JIT vs AOT (`sgl-kernel`)
+
+- **JIT (`jit_kernel`)**: prefer this first for kernels that do **not** depend on CUTLASS or another large C++ project. It is the default choice for lightweight kernels that benefit from rapid iteration and first-use compilation.
+- **AOT (`sgl-kernel`)**: prefer this when the kernel **does** depend on CUTLASS or another large C++ project, or when it should live in `sgl-kernel/` and participate in the wheel build / torch op registration flow.
+- **Exception**: kernels that depend on `flashinfer`, or on CUTLASS that is already provided through `flashinfer`, can still be implemented as `jit_kernel`.
+
+---
+
+## Common Abstractions in `python/sglang/jit_kernel/include/sgl_kernel/`
+
+**Always prefer these abstractions over raw CUDA primitives.** They provide safety, readability, and consistency with the rest of the codebase.
+
+**Important include rule:** for every `#include <sgl_kernel/...>` line, add a short trailing comment explaining why that header is included (for example `// For TensorMatcher, SymbolicSize, SymbolicDevice`). This matches the current JIT kernel style and keeps include usage self-documenting.
+
+### `utils.h` — Host-side utilities
+
+```cpp
+#include <sgl_kernel/utils.h>
+```
+
+- **`host::RuntimeCheck(cond, args...)`** — Assert a condition at runtime; throws `PanicError` with file/line info on failure. Prefer this over bare `assert`.
+- **`host::Panic(args...)`** — Unconditionally throw a `PanicError` with a descriptive message.
+- **`host::div_ceil(a, b)`** — Integer ceiling division `(a + b - 1) / b`.
+- **`host::irange(n)`** / **`host::irange(start, end)`** — Range views for cleaner loops.
+- **`host::pointer::offset(ptr, offsets...)`** — Byte-safe pointer arithmetic on `void*`. Use this instead of raw casts.
+
+### `utils.cuh` — Device-side utilities + `LaunchKernel`
+
+```cpp
+#include <sgl_kernel/utils.cuh>
+```
+
+- **Type aliases**: `fp16_t`, `bf16_t`, `fp32_t`, `fp8_e4m3_t`, `fp8_e5m2_t` and their packed variants `fp16x2_t`, `bf16x2_t`, `fp32x2_t`, etc.
+- **`SGL_DEVICE`** — Expands to `__forceinline__ __device__`. Use on all device functions.
+- **`device::kWarpThreads`** — Constant `32`.
+- **`device::load_as<T>(ptr, offset)`** / **`device::store_as<T>(ptr, val, offset)`** — Type-safe loads/stores from `void*`.
+- **`device::pointer::offset(ptr, offsets...)`** — Pointer arithmetic on device.
+- **`host::LaunchKernel(grid, block, device_or_stream [, smem])`** — RAII kernel launcher that:
+  - Resolves the CUDA stream from a `DLDevice` via TVM-FFI automatically.
+  - Checks the CUDA error with file/line info after launch via `operator()(kernel, args...)`.
+  - Supports `.enable_pdl(bool)` for PDL (Programmatic Dependent Launch, SM90+).
+- **`host::RuntimeDeviceCheck(cudaError_t)`** — Check a CUDA error; throw on failure.
+
+### `tensor.h` — Tensor validation (`TensorMatcher`, Symbolic types)
+
+```cpp
+#include <sgl_kernel/tensor.h>
+```
+
+This is the **primary validation API** for all kernel launchers. Use it to validate every `tvm::ffi::TensorView` argument.
+
+- **`host::SymbolicSize{"name"}`** — A named symbolic dimension. Call `.set_value(n)` to pin it, `.unwrap()` to extract after verification.
+- **`host::SymbolicDType`** — Symbolic dtype. Use `.set_options<Ts...>()` to restrict allowed types.
+- **`host::SymbolicDevice`** — Symbolic device. Use `.set_options<kDLCUDA>()` to restrict to CUDA.
+- **`host::TensorMatcher({dims...})`** — Fluent builder for tensor validation:
+  - `.with_dtype<T>()` — require a specific C++ type (e.g. `fp16_t`)
+  - `.with_dtype<T1, T2, ...>()` — allow a set of types
+  - `.with_device<kDLCUDA>(device_sym)` — require CUDA and bind the checked device to a `SymbolicDevice`
+  - `.with_strides({strides...})` — validate strides (omit to require contiguous)
+  - `.verify(tensor_view)` — execute the check; throws `PanicError` with full context on failure; **chainable** (`verify(a).verify(b)` to check multiple tensors with the same shape)
+
+**Typical pattern:**
+```cpp
+auto N = SymbolicSize{"num_elements"};
+auto device = SymbolicDevice{};
+device.set_options<kDLCUDA>();
+TensorMatcher({N})  //
+    .with_dtype<fp16_t>()
+    .with_device<kDLCUDA>(device)
+    .verify(dst)
+    .verify(src);  // same shape, dtype, device as dst
+const size_t n = N.unwrap();
+const DLDevice dev = device.unwrap();
+```
+
+### `type.cuh` — `dtype_trait<T>` and `packed_t<T>`
+
+```cpp
+#include <sgl_kernel/type.cuh>
+```
+
+- **`dtype_trait<T>`** — Static trait struct for each scalar type. Provides:
+  - `dtype_trait<T>::from(value)` — convert from another type (e.g. `fp32_t` → `fp16_t`)
+  - `dtype_trait<T>::abs/sqrt/rsqrt/exp/sin/cos(x)` — type-dispatched unary math (primarily for `fp32_t`)
+  - `dtype_trait<T>::max/min(x, y)` — type-dispatched binary math (primarily for `fp32_t`)
+- **`packed_t<T>`** — Two-element packed alias: `packed_t<fp16_t>` = `fp16x2_t`, `packed_t<bf16_t>` = `bf16x2_t`, `packed_t<fp32_t>` = `fp32x2_t`. Use for vectorized loads/stores.
+- **`device::cast<To, From>(value)`** — Type-safe cast using `dtype_trait`, e.g. `cast<fp32x2_t, fp16x2_t>(v)`.
+
+### `vec.cuh` — Vectorized memory access (`AlignedVector`)
+
+```cpp
+#include <sgl_kernel/vec.cuh>
+```
+
+- **`device::AlignedVector<T, N>`** — Aligned storage for N elements of type T. N must be a power of two, `sizeof(T)*N <= 32`. Enables vectorized loads/stores for bandwidth efficiency. In terms of API/codegen constraints, the upper bound is 256-bit; in practice, 128-bit is the portable default, while 256-bit vectorization is typically only viable on `SM100+` and should be gated by an architecture check when needed.
+  - `.load(ptr, offset)` — vectorized load from `ptr[offset]`
+  - `.store(ptr, offset)` — vectorized store to `ptr[offset]`
+  - `.fill(value)` — fill all lanes
+  - `operator[](i)` — element access
+
+### `tile.cuh` — `tile::Memory` (strided memory access pattern)
+
+```cpp
+#include <sgl_kernel/tile.cuh>
+```
+
+- `tile::Memory<T>` is fundamentally a **1D cooperative accessor** over a contiguous region.
+- **`device::tile::Memory<T>::cta(blockDim.x)`** — Creates a tile accessor where each thread handles `tid = threadIdx.x` with stride `tsize` (for `cta(blockDim.x)`, this is `blockDim.x`). Common for loops over a 1D array.
+- **`.load(ptr, offset)`** — loads `ptr[tid + offset * tsize]`
+- **`.store(ptr, val, offset)`** — stores to `ptr[tid + offset * tsize]`
+- **`.in_bound(n, offset)`** — boundary check
+
+For a **2D tile**, either flatten `(row, col)` into a linear tile index first, or compute the address manually with `ptr[row * stride + col]` using your thread/block coordinates.
+
+### `math.cuh` — Device math (`device::math::`)
+
+```cpp
+#include <sgl_kernel/math.cuh>
+```
+
+- `device::math::max/min<T>(a, b)` — type-dispatched binary math via `dtype_trait`
+- `device::math::abs/sqrt/rsqrt/exp/sin/cos<T>(x)` — type-dispatched unary math via `dtype_trait`
+
+### `warp.cuh` — Warp-level primitives
+
+```cpp
+#include <sgl_kernel/warp.cuh>
+```
+
+- `device::warp::reduce_sum<T>(value)` — warp-level sum reduction via `__shfl_xor_sync`
+- `device::warp::reduce_max<T>(value)` — warp-level max reduction
+
+### `cta.cuh` — CTA-level primitives
+
+```cpp
+#include <sgl_kernel/cta.cuh>
+```
+
+- `device::cta::reduce_max<T>(value, smem, min_value)` — CTA-wide max using shared memory + warp reduction. Caller is responsible for a `__syncthreads()` after if the result in `smem[0]` is needed.
+
+### `atomic.cuh` — Atomic operations
+
+```cpp
+#include <sgl_kernel/atomic.cuh>
+```
+
+- `device::atomic::max(float* addr, float value)` — float atomic max (handles negative values correctly via bit tricks).
+
+### `runtime.cuh` — Occupancy and device info
+
+```cpp
+#include <sgl_kernel/runtime.cuh>
+```
+
+- `host::runtime::get_blocks_per_sm(kernel, block_dim)` — max active blocks per SM (occupancy)
+- `host::runtime::get_sm_count(device_id)` — number of SMs on the device
+- `host::runtime::get_cc_major(device_id)` — compute capability major version
+
+**Persistent kernel pattern** (cap blocks to SM count × occupancy):
+```cpp
+static const uint32_t max_occ = runtime::get_blocks_per_sm(kernel, kBlockSize);
+static const uint32_t num_sm  = runtime::get_sm_count(device.unwrap().device_id);
+const auto num_blocks = std::min(num_sm * max_occ, div_ceil(n, kBlockSize));
+LaunchKernel(num_blocks, kBlockSize, device.unwrap())(kernel, params);
+```
+
+---
+
+## Step 0 (optional): Generate a `.clangd` config for better IDE support
+
+```bash
+python -m sglang.jit_kernel
+```
+
+---
+
+## Step 1: Implement the CUDA kernel in `jit_kernel/csrc/`
+
+Create `python/sglang/jit_kernel/csrc/elementwise/scale.cuh`.
+
+The implementation fully uses the project abstractions described above:
+
+```cpp
+#include <sgl_kernel/tensor.h>   // For TensorMatcher, SymbolicSize, SymbolicDevice
+#include <sgl_kernel/type.cuh>   // For dtype_trait, fp16_t, bf16_t, fp32_t
+#include <sgl_kernel/utils.h>    // For RuntimeCheck, div_ceil
+#include <sgl_kernel/utils.cuh>  // For LaunchKernel, SGL_DEVICE
+#include <sgl_kernel/vec.cuh>    // For AlignedVector
+
+#include <dlpack/dlpack.h>
+#include <tvm/ffi/container/tensor.h>
+
+namespace {
+
+// ----------------------------------------------------------------
+// Kernel: element-wise scale using vectorized 128-bit loads/stores
+// T       = fp16_t | bf16_t | fp32_t
+// kVecN   = number of elements per vector load (e.g. 8 for fp16)
+// factor  = runtime scale factor
+// ----------------------------------------------------------------
+template <typename T, int kVecN>
+__global__ void scale_kernel(T* __restrict__ dst,
+                              const T* __restrict__ src,
+                              float factor,
+                              uint32_t n_total) {
+  using vec_t = device::AlignedVector<T, kVecN>;
+  const uint32_t n_vecs = n_total / kVecN;
+
+  // --- vectorised body ---
+  const uint32_t vec_stride = blockDim.x * gridDim.x;
+  for (uint32_t vi = blockIdx.x * blockDim.x + threadIdx.x;
+       vi < n_vecs;
+       vi += vec_stride) {
+    vec_t v;
+    v.load(src, vi);
+#pragma unroll
+    for (int i = 0; i < kVecN; ++i) {
+      v[i] = static_cast<T>(static_cast<float>(v[i]) * factor);
+    }
+    v.store(dst, vi);
+  }
+
+  // --- scalar tail ---
+  const uint32_t base = n_vecs * kVecN;
+  const uint32_t scalar_stride = blockDim.x * gridDim.x;
+  for (uint32_t i = blockIdx.x * blockDim.x + threadIdx.x;
+       base + i < n_total;
+       i += scalar_stride) {
+    dst[base + i] = static_cast<T>(static_cast<float>(src[base + i]) * factor);
+  }
+}
+
+// ----------------------------------------------------------------
+// Launcher: validates tensors, selects vector width, launches kernel
+// ----------------------------------------------------------------
+template <typename T>
+void scale(tvm::ffi::TensorView dst, tvm::ffi::TensorView src, float factor) {
+  using namespace host;
+
+  // 1. Validate input tensors with TensorMatcher
+  SymbolicSize N = {"num_elements"};
+  SymbolicDevice device_;
+  device_.set_options<kDLCUDA>();
+
+  TensorMatcher({N})  //
+      .with_dtype<T>()
+      .with_device<kDLCUDA>(device_)
+      .verify(dst)
+      .verify(src);  // same shape / dtype / device as dst
+
+  const uint32_t n = static_cast<uint32_t>(N.unwrap());
+  const DLDevice device = device_.unwrap();
+
+  RuntimeCheck(n > 0, "scale: num_elements must be > 0, got ", n);
+
+  // 2. Choose vector width for 128-bit loads (16 bytes)
+  //    fp16/bf16: 8 elements × 2 bytes = 16 bytes
+  //    fp32:      4 elements × 4 bytes = 16 bytes
+  constexpr int kVecN = 16 / sizeof(T);
+  const uint32_t n_work_items = div_ceil(n, static_cast<uint32_t>(kVecN));
+
+  // 3. Launch
+  constexpr uint32_t kBlockSize = 256;
+  const uint32_t grid = div_ceil(n_work_items, kBlockSize);
+
+  LaunchKernel(grid, kBlockSize, device)(
+      scale_kernel<T, kVecN>,
+      static_cast<T*>(dst.data_ptr()),
+      static_cast<const T*>(src.data_ptr()),
+      factor,
+      n);
+}
+
+}  // namespace
+```
+
+**Key points:**
+
+- Include headers from `sgl_kernel/` — **not** raw CUDA headers for anything already covered
+- Add a short trailing `// For ...` explanation to every `#include <sgl_kernel/...>` line
+- Use `TensorMatcher` for all tensor validation; never manually check shape/dtype/device
+- Use `AlignedVector` for vectorised 128-bit loads/stores — significant bandwidth win
+- Use `LaunchKernel` — it resolves the stream and checks errors automatically
+- Use `RuntimeCheck` for runtime assertions with useful error messages
+- Prefer passing runtime scalars like `factor` directly unless compile-time specialisation is genuinely required
+- `fp16_t` / `bf16_t` / `fp32_t` are the project's type aliases (from `utils.cuh`)
+- `device::cast<To, From>` or `dtype_trait<T>::from(val)` for cross-type conversions
+- `device::math::` functions for device math instead of bare `__` intrinsics
+
+---
+
+## Step 2: Add the Python wrapper in `jit_kernel/`
+
+Create `python/sglang/jit_kernel/scale.py`:
+
+```python
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import torch
+
+from sglang.jit_kernel.utils import cache_once, load_jit, make_cpp_args
+
+if TYPE_CHECKING:
+    from tvm_ffi.module import Module
+
+
+@cache_once
+def _jit_scale_module(dtype: torch.dtype) -> Module:
+    """Compile and cache the JIT scale module for a given dtype."""
+    args = make_cpp_args(dtype)
+    return load_jit(
+        "scale",
+        *args,
+        cuda_files=["elementwise/scale.cuh"],
+        cuda_wrappers=[("scale", f"scale<{args}>")],
+    )
+
+
+def scale(src: torch.Tensor, factor: float, out: torch.Tensor | None = None) -> torch.Tensor:
+    """
+    Element-wise scale: dst = src * factor.
+
+    Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
+
+    Parameters
+    ----------
+    src    : CUDA tensor (FP16 / BF16 / FP32)
+    factor : scale factor
+    out    : optional pre-allocated output tensor (same shape/dtype as src)
+
+    Returns
+    -------
+    Scaled tensor (dst = src * factor).
+    """
+    if not src.is_cuda:
+        raise RuntimeError("src must be a CUDA tensor")
+    if src.dtype not in (torch.float16, torch.bfloat16, torch.float32):
+        raise RuntimeError(
+            f"Unsupported dtype {src.dtype}. Supported: float16, bfloat16, float32"
+        )
+    if out is None:
+        out = torch.empty_like(src)
+    else:
+        if out.shape != src.shape:
+            raise RuntimeError("out shape must match src")
+        if out.dtype != src.dtype:
+            raise RuntimeError("out dtype must match src")
+        if out.device != src.device:
+            raise RuntimeError("out device must match src")
+
+    # Keep the Python wrapper thin, but still enforce the basic preconditions
+    # that the current JIT/FFI path does not reject safely on its own.
+    module = _jit_scale_module(src.dtype)
+    module.scale(out, src, factor)
+    return out
+```
+
+**Key points:**
+
+- Use `cache_once` — **not** `functools.lru_cache` (incompatible with `torch.compile`)
+- `load_jit` first arg(s) form the unique build marker; same marker = same cached binary
+- Only include compile-time specialisation knobs in the build marker; runtime values like `factor` should stay runtime unless the kernel truly needs templating
+- `cuda_wrappers`: `(export_name, kernel_symbol)` — `export_name` is called from Python
+- `make_cpp_args(dtype, ...)` converts `torch.dtype` to C++ type alias:
+- Keep Python launchers thin, but still validate the basic invariants (`is_cuda`, supported dtype, `out` metadata). In the current JIT/FFI path, invalid tensors are not always rejected safely before launch
+
+| `torch.dtype`      | C++ type   |
+|--------------------|------------|
+| `torch.float16`    | `fp16_t`   |
+| `torch.bfloat16`   | `bf16_t`   |
+| `torch.float32`    | `fp32_t`   |
+
+---
+
+## Step 3 (optional): Tune JIT build flags
+
+```python
+return load_jit(
+    "scale",
+    *args,
+    cuda_files=["elementwise/scale.cuh"],
+    cuda_wrappers=[("scale", f"scale<{args}>")],
+    extra_cuda_cflags=["-O3", "--use_fast_math"],
+)
+```
+
+If your kernel requires SM90+, raise a clear Python error before calling `load_jit`:
+
+```python
+if torch.cuda.get_device_capability()[0] < 9:
+    raise RuntimeError("This kernel requires SM90 (Hopper) or later")
+```
+
+---
+
+## Step 4: Write tests (required)
+
+JIT kernel tests live under `python/sglang/jit_kernel/tests/`. **CI does not run `pytest` in that directory directly.** The unified runner `test/run_suite.py` discovers every `test_*.py` there (and every `bench_*.py` under `benchmark/`), collects `register_*_ci(...)` calls by **statically parsing each file’s AST**, and executes the selected suite. Every test file must register at least one CUDA entry or the collector fails its sanity check.
+
+- **PR / per-commit CUDA suites** (see `test/run_suite.py` → `PER_COMMIT_SUITES`): JIT unit tests use `stage-b-kernel-unit-1-gpu-large` (see `.github/workflows/pr-test-jit-kernel.yml`: `python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-large`).
+- **Nightly kernel suite**: `nightly-kernel-1-gpu` with `--nightly` — typically used with `SGLANG_JIT_KERNEL_RUN_FULL_TESTS=1` in CI for expanded parameter grids (see `python/sglang/jit_kernel/utils.py` → `should_run_full_tests` / `get_ci_test_range`). Wired in `.github/workflows/nightly-test-nvidia.yml` (e.g. `python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error`).
+
+Registration pattern (module level, **literal** `est_time` and `suite` strings — required for AST parsing):
+
+```python
+from sglang.test.ci.ci_register import register_cuda_ci
+
+register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-large")
+# Optional second registration: same file also listed under the nightly kernel suite
+# register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True)
+```
+
+Keep `est_time` and `suite` as literal values. `run_suite.py` collects them from the file AST, so computed values and helper wrappers can break CI discovery.
+
+Use `register_cuda_ci(..., disabled="reason")` if the file must stay in-tree but should be skipped in CI (e.g. multi-GPU only).
+
+**Run like CI** (from repo root):
+
+```bash
+cd test && python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-large
+```
+
+For fast iteration you can still run `pytest` on a single file locally; CI coverage is via `run_suite.py`.
+
+Create `python/sglang/jit_kernel/tests/test_scale.py`:
+
+```python
+import pytest
+import torch
+from sglang.jit_kernel.scale import scale
+from sglang.test.ci.ci_register import register_cuda_ci
+
+register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-large")
+
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
+@pytest.mark.parametrize("size", [1, 127, 128, 1024, 4097])  # cover tail remainder
+@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0, 3.0])
+def test_scale_correctness(dtype, size, factor):
+    src = torch.randn(size, dtype=dtype, device="cuda")
+    out = scale(src, factor)
+    expected = src * factor
+
+    rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
+    torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
+
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
+def test_scale_out_param(dtype):
+    src = torch.randn(1024, dtype=dtype, device="cuda")
+    out = torch.empty_like(src)
+    result = scale(src, 2.0, out=out)
+    assert result is out
+    torch.testing.assert_close(out, src * 2.0, rtol=1e-2, atol=1e-2)
+
+
+def test_scale_cpu_error():
+    src = torch.randn(128, dtype=torch.float16)  # CPU tensor
+    with pytest.raises(RuntimeError, match="CUDA"):
+        scale(src, 2.0)
+
+
+def test_scale_unsupported_dtype():
+    src = torch.randint(0, 10, (128,), dtype=torch.int32, device="cuda")
+    with pytest.raises(RuntimeError, match="dtype"):
+        scale(src, 2.0)
+
+
+if __name__ == "__main__":
+    import sys
+    sys.exit(pytest.main([__file__, "-v", "-s"]))
+```
+
+---
+
+## Step 5: Add a benchmark (required)
+
+Benchmarks are `bench_*.py` files under `python/sglang/jit_kernel/benchmark/`. They are picked up by the same `run_suite.py` machinery as unit tests. Register them for **`stage-b-kernel-benchmark-1-gpu-large`** (PR JIT benchmark job: `python3 run_suite.py --hw cuda --suite stage-b-kernel-benchmark-1-gpu-large`).
+
+Create `python/sglang/jit_kernel/benchmark/bench_scale.py`:
+
+```python
+import itertools
+
+import torch
+import triton
+import triton.testing
+
+from sglang.jit_kernel.benchmark.utils import (
+    DEFAULT_DEVICE,
+    DEFAULT_DTYPE,
+    get_benchmark_range,
+    run_benchmark,
+)
+from sglang.jit_kernel.scale import scale as jit_scale
+from sglang.test.ci.ci_register import register_cuda_ci
+
+register_cuda_ci(est_time=6, suite="stage-b-kernel-benchmark-1-gpu-large")
+
+SIZE_LIST = get_benchmark_range(
+    full_range=[2**n for n in range(10, 20)],  # 1K … 512K elements
+    ci_range=[4096, 65536],
+)
+
+configs = list(itertools.product(SIZE_LIST))
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["size"],
+        x_vals=configs,
+        line_arg="provider",
+        line_vals=["jit", "torch"],
+        line_names=["SGL JIT Kernel", "PyTorch"],
+        styles=[("blue", "-"), ("red", "--")],
+        ylabel="us",
+        plot_name="scale-performance",
+        args={},
+    )
+)
+def benchmark(size: int, provider: str):
+    src = torch.randn(size, dtype=DEFAULT_DTYPE, device=DEFAULT_DEVICE)
+    factor = 2.0
+
+    if provider == "jit":
+        fn = lambda: jit_scale(src, factor)
+    else:
+        fn = lambda: src * factor
+
+    return run_benchmark(fn)
+
+
+if __name__ == "__main__":
+    benchmark.run(print_data=True)
+```
+
+Run locally:
+
+```bash
+python python/sglang/jit_kernel/benchmark/bench_scale.py
+```
+
+Run the benchmark suite the way CI does:
+
+```bash
+cd test && python3 run_suite.py --hw cuda --suite stage-b-kernel-benchmark-1-gpu-large
+```
+
+---
+
+## Troubleshooting
+
+- **`No CI registry found in ...` from `run_suite.py`**: add a module-level `register_cuda_ci(...)` with literal `est_time` and `suite` (and optional `nightly=True`); starred args and non-literal values break AST collection
+- **JIT compilation fails**: ensure the `.cuh` file is under `python/sglang/jit_kernel/csrc/`; reduce template argument combinations
+- **CUDA crash / illegal memory access**: `CUDA_LAUNCH_BLOCKING=1`; `compute-sanitizer --tool memcheck python ...`
+- **Unstable benchmark results**: `run_benchmark` uses CUDA-graph-based timing by default
+
+---
+
+## References
+
+- `docs/developer_guide/development_jit_kernel_guide.md`
+- `test/run_suite.py` — suite names, discovery of `jit_kernel/tests/` and `jit_kernel/benchmark/`, execution entrypoint for CI
+- `python/sglang/test/ci/ci_register.py` — `register_cuda_ci` and AST registration rules
+- `python/sglang/jit_kernel/utils.py` — `cache_once`, `load_jit`, `make_cpp_args`, `should_run_full_tests`, `get_ci_test_range`
+- `python/sglang/jit_kernel/include/sgl_kernel/tensor.h` — `TensorMatcher`, `SymbolicSize/DType/Device`
+- `python/sglang/jit_kernel/include/sgl_kernel/utils.cuh` — type aliases, `LaunchKernel`, `SGL_DEVICE`
+- `python/sglang/jit_kernel/include/sgl_kernel/vec.cuh` — `AlignedVector`
+- `python/sglang/jit_kernel/include/sgl_kernel/tile.cuh` — `tile::Memory`
+- `python/sglang/jit_kernel/include/sgl_kernel/type.cuh` — `dtype_trait`, `packed_t`, `device::cast`
+- `python/sglang/jit_kernel/include/sgl_kernel/math.cuh` — `device::math::`
+- `python/sglang/jit_kernel/include/sgl_kernel/warp.cuh` — `warp::reduce_sum/max`
+- `python/sglang/jit_kernel/include/sgl_kernel/cta.cuh` — `cta::reduce_max`
+- `python/sglang/jit_kernel/include/sgl_kernel/atomic.cuh` — `atomic::max`
+- `python/sglang/jit_kernel/include/sgl_kernel/runtime.cuh` — occupancy / SM count helpers
+- `python/sglang/jit_kernel/csrc/add_constant.cuh` — minimal runnable reference
+- `python/sglang/jit_kernel/csrc/elementwise/rmsnorm.cuh` — real example using `TensorMatcher` + `LaunchKernel` + `tile::Memory`
+- `python/sglang/jit_kernel/csrc/elementwise/qknorm.cuh` — real example using `runtime::get_blocks_per_sm` + persistent kernel pattern
+- `python/sglang/jit_kernel/benchmark/utils.py` — benchmark helpers
+
+## Summary of Files Created
+
+```
+python/sglang/jit_kernel/csrc/elementwise/scale.cuh   # NEW: CUDA kernel
+python/sglang/jit_kernel/scale.py                     # NEW: Python wrapper
+python/sglang/jit_kernel/tests/test_scale.py          # NEW: Tests
+python/sglang/jit_kernel/benchmark/bench_scale.py     # NEW: Benchmark
+```

third_party/sglang/.claude/skills/add-sgl-kernel/SKILL.md

@@ -0,0 +1,363 @@
+---
+name: add-sgl-kernel
+description: Step-by-step tutorial for adding a heavyweight AOT CUDA/C++ kernel to sgl-kernel (including tests & benchmarks)
+---
+
+# Tutorial: Adding a New Kernel to `sgl-kernel` (AOT / Heavyweight)
+
+This tutorial walks through adding a simple element-wise scale operation as an AOT kernel. We'll implement `scale(x, factor) = x * factor` to demonstrate the complete workflow.
+
+## Goal
+
+Add a new operation that scales each element of a tensor by a scalar factor:
+
+- Input: tensor `x` (CUDA) and scalar `factor` (float)
+- Output: `x * factor` (element-wise, in-place or into pre-allocated `out`)
+- Supported dtypes: **FP16 (`torch.float16`), BF16 (`torch.bfloat16`), FP32 (`torch.float32`)**
+  - Dispatched via `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro (defined in `sgl-kernel/include/utils.h`)
+
+## Two rules of thumb (must follow)
+
+1. **Prefer `python/sglang/jit_kernel` first** when the kernel does **not** depend on CUTLASS or another large C++ project. This is the default path for lightweight kernels that benefit from rapid iteration.
+2. **Prefer `sgl-kernel`** when the kernel **does** depend on CUTLASS or another large C++ project, or when it should be part of the AOT wheel / torch op registration flow.
+3. **Exception**: if the dependency is `flashinfer`, or CUTLASS that is already provided through `flashinfer`, the kernel can still be implemented as `jit_kernel`.
+
+In addition, every new kernel must ship with:
+
+- **Tests** (pytest)
+- **A benchmark script** (triton.testing)
+
+---
+
+## Repository integration map
+
+You will typically touch these files/areas:
+
+- Implementation: `sgl-kernel/csrc/elementwise/scale.cu` (pick the right subdirectory)
+- Public declarations: `sgl-kernel/include/sgl_kernel_ops.h`
+- Torch extension registration: `sgl-kernel/csrc/common_extension.cc`
+- Build: `sgl-kernel/CMakeLists.txt` (`set(SOURCES ...)`)
+- Python API: `sgl-kernel/python/sgl_kernel/` and `sgl-kernel/python/sgl_kernel/__init__.py`
+- Tests: `sgl-kernel/tests/test_scale.py`
+- Benchmarks: `sgl-kernel/benchmark/bench_scale.py`
+
+---
+
+## Step 1: Implement the kernel in `csrc/`
+
+Pick the right subdirectory:
+
+- `csrc/elementwise/` — for element-wise ops (our example)
+- `csrc/gemm/`, `csrc/attention/`, `csrc/moe/` — for other categories
+
+Create `sgl-kernel/csrc/elementwise/scale.cu`:
+
+```cpp
+#include <ATen/cuda/CUDAContext.h>
+#include <c10/cuda/CUDAGuard.h>
+#include <torch/all.h>
+
+#include "utils.h"  // DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16
+
+// scale_kernel: out[i] = input[i] * factor
+// Supports float, half (__half), __nv_bfloat16 via template T
+template <typename T>
+__global__ void scale_kernel(T* __restrict__ out,
+                              const T* __restrict__ input,
+                              float factor,
+                              int64_t n) {
+  int64_t idx = static_cast<int64_t>(blockIdx.x) * blockDim.x + threadIdx.x;
+  if (idx < n) {
+    out[idx] = static_cast<T>(static_cast<float>(input[idx]) * factor);
+  }
+}
+
+void scale(at::Tensor& out, const at::Tensor& input, double factor) {
+  TORCH_CHECK(input.is_cuda(),       "input must be a CUDA tensor");
+  TORCH_CHECK(input.is_contiguous(), "input must be contiguous");
+  TORCH_CHECK(out.is_cuda(),         "out must be a CUDA tensor");
+  TORCH_CHECK(out.is_contiguous(),   "out must be contiguous");
+  TORCH_CHECK(out.sizes() == input.sizes(),  "out and input must have the same shape");
+  TORCH_CHECK(out.scalar_type() == input.scalar_type(),
+              "out and input must have the same dtype");
+
+  const int64_t n = input.numel();
+  const int threads = 256;
+  const int blocks  = (n + threads - 1) / threads;
+
+  const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
+  const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
+
+  // Dispatches over float, float16, bfloat16
+  DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16(input.scalar_type(), c_type, [&] {
+    scale_kernel<c_type><<<blocks, threads, 0, stream>>>(
+        static_cast<c_type*>(out.data_ptr()),
+        static_cast<const c_type*>(input.data_ptr()),
+        static_cast<float>(factor),
+        n);
+    cudaError_t status = cudaGetLastError();
+    TORCH_CHECK(status == cudaSuccess,
+                "scale_kernel launch failed: ", cudaGetErrorString(status));
+    return true;
+  });
+}
+```
+
+**Key points:**
+
+- Use `at::Tensor` (PyTorch tensors), `TORCH_CHECK` for validation, `at::cuda::getCurrentCUDAStream()` for stream
+- Keep Python wrappers thin; do shape/dtype/device validation in C++ right around the launch path
+- `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` covers `float`, `half` (FP16), `__nv_bfloat16` (BF16)
+- Add device error checking after every kernel launch
+- If a kernel only works on certain architectures, enforce that with `TORCH_CHECK` and skip logic in tests
+
+---
+
+## Step 2: Add a C++ declaration in `include/sgl_kernel_ops.h`
+
+Edit `sgl-kernel/include/sgl_kernel_ops.h`, add to the elementwise section:
+
+```cpp
+void scale(at::Tensor& out, const at::Tensor& input, double factor);
+```
+
+---
+
+## Step 3: Register the op in `csrc/common_extension.cc`
+
+Edit `sgl-kernel/csrc/common_extension.cc`, inside `TORCH_LIBRARY_FRAGMENT(sgl_kernel, m)`:
+
+```cpp
+// From csrc/elementwise
+m.def("scale(Tensor! out, Tensor input, float factor) -> ()");
+m.impl("scale", torch::kCUDA, &scale);
+```
+
+**Key points:**
+
+- `Tensor!` means in-place / mutable output argument
+- The schema is important for `torch.compile` and for consistent call signatures
+- Keep the torch schema in PyTorch scalar types (`float` here), but note that the C++ launcher signature still needs `double` for scalar arguments accepted by `torch::Library`
+
+---
+
+## Step 4: Add the new source file to `CMakeLists.txt`
+
+Edit `sgl-kernel/CMakeLists.txt`, add to `set(SOURCES ...)`:
+
+```cmake
+csrc/elementwise/scale.cu
+```
+
+**Key points:**
+
+- Keep the list **alphabetically sorted** (the file explicitly requires this)
+- If the kernel has arch constraints, reflect that in tests/benchmarks via skip logic
+
+---
+
+## Step 5: Expose a Python API under `sgl-kernel/python/sgl_kernel/`
+
+Prefer following the existing module organization first. For elementwise kernels, the usual pattern is:
+
+- implement the Python wrapper in `sgl-kernel/python/sgl_kernel/elementwise.py`
+- then re-export it from `sgl-kernel/python/sgl_kernel/__init__.py`
+
+For example, in `sgl-kernel/python/sgl_kernel/elementwise.py`, add:
+
+```python
+import torch
+
+def scale(
+    input: torch.Tensor,
+    factor: float,
+    out: torch.Tensor | None = None,
+) -> torch.Tensor:
+    """
+    Element-wise scale: out = input * factor.
+
+    Supported dtypes: torch.float16, torch.bfloat16, torch.float32.
+
+    Parameters
+    ----------
+    input  : CUDA input tensor
+    factor : scale factor (float)
+    out    : optional pre-allocated CUDA output tensor (same shape/dtype as input)
+    """
+    if out is None:
+        out = torch.empty_like(input)
+    torch.ops.sgl_kernel.scale.default(out, input, factor)
+    return out
+```
+
+Then re-export it from `sgl-kernel/python/sgl_kernel/__init__.py` following the existing import style used by other kernels.
+
+---
+
+## Step 6: Write tests (required)
+
+Create `sgl-kernel/tests/test_scale.py`:
+```python
+import pytest
+
+import torch
+import sgl_kernel
+
+@pytest.mark.parametrize("dtype", [torch.float16, torch.bfloat16, torch.float32])
+@pytest.mark.parametrize("size", [128, 1024, 4096, 65536])
+@pytest.mark.parametrize("factor", [0.5, 1.0, 2.0])
+def test_scale_correctness(dtype, size, factor):
+    input = torch.randn(size, dtype=dtype, device="cuda")
+    out   = torch.empty_like(input)
+
+    result = sgl_kernel.scale(input, factor, out=out)
+    assert result is out
+
+    expected = input * factor
+    rtol, atol = (1e-5, 1e-6) if dtype == torch.float32 else (1e-2, 1e-2)
+    torch.testing.assert_close(out, expected, rtol=rtol, atol=atol)
+
+
+def test_scale_shape_mismatch():
+    input = torch.randn(128, dtype=torch.float16, device="cuda")
+    out   = torch.empty(256, dtype=torch.float16, device="cuda")
+    with pytest.raises(RuntimeError, match="same shape"):
+        sgl_kernel.scale(input, 2.0, out=out)
+
+
+def test_scale_cpu_input():
+    input = torch.randn(128, dtype=torch.float16)  # CPU
+    out   = torch.empty_like(input)
+    with pytest.raises(RuntimeError, match="CUDA"):
+        sgl_kernel.scale(input, 2.0, out=out)
+
+
+if __name__ == "__main__":
+    import sys
+    sys.exit(pytest.main([__file__, "-q"]))
+```
+
+---
+
+## Step 7: Add a benchmark (required)
+
+Create `sgl-kernel/benchmark/bench_scale.py`:
+
+```python
+import itertools
+
+import torch
+import triton
+import triton.testing
+
+import sgl_kernel
+from sglang.utils import is_in_ci
+
+IS_CI = is_in_ci()
+
+dtypes  = [torch.float16] if IS_CI else [torch.float16, torch.bfloat16, torch.float32]
+sizes   = [4096] if IS_CI else [2**n for n in range(10, 20)]  # 1K … 512K
+factors = [2.0]
+
+configs = list(itertools.product(dtypes, sizes))
+
+
+def torch_scale(input: torch.Tensor, factor: float) -> torch.Tensor:
+    return input * factor
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["dtype", "size"],
+        x_vals=configs,
+        line_arg="provider",
+        line_vals=["sglang", "torch"],
+        line_names=["SGL Kernel", "PyTorch"],
+        styles=[("green", "-"), ("red", "--")],
+        ylabel="µs (median)",
+        plot_name="scale-performance",
+        args={},
+    )
+)
+def benchmark(dtype, size, provider):
+    input  = torch.randn(size, dtype=dtype, device="cuda")
+    out    = torch.empty_like(input)
+    factor = 2.0
+
+    if provider == "sglang":
+        fn = lambda: sgl_kernel.scale(input, factor, out=out)
+    else:
+        fn = lambda: torch_scale(input, factor)
+
+    ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
+        fn, quantiles=[0.5, 0.2, 0.8]
+    )
+    return 1000 * ms, 1000 * max_ms, 1000 * min_ms
+
+
+if __name__ == "__main__":
+    benchmark.run(print_data=True)
+```
+
+---
+
+## Step 8: Build
+
+Build:
+
+```bash
+cd sgl-kernel
+make build -j16
+```
+
+If you need to limit host resource usage:
+
+```bash
+cd sgl-kernel
+make build -j1 MAX_JOBS=2 CMAKE_ARGS="-DSGL_KERNEL_COMPILE_THREADS=1"
+```
+
+---
+
+## Step 9: Validate
+
+After building successfully, run the test and benchmark:
+
+```bash
+pytest sgl-kernel/tests/test_scale.py -q
+python sgl-kernel/benchmark/bench_scale.py
+```
+
+---
+
+## Troubleshooting
+
+- **Async CUDA errors**: `CUDA_LAUNCH_BLOCKING=1`
+- **Memory errors**: `compute-sanitizer --tool memcheck python ...`
+- **Build is too slow / OOM**: reduce `MAX_JOBS` and `SGL_KERNEL_COMPILE_THREADS`
+- **Binary bloat**: use `sgl-kernel/analyze_whl_kernel_sizes.py`
+- **CMake sources list**: if your `.cu` file is missing from `SOURCES`, the symbol will be undefined at link time
+
+---
+
+## References
+
+- `sgl-kernel/README.md`
+- `sgl-kernel/include/sgl_kernel_ops.h`
+- `sgl-kernel/csrc/common_extension.cc`
+- `sgl-kernel/CMakeLists.txt`
+- `sgl-kernel/include/utils.h` — `DISPATCH_PYTORCH_DTYPE_TO_CTYPE_FLOAT_FP16` macro and friends
+- `sgl-kernel/csrc/elementwise/activation.cu` — reference for the FP16/BF16/FP32 dispatch pattern
+
+## Summary of Files Created/Modified
+
+```
+sgl-kernel/csrc/elementwise/scale.cu          # NEW: CUDA kernel + launcher
+sgl-kernel/include/sgl_kernel_ops.h           # MODIFIED: C++ declaration
+sgl-kernel/csrc/common_extension.cc           # MODIFIED: schema + dispatch registration
+sgl-kernel/CMakeLists.txt                     # MODIFIED: add source file (alphabetical)
+sgl-kernel/python/sgl_kernel/elementwise.py   # MODIFIED: Python wrapper
+sgl-kernel/python/sgl_kernel/__init__.py      # MODIFIED: re-export Python API
+sgl-kernel/tests/test_scale.py                # NEW: tests
+sgl-kernel/benchmark/bench_scale.py           # NEW: benchmark
+```

third_party/sglang/.claude/skills/ci-workflow-guide/SKILL.md

@@ -0,0 +1,386 @@
+---
+name: ci-workflow-guide
+description: Guide to SGLang CI workflow orchestration — stage ordering, fast-fail, gating, partitioning, execution modes, and debugging CI failures. Use when modifying CI workflows, adding stages, debugging CI pipeline issues, or understanding how tests are dispatched and gated across stages.
+---
+
+# SGLang CI Workflow Orchestration Guide
+
+This skill covers the CI **infrastructure** layer — how tests are dispatched, gated, and fast-failed across stages. For test authoring (templates, fixtures, registration, model selection), see the [write-sglang-test skill](../write-sglang-test/SKILL.md).
+
+---
+
+## Naming Conventions
+
+- **Suite**: `stage-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `stage-b-test-1-gpu-small`)
+- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
+
+---
+
+## Key Files
+
+| File | Role |
+|------|------|
+| `.github/workflows/pr-test.yml` | Main workflow — all stages, jobs, conditions, matrix definitions |
+| `.github/workflows/pr-gate.yml` | PR gating: draft check, `run-ci` label, per-user rate limiting |
+| `.github/actions/check-stage-health/action.yml` | Cross-job fast-fail: queries API for any failed job |
+| `.github/actions/wait-for-jobs/action.yml` | Stage gating: polls API until stage jobs complete |
+| `.github/actions/check-maintenance/action.yml` | Maintenance mode check |
+| `test/run_suite.py` | Suite runner: collects, filters, partitions, executes tests |
+| `python/sglang/test/ci/ci_register.py` | Test registration (AST-parsed markers), LPT auto-partition |
+| `python/sglang/test/ci/ci_utils.py` | `run_unittest_files()`: execution, retry, continue-on-error |
+| `scripts/ci/utils/slash_command_handler.py` | Handles slash commands from PR comments |
+
+---
+
+## Architecture Overview
+
+```
+ ┌──────────────┐
+ │ build kernel │
+ └──────┬───────┘
+        │
+        ├─ check-changes ──── detects which packages changed
+        │                      (main_package, sgl_kernel, jit_kernel, multimodal_gen)
+        │
+        ├─ call-gate ──────── pr-gate.yml (draft? label? rate limit?)
+        │
+        ├─────────────────────────────────────────────────────┐
+        │                                                     │
+        ▼                                                     │
+ ┌─────────────────────────────────────┐                      │
+ │          Stage A (~3 min)           │                      │
+ │         pre-flight check            │                      │
+ │                                     │                      │
+ │  ┌─────────────────────────────┐    │                      │
+ │  │ stage-a-test-1-gpu-small    │    │                      │
+ │  │ (small GPUs)                │    │                      │
+ │  └─────────────────────────────┘    │                      │
+ │  ┌─────────────────────────────┐    │                      │
+ │  │ stage-a-test-cpu            │    │                      │
+ │  │ (CPU)                       │    │                      │
+ │  └─────────────────────────────┘    │                      │
+ └──────┬──────────────────────────────┘                      │
+        │                                                     │
+        ▼                                                     ▼
+ ┌─────────────────────────────────────┐          ┌──────────────────────────┐
+ │          Stage B (~30 min)          │          │      kernel test         │
+ │           basic tests               │          └──────────────────────────┘
+ │                                     │          ┌──────────────────────────┐
+ │  ┌─────────────────────────────┐    │          │   multimodal gen test    │
+ │  │ stage-b-test-1-gpu-small    │    │          └──────────────────────────┘
+ │  │ (small GPUs, e.g. 5090)     │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ stage-b-test-1-gpu-large    │    │
+ │  │ (large GPUs, e.g. H100)     │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ stage-b-test-2-gpu-large    │    │
+ │  │ (large GPUs, e.g. H100)     │    │
+ │  └─────────────────────────────┘    │
+ └──────┬──────────────────────────────┘
+        │
+        ▼
+ ┌─────────────────────────────────────┐
+ │          Stage C (~30 min)          │
+ │          advanced tests             │
+ │                                     │
+ │  ┌─────────────────────────────┐    │
+ │  │ stage-c-test-4-gpu-h100     │    │
+ │  │ (H100 GPUs)                 │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ stage-c-test-8-gpu-h200     │    │
+ │  │ (8 x H200 GPUs)             │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ stage-c-test-4-gpu-b200     │    │
+ │  │ (4 x B200 GPUs)             │    │
+ │  └─────────────────────────────┘    │
+ │  ┌─────────────────────────────┐    │
+ │  │ Other advanced tests        │    │
+ │  │ (DeepEP, PD Disagg, GB300)  │    │
+ │  └─────────────────────────────┘    │
+ └──────┬──────────────────────────────┘
+        │
+        ▼
+ ┌─────────────────────────────────────┐
+ │         pr-test-finish              │
+ │  aggregates all results, fails if   │
+ │  any job failed/cancelled           │
+ └─────────────────────────────────────┘
+```
+
+**Every stage test job** includes a `check-stage-health` step after checkout — if any job in the run has already failed, the job fast-fails (red X) with a root cause annotation.
+
+**Scheduled runs** skip `wait-for-stage-*` jobs, running all stages in parallel. Fast-fail is also disabled.
+
+---
+
+## Fast-Fail Layers
+
+4 layers of fast-fail, from fine to coarse:
+
+| Layer | Mechanism | Granularity | Disabled on schedule? |
+|-------|-----------|-------------|----------------------|
+| **1. Test method → file** | `unittest -f` (failfast) | One test method fails → entire test file stops immediately | Yes |
+| **2. File → suite** | `run_unittest_files()` default | One test file fails → entire suite stops (`--continue-on-error` off) | Yes |
+| **3. Job → job (same stage)** | `check-stage-health` action | One job fails → other waiting jobs in same stage fast-fail (red X) | Yes |
+| **4. Stage → stage (cross-stage)** | `wait-for-stage` + `needs` | Stage A fails → stage B/C jobs skip entirely (never get a runner) | Yes (wait jobs skipped) |
+
+- **Layer 1**: `-f` flag appended to all `python3 -m pytest` / `unittest` invocations in `ci_utils.py`
+- **Layer 2**: `--continue-on-error` flag in `run_suite.py` — off for PRs, on for scheduled runs
+- **Layer 3**: `check-stage-health` auto-detects `schedule` event and skips; filters out cascade failures to show only root cause jobs
+- **Layer 4**: `wait-for-stage-*` jobs are conditioned on `github.event_name == 'pull_request'` — skipped for scheduled runs
+
+---
+
+## Execution Modes
+
+| Aspect | PR (`pull_request`) | Scheduled (`cron`, every 6h) | `/rerun-stage` (`workflow_dispatch`) |
+|--------|---------------------|------------------------------|--------------------------------------|
+| **Stage ordering** | Sequential: A → B → C via `wait-for-stage-*` | Parallel (all at once) | Single target stage only |
+| **Cross-job fast-fail** | Yes (`check-stage-health`) | Yes | Yes |
+| **continue-on-error** | No (stop at first failure within suite) | Yes (run all tests) | No |
+| **Retry** | Enabled | Enabled | Enabled |
+| **max_parallel** | 3 (default), 14 if `high priority` label | 14 | 3 (default), 14 if `high priority` |
+| **PR gate** | Yes (draft, label, rate limit) | Skipped | Skipped |
+| **Concurrency** | `cancel-in-progress: true` per branch | Queue (no cancel) | Isolated per stage+SHA |
+
+---
+
+## Stage Gating (`wait-for-jobs` action)
+
+`wait-for-stage-a` and `wait-for-stage-b` are lightweight `ubuntu-latest` jobs that poll the GitHub Actions API.
+
+**How it works:**
+1. Calls `listJobsForWorkflowRun` to list all jobs in the current run
+2. Matches jobs by exact name or prefix (for matrix jobs, e.g., `stage-b-test-1-gpu-small (3)`)
+3. If any matched job has `conclusion === 'failure'` → fail immediately (fast-fail)
+4. If all matched jobs are completed and count matches `expected_count` → success
+5. Otherwise → sleep `poll-interval-seconds` (default: 60s) and retry
+6. Timeout after `max-wait-minutes` (240 min for stage-a, 480 min for stage-b)
+
+**Job specs example** (stage-b):
+```json
+[
+  {"prefix": "stage-b-test-1-gpu-small", "expected_count": 8},
+  {"prefix": "stage-b-test-1-gpu-large", "expected_count": 14},
+  {"prefix": "stage-b-test-2-gpu-large", "expected_count": 4},
+  {"prefix": "stage-b-test-4-gpu-b200", "expected_count": 1}
+]
+```
+
+> **Critical**: `expected_count` must match the matrix size. If you add/remove matrix entries, update the wait job's spec accordingly.
+
+**PR only**: Condition `github.event_name == 'pull_request' && !inputs.target_stage` — scheduled runs and `/rerun-stage` skip these entirely, allowing parallel execution.
+
+---
+
+## Cross-Job Fast-Fail (`check-stage-health` action)
+
+Composite action called after checkout in every stage test job (21 jobs total across `pr-test.yml`, `pr-test-multimodal-gen.yml`, `pr-test-sgl-kernel.yml`, `pr-test-jit-kernel.yml`).
+
+**How it works:**
+1. Queries `listJobsForWorkflowRun` for the current workflow run
+2. Filters for **root cause failures only** — jobs with `conclusion === 'failure'` whose failing step is NOT `check-stage-health` (excludes cascade failures)
+3. If root cause failures found → calls `core.setFailed()` with the list of root cause job names
+4. If none → does nothing (step succeeds)
+
+**Cascade filtering**: When job A fast-fails due to health check, it also has `conclusion: failure`. Without filtering, job B would list both the original failure AND job A's fast-fail. The filter checks each failed job's `steps` array — if the failing step name contains `check-stage-health` or `Check stage health`, it's excluded from the root cause list.
+
+**Usage pattern:**
+```yaml
+steps:
+  - name: Checkout code
+    uses: actions/checkout@v4
+    ...
+
+  - uses: ./.github/actions/check-stage-health
+    id: stage-health
+
+  - name: Install dependencies        # skipped automatically if health check failed
+    ...                                # (default if: success() is false)
+
+  - name: Run test                     # also skipped
+    ...
+```
+
+**Visual effect**: Job shows **red X** (failure) with error annotation showing root cause job names. Subsequent steps are naturally skipped (default `if: success()` is false after a failed step). No per-step `if` guards needed.
+
+**No stage filtering**: Checks ALL jobs in the run, not just the current stage. Any failure anywhere triggers fast-fail.
+
+**Error message example:**
+```
+Fast-fail: skipping — root cause job(s): stage-b-test-1-gpu-small (0), stage-b-test-1-gpu-small (1)
+```
+
+---
+
+## Within-Suite Failure Handling
+
+Controlled by `run_unittest_files()` in `python/sglang/test/ci/ci_utils.py`.
+
+### Flags
+
+| Flag | PR default | Scheduled default | Effect |
+|------|------------|-------------------|--------|
+| `--continue-on-error` | Off | On | Off: stop at first failure. On: run all files, report all failures at end |
+| `--enable-retry` | On | On | Retry retriable failures (accuracy/perf assertions) |
+| `--max-attempts` | 2 | 2 | Max attempts per file including initial run |
+
+### Retry Classification
+
+When a test fails and retry is enabled, the output is classified:
+
+**Non-retriable** (checked first — real code errors):
+`SyntaxError`, `ImportError`, `ModuleNotFoundError`, `NameError`, `TypeError`, `AttributeError`, `RuntimeError`, `CUDA out of memory`, `OOM`, `Segmentation fault`, `core dumped`, `ConnectionRefusedError`, `FileNotFoundError`
+
+**Retriable** (accuracy/performance):
+`AssertionError` with comparison patterns (`not greater than`, `not less than`, `not equal to`), `accuracy`, `score`, `latency`, `throughput`, `timeout`
+
+**Default**: Unknown `AssertionError` → retriable. Other unknown failures → not retriable.
+
+### How `continue_on_error` is set
+
+In `pr-test.yml`'s `check-changes` job:
+- `schedule` runs or `run_all_tests` flag → `continue_on_error = 'true'`
+- PR runs → `continue_on_error = 'false'`
+
+Each test job propagates via:
+```yaml
+env:
+  CONTINUE_ON_ERROR_FLAG: ${{ needs.check-changes.outputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
+run: |
+  python3 run_suite.py --hw cuda --suite <name> $CONTINUE_ON_ERROR_FLAG
+```
+
+---
+
+## Test Partitioning
+
+Large suites are split across matrix jobs using the **LPT (Longest Processing Time) heuristic** in `ci_register.py:auto_partition()`:
+
+1. Sort tests by `est_time` descending, filename as tie-breaker (deterministic)
+2. Greedily assign each test to the partition with smallest cumulative time
+3. Result: roughly equal total time per partition
+
+**Partition table** (CUDA per-commit suites):
+
+| Suite | Partitions | Runner | max_parallel |
+|-------|-----------|--------|-------------|
+| `stage-a-test-1-gpu-small` | 1 (no matrix) | `1-gpu-5090` | — |
+| `stage-a-test-cpu` | 1 (no matrix) | `ubuntu-latest` | — |
+| `stage-b-test-1-gpu-small` | 8 | `1-gpu-5090` | 8 |
+| `stage-b-test-1-gpu-large` | 14 | `1-gpu-h100` | dynamic (3 or 14) |
+| `stage-b-test-2-gpu-large` | 4 | `2-gpu-h100` | — |
+| `stage-b-test-4-gpu-b200` | 1 (no matrix) | `4-gpu-b200` | — |
+| `stage-b-kernel-unit-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
+| `stage-b-kernel-unit-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
+| `stage-b-kernel-benchmark-1-gpu-large` | 1 (no matrix) | `1-gpu-h100` | — |
+| `stage-c-test-4-gpu-h100` | 3 | `4-gpu-h100` | — |
+| `stage-c-test-8-gpu-h200` | 4 | `8-gpu-h200` | — |
+| `stage-c-test-8-gpu-h20` | 2 | `8-gpu-h20` | — |
+| `stage-c-test-deepep-4-gpu-h100` | 1 (no matrix) | `4-gpu-h100` | — |
+| `stage-c-test-deepep-8-gpu-h200` | 1 (no matrix) | `8-gpu-h200` | — |
+| `stage-c-test-4-gpu-b200` | 4 | `4-gpu-b200` | — |
+| `stage-c-test-4-gpu-gb200` | 1 (no matrix) | `4-gpu-gb200` | — |
+
+> **Note**: Kernel suites (`stage-b-kernel-*`) run via `pr-test-jit-kernel.yml` and `pr-test-sgl-kernel.yml`, not the main `pr-test.yml`. Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
+
+**Workflow usage:**
+```yaml
+strategy:
+  matrix:
+    partition: [0, 1, 2, 3, 4, 5, 6, 7]
+steps:
+  - run: python3 run_suite.py --hw cuda --suite stage-b-test-1-gpu-small \
+           --auto-partition-id ${{ matrix.partition }} --auto-partition-size 8
+```
+
+---
+
+## check-changes Job
+
+Determines which test suites to run based on file changes.
+
+### Detection Methods
+
+| Trigger | Method | Details |
+|---------|--------|---------|
+| `pull_request` | `dorny/paths-filter` | Detects changes via GitHub diff |
+| `workflow_dispatch` (with `pr_head_sha`) | GitHub API | `repos/{repo}/compare/main...{sha}` |
+| `schedule` / `run_all_tests` | Force all true | Runs everything |
+
+### Output Flags
+
+| Output | Triggers |
+|--------|----------|
+| `main_package` | Stage A/B/C test suites |
+| `sgl_kernel` | Kernel wheel builds + kernel test suites |
+| `jit_kernel` | JIT kernel test workflow |
+| `multimodal_gen` | Multimodal-gen test workflow |
+
+> **Note**: `sgl_kernel` is forced to `false` when `target_stage` is set, because `sgl-kernel-build-wheels` won't run and wheel artifacts won't be available.
+
+---
+
+## Concurrency Control
+
+```
+group: pr-test-{event_name}-{branch}-{pr_sha}-{stage}
+```
+
+| Segment | Source | Purpose |
+|---------|--------|---------|
+| `event_name` | `github.event_name` | Prevents scheduled runs colliding with fork PRs named `main` |
+| `branch` | `github.head_ref \|\| github.ref_name` | Per-branch isolation |
+| `pr_sha` | `inputs.pr_head_sha \|\| 'current'` | Isolates `/rerun-stage` from main runs |
+| `stage` | `inputs.target_stage \|\| 'all'` | Allows parallel stage dispatches |
+
+`cancel-in-progress: true` for `pull_request` events (new push cancels old run), `false` for `workflow_call`.
+
+---
+
+## How To: Add a New Stage Job
+
+1. Define the job in `pr-test.yml` with `needs: [check-changes, call-gate, wait-for-stage-X, ...]`
+2. Copy the `if:` condition pattern from an existing same-stage job (handles `target_stage`, `schedule`, `main_package`)
+3. Add `checkout` step
+4. Add `check-stage-health` step (after checkout) — if any prior job failed, `core.setFailed()` fires and all subsequent steps auto-skip via default `if: success()`
+5. Add `check-maintenance` step
+6. Add `download-artifact` step if `sgl_kernel` changed
+7. Add `install dependencies` step
+8. Add `run test` step with `$CONTINUE_ON_ERROR_FLAG`
+9. Add `upload-cuda-coredumps` step with `if: always()`
+10. Register the suite name in `PER_COMMIT_SUITES` in `test/run_suite.py`
+11. If using matrix, add `--auto-partition-id` and `--auto-partition-size` to the run command
+12. **Update `wait-for-stage-X`** job spec with the new job name and `expected_count` (if matrix)
+13. **Add the job to `pr-test-finish.needs`** list
+
+---
+
+## How To: Debug CI Failures
+
+| Symptom | Likely cause | What to check |
+|---------|-------------|---------------|
+| All stage-B/C jobs green but steps skipped | Earlier job failed, `check-stage-health` triggered | Find the actual failed job (red X) |
+| `wait-for-stage-b` timeout | `expected_count` doesn't match matrix size | Verify job spec counts match `matrix:` array length |
+| `pr-test-finish` fails but all jobs green | A job was `cancelled` (counts as failure in finish) | Check concurrency cancellation |
+| Tests pass locally but fail in CI | Partition assignment, runner GPU type, or `est_time` inaccuracy | Check which partition the test lands in; verify runner label |
+| Flaky test retried and passed | Retriable failure (accuracy/perf) | Check `[CI Retry]` markers in job logs |
+| Flaky test NOT retried | Matched non-retriable pattern | Check if error matches `NON_RETRIABLE_PATTERNS` in `ci_utils.py` |
+
+---
+
+## Slash Commands
+
+| Command | Effect |
+|---------|--------|
+| `/tag-run-ci-label` | Adds `run-ci` label to PR |
+| `/rerun-failed-ci` | Reruns failed jobs in the latest workflow run |
+| `/tag-and-rerun-ci` | Adds label + reruns |
+| `/rerun-stage <stage>` | Dispatches `pr-test.yml` with `target_stage=<stage>` |
+| `/rerun-test <test-file>` | Reruns a specific test file via `rerun-test.yml` |
+
+Handled by `scripts/ci/utils/slash_command_handler.py` → `.github/workflows/slash-command-handler.yml`.

third_party/sglang/.claude/skills/debug-cuda-crash/SKILL.md

@@ -0,0 +1,657 @@
+---
+name: debug-cuda-crash
+description: Call this skill when you need to debug CUDA crashes in SGLang using kernel API logging
+---
+
+# Tutorial: Debugging CUDA Crashes with Kernel API Logging
+
+This tutorial shows you how to debug CUDA crashes and errors in SGLang using the `@debug_kernel_api` logging decorator.
+
+## Goal
+
+When your code crashes with CUDA errors such as illegal memory access, device-side assert, out-of-bounds, or NaN/Inf, use kernel API logging to:
+- Capture input tensors BEFORE the crash occurs
+- Understand what data caused the problem
+- Track tensor shapes, dtypes, and values through the call boundary that triggered the crash
+- Detect numerical issues such as NaN, Inf, or obviously wrong shapes
+
+## Why Use Kernel API Logging?
+
+**Problem**: CUDA errors often crash the program before normal debugging output is flushed.
+
+**Solution**: SGLang's `@debug_kernel_api` decorator logs inputs before execution, so you can still see what caused the crash even after the program aborts.
+
+## What Is Covered?
+
+The current logging coverage focuses on the highest-value kernel boundaries in SGLang:
+- Custom ops registered through `register_custom_op(...)`
+- External custom ops registered through `register_custom_op_from_extern(...)`
+- LLM attention, linear, quantization, and multi-platform wrapper entry points
+- Diffusion attention impl, linear, rotary, and custom-op wrapper entry points
+- Selected direct `torch.ops.sglang.*` hotspots and model-specific bypasses
+
+This means the logging is useful for both LLM and diffusion kernel debugging, but it does not automatically cover every pure PyTorch call in the repository.
+
+## Step 1: Enable Kernel API Logging
+
+### Basic Logging (Function Names Only)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=1
+export SGLANG_KERNEL_API_LOGDEST=stdout
+
+python my_script.py
+```
+
+Output:
+```
+================================================================================
+[2026-03-19 00:47:06] SGLang Kernel API Call: RMSNorm.forward
+================================================================================
+[2026-03-19 00:47:06] SGLang Kernel API Call: sglang.quant_method.UnquantizedLinearMethod.apply
+================================================================================
+[2026-03-19 00:47:06] SGLang Kernel API Call: sglang.custom_op.fused_inplace_qknorm
+```
+
+This is a real level-1 excerpt captured from `Qwen/Qwen3-0.6B`.
+
+### Detailed Logging (Inputs with Metadata)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+python my_script.py
+```
+
+Output in `debug.log`:
+```
+================================================================================
+[2026-03-19 00:47:30] SGLang Kernel API Call: sglang.quant_method.UnquantizedLinearMethod.apply
+Positional input arguments:
+  arg[0]=QKVParallelLinear(
+      repr=QKVParallelLinear(in_features=1024, output_features=4096, bias=False, tp_size=1, gather_output=False)
+    )
+  arg[1]=Tensor(
+      shape=(1, 1024)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+    )
+  arg[2]=None
+Output:
+  return=Tensor(
+      shape=(1, 4096)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+    )
+```
+
+This is a real level-3 excerpt captured from `Qwen/Qwen3-0.6B`.
+
+### Full Logging (With Tensor Statistics)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=5
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+python my_script.py
+```
+
+Additional output:
+```
+================================================================================
+[2026-03-19 01:00:42] SGLang Kernel API Call: diffusion.quant_method.UnquantizedLinearMethod.apply
+Positional input arguments:
+  arg[1]=Tensor(
+      shape=(1, 77, 768)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+      min=-27.250000
+      max=28.500000
+      mean=0.011723
+      nan_count=0
+      inf_count=0
+    )
+Output:
+  return=Tensor(
+      shape=(1, 77, 2304)
+      dtype=torch.bfloat16
+      device=cuda:0
+      requires_grad=False
+      is_contiguous=True
+      min=-8.937500
+      max=9.375000
+      mean=0.009460
+      nan_count=0
+      inf_count=0
+    )
+```
+
+This is a real level-5 excerpt captured from `black-forest-labs/FLUX.1-dev`.
+
+### Crash-Safe Dumps (Inputs Saved Before Execution)
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+export SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_kernel_api_dumps
+
+python my_script.py
+```
+
+At level 10, SGLang saves the inputs before execution. If the kernel crashes, the dump directory still contains the inputs and exception metadata.
+
+If CUDA graph capture is active, tensor dumps are skipped automatically to avoid capture-time CUDA errors. In that case, you still get the kernel API call log, but not `inputs.pt` / `outputs.pt`.
+
+Level-10 dumps are best understood as crash-safe call snapshots. They always preserve the observed call boundary. They do not guarantee one-click replay for every method, because some methods depend on module state that is not serialized into the dump.
+
+Real level-10 dump layout from `Qwen/Qwen3-0.6B`:
+
+```text
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001/inputs.pt
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001/metadata.json
+/tmp/sglang_kernel_api_validation/qwen_qwen3_0_6b_level10_dumps/20260319_004821_182_pid919286_RotaryEmbedding.forward_call0001/outputs.pt
+```
+
+Real `metadata.json` excerpt:
+
+```json
+{
+  "function_name": "RotaryEmbedding.forward",
+  "timestamp": "20260319_004821_182",
+  "process_id": 919286,
+  "execution_status": "completed",
+  "input_tensor_keys": ["arg_0", "arg_1", "arg_2"],
+  "output_tensor_keys": ["result_0", "result_1"]
+}
+```
+
+## Step 2: Reproduce an LLM CUDA Crash
+
+Create a temporary reproducer:
+
+```bash
+python3 - <<'PY'
+from pathlib import Path
+Path("/tmp/sglang_llm_crash.py").write_text(
+    "import torch\\n"
+    "import torch.nn.functional as F\\n"
+    "from sglang.srt.utils.custom_op import register_custom_op\\n\\n"
+    "def _fake_embedding(indices, table):\\n"
+    "    return torch.empty((*indices.shape, table.shape[-1]), device=table.device, dtype=table.dtype)\\n\\n"
+    "@register_custom_op(op_name='mock_llm_cuda_crash', fake_impl=_fake_embedding)\\n"
+    "def mock_llm_cuda_crash(indices, table):\\n"
+    "    out = F.embedding(indices, table)\\n"
+    "    torch.cuda.synchronize()\\n"
+    "    return out\\n\\n"
+    "table = torch.randn(4, 8, device='cuda', dtype=torch.float16)\\n"
+    "indices = torch.tensor([0, 7], device='cuda', dtype=torch.long)\\n"
+    "mock_llm_cuda_crash(indices, table)\\n"
+)
+PY
+
+SGLANG_KERNEL_API_LOGLEVEL=1 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_llm_level1.log \
+python3 /tmp/sglang_llm_crash.py
+```
+
+What to expect:
+- The script exits with a CUDA `device-side assert`
+- The log still contains the last API boundary before the crash
+
+Try the same example at level 3:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=3 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_llm_level3.log \
+python3 /tmp/sglang_llm_crash.py
+```
+
+Now the log shows tensor metadata before the crash.
+
+Try level 10:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=10 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_llm_level10.log \
+SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_llm_level10_dumps \
+python3 /tmp/sglang_llm_crash.py
+```
+
+Now you should see:
+- A log entry for `sglang.custom_op.mock_llm_cuda_crash`
+- A dump directory with `inputs.pt`
+- `metadata.json` showing `execution_status: "exception"`
+- No `outputs.pt`, because the kernel crashed before producing output
+
+For real-model success-path level-10 dumps, it is often easier to temporarily disable CUDA graph and piecewise CUDA graph for the debug run.
+
+## Step 3: Reproduce a Diffusion CUDA Crash
+
+Create a temporary diffusion-side reproducer:
+
+```bash
+python3 - <<'PY'
+from pathlib import Path
+Path("/tmp/sglang_diffusion_crash.py").write_text(
+    "import torch\\n"
+    "import torch.nn.functional as F\\n"
+    "from sglang.multimodal_gen.runtime.layers.utils import register_custom_op\\n\\n"
+    "def _fake_embedding(positions, cache):\\n"
+    "    return torch.empty((*positions.shape, cache.shape[-1]), device=cache.device, dtype=cache.dtype)\\n\\n"
+    "@register_custom_op(op_name='mock_diffusion_cuda_crash', fake_impl=_fake_embedding)\\n"
+    "def mock_diffusion_cuda_crash(positions, cache):\\n"
+    "    out = F.embedding(positions, cache)\\n"
+    "    torch.cuda.synchronize()\\n"
+    "    return out\\n\\n"
+    "cache = torch.randn(4, 64, device='cuda', dtype=torch.float16)\\n"
+    "positions = torch.tensor([0, 9], device='cuda', dtype=torch.long)\\n"
+    "mock_diffusion_cuda_crash(positions, cache)\\n"
+)
+PY
+
+SGLANG_KERNEL_API_LOGLEVEL=1 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_diffusion_level1.log \
+python3 /tmp/sglang_diffusion_crash.py
+```
+
+Try level 3:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=3 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_diffusion_level3.log \
+python3 /tmp/sglang_diffusion_crash.py
+```
+
+Try level 10:
+
+```bash
+SGLANG_KERNEL_API_LOGLEVEL=10 \
+SGLANG_KERNEL_API_LOGDEST=/tmp/sglang_diffusion_level10.log \
+SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_diffusion_level10_dumps \
+python3 /tmp/sglang_diffusion_crash.py
+```
+
+If your local environment has unrelated FlashInfer import issues, resolve them in the shell before running the example. The example itself does not set any `FLASHINFER_*` environment variable.
+
+## Step 4: Multi-Process Debugging
+
+When running with multiple GPUs or worker processes, use `%i` in the log path:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug_rank_%i.log
+
+torchrun --nproc_per_node=4 my_script.py
+```
+
+This creates separate logs such as:
+- `debug_rank_12345.log`
+- `debug_rank_12346.log`
+- `debug_rank_12347.log`
+- `debug_rank_12348.log`
+
+Real multi-process example from a 2-GPU `Qwen/Qwen2.5-0.5B-Instruct` run:
+
+```text
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950201.log
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950349.log
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950350.log
+/tmp/sglang_kernel_api_validation_multi/qwen_qwen2_5_0_5b_instruct_level3_950351.log
+```
+
+You should usually do the same for level-10 dump directories:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+export SGLANG_KERNEL_API_LOGDEST=debug_rank_%i.log
+export SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_kernel_api_dumps_%i
+```
+
+This avoids multiple ranks writing into the same dump directory tree.
+
+## Step 5: Filter Level-10 Dumps
+
+If level 10 is too noisy, restrict dumps to specific APIs:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+export SGLANG_KERNEL_API_DUMP_DIR=/tmp/sglang_kernel_api_dumps
+export SGLANG_KERNEL_API_DUMP_INCLUDE='sglang.custom_op.*'
+export SGLANG_KERNEL_API_DUMP_EXCLUDE='*.fake_impl'
+```
+
+`SGLANG_KERNEL_API_DUMP_INCLUDE` and `SGLANG_KERNEL_API_DUMP_EXCLUDE` use shell-style wildcard matching.
+
+## Step 6: Common CUDA Errors and What to Check
+
+### Illegal Memory Access or Device-Side Assert
+
+**Typical errors**:
+```
+RuntimeError: CUDA error: an illegal memory access was encountered
+torch.AcceleratorError: CUDA error: device-side assert triggered
+```
+
+Use:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+Check in the logs:
+- ✅ Tensor shapes
+- ✅ Tensor dtypes
+- ✅ CUDA vs CPU device placement
+- ✅ Tensor stride / contiguity
+- ✅ Whether the failing call has inputs logged but no outputs logged
+
+Typical shape-mismatch pattern:
+
+```text
+SGLang Kernel API Call: ...
+arg[0]=Tensor(shape=(..., 128), ...)   # ✅ expected dimension
+arg[1]=Tensor(shape=(..., 64), ...)    # ❌ mismatch
+```
+
+This often points to head-dim, hidden-dim, or cache-layout mismatch rather than a random CUDA failure.
+
+### NaN or Inf
+
+Use:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=5
+```
+
+Check:
+- `min`
+- `max`
+- `mean`
+- `nan_count`
+- `inf_count`
+
+Typical bad pattern:
+
+```text
+Tensor(
+  ...
+  min=-1234567.000000   # ❌ suspiciously large
+  max=9876543.000000    # ❌ suspiciously large
+  mean=nan              # ❌ bad
+  nan_count=128         # ❌ found NaNs
+  inf_count=0           # ✅ no Infs here
+)
+```
+
+This usually means the bad values were already present before the crashing kernel.
+
+### Out of Memory
+
+Use:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+Check:
+- Unexpectedly large tensor shapes
+- Batch size
+- Sequence length
+- Frame count or image resolution in diffusion workloads
+
+Also check whether a supposedly per-token or per-frame tensor accidentally became full-sequence or full-image sized.
+
+Typical bad pattern:
+
+```text
+Tensor(
+  shape=(1024, 8192, 128, 128)   # ❌ way too large
+  ...
+)
+```
+
+### Example: Spot a Shape Bug from the Log
+
+Suppose the failing API log looks like this:
+
+```text
+[2026-03-19 00:47:30] SGLang Kernel API Call: RotaryEmbedding.forward
+Positional input arguments:
+  arg[0]=Tensor(shape=(1, 8), dtype=torch.int64, ...)
+  arg[1]=Tensor(shape=(1, 8, 8, 256), dtype=torch.bfloat16, ...)    # ✅ query
+  arg[2]=Tensor(shape=(1, 8, 4, 64), dtype=torch.bfloat16, ...)     # ❌ key head_dim mismatch
+```
+
+What this tells you:
+- ✅ positions look reasonable
+- ✅ query looks plausible
+- ❌ key last dimension is inconsistent with the expected rotary/head dimension
+
+That usually means the bug is in projection layout, head packing, or cache format rather than in the rotary kernel itself.
+
+## Step 7: Combine with compute-sanitizer
+
+For harder bugs, combine kernel API logging with CUDA memory checking:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+compute-sanitizer --tool memcheck python3 /tmp/sglang_llm_crash.py
+```
+
+Use `debug.log` to see the exact inputs that reached the crashing API boundary.
+
+Typical `compute-sanitizer` output:
+
+```text
+========= COMPUTE-SANITIZER
+========= Invalid __global__ write of size 4 bytes
+=========     at 0x1234 in SomeKernel
+=========     by thread (256,0,0) in block (10,0,0)
+=========     Address 0x... is out of bounds
+```
+
+Use the sanitizer output to identify the failing kernel and use `debug.log` to identify the exact tensors that reached the API boundary right before it.
+
+If you need more synchronous host-side error reporting, you can try `CUDA_LAUNCH_BLOCKING=1` as a separate follow-up experiment. It is not part of the default workflow because it changes execution timing and can hide concurrency-related behavior.
+
+## Step 8: Combine with cuda-gdb
+
+For crashes that need a stack trace instead of only memory diagnostics:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+export SGLANG_KERNEL_API_LOGDEST=debug.log
+
+cuda-gdb --args python3 /tmp/sglang_llm_crash.py
+```
+
+Inside `cuda-gdb`:
+
+```text
+(cuda-gdb) run
+(cuda-gdb) where
+```
+
+Then correlate the backtrace with `debug.log`.
+
+## Step 9: Kernel-Level Debugging with printf()
+
+When you own the CUDA kernel, `printf()` is still useful for narrowing down bad indices, bad launch geometry, or broken state propagation.
+
+Basic pattern:
+
+```cpp
+__global__ void MyKernel(const float* input, float* output, int n) {
+  int idx = blockIdx.x * blockDim.x + threadIdx.x;
+
+  if (threadIdx.x == 0 && blockIdx.x == 0) {
+    printf("n=%d input0=%f\n", n, input[0]);
+  }
+
+  if (idx < n) {
+    output[idx] = input[idx] * 2.0f;
+  }
+}
+```
+
+After launch, force the output to flush:
+
+```python
+my_kernel(...)
+torch.cuda.synchronize()
+```
+
+For warp-specialized kernels, do not blindly print only on `threadIdx.x == 0`. Pick one representative thread per warp or per specialization group instead.
+
+### Warp-Specialized Kernels: Choosing the Right Print Thread
+
+Problem:
+- `threadIdx.x == 0` only prints from the first warp in the block
+- for warp-specialized kernels, that often misses the warp or group that is actually wrong
+
+Better pattern:
+
+```cpp
+__global__ void WarpSpecializedKernel(...) {
+  // Example: first lane of each warp
+  if ((threadIdx.x % 32) == 0) {
+    printf("warp=%d\n", threadIdx.x / 32);
+  }
+}
+```
+
+Or, if the kernel is organized in larger specialization groups, print once per group instead of once per block.
+
+Common mistake:
+
+```cpp
+// Only warp 0 prints
+if (threadIdx.x == 0) {
+  printf("warp=%d\n", threadIdx.x / 32);
+}
+```
+
+### Quick Reference
+
+| Kernel Type | Print Condition | Notes |
+|----------|----------|-------------|
+| Simple kernel | `threadIdx.x == 0` | One thread per block is usually enough |
+| Warp-specialized kernel | one representative lane per warp | e.g. `threadIdx.x % 32 == 0` |
+| Group-specialized kernel | one representative lane per group | choose based on the kernel's scheduling layout |
+
+### Other Kernel Debugging Tools
+
+```cpp
+assert(value >= 0.0f && "value must be non-negative");
+static_assert(BLOCK_SIZE % 32 == 0, "BLOCK_SIZE must be warp aligned");
+```
+
+## Environment Variables Reference
+
+| Variable | Values | Description |
+|----------|--------|-------------|
+| `SGLANG_KERNEL_API_LOGLEVEL` | `0` | No logging (default) |
+|  | `1` | Function names only |
+|  | `3` | Inputs and outputs with metadata |
+|  | `5` | Level 3 plus tensor statistics |
+|  | `10` | Level 5 plus crash-safe tensor dumps |
+| `SGLANG_KERNEL_API_LOGDEST` | `stdout` | Log to stdout |
+|  | `stderr` | Log to stderr |
+|  | `<path>` | Log to file |
+|  | `log_%i.txt` | `%i` expands to process ID |
+| `SGLANG_KERNEL_API_DUMP_DIR` | `<path>` | Directory for level-10 dumps |
+| `SGLANG_KERNEL_API_DUMP_INCLUDE` | wildcard list | Only dump matching API names |
+| `SGLANG_KERNEL_API_DUMP_EXCLUDE` | wildcard list | Skip matching API names |
+
+## Best Practices
+
+### 1. Start with Level 3
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+Level 3 is usually enough to catch wrong shapes, wrong dtypes, and wrong devices.
+
+### 2. Use Level 5 for Numerical Issues
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=5
+```
+
+Use it when you suspect NaN or Inf values.
+
+### 3. Use Level 10 for Crash Reproduction
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=10
+```
+
+This is the most useful mode when the process crashes before you can inspect live tensors.
+
+If you need successful input/output dumps from a real model run, temporarily disable CUDA graph for that debug session.
+
+When level 10 is too noisy, pair it with `SGLANG_KERNEL_API_DUMP_INCLUDE` / `SGLANG_KERNEL_API_DUMP_EXCLUDE` instead of dumping every covered API.
+
+### 4. Log to File for Crashes
+
+```bash
+export SGLANG_KERNEL_API_LOGDEST=crash.log
+```
+
+File logs are safer than stdout when the process aborts.
+
+### 5. Disable Logging in Production
+
+```bash
+unset SGLANG_KERNEL_API_LOGLEVEL
+```
+
+When disabled, the decorator returns the original callable and adds no runtime logging overhead.
+
+## Troubleshooting
+
+### No Logs Appear
+
+Check:
+1. `echo $SGLANG_KERNEL_API_LOGLEVEL`
+2. `echo $SGLANG_KERNEL_API_LOGDEST`
+3. Whether the failing path goes through a covered API boundary
+
+### Too Much Output
+
+Reduce the level:
+
+```bash
+export SGLANG_KERNEL_API_LOGLEVEL=3
+```
+
+### Statistics Are Skipped During CUDA Graph Capture
+
+If you see:
+```text
+statistics=[skipped: CUDA graph capture in progress]
+```
+
+That is expected. Level-5 statistics are intentionally skipped during CUDA graph capture to avoid synchronization side effects.
+
+### Tensor Dumps Are Skipped During CUDA Graph Capture
+
+If you see:
+```text
+Tensor dump skipped: CUDA graph capture in progress
+```
+
+That is also expected. Level-10 dumps require copying tensors to CPU, which is not allowed during CUDA graph capture.

third_party/sglang/.claude/skills/generate-profile/SKILL.md

@@ -0,0 +1,141 @@
+---
+name: generate-profile
+description: Generate an e2e profiling trace of an SGLang server run. Launches a server, validates accuracy, captures a Chrome-compatible trace, and returns the profile path.
+---
+
+# Generate an E2E Profile of an SGLang Server Run
+
+This skill launches an SGLang server, validates it with a quick accuracy test, generates a profiling trace, and returns the profile file path.
+
+## Prerequisites
+
+- A working SGLang installation (`pip install -e .` or equivalent)
+- At least one available CUDA GPU
+
+## Step-by-step Workflow
+
+### Step 1: Launch the server
+
+```bash
+CUDA_VISIBLE_DEVICES=<gpu_id> sglang serve --model-path <model> --port <port> &
+```
+
+- Default model: `Qwen/Qwen3-8B` (good balance of speed and quality)
+- Default port: `30000`
+- The server runs in the background. Save the PID for cleanup.
+- Use the GPU specified by the user's preferences (check memory files for GPU preferences).
+
+### Step 2: Wait for server readiness
+
+Poll the health endpoint until the server is ready:
+
+```bash
+for i in $(seq 1 120); do
+  if curl -s http://127.0.0.1:<port>/health 2>/dev/null | grep -q "ok\|healthy"; then
+    echo "Server ready"
+    break
+  fi
+  sleep 5
+done
+```
+
+The server prints **"The server is fired up and ready to roll!"** to stdout when ready. The health endpoint returns 200 once the server can accept requests.
+
+Typical startup time: 30-90 seconds depending on model size and whether CUDA graphs are being compiled.
+
+### Step 3: Validate accuracy (sanity check)
+
+```bash
+python3 -m sglang.test.few_shot_gsm8k --num-q 20
+```
+
+- Expected accuracy: **> 0.8** for capable models (Qwen3-8B, Llama-3.1-8B-Instruct, etc.)
+- This is a quick sanity check, not a rigorous benchmark.
+- If accuracy is unexpectedly low, something is wrong — do not proceed to profiling.
+
+### Step 4: Generate the profile
+
+```bash
+python3 -m sglang.test.send_one --profile
+```
+
+This command:
+1. Sends a request to the server
+2. Triggers the profiler for 5 steps (default)
+3. Generates a trace file under `/tmp/<timestamp>/`
+4. The trace directory contains:
+   - `<timestamp>-TP-0.trace.json.gz` — Chrome trace format (open in `chrome://tracing` or Perfetto)
+   - `server_args.json` — the server configuration used
+
+**Output format:**
+```
+Dump profiling traces to /tmp/<timestamp>
+```
+
+The profile path is printed to stdout. Parse it from the output.
+
+**Optional flags:**
+- `--profile-steps N` — number of profiling steps (default: 5)
+- `--profile-by-stage` — profile by stage (prefill/decode separately)
+- `--profile-prefix <path>` — custom output prefix
+
+### Step 5: Kill the server
+
+```bash
+pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
+```
+
+Wait a moment and verify no sglang processes remain:
+```bash
+sleep 2 && pgrep -af "sglang serve" || echo "Server killed"
+```
+
+### Step 6: Report the profile path
+
+Return the profile directory path (e.g., `/tmp/1773999986.4769795`) and list its contents so the user knows what files were generated.
+
+## Example Full Run
+
+```bash
+# 1. Launch server
+source cleanup/bin/activate
+CUDA_VISIBLE_DEVICES=1 sglang serve --model-path Qwen/Qwen3-8B --port 30000 &
+
+# 2. Wait for ready
+for i in $(seq 1 120); do
+  curl -s http://127.0.0.1:30000/health | grep -q "ok" && break
+  sleep 5
+done
+
+# 3. Accuracy check
+python3 -m sglang.test.few_shot_gsm8k --num-q 20
+# Expected: Accuracy > 0.8
+
+# 4. Profile
+python3 -m sglang.test.send_one --profile
+# Output: "Dump profiling traces to /tmp/1773999986.4769795"
+
+# 5. Cleanup
+pkill -9 -f "sglang.launch_server\|sglang serve\|sglang.srt"
+sleep 2
+
+# 6. Check output
+ls -la /tmp/1773999986.4769795/
+# 1773999986.4851577-TP-0.trace.json.gz  (Chrome trace)
+# server_args.json                        (server config)
+```
+
+## Customization
+
+- **Different port**: Pass `--port <port>` and use `--host 127.0.0.1 --port <port>` for test commands
+- **Multi-GPU**: Use `--tp <N>` for tensor parallelism; trace files will be generated per TP rank
+- **Longer profile**: Use `--profile-steps 10` for more steps in the trace
+- **Stage profiling**: Use `--profile-by-stage` to separate prefill and decode phases
+
+## Viewing the Profile
+
+Open the `.trace.json.gz` file in:
+- **Perfetto UI**: https://ui.perfetto.dev/ (drag and drop the file)
+- **Chrome tracing**: `chrome://tracing` (load the file)
+
+Both support the gzipped Chrome trace format natively.

third_party/sglang/.claude/skills/sglang-bisect-ci-regression/SKILL.md

@@ -0,0 +1,219 @@
+# SGLang Bisect CI Regression
+
+Investigate a consistently failing CI test to find the root cause - whether it's a code regression from a specific PR, a hardware/runner-specific issue, or an environment change. Optionally reproduce the failure on a remote GPU server.
+
+## Slash Command
+
+`/sglang-bisect-ci-regression <test_name_or_ci_url> [ssh_target] [docker_container]`
+
+## When to Use This Skill
+
+- A CI test is failing consistently on main (scheduled runs)
+- You need to find which PR introduced a regression
+- You suspect a runner-specific or GPU-specific issue
+- You want to reproduce a CI failure on a remote server
+
+## Arguments
+
+- **First argument (required)**: Test file name (e.g. `test_lora_tp.py`) or a GitHub Actions job URL
+- **Second argument (optional)**: SSH target for remote reproduction (e.g. `user@host`)
+- **Third argument (optional)**: Docker container name on the SSH target (e.g. `sglang_dev`)
+
+If SSH target and docker container are not provided, the skill will only perform the CI log analysis and bisection, without remote reproduction. **Ask the user** for these if reproduction is needed and they weren't provided.
+
+## Background: Scheduled CI Runs
+
+SGLang uses the `pr-test.yml` workflow with **scheduled runs** (cron-triggered) to periodically test the `main` branch. These runs are the primary data source for detecting regressions:
+
+- **Workflow**: `pr-test.yml` with `event: schedule`
+- **Branch**: `main`
+- **Dashboard**: https://github.com/sgl-project/sglang/actions/workflows/pr-test.yml?query=event%3Aschedule
+- **Frequency**: Runs multiple times daily, each pinned to the HEAD of `main` at trigger time
+- **Purpose**: Catches regressions that slip through PR-level CI (e.g., interaction bugs between merged PRs, hardware-specific issues)
+
+Always use these scheduled runs (not PR-triggered runs) when bisecting regressions on `main`. The `--event schedule` filter in `gh run list` ensures you only see these periodic main-branch runs.
+
+## Workflow
+
+### Phase 1: Extract the Failure Signature
+
+1. **Get the failing test details from CI logs.** If given a URL, fetch logs directly. If given a test name, find recent scheduled runs of `pr-test.yml` on `main` that failed:
+
+```bash
+# List recent scheduled runs targeting main (the primary source of truth for regressions)
+# These are cron-triggered runs visible at:
+# https://github.com/sgl-project/sglang/actions/workflows/pr-test.yml?query=event%3Aschedule
+gh run list --repo sgl-project/sglang --workflow="pr-test.yml" --event schedule --branch main --limit 20 --json databaseId,conclusion,createdAt,headSha
+
+# Find the job containing the test
+gh run view {RUN_ID} --repo sgl-project/sglang --json jobs --jq '.jobs[] | select(.conclusion == "failure") | {name, conclusion, databaseId}'
+
+# Get the failure details
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E -B 5 -A 30 "AssertionError|FAIL|Error|{TEST_NAME}"
+```
+
+2. **Record the failure signature:**
+   - Exact error message and assertion
+   - Affected test method name
+   - Model/config involved
+   - Numeric values (e.g., tolerance diffs, scores)
+   - Whether the failure is deterministic (same values across runs)
+
+### Phase 2: Temporal Bisection
+
+3. **Find the boundary between passing and failing runs.** Walk through the scheduled run history (from the `pr-test.yml` schedule runs on `main`) to identify:
+   - Last known PASSING run (sha + date)
+   - First known FAILING run (sha + date)
+
+```bash
+# For each scheduled run, check the specific partition/job status
+gh run view {RUN_ID} --repo sgl-project/sglang --json jobs --jq '.jobs[] | select(.name == "{JOB_NAME}") | {conclusion, databaseId}'
+
+# Verify a specific test passed or failed in a run
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E "{TEST_NAME}|PASSED|FAILED|logprobs mismatch" | head -10
+```
+
+4. **List commits between the boundary:**
+
+```bash
+git log --oneline {LAST_PASS_SHA}..{FIRST_FAIL_SHA}
+```
+
+5. **Filter for relevant commits** that touch files related to the failing test (model layers, kernels, test utilities, etc.):
+
+```bash
+git log --oneline {LAST_PASS_SHA}..{FIRST_FAIL_SHA} -- {relevant_paths}
+```
+
+### Phase 3: Runner/Hardware Analysis
+
+6. **Check if the failure is runner-specific.** Extract the runner identity from each failing and passing run:
+
+```bash
+# Get runner name and machine
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E "Runner name|Machine name" | head -5
+
+# Get GPU/driver info
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -i -E "NVIDIA-SMI|Driver Version|CUDA Version" | head -5
+
+# Get package versions
+gh run view {RUN_ID} --repo sgl-project/sglang --job {JOB_ID} --log 2>&1 | grep -E "sgl.kernel.*==|flashinfer.*==" | head -5
+```
+
+7. **Correlate runners with pass/fail outcomes.** Build a table:
+
+| Run ID | Date | Runner | GPU Type | Driver | Result |
+|--------|------|--------|----------|--------|--------|
+
+If all failures map to a specific runner type/GPU and all passes map to another, the issue is **hardware-specific**, not a code regression.
+
+### Phase 4: Code Analysis
+
+8. **If a code regression is suspected** (failures not runner-specific), examine the candidate commits:
+   - Read the changed files
+   - Understand how the changes could affect the failing test
+   - Look for prefill-vs-decode differences, TP-specific paths, kernel changes
+
+9. **If a hardware issue is suspected**, analyze:
+   - Kernel compatibility (CUDA compute capability)
+   - Driver version differences
+   - All-reduce / NCCL behavior differences
+   - CUDA graph capture differences across GPU architectures
+
+### Phase 5: Remote Reproduction (Optional)
+
+Only if SSH target and docker container were provided.
+
+10. **Verify the remote environment:**
+
+```bash
+ssh {SSH_TARGET} "docker exec {CONTAINER} nvidia-smi --query-gpu=name,driver_version --format=csv"
+ssh {SSH_TARGET} "docker exec {CONTAINER} pip show sgl-kernel sglang flashinfer-python 2>&1 | grep -E 'Name:|Version:'"
+```
+
+11. **Ensure latest code is installed.** If the container is stale, update:
+
+```bash
+# Try fetching latest main
+ssh {SSH_TARGET} "docker exec {CONTAINER} bash -c 'cd /path/to/sglang && git fetch origin main && git checkout origin/main'"
+# Or download and install from tarball if git auth fails
+ssh {SSH_TARGET} "docker exec {CONTAINER} bash -c 'cd /tmp && curl -L https://github.com/sgl-project/sglang/archive/refs/heads/main.tar.gz | tar xz && cd sglang-main && pip install -e \"python[all]\"'"
+# Reinstall (after git fetch)
+ssh {SSH_TARGET} "docker exec {CONTAINER} bash -c 'cd /path/to/sglang && pip install -e \"python[all]\"'"
+# Install test dependencies if needed
+ssh {SSH_TARGET} "docker exec {CONTAINER} pip install peft rouge-score"
+```
+
+12. **Create a minimal reproduction script** that:
+    - Uses `if __name__ == '__main__'` with `mp.set_start_method("spawn")`
+    - Runs the specific failing test configuration
+    - Prints key metrics (diffs, scores, outputs)
+    - Exits with code 1 on failure
+
+13. **Copy and run the reproduction script:**
+
+```bash
+scp /tmp/repro_script.py {SSH_TARGET}:/tmp/
+ssh {SSH_TARGET} "docker cp /tmp/repro_script.py {CONTAINER}:/tmp/"
+ssh {SSH_TARGET} "docker exec -e CUDA_VISIBLE_DEVICES=0,1 {CONTAINER} python3 /tmp/repro_script.py"
+```
+
+14. **Run control experiments** to isolate the variable:
+    - If suspecting TP issue: run with TP=1 as control
+    - If suspecting GPU issue: compare same code on different GPU
+    - If suspecting a specific commit: test before/after that commit
+
+### Phase 6: Report
+
+15. **Produce a structured report:**
+
+```markdown
+## CI Regression Bisection Report
+
+### Failure Signature
+- **Test**: {test_file}::{test_method}
+- **Error**: {exact error message}
+- **Key metrics**: {numeric values}
+- **Deterministic**: Yes/No
+
+### Root Cause Classification
+One of:
+- **Code Regression**: PR #{number} introduced the bug
+- **Hardware-Specific**: Fails on {GPU_TYPE}, passes on others
+- **Environment Change**: New runner/driver/package version
+- **Pre-existing Flakiness**: Intermittent, not a new regression
+
+### Evidence
+| Condition | Result |
+|-----------|--------|
+| {condition1} | PASS/FAIL |
+| {condition2} | PASS/FAIL |
+
+### Timeline
+- {date}: Last known pass ({sha}, {runner})
+- {date}: First known fail ({sha}, {runner})
+- {date}: Confirmed reproduction on {server}
+
+### Recommended Fix
+- **Short-term**: {workaround}
+- **Long-term**: {proper fix}
+```
+
+## Key Patterns to Recognize
+
+| Pattern | Diagnosis |
+|---------|-----------|
+| Same SHA passes on runner A, fails on runner B | Hardware/runner-specific |
+| All runners fail after commit X | Code regression from commit X |
+| Intermittent - same runner sometimes passes/fails | Flaky test or race condition |
+| Prefill OK but decode fails | TP/all-reduce issue in decode path |
+| Works with TP=1, fails with TP>1 | Tensor parallelism bug |
+| Exact same numeric diff every time | Deterministic bug, not flakiness |
+
+## Important Notes
+
+- **Always check runner identity** before concluding it's a code regression. Many "consistent" failures are actually runner-specific.
+- **Test partition assignments change over time** as tests are added/removed. A test may move between partitions, landing on different runner types.
+- **H200 runners** use `/root/actions-runner/` path and machine names like `gpu-h200-worker-*`. Non-H200 runners use `/public_sglang_ci/runner-*` paths.
+- When running remote reproduction, use `run_in_background` for long-running tests and check output with `TaskOutput`.
+- Container environments may be stale - always verify package versions match CI before drawing conclusions.

third_party/sglang/.claude/skills/write-sglang-test/SKILL.md

@@ -0,0 +1,444 @@
+---
+name: write-sglang-test
+description: Guide for writing SGLang CI/UT tests. Covers CustomTestCase, CI registration, server fixtures, model selection, mock testing, and test placement. Always read test/README.md for the full CI layout, how to run tests, and extra tips. Use when creating new tests, adding CI test cases, writing unit tests, or when the user asks to add tests for SGLang features.
+---
+
+# Writing SGLang CI / UT Tests
+
+This skill covers **how to write and register tests**. For CI pipeline internals (stage ordering, fast-fail, gating, partitioning, debugging CI failures), see the [CI workflow guide](../ci-workflow-guide/SKILL.md).
+
+## Core Rules
+
+1. **Always use `CustomTestCase`** — never raw `unittest.TestCase`. It ensures `tearDownClass` runs even when `setUpClass` fails, preventing resource leaks in CI.
+2. **`tearDownClass` must be defensive** — use `hasattr`/null checks before accessing resources (e.g. `cls.process`) that `setUpClass` may not have finished allocating.
+3. **Place tests in `test/registered/<category>/`** — except JIT kernel tests and benchmarks, which live in `python/sglang/jit_kernel/tests/` and `python/sglang/jit_kernel/benchmark/` (nested subfolders are allowed)
+4. **Reuse server fixtures** — inherit from `DefaultServerBase` or write `setUpClass`/`tearDownClass` with `popen_launch_server`
+5. **Prefer mock over real server** — when testing logic that doesn't need a server / engine launch (middleware, request routing, config validation, argument parsing), use `unittest.mock.patch` / `MagicMock` and place tests in `test/registered/unit/`. Only launch a real server when the test genuinely needs inference results or server lifecycle behavior.
+
+JIT kernel exception:
+- If the task is adding or updating code under `python/sglang/jit_kernel/`, prefer the `add-jit-kernel` skill first.
+- JIT kernel correctness tests use `python/sglang/jit_kernel/tests/**/test_*.py`.
+- JIT kernel benchmarks use `python/sglang/jit_kernel/benchmark/**/bench_*.py`.
+- Those files are still executed by `test/run_suite.py`, but through dedicated kernel suites rather than `test/registered/`.
+
+---
+
+## Model & Backend Selection
+
+| Scenario | Model | CI Registration | Suite |
+|----------|-------|-----------------|-------|
+| **Unit tests** (no server / engine launch) | None | `register_cpu_ci` (prefer) or `register_cuda_ci` | `stage-a-test-cpu` or `stage-b-test-1-gpu-small` |
+| **Common / backend-independent** (middleware, abort, routing, config, arg parsing) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` only | `stage-b-test-1-gpu-small` |
+| **Model-agnostic functionality** (sampling, session, OpenAI API features) | `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` (1B) | `register_cuda_ci` (+ AMD if relevant) | `stage-b-test-1-gpu-small` |
+| **General performance** (single node, no spec/DP/parallelism) | `DEFAULT_MODEL_NAME_FOR_TEST` (8B) | `register_cuda_ci` | `stage-b-test-1-gpu-large` |
+| **Bigger features** (spec, DP, TP, disaggregation) | Case by case | Case by case | See suite table below |
+
+**Key principle for E2E tests**: Do NOT add `register_amd_ci` unless the test specifically exercises AMD/ROCm code paths. Common E2E tests just need any GPU to run — duplicating across backends wastes CI time with no extra coverage.
+
+### All model constants
+
+Defined in `python/sglang/test/test_utils.py`:
+
+| Constant | Model | When to use |
+|----------|-------|-------------|
+| `DEFAULT_SMALL_MODEL_NAME_FOR_TEST` | Llama-3.2-1B-Instruct | Common features, model-agnostic tests |
+| `DEFAULT_SMALL_MODEL_NAME_FOR_TEST_BASE` | Llama-3.2-1B | Base (non-instruct) model tests |
+| `DEFAULT_MODEL_NAME_FOR_TEST` | Llama-3.1-8B-Instruct | General performance (single node) |
+| `DEFAULT_MOE_MODEL_NAME_FOR_TEST` | Mixtral-8x7B-Instruct | MoE-specific tests |
+| `DEFAULT_SMALL_EMBEDDING_MODEL_NAME_FOR_TEST` | — | Embedding tests |
+| `DEFAULT_SMALL_VLM_MODEL_NAME_FOR_TEST` | — | Vision-language tests |
+
+### Naming Conventions
+
+- **Suite**: `stage-{a,b,c}-test-{gpu_count}-gpu-{hardware}` (e.g., `stage-b-test-1-gpu-small`)
+- **CI runner**: `{gpu_count}-gpu-{hardware}` (e.g., `1-gpu-5090`, `4-gpu-h100`, `8-gpu-h200`)
+
+### All CI Suites
+
+#### Per-commit (CUDA)
+
+| Suite | Runner (label) | Description |
+|-------|----------------|-------------|
+| `stage-a-test-1-gpu-small` | `1-gpu-5090` | Quick checks on a small NVIDIA GPU before heavier stages |
+| `stage-a-test-cpu` | `ubuntu-latest` | CPU-only unit tests |
+| `stage-b-test-1-gpu-small` | `1-gpu-5090` | Core engine tests that fit a 5090-class card |
+| `stage-b-test-1-gpu-large` | `1-gpu-h100` | Tests that need H100-class memory or kernels (e.g. FA3) |
+| `stage-b-test-2-gpu-large` | `2-gpu-h100` | Two-GPU correctness and parallelism (TP/PP) on H100 |
+| `stage-b-test-4-gpu-b200` | `4-gpu-b200` | Early Blackwell coverage (SM100+ paths) on four GPUs |
+| `stage-b-kernel-unit-1-gpu-large` | `1-gpu-h100` | JIT kernel correctness tests under `python/sglang/jit_kernel/tests/` |
+| `stage-b-kernel-unit-8-gpu-h200` | `8-gpu-h200` | Multi-GPU JIT kernel correctness tests under `python/sglang/jit_kernel/tests/` |
+| `stage-b-kernel-benchmark-1-gpu-large` | `1-gpu-h100` | JIT kernel benchmark files under `python/sglang/jit_kernel/benchmark/` |
+| `stage-c-test-4-gpu-h100` | `4-gpu-h100` | Large 4-GPU H100 integration and scaling tests |
+| `stage-c-test-8-gpu-h200` | `8-gpu-h200` | Large 8-GPU H200 runs for big models and parallelism |
+| `stage-c-test-8-gpu-h20` | `8-gpu-h20` | Large 8-GPU H20 runs for big models |
+| `stage-c-test-deepep-4-gpu-h100` | `4-gpu-h100` | DeepEP expert-parallel and networking on four H100s |
+| `stage-c-test-deepep-8-gpu-h200` | `8-gpu-h200` | DeepEP at 8-GPU H200 scale |
+| `stage-c-test-8-gpu-b200` | `8-gpu-b200` | 8-GPU B200 suite (registered but not yet wired to a workflow) |
+| `stage-c-test-4-gpu-b200` | `4-gpu-b200` | 4-GPU B200 suite for large models on Blackwell |
+| `stage-c-test-4-gpu-gb200` | `4-gpu-gb200` | 4-GPU GB200 suite for large models on Grace Blackwell |
+
+#### Per-commit (AMD)
+
+| Suite | Runner (label) | Description |
+|-------|----------------|-------------|
+| `stage-a-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Quick checks on one MI325-class GPU |
+| `stage-b-test-1-gpu-small-amd` | `linux-mi325-1gpu-sglang` | Core 1-GPU AMD tests (14 partitions) |
+| `stage-b-test-1-gpu-small-amd-nondeterministic` | `linux-mi325-1gpu-sglang` | Non-deterministic 1-GPU AMD tests |
+| `stage-b-test-1-gpu-small-amd-mi35x` | `linux-mi35x-gpu-1` | 1-GPU tests on MI35x hardware |
+| `stage-b-test-1-gpu-large-amd` | `linux-mi325-1gpu-sglang` | Large 1-GPU AMD tests (2 partitions) |
+| `stage-b-test-2-gpu-large-amd` | `linux-mi325-2gpu-sglang` | 2-GPU ROCm correctness and parallel setups |
+| `stage-b-test-large-8-gpu-35x-disaggregation-amd` | `linux-mi35x-gpu-8.fabric` | PD disaggregation and RDMA on 8×MI35x fabric |
+| `stage-c-test-4-gpu-amd` | `linux-mi325-4gpu-sglang` | 4-GPU AMD integration (2 partitions) |
+| `stage-c-test-large-8-gpu-amd` | `linux-mi325-8gpu-sglang` | 8-GPU MI325 scaling and integration |
+| `stage-c-test-large-8-gpu-amd-mi35x` | `linux-mi35x-gpu-8` | 8-GPU MI35x scaling (2 partitions) |
+
+
+### Per-commit (Ascend NPU)
+
+| Suite | Runner (label) | Description |
+| --- | --- | --- |
+| `per-commit-1-npu-a2` | `linux-aarch64-a2-1` | 1-NPU LLM CI machine |
+| `per-commit-2-npu-a2` | `linux-aarch64-a2-2` | 2-NPU LLM CI machine |
+| `per-commit-4-npu-a3` | `linux-aarch64-a3-4` | 4-NPU LLM CI machine |
+| `per-commit-16-npu-a3` | `linux-aarch64-a3-16` | 16-NPU LLM CI machine  |
+| `multimodal-gen-test-1-npu-a3` | `linux-aarch64-a3-2` | 1-NPU multimodal CI machine |
+| `multimodal-gen-test-2-npu-a3` | `linux-aarch64-a3-16` | 2-NPU multimodal CI machine |
+| `multimodal-gen-test-8-npu-a3` | `linux-aarch64-a3-16` | 8-NPU multimodal CI machine |
+
+#### Nightly
+
+Nightly suites are listed in `NIGHTLY_SUITES` in [`test/run_suite.py`](../../../test/run_suite.py). They run via `nightly-test-nvidia.yml`, `nightly-test-amd.yml` amd `nightly-test-npu.yml`, not `pr-test.yml`. Examples:
+
+- `nightly-1-gpu` (CUDA)
+- `nightly-kernel-1-gpu` (CUDA, JIT kernel full grids)
+- `nightly-kernel-8-gpu-h200` (CUDA, multi-GPU JIT kernel nightly)
+- `nightly-8-gpu-h200` (CUDA)
+- `nightly-eval-vlm-2-gpu` (CUDA)
+- `nightly-amd` (AMD)
+- `nightly-amd-8-gpu-mi35x` (AMD)
+- `nightly-1-npu-a3` (NPU)
+- `nightly-2-npu-a3` (NPU)
+- `nightly-4-npu-a3` (NPU)
+- `nightly-8-npu-a3` (NPU)
+- `nightly-16-npu-a3` (NPU)
+
+> **Note**: Multimodal diffusion uses `python/sglang/multimodal_gen/test/run_suite.py`, not `test/run_suite.py`.
+
+### Choosing a Suite
+
+Use the lightest suite that meets your test's needs:
+
+- **No GPU required** → `stage-a-test-cpu`
+- **Most small GPU tests** → `stage-b-test-1-gpu-small` (default choice)
+- **Need H100 memory or Hopper features** → `stage-b-test-1-gpu-large`
+- **JIT kernel correctness** → `stage-b-kernel-unit-1-gpu-large`
+- **JIT kernel benchmarks** → `stage-b-kernel-benchmark-1-gpu-large`
+- **Multi-GPU** → only when the test actually needs multiple GPUs
+
+---
+
+## Test File Templates
+
+### Unit Tests (no server / engine launch)
+
+See `test/registered/unit/README.md` for quick-start and rules. Unit tests live in `test/registered/unit/`, mirroring `python/sglang/srt/`:
+
+```python
+"""Unit tests for srt/<module>"""
+
+import unittest
+from unittest.mock import MagicMock, patch
+
+from sglang.srt.<module> import TargetClass
+from sglang.test.ci.ci_register import register_cpu_ci
+from sglang.test.test_utils import CustomTestCase
+
+register_cpu_ci(est_time=5, suite="stage-a-test-cpu")
+# Prefer CPU. Only use register_cuda_ci when the test truly needs a GPU.
+
+class TestTargetClass(CustomTestCase):
+    def test_basic_behavior(self):
+        obj = TargetClass(...)
+        self.assertEqual(obj.method(), expected)
+
+    @patch("sglang.srt.<module>.some_dependency")
+    def test_with_mock(self, mock_dep):
+        mock_dep.return_value = MagicMock()
+        # test logic with dependency mocked
+        ...
+
+
+if __name__ == "__main__":
+    unittest.main()
+```
+
+Use `unittest.mock.patch` / `MagicMock` to mock dependencies and isolate the logic under test. If the module transitively imports GPU-only packages (e.g. `sgl_kernel`), they can be stubbed so the test runs on CPU CI. See `test/registered/unit/README.md` for details and examples.
+
+**Quality bar** — test real logic (validation boundaries, state transitions, error paths, branching, etc.). Skip tests that just verify Python itself works (e.g., "does calling an abstract method raise `NotImplementedError`?", "does a dataclass store the field I assigned?"). Consolidate repetitive patterns into parameterized tests. No production code changes in test PRs.
+
+### E2E test (small model, server needed)
+
+```python
+import unittest
+
+import requests
+
+from sglang.srt.utils import kill_process_tree
+from sglang.test.ci.ci_register import register_cuda_ci
+from sglang.test.test_utils import (
+    DEFAULT_SMALL_MODEL_NAME_FOR_TEST,
+    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+    DEFAULT_URL_FOR_TEST,
+    CustomTestCase,
+    popen_launch_server,
+)
+
+register_cuda_ci(est_time=60, suite="stage-b-test-1-gpu-small")
+
+
+class TestMyFeature(CustomTestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
+        cls.base_url = DEFAULT_URL_FOR_TEST
+        cls.process = popen_launch_server(
+            cls.model,
+            cls.base_url,
+            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+            other_args=["--arg1", "value1"],  # feature-specific args
+        )
+
+    @classmethod
+    def tearDownClass(cls):
+        if hasattr(cls, "process") and cls.process:
+            kill_process_tree(cls.process.pid)
+
+    def test_basic_functionality(self):
+        response = requests.post(
+            self.base_url + "/generate",
+            json={"text": "Hello", "sampling_params": {"max_new_tokens": 32}},
+        )
+        self.assertEqual(response.status_code, 200)
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=3)
+```
+
+### E2E test (8B model, server needed, performance)
+
+```python
+import time
+import unittest
+
+import requests
+
+from sglang.srt.utils import kill_process_tree
+from sglang.test.ci.ci_register import register_cuda_ci
+from sglang.test.test_utils import (
+    DEFAULT_MODEL_NAME_FOR_TEST,
+    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+    DEFAULT_URL_FOR_TEST,
+    CustomTestCase,
+    popen_launch_server,
+)
+
+register_cuda_ci(est_time=300, suite="stage-b-test-1-gpu-large")
+
+
+class TestMyFeaturePerf(CustomTestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.model = DEFAULT_MODEL_NAME_FOR_TEST
+        cls.base_url = DEFAULT_URL_FOR_TEST
+        cls.process = popen_launch_server(
+            cls.model,
+            cls.base_url,
+            timeout=DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,
+        )
+
+    @classmethod
+    def tearDownClass(cls):
+        if hasattr(cls, "process") and cls.process:
+            kill_process_tree(cls.process.pid)
+
+    def test_latency(self):
+        start = time.perf_counter()
+        response = requests.post(
+            self.base_url + "/generate",
+            json={"text": "Hello", "sampling_params": {"max_new_tokens": 128}},
+        )
+        elapsed = time.perf_counter() - start
+        self.assertEqual(response.status_code, 200)
+        self.assertLess(elapsed, 5.0, "Latency exceeded threshold")
+
+
+if __name__ == "__main__":
+    unittest.main(verbosity=3)
+```
+
+---
+
+## Server Fixture Reuse
+
+For tests that only need a standard server, inherit from `DefaultServerBase` and override class attributes:
+
+```python
+from sglang.test.server_fixtures.default_fixture import DefaultServerBase
+
+class TestMyFeature(DefaultServerBase):
+    model = DEFAULT_SMALL_MODEL_NAME_FOR_TEST
+    other_args = ["--enable-my-feature"]
+
+    def test_something(self):
+        ...
+```
+
+Available fixtures in `python/sglang/test/server_fixtures/`:
+
+| Fixture | Use case |
+|---------|----------|
+| `DefaultServerBase` | Standard single-server tests |
+| `EagleServerBase` | EAGLE speculative decoding |
+| `PDDisaggregationServerBase` | Disaggregated prefill/decode |
+| `MMMUServerBase` | Multimodal VLM tests |
+
+---
+
+## CI Registration
+
+Every CI-discovered test file must call a registration function at module level:
+
+```python
+from sglang.test.ci.ci_register import (
+    register_cuda_ci,
+    register_amd_ci,
+    register_cpu_ci,
+    register_npu_ci,
+)
+
+# Per-commit test (small 1-gpu, runs on 5090)
+register_cuda_ci(est_time=80, suite="stage-b-test-1-gpu-small")
+
+# Per-commit test (large 1-gpu, runs on H100)
+register_cuda_ci(est_time=120, suite="stage-b-test-1-gpu-large")
+
+# Nightly-only test
+register_cuda_ci(est_time=200, suite="nightly-1-gpu", nightly=True)
+
+# Multi-backend test (only when testing backend-specific code paths)
+register_cuda_ci(est_time=80, suite="stage-a-test-1-gpu-small")
+register_amd_ci(est_time=120, suite="stage-a-test-1-gpu-small-amd")
+register_npu_ci(est_time=400, suite="nightly-8-npu-a3", nightly=True)
+
+# Temporarily disabled test
+register_cuda_ci(est_time=80, suite="stage-b-test-1-gpu-small", disabled="flaky - see #12345")
+```
+
+Parameters:
+- `est_time`: estimated runtime in seconds (used for CI partitioning)
+- `suite`: which CI suite to run in (see suite tables above)
+- `nightly=True`: for nightly-only tests (default `False` = per-commit)
+- `disabled="reason"`: temporarily disable with explanation
+
+**Key principle**: Only add `register_amd_ci` / `register_npu_ci` when the test exercises backend-specific code paths. Common E2E tests just need `register_cuda_ci` — duplicating across backends wastes CI time.
+
+### JIT Kernel Registration
+
+JIT kernel files live outside `test/registered/` but still use registration:
+
+```python
+from sglang.test.ci.ci_register import register_cuda_ci
+
+# Correctness tests in python/sglang/jit_kernel/tests/
+register_cuda_ci(est_time=30, suite="stage-b-kernel-unit-1-gpu-large")
+register_cuda_ci(est_time=120, suite="stage-b-kernel-unit-8-gpu-h200")
+
+# Benchmarks in python/sglang/jit_kernel/benchmark/
+register_cuda_ci(est_time=6, suite="stage-b-kernel-benchmark-1-gpu-large")
+
+# Optional nightly registration
+register_cuda_ci(est_time=120, suite="nightly-kernel-1-gpu", nightly=True)
+register_cuda_ci(est_time=120, suite="nightly-kernel-8-gpu-h200", nightly=True)
+```
+
+Keep `est_time` and `suite` as **literal values** — `run_suite.py` collects them by AST parsing
+
+---
+
+## Test Placement
+
+```
+test/
+├── registered/          # CI tests (auto-discovered by run_suite.py)
+│   ├── unit/            # No server / engine launch (see test/registered/unit/README.md)
+│   ├── kernels/         # CUDA kernel correctness (no server, GPU required)
+│   ├── sampling/        # test_penalty.py, test_sampling_params.py ...
+│   ├── sessions/        # test_session_control.py ...
+│   ├── openai_server/   # basic/, features/, validation/ ...
+│   ├── spec/            # eagle/, utils/ ...
+│   ├── models/          # model-specific accuracy tests
+│   ├── perf/            # performance benchmarks
+│   └── <category>/      # create new category if needed
+├── manual/              # Non-CI: debugging, one-off, manual verification
+└── run_suite.py         # CI runner (scans registered/ plus jit_kernel test/benchmark files)
+
+python/sglang/jit_kernel/
+├── tests/               # JIT kernel correctness tests (CI-discovered by test/run_suite.py)
+└── benchmark/           # JIT kernel benchmarks (CI-discovered by test/run_suite.py)
+```
+
+**Decision rule** (see also `test/registered/README.md`):
+- Component logic, no server → `registered/unit/`
+- JIT kernel correctness / benchmarks → `python/sglang/jit_kernel/tests/` or `python/sglang/jit_kernel/benchmark/`
+- Other kernel correctness → `registered/kernels/`
+- Server needed → `registered/<category>/`
+- Local debugging → `manual/`
+
+---
+
+## Eval Accuracy Mixins
+
+**Design philosophy**: Most test files don't care about eval logic — they only need a "does this feature break model output quality?" sanity check. The mixin pattern separates **what to test** (threshold) from **how to test** (run_eval, assertions, CI summary). Test classes declare thresholds as class attributes; the mixin provides the `test_*` method. Override when you need extra assertions (e.g. EAGLE accept length).
+
+Available mixins in `python/sglang/test/kits/eval_accuracy_kit.py`: `MMLUMixin`, `HumanEvalMixin`, `MGSMEnMixin`, `GSM8KMixin`. Can be combined freely. Read the source for attrs and defaults.
+
+```python
+class TestMyFeature(CustomTestCase, MMLUMixin):
+    mmlu_score_threshold = 0.65
+    mmlu_num_examples = 64
+    mmlu_num_threads = 32
+    # test_mmlu is inherited — no code needed
+```
+
+---
+
+## Key Utilities
+
+```python
+from sglang.test.test_utils import (
+    CustomTestCase,              # base class with retry logic
+    popen_launch_server,         # launch server subprocess
+    DEFAULT_URL_FOR_TEST,        # auto-configured base URL
+    DEFAULT_TIMEOUT_FOR_SERVER_LAUNCH,  # 600s default
+    run_bench_serving,           # benchmark helper (launch + bench)
+)
+from sglang.srt.utils import kill_process_tree  # cleanup server
+```
+
+---
+
+## Checklist
+
+Before submitting a test:
+
+- [ ] Inherits from `CustomTestCase` (not `unittest.TestCase`)
+- [ ] Has `register_*_ci(...)` call at module level
+- [ ] Placed in `test/registered/<category>/`, unless this is a JIT kernel test/benchmark
+- [ ] JIT kernel work: files live in `python/sglang/jit_kernel/tests/` or `python/sglang/jit_kernel/benchmark/`
+- [ ] Backend-independent tests: `register_cuda_ci` only + smallest model
+- [ ] Logic that doesn't need a server / engine launch → unit test in `registered/unit/` (see Unit Tests section)
+- [ ] `setUpClass` launches server, `tearDownClass` kills it (if server-based)
+- [ ] `tearDownClass` is defensive — uses `hasattr`/null checks before accessing resources that may not have been allocated
+- [ ] Has `if __name__ == "__main__": unittest.main()`
+- [ ] `est_time` is reasonable (measure locally)

third_party/sglang/.codespellrc

@@ -0,0 +1,3 @@
+[codespell]
+ignore-words-list = ans, als, hel, boostrap, childs, te, vas, hsa, ment, cann, thi, makro, wil, rouge, PRIS
+skip = *.json,*.jsonl,*.patch,*.txt

third_party/sglang/.coveragerc

@@ -0,0 +1,16 @@
+[run]
+source = python/sglang/srt
+omit =
+    */test/*
+    */__pycache__/*
+
+[report]
+show_missing = true
+exclude_lines =
+    pragma: no cover
+    if __name__ == .__main__.:
+    raise NotImplementedError
+    if TYPE_CHECKING
+
+[html]
+directory = htmlcov

third_party/sglang/.devcontainer/Dockerfile

@@ -0,0 +1,35 @@
+FROM lmsysorg/sglang:dev
+
+# Create non-root user with specified UID and GID
+# NOTE: Replace with your own UID and GID. This is a workaround from https://github.com/microsoft/vscode-remote-release/issues/49#issuecomment-489060908.
+ARG HOST_UID=1003
+ARG HOST_GID=1003
+RUN groupadd -g $HOST_GID devuser && \
+    useradd -m -u $HOST_UID -g $HOST_GID -s /bin/zsh devuser
+
+# Give devuser sudo access
+RUN apt-get update && apt-get install -y sudo && \
+    echo "devuser ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/devuser && \
+    rm -rf /var/lib/apt/lists/* && \
+    apt-get clean
+
+# Set up oh-my-zsh for devuser
+RUN cp -r /root/.oh-my-zsh /home/devuser/.oh-my-zsh && \
+    cp /root/.zshrc /home/devuser/.zshrc && \
+    cp /root/.vimrc /home/devuser/.vimrc && \
+    cp /root/.tmux.conf /home/devuser/.tmux.conf && \
+    sed -i 's|/root/.oh-my-zsh|/home/devuser/.oh-my-zsh|g' /home/devuser/.zshrc && \
+    chown -R devuser:devuser /home/devuser/
+
+# Set workspace directory and ownership
+WORKDIR /sgl-workspace/sglang
+RUN chown -R devuser:devuser /sgl-workspace
+
+# Switch to devuser
+USER devuser
+
+# Install uv
+RUN curl -LsSf https://astral.sh/uv/install.sh | sh
+
+# Install rust
+RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

third_party/sglang/.devcontainer/devcontainer.json

@@ -0,0 +1,30 @@
+{
+    "name": "sglang",
+    "build": {
+        "dockerfile": "Dockerfile"
+    },
+    "remoteUser": "devuser",
+    "customizations": {
+        "vscode": {
+            "extensions": [
+                // Python development
+                "ms-python.python",
+                "charliermarsh.ruff",
+                // Rust development
+                "rust-lang.rust-analyzer",
+                "tamasfe.even-better-toml"
+            ]
+        }
+    },
+    "forwardPorts": [],
+    "runArgs": [
+        "--gpus",
+        "all"
+    ],
+    // The two lines below ensures that your local changes in the sglang
+    // repo is automatically synced to the sglang pip package installed
+    // in the dev docker container. You can remove / comment out these
+    // two lines if you prefer to sync code changes manually.
+    "workspaceMount": "source=${localWorkspaceFolder},target=/sgl-workspace/sglang,type=bind",
+    "workspaceFolder": "/sgl-workspace/sglang"
+}

third_party/sglang/.dockerignore

@@ -0,0 +1 @@
+.gitignore

third_party/sglang/.github/CODEOWNERS

@@ -0,0 +1,74 @@
+.github @merrymercy @Fridge003 @ispobock @Kangyan-Zhou @bingxche
+/docker @Fridge003 @ispobock @HaiShaw @ishandhanani @yctseng0211
+/docker/npu.Dockerfile @ping1jing2 @iforgetmyname
+/python/pyproject.toml @merrymercy @Fridge003 @ispobock
+/python/sglang/jit_kernel @DarkSharpness @BBuf @celve @HydraQYH @yuan-luo
+/python/sglang/jit_kernel/diffusion @yingluosanqian @BBuf @mickqian
+/python/sglang/multimodal_gen @mickqian @yhyang201 @ping1jing2
+/python/sglang/multimodal_gen/runtime/cache @DefTruth
+/python/sglang/multimodal_gen/runtime/layers @mickqian @yhyang201 @BBuf @yingluosanqian @ping1jing2
+/python/sglang/multimodal_gen/runtime/models/dits @mickqian @yhyang201 @BBuf @yingluosanqian @ping1jing2
+/python/sglang/srt/batch_invariant_ops @Fridge003 @hebiao064
+/python/sglang/srt/compilation @hebiao064 @Oasis-Git
+/python/sglang/srt/constrained @hnyls2002 @DarkSharpness
+/python/sglang/srt/disaggregation @ByronHsu @hnyls2002 @ShangmingCai
+/python/sglang/srt/disaggregation/ascend @ping1jing2 @iforgetmyname
+/python/sglang/srt/distributed @yizhang2077 @merrymercy @ch-wan
+/python/sglang/srt/distributed/device_communicators/mooncake_transfer_engine.py @ShangmingCai @stmatengss
+/python/sglang/srt/dllm @ClawSeven @btw616
+/python/sglang/srt/entrypoints @ispobock @CatherineSue @slin1237 @merrymercy @JustinTong0323
+/python/sglang/srt/entrypoints/engine_score_mixin.py @sundar24295s @chanh @fortunecookiee
+/python/sglang/srt/entrypoints/grpc_server.py @CatherineSue @slin1237
+/python/sglang/srt/entrypoints/openai/serving_score.py @sundar24295s @chanh @fortunecookiee
+/python/sglang/srt/eplb @fzyzcjy @ch-wan
+/python/sglang/srt/function_call @CatherineSue @JustinTong0323
+/python/sglang/srt/grpc @CatherineSue @slin1237
+/python/sglang/srt/hardware_backend/npu @ping1jing2 @iforgetmyname
+/python/sglang/srt/hardware_backend/npu/quantization @OrangeRedeng @TamirBaydasov @iforgetmyname
+/python/sglang/srt/layers @merrymercy @Ying1123 @Fridge003 @ispobock @HaiShaw @ch-wan @BBuf @Edwardf0t1
+/python/sglang/srt/layers/attention @merrymercy @Fridge003 @ispobock @Qiaolin-Yu @hebiao064 @HaiShaw
+/python/sglang/srt/layers/attention/fla @yizhang2077 @hebiao064 @yuan-luo
+/python/sglang/srt/layers/attention/hybrid_linear_attn_backend.py @yizhang2077 @hebiao064 @hanming-lu @yuan-luo
+/python/sglang/srt/layers/attention/mamba @yizhang2077 @hebiao064
+/python/sglang/srt/layers/attention/nsa @1am9trash @hubertlu-tw @kkHuang-amd @HaiShaw @Fridge003 @hlu1 @rainj-me
+/python/sglang/srt/layers/attention/vision.py @mickqian @yuan-luo @yhyang201
+/python/sglang/srt/layers/quantization @ch-wan @BBuf @Edwardf0t1 @FlamingoPg @AniZpZ @HaiShaw @b8zhong
+/python/sglang/srt/layers/quantization/quark @kkHuang-amd @yichiche @hubertlu-tw @1am9trash @BowenBao
+/python/sglang/srt/lora @Ying1123 @Fridge003 @lifuhuang @yushengsu-thu
+/python/sglang/srt/managers @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann
+/python/sglang/srt/managers/scheduler_pp_mixin.py @ShangmingCai @XucSh
+/python/sglang/srt/managers/tokenizer_manager_score_mixin.py @sundar24295s @chanh @fortunecookiee
+/python/sglang/srt/mem_cache @merrymercy @Ying1123 @hnyls2002 @xiezhq-hermann @hanming-lu @yizhang2077 @hzh0425 @ispobock
+/python/sglang/srt/model_executor @merrymercy @Ying1123 @hnyls2002 @Fridge003 @ispobock
+/python/sglang/srt/model_executor/piecewise_cuda_graph_runner.py @hebiao064
+/python/sglang/srt/models/deepseek_common @Fridge003 @ispobock @fzyzcjy @ch-wan
+/python/sglang/srt/models/deepseek_v2.py @fzyzcjy @zhyncs @ispobock @ch-wan @merrymercy @Fridge003
+/python/sglang/srt/models/transformers.py @adarshxs
+/python/sglang/srt/multimodal @mickqian @JustinTong0323 @yhyang201 @yuan-luo
+/python/sglang/srt/observability @merrymercy @fzyzcjy @sufeng-buaa
+/python/sglang/srt/ray @Qiaolin-Yu @xyuzh
+/python/sglang/srt/speculative @Ying1123 @merrymercy @hnyls2002
+/sgl-kernel @ispobock @BBuf @yizhang2077 @merrymercy @FlamingoPg @HaiShaw
+/sgl-model-gateway @slin1237 @CatherineSue
+/sgl-model-gateway/benches @slin1237
+/sgl-model-gateway/bindings/python @CatherineSue @key4ng @slin1237
+/sgl-model-gateway/e2e_test @CatherineSue @key4ng
+/sgl-model-gateway/examples/wasm @slin1237
+/sgl-model-gateway/src/config @slin1237
+/sgl-model-gateway/src/core @slin1237
+/sgl-model-gateway/src/data_connector @key4ng
+/sgl-model-gateway/src/grpc_client @CatherineSue @slin1237
+/sgl-model-gateway/src/mcp @key4ng @slin1237
+/sgl-model-gateway/src/policies @slin1237 @ByronHsu
+/sgl-model-gateway/src/proto @CatherineSue @slin1237
+/sgl-model-gateway/src/protocols @CatherineSue @key4ng
+/sgl-model-gateway/src/reasoning_parser @CatherineSue
+/sgl-model-gateway/src/routers @CatherineSue @key4ng @slin1237
+/sgl-model-gateway/src/tokenizer @slin1237 @CatherineSue
+/sgl-model-gateway/src/tool_parser @slin1237 @CatherineSue
+/sgl-model-gateway/src/wasm @slin1237
+/sgl-model-gateway/examples/wasm @slin1237
+/test/registered/core/test_score_api.py @sundar24295s @chanh @fortunecookiee
+/benchmark/prefill_only/bench_score.py @sundar24295s @chanh @fortunecookiee
+/test/srt/ascend @ping1jing2 @iforgetmyname
+/test/srt/test_modelopt* @Edwardf0t1

third_party/sglang/.github/FOLDER_README.md

@@ -0,0 +1,12 @@
+# Maintenance Tools
+
+This folder contains tools and workflows for automating maintenance tasks.
+
+## CI Permissions
+
+`CI_PERMISSIONS.json` defines the CI permissions granted to each user.
+Maintainers can directly edit the file to add entries with `"reason": "custom override"`.
+Maintainers can also run `update_ci_permission.py` to update it with some auto rules (e.g., top contributors in the last 90 days get full permissions).
+
+## Others
+- `MAINTAINER.md` defines the code maintenance model.

third_party/sglang/.github/ISSUE_TEMPLATE/1-bug-report.yml

@@ -0,0 +1,35 @@
+name: 🐞 Bug report
+description: Report a bug to help us reproduce and fix it.
+title: "[Bug] "
+labels: ['Bug']
+
+body:
+- type: checkboxes
+  attributes:
+    label: Checklist
+    options:
+      - label: I searched related issues but found no solution.
+      - label: The bug persists in the latest version.
+      - label: Issues without environment info and a minimal reproducible demo are hard to resolve and may receive no feedback.
+      - label: If this is not a bug report but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
+      - label: Please use English. Otherwise, it will be closed.
+- type: textarea
+  attributes:
+    label: Describe the bug
+    description: A clear, concise description of the bug.
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Reproduction
+    description: Command/script run and model used.
+    placeholder: Paste the command here.
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Environment
+    description: Run `python3 -m sglang.check_env` and paste output here. Issues without this will be closed.
+    placeholder: Paste environment output here.
+  validations:
+    required: true

third_party/sglang/.github/ISSUE_TEMPLATE/2-feature-request.yml

@@ -0,0 +1,23 @@
+name: 🚀 Feature request
+description: Suggest an idea for this project
+title: "[Feature] "
+
+body:
+- type: checkboxes
+  attributes:
+    label: Checklist
+    options:
+      - label: If this is not a feature request but a general question, please start a discussion at https://github.com/sgl-project/sglang/discussions. Otherwise, it will be closed.
+      - label: Please use English. Otherwise, it will be closed.
+- type: textarea
+  attributes:
+    label: Motivation
+    description: |
+      Clearly and concisely describe the feature's motivation.
+  validations:
+    required: true
+- type: textarea
+  attributes:
+    label: Related resources
+    description: |
+      Provide official releases or third-party implementations if available.

third_party/sglang/.github/MAINTAINER.md

@@ -0,0 +1,154 @@
+# SGLang Code Maintenance Model
+This document describes the code maintenance model for the SGLang project.
+Since SGLang is a large project involving multiple organizations and hardware platforms, we designed this model with the following goals:
+- Ensure a responsive and smooth review process.
+- Allow for fast iteration, so maintainers can sometimes bypass flaky CI tests for important PRs.
+
+## Role Descriptions
+There are four roles in this maintenance model. Some are custom roles, while others are predefined by GitHub.
+
+- **Merge Oncall**: The person who drives the PR merge process. They have strong area-specific expertise and uphold a high bar for code quality.
+  - Permission: Merge PRs. Bypass branch protection rules if needed.
+  - Responsibility: Shepherd the merge of PRs assigned to their area. Revert or hotfix any issues related to their merge (especially if they bypass).
+- **Codeowner**: The person who protects critical code. Without a bypass, each PR needs at least one Codeowner approval for each modified file protected by [CODEOWNERS](./CODEOWNERS). Please note that this role is not an honor but a significant responsibility because PRs cannot be merged without your approval (except when bypassed by a Merge Oncall).
+  - Permission: Approve PRs, allowing them to be merged without a bypass.
+  - Responsibility: Review PRs in a timely manner.
+- **Write**: A person with write permission to the SGLang repo.
+  - Permission: Merge PRs if they have passed required tests and been approved by Codeowners. This role cannot bypass branch protection rules.
+  - Responsibility: Review and merge PRs in a timely manner.
+- **CI Oncall**: A person who manages CI runners for specific hardware platforms.
+  - Permission: Add CI runners.
+  - Responsibility: Keep the CI runners up and running.
+
+__Note__: Difference between Merge Oncall and Codeowner
+- The Merge Oncall is an active role held by someone who actively tries to help merge PRs and can bypass CI if needed.
+- The Codeowner is a passive protection role provided by GitHub; it prevents accidental changes to critical code.
+- The list of Merge Oncalls is attached below. The list of Codeowners is in the [CODEOWNERS](./CODEOWNERS) file.
+
+__Note__: The permissions to trigger CI tests are defined separately according to these [rules](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests).
+
+
+## Pull Request Merge Process
+1. The author submits a pull request (PR) and fills out the PR checklist.
+2. A bot assigns this PR to a Merge Oncall and @-mentions them. At the same time, GitHub will automatically request reviews from Codeowners.
+3. Someone tags the PR with a `run-ci` label ([help](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests)). Then the author can trigger CI by pushing new commits.
+4. The Merge Oncall coordinates the review (e.g., asking people to review) and approves the PR; the Codeowners also approve the PR. If the assigned Merge Oncall is not responsive, the author can ping other related Merge Oncalls and Reviewers in the list below.
+5. The code can now be merged:
+   - **Ideal case:** For each modified file, one Codeowner has approved the PR. The PR has also passed the required CI tests. Then, anyone with write permission can merge the PR.
+   - **Exception:** In cases where it is difficult to meet all requirements (due to flaky CI or slow responses), a Merge Oncall can bypass branch protection to merge the PR.
+
+If you meet any issues during the merge, you can discuss in [slack channels](https://slack.sglang.io/): #pull-request, #ci-cd-build-release, #dev.
+
+## The List of Merge Oncalls and Reviewers
+This section lists the oncalls for each module or feature.
+The format is @github-username (Slack username).
+
+### Scheduler
+[@merrymercy](https://github.com/merrymercy) (Lianmin Zheng), [@hnyls2002](https://github.com/hnyls2002) (Liangsheng Yin), [@cctry](https://github.com/cctry) (Shiyang Chen)
+
+related files
+- python/sglang/srt/managers
+- python/sglang/srt/model_executor
+
+### Diffusion
+[@mickqian](https://github.com/mickqian) (Mick), [@BBuf](https://github.com/BBuf) (BBuf)
+
+related files
+- python/sglang/multimodal_gen
+
+### PD disaggregation
+[@ByronHsu](https://github.com/ByronHsu) (Byron Hsu), [@cctry](https://github.com/cctry) (Shiyang Chen), [@ShangmingCai](https://github.com/ShangmingCai) (Shangming Cai)
+
+related files
+- python/sglang/srt/disaggregation
+
+### KV Cache
+[@ispobock](https://github.com/ispobock) (Ke Bao), [@xiezhq-hermann](https://github.com/xiezhq-hermann) (Zhiqiang Xie)
+
+related files
+- python/sglang/srt/mem_cache
+
+### Parallelism
+[@ch-wan](https://github.com/ch-wan) (Cheng Wan), [@fzyzcjy](https://github.com/fzyzcjy) (Tom)
+
+related files
+- python/sglang/srt/eplb
+- python/sglang/srt/distributed
+- python/sglang/srt/layers/dp_attention.py
+
+### Kernel
+[@BBuf](https://github.com/BBuf) (BBuf)
+
+related files
+- python/sglang/jit_kernel
+- sgl-kernel
+
+### Speculative decoding
+[@hnyls2002](https://github.com/hnyls2002) (Liangsheng Yin), [@Qiaolin-Yu](https://github.com/Qiaolin-Yu) (Qiaolin Yu)
+
+related files
+- python/sglang/srt/speculative
+
+### NV and model-specific optimizations
+[@Fridge003](https://github.com/Fridge003) (Baizhou Zhang), [@ishandhanani](https://github.com/ishandhanani) (Ishan Dhanani), [@Qiaolin-Yu](https://github.com/Qiaolin-Yu) (Qiaolin Yu)
+
+related files
+- python/sglang/srt/models
+- python/sglang/srt/layers/attention
+
+### AMD optimizations
+[@HaiShaw](https://github.com/HaiShaw) (Henry HAI)
+
+### NPU optimizations
+[@iforgetmyname](https://github.com/iforgetmyname) (Even Zhou)
+
+related files
+- python/sglang/srt/hardware_backend/npu
+
+### CI, Release, Package
+[@Kangyan-Zhou](https://github.com/Kangyan-Zhou) (Kangyan Zhou), [@Fridge003](https://github.com/Fridge003) (Baizhou Zhang)
+
+related files
+- .github/workflows
+
+### Router, API
+[@slin1237](https://github.com/slin1237) (Simo Lin)
+
+related files
+- sgl-model-gateway
+- python/sglang/srt/grpc
+- python/sglang/srt/entrypoints
+
+### Other Notes
+
+Now we have many Merge Oncalls mainly because the CI is flaky and the CODEOWNERS is too coarse-grained.
+In the future, we hope the CI can be improved and we only need bypass rarely. After that, most Merge Oncalls can be converted back to Write and CODEOWNERS.
+
+This list is based on the current situation. If you or someone you know would like to take on more responsibility and are qualified, please ping [Lianmin Zheng](https://github.com/merrymercy) and [Ying Sheng](https://github.com/Ying1123) in the Slack channel. They will start a nomination and internal review process.
+
+## The List of CI Oncalls
+This section lists the oncalls for each hardware platform. The format is @github-username (Slack username).
+
+### NVIDIA GPUs
+[@Kangyan-Zhou](https://github.com/Kangyan-Zhou) (Kangyan Zhou), [@ch-wan](https://github.com/ch-wan) (Cheng Wan), [@HanHan009527](https://github.com/HanHan009527) (hanhan), [@ishandhanani](https://github.com/ishandhanani) (Ishan Dhanani), [@ShangmingCai](https://github.com/ShangmingCai) (Shangming Cai), [@alisonshao](https://github.com/alisonshao) (Alison Shao).
+
+### AMD GPUs
+[@saienduri](https://github.com/saienduri) (Sai Enduri), [@HaiShaw](https://github.com/HaiShaw) (Henry HAI)
+
+### Intel CPU and XPU
+[@mingfeima](https://github.com/mingfeima) (Mingfei Ma), [@DiweiSun](https://github.com/DiweiSun) (Diwei Sun)
+
+### Ascend NPUs
+[@iforgetmyname](https://github.com/iforgetmyname) (Even Zhou)
+
+This list is based on the current situation. If you or someone you know would like to donate machines for CI, they can serve as the CI oncalls for their machines. Please ping [Lianmin Zheng](https://github.com/merrymercy) and [Ying Sheng](https://github.com/Ying1123) in the Slack channel. They will start a nomination and internal review process.
+
+## CI Maintenance Mode
+When the CI is unhealthy (e.g., the scheduled pr-test on `main` is broken for consecutive runs), the project enters **CI Maintenance Mode** by opening [issue #21065](https://github.com/sgl-project/sglang/issues/21065). While active:
+- All PR CI runs are paused. Resources are allocated to PRs that fix the CI.
+- **Merging non-CI-fix PRs is prohibited.** Only PRs that fix the CI may be merged. In severe cases, merge permissions may be revoked.
+
+Maintenance mode ends when `pr-test.yml` is all green on `main` and the issue is closed.
+
+## Suspending Permissions
+If a Merge Oncall bypasses checks to merge a PR that breaks the `main` branch, merges a non-CI-fix PR during CI Maintenance Mode, or repeatedly breaks the CI due to various reasons, their privileges will be suspended for at least two days, depending on the severity of the incident.

third_party/sglang/.github/actions/check-maintenance/action.yml

@@ -0,0 +1,63 @@
+name: Check Maintenance Mode
+description: Blocks CI when maintenance mode is active (issue #21065 is open), unless the PR has the bypass-maintenance label, or env SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN=true (PR Test workflow on main only). Merging non-CI-fix PRs is prohibited during maintenance mode; in severe cases, merge permissions may be revoked.
+
+inputs:
+  github-token:
+    description: GitHub token for API access
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Check maintenance mode
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.github-token }}
+      run: |
+        MAINTENANCE_ISSUE=21065
+        REPO="${{ github.repository }}"
+        PR_NUMBER="${{ github.event.pull_request.number }}"
+
+        # PR Test workflow only: scheduled runs and runs on main (dispatch / workflow_call) set this env
+        if [[ "${SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN:-}" == "true" ]]; then
+          echo "✅ PR Test on main branch; bypassing maintenance gate."
+          exit 0
+        fi
+
+        # Check if maintenance issue is open (fail-open: if API errors, allow CI to proceed)
+        ISSUE_STATE=$(gh issue view "$MAINTENANCE_ISSUE" --repo "$REPO" --json state --jq '.state' 2>/dev/null || echo "UNKNOWN")
+
+        if [[ "$ISSUE_STATE" != "OPEN" ]]; then
+          echo "✅ Maintenance mode is OFF. Proceeding with CI."
+          exit 0
+        fi
+
+        # For PRs, check if bypass-maintenance label is present
+        if [[ -n "$PR_NUMBER" ]]; then
+          HAS_BYPASS=$(gh pr view "$PR_NUMBER" --repo "$REPO" --json labels --jq '[.labels[].name] | map(select(. == "bypass-maintenance")) | length' 2>/dev/null || echo "0")
+          if [[ "$HAS_BYPASS" -gt 0 ]]; then
+            echo "✅ PR #$PR_NUMBER has 'bypass-maintenance' label. Bypassing maintenance mode."
+            exit 0
+          fi
+        fi
+
+        MSG=$(printf "%s\n" \
+          "## ⚠️ CI Maintenance Mode is Active" \
+          "The CI infrastructure is currently under maintenance." \
+          "All PR CI runs are paused until maintenance is complete." \
+          "**Merging non-CI-fix PRs is prohibited during maintenance mode.** In severe cases, merge permissions may be revoked." \
+          "You might also experience unexpected failures during this period." \
+          "The team is working on the issue and will update the status as soon as possible." \
+          "" \
+          "What should you do?" \
+          "- **Do NOT merge non-CI-fix PRs** until maintenance mode is lifted" \
+          "- Check back later (~12 hours)" \
+          "- Follow CI Maintenance Mode issue: https://github.com/$REPO/issues/$MAINTENANCE_ISSUE for status updates")
+
+        echo "$MSG" >> "$GITHUB_STEP_SUMMARY"
+        while IFS= read -r line; do
+          echo "::error::$line"
+        done <<< "$MSG"
+
+        exit 1

third_party/sglang/.github/actions/check-stage-health/action.yml

@@ -0,0 +1,50 @@
+name: Check Stage Health
+description: Fail fast if any job in the current workflow run has already failed. Auto-skips for scheduled runs.
+
+inputs:
+  github-token:
+    description: 'GitHub token for API calls'
+    required: false
+    default: ${{ github.token }}
+
+runs:
+  using: composite
+  steps:
+    - name: Check stage health
+      uses: actions/github-script@v7
+      env:
+        SKIP_STAGE_HEALTH_CHECK: ${{ env.SKIP_STAGE_HEALTH_CHECK }}
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          // Skip when explicitly requested via env var (e.g. release branch cut)
+          if (process.env.SKIP_STAGE_HEALTH_CHECK === 'true') {
+            core.info('Skipping health check (SKIP_STAGE_HEALTH_CHECK=true)');
+            return;
+          }
+
+          // Skip for scheduled runs — they should collect all failures, not fast-fail
+          if (context.eventName === 'schedule') {
+            core.info('Skipping health check for scheduled run');
+            return;
+          }
+
+          const jobs = await github.paginate(github.rest.actions.listJobsForWorkflowRun, {
+            owner: context.repo.owner,
+            repo: context.repo.repo,
+            run_id: context.runId,
+            per_page: 100,
+          });
+          // Find jobs that failed from a real error, not from fast-fail cascade
+          const rootCauseFailures = jobs.filter(j => {
+            if (j.status !== 'completed' || j.conclusion !== 'failure') return false;
+            // If the failing step is the health check, it's a cascade — skip it
+            const failedStep = (j.steps || []).find(s => s.conclusion === 'failure');
+            if (failedStep && (failedStep.name.includes('check-stage-health') || failedStep.name.includes('Check stage health'))) {
+              return false;
+            }
+            return true;
+          });
+          if (rootCauseFailures.length > 0) {
+            core.setFailed(`Fast-fail: skipping — root cause job(s): ${rootCauseFailures.map(j => j.name).join(', ')}`);
+          }

third_party/sglang/.github/actions/upload-cuda-coredumps/action.yml

@@ -0,0 +1,27 @@
+name: Upload CUDA Coredumps
+description: Upload CUDA coredump files as artifacts and clean up the directory.
+
+inputs:
+  artifact-suffix:
+    description: Suffix appended to the artifact name (e.g. matrix partition id)
+    required: false
+    default: ""
+  retention-days:
+    description: Number of days to retain the artifact
+    required: false
+    default: "7"
+
+runs:
+  using: composite
+  steps:
+    - name: Upload CUDA coredumps
+      uses: actions/upload-artifact@v4
+      with:
+        name: cuda-coredumps-${{ github.job }}${{ inputs.artifact-suffix && format('-{0}', inputs.artifact-suffix) }}
+        path: ${{ env.SGLANG_CUDA_COREDUMP_DIR || '/tmp/sglang_cuda_coredumps' }}/
+        retention-days: ${{ inputs.retention-days }}
+        if-no-files-found: ignore
+
+    - name: Cleanup CUDA coredumps
+      shell: bash
+      run: rm -rf "${{ env.SGLANG_CUDA_COREDUMP_DIR || '/tmp/sglang_cuda_coredumps' }}"

third_party/sglang/.github/actions/wait-for-jobs/action.yml

@@ -0,0 +1,177 @@
+name: Wait for Jobs
+description: Poll and wait for specified jobs in the current workflow run to complete
+
+inputs:
+  stage-name:
+    description: 'Human-readable stage name for log messages (e.g. "stage-a")'
+    required: true
+  jobs:
+    description: |
+      JSON array of job specs to wait for. Each element is either:
+        - a string: exact job name (e.g. "stage-a-test-1-gpu-small")
+        - an object { "prefix": "...", "expected_count": N }: for matrix jobs
+    required: true
+  max-wait-minutes:
+    description: 'Maximum time to wait before timing out'
+    required: false
+    default: '240'
+  poll-interval-seconds:
+    description: 'Seconds between polling attempts'
+    required: false
+    default: '60'
+  github-token:
+    description: 'GitHub token for API calls'
+    required: false
+    default: ${{ github.token }}
+
+outputs:
+  result:
+    description: 'Overall result: success, failure, or timeout'
+    value: ${{ steps.wait.outputs.result }}
+
+runs:
+  using: composite
+  steps:
+    - name: Wait for jobs to complete
+      id: wait
+      uses: actions/github-script@v7
+      env:
+        INPUT_STAGE_NAME: ${{ inputs.stage-name }}
+        INPUT_JOBS: ${{ inputs.jobs }}
+        INPUT_MAX_WAIT_MINUTES: ${{ inputs.max-wait-minutes }}
+        INPUT_POLL_INTERVAL_SECONDS: ${{ inputs.poll-interval-seconds }}
+      with:
+        github-token: ${{ inputs.github-token }}
+        script: |
+          const stageName = process.env.INPUT_STAGE_NAME;
+          const jobSpecs = JSON.parse(process.env.INPUT_JOBS);
+          const maxWaitMinutes = parseInt(process.env.INPUT_MAX_WAIT_MINUTES);
+          const pollIntervalSeconds = parseInt(process.env.INPUT_POLL_INTERVAL_SECONDS);
+          const maxAttempts = (maxWaitMinutes * 60) / pollIntervalSeconds;
+
+          // Normalize job specs into a uniform format
+          const normalizedSpecs = jobSpecs.map(spec => {
+            if (typeof spec === 'string') {
+              return { prefix: spec, expected_count: 1, exact: true };
+            }
+            return { ...spec, exact: false };
+          });
+
+          const totalExpectedJobs = normalizedSpecs.reduce((sum, s) => sum + s.expected_count, 0);
+
+          const matchesSpec = (jobName, spec) => {
+            if (spec.exact) {
+              return jobName === spec.prefix;
+            }
+            return jobName === spec.prefix || jobName.startsWith(spec.prefix + ' (');
+          };
+
+          // Use ETag conditional requests to avoid consuming rate limit when nothing changed.
+          // GitHub returns 304 Not Modified for unchanged data, which is FREE (no rate limit cost).
+          let lastEtag = '';
+          let lastJobs = null;
+          let apiCalls = 0;
+          let cachedCalls = 0;
+
+          async function fetchJobs() {
+            const url = `GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs`;
+            const params = {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              run_id: context.runId,
+              per_page: 100,
+              headers: {},
+            };
+            if (lastEtag) {
+              params.headers['if-none-match'] = lastEtag;
+            }
+
+            try {
+              const response = await github.request(url, params);
+              apiCalls++;
+              const rateRemaining = response.headers['x-ratelimit-remaining'] || '?';
+              const rateLimit = response.headers['x-ratelimit-limit'] || '?';
+              console.log(`[rate-limit] ${rateRemaining}/${rateLimit} remaining (ETag: ${lastEtag ? 'sent' : 'none'}) | this session: ${apiCalls} paid, ${cachedCalls} free`);
+              lastEtag = response.headers.etag || '';
+              const jobs = response.data.jobs;
+
+              // Handle pagination if >100 jobs
+              // ETag only covers page 1, so invalidate it to avoid stale cache
+              // when later pages change but page 1 doesn't.
+              if (response.data.total_count > 100) {
+                lastEtag = '';
+                for (let page = 2; page <= Math.ceil(response.data.total_count / 100); page++) {
+                  const { data: pageData } = await github.request(url, {
+                    ...params,
+                    page,
+                    headers: {},
+                  });
+                  jobs.push(...pageData.jobs);
+                }
+              }
+
+              lastJobs = jobs;
+              return { jobs, cached: false };
+            } catch (err) {
+              if (err.status === 304 && lastJobs) {
+                cachedCalls++;
+                console.log(`[rate-limit] 304 Not Modified | this session: ${apiCalls} paid, ${cachedCalls} free`);
+                return { jobs: lastJobs, cached: true };
+              }
+              throw err;
+            }
+          }
+
+          for (let attempt = 0; attempt < maxAttempts; attempt++) {
+            const { jobs, cached } = await fetchJobs();
+
+            let allCompleted = true;
+            let failedJobs = [];
+            let completedCount = 0;
+            let totalCount = 0;
+
+            for (const spec of normalizedSpecs) {
+              const matchingJobs = jobs.filter(job => matchesSpec(job.name, spec));
+
+              for (const job of matchingJobs) {
+                totalCount++;
+                if (!cached) {
+                  console.log(`${job.name}: status=${job.status}, conclusion=${job.conclusion}`);
+                }
+
+                if (job.status === 'completed') {
+                  completedCount++;
+                  if (job.conclusion !== 'success' && job.conclusion !== 'skipped') {
+                    failedJobs.push(job.name);
+                  }
+                } else {
+                  allCompleted = false;
+                }
+              }
+
+              if (matchingJobs.length < spec.expected_count) {
+                console.log(`${spec.prefix}: found ${matchingJobs.length}/${spec.expected_count} jobs (waiting for more)`);
+                allCompleted = false;
+              }
+            }
+
+            console.log(`[${stageName}] Progress: ${completedCount}/${totalCount} jobs completed (expected ${totalExpectedJobs})${cached ? ' (cached, no rate limit cost)' : ''}`);
+
+            // Fail fast if any jobs failed
+            if (failedJobs.length > 0) {
+              core.setOutput('result', 'failure');
+              core.setFailed(`${stageName} jobs failed: ${failedJobs.join(', ')}`);
+              return;
+            }
+
+            if (allCompleted && totalCount >= totalExpectedJobs) {
+              core.setOutput('result', 'success');
+              return;
+            }
+
+            console.log(`Waiting ${pollIntervalSeconds}s... (attempt ${attempt + 1}/${maxAttempts})`);
+            await new Promise(resolve => setTimeout(resolve, pollIntervalSeconds * 1000));
+          }
+
+          core.setFailed(`Timeout waiting for ${stageName} jobs`);
+          core.setOutput('result', 'timeout');

third_party/sglang/.github/audit_permission.py

@@ -0,0 +1,411 @@
+"""
+Audit GitHub repository collaborators with elevated access.
+
+This script will:
+1. Fetch all collaborators with write permission to this repo.
+2. Show their github username, Nickname and the role (e.g., admin, maintain,
+   custom org role, write, triage).
+3. Show their last activity related to this repo (last commit, last issue,
+   last pull request). Put the data in YYYY-MM-DD format. Add a column "last activity date" to the CSV, before the above three breakdown columns.
+4. Show activity on other repos: repos touched via public events in the last 90 days (Push, PR, Issues, etc.). Sort the repos by the number of activities.
+5. Write results to a CSV sorted by the roles (admin, maintain, custom org role, write, triage) and the last activity date (most recent first).
+
+Usage:
+    export GH_TOKEN="your_github_token"
+    python3 audit_permission.py [--output path] [--repo owner/name]
+
+Requires: requests, and a token with permission to list collaborators (push+
+access to the repo).
+"""
+
+from __future__ import annotations
+
+import argparse
+import csv
+import os
+import sys
+import time
+from collections import Counter
+from datetime import datetime, timedelta, timezone
+from typing import Any
+
+try:
+    import requests
+except ImportError:
+    requests = None  # type: ignore
+
+DEFAULT_OWNER = "sgl-project"
+DEFAULT_NAME = "sglang"
+
+HEADERS: dict[str, str] = {}
+
+
+def _request(
+    method: str,
+    url: str,
+    *,
+    params: dict[str, Any] | None = None,
+    max_retries: int = 3,
+) -> requests.Response:
+    if requests is None:
+        raise RuntimeError("Install the requests package: pip install requests")
+    for attempt in range(max_retries):
+        r = requests.request(method, url, headers=HEADERS, params=params, timeout=60)
+        if r.status_code == 403 and "rate limit" in (r.text or "").lower():
+            reset = r.headers.get("X-RateLimit-Reset")
+            wait = 60
+            if reset:
+                try:
+                    wait = max(1, int(reset) - int(time.time()) + 2)
+                except ValueError:
+                    pass
+            print(f"Rate limited; sleeping {wait}s...", file=sys.stderr)
+            time.sleep(min(wait, 3600))
+            continue
+        return r
+    return r
+
+
+def paginate_list(url: str, params: dict[str, Any] | None = None) -> list[Any]:
+    out: list[Any] = []
+    next_url: str | None = url
+    next_params = params
+    while next_url:
+        r = _request("GET", next_url, params=next_params)
+        next_params = None
+        if r.status_code != 200:
+            print(
+                f"Error {r.status_code} GET {next_url}: {r.text[:500]}",
+                file=sys.stderr,
+            )
+            break
+        data = r.json()
+        if isinstance(data, list):
+            out.extend(data)
+        else:
+            break
+        next_url = None
+        link = r.headers.get("Link", "")
+        for part in link.split(", "):
+            if 'rel="next"' in part:
+                start = part.find("<") + 1
+                end = part.find(">")
+                if start > 0 and end > start:
+                    next_url = part[start:end]
+                break
+    return out
+
+
+def collaborator_role(collab: dict[str, Any]) -> str:
+    role_name = collab.get("role_name")
+    if isinstance(role_name, str) and role_name.strip():
+        return role_name.strip()
+    perms = collab.get("permissions") or {}
+    if perms.get("admin"):
+        return "admin"
+    if perms.get("maintain"):
+        return "maintain"
+    if perms.get("push"):
+        return "write"
+    if perms.get("triage"):
+        return "triage"
+    return "read"
+
+
+def has_write_plus(collab: dict[str, Any]) -> bool:
+    perms = collab.get("permissions") or {}
+    return bool(
+        perms.get("admin")
+        or perms.get("maintain")
+        or perms.get("push")
+        or perms.get("triage")
+    )
+
+
+def role_sort_tier(collab: dict[str, Any]) -> int:
+    """Sort order: admin (0), maintain (1), custom org role (2), write (3), triage (4)."""
+    rn = collab.get("role_name")
+    if isinstance(rn, str) and rn.strip():
+        k = rn.strip().lower()
+        if k == "admin":
+            return 0
+        if k == "maintain":
+            return 1
+        if k == "write":
+            return 3
+        if k == "triage":
+            return 4
+        if k == "read":
+            return 5
+        return 2
+    perms = collab.get("permissions") or {}
+    if perms.get("admin"):
+        return 0
+    if perms.get("maintain"):
+        return 1
+    if perms.get("push"):
+        return 3
+    if perms.get("triage"):
+        return 4
+    return 5
+
+
+def fetch_display_name(login: str) -> str:
+    url = f"https://api.github.com/users/{login}"
+    r = _request("GET", url)
+    if r.status_code != 200:
+        return ""
+    data = r.json()
+    if not isinstance(data, dict):
+        return ""
+    n = data.get("name")
+    return n.strip() if isinstance(n, str) else ""
+
+
+def parse_github_ts(s: str) -> datetime | None:
+    if not s:
+        return None
+    s = s.replace("Z", "+00:00")
+    try:
+        return datetime.fromisoformat(s)
+    except ValueError:
+        return None
+
+
+def iso_timestamp_to_ymd(iso: str | None) -> str:
+    if not iso:
+        return ""
+    p = parse_github_ts(iso)
+    if not p:
+        return ""
+    return p.date().isoformat()
+
+
+def max_date_ymd(*iso_dates: str | None) -> str:
+    best: datetime | None = None
+    for d in iso_dates:
+        p = parse_github_ts(d or "")
+        if p and (best is None or p > best):
+            best = p
+    return best.date().isoformat() if best else ""
+
+
+def parse_ymd(s: str) -> datetime | None:
+    if not s:
+        return None
+    try:
+        return datetime.strptime(s, "%Y-%m-%d").replace(tzinfo=timezone.utc)
+    except ValueError:
+        return None
+
+
+def last_commit_date(owner: str, repo: str, login: str) -> str | None:
+    url = f"https://api.github.com/repos/{owner}/{repo}/commits"
+    r = _request("GET", url, params={"author": login, "per_page": 1})
+    if r.status_code != 200:
+        return None
+    data = r.json()
+    if not isinstance(data, list) or not data:
+        return None
+    commit = data[0].get("commit") or {}
+    c = commit.get("committer") or commit.get("author") or {}
+    d = c.get("date")
+    return d if isinstance(d, str) else None
+
+
+def search_repo_item(
+    owner: str, repo: str, login: str, kind: str
+) -> dict[str, Any] | None:
+    q = f"repo:{owner}/{repo} is:{kind} author:{login}"
+    url = "https://api.github.com/search/issues"
+    r = _request(
+        "GET",
+        url,
+        params={"q": q, "sort": "updated", "order": "desc", "per_page": 1},
+    )
+    if r.status_code != 200:
+        return None
+    payload = r.json()
+    items = payload.get("items")
+    if not items:
+        return None
+    return items[0] if isinstance(items[0], dict) else None
+
+
+def last_issue_pr_dates(
+    owner: str, repo: str, login: str
+) -> tuple[str | None, str | None]:
+    issue = search_repo_item(owner, repo, login, "issue")
+    pr = search_repo_item(owner, repo, login, "pr")
+    issue_dt = None
+    pr_dt = None
+    if issue:
+        issue_dt = issue.get("updated_at") or issue.get("created_at")
+        if not isinstance(issue_dt, str):
+            issue_dt = None
+    if pr:
+        pr_dt = pr.get("updated_at") or pr.get("created_at")
+        if not isinstance(pr_dt, str):
+            pr_dt = None
+    return issue_dt, pr_dt
+
+
+def other_repos_activity_column(
+    login: str, owner: str, repo: str, days: int = 90
+) -> str:
+    """Repos other than this one touched in the window, sorted by event count (desc)."""
+    cutoff = datetime.now(timezone.utc) - timedelta(days=days)
+    full = f"{owner}/{repo}"
+    counts: Counter[str] = Counter()
+    url: str | None = f"https://api.github.com/users/{login}/events/public"
+    params: dict[str, Any] = {"per_page": 100}
+
+    while url:
+        r = _request("GET", url, params=params)
+        params = {}
+        if r.status_code != 200:
+            break
+        events = r.json()
+        if not isinstance(events, list):
+            break
+        oldest_in_page: datetime | None = None
+        for ev in events:
+            if not isinstance(ev, dict):
+                continue
+            created = parse_github_ts(ev.get("created_at") or "")
+            if created:
+                if oldest_in_page is None or created < oldest_in_page:
+                    oldest_in_page = created
+            if created and created < cutoff:
+                continue
+            rinfo = ev.get("repo")
+            name = None
+            if isinstance(rinfo, dict):
+                name = rinfo.get("name")
+            if isinstance(name, str) and name and name != full:
+                counts[name] += 1
+        next_url = None
+        link = r.headers.get("Link", "")
+        for part in link.split(", "):
+            if 'rel="next"' in part:
+                s, e = part.find("<") + 1, part.find(">")
+                if s > 0 and e > s:
+                    next_url = part[s:e]
+                break
+        if oldest_in_page and oldest_in_page < cutoff:
+            break
+        url = next_url
+        if not events:
+            break
+
+    ordered = sorted(counts.items(), key=lambda x: (-x[1], x[0]))
+    return ";".join(f"{n}:{c}" for n, c in ordered)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Audit repo collaborator permissions.")
+    parser.add_argument(
+        "--repo",
+        default=f"{DEFAULT_OWNER}/{DEFAULT_NAME}",
+        help=f"owner/name (default: {DEFAULT_OWNER}/{DEFAULT_NAME})",
+    )
+    parser.add_argument(
+        "--output",
+        "-o",
+        default=os.path.join(os.path.dirname(__file__), "permission_audit.csv"),
+        help="Output CSV path",
+    )
+    parser.add_argument(
+        "--events-days",
+        type=int,
+        default=90,
+        help="Window for other-repo activity via public events",
+    )
+    args = parser.parse_args()
+
+    if "/" not in args.repo:
+        print("Error: --repo must be owner/name", file=sys.stderr)
+        sys.exit(1)
+    owner, name = args.repo.split("/", 1)
+
+    gh_token = os.getenv("GH_TOKEN")
+    if not gh_token:
+        print("Error: GH_TOKEN environment variable is not set.", file=sys.stderr)
+        sys.exit(1)
+
+    global HEADERS
+    HEADERS = {
+        "Authorization": f"Bearer {gh_token}",
+        "Accept": "application/vnd.github+json",
+        "X-GitHub-Api-Version": "2022-11-28",
+    }
+
+    collab_url = f"https://api.github.com/repos/{owner}/{name}/collaborators"
+    print(f"Fetching collaborators for {owner}/{name}...", file=sys.stderr)
+    collaborators = paginate_list(
+        collab_url, params={"per_page": 100, "affiliation": "all"}
+    )
+
+    rows: list[dict[str, Any]] = []
+    elevated = [c for c in collaborators if isinstance(c, dict) and has_write_plus(c)]
+    print(
+        f"Found {len(elevated)} collaborators with admin/maintain/write/triage.",
+        file=sys.stderr,
+    )
+
+    for i, col in enumerate(elevated, start=1):
+        login = col.get("login")
+        if not isinstance(login, str):
+            continue
+        print(f"  [{i}/{len(elevated)}] {login}", file=sys.stderr)
+
+        role = collaborator_role(col)
+        nickname = fetch_display_name(login)
+        cd = last_commit_date(owner, name, login)
+        issue_dt, pr_dt = last_issue_pr_dates(owner, name, login)
+        last_act_ymd = max_date_ymd(cd, issue_dt, pr_dt)
+        others = other_repos_activity_column(login, owner, name, days=args.events_days)
+        rows.append(
+            {
+                "_role_tier": role_sort_tier(col),
+                "github_username": login,
+                "nickname": nickname,
+                "role": role,
+                "last_activity_date": last_act_ymd,
+                "last_commit_date": iso_timestamp_to_ymd(cd),
+                "last_issue_date": iso_timestamp_to_ymd(issue_dt),
+                "last_pr_date": iso_timestamp_to_ymd(pr_dt),
+                "other_repos_90d": others,
+            }
+        )
+
+    def sort_key(r: dict[str, Any]) -> tuple[int, float]:
+        tier = r["_role_tier"]
+        act = parse_ymd(r.get("last_activity_date") or "")
+        ts = act.timestamp() if act else 0.0
+        return (tier, -ts)
+
+    rows.sort(key=sort_key)
+
+    fieldnames = [
+        "github_username",
+        "nickname",
+        "role",
+        "last_activity_date",
+        "last_commit_date",
+        "last_issue_date",
+        "last_pr_date",
+        "other_repos_90d",
+    ]
+    for r in rows:
+        del r["_role_tier"]
+    with open(args.output, "w", newline="", encoding="utf-8") as f:
+        w = csv.DictWriter(f, fieldnames=fieldnames)
+        w.writeheader()
+        w.writerows(rows)
+
+    print(f"Wrote {len(rows)} rows to {args.output}", file=sys.stderr)
+
+
+if __name__ == "__main__":
+    main()

third_party/sglang/.github/labeler.yml

@@ -0,0 +1,122 @@
+# Configuration for the GitHub Labeler action
+# Automatically adds labels to PRs based on the files changed
+
+# Router specific (Rust code in sgl-model-gateway)
+model-gateway:
+  - changed-files:
+    - any-glob-to-any-file: 'sgl-model-gateway/**/*'
+
+# Kernel specific
+sgl-kernel:
+  - changed-files:
+    - any-glob-to-any-file: 'sgl-kernel/**/*'
+
+# JIT kernel specific
+jit-kernel:
+  - changed-files:
+    - any-glob-to-any-file: 'python/sglang/jit_kernel/**/*'
+
+# Documentation
+documentation:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*.md'
+      - 'docs/**/*'
+      - 'README*'
+
+# Dependencies
+dependencies:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/requirements*.txt'
+      - '**/Cargo.toml'
+      - '**/Cargo.lock'
+      - '**/pyproject*.toml'
+      - '**/setup.py'
+      - '**/poetry.lock'
+      - '**/package.json'
+      - '**/package-lock.json'
+
+# Multi-modal
+Multi-modal:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*multimodal*'
+      - '**/*vision*'
+      - '**/*vlm*'
+
+# Diffusion
+diffusion:
+  - changed-files:
+    - any-glob-to-any-file: 'python/sglang/multimodal_gen/**/*'
+
+# LoRA
+lora:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*lora*'
+
+# Quantization
+quant:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*quant*'
+      - '**/*quantization*'
+
+# Speculative decoding
+speculative-decoding:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*speculative*'
+
+# AMD specific
+amd:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*amd*'
+      - '**/*rocm*'
+
+# NPU specific
+npu:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*npu*'
+      - '**/*ascend*'
+
+# Blackwell
+blackwell:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*nvfp4*'
+      - 'sgl-kernel/csrc/attention/cutlass_sm100_mla/**/*'
+      - 'python/sglang/srt/layers/attention/trtllm_mla_backend.py'
+      - 'python/sglang/srt/layers/attention/trtllm_mha_backend.py'
+
+# DeepSeek specific
+deepseek:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*deepseek*'
+
+# HiCache
+hicache:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*hicache*'
+
+# Deterministic
+deterministic:
+  - changed-files:
+    - any-glob-to-any-file: 'python/sglang/srt/batch_invariant_ops/**/*'
+
+# Piecewise CUDA Graph
+piecewise-cuda-graph:
+  - changed-files:
+    - any-glob-to-any-file: 'python/sglang/srt/compilation/**/*'
+
+# Moore Threads specific
+mthreads:
+  - changed-files:
+    - any-glob-to-any-file:
+      - '**/*mthreads*'
+      - '**/*musa*'

third_party/sglang/.github/linters/lychee-ci.toml

@@ -0,0 +1,42 @@
+no_progress = true
+verbose = "warn"
+timeout = 20
+max_concurrency = 8
+retry_wait_time = 2
+max_retries = 2
+
+# CI should validate external links over the network.
+offline = false
+scheme = ["http", "https"]
+
+exclude_path = [
+  # Exclude generated Sphinx build artifacts.
+  # - "(\\./)?" allows both "docs/..." and "./docs/..."
+  # - "[/\\\\]" supports both slash styles in CI environments
+  "^(\\./)?docs[/\\\\]_build[/\\\\]",
+]
+
+exclude = [
+  # Local-only endpoints referenced in docs/examples.
+  # These are expected to be unreachable in GitHub-hosted CI.
+  "^https?://localhost(:[0-9]+)?(/|$)",
+  "^http://127\\.0\\.0\\.1(:[0-9]+)?(/|$)",
+  # Vendor pages that frequently block/deny CI user-agents (transient 403/anti-bot).
+  "^https://www\\.intel\\.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics\\.html$",
+  "^https://www\\.intel\\.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics\\.html$",
+  "^https://www\\.intel\\.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications\\.html$",
+
+  # Non-routable bind address used in examples, never externally reachable.
+  "^http://0\\.0\\.0\\.0(/|$)",
+
+  # Large doc portals with anti-bot/rate-limit behavior in CI.
+  # We keep API docs references in content but do not fail CI on access policy.
+  "^https://platform\\.openai\\.com/docs/",
+  "^https://gamma\\.app/docs/Optimizing-RL-with-SGLang-y0kqgj877k34779$",
+  "^https://aflah02\\.substack\\.com/p/multi-node-llm-inference-with-sglang/?$",
+
+  # Known noisy image URLs used in notebook-rendered examples.
+  "^https://github\\.com/sgl-project/sglang/blob/main/examples/assets/example_image\\.png\\?raw=true$",
+  "^https://raw\\.githubusercontent\\.com/sgl-project/sglang/main/examples/assets/example_image\\.png/?$",
+  "^https://raw\\.githubusercontent\\.com/sgl-project/sglang/main/assets/logo\\.png/?$",
+]

third_party/sglang/.github/linters/lychee.toml

@@ -0,0 +1,18 @@
+# .github/linters/lychee.toml
+no_progress = true
+verbose = "warn"
+timeout = 20
+max_concurrency = 8
+
+offline = true
+
+# Ignore generated docs output; check source docs only.
+exclude_path = [
+  "^(\\./)?docs[/\\\\]_build[/\\\\]",
+]
+
+exclude = [
+  "^https?://localhost(:[0-9]+)?(/|$)",
+  "^http://127\\.0\\.0\\.1(:[0-9]+)?(/|$)",
+  "^http://0\\.0\\.0\\.0(/|$)",
+]

third_party/sglang/.github/pull_request_template.md

@@ -0,0 +1,33 @@
+<!-- Thank you for your contribution! Please follow these guidelines to enhance your pull request. If anything is unclear, submit your PR and reach out to maintainers for assistance. Join our Slack community at https://slack.sglang.io to discuss further. -->
+
+## Motivation
+
+<!-- Describe the purpose and goals of this pull request. -->
+
+## Modifications
+
+<!-- Detail the changes made in this pull request. -->
+
+## Accuracy Tests
+
+<!-- If this pull request affects model outputs (e.g., changes to the kernel or model forward code), provide accuracy test results. -->
+
+## Speed Tests and Profiling
+
+<!-- If this pull request impacts inference speed, provide benchmarking and profiling results. -->
+
+## Checklist
+
+- [ ] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit).
+- [ ] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests).
+- [ ] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations).
+- [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed).
+- [ ] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance).
+
+## Review and Merge Process
+
+1. Ping Merge Oncalls to start the process. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process).
+2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers.
+3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so.
+   - Common commands include `/tag-and-rerun-ci`, `/tag-run-ci-label`, `/rerun-failed-ci`
+4. After green CI and required approvals, ask Merge Oncalls or people with Write permission to merge the PR.

third_party/sglang/.github/update_ci_permission.py

@@ -0,0 +1,244 @@
+"""
+Update the CI permissions configuration file.
+
+This script updates the `CI_PERMISSIONS.json` file, which defines the CI permissions granted to each user.
+
+The format of `CI_PERMISSIONS.json` is as follows:
+
+{
+    "username1": {
+        "can_tag_run_ci_label": true,
+        "can_rerun_failed_ci": true,
+        "cooldown_interval_minutes": 0,
+        "reason": "top contributor"
+    },
+    "username2": {
+        "can_tag_run_ci_label": true,
+        "can_rerun_failed_ci": true,
+        "cooldown_interval_minutes": 60,
+        "reason": "custom override"
+    }
+}
+
+Permissions are assigned according to the following rules:
+
+1. Add the top 50 contributors from the last 120 days with full permissions, no cooldown, and the reason "top contributor".
+2. Load all users from the existing `CI_PERMISSIONS.json` file and update their entries as follows:
+   - If a user is already covered by rule 1, skip that user.
+   - If the old reason of a user is "top contributor" but they are not in the current top contributors list, change their configuration to:
+       {
+           "can_tag_run_ci_label": true,
+           "can_rerun_failed_ci": true,
+           "cooldown_interval_minutes": 60,
+           "reason": "custom override"
+       }
+    - For all other cases, preserve the original configuration unchanged.
+3. All other users receive no permissions and a 120-minute cooldown (they are omitted from the file).
+
+Usage:
+    export GH_TOKEN="your_github_token"
+    python3 update_ci_permission.py
+
+    # Sort-only mode (no network calls, no GH_TOKEN required)
+    python3 update_ci_permission.py --sort-only
+"""
+
+import argparse
+import json
+import os
+from collections import Counter
+from datetime import datetime, timedelta, timezone
+
+try:
+    import requests
+except ImportError:
+    requests = None  # Only needed for non-sort-only runs
+
+# Configuration
+REPO_OWNER = "sgl-project"
+REPO_NAME = "sglang"
+FILE_NAME = os.path.join(os.path.dirname(__file__), "CI_PERMISSIONS.json")
+HEADERS = {}
+
+
+def github_api_get(endpoint, params=None):
+    """Helper to make paginated GitHub API requests."""
+    if requests is None:
+        raise RuntimeError(
+            "The requests package is required. Install it or use --sort-only."
+        )
+    if not HEADERS:
+        raise RuntimeError(
+            "GitHub headers not initialized. Set GH_TOKEN or use --sort-only."
+        )
+
+    results = []
+    url = f"https://api.github.com/repos/{REPO_OWNER}/{REPO_NAME}/{endpoint}"
+
+    while url:
+        response = requests.get(url, headers=HEADERS, params=params)
+        if response.status_code != 200:
+            print(f"Error fetching {url}: {response.status_code} {response.text}")
+            # If we fail to fetch, strictly return what we have or empty to avoid crashing logic
+            break
+
+        data = response.json()
+        if isinstance(data, list):
+            results.extend(data)
+        else:
+            return data  # Non-list response (not paginated usually)
+
+        # Handle pagination
+        url = None
+        if "link" in response.headers:
+            links = response.headers["link"].split(", ")
+            for link in links:
+                if 'rel="next"' in link:
+                    url = link[link.find("<") + 1 : link.find(">")]
+                    params = None  # Params are included in the next link
+                    break
+    return results
+
+
+def get_write_access_users():
+    """Fetches users with push (write) or admin access."""
+    print("Fetching collaborators with write access...")
+    # Note: This endpoint usually requires admin rights on the token.
+    collaborators = github_api_get("collaborators", params={"per_page": 100})
+
+    writers = set()
+    for col in collaborators:
+        perms = col.get("permissions", {})
+        # Check for admin, maintain, or push rights
+        if perms.get("admin") or perms.get("maintain") or perms.get("push"):
+            writers.add(col["login"])
+
+    print(f"Found {len(writers)} users with write access.")
+    return writers
+
+
+def get_top_contributors(days, limit):
+    """Fetches top contributors based on commit count in the last N days."""
+    print(f"Fetching commits from the last {days} days...")
+    since_date = (datetime.now(timezone.utc) - timedelta(days=days)).isoformat()
+
+    # Fetch commits
+    commits = github_api_get("commits", params={"since": since_date, "per_page": 100})
+
+    author_counts = Counter()
+    for commit in commits:
+        # commit['author'] contains the GitHub user object (can be None if not linked)
+        if commit.get("author") and "login" in commit["author"]:
+            author_counts[commit["author"]["login"]] += 1
+
+    top_users = [user for user, _ in author_counts.most_common(limit)]
+    print(f"Found {len(top_users)} top contributors in the last {days} days.")
+    return set(top_users)
+
+
+def load_existing_permissions():
+    if os.path.exists(FILE_NAME):
+        try:
+            with open(FILE_NAME, "r") as f:
+                return json.load(f)
+        except json.JSONDecodeError:
+            print(f"Warning: {FILE_NAME} is invalid JSON. Starting fresh.")
+    return {}
+
+
+def sort_permissions_file():
+    """Sort the existing CI permissions file alphabetically and exit."""
+    if not os.path.exists(FILE_NAME):
+        print(f"{FILE_NAME} not found. Nothing to sort.")
+        return
+
+    old_permissions = load_existing_permissions()
+    sorted_permissions = dict(sorted(old_permissions.items()))
+
+    with open(FILE_NAME, "w") as f:
+        json.dump(sorted_permissions, f, indent=4)
+        f.write("\n")
+
+    print(f"Sorted {FILE_NAME}. Total users: {len(sorted_permissions)}")
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Update or sort CI permissions.")
+    parser.add_argument(
+        "--sort-only",
+        action="store_true",
+        help="Only sort CI_PERMISSIONS.json alphabetically without fetching data.",
+    )
+    args = parser.parse_args()
+
+    if args.sort_only:
+        sort_permissions_file()
+        return
+
+    gh_token = os.getenv("GH_TOKEN")
+    if not gh_token:
+        raise ValueError("Error: GH_TOKEN environment variable is not set.")
+
+    global HEADERS
+    HEADERS = {
+        "Authorization": f"Bearer {gh_token}",
+        "Accept": "application/vnd.github+json",
+        "X-GitHub-Api-Version": "2022-11-28",
+    }
+
+    # Gather Data
+    try:
+        write_access_users = get_write_access_users()
+    except Exception as e:
+        print(f"Warning: Could not fetch collaborators (check token scope). Error: {e}")
+        write_access_users = set()
+
+    top_contributors = get_top_contributors(days=120, limit=50)
+    old_permissions = load_existing_permissions()
+
+    new_permissions = {}
+
+    # Rule 1: Add Top 50 Contributors
+    for user in top_contributors:
+        new_permissions[user] = {
+            "can_tag_run_ci_label": True,
+            "can_rerun_failed_ci": True,
+            "can_rerun_stage": True,
+            "cooldown_interval_minutes": 0,
+            "reason": "top contributor",
+        }
+
+    # Rule 2: Process Existing Users (Merge Logic)
+    for user, config in old_permissions.items():
+        if user in new_permissions:
+            # Already handled by Rule 1 or 2
+            continue
+
+        old_reason = config.get("reason", "")
+
+        # If they fell off the top contributor list
+        if old_reason in ["top contributor"]:
+            new_permissions[user] = {
+                "can_tag_run_ci_label": True,
+                "can_rerun_failed_ci": True,
+                "can_rerun_stage": True,
+                "cooldown_interval_minutes": 60,
+                "reason": "custom override",
+            }
+        else:
+            # Preserve custom overrides
+            new_permissions[user] = config
+
+    # Save and Sort
+    # Sorting keys for cleaner diffs
+    sorted_permissions = dict(sorted(new_permissions.items()))
+
+    with open(FILE_NAME, "w") as f:
+        json.dump(sorted_permissions, f, indent=4)
+        f.write("\n")  # Add trailing newline
+
+    print(f"Successfully updated {FILE_NAME}. Total users: {len(sorted_permissions)}")
+
+
+if __name__ == "__main__":
+    main()

third_party/sglang/.github/workflows/amd-aiter-scout.yml

@@ -0,0 +1,161 @@
+name: AMD AITER Scout
+
+on:
+  schedule:
+    - cron: '0 20 * * 1'   # Monday 20:00 UTC
+    - cron: '0 20 * * 4'   # Thursday 20:00 UTC
+  workflow_dispatch:
+    inputs:
+      aiter_ref:
+        description: 'AITER git ref (branch, tag, or SHA). Default: main (latest commit)'
+        required: false
+        type: string
+        default: 'main'
+      job_filter:
+        description: 'Comma-separated workflows to run: nightly-amd, nightly-amd-rocm720, pr-test-amd, pr-test-amd-rocm720. Default: all'
+        required: false
+        type: string
+        default: 'all'
+      continue_on_error:
+        description: 'Continue running other workflows even if one fails'
+        required: false
+        type: boolean
+        default: true
+
+concurrency:
+  group: amd-aiter-scout-${{ github.run_id }}
+  cancel-in-progress: true
+
+jobs:
+  resolve-aiter:
+    runs-on: ubuntu-latest
+    outputs:
+      aiter_sha: ${{ steps.resolve.outputs.sha }}
+      run_nightly_amd: ${{ steps.parse.outputs.run_nightly_amd }}
+      run_nightly_amd_rocm720: ${{ steps.parse.outputs.run_nightly_amd_rocm720 }}
+      run_pr_test_amd: ${{ steps.parse.outputs.run_pr_test_amd }}
+      run_pr_test_amd_rocm720: ${{ steps.parse.outputs.run_pr_test_amd_rocm720 }}
+    steps:
+      - name: Resolve AITER commit
+        id: resolve
+        run: |
+          REF="${{ inputs.aiter_ref || 'main' }}"
+          echo "Resolving AITER ref: ${REF}"
+
+          SHA=$(git ls-remote https://github.com/ROCm/aiter.git "refs/heads/${REF}" | head -1 | cut -f1)
+          if [ -z "$SHA" ]; then
+            SHA=$(git ls-remote https://github.com/ROCm/aiter.git "refs/tags/${REF}" | head -1 | cut -f1)
+          fi
+          if [ -z "$SHA" ]; then
+            SHA=$(git ls-remote https://github.com/ROCm/aiter.git "${REF}" | head -1 | cut -f1)
+          fi
+          if [ -z "$SHA" ]; then
+            SHA="${REF}"
+          fi
+
+          echo "sha=${SHA}" >> $GITHUB_OUTPUT
+          echo "### AITER Ref Resolution" >> $GITHUB_STEP_SUMMARY
+          echo "- **Requested ref:** \`${REF}\`" >> $GITHUB_STEP_SUMMARY
+          echo "- **Resolved SHA:** \`${SHA}\`" >> $GITHUB_STEP_SUMMARY
+          echo "- **AITER commit:** https://github.com/ROCm/aiter/commit/${SHA}" >> $GITHUB_STEP_SUMMARY
+
+      - name: Parse job filter
+        id: parse
+        run: |
+          FILTER="${{ inputs.job_filter || 'all' }}"
+          echo "Job filter: ${FILTER}"
+
+          if [[ "$FILTER" == "all" ]]; then
+            echo "run_nightly_amd=true" >> $GITHUB_OUTPUT
+            echo "run_nightly_amd_rocm720=true" >> $GITHUB_OUTPUT
+            echo "run_pr_test_amd=true" >> $GITHUB_OUTPUT
+            echo "run_pr_test_amd_rocm720=true" >> $GITHUB_OUTPUT
+          else
+            # Wrap with commas for exact substring matching (avoids "nightly-amd" matching "nightly-amd-rocm720")
+            PADDED=",${FILTER// /},"
+            echo "run_nightly_amd=$(echo "$PADDED" | grep -q ',nightly-amd,' && echo true || echo false)" >> $GITHUB_OUTPUT
+            echo "run_nightly_amd_rocm720=$(echo "$PADDED" | grep -q ',nightly-amd-rocm720,' && echo true || echo false)" >> $GITHUB_OUTPUT
+            echo "run_pr_test_amd=$(echo "$PADDED" | grep -q ',pr-test-amd,' && echo true || echo false)" >> $GITHUB_OUTPUT
+            echo "run_pr_test_amd_rocm720=$(echo "$PADDED" | grep -q ',pr-test-amd-rocm720,' && echo true || echo false)" >> $GITHUB_OUTPUT
+          fi
+
+          echo "### Job Filter" >> $GITHUB_STEP_SUMMARY
+          echo "- **Filter:** \`${FILTER}\`" >> $GITHUB_STEP_SUMMARY
+
+  call-nightly-amd:
+    if: needs.resolve-aiter.outputs.run_nightly_amd == 'true'
+    needs: resolve-aiter
+    uses: ./.github/workflows/nightly-test-amd.yml
+    secrets: inherit
+    with:
+      ref: ${{ github.sha }}
+      aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }}
+      job_filter: 'all'
+      continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }}
+
+  call-nightly-amd-rocm720:
+    if: needs.resolve-aiter.outputs.run_nightly_amd_rocm720 == 'true'
+    needs: resolve-aiter
+    uses: ./.github/workflows/nightly-test-amd-rocm720.yml
+    secrets: inherit
+    with:
+      ref: ${{ github.sha }}
+      aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }}
+      job_filter: 'all'
+      continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }}
+
+  call-pr-test-amd:
+    if: needs.resolve-aiter.outputs.run_pr_test_amd == 'true'
+    needs: resolve-aiter
+    uses: ./.github/workflows/pr-test-amd.yml
+    secrets: inherit
+    with:
+      run_all_tests: true
+      aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }}
+      continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }}
+
+  call-pr-test-amd-rocm720:
+    if: needs.resolve-aiter.outputs.run_pr_test_amd_rocm720 == 'true'
+    needs: resolve-aiter
+    uses: ./.github/workflows/pr-test-amd-rocm720.yml
+    secrets: inherit
+    with:
+      run_all_tests: true
+      aiter_ref: ${{ needs.resolve-aiter.outputs.aiter_sha }}
+      continue_on_error: ${{ inputs.continue_on_error == '' && true || inputs.continue_on_error }}
+
+  check-all-jobs:
+    if: always()
+    needs:
+      - resolve-aiter
+      - call-nightly-amd
+      - call-nightly-amd-rocm720
+      - call-pr-test-amd
+      - call-pr-test-amd-rocm720
+    runs-on: ubuntu-latest
+    steps:
+      - name: Summary
+        run: |
+          echo "## AMD AITER Scout Results" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "- **AITER SHA:** \`${{ needs.resolve-aiter.outputs.aiter_sha }}\`" >> $GITHUB_STEP_SUMMARY
+          echo "- **AITER commit:** https://github.com/ROCm/aiter/commit/${{ needs.resolve-aiter.outputs.aiter_sha }}" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "| Workflow | Result |" >> $GITHUB_STEP_SUMMARY
+          echo "|----------|--------|" >> $GITHUB_STEP_SUMMARY
+          echo "| Nightly AMD (AITER Latest) | \`${{ needs.call-nightly-amd.result }}\` |" >> $GITHUB_STEP_SUMMARY
+          echo "| Nightly AMD ROCm 7.2 | \`${{ needs.call-nightly-amd-rocm720.result }}\` |" >> $GITHUB_STEP_SUMMARY
+          echo "| PR Test AMD (AITER Latest) | \`${{ needs.call-pr-test-amd.result }}\` |" >> $GITHUB_STEP_SUMMARY
+          echo "| PR Test AMD ROCm 7.2 | \`${{ needs.call-pr-test-amd-rocm720.result }}\` |" >> $GITHUB_STEP_SUMMARY
+
+      - name: Check if any job failed
+        run: |
+          if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then
+            echo "One or more workflows failed"
+            exit 1
+          fi
+          if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
+            echo "One or more workflows were cancelled"
+            exit 1
+          fi
+          echo "All workflows passed"

third_party/sglang/.github/workflows/amd-ci-job-monitor.yml

@@ -0,0 +1,338 @@
+name: AMD CI Job Monitor
+
+on:
+  schedule:
+    - cron: '0 0 * * *'  # Daily at midnight UTC
+  pull_request:
+    paths:
+      - '.github/workflows/amd-ci-job-monitor.yml'
+      - 'scripts/ci/utils/query_job_status.py'
+  workflow_dispatch:
+    inputs:
+      hours:
+        description: 'Time window in hours'
+        required: false
+        default: '24'
+        type: string
+      job_filter:
+        description: 'Job name filter (leave empty for all AMD jobs)'
+        required: false
+        type: string
+
+jobs:
+  fetch-actions-data:
+    name: Fetch Actions Snapshot
+    runs-on: ubuntu-latest
+    env:
+      GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Select workflows for snapshot
+        id: select-workflows
+        run: |
+          if [[ -n "${{ inputs.job_filter }}" ]]; then
+            echo "workflows=pr-test-amd.yml" >> "$GITHUB_OUTPUT"
+          else
+            echo "workflows=pr-test-amd.yml,nightly-test-amd.yml,pr-test-amd-rocm720.yml,nightly-test-amd-rocm720.yml" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Fetch Actions data snapshot
+        timeout-minutes: 30
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --workflow "${{ steps.select-workflows.outputs.workflows }}" \
+            --hours ${{ inputs.hours || '24' }} \
+            --dump-data-file actions-job-snapshot.json
+
+      - name: Upload Actions data snapshot
+        uses: actions/upload-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: actions-job-snapshot.json
+          if-no-files-found: error
+
+  # Single job filter mode
+  custom-report:
+    name: Custom Job Report
+    if: ${{ inputs.job_filter }}
+    needs: fetch-actions-data
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Download Actions data snapshot
+        uses: actions/download-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: ci-data
+
+      - name: Generate Custom Job Report
+        timeout-minutes: 30
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --job "${{ inputs.job_filter }}" \
+            --workflow "pr-test-amd.yml" \
+            --hours ${{ inputs.hours || '24' }} \
+            --input-data-file ci-data/actions-job-snapshot.json \
+            --summary
+
+  # Parse workflow files to get job names dynamically
+  parse-workflows:
+    name: Parse Workflow Jobs
+    if: ${{ !inputs.job_filter }}
+    runs-on: ubuntu-latest
+    outputs:
+      pr_jobs: ${{ steps.parse.outputs.pr_jobs }}
+      nightly_jobs: ${{ steps.parse.outputs.nightly_jobs }}
+      pr_rocm720_jobs: ${{ steps.parse.outputs.pr_rocm720_jobs }}
+      nightly_rocm720_jobs: ${{ steps.parse.outputs.nightly_rocm720_jobs }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Parse workflow files
+        id: parse
+        run: |
+          # Parse pr-test-amd.yml and extract job names (exclude utility jobs)
+          # Excluded: call-gate, check-changes, pr-test-amd-finish, cancel, check-all-jobs
+          pr_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/pr-test-amd.yml | \
+            grep -v -E '^(call-gate|check-changes|pr-test-amd-finish|cancel|check-all-jobs)$' | \
+            jq -R -s -c 'split("\n") | map(select(length > 0))')
+          echo "pr_jobs=$pr_jobs" >> $GITHUB_OUTPUT
+          echo "PR jobs: $pr_jobs"
+
+          # Parse nightly-test-amd.yml and extract job names (exclude utility jobs)
+          # Excluded: check-all-jobs
+          nightly_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/nightly-test-amd.yml | \
+            grep -v -E '^(check-all-jobs)$' | \
+            jq -R -s -c 'split("\n") | map(select(length > 0))')
+          echo "nightly_jobs=$nightly_jobs" >> $GITHUB_OUTPUT
+          echo "Nightly jobs: $nightly_jobs"
+
+          # Parse pr-test-amd-rocm720.yml (exclude utility jobs)
+          # Excluded: call-gate, check-changes, pr-test-amd-finish, cancel, check-all-jobs
+          pr_rocm720_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/pr-test-amd-rocm720.yml | \
+            grep -v -E '^(call-gate|check-changes|pr-test-amd-finish|cancel|check-all-jobs)$' | \
+            jq -R -s -c 'split("\n") | map(select(length > 0))')
+          echo "pr_rocm720_jobs=$pr_rocm720_jobs" >> $GITHUB_OUTPUT
+          echo "PR ROCm 7.2 jobs: $pr_rocm720_jobs"
+
+          # Parse nightly-test-amd-rocm720.yml (exclude utility jobs)
+          # Excluded: check-all-jobs
+          nightly_rocm720_jobs=$(yq -r '.jobs | keys | .[]' .github/workflows/nightly-test-amd-rocm720.yml | \
+            grep -v -E '^(check-all-jobs)$' | \
+            jq -R -s -c 'split("\n") | map(select(length > 0))')
+          echo "nightly_rocm720_jobs=$nightly_rocm720_jobs" >> $GITHUB_OUTPUT
+          echo "Nightly ROCm 7.2 jobs: $nightly_rocm720_jobs"
+
+  # PR CI reports using dynamic matrix
+  pr-ci-reports:
+    name: PR - ${{ matrix.job_name }}
+    needs: [parse-workflows, fetch-actions-data]
+    if: ${{ !inputs.job_filter }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        job_name: ${{ fromJson(needs.parse-workflows.outputs.pr_jobs) }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Download Actions data snapshot
+        uses: actions/download-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: ci-data
+
+      - name: Generate Report
+        timeout-minutes: 15
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --job "${{ matrix.job_name }}" \
+            --workflow "pr-test-amd.yml" \
+            --hours ${{ inputs.hours || '24' }} \
+            --input-data-file ci-data/actions-job-snapshot.json \
+            --summary
+
+  # Nightly AMD test reports using dynamic matrix
+  nightly-reports:
+    name: Nightly - ${{ matrix.job_name }}
+    needs: [parse-workflows, fetch-actions-data]
+    if: ${{ !inputs.job_filter }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        job_name: ${{ fromJson(needs.parse-workflows.outputs.nightly_jobs) }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Download Actions data snapshot
+        uses: actions/download-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: ci-data
+
+      - name: Generate Nightly Report
+        timeout-minutes: 15
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --job "${{ matrix.job_name }}" \
+            --workflow "nightly-test-amd.yml" \
+            --hours ${{ inputs.hours || '24' }} \
+            --input-data-file ci-data/actions-job-snapshot.json \
+            --summary
+
+  # PR ROCm 7.2 CI reports using dynamic matrix
+  pr-rocm720-ci-reports:
+    name: PR ROCm720 - ${{ matrix.job_name }}
+    needs: [parse-workflows, fetch-actions-data]
+    if: ${{ !inputs.job_filter }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        job_name: ${{ fromJson(needs.parse-workflows.outputs.pr_rocm720_jobs) }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Download Actions data snapshot
+        uses: actions/download-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: ci-data
+
+      - name: Generate PR ROCm 7.2 Report
+        timeout-minutes: 15
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --job "${{ matrix.job_name }}" \
+            --workflow "pr-test-amd-rocm720.yml" \
+            --hours ${{ inputs.hours || '24' }} \
+            --input-data-file ci-data/actions-job-snapshot.json \
+            --summary
+
+  # Nightly ROCm 7.2 reports using dynamic matrix
+  nightly-rocm720-reports:
+    name: Nightly ROCm720 - ${{ matrix.job_name }}
+    needs: [parse-workflows, fetch-actions-data]
+    if: ${{ !inputs.job_filter }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        job_name: ${{ fromJson(needs.parse-workflows.outputs.nightly_rocm720_jobs) }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Download Actions data snapshot
+        uses: actions/download-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: ci-data
+
+      - name: Generate Nightly ROCm 7.2 Report
+        timeout-minutes: 15
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --job "${{ matrix.job_name }}" \
+            --workflow "nightly-test-amd-rocm720.yml" \
+            --hours ${{ inputs.hours || '24' }} \
+            --input-data-file ci-data/actions-job-snapshot.json \
+            --summary
+
+  # Runner fleet report - cross-workflow runner analytics in a single pass
+  runner-fleet-report:
+    name: Runner Fleet Report
+    if: ${{ !inputs.job_filter }}
+    needs: fetch-actions-data
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install dependencies
+        run: pip install tabulate
+
+      - name: Download Actions data snapshot
+        uses: actions/download-artifact@v4
+        with:
+          name: actions-job-snapshot
+          path: ci-data
+
+      - name: Generate Runner Fleet Report
+        timeout-minutes: 30
+        run: |
+          python scripts/ci/utils/query_job_status.py \
+            --repo ${{ github.repository }} \
+            --runner-report \
+            --workflow "pr-test-amd.yml,nightly-test-amd.yml,pr-test-amd-rocm720.yml,nightly-test-amd-rocm720.yml" \
+            --hours ${{ inputs.hours || '24' }} \
+            --input-data-file ci-data/actions-job-snapshot.json \
+            --summary

third_party/sglang/.github/workflows/auto-tune.yml

@@ -0,0 +1,10 @@
+name: Auto tune
+
+on:
+  workflow_dispatch:
+
+jobs:
+  auto-tune-lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4

third_party/sglang/.github/workflows/bot-bump-flashinfer-version.yml

@@ -0,0 +1,50 @@
+name: Bot Bump Flashinfer Version
+
+on:
+  workflow_dispatch:
+    inputs:
+      new_version:
+        description: 'New flashinfer version (e.g., 0.6.4)'
+        required: true
+        type: string
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  bump-flashinfer-version:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install Python dependencies
+        run: |
+            pip install tomli
+
+      - name: Configure Git and branch
+        run: |
+          git config user.name "sglang-bot"
+          git config user.email "sglang-bot@users.noreply.github.com"
+          RANDOM_SUFFIX=$(echo $RANDOM | md5sum | head -c 4)
+          BRANCH_NAME="bot/bump-flashinfer-version-${{ github.event.inputs.new_version }}-${RANDOM_SUFFIX}"
+          git checkout -b "$BRANCH_NAME"
+          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
+
+      - name: Run flashinfer version bump script
+        run: |
+          python scripts/release/bump_flashinfer_version.py "${{ github.event.inputs.new_version }}"
+
+      - name: Commit and create PR
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_PULL_REQUEST }}
+        run: |
+          bash scripts/release/commit_and_pr.sh "flashinfer" "${{ github.event.inputs.new_version }}" "$BRANCH_NAME"

third_party/sglang/.github/workflows/bot-bump-kernel-version-to-sglang.yml

@@ -0,0 +1,60 @@
+name: Bot Bump Kernel Version to SGLang
+
+on:
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  bump-kernel-version-to-sglang:
+    runs-on: ubuntu-latest
+    outputs:
+      branch_name: ${{ steps.set_output.outputs.branch_name }}
+      needs_sync: ${{ steps.check_sync.outputs.needs_sync }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install Python dependencies
+        run: |
+          pip install tomli
+
+      - name: Check if sync is needed
+        id: check_sync
+        run: |
+          python scripts/release/check_kernel_version_to_sglang.py
+
+      - name: Configure Git and branch
+        if: steps.check_sync.outputs.needs_sync == 'true'
+        id: set_output
+        run: |
+          git config user.name "sglang-bot"
+          git config user.email "sglang-bot@users.noreply.github.com"
+          RANDOM_SUFFIX=$(echo $RANDOM | md5sum | head -c 4)
+          KERNEL_VERSION="${{ steps.check_sync.outputs.kernel_version }}"
+          BRANCH_NAME="bot/bump-kernel-version-to-sglang-${KERNEL_VERSION}-${RANDOM_SUFFIX}"
+          git checkout -b "$BRANCH_NAME"
+          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
+          echo "KERNEL_VERSION=$KERNEL_VERSION" >> $GITHUB_ENV
+          echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT
+
+      - name: Run kernel version bump script
+        if: steps.check_sync.outputs.needs_sync == 'true'
+        run: |
+          python scripts/release/bump_kernel_version_to_sglang.py
+
+      - name: Commit and create PR
+        if: steps.check_sync.outputs.needs_sync == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_PULL_REQUEST }}
+        run: |
+          bash scripts/release/commit_and_pr_kernel_to_sglang.sh "$KERNEL_VERSION" "$BRANCH_NAME"

third_party/sglang/.github/workflows/bot-bump-kernel-version.yml

@@ -0,0 +1,50 @@
+name: Bot Bump Kernel Version
+
+on:
+  workflow_dispatch:
+    inputs:
+      new_version:
+        description: 'New sgl-kernel version (e.g., 0.3.12)'
+        required: true
+        type: string
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  bump-kernel-version:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install Python dependencies
+        run: |
+          pip install tomli
+
+      - name: Configure Git and branch
+        run: |
+          git config user.name "sglang-bot"
+          git config user.email "sglang-bot@users.noreply.github.com"
+          RANDOM_SUFFIX=$(echo $RANDOM | md5sum | head -c 4)
+          BRANCH_NAME="bot/bump-kernel-version-${{ github.event.inputs.new_version }}-${RANDOM_SUFFIX}"
+          git checkout -b "$BRANCH_NAME"
+          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
+
+      - name: Run kernel version bump script
+        run: |
+          python scripts/release/bump_kernel_version.py "${{ github.event.inputs.new_version }}"
+
+      - name: Commit and create PR
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_PULL_REQUEST }}
+        run: |
+          bash scripts/release/commit_and_pr.sh "sgl-kernel" "${{ github.event.inputs.new_version }}" "$BRANCH_NAME"

third_party/sglang/.github/workflows/bot-bump-sglang-version.yml

@@ -0,0 +1,89 @@
+name: Bot Bump SGLang Version
+
+on:
+  workflow_dispatch:
+    inputs:
+      new_version:
+        description: 'New SGLang version (e.g., 0.5.3 or 0.5.3rc0)'
+        required: true
+        type: string
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  bump-sglang-version:
+    runs-on: ubuntu-latest
+    outputs:
+      branch_name: ${{ steps.set_output.outputs.branch_name }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install Python dependencies
+        run: |
+          pip install tomli
+
+      - name: Configure Git and branch
+        id: set_output
+        run: |
+          git config user.name "sglang-bot"
+          git config user.email "sglang-bot@users.noreply.github.com"
+          RANDOM_SUFFIX=$(echo $RANDOM | md5sum | head -c 4)
+          BRANCH_NAME="bot/bump-sglang-version-${{ github.event.inputs.new_version }}-${RANDOM_SUFFIX}"
+          git checkout -b "$BRANCH_NAME"
+          echo "BRANCH_NAME=$BRANCH_NAME" >> $GITHUB_ENV
+          echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT
+
+      - name: Run SGLang version bump script
+        run: |
+          python scripts/release/bump_sglang_version.py "${{ github.event.inputs.new_version }}"
+
+      - name: Commit and create PR
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_PULL_REQUEST }}
+        run: |
+          bash scripts/release/commit_and_pr.sh "SGLang" "${{ github.event.inputs.new_version }}" "$BRANCH_NAME"
+
+  run-nightly-tests-nvidia:
+    needs: bump-sglang-version
+    uses: ./.github/workflows/nightly-test-nvidia.yml
+    with:
+      ref: ${{ needs.bump-sglang-version.outputs.branch_name }}
+    secrets: inherit
+
+  run-nightly-tests-amd:
+    needs: bump-sglang-version
+    uses: ./.github/workflows/nightly-test-amd.yml
+    with:
+      ref: ${{ needs.bump-sglang-version.outputs.branch_name }}
+    secrets: inherit
+
+  run-nightly-tests-npu:
+    needs: bump-sglang-version
+    uses: ./.github/workflows/nightly-test-npu.yml
+    with:
+      ref: ${{ needs.bump-sglang-version.outputs.branch_name }}
+    secrets: inherit
+
+  run-pr-tests-xeon:
+    needs: bump-sglang-version
+    uses: ./.github/workflows/pr-test-xeon.yml
+    with:
+      ref: ${{ needs.bump-sglang-version.outputs.branch_name }}
+    secrets: inherit
+
+  run-pr-tests-xpu:
+    needs: bump-sglang-version
+    uses: ./.github/workflows/pr-test-xpu.yml
+    with:
+      ref: ${{ needs.bump-sglang-version.outputs.branch_name }}
+    secrets: inherit

third_party/sglang/.github/workflows/bot-cherry-pick.yml

@@ -0,0 +1,182 @@
+name: Bot Cherry Pick to Release Branch
+
+on:
+  workflow_dispatch:
+    inputs:
+      commit_sha:
+        description: 'Commit SHA to cherry-pick (full or short hash)'
+        required: true
+        type: string
+      target_branch:
+        description: 'Target release branch (e.g., release/v0.5.7)'
+        required: true
+        type: string
+      create_pr:
+        description: 'Create a PR instead of pushing directly'
+        required: false
+        type: boolean
+        default: true
+
+permissions:
+  contents: write
+  pull-requests: write
+
+concurrency:
+  group: cherry-pick-${{ github.event.inputs.target_branch }}
+  cancel-in-progress: false
+
+jobs:
+  cherry-pick:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-latest
+    environment: 'prod'
+    steps:
+      - name: Validate inputs
+        env:
+          TARGET_BRANCH: ${{ github.event.inputs.target_branch }}
+        run: |
+          if [[ ! "$TARGET_BRANCH" =~ ^release/v[0-9]+\.[0-9]+(\.[0-9]+)?$ ]]; then
+            echo "::error::Target branch must match pattern 'release/vX.Y' or 'release/vX.Y.Z' (e.g., release/v0.5.7)"
+            exit 1
+          fi
+
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          token: ${{ secrets.GH_PAT_FOR_PULL_REQUEST }}
+
+      - name: Configure Git
+        run: |
+          git config user.name "sglang-bot"
+          git config user.email "sglang-bot@users.noreply.github.com"
+
+      - name: Validate target branch exists
+        env:
+          TARGET_BRANCH: ${{ github.event.inputs.target_branch }}
+        run: |
+          git fetch origin
+          if ! git ls-remote --exit-code --heads origin "$TARGET_BRANCH" > /dev/null 2>&1; then
+            echo "::error::Target branch '$TARGET_BRANCH' does not exist on remote"
+            exit 1
+          fi
+
+      - name: Get commit info
+        id: commit_info
+        env:
+          COMMIT_SHA_INPUT: ${{ github.event.inputs.commit_sha }}
+        run: |
+          # Verify commit exists
+          if ! git cat-file -t "$COMMIT_SHA_INPUT" > /dev/null 2>&1; then
+            echo "::error::Commit SHA '$COMMIT_SHA_INPUT' does not exist"
+            exit 1
+          fi
+
+          # Get full SHA if short hash provided
+          FULL_SHA=$(git rev-parse "$COMMIT_SHA_INPUT")
+          COMMIT_TITLE=$(git log -1 --format="%s" "$FULL_SHA")
+          SHORT_SHA=$(git rev-parse --short "$FULL_SHA")
+          echo "full_sha=$FULL_SHA" >> $GITHUB_OUTPUT
+          echo "short_sha=$SHORT_SHA" >> $GITHUB_OUTPUT
+          # Use delimiter for multiline-safe output
+          {
+            echo "commit_title<<EOF"
+            echo "$COMMIT_TITLE"
+            echo "EOF"
+          } >> $GITHUB_OUTPUT
+          echo "Cherry-picking commit: $SHORT_SHA - $COMMIT_TITLE"
+
+      - name: Cherry-pick commit
+        id: cherry_pick
+        env:
+          TARGET_BRANCH: ${{ github.event.inputs.target_branch }}
+          FULL_SHA: ${{ steps.commit_info.outputs.full_sha }}
+          SHORT_SHA: ${{ steps.commit_info.outputs.short_sha }}
+          CREATE_PR: ${{ github.event.inputs.create_pr }}
+        run: |
+          if [[ "$CREATE_PR" == "true" ]]; then
+            # Create a new branch for the PR
+            RANDOM_SUFFIX=$(head -c 4 /dev/urandom | xxd -p)
+            NEW_BRANCH="cherry-pick/${SHORT_SHA}-to-${TARGET_BRANCH#release/}-${RANDOM_SUFFIX}"
+            git checkout -b "$NEW_BRANCH" "origin/$TARGET_BRANCH"
+            echo "new_branch=$NEW_BRANCH" >> $GITHUB_OUTPUT
+          else
+            # Checkout target branch directly
+            git checkout "$TARGET_BRANCH"
+          fi
+
+          # Attempt cherry-pick
+          if git cherry-pick "$FULL_SHA"; then
+            echo "cherry_pick_success=true" >> $GITHUB_OUTPUT
+          else
+            echo "::error::Cherry-pick failed due to conflicts. Please resolve manually."
+            git cherry-pick --abort || true
+            echo "cherry_pick_success=false" >> $GITHUB_OUTPUT
+            exit 1
+          fi
+
+      - name: Push changes
+        if: steps.cherry_pick.outputs.cherry_pick_success == 'true'
+        env:
+          CREATE_PR: ${{ github.event.inputs.create_pr }}
+          TARGET_BRANCH: ${{ github.event.inputs.target_branch }}
+          NEW_BRANCH: ${{ steps.cherry_pick.outputs.new_branch }}
+        run: |
+          if [[ "$CREATE_PR" == "true" ]]; then
+            git push origin "$NEW_BRANCH"
+          else
+            git push origin "$TARGET_BRANCH"
+          fi
+
+      - name: Create Pull Request
+        if: steps.cherry_pick.outputs.cherry_pick_success == 'true' && github.event.inputs.create_pr == 'true'
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_PULL_REQUEST }}
+          TARGET_BRANCH: ${{ github.event.inputs.target_branch }}
+          SHORT_SHA: ${{ steps.commit_info.outputs.short_sha }}
+          COMMIT_TITLE: ${{ steps.commit_info.outputs.commit_title }}
+          FULL_SHA: ${{ steps.commit_info.outputs.full_sha }}
+          NEW_BRANCH: ${{ steps.cherry_pick.outputs.new_branch }}
+        run: |
+          PR_TITLE="[Cherry-pick] ${COMMIT_TITLE} to ${TARGET_BRANCH}"
+
+          gh pr create \
+            --title "$PR_TITLE" \
+            --base "$TARGET_BRANCH" \
+            --head "$NEW_BRANCH" \
+            --label "cherry-pick" \
+            --body-file - <<EOF
+          Cherry-pick of commit ${FULL_SHA} to \`${TARGET_BRANCH}\`
+
+          **Original commit:** ${FULL_SHA}
+          **Original title:** ${COMMIT_TITLE}
+
+          ---
+          *This PR was automatically created by the cherry-pick workflow.*
+          EOF
+
+      - name: Summary
+        if: always()
+        env:
+          FULL_SHA: ${{ steps.commit_info.outputs.full_sha }}
+          COMMIT_TITLE: ${{ steps.commit_info.outputs.commit_title }}
+          TARGET_BRANCH: ${{ github.event.inputs.target_branch }}
+          CHERRY_PICK_SUCCESS: ${{ steps.cherry_pick.outputs.cherry_pick_success }}
+          CREATE_PR: ${{ github.event.inputs.create_pr }}
+          NEW_BRANCH: ${{ steps.cherry_pick.outputs.new_branch }}
+          ACTOR: ${{ github.actor }}
+        run: |
+          echo "## Cherry-Pick Summary" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "- **Triggered by:** @${ACTOR}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Commit:** ${FULL_SHA}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Title:** ${COMMIT_TITLE}" >> $GITHUB_STEP_SUMMARY
+          echo "- **Target Branch:** ${TARGET_BRANCH}" >> $GITHUB_STEP_SUMMARY
+          if [[ "$CHERRY_PICK_SUCCESS" == "true" ]]; then
+            echo "- **Status:** ✅ Success" >> $GITHUB_STEP_SUMMARY
+          else
+            echo "- **Status:** ❌ Failed" >> $GITHUB_STEP_SUMMARY
+          fi
+          if [[ "$CREATE_PR" == "true" && "$CHERRY_PICK_SUCCESS" == "true" ]]; then
+            echo "- **PR Branch:** ${NEW_BRANCH}" >> $GITHUB_STEP_SUMMARY
+          fi

third_party/sglang/.github/workflows/cancel-pr-workflow-on-merge.yml

@@ -0,0 +1,22 @@
+name: Cancel PR Workflows on Merge
+
+on:
+  pull_request_target:
+    types:
+      - closed
+
+permissions:
+  actions: write
+
+jobs:
+  cancel:
+    if: github.event.pull_request.merged == true
+    runs-on: ubuntu-latest
+    steps:
+      - name: Cancel Previous Runs
+        uses: styfle/cancel-workflow-action@0.12.1
+        with:
+          workflow_id: all
+          access_token: ${{ secrets.GITHUB_TOKEN }}
+          ignore_sha: true
+          pr_number: ${{ github.event.pull_request.number }}

third_party/sglang/.github/workflows/cancel-unfinished-pr-tests.yml

@@ -0,0 +1,155 @@
+name: Cancel Unfinished PR Runs
+
+on:
+  workflow_dispatch:
+    inputs:
+      workflows:
+        description: 'Space-separated list of workflow filenames to cancel'
+        required: true
+        type: string
+        default: 'pr-test.yml'
+      include_high_priority:
+        description: 'Also cancel runs from high-priority PRs'
+        required: false
+        type: boolean
+        default: false
+
+permissions:
+  actions: write   # Needed to cancel runs
+  contents: read   # Needed to read repo info
+  pull-requests: read  # needed for gh pr view (labels)
+
+jobs:
+  cancel-unfinished-pr-runs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install GitHub CLI
+        run: sudo apt-get install -y gh jq
+
+      - name: Cancel unfinished PR-associated runs (skip high-priority PRs)
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          REPO: ${{ github.repository }}
+          WORKFLOWS: ${{ github.event.inputs.workflows || 'pr-test.yml' }}
+          INCLUDE_HIGH_PRIORITY: ${{ github.event.inputs.include_high_priority || 'false' }}
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          # Read the space-separated string from the input into a bash array
+          read -r -a WORKFLOW_FILES <<< "${WORKFLOWS}"
+
+          echo "Targeting ${#WORKFLOW_FILES[@]} workflow(s): ${WORKFLOWS}"
+          echo ""
+
+          for workflow_file in "${WORKFLOW_FILES[@]}"; do
+            echo "========================================="
+            echo "Workflow: $workflow_file"
+            echo "========================================="
+
+            # Get all unfinished runs
+            all_runs=$(gh run list \
+              --repo "$REPO" \
+              --workflow "$workflow_file" \
+              --json databaseId,status,event,url,createdAt \
+              --limit 1000 \
+            | jq -c '.[] | select(.status=="queued" or .status=="waiting" or .status=="in_progress")')
+
+            if [ -z "$all_runs" ]; then
+              echo "✅ No unfinished runs found"
+              echo ""
+              continue
+            fi
+
+            # Count runs by event type
+            total_runs=$(echo "$all_runs" | wc -l)
+            pr_runs=$(echo "$all_runs" | jq -s '[.[] | select(.event=="pull_request")] | length')
+            other_runs=$(echo "$all_runs" | jq -s '[.[] | select(.event!="pull_request")] | length')
+
+            echo "📊 Summary: $total_runs unfinished runs ($pr_runs PR-related, $other_runs other)"
+            echo ""
+
+            # Process non-PR runs first
+            if [ "$other_runs" -gt 0 ]; then
+              echo "--- Non-PR Runs ---"
+              echo "$all_runs" | jq -c 'select(.event!="pull_request")' | while read -r run; do
+                run_url=$(echo "$run" | jq -r '.url')
+                run_event=$(echo "$run" | jq -r '.event')
+                run_status=$(echo "$run" | jq -r '.status')
+                echo "  • $run_event ($run_status): $run_url"
+              done
+              echo ""
+            fi
+
+            # Process PR runs
+            if [ "$pr_runs" -gt 0 ]; then
+              echo "--- PR Runs (checking for cancellation) ---"
+              echo "$all_runs" | jq -c 'select(.event=="pull_request")' | while read -r run; do
+                run_id=$(echo "$run" | jq -r '.databaseId')
+                run_url=$(echo "$run" | jq -r '.url')
+                run_status=$(echo "$run" | jq -r '.status')
+
+                echo ""
+                echo "Run ($run_status): $run_url"
+
+                # Fetch full run details to get head repository and branch info
+                run_details=$(gh api -H "Accept: application/vnd.github+json" \
+                  "repos/$REPO/actions/runs/$run_id" 2>/dev/null || true)
+
+                if [ -z "$run_details" ]; then
+                  echo "  ⚠️  Could not fetch run details, skipping"
+                  continue
+                fi
+
+                # Get head owner and branch (works for both fork and non-fork PRs)
+                head_owner=$(echo "$run_details" | jq -r '.head_repository.owner.login // empty')
+                head_branch=$(echo "$run_details" | jq -r '.head_branch // empty')
+
+                if [ -z "$head_owner" ] || [ -z "$head_branch" ]; then
+                  echo "  ⚠️  Missing head info, skipping"
+                  continue
+                fi
+
+                echo "  Branch: ${head_owner}:${head_branch}"
+
+                # Find PR by searching with head=owner:branch
+                pr_number=$(gh api -H "Accept: application/vnd.github+json" \
+                  "repos/$REPO/pulls?state=open&head=${head_owner}:${head_branch}" \
+                  --jq '.[0].number // empty' 2>/dev/null || true)
+
+                if [ -z "$pr_number" ]; then
+                  echo "  ⚠️  No open PR found, skipping"
+                  continue
+                fi
+
+                pr_url="https://github.com/$REPO/pull/$pr_number"
+                echo "  PR: $pr_url"
+
+                # Check for high priority label
+                labels=$(gh pr view "$pr_number" --repo "$REPO" --json labels \
+                  | jq -r '.labels[].name' 2>/dev/null || true)
+
+                if echo "$labels" | grep -Fxq "bypass-maintenance"; then
+                  echo "  🛑 Skipping (bypass-maintenance label, never cancelled)"
+                  continue
+                fi
+
+                if echo "$labels" | grep -Fxq "high priority"; then
+                  if [ "$INCLUDE_HIGH_PRIORITY" != "true" ]; then
+                    echo "  🛑 Skipping (high priority label)"
+                    continue
+                  fi
+                  echo "  ⚠️  High priority PR, but include_high_priority is enabled"
+                fi
+
+                echo "  🚫 Cancelling..."
+                gh run cancel "$run_id" --repo "$REPO" || echo "  ⚠️  Cancellation failed"
+              done
+            fi
+
+            echo ""
+          done
+
+          echo "========================================="
+          echo "✅ Processing complete"
+          echo "========================================="

third_party/sglang/.github/workflows/ci-coverage-overview.yml

@@ -0,0 +1,154 @@
+name: CI Coverage Overview
+
+on:
+  schedule:
+    - cron: '0 6 * * *'  # Daily at 6 AM UTC
+  pull_request:
+    paths:
+      - '.github/workflows/ci-coverage-overview.yml'
+      - 'scripts/ci/utils/ci_coverage_report.py'
+      - 'test/registered/**'
+  workflow_dispatch:
+    inputs:
+      output_format:
+        description: 'Output format'
+        required: false
+        default: 'markdown'
+        type: choice
+        options:
+          - markdown
+          - json
+
+jobs:
+  summary:
+    name: Summary
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Generate Summary Report
+        run: |
+          python scripts/ci/utils/ci_coverage_report.py --section summary
+
+  by-folder:
+    name: Tests by Folder
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Generate Tests by Folder Report
+        run: |
+          python scripts/ci/utils/ci_coverage_report.py --section by-folder
+
+  by-suite:
+    name: Tests by Suite
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Generate Tests by Suite Report
+        run: |
+          python scripts/ci/utils/ci_coverage_report.py --section by-suite
+
+  unit-test-coverage:
+    name: Unit Test Code Coverage
+    if: github.event_name != 'pull_request'
+    runs-on: 1-gpu-h100
+    timeout-minutes: 30
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install dependencies
+        timeout-minutes: 10
+        run: |
+          pip install -e "python/[test]"
+
+      - name: Run unit tests with coverage
+        timeout-minutes: 10
+        run: |
+          pytest test/registered/unit/ \
+            --cov --cov-config=.coveragerc \
+            --cov-report=term-missing:skip-covered \
+            --continue-on-collection-errors \
+            -v | tee coverage_output.txt
+
+      - name: Write coverage to summary
+        if: always()
+        run: |
+          echo "## Unit Test Code Coverage" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "**Commit:** \`${GITHUB_SHA::8}\` | **Branch:** \`${GITHUB_REF_NAME}\`" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+
+          # Test result line (e.g., "== 42 passed, 1 failed in 23.5s ==")
+          echo '```' >> $GITHUB_STEP_SUMMARY
+          grep -E '^=+.*passed' coverage_output.txt >> $GITHUB_STEP_SUMMARY || true
+          echo "" >> $GITHUB_STEP_SUMMARY
+          # Coverage total
+          grep -E '^TOTAL ' coverage_output.txt >> $GITHUB_STEP_SUMMARY || true
+          echo '```' >> $GITHUB_STEP_SUMMARY
+
+          # Partially covered core modules (1-49%) — most actionable for contributors
+          # Only show modules with testable logic; skip configs, models, layers, etc.
+          LOW_COV=$(awk '/^python\/.*%/ {
+            for (i=1; i<=NF; i++) {
+              if ($i ~ /^[0-9]+%$/) {
+                pct = $i + 0
+                if (pct >= 1 && pct < 50) printf "%-80s %5s  %s\n", $1, $(i-2), $i
+                break
+              }
+            }
+          }' coverage_output.txt \
+            | grep -E '/(mem_cache|managers|sampling|parser|observability|function_call|entrypoints|speculative|multimodal|utils)/' \
+            | head -40 || true)
+          if [ -n "$LOW_COV" ]; then
+            echo "" >> $GITHUB_STEP_SUMMARY
+            echo "<details><summary>Core modules with coverage below 50% — good candidates for more unit tests</summary>" >> $GITHUB_STEP_SUMMARY
+            echo "" >> $GITHUB_STEP_SUMMARY
+            echo '```' >> $GITHUB_STEP_SUMMARY
+            echo "$LOW_COV" >> $GITHUB_STEP_SUMMARY
+            echo '```' >> $GITHUB_STEP_SUMMARY
+            echo "</details>" >> $GITHUB_STEP_SUMMARY
+          fi
+
+  json-export:
+    name: JSON Export
+    runs-on: ubuntu-latest
+    if: inputs.output_format == 'json'
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Generate JSON Report
+        run: |
+          python scripts/ci/utils/ci_coverage_report.py --output-format json > ci_coverage.json
+
+      - name: Upload JSON artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: ci-coverage-report
+          path: ci_coverage.json

third_party/sglang/.github/workflows/ci-failure-monitor.yml

@@ -0,0 +1,72 @@
+name: CI Failure Monitor
+
+on:
+  schedule:
+    - cron: '0 */12 * * *' # Every 12 hour
+  workflow_dispatch:
+
+concurrency:
+  group: ci-failure-monitor-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+  actions: read
+
+jobs:
+  failure-analysis:
+    if: github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.14'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install requests slack_sdk
+
+      - name: Run Failure Analysis
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+          GH_PAT_FOR_RUNNER_ADMIN: ${{ secrets.GH_PAT_FOR_RUNNER_ADMIN }}
+          PYTHONUNBUFFERED: 1
+          PYTHONIOENCODING: utf-8
+        run: |
+          cd scripts/ci_monitor
+          python ci_failures_analysis.py \
+            --token $GITHUB_TOKEN \
+            --limit 100 \
+            --output ci_failure_analysis_$(date +%Y%m%d_%H%M%S).json
+
+      - name: Upload Analysis Results
+        uses: actions/upload-artifact@v4
+        with:
+          name: ci-failure-analysis-${{ github.run_number }}
+          path: |
+            scripts/ci_monitor/ci_failure_analysis_*.json
+          retention-days: 7
+
+      - name: Send Slack Notification
+        if: always()
+        env:
+          SGLANG_DIFFUSION_SLACK_TOKEN: ${{ secrets.SGLANG_DIFFUSION_SLACK_TOKEN }}
+        run: |
+          cd scripts/ci_monitor
+          LATEST_REPORT=$(ls -t ci_failure_analysis_*.json | head -1)
+
+          if [ ! -f "$LATEST_REPORT" ]; then
+            echo "No report found, so skipping Slack notification"
+            exit 0
+          fi
+
+          if [ -n "$SGLANG_DIFFUSION_SLACK_TOKEN" ]; then
+            python3 post_ci_failures_to_slack.py --report-file "$LATEST_REPORT"
+          else
+            echo "SGLANG_DIFFUSION_SLACK_TOKEN not configured, skipping notification"
+          fi

third_party/sglang/.github/workflows/close-inactive-issues.yml

@@ -0,0 +1,96 @@
+name: Close Inactive Issues
+
+on:
+  schedule:
+    - cron: '0 0 * * *'
+  workflow_dispatch:
+
+permissions:
+  issues: write
+  contents: read
+
+jobs:
+  close-inactive-issues:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check and close inactive issues
+        uses: actions/github-script@v6
+        with:
+          github-token: ${{secrets.GITHUB_TOKEN}}
+          script: |
+            const sixtyDaysAgo = new Date(Date.now() - 60 * 24 * 60 * 60 * 1000);
+
+            const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/');
+            console.log(`Owner: ${owner}, Repo: ${repo}`);
+
+            async function fetchIssues(page = 1) {
+              console.log(`Fetching issues for ${owner}/${repo}, page ${page}`);
+              return await github.rest.issues.listForRepo({
+                owner,
+                repo,
+                state: 'open',
+                sort: 'updated',
+                direction: 'asc',
+                per_page: 100,
+                page: page
+              });
+            }
+
+            async function processIssues() {
+              console.log('Starting to process issues');
+              console.log(`Repository: ${owner}/${repo}`);
+
+              let page = 1;
+              let hasMoreIssues = true;
+              while (hasMoreIssues) {
+                try {
+                  const issues = await fetchIssues(page);
+                  console.log(`Fetched ${issues.data.length} issues on page ${page}`);
+
+                  if (issues.data.length === 0) {
+                    hasMoreIssues = false;
+                    break;
+                  }
+
+                  for (const issue of issues.data) {
+                    // Skip if the issue has 'good first issue' label
+                    if (issue.labels.some(label => label.name === 'good first issue')) {
+                      console.log(`Skipping issue #${issue.number} as it's marked as 'good first issue'`);
+                      continue;
+                    }
+                    if (new Date(issue.updated_at) < sixtyDaysAgo) {
+                      try {
+                        await github.rest.issues.update({
+                          owner,
+                          repo,
+                          issue_number: issue.number,
+                          state: 'closed',
+                          labels: [...issue.labels.map(l => l.name), 'inactive']
+                        });
+                        await github.rest.issues.createComment({
+                          owner,
+                          repo,
+                          issue_number: issue.number,
+                          body: 'This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.'
+                        });
+                        console.log(`Closed issue #${issue.number} due to inactivity.`);
+                      } catch (error) {
+                        console.error(`Failed to close issue #${issue.number}: ${error.message}`);
+                      }
+                    } else {
+                      console.log(`Issue #${issue.number} is still active. Stopping processing.`);
+                      hasMoreIssues = false;
+                      break;
+                    }
+                  }
+                  page += 1;
+                } catch (error) {
+                  console.error(`Error fetching issues on page ${page}: ${error.message}`);
+                  hasMoreIssues = false;
+                }
+              }
+              console.log('Finished processing issues');
+            }
+
+            await processIssues();

third_party/sglang/.github/workflows/diffusion-ci-gt-gen.yml

@@ -0,0 +1,115 @@
+name: Diffusion CI Ground Truth Generation
+
+on:
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: 'Git ref to checkout'
+        required: false
+        default: ''
+        type: string
+      case_ids:
+        description: 'Specific case IDs to run (space-separated, optional)'
+        required: false
+        default: ''
+        type: string
+
+concurrency:
+  group: diffusion-ci-gt-gen-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: write
+  actions: read
+
+jobs:
+  multimodal-diffusion-gen-1gpu:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: 1-gpu-h100
+    strategy:
+      matrix:
+        part: [0, 1]
+    timeout-minutes: 150
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Install dependencies
+        run: bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Generate outputs
+        run: |
+          cd python
+          python -m sglang.multimodal_gen.test.scripts.gen_diffusion_ci_outputs \
+            --suite 1-gpu \
+            --partition-id ${{ matrix.part }} \
+            --total-partitions 2 \
+            --out-dir ./diffusion-ci-outputs \
+            ${{ inputs.case_ids != '' && format('--case-ids {0}', inputs.case_ids) || '' }}
+
+      - name: Upload artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: diffusion-gen-1gpu-part${{ matrix.part }}
+          path: python/diffusion-ci-outputs
+          retention-days: 7
+
+  multimodal-diffusion-gen-2gpu:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: 2-gpu-h100
+    strategy:
+      matrix:
+        part: [0, 1]
+    timeout-minutes: 150
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Install dependencies
+        run: bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Generate outputs
+        run: |
+          cd python
+          python -m sglang.multimodal_gen.test.scripts.gen_diffusion_ci_outputs \
+            --suite 2-gpu \
+            --partition-id ${{ matrix.part }} \
+            --total-partitions 2 \
+            --out-dir ./diffusion-ci-outputs \
+            ${{ inputs.case_ids != '' && format('--case-ids {0}', inputs.case_ids) || '' }}
+
+      - name: Upload artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: diffusion-gen-2gpu-part${{ matrix.part }}
+          path: python/diffusion-ci-outputs
+          retention-days: 7
+
+  diffusion-ci-push:
+    needs: [multimodal-diffusion-gen-1gpu, multimodal-diffusion-gen-2gpu]
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          pattern: diffusion-gen-*
+          path: combined
+          merge-multiple: true
+
+      - name: Collect image files
+        run: |
+          mkdir -p gt_images
+          find combined \( -name "*.png" -o -name "*.jpg" -o -name "*.jpeg" -o -name "*.webp" \) -type f -exec cp -f {} gt_images/ \;
+
+      - name: Publish GT images to sglang-bot/sglang-ci-data
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+        run: python scripts/ci/utils/diffusion/publish_diffusion_gt.py --source-dir gt_images

third_party/sglang/.github/workflows/execute-notebook.yml

@@ -0,0 +1,74 @@
+name: Execute Notebooks
+
+on:
+  pull_request:
+    branches: [ main ]
+    types: [opened, synchronize, reopened, labeled]
+    paths:
+      - "python/sglang/**"
+      - "docs/**"
+      - "!python/sglang/**/*.md"
+      - "!docs/**/*.md"
+  workflow_dispatch:
+
+
+concurrency:
+  group: execute-notebook-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  SGLANG_IS_IN_CI: true
+
+jobs:
+  call-gate:
+    # Align with PR Test: fail fast if PR doesn't have run-ci label.
+    # This makes /tag-and-rerun-ci work by rerunning this failed workflow.
+    uses: ./.github/workflows/pr-gate.yml
+    secrets: inherit
+
+  run-all-notebooks:
+    needs: [call-gate]
+    runs-on: 1-gpu-h100
+    if: github.event_name != 'pull_request' || needs.call-gate.result == 'success'
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+          pip install -r docs/requirements.txt
+          apt-get update && apt-get install -y pandoc parallel retry
+          ln -sf "$(which python3)" /usr/bin/python
+
+      - name: Setup Jupyter Kernel
+        run: |
+          python -m ipykernel install --user --name python3 --display-name "Python 3"
+
+      - name: Execute notebooks
+        timeout-minutes: 40
+        run: |
+          cd docs
+          make clean
+          make compile
+
+
+  notebook-finish:
+    needs: [
+      call-gate,
+      run-all-notebooks
+    ]
+    runs-on: ubuntu-latest
+    if: always() && needs.run-all-notebooks.result != 'skipped'
+    steps:
+      - name: Check all dependent job statuses
+        run: |
+          results=(${{ join(needs.*.result, ' ') }})
+          for result in "${results[@]}"; do
+            if [ "$result" = "failure" ] || [ "$result" = "cancelled" ]; then
+              echo "Job failed with result: $result"
+              exit 1
+            fi
+          done
+          echo "All jobs completed successfully"
+          exit 0

third_party/sglang/.github/workflows/full-test-npu.yml

@@ -0,0 +1,355 @@
+name: Full Test (NPU)
+
+on:
+#  pull_request:
+#    branches:
+#      - main
+#    paths:
+#      - ".github/workflows/full-test-npu.yml"
+  workflow_dispatch:
+    inputs:
+      ref:
+        description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
+        required: false
+        type: string
+        default: ''
+      job_filter:
+        description: 'Select which job to run (leave empty or "all" to run all jobs)'
+        required: false
+        type: string
+        default: 'all'
+      image_a3:
+        description: 'The a3 running docker image of the test task.'
+        required: false
+        type: string
+        default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11'
+      skip_install_flag:
+        description: 'Indicates whether to skip the installation of sglang, defaulting to false.'
+        required: false
+        type: string
+        default: 'false'
+
+concurrency:
+  group: full-test-npu-${{ inputs.ref || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
+
+jobs:
+  set-image-config:
+    runs-on: ubuntu-latest
+    outputs:
+      ref: ${{ steps.set-vars.outputs.ref }}
+      job_filter: ${{ steps.set-vars.outputs.job_filter }}
+      image_a3: ${{ steps.set-vars.outputs.image_a3 }}
+      skip_install_flag: ${{ steps.set-vars.outputs.skip_install_flag }}
+    steps:
+      # When triggered by PR, no inputs parameters are used. The latest community code is tested by default.
+      - name: Set image config
+        id: set-vars
+        run: |
+          if [ -z "${{ inputs.ref }}" ]; then
+            echo "ref=" >> $GITHUB_OUTPUT
+          else
+            echo "ref=${{ inputs.ref }}" >> $GITHUB_OUTPUT
+          fi
+
+          if [ -z "${{ inputs.job_filter }}" ]; then
+            echo "job_filter=all" >> $GITHUB_OUTPUT
+          else
+            echo "job_filter=${{ inputs.job_filter }}" >> $GITHUB_OUTPUT
+          fi
+
+          if [ -z "${{ inputs.image_a3 }}" ]; then
+            echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT
+          else
+            echo "image_a3=${{ inputs.image_a3 }}" >> $GITHUB_OUTPUT
+          fi
+
+          if [ -z "${{ inputs.skip_install_flag }}" ]; then
+            echo "skip_install_flag=false" >> $GITHUB_OUTPUT
+          else
+            echo "skip_install_flag=${{ inputs.skip_install_flag }}" >> $GITHUB_OUTPUT
+          fi
+
+  nighly-test-npu:
+    needs: [set-image-config]
+    name: nightly-test-npu
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    uses: ./.github/workflows/nightly-test-npu.yml
+    with:
+      ref:  ${{ needs.set-image-config.outputs.ref }}
+      job_filter: ${{ needs.set-image-config.outputs.job_filter }}
+      image_a3: ${{ needs.set-image-config.outputs.image_a3 }}
+      skip_install_flag: ${{ needs.set-image-config.outputs.skip_install_flag }}
+    secrets: inherit
+
+  full-1-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-2
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.25 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite full-1-npu-a3 --nightly --continue-on-error --timeout-per-file 3600
+
+  full-2-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-2
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.25 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite full-2-npu-a3 --nightly --continue-on-error --timeout-per-file 3600
+
+  full-4-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-4
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.25 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite full-4-npu-a3 --nightly --continue-on-error --timeout-per-file 3600
+
+  full-16-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-16
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.25 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite full-16-npu-a3 --nightly --continue-on-error --timeout-per-file 3600
+
+  check-all-jobs:
+    if: github.repository == 'sgl-project/sglang' && always()
+    needs:
+      - nighly-test-npu
+      - full-1-npu-a3
+      - full-2-npu-a3
+      - full-4-npu-a3
+      - full-16-npu-a3
+    runs-on: ubuntu-latest
+    container:
+      image: docker.m.daocloud.io/ubuntu:22.04
+    steps:
+      - name: Check if any job failed
+        run: |
+          if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then
+            echo "One or more nightly test jobs failed"
+            exit 1
+          fi
+          if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
+            echo "One or more nightly test jobs were cancelled"
+            exit 1
+          fi
+          echo "All nightly test jobs passed"

third_party/sglang/.github/workflows/labeler.yml

@@ -0,0 +1,20 @@
+name: Auto Label PRs
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+
+permissions:
+  contents: read
+  pull-requests: write
+
+jobs:
+  label:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Auto-label by file changes
+        uses: actions/labeler@v5
+        with:
+          repo-token: "${{ secrets.GITHUB_TOKEN }}"
+          configuration-path: .github/labeler.yml
+          sync-labels: false

third_party/sglang/.github/workflows/lint.yml

@@ -0,0 +1,39 @@
+name: Lint
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.12"
+
+      - name: Install pre-commit hook
+        run: |
+          python -m pip install pre-commit
+          pre-commit install
+
+      - name: Run pre-commit checks
+        run: SKIP=no-commit-to-branch pre-commit run --all-files --show-diff-on-failure
+
+      - name: Run lychee docs checks (offline references)
+        uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2
+        with:
+          args: --config .github/linters/lychee.toml README.md "docs/**/*.md" "docs/**/*.rst" "docs/**/*.ipynb"
+
+      - name: Run sgl-kernel clang-format checks
+        uses: DoozyX/clang-format-lint-action@v0.20
+        with:
+          source: sgl-kernel
+          extensions: h,c,cpp,hpp,cu,cuh,cc
+          clangFormatVersion: 20
+          style: file

third_party/sglang/.github/workflows/list-active-pr-runs.yml

@@ -0,0 +1,317 @@
+name: List Active Runs
+
+on:
+  workflow_dispatch:
+    inputs:
+      workflows:
+        description: 'Space-separated list of workflow filenames to check'
+        required: false
+        type: string
+        default: 'pr-test.yml'
+
+permissions:
+  actions: read
+  contents: read
+  pull-requests: read
+
+jobs:
+  list-active-runs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Install GitHub CLI
+        run: sudo apt-get install -y gh jq
+
+      - name: List active runs grouped by PR
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          REPO: ${{ github.repository }}
+          WORKFLOWS: ${{ github.event.inputs.workflows || 'pr-test.yml' }}
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          echo "========================================="
+          echo "🔍 Active Workflow Runs Report"
+          echo "========================================="
+          echo ""
+
+          # Get all workflows or specific ones
+          read -r -a workflow_files <<< "${WORKFLOWS}"
+          echo "📋 Checking specified workflows: ${WORKFLOWS}"
+
+          echo ""
+
+          # Create a temporary file to store PR data
+          pr_data_file=$(mktemp)
+
+          # Process each workflow
+          for workflow_file in ${workflow_files[@]}; do
+            echo "Scanning workflow: $workflow_file"
+
+            # Get all active runs (queued, waiting, in_progress)
+            active_runs=$(gh run list \
+              --repo "$REPO" \
+              --workflow "$workflow_file" \
+              --json databaseId,status,event,headBranch,createdAt,updatedAt,headSha,number,attempt \
+              --limit 500 \
+              | jq -c '.[] | select(.status=="queued" or .status=="waiting" or .status=="in_progress")')
+
+            if [ -z "$active_runs" ]; then
+              continue
+            fi
+
+            # Process each run
+            echo "$active_runs" | while read -r run; do
+              run_id=$(echo "$run" | jq -r '.databaseId')
+              run_status=$(echo "$run" | jq -r '.status')
+              run_event=$(echo "$run" | jq -r '.event')
+              created_at=$(echo "$run" | jq -r '.createdAt')
+              head_sha=$(echo "$run" | jq -r '.headSha')
+              run_number=$(echo "$run" | jq -r '.number')
+              run_attempt=$(echo "$run" | jq -r '.attempt // 1')
+
+              # Get detailed run information including jobs
+              run_details=$(gh api "repos/$REPO/actions/runs/$run_id" 2>/dev/null || true)
+
+              if [ -z "$run_details" ]; then
+                continue
+              fi
+
+              head_owner=$(echo "$run_details" | jq -r '.head_repository.owner.login // empty')
+              head_branch=$(echo "$run_details" | jq -r '.head_branch // empty')
+
+              if [ -z "$head_owner" ] || [ -z "$head_branch" ]; then
+                continue
+              fi
+
+              # Find PR number (may be empty for non-PR runs)
+              pr_number=$(gh api "repos/$REPO/pulls?state=open&head=${head_owner}:${head_branch}" \
+                --jq '.[0].number // empty' 2>/dev/null || true)
+
+              if [ -z "$pr_number" ]; then
+                pr_number="NO_PR"
+              fi
+
+              # Get jobs for this run (with pagination to avoid missing jobs)
+              jobs=$(gh api "repos/$REPO/actions/runs/$run_id/jobs" --paginate --jq '.jobs[]' | jq -s '.')
+
+              running_jobs=$(echo "$jobs" | jq '[.[] | select(.status=="in_progress")] | length')
+              queued_jobs=$(echo "$jobs" | jq '[.[] | select(.status=="queued" or .status=="waiting")] | length')
+
+              # Get runner info for running jobs
+              runners=$(echo "$jobs" | jq -r '.[] | select(.status=="in_progress") | .runner_name // "N/A"' | paste -sd "," -)
+
+              # Calculate queue time
+              current_time=$(date -u +%s)
+              created_time=$(date -u -d "$created_at" +%s 2>/dev/null || echo "$current_time")
+              queue_time=$((current_time - created_time))
+              queue_minutes=$((queue_time / 60))
+
+              # Store data in temporary file (unified format with event and branch)
+              echo "$pr_number|$workflow_file|$run_id|$run_status|$running_jobs|$queued_jobs|$runners|$queue_minutes|$created_at|$head_sha|$run_attempt|$run_event|$head_branch" >> "$pr_data_file"
+            done
+          done
+
+          echo ""
+          echo "========================================="
+          echo "📊 Active Runs Summary"
+          echo "========================================="
+          echo ""
+
+          if [ ! -s "$pr_data_file" ]; then
+            echo "✅ No active runs found"
+            rm -f "$pr_data_file"
+            exit 0
+          fi
+
+          # Get unique PR numbers (exclude NO_PR entries)
+          pr_numbers=$(cut -d'|' -f1 < "$pr_data_file" | grep -v '^NO_PR$' | sort -u || true)
+
+          # Separate high priority and normal PRs
+          high_priority_prs=()
+          normal_prs=()
+
+          for pr_num in $pr_numbers; do
+            labels=$(gh pr view "$pr_num" --repo "$REPO" --json labels \
+              | jq -r '.labels[].name' 2>/dev/null || true)
+
+            if echo "$labels" | grep -Fxq "high priority"; then
+              high_priority_prs+=($pr_num)
+            else
+              normal_prs+=($pr_num)
+            fi
+          done
+
+          # Combine: high priority first, then normal
+          sorted_pr_numbers=("${high_priority_prs[@]}" "${normal_prs[@]}")
+
+          pr_count=0
+          total_running=0
+          total_queued=0
+
+          for pr_num in "${sorted_pr_numbers[@]}"; do
+            pr_count=$((pr_count + 1))
+
+            # Get PR details
+            pr_info=$(gh pr view "$pr_num" --repo "$REPO" --json title,author,labels,url 2>/dev/null || true)
+
+            if [ -z "$pr_info" ]; then
+              continue
+            fi
+
+            pr_title=$(echo "$pr_info" | jq -r '.title')
+            pr_author=$(echo "$pr_info" | jq -r '.author.login')
+            pr_url=$(echo "$pr_info" | jq -r '.url')
+            pr_labels=$(echo "$pr_info" | jq -r '.labels[].name' | paste -sd ", " -)
+
+            if [ -z "$pr_labels" ]; then
+              pr_labels="(no labels)"
+            fi
+
+            # Add priority indicator
+            priority_indicator=""
+            if echo "$pr_labels" | grep -q "high priority"; then
+              priority_indicator="🔴 [HIGH PRIORITY] "
+            fi
+
+            echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+            echo "🔗 ${priority_indicator}PR #$pr_num: $pr_title"
+            echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+            echo "👤 Author: $pr_author"
+            echo "🏷️  Labels: $pr_labels"
+            echo "🔗 URL: $pr_url"
+            echo ""
+
+            # Get all runs for this PR
+            pr_runs=$(grep "^$pr_num|" "$pr_data_file")
+
+            pr_running_total=0
+            pr_queued_total=0
+
+            echo "$pr_runs" | while read -r line; do
+              workflow=$(echo "$line" | cut -d'|' -f2)
+              run_id=$(echo "$line" | cut -d'|' -f3)
+              status=$(echo "$line" | cut -d'|' -f4)
+              running=$(echo "$line" | cut -d'|' -f5)
+              queued=$(echo "$line" | cut -d'|' -f6)
+              runners=$(echo "$line" | cut -d'|' -f7)
+              queue_min=$(echo "$line" | cut -d'|' -f8)
+              created=$(echo "$line" | cut -d'|' -f9)
+              attempt=$(echo "$line" | cut -d'|' -f11)
+
+              pr_running_total=$((pr_running_total + running))
+              pr_queued_total=$((pr_queued_total + queued))
+
+              run_url="https://github.com/$REPO/actions/runs/$run_id"
+
+              # Calculate retry count for this specific run
+              retry_count=$((attempt - 1))
+
+              # Show retry indicator
+              retry_indicator=""
+              if [ "$retry_count" -gt 0 ]; then
+                retry_indicator=" 🔄 Retry #$retry_count"
+              fi
+
+              echo "  📦 Workflow: $workflow (Run #$run_id)$retry_indicator"
+              echo "     Status: $status"
+              echo "     🟢 Running jobs: $running"
+              echo "     🟡 Queued jobs: $queued"
+
+              if [ "$running" -gt 0 ] && [ "$runners" != "" ]; then
+                echo "     🖥️  Runners: $runners"
+              fi
+
+              if [ "$queue_min" -gt 0 ]; then
+                echo "     ⏱️  Queue time: ${queue_min} minutes"
+              fi
+
+              echo "     🔗 Run URL: $run_url"
+              echo ""
+            done
+
+            # Summary for this PR
+            pr_running_total=$(grep "^$pr_num|" "$pr_data_file" | cut -d'|' -f5 | awk '{sum+=$1} END {print sum+0}')
+            pr_queued_total=$(grep "^$pr_num|" "$pr_data_file" | cut -d'|' -f6 | awk '{sum+=$1} END {print sum+0}')
+
+            total_running=$((total_running + pr_running_total))
+            total_queued=$((total_queued + pr_queued_total))
+
+            echo "  📊 PR Total: $pr_running_total running, $pr_queued_total queued"
+            echo ""
+          done
+
+          # --- Non-PR Runs Section ---
+          non_pr_runs=$(grep '^NO_PR|' "$pr_data_file" 2>/dev/null || true)
+          non_pr_running=0
+          non_pr_queued=0
+
+          if [ -n "$non_pr_runs" ]; then
+            echo "========================================="
+            echo "📦 Non-PR Runs (manual / scheduled / other)"
+            echo "========================================="
+            echo ""
+
+            echo "$non_pr_runs" | while read -r line; do
+              workflow=$(echo "$line" | cut -d'|' -f2)
+              run_id=$(echo "$line" | cut -d'|' -f3)
+              status=$(echo "$line" | cut -d'|' -f4)
+              running=$(echo "$line" | cut -d'|' -f5)
+              queued=$(echo "$line" | cut -d'|' -f6)
+              runners=$(echo "$line" | cut -d'|' -f7)
+              queue_min=$(echo "$line" | cut -d'|' -f8)
+              created=$(echo "$line" | cut -d'|' -f9)
+              attempt=$(echo "$line" | cut -d'|' -f11)
+              event=$(echo "$line" | cut -d'|' -f12)
+              branch=$(echo "$line" | cut -d'|' -f13)
+
+              run_url="https://github.com/$REPO/actions/runs/$run_id"
+
+              retry_count=$((attempt - 1))
+              retry_indicator=""
+              if [ "$retry_count" -gt 0 ]; then
+                retry_indicator=" 🔄 Retry #$retry_count"
+              fi
+
+              echo "  📦 Workflow: $workflow (Run #$run_id)$retry_indicator"
+              echo "     Event: $event"
+              echo "     Branch: $branch"
+              echo "     Status: $status"
+              echo "     🟢 Running jobs: $running"
+              echo "     🟡 Queued jobs: $queued"
+
+              if [ "$running" -gt 0 ] && [ "$runners" != "" ]; then
+                echo "     🖥️  Runners: $runners"
+              fi
+
+              if [ "$queue_min" -gt 0 ]; then
+                echo "     ⏱️  Queue time: ${queue_min} minutes"
+              fi
+
+              echo "     🔗 Run URL: $run_url"
+              echo ""
+            done
+
+            non_pr_running=$(echo "$non_pr_runs" | cut -d'|' -f5 | awk '{sum+=$1} END {print sum+0}')
+            non_pr_queued=$(echo "$non_pr_runs" | cut -d'|' -f6 | awk '{sum+=$1} END {print sum+0}')
+            non_pr_count=$(echo "$non_pr_runs" | wc -l | tr -d ' ')
+
+            total_running=$((total_running + non_pr_running))
+            total_queued=$((total_queued + non_pr_queued))
+
+            echo "  📊 Non-PR Total: $non_pr_running running, $non_pr_queued queued"
+            echo ""
+          fi
+
+          # Overall summary
+          echo "========================================="
+          echo "📈 Overall Summary"
+          echo "========================================="
+          echo "Total PRs with active runs: $pr_count"
+          echo "Total non-PR active runs: ${non_pr_count:-0}"
+          echo "Total running jobs: $total_running"
+          echo "Total queued jobs: $total_queued"
+          echo "========================================="
+
+          # Cleanup
+          rm -f "$pr_data_file"

third_party/sglang/.github/workflows/nightly-link-check.yml

@@ -0,0 +1,32 @@
+name: Nightly Link Check
+
+on:
+  schedule:
+    - cron: "0 2 * * *"
+  workflow_dispatch:
+
+concurrency:
+  group: nightly-link-check-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  lychee-online:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Run lychee online link checks
+        uses: lycheeverse/lychee-action@8646ba30535128ac92d33dfc9133794bfdd9b411 # v2
+        with:
+          fail: true
+          args: >-
+            --config .github/linters/lychee-ci.toml
+            README.md
+            docs/**/*.md
+            docs/**/*.rst
+            docs/**/*.ipynb
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

third_party/sglang/.github/workflows/nightly-release-gateway.yml

@@ -0,0 +1,196 @@
+# Nightly release workflow for SGLang Model Gateway
+
+name: Nightly Release SGLang Model Gateway to PyPI
+
+on:
+  schedule:
+    # Run at 2 AM UTC every day
+    - cron: '0 2 * * *'
+  workflow_dispatch:  # Allow manual trigger
+
+jobs:
+  build:
+    name: build on ${{ matrix.platform || matrix.os }} (${{ matrix.target }} - ${{ matrix.manylinux || 'auto' }})
+    runs-on: ${{ matrix.os }}-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu, macos, windows]
+        target: [x86_64, aarch64]
+        manylinux: [auto]
+        include:
+          - os: ubuntu
+            platform: linux
+          - os: windows
+            ls: dir
+            target: x86_64
+            python-architecture: x64
+            interpreter: 3.9 3.10 3.11 3.12 3.13
+          - os: macos
+            target: aarch64
+            interpreter: 3.9 3.10 3.11 3.12 3.13
+          - os: ubuntu
+            platform: linux
+            target: aarch64
+          # musllinux
+          - os: ubuntu
+            platform: linux
+            target: x86_64
+            manylinux: musllinux_1_1
+          - os: ubuntu
+            platform: linux
+            target: aarch64
+            manylinux: musllinux_1_1
+        exclude:
+          - os: windows
+            target: aarch64
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          path: sglang-repo
+
+      - name: Move sgl-model-gateway folder to root and delete sglang-repo
+        run: |
+          mv sglang-repo/sgl-model-gateway/* .
+          rm -rf sglang-repo
+          ls -alt
+        shell: bash
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          architecture: ${{ matrix.python-architecture || 'x64' }}
+
+      - name: Modify version for nightly release
+        run: |
+          # Get current version from pyproject.toml
+          CURRENT_VERSION=$(python -c "import tomllib; print(tomllib.load(open('bindings/python/pyproject.toml', 'rb'))['project']['version'])" 2>/dev/null || python -c "import tomli; print(tomli.load(open('bindings/python/pyproject.toml', 'rb'))['project']['version'])")
+          # Create nightly version with date: e.g., 0.2.1.dev20250128
+          NIGHTLY_VERSION="${CURRENT_VERSION}.dev$(date +%Y%m%d)"
+          echo "Nightly version: $NIGHTLY_VERSION"
+
+          # Update pyproject.toml with nightly version (temporary, not committed)
+          sed -i.bak "s/version = \"${CURRENT_VERSION}\"/version = \"${NIGHTLY_VERSION}\"/" bindings/python/pyproject.toml
+
+          # Verify the change
+          cat bindings/python/pyproject.toml | grep "^version"
+        shell: bash
+
+      - name: Install twine and tomli
+        run: pip install -U twine tomli
+
+      - name: Install protoc (macOS)
+        if: matrix.os == 'macos'
+        run: brew install protobuf
+
+      - name: Install protoc (Windows)
+        if: matrix.os == 'windows'
+        run: choco install protoc -y
+
+      - name: Build wheels
+        uses: PyO3/maturin-action@v1
+        with:
+          working-directory: bindings/python
+          target: ${{ matrix.target }}
+          manylinux: ${{ matrix.manylinux || 'auto' }}
+          args: --release --out dist --features vendored-openssl --interpreter ${{ matrix.interpreter || '3.9 3.10 3.11 3.12 3.13 3.14' }}
+          rust-toolchain: stable
+          docker-options: -e CI -e CC_aarch64_unknown_linux_gnu=aarch64-linux-gnu-gcc -e CXX_aarch64_unknown_linux_gnu=aarch64-linux-gnu-g++
+          before-script-linux: |
+            # Install build dependencies (perl/make for vendored OpenSSL, protoc for gRPC)
+            if command -v yum &> /dev/null; then
+              yum update -y && yum install -y wget unzip gcc gcc-c++ perl-core make
+              # Install cross-compilation toolchain for aarch64 if needed
+              if [ "${{ matrix.target }}" = "aarch64" ]; then
+                yum install -y gcc-aarch64-linux-gnu gcc-c++-aarch64-linux-gnu || true
+              fi
+            elif command -v apt-get &> /dev/null; then
+              apt-get update && apt-get install -y wget unzip gcc g++ perl make
+              # Install cross-compilation toolchain for aarch64 if needed
+              if [ "${{ matrix.target }}" = "aarch64" ]; then
+                apt-get install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu || true
+              fi
+            fi
+            (cd /tmp && \
+             wget https://github.com/protocolbuffers/protobuf/releases/download/v32.0/protoc-32.0-linux-x86_64.zip && \
+             unzip protoc-32.0-linux-x86_64.zip -d /usr/local && \
+             rm protoc-32.0-linux-x86_64.zip)
+            protoc --version
+
+      - name: List built packages
+        run: ${{ matrix.ls || 'ls -lh' }} bindings/python/dist/
+
+      - name: Check packages
+        run: twine check --strict bindings/python/dist/*
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: packages-${{ matrix.os }}-${{ matrix.target }}-${{ matrix.manylinux || 'auto' }}
+          path: bindings/python/dist/
+
+  build-sdist:
+    name: Build SDist
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          path: sglang-repo
+
+      - name: Move sgl-model-gateway folder to root and delete sglang-repo
+        run: |
+          mv sglang-repo/sgl-model-gateway/* .
+          rm -rf sglang-repo
+          ls -alt
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Modify version for nightly release
+        run: |
+          # Get current version from pyproject.toml
+          CURRENT_VERSION=$(python -c "import tomllib; print(tomllib.load(open('bindings/python/pyproject.toml', 'rb'))['project']['version'])" 2>/dev/null || python -c "import tomli; print(tomli.load(open('bindings/python/pyproject.toml', 'rb'))['project']['version'])")
+          # Create nightly version with date: e.g., 0.2.1.dev20250128
+          NIGHTLY_VERSION="${CURRENT_VERSION}.dev$(date +%Y%m%d)"
+          echo "Nightly version: $NIGHTLY_VERSION"
+
+          # Update pyproject.toml with nightly version (temporary, not committed)
+          sed -i "s/version = \"${CURRENT_VERSION}\"/version = \"${NIGHTLY_VERSION}\"/" bindings/python/pyproject.toml
+
+          # Verify the change
+          cat bindings/python/pyproject.toml | grep "^version"
+
+      - name: Build SDist
+        uses: PyO3/maturin-action@v1
+        with:
+          working-directory: bindings/python
+          command: sdist
+          args: --out dist
+          rust-toolchain: stable
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: sdist
+          path: bindings/python/dist/*.tar.gz
+
+  upload:
+    name: Upload to TestPyPI
+    if: github.repository == 'sgl-project/sglang'  # Ensure this job only runs for the sgl-project/sglang repository
+    needs: [build, build-sdist]
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          path: dist
+          merge-multiple: true
+
+      - name: Upload to TestPyPI
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.TEST_PYPI_TOKEN_ROUTER }}
+        run: |
+          pip install twine
+          twine upload --repository testpypi dist/* --verbose

third_party/sglang/.github/workflows/nightly-test-intel.yml

@@ -0,0 +1,33 @@
+name: Nightly Test (Intel)
+
+on:
+  schedule:
+    - cron: '0 0 * * *'
+  push:
+    branches:
+      - main
+    paths:
+      - "python/sglang/version.py"
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: "Branch, tag or SHA to checkout"
+        required: false
+        type: string
+        default: ""
+
+concurrency:
+  group: nightly-test-intel-${{ inputs.ref || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
+
+jobs:
+  # Placeholder for Intel GPU tests
+  # Add Intel-specific nightly test workflows here when available
+
+  placeholder:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Placeholder
+        run: echo "Intel nightly tests will be added here"

third_party/sglang/.github/workflows/nightly-test-npu.yml

@@ -0,0 +1,428 @@
+name: Nightly Test (NPU)
+
+on:
+  schedule:
+    - cron: '0 18 * * *'  # Execute at 2:00 a.m. Beijing Time every day
+  pull_request:
+    branches:
+      - main
+    paths:
+      - ".github/workflows/nightly-test-npu.yml"
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
+        required: false
+        type: string
+        default: ''
+      job_filter:
+        description: 'Select which job to run (leave empty or "all" to run all jobs)'
+        required: false
+        type: string
+        default: 'all'
+      image_a3:
+        description: 'The a3 running docker image of the test task.'
+        required: false
+        type: string
+        default: 'swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11'
+      skip_install_flag:
+        description: 'Indicates whether to skip the installation of sglang, defaulting to false.'
+        required: false
+        type: string
+        default: 'false'
+
+
+concurrency:
+  group: nightly-test-npu-${{ inputs.ref || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
+
+jobs:
+  set-image-config:
+    runs-on: ubuntu-latest
+    outputs:
+      ref: ${{ steps.set-vars.outputs.ref }}
+      job_filter: ${{ steps.set-vars.outputs.job_filter }}
+      image_a3: ${{ steps.set-vars.outputs.image_a3 }}
+      skip_install_flag: ${{ steps.set-vars.outputs.skip_install_flag }}
+    steps:
+      # When triggered by PR, no inputs parameters are used. The latest community code is tested by default.
+      - name: Set image config
+        id: set-vars
+        run: |
+          if [ -z "${{ inputs.ref }}" ]; then
+            echo "ref=" >> $GITHUB_OUTPUT
+          else
+            echo "ref=${{ inputs.ref }}" >> $GITHUB_OUTPUT
+          fi
+
+          if [ -z "${{ inputs.job_filter }}" ]; then
+            echo "job_filter=all" >> $GITHUB_OUTPUT
+          else
+            echo "job_filter=${{ inputs.job_filter }}" >> $GITHUB_OUTPUT
+          fi
+
+          if [ -z "${{ inputs.image_a3 }}" ]; then
+            echo "image_a3=swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11" >> $GITHUB_OUTPUT
+          else
+            echo "image_a3=${{ inputs.image_a3 }}" >> $GITHUB_OUTPUT
+          fi
+
+          if [ -z "${{ inputs.skip_install_flag }}" ]; then
+            echo "skip_install_flag=false" >> $GITHUB_OUTPUT
+          else
+            echo "skip_install_flag=${{ inputs.skip_install_flag }}" >> $GITHUB_OUTPUT
+          fi
+
+  nightly-1-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-2
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0, 1]
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.32 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite nightly-1-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 --auto-partition-id ${{ matrix.part }} --auto-partition-size 2
+
+  nightly-2-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-2
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0]
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.32 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite nightly-2-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 --auto-partition-id ${{ matrix.part }} --auto-partition-size 1
+
+  nightly-4-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-4
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0]
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref|| github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.32 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite nightly-4-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 --auto-partition-id ${{ matrix.part }} --auto-partition-size 1
+
+  nightly-8-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-8
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0]
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.32 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite nightly-8-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 --auto-partition-id ${{ matrix.part }} --auto-partition-size 1
+
+  nightly-16-npu-a3:
+    needs: [set-image-config]
+    if: ${{ (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request') }}
+    runs-on: linux-aarch64-a3-16
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0, 1]
+    container:
+      image: ${{ needs.set-image-config.outputs.image_a3 }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ needs.set-image-config.outputs.ref || github.ref }}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          if [ ${{ needs.set-image-config.outputs.skip_install_flag }} != "true" ];then
+            bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          fi
+
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Print Log Information
+        run: |
+          bash scripts/ci/npu/npu_log_print.sh
+
+      - name: Run test
+        timeout-minutes: 240
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          pip install sglang_router
+          hf download lmms-lab/MMMU --repo-type dataset
+          pip install sentence_transformers torchaudio==2.8.0
+          pip install protobuf==6.31.1 zss pre-commit wandb>=0.16.0 tenacity==8.3.0 loguru openpyxl latex2sympy2 zstandard transformers-stream-generator tqdm-multiprocess pycocoevalcap
+          pip install yt-dlp sentencepiece==0.1.99 nltk av ftfy sqlitedict==2.1.0 sacrebleu>=1.5.0 pytablewriter black==24.1.0 isort==5.13.2 peft>=0.2.0 accelerate>=0.29.1
+          pip install jsonlines httpx==0.25.0 evaluate>=0.4.0 datasets==2.16.1 numexpr xgrammar==0.1.32 numpy==1.26.4 dotenv
+          git clone --branch v0.3.3 --depth 1 https://github.com/EvolvingLMMs-Lab/lmms-eval.git
+          cd ./lmms-eval
+          nohup pip install . > lmmslog.txt 2>&1 &
+          sleep 120
+          export PYTHONPATH=$PYTHONPATH:$(pwd)
+          cd ../
+          cd test
+          python3 run_suite.py --hw npu --suite nightly-16-npu-a3 --nightly --continue-on-error --timeout-per-file 3600 --auto-partition-id ${{ matrix.part }} --auto-partition-size 2
+
+  check-all-jobs:
+    if: github.repository == 'sgl-project/sglang' && always()
+    needs:
+      - nightly-1-npu-a3
+      - nightly-2-npu-a3
+      - nightly-4-npu-a3
+      - nightly-8-npu-a3
+      - nightly-16-npu-a3
+    runs-on: ubuntu-latest
+    container:
+      image: docker.m.daocloud.io/ubuntu:22.04
+    steps:
+      - name: Check if any job failed
+        run: |
+          if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then
+            echo "One or more nightly test jobs failed"
+            exit 1
+          fi
+          if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
+            echo "One or more nightly test jobs were cancelled"
+            exit 1
+          fi
+          echo "All nightly test jobs passed"

third_party/sglang/.github/workflows/nightly-test-nvidia.yml

@@ -0,0 +1,796 @@
+name: Nightly Test (Nvidia)
+
+on:
+  schedule:
+    - cron: '0 0 * * *'
+  workflow_dispatch:
+    inputs:
+      job_filter:
+        description: 'Select which job to run (leave empty or "all" to run all jobs)'
+        required: false
+        type: choice
+        default: 'all'
+        options:
+          - 'all'
+          - 'nightly-test-general-1-gpu-h100'
+          - 'nightly-test-general-4-gpu-h100'
+          - 'nightly-test-general-8-gpu-h200'
+          - 'nightly-test-general-8-gpu-h20'
+          - 'nightly-test-general-8-gpu-b200'
+          - 'nightly-test-text-accuracy-2-gpu-h100'
+          - 'nightly-test-text-perf-2-gpu-h100'
+          - 'nightly-test-vlm-accuracy-2-gpu-h100'
+          - 'nightly-test-vlm-perf-2-gpu-h100'
+          - 'nightly-test-multimodal-server-1-gpu'
+          - 'nightly-test-multimodal-server-2-gpu'
+          - 'nightly-test-perf-4-gpu-b200'
+          - 'nightly-test-perf-8-gpu-b200'
+          - 'nightly-test-specialized-8-gpu-b200'
+          - 'nightly-test-kernel-1-gpu-h100'
+          - 'nightly-test-diffusion-comparison'
+          - 'nightly-test-kernel-8-gpu-h200'
+  workflow_call:
+    inputs:
+      ref:
+        description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
+        required: false
+        type: string
+        default: ''
+      job_filter:
+        description: 'Select which job to run (leave empty or "all" to run all jobs)'
+        required: false
+        type: string
+        default: 'all'
+
+concurrency:
+  group: nightly-test-nvidia-${{ inputs.ref || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
+
+env:
+  SGLANG_IS_IN_CI: true
+  SGLANG_CUDA_COREDUMP: "1"
+  HF_HUB_DOWNLOAD_TIMEOUT: 300
+  HF_HUB_ETAG_TIMEOUT: 300
+
+jobs:
+  # General tests - 1 GPU
+  nightly-test-general-1-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-1-gpu-h100')
+    runs-on: 1-gpu-h100
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run test
+        timeout-minutes: 60
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-1-gpu --nightly --continue-on-error
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # JIT kernel full unit tests (expanded parameter ranges via SGLANG_JIT_KERNEL_RUN_FULL_TESTS)
+  nightly-test-kernel-1-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-kernel-1-gpu-h100')
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    env:
+      # Full jit_kernel test grids (see sglang.jit_kernel.utils.should_run_full_tests)
+      SGLANG_JIT_KERNEL_RUN_FULL_TESTS: "1"
+      # Match pr-test-jit-kernel workflow for consistent JIT warmup behavior
+      SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true
+      # Allow maintenance bypass on default branch (same semantics as PR JIT workflow)
+      SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run jit kernel nightly suite
+        timeout-minutes: 60
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-kernel-1-gpu --nightly --continue-on-error
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  nightly-test-kernel-8-gpu-h200:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-kernel-8-gpu-h200')
+    runs-on: 8-gpu-h200
+    timeout-minutes: 240
+    env:
+      SGLANG_JIT_KERNEL_RUN_FULL_TESTS: "1"
+      SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true
+      SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run multi-GPU jit kernel nightly suite
+        timeout-minutes: 90
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-kernel-8-gpu-h200 --nightly --continue-on-error
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # General tests - 4 GPU H100
+  nightly-test-general-4-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-4-gpu-h100')
+    runs-on: 4-gpu-h100
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run test
+        timeout-minutes: 30
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-4-gpu --nightly --continue-on-error
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # General tests - 8 GPU H200
+  nightly-test-general-8-gpu-h200:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-h200')
+    runs-on: 8-gpu-h200
+    strategy:
+      fail-fast: false
+      matrix:
+        partition: [0, 1, 2, 3]
+    env:
+      RUNNER_LABELS: 8-gpu-h200
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run common 8-GPU model tests
+        if: always()
+        timeout-minutes: 300
+        env:
+          TRACE_BASE_URL: https://raw.githubusercontent.com/sglang-bot/sglang-ci-data/main/traces/${{ github.run_id }}
+          PERFETTO_RELAY_URL: ${{ vars.PERFETTO_RELAY_URL }}
+          GPU_CONFIG: "8-gpu-h200"
+          IS_H200: "1"
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-8-gpu-common --nightly --timeout-per-file=18000 --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4
+
+      - name: Publish traces to storage repo
+        if: always()
+        continue-on-error: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GITHUB_RUN_NUMBER: ${{ github.run_number }}
+        run: |
+          TRACE_ARGS=""
+          for dir in test/performance_profiles_*/; do
+            [ -d "$dir" ] && TRACE_ARGS="$TRACE_ARGS --traces-dir $dir"
+          done
+          if [ -n "$TRACE_ARGS" ]; then
+            python3 scripts/ci/utils/publish_traces.py $TRACE_ARGS
+            find test/performance_profiles_*/ -name '*.json.gz' -delete
+          else
+            echo "No trace directories found, skipping publish"
+          fi
+
+      - name: Run test
+        timeout-minutes: 30
+        env:
+          GPU_CONFIG: "8-gpu-h200"
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-8-gpu-h200 --nightly --continue-on-error
+
+      - name: Collect performance metrics
+        if: always()
+        run: |
+          python3 scripts/ci/utils/save_metrics.py \
+            --gpu-config 8-gpu-h200 \
+            --partition ${{ matrix.partition }} \
+            --run-id ${{ github.run_id }} \
+            --output test/metrics-8gpu-h200-partition-${{ matrix.partition }}.json \
+            --search-dir test/performance_profiles_8_gpu \
+            --search-dir test
+
+      - name: Upload partition metrics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: metrics-8gpu-h200-partition-${{ matrix.partition }}
+          path: test/metrics-8gpu-h200-partition-${{ matrix.partition }}.json
+          retention-days: 5
+          if-no-files-found: ignore
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+        with:
+          artifact-suffix: ${{ matrix.partition }}
+
+  # General tests - 8 GPU H20
+  nightly-test-general-8-gpu-h20:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-h20')
+    runs-on: 8-gpu-h20
+    env:
+      SGLANG_CI_RDMA_ALL_DEVICES: "mlx5_1,mlx5_2,mlx5_3,mlx5_4"
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run test
+        timeout-minutes: 30
+        env:
+          GPU_CONFIG: "8-gpu-h20"
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-8-gpu-h20 --nightly --continue-on-error
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # General tests - 8 GPU B200
+  nightly-test-general-8-gpu-b200:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-general-8-gpu-b200')
+    runs-on: 8-gpu-b200
+    strategy:
+      fail-fast: false
+      matrix:
+        partition: [0, 1, 2, 3]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run common 8-GPU model tests
+        if: always()
+        timeout-minutes: 300
+        env:
+          TRACE_BASE_URL: https://raw.githubusercontent.com/sglang-bot/sglang-ci-data/main/traces/${{ github.run_id }}
+          PERFETTO_RELAY_URL: ${{ vars.PERFETTO_RELAY_URL }}
+          GPU_CONFIG: "8-gpu-b200"
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-8-gpu-common --nightly --timeout-per-file=12000 --continue-on-error --auto-partition-id=${{ matrix.partition }} --auto-partition-size=4
+
+      - name: Publish traces to storage repo
+        if: always()
+        continue-on-error: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GITHUB_RUN_NUMBER: ${{ github.run_number }}
+        run: |
+          TRACE_ARGS=""
+          for dir in test/performance_profiles_*/; do
+            [ -d "$dir" ] && TRACE_ARGS="$TRACE_ARGS --traces-dir $dir"
+          done
+          if [ -n "$TRACE_ARGS" ]; then
+            python3 scripts/ci/utils/publish_traces.py $TRACE_ARGS
+            find test/performance_profiles_*/ -name '*.json.gz' -delete
+          else
+            echo "No trace directories found, skipping publish"
+          fi
+
+      - name: Collect performance metrics
+        if: always()
+        run: |
+          python3 scripts/ci/utils/save_metrics.py \
+            --gpu-config 8-gpu-b200 \
+            --partition ${{ matrix.partition }} \
+            --run-id ${{ github.run_id }} \
+            --output test/metrics-8gpu-b200-partition-${{ matrix.partition }}.json \
+            --search-dir test/performance_profiles_8_gpu \
+            --search-dir test
+
+      - name: Upload partition metrics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: metrics-8gpu-b200-partition-${{ matrix.partition }}
+          path: test/metrics-8gpu-b200-partition-${{ matrix.partition }}.json
+          retention-days: 5
+          if-no-files-found: ignore
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+        with:
+          artifact-suffix: ${{ matrix.partition }}
+
+  # Text model accuracy tests
+  nightly-test-text-accuracy-2-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-text-accuracy-2-gpu-h100')
+    runs-on: 2-gpu-h100
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run eval test for text models
+        timeout-minutes: 120
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-eval-text-2-gpu --nightly --continue-on-error --timeout-per-file 4500
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # Text model performance tests
+  nightly-test-text-perf-2-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-text-perf-2-gpu-h100')
+    runs-on: 2-gpu-h100
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run performance test for text models
+        timeout-minutes: 180
+        env:
+          TRACE_BASE_URL: https://raw.githubusercontent.com/sglang-bot/sglang-ci-data/main/traces/${{ github.run_id }}
+          PERFETTO_RELAY_URL: ${{ vars.PERFETTO_RELAY_URL }}
+          GPU_CONFIG: "2-gpu-h100"
+        run: |
+          cd test
+          rm -rf performance_profiles_text_models/
+          python3 run_suite.py --hw cuda --suite nightly-perf-text-2-gpu --nightly --continue-on-error --timeout-per-file 3600
+
+      - name: Publish traces to storage repo
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GITHUB_RUN_NUMBER: ${{ github.run_number }}
+        run: |
+          python3 scripts/ci/utils/publish_traces.py --traces-dir test/performance_profiles_text_models
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # VLM accuracy tests
+  nightly-test-vlm-accuracy-2-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-vlm-accuracy-2-gpu-h100')
+    runs-on: 2-gpu-h100
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run eval test for VLM models (fixed MMMU-100)
+        timeout-minutes: 240
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-eval-vlm-2-gpu --nightly --continue-on-error --timeout-per-file 9000
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # VLM performance tests
+  nightly-test-vlm-perf-2-gpu-h100:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-vlm-perf-2-gpu-h100')
+    runs-on: 2-gpu-h100
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run perf test for VLM models (MMMU)
+        timeout-minutes: 240
+        env:
+          TRACE_BASE_URL: https://raw.githubusercontent.com/sglang-bot/sglang-ci-data/main/traces/${{ github.run_id }}
+          PERFETTO_RELAY_URL: ${{ vars.PERFETTO_RELAY_URL }}
+          GPU_CONFIG: "2-gpu-h100"
+        run: |
+          cd test
+          rm -rf performance_profiles_vlms/
+          python3 run_suite.py --hw cuda --suite nightly-perf-vlm-2-gpu --nightly --continue-on-error --timeout-per-file 3600
+
+      - name: Publish traces to storage repo
+        env:
+          GITHUB_TOKEN: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GITHUB_RUN_NUMBER: ${{ github.run_number }}
+        run: |
+          python3 scripts/ci/utils/publish_traces.py --traces-dir test/performance_profiles_vlms
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # diffusion performance tests
+  nightly-test-multimodal-server-1-gpu:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-multimodal-server-1-gpu')
+    runs-on: 1-gpu-h100
+    strategy:
+      fail-fast: false
+      max-parallel: 5
+      matrix:
+        part: [0, 1]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+          pip install slack_sdk
+
+      - name: Run diffusion server tests
+        env:
+          SGLANG_DIFFUSION_SLACK_TOKEN: ${{ secrets.SGLANG_DIFFUSION_SLACK_TOKEN }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GPU_CONFIG: "1-gpu-h100"
+
+        timeout-minutes: 90
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py \
+            --suite 1-gpu \
+            --partition-id ${{ matrix.part }} \
+            --total-partitions 2
+
+      - name: Collect diffusion performance metrics
+        if: always()
+        run: |
+          python3 scripts/ci/utils/diffusion/save_diffusion_metrics.py \
+            --gpu-config 1-gpu-h100 \
+            --run-id ${{ github.run_id }} \
+            --output python/diffusion-metrics-1gpu-partition-${{ matrix.part }}.json \
+            --results-json python/diffusion-results.json
+
+      - name: Upload diffusion metrics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: diffusion-metrics-1gpu-partition-${{ matrix.part }}
+          path: python/diffusion-metrics-1gpu-partition-${{ matrix.part }}.json
+          retention-days: 90
+          if-no-files-found: ignore
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+        with:
+          artifact-suffix: ${{ matrix.part }}
+
+  nightly-test-multimodal-server-2-gpu:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-multimodal-server-2-gpu')
+    runs-on: 2-gpu-h100
+    strategy:
+      fail-fast: false
+      max-parallel: 5
+      matrix:
+        part: [0, 1]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+          pip install slack_sdk
+
+      - name: Run diffusion server tests
+        env:
+          SGLANG_DIFFUSION_SLACK_TOKEN: ${{ secrets.SGLANG_DIFFUSION_SLACK_TOKEN }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          GPU_CONFIG: "2-gpu-h100"
+
+        timeout-minutes: 90
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py \
+            --suite 2-gpu \
+            --partition-id ${{ matrix.part }} \
+            --total-partitions 2
+
+      - name: Collect diffusion performance metrics
+        if: always()
+        run: |
+          python3 scripts/ci/utils/diffusion/save_diffusion_metrics.py \
+            --gpu-config 2-gpu-h100 \
+            --run-id ${{ github.run_id }} \
+            --output python/diffusion-metrics-2gpu-partition-${{ matrix.part }}.json \
+            --results-json python/diffusion-results.json
+
+      - name: Upload diffusion metrics
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: diffusion-metrics-2gpu-partition-${{ matrix.part }}
+          path: python/diffusion-metrics-2gpu-partition-${{ matrix.part }}.json
+          retention-days: 90
+          if-no-files-found: ignore
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+        with:
+          artifact-suffix: ${{ matrix.part }}
+
+  # B200 Performance tests - 4 GPU
+  nightly-test-perf-4-gpu-b200:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-4-gpu-b200')
+    runs-on: 4-gpu-b200
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run test
+        timeout-minutes: 300
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-4-gpu-b200 --nightly --continue-on-error --timeout-per-file 12000
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # Specialized B200 tests - 8 GPU, for specific backends and configs
+  nightly-test-specialized-8-gpu-b200:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-perf-8-gpu-b200' || inputs.job_filter == 'nightly-test-specialized-8-gpu-b200')
+    runs-on: 8-gpu-b200
+    env:
+      RUNNER_LABELS: 8-gpu-b200
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run test
+        timeout-minutes: 120
+        env:
+          GPU_CONFIG: "8-gpu-b200"
+        run: |
+          cd test
+          python3 run_suite.py --hw cuda --suite nightly-8-gpu-b200 --nightly --continue-on-error --timeout-per-file 2400
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # Diffusion cross-framework comparison
+  nightly-test-diffusion-comparison:
+    if: github.repository == 'sgl-project/sglang' && (inputs.job_filter == '' || inputs.job_filter == 'all' || inputs.job_filter == 'nightly-test-diffusion-comparison')
+    runs-on: 4-gpu-h100
+    timeout-minutes: 240
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run cross-framework comparison
+        env:
+          GITHUB_SHA: ${{ github.sha }}
+          GITHUB_RUN_ID: ${{ github.run_id }}
+          PYTHONUNBUFFERED: "1"
+        timeout-minutes: 210
+        run: |
+          python3 -u scripts/ci/utils/diffusion/run_comparison.py \
+            --output comparison-results.json
+
+      - name: Generate dashboard
+        if: always()
+        env:
+          GH_PAT_FOR_NIGHTLY_CI_DATA: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          python3 scripts/ci/utils/diffusion/generate_diffusion_dashboard.py \
+            --results comparison-results.json \
+            --output dashboard.md \
+            --charts-dir comparison-charts \
+            --fetch-history \
+            --step-summary
+
+      - name: Publish to sglang-ci-data
+        if: always()
+        env:
+          GH_PAT_FOR_NIGHTLY_CI_DATA: ${{ secrets.GH_PAT_FOR_NIGHTLY_CI_DATA }}
+        run: |
+          python3 scripts/ci/utils/diffusion/publish_comparison_results.py \
+            --results comparison-results.json \
+            --dashboard dashboard.md \
+            --charts-dir comparison-charts
+
+      - name: Upload comparison artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: diffusion-comparison-${{ github.run_id }}
+          path: |
+            comparison-results.json
+            dashboard.md
+            comparison-charts/
+            comparison-logs/
+          retention-days: 90
+          if-no-files-found: ignore
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  # Consolidate performance metrics from all jobs
+  consolidate-metrics:
+    if: github.repository == 'sgl-project/sglang' && always()
+    needs:
+      - nightly-test-general-8-gpu-h200
+      - nightly-test-general-8-gpu-b200
+      - nightly-test-multimodal-server-1-gpu
+      - nightly-test-multimodal-server-2-gpu
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Download all partition metrics
+        uses: actions/download-artifact@v4
+        with:
+          pattern: "*metrics-*"
+          path: metrics/
+          merge-multiple: true
+
+      - name: List downloaded metrics
+        run: |
+          echo "Downloaded metrics files:"
+          find metrics/ -name "*.json" -type f 2>/dev/null || echo "No metrics files found"
+
+      - name: Merge metrics
+        run: |
+          python3 scripts/ci/utils/merge_metrics.py \
+            --input-dir metrics/ \
+            --output consolidated-metrics-${{ github.run_id }}.json \
+            --run-id ${{ github.run_id }} \
+            --commit-sha ${{ github.sha }} \
+            --branch ${{ github.ref_name }}
+
+      - name: Upload consolidated metrics
+        uses: actions/upload-artifact@v4
+        with:
+          name: consolidated-metrics-${{ github.run_id }}
+          path: consolidated-metrics-${{ github.run_id }}.json
+          retention-days: 90
+          if-no-files-found: warn
+
+  # Final check job
+  check-all-jobs:
+    if: github.repository == 'sgl-project/sglang' && always()
+    needs:
+      - nightly-test-general-1-gpu-h100
+      - nightly-test-general-4-gpu-h100
+      - nightly-test-general-8-gpu-h200
+      - nightly-test-general-8-gpu-h20
+      - nightly-test-general-8-gpu-b200
+      - nightly-test-text-accuracy-2-gpu-h100
+      - nightly-test-text-perf-2-gpu-h100
+      - nightly-test-vlm-accuracy-2-gpu-h100
+      - nightly-test-vlm-perf-2-gpu-h100
+      - nightly-test-multimodal-server-1-gpu
+      - nightly-test-multimodal-server-2-gpu
+      - nightly-test-perf-4-gpu-b200
+      - nightly-test-specialized-8-gpu-b200
+      - nightly-test-diffusion-comparison
+      - consolidate-metrics
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check if any job failed
+        run: |
+          if [[ "${{ contains(needs.*.result, 'failure') }}" == "true" ]]; then
+            echo "One or more nightly test jobs failed"
+            exit 1
+          fi
+          if [[ "${{ contains(needs.*.result, 'cancelled') }}" == "true" ]]; then
+            echo "One or more nightly test jobs were cancelled"
+            exit 1
+          fi
+          echo "All nightly test jobs passed"

third_party/sglang/.github/workflows/open-pr-copy-from-oss.yml

@@ -0,0 +1,28 @@
+name: Open A PR to Copy Code From OSS
+
+on:
+  workflow_dispatch:
+  # schedule:
+  #   - cron: '0 10 * * *'
+
+permissions:
+  contents: write
+
+jobs:
+  copy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          ref: 'main'
+
+      - name: Install GitHub CLI (if not present)
+        run: |
+          bash scripts/code_sync/install_github_cli.sh
+
+      - name: Copy from OSS code
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_OPEN_PR_TO_PRIVATE }}
+        run: |
+          python3 scripts/code_sync/copy_from_oss.py

third_party/sglang/.github/workflows/open-pr-copy-to-oss.yml

@@ -0,0 +1,31 @@
+name: Open A PR to Copy Diff To OSS
+
+on:
+  workflow_dispatch:
+    inputs:
+      commit_sha:
+        description: 'The commit SHA to copy. Defaults to LAST to copy the latest commit.'
+        required: false
+        default: 'LAST'
+
+permissions:
+  contents: write
+
+jobs:
+  copy:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Install GitHub CLI (if not present)
+        run: |
+          bash scripts/code_sync/install_github_cli.sh
+
+      - name: Copy to OSS code
+        env:
+          GH_TOKEN: ${{ secrets.GH_PAT_FOR_OPEN_PR_TO_OSS }}
+        run: |
+          python3 scripts/code_sync/copy_to_oss.py --commit ${{ github.event.inputs.commit_sha }}

third_party/sglang/.github/workflows/patch-docker-dev.yml

@@ -0,0 +1,115 @@
+name: Patch Docker Image
+
+on:
+  workflow_dispatch:
+    inputs:
+      pr_numbers:
+        description: "Comma-separated PR numbers to apply (e.g. 18962,19010)"
+        required: false
+        default: ""
+      image_tag:
+        description: "Base image tag to patch (e.g. dev-x86, dev-x86-cu13)"
+        required: true
+
+concurrency:
+  group: patch-docker-${{ inputs.image_tag }}
+  cancel-in-progress: true
+
+jobs:
+  patch:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: x64-docker-build-node
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Pull base image and extract commit
+        run: |
+          IMAGE="lmsysorg/sglang:${{ inputs.image_tag }}"
+          docker pull "${IMAGE}"
+          if BASE_SHA=$(docker run --rm "${IMAGE}" git -C /sgl-workspace/sglang rev-parse HEAD 2>/dev/null); then
+            echo "Image built from commit: ${BASE_SHA}"
+          else
+            BASE_SHA=""
+            echo "::warning::Image has no .git directory — cannot extract base commit"
+          fi
+          echo "BASE_SHA=${BASE_SHA}" >> "$GITHUB_ENV"
+
+      - name: Generate patches
+        run: |
+          git config --global --add safe.directory "$GITHUB_WORKSPACE"
+          git fetch origin main
+          mkdir -p /tmp/patch-ctx
+
+          if [ -n "${{ inputs.pr_numbers }}" ]; then
+            IFS=',' read -ra PRS <<< "${{ inputs.pr_numbers }}"
+            for pr in "${PRS[@]}"; do
+              pr=$(echo "${pr}" | xargs)
+              echo "Fetching PR #${pr}"
+              git fetch origin "pull/${pr}/head:pr-${pr}"
+              MERGE_BASE=$(git merge-base origin/main "pr-${pr}")
+              echo "  PR #${pr}: merge-base=${MERGE_BASE}"
+              git diff "${MERGE_BASE}..pr-${pr}" > "/tmp/patch-ctx/${pr}.patch"
+              echo "  PR #${pr}: $(wc -l < /tmp/patch-ctx/${pr}.patch) lines"
+            done
+          elif [ -n "${BASE_SHA}" ]; then
+            echo "Generating diff: image ${BASE_SHA} → latest main"
+            git fetch origin "${BASE_SHA}"
+            git diff "${BASE_SHA}..origin/main" > /tmp/patch-ctx/main.patch
+            echo "  main: $(wc -l < /tmp/patch-ctx/main.patch) lines"
+          else
+            echo "::error::No PR numbers specified and image has no .git — cannot generate diff against main"
+            exit 1
+          fi
+
+          TOTAL=$(cat /tmp/patch-ctx/*.patch | wc -l)
+          if [ "${TOTAL}" -eq 0 ]; then
+            echo "::warning::All patches are empty — image is already up to date"
+            echo "SKIP_BUILD=true" >> "$GITHUB_ENV"
+          fi
+
+      - name: Build patched image
+        if: env.SKIP_BUILD != 'true'
+        run: |
+          IMAGE="lmsysorg/sglang:${{ inputs.image_tag }}"
+
+          cat <<'DOCKERFILE' > /tmp/patch-ctx/Dockerfile
+          ARG BASE_IMAGE
+          FROM ${BASE_IMAGE}
+          COPY *.patch /tmp/patches/
+          RUN cd /sgl-workspace/sglang \
+              && for p in /tmp/patches/*.patch; do \
+                   if [ ! -s "${p}" ]; then \
+                     echo "Skipping ${p} (empty)"; \
+                   else \
+                     echo "Applying ${p}..." \
+                     && patch -p1 --fuzz=2 --no-backup-if-mismatch -f < "${p}" \
+                     || { echo "ERROR: Failed to apply ${p}"; exit 1; }; \
+                   fi; \
+                 done \
+              && rm -rf /tmp/patches
+          DOCKERFILE
+
+          docker build \
+            --no-cache \
+            --build-arg BASE_IMAGE="${IMAGE}" \
+            -t "${IMAGE}" \
+            /tmp/patch-ctx/
+
+      - name: Push patched image
+        if: env.SKIP_BUILD != 'true'
+        run: |
+          IMAGE="lmsysorg/sglang:${{ inputs.image_tag }}"
+          docker push "${IMAGE}"
+
+          echo "### Patched \`${IMAGE}\`" >> "$GITHUB_STEP_SUMMARY"
+          echo "- **Base commit:** \`${BASE_SHA:-unknown (no .git)}\`" >> "$GITHUB_STEP_SUMMARY"
+          echo "- **Source:** ${{ inputs.pr_numbers && format('PRs: {0}', inputs.pr_numbers) || 'latest main' }}" >> "$GITHUB_STEP_SUMMARY"

third_party/sglang/.github/workflows/pr-benchmark-rust.yml

@@ -0,0 +1,198 @@
+name: PR Benchmark (SMG Components)
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - "sgl-model-gateway/**"
+  pull_request:
+    branches: [ main ]
+    paths:
+      - "sgl-model-gateway/**"
+  workflow_dispatch:
+
+concurrency:
+  group: pr-benchmark-rust-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  RUSTC_WRAPPER: sccache
+  SCCACHE_GHA_ENABLED: "true"
+
+permissions:
+  contents: read
+  pull-requests: write
+  issues: write
+
+jobs:
+  benchmark-compile-check:
+    name: Benchmark Compilation Check
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_gateway_dependencies.sh
+
+      - name: Configure sccache
+        uses: mozilla-actions/sccache-action@v0.0.9
+        with:
+          version: "v0.12.0"
+          disable_annotations: true
+
+      - name: Rust cache
+        uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: sgl-model-gateway
+          shared-key: "rust-cache"
+          save-if: true
+          cache-all-crates: true
+          cache-on-failure: true
+
+      - name: Check benchmarks compile
+        run: |
+          source "$HOME/.cargo/env"
+          cd sgl-model-gateway/
+          cargo check --benches
+
+      - name: Show sccache stats
+        if: always()
+        run: sccache --show-stats
+
+  benchmark:
+    name: Benchmark - ${{ matrix.name }}
+    if: |
+      github.repository == 'sgl-project/sglang' &&
+      (github.event_name == 'push' ||
+       github.event_name == 'workflow_dispatch' ||
+       (contains(github.event.pull_request.labels.*.name, 'router-benchmark') &&
+        contains(github.event.pull_request.labels.*.name, 'run-ci')))
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - name: Request Processing
+            bench_name: request_processing
+            bench_args: "benchmark_summary --exact"
+            runner: ubuntu-latest
+            sccache_version: "v0.12.0"
+            artifact_name: request-processing-results
+            artifact_path: criterion/benchmark_summary/
+          - name: Manual Policy
+            bench_name: manual_policy_benchmark
+            bench_args: ""
+            runner: ubuntu-latest
+            sccache_version: "v0.12.0"
+            artifact_name: manual-policy-results
+            artifact_path: criterion/manual_policy*/
+    runs-on: ${{ matrix.runner }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 100
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_gateway_dependencies.sh
+
+      - name: Configure sccache
+        uses: mozilla-actions/sccache-action@v0.0.9
+        with:
+          version: ${{ matrix.sccache_version }}
+          disable_annotations: true
+
+      - name: Rust cache
+        uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: sgl-model-gateway
+          shared-key: "rust-cache"
+          cache-all-crates: true
+          cache-on-failure: true
+          save-if: true
+
+      - name: Run benchmark
+        timeout-minutes: 30
+        run: |
+          source "$HOME/.cargo/env"
+          cd sgl-model-gateway/
+          if command -v sccache &> /dev/null; then
+            echo "Testing sccache availability..."
+            export RUSTC_WRAPPER=sccache
+            export SCCACHE_GHA_ENABLED="true"
+            if sccache --start-server 2>/dev/null && sccache --show-stats 2>/dev/null; then
+              echo "sccache is working, using it for compilation"
+            else
+              echo "sccache failed to start, falling back to regular cargo"
+              unset RUSTC_WRAPPER
+              unset SCCACHE_GHA_ENABLED
+            fi
+          else
+            echo "sccache not available, using regular cargo"
+          fi
+          cargo bench --bench ${{ matrix.bench_name }} -- ${{ matrix.bench_args }} 2>&1 | tee benchmark_output.txt
+
+      - name: Upload benchmark results
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: ${{ matrix.artifact_name }}-${{ github.sha }}
+          path: |
+            sgl-model-gateway/target/${{ matrix.artifact_path }}
+            sgl-model-gateway/benchmark_output.txt
+          retention-days: 30
+
+      - name: Show sccache stats
+        if: always()
+        run: sccache --show-stats
+
+  benchmark-summary:
+    name: Benchmark Summary
+    needs: [benchmark]
+    if: always() && (github.repository == 'sgl-project/sglang' || github.event_name == 'pull_request')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Download all benchmark results
+        uses: actions/download-artifact@v4
+        with:
+          pattern: '*-results-${{ github.sha }}'
+          path: benchmark-results
+
+      - name: Generate summary
+        run: |
+          generate_section() {
+            local title="$1" dir_name="$2" lines="${3:-100}"
+            local dir="benchmark-results/${dir_name}-${{ github.sha }}"
+            echo "### $title" >> summary.md
+            if [ -d "$dir" ]; then
+              echo "✅ **Completed**" >> summary.md
+              if [ -f "$dir/benchmark_output.txt" ]; then
+                echo -e "\n<details>\n<summary>View Results</summary>\n\n\`\`\`" >> summary.md
+                tail -"$lines" "$dir/benchmark_output.txt" >> summary.md
+                echo -e "\`\`\`\n</details>" >> summary.md
+              fi
+            else
+              echo "❌ Failed or skipped" >> summary.md
+            fi
+            echo "" >> summary.md
+          }
+
+          echo "## 🚀 Benchmark Results Summary" > summary.md
+          echo "" >> summary.md
+
+          generate_section "Request Processing" "request-processing-results" 60
+          generate_section "Manual Policy (Sticky Sessions)" "manual-policy-results" 100
+
+          echo -e "---\n_Generated at $(date -u '+%Y-%m-%d %H:%M:%S UTC')_" >> summary.md
+
+          cat summary.md
+          cat summary.md >> $GITHUB_STEP_SUMMARY
+
+      - name: Upload summary
+        uses: actions/upload-artifact@v4
+        with:
+          name: benchmark-summary-${{ github.sha }}
+          path: summary.md
+          retention-days: 30

third_party/sglang/.github/workflows/pr-gate.yml

@@ -0,0 +1,254 @@
+on:
+  workflow_call:
+    inputs:
+      require-run-ci:
+        description: "Whether the PR must have the run-ci label"
+        type: boolean
+        default: true
+      cool-down-minutes:
+        description: "Cooldown period in minutes for low-permission users; 0 disables rate limiting"
+        type: number
+        default: 120
+
+jobs:
+  pr-gate:
+    # 1. for commits on main: no gating needed
+    # 2. for workflow_dispatch: this can only be triggered by users with write access
+    runs-on: ubuntu-latest
+    steps:
+      - name: Fetch latest PR info
+        if: github.event_name == 'pull_request'
+        id: pr
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const pr = await github.rest.pulls.get({
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              pull_number: context.issue.number
+            });
+            core.setOutput("labels", JSON.stringify(pr.data.labels.map(l => l.name)));
+            core.setOutput("draft", pr.data.draft);
+            core.setOutput("user", pr.data.user.login);
+
+      - name: Log PR info
+        if: github.event_name == 'pull_request'
+        run: |
+          echo "===== PR Info ====="
+          echo "PR Event: ${{ github.event_name }}"
+          echo "PR Labels: ${{ steps.pr.outputs.labels }}"
+          echo "PR Draft: ${{ steps.pr.outputs.draft }}"
+          echo "PR User: ${{ steps.pr.outputs.user }}"
+          echo "Require run-ci: ${{ inputs.require-run-ci }}"
+          echo "Cool down minutes: ${{ inputs.cool-down-minutes }}"
+          echo "==================="
+
+      - name: Block draft PR
+        if: github.event_name == 'pull_request' && fromJson(steps.pr.outputs.draft)
+        run: |
+          echo "PR is draft. Blocking CI."
+          exit 1
+
+      - name: Require run-ci label (optional)
+        if:  github.event_name == 'pull_request' && inputs.require-run-ci == true
+        run: |
+          labels='${{ steps.pr.outputs.labels }}'
+          if [[ "${{ contains(fromJson(steps.pr.outputs.labels), 'run-ci') }}" == "false" ]]; then
+            echo "Missing required label 'run-ci'. See https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests for more details."
+            exit 1
+          fi
+
+      - name: Enforce rate limit for low-permission actors (optional)
+        if: github.event_name == 'pull_request' && inputs.cool-down-minutes > 0
+        uses: actions/github-script@v7
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const DEFAULT_MINUTES = Number("${{ inputs.cool-down-minutes }}");
+            const owner = context.repo.owner;
+            const repo = context.repo.repo;
+            const eventName = context.eventName;
+            const curRun = await github.rest.actions.getWorkflowRun({
+              owner, repo, run_id: context.runId
+            });
+            let triggeringActor = curRun.data.triggering_actor?.login || context.actor;
+            if (triggeringActor === "github-actions[bot]") {
+              triggeringActor = `${{ steps.pr.outputs.user }}`;
+              core.info(
+                `triggering_actor is github-actions[bot]; substituting PR author '${triggeringActor}'.`
+              );
+            }
+
+            async function hasHighPermission(username) {
+              try {
+                const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ owner, repo, username });
+                const perm = data.permission || 'none';
+                return perm === 'write' || perm === 'maintain' || perm === 'admin';
+              } catch (e) {
+                if (e.status === 404 || e.status === 403) return false;
+                throw e;
+              }
+            }
+
+            if (await hasHighPermission(triggeringActor)) {
+              core.info(`Triggering user '${triggeringActor}' has high permission. No rate limit applied.`);
+              return;
+            }
+
+            let effectiveCooldownMinutes = DEFAULT_MINUTES;
+            let perUserCooldownMinutes = null;
+
+            try {
+              const contentResp = await github.rest.repos.getContent({
+                owner,
+                repo,
+                path: ".github/CI_PERMISSIONS.json",
+                ref: "main",
+              });
+
+              if (!Array.isArray(contentResp.data) && contentResp.data && "content" in contentResp.data) {
+                const raw = Buffer.from(
+                  contentResp.data.content,
+                  contentResp.data.encoding || "base64"
+                ).toString();
+                const ciPermissions = JSON.parse(raw);
+
+                const userPerm = ciPermissions[triggeringActor];
+                if (userPerm && typeof userPerm.cooldown_interval_minutes === "number") {
+                  perUserCooldownMinutes = userPerm.cooldown_interval_minutes;
+                  core.info(
+                    `Per-user cooldown for '${triggeringActor}' from CI_PERMISSIONS.json: ${perUserCooldownMinutes} minutes.`
+                  );
+                } else {
+                  core.info(`No per-user cooldown found for '${triggeringActor}' in CI_PERMISSIONS.json.`);
+                }
+              } else {
+                core.info("CI_PERMISSIONS.json content response is not a file; skipping per-user cooldown.");
+              }
+            } catch (e) {
+              core.info(`CI_PERMISSIONS.json not found or unreadable: ${e.message}. Using default rate limit only.`);
+            }
+
+            if (perUserCooldownMinutes !== null) {
+              effectiveCooldownMinutes = Math.min(effectiveCooldownMinutes, perUserCooldownMinutes);
+            }
+
+            if (effectiveCooldownMinutes <= 0) {
+              core.info(
+                `Effective cooldown for '${triggeringActor}' is 0 minutes; no rate limit enforced for this user.`
+              );
+              return;
+            }
+
+            const cutoff = new Date(Date.now() - effectiveCooldownMinutes * 60 * 1000);
+            core.info(
+              `Checking for workflow runs since ${cutoff.toISOString()} (last ${effectiveCooldownMinutes} minutes) for event '${eventName}'.`
+            );
+
+            const { data } = await github.rest.actions.listWorkflowRuns({
+              owner,
+              repo,
+              workflow_id: 'pr-test.yml',
+              event: eventName,
+              per_page: 100,
+            });
+
+            const runs = data.workflow_runs || [];
+
+            // Rate Limiting Logic:
+            // We only count workflow runs that actually consumed CI resources (i.e., passed the gate).
+            // A run "passes the gate" if any jobs beyond the gate jobs (check-changes, pr-gate, call-gate)
+            // actually executed (not skipped/cancelled). This prevents scenarios where:
+            // - User has PR A with missing 'run-ci' label (fails at gate)
+            // - User opens PR B with 'run-ci' label
+            // - PR B should be able to run even though PR A triggered a run recently
+
+            // Helper function to check if a run passed the gate (i.e., actually consumed CI resources)
+            async function didRunPassGate(run) {
+              try {
+                // Note: Fetching up to 100 jobs (API maximum). If a workflow has >100 jobs,
+                // we may miss some, but this is unlikely in practice.
+                const { data: jobsData } = await github.rest.actions.listJobsForWorkflowRun({
+                  owner, repo, run_id: run.id, per_page: 100
+                });
+                const jobs = jobsData.jobs || [];
+
+                // If no jobs exist yet, the run hasn't started consuming resources
+                if (jobs.length === 0) {
+                  core.info(`Run ${run.id} has no jobs yet; not counting against rate limit.`);
+                  return false;
+                }
+
+                // Gate jobs that don't consume significant CI resources
+                const gateJobs = ['check-changes', 'pr-gate', 'call-gate', 'pr-test-finish'];
+                const jobsBeyondGate = jobs.filter(j => !gateJobs.some(g => j.name === g || j.name.startsWith(g + ' ')));
+
+                // A job "ran" if it reached a terminal conclusion state that indicates actual execution
+                const ranStates = ['success', 'failure', 'timed_out', 'action_required'];
+                const hasJobsThatRan = jobsBeyondGate.some(j => j.conclusion && ranStates.includes(j.conclusion));
+                return hasJobsThatRan;
+              } catch (e) {
+                core.warning(`Could not check jobs for run ${run.id}: ${e.message}`);
+
+                // If it's a rate limit error, count it conservatively to prevent abuse
+                if (e.status === 429) {
+                  core.warning(`Hit rate limit checking run ${run.id}; counting it to be safe.`);
+                  return true;
+                }
+
+                // For cancelled/skipped runs, they likely didn't consume resources
+                if (run.conclusion === 'cancelled' || run.conclusion === 'skipped') {
+                  return false;
+                }
+
+                // Default to counting it to prevent abuse
+                return true;
+              }
+            }
+
+            // Limit the number of runs we'll check in detail to avoid API rate limits
+            const MAX_RUNS_TO_CHECK = 5;
+            let runsChecked = 0;
+            let runsSkippedAtGate = 0;
+            let recentFound = null;
+
+            for (const run of runs) {
+              if (String(run.id) === String(context.runId)) continue;
+              if (new Date(run.created_at) < cutoff) continue;
+              const isUserRun = (run.actor?.login === triggeringActor) || (run.triggering_actor?.login === triggeringActor);
+              if (!isUserRun) continue;
+
+              runsChecked++;
+              core.info(`Checking run ${run.id} (created: ${run.created_at}, conclusion: ${run.conclusion})`);
+
+              // Safety limit: if we've checked too many runs, assume the next one passed to be conservative
+              if (runsChecked > MAX_RUNS_TO_CHECK) {
+                core.warning(`Checked ${MAX_RUNS_TO_CHECK} runs; assuming this one passed gate to avoid API limits.`);
+                recentFound = run;
+                break;
+              }
+
+              // Only count runs that actually passed the gate and consumed CI resources
+              if (await didRunPassGate(run)) {
+                recentFound = run;
+                core.info(`Found recent run ${run.id} that passed gate.`);
+                break;
+              } else {
+                runsSkippedAtGate++;
+                core.info(`Run ${run.id} failed at gate; not counting against rate limit.`);
+              }
+            }
+
+            core.info(`Rate limit check summary: checked ${runsChecked} runs, ${runsSkippedAtGate} failed at gate.`);
+
+            if (recentFound) {
+              core.setFailed(
+                `User '${triggeringActor}' already triggered '${context.workflow}' via '${eventName}' at ${recentFound.created_at}. ` +
+                `Please wait ${effectiveCooldownMinutes} minutes before triggering again.`
+              );
+            } else {
+              core.info(
+                `No recent runs detected for '${triggeringActor}' within the last ${effectiveCooldownMinutes} minutes; proceeding.`
+              );
+            }

third_party/sglang/.github/workflows/pr-test-jit-kernel.yml

@@ -0,0 +1,117 @@
+name: PR Test - JIT Kernel
+
+on:
+  workflow_call:
+    inputs:
+      jit_kernel:
+        required: true
+        type: string
+      pr_head_sha:
+        required: false
+        type: string
+        default: ''
+      git_ref:
+        required: false
+        type: string
+        default: ''
+      target_stage:
+        required: false
+        type: string
+        default: ''
+      test_parallel_dispatch:
+        required: false
+        type: string
+        default: 'false'
+      skip_stage_health_check:
+        required: false
+        type: boolean
+        default: false
+
+# Workflow-level env is NOT inherited from the caller in reusable workflows (verified by CI test).
+# The github context (including github.event_name) IS inherited from the caller.
+env:
+  SGLANG_IS_IN_CI: true
+  SGLANG_CUDA_COREDUMP: "1"
+  SGLANG_JIT_DEEPGEMM_FAST_WARMUP: true
+  SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
+  SKIP_STAGE_HEALTH_CHECK: ${{ inputs.skip_stage_health_check == true && 'true' || 'false' }}
+
+jobs:
+  jit-kernel-unit-test:
+    if: |
+      github.event_name != 'schedule' &&
+      inputs.test_parallel_dispatch != 'true' &&
+      !inputs.target_stage
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run test
+        timeout-minutes: 30
+        run: |
+          cd test/
+          python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-1-gpu-large
+
+  jit-kernel-multigpu-unit-test:
+    if: |
+      github.event_name != 'schedule' &&
+      inputs.test_parallel_dispatch != 'true' &&
+      !inputs.target_stage
+    runs-on: 8-gpu-h200
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run multi-GPU test
+        timeout-minutes: 45
+        run: |
+          cd test/
+          python3 run_suite.py --hw cuda --suite stage-b-kernel-unit-8-gpu-h200
+
+  jit-kernel-benchmark-test:
+    if: |
+      github.event_name != 'schedule' &&
+      inputs.test_parallel_dispatch != 'true' &&
+      !inputs.target_stage
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run benchmark tests
+        timeout-minutes: 45
+        run: |
+          cd test/
+          python3 run_suite.py --hw cuda --suite stage-b-kernel-benchmark-1-gpu-large

third_party/sglang/.github/workflows/pr-test-multimodal-gen.yml

@@ -0,0 +1,245 @@
+name: PR Test - Multimodal Gen
+
+on:
+  workflow_call:
+    inputs:
+      multimodal_gen:
+        required: true
+        type: string
+      sgl_kernel:
+        required: true
+        type: string
+      b200_runner:
+        required: true
+        type: string
+      continue_on_error:
+        required: false
+        type: string
+        default: 'false'
+      pr_head_sha:
+        required: false
+        type: string
+        default: ''
+      git_ref:
+        required: false
+        type: string
+        default: ''
+      target_stage:
+        required: false
+        type: string
+        default: ''
+      test_parallel_dispatch:
+        required: false
+        type: string
+        default: 'false'
+      caller_needs_failure:
+        required: false
+        type: string
+        default: 'false'
+      skip_stage_health_check:
+        required: false
+        type: string
+        default: 'false'
+
+# Workflow-level env is NOT inherited from the caller in reusable workflows.
+# The github context (including github.event_name) IS inherited from the caller.
+env:
+  SGLANG_IS_IN_CI: true
+  SGLANG_CUDA_COREDUMP: "1"
+  SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
+  SKIP_STAGE_HEALTH_CHECK: ${{ inputs.skip_stage_health_check == 'true' }}
+
+jobs:
+  multimodal-gen-test-1-gpu:
+    if: |
+      (inputs.target_stage == 'multimodal-gen-test-1-gpu') ||
+      (
+        !inputs.target_stage &&
+        ((github.event_name == 'schedule' || inputs.test_parallel_dispatch == 'true') || (inputs.caller_needs_failure != 'true' && !cancelled())) &&
+        inputs.multimodal_gen == 'true'
+      )
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0, 1]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Download artifacts
+        if: inputs.sgl_kernel == 'true'
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+      - name: Run diffusion server tests
+        timeout-minutes: 240
+        env:
+          RUNAI_STREAMER_MEMORY_LIMIT: 0
+          CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py \
+            --suite 1-gpu \
+            --partition-id ${{ matrix.part }} \
+            --total-partitions 2 \
+            $CONTINUE_ON_ERROR_FLAG
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+        with:
+          artifact-suffix: ${{ matrix.part }}
+
+  multimodal-gen-test-2-gpu:
+    if: |
+      (inputs.target_stage == 'multimodal-gen-test-2-gpu') ||
+      (
+        !inputs.target_stage &&
+        ((github.event_name == 'schedule' || inputs.test_parallel_dispatch == 'true') || (inputs.caller_needs_failure != 'true' && !cancelled())) &&
+        inputs.multimodal_gen == 'true'
+      )
+    runs-on: 2-gpu-h100
+    timeout-minutes: 240
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [0, 1]
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Download artifacts
+        if: inputs.sgl_kernel == 'true'
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run diffusion server tests
+        timeout-minutes: 240
+        env:
+          RUNAI_STREAMER_MEMORY_LIMIT: 0
+          CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py \
+            --suite 2-gpu \
+            --partition-id ${{ matrix.part }} \
+            --total-partitions 2 \
+            $CONTINUE_ON_ERROR_FLAG
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+        with:
+          artifact-suffix: ${{ matrix.part }}
+
+  multimodal-gen-test-1-b200:
+    if: |
+      (inputs.target_stage == 'multimodal-gen-test-1-b200') ||
+      (
+        !inputs.target_stage &&
+        ((github.event_name == 'schedule' || inputs.test_parallel_dispatch == 'true') || (inputs.caller_needs_failure != 'true' && !cancelled())) &&
+        inputs.multimodal_gen == 'true'
+      )
+    runs-on: ${{ inputs.b200_runner }}
+    timeout-minutes: 240
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Download artifacts
+        if: inputs.sgl_kernel == 'true'
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run diffusion server tests
+        timeout-minutes: 240
+        env:
+          RUNAI_STREAMER_MEMORY_LIMIT: 0
+          CONTINUE_ON_ERROR_FLAG: ${{ inputs.continue_on_error == 'true' && '--continue-on-error' || '' }}
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py \
+            --suite 1-gpu-b200 \
+            $CONTINUE_ON_ERROR_FLAG
+
+      - uses: ./.github/actions/upload-cuda-coredumps
+        if: always()
+
+  multimodal-gen-unit-test:
+    if: |
+      (inputs.target_stage == 'multimodal-gen-unit-test') ||
+      (
+        !inputs.target_stage &&
+        ((github.event_name == 'schedule' || inputs.test_parallel_dispatch == 'true') || (inputs.caller_needs_failure != 'true' && !cancelled())) &&
+        inputs.multimodal_gen == 'true'
+      )
+    runs-on: 1-gpu-h100
+    timeout-minutes: 120
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Download artifacts
+        if: inputs.sgl_kernel == 'true'
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run diffusion unit tests
+        timeout-minutes: 60
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py --suite unit

third_party/sglang/.github/workflows/pr-test-npu.yml

@@ -0,0 +1,453 @@
+name: PR Test (NPU)
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
+        required: false
+        type: string
+        default: ''
+      run_all_tests:
+        description: "Run all tests (for releasing or testing purpose)"
+        required: false
+        type: boolean
+        default: false
+
+concurrency:
+  group: pr-test-npu-${{ inputs.ref || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
+
+jobs:
+  # ==================== Check Changes ==================== #
+  check-changes:
+    runs-on: ubuntu-latest
+    outputs:
+      changes_exist: ${{ steps.filter.outputs.main_package == 'true' || steps.filter.outputs.multimodal_gen == 'true' || steps.run-mode.outputs.run_all_tests == 'true'}}
+      main_package: ${{ steps.filter.outputs.main_package == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
+      multimodal_gen: ${{ steps.filter.outputs.multimodal_gen == 'true' || steps.run-mode.outputs.run_all_tests == 'true' }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Determine run mode
+        id: run-mode
+        run: |
+          # Run all tests for workflow_call (when ref input is provided)
+          # Note: github.event_name is inherited from caller, so we detect workflow_call by checking inputs.ref
+          if [[ "${{ inputs.run_all_tests }}" == "true" ]]; then
+            echo "run_all_tests=true" >> $GITHUB_OUTPUT
+            echo "Run mode: ALL TESTS (run_all_tests=${{ inputs.run_all_tests }})"
+          else
+            echo "run_all_tests=false" >> $GITHUB_OUTPUT
+            echo "Run mode: FILTERED (triggered by ${{ github.event_name }})"
+          fi
+
+      - name: Detect file changes
+        id: filter
+        uses: dorny/paths-filter@v3
+        if: steps.run-mode.outputs.run_all_tests != 'true'
+        with:
+          filters: |
+            main_package:
+              - "python/sglang/!(multimodal_gen)/**/!(*.md)"
+              - "python/pyproject_npu.toml"
+              - "scripts/ci/npu/npu_ci_install_dependency.sh"
+              - "test/srt/ascend/**"
+              - ".github/workflows/pr-test-npu.yml"
+            multimodal_gen:
+              - "python/sglang/multimodal_gen/**/*.!(md|ipynb)"
+              - "python/sglang/srt/**"
+              - "python/pyproject_npu.toml"
+              - "scripts/ci/npu/npu_ci_install_dependency.sh"
+              - ".github/workflows/pr-test-npu.yml"
+
+  # ==================== PR Gate ==================== #
+  pr-gate:
+    needs: check-changes
+    if: needs.check-changes.outputs.changes_exist == 'true'
+    uses: ./.github/workflows/pr-gate.yml
+    secrets: inherit
+
+  stage-b-test-1-npu-a2:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.main_package == 'true'
+    runs-on: linux-aarch64-a2-1
+    strategy:
+      fail-fast: false
+      matrix:
+        part: [ 0, 1 ]
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-910b-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh 910b
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          cd test
+          python3 run_suite.py --hw npu --suite stage-b-test-1-npu-a2 --auto-partition-id ${{ matrix.part }} --auto-partition-size 2
+
+  stage-b-test-2-npu-a2:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.main_package == 'true'
+    runs-on: linux-aarch64-a2-2
+    strategy:
+      fail-fast: true
+      matrix:
+        part: [0, 1]
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-910b-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh 910b
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          cd test
+          python3 run_suite.py --hw npu --suite stage-b-test-2-npu-a2 --auto-partition-id ${{ matrix.part }} --auto-partition-size 2
+
+  stage-b-test-4-npu-a3:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.main_package == 'true'
+    runs-on: linux-aarch64-a3-4
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          cd test
+          python3 run_suite.py --hw npu --suite stage-b-test-4-npu-a3 --timeout-per-file 3600
+
+
+  stage-b-test-16-npu-a3:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.main_package == 'true'
+    runs-on: linux-aarch64-a3-16
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh a3
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          cd test
+          python3 run_suite.py --hw npu --suite stage-b-test-16-npu-a3 --timeout-per-file 3600
+
+  multimodal-gen-test-1-npu-a3:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.multimodal_gen == 'true'
+    runs-on: linux-aarch64-a3-2
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.3.rc2-a3-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh a3 diffusion
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          export PATH="/usr/local/Ascend/8.3.RC1/compiler/bishengir/bin:${PATH}"
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py --suite 1-npu
+
+  multimodal-gen-test-2-npu-a3:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.multimodal_gen == 'true'
+    runs-on: linux-aarch64-a3-16
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.3.rc2-a3-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh a3 diffusion
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          export PATH="/usr/local/Ascend/8.3.RC1/compiler/bishengir/bin:${PATH}"
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py --suite 2-npu
+
+  multimodal-gen-test-8-npu-a3:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.multimodal_gen == 'true'
+    runs-on: linux-aarch64-a3-8
+    container:
+      image: swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/cann:8.5.0-a3-ubuntu22.04-py3.11
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Mark repository safe
+        run: |
+          git config --system --add safe.directory ${GITHUB_WORKSPACE}
+
+      - name: Install dependencies
+        env:
+          TORCH_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/whl/cpu"
+          PYPI_CACHE_URL: "http://cache-service.nginx-pypi-cache.svc.cluster.local/pypi/simple"
+          GITHUB_PROXY_URL: "https://gh-proxy.test.osinfra.cn/"
+        run: |
+          # speed up by using infra cache services
+          CACHING_URL="cache-service.nginx-pypi-cache.svc.cluster.local"
+          sed -Ei "s@(ports|archive).ubuntu.com@${CACHING_URL}:8081@g" /etc/apt/sources.list
+          pip config set global.index-url http://${CACHING_URL}/pypi/simple
+          pip config set global.trusted-host "${CACHING_URL}"
+
+          bash scripts/ci/npu/npu_ci_install_dependency.sh a3 diffusion
+          # copy required file from our daily cache
+          cp ~/.cache/modelscope/hub/datasets/otavia/ShareGPT_Vicuna_unfiltered/ShareGPT_V3_unfiltered_cleaned_split.json /tmp
+          # copy gsm8k dataset
+          cp ~/.cache/modelscope/hub/datasets/tmp/test.jsonl /tmp
+
+      - name: Run test
+        timeout-minutes: 60
+        env:
+          SGLANG_USE_MODELSCOPE: true
+          SGLANG_IS_IN_CI: true
+          HF_ENDPOINT: https://hf-mirror.com
+          TORCH_EXTENSIONS_DIR: /tmp/torch_extensions
+          PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
+          STREAMS_PER_DEVICE: 32
+        run: |
+          cd python
+          python3 sglang/multimodal_gen/test/run_suite.py --suite 8-npu
+
+  pr-test-npu-finish:
+    needs:
+      [
+        check-changes,
+
+        stage-b-test-1-npu-a2,
+        stage-b-test-2-npu-a2,
+        stage-b-test-4-npu-a3,
+        stage-b-test-16-npu-a3,
+
+        multimodal-gen-test-1-npu-a3,
+        multimodal-gen-test-2-npu-a3,
+        multimodal-gen-test-8-npu-a3,
+      ]
+    if: always()
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check all dependent job statuses
+        run: |
+          # Convert the 'needs' context to a JSON string
+          json_needs='${{ toJson(needs) }}'
+
+          # Get a list of all job names from the JSON keys
+          job_names=$(echo "$json_needs" | jq -r 'keys_unsorted[]')
+
+          for job in $job_names; do
+            # For each job, extract its result
+            result=$(echo "$json_needs" | jq -r --arg j "$job" '.[$j].result')
+
+            # Print the job name and its result
+            echo "$job: $result"
+
+            # Check for failure or cancellation and exit if found
+            if [[ "$result" == "failure" || "$result" == "cancelled" ]]; then
+              echo "The above jobs failed."
+              exit 1
+            fi
+          done
+          # If the loop completes, all jobs were successful
+          echo "All jobs completed successfully"
+          exit 0

third_party/sglang/.github/workflows/pr-test-rust.yml

@@ -0,0 +1,359 @@
+name: PR Test (SMG)
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - "sgl-model-gateway/**"
+  pull_request:
+    branches: [ main ]
+    types: [opened, synchronize, reopened, labeled]
+    paths:
+      - "sgl-model-gateway/**"
+  workflow_dispatch:
+
+concurrency:
+  group: gateway-tests-${{ github.ref }}
+  cancel-in-progress: true
+
+env:
+  RUSTC_WRAPPER: sccache
+  SCCACHE_GHA_ENABLED: "true"
+  SGLANG_IS_IN_CI: true
+
+jobs:
+  build-wheel:
+    if: |
+      github.event_name != 'pull_request' ||
+      (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'run-ci')) ||
+      (github.event.action == 'labeled' && github.event.label.name == 'run-ci')
+    runs-on: 4-gpu-a10
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install rust dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_gateway_dependencies.sh
+
+      - name: Configure sccache
+        uses: mozilla-actions/sccache-action@v0.0.9
+        with:
+          version: "v0.12.0"
+          disable_annotations: true
+
+      - name: Rust cache
+        uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: sgl-model-gateway
+          shared-key: "rust-cache"
+          cache-all-crates: true
+          cache-on-failure: true
+          save-if: true
+
+      - name: Build python binding
+        run: |
+          source "$HOME/.cargo/env"
+          export RUSTC_WRAPPER=sccache
+          cd sgl-model-gateway/bindings/python
+          python3 -m pip install --upgrade pip maturin
+          maturin build --profile ci --features vendored-openssl --out dist
+
+      - name: List built wheel
+        run: ls -lh sgl-model-gateway/bindings/python/dist/
+
+      - name: Upload wheel artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: smg-wheel
+          path: sgl-model-gateway/bindings/python/dist/*.whl
+          retention-days: 1
+
+      - name: Test wheel install
+        run: |
+          pip install sgl-model-gateway/bindings/python/dist/*.whl
+          python3 -c "import sglang_router; print('Python package: OK')"
+          python3 -c "from sglang_router.sglang_router_rs import Router; print('Rust extension: OK')"
+          python3 -m sglang_router.launch_router --help > /dev/null && echo "Entry point: OK"
+
+  python-unit-tests:
+    needs: build-wheel
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          path: sglang-repo
+
+      - name: Move sgl-model-gateway folder to root
+        run: |
+          mv sglang-repo/sgl-model-gateway/* .
+          rm -rf sglang-repo
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+
+      - name: Download wheel artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: smg-wheel
+          path: dist/
+
+      - name: Install wheel
+        run: pip install dist/*.whl
+
+      - name: Run Python unit tests
+        run: |
+          cd bindings/python
+          python3 -m pip install pytest pytest-cov pytest-xdist
+          pytest -q tests --cov=sglang_router --cov-config=.coveragerc --cov-report=term-missing --cov-fail-under=80
+
+  unit-tests:
+    if: |
+      github.event_name != 'pull_request' ||
+      (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'run-ci')) ||
+      (github.event.action == 'labeled' && github.event.label.name == 'run-ci')
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install dependencies
+        run: |
+          bash scripts/ci/cuda/ci_install_gateway_dependencies.sh
+
+      - name: Configure sccache
+        uses: mozilla-actions/sccache-action@v0.0.9
+        with:
+          version: "v0.12.0"
+          disable_annotations: true
+
+      - name: Rust cache
+        uses: Swatinem/rust-cache@v2
+        with:
+          workspaces: sgl-model-gateway
+          shared-key: "rust-cache"
+          cache-all-crates: true
+          cache-on-failure: true
+          save-if: true
+
+      - name: Run lint
+        run: |
+          source "$HOME/.cargo/env"
+          cd sgl-model-gateway/
+          rustup component add clippy
+          cargo clippy --all-targets --all-features -- -D warnings
+
+      - name: Run fmt
+        run: |
+          source "$HOME/.cargo/env"
+          cd sgl-model-gateway/
+          rustup component add --toolchain nightly-x86_64-unknown-linux-gnu rustfmt
+          rustup toolchain install nightly --profile minimal
+          cargo +nightly fmt -- --check
+
+      - name: Generate vision golden fixtures
+        run: |
+          pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
+
+          pip install transformers pillow numpy scipy
+          pip install transformers pillow numpy
+          cd sgl-model-gateway/
+          python scripts/generate_vision_golden.py
+
+      - name: Run Rust tests
+        timeout-minutes: 20
+        run: |
+          source "$HOME/.cargo/env"
+          cd sgl-model-gateway/
+          cargo test
+
+      - name: Show sccache stats
+        if: always()
+        run: sccache --show-stats
+
+  gateway-e2e:
+    name: ${{ matrix.name }}
+    needs: build-wheel
+    if: |
+      github.event_name != 'pull_request' ||
+      (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'run-ci')) ||
+      (github.event.action == 'labeled' && github.event.label.name == 'run-ci')
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - name: benchmarks
+            timeout: 32
+            test_dirs: "e2e_test/benchmarks"
+            extra_deps: "genai-bench==0.0.3"
+            env_vars: ""
+            reruns: ""
+            upload_benchmarks: true
+            parallel_opts: ""  # No parallel for benchmarks (performance measurement)
+          - name: responses
+            timeout: 45
+            test_dirs: "e2e_test/responses"
+            extra_deps: ""
+            env_vars: "SHOW_WORKER_LOGS=0 SHOW_ROUTER_LOGS=1"
+            reruns: "--reruns 2 --reruns-delay 5"
+            setup_oracle: true
+            setup_brave: true
+            parallel_opts: ""  # Cloud backend tests not compatible with parallel execution
+          - name: e2e
+            timeout: 45
+            test_dirs: "e2e_test/router e2e_test/embeddings"
+            extra_deps: "pytest-parallel py"  # py is required for pytest-parallel with newer pytest
+            env_vars: "SHOW_WORKER_LOGS=0 SHOW_ROUTER_LOGS=1"
+            reruns: "--reruns 2 --reruns-delay 5"
+            parallel_opts: "--workers 1 --tests-per-worker 4"  # Thread-based parallelism
+          - name: chat-completions
+            timeout: 45
+            test_dirs: "e2e_test/chat_completions"
+            extra_deps: ""
+            env_vars: "SHOW_WORKER_LOGS=0 SHOW_ROUTER_LOGS=1"
+            reruns: "--reruns 2 --reruns-delay 5"
+            parallel_opts: ""
+    runs-on: 4-gpu-a10
+    timeout-minutes: ${{ matrix.timeout }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Install SGLang dependencies
+        run: |
+          sudo --preserve-env=PATH bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Setup Oracle Instant Client
+        if: matrix.setup_oracle
+        run: |
+          sudo apt-get install -y unzip
+          INSTANT_CLIENT_DIR="/home/ubuntu/instant-client"
+          INSTANT_CLIENT_ZIP="instantclient-basic-linux.x64-23.9.0.25.07.zip"
+
+          if [ ! -d "$INSTANT_CLIENT_DIR/instantclient_23_9" ]; then
+            echo "Downloading Oracle Instant Client..."
+            mkdir -p "$INSTANT_CLIENT_DIR"
+            cd "$INSTANT_CLIENT_DIR"
+            wget https://download.oracle.com/otn_software/linux/instantclient/2390000/$INSTANT_CLIENT_ZIP
+            unzip $INSTANT_CLIENT_ZIP
+            rm $INSTANT_CLIENT_ZIP
+          else
+            echo "Oracle Instant Client already exists, skipping download"
+          fi
+
+          echo "LD_LIBRARY_PATH=/home/ubuntu/instant-client/instantclient_23_9:\$LD_LIBRARY_PATH" >> $GITHUB_ENV
+
+      - name: Start Oracle Database
+        if: matrix.setup_oracle
+        run: |
+          docker run -d -p 1521:1521 -e ORACLE_PASSWORD=oracle --name oracle-db gvenzl/oracle-xe:21-slim
+          echo "Starting Oracle DB..."
+
+          # Export Oracle connection environment variables
+          echo "ATP_USER=system" >> $GITHUB_ENV
+          echo "ATP_PASSWORD=oracle" >> $GITHUB_ENV
+          echo "ATP_DSN=localhost:1521/XEPDB1" >> $GITHUB_ENV
+
+      - name: Start Brave MCP Server
+        if: matrix.setup_brave
+        run: |
+          docker run -d --rm \
+            -p 8001:8080 \
+            -e BRAVE_API_KEY \
+            --name brave-search-server \
+            shoofio/brave-search-mcp-sse:1.0.10
+          echo "Starting Brave MCP Server..."
+          sleep 2
+          curl -f --max-time 1 http://localhost:8001/sse > /dev/null 2>&1 && echo "Brave MCP Server is healthy!" || echo "Brave MCP Server responded"
+
+      - name: Download wheel artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: smg-wheel
+          path: wheel/
+
+      - name: Install wheel
+        run: |
+          pip uninstall -y sglang-router || true
+          pip install wheel/*.whl
+
+      - name: Install e2e test dependencies
+        run: |
+          python3 -m pip install pytest pytest-rerunfailures httpx openai grpcio grpcio-health-checking numpy
+          if [ -n "${{ matrix.extra_deps }}" ]; then
+            python3 -m pip --no-cache-dir install --upgrade ${{ matrix.extra_deps }}
+          fi
+
+      - name: Run E2E tests
+        run: |
+          python3 python/sglang/cli/killall.py
+          cd sgl-model-gateway
+          ${{ matrix.env_vars }} ROUTER_LOCAL_MODEL_PATH="/home/ubuntu/models" pytest ${{ matrix.reruns }} ${{ matrix.parallel_opts }} ${{ matrix.test_dirs }} -s -vv -o log_cli=true --log-cli-level=INFO
+
+      - name: Upload benchmark results
+        if: matrix.upload_benchmarks && success()
+        uses: actions/upload-artifact@v4
+        with:
+          name: genai-bench-results-all-policies
+          path: sgl-model-gateway/benchmark_**/
+
+      - name: Cleanup Brave MCP Server
+        if: always() && matrix.setup_brave
+        run: |
+          docker stop brave-search-server || true
+          docker rm brave-search-server || true
+
+      - name: Cleanup Oracle Database
+        if: always() && matrix.setup_oracle
+        run: |
+          docker stop oracle-db || true
+          docker rm oracle-db || true
+
+  docker-build-test:
+    if: |
+      github.event_name != 'pull_request' ||
+      (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'run-ci')) ||
+      (github.event.action == 'labeled' && github.event.label.name == 'run-ci')
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build Docker image (no push)
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: docker/gateway.Dockerfile
+          push: false
+          tags: sgl-model-gateway:test
+          cache-from: type=gha
+          cache-to: type=gha,mode=max
+
+  finish:
+    needs: [build-wheel, python-unit-tests, unit-tests, gateway-e2e, docker-build-test]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Finish
+        run: echo "This is an empty step to ensure that all jobs are completed."
+
+  summarize-benchmarks:
+    needs: gateway-e2e
+    runs-on: ubuntu-latest
+    if: success()
+
+    steps:
+    - name: Checkout code
+      uses: actions/checkout@v4
+
+    - name: Download benchmark results
+      uses: actions/download-artifact@v4
+      with:
+        name: genai-bench-results-all-policies
+
+    - name: Create benchmark summary
+      run: python3 sgl-model-gateway/e2e_test/benchmarks/summarize.py .

third_party/sglang/.github/workflows/pr-test-sgl-kernel.yml

@@ -0,0 +1,214 @@
+name: PR Test - SGL Kernel
+
+on:
+  workflow_call:
+    inputs:
+      sgl_kernel:
+        required: true
+        type: string
+      b200_runner:
+        required: true
+        type: string
+      pr_head_sha:
+        required: false
+        type: string
+        default: ''
+      git_ref:
+        required: false
+        type: string
+        default: ''
+      skip_stage_health_check:
+        required: false
+        type: boolean
+        default: false
+
+# Workflow-level env is NOT inherited from the caller in reusable workflows.
+# The github context (including github.event_name) IS inherited from the caller.
+env:
+  SGLANG_IS_IN_CI: true
+  SGLANG_CUDA_COREDUMP: "1"
+  SGLANG_PR_TEST_BYPASS_MAINTENANCE_ON_MAIN: ${{ github.ref == 'refs/heads/main' && 'true' || 'false' }}
+  SKIP_STAGE_HEALTH_CHECK: ${{ inputs.skip_stage_health_check == true && 'true' || 'false' }}
+
+jobs:
+  sgl-kernel-unit-test:
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Cleanup
+        run: |
+          ls -alh sgl-kernel/dist || true
+          rm -rf sgl-kernel/dist/* || true
+
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run test
+        timeout-minutes: 30
+        run: |
+          cd sgl-kernel
+          pytest tests/
+
+  sgl-kernel-mla-test:
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Cleanup
+        run: |
+          ls -alh sgl-kernel/dist || true
+          rm -rf sgl-kernel/dist/* || true
+
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run test
+        timeout-minutes: 30
+        run: |
+          cd test/registered/mla
+          python3 test_mla_deepseek_v3.py
+
+  sgl-kernel-benchmark-test:
+    runs-on: 1-gpu-h100
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Cleanup
+        run: |
+          ls -alh sgl-kernel/dist || true
+          rm -rf sgl-kernel/dist/* || true
+
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh
+
+      - name: Run benchmark tests
+        timeout-minutes: 45
+        run: |
+          cd sgl-kernel/benchmark
+          echo "Running sgl-kernel benchmark tests in CI mode..."
+
+          echo "CI environment variable: $CI"
+          echo "GITHUB_ACTIONS environment variable: $GITHUB_ACTIONS"
+
+          for bench_file in bench_*.py; do
+            echo "Testing $bench_file..."
+            timeout 60 python3 "$bench_file" || echo "Warning: $bench_file timed out or failed, continuing..."
+            echo "Completed $bench_file"
+            echo "---"
+          done
+
+          echo "All benchmark tests completed!"
+
+  sgl-kernel-b200-test:
+    runs-on: ${{ inputs.b200_runner }}
+    timeout-minutes: 240
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_head_sha || inputs.git_ref || github.sha }}
+
+      - uses: ./.github/actions/check-stage-health
+
+      - uses: ./.github/actions/check-maintenance
+
+      - name: Cleanup
+        run: |
+          ls -alh sgl-kernel/dist || true
+          rm -rf sgl-kernel/dist/* || true
+
+      - name: Download artifacts
+        uses: actions/download-artifact@v4
+        with:
+          path: sgl-kernel/dist/
+          merge-multiple: true
+          pattern: wheel-python3.10-cuda12.9
+
+      - name: Install dependencies
+        timeout-minutes: 20
+        run: |
+          CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh diffusion
+
+      - name: Run sgl-kernel unit tests on B200
+        timeout-minutes: 30
+        run: |
+          cd sgl-kernel
+          pytest tests/
+
+  # Adding a single CUDA13 smoke test to verify that the kernel builds and runs
+  # TODO: Add back this test when it can pass on CI
+  # cuda13-kernel-smoke-test:
+  #   if: inputs.sgl_kernel == 'true'
+  #   runs-on: x64-cu13-kernel-tests
+  #   steps:
+  #     - uses: actions/checkout@v4
+
+  #     - name: Cleanup
+  #       run: |
+  #         ls -alh sgl-kernel/dist || true
+  #         rm -rf sgl-kernel/dist/* || true
+
+  #     - name: Download CUDA 13.0 artifacts
+  #       uses: actions/download-artifact@v4
+  #       with:
+  #         path: sgl-kernel/dist/
+  #         merge-multiple: true
+  #         pattern: wheel-python3.10-cuda13.0
+
+  #     - name: Install dependencies
+  #       run: |
+  #         CUSTOM_BUILD_SGL_KERNEL=${{inputs.sgl_kernel}} bash scripts/ci/cuda/ci_install_dependency.sh
+
+  #     - name: Run kernel unit tests
+  #       timeout-minutes: 30
+  #       run: |
+  #         cd sgl-kernel
+  #         pytest tests/

third_party/sglang/.github/workflows/pr-test-xeon.yml

@@ -0,0 +1,131 @@
+name: PR Test (Xeon)
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
+        required: false
+        type: string
+        default: ''
+      run_all_tests:
+        description: "Run all tests (for releasing or testing purpose)"
+        required: false
+        type: boolean
+        default: false
+
+concurrency:
+  group: pr-test-xeon-${{ inputs.ref || github.ref }}
+  cancel-in-progress: false
+
+jobs:
+  # ==================== Check Changes ==================== #
+  check-changes:
+    runs-on: ubuntu-latest
+    outputs:
+      main_package: ${{ steps.filter.outputs.main_package || steps.run-mode.outputs.run_all_tests}}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Determine run mode
+        id: run-mode
+        run: |
+          # Run all tests for workflow_call (when ref input is provided)
+          # Note: github.event_name is inherited from caller, so we detect workflow_call by checking inputs.ref
+          if [[ "${{ inputs.run_all_tests }}" == "true" ]]; then
+            echo "run_all_tests=true" >> $GITHUB_OUTPUT
+            echo "Run mode: ALL TESTS (run_all_tests=${{ inputs.run_all_tests }})"
+          else
+            echo "run_all_tests=false" >> $GITHUB_OUTPUT
+            echo "Run mode: FILTERED (triggered by ${{ github.event_name }})"
+          fi
+
+      - name: Detect file changes
+        id: filter
+        uses: dorny/paths-filter@v3
+        if: steps.run-mode.outputs.run_all_tests != 'true'
+        with:
+          filters: |
+            main_package:
+              - "python/sglang/!(multimodal_gen)/**/!(*.md)"
+              - "python/pyproject_cpu.toml"
+              - "test/**/!(*.md)"
+              - "sgl-kernel/**/*.!(md|txt)"
+              - ".github/workflows/pr-test-xeon.yml"
+              - "docker/xeon.Dockerfile"
+
+  # ==================== PR Gate ==================== #
+  pr-gate:
+    needs: check-changes
+    if: needs.check-changes.outputs.main_package == 'true'
+    uses: ./.github/workflows/pr-gate.yml
+    secrets: inherit
+
+  build-test:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.main_package == 'true'
+    runs-on: xeon-gnr
+    env:
+      HF_HOME: /home/sdp/.cache/huggingface
+    strategy:
+      matrix:
+        build_type: ['all']
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Build and Push
+        run: |
+          version=$(cat python/sglang/version.py | cut -d'"' -f2)
+          tag=v${version}-xeon
+          PR_REPO=${{ github.event.pull_request.head.repo.clone_url }}
+          PR_HEAD_REF=${{ github.head_ref }}
+
+          docker build \
+            ${PR_REPO:+--build-arg SGLANG_REPO=$PR_REPO} \
+            ${PR_HEAD_REF:+--build-arg VER_SGLANG=$PR_HEAD_REF} \
+            . -f docker/xeon.Dockerfile  -t sglang_xeon --no-cache
+
+      - name: Run container
+        run: |
+          docker run -dt \
+            -v ${{ github.workspace }}:/sglang-checkout/ --ipc=host \
+            -v ${HF_HOME}:/root/.cache/huggingface \
+            --name ci_sglang_xeon \
+            sglang_xeon
+
+      - name: Check AMX support
+        id: check_amx
+        timeout-minutes: 5
+        run: |
+          docker exec -w /sglang-checkout/ ci_sglang_xeon \
+            bash -c "source /opt/.venv/bin/activate && python3 -c 'import torch; import sgl_kernel; assert torch._C._cpu._is_amx_tile_supported(); assert hasattr(torch.ops.sgl_kernel, \"convert_weight_packed\"); '"
+
+      - name: Run unit tests
+        timeout-minutes: 36
+        run: |
+          docker exec -w /sglang-checkout/ ci_sglang_xeon \
+            bash -c "source /opt/.venv/bin/activate && cd ./test/srt && python3 run_suite.py --suite per-commit-cpu --timeout-per-file 1500"
+
+      - name: Change permission
+        timeout-minutes: 2
+        run: |
+          docker exec -u root ci_sglang_xeon bash -c "
+            rm -rf /tmp/ci-home  &&
+            chown -R  $(id -u):$(id -g) /sglang-checkout/ 2>/dev/null || true
+          "
+
+      - name: Cleanup container
+        if: always()
+        run: |
+          docker rm -f ci_sglang_xeon || true

third_party/sglang/.github/workflows/pr-test-xpu.yml

@@ -0,0 +1,143 @@
+name: PR Test (XPU)
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+  workflow_dispatch:
+  workflow_call:
+    inputs:
+      ref:
+        description: 'Git ref (branch, tag, or SHA) to test. If not provided, uses the default branch.'
+        required: false
+        type: string
+        default: ''
+      run_all_tests:
+        description: "Run all tests (for releasing or testing purpose)"
+        required: false
+        type: boolean
+        default: false
+
+concurrency:
+  group: pr-test-xpu-${{ inputs.ref || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_call' }}
+
+jobs:
+  # ==================== Check Changes ==================== #
+  check-changes:
+    runs-on: ubuntu-latest
+    outputs:
+      main_package: ${{ steps.filter.outputs.main_package || steps.run-mode.outputs.run_all_tests }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Determine run mode
+        id: run-mode
+        run: |
+          # Run all tests for workflow_call (when ref input is provided)
+          # Note: github.event_name is inherited from caller, so we detect workflow_call by checking inputs.ref
+          if [[ "${{ inputs.run_all_tests }}" == "true" ]]; then
+            echo "run_all_tests=true" >> $GITHUB_OUTPUT
+            echo "Run mode: ALL TESTS (run_all_tests=${{ inputs.run_all_tests }})"
+          else
+            echo "run_all_tests=false" >> $GITHUB_OUTPUT
+            echo "Run mode: FILTERED (triggered by ${{ github.event_name }})"
+          fi
+      - name: Detect file changes
+        id: filter
+        uses: dorny/paths-filter@v3
+        if: steps.run-mode.outputs.run_all_tests != 'true'
+        with:
+          filters: |
+            main_package:
+              - "python/sglang/!(multimodal_gen)/**/!(*.md)"
+              - "python/pyproject_xpu.toml"
+              - "test/**/!(*.md)"
+              - "sgl-kernel/**/*.!(md|txt)"
+              - ".github/workflows/pr-test-xpu.yml"
+              - "docker/xpu.Dockerfile"
+
+  # ==================== PR Gate ==================== #
+  pr-gate:
+    needs: check-changes
+    if: needs.check-changes.outputs.main_package == 'true'
+    uses: ./.github/workflows/pr-gate.yml
+    secrets: inherit
+
+  build-and-test:
+    needs: [check-changes, pr-gate]
+    if: needs.check-changes.outputs.main_package == 'true'
+    runs-on: intel-bmg
+    env:
+      HF_HOME: /home/sdp/.cache/huggingface
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          ref: ${{ inputs.ref || github.ref }}
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build Docker image
+        run: |
+          PR_REPO=${{ github.event.pull_request.head.repo.clone_url }}
+          PR_HEAD_REF=${{ github.head_ref }}
+          docker build \
+            ${PR_REPO:+--build-arg SG_LANG_REPO=$PR_REPO} \
+            ${PR_HEAD_REF:+--build-arg SG_LANG_BRANCH=$PR_HEAD_REF} \
+            --no-cache --progress=plain -f docker/xpu.Dockerfile -t xpu_sglang_main:bmg .
+
+      - name: Run container
+        id: start_container
+        run: |
+          container_id=$(docker run -dt \
+            --group-add 992 \
+            --group-add $(getent group video | cut -d: -f3) \
+            -v ${HF_HOME}:/root/.cache/huggingface \
+            --device /dev/dri \
+            -e HF_TOKEN="$(cat ~/huggingface_token.txt)" \
+            xpu_sglang_main:bmg)
+          echo "Started container: $container_id"
+          echo "container_id=$container_id" >> "$GITHUB_OUTPUT"
+
+      - name: Install Dependency
+        timeout-minutes: 20
+        run: |
+          cid="${{ steps.start_container.outputs.container_id }}"
+          docker exec "$cid" /home/sdp/miniforge3/envs/py3.10/bin/python3 -m pip install --upgrade pip
+          docker exec "$cid" /home/sdp/miniforge3/envs/py3.10/bin/python3 -m pip install pytest expecttest ray huggingface_hub
+          docker exec "$cid" /home/sdp/miniforge3/envs/py3.10/bin/python3 -m pip uninstall -y flashinfer-python
+          docker exec "$cid" /bin/bash -c '/home/sdp/miniforge3/envs/py3.10/bin/hf auth login --token ${HF_TOKEN} '
+
+
+      - name: Run E2E Bfloat16 tests
+        timeout-minutes: 20
+        run: |
+          cid="${{ steps.start_container.outputs.container_id }}"
+          docker exec "$cid" bash -c "source /home/sdp/miniforge3/bin/activate && conda activate py3.10 && cd /home/sdp/sglang/test/srt && python3 run_suite.py --suite per-commit-xpu"
+      - name: Cleanup container
+        if: always()
+        run: |
+          cid="${{ steps.start_container.outputs.container_id }}"
+          docker rm -f "$cid" || true
+
+  finish:
+    if: always()
+    needs: [build-and-test, pr-gate]
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check job status
+        run: |
+          result="${{ needs.build-and-test.result }}"
+          if [ "$result" != "success" ] && [ "$result" != "skipped" ]; then
+            echo "Job failed with result: $result"
+            exit 1
+          fi
+          echo "All jobs completed successfully (result: $result)"
+          exit 0

third_party/sglang/.github/workflows/release-branch-cut.yml

@@ -0,0 +1,215 @@
+name: Release Branch Cut
+
+on:
+  workflow_dispatch:
+    inputs:
+      branch_name:
+        description: 'Branch name to create (e.g., release/v0.5.7)'
+        required: true
+        type: string
+      commit_sha:
+        description: 'Commit SHA from main to cut the release branch from (defaults to latest main)'
+        required: false
+        type: string
+        default: ''
+
+permissions:
+  actions: write
+  contents: write
+  issues: read
+  pull-requests: read
+
+jobs:
+  cut-release-branch:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-latest
+    environment: 'prod'
+    outputs:
+      branch_name: ${{ steps.set_output.outputs.branch_name }}
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          ref: main
+          fetch-depth: 0
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Validate branch name
+        run: |
+          BRANCH_NAME="${{ github.event.inputs.branch_name }}"
+
+          if [ -z "$BRANCH_NAME" ]; then
+            echo "::error::Branch name is required"
+            exit 1
+          fi
+
+          # Validate branch name format (should start with release/)
+          if [[ ! "$BRANCH_NAME" =~ ^release/ ]]; then
+            echo "::warning::Branch name '$BRANCH_NAME' does not follow convention 'release/vX.Y.Z'"
+          fi
+
+          echo "Branch name: $BRANCH_NAME"
+
+      - name: Validate commit SHA
+        id: validate
+        run: |
+          COMMIT_SHA="${{ github.event.inputs.commit_sha }}"
+
+          # If no commit SHA provided, use latest main
+          if [ -z "$COMMIT_SHA" ]; then
+            COMMIT_SHA=$(git rev-parse HEAD)
+            echo "No commit SHA provided, using latest main: $COMMIT_SHA"
+          fi
+
+          # Verify the commit exists and is on main
+          if ! git cat-file -t "$COMMIT_SHA" > /dev/null 2>&1; then
+            echo "::error::Commit SHA '$COMMIT_SHA' does not exist"
+            exit 1
+          fi
+
+          # Check if commit is an ancestor of main (i.e., is on main branch)
+          if ! git merge-base --is-ancestor "$COMMIT_SHA" main; then
+            echo "::error::Commit SHA '$COMMIT_SHA' is not on the main branch"
+            exit 1
+          fi
+
+          echo "COMMIT_SHA=$COMMIT_SHA" >> $GITHUB_OUTPUT
+          echo "Validated commit SHA: $COMMIT_SHA"
+
+      - name: Check if branch already exists
+        run: |
+          BRANCH_NAME="${{ github.event.inputs.branch_name }}"
+
+          if git ls-remote --heads origin "$BRANCH_NAME" | grep -q "$BRANCH_NAME"; then
+            echo "::error::Branch '$BRANCH_NAME' already exists"
+            exit 1
+          fi
+
+          echo "Branch '$BRANCH_NAME' does not exist, proceeding with creation"
+
+      - name: Create release branch
+        id: set_output
+        run: |
+          COMMIT_SHA="${{ steps.validate.outputs.COMMIT_SHA }}"
+          BRANCH_NAME="${{ github.event.inputs.branch_name }}"
+
+          git config user.name "sglang-bot"
+          git config user.email "sglang-bot@users.noreply.github.com"
+
+          # Create branch from the specified commit
+          git checkout -b "$BRANCH_NAME" "$COMMIT_SHA"
+
+          echo "branch_name=$BRANCH_NAME" >> $GITHUB_OUTPUT
+          echo "Successfully created branch '$BRANCH_NAME' from commit '$COMMIT_SHA'"
+
+      - name: Update version references in documentation
+        run: |
+          BRANCH_NAME="${{ github.event.inputs.branch_name }}"
+          # Extract version from branch name (e.g., release/v0.5.8 -> v0.5.8)
+          VERSION=$(echo "$BRANCH_NAME" | sed 's/release\///')
+
+          # Update git clone version references in docs
+          sed -i "s/git clone -b v[0-9]\+\.[0-9]\+\.[0-9]\+\.\?post\?[0-9]*/git clone -b $VERSION/" docs/get_started/install.md
+          sed -i "s/git clone -b v[0-9]\+\.[0-9]\+\.[0-9]\+\.\?post\?[0-9]*/git clone -b $VERSION/" docs/platforms/amd_gpu.md
+
+          # Check if any changes were made
+          if git diff --quiet; then
+            echo "No version references needed updating"
+          else
+            git add docs/get_started/install.md docs/platforms/amd_gpu.md
+            git commit -m "docs: update version references to $VERSION"
+            echo "Updated version references to $VERSION"
+          fi
+
+      - name: Push release branch
+        run: |
+          BRANCH_NAME="${{ steps.set_output.outputs.branch_name }}"
+          git push origin "$BRANCH_NAME"
+          echo "Successfully pushed branch '$BRANCH_NAME'"
+
+      - name: Summary
+        run: |
+          COMMIT_SHA="${{ steps.validate.outputs.COMMIT_SHA }}"
+          BRANCH_NAME="${{ github.event.inputs.branch_name }}"
+
+          echo "## Release Branch Cut Summary" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "| Property | Value |" >> $GITHUB_STEP_SUMMARY
+          echo "|----------|-------|" >> $GITHUB_STEP_SUMMARY
+          echo "| Branch | \`$BRANCH_NAME\` |" >> $GITHUB_STEP_SUMMARY
+          echo "| Commit | \`$COMMIT_SHA\` |" >> $GITHUB_STEP_SUMMARY
+          echo "| Triggered by | @${{ github.actor }} |" >> $GITHUB_STEP_SUMMARY
+          echo "" >> $GITHUB_STEP_SUMMARY
+          echo "### Next Steps" >> $GITHUB_STEP_SUMMARY
+          echo "1. Tests are automatically triggered on the release branch" >> $GITHUB_STEP_SUMMARY
+          echo "2. Apply any hotfixes if needed" >> $GITHUB_STEP_SUMMARY
+          echo "3. Create a tag to trigger release: \`gh workflow run release-tag.yml -f version=X.Y.Z -f ref=$BRANCH_NAME\`" >> $GITHUB_STEP_SUMMARY
+
+  run-pr-tests-nvidia:
+    needs: cut-release-branch
+    uses: ./.github/workflows/pr-test.yml
+    with:
+      git_ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+      run_all_tests: true
+      skip_stage_health_check: true
+    secrets: inherit
+
+  run-pr-tests-amd:
+    needs: cut-release-branch
+    uses: ./.github/workflows/pr-test-amd.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+      run_all_tests: true
+    secrets: inherit
+
+  run-pr-test-npu:
+    needs: cut-release-branch
+    uses: ./.github/workflows/pr-test-npu.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+      run_all_tests: true
+    secrets: inherit
+
+  run-pr-tests-xeon:
+    needs: cut-release-branch
+    uses: ./.github/workflows/pr-test-xeon.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+      run_all_tests: true
+    secrets: inherit
+
+  run-pr-tests-xpu:
+    needs: cut-release-branch
+    uses: ./.github/workflows/pr-test-xpu.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+      run_all_tests: true
+    secrets: inherit
+
+  run-nightly-tests-nvidia:
+    needs: cut-release-branch
+    uses: ./.github/workflows/nightly-test-nvidia.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+    secrets: inherit
+
+  run-nightly-tests-amd:
+    needs: cut-release-branch
+    uses: ./.github/workflows/nightly-test-amd.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+    secrets: inherit
+
+  run-nightly-tests-npu:
+    needs: cut-release-branch
+    uses: ./.github/workflows/nightly-test-npu.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+    secrets: inherit
+
+  run-nightly-tests-intel:
+    needs: cut-release-branch
+    uses: ./.github/workflows/nightly-test-intel.yml
+    with:
+      ref: ${{ needs.cut-release-branch.outputs.branch_name }}
+    secrets: inherit

third_party/sglang/.github/workflows/release-docker-amd-nightly.yml

@@ -0,0 +1,182 @@
+name: Release Docker Images Nightly (AMD)
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 12 * * *'
+
+concurrency:
+  # A PR number if a pull request and otherwise the commit hash. This cancels
+  # queued and in-progress runs for the same PR (presubmit) or commit
+  # (postsubmit). The workflow name is prepended to avoid conflicts between
+  # different workflows.
+  group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
+  cancel-in-progress: true
+
+jobs:
+  publish:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: amd-docker-scale
+    environment: 'prod'
+    strategy:
+      fail-fast: false
+      matrix:
+        gpu_arch: ['gfx942', 'gfx950']
+        build_type: ['all']
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Required for git describe to find tags
+
+      - name: "Set Date"
+        run: |
+          echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV
+
+      - name: Get version from latest tag
+        id: version
+        run: |
+          # Get the latest version tag sorted by version number (e.g., v0.5.7 -> 0.5.7)
+          VERSION=$(git tag -l 'v[0-9]*' --sort=-v:refname | head -1 | sed 's/^v//')
+
+          if [ -z "$VERSION" ]; then
+            echo "::error::Could not determine version from git tags"
+            exit 1
+          fi
+
+          # Get short commit hash of current HEAD
+          COMMIT_HASH=$(git rev-parse --short HEAD)
+
+          # Compose pretend version for setuptools_scm: e.g., 0.5.8.dev20260129+g1a2b3c4
+          PRETEND_VERSION="${VERSION}.dev${{ env.DATE }}+g${COMMIT_HASH}"
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+          echo "pretend_version=${PRETEND_VERSION}" >> $GITHUB_OUTPUT
+          echo "Detected version: ${VERSION}"
+          echo "Pretend version for pip: ${PRETEND_VERSION}"
+
+      - name: Login to Docker Hub (AMD)
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_AMD_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_AMD_TOKEN }}
+
+      - name: Build and Push to rocm/sgl-dev
+        run: |
+          version=${{ steps.version.outputs.version }}
+          pretend_version=${{ steps.version.outputs.pretend_version }}
+          echo "Version: ${version}"
+          echo "Pretend version: ${pretend_version}"
+
+          if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then
+            rocm_tag="rocm700-mi30x"
+          elif [ "${{ matrix.gpu_arch }}" = "gfx950" ]; then
+            rocm_tag="rocm700-mi35x"
+          else
+            echo "Unsupported gfx arch"
+            exit 1
+          fi
+
+          tag=v${version}-${rocm_tag}
+          echo "IMAGE_TAG=${tag}-${{ env.DATE }}" >> $GITHUB_ENV
+
+          docker build . -f docker/rocm.Dockerfile --build-arg SGL_BRANCH=${{ github.ref_name }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }} --build-arg ENABLE_MORI=1 --build-arg NIC_BACKEND=ainic --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=${pretend_version} -t rocm/sgl-dev:${tag}-${{ env.DATE }} --no-cache
+          docker push rocm/sgl-dev:${tag}-${{ env.DATE }}
+
+      - name: Login to Docker Hub (lmsys)
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Push to lmsysorg/sglang-rocm
+        run: |
+          docker tag rocm/sgl-dev:${{ env.IMAGE_TAG }} lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
+          docker push lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
+
+  # Temporarily disable docker cache seeding until performant storage is in place
+  cache:
+    if: false
+    # if: always() && github.repository == 'sgl-project/sglang'
+    runs-on: linux-mi300-gpu-1
+    environment: 'prod'
+    needs: publish
+    strategy:
+      fail-fast: false
+      matrix:
+        gpu_arch: ['gfx942']
+        build_type: ['all']
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Required for git describe to find tags
+
+      - name: "Set Date"
+        run: |
+          echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV
+
+      - name: Get version from latest tag
+        id: version
+        run: |
+          # Get the latest version tag sorted by version number (e.g., v0.5.7 -> 0.5.7)
+          VERSION=$(git tag -l 'v[0-9]*' --sort=-v:refname | head -1 | sed 's/^v//')
+
+          if [ -z "$VERSION" ]; then
+            echo "::error::Could not determine version from git tags"
+            exit 1
+          fi
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+          echo "Detected version: ${VERSION}"
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_AMD_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_AMD_TOKEN }}
+
+      - name: Pull and Save Docker Image to Cache
+        run: |
+          set -euxo pipefail
+
+          version=${{ steps.version.outputs.version }}
+          echo "Version: ${version}"
+
+          if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then
+            rocm_tag="rocm700-mi30x"
+          else
+            echo "Unsupported gfx arch"
+            exit 1
+          fi
+
+          tag=v${version}-${rocm_tag}
+
+          if [ "${{ matrix.build_type }}" = "all" ]; then
+            tag_suffix=""
+          else
+            echo "Unsupported build type"
+            exit 1
+          fi
+
+          image="rocm/sgl-dev:${tag}-${{ env.DATE }}${tag_suffix}"
+
+          # Determine target cache file name based on ROCm variant
+          if [[ "${rocm_tag}" == rocm700* ]]; then
+            final_path="/home/runner/sgl-data/docker/image-700.tar"
+          else
+            echo "Unexpected ROCm tag: ${rocm_tag}"
+            exit 1
+          fi
+
+          tmp_path="${final_path}.tmp"
+
+          echo "Pulling image: ${image}"
+          docker pull "${image}"
+
+          echo "Saving to temp file: ${tmp_path}"
+          docker save "${image}" -o "${tmp_path}"
+
+          echo "Moving to final path: ${final_path}"
+          mv -f "${tmp_path}" "${final_path}"
+
+          echo "Cache populated successfully at ${final_path}"

third_party/sglang/.github/workflows/release-docker-amd-rocm720-nightly.yml

@@ -0,0 +1,94 @@
+name: Release Docker Images ROCm 7.2.0 Nightly Preview (AMD)
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: '0 12 * * *'
+
+concurrency:
+  # A PR number if a pull request and otherwise the commit hash. This cancels
+  # queued and in-progress runs for the same PR (presubmit) or commit
+  # (postsubmit). The workflow name is prepended to avoid conflicts between
+  # different workflows.
+  group: ${{ github.workflow }}-${{ github.event.number || github.sha }}
+  cancel-in-progress: True
+
+jobs:
+  publish:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: amd-docker-scale
+    environment: 'prod'
+    strategy:
+      fail-fast: false
+      matrix:
+        gpu_arch: ['gfx942-rocm720', 'gfx950-rocm720']
+        build_type: ['all']
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0  # Required for git describe to find tags
+
+      - name: "Set Date"
+        run: |
+          echo "DATE=$(date +%Y%m%d)" >> $GITHUB_ENV
+
+      - name: Get version from latest tag
+        id: version
+        run: |
+          # Get the latest version tag sorted by version number (e.g., v0.5.7 -> 0.5.7)
+          VERSION=$(git tag -l 'v[0-9]*' --sort=-v:refname | head -1 | sed 's/^v//')
+
+          if [ -z "$VERSION" ]; then
+            echo "::error::Could not determine version from git tags"
+            exit 1
+          fi
+
+          # Get short commit hash of current HEAD
+          COMMIT_HASH=$(git rev-parse --short HEAD)
+
+          # Compose pretend version for setuptools_scm: e.g., 0.5.8.post1.dev20260211+g1a2b3c4
+          PRETEND_VERSION="${VERSION}.dev${{ env.DATE }}+g${COMMIT_HASH}"
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+          echo "pretend_version=${PRETEND_VERSION}" >> $GITHUB_OUTPUT
+          echo "Detected version: ${VERSION}"
+          echo "Pretend version for pip: ${PRETEND_VERSION}"
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_AMD_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_AMD_TOKEN }}
+
+      - name: Build and Push to rocm/sgl-dev
+        run: |
+          version=${{ steps.version.outputs.version }}
+          pretend_version=${{ steps.version.outputs.pretend_version }}
+          echo "Version: ${version}"
+          echo "Pretend version: ${pretend_version}"
+
+          if [ "${{ matrix.gpu_arch }}" = "gfx942-rocm720" ]; then
+            rocm_tag="rocm720-mi30x"
+          elif [ "${{ matrix.gpu_arch }}" = "gfx950-rocm720" ]; then
+            rocm_tag="rocm720-mi35x"
+          else
+            echo "Unsupported gfx arch"
+            exit 1
+          fi
+
+          tag=v${version}-${rocm_tag}
+          echo "IMAGE_TAG=${tag}-${{ env.DATE }}" >> $GITHUB_ENV
+
+          docker build . -f docker/rocm.Dockerfile --build-arg SGL_BRANCH=${{ github.ref_name }} --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }} --build-arg ENABLE_MORI=1 --build-arg NIC_BACKEND=ainic --build-arg SETUPTOOLS_SCM_PRETEND_VERSION=${pretend_version} -t rocm/sgl-dev:${tag}-${{ env.DATE }} --no-cache
+          docker push rocm/sgl-dev:${tag}-${{ env.DATE }}
+
+      - name: Login to Docker Hub (lmsys)
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Push to lmsysorg/sglang-rocm
+        run: |
+          docker tag rocm/sgl-dev:${{ env.IMAGE_TAG }} lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}
+          docker push lmsysorg/sglang-rocm:${{ env.IMAGE_TAG }}

third_party/sglang/.github/workflows/release-docker-amd.yml

@@ -0,0 +1,88 @@
+name: Release Docker Images (AMD)
+on:
+  push:
+    tags:
+      - 'v[0-9]+.*'
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version to build (without v prefix, e.g., 0.5.7)'
+        required: true
+
+jobs:
+  publish:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: amd-docker-scale
+    environment: 'prod'
+    strategy:
+      matrix:
+        rocm_version: ['rocm700', 'rocm720']
+        gpu_arch: ['gfx942', 'gfx950']
+        build_type: ['all']
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Get version from tag
+        id: version
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            VERSION="${{ github.event.inputs.version }}"
+          else
+            # Extract version from tag (e.g., v0.5.7 -> 0.5.7)
+            VERSION="${GITHUB_REF_NAME#v}"
+          fi
+
+          # Validate version format
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+      - name: Build and Push
+        run: |
+          version=${{ steps.version.outputs.version }}
+          echo "Version: ${version}"
+
+          gpu_arch_suffix=""
+          if [ "${{ matrix.rocm_version }}" = "rocm700" ]; then
+            if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then
+              rocm_tag="rocm700-mi30x"
+            elif [ "${{ matrix.gpu_arch }}" = "gfx950" ]; then
+              rocm_tag="rocm700-mi35x"
+            else
+              echo "Unsupported gfx arch"
+              exit 1
+            fi
+          elif [ "${{ matrix.rocm_version }}" = "rocm720" ]; then
+            gpu_arch_suffix="-${{ matrix.rocm_version }}"
+            if [ "${{ matrix.gpu_arch }}" = "gfx942" ]; then
+              rocm_tag="rocm720-mi30x"
+            elif [ "${{ matrix.gpu_arch }}" = "gfx950" ]; then
+              rocm_tag="rocm720-mi35x"
+            else
+              echo "Unsupported gfx arch"
+              exit 1
+            fi
+          else
+            echo "Unsupported rocm version"
+            exit 1
+          fi
+
+          tag=v${version}-${rocm_tag}
+
+          # rocm.Dockerfile expects SGL_BRANCH with 'v' prefix for git tag checkout
+          docker build . -f docker/rocm.Dockerfile --build-arg BUILD_TYPE=${{ matrix.build_type }} --build-arg GPU_ARCH=${{ matrix.gpu_arch }}${gpu_arch_suffix} --build-arg SGL_BRANCH=v${version} --build-arg ENABLE_MORI=1 --build-arg NIC_BACKEND=ainic -t lmsysorg/sglang:${tag} --no-cache
+          docker push lmsysorg/sglang:${tag}

third_party/sglang/.github/workflows/release-docker-cu13-framework.yml

@@ -0,0 +1,190 @@
+name: Release CUDA 13 Framework Docker Images (Temporary)
+
+# Temporary workflow to build only versioned cu13 framework images
+# Can be deleted after use
+
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Version to build (without v prefix, e.g., 0.5.8)"
+        required: true
+jobs:
+  publish-x86:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: x64-docker-build-node
+    steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Free disk space
+        uses: jlumbroso/free-disk-space@main
+        with:
+          tool-cache: false
+          docker-images: false
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          swap-storage: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Validate version
+        id: version
+        run: |
+          VERSION="${{ github.event.inputs.version }}"
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+      - name: Build and Push AMD64 Framework (CUDA 13)
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          docker buildx build \
+            --target framework \
+            --platform linux/amd64 \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=13.0.1 \
+            --build-arg BUILD_TYPE=all \
+            --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
+            --build-arg GRACE_BLACKWELL=0 \
+            --build-arg SGL_VERSION=${version} \
+            --metadata-file /tmp/metadata.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest-cu130-amd64-framework.txt
+
+      - name: Upload digest
+        uses: actions/upload-artifact@v4
+        with:
+          name: digest-cu130-amd64
+          path: /tmp/digest-cu130-amd64-framework.txt
+          retention-days: 1
+
+  publish-arm64:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: arm-docker-build-node
+    steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Validate version
+        id: version
+        run: |
+          VERSION="${{ github.event.inputs.version }}"
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+      - name: Build and Push ARM64 Framework (CUDA 13)
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          docker buildx build \
+            --target framework \
+            --platform linux/arm64 \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=13.0.1 \
+            --build-arg BUILD_TYPE=all \
+            --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
+            --build-arg GRACE_BLACKWELL=1 \
+            --build-arg SGL_VERSION=${version} \
+            --metadata-file /tmp/metadata.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest-cu130-arm64-framework.txt
+
+      - name: Upload digest
+        uses: actions/upload-artifact@v4
+        with:
+          name: digest-cu130-arm64
+          path: /tmp/digest-cu130-arm64-framework.txt
+          retention-days: 1
+
+  create-manifest:
+    runs-on: ubuntu-22.04
+    needs: [publish-x86, publish-arm64]
+    if: github.repository == 'sgl-project/sglang'
+    steps:
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Download amd64 digest
+        uses: actions/download-artifact@v4
+        with:
+          name: digest-cu130-amd64
+          path: /tmp/digests/amd64
+
+      - name: Download arm64 digest
+        uses: actions/download-artifact@v4
+        with:
+          name: digest-cu130-arm64
+          path: /tmp/digests/arm64
+
+      - name: Create multi-arch manifest
+        run: |
+          version=${{ github.event.inputs.version }}
+          AMD64_DIGEST=$(cat /tmp/digests/amd64/digest-cu130-amd64-framework.txt)
+          ARM64_DIGEST=$(cat /tmp/digests/arm64/digest-cu130-arm64-framework.txt)
+
+          # Create versioned CUDA 13 framework manifest
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:v${version}-cu130 \
+            lmsysorg/sglang@${AMD64_DIGEST} \
+            lmsysorg/sglang@${ARM64_DIGEST}
+
+          # Create latest CUDA 13 framework manifest
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:latest-cu130 \
+            lmsysorg/sglang@${AMD64_DIGEST} \
+            lmsysorg/sglang@${ARM64_DIGEST}

third_party/sglang/.github/workflows/release-docker-dev.yml

@@ -0,0 +1,209 @@
+name: Build and Push Development Docker Images
+
+on:
+  workflow_dispatch:
+    inputs:
+      pr_number:
+        description: "PR number to build from (leave empty to use current branch)"
+        required: false
+        default: ""
+      tag:
+        description: "Custom tag suffix (overrides pr_number in tag). E.g. 'my-test' → dev-my-test, dev-cu13-my-test, etc."
+        required: false
+        default: ""
+  schedule:
+    - cron: "0 0 * * *"
+
+concurrency:
+  group: release-docker-dev-${{ inputs.tag || inputs.pr_number || 'nightly' }}
+  cancel-in-progress: true
+
+jobs:
+  build-dev:
+    if: ${{ github.repository == 'sgl-project/sglang' }}
+    runs-on: ${{ matrix.runner }}
+    strategy:
+      matrix:
+        include:
+          - runner: x64-docker-build-node
+            platform: linux/amd64
+            build_type: all
+            grace_blackwell: 0
+            arch_tag: x86
+            version: 12.9.1
+          - runner: arm-docker-build-node
+            platform: linux/arm64
+            build_type: all
+            grace_blackwell: 1
+            arch_tag: arm64
+            version: 12.9.1
+          - runner: x64-docker-build-node
+            platform: linux/amd64
+            build_type: all
+            grace_blackwell: 0
+            arch_tag: x86-cu13
+            version: 13.0.1
+          - runner: arm-docker-build-node
+            platform: linux/arm64
+            build_type: all
+            grace_blackwell: 1
+            arch_tag: arm64-cu13
+            version: 13.0.1
+    steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ inputs.pr_number && format('refs/pull/{0}/head', inputs.pr_number) || github.ref }}
+
+      - name: Free disk space
+        uses: jlumbroso/free-disk-space@main
+        with:
+          tool-cache: true
+          docker-images: true
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          swap-storage: true
+
+      - name: Prune Docker to reclaim disk space
+        run: |
+          docker buildx prune --filter "until=72h" -f
+          docker system prune -af --filter "until=72h"
+          docker volume prune -af
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and Push Dev Image
+        run: |
+          # Nightly (schedule) installs latest release; manual dispatch builds from checked-out source
+          if [ "${{ github.event_name }}" = "schedule" ]; then
+            SOURCE_ARG="--build-arg USE_LATEST_SGLANG=1"
+          else
+            SOURCE_ARG="--build-arg BRANCH_TYPE=local"
+          fi
+
+          docker buildx build \
+            --platform ${{ matrix.platform }} \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            --target framework \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=${{ matrix.version }} \
+            --build-arg BUILD_TYPE=${{ matrix.build_type }} \
+            --build-arg CMAKE_BUILD_PARALLEL_LEVEL=$(nproc) \
+            --build-arg GRACE_BLACKWELL=${{ matrix.grace_blackwell }} \
+            ${SOURCE_ARG} \
+            --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
+            --metadata-file /tmp/metadata.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest.txt
+
+      - name: Upload digest
+        uses: actions/upload-artifact@v4
+        with:
+          name: digest-${{ matrix.arch_tag }}
+          path: /tmp/digest.txt
+          retention-days: 1
+
+  create-manifests:
+    runs-on: ubuntu-22.04
+    needs: [build-dev]
+    if: ${{ github.repository == 'sgl-project/sglang' }}
+    strategy:
+      matrix:
+        variant:
+          - base: dev
+            x86: x86
+            arm64: arm64
+          - base: dev-cu13
+            x86: x86-cu13
+            arm64: arm64-cu13
+    steps:
+      - uses: docker/setup-buildx-action@v3
+
+      - uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Download x86 digest
+        uses: actions/download-artifact@v4
+        with:
+          name: digest-${{ matrix.variant.x86 }}
+          path: /tmp/digests/x86
+
+      - name: Download arm64 digest
+        uses: actions/download-artifact@v4
+        with:
+          name: digest-${{ matrix.variant.arm64 }}
+          path: /tmp/digests/arm64
+
+      - name: Create multi-arch manifest
+        run: |
+          X86_DIGEST=$(cat /tmp/digests/x86/digest.txt)
+          ARM64_DIGEST=$(cat /tmp/digests/arm64/digest.txt)
+
+          SUFFIX=""
+          if [ -n "${{ inputs.tag }}" ]; then
+            SUFFIX="-${{ inputs.tag }}"
+          elif [ -n "${{ inputs.pr_number }}" ]; then
+            SUFFIX="-pr-${{ inputs.pr_number }}"
+          fi
+
+          TAG="${{ matrix.variant.base }}${SUFFIX}"
+
+          # For nightly (no suffix), also stamp a dated tag
+          EXTRA_TAG=""
+          if [ -z "${SUFFIX}" ]; then
+            SHORT_SHA="${{ github.sha }}"
+            EXTRA_TAG="-t lmsysorg/sglang:nightly-${TAG}-$(date +%Y%m%d)-${SHORT_SHA:0:8}"
+          fi
+
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:${TAG} \
+            ${EXTRA_TAG} \
+            lmsysorg/sglang@${X86_DIGEST} \
+            lmsysorg/sglang@${ARM64_DIGEST}
+
+          echo "✓ Published lmsysorg/sglang:${TAG}"
+
+      - name: Cleanup Old Nightly Builds
+        if: ${{ !inputs.tag && !inputs.pr_number }}
+        run: |
+          TOKEN=$(curl -s -H "Content-Type: application/json" \
+            -X POST -d '{"username": "${{ secrets.DOCKERHUB_USERNAME }}", "password": "${{ secrets.DOCKERHUB_TOKEN }}"}' \
+            https://hub.docker.com/v2/users/login/ | jq -r .token)
+
+          TAGS_RESPONSE=$(curl -s -H "Authorization: JWT $TOKEN" \
+            "https://hub.docker.com/v2/repositories/lmsysorg/sglang/tags/?page_size=100")
+
+          TAGS=$(echo "$TAGS_RESPONSE" | jq -r \
+            '.results[] | select(.name | test("^nightly-${{ matrix.variant.base }}-[0-9]")) | "\(.last_updated)|\(.name)"' \
+            | sort -r | cut -d'|' -f2)
+
+          TAG_COUNT=$(echo "$TAGS" | wc -l)
+          if [ "$TAG_COUNT" -gt 14 ]; then
+            echo "Found $TAG_COUNT nightly builds, keeping only the 14 most recent"
+            TAGS_TO_DELETE=$(echo "$TAGS" | tail -n +15)
+            for tag in $TAGS_TO_DELETE; do
+              echo "Deleting tag: $tag"
+              curl -X DELETE -H "Authorization: JWT $TOKEN" \
+                "https://hub.docker.com/v2/repositories/lmsysorg/sglang/tags/$tag/"
+            done
+          else
+            echo "Only $TAG_COUNT nightly builds found, no cleanup needed"
+          fi

third_party/sglang/.github/workflows/release-docker-gateway.yml

@@ -0,0 +1,39 @@
+name: Release SGLang Model Gateway Docker Image
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - sgl-model-gateway/bindings/python/pyproject.toml
+  workflow_dispatch:
+
+jobs:
+  publish:
+    if: github.repository == 'sgl-project/sglang'
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and Push
+        run: |
+          version=$(cat sgl-model-gateway/bindings/python/src/sglang_router/version.py | cut -d'"' -f2)
+          tag=v${version}
+
+          docker buildx build . -f docker/gateway.Dockerfile \
+            --platform linux/amd64,linux/arm64 \
+            -t lmsysorg/sgl-model-gateway:${tag} \
+            -t lmsysorg/sgl-model-gateway:latest \
+            --push

third_party/sglang/.github/workflows/release-docker-npu-nightly.yml

@@ -0,0 +1,85 @@
+name: Release Docker Images Nightly (NPU)
+on:
+  pull_request:
+    branches:
+      - 'main'
+    paths:
+      - '.github/workflows/release-docker-npu-nightly.yml'
+      - 'docker/npu.Dockerfile'
+  workflow_dispatch:
+  schedule:
+    - cron: "0 16 * * *" # Execute at 0:00 a.m. Beijing Time every day
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.sha }}
+  cancel-in-progress: true
+
+jobs:
+  build:
+    runs-on: ubuntu-22.04-arm
+    strategy:
+      matrix:
+        cann_version: ["8.5.0"]
+        device_type: ["910b", "a3"]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Free up disk space
+        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
+        with:
+          tool-cache: true
+          docker-images: false
+
+      - name: Setup Docker buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            lmsysorg/sglang
+          # push with schedule event
+          # push with workflow_dispatch event
+          tags: |
+            type=ref,event=pr
+            type=ref,event=branch
+            type=schedule,pattern=main
+          flavor: |
+            latest=false
+            suffix=-cann${{ matrix.cann_version }}-${{ matrix.device_type }},onlatest=true
+      # Login against a Docker registry except on PR
+      # https://github.com/docker/login-action
+      - name: Log into docker hub
+        uses: docker/login-action@v3
+        if: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      # Enable Docker multi-architecture build environment
+      # Emulate non-native architectures
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+      # Required for building and pushing multi-arch Docker images
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      # Build and push Docker image with Buildx (don't push on PR)
+      # https://github.com/docker/build-push-action
+      - name: Build and push Docker image
+        id: build-and-push
+        uses: docker/build-push-action@v6
+        with:
+          context: docker
+          file: docker/npu.Dockerfile
+          platforms: linux/arm64,linux/amd64
+          labels: ${{ steps.meta.outputs.labels }}
+          tags: ${{ steps.meta.outputs.tags }}
+          push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
+          provenance: false
+          build-args: |
+            SGLANG_KERNEL_NPU_TAG=2026.03.10.rc1
+            CANN_VERSION=${{ matrix.cann_version }}
+            DEVICE_TYPE=${{ matrix.device_type }}

third_party/sglang/.github/workflows/release-docker-npu.yml

@@ -0,0 +1,93 @@
+name: Release Docker Images (NPU)
+on:
+  push:
+    tags:
+      - 'v[0-9]+.*'
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version to build (without v prefix, e.g., 0.5.7)'
+        required: true
+
+jobs:
+  build:
+    runs-on: ubuntu-22.04-arm
+    strategy:
+      matrix:
+        cann_version: ["8.5.0"]
+        device_type: ["910b", "a3"]
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Free up disk space
+        uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
+        with:
+          tool-cache: true
+          docker-images: false
+
+        # push with tag
+      - name: Docker meta
+        id: meta
+        uses: docker/metadata-action@v5
+        with:
+          images: |
+            lmsysorg/sglang
+          tags: |
+            type=ref,event=pr
+          flavor: |
+            latest=false
+
+      # Login against a Docker registry except on PR
+      # https://github.com/docker/login-action
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        if: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Get version from tag
+        id: version
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            VERSION="${{ github.event.inputs.version }}"
+          else
+            # Extract version from tag (e.g., v0.5.7 -> 0.5.7)
+            VERSION="${GITHUB_REF_NAME#v}"
+          fi
+          # Validate version format
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+          echo "version=v${VERSION}" >> $GITHUB_OUTPUT
+          echo "TAG=lmsysorg/sglang:v${VERSION}-cann${{ matrix.cann_version }}-${{ matrix.device_type }}" >> $GITHUB_OUTPUT
+      # Enable Docker multi-architecture build environment
+      # Emulate non-native architectures
+      - name: Set up QEMU
+        uses: docker/setup-qemu-action@v3
+      # Required for building and pushing multi-arch Docker images
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build and push Docker image
+        id: build-and-push
+        uses: docker/build-push-action@v6
+        with:
+          context: docker
+          file: docker/npu.Dockerfile
+          platforms: linux/arm64,linux/amd64
+          labels: ${{ steps.meta.outputs.labels }}
+          tags: ${{ steps.meta.outputs.tags || steps.version.outputs.TAG }}
+          push: ${{ github.repository == 'sgl-project/sglang' && github.event_name != 'pull_request' }}
+          provenance: false
+          build-args: |
+            SGLANG_KERNEL_NPU_TAG=2026.03.10.rc1
+            CANN_VERSION=${{ matrix.cann_version }}
+            DEVICE_TYPE=${{ matrix.device_type }}
+            SGLANG_TAG=${{ steps.version.outputs.version }}

third_party/sglang/.github/workflows/release-docker-runtime.yml

@@ -0,0 +1,309 @@
+name: Release Docker Runtime Images
+#
+# This workflow builds and publishes runtime Docker images (production-optimized, ~50% smaller):
+#   - lmsysorg/sglang:v{version}-runtime, lmsysorg/sglang:latest-runtime
+#   - lmsysorg/sglang:v{version}-cu130-runtime, lmsysorg/sglang:latest-cu130-runtime
+#
+on:
+  push:
+    tags:
+      - "v[0-9]+.*"
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Version to build (without v prefix, e.g., 0.5.7)"
+        required: true
+
+jobs:
+  publish-x86:
+    if: github.repository == 'sgl-project/sglang'
+    environment: "prod"
+    strategy:
+      matrix:
+        variant:
+          - cuda_version: "12.9.1"
+            build_type: "all"
+            grace_blackwell: 0
+    runs-on: x64-docker-build-node
+    steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Free disk space
+        uses: jlumbroso/free-disk-space@main
+        with:
+          tool-cache: false
+          docker-images: false
+          android: true
+          dotnet: true
+          haskell: true
+          large-packages: true
+          swap-storage: false
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Get version from tag
+        id: version
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            VERSION="${{ github.event.inputs.version }}"
+          else
+            # Extract version from tag (e.g., v0.5.7 -> 0.5.7)
+            VERSION="${GITHUB_REF_NAME#v}"
+          fi
+
+          # Validate version format
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+      - name: Build and Push AMD64 Runtime
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          docker buildx build \
+            --target runtime \
+            --platform linux/amd64 \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=${{ matrix.variant.cuda_version }} \
+            --build-arg BUILD_TYPE=${{ matrix.variant.build_type }} \
+            --build-arg GRACE_BLACKWELL=${{ matrix.variant.grace_blackwell }} \
+            --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
+            --build-arg SGL_VERSION=${version} \
+            --metadata-file /tmp/metadata-cu129-runtime.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata-cu129-runtime.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest-cu129-amd64-runtime.txt
+
+      - name: Build and Push AMD64 Runtime (CUDA 13)
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          docker buildx build \
+            --target runtime \
+            --platform linux/amd64 \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=13.0.1 \
+            --build-arg BUILD_TYPE=${{ matrix.variant.build_type }} \
+            --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
+            --build-arg GRACE_BLACKWELL=0 \
+            --build-arg SGL_VERSION=${version} \
+            --metadata-file /tmp/metadata-cu130-runtime.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata-cu130-runtime.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest-cu130-amd64-runtime.txt
+
+      - name: Upload digests
+        uses: actions/upload-artifact@v4
+        with:
+          name: digests-amd64
+          path: /tmp/digest-*.txt
+          retention-days: 1
+
+  publish-arm64:
+    if: github.repository == 'sgl-project/sglang'
+    environment: "prod"
+    strategy:
+      matrix:
+        variant:
+          - cuda_version: "12.9.1"
+            build_type: "all"
+            grace_blackwell: 1
+    runs-on: arm-docker-build-node
+    steps:
+      - name: Delete huge unnecessary tools folder
+        run: rm -rf /opt/hostedtoolcache
+
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Get version from tag
+        id: version
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            VERSION="${{ github.event.inputs.version }}"
+          else
+            # Extract version from tag (e.g., v0.5.7 -> 0.5.7)
+            VERSION="${GITHUB_REF_NAME#v}"
+          fi
+
+          # Validate version format
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+      - name: Build and Push ARM64 Runtime
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          docker buildx build \
+            --target runtime \
+            --platform linux/arm64 \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=${{ matrix.variant.cuda_version }} \
+            --build-arg BUILD_TYPE=${{ matrix.variant.build_type }} \
+            --build-arg GRACE_BLACKWELL=${{ matrix.variant.grace_blackwell }} \
+            --build-arg INSTALL_FLASHINFER_JIT_CACHE=1 \
+            --build-arg SGL_VERSION=${version} \
+            --metadata-file /tmp/metadata-cu129-runtime.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata-cu129-runtime.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest-cu129-arm64-runtime.txt
+
+      - name: Build and Push ARM64 Runtime (CUDA 13)
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          docker buildx build \
+            --target runtime \
+            --platform linux/arm64 \
+            --output type=image,name=lmsysorg/sglang,push-by-digest=true,name-canonical=true,push=true \
+            -f docker/Dockerfile \
+            --build-arg CUDA_VERSION=13.0.1 \
+            --build-arg BUILD_TYPE=${{ matrix.variant.build_type }} \
+            --build-arg GRACE_BLACKWELL=1 \
+            --build-arg SGL_VERSION=${version} \
+            --metadata-file /tmp/metadata-cu130-runtime.json \
+            --no-cache \
+            .
+
+          DIGEST=$(python3 -c "import json; print(json.load(open('/tmp/metadata-cu130-runtime.json'))['containerimage.digest'])")
+          echo "Pushed digest: ${DIGEST}"
+          echo "${DIGEST}" > /tmp/digest-cu130-arm64-runtime.txt
+
+      - name: Upload digests
+        uses: actions/upload-artifact@v4
+        with:
+          name: digests-arm64
+          path: /tmp/digest-*.txt
+          retention-days: 1
+
+  create-manifests:
+    runs-on: ubuntu-22.04
+    needs: [publish-x86, publish-arm64]
+    if: github.repository == 'sgl-project/sglang'
+    environment: "prod"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Login to Docker Hub
+        uses: docker/login-action@v2
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Get version from tag
+        id: version
+        run: |
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            VERSION="${{ github.event.inputs.version }}"
+          else
+            # Extract version from tag (e.g., v0.5.7 -> 0.5.7)
+            VERSION="${GITHUB_REF_NAME#v}"
+          fi
+
+          # Validate version format
+          if [ -z "$VERSION" ]; then
+            echo "::error::Version is empty"
+            exit 1
+          fi
+          if ! echo "$VERSION" | grep -qE '^[0-9]+\.[0-9]+\.[0-9]+'; then
+            echo "::error::Invalid version format: $VERSION (expected: X.Y.Z)"
+            exit 1
+          fi
+
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+
+      - name: Download amd64 digests
+        uses: actions/download-artifact@v4
+        with:
+          name: digests-amd64
+          path: /tmp/digests/amd64
+
+      - name: Download arm64 digests
+        uses: actions/download-artifact@v4
+        with:
+          name: digests-arm64
+          path: /tmp/digests/arm64
+
+      - name: Create multi-arch manifests
+        run: |
+          version=${{ steps.version.outputs.version }}
+
+          CU129_AMD64_RT=$(cat /tmp/digests/amd64/digest-cu129-amd64-runtime.txt)
+          CU130_AMD64_RT=$(cat /tmp/digests/amd64/digest-cu130-amd64-runtime.txt)
+          CU129_ARM64_RT=$(cat /tmp/digests/arm64/digest-cu129-arm64-runtime.txt)
+          CU130_ARM64_RT=$(cat /tmp/digests/arm64/digest-cu130-arm64-runtime.txt)
+
+          # Create versioned runtime manifest
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:v${version}-runtime \
+            lmsysorg/sglang@${CU129_AMD64_RT} \
+            lmsysorg/sglang@${CU129_ARM64_RT}
+
+          # Create latest runtime manifest
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:latest-runtime \
+            lmsysorg/sglang@${CU129_AMD64_RT} \
+            lmsysorg/sglang@${CU129_ARM64_RT}
+
+          # Create versioned CUDA 13 runtime manifest
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:v${version}-cu130-runtime \
+            lmsysorg/sglang@${CU130_AMD64_RT} \
+            lmsysorg/sglang@${CU130_ARM64_RT}
+
+          # Create latest CUDA 13 runtime manifest
+          docker buildx imagetools create \
+            -t lmsysorg/sglang:latest-cu130-runtime \
+            lmsysorg/sglang@${CU130_AMD64_RT} \
+            lmsysorg/sglang@${CU130_ARM64_RT}