chore: vendor sglang v0.5.10 snapshot

third_party/sglang/docs/diffusion/api/cli.md

@@ -0,0 +1,230 @@
+# SGLang Diffusion CLI
+
+Use the CLI for one-off generation with `sglang generate` or to start a persistent HTTP server with `sglang serve`.
+
+### Overlay repos for non-diffusers models
+
+If `--model-path` points to a supported non-diffusers source repo, SGLang can resolve it
+through a self-hosted overlay repo.
+
+SGLang first checks a built-in overlay registry. Concrete built-in mappings can be added over time without changing the CLI surface.
+
+Override example:
+
+```bash
+export SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY='{
+  "Wan-AI/Wan2.2-S2V-14B": {
+    "overlay_repo_id": "your-org/Wan2.2-S2V-14B-overlay",
+    "overlay_revision": "main"
+  }
+}'
+
+sglang generate \
+  --model-path Wan-AI/Wan2.2-S2V-14B \
+  --config configs/wan_s2v.yaml
+```
+
+The overlay repo should be a complete diffusers-style/componentized repo
+
+You can also pass the overlay repo itself as `--model-path` if it contains `_overlay/overlay_manifest.json`.
+
+Notes:
+1. `SGLANG_DIFFUSION_MODEL_OVERLAY_REGISTRY` is only an optional override for
+development and debugging. It accepts either a JSON object or a path to a JSON
+file, and can extend or replace built-in entries for the current process.
+2. On the first load, SGLang will:
+   - download overlay metadata from the overlay repo
+   - download the required files from the original source repo
+   - materialize a local standard component repo under `~/.cache/sgl_diffusion/materialized_models/`
+3. Later loads reuse the materialized local repo. The materialized repo is what the runtime loads as a normal componentized model directory.
+
+
+## Quick Start
+
+### Generate
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --prompt "A beautiful sunset over the mountains" \
+  --save-output
+```
+
+### Serve
+
+```bash
+sglang serve \
+  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --num-gpus 4 \
+  --ulysses-degree 2 \
+  --ring-degree 2 \
+  --port 30010
+```
+
+For request and response examples, see [OpenAI-Compatible API](openai_api.md).
+
+```{tip}
+Use `sglang generate --help` and `sglang serve --help` for the full argument list. The CLI help output is the source of truth for exhaustive flags.
+```
+
+## Common Options
+
+### Model and runtime
+
+- `--model-path {MODEL}`: model path or Hugging Face model ID
+- `--lora-path {PATH}` and `--lora-nickname {NAME}`: load a LoRA adapter
+- `--num-gpus {N}`: number of GPUs to use
+- `--tp-size {N}`: tensor parallelism size, mainly for encoders
+- `--sp-degree {N}`: sequence parallelism size
+- `--ulysses-degree {N}` and `--ring-degree {N}`: USP parallelism controls
+- `--attention-backend {BACKEND}`: attention backend for native SGLang pipelines
+- `--attention-backend-config {CONFIG}`: attention backend configuration
+
+### Sampling and output
+
+- `--prompt {PROMPT}` and `--negative-prompt {PROMPT}`
+- `--num-inference-steps {STEPS}` and `--seed {SEED}`
+- `--height {HEIGHT}`, `--width {WIDTH}`, `--num-frames {N}`, `--fps {FPS}`
+- `--output-path {PATH}`, `--output-file-name {NAME}`, `--save-output`, `--return-frames`
+
+For frame interpolation and upscaling, see [Post-Processing](post_processing.md).
+
+### Quantized transformers
+
+For quantized transformer checkpoints, prefer:
+
+- `--model-path` for the base pipeline
+- `--transformer-path` for a quantized `transformers` transformer component folder
+- `--transformer-weights-path` for a quantized safetensors file, directory, or repo
+
+See [Quantization](../quantization.md) for supported quantization families and examples.
+
+## Configuration Files
+
+Use `--config` to load JSON or YAML configuration. Command-line flags override values from the config file.
+
+```bash
+sglang generate --config config.yaml
+```
+
+Example:
+
+```yaml
+model_path: FastVideo/FastHunyuan-diffusers
+prompt: A beautiful woman in a red dress walking down a street
+output_path: outputs/
+num_gpus: 2
+sp_size: 2
+tp_size: 1
+num_frames: 45
+height: 720
+width: 1280
+num_inference_steps: 6
+seed: 1024
+fps: 24
+precision: bf16
+vae_precision: fp16
+vae_tiling: true
+vae_sp: true
+enable_torch_compile: false
+```
+
+## Generate
+
+`sglang generate` runs a single generation job and exits when the job finishes.
+
+```bash
+sglang generate \
+  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
+  --text-encoder-cpu-offload \
+  --pin-cpu-memory \
+  --num-gpus 4 \
+  --ulysses-degree 2 \
+  --ring-degree 2 \
+  --prompt "A curious raccoon" \
+  --save-output \
+  --output-path outputs \
+  --output-file-name "a-curious-raccoon.mp4"
+```
+
+```{note}
+HTTP server-only arguments are ignored by `sglang generate`.
+```
+
+For diffusers pipelines, Cache-DiT can be enabled with `SGLANG_CACHE_DIT_ENABLED=true` or `--cache-dit-config`. See [Cache-DiT](../performance/cache/cache_dit.md).
+
+## Serve
+
+`sglang serve` starts the HTTP server and keeps the model loaded for repeated requests.
+
+```bash
+sglang serve \
+  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --text-encoder-cpu-offload \
+  --pin-cpu-memory \
+  --num-gpus 4 \
+  --ulysses-degree 2 \
+  --ring-degree 2 \
+  --port 30010
+```
+
+### Cloud Storage
+
+SGLang Diffusion can upload generated images and videos to S3-compatible object storage after generation.
+
+```bash
+export SGLANG_CLOUD_STORAGE_TYPE=s3
+export SGLANG_S3_BUCKET_NAME=my-bucket
+export SGLANG_S3_ACCESS_KEY_ID=your-access-key
+export SGLANG_S3_SECRET_ACCESS_KEY=your-secret-key
+export SGLANG_S3_ENDPOINT_URL=https://minio.example.com
+```
+
+See [Environment Variables](../environment_variables.md) for the full set of storage options.
+
+## Component Path Overrides
+
+Override individual pipeline components such as `vae`, `transformer`, or `text_encoder` with `--<component>-path`.
+
+```bash
+sglang serve \
+  --model-path black-forest-labs/FLUX.2-dev \
+  --vae-path fal/FLUX.2-Tiny-AutoEncoder
+```
+
+The component key must match the key in the model's `model_index.json`, and the path must be either a Hugging Face repo ID or a complete component directory.
+
+## Diffusers Backend
+
+Use `--backend diffusers` to force vanilla diffusers pipelines when no native SGLang implementation exists or when a model requires a custom pipeline class.
+
+### Key Options
+
+| Argument | Values | Description |
+|----------|--------|-------------|
+| `--backend` | `auto`, `sglang`, `diffusers` | Choose native SGLang, force native, or force diffusers |
+| `--diffusers-attention-backend` | `flash`, `_flash_3_hub`, `sage`, `xformers`, `native` | Attention backend for diffusers pipelines |
+| `--trust-remote-code` | flag | Required for models with custom pipeline classes |
+| `--vae-tiling` and `--vae-slicing` | flag | Lower memory usage for VAE decode |
+| `--dit-precision` and `--vae-precision` | `fp16`, `bf16`, `fp32` | Precision controls |
+| `--enable-torch-compile` | flag | Enable `torch.compile` |
+| `--cache-dit-config` | `{PATH}` | Cache-DiT config for diffusers pipelines |
+
+### Example
+
+```bash
+sglang generate \
+  --model-path AIDC-AI/Ovis-Image-7B \
+  --backend diffusers \
+  --trust-remote-code \
+  --diffusers-attention-backend flash \
+  --prompt "A serene Japanese garden with cherry blossoms" \
+  --height 1024 \
+  --width 1024 \
+  --num-inference-steps 30 \
+  --save-output \
+  --output-path outputs \
+  --output-file-name ovis_garden.png
+```
+
+For pipeline-specific arguments not exposed in the CLI, pass `diffusers_kwargs` in a config file.

third_party/sglang/docs/diffusion/api/openai_api.md

@@ -0,0 +1,420 @@
+# SGLang Diffusion OpenAI API
+
+The SGLang diffusion HTTP server implements an OpenAI-compatible API for image and video generation, as well as LoRA adapter management.
+
+## Prerequisites
+
+- Python 3.11+ if you plan to use the OpenAI Python SDK.
+
+## Serve
+
+Launch the server using the `sglang serve` command.
+
+### Start the server
+
+```bash
+SERVER_ARGS=(
+  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers
+  --text-encoder-cpu-offload
+  --pin-cpu-memory
+  --num-gpus 4
+  --ulysses-degree=2
+  --ring-degree=2
+  --port 30010
+)
+
+sglang serve "${SERVER_ARGS[@]}"
+```
+
+- **--model-path**: Path to the model or model ID.
+- **--port**: HTTP port to listen on (default: `30000`).
+
+**Get Model Information**
+
+**Endpoint:** `GET /models`
+
+Returns information about the model served by this server, including model path, task type, pipeline configuration, and precision settings.
+
+**Curl Example:**
+
+```bash
+curl -sS -X GET "http://localhost:30010/models"
+```
+
+**Response Example:**
+
+```json
+{
+  "model_path": "Wan-AI/Wan2.1-T2V-1.3B-Diffusers",
+  "task_type": "T2V",
+  "pipeline_name": "wan_pipeline",
+  "pipeline_class": "WanPipeline",
+  "num_gpus": 4,
+  "dit_precision": "bf16",
+  "vae_precision": "fp16"
+}
+```
+
+---
+
+## Endpoints
+
+### Image Generation
+
+The server implements an OpenAI-compatible Images API under the `/v1/images` namespace.
+
+**Create an image**
+
+**Endpoint:** `POST /v1/images/generations`
+
+**Python Example (b64_json response):**
+
+```python
+import base64
+from openai import OpenAI
+
+client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
+
+img = client.images.generate(
+    prompt="A calico cat playing a piano on stage",
+    size="1024x1024",
+    n=1,
+    response_format="b64_json",
+)
+
+image_bytes = base64.b64decode(img.data[0].b64_json)
+with open("output.png", "wb") as f:
+    f.write(image_bytes)
+```
+
+**Curl Example:**
+
+```bash
+curl -sS -X POST "http://localhost:30010/v1/images/generations" \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer sk-proj-1234567890" \
+  -d '{
+        "prompt": "A calico cat playing a piano on stage",
+        "size": "1024x1024",
+        "n": 1,
+        "response_format": "b64_json"
+      }'
+```
+
+> **Note**
+> If `response_format=url` is used and cloud storage is not configured, the API returns
+> a relative URL like `/v1/images/<IMAGE_ID>/content`.
+
+**Edit an image**
+
+**Endpoint:** `POST /v1/images/edits`
+
+This endpoint accepts a multipart form upload with input images and a text prompt. The server can return either a base64-encoded image or a URL to download the image.
+
+**Curl Example (b64_json response):**
+
+```bash
+curl -sS -X POST "http://localhost:30010/v1/images/edits" \
+  -H "Authorization: Bearer sk-proj-1234567890" \
+  -F "image=@local_input_image.png" \
+  -F "url=image_url.jpg" \
+  -F "prompt=A calico cat playing a piano on stage" \
+  -F "size=1024x1024" \
+  -F "response_format=b64_json"
+```
+
+**Curl Example (URL response):**
+
+```bash
+curl -sS -X POST "http://localhost:30010/v1/images/edits" \
+  -H "Authorization: Bearer sk-proj-1234567890" \
+  -F "image=@local_input_image.png" \
+  -F "url=image_url.jpg" \
+  -F "prompt=A calico cat playing a piano on stage" \
+  -F "size=1024x1024" \
+  -F "response_format=url"
+```
+
+**Download image content**
+
+When `response_format=url` is used with `POST /v1/images/generations` or `POST /v1/images/edits`,
+the API returns a relative URL like `/v1/images/<IMAGE_ID>/content`.
+
+**Endpoint:** `GET /v1/images/{image_id}/content`
+
+**Curl Example:**
+
+```bash
+curl -sS -L "http://localhost:30010/v1/images/<IMAGE_ID>/content" \
+  -H "Authorization: Bearer sk-proj-1234567890" \
+  -o output.png
+```
+
+### Video Generation
+
+The server implements a subset of the OpenAI Videos API under the `/v1/videos` namespace.
+
+**Create a video**
+
+**Endpoint:** `POST /v1/videos`
+
+**Python Example:**
+
+```python
+from openai import OpenAI
+
+client = OpenAI(api_key="sk-proj-1234567890", base_url="http://localhost:30010/v1")
+
+video = client.videos.create(
+    prompt="A calico cat playing a piano on stage",
+    size="1280x720"
+)
+print(f"Video ID: {video.id}, Status: {video.status}")
+```
+
+**Curl Example:**
+
+```bash
+curl -sS -X POST "http://localhost:30010/v1/videos" \
+  -H "Content-Type: application/json" \
+  -H "Authorization: Bearer sk-proj-1234567890" \
+  -d '{
+        "prompt": "A calico cat playing a piano on stage",
+        "size": "1280x720"
+      }'
+```
+
+**List videos**
+
+**Endpoint:** `GET /v1/videos`
+
+**Python Example:**
+
+```python
+videos = client.videos.list()
+for item in videos.data:
+    print(item.id, item.status)
+```
+
+**Curl Example:**
+
+```bash
+curl -sS -X GET "http://localhost:30010/v1/videos" \
+  -H "Authorization: Bearer sk-proj-1234567890"
+```
+
+**Download video content**
+
+**Endpoint:** `GET /v1/videos/{video_id}/content`
+
+**Python Example:**
+
+```python
+import time
+
+# Poll for completion
+while True:
+    page = client.videos.list()
+    item = next((v for v in page.data if v.id == video_id), None)
+    if item and item.status == "completed":
+        break
+    time.sleep(5)
+
+# Download content
+resp = client.videos.download_content(video_id=video_id)
+with open("output.mp4", "wb") as f:
+    f.write(resp.read())
+```
+
+**Curl Example:**
+
+```bash
+curl -sS -L "http://localhost:30010/v1/videos/<VIDEO_ID>/content" \
+  -H "Authorization: Bearer sk-proj-1234567890" \
+  -o output.mp4
+```
+
+---
+
+### LoRA Management
+
+The server supports dynamic loading, merging, and unmerging of LoRA adapters.
+
+**Important Notes:**
+- Mutual Exclusion: Only one LoRA can be *merged* (active) at a time
+- Switching: To switch LoRAs, you must first `unmerge` the current one, then `set` the new one
+- Caching: The server caches loaded LoRA weights in memory. Switching back to a previously loaded LoRA (same path) has little cost
+
+**Set LoRA Adapter**
+
+Loads one or more LoRA adapters and merges their weights into the model. Supports both single LoRA (backward compatible) and multiple LoRA adapters.
+
+**Endpoint:** `POST /v1/set_lora`
+
+**Parameters:**
+- `lora_nickname` (string or list of strings, required): A unique identifier for the LoRA adapter(s). Can be a single string or a list of strings for multiple LoRAs
+- `lora_path` (string or list of strings/None, optional): Path to the `.safetensors` file(s) or Hugging Face repo ID(s). Required for the first load; optional if re-activating a cached nickname. If a list, must match the length of `lora_nickname`
+- `target` (string or list of strings, optional): Which transformer(s) to apply the LoRA to. If a list, must match the length of `lora_nickname`. Valid values:
+  - `"all"` (default): Apply to all transformers
+  - `"transformer"`: Apply only to the primary transformer (high noise for Wan2.2)
+  - `"transformer_2"`: Apply only to transformer_2 (low noise for Wan2.2)
+  - `"critic"`: Apply only to the critic model
+- `strength` (float or list of floats, optional): LoRA strength for merge, default 1.0. If a list, must match the length of `lora_nickname`. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
+
+**Single LoRA Example:**
+
+```bash
+curl -X POST http://localhost:30010/v1/set_lora \
+  -H "Content-Type: application/json" \
+  -d '{
+        "lora_nickname": "lora_name",
+        "lora_path": "/path/to/lora.safetensors",
+        "target": "all",
+        "strength": 0.8
+      }'
+```
+
+**Multiple LoRA Example:**
+
+```bash
+curl -X POST http://localhost:30010/v1/set_lora \
+  -H "Content-Type: application/json" \
+  -d '{
+        "lora_nickname": ["lora_1", "lora_2"],
+        "lora_path": ["/path/to/lora1.safetensors", "/path/to/lora2.safetensors"],
+        "target": ["transformer", "transformer_2"],
+        "strength": [0.8, 1.0]
+      }'
+```
+
+**Multiple LoRA with Same Target:**
+
+```bash
+curl -X POST http://localhost:30010/v1/set_lora \
+  -H "Content-Type: application/json" \
+  -d '{
+        "lora_nickname": ["style_lora", "character_lora"],
+        "lora_path": ["/path/to/style.safetensors", "/path/to/character.safetensors"],
+        "target": "all",
+        "strength": [0.7, 0.9]
+      }'
+```
+
+> [!NOTE]
+> When using multiple LoRAs:
+> - All list parameters (`lora_nickname`, `lora_path`, `target`, `strength`) must have the same length
+> - If `target` or `strength` is a single value, it will be applied to all LoRAs
+> - Multiple LoRAs applied to the same target will be merged in order
+
+
+**Merge LoRA Weights**
+
+Manually merges the currently set LoRA weights into the base model.
+
+> [!NOTE]
+> `set_lora` automatically performs a merge, so this is typically only needed if you have manually unmerged but want to re-apply the same LoRA without calling `set_lora` again.*
+
+**Endpoint:** `POST /v1/merge_lora_weights`
+
+**Parameters:**
+- `target` (string, optional): Which transformer(s) to merge. One of "all" (default), "transformer", "transformer_2", "critic"
+- `strength` (float, optional): LoRA strength for merge, default 1.0. Values < 1.0 reduce the effect, values > 1.0 amplify the effect
+
+**Curl Example:**
+
+```bash
+curl -X POST http://localhost:30010/v1/merge_lora_weights \
+  -H "Content-Type: application/json" \
+  -d '{"strength": 0.8}'
+```
+
+
+**Unmerge LoRA Weights**
+
+Unmerges the currently active LoRA weights from the base model, restoring it to its original state. This **must** be called before setting a different LoRA.
+
+**Endpoint:** `POST /v1/unmerge_lora_weights`
+
+**Curl Example:**
+
+```bash
+curl -X POST http://localhost:30010/v1/unmerge_lora_weights \
+  -H "Content-Type: application/json"
+```
+
+**List LoRA Adapters**
+
+Returns loaded LoRA adapters and current application status per module.
+
+**Endpoint:** `GET /v1/list_loras`
+
+**Curl Example:**
+
+```bash
+curl -sS -X GET "http://localhost:30010/v1/list_loras"
+```
+
+**Response Example:**
+
+```json
+{
+  "loaded_adapters": [
+    { "nickname": "lora_a", "path": "/weights/lora_a.safetensors" },
+    { "nickname": "lora_b", "path": "/weights/lora_b.safetensors" }
+  ],
+  "active": {
+    "transformer": [
+      {
+        "nickname": "lora2",
+        "path": "tarn59/pixel_art_style_lora_z_image_turbo",
+        "merged": true,
+        "strength": 1.0
+      }
+    ]
+  }
+}
+```
+
+Notes:
+- If LoRA is not enabled for the current pipeline, the server will return an error.
+- `num_lora_layers_with_weights` counts only layers that have LoRA weights applied for the active adapter.
+
+### Example: Switching LoRAs
+
+1.  Set LoRA A:
+    ```bash
+    curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_a", "lora_path": "path/to/A"}'
+    ```
+2.  Generate with LoRA A...
+3.  Unmerge LoRA A:
+    ```bash
+    curl -X POST http://localhost:30010/v1/unmerge_lora_weights
+    ```
+4.  Set LoRA B:
+    ```bash
+    curl -X POST http://localhost:30010/v1/set_lora -d '{"lora_nickname": "lora_b", "lora_path": "path/to/B"}'
+    ```
+5.  Generate with LoRA B...
+
+### Adjust Output Quality
+
+The server supports adjusting output quality and compression levels for both image and video generation through the `output-quality` and `output-compression` parameters.
+
+#### Parameters
+
+- **`output-quality`** (string, optional): Preset quality level that automatically sets compression. **Default is `"default"`**. Valid values:
+  - `"maximum"`: Highest quality (100)
+  - `"high"`: High quality (90)
+  - `"medium"`: Medium quality (55)
+  - `"low"`: Lower quality (35)
+  - `"default"`: Auto-adjust based on media type (50 for video, 75 for image)
+
+- **`output-compression`** (integer, optional): Direct compression level override (0-100). **Default is `None`**. When provided (not `None`), takes precedence over `output-quality`.
+  - `0`: Lowest quality, smallest file size
+  - `100`: Highest quality, largest file size
+
+#### Notes
+
+- **Precedence**: When both `output-quality` and `output-compression` are provided, `output-compression` takes precedence
+- **Format Support**: Quality settings apply to JPEG, and video formats. PNG uses lossless compression and ignores these settings
+- **File Size vs Quality**: Lower compression values (or "low" quality preset) produce smaller files but may show visible artifacts

third_party/sglang/docs/diffusion/api/post_processing.md

@@ -0,0 +1,148 @@
+# Post-Processing
+
+SGLang diffusion supports optional post-processing steps that run after
+generation to improve temporal smoothness (frame interpolation) or spatial
+resolution (upscaling). These steps are independent of the diffusion model and
+can be combined in a single run.
+
+When both are enabled, **frame interpolation runs first** (increasing the frame
+count), then **upscaling runs on every frame** (increasing the spatial
+resolution).
+
+---
+
+## Frame Interpolation (video only)
+
+Frame interpolation synthesizes new frames between each pair of consecutive
+generated frames, producing smoother motion without re-running the diffusion
+model.
+
+The `--frame-interpolation-exp` flag controls how many rounds of interpolation
+to apply: each round inserts one new frame into every gap between adjacent
+frames, so the output frame count follows the formula:
+
+> **(N − 1) × 2^exp + 1**
+>
+> e.g. 5 original frames with `exp=1` → 4 gaps × 1 new frame + 5 originals = **9** frames;
+> with `exp=2` → **17** frames.
+
+### CLI Arguments
+
+| Argument | Description |
+|----------|-------------|
+| `--enable-frame-interpolation` | Enable frame interpolation. Model weights are downloaded automatically on first use. |
+| `--frame-interpolation-exp {EXP}` | Interpolation exponent — `1` = 2× temporal resolution, `2` = 4×, etc. (default: `1`) |
+| `--frame-interpolation-scale {SCALE}` | RIFE inference scale; use `0.5` for high-resolution inputs to save memory (default: `1.0`) |
+| `--frame-interpolation-model-path {PATH}` | Local directory or HuggingFace repo ID containing RIFE `flownet.pkl` weights (default: `elfgum/RIFE-4.22.lite`, downloaded automatically) |
+
+### Supported Models
+
+Frame interpolation uses the [RIFE](https://github.com/hzwer/Practical-RIFE)
+(Real-Time Intermediate Flow Estimation) architecture. Only **RIFE 4.22.lite**
+(`IFNet` with 4-scale `IFBlock` backbone) is supported. The network topology is
+hard-coded, so custom weights provided via `--frame-interpolation-model-path`
+must be a `flownet.pkl` checkpoint that is compatible with this architecture.
+
+Other RIFE versions (e.g., older `v4.x` variants with different block counts)
+or entirely different frame interpolation methods (FILM, AMT, etc.) are **not
+supported**.
+
+| Weight | HuggingFace Repo | Description |
+|--------|------------------|-------------|
+| RIFE 4.22.lite *(default)* | [`elfgum/RIFE-4.22.lite`](https://huggingface.co/elfgum/RIFE-4.22.lite) | Lightweight model, downloaded automatically on first use |
+
+### Example
+
+Generate a 5-frame video and interpolate to 9 frames ((5 − 1) × 2¹ + 1 = 9):
+
+```bash
+sglang generate \
+  --model-path Wan-AI/Wan2.2-T2V-A14B-Diffusers \
+  --prompt "A dog running through a park" \
+  --num-frames 5 \
+  --enable-frame-interpolation \
+  --frame-interpolation-exp 1 \
+  --save-output
+```
+
+---
+
+## Upscaling (image and video)
+
+Upscaling increases the spatial resolution of generated images or video frames
+using [Real-ESRGAN](https://github.com/xinntao/Real-ESRGAN). The model weights
+are downloaded automatically on first use and cached for subsequent runs.
+
+### CLI Arguments
+
+| Argument | Description |
+|----------|-------------|
+| `--enable-upscaling` | Enable post-generation upscaling using Real-ESRGAN. |
+| `--upscaling-scale {SCALE}` | Desired upscaling factor (default: `4`). The 4× model is used internally; if a different scale is requested, a bicubic resize is applied after the network output. |
+| `--upscaling-model-path {PATH}` | Local `.pth` file, HuggingFace repo ID, or `repo_id:filename` for Real-ESRGAN weights (default: `ai-forever/Real-ESRGAN` with `RealESRGAN_x4.pth`, downloaded automatically). Use the `repo_id:filename` format to specify a custom weight file from a HuggingFace repo (e.g. `my-org/my-esrgan:weights.pth`). |
+
+### Supported Models
+
+Upscaling supports two Real-ESRGAN network architectures. The correct
+architecture is **auto-detected** from the checkpoint keys, so you only need to
+point `--upscaling-model-path` at a valid `.pth` file:
+
+| Architecture | Example Weights | Description |
+|--------------|-----------------|-------------|
+| **RRDBNet** | `RealESRGAN_x4plus.pth` | Heavier model with higher quality; best for photos |
+| **SRVGGNetCompact** | `RealESRGAN_x4.pth` *(default)*, `realesr-animevideov3.pth`, `realesr-general-x4v3.pth` | Lightweight model; faster inference, good for video |
+
+The default weight file is
+[`ai-forever/Real-ESRGAN`](https://huggingface.co/ai-forever/Real-ESRGAN) with
+`RealESRGAN_x4.pth` (SRVGGNetCompact, 4× native scale).
+
+Other super-resolution models (e.g., SwinIR, HAT, BSRGAN) are **not supported**
+— only Real-ESRGAN checkpoints using the two architectures above are
+compatible.
+
+### Examples
+
+Generate a 1024×1024 image and upscale to 4096×4096:
+
+```bash
+sglang generate \
+  --model-path black-forest-labs/FLUX.2-dev \
+  --prompt "A cat sitting on a windowsill" \
+  --output-size 1024x1024 \
+  --enable-upscaling \
+  --save-output
+```
+
+Generate a video and upscale each frame by 4×:
+
+```bash
+sglang generate \
+  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --prompt "A curious raccoon" \
+  --enable-upscaling \
+  --upscaling-scale 4 \
+  --save-output
+```
+
+---
+
+## Combining Frame Interpolation and Upscaling
+
+Frame interpolation and upscaling can be combined in a single run.
+Interpolation is applied first (increasing the frame count), then upscaling is
+applied to every frame (increasing the spatial resolution).
+
+Example — generate 5 frames, interpolate to 9 frames, and upscale each frame
+by 4×:
+
+```bash
+sglang generate \
+  --model-path Wan-AI/Wan2.1-T2V-1.3B-Diffusers \
+  --prompt "A curious raccoon" \
+  --num-frames 5 \
+  --enable-frame-interpolation \
+  --frame-interpolation-exp 1 \
+  --enable-upscaling \
+  --upscaling-scale 4 \
+  --save-output
+```

third_party/sglang/docs/diffusion/ci_perf.md

@@ -0,0 +1,31 @@
+# CI Performance
+
+## Perf Baseline Generation Script
+
+`python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py` starts a local diffusion server, issues requests for selected test cases, aggregates stage/denoise-step/E2E timings from the perf log, and writes the results back to the `scenarios` section of `perf_baselines.json`.
+
+### Usage
+
+Update a single case:
+
+```bash
+python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --case qwen_image_t2i
+```
+
+Select by regex:
+
+```bash
+python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --match 'qwen_image_.*'
+```
+
+Run all keys from the baseline file `scenarios`:
+
+```bash
+python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --all-from-baseline
+```
+
+Specify input/output paths and timeout:
+
+```bash
+python python/sglang/multimodal_gen/test/scripts/gen_perf_baselines.py --baseline python/sglang/multimodal_gen/test/server/perf_baselines.json --out /tmp/perf_baselines.json --timeout 600
+```

third_party/sglang/docs/diffusion/compatibility_matrix.md

@@ -0,0 +1,81 @@
+# Compatibility Matrix
+
+The table below shows every supported model and the optimizations supported for them.
+
+The symbols used have the following meanings:
+
+- ✅ = Full compatibility
+- ❌ = No compatibility
+- ⭕ = Does not apply to this model
+
+## Models x Optimization
+
+The `HuggingFace Model ID` can be passed directly to `from_pretrained()` methods, and sglang-diffusion will use the
+optimal
+default parameters when initializing and generating videos.
+
+### Video Generation Models
+
+| Model Name                   | Hugging Face Model ID                             | Resolutions         | TeaCache | Sliding Tile Attn | Sage Attn | Video Sparse Attention (VSA) | Sparse Linear Attention (SLA) | Sage Sparse Linear Attention (SageSLA) | Sparse Video Gen 2 (SVG2) |
+|:-----------------------------|:--------------------------------------------------|:--------------------|:--------:|:-----------------:|:---------:|:----------------------------:|:----------------------------:|:-----------------------------------------------:|:----------------------------------:|
+| FastWan2.1 T2V 1.3B          | `FastVideo/FastWan2.1-T2V-1.3B-Diffusers`         | 480p                |    ⭕     |         ⭕         |      ⭕     |              ✅               |              ❌               |              ❌               |    ❌     |
+| FastWan2.2 TI2V 5B Full Attn | `FastVideo/FastWan2.2-TI2V-5B-FullAttn-Diffusers` | 720p                |    ⭕     |         ⭕         |     ⭕     |              ✅               |              ❌               |              ❌               |    ❌     |
+| Wan2.2 TI2V 5B               | `Wan-AI/Wan2.2-TI2V-5B-Diffusers`                 | 720p                |    ⭕     |         ⭕         |     ✅     |              ⭕               |              ❌               |              ❌               |    ❌     |
+| Wan2.2 T2V A14B              | `Wan-AI/Wan2.2-T2V-A14B-Diffusers`                | 480p<br>720p        |    ❌     |         ❌         |     ✅     |              ⭕               |              ❌               |              ❌               |    ❌     |
+| Wan2.2 I2V A14B              | `Wan-AI/Wan2.2-I2V-A14B-Diffusers`                | 480p<br>720p        |    ❌     |         ❌         |     ✅     |              ⭕               |              ❌               |              ❌               |    ❌     |
+| HunyuanVideo                 | `hunyuanvideo-community/HunyuanVideo`             | 720×1280<br>544×960 |    ❌     |         ✅         |     ✅     |              ⭕               |              ❌               |              ❌               |    ✅     |
+| FastHunyuan                  | `FastVideo/FastHunyuan-diffusers`                 | 720×1280<br>544×960 |    ❌     |         ✅         |     ✅     |              ⭕               |              ❌               |              ❌               |    ✅     |
+| Wan2.1 T2V 1.3B              | `Wan-AI/Wan2.1-T2V-1.3B-Diffusers`                | 480p                |    ✅     |         ✅         |     ✅     |              ⭕               |              ❌               |              ❌               |    ✅     |
+| Wan2.1 T2V 14B               | `Wan-AI/Wan2.1-T2V-14B-Diffusers`                 | 480p, 720p          |    ✅     |         ✅         |     ✅     |              ⭕               |              ❌               |              ❌               |    ✅     |
+| Wan2.1 I2V 480P              | `Wan-AI/Wan2.1-I2V-14B-480P-Diffusers`            | 480p                |    ✅     |         ✅         |     ✅     |              ⭕               |              ❌               |              ❌               |    ✅     |
+| Wan2.1 I2V 720P              | `Wan-AI/Wan2.1-I2V-14B-720P-Diffusers`            | 720p                |    ✅     |         ✅         |     ✅     |              ⭕               |              ❌               |              ❌               |    ✅     |
+| TurboWan2.1 T2V 1.3B         | `IPostYellow/TurboWan2.1-T2V-1.3B-Diffusers`      | 480p                |    ✅     |         ❌         |     ❌     |              ❌               |              ✅               |              ✅               |    ⭕     |
+| TurboWan2.1 T2V 14B          | `IPostYellow/TurboWan2.1-T2V-14B-Diffusers`       | 480p                |    ✅     |         ❌         |     ❌     |              ❌               |              ✅               |              ✅               |    ⭕     |
+| TurboWan2.1 T2V 14B 720P     | `IPostYellow/TurboWan2.1-T2V-14B-720P-Diffusers`  | 720p                |    ✅     |         ❌         |     ❌     |              ❌               |              ✅               |              ✅               |    ⭕     |
+| TurboWan2.2 I2V A14B         | `IPostYellow/TurboWan2.2-I2V-A14B-Diffusers`      | 720p                |    ✅     |         ❌         |     ❌     |              ❌               |              ✅               |              ✅               |    ⭕     |
+| LTX-2                        | `Lightricks/LTX-2`                                | 1536×1024           |    ❌     |         ❌         |     ❌     |              ❌               |              ❌               |              ❌               |    ❌     |
+
+**Note**:
+1. Wan2.2 TI2V 5B has some quality issues when performing I2V generation. We are working on fixing this issue.
+2. SageSLA is based on SpargeAttn. Install it first with `pip install git+https://github.com/thu-ml/SpargeAttn.git --no-build-isolation`
+3. LTX-2 two-stage generation uses `--pipeline-class-name LTX2TwoStagePipeline`. The spatial upsampler and distilled LoRA are auto-resolved from the model snapshot by default, and can still be overridden with `--spatial-upsampler-path` and `--distilled-lora-path`.
+
+### Image Generation Models
+
+| Model Name           | HuggingFace Model ID                |
+|:---------------------|:------------------------------------|
+| FLUX.1-dev           | `black-forest-labs/FLUX.1-dev`      |
+| FLUX.2-dev           | `black-forest-labs/FLUX.2-dev`      |
+| FLUX.2-Klein         | `black-forest-labs/FLUX.2-klein-4B` |
+| Z-Image-Turbo        | `Tongyi-MAI/Z-Image-Turbo`          |
+| GLM-Image            | `zai-org/GLM-Image`                 |
+| Qwen Image           | `Qwen/Qwen-Image`                   |
+| Qwen Image 2512      | `Qwen/Qwen-Image-2512`              |
+| Qwen Image Edit      | `Qwen/Qwen-Image-Edit`              |
+| Qwen Image Edit 2511 | `Qwen/Qwen-Image-Edit-2511`         |
+
+## Verified LoRA Examples
+
+This section lists example LoRAs that have been explicitly tested and verified with each base model in the **SGLang Diffusion** pipeline.
+
+> Important:
+> LoRAs that are not listed here are not necessarily incompatible.
+> In practice, most standard LoRAs are expected to work, especially those following common Diffusers or SD-style conventions.
+> The entries below simply reflect configurations that have been manually validated by the SGLang team.
+
+### Verified LoRAs by Base Model
+
+| Base Model       | Supported LoRAs |
+|:-----------------|:----------------|
+| Wan2.2           | `lightx2v/Wan2.2-Distill-Loras`<br>`Cseti/wan2.2-14B-Arcane_Jinx-lora-v1` |
+| Wan2.1           | `lightx2v/Wan2.1-Distill-Loras` |
+| Z-Image-Turbo    | `tarn59/pixel_art_style_lora_z_image_turbo`<br>`wcde/Z-Image-Turbo-DeJPEG-Lora` |
+| Qwen-Image       | `lightx2v/Qwen-Image-Lightning`<br>`flymy-ai/qwen-image-realism-lora`<br>`prithivMLmods/Qwen-Image-HeadshotX`<br>`starsfriday/Qwen-Image-EVA-LoRA` |
+| Qwen-Image-Edit  | `ostris/qwen_image_edit_inpainting`<br>`lightx2v/Qwen-Image-Edit-2511-Lightning` |
+| Flux             | `dvyio/flux-lora-simple-illustration`<br>`XLabs-AI/flux-furry-lora`<br>`XLabs-AI/flux-RealismLora` |
+
+## Special requirements
+
+### Sliding Tile Attention
+
+- Currently, only Hopper GPUs (H100s) are supported.

third_party/sglang/docs/diffusion/contributing.md

@@ -0,0 +1,79 @@
+# Contributing to SGLang Diffusion
+
+This guide outlines the requirements for contributing to the SGLang Diffusion module (`sglang.multimodal_gen`).
+
+## Contributor Guides
+
+- [Support New Models](support_new_models.md): implementation guide for adding new diffusion pipelines
+- [CI Performance](ci_perf.md): update and regenerate perf baselines
+
+```{toctree}
+:maxdepth: 1
+
+support_new_models
+ci_perf
+```
+
+## On AI-Assisted ("Vibe Coding") PRs
+
+Vibe-coded PRs are welcome — we judge code quality, not how it was produced. The bar is the same for all PRs:
+
+- **No over-commenting.** If the name says it all, skip the docstring.
+- **No over-catching.** Don't guard against errors that virtually never happen in practice.
+- **Test before submitting.** AI-generated code can be subtly wrong — verify correctness end-to-end.
+
+## Commit Message Convention
+
+We follow a structured commit message format to maintain a clean history.
+
+**Format:**
+```text
+[diffusion] <scope>: <subject>
+```
+
+**Examples:**
+- `[diffusion] cli: add --perf-dump-path argument`
+- `[diffusion] scheduler: fix deadlock in batch processing`
+- `[diffusion] model: support Stable Diffusion 3.5`
+
+**Rules:**
+- **Prefix**: Always start with `[diffusion]`.
+- **Scope** (Optional): `cli`, `scheduler`, `model`, `pipeline`, `docs`, etc.
+- **Subject**: Imperative mood, short and clear (e.g., "add feature" not "added feature").
+
+## Performance Reporting
+
+For PRs that impact **latency**, **throughput**, or **memory usage**, you **should** provide a performance comparison report.
+
+### How to Generate a Report
+
+1.  **Baseline**: run the benchmark (for a single generation task)
+    ```bash
+    $ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path baseline.json
+    ```
+
+2.  **New**: run the same benchmark, without modifying any server_args or sampling_params
+    ```bash
+    $ sglang generate --model-path <model> --prompt "A benchmark prompt" --perf-dump-path new.json
+    ```
+
+3.  **Compare**: run the compare script, which will print a Markdown table to the console
+    ```bash
+    $ python python/sglang/multimodal_gen/benchmarks/compare_perf.py baseline.json new.json [new2.json ...]
+    ### Performance Comparison Report
+    ...
+    ```
+4. **Paste**: paste the table into the PR description
+
+## CI-Based Change Protection
+
+Consider adding tests to the `pr-test` or `nightly-test` suites to safeguard your changes, especially for PRs that:
+
+- support a new model
+    - add a testcase for this new model to `testcase_configs.py`
+- support or fix important features
+- significantly improve performance
+
+Please run the according testcase, then update/add the baseline to `perf_baselines.json` by following the instruction in console if applicable.
+
+See [test](https://github.com/sgl-project/sglang/tree/main/python/sglang/multimodal_gen/test) for examples

third_party/sglang/docs/diffusion/development.md

@@ -0,0 +1,5 @@
+# Development
+
+This page collects lower-level development material for SGLang Diffusion.
+
+- [Contributing](contributing.md): contribution workflow, adding new models, and CI perf baselines

third_party/sglang/docs/diffusion/environment_variables.md

@@ -0,0 +1,56 @@
+# Environment Variables
+
+## Apple MPS
+
+| Environment Variable | Default | Description                                                  |
+|----------------------|---------|--------------------------------------------------------------|
+| `SGLANG_USE_MLX`     | not set | Set to `1` to enable MLX fused Metal kernels for norm ops on MPS |
+
+## Caching Acceleration
+
+These variables configure caching acceleration for Diffusion Transformer (DiT) models.
+SGLang supports multiple caching strategies - see [caching documentation](performance/cache/index.md) for an overview.
+
+### Cache-DiT Configuration
+
+See [cache-dit documentation](performance/cache/cache_dit.md) for detailed configuration.
+
+| Environment Variable                | Default | Description                              |
+|-------------------------------------|---------|------------------------------------------|
+| `SGLANG_CACHE_DIT_ENABLED`          | false   | Enable Cache-DiT acceleration            |
+| `SGLANG_CACHE_DIT_FN`               | 1       | First N blocks to always compute         |
+| `SGLANG_CACHE_DIT_BN`               | 0       | Last N blocks to always compute          |
+| `SGLANG_CACHE_DIT_WARMUP`           | 4       | Warmup steps before caching              |
+| `SGLANG_CACHE_DIT_RDT`              | 0.24    | Residual difference threshold            |
+| `SGLANG_CACHE_DIT_MC`               | 3       | Max continuous cached steps              |
+| `SGLANG_CACHE_DIT_TAYLORSEER`       | false   | Enable TaylorSeer calibrator             |
+| `SGLANG_CACHE_DIT_TS_ORDER`         | 1       | TaylorSeer order (1 or 2)                |
+| `SGLANG_CACHE_DIT_SCM_PRESET`       | none    | SCM preset (none/slow/medium/fast/ultra) |
+| `SGLANG_CACHE_DIT_SCM_POLICY`       | dynamic | SCM caching policy                       |
+| `SGLANG_CACHE_DIT_SCM_COMPUTE_BINS` | not set | Custom SCM compute bins                  |
+| `SGLANG_CACHE_DIT_SCM_CACHE_BINS`   | not set | Custom SCM cache bins                    |
+
+## Cloud Storage
+
+These variables configure S3-compatible cloud storage for automatically uploading generated images and videos.
+
+| Environment Variable            | Default | Description                                            |
+|---------------------------------|---------|--------------------------------------------------------|
+| `SGLANG_CLOUD_STORAGE_TYPE`     | not set | Set to `s3` to enable cloud storage                    |
+| `SGLANG_S3_BUCKET_NAME`         | not set | The name of the S3 bucket                              |
+| `SGLANG_S3_ENDPOINT_URL`        | not set | Custom endpoint URL (for MinIO, OSS, etc.)             |
+| `SGLANG_S3_REGION_NAME`         | us-east-1 | AWS region name                                      |
+| `SGLANG_S3_ACCESS_KEY_ID`       | not set | AWS Access Key ID                                      |
+| `SGLANG_S3_SECRET_ACCESS_KEY`   | not set | AWS Secret Access Key                                  |
+
+## CUDA Crash Debugging
+
+These variables enable kernel API logging and optional input/output dumps around diffusion CUDA kernel call boundaries. They are useful when tracking down CUDA crashes such as illegal memory access, device-side assert, or shape mismatches in custom kernels.
+
+| Environment Variable | Default | Description |
+|----------------------|---------|-------------|
+| `SGLANG_KERNEL_API_LOGLEVEL` | `0` | Controls crash-debug kernel API logging. `1` logs API names, `3` logs tensor metadata, `5` adds tensor statistics, and `10` also writes dump snapshots. |
+| `SGLANG_KERNEL_API_LOGDEST` | `stdout` | Destination for crash-debug kernel API logs. Use `stdout`, `stderr`, or a file path. `%i` is replaced with the process PID. |
+| `SGLANG_KERNEL_API_DUMP_DIR` | `sglang_kernel_api_dumps` | Output directory for level-10 kernel API dumps. `%i` is replaced with the process PID. |
+| `SGLANG_KERNEL_API_DUMP_INCLUDE` | not set | Comma-separated wildcard patterns for kernel API names to include in level-10 dumps. |
+| `SGLANG_KERNEL_API_DUMP_EXCLUDE` | not set | Comma-separated wildcard patterns for kernel API names to exclude from level-10 dumps. |

third_party/sglang/docs/diffusion/index.md

@@ -0,0 +1,53 @@
+# SGLang Diffusion
+
+SGLang Diffusion is a high-performance inference framework for image and video generation. It provides native SGLang pipelines, diffusers backend support, an OpenAI-compatible server, and an optimized kernel stack built on both precompiled `sgl-kernel` operators and JIT kernels for key inference paths.
+
+## Key Features
+
+- Broad model support across Wan, Hunyuan, Qwen-Image, FLUX, Z-Image, GLM-Image, and more
+- Fast inference with `sgl-kernel`, JIT kernels, scheduler improvements, and caching acceleration
+- Multiple interfaces: `sglang generate`, `sglang serve`, and an OpenAI-compatible API
+- Multi-platform support for NVIDIA, AMD, Ascend, Apple Silicon, and Moore Threads
+
+## Quick Start
+
+```bash
+uv pip install "sglang[diffusion]" --prerelease=allow
+```
+
+```bash
+sglang generate --model-path Qwen/Qwen-Image \
+  --prompt "A beautiful sunset over the mountains" \
+  --save-output
+```
+
+```bash
+sglang serve --model-path Qwen/Qwen-Image --port 30010
+```
+
+## Start Here
+
+- [Installation](installation.md): install SGLang Diffusion and platform dependencies
+- [Compatibility Matrix](compatibility_matrix.md): check model and optimization support
+- [CLI](api/cli.md): run one-off generation jobs or launch a persistent server
+- [OpenAI-Compatible API](api/openai_api.md): send image and video requests to the HTTP server
+- [Attention Backends](performance/attention_backends.md): choose the best backend for your model and hardware
+- [Caching Acceleration](performance/cache/index.md): use Cache-DiT or TeaCache to reduce denoising cost
+- [Quantization](quantization.md): load quantized transformer checkpoints
+- [Contributing](contributing.md): contribution workflow, adding new models, and CI perf baselines
+
+## Additional Documentation
+
+- [Post-Processing](api/post_processing.md): frame interpolation and upscaling
+- [Performance Overview](performance/index.md): overview of attention, caching, and profiling
+- [Environment Variables](environment_variables.md): platform, caching, storage, and debugging configuration
+- [Support New Models](support_new_models.md): implementation guide for new diffusion pipelines
+- [CI Performance](ci_perf.md): performance baseline generation
+
+## References
+
+- [SGLang GitHub](https://github.com/sgl-project/sglang)
+- [Cache-DiT](https://github.com/vipshop/cache-dit)
+- [FastVideo](https://github.com/hao-ai-lab/FastVideo)
+- [xDiT](https://github.com/xdit-project/xDiT)
+- [Diffusers](https://github.com/huggingface/diffusers)

third_party/sglang/docs/diffusion/installation.md

@@ -0,0 +1,120 @@
+# Install SGLang-Diffusion
+
+You can install SGLang-Diffusion using one of the methods below. The standard installation already includes SGLang's optimized kernel stack, including both `sgl-kernel` and JIT kernels used by diffusion workloads.
+
+## Standard Installation (NVIDIA GPUs)
+
+### Method 1: With pip or uv
+
+It is recommended to use uv for a faster installation:
+
+```bash
+pip install --upgrade pip
+pip install uv
+uv pip install "sglang[diffusion]" --prerelease=allow
+```
+
+### Method 2: From source
+
+```bash
+# Use the latest release branch
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+
+# Install the Python packages
+pip install --upgrade pip
+pip install -e "python[diffusion]"
+
+# With uv
+uv pip install -e "python[diffusion]" --prerelease=allow
+```
+
+### Method 3: Using Docker
+
+The Docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang), built from the [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/Dockerfile).
+Replace `<secret>` below with your HuggingFace Hub [token](https://huggingface.co/docs/hub/en/security-tokens).
+
+```bash
+docker run --gpus all \
+    --shm-size 32g \
+    -p 30000:30000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    --env "HF_TOKEN=<secret>" \
+    --ipc=host \
+    lmsysorg/sglang:dev \
+    zsh -c '\
+        echo "Installing diffusion dependencies..." && \
+        pip install -e "python[diffusion]" && \
+        echo "Starting SGLang-Diffusion..." && \
+        sglang generate \
+            --model-path black-forest-labs/FLUX.1-dev \
+            --prompt "A logo With Bold Large text: SGL Diffusion" \
+            --save-output \
+    '
+```
+
+## Platform-Specific: ROCm (AMD GPUs)
+
+For AMD Instinct GPUs (e.g., MI300X), you can use the ROCm-enabled Docker image:
+
+```bash
+docker run --device=/dev/kfd --device=/dev/dri --ipc=host \
+  -v ~/.cache/huggingface:/root/.cache/huggingface \
+  --env HF_TOKEN=<secret> \
+  lmsysorg/sglang:v0.5.5.post2-rocm700-mi30x \
+  sglang generate --model-path black-forest-labs/FLUX.1-dev --prompt "A logo With Bold Large text: SGL Diffusion" --save-output
+```
+
+For detailed ROCm system configuration and installation from source, see [AMD GPUs](../platforms/amd_gpu.md).
+
+## Platform-Specific: MUSA (Moore Threads GPUs)
+
+For Moore Threads GPUs (MTGPU) with the MUSA software stack, please follow the instructions below to install from source:
+
+```bash
+# Clone the repository
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+
+# Install the Python packages
+pip install --upgrade pip
+rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
+pip install -e "python[all_musa]"
+```
+
+## Platform-Specific: Ascend NPU
+
+For Ascend NPU, please follow the [NPU installation guide](../platforms/ascend/ascend_npu.md).
+
+Quick test:
+
+```bash
+sglang generate --model-path black-forest-labs/FLUX.1-dev \
+    --prompt "A logo With Bold Large text: SGL Diffusion" \
+    --save-output
+```
+
+## Platform-Specific: Apple MPS
+
+For Apple MPS, please follow the instructions below to install from source:
+
+```bash
+# Install ffmpeg
+brew install ffmpeg
+
+# Install uv
+brew install uv
+
+# Clone the repository
+git clone https://github.com/sgl-project/sglang.git
+cd sglang
+
+# Create and activate a virtual environment
+uv venv -p 3.11 sglang-diffusion
+source sglang-diffusion/bin/activate
+
+# Install the Python packages
+uv pip install --upgrade pip
+rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
+uv pip install -e "python[all_mps]"
+```

third_party/sglang/docs/diffusion/performance/attention_backends.md

@@ -0,0 +1,133 @@
+# Attention Backends
+
+This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them.
+
+## Overview
+
+Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`.
+
+Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`) and therefore applies to any model component using these layers (e.g. diffusion transformer / DiT and encoders).
+
+When using the diffusers backend, `--attention-backend` is passed through to diffusers'
+`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).
+
+- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA.
+- **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
+- **MPS**: always uses PyTorch SDPA.
+- **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA.
+
+## Backend options
+
+For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
+
+| CLI value | Enum value | Notes |
+|---|---|---|
+| `fa` / `fa3` / `fa4` | `FA` | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`). |
+| `torch_sdpa` | `TORCH_SDPA` | PyTorch `scaled_dot_product_attention`. |
+| `sliding_tile_attn` | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn`. Configure via `--attention-backend-config`. |
+| `sage_attn` | `SAGE_ATTN` | Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py. |
+| `sage_attn_3` | `SAGE_ATTN_3` | Requires SageAttention3 installed per upstream instructions. |
+| `video_sparse_attn` | `VIDEO_SPARSE_ATTN` | Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
+| `vmoba_attn` | `VMOBA_ATTN` | Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
+| `aiter` | `AITER` | Requires `aiter`. |
+| `aiter_sage` | `AITER_SAGE` | Requires `aiter`. |
+| `sparse_video_gen_2_attn` | `SPARSE_VIDEO_GEN_2_ATTN` | Requires `svg`. See installation instructions at https://github.com/svg-project/Sparse-VideoGen. |
+
+## Selection priority
+
+The selection order in `runtime/layers/attention/selector.py` is:
+
+1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
+2. CLI `--attention-backend` (`ServerArgs.attention_backend`)
+3. Auto selection (platform capability, dtype, and installed packages)
+
+## Configuration
+
+Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts:
+- A path to a JSON or YAML configuration file.
+- A JSON string (e.g., `'{"sparsity": 0.5}'`).
+- Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`).
+
+### Supported Configuration Parameters
+
+**Sliding Tile Attention (`sliding_tile_attn`)**
+
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `mask_strategy_file_path` | `str` | **Required.** Path to the mask strategy JSON file. | - |
+| `sta_mode` | `str` | Mode of STA. | `STA_inference` |
+| `skip_time_steps` | `int` | Number of steps to use full attention before switching to sparse attention. | `15` |
+
+**Video Sparse Attention (`video_sparse_attn`)**
+
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `sparsity` | `float` | Validation sparsity (0.0 - 1.0). | `0.0` |
+
+**V-MoBA (`vmoba_attn`)**
+
+| Parameter | Type | Description | Default |
+| :--- | :--- | :--- | :--- |
+| `temporal_chunk_size` | `int` | Chunk size for temporal dimension. | - |
+| `temporal_topk` | `int` | Top-K tokens to select in temporal dimension. | - |
+| `spatial_chunk_size` | `list[int]` | Chunk size for spatial dimension (H, W). | - |
+| `spatial_topk` | `int` | Top-K tokens to select in spatial dimension. | - |
+| `st_chunk_size` | `list[int]` | Chunk size for spatiotemporal dimension (T, H, W). | - |
+| `st_topk` | `int` | Top-K tokens to select in spatiotemporal dimension. | - |
+| `moba_select_mode` | `str` | Selection mode (e.g., `threshold`). | `threshold` |
+| `moba_threshold` | `float` | Threshold value for selection. | `0.25` |
+| `moba_threshold_type` | `str` | Type of thresholding (e.g., `query_head`). | `query_head` |
+| `first_full_step` | `int` | Number of initial steps to use full attention. | `12` |
+| `first_full_layer` | `int` | Number of initial layers to use full attention. | `0` |
+| `temporal_layer` | `int` | Number of temporal layers. | `1` |
+| `spatial_layer` | `int` | Number of spatial layers. | `1` |
+| `st_layer` | `int` | Number of spatiotemporal layers. | `1` |
+
+## Platform support matrix
+
+| Backend | CUDA | ROCm | MPS | NPU | Notes |
+|---|---:|---:|---:|---:|---|
+| `fa` | ✅ | ✅ | ❌ | ✅ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU |
+| `torch_sdpa` | ✅ | ✅ | ✅ | ✅ | Most compatible option across platforms. |
+| `sliding_tile_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`. |
+| `sage_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). |
+| `sage_attn_3` | ✅ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). |
+| `video_sparse_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
+| `vmoba_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
+| `aiter` | ❌ | ✅ | ❌ | ❌ | Requires `aiter`. |
+| `aiter_sage` | ❌ | ✅ | ❌ | ❌ | Requires `aiter`. |
+| `sparse_video_gen_2_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `svg`. |
+
+## Usage
+
+### Select a backend via CLI
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend fa
+```
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend torch_sdpa
+```
+
+### Using Sliding Tile Attention (STA)
+
+```bash
+# Pass the mask strategy file path via config
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "..." \
+  --attention-backend sliding_tile_attn \
+  --attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"
+```
+
+### Notes for ROCm / MPS
+
+- ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment.
+- MPS: the platform implementation always uses `torch_sdpa`.

third_party/sglang/docs/diffusion/performance/cache/cache_dit.md

@@ -0,0 +1,418 @@
+# Cache-DiT
+
+SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.
+
+## Overview
+
+**Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop:
+
+- **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
+- **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
+- **SCM (Step Computation Masking)**: Step-level caching control for additional speedup
+
+## Basic Usage
+
+Enable Cache-DiT by exporting the environment variable and using `sglang generate` or `sglang serve` :
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A beautiful sunset over the mountains"
+```
+
+## Diffusers Backend
+
+Cache-DiT supports loading acceleration configs from a custom YAML file. For
+diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This
+flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`).
+
+### Single GPU inference
+
+Define a `cache.yaml` file that contains:
+
+- DBCache + TaylorSeer
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+```
+
+Then apply the config with:
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --model-path Qwen/Qwen-Image \
+  --cache-dit-config cache.yaml \
+  --prompt "A beautiful sunset over the mountains"
+```
+
+- DBCache + TaylorSeer + SCM (Step Computation Mask)
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+  # Must set the num_inference_steps for SCM. The SCM will automatically
+  # generate the steps computation mask based on the num_inference_steps.
+  # Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking
+  num_inference_steps: 28
+  steps_computation_mask: fast
+```
+
+- DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+  num_inference_steps: 28
+  steps_computation_mask: fast
+  enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc.
+```
+
+### Distributed inference
+
+- 1D Parallelism
+
+Define a parallelism only config yaml `parallel.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+```
+
+Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses_size. Otherwise, you should manually set it as specific int number, e.g, 4.
+
+Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference)
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --num-gpus 4 \
+  --model-path Qwen/Qwen-Image \
+  --cache-dit-config parallel.yaml \
+  --prompt "A futuristic cityscape at sunset"
+```
+
+- 2D Parallelism
+
+You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  tp_size: 2
+  attention_backend: native
+```
+Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses_size.
+
+- 3D Parallelism
+
+You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: 2
+  ring_size: 2
+  tp_size: 2
+  attention_backend: native
+```
+Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2.
+
+- Ulysses Anything Attention
+
+To enable Ulysses Anything Attention, you can define a parallelism config yaml `parallel_uaa.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  ulysses_anything: true
+```
+
+- Ulysses FP8 Communication
+
+For device that don't have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml `parallel_fp8.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  ulysses_float8: true
+```
+
+- Async Ulysses CP
+
+You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml `parallel_async.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image.
+```
+Then, apply the config from yaml. Here `ulysses_async: true` means enabling async ulysses CP.
+
+- TE-P and VAE-P
+
+You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml `parallel_extra.yaml` file that contains:
+
+```yaml
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  extra_parallel_modules: ["text_encoder", "vae"]
+```
+
+
+### Hybrid Cache and Parallelism
+
+Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains:
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  extra_parallel_modules: ["text_encoder", "vae"]
+```
+
+Then, apply the hybrid cache and parallel acceleration config from yaml.
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --num-gpus 4 \
+  --model-path Qwen/Qwen-Image \
+  --cache-dit-config hybrid.yaml \
+  --prompt "A beautiful sunset over the mountains"
+```
+
+### Attention Backend
+
+In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file `attention.yaml` that only contains:
+
+```yaml
+attention_backend: "flash" # '_flash_3' for Hopper
+```
+
+### Quantization
+
+You can also specify the quantization config in the yaml file, required `torchao>=0.16.0`. For example, define a yaml file `quantize.yaml` that contains:
+
+```yaml
+quantize_config: # quantization configuration for transformer modules
+  # float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc.
+  quant_type: "float8"
+  # layers to exclude from quantization (transformer). layers that contains any of the
+  # keywords in the exclude_layers list will be excluded from quantization. This is useful
+  # for some sensitive layers that are not robust to quantization, e.g., embedding layers.
+  exclude_layers:
+    - "embedder"
+    - "embed"
+  verbose: false # whether to print verbose logs during quantization
+```
+Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example:
+
+```bash
+sglang generate \
+  --backend diffusers \
+  --model-path Qwen/Qwen-Image \
+  --warmup \
+  --cache-dit-config quantize.yaml \
+  --enable-torch-compile \
+  --dit-cpu-offload false \
+  --text-encoder-cpu-offload false \
+  --prompt "A beautiful sunset over the mountains"
+```
+
+### Combined Configs: Cache + Parallelism + Quantization
+
+You can also combine all the above configs together in a single yaml file `combined.yaml` that contains:
+
+```yaml
+cache_config:
+  max_warmup_steps: 8
+  warmup_interval: 2
+  max_cached_steps: -1
+  max_continuous_cached_steps: 2
+  Fn_compute_blocks: 1
+  Bn_compute_blocks: 0
+  residual_diff_threshold: 0.12
+  enable_taylorseer: true
+  taylorseer_order: 1
+parallelism_config:
+  ulysses_size: auto
+  attention_backend: native
+  extra_parallel_modules: ["text_encoder", "vae"]
+quantize_config:
+  quant_type: "float8"
+  exclude_layers:
+    - "embedder"
+    - "embed"
+  verbose: false
+```
+Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization.
+
+## Advanced Configuration
+
+### DBCache Parameters
+
+DBCache controls block-level caching behavior:
+
+| Parameter | Env Variable              | Default | Description                              |
+|-----------|---------------------------|---------|------------------------------------------|
+| Fn        | `SGLANG_CACHE_DIT_FN`     | 1       | Number of first blocks to always compute |
+| Bn        | `SGLANG_CACHE_DIT_BN`     | 0       | Number of last blocks to always compute  |
+| W         | `SGLANG_CACHE_DIT_WARMUP` | 4       | Warmup steps before caching starts       |
+| R         | `SGLANG_CACHE_DIT_RDT`    | 0.24    | Residual difference threshold            |
+| MC        | `SGLANG_CACHE_DIT_MC`     | 3       | Maximum continuous cached steps          |
+
+### TaylorSeer Configuration
+
+TaylorSeer improves caching accuracy using Taylor expansion:
+
+| Parameter | Env Variable                  | Default | Description                     |
+|-----------|-------------------------------|---------|---------------------------------|
+| Enable    | `SGLANG_CACHE_DIT_TAYLORSEER` | false   | Enable TaylorSeer calibrator    |
+| Order     | `SGLANG_CACHE_DIT_TS_ORDER`   | 1       | Taylor expansion order (1 or 2) |
+
+### Combined Configuration Example
+
+DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
+simultaneously:
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+SGLANG_CACHE_DIT_FN=2 \
+SGLANG_CACHE_DIT_BN=1 \
+SGLANG_CACHE_DIT_WARMUP=4 \
+SGLANG_CACHE_DIT_RDT=0.4 \
+SGLANG_CACHE_DIT_MC=4 \
+SGLANG_CACHE_DIT_TAYLORSEER=true \
+SGLANG_CACHE_DIT_TS_ORDER=2 \
+sglang generate --model-path black-forest-labs/FLUX.1-dev \
+    --prompt "A curious raccoon in a forest"
+```
+
+### SCM (Step Computation Masking)
+
+SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and
+which to use cached results.
+
+**SCM Presets**
+
+SCM is configured with presets:
+
+| Preset   | Compute Ratio | Speed    | Quality    |
+|----------|---------------|----------|------------|
+| `none`   | 100%          | Baseline | Best       |
+| `slow`   | ~75%          | ~1.3x    | High       |
+| `medium` | ~50%          | ~2x      | Good       |
+| `fast`   | ~35%          | ~3x      | Acceptable |
+| `ultra`  | ~25%          | ~4x      | Lower      |
+
+**Usage**
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+SGLANG_CACHE_DIT_SCM_PRESET=medium \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A futuristic cityscape at sunset"
+```
+
+**Custom SCM Bins**
+
+For fine-grained control over which steps to compute vs cache:
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
+SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A futuristic cityscape at sunset"
+```
+
+**SCM Policy**
+
+| Policy    | Env Variable                          | Description                                 |
+|-----------|---------------------------------------|---------------------------------------------|
+| `dynamic` | `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` | Adaptive caching based on content (default) |
+| `static`  | `SGLANG_CACHE_DIT_SCM_POLICY=static`  | Fixed caching pattern                       |
+
+## Environment Variables
+
+All Cache-DiT parameters can be configured via environment variables.
+See [Environment Variables](../../environment_variables.md) for the complete list.
+
+## Supported Models
+
+SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
+
+| Model Family | Example Models              |
+|--------------|-----------------------------|
+| Wan          | Wan2.1, Wan2.2              |
+| Flux         | FLUX.1-dev, FLUX.2-dev      |
+| Z-Image      | Z-Image-Turbo               |
+| Qwen         | Qwen-Image, Qwen-Image-Edit |
+| Hunyuan      | HunyuanVideo                |
+
+## Performance Tips
+
+1. **Start with defaults**: The default parameters work well for most models
+2. **Use TaylorSeer**: It typically improves both speed and quality
+3. **Tune R threshold**: Lower values = better quality, higher values = faster
+4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance
+5. **Warmup matters**: Higher warmup = more stable caching decisions
+
+## Limitations
+
+- **SGLang-native pipelines**: Distributed support (TP/SP) is not yet validated; Cache-DiT will be automatically
+  disabled when `world_size > 1`.
+- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective
+- **Model support**: Only models registered in Cache-DiT's BlockAdapterRegister are supported
+
+## Troubleshooting
+
+### SCM disabled for low step count
+
+For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
+acceleration still works.
+
+## References
+
+- [Cache-DiT](https://github.com/vipshop/cache-dit)
+- [SGLang Diffusion](../index.md)

third_party/sglang/docs/diffusion/performance/cache/index.md

@@ -0,0 +1,65 @@
+# Caching Acceleration
+
+SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels.
+
+## Overview
+
+SGLang supports two complementary caching approaches:
+
+| Strategy | Scope | Mechanism | Best For |
+|----------|-------|-----------|----------|
+| **Cache-DiT** | Block-level | Skip individual transformer blocks dynamically | Advanced, higher speedup |
+| **TeaCache** | Timestep-level | Skip entire denoising steps based on L1 similarity | Simple, built-in |
+
+## Cache-DiT
+
+[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
+advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**.
+
+See [cache_dit.md](cache_dit.md) for detailed configuration.
+
+### Quick Start
+
+```bash
+SGLANG_CACHE_DIT_ENABLED=true \
+sglang generate --model-path Qwen/Qwen-Image \
+    --prompt "A beautiful sunset over the mountains"
+```
+
+### Key Features
+
+- **DBCache**: Dynamic block-level caching based on residual differences
+- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
+- **SCM**: Step-level computation masking for additional speedup
+
+## TeaCache
+
+TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
+
+See [teacache.md](teacache.md) for detailed documentation.
+
+### Quick Overview
+
+- Tracks L1 distance between modulated inputs across timesteps
+- When accumulated distance is below threshold, reuses cached residual
+- Supports CFG with separate positive/negative caches
+
+### Supported Models
+
+- Wan (wan2.1, wan2.2)
+- Hunyuan (HunyuanVideo)
+- Z-Image
+
+For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
+
+```{toctree}
+:maxdepth: 1
+
+cache_dit
+teacache
+```
+
+## References
+
+- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
+- [TeaCache Paper](https://arxiv.org/abs/2411.14324)

third_party/sglang/docs/diffusion/performance/cache/teacache.md

@@ -0,0 +1,84 @@
+# TeaCache
+
+> **Note**: This is one of two caching strategies available in SGLang.
+> For an overview of all caching options, see [caching](../index.md).
+
+TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
+
+## Overview
+
+TeaCache works by:
+1. Tracking the L1 distance between modulated inputs across consecutive timesteps
+2. Accumulating the rescaled L1 distance over steps
+3. When accumulated distance is below a threshold, reusing the cached residual
+4. Supporting CFG (Classifier-Free Guidance) with separate positive/negative caches
+
+## How It Works
+
+### L1 Distance Tracking
+
+At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs:
+
+```
+rel_l1 = |current - previous|.mean() / |previous|.mean()
+```
+
+This distance is then rescaled using polynomial coefficients and accumulated:
+
+```
+accumulated += poly(coefficients)(rel_l1)
+```
+
+### Cache Decision
+
+- If `accumulated >= threshold`: Force computation, reset accumulator
+- If `accumulated < threshold`: Skip computation, use cached residual
+
+### CFG Support
+
+For models that support CFG cache separation (Wan, Hunyuan, Z-Image), TeaCache maintains separate caches for positive and negative branches:
+- `previous_modulated_input` / `previous_residual` for positive branch
+- `previous_modulated_input_negative` / `previous_residual_negative` for negative branch
+
+For models that don't support CFG separation (Flux, Qwen), TeaCache is automatically disabled when CFG is enabled.
+
+## Configuration
+
+TeaCache is configured via `TeaCacheParams` in the sampling parameters:
+
+```python
+from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
+
+params = TeaCacheParams(
+    teacache_thresh=0.1,           # Threshold for accumulated L1 distance
+    coefficients=[1.0, 0.0, 0.0],  # Polynomial coefficients for L1 rescaling
+)
+```
+
+### Parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `teacache_thresh` | float | Threshold for accumulated L1 distance. Lower = more caching, faster but potentially lower quality |
+| `coefficients` | list[float] | Polynomial coefficients for L1 rescaling. Model-specific tuning |
+
+### Model-Specific Configurations
+
+Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality.
+
+## Supported Models
+
+TeaCache is built into the following model families:
+
+| Model Family | CFG Cache Separation | Notes |
+|--------------|---------------------|-------|
+| Wan (wan2.1, wan2.2) | Yes | Full support |
+| Hunyuan (HunyuanVideo) | Yes | To be supported |
+| Z-Image | Yes | To be supported |
+| Flux | No | To be supported |
+| Qwen | No | To be supported |
+
+
+## References
+
+- [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324)

third_party/sglang/docs/diffusion/performance/index.md

@@ -0,0 +1,42 @@
+# Performance
+
+This section covers the main performance levers for SGLang Diffusion: attention backends, caching acceleration, and profiling.
+
+## Overview
+
+| Optimization | Type | Description |
+|--------------|------|-------------|
+| **Cache-DiT** | Caching | Block-level caching with DBCache, TaylorSeer, and SCM |
+| **TeaCache** | Caching | Timestep-level caching based on temporal similarity |
+| **Attention Backends** | Kernel | Optimized attention implementations (FlashAttention, SageAttention, etc.) |
+| **Profiling** | Diagnostics | PyTorch Profiler and Nsight Systems guidance |
+
+## Start Here
+
+- Use [Attention Backends](attention_backends.md) to choose the best backend for your model and hardware.
+- Use [Caching Acceleration](cache/index.md) to reduce denoising cost with Cache-DiT or TeaCache.
+- Use [Profiling](profiling.md) when you need to diagnose a bottleneck rather than guess.
+
+## Caching at a Glance
+
+- [Cache-DiT](cache/cache_dit.md) is block-level caching for diffusers pipelines and higher speedup-oriented tuning.
+- [TeaCache](cache/teacache.md) is timestep-level caching built into SGLang model families.
+
+```{toctree}
+:maxdepth: 1
+
+attention_backends
+cache/index
+profiling
+```
+
+## Current Baseline Snapshot
+
+For Ring SP benchmark details, see:
+
+- [Ring SP Performance](ring_sp_performance.md)
+
+## References
+
+- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
+- [TeaCache Paper](https://arxiv.org/abs/2411.14324)

third_party/sglang/docs/diffusion/performance/profiling.md

@@ -0,0 +1,136 @@
+# Profiling Multimodal Generation
+
+This guide covers profiling techniques for multimodal generation pipelines in SGLang.
+
+## PyTorch Profiler
+
+PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics.
+
+### Denoising Stage Profiling
+
+Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step):
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+  --seed 0 \
+  --profile
+```
+
+**Parameters:**
+- `--profile`: Enable profiling for the denoising stage
+- `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
+  - Smaller values reduce trace file size
+  - Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step
+
+### Full Pipeline Profiling
+
+Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.):
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+  --seed 0 \
+  --profile \
+  --profile-all-stages
+```
+
+**Parameters:**
+- `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
+
+### Output Location
+
+By default, trace files are saved in the ./logs/ directory.
+
+The exact output file path will be shown in the console output, for example:
+
+```bash
+[mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz
+```
+
+### View Traces
+
+Load and visualize trace files at:
+- https://ui.perfetto.dev/ (recommended)
+- chrome://tracing (Chrome only)
+
+For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
+
+
+### `--perf-dump-path` (Stage/Step Timing Dump)
+
+Besides profiler traces, you can also dump a lightweight JSON report that contains:
+- stage-level timing breakdown for the full pipeline
+- step-level timing breakdown for the denoising stage (per diffusion step)
+
+This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike).
+
+The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key.
+
+Example:
+
+```bash
+sglang generate \
+  --model-path <MODEL_PATH_OR_ID> \
+  --prompt "<PROMPT>" \
+  --perf-dump-path perf.json
+```
+
+## Nsight Systems
+
+Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns.
+
+### Installation
+
+See the [SGLang profiling guide](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/benchmark_and_profiling.md#profile-with-nsight) for installation instructions.
+
+### Basic Profiling
+
+Profile the entire pipeline execution:
+
+```bash
+nsys profile \
+  --trace-fork-before-exec=true \
+  --cuda-graph-trace=node \
+  --force-overwrite=true \
+  -o QwenImage \
+  sglang generate \
+    --model-path Qwen/Qwen-Image \
+    --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+    --seed 0
+```
+
+### Targeted Stage Profiling
+
+Use `--delay` and `--duration` to capture specific stages and reduce file size:
+
+```bash
+nsys profile \
+  --trace-fork-before-exec=true \
+  --cuda-graph-trace=node \
+  --force-overwrite=true \
+  --delay 10 \
+  --duration 30 \
+  -o QwenImage_denoising \
+  sglang generate \
+    --model-path Qwen/Qwen-Image \
+    --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+    --seed 0
+```
+
+**Parameters:**
+- `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
+- `--duration N`: Capture for N seconds (focus on specific stages)
+- `--force-overwrite`: Overwrite existing output files
+
+## Notes
+
+- **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems
+- **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline
+- **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads
+
+## FAQ
+
+- If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time.

third_party/sglang/docs/diffusion/performance/ring_sp_performance.md

@@ -0,0 +1,67 @@
+# Ring SP Benchmark: Wan2.2-TI2V-5B (u1r2 vs Baseline)
+
+This page reports Ring-SP performance for `Wan2.2-TI2V-5B-Diffusers` using:
+
+- Parallel config: `sp=2, ulysses=1, ring=2` (short: `u1r2`)
+- Baseline config: `sp=1, ulysses=1, ring=1` (short: `u1r1`)
+
+## Benchmark Setup
+
+- Model: `Wan2.2-TI2V-5B-Diffusers`
+- GPU: `48G RTX40 series * 2`
+
+## Online Serving
+
+### Ring SP (`u1r2`)
+
+```bash
+sglang serve \
+  --model-type diffusion \
+  --model-path /model/HuggingFace/Wan-AI/Wan2.2-TI2V-5B-Diffusers \
+  --num-gpus 2 --sp-degree 2 --ulysses-degree 1 --ring-degree 2 \
+  --port 8898
+```
+
+### Baseline (`u1r1`)
+
+```bash
+sglang serve \
+  --model-type diffusion \
+  --model-path /model/HuggingFace/Wan-AI/Wan2.2-TI2V-5B-Diffusers \
+  --num-gpus 1 --sp-degree 1 --ulysses-degree 1 --ring-degree 1 \
+  --port 8898
+```
+
+## Benchmarks
+
+### Benchmark Disclaimer
+
+These benchmarks are provided for reference under one specific setup and command configuration. Actual performance may vary with model settings, runtime environment, and request patterns.
+
+### Stage Time Breakdown
+
+| Stage / Metric | `u1r2` (s) | `u1r1` baseline (s) | Speedup |
+|---|---:|---:|---:|
+| InputValidation | 0.1060 | 0.1029 | 0.97x |
+| TextEncoding | 1.3965 | 2.2261 | 1.59x |
+| LatentPreparation | 0.0002 | 0.0002 | 1.00x |
+| TimestepPreparation | 0.0003 | 0.0004 | 1.33x |
+| Denoising | 52.6358 | 71.6785 | 1.36x |
+| Decoding | 7.6708 | 13.4314 | 1.75x |
+| **Total** | **63.74** | **90.63** | **1.42x** |
+
+### Memory Usage
+
+| Memory Metric | `u1r2` (GB) | `u1r1` baseline (GB) | Delta |
+|---|---:|---:|---:|
+| Peak GPU Memory | 20.07 | 27.40 | -7.33 |
+| Peak Allocated | 13.35 | 20.40 | -7.05 |
+| Memory Overhead | 6.72 | 7.00 | -0.28 |
+| Overhead Ratio | 33.5% | 25.6% | +7.9pp |
+
+## Summary
+
+- End-to-end latency improves from `90.63s` to `63.74s` (`1.42x`).
+- Main gains come from `Denoising` (`1.36x`) and `Decoding` (`1.75x`).
+- Absolute memory usage drops noticeably on Ring-SP (`Peak GPU Memory -7.33GB`, `Peak Allocated -7.05GB`).
+- Overhead ratio rises (`+7.9pp`), so future tuning can focus on reducing communication/runtime overhead while preserving the latency gain.

third_party/sglang/docs/diffusion/quantization.md

@@ -0,0 +1,239 @@
+# Quantization
+
+SGLang-Diffusion supports quantized transformer checkpoints. In most cases, keep
+the base model and the quantized transformer override separate.
+
+## Quick Reference
+
+Use these paths:
+
+- `--model-path`: the base or original model
+- `--transformer-path`: a quantized transformers-style transformer component directory that already contains its own `config.json`
+- `--transformer-weights-path`: quantized transformer weights provided as a single safetensors file, a sharded safetensors directory, a local path, or a Hugging Face repo ID
+
+Recommended example:
+
+```bash
+sglang generate \
+  --model-path black-forest-labs/FLUX.2-dev \
+  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
+  --prompt "a curious pikachu"
+```
+
+For quantized transformers-style transformer component folders:
+
+```bash
+sglang generate \
+  --model-path /path/to/base-model \
+  --transformer-path /path/to/quantized-transformer \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion"
+```
+
+NOTE: Some model-specific integrations also accept a quantized repo or local
+directory directly as `--model-path`, but that is a compatibility path. If a
+repo contains multiple candidate checkpoints, pass
+`--transformer-weights-path` explicitly.
+
+## Quant Families
+
+Here, `quant_family` means a checkpoint and loading family with shared CLI
+usage and loader behavior. It is not just the numeric precision or a kernel
+backend.
+
+| quant_family     | checkpoint form                                                                            | canonical CLI                                        | supported models                                             | extra dependency                      | platform / notes                                                                                                      |
+|------------------|--------------------------------------------------------------------------------------------|------------------------------------------------------|--------------------------------------------------------------|---------------------------------------|-----------------------------------------------------------------------------------------------------------------------|
+| `fp8`            | Quantized transformer component folder, or safetensors with `quantization_config` metadata | `--transformer-path` or `--transformer-weights-path` | ALL                                                          | None                                  | Component-folder and single-file flows are both supported                                                             |
+| `nvfp4-modelopt` | NVFP4 safetensors file, sharded directory, or repo providing transformer weights           | `--transformer-weights-path`                         | FLUX.2                                                       | `comfy-kitchen` optional on Blackwell | Blackwell can use a best-performance kit when available; otherwise SGLang falls back to the generic ModelOpt FP4 path |
+| `nunchaku-svdq`  | Pre-quantized Nunchaku transformer weights, usually named `svdq-{int4\|fp4}_r{rank}-...`   | `--transformer-weights-path`                         | Model-specific support such as Qwen-Image, FLUX, and Z-Image | `nunchaku`                            | SGLang can infer precision and rank from the filename and supports both `int4` and `nvfp4`                            |
+| `msmodelslim`    | Pre-quantized msmodelslim transformer weights                                              | `--model-path`                                       | Wan2.2 family                                                | None                                  | Currently only compatible with the Ascend NPU family and supports both `w8a8` and `w4a4`                              |
+
+## NVFP4
+
+### Usage Examples
+
+Recommended usage keeps the base model and quantized transformer override
+separate:
+
+```bash
+sglang generate \
+  --model-path black-forest-labs/FLUX.2-dev \
+  --transformer-weights-path black-forest-labs/FLUX.2-dev-NVFP4 \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+  --save-output
+```
+
+SGLang also supports passing the NVFP4 repo or local directory directly as
+`--model-path`:
+
+```bash
+sglang generate \
+  --model-path black-forest-labs/FLUX.2-dev-NVFP4 \
+  --prompt "A Logo With Bold Large Text: SGL Diffusion" \
+  --save-output
+```
+
+### Notes
+
+- `--transformer-weights-path` is still the canonical CLI for NVFP4
+  transformer checkpoints.
+- Direct `--model-path` loading is a compatibility path for FLUX.2 NVFP4-style
+  repos or local directories.
+- If `--transformer-weights-path` is provided explicitly, it takes precedence
+  over the compatibility `--model-path` flow.
+- For local directories, SGLang first looks for `*-mixed.safetensors`, then
+  falls back to loading from the directory.
+- On Blackwell, `comfy-kitchen` can provide the best-performance path when
+  available; otherwise SGLang falls back to the generic ModelOpt FP4 path.
+
+## Nunchaku (SVDQuant)
+
+### Install
+
+Install the runtime dependency first:
+
+```bash
+pip install nunchaku
+```
+
+For platform-specific installation methods and troubleshooting, see the
+[Nunchaku installation guide](https://nunchaku.tech/docs/nunchaku/installation/installation.html).
+
+### File Naming and Auto-Detection
+
+For Nunchaku checkpoints, `--model-path` should still point to the original
+base model, while `--transformer-weights-path` points to the quantized
+transformer weights.
+
+If the basename of `--transformer-weights-path` contains the pattern
+`svdq-(int4|fp4)_r{rank}`, SGLang will automatically:
+- enable SVDQuant
+- infer `--quantization-precision`
+- infer `--quantization-rank`
+
+Examples:
+
+| checkpoint name fragment | inferred precision | inferred rank | notes |
+|--------------------------|--------------------|---------------|-------|
+| `svdq-int4_r32`          | `int4`             | `32`          | Standard INT4 checkpoint |
+| `svdq-int4_r128`         | `int4`             | `128`         | Higher-quality INT4 checkpoint |
+| `svdq-fp4_r32`           | `nvfp4`            | `32`          | `fp4` in the filename maps to CLI value `nvfp4` |
+| `svdq-fp4_r128`          | `nvfp4`            | `128`         | Higher-quality NVFP4 checkpoint |
+
+Common filenames:
+
+| filename | precision | rank | typical use |
+|----------|-----------|------|-------------|
+| `svdq-int4_r32-qwen-image.safetensors` | `int4` | `32` | Balanced default |
+| `svdq-int4_r128-qwen-image.safetensors` | `int4` | `128` | Quality-focused |
+| `svdq-fp4_r32-qwen-image.safetensors` | `nvfp4` | `32` | RTX 50-series / NVFP4 path |
+| `svdq-fp4_r128-qwen-image.safetensors` | `nvfp4` | `128` | Quality-focused NVFP4 |
+| `svdq-int4_r32-qwen-image-lightningv1.0-4steps.safetensors` | `int4` | `32` | Lightning 4-step |
+| `svdq-int4_r128-qwen-image-lightningv1.1-8steps.safetensors` | `int4` | `128` | Lightning 8-step |
+
+If your checkpoint name does not follow this convention, pass
+`--enable-svdquant`, `--quantization-precision`, and `--quantization-rank`
+explicitly.
+
+### Usage Examples
+
+Recommended auto-detected flow:
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --transformer-weights-path /path/to/svdq-int4_r32-qwen-image.safetensors \
+  --prompt "a beautiful sunset" \
+  --save-output
+```
+
+Manual override when the filename does not encode the quant settings:
+
+```bash
+sglang generate \
+  --model-path Qwen/Qwen-Image \
+  --transformer-weights-path /path/to/custom_nunchaku_checkpoint.safetensors \
+  --enable-svdquant \
+  --quantization-precision int4 \
+  --quantization-rank 128 \
+  --prompt "a beautiful sunset" \
+  --save-output
+```
+
+### Notes
+
+- `--transformer-weights-path` is the canonical flag for Nunchaku checkpoints.
+  Older config names such as `quantized_model_path` are treated as
+  compatibility aliases.
+- Auto-detection only happens when the checkpoint basename matches
+  `svdq-(int4|fp4)_r{rank}`.
+- The CLI values are `int4` and `nvfp4`. In filenames, the NVFP4 variant is
+  written as `fp4`.
+- Lightning checkpoints usually expect matching `--num-inference-steps`, such
+  as `4` or `8`.
+- Current runtime validation only allows Nunchaku on NVIDIA CUDA Ampere (SM8x)
+  or SM12x GPUs. Hopper (SM90) is currently rejected.
+
+## [ModelSlim](https://gitcode.com/Ascend/msmodelslim)
+MindStudio-ModelSlim (msModelSlim) is a model offline quantization compression tool launched by MindStudio and optimized for Ascend hardware.
+
+- **Installation**
+
+    ```bash
+    # Clone repo and install msmodelslim:
+    git clone https://gitcode.com/Ascend/msmodelslim.git
+    cd msmodelslim
+    bash install.sh
+    ```
+
+- **Multimodal_sd quantization**
+
+    Download the original floating-point weights of the large model. Taking Wan2.2-T2V-A14B as an example, you can go to [Wan2.2-T2V-A14B](https://modelscope.cn/models/Wan-AI/Wan2.2-T2V-A14B) to obtain the original model weights. Then install other dependencies (related to the model, refer to the modelscope model card).
+    > Note: You can find pre-quantized validated models on [modelscope/Eco-Tech](https://modelscope.cn/models/Eco-Tech).
+
+  Run quantization using one-click quantization (recommended):
+
+  ```bash
+  msmodelslim quant \
+    --model_path /path/to/wan2_2_float_weights \
+    --save_path /path/to/wan2_2_quantized_weights \
+    --device npu \
+    --model_type Wan2_2 \
+    --quant_type w8a8 \
+    --trust_remote_code True
+  ```
+
+  For more detailed examples of quantization of models, as well as information about their support, see the [examples](https://gitcode.com/Ascend/msmodelslim/blob/master/example/multimodal_sd/README.md) section in ModelSLim repo.
+
+  > Note: SGLang does not support quantized embeddings, please disable this option when quantizing using msmodelslim.
+
+- **Auto-Detection and different formats**
+
+    For msmodelslim checkpoints, it's enough to specify only ```--model-path```, the detection of quantization occurs automatically for each layer using parsing of      `quant_model_description.json` config.
+
+    In the case of `Wan2.2` only `Diffusers` weights storage format are supported, whereas modelslim saves the quantized model in the original `Wan2.2` format,
+    for conversion in use `python/sglang/multimodal_gen/tools/wan_repack.py` script:
+
+    ```bash
+    python wan_repack.py \
+      --input-path {path_to_quantized_model} \
+      --output-path {path_to_converted_model}
+    ```
+
+    After that, please copy all files from original `Diffusers` checkpoint (instead of `transformer`/`tranfsormer_2` folders)
+
+- **Usage Example**
+
+    With auto-detected flow:
+
+    ```bash
+    sglang generate \
+      --model-path Eco-Tech/Wan2.2-T2V-A14B-Diffusers-w8a8 \
+      --prompt "a beautiful sunset" \
+      --save-output
+    ```
+
+- **Available Quantization Methods**:
+    - [x]  ```W4A4_DYNAMIC``` linear with online quantization of activations
+    - [x]  ```W8A8``` linear with offline quantization of activations
+    - [x]  ```W8A8_DYNAMIC``` linear with online quantization of activations
+    - [ ]  ```mxfp8``` linear in progress

third_party/sglang/docs/diffusion/reference.md

@@ -0,0 +1,11 @@
+# Reference
+
+Reference material for environment-based configuration and runtime behavior.
+
+- [Environment Variables](environment_variables.md): platform, caching, cloud storage, and debugging variables
+
+```{toctree}
+:maxdepth: 1
+
+environment_variables
+```

third_party/sglang/docs/diffusion/support_new_models.md

@@ -0,0 +1,388 @@
+# How to Support New Diffusion Models
+
+This document explains how to add support for new diffusion models in SGLang Diffusion.
+
+## Architecture Overview
+
+SGLang Diffusion is engineered for both performance and flexibility, built upon a pipeline architecture. This
+design allows developers to construct pipelines for various diffusion models while keeping the core generation
+loop standardized for optimization.
+
+At its core, the architecture revolves around two key concepts, as highlighted in our [blog post](https://lmsys.org/blog/2025-11-07-sglang-diffusion/#architecture):
+
+-   **`ComposedPipeline`**: This class orchestrates a series of `PipelineStage`s to define the complete generation process for a specific model. It acts as the main entry point for a model and manages the data flow between the different stages of the diffusion process.
+-   **`PipelineStage`**: Each stage is a modular component that encapsulates a function within the diffusion process. Examples include prompt encoding, the denoising loop, or VAE decoding.
+
+### Two Pipeline Styles
+
+SGLang Diffusion supports two pipeline composition styles. Both are valid; choose the one that best fits your model.
+
+#### Style A: Hybrid Monolithic Pipeline (Recommended Default)
+
+The recommended default for most new models. Uses a three-stage structure:
+
+```
+BeforeDenoisingStage (model-specific)  →  DenoisingStage (standard)  →  DecodingStage (standard)
+```
+
+| Stage | Ownership | Responsibility |
+|-------|-----------|----------------|
+| `{Model}BeforeDenoisingStage` | Model-specific | All pre-processing: input validation, text/image encoding, latent preparation, timestep computation |
+| `DenoisingStage` | Framework-standard | The denoising loop (DiT/UNet forward passes), shared across all models |
+| `DecodingStage` | Framework-standard | VAE decoding from latent space to pixel space, shared across all models |
+
+**Why recommended?** Modern diffusion models often have highly heterogeneous pre-processing requirements — different text encoders, different latent formats, different conditioning mechanisms. The Hybrid approach keeps pre-processing isolated per model, avoids fragile shared stages with excessive conditional logic, and lets developers port Diffusers reference code quickly.
+
+#### Style B: Modular Composition Style
+
+Uses the framework's fine-grained standard stages (`TextEncodingStage`, `LatentPreparationStage`, `TimestepPreparationStage`, etc.) to build the pipeline by composition. Convenience methods like `add_standard_t2i_stages()` and `add_standard_ti2i_stages()` make this very concise.
+
+This style is appropriate when:
+- **The new model's pre-processing can largely reuse existing stages** — e.g., a model that uses standard CLIP/T5 text encoding + standard latent preparation with minimal customization.
+- **A model-specific optimization needs to be extracted as a standalone stage** — e.g., a specialized encoding or conditioning step that benefits from being a separate stage for profiling, parallelism control, or reuse across multiple pipeline variants.
+
+#### How to Choose
+
+| Situation | Recommended Style |
+|-----------|-------------------|
+| Model has unique/complex pre-processing (VLM captioning, AR token generation, custom latent packing, etc.) | **Hybrid** — consolidate into a BeforeDenoisingStage |
+| Model fits neatly into standard text-to-image or text+image-to-image pattern | **Modular** — use `add_standard_t2i_stages()` / `add_standard_ti2i_stages()` |
+| Porting a Diffusers pipeline with many custom steps | **Hybrid** — copy the `__call__` logic into a single stage |
+| Adding a variant of an existing model that shares most logic | **Modular** — reuse existing stages, customize via PipelineConfig callbacks |
+| A specific pre-processing step needs special parallelism or profiling isolation | **Modular** — extract that step as a dedicated stage |
+
+## Key Components for Implementation
+
+To add support for a new diffusion model, you will need to define or configure the following components:
+
+1.  **`PipelineConfig`**: A dataclass holding static configurations for your model pipeline — precision settings, model architecture parameters, and callback methods used by the standard `DenoisingStage` and `DecodingStage`. Each model has its own subclass.
+
+2.  **`SamplingParams`**: A dataclass defining runtime generation parameters — `prompt`, `negative_prompt`, `guidance_scale`, `num_inference_steps`, `seed`, `height`, `width`, etc.
+
+3.  **Pre-processing stage(s)**: Either a single model-specific `{Model}BeforeDenoisingStage` (Hybrid style) or a combination of standard stages (Modular style). See [Two Pipeline Styles](#two-pipeline-styles) above.
+
+4.  **`ComposedPipeline`**: A class that wires together your pre-processing stage(s) with the standard `DenoisingStage` and `DecodingStage`. See base definitions:
+    - [`ComposedPipelineBase`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py)
+    - [`PipelineStage`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines_core/stages/base.py)
+    - [Central registry](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/registry.py)
+
+5.  **Modules (model components)**: Each pipeline references modules loaded from the model repository (e.g., Diffusers `model_index.json`):
+    - `text_encoder`: Encodes text prompts into embeddings.
+    - `tokenizer`: Tokenizes raw text input for the text encoder(s).
+    - `processor`: Preprocesses images and extracts features; often used in image-to-image tasks.
+    - `image_encoder`: Specialized image feature extractor.
+    - `dit/transformer`: The core denoising network (DiT/UNet architecture) operating in latent space.
+    - `scheduler`: Controls the timestep schedule and denoising dynamics.
+    - `vae`: Variational Autoencoder for encoding/decoding between pixel space and latent space.
+
+## Pipeline Stages Reference
+
+### Core Stages (used by all pipelines)
+
+| Stage Class                      | Description                                                                                             |
+| -------------------------------- | ------------------------------------------------------------------------------------------------------- |
+| `DenoisingStage`                 | Executes the main denoising loop, iteratively applying the model (DiT/UNet) to refine the latents.      |
+| `DecodingStage`                  | Decodes the final latent tensor back into pixel space using the VAE.                                    |
+| `DmdDenoisingStage`              | A specialized denoising stage for DMD model architectures.                                              |
+| `CausalDMDDenoisingStage`        | A specialized causal denoising stage for specific video models.                                         |
+
+### Pre-processing Stages (for Modular Composition Style)
+
+The following fine-grained stages can be composed to build the pre-processing portion of a pipeline. They are best suited for models whose pre-processing largely fits the standard patterns. If your model requires significant customization, consider the Hybrid style with a single `BeforeDenoisingStage` instead.
+
+| Stage Class                      | Description                                                                                             |
+| -------------------------------- | ------------------------------------------------------------------------------------------------------- |
+| `InputValidationStage`           | Validates user-provided `SamplingParams`.                                                               |
+| `TextEncodingStage`              | Encodes text prompts into embeddings using one or more text encoders.                                   |
+| `ImageEncodingStage`             | Encodes input images into embeddings, often used in image-to-image tasks.                               |
+| `ImageVAEEncodingStage`          | Encodes an input image into latent space using the VAE.                                                 |
+| `TimestepPreparationStage`       | Prepares the scheduler's timesteps for the diffusion process.                                           |
+| `LatentPreparationStage`         | Creates the initial noisy latent tensor that will be denoised.                                          |
+
+## Implementation Guide
+
+### Step 1: Obtain and Study the Reference Implementation
+
+Before writing any code, obtain the model's original implementation or Diffusers pipeline code:
+- The model's Diffusers pipeline source (e.g., the `pipeline_*.py` file from the `diffusers` library or HuggingFace repo)
+- Or the model's official reference implementation (e.g., from the model author's GitHub repo)
+- Or the HuggingFace model ID to look up `model_index.json` and the associated pipeline class
+
+Once you have the reference code, study it thoroughly:
+
+1. Find the model's `model_index.json` to identify required modules.
+2. Read the Diffusers pipeline's `__call__` method to understand:
+   - How text prompts are encoded
+   - How latents are prepared (shape, dtype, scaling)
+   - How timesteps/sigmas are computed
+   - What conditioning kwargs the DiT expects
+   - How the denoising loop works
+   - How VAE decoding is done
+
+### Step 2: Evaluate Reuse of Existing Pipelines and Stages
+
+Before creating any new files, check whether an existing pipeline or stage can be reused or extended. Only create new pipelines/stages when the existing ones would need substantial structural changes or when no architecturally similar implementation exists.
+
+- **Compare against existing pipelines** (Flux, Wan, Qwen-Image, GLM-Image, HunyuanVideo, LTX, etc.). If the new model shares most of its structure with an existing one, prefer adding a new config variant or reusing existing stages.
+- **Check existing stages** in `runtime/pipelines_core/stages/` and `stages/model_specific_stages/`.
+- **Check existing model components** — many models share VAEs (e.g., `AutoencoderKL`), text encoders (CLIP, T5), and schedulers. Reuse these directly.
+
+### Step 3: Implement Model Components
+
+Adapt the model's core components:
+
+- **DiT/Transformer**: Implement in [`runtime/models/dits/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/dits/)
+- **Encoders**: Implement in [`runtime/models/encoders/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/encoders/)
+- **VAEs**: Implement in [`runtime/models/vaes/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/vaes/)
+- **Schedulers**: Implement in [`runtime/models/schedulers/`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/runtime/models/schedulers/) if needed
+
+Use SGLang's fused kernels where possible (see `LayerNormScaleShift`, `RMSNormScaleShift`, `apply_qk_norm`, etc.).
+
+**Tensor Parallel (TP) and Sequence Parallel (SP)**: For multi-GPU deployment, it is recommended to add TP/SP support to the DiT model. This can be done incrementally after the single-GPU implementation is verified. Reference implementations:
+- **Wan model** (`runtime/models/dits/wanvideo.py`) — Full TP + SP: `ColumnParallelLinear`/`RowParallelLinear` for attention, sequence dimension sharding via `get_sp_world_size()`
+- **Qwen-Image model** (`runtime/models/dits/qwen_image.py`) — SP via `USPAttention` (Ulysses + Ring Attention)
+
+### Step 4: Create Configs
+
+- **DiT Config**: `configs/models/dits/{model_name}.py`
+- **VAE Config**: `configs/models/vaes/{model_name}.py`
+- **SamplingParams**: `configs/sample/{model_name}.py`
+
+### Step 5: Create PipelineConfig
+
+The `PipelineConfig` provides callbacks that the standard `DenoisingStage` and `DecodingStage` use:
+
+```python
+# python/sglang/multimodal_gen/configs/pipeline_configs/my_model.py
+
+@dataclass
+class MyModelPipelineConfig(ImagePipelineConfig):
+    task_type: ModelTaskType = ModelTaskType.T2I
+    vae_precision: str = "bf16"
+    should_use_guidance: bool = True
+    dit_config: DiTConfig = field(default_factory=MyModelDitConfig)
+    vae_config: VAEConfig = field(default_factory=MyModelVAEConfig)
+
+    def get_freqs_cis(self, batch, device, rotary_emb, dtype):
+        """Prepare rotary position embeddings for the DiT."""
+        ...
+
+    def prepare_pos_cond_kwargs(self, batch, latent_model_input, t, **kwargs):
+        """Build positive conditioning kwargs for each denoising step."""
+        return {
+            "hidden_states": latent_model_input,
+            "encoder_hidden_states": batch.prompt_embeds[0],
+            "timestep": t,
+        }
+
+    def prepare_neg_cond_kwargs(self, batch, latent_model_input, t, **kwargs):
+        """Build negative conditioning kwargs for CFG."""
+        return {
+            "hidden_states": latent_model_input,
+            "encoder_hidden_states": batch.negative_prompt_embeds[0],
+            "timestep": t,
+        }
+
+    def get_decode_scale_and_shift(self):
+        """Return (scale, shift) for latent denormalization before VAE decode."""
+        ...
+```
+
+### Step 6: Implement Pre-processing
+
+Choose based on your model's needs (see [How to Choose](#how-to-choose)):
+
+#### Option A: BeforeDenoisingStage (Hybrid Style)
+
+Create a single stage that handles all pre-processing. Best when the model has custom/complex pre-processing logic.
+
+```python
+# python/sglang/multimodal_gen/runtime/pipelines_core/stages/model_specific_stages/my_model.py
+
+class MyModelBeforeDenoisingStage(PipelineStage):
+    """Monolithic pre-processing stage for MyModel.
+
+    Consolidates: input validation, text/image encoding, latent
+    preparation, and timestep computation.
+    """
+
+    def __init__(self, vae, text_encoder, tokenizer, transformer, scheduler):
+        super().__init__()
+        self.vae = vae
+        self.text_encoder = text_encoder
+        self.tokenizer = tokenizer
+        self.transformer = transformer
+        self.scheduler = scheduler
+
+    @torch.no_grad()
+    def forward(self, batch: Req, server_args: ServerArgs) -> Req:
+        device = get_local_torch_device()
+
+        # 1. Encode prompt (model-specific logic)
+        prompt_embeds, negative_prompt_embeds = self._encode_prompt(...)
+
+        # 2. Prepare latents
+        latents = self._prepare_latents(...)
+
+        # 3. Prepare timesteps
+        timesteps, sigmas = self._prepare_timesteps(...)
+
+        # 4. Populate batch for DenoisingStage
+        batch.prompt_embeds = [prompt_embeds]
+        batch.negative_prompt_embeds = [negative_prompt_embeds]
+        batch.latents = latents
+        batch.timesteps = timesteps
+        batch.num_inference_steps = len(timesteps)
+        batch.sigmas = sigmas.tolist()
+        batch.generator = generator
+        batch.raw_latent_shape = latents.shape
+        return batch
+```
+
+#### Option B: Standard Stages (Modular Style)
+
+Skip creating a custom stage entirely — configure via `PipelineConfig` callbacks and use framework helpers. Best when the model fits standard patterns.
+
+(This option has no separate stage file; the pipeline class in Step 7 calls `add_standard_t2i_stages()` directly.)
+
+**Key batch fields that `DenoisingStage` expects** (regardless of which option you choose):
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `batch.latents` | `torch.Tensor` | Initial noisy latent tensor |
+| `batch.timesteps` | `torch.Tensor` | Timestep schedule |
+| `batch.num_inference_steps` | `int` | Number of denoising steps |
+| `batch.sigmas` | `list[float]` | Sigma schedule (must be a Python list, not numpy) |
+| `batch.prompt_embeds` | `list[torch.Tensor]` | Positive prompt embeddings (wrapped in a list) |
+| `batch.negative_prompt_embeds` | `list[torch.Tensor]` | Negative prompt embeddings (wrapped in a list) |
+| `batch.generator` | `torch.Generator` | RNG generator for reproducibility |
+| `batch.raw_latent_shape` | `tuple` | Original latent shape before any packing |
+
+### Step 7: Define the Pipeline Class
+
+#### Hybrid Style
+
+```python
+# python/sglang/multimodal_gen/runtime/pipelines/my_model.py
+
+class MyModelPipeline(LoRAPipeline, ComposedPipelineBase):
+    pipeline_name = "MyModelPipeline"  # Must match model_index.json _class_name
+
+    _required_config_modules = [
+        "text_encoder", "tokenizer", "vae", "transformer", "scheduler",
+    ]
+
+    def create_pipeline_stages(self, server_args: ServerArgs):
+        # 1. Monolithic pre-processing (model-specific)
+        self.add_stage(
+            MyModelBeforeDenoisingStage(
+                vae=self.get_module("vae"),
+                text_encoder=self.get_module("text_encoder"),
+                tokenizer=self.get_module("tokenizer"),
+                transformer=self.get_module("transformer"),
+                scheduler=self.get_module("scheduler"),
+            ),
+        )
+
+        # 2. Standard denoising loop (framework-provided)
+        self.add_stage(
+            DenoisingStage(
+                transformer=self.get_module("transformer"),
+                scheduler=self.get_module("scheduler"),
+            ),
+        )
+
+        # 3. Standard VAE decoding (framework-provided)
+        self.add_standard_decoding_stage()
+
+
+EntryClass = [MyModelPipeline]
+```
+
+#### Modular Style
+
+```python
+# python/sglang/multimodal_gen/runtime/pipelines/my_model.py
+
+class MyModelPipeline(LoRAPipeline, ComposedPipelineBase):
+    pipeline_name = "MyModelPipeline"
+
+    _required_config_modules = [
+        "text_encoder", "tokenizer", "vae", "transformer", "scheduler",
+    ]
+
+    def create_pipeline_stages(self, server_args: ServerArgs):
+        # All pre-processing + denoising + decoding in one call
+        self.add_standard_t2i_stages(
+            prepare_extra_timestep_kwargs=[prepare_mu],  # model-specific hooks
+        )
+
+
+EntryClass = [MyModelPipeline]
+```
+
+### Step 8: Register the Model
+
+Register your configs in [`registry.py`](https://github.com/sgl-project/sglang/blob/main/python/sglang/multimodal_gen/registry.py):
+
+```python
+register_configs(
+    model_family="my_model",
+    sampling_param_cls=MyModelSamplingParams,
+    pipeline_config_cls=MyModelPipelineConfig,
+    hf_model_paths=["org/my-model-name"],
+)
+```
+
+The `EntryClass` in your pipeline file is automatically discovered by the registry — no additional registration needed for the pipeline class itself.
+
+### Step 9: Verify Output Quality
+
+After implementation, verify that the generated output is not noise. A noisy or garbled output is the most common sign of an incorrect implementation. Common causes include:
+
+- Incorrect latent scale/shift factors
+- Wrong timestep/sigma schedule (order, dtype, or value range)
+- Mismatched conditioning kwargs
+- Rotary embedding style mismatch (`is_neox_style`)
+
+Debug by comparing intermediate tensor values against the Diffusers reference pipeline with the same seed.
+
+## Reference Implementations
+
+### Hybrid Style
+
+| Model | Pipeline | BeforeDenoisingStage | PipelineConfig |
+|-------|----------|---------------------|----------------|
+| GLM-Image | `runtime/pipelines/glm_image.py` | `stages/model_specific_stages/glm_image.py` | `configs/pipeline_configs/glm_image.py` |
+| Qwen-Image-Layered | `runtime/pipelines/qwen_image.py` | `stages/model_specific_stages/qwen_image_layered.py` | `configs/pipeline_configs/qwen_image.py` |
+
+### Modular Style
+
+| Model | Pipeline | Notes |
+|-------|----------|-------|
+| Qwen-Image (T2I) | `runtime/pipelines/qwen_image.py` | Uses `add_standard_t2i_stages()` |
+| Qwen-Image-Edit | `runtime/pipelines/qwen_image.py` | Uses `add_standard_ti2i_stages()` |
+| Flux | `runtime/pipelines/flux.py` | Uses `add_standard_t2i_stages()` with custom `prepare_mu` |
+| Wan | `runtime/pipelines/wan_pipeline.py` | Uses `add_standard_ti2v_stages()` |
+
+## Checklist
+
+Before submitting your implementation, verify:
+
+**Common (both styles):**
+- [ ] **Pipeline file** at `runtime/pipelines/{model_name}.py` with `EntryClass`
+- [ ] **PipelineConfig** at `configs/pipeline_configs/{model_name}.py`
+- [ ] **SamplingParams** at `configs/sample/{model_name}.py`
+- [ ] **DiT model** at `runtime/models/dits/{model_name}.py`
+- [ ] **Model configs** (DiT, VAE) at `configs/models/dits/` and `configs/models/vaes/`
+- [ ] **Registry entry** in `registry.py` via `register_configs()`
+- [ ] `pipeline_name` matches Diffusers `model_index.json` `_class_name`
+- [ ] `_required_config_modules` lists all modules from `model_index.json`
+- [ ] `PipelineConfig` callbacks (`prepare_pos_cond_kwargs`, etc.) match the DiT's `forward()` signature
+- [ ] Uses framework-standard `DenoisingStage` and `DecodingStage` (not custom denoising loops)
+- [ ] **TP/SP support** considered for DiT model (recommended; reference `wanvideo.py` for TP+SP, `qwen_image.py` for USPAttention)
+- [ ] **Output quality verified** — generated images/videos are not noise; compared against Diffusers reference output
+
+**Hybrid style only:**
+- [ ] **BeforeDenoisingStage** at `stages/model_specific_stages/{model_name}.py`
+- [ ] `BeforeDenoisingStage.forward()` populates all batch fields required by `DenoisingStage`

third_party/sglang/docs/diffusion/usage.md

@@ -0,0 +1,17 @@
+# Usage
+
+Use this section for day-to-day inference workflows with SGLang Diffusion.
+
+- [CLI](api/cli.md): run one-off jobs with `sglang generate` or start a server with `sglang serve`
+- [OpenAI-Compatible API](api/openai_api.md): request format, endpoints, and SDK examples
+- [Post-Processing](api/post_processing.md): frame interpolation and upscaling
+- [Quantization](quantization.md): quantized transformer checkpoints and supported quantization families
+
+```{toctree}
+:maxdepth: 1
+
+api/cli
+api/openai_api
+api/post_processing
+quantization
+```