gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: vendor sglang v0.5.10 snapshot

Browse Source
This commit is contained in:
Gahow Wang
2026-04-24 12:29:36 +00:00
parent 78f0d15221
commit bded08301f
4308 changed files with 1200894 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

133
third_party/sglang/docs/diffusion/performance/attention_backends.md vendored Normal file
View File

@@ -0,0 +1,133 @@
# Attention Backends
This document describes the attention backends available in sglang diffusion (`sglang.multimodal_gen`) and how to select them.
## Overview
Attention backends are defined by `AttentionBackendEnum` (`sglang.multimodal_gen.runtime.platforms.interface.AttentionBackendEnum`) and selected via the CLI flag `--attention-backend`.
Backend selection is performed by the shared attention layers (e.g. `LocalAttention` / `USPAttention` / `UlyssesAttention` in `sglang.multimodal_gen.runtime.layers.attention.layer`) and therefore applies to any model component using these layers (e.g. diffusion transformer / DiT and encoders).
When using the diffusers backend, `--attention-backend` is passed through to diffusers'
`set_attention_backend` (e.g., `flash`, `_flash_3_hub`, `sage`, `xformers`, `native`).
- **CUDA**: prefers FlashAttention (FA3/FA4) when supported; otherwise falls back to PyTorch SDPA.
- **ROCm**: uses FlashAttention when available; otherwise falls back to PyTorch SDPA.
- **MPS**: always uses PyTorch SDPA.
- **NPU**: for ring attention uses FA otherwise uses PyTorch SDPA.
## Backend options
For SGLang-native pipelines, the CLI accepts the lowercase names of `AttentionBackendEnum`. The table below lists the backends implemented by the built-in platforms. `fa3`/`fa4` are accepted as aliases for `fa`.
| CLI value | Enum value | Notes |
|---|---|---|
| `fa` / `fa3` / `fa4` | `FA` | FlashAttention. `fa3/fa4` are normalized to `fa` during argument parsing (`ServerArgs.__post_init__`). |
| `torch_sdpa` | `TORCH_SDPA` | PyTorch `scaled_dot_product_attention`. |
| `sliding_tile_attn` | `SLIDING_TILE_ATTN` | Sliding Tile Attention (STA). Requires `st_attn`. Configure via `--attention-backend-config`. |
| `sage_attn` | `SAGE_ATTN` | Requires `sageattention`. Upstream SageAttention CUDA extensions target SM80/SM86/SM89/SM90/SM120 (compute capability 8.0/8.6/8.9/9.0/12.0); see upstream `setup.py`: https://github.com/thu-ml/SageAttention/blob/main/setup.py. |
| `sage_attn_3` | `SAGE_ATTN_3` | Requires SageAttention3 installed per upstream instructions. |
| `video_sparse_attn` | `VIDEO_SPARSE_ATTN` | Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
| `vmoba_attn` | `VMOBA_ATTN` | Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
| `aiter` | `AITER` | Requires `aiter`. |
| `aiter_sage` | `AITER_SAGE` | Requires `aiter`. |
| `sparse_video_gen_2_attn` | `SPARSE_VIDEO_GEN_2_ATTN` | Requires `svg`. See installation instructions at https://github.com/svg-project/Sparse-VideoGen. |
## Selection priority
The selection order in `runtime/layers/attention/selector.py` is:
1. `global_force_attn_backend(...)` / `global_force_attn_backend_context_manager(...)`
2. CLI `--attention-backend` (`ServerArgs.attention_backend`)
3. Auto selection (platform capability, dtype, and installed packages)
## Configuration
Some backends require additional configuration. You can pass these parameters via `--attention-backend-config`. This argument accepts:
- A path to a JSON or YAML configuration file.
- A JSON string (e.g., `'{"sparsity": 0.5}'`).
- Key-value pairs (e.g., `"sparsity=0.5,enable_x=true"`).
### Supported Configuration Parameters
**Sliding Tile Attention (`sliding_tile_attn`)**
| Parameter | Type | Description | Default |
| :--- | :--- | :--- | :--- |
| `mask_strategy_file_path` | `str` | **Required.** Path to the mask strategy JSON file. | - |
| `sta_mode` | `str` | Mode of STA. | `STA_inference` |
| `skip_time_steps` | `int` | Number of steps to use full attention before switching to sparse attention. | `15` |
**Video Sparse Attention (`video_sparse_attn`)**
| Parameter | Type | Description | Default |
| :--- | :--- | :--- | :--- |
| `sparsity` | `float` | Validation sparsity (0.0 - 1.0). | `0.0` |
**V-MoBA (`vmoba_attn`)**
| Parameter | Type | Description | Default |
| :--- | :--- | :--- | :--- |
| `temporal_chunk_size` | `int` | Chunk size for temporal dimension. | - |
| `temporal_topk` | `int` | Top-K tokens to select in temporal dimension. | - |
| `spatial_chunk_size` | `list[int]` | Chunk size for spatial dimension (H, W). | - |
| `spatial_topk` | `int` | Top-K tokens to select in spatial dimension. | - |
| `st_chunk_size` | `list[int]` | Chunk size for spatiotemporal dimension (T, H, W). | - |
| `st_topk` | `int` | Top-K tokens to select in spatiotemporal dimension. | - |
| `moba_select_mode` | `str` | Selection mode (e.g., `threshold`). | `threshold` |
| `moba_threshold` | `float` | Threshold value for selection. | `0.25` |
| `moba_threshold_type` | `str` | Type of thresholding (e.g., `query_head`). | `query_head` |
| `first_full_step` | `int` | Number of initial steps to use full attention. | `12` |
| `first_full_layer` | `int` | Number of initial layers to use full attention. | `0` |
| `temporal_layer` | `int` | Number of temporal layers. | `1` |
| `spatial_layer` | `int` | Number of spatial layers. | `1` |
| `st_layer` | `int` | Number of spatiotemporal layers. | `1` |
## Platform support matrix
| Backend | CUDA | ROCm | MPS | NPU | Notes |
|---|---:|---:|---:|---:|---|
| `fa` | ✅ | ✅ | ❌ | ✅ | CUDA requires SM80+ and fp16/bf16. FlashAttention is only used when the required runtime is installed; otherwise it falls back to `torch_sdpa`. No extra installations are required for NPU |
| `torch_sdpa` | ✅ | ✅ | ✅ | ✅ | Most compatible option across platforms. |
| `sliding_tile_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `st_attn`. Configure via `--attention-backend-config`. |
| `sage_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). |
| `sage_attn_3` | ✅ | ❌ | ❌ | ❌ | CUDA-only (optional dependency). |
| `video_sparse_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `vsa`. Configure `sparsity` via `--attention-backend-config`. |
| `vmoba_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `kernel.attn.vmoba_attn.vmoba`. Configure via `--attention-backend-config`. |
| `aiter` | ❌ | ✅ | ❌ | ❌ | Requires `aiter`. |
| `aiter_sage` | ❌ | ✅ | ❌ | ❌ | Requires `aiter`. |
| `sparse_video_gen_2_attn` | ✅ | ❌ | ❌ | ❌ | CUDA-only. Requires `svg`. |
## Usage
### Select a backend via CLI
```bash
sglang generate \
--model-path <MODEL_PATH_OR_ID> \
--prompt "..." \
--attention-backend fa
```
```bash
sglang generate \
--model-path <MODEL_PATH_OR_ID> \
--prompt "..." \
--attention-backend torch_sdpa
```
### Using Sliding Tile Attention (STA)
```bash
# Pass the mask strategy file path via config
sglang generate \
--model-path <MODEL_PATH_OR_ID> \
--prompt "..." \
--attention-backend sliding_tile_attn \
--attention-backend-config "mask_strategy_file_path=/abs/path/to/mask_strategy.json"
```
### Notes for ROCm / MPS
- ROCm: use `--attention-backend torch_sdpa` or `fa` depending on what is available in your environment.
- MPS: the platform implementation always uses `torch_sdpa`.

418
third_party/sglang/docs/diffusion/performance/cache/cache_dit.md vendored Normal file
View File

@@ -0,0 +1,418 @@
# Cache-DiT
SGLang integrates [Cache-DiT](https://github.com/vipshop/cache-dit), a caching acceleration engine for Diffusion Transformers (DiT), to achieve up to **1.69x inference speedup** with minimal quality loss.
## Overview
**Cache-DiT** uses intelligent caching strategies to skip redundant computation in the denoising loop:
- **DBCache (Dual Block Cache)**: Dynamically decides when to cache transformer blocks based on residual differences
- **TaylorSeer**: Uses Taylor expansion for calibration to optimize caching decisions
- **SCM (Step Computation Masking)**: Step-level caching control for additional speedup
## Basic Usage
Enable Cache-DiT by exporting the environment variable and using `sglang generate` or `sglang serve` :
```bash
SGLANG_CACHE_DIT_ENABLED=true \
sglang generate --model-path Qwen/Qwen-Image \
--prompt "A beautiful sunset over the mountains"
```
## Diffusers Backend
Cache-DiT supports loading acceleration configs from a custom YAML file. For
diffusers pipelines (`diffusers` backend), pass the YAML/JSON path via `--cache-dit-config`. This
flow requires cache-dit >= 1.2.0 (`cache_dit.load_configs`).
### Single GPU inference
Define a `cache.yaml` file that contains:
- DBCache + TaylorSeer
```yaml
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0
residual_diff_threshold: 0.12
enable_taylorseer: true
taylorseer_order: 1
```
Then apply the config with:
```bash
sglang generate \
--backend diffusers \
--model-path Qwen/Qwen-Image \
--cache-dit-config cache.yaml \
--prompt "A beautiful sunset over the mountains"
```
- DBCache + TaylorSeer + SCM (Step Computation Mask)
```yaml
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0
residual_diff_threshold: 0.12
enable_taylorseer: true
taylorseer_order: 1
# Must set the num_inference_steps for SCM. The SCM will automatically
# generate the steps computation mask based on the num_inference_steps.
# Reference: https://cache-dit.readthedocs.io/en/latest/user_guide/CACHE_API/#scm-steps-computation-masking
num_inference_steps: 28
steps_computation_mask: fast
```
- DBCache + TaylorSeer + SCM (Step Computation Mask) + Cache CFG
```yaml
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0
residual_diff_threshold: 0.12
enable_taylorseer: true
taylorseer_order: 1
num_inference_steps: 28
steps_computation_mask: fast
enable_sperate_cfg: true # e.g, Qwen-Image, Wan, Chroma, Ovis-Image, etc.
```
### Distributed inference
- 1D Parallelism
Define a parallelism only config yaml `parallel.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: auto
attention_backend: native
```
Then, apply the distributed inference acceleration config from yaml. `ulysses_size: auto` means that cache-dit will auto detect the `world_size` as the ulysses_size. Otherwise, you should manually set it as specific int number, e.g, 4.
Then apply the distributed config with: (Note: please add `--num-gpus N` to specify the number of gpus for distributed inference)
```bash
sglang generate \
--backend diffusers \
--num-gpus 4 \
--model-path Qwen/Qwen-Image \
--cache-dit-config parallel.yaml \
--prompt "A futuristic cityscape at sunset"
```
- 2D Parallelism
You can also define a 2D parallelism config yaml `parallel_2d.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: auto
tp_size: 2
attention_backend: native
```
Then, apply the 2D parallelism config from yaml. Here `tp_size: 2` means using tensor parallelism with size 2. The `ulysses_size: auto` means that cache-dit will auto detect the `world_size // tp_size` as the ulysses_size.
- 3D Parallelism
You can also define a 3D parallelism config yaml `parallel_3d.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: 2
ring_size: 2
tp_size: 2
attention_backend: native
```
Then, apply the 3D parallelism config from yaml. Here `ulysses_size: 2`, `ring_size: 2`, `tp_size: 2` means using ulysses parallelism with size 2, ring parallelism with size 2 and tensor parallelism with size 2.
- Ulysses Anything Attention
To enable Ulysses Anything Attention, you can define a parallelism config yaml `parallel_uaa.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: auto
attention_backend: native
ulysses_anything: true
```
- Ulysses FP8 Communication
For device that don't have NVLink support, you can enable Ulysses FP8 Communication to further reduce the communication overhead. You can define a parallelism config yaml `parallel_fp8.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: auto
attention_backend: native
ulysses_float8: true
```
- Async Ulysses CP
You can also enable async ulysses CP to overlap the communication and computation. Define a parallelism config yaml `parallel_async.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: auto
attention_backend: native
ulysses_async: true # Now, only support for FLUX.1, Qwen-Image, Ovis-Image and Z-Image.
```
Then, apply the config from yaml. Here `ulysses_async: true` means enabling async ulysses CP.
- TE-P and VAE-P
You can also specify the extra parallel modules in the yaml config. For example, define a parallelism config yaml `parallel_extra.yaml` file that contains:
```yaml
parallelism_config:
ulysses_size: auto
attention_backend: native
extra_parallel_modules: ["text_encoder", "vae"]
```
### Hybrid Cache and Parallelism
Define a hybrid cache and parallel acceleration config yaml `hybrid.yaml` file that contains:
```yaml
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0
residual_diff_threshold: 0.12
enable_taylorseer: true
taylorseer_order: 1
parallelism_config:
ulysses_size: auto
attention_backend: native
extra_parallel_modules: ["text_encoder", "vae"]
```
Then, apply the hybrid cache and parallel acceleration config from yaml.
```bash
sglang generate \
--backend diffusers \
--num-gpus 4 \
--model-path Qwen/Qwen-Image \
--cache-dit-config hybrid.yaml \
--prompt "A beautiful sunset over the mountains"
```
### Attention Backend
In some cases, users may want to only specify the attention backend without any other optimization configs. In this case, you can define a yaml file `attention.yaml` that only contains:
```yaml
attention_backend: "flash" # '_flash_3' for Hopper
```
### Quantization
You can also specify the quantization config in the yaml file, required `torchao>=0.16.0`. For example, define a yaml file `quantize.yaml` that contains:
```yaml
quantize_config: # quantization configuration for transformer modules
# float8 (DQ), float8_weight_only, float8_blockwise, int8 (DQ), int8_weight_only, etc.
quant_type: "float8"
# layers to exclude from quantization (transformer). layers that contains any of the
# keywords in the exclude_layers list will be excluded from quantization. This is useful
# for some sensitive layers that are not robust to quantization, e.g., embedding layers.
exclude_layers:
- "embedder"
- "embed"
verbose: false # whether to print verbose logs during quantization
```
Then, apply the quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization. For example:
```bash
sglang generate \
--backend diffusers \
--model-path Qwen/Qwen-Image \
--warmup \
--cache-dit-config quantize.yaml \
--enable-torch-compile \
--dit-cpu-offload false \
--text-encoder-cpu-offload false \
--prompt "A beautiful sunset over the mountains"
```
### Combined Configs: Cache + Parallelism + Quantization
You can also combine all the above configs together in a single yaml file `combined.yaml` that contains:
```yaml
cache_config:
max_warmup_steps: 8
warmup_interval: 2
max_cached_steps: -1
max_continuous_cached_steps: 2
Fn_compute_blocks: 1
Bn_compute_blocks: 0
residual_diff_threshold: 0.12
enable_taylorseer: true
taylorseer_order: 1
parallelism_config:
ulysses_size: auto
attention_backend: native
extra_parallel_modules: ["text_encoder", "vae"]
quantize_config:
quant_type: "float8"
exclude_layers:
- "embedder"
- "embed"
verbose: false
```
Then, apply the combined cache, parallelism and quantization config from yaml. Please also enable torch.compile for better performance if you are using quantization.
## Advanced Configuration
### DBCache Parameters
DBCache controls block-level caching behavior:
| Parameter | Env Variable | Default | Description |
|-----------|---------------------------|---------|------------------------------------------|
| Fn | `SGLANG_CACHE_DIT_FN` | 1 | Number of first blocks to always compute |
| Bn | `SGLANG_CACHE_DIT_BN` | 0 | Number of last blocks to always compute |
| W | `SGLANG_CACHE_DIT_WARMUP` | 4 | Warmup steps before caching starts |
| R | `SGLANG_CACHE_DIT_RDT` | 0.24 | Residual difference threshold |
| MC | `SGLANG_CACHE_DIT_MC` | 3 | Maximum continuous cached steps |
### TaylorSeer Configuration
TaylorSeer improves caching accuracy using Taylor expansion:
| Parameter | Env Variable | Default | Description |
|-----------|-------------------------------|---------|---------------------------------|
| Enable | `SGLANG_CACHE_DIT_TAYLORSEER` | false | Enable TaylorSeer calibrator |
| Order | `SGLANG_CACHE_DIT_TS_ORDER` | 1 | Taylor expansion order (1 or 2) |
### Combined Configuration Example
DBCache and TaylorSeer are complementary strategies that work together, you can configure both sets of parameters
simultaneously:
```bash
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_FN=2 \
SGLANG_CACHE_DIT_BN=1 \
SGLANG_CACHE_DIT_WARMUP=4 \
SGLANG_CACHE_DIT_RDT=0.4 \
SGLANG_CACHE_DIT_MC=4 \
SGLANG_CACHE_DIT_TAYLORSEER=true \
SGLANG_CACHE_DIT_TS_ORDER=2 \
sglang generate --model-path black-forest-labs/FLUX.1-dev \
--prompt "A curious raccoon in a forest"
```
### SCM (Step Computation Masking)
SCM provides step-level caching control for additional speedup. It decides which denoising steps to compute fully and
which to use cached results.
**SCM Presets**
SCM is configured with presets:
| Preset | Compute Ratio | Speed | Quality |
|----------|---------------|----------|------------|
| `none` | 100% | Baseline | Best |
| `slow` | ~75% | ~1.3x | High |
| `medium` | ~50% | ~2x | Good |
| `fast` | ~35% | ~3x | Acceptable |
| `ultra` | ~25% | ~4x | Lower |
**Usage**
```bash
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_PRESET=medium \
sglang generate --model-path Qwen/Qwen-Image \
--prompt "A futuristic cityscape at sunset"
```
**Custom SCM Bins**
For fine-grained control over which steps to compute vs cache:
```bash
SGLANG_CACHE_DIT_ENABLED=true \
SGLANG_CACHE_DIT_SCM_COMPUTE_BINS="8,3,3,2,2" \
SGLANG_CACHE_DIT_SCM_CACHE_BINS="1,2,2,2,3" \
sglang generate --model-path Qwen/Qwen-Image \
--prompt "A futuristic cityscape at sunset"
```
**SCM Policy**
| Policy | Env Variable | Description |
|-----------|---------------------------------------|---------------------------------------------|
| `dynamic` | `SGLANG_CACHE_DIT_SCM_POLICY=dynamic` | Adaptive caching based on content (default) |
| `static` | `SGLANG_CACHE_DIT_SCM_POLICY=static` | Fixed caching pattern |
## Environment Variables
All Cache-DiT parameters can be configured via environment variables.
See [Environment Variables](../../environment_variables.md) for the complete list.
## Supported Models
SGLang Diffusion x Cache-DiT supports almost all models originally supported in SGLang Diffusion:
| Model Family | Example Models |
|--------------|-----------------------------|
| Wan | Wan2.1, Wan2.2 |
| Flux | FLUX.1-dev, FLUX.2-dev |
| Z-Image | Z-Image-Turbo |
| Qwen | Qwen-Image, Qwen-Image-Edit |
| Hunyuan | HunyuanVideo |
## Performance Tips
1. **Start with defaults**: The default parameters work well for most models
2. **Use TaylorSeer**: It typically improves both speed and quality
3. **Tune R threshold**: Lower values = better quality, higher values = faster
4. **SCM for extra speed**: Use `medium` preset for good speed/quality balance
5. **Warmup matters**: Higher warmup = more stable caching decisions
## Limitations
- **SGLang-native pipelines**: Distributed support (TP/SP) is not yet validated; Cache-DiT will be automatically
disabled when `world_size > 1`.
- **SCM minimum steps**: SCM requires >= 8 inference steps to be effective
- **Model support**: Only models registered in Cache-DiT's BlockAdapterRegister are supported
## Troubleshooting
### SCM disabled for low step count
For models with < 8 inference steps (e.g., DMD distilled models), SCM will be automatically disabled. DBCache
acceleration still works.
## References
- [Cache-DiT](https://github.com/vipshop/cache-dit)
- [SGLang Diffusion](../index.md)

65
third_party/sglang/docs/diffusion/performance/cache/index.md vendored Normal file
View File

@@ -0,0 +1,65 @@
# Caching Acceleration
SGLang provides two complementary caching strategies for Diffusion Transformer (DiT) models. Both reduce denoising cost by skipping redundant computation, but they operate at different levels.
## Overview
SGLang supports two complementary caching approaches:
| Strategy | Scope | Mechanism | Best For |
|----------|-------|-----------|----------|
| **Cache-DiT** | Block-level | Skip individual transformer blocks dynamically | Advanced, higher speedup |
| **TeaCache** | Timestep-level | Skip entire denoising steps based on L1 similarity | Simple, built-in |
## Cache-DiT
[Cache-DiT](https://github.com/vipshop/cache-dit) provides block-level caching with
advanced strategies like DBCache and TaylorSeer. It can achieve up to **1.69x speedup**.
See [cache_dit.md](cache_dit.md) for detailed configuration.
### Quick Start
```bash
SGLANG_CACHE_DIT_ENABLED=true \
sglang generate --model-path Qwen/Qwen-Image \
--prompt "A beautiful sunset over the mountains"
```
### Key Features
- **DBCache**: Dynamic block-level caching based on residual differences
- **TaylorSeer**: Taylor expansion-based calibration for optimized caching
- **SCM**: Step-level computation masking for additional speedup
## TeaCache
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
See [teacache.md](teacache.md) for detailed documentation.
### Quick Overview
- Tracks L1 distance between modulated inputs across timesteps
- When accumulated distance is below threshold, reuses cached residual
- Supports CFG with separate positive/negative caches
### Supported Models
- Wan (wan2.1, wan2.2)
- Hunyuan (HunyuanVideo)
- Z-Image
For Flux and Qwen models, TeaCache is automatically disabled when CFG is enabled.
```{toctree}
:maxdepth: 1
cache_dit
teacache
```
## References
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
- [TeaCache Paper](https://arxiv.org/abs/2411.14324)

84
third_party/sglang/docs/diffusion/performance/cache/teacache.md vendored Normal file
View File

@@ -0,0 +1,84 @@
# TeaCache
> **Note**: This is one of two caching strategies available in SGLang.
> For an overview of all caching options, see [caching](../index.md).
TeaCache (Temporal similarity-based caching) accelerates diffusion inference by detecting when consecutive denoising steps are similar enough to skip computation entirely.
## Overview
TeaCache works by:
1. Tracking the L1 distance between modulated inputs across consecutive timesteps
2. Accumulating the rescaled L1 distance over steps
3. When accumulated distance is below a threshold, reusing the cached residual
4. Supporting CFG (Classifier-Free Guidance) with separate positive/negative caches
## How It Works
### L1 Distance Tracking
At each denoising step, TeaCache computes the relative L1 distance between the current and previous modulated inputs:
```
rel_l1 = |current - previous|.mean() / |previous|.mean()
```
This distance is then rescaled using polynomial coefficients and accumulated:
```
accumulated += poly(coefficients)(rel_l1)
```
### Cache Decision
- If `accumulated >= threshold`: Force computation, reset accumulator
- If `accumulated < threshold`: Skip computation, use cached residual
### CFG Support
For models that support CFG cache separation (Wan, Hunyuan, Z-Image), TeaCache maintains separate caches for positive and negative branches:
- `previous_modulated_input` / `previous_residual` for positive branch
- `previous_modulated_input_negative` / `previous_residual_negative` for negative branch
For models that don't support CFG separation (Flux, Qwen), TeaCache is automatically disabled when CFG is enabled.
## Configuration
TeaCache is configured via `TeaCacheParams` in the sampling parameters:
```python
from sglang.multimodal_gen.configs.sample.teacache import TeaCacheParams
params = TeaCacheParams(
teacache_thresh=0.1, # Threshold for accumulated L1 distance
coefficients=[1.0, 0.0, 0.0], # Polynomial coefficients for L1 rescaling
)
```
### Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `teacache_thresh` | float | Threshold for accumulated L1 distance. Lower = more caching, faster but potentially lower quality |
| `coefficients` | list[float] | Polynomial coefficients for L1 rescaling. Model-specific tuning |
### Model-Specific Configurations
Different models may have different optimal configurations. The coefficients are typically tuned per-model to balance speed and quality.
## Supported Models
TeaCache is built into the following model families:
| Model Family | CFG Cache Separation | Notes |
|--------------|---------------------|-------|
| Wan (wan2.1, wan2.2) | Yes | Full support |
| Hunyuan (HunyuanVideo) | Yes | To be supported |
| Z-Image | Yes | To be supported |
| Flux | No | To be supported |
| Qwen | No | To be supported |
## References
- [TeaCache: Accelerating Diffusion Models with Temporal Similarity](https://arxiv.org/abs/2411.14324)

42
third_party/sglang/docs/diffusion/performance/index.md vendored Normal file
View File

@@ -0,0 +1,42 @@
# Performance
This section covers the main performance levers for SGLang Diffusion: attention backends, caching acceleration, and profiling.
## Overview
| Optimization | Type | Description |
|--------------|------|-------------|
| **Cache-DiT** | Caching | Block-level caching with DBCache, TaylorSeer, and SCM |
| **TeaCache** | Caching | Timestep-level caching based on temporal similarity |
| **Attention Backends** | Kernel | Optimized attention implementations (FlashAttention, SageAttention, etc.) |
| **Profiling** | Diagnostics | PyTorch Profiler and Nsight Systems guidance |
## Start Here
- Use [Attention Backends](attention_backends.md) to choose the best backend for your model and hardware.
- Use [Caching Acceleration](cache/index.md) to reduce denoising cost with Cache-DiT or TeaCache.
- Use [Profiling](profiling.md) when you need to diagnose a bottleneck rather than guess.
## Caching at a Glance
- [Cache-DiT](cache/cache_dit.md) is block-level caching for diffusers pipelines and higher speedup-oriented tuning.
- [TeaCache](cache/teacache.md) is timestep-level caching built into SGLang model families.
```{toctree}
:maxdepth: 1
attention_backends
cache/index
profiling
```
## Current Baseline Snapshot
For Ring SP benchmark details, see:
- [Ring SP Performance](ring_sp_performance.md)
## References
- [Cache-DiT Repository](https://github.com/vipshop/cache-dit)
- [TeaCache Paper](https://arxiv.org/abs/2411.14324)

136
third_party/sglang/docs/diffusion/performance/profiling.md vendored Normal file
View File

@@ -0,0 +1,136 @@
# Profiling Multimodal Generation
This guide covers profiling techniques for multimodal generation pipelines in SGLang.
## PyTorch Profiler
PyTorch Profiler provides detailed kernel execution time, call stack, and GPU utilization metrics.
### Denoising Stage Profiling
Profile the denoising stage with sampled timesteps (default: 5 steps after 1 warmup step):
```bash
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--seed 0 \
--profile
```
**Parameters:**
- `--profile`: Enable profiling for the denoising stage
- `--num-profiled-timesteps N`: Number of timesteps to profile after warmup (default: 5)
- Smaller values reduce trace file size
- Example: `--num-profiled-timesteps 10` profiles 10 steps after 1 warmup step
### Full Pipeline Profiling
Profile all pipeline stages (text encoding, denoising, VAE decoding, etc.):
```bash
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--seed 0 \
--profile \
--profile-all-stages
```
**Parameters:**
- `--profile-all-stages`: Used with `--profile`, profile all pipeline stages instead of just denoising
### Output Location
By default, trace files are saved in the ./logs/ directory.
The exact output file path will be shown in the console output, for example:
```bash
[mm-dd hh:mm:ss] Saved profiler traces to: /sgl-workspace/sglang/logs/mocked_fake_id_for_offline_generate-5_steps-global-rank0.trace.json.gz
```
### View Traces
Load and visualize trace files at:
- https://ui.perfetto.dev/ (recommended)
- chrome://tracing (Chrome only)
For large trace files, reduce `--num-profiled-timesteps` or avoid using `--profile-all-stages`.
### `--perf-dump-path` (Stage/Step Timing Dump)
Besides profiler traces, you can also dump a lightweight JSON report that contains:
- stage-level timing breakdown for the full pipeline
- step-level timing breakdown for the denoising stage (per diffusion step)
This is useful to quickly identify which stage dominates end-to-end latency, and whether denoising steps have uniform runtimes (and if not, which step has an abnormal spike).
The dumped JSON contains a `denoise_steps_ms` field formatted as an array of objects, each with a `step` key (the step index) and a `duration_ms` key.
Example:
```bash
sglang generate \
--model-path <MODEL_PATH_OR_ID> \
--prompt "<PROMPT>" \
--perf-dump-path perf.json
```
## Nsight Systems
Nsight Systems provides low-level CUDA profiling with kernel details, register usage, and memory access patterns.
### Installation
See the [SGLang profiling guide](https://github.com/sgl-project/sglang/blob/main/docs/developer_guide/benchmark_and_profiling.md#profile-with-nsight) for installation instructions.
### Basic Profiling
Profile the entire pipeline execution:
```bash
nsys profile \
--trace-fork-before-exec=true \
--cuda-graph-trace=node \
--force-overwrite=true \
-o QwenImage \
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--seed 0
```
### Targeted Stage Profiling
Use `--delay` and `--duration` to capture specific stages and reduce file size:
```bash
nsys profile \
--trace-fork-before-exec=true \
--cuda-graph-trace=node \
--force-overwrite=true \
--delay 10 \
--duration 30 \
-o QwenImage_denoising \
sglang generate \
--model-path Qwen/Qwen-Image \
--prompt "A Logo With Bold Large Text: SGL Diffusion" \
--seed 0
```
**Parameters:**
- `--delay N`: Wait N seconds before starting capture (skip initialization overhead)
- `--duration N`: Capture for N seconds (focus on specific stages)
- `--force-overwrite`: Overwrite existing output files
## Notes
- **Reduce trace size**: Use `--num-profiled-timesteps` with smaller values or `--delay`/`--duration` with Nsight Systems
- **Stage-specific analysis**: Use `--profile` alone for denoising stage, add `--profile-all-stages` for full pipeline
- **Multiple runs**: Profile with different prompts and resolutions to identify bottlenecks across workloads
## FAQ
- If you are profiling `sglang generate` with Nsight Systems and find that the generated profiler file did not capture any CUDA kernels, you can resolve this issue by increasing the model's inference steps to extend the execution time.

67
third_party/sglang/docs/diffusion/performance/ring_sp_performance.md vendored Normal file
View File

@@ -0,0 +1,67 @@
# Ring SP Benchmark: Wan2.2-TI2V-5B (u1r2 vs Baseline)
This page reports Ring-SP performance for `Wan2.2-TI2V-5B-Diffusers` using:
- Parallel config: `sp=2, ulysses=1, ring=2` (short: `u1r2`)
- Baseline config: `sp=1, ulysses=1, ring=1` (short: `u1r1`)
## Benchmark Setup
- Model: `Wan2.2-TI2V-5B-Diffusers`
- GPU: `48G RTX40 series * 2`
## Online Serving
### Ring SP (`u1r2`)
```bash
sglang serve \
--model-type diffusion \
--model-path /model/HuggingFace/Wan-AI/Wan2.2-TI2V-5B-Diffusers \
--num-gpus 2 --sp-degree 2 --ulysses-degree 1 --ring-degree 2 \
--port 8898
```
### Baseline (`u1r1`)
```bash
sglang serve \
--model-type diffusion \
--model-path /model/HuggingFace/Wan-AI/Wan2.2-TI2V-5B-Diffusers \
--num-gpus 1 --sp-degree 1 --ulysses-degree 1 --ring-degree 1 \
--port 8898
```
## Benchmarks
### Benchmark Disclaimer
These benchmarks are provided for reference under one specific setup and command configuration. Actual performance may vary with model settings, runtime environment, and request patterns.
### Stage Time Breakdown
| Stage / Metric | `u1r2` (s) | `u1r1` baseline (s) | Speedup |
|---|---:|---:|---:|
| InputValidation | 0.1060 | 0.1029 | 0.97x |
| TextEncoding | 1.3965 | 2.2261 | 1.59x |
| LatentPreparation | 0.0002 | 0.0002 | 1.00x |
| TimestepPreparation | 0.0003 | 0.0004 | 1.33x |
| Denoising | 52.6358 | 71.6785 | 1.36x |
| Decoding | 7.6708 | 13.4314 | 1.75x |
| **Total** | **63.74** | **90.63** | **1.42x** |
### Memory Usage
| Memory Metric | `u1r2` (GB) | `u1r1` baseline (GB) | Delta |
|---|---:|---:|---:|
| Peak GPU Memory | 20.07 | 27.40 | -7.33 |
| Peak Allocated | 13.35 | 20.40 | -7.05 |
| Memory Overhead | 6.72 | 7.00 | -0.28 |
| Overhead Ratio | 33.5% | 25.6% | +7.9pp |
## Summary
- End-to-end latency improves from `90.63s` to `63.74s` (`1.42x`).
- Main gains come from `Denoising` (`1.36x`) and `Decoding` (`1.75x`).
- Absolute memory usage drops noticeably on Ring-SP (`Peak GPU Memory -7.33GB`, `Peak Allocated -7.05GB`).
- Overhead ratio rises (`+7.9pp`), so future tuning can focus on reducing communication/runtime overhead while preserving the latency gain.