chore: vendor sglang v0.5.10 snapshot
This commit is contained in:
195
third_party/sglang/docs/platforms/amd_gpu.md
vendored
Normal file
195
third_party/sglang/docs/platforms/amd_gpu.md
vendored
Normal file
@@ -0,0 +1,195 @@
|
||||
# AMD GPUs
|
||||
|
||||
This document describes how to run SGLang on AMD GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
## System Configuration
|
||||
|
||||
When using AMD GPUs (such as MI300X), certain system-level optimizations help ensure stable performance. Here we take MI300X as an example. AMD provides official documentation for MI300X optimization and system tuning:
|
||||
|
||||
- [AMD MI300X Tuning Guides](https://rocm.docs.amd.com/en/latest/how-to/tuning-guides/mi300x/index.html)
|
||||
- [LLM inference performance validation on AMD Instinct MI300X](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference/vllm-benchmark.html)
|
||||
- [AMD Instinct MI300X System Optimization](https://rocm.docs.amd.com/en/latest/how-to/system-optimization/mi300x.html)
|
||||
- [AMD Instinct MI300X Workload Optimization](https://rocm.docs.amd.com/en/latest/how-to/rocm-for-ai/inference-optimization/workload.html)
|
||||
- [Supercharge DeepSeek-R1 Inference on AMD Instinct MI300X](https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/README.html)
|
||||
|
||||
**NOTE:** We strongly recommend reading these docs and guides entirely to fully utilize your system.
|
||||
|
||||
Below are a few key settings to confirm or enable for SGLang:
|
||||
|
||||
### Update GRUB Settings
|
||||
|
||||
In `/etc/default/grub`, append the following to `GRUB_CMDLINE_LINUX`:
|
||||
|
||||
```text
|
||||
pci=realloc=off iommu=pt
|
||||
```
|
||||
|
||||
Afterward, run `sudo update-grub` (or your distro’s equivalent) and reboot.
|
||||
|
||||
### Disable NUMA Auto-Balancing
|
||||
|
||||
```bash
|
||||
sudo sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
|
||||
```
|
||||
|
||||
You can automate or verify this change using [this helpful script](https://github.com/ROCm/triton/blob/rocm_env/scripts/amd/env_check.sh).
|
||||
|
||||
Again, please go through the entire documentation to confirm your system is using the recommended configuration.
|
||||
|
||||
## Install SGLang
|
||||
|
||||
You can install SGLang using one of the methods below.
|
||||
|
||||
### Install from Source
|
||||
|
||||
```bash
|
||||
# Use the last release branch
|
||||
git clone -b v0.5.9 https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
|
||||
# Compile sgl-kernel
|
||||
pip install --upgrade pip
|
||||
cd sgl-kernel
|
||||
python setup_rocm.py install
|
||||
|
||||
# Install sglang python package along with diffusion support
|
||||
cd ..
|
||||
rm -rf python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e "python[all_hip]"
|
||||
```
|
||||
|
||||
### Install Using Docker (Recommended)
|
||||
|
||||
The docker images are available on Docker Hub at [lmsysorg/sglang](https://hub.docker.com/r/lmsysorg/sglang/tags), built from [rocm.Dockerfile](https://github.com/sgl-project/sglang/tree/main/docker).
|
||||
|
||||
The steps below show how to build and use an image.
|
||||
|
||||
1. Build the docker image.
|
||||
If you use pre-built images, you can skip this step and replace `sglang_image` with the pre-built image names in the steps below.
|
||||
|
||||
```bash
|
||||
docker build -t sglang_image -f rocm.Dockerfile .
|
||||
```
|
||||
|
||||
2. Create a convenient alias.
|
||||
|
||||
```bash
|
||||
alias drun='docker run -it --rm --network=host --privileged --device=/dev/kfd --device=/dev/dri \
|
||||
--ipc=host --shm-size 16G --group-add video --cap-add=SYS_PTRACE \
|
||||
--security-opt seccomp=unconfined \
|
||||
-v $HOME/dockerx:/dockerx \
|
||||
-v /data:/data'
|
||||
```
|
||||
|
||||
If you are using RDMA, please note that:
|
||||
- `--network host` and `--privileged` are required by RDMA. If you don't need RDMA, you can remove them.
|
||||
- You may need to set `NCCL_IB_GID_INDEX` if you are using RoCE, for example: `export NCCL_IB_GID_INDEX=3`.
|
||||
|
||||
3. Launch the server.
|
||||
|
||||
**NOTE:** Replace `<secret>` below with your [huggingface hub token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
```bash
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path NousResearch/Meta-Llama-3.1-8B \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
4. To verify the utility, you can run a benchmark in another terminal or refer to [other docs](https://docs.sglang.io/basic_usage/openai_api_completions.html) to send requests to the engine.
|
||||
|
||||
```bash
|
||||
drun sglang_image \
|
||||
python3 -m sglang.bench_serving \
|
||||
--backend sglang \
|
||||
--dataset-name random \
|
||||
--num-prompts 4000 \
|
||||
--random-input 128 \
|
||||
--random-output 128
|
||||
```
|
||||
|
||||
With your AMD system properly configured and SGLang installed, you can now fully leverage AMD hardware to power SGLang’s machine learning capabilities.
|
||||
|
||||
## Quantization on AMD GPUs
|
||||
|
||||
The [Quantization documentation](../advanced_features/quantization.md#platform-compatibility) has a full compatibility matrix. The short version: FP8, AWQ, MXFP4, W8A8, GPTQ, compressed-tensors, Quark, and **petit_nvfp4** (NVFP4 on ROCm via [Petit](https://github.com/causalflow-ai/petit-kernel)) all work on AMD. Methods that depend on Marlin or NVIDIA-specific kernels (`awq_marlin`, `gptq_marlin`, `gguf`, `modelopt_fp8`, `modelopt_fp4`) do not.
|
||||
|
||||
A few things to keep in mind:
|
||||
|
||||
- FP8 works via Aiter or Triton. Pre-quantized FP8 models like DeepSeek-V3/R1 work out of the box.
|
||||
- AWQ uses Triton dequantization kernels on AMD. The faster Marlin path is not available.
|
||||
- MXFP4 requires CDNA3/CDNA4 and `SGLANG_USE_AITER=1`.
|
||||
- `petit_nvfp4` enables NVFP4 models (e.g., [Llama 3.3 70B FP4](https://huggingface.co/nvidia/Llama-3.3-70B-Instruct-FP4)) on MI250/MI300X via [Petit](https://github.com/causalflow-ai/petit-kernel). Install with `pip install petit-kernel`; no `--quantization` flag needed when loading pre-quantized NVFP4 models.
|
||||
- `quark_int4fp8_moe` is an AMD-only online quantization method for MoE models on CDNA3/CDNA4.
|
||||
|
||||
Several of these backends are accelerated by [Aiter](https://github.com/ROCm/aiter). Enable it with:
|
||||
|
||||
```bash
|
||||
export SGLANG_USE_AITER=1
|
||||
```
|
||||
|
||||
Example -- serving an AWQ model:
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path hugging-quants/Mixtral-8x7B-Instruct-v0.1-AWQ-INT4 \
|
||||
--trust-remote-code \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
Example -- FP8 online quantization:
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--quantization fp8 \
|
||||
--port 30000 --host 0.0.0.0
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Running DeepSeek-V3
|
||||
|
||||
The only difference when running DeepSeek-V3 is in how you start the server. Here's an example command:
|
||||
|
||||
```bash
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--ipc=host \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-V3 \ # <- here
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
[Running DeepSeek-R1 on a single NDv5 MI300X VM](https://techcommunity.microsoft.com/blog/azurehighperformancecomputingblog/running-deepseek-r1-on-a-single-ndv5-mi300x-vm/4372726) could also be a good reference.
|
||||
|
||||
### Running Llama3.1
|
||||
|
||||
Running Llama3.1 is nearly identical to running DeepSeek-V3. The only difference is in the model specified when starting the server, shown by the following example command:
|
||||
|
||||
```bash
|
||||
drun -p 30000:30000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
--ipc=host \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
sglang_image \
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \ # <- here
|
||||
--tp 8 \
|
||||
--trust-remote-code \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
### Warmup Step
|
||||
|
||||
When the server displays `The server is fired up and ready to roll!`, it means the startup is successful.
|
||||
20
third_party/sglang/docs/platforms/apple_metal.md
vendored
Normal file
20
third_party/sglang/docs/platforms/apple_metal.md
vendored
Normal file
@@ -0,0 +1,20 @@
|
||||
# Apple Silicon with Metal
|
||||
|
||||
This document describes how run SGLang on Apple Silicon using [Metal](https://developer.apple.com/metal/). If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
## Install SGLang
|
||||
|
||||
You can install SGLang using one of the methods below.
|
||||
|
||||
### Install from Source
|
||||
|
||||
```bash
|
||||
# Use the default branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
|
||||
# Install sglang python package
|
||||
pip install --upgrade pip
|
||||
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
uv pip install -e "python[all_mps]"
|
||||
```
|
||||
163
third_party/sglang/docs/platforms/ascend/ascend_contribution_guide.md
vendored
Normal file
163
third_party/sglang/docs/platforms/ascend/ascend_contribution_guide.md
vendored
Normal file
@@ -0,0 +1,163 @@
|
||||
# Contribution Guide
|
||||
|
||||
Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
|
||||
|
||||
## Install SGLang from Source
|
||||
|
||||
### Prepare Environment
|
||||
|
||||
Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](ascend_npu.md) to install the necessary dependencies. We recommend [using docker](ascend_npu.md#method-2-using-docker-image) to build the environment.
|
||||
|
||||
### Fork and clone the repository
|
||||
|
||||
**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<your_user_name>/sglang.git
|
||||
# if you are using docker, the environment is already set up.
|
||||
cd sglang
|
||||
export PYTHONPATH=$PWD/python:$PYTHONPATH
|
||||
```
|
||||
|
||||
## Format code with pre-commit
|
||||
|
||||
We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
|
||||
|
||||
```bash
|
||||
pip3 install pre-commit
|
||||
pre-commit install
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
|
||||
- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
|
||||
|
||||
## Run and add unit tests
|
||||
|
||||
If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
|
||||
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
|
||||
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
|
||||
|
||||
If you need to use model which is not in ```python/sglang/test/ascend/test_ascend_utils.py`` list. Follow these steps:
|
||||
1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
|
||||
2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
|
||||
If this is not the case, use following command on CI server:
|
||||
```bash
|
||||
modelscope download
|
||||
--model {your_model_repo}/{your_model}
|
||||
--local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
|
||||
```
|
||||
> Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model.
|
||||
4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path).
|
||||
|
||||
## Write documentations
|
||||
|
||||
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
|
||||
For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
|
||||
|
||||
## Test the accuracy
|
||||
If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
|
||||
|
||||
```
|
||||
# Launch a server
|
||||
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
|
||||
|
||||
# Evaluate
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
|
||||
This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
|
||||
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
|
||||
|
||||
GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
|
||||
You can find additional accuracy eval examples in:
|
||||
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_eval_accuracy_large.py)
|
||||
- [test_moe_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_moe_eval_accuracy_large.py)
|
||||
|
||||
## Benchmark the speed
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md).
|
||||
|
||||
## Requesting a review for merge
|
||||
You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
|
||||
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
|
||||
Then your PR can be merged.
|
||||
|
||||
## How to Trigger CI Tests
|
||||
|
||||
We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
|
||||
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
|
||||
|
||||
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
|
||||
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI.
|
||||
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
|
||||
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`.
|
||||
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
|
||||
To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
|
||||
|
||||
Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
|
||||
|
||||
If you don’t have permission, please ask maintainers to trigger CI for you.
|
||||
|
||||
### CI rate limits
|
||||
|
||||
Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
|
||||
|
||||
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
|
||||
|
||||
Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
|
||||
|
||||
```yaml
|
||||
cool-down-minutes:
|
||||
description: "Default cooldown period in minutes; 0 disables rate limiting"
|
||||
type: number
|
||||
default: 120
|
||||
```
|
||||
|
||||
Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
|
||||
|
||||
## Code style guidance
|
||||
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
|
||||
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
|
||||
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
|
||||
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
|
||||
- Make functions as pure as possible. Avoid in-place modification of arguments.
|
||||
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
|
||||
- Keep tests run fast.
|
||||
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
|
||||
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
|
||||
- Reuse server launches in your unit tests to make tests run faster.
|
||||
- When supporting new hardware or features, follow these guidelines:
|
||||
- Do not drastically change existing code.
|
||||
- Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`).
|
||||
- If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
|
||||
|
||||
## How to update sgl-kernel
|
||||
Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
|
||||
To add a new kernel or modify an existing one in the `sgl-kernel/` source tree, you must use multiple PRs.
|
||||
|
||||
Follow these steps:
|
||||
|
||||
1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
|
||||
2. Bump the version of the kernel package (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
|
||||
- Once merged, this will trigger an automatic release of the `sglang-kernel` wheel to PyPI.
|
||||
- If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
|
||||
3. Apply the changes:
|
||||
- Update the `sglang-kernel` version in `sglang/python/pyproject.toml` to use the modified kernels.
|
||||
- Update the related caller code in the sglang to use the new kernel.
|
||||
|
||||
## How to update sgl-kernel-npu
|
||||
|
||||
Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md).
|
||||
|
||||
## Tips for newcomers
|
||||
|
||||
If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
|
||||
|
||||
If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
|
||||
|
||||
Thank you for your interest in SGLang. Happy coding!
|
||||
244
third_party/sglang/docs/platforms/ascend/ascend_npu.md
vendored
Normal file
244
third_party/sglang/docs/platforms/ascend/ascend_npu.md
vendored
Normal file
@@ -0,0 +1,244 @@
|
||||
|
||||
# SGLang installation with NPUs support
|
||||
|
||||
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
|
||||
|
||||
## Component Version Mapping For SGLang
|
||||
| Component | Version | Obtain Way |
|
||||
|-------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| HDK | 25.3.RC1 | [link](https://www.hiascend.com/hardware/firmware-drivers/commercial?product=7&model=33) |
|
||||
| CANN | 8.5.0 | [Obtain Images](#obtain-cann-image) |
|
||||
| Pytorch Adapter | 7.3.0 | [link](https://gitcode.com/Ascend/pytorch/releases) |
|
||||
| MemFabric | 1.0.5 | `pip install memfabric-hybrid==1.0.5` |
|
||||
| Triton | 3.2.0 | `pip install triton-ascend`|
|
||||
| SGLang NPU Kernel | NA | [link](https://github.com/sgl-project/sgl-kernel-npu/releases) |
|
||||
|
||||
<a id="obtain-cann-image"></a>
|
||||
### Obtain CANN Image
|
||||
You can obtain the dependency of a specified version of CANN through an image.
|
||||
```shell
|
||||
# for Atlas 800I A3 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
|
||||
# for Atlas 800I A2 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
|
||||
```
|
||||
|
||||
## Preparing the Running Environment
|
||||
|
||||
### Method 1: Installing from source with prerequisites
|
||||
|
||||
#### Python Version
|
||||
|
||||
Only `python==3.11` is supported currently. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
|
||||
|
||||
```shell
|
||||
conda create --name sglang_npu python=3.11
|
||||
conda activate sglang_npu
|
||||
```
|
||||
|
||||
#### CANN
|
||||
|
||||
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit)
|
||||
|
||||
#### MemFabric-Hybrid
|
||||
|
||||
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
|
||||
|
||||
```shell
|
||||
pip install memfabric-hybrid==1.0.5
|
||||
```
|
||||
|
||||
#### Pytorch and Pytorch Framework Adaptor on Ascend
|
||||
|
||||
```shell
|
||||
PYTORCH_VERSION=2.8.0
|
||||
TORCHVISION_VERSION=0.23.0
|
||||
TORCH_NPU_VERSION=2.8.0
|
||||
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install torch_npu==$TORCH_NPU_VERSION
|
||||
```
|
||||
|
||||
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)
|
||||
|
||||
#### Triton on Ascend
|
||||
|
||||
We provide our own implementation of Triton for Ascend.
|
||||
|
||||
```shell
|
||||
pip install triton-ascend
|
||||
```
|
||||
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
|
||||
|
||||
#### SGLang Kernels NPU
|
||||
We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md).
|
||||
|
||||
#### DeepEP-compatible Library
|
||||
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).
|
||||
|
||||
#### Installing SGLang from source
|
||||
|
||||
```shell
|
||||
# Use the last release branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
mv python/pyproject_npu.toml python/pyproject.toml
|
||||
pip install -e python[all_npu]
|
||||
```
|
||||
|
||||
### Method 2: Using Docker Image
|
||||
#### Obtain Image
|
||||
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
|
||||
1. Download SGLang image
|
||||
```angular2html
|
||||
dockerhub: docker.io/lmsysorg/sglang:$tag
|
||||
# Main-based tag, change main to specific version like v0.5.6,
|
||||
# you can get image for specific version
|
||||
Atlas 800I A3 : {main}-cann8.5.0-a3
|
||||
Atlas 800I A2: {main}-cann8.5.0-910b
|
||||
```
|
||||
2. Build an image based on Dockerfile
|
||||
```shell
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
|
||||
docker build -t <image_name> -f npu.Dockerfile .
|
||||
```
|
||||
|
||||
#### Create Docker
|
||||
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
|
||||
|
||||
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.
|
||||
|
||||
```shell
|
||||
|
||||
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
|
||||
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
|
||||
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
|
||||
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
|
||||
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
|
||||
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
|
||||
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--volume /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
|
||||
|
||||
# Add HF_TOKEN env for download model by SGLang.
|
||||
drun --env "HF_TOKEN=<secret>" \
|
||||
<image_name> \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
## System Settings
|
||||
|
||||
### CPU performance power scheme
|
||||
|
||||
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
|
||||
|
||||
```shell
|
||||
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
|
||||
# Make sure changes are applied successfully
|
||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
|
||||
```
|
||||
|
||||
### Disable NUMA balancing
|
||||
|
||||
```shell
|
||||
sudo sysctl -w kernel.numa_balancing=0
|
||||
# Check
|
||||
cat /proc/sys/kernel/numa_balancing # shows 0
|
||||
```
|
||||
|
||||
### Prevent swapping out system memory
|
||||
|
||||
```shell
|
||||
sudo sysctl -w vm.swappiness=10
|
||||
|
||||
# Check
|
||||
cat /proc/sys/vm/swappiness # shows 10
|
||||
```
|
||||
|
||||
## Running SGLang Service
|
||||
### Running Service For Large Language Models
|
||||
#### PD Mixed Scene
|
||||
```shell
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
#### PD Disaggregation Scene
|
||||
1. Launch Prefill Server
|
||||
```shell
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
2. Launch Decode Server
|
||||
```shell
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 1 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
3. Launch Router
|
||||
```shell
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://127.0.0.1:8000 8995 \
|
||||
--decode http://127.0.0.1:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
|
||||
### Running Service For Multimodal Language Models
|
||||
#### PD Mixed Scene
|
||||
```shell
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen3-VL-30B-A3B-Instruct \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--tp 4 \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--enable-multimodal \
|
||||
--sampling-backend ascend
|
||||
```
|
||||
2485
third_party/sglang/docs/platforms/ascend/ascend_npu_best_practice.md
vendored
Normal file
2485
third_party/sglang/docs/platforms/ascend/ascend_npu_best_practice.md
vendored
Normal file
File diff suppressed because it is too large
Load Diff
297
third_party/sglang/docs/platforms/ascend/ascend_npu_deepseek_example.md
vendored
Normal file
297
third_party/sglang/docs/platforms/ascend/ascend_npu_deepseek_example.md
vendored
Normal file
@@ -0,0 +1,297 @@
|
||||
## DeepSeek examples
|
||||
|
||||
### Running DeepSeek-V3
|
||||
|
||||
#### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3.
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
```shell
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export HCCL_BUFFSIZE=1600
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--tp 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 8 16 24 28 32 \
|
||||
--mem-fraction-static 0.68 \
|
||||
--max-running-requests 128 \
|
||||
--context-length 8188 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 16384 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--enable-dp-attention \
|
||||
--dp-size 4 \
|
||||
--enable-dp-lm-head \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Running DeepSeek with PD disaggregation mode on 2 x Atlas 800I A3.
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
1. Prefill:
|
||||
|
||||
```shell
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export HCCL_BUFFSIZE=1536
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--host $PREFILL_HOST_IP \
|
||||
--port 8000 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-bootstrap-port 8996 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--load-balance-method round_robin \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 28680 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dp-size 2 \
|
||||
--enable-dp-attention \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
2. Decode:
|
||||
|
||||
```shell
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export HCCL_BUFFSIZE=720
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
unset TASK_QUEUE_ENABLE
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode decode \
|
||||
--host $DECODE_HOST_IP \
|
||||
--port 8001 \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 16 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 352 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--tokenizer-worker-num 4
|
||||
```
|
||||
|
||||
3. SGLang Model Gateway (former Router)
|
||||
|
||||
```shell
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
|
||||
--decode http://<DECODE_HOST_IP>:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
|
||||
#### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3.
|
||||
|
||||
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
|
||||
|
||||
1. Prefill & Decode:
|
||||
|
||||
```shell
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
|
||||
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
|
||||
|
||||
P_IP=('your prefill ip1' 'your prefill ip2')
|
||||
|
||||
D_IP=('your decode ip1' 'your decode ip2')
|
||||
|
||||
MODEL_PATH=xxx
|
||||
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
echo "${LOCAL_HOST1}"
|
||||
echo "${LOCAL_HOST2}"
|
||||
# prefill
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
|
||||
--port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
|
||||
--tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
|
||||
--chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
|
||||
--dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# decode
|
||||
for i in "${!D_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
|
||||
then
|
||||
echo "${D_IP[$i]}"
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export HCCL_BUFFSIZE=650
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
|
||||
export TASK_QUEUE_ENABLE=1
|
||||
export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
|
||||
export HCCL_SOCKET_IFNAME=xxx
|
||||
export GLOO_SOCKET_IFNAME=xxx
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
|
||||
--port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
|
||||
--mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
|
||||
--cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
|
||||
--tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \
|
||||
--load-balance-method decode_round_robin
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
2. SGLang Model Gateway (former Router):
|
||||
|
||||
```shell
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://P_IP:8000 8998 \
|
||||
--prefill http://P_IP:8000 8999 \
|
||||
--decode http://D_IP:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688 \
|
||||
--mini-lb
|
||||
```
|
||||
|
||||
#### test gsm8k
|
||||
|
||||
```python
|
||||
from types import SimpleNamespace
|
||||
from sglang.test.few_shot_gsm8k import run_eval
|
||||
|
||||
def gsm8k():
|
||||
args = SimpleNamespace(
|
||||
num_shots=5,
|
||||
data_path=None,
|
||||
num_questions=200,
|
||||
max_new_tokens=512,
|
||||
parallel=32,
|
||||
host=f"http://127.0.0.1",
|
||||
port=6688,
|
||||
)
|
||||
metrics = run_eval(args)
|
||||
print(f"{metrics=}")
|
||||
print(f"{metrics['accuracy']=}")
|
||||
if __name__ == "__main__":
|
||||
gsm8k()
|
||||
```
|
||||
39
third_party/sglang/docs/platforms/ascend/ascend_npu_environment_variables.md
vendored
Normal file
39
third_party/sglang/docs/platforms/ascend/ascend_npu_environment_variables.md
vendored
Normal file
@@ -0,0 +1,39 @@
|
||||
# Environment Variables
|
||||
|
||||
SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior.
|
||||
This document provides a list of commonly used environment variables and aims to stay updated over time.
|
||||
|
||||
## Directly Used in SGLang
|
||||
|
||||
| Environment Variable | Description | Default Value |
|
||||
|--------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
|
||||
| `SGLANG_NPU_USE_MLAPO` | Adopts the `MLAPO` fusion operator in attention <br/> preprocessing stage of the MLA model. | `false` |
|
||||
| `SGLANG_USE_FIA_NZ` | Reshapes KV Cache for FIA NZ format.<br/> `SGLANG_USE_FIA_NZ` must be enabled with `SGLANG_NPU_USE_MLAPO` | `false` |
|
||||
| `SGLANG_NPU_USE_MULTI_STREAM` | Enable dual-stream computation of shared experts <br/> and routing experts in DeepSeek models.<br/> Enable dual-stream computation in DeepSeek NSA Indexer. | `false` |
|
||||
| `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT` | Disable cast model weight tensor to a specific NPU <br/> ACL format. | `false` |
|
||||
| `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` | The maximum number of dispatched tokens on each rank. | `128` |
|
||||
|
||||
## Used in DeepEP Ascend
|
||||
|
||||
| Environment Variable | Description | Default Value |
|
||||
|-------------------------------------------|------------------------------------------------------------------------------------------------------------------------|---------------|
|
||||
| `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS` | Enable ant-moving function in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank. | `8192` |
|
||||
| `DEEPEP_NORMAL_LONG_SEQ_ROUND` | Enable ant-moving function in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank. | `1` |
|
||||
| `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ` | Enable ant-moving function in combine stage. <br/> The value `0` means disabled. | `0` |
|
||||
| `MOE_ENABLE_TOPK_NEG_ONE` | Needs to be enabled when the expert ID to be processed by <br/> DEEPEP contains -1. | `0` |
|
||||
| `DEEP_NORMAL_MODE_USE_INT8_QUANT` | Quantizes x to int8 and returns (tensor, scales) in dispatch operator. | `0` |
|
||||
|
||||
## Others
|
||||
|
||||
| Environment Variable | Description | Default Value |
|
||||
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
|
||||
| `TASK_QUEUE_ENABLE` | Used to control the optimization level of the dispatch queue<br/> about the task_queue operator. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/730/comref/Envvariables/docs/zh/environment_variable_reference/TASK_QUEUE_ENABLE.md) | `1` |
|
||||
| `INF_NAN_MODE_ENABLE` | Controls whether the chip uses saturation mode or INF_NAN mode. [Detail](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0056.html) | `1` |
|
||||
| `STREAMS_PER_DEVICE` | Configures the maximum number of streams for the stream pool. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/720/comref/Envvariables/Envir_041.html) | `32` |
|
||||
| `PYTORCH_NPU_ALLOC_CONF` | Controls the behavior of the cache allocator. <br/>This variable changes memory usage and may cause performance fluctuations. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html) | |
|
||||
| `ASCEND_MF_STORE_URL` | The address of config store in MemFabric during PD separation, <br/>which is generally set to the IP address of the P primary node<br/> with an arbitrary port number. | |
|
||||
| `ASCEND_LAUNCH_BLOCKING` | Controls whether synchronous mode is enabled during operator execution. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/710/comref/Envvariables/Envir_006.html) | `0` |
|
||||
| `HCCL_OP_EXPANSION_MODE` | Configures the expansion position for communication algorithm scheduling. [Detail](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0094.html) | |
|
||||
| `HCCL_BUFFSIZE` | Controls the size of the buffer area for shared data between two NPUs. <br/>The unit is MB, and the value must be greater than or equal to 1. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/60RC3/ptmoddevg/trainingmigrguide/performance_tuning_0047.html) | `200` |
|
||||
| `HCCL_SOCKET_IFNAME` | Configures the name of the network card used by the Host <br/>during HCCL initialization. [Detail](https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/apiref/envvar/envref_07_0075.html) | |
|
||||
| `GLOO_SOCKET_IFNAME` | Configures the network interface name for GLOO communication. | |
|
||||
194
third_party/sglang/docs/platforms/ascend/ascend_npu_glm5_examples.md
vendored
Normal file
194
third_party/sglang/docs/platforms/ascend/ascend_npu_glm5_examples.md
vendored
Normal file
@@ -0,0 +1,194 @@
|
||||
# GLM-5 examples
|
||||
|
||||
## Introduction
|
||||
|
||||
The GLM (General Language Model) series is an open-source bilingual large language model family jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. This series of models has performed outstandingly in the field of Chinese NLP with its unique unified pre-training framework and bilingual capabilities. [GLM-5](https://huggingface.co/zai-org/GLM-5) adopts the DeepSeek-V3/V3.2 architecture, including the sparse attention (DSA) and multi-token prediction (MTP). Ascend supports GLM-5 with 0Day based on the SGLang inference framework, achieving low-code seamless enablement and compatibility with the mainstream distributed parallel capabilities within the current SGLang framework. We welcome developers to download and experience it.
|
||||
|
||||
## Environment Preparation
|
||||
|
||||
### Model Weight
|
||||
|
||||
- `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5).
|
||||
- `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8).
|
||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
|
||||
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
|
||||
|
||||
```{code-block} bash
|
||||
#Atlas 800 A3
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
|
||||
#Atlas 800 A2
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
|
||||
```
|
||||
|
||||
Note: Using this image, you need to update transformers to main branch
|
||||
``` shell
|
||||
# reinstall transformers
|
||||
pip install git+https://github.com/huggingface/transformers.git
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
- Quantized model `glm5_w4a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep --deepep-mode auto
|
||||
```
|
||||
|
||||
### Multi-node Deployment
|
||||
|
||||
- `GLM-5-bf16`: require at least 2 Atlas 800 A3 (64G × 16).
|
||||
|
||||
**A3 series**
|
||||
|
||||
Modify the IP of 2 nodes, then run the same scripts on two nodes.
|
||||
|
||||
**node 0/1**
|
||||
|
||||
```shell
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
# Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
|
||||
P_IP=('your ip1' 'your ip2')
|
||||
P_MASTER="${P_IP[0]}:your port"
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $P_MASTER \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 131072 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.8\
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-max-bs 16 \
|
||||
--disable-radix-cache
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
```
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
||||
52
third_party/sglang/docs/platforms/ascend/ascend_npu_quantization.md
vendored
Normal file
52
third_party/sglang/docs/platforms/ascend/ascend_npu_quantization.md
vendored
Normal file
@@ -0,0 +1,52 @@
|
||||
# Quantization on Ascend
|
||||
|
||||
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
||||
|
||||
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers).
|
||||
|
||||
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | Diffusion models |
|
||||
|-----------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|:------------------------------------------:|
|
||||
| W4A4 dynamic | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| W8A8 static | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| W8A8 dynamic | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | Linear | **<span style="color: red;">x</span>** | **<span style="color: red;">x</span>** | **<span style="color: blue;">WIP</span>** | **<span style="color: blue;">WIP</span>** |
|
||||
| W4A4 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| W4A8 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| W8A8 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | MoE | **<span style="color: red;">x</span>** | **<span style="color: red;">x</span>** | **<span style="color: blue;">WIP</span>** | **<span style="color: red;">x</span>** |
|
||||
|
||||
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|
|
||||
| W4A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W4A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
GPTQ on Ascend support
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|----------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| [W4A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| W4A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W4A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
Compressed-tensors (LLM Compressor) on Ascend support:
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|-----------------------------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A8 dynamic with/without activation clip](https://github.com/sgl-project/sglang/pull/14736) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
|
||||
|
||||
in progress
|
||||
231
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_5_examples.md
vendored
Normal file
231
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_5_examples.md
vendored
Normal file
@@ -0,0 +1,231 @@
|
||||
# Qwen3.5 examples
|
||||
|
||||
## Environment Preparation
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
|
||||
|
||||
```{code-block} bash
|
||||
#Atlas 800 A3
|
||||
docker pull quay.io/ascend/sglang:main-cann8.5.0-a3
|
||||
#Atlas 800 A2
|
||||
docker pull quay.io/ascend/sglang:main-cann8.5.0-910b
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
quay.io/ascend/sglang:${tag}
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
#### Qwen3.5 397B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Qwen3.5 122B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 8 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Qwen3.5 35B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 2 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Qwen3.5 27B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 2 \
|
||||
--chunked-prefill-size -1 --max-prefill-tokens 120000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 32 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn
|
||||
```
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
||||
207
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_examples.md
vendored
Normal file
207
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_examples.md
vendored
Normal file
@@ -0,0 +1,207 @@
|
||||
## Qwen3 examples
|
||||
|
||||
### Running Qwen3
|
||||
|
||||
#### Running Qwen3-32B on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
|
||||
#### Running Qwen3-32B on 1 x Atlas 800I A3 with Qwen3-32B-Eagle3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
Speculative model weights could be found [here](https://huggingface.co/Zhihu-ai/Zhi-Create-Qwen3-32B-Eagle3)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path Qwen/Qwen3-32B-Eagle3 \
|
||||
--speculative-num-steps 1 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 2
|
||||
```
|
||||
|
||||
#### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-30B-A3B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
|
||||
#### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--tp-size 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--watchdog-timeout 9000 \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
|
||||
#### Running Qwen3-235B-A22B-Instruct-2507 with 256K long sequence on 2 x Atlas 800I A3 without CP
|
||||
|
||||
This example uses **PD disaggregation** for long-sequence inference and keeps **context parallel disabled**.
|
||||
|
||||
Set the shared environment variables on both nodes first:
|
||||
|
||||
```shell
|
||||
export ASCEND_USE_FIA=1
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:12345"
|
||||
export HCCL_SOCKET_IFNAME=<NETWORK_IFACE>
|
||||
export GLOO_SOCKET_IFNAME=<NETWORK_IFACE>
|
||||
|
||||
MODEL_PATH=/root/.cache/modelscope/hub/models/zcgy26/Qwen3-235B-A22B-Instruct-2507-w8a8
|
||||
```
|
||||
|
||||
**Prefill node:**
|
||||
|
||||
```shell
|
||||
export ASCEND_LAUNCH_BLOCKING=1
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export HCCL_BUFFSIZE=1500
|
||||
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
|
||||
export DEEPEP_NORMAL_LONG_SEQ_ROUND=128
|
||||
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--attention-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--quantization modelslim \
|
||||
--chunked-prefill-size -1 \
|
||||
--skip-server-warmup \
|
||||
--device npu \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.45 \
|
||||
--max-running-requests 1 \
|
||||
--host <PREFILL_HOST_IP> \
|
||||
--port 8000 \
|
||||
--dist-init-addr <PREFILL_HOST_IP>:5000 \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal
|
||||
```
|
||||
|
||||
**Decode node:**
|
||||
|
||||
```shell
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=0
|
||||
export HCCL_BUFFSIZE=4000
|
||||
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
|
||||
export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--attention-backend ascend \
|
||||
--mem-fraction-static 0.8 \
|
||||
--disable-cuda-graph \
|
||||
--device npu \
|
||||
--disable-radix-cache \
|
||||
--quantization modelslim \
|
||||
--chunked-prefill-size 8192 \
|
||||
--skip-server-warmup \
|
||||
--tp-size 16 \
|
||||
--max-running-requests 1 \
|
||||
--host <DECODE_HOST_IP> \
|
||||
--port 8232 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--disable-overlap-schedule
|
||||
```
|
||||
|
||||
**Router:**
|
||||
|
||||
```shell
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://<PREFILL_HOST_IP>:8000 8995 \
|
||||
--decode http://<DECODE_HOST_IP>:8232 \
|
||||
--host <ROUTER_HOST_IP> \
|
||||
--port 6689 \
|
||||
--prometheus-port 29010
|
||||
```
|
||||
|
||||
#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--enable-multimodal \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
19
third_party/sglang/docs/platforms/ascend/ascend_npu_support.rst
vendored
Normal file
19
third_party/sglang/docs/platforms/ascend/ascend_npu_support.rst
vendored
Normal file
@@ -0,0 +1,19 @@
|
||||
Ascend NPUs
|
||||
===============================================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
ascend_npu.md
|
||||
ascend_npu_support_features.md
|
||||
ascend_npu_support_models.md
|
||||
ascend_npu_quantization.md
|
||||
ascend_npu_deepseek_example.md
|
||||
ascend_npu_qwen3_examples.md
|
||||
mindspore_backend.md
|
||||
ascend_contribution_guide.md
|
||||
ascend_npu_best_practice.md
|
||||
ascend_npu_ring_sp_performance.md
|
||||
ascend_npu_qwen3_5_examples.md
|
||||
ascend_npu_glm5_examples.md
|
||||
ascend_npu_environment_variables.md
|
||||
488
third_party/sglang/docs/platforms/ascend/ascend_npu_support_features.md
vendored
Normal file
488
third_party/sglang/docs/platforms/ascend/ascend_npu_support_features.md
vendored
Normal file
@@ -0,0 +1,488 @@
|
||||
# Support Features on Ascend NPU
|
||||
|
||||
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
|
||||
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
If you want to know the meaning and usage of each parameter,
|
||||
click [Server Arguments](https://docs.sglang.io/advanced_features/server_arguments.html).
|
||||
|
||||
## Model and tokenizer
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------|----------|---------------------------------------|:----------------:|
|
||||
| `--model-path`<br/>`--model` | `None` | Type: str | A2, A3 |
|
||||
| `--tokenizer-path` | `None` | Type: str | A2, A3 |
|
||||
| `--tokenizer-mode` | `auto` | `auto`, `slow` | A2, A3 |
|
||||
| `--tokenizer-worker-num` | `1` | Type: int | A2, A3 |
|
||||
| `--skip-tokenizer-init` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--load-format` | `auto` | `auto`, `safetensors` | A2, A3 |
|
||||
| `--model-loader-` <br/> `extra-config` | `{}` | Type: str | A2, A3 |
|
||||
| `--trust-remote-code` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--context-length` | `None` | Type: int | A2, A3 |
|
||||
| `--is-embedding` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--enable-multimodal` | `None` | bool flag (set to enable) | A2, A3 |
|
||||
| `--revision` | `None` | Type: str | A2, A3 |
|
||||
| `--model-impl` | `auto` | `auto`, `sglang`,<br/> `transformers` | A2, A3 |
|
||||
|
||||
## HTTP server
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------|-------------|---------------------------|:----------------:|
|
||||
| `--host` | `127.0.0.1` | Type: str | A2, A3 |
|
||||
| `--port` | `30000` | Type: int | A2, A3 |
|
||||
| `--skip-server-warmup` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--warmups` | `None` | Type: str | A2, A3 |
|
||||
| `--nccl-port` | `None` | Type: int | A2, A3 |
|
||||
| `--fastapi-root-path` | `None` | Type: str | A2, A3 |
|
||||
| `--grpc-mode` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
|
||||
## Quantization and data type
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------------|----------|-----------------------------------------|:----------------:|
|
||||
| `--dtype` | `auto` | `auto`,<br/> `float16`,<br/> `bfloat16` | A2, A3 |
|
||||
| `--quantization` | `None` | `modelslim` | A2, A3 |
|
||||
| `--quantization-param-path` | `None` | Type: str | Special For GPU |
|
||||
| `--kv-cache-dtype` | `auto` | `auto` | A2, A3 |
|
||||
| `--enable-fp32-lm-head` | `False` | bool flag <br/> (set to enable) | A2, A3 |
|
||||
| `--modelopt-quant` | `None` | Type: str | Special For GPU |
|
||||
| `--modelopt-checkpoint-`<br/>`restore-path` | `None` | Type: str | Special For GPU |
|
||||
| `--modelopt-checkpoint-`<br/>`save-path` | `None` | Type: str | Special For GPU |
|
||||
| `--modelopt-export-path` | `None` | Type: str | Special For GPU |
|
||||
| `--quantize-and-serve` | `False` | bool flag <br/> (set to enable) | Special For GPU |
|
||||
| `--rl-quant-profile` | `None` | Type: str | Special For GPU |
|
||||
|
||||
## Memory and scheduling
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-----------------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--mem-fraction-static` | `None` | Type: float | A2, A3 |
|
||||
| `--max-running-requests` | `None` | Type: int | A2, A3 |
|
||||
| `--prefill-max-requests` | `None` | Type: int | A2, A3 |
|
||||
| `--max-queued-requests` | `None` | Type: int | A2, A3 |
|
||||
| `--max-total-tokens` | `None` | Type: int | A2, A3 |
|
||||
| `--chunked-prefill-size` | `None` | Type: int | A2, A3 |
|
||||
| `--max-prefill-tokens` | `16384` | Type: int | A2, A3 |
|
||||
| `--schedule-policy` | `fcfs` | `lpm`, `fcfs` | A2, A3 |
|
||||
| `--enable-priority-`<br/>`scheduling` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--schedule-low-priority-`<br/>`values-first` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--priority-scheduling-`<br/>`preemption-threshold` | `10` | Type: int | A2, A3 |
|
||||
| `--schedule-conservativeness` | `1.0` | Type: float | A2, A3 |
|
||||
| `--page-size` | `128` | Type: int | A2, A3 |
|
||||
| `--swa-full-tokens-ratio` | `0.8` | Type: float | A2, A3 |
|
||||
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--radix-eviction-policy` | `lru` | `lru`,<br/>`lfu` | A2, A3 |
|
||||
| `--enable-prefill-delayer` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--prefill-delayer-max-delay-passes` | `30` | Type: int | A2, A3 |
|
||||
| `--prefill-delayer-token-usage-low-watermark` | `None` | Type: float | A2, A3 |
|
||||
| `--prefill-delayer-forward-passes-buckets` | `None` | List[float] | A2, A3 |
|
||||
| `--prefill-delayer-wait-seconds-buckets` | `None` | List[float] | A2, A3 |
|
||||
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-dynamic-chunking` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
|
||||
## Runtime options
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------------------|----------|---------------------------|:----------------:|
|
||||
| `--device` | `None` | Type: str | A2, A3 |
|
||||
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--attention-context-parallel-size`<br/>`--attn-cp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--moe-data-parallel-size`<br/>`--moe-dp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 |
|
||||
| `--pp-async-batch-depth` | `None` | Type: int | A2, A3 |
|
||||
| `--stream-interval` | `1` | Type: int | A2, A3 |
|
||||
| `--incremental-streaming-output` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--random-seed` | `None` | Type: int | A2, A3 |
|
||||
| `--constrained-json-`<br/>`whitespace-pattern` | `None` | Type: str | A2, A3 |
|
||||
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--watchdog-timeout` | `300` | Type: float | A2, A3 |
|
||||
| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 |
|
||||
| `--dist-timeout` | `None` | Type: int | A2, A3 |
|
||||
| `--download-dir` | `None` | Type: str | A2, A3 |
|
||||
| `--model-checksum` | `None` | Type: str | A2, A3 |
|
||||
| `--base-gpu-id` | `0` | Type: int | A2, A3 |
|
||||
| `--gpu-id-step` | `1` | Type: int | A2, A3 |
|
||||
| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
|
||||
## Logging
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------------------|-------------------|--------------------------------|:----------------:|
|
||||
| `--log-level` | `info` | Type: str | A2, A3 |
|
||||
| `--log-level-http` | `None` | Type: str | A2, A3 |
|
||||
| `--log-requests` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | A2, A3 |
|
||||
| `--log-requests-format` | `text` | `text`, `json` | A2, A3 |
|
||||
| `--crash-dump-folder` | `None` | Type: str | A2, A3 |
|
||||
| `--enable-metrics` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-metrics-for-`<br/>`all-schedulers` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--tokenizer-metrics-`<br/>`custom-labels-header` | `x-custom-labels` | Type: str | A2, A3 |
|
||||
| `--tokenizer-metrics-`<br/>`allowed-custom-labels` | `None` | List[str] | A2, A3 |
|
||||
| `--bucket-time-to-`<br/>`first-token` | `None` | List[float] | A2, A3 |
|
||||
| `--bucket-inter-token-`<br/>`latency` | `None` | List[float] | A2, A3 |
|
||||
| `--bucket-e2e-request-`<br/>`latency` | `None` | List[float] | A2, A3 |
|
||||
| `--collect-tokens-`<br/>`histogram` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--prompt-tokens-buckets` | `None` | List[str] | A2, A3 |
|
||||
| `--generation-tokens-buckets` | `None` | List[str] | A2, A3 |
|
||||
| `--gc-warning-threshold-secs` | `0.0` | Type: float | A2, A3 |
|
||||
| `--decode-log-interval` | `40` | Type: int | A2, A3 |
|
||||
| `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--kv-events-config` | `None` | Type: str | Special for GPU |
|
||||
| `--enable-trace` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--oltp-traces-endpoint` | `localhost:4317` | Type: str | A2, A3 |
|
||||
| `--log-requests-target` | `None` | Type: str | A2, A3 |
|
||||
| `--uvicorn-access-log-exclude-prefixes` | `[]` | List[str] | A2, A3 |
|
||||
|
||||
## RequestMetricsExporter configuration
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--export-metrics-to-`<br/>`file` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--export-metrics-to-`<br/>`file-dir` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## API related
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------|-----------|--------------------------------|:----------------:|
|
||||
| `--api-key` | `None` | Type: str | A2, A3 |
|
||||
| `--admin-api-key` | `None` | Type: str | A2, A3 |
|
||||
| `--served-model-name` | `None` | Type: str | A2, A3 |
|
||||
| `--weight-version` | `default` | Type: str | A2, A3 |
|
||||
| `--chat-template` | `None` | Type: str | A2, A3 |
|
||||
| `--hf-chat-template-name` | `None` | Type: str | A2, A3 |
|
||||
| `--completion-template` | `None` | Type: str | A2, A3 |
|
||||
| `--enable-cache-report` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--reasoning-parser` | `None` | `deepseek-r1`<br/>`deepseek-v3`<br/>`glm45`<br/>`gpt-oss`<br/>`kimi`<br/>`qwen3`<br/>`qwen3-thinking`<br/>`step3` | A2, A3 |
|
||||
| `--tool-call-parser` | `None` | `deepseekv3`<br/>`deepseekv31`<br/>`glm`<br/>`glm45`<br/>`glm47`<br/>`gpt-oss`<br/>`kimi_k2`<br/>`llama3`<br/>`mistral`<br/>`pythonic`<br/>`qwen`<br/>`qwen25`<br/>`qwen3_coder`<br/>`step3`<br/>`gigachat3` | A2, A3 |
|
||||
| `--sampling-defaults` | `model` | `openai`, `model` | A2, A3 |
|
||||
|
||||
## Data parallelism
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------|---------------|-----------------------------------------------------------|:----------------:|
|
||||
| `--data-parallel-size`<br/>`--dp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--load-balance-method` | `auto` | `auto`,<br/> `round_robin`,<br/> `follow_bootstrap_room`,<br/> `total_requests`,<br/> `total_tokens` | A2, A3 |
|
||||
|
||||
## Multi-node distributed serving
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------|----------|-----------|:----------------:|
|
||||
| `--dist-init-addr`<br/>`--nccl-init-addr` | `None` | Type: str | A2, A3 |
|
||||
| `--nnodes` | `1` | Type: int | A2, A3 |
|
||||
| `--node-rank` | `0` | Type: int | A2, A3 |
|
||||
|
||||
## Model override args
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------------------|----------|-----------|:----------------:|
|
||||
| `--json-model-override-`<br/>`args` | `{}` | Type: str | A2, A3 |
|
||||
| `--preferred-sampling-`<br/>`params` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## LoRA
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------|----------|-------------------------------------|:----------------:|
|
||||
| `--enable-lora` | `False` | Bool flag <br/>(set to enable) | A2, A3 |
|
||||
| `--max-lora-rank` | `None` | Type: int | A2, A3 |
|
||||
| `--lora-target-modules` | `None` | `all` | A2, A3 |
|
||||
| `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | A2, A3 |
|
||||
| `--max-loras-per-batch` | `8` | Type: int | A2, A3 |
|
||||
| `--max-loaded-loras` | `None` | Type: int | A2, A3 |
|
||||
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | A2, A3 |
|
||||
| `--lora-backend` | `csgmv` | `triton`,<br/>`csgmv`,<br/>`ascend`,<br/>`torch_native` | A2, A3 |
|
||||
| `--max-lora-chunk-size` | `16` | `16`, `32`,<br/> `64`, `128` | Special for GPU |
|
||||
|
||||
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------|-------------------|------------------------------------------------------------------------------------------------|:----------------:|
|
||||
| `--attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--prefill-attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--decode-attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--sampling-backend` | `None` | `pytorch`,<br/>`ascend` | A2, A3 |
|
||||
| `--grammar-backend` | `None` | `xgrammar` | A2, A3 |
|
||||
| `--mm-attention-backend` | `None` | `ascend_attn` | A2, A3 |
|
||||
| `--nsa-prefill-backend` | `flashmla_sparse` | `flashmla_sparse`,<br/> `flashmla_decode`,<br/>`fa3`,<br/> `tilelang`,<br/> `aiter` | Special for GPU |
|
||||
| `--nsa-decode-backend` | `fa3` | `flashmla_prefill`,<br/> `flashmla_kv`,<br/> `fa3`,<br/>`tilelang`,<br/> `aiter` | Special for GPU |
|
||||
| `--fp8-gemm-backend` | `auto` | `auto`,<br/> `deep_gemm`,<br/> `flashinfer_trtllm`,<br/>`flashinfer_cutlass`,<br/>`flashinfer_deepgemm`,<br/>`cutlass`,<br/> `triton`,<br/> `aiter` | Special for GPU |
|
||||
| `--disable-flashinfer-`<br/>`autotune` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## Speculative decoding
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------------------------------------------------|-----------|--------------------------|:----------------:|
|
||||
| `--speculative-algorithm` | `None` | `EAGLE3`,<br/> `NEXTN` | A2, A3 |
|
||||
| `--speculative-draft-model-path`<br/>`--speculative-draft-model` | `None` | Type: str | A2, A3 |
|
||||
| `--speculative-draft-model-`<br/>`revision` | `None` | Type: str | A2, A3 |
|
||||
| `--speculative-draft-load-format` | `None` | `auto` | A2, A3 |
|
||||
| `--speculative-num-steps` | `None` | Type: int | A2, A3 |
|
||||
| `--speculative-eagle-topk` | `None` | Type: int | A2, A3 |
|
||||
| `--speculative-num-draft-tokens` | `None` | Type: int | A2, A3 |
|
||||
| `--speculative-accept-`<br/>`threshold-single` | `1.0` | Type: float | Special for GPU |
|
||||
| `--speculative-accept-`<br/>`threshold-acc` | `1.0` | Type: float | Special for GPU |
|
||||
| `--speculative-token-map` | `None` | Type: str | A2, A3 |
|
||||
| `--speculative-attention-`<br/>`mode` | `prefill` | `prefill`,<br/> `decode` | A2, A3 |
|
||||
| `--speculative-moe-runner-`<br/>`backend` | `None` | `auto` | A2, A3 |
|
||||
| `--speculative-moe-a2a-`<br/>`backend` | `None` | `ascend_fuseep` | A2, A3 |
|
||||
| `--speculative-draft-attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--speculative-draft-model-quantization` | `None` | `unquant` | A2, A3 |
|
||||
|
||||
## Ngram speculative decoding
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------------------|------------|--------------------|:----------------:|
|
||||
| `--speculative-ngram-`<br/>`min-match-window-size` | `1` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`max-match-window-size` | `12` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`min-bfs-breadth` | `1` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`max-bfs-breadth` | `10` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`match-type` | `BFS` | `BFS`,<br/> `PROB` | Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion. |
|
||||
| `--speculative-ngram-`<br/>`max-trie-depth` | `18` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`capacity` | `10000000` | Type: int | Experimental |
|
||||
|
||||
## Expert parallelism
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------------------|-----------|---------------------------------------------|:----------------:|
|
||||
| `--expert-parallel-size`<br/>`--ep-size`<br/>`--ep` | `1` | Type: int | A2, A3 |
|
||||
| `--moe-a2a-backend` | `none` | `none`,<br/> `deepep`,<br/> `ascend_fuseep` | A2, A3 |
|
||||
| `--moe-runner-backend` | `auto` | `auto`, `triton` | A2, A3 |
|
||||
| `--flashinfer-mxfp4-`<br/>`moe-precision` | `default` | `default`,<br/> `bf16` | Special for GPU |
|
||||
| `--enable-flashinfer-`<br/>`allreduce-fusion` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--deepep-mode` | `auto` | `normal`, <br/>`low_latency`,<br/> `auto` | A2, A3 |
|
||||
| `--deepep-config` | `None` | Type: str | Special for GPU |
|
||||
| `--ep-num-redundant-experts` | `0` | Type: int | A2, A3 |
|
||||
| `--ep-dispatch-algorithm` | `None` | Type: str | A2, A3 |
|
||||
| `--init-expert-location` | `trivial` | Type: str | A2, A3 |
|
||||
| `--enable-eplb` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--eplb-algorithm` | `auto` | Type: str | A2, A3 |
|
||||
| `--eplb-rebalance-layers-`<br/>`per-chunk` | `None` | Type: int | A2, A3 |
|
||||
| `--eplb-min-rebalancing-`<br/>`utilization-threshold` | `1.0` | Type: float | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-mode` | `None` | Type: str | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-buffer-size` | `None` | Type: int | A2, A3 |
|
||||
| `--enable-expert-distribution-`<br/>`metrics` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--moe-dense-tp-size` | `None` | Type: int | A2, A3 |
|
||||
| `--elastic-ep-backend` | `None` | `none`, `mooncake` | Special for GPU |
|
||||
| `--mooncake-ib-device` | `None` | Type: str | Special for GPU |
|
||||
|
||||
## Mamba Cache
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------------|-----------|-----------------------------------------------|:----------------:|
|
||||
| `--max-mamba-cache-size` | `None` | Type: int | A2, A3 |
|
||||
| `--mamba-ssm-dtype` | `float32` | `float32`,<br/>`bfloat16`,<br/>`float16` | A2, A3 |
|
||||
| `--mamba-full-memory-ratio` | `0.9` | Type: float | A2, A3 |
|
||||
| `--mamba-scheduler-strategy` | `auto` | Only `auto`, `no_buffer` supported | A2, A3 |
|
||||
| `--mamba-track-interval` | `256` | Type: int | A2, A3 |
|
||||
|
||||
## Hierarchical cache
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------------|-----------------|---------------------------------------------------------------------|:----------------:|
|
||||
| `--enable-hierarchical-`<br/>`cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--hicache-ratio` | `2.0` | Type: float | A2, A3 |
|
||||
| `--hicache-size` | `0` | Type: int | A2, A3 |
|
||||
| `--hicache-write-policy` | `write_through` | Currently only `write_back` supported | A2, A3 |
|
||||
| `--hicache-io-backend` | `kernel` | `kernel_ascend`,<br/> `direct` | A2, A3 |
|
||||
| `--hicache-mem-layout` | `layer_first` | `page_first_direct`,<br/> `page_first_kv_split` | A2, A3 |
|
||||
| `--hicache-storage-`<br/>`backend` | `None` | `file` | A2, A3 |
|
||||
| `--hicache-storage-`<br/>`prefetch-policy` | `best_effort` | `best_effort`,<br/> `wait_complete`,<br/> `timeout` | Special for GPU |
|
||||
| `--hicache-storage-`<br/>`backend-extra-config` | `None` | Type: str | Special for GPU |
|
||||
|
||||
## LMCache
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------|----------|--------------------------------|:----------------:|
|
||||
| `--enable-lmcache` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## Offloading
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------|----------|-----------|:----------------:|
|
||||
| `--cpu-offload-gb` | `0` | Type: int | A2, A3 |
|
||||
| `--offload-group-size` | `-1` | Type: int | Planned |
|
||||
| `--offload-num-in-group` | `1` | Type: int | Planned |
|
||||
| `--offload-prefetch-step` | `1` | Type: int | Planned |
|
||||
| `--offload-mode` | `cpu` | Type: str | Planned |
|
||||
|
||||
## Args for multi-item scoring
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------|----------|-----------|:----------------:|
|
||||
| `--multi-item-scoring-delimiter` | `None` | Type: int | A2, A3 |
|
||||
|
||||
## Optimization/debug options
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--disable-radix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 |
|
||||
| `--cuda-graph-bs` | `None` | List[int] | A2, A3 |
|
||||
| `--disable-cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-cuda-graph-`<br/>`padding` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-profile-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-cudagraph-gc` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-nccl-nvls` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-tokenizer-`<br/>`batch-decode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-torch-`<br/>`symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-overlap`<br/>`-schedule` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-mixed-`<br/>`chunk` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-dp-attention` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-dp-lm-head` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-two-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | Planned |
|
||||
| `--enable-single-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--tbo-token-`<br/>`distribution-threshold` | `0.48` | Type: float | Planned |
|
||||
| `--enable-torch-`<br/>`compile` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | A2, A3 |
|
||||
| `--piecewise-cuda-`<br/>`graph-compiler` | `eager` | ["eager", "inductor"] | A2, A3 |
|
||||
| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 |
|
||||
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `None` | Type: int | A2, A3 |
|
||||
| `--torchao-config` | `` | Type: str | Special for GPU |
|
||||
| `--enable-nan-detection` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-p2p-check` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--triton-attention-`<br/>`reduce-in-fp32` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--triton-attention-`<br/>`num-kv-splits` | `8` | Type: int | Special for GPU |
|
||||
| `--triton-attention-`<br/>`split-tile-size` | `None` | Type: int | Special for GPU |
|
||||
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-shared-`<br/>`experts-fusion` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-chunked-`<br/>`prefix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | A2, A3 |
|
||||
| `--numa-node` | `None` | List[int] | A2, A3 |
|
||||
| `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | Planned |
|
||||
| `--rl-on-policy-target` | `None` | `fsdp` | Planned |
|
||||
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | Experimental |
|
||||
| `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-fused-qk-`<br/>`norm-rope` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## Dynamic batch tokenizer
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--enable-dynamic-`<br/>`batch-tokenizer` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--dynamic-batch-`<br/>`tokenizer-batch-size` | `32` | Type: int | A2, A3 |
|
||||
| `--dynamic-batch-`<br/>`tokenizer-batch-timeout` | `0.002` | Type: float | A2, A3 |
|
||||
|
||||
## Debug tensor dumps
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------------------------|----------|-----------|:----------------:|
|
||||
| `--debug-tensor-dump-`<br/>`output-folder` | `None` | Type: str | A2, A3 |
|
||||
| `--debug-tensor-dump-`<br/>`layers` | `None` | List[int] | A2, A3 |
|
||||
| `--debug-tensor-dump-`<br/>`input-file` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## PD disaggregation
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------------------------|------------|---------------------------------------|:----------------:|
|
||||
| `--disaggregation-mode` | `null` | `null`,<br/> `prefill`,<br/> `decode` | A2, A3 |
|
||||
| `--disaggregation-transfer-backend` | `mooncake` | `ascend` | A2, A3 |
|
||||
| `--disaggregation-bootstrap-port` | `8998` | Type: int | A2, A3 |
|
||||
| `--disaggregation-ib-device` | `None` | Type: str | Special for GPU |
|
||||
| `--disaggregation-decode-`<br/>`enable-offload-kvcache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--num-reserved-decode-tokens` | `512` | Type: int | A2, A3 |
|
||||
| `--disaggregation-decode-`<br/>`polling-interval` | `1` | Type: int | A2, A3 |
|
||||
|
||||
## Encode prefill disaggregation
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------------|--------------------|----------------------------------------------------------------|:----------------:|
|
||||
| `--encoder-only` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--language-only` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--encoder-transfer-backend` | `zmq_to_scheduler` | `zmq_to_scheduler`, <br/> `zmq_to_tokenizer`,<br/> `mooncake` | A2, A3 |
|
||||
| `--encoder-urls` | `[]` | List[str] | A2, A3 |
|
||||
|
||||
## Custom weight loader
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------------------------------------|----------|---------------------------------|:----------------:|
|
||||
| `--custom-weight-loader` | `None` | List[str] | A2, A3 |
|
||||
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-backend` | `nccl` | `transfer_engine`, <br/> `nccl` | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-start-seed-via-transfer-engine` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## For PD-Multiplexing
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-----------------------|----------|--------------------------------|:----------------:|
|
||||
| `--enable-pdmux` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--pdmux-config-path` | `None` | Type: str | Special for GPU |
|
||||
| `--sm-group-num` | `8` | Type: int | Special for GPU |
|
||||
|
||||
## For Multi-Modal
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-----------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--mm-max-concurrent-calls` | `32` | Type: int | A2, A3 |
|
||||
| `--mm-per-request-timeout` | `10.0` | Type: float | A2, A3 |
|
||||
| `--enable-broadcast-mm-`<br/>`inputs-process` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--mm-process-config` | `None` | Type: JSON / Dict | A2, A3 |
|
||||
| `--mm-enable-dp-encoder` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--limit-mm-data-per-request` | `None` | Type: JSON / Dict | A2, A3 |
|
||||
|
||||
## For checkpoint decryption
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--decrypted-config-file` | `None` | Type: str | A2, A3 |
|
||||
| `--decrypted-draft-config-file` | `None` | Type: str | A2, A3 |
|
||||
| `--enable-prefix-mm-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
|
||||
## Forward hooks
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------|----------|-----------------|:----------------:|
|
||||
| `--forward-hooks` | `None` | Type: JSON list | A2, A3 |
|
||||
|
||||
## Configuration file support
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------|----------|-----------|:----------------:|
|
||||
| `--config` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## Other Params
|
||||
|
||||
The following parameters are not supported because the third-party components that depend on are not compatible with the
|
||||
NPU, like Ktransformer, checkpoint-engine etc.
|
||||
|
||||
| Argument | Defaults | Options |
|
||||
|-------------------------------------------------------------------|-----------|---------------------------|
|
||||
| `--checkpoint-engine-` <br/> `wait-weights-` <br/> `before-ready` | `False` | bool flag (set to enable) |
|
||||
| `--kt-weight-path` | `None` | Type: str |
|
||||
| `--kt-method` | `AMXINT4` | Type: str |
|
||||
| `--kt-cpuinfer` | `None` | Type: int |
|
||||
| `--kt-threadpool-count` | `2` | Type: int |
|
||||
| `--kt-num-gpu-experts` | `None` | Type: int |
|
||||
| `--kt-max-deferred-`<br/>`experts-per-token` | `None` | Type: int |
|
||||
|
||||
The following parameters have some functional deficiencies on community
|
||||
|
||||
| Argument | Defaults | Options |
|
||||
|---------------------------------------|----------|--------------------------------|
|
||||
| `--enable-double-sparsity` | `False` | bool flag<br/> (set to enable) |
|
||||
| `--ds-channel-config-path` | `None` | Type: str |
|
||||
| `--ds-heavy-channel-num` | `32` | Type: int |
|
||||
| `--ds-heavy-token-num` | `256` | Type: int |
|
||||
| `--ds-heavy-channel-type` | `qk` | Type: str |
|
||||
| `--ds-sparse-decode-`<br/>`threshold` | `4096` | Type: int |
|
||||
| `--tool-server` | `None` | Type: str |
|
||||
110
third_party/sglang/docs/platforms/ascend/ascend_npu_support_models.md
vendored
Normal file
110
third_party/sglang/docs/platforms/ascend/ascend_npu_support_models.md
vendored
Normal file
@@ -0,0 +1,110 @@
|
||||
# Support Models on Ascend NPU
|
||||
|
||||
This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
|
||||
Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
|
||||
You are welcome to enable various models based on your business requirements.
|
||||
|
||||
## Large Language Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|--------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek-V3.2-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek-R1-0528-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek-V2-Lite-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-30B-A3B-Instruct-2507 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-32B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-0.6B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-235B-A22B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-7B-Instruct | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| QWQ-32B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/llama-2-7b | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/Llama-3.2-1B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| huihui-ai/grok-2 | Grok | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| HuggingFaceTB/SmolLM-1.7B | SmolLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/glm-4-9b-chat | GLM-4 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| XiaomiMiMo/MiMo-7B-RL | MiMo | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| arcee-ai/AFM-4.5B-Base | Arcee AFM-4.5B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Howeee/persimmon-8b-chat | Persimmon | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| inclusionAI/Ling-lite | Ling | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ibm-granite/granite-3.1-8b-instruct | Granite | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/dbrx-instruct | DBRX (Databricks) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| OpenBMB/MiniCPM3-4B | MiniCPM (v3, 4B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-K2-Thinking | Kimi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| eigen-ai-labs/gpt-oss-120b-bf16 | GPTOSS | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| allenai/OLMo-2-1124-7B-Instruct | OLMo | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| cyankiwi/MiniMax-M2-BF16 | MiniMax-M2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| upstage/SOLAR-10.7B-Instruct-v1.0 | Solar | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| bigcode/starcoder2-7b | StarCoder2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| arcee-ai/Trinity-Mini | Trinity (Nano, Mini) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Multimodal Language Models
|
||||
|
||||
| Models | Model Family (Variants) | A2 Supported | A3 Supported |
|
||||
|-----------------------------------------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| Qwen/Qwen2.5-VL-3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-VL-72B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-1B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-o-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma 3 (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| XiaomiMiMo/MiMo-VL-7B-RL | MiMo-VL (7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/llava-v1.6-34b | LLaVA (v1.5 & v1.6) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-next-72b | LLaVA-NeXT (8B, 72B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-onevision-qwen2-7b-ov | LLaVA-OneVision | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/Llama-3.2-11B-Vision-Instruct | Llama 3.2 Vision (11B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| rednote-hilab/dots.ocr | DotsVLM-OCR | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Embedding Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|-------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| intfloat/e5-mistral-7b-instruct | E5 (Llama/Mistral based) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| iic/gte_Qwen2-1.5B-instruct | GTE-Qwen2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-Embedding-8B | Qwen3-Embedding | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | GME (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/clip-vit-large-patch14-336 | CLIP | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| BAAI/bge-large-en-v1.5 | BGE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Reward Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|------------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
|
||||
| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Howeee/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Rerank Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|-------------------------|--------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| BAAI/bge-reranker-v2-m3 | BGE-Reranker | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
151
third_party/sglang/docs/platforms/ascend/mindspore_backend.md
vendored
Normal file
151
third_party/sglang/docs/platforms/ascend/mindspore_backend.md
vendored
Normal file
@@ -0,0 +1,151 @@
|
||||
# MindSpore Models
|
||||
|
||||
## Introduction
|
||||
|
||||
MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang.
|
||||
|
||||
## Requirements
|
||||
|
||||
MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
|
||||
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.
|
||||
|
||||
## Supported Models
|
||||
|
||||
Currently, the following models are supported:
|
||||
|
||||
- **Qwen3**: Dense and MoE models
|
||||
- **DeepSeek V3/R1**
|
||||
- *More models coming soon...*
|
||||
|
||||
## Installation
|
||||
|
||||
> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](ascend_npu.md) and then install `sgl-mindspore`:
|
||||
|
||||
```shell
|
||||
git clone https://github.com/mindspore-lab/sgl-mindspore.git
|
||||
cd sgl-mindspore
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
|
||||
## Run Model
|
||||
|
||||
Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.
|
||||
|
||||
### Offline infer
|
||||
|
||||
Use the following script for offline infer:
|
||||
|
||||
```python
|
||||
import sglang as sgl
|
||||
|
||||
# Initialize the engine with MindSpore backend
|
||||
llm = sgl.Engine(
|
||||
model_path="/path/to/your/model", # Local model path
|
||||
device="npu", # Use NPU device
|
||||
model_impl="mindspore", # MindSpore implementation
|
||||
attention_backend="ascend", # Attention backend
|
||||
tp_size=1, # Tensor parallelism size
|
||||
dp_size=1 # Data parallelism size
|
||||
)
|
||||
|
||||
# Generate text
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The capital of France is",
|
||||
"The future of AI is"
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0, "top_p": 0.9}
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"Prompt: {prompt}")
|
||||
print(f"Generated: {output['text']}")
|
||||
print("---")
|
||||
```
|
||||
|
||||
### Start server
|
||||
|
||||
Launch a server with MindSpore backend:
|
||||
|
||||
```bash
|
||||
# Basic server startup
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--tp-size 1 \
|
||||
--dp-size 1
|
||||
```
|
||||
|
||||
For distributed server with multiple nodes:
|
||||
|
||||
```bash
|
||||
# Multi-node distributed server
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--dist-init-addr 127.0.0.1:29500 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 4 \
|
||||
--dp-size 2
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
#### Debug Mode
|
||||
|
||||
Enable sglang debug logging by log-level argument.
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--log-level DEBUG
|
||||
```
|
||||
|
||||
Enable mindspore info and debug logging by setting environments.
|
||||
|
||||
```bash
|
||||
export GLOG_v=1 # INFO
|
||||
export GLOG_v=0 # DEBUG
|
||||
```
|
||||
|
||||
#### Explicitly select devices
|
||||
|
||||
Use the following environment variable to explicitly select the devices to use.
|
||||
|
||||
```shell
|
||||
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device
|
||||
```
|
||||
|
||||
#### Some communication environment issues
|
||||
|
||||
In case of some environment with special communication environment, users need set some environment variables.
|
||||
|
||||
```shell
|
||||
export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore
|
||||
```
|
||||
|
||||
#### Some dependencies of protobuf
|
||||
|
||||
In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch.
|
||||
|
||||
```shell
|
||||
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch
|
||||
```
|
||||
|
||||
## Support
|
||||
For MindSpore-specific issues:
|
||||
|
||||
- Refer to the [MindSpore documentation](https://www.mindspore.cn/)
|
||||
55
third_party/sglang/docs/platforms/ascend_npu_ring_sp_performance.md
vendored
Normal file
55
third_party/sglang/docs/platforms/ascend_npu_ring_sp_performance.md
vendored
Normal file
@@ -0,0 +1,55 @@
|
||||
# Ascend NPU Ring-SP Performance (Wan2.1-T2V-1.3B)
|
||||
|
||||
This page reports Ring-SP performance on Ascend NPU with `torch_npu==2.10.0`.
|
||||
|
||||
- Baseline config: `ulysses=1, ring=1` (short: `u1r1`)
|
||||
- Ring-SP config: `ulysses=1, ring=2` (short: `u1r2`)
|
||||
|
||||
## Benchmark Setup
|
||||
|
||||
- Model: `Wan2.1-T2V-1.3B-Diffusers`
|
||||
- Prompt: `"a cat is playing piano"`
|
||||
- Framework command: `sglang generate`
|
||||
- Runtime: `torch_npu==2.10.0`
|
||||
|
||||
## Generate Commands
|
||||
|
||||
### Baseline (`u1r1`)
|
||||
|
||||
```bash
|
||||
sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--prompt "a cat is playing piano" --num-gpus 1 --ring-degree 1 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
### Ring-SP (`u1r2`)
|
||||
|
||||
```bash
|
||||
sglang generate --model-path /nas/disk1/Wan2.1-T2V-1.3B-Diffusers \
|
||||
--prompt "a cat is playing piano" --num-gpus 2 --ring-degree 2 \
|
||||
--save-output
|
||||
```
|
||||
|
||||
## Benchmarks
|
||||
|
||||
Benchmark Disclaimer
|
||||
|
||||
These numbers are from one fixed setup and one prompt case. Actual performance may vary by model settings, environment, and workload.
|
||||
|
||||
### Stage Time Breakdown
|
||||
|
||||
| Stage / Metric | `u1r2` (s) | `u1r1` baseline (s) | Speedup |
|
||||
|---|---:|---:|---:|
|
||||
| InputValidation | 0.0003 | 0.0002 | 0.67x |
|
||||
| TextEncoding | 3.5936 | 3.5820 | 1.00x |
|
||||
| LatentPreparation | 0.0007 | 0.0055 | 7.86x |
|
||||
| TimestepPreparation | 0.0008 | 0.0007 | 0.88x |
|
||||
| Denoising | 121.2788 | 239.2580 | 1.97x |
|
||||
| Decoding | 13.8685 | 16.4969 | 1.19x |
|
||||
| **Total (Pixel data generated)** | **141.86** | **266.50** | **1.88x** |
|
||||
|
||||
## Summary
|
||||
|
||||
- With `torch_npu==2.10.0`, Ring-SP (`u1r2`) runs successfully on NPU for this case.
|
||||
- End-to-end generation time improves from `266.50s` to `141.86s` (`1.88x`).
|
||||
- The main gain comes from `DenoisingStage` (`1.97x`), while decoding also improves (`1.19x`).
|
||||
335
third_party/sglang/docs/platforms/cpu_server.md
vendored
Normal file
335
third_party/sglang/docs/platforms/cpu_server.md
vendored
Normal file
@@ -0,0 +1,335 @@
|
||||
# CPU Servers
|
||||
|
||||
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on CPU servers.
|
||||
SGLang is enabled and optimized on the CPUs equipped with Intel® AMX® Instructions,
|
||||
which are 4th generation or newer Intel® Xeon® Scalable Processors.
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
A list of popular LLMs are optimized and run efficiently on CPU,
|
||||
including the most notable open-source models like Llama series, Qwen series,
|
||||
and DeepSeek series like DeepSeek-R1 and DeepSeek-V3.1-Terminus.
|
||||
|
||||
| Model Name | BF16 | W8A8_INT8 | FP8 |
|
||||
|:---:|:---:|:---:|:---:|
|
||||
| DeepSeek-R1 | | [meituan/DeepSeek-R1-Channel-INT8](https://huggingface.co/meituan/DeepSeek-R1-Channel-INT8) | [deepseek-ai/DeepSeek-R1](https://huggingface.co/deepseek-ai/DeepSeek-R1) |
|
||||
| DeepSeek-V3.1-Terminus | | [IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8](https://huggingface.co/IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8) | [deepseek-ai/DeepSeek-V3.1-Terminus](https://huggingface.co/deepseek-ai/DeepSeek-V3.1-Terminus) |
|
||||
| Llama-3.2-3B | [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) | [RedHatAI/Llama-3.2-3B-quantized.w8a8](https://huggingface.co/RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8) | |
|
||||
| Llama-3.1-8B | [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) | [RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8](https://huggingface.co/RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8) | |
|
||||
| QwQ-32B | | [RedHatAI/QwQ-32B-quantized.w8a8](https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8) | |
|
||||
| DeepSeek-Distilled-Llama | | [RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8](https://huggingface.co/RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8) | |
|
||||
| Qwen3-235B | | | [Qwen/Qwen3-235B-A22B-FP8](https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8) |
|
||||
|
||||
**Note:** The model identifiers listed in the table above
|
||||
have been verified on 6th Gen Intel® Xeon® P-core platforms.
|
||||
|
||||
## Installation
|
||||
|
||||
### Install Using Docker
|
||||
|
||||
It is recommended to use Docker for setting up the SGLang environment.
|
||||
A [Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile) is provided to facilitate the installation.
|
||||
Replace `<secret>` below with your [HuggingFace access token](https://huggingface.co/docs/hub/en/security-tokens).
|
||||
|
||||
```bash
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
docker build -t sglang-cpu:latest -f xeon.Dockerfile .
|
||||
|
||||
# Initiate a docker container
|
||||
docker run \
|
||||
-it \
|
||||
--privileged \
|
||||
--ipc=host \
|
||||
--network=host \
|
||||
-v /dev/shm:/dev/shm \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 30000:30000 \
|
||||
-e "HF_TOKEN=<secret>" \
|
||||
sglang-cpu:latest /bin/bash
|
||||
```
|
||||
|
||||
### Install From Source
|
||||
|
||||
If you prefer to install SGLang in a bare metal environment,
|
||||
the setup process is as follows:
|
||||
|
||||
Please install the required packages and libraries beforehand if
|
||||
they are not already present on your system.
|
||||
You can refer to the Ubuntu-based installation commands in
|
||||
[the Dockerfile](https://github.com/sgl-project/sglang/blob/main/docker/xeon.Dockerfile#L11)
|
||||
for guidance.
|
||||
|
||||
1. Install `uv` package manager, then create and activate a virtual environment:
|
||||
|
||||
```bash
|
||||
# Taking '/opt' as the example uv env folder, feel free to change it as needed
|
||||
cd /opt
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
source $HOME/.local/bin/env
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
2. Create a config file to direct the installation channel
|
||||
(a.k.a. index-url) of `torch` related packages:
|
||||
|
||||
```bash
|
||||
vim .venv/uv.toml
|
||||
```
|
||||
|
||||
Press 'a' to enter insert mode of `vim`, paste the following content into the created file
|
||||
|
||||
```file
|
||||
[[index]]
|
||||
name = "torch"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "torchvision"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "torchaudio"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
[[index]]
|
||||
name = "triton"
|
||||
url = "https://download.pytorch.org/whl/cpu"
|
||||
|
||||
```
|
||||
|
||||
Save the file (in `vim`, press 'esc' to exit insert mode, then ':x+Enter'),
|
||||
and set it as the default `uv` config.
|
||||
|
||||
```bash
|
||||
export UV_CONFIG_FILE=/opt/.venv/uv.toml
|
||||
```
|
||||
|
||||
3. Clone the `sglang` source code and build the packages
|
||||
|
||||
```bash
|
||||
# Clone the SGLang code
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git checkout <YOUR-DESIRED-VERSION>
|
||||
|
||||
# Use dedicated toml file
|
||||
cd python
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
# Install SGLang dependent libs, and build SGLang main package
|
||||
uv pip install --upgrade pip setuptools
|
||||
uv pip install .
|
||||
|
||||
# Build the CPU backend kernels
|
||||
cd ../sgl-kernel
|
||||
cp pyproject_cpu.toml pyproject.toml
|
||||
uv pip install .
|
||||
```
|
||||
|
||||
4. Set the required environment variables
|
||||
|
||||
```bash
|
||||
export SGLANG_USE_CPU_ENGINE=1
|
||||
|
||||
# Set 'LD_LIBRARY_PATH' and 'LD_PRELOAD' to ensure the libs can be loaded by sglang processes
|
||||
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu
|
||||
export LD_PRELOAD=${LD_PRELOAD}:/opt/.venv/lib/libiomp5.so:${LD_LIBRARY_PATH}/libtcmalloc.so.4:${LD_LIBRARY_PATH}/libtbbmalloc.so.2
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- Note that the environment variable `SGLANG_USE_CPU_ENGINE=1`
|
||||
is required to enable the SGLang service with the CPU engine.
|
||||
|
||||
- If you encounter code compilation issues during the `sgl-kernel` building process,
|
||||
please check your `gcc` and `g++` versions and upgrade them if they are outdated.
|
||||
It is recommended to use `gcc-13` and `g++-13` as they have been verified
|
||||
in the official Docker container.
|
||||
|
||||
- The system library path is typically located in one of the following directories:
|
||||
`~/.local/lib/`, `/usr/local/lib/`, `/usr/local/lib64/`, `/usr/lib/`, `/usr/lib64/`
|
||||
and `/usr/lib/x86_64-linux-gnu/`. In the above example commands, `/usr/lib/x86_64-linux-gnu`
|
||||
is used. Please adjust the path according to your server configuration.
|
||||
|
||||
- It is recommended to add the following to your `~/.bashrc` file to
|
||||
avoid setting these variables every time you open a new terminal:
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
export SGLANG_USE_CPU_ENGINE=1
|
||||
export LD_LIBRARY_PATH=<YOUR-SYSTEM-LIBRARY-FOLDER>
|
||||
export LD_PRELOAD=<YOUR-LIBS-PATHS>
|
||||
```
|
||||
|
||||
## Launch of the Serving Engine
|
||||
|
||||
Example command to launch SGLang serving:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model <MODEL_ID_OR_PATH> \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--tp 6
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
1. For running W8A8 quantized models, please add the flag `--quantization w8a8_int8`.
|
||||
|
||||
2. The flag `--tp 6` specifies that tensor parallelism will be applied using 6 ranks (TP6).
|
||||
The number of TP specified is how many TP ranks will be used during the execution.
|
||||
On a CPU platform, a TP rank means a sub-NUMA cluster (SNC).
|
||||
Usually we can get the SNC information (How many available) from the Operating System with e.g. `lscpu` command.
|
||||
|
||||
If the specified TP rank number differs from the total SNC count,
|
||||
the system will automatically utilize the first `n` SNCs.
|
||||
Note that `n` cannot exceed the total SNC number, doing so will result in an error.
|
||||
|
||||
`SGLANG_CPU_OMP_THREADS_BIND` allows explicit control of CPU cores for each tensor parallel (TP) rank.
|
||||
|
||||
**example 1**: Run SGLang service with TP=6, using the first 40 cores of each SNC on a Xeon® 6980P server,
|
||||
which has 43-43-42 cores on the 3 SNCs of a socket, we should set:
|
||||
|
||||
```bash
|
||||
export SGLANG_CPU_OMP_THREADS_BIND="0-39|43-82|86-125|128-167|171-210|214-253"
|
||||
```
|
||||
This configuration is equivalent to:
|
||||
- rank 0: `numactl -C 0-39 -m 0`
|
||||
- rank 1: `numactl -C 43-82 -m 1`
|
||||
- rank 2: `numactl -C 86-125 -m 2`
|
||||
- rank 3: `numactl -C 128-167 -m 3`
|
||||
- rank 4: `numactl -C 171-210 -m 4`
|
||||
- rank 5: `numactl -C 214-253 -m 5`
|
||||
|
||||
|
||||
**example 2**: Run SGLang service with TP=2, using 96 cores cross 3 SNCs on a Xeon® 6972P server,
|
||||
which has 32-32-32 cores on the 3 SNCs in a socket, we should set:
|
||||
```bash
|
||||
export SGLANG_CPU_OMP_THREADS_BIND="0-95|96-191"
|
||||
```
|
||||
This configuration is equivalent to:
|
||||
- rank 0: `numactl -C 0-95 -m 0-2`
|
||||
- rank 1: `numactl -C 96-191 -m 3-5`
|
||||
|
||||
Please beware that with SGLANG_CPU_OMP_THREADS_BIND set,
|
||||
the available memory amounts of the ranks may not be determined in prior.
|
||||
You may need to set proper `--max-total-tokens` to avoid the out-of-memory error.
|
||||
|
||||
3. For optimizing decoding with torch.compile, please add the flag `--enable-torch-compile`.
|
||||
To specify the maximum batch size when using `torch.compile`, set the flag `--torch-compile-max-bs`.
|
||||
For example, `--enable-torch-compile --torch-compile-max-bs 4` means using `torch.compile`
|
||||
and setting the maximum batch size to 4.
|
||||
|
||||
4. A warmup step is automatically triggered when the service is started.
|
||||
The server is ready when you see the log `The server is fired up and ready to roll!`.
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script.
|
||||
Run the command in another terminal. An example command would be:
|
||||
|
||||
```bash
|
||||
python -m sglang.bench_serving \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1 \
|
||||
--request-rate inf \
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
|
||||
Detailed parameter descriptions are available via the command:
|
||||
|
||||
```bash
|
||||
python -m sglang.bench_serving -h
|
||||
```
|
||||
|
||||
Additionally, requests can be formatted using
|
||||
[the OpenAI Completions API](https://docs.sglang.io/basic_usage/openai_api_completions.html)
|
||||
and sent via the command line (e.g., using `curl`) or through your own scripts.
|
||||
|
||||
## Example Usage Commands
|
||||
|
||||
Large Language Models can range from fewer than 1 billion to several hundred billion parameters.
|
||||
Dense models larger than 20B are expected to run on flagship 6th Gen Intel® Xeon® processors
|
||||
with dual sockets and a total of 6 sub-NUMA clusters. Dense models of approximately 10B parameters or fewer,
|
||||
or MoE (Mixture of Experts) models with fewer than 10B activated parameters, can run on more common
|
||||
4th generation or newer Intel® Xeon® processors, or utilize a single socket of the flagship 6th Gen Intel® Xeon® processors.
|
||||
|
||||
### Example: Running DeepSeek-V3.1-Terminus
|
||||
|
||||
An example command to launch service of W8A8_INT8 DeepSeek-V3.1-Terminus on a Xeon® 6980P server:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model IntervitensInc/DeepSeek-V3.1-Terminus-Channel-int8 \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--quantization w8a8_int8 \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 4 \
|
||||
--tp 6
|
||||
```
|
||||
|
||||
Similarly, an example command to launch service of FP8 DeepSeek-V3.1-Terminus would be:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model deepseek-ai/DeepSeek-V3.1-Terminus \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 4 \
|
||||
--tp 6
|
||||
```
|
||||
|
||||
Note: Please set `--torch-compile-max-bs` to the maximum desired batch size for your deployment,
|
||||
which can be up to 16. The value `4` in the examples is illustrative.
|
||||
|
||||
### Example: Running Llama-3.2-3B
|
||||
|
||||
An example command to launch service of Llama-3.2-3B with BF16 precision:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model meta-llama/Llama-3.2-3B-Instruct \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
The example command to launch service of W8A8_INT8 version of Llama-3.2-3B:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model RedHatAI/Llama-3.2-3B-quantized.w8a8 \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device cpu \
|
||||
--quantization w8a8_int8 \
|
||||
--host 0.0.0.0 \
|
||||
--enable-torch-compile \
|
||||
--torch-compile-max-bs 16 \
|
||||
--tp 2
|
||||
```
|
||||
|
||||
Note: The `--torch-compile-max-bs` and `--tp` settings are examples that should be adjusted for your setup.
|
||||
For instance, use `--tp 3` to utilize 1 socket with 3 sub-NUMA clusters on an Intel® Xeon® 6980P server.
|
||||
|
||||
Once the server have been launched, you can test it using the `bench_serving` command or create
|
||||
your own commands or scripts following [the benchmarking example](#benchmarking-with-requests).
|
||||
25
third_party/sglang/docs/platforms/mthreads_gpu.md
vendored
Normal file
25
third_party/sglang/docs/platforms/mthreads_gpu.md
vendored
Normal file
@@ -0,0 +1,25 @@
|
||||
# Moore Threads GPUs
|
||||
|
||||
This document describes how run SGLang on Moore Threads GPUs. If you encounter issues or have questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
## Install SGLang
|
||||
|
||||
You can install SGLang using one of the methods below.
|
||||
|
||||
### Install from Source
|
||||
|
||||
```bash
|
||||
# Use the default branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
|
||||
# Compile sgl-kernel
|
||||
pip install --upgrade pip
|
||||
cd sgl-kernel
|
||||
python setup_musa.py install
|
||||
|
||||
# Install sglang python package
|
||||
cd ..
|
||||
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
|
||||
pip install -e "python[all_musa]"
|
||||
```
|
||||
80
third_party/sglang/docs/platforms/nvidia_jetson.md
vendored
Normal file
80
third_party/sglang/docs/platforms/nvidia_jetson.md
vendored
Normal file
@@ -0,0 +1,80 @@
|
||||
# NVIDIA Jetson Orin
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure the following:
|
||||
|
||||
- [**NVIDIA Jetson AGX Orin Devkit**](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/) is set up with **JetPack 6.1** or later.
|
||||
- **CUDA Toolkit** and **cuDNN** are installed.
|
||||
- Verify that the Jetson AGX Orin is in **high-performance mode**:
|
||||
```bash
|
||||
sudo nvpmodel -m 0
|
||||
```
|
||||
* * * * *
|
||||
## Installing and running SGLang with Jetson Containers
|
||||
Clone the jetson-containers github repository:
|
||||
```
|
||||
git clone https://github.com/dusty-nv/jetson-containers.git
|
||||
```
|
||||
Run the installation script:
|
||||
```
|
||||
bash jetson-containers/install.sh
|
||||
```
|
||||
Build the container image:
|
||||
```
|
||||
jetson-containers build sglang
|
||||
```
|
||||
Run the container:
|
||||
```
|
||||
jetson-containers run $(autotag sglang)
|
||||
```
|
||||
Or you can also manually run a container with this command:
|
||||
```
|
||||
docker run --runtime nvidia -it --rm --network=host IMAGE_NAME
|
||||
```
|
||||
* * * * *
|
||||
|
||||
Running Inference
|
||||
-----------------------------------------
|
||||
|
||||
Launch the server:
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path deepseek-ai/DeepSeek-R1-Distill-Llama-8B \
|
||||
--device cuda \
|
||||
--dtype half \
|
||||
--attention-backend flashinfer \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 8192
|
||||
```
|
||||
The quantization and limited context length (`--dtype half --context-length 8192`) are due to the limited computational resources in [Nvidia jetson kit](https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/). A detailed explanation can be found in [Server Arguments](../advanced_features/server_arguments.md).
|
||||
|
||||
After launching the engine, refer to [Chat completions](https://docs.sglang.io/basic_usage/openai_api_completions.html#Usage) to test the usability.
|
||||
* * * * *
|
||||
Running quantization with TorchAO
|
||||
-------------------------------------
|
||||
TorchAO is suggested to NVIDIA Jetson Orin.
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model-path meta-llama/Meta-Llama-3.1-8B-Instruct \
|
||||
--device cuda \
|
||||
--dtype bfloat16 \
|
||||
--attention-backend flashinfer \
|
||||
--mem-fraction-static 0.8 \
|
||||
--context-length 8192 \
|
||||
--torchao-config int4wo-128
|
||||
```
|
||||
This enables TorchAO's int4 weight-only quantization with a 128-group size. The usage of `--torchao-config int4wo-128` is also for memory efficiency.
|
||||
|
||||
|
||||
* * * * *
|
||||
Structured output with XGrammar
|
||||
-------------------------------
|
||||
Please refer to [SGLang doc structured output](../advanced_features/structured_outputs.ipynb).
|
||||
* * * * *
|
||||
|
||||
Thanks to the support from [Nurgaliyev Shakhizat](https://github.com/shahizat), [Dustin Franklin](https://github.com/dusty-nv) and [Johnny Núñez Cano](https://github.com/johnnynunez).
|
||||
|
||||
References
|
||||
----------
|
||||
- [NVIDIA Jetson AGX Orin Documentation](https://developer.nvidia.com/embedded/jetson-agx-orin)
|
||||
477
third_party/sglang/docs/platforms/tpu.md
vendored
Normal file
477
third_party/sglang/docs/platforms/tpu.md
vendored
Normal file
@@ -0,0 +1,477 @@
|
||||
# TPU
|
||||
|
||||
SGLang supports high-performance TPU inference through the SGLang-JAX backend, which is specifically optimized for Google Cloud TPUs. The JAX-based implementation delivers exceptional throughput and low latency for Large Language Model (LLM) serving workloads on TPU hardware.
|
||||
|
||||
For TPU-specific issues or feature requests, please visit the [sglang-jax GitHub issues page](https://github.com/sgl-project/sglang-jax/issues).
|
||||
|
||||
**NOTE:** SGLang TPU support is implemented via the SGLang-JAX backend, a dedicated JAX-based inference engine maintained as a separate repository at [https://github.com/sgl-project/sglang-jax](https://github.com/sgl-project/sglang-jax).
|
||||
|
||||
## System Requirements
|
||||
|
||||
### Supported TPU Hardware
|
||||
|
||||
| TPU Type | HBM Memory | Availability |
|
||||
|----------|-----------|--------------|
|
||||
| TPU v6e | 32 GB | Google Cloud |
|
||||
| TPU v7 | 96 GB per core | Google Cloud |
|
||||
|
||||
### Software Requirements
|
||||
|
||||
- **Python:** 3.12 or higher
|
||||
- **JAX:** Latest version with TPU support
|
||||
- **Environment:** Google Cloud TPU VM or compatible TPU runtime
|
||||
- **Optional:** SkyPilot for simplified cloud deployment
|
||||
|
||||
## Feature Support Matrix
|
||||
|
||||
SGLang-JAX provides comprehensive TPU-optimized features for production LLM serving:
|
||||
|
||||
| Feature | Support Status | Description |
|
||||
|---------|---------------|-------------|
|
||||
| High-Throughput Continuous Batching | ✅ | Dynamic request batching for maximum TPU utilization |
|
||||
| Radix Tree KV Cache | ✅ | Memory-efficient prefix sharing between requests |
|
||||
| FlashAttention Backend | ✅ | TPU-optimized attention kernel for long sequences |
|
||||
| Tensor Parallelism | ✅ | Distribute models across multiple TPU cores |
|
||||
| Paged Attention | ✅ | Flexible KV cache management with paging |
|
||||
| Speculative Decoding (EAGLE/EAGLE3) | ✅ | 20-40% throughput improvement for compatible models |
|
||||
| Chunked Prefill | ✅ | Mixed prefill-decode batching |
|
||||
| OpenAI-Compatible API | ✅ | Drop-in replacement for OpenAI API |
|
||||
| Data Parallel Attention | 🚧 | In development - Attention computation with data parallelism |
|
||||
| Quantization | 🚧 | In development - Model quantization for reduced memory usage |
|
||||
| Multi-LoRA | 🚧 | In development - Serve multiple LoRA adapters simultaneously |
|
||||
|
||||
### Attention Backend Comparison
|
||||
|
||||
| Backend | Paged Attention | Spec Decoding | MLA | Sliding Window |
|
||||
|---------|----------------|---------------|-----|----------------|
|
||||
| FlashAttention (fa) | ✅ | ✅ | ❌ | ✅ |
|
||||
| Native | ❌ | ❌ | ❌ | ❌ |
|
||||
|
||||
**NOTE:** FlashAttention backend is recommended for production workloads due to superior memory efficiency and performance.
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
The following models have been tested and optimized for TPU deployment:
|
||||
|
||||
| Model Family | Performance Status |
|
||||
|--------------|-------------------|
|
||||
| [Qwen 3](https://huggingface.co/Qwen) | ⭐ Recommended for production |
|
||||
| [Qwen 3 MoE](https://huggingface.co/Qwen) | ⭐ Best performance |
|
||||
| [Qwen 2](https://huggingface.co/Qwen) | Needs improvement |
|
||||
| [Qwen 2 MoE](https://huggingface.co/Qwen) | Needs improvement |
|
||||
| [Qwen 1.5](https://huggingface.co/Qwen) | Needs improvement |
|
||||
| [Llama/LLaMA](https://huggingface.co/meta-llama) | Needs improvement |
|
||||
| [Grok-2](https://huggingface.co/xai-org) | Needs improvement |
|
||||
| [Gemma 2](https://huggingface.co/google) | Verified on TPU |
|
||||
| Bailing MoE | Needs improvement |
|
||||
|
||||
## Installation
|
||||
|
||||
### Method 1: Using PyPI (Recommended)
|
||||
|
||||
```bash
|
||||
pip install sglang-jax
|
||||
```
|
||||
|
||||
### Method 2: From Source
|
||||
|
||||
```bash
|
||||
git clone https://github.com/sgl-project/sglang-jax
|
||||
cd sglang-jax
|
||||
uv venv --python 3.12 && source .venv/bin/activate
|
||||
uv pip install -e "python[all]"
|
||||
```
|
||||
|
||||
### Method 3: Using Docker
|
||||
|
||||
**NOTE:** Docker support for TPU is currently under development. Please use PyPI or source installation methods.
|
||||
|
||||
### Method 4: Cloud TPU with SkyPilot
|
||||
|
||||
[SkyPilot](https://github.com/skypilot-org/skypilot) provides simplified deployment on Google Cloud TPU:
|
||||
|
||||
1. Install SkyPilot and configure GCP access (see [SkyPilot documentation](https://skypilot.readthedocs.io/))
|
||||
|
||||
2. Create a SkyPilot configuration file:
|
||||
|
||||
<details>
|
||||
<summary>SkyPilot YAML: <code>sglang-jax.sky.yaml</code></summary>
|
||||
|
||||
```yaml
|
||||
# sglang-jax.sky.yaml
|
||||
resources:
|
||||
accelerators: tpu-v6e-4
|
||||
accelerator_args:
|
||||
tpu_vm: True
|
||||
runtime_version: v2-alpha-tpuv6e
|
||||
|
||||
run: |
|
||||
git clone https://github.com/sgl-project/sglang-jax.git
|
||||
cd sglang-jax
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install -e "python[all]"
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
3. Launch your TPU cluster:
|
||||
|
||||
```bash
|
||||
# Standard deployment
|
||||
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp
|
||||
|
||||
# With spot instances for cost savings
|
||||
sky launch -c sglang-jax sglang-jax.sky.yaml --infra=gcp --use-spot
|
||||
```
|
||||
|
||||
## Launch of the Serving Engine
|
||||
|
||||
### Basic Example: Qwen-7B
|
||||
|
||||
```bash
|
||||
JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen-7B-Chat \
|
||||
--trust-remote-code \
|
||||
--dist-init-addr=0.0.0.0:10011 \
|
||||
--nnodes=1 \
|
||||
--tp-size=4 \
|
||||
--device=tpu \
|
||||
--random-seed=3 \
|
||||
--node-rank=0 \
|
||||
--mem-fraction-static=0.8 \
|
||||
--max-prefill-tokens=8192 \
|
||||
--download-dir=/tmp \
|
||||
--dtype=bfloat16 \
|
||||
--skip-server-warmup \
|
||||
--host 0.0.0.0 \
|
||||
--port 30000
|
||||
```
|
||||
|
||||
**Key Parameters Explained:**
|
||||
|
||||
1. `JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache` - Enables JIT compilation caching to accelerate server startup on subsequent runs
|
||||
2. `--tp-size=4` - Tensor parallelism size; match this to your TPU core count (typically 1, 4, or 8)
|
||||
3. `--device=tpu` - Specifies TPU device (this is the default for sglang-jax)
|
||||
4. `--dtype=bfloat16` - Uses bfloat16 precision, which TPUs are optimized for
|
||||
5. `--mem-fraction-static=0.8` - Allocates 80% of TPU HBM for static memory (adjustable from 0.2 to 0.9)
|
||||
6. `--max-prefill-tokens=8192` - Maximum number of tokens processed in the prefill phase
|
||||
|
||||
### High-Performance Configuration: Qwen3-8B
|
||||
|
||||
For production workloads with optimal throughput:
|
||||
|
||||
```bash
|
||||
python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen3-8B \
|
||||
--trust-remote-code \
|
||||
--tp-size=4 \
|
||||
--device=tpu \
|
||||
--mem-fraction-static=0.8 \
|
||||
--chunked-prefill-size=2048 \
|
||||
--dtype=bfloat16 \
|
||||
--max-running-requests=256 \
|
||||
--page-size=128 \
|
||||
--attention-backend=fa
|
||||
```
|
||||
|
||||
### Advanced: Speculative Decoding (EAGLE3)
|
||||
|
||||
Speculative decoding can improve throughput by 20-40% for compatible models:
|
||||
|
||||
```bash
|
||||
python3 -u -m sgl_jax.launch_server \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--trust-remote-code \
|
||||
--device=tpu \
|
||||
--tp-size=4 \
|
||||
--mem-fraction-static=0.8 \
|
||||
--max-prefill-tokens=4096 \
|
||||
--attention-backend=fa \
|
||||
--dtype=bfloat16 \
|
||||
--port=30000 \
|
||||
--host=0.0.0.0 \
|
||||
--disable-overlap-schedule \
|
||||
--speculative-algorithm=EAGLE3 \
|
||||
--speculative-draft-model-path=AngelSlim/Qwen3-32B_eagle3 \
|
||||
--page-size=64 \
|
||||
--speculative-eagle-topk=1 \
|
||||
--speculative-num-steps=3 \
|
||||
--speculative-num-draft-tokens=4
|
||||
```
|
||||
|
||||
**NOTE:** Speculative decoding is currently supported for Qwen3 and LLaMA model families. See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration guidance.
|
||||
|
||||
|
||||
### Multi-Node Distributed Serving
|
||||
|
||||
For large models requiring multiple TPU VMs:
|
||||
|
||||
```bash
|
||||
# Node 0 (coordinator)
|
||||
python3 -m sgl_jax.launch_server \
|
||||
--model-path MODEL_PATH \
|
||||
--dist-init-addr=NODE0_IP:10011 \
|
||||
--nnodes=2 \
|
||||
--node-rank=0 \
|
||||
--tp-size=8 \
|
||||
[other parameters...]
|
||||
|
||||
# Node 1 (worker)
|
||||
python3 -m sgl_jax.launch_server \
|
||||
--model-path MODEL_PATH \
|
||||
--dist-init-addr=NODE0_IP:10011 \
|
||||
--nnodes=2 \
|
||||
--node-rank=1 \
|
||||
--tp-size=8 \
|
||||
[other parameters...]
|
||||
```
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
### Throughput Testing
|
||||
|
||||
Basic throughput benchmark:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_serving \
|
||||
--backend sgl-jax \
|
||||
--dataset-name random \
|
||||
--num-prompts=100 \
|
||||
--random-input=512 \
|
||||
--random-output=128 \
|
||||
--max-concurrency=8 \
|
||||
--random-range-ratio=1 \
|
||||
--warmup-requests=0
|
||||
```
|
||||
|
||||
### Latency Testing
|
||||
|
||||
Measure single-batch latency:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_one_batch_server \
|
||||
--base-url http://127.0.0.1:30000 \
|
||||
--model-path Qwen/Qwen-7B-Chat \
|
||||
--batch-size=32 \
|
||||
--input-len=256 \
|
||||
--output-len=32
|
||||
```
|
||||
|
||||
### Comprehensive Benchmark Script
|
||||
|
||||
For systematic performance evaluation across different configurations:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
backend=${1:-sgl-jax}
|
||||
num_prompts_per_concurrency=3
|
||||
input_seq_lens=(1024 4096 8192)
|
||||
output_seq_lens=(1 1024)
|
||||
max_concurrencies=(8 16 32 64 128 256)
|
||||
|
||||
for input_seq_len in "${input_seq_lens[@]}"; do
|
||||
for output_seq_len in "${output_seq_lens[@]}"; do
|
||||
echo "======================================="
|
||||
echo "Testing ISL/OSL: $input_seq_len/$output_seq_len"
|
||||
echo "======================================="
|
||||
for max_concurrency in "${max_concurrencies[@]}"; do
|
||||
num_prompts=$((num_prompts_per_concurrency * max_concurrency))
|
||||
python3 -m sgl_jax.bench_serving \
|
||||
--backend ${backend} \
|
||||
--dataset-name random \
|
||||
--num-prompts ${num_prompts} \
|
||||
--random-input ${input_seq_len} \
|
||||
--random-output ${output_seq_len} \
|
||||
--max-concurrency ${max_concurrency} \
|
||||
--random-range-ratio 1 \
|
||||
--disable-ignore-eos \
|
||||
--warmup-requests 0
|
||||
done
|
||||
done
|
||||
done
|
||||
```
|
||||
|
||||
For detailed help on all benchmark parameters:
|
||||
|
||||
```bash
|
||||
python3 -m sgl_jax.bench_serving --help
|
||||
```
|
||||
|
||||
See the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for advanced benchmarking techniques and profiling with JAX Profiler.
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Memory Optimization
|
||||
|
||||
**Reduce memory usage:**
|
||||
- Lower `--mem-fraction-static` (from 0.8 → 0.5 → 0.3)
|
||||
- Decrease `--max-prefill-tokens` (from 16384 → 8192 → 4096)
|
||||
- Reduce `--max-running-requests`
|
||||
|
||||
**Handle OOM errors:**
|
||||
- Start with conservative memory settings (`--mem-fraction-static=0.5`)
|
||||
- Gradually increase until you find the optimal balance
|
||||
- Increase `--page-size` for better memory locality (1 → 16 → 64 → 128)
|
||||
|
||||
### Throughput Optimization
|
||||
|
||||
To maximize tokens per second:
|
||||
|
||||
- Use FlashAttention backend: `--attention-backend=fa`
|
||||
- Enable speculative decoding (EAGLE3) for Qwen3 models (20-40% improvement)
|
||||
- Increase `--max-running-requests` to 256+
|
||||
- Set `--mem-fraction-static` to 0.8+ (if memory allows)
|
||||
- Use larger page sizes (64-128)
|
||||
- Enable chunked prefill: `--chunked-prefill-size=2048`
|
||||
|
||||
### Latency Optimization
|
||||
|
||||
To minimize time-to-first-token (TTFT) and inter-token latency:
|
||||
|
||||
- Reduce `--page-size` to 1-4
|
||||
- Lower `--max-running-requests` (16-32) for smaller batches
|
||||
- Reduce `--chunked-prefill-size`
|
||||
- Use conservative memory settings to avoid GC pauses
|
||||
|
||||
### TPU-Specific Optimizations
|
||||
|
||||
1. **JIT Compilation Cache:**
|
||||
```bash
|
||||
export JAX_COMPILATION_CACHE_DIR=/tmp/jit_cache
|
||||
```
|
||||
Always set this environment variable to cache compiled kernels and accelerate server startup.
|
||||
|
||||
2. **Data Type Optimization:**
|
||||
Use `--dtype=bfloat16` for TPU native optimization. TPUs are specifically designed for bfloat16 computations.
|
||||
|
||||
3. **Tensor Parallelism:**
|
||||
Match `--tp-size` to your TPU core configuration (1, 4, or 8) for optimal model distribution.
|
||||
|
||||
4. **Attention Backend:**
|
||||
Always use `--attention-backend=fa` (FlashAttention) for production workloads.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### OOM (Out of Memory) Errors
|
||||
|
||||
If you encounter out-of-memory errors:
|
||||
|
||||
1. Reduce `--mem-fraction-static` from 0.8 to 0.5 or lower
|
||||
2. Decrease `--max-prefill-tokens` from 8192 to 4096 or 2048
|
||||
3. Lower `--max-running-requests` to reduce concurrent batch size
|
||||
4. Increase `--page-size` for better memory layout efficiency
|
||||
|
||||
### Compilation Long-Time
|
||||
|
||||
If the server takes too long to start:
|
||||
|
||||
1. Ensure `JAX_COMPILATION_CACHE_DIR` is properly set
|
||||
2. Understand that the first run requires JIT compilation (this is normal)
|
||||
3. Subsequent runs will be significantly faster with cached compilations
|
||||
4. Consider using `--skip-server-warmup` to defer compilation until first request
|
||||
|
||||
### Low Throughput
|
||||
|
||||
If you're not achieving expected throughput:
|
||||
|
||||
1. Verify `--tp-size` matches your TPU core configuration
|
||||
2. Check that `--attention-backend=fa` is enabled
|
||||
3. Increase `--max-running-requests` to enable larger batch formation
|
||||
4. Consider enabling speculative decoding for compatible models
|
||||
5. Ensure memory settings allow for sufficient batch sizes
|
||||
|
||||
### Connection Issues
|
||||
|
||||
If clients cannot connect to the server:
|
||||
|
||||
1. Ensure `--host=0.0.0.0` for external access (not just `127.0.0.1`)
|
||||
2. Verify firewall rules allow traffic on the specified port (default: 30000)
|
||||
3. Check that the server process is running: `curl http://localhost:30000/health`
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Speculative Decoding
|
||||
|
||||
SGLang-JAX supports EAGLE and EAGLE3 speculative decoding algorithms for Qwen3 and LLaMA model families. Speculative decoding can improve throughput by 20-40% without affecting output quality.
|
||||
|
||||
See the [Speculative Decoding documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md) for detailed configuration and supported model combinations.
|
||||
|
||||
### Chunked Prefill
|
||||
|
||||
Enable mixed prefill-decode batching for better TPU utilization:
|
||||
|
||||
```bash
|
||||
--chunked-prefill-size=2048 --enable-mixed-chunk
|
||||
```
|
||||
|
||||
This allows the scheduler to mix prefill operations with decode operations in the same batch, improving overall throughput.
|
||||
|
||||
### Custom Attention Backends
|
||||
|
||||
SGLang-JAX supports a plugin-based attention backend system. You can implement custom attention kernels optimized for specific use cases.
|
||||
|
||||
See the [Attention Backend documentation](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/attention_backend.md) for implementation details.
|
||||
|
||||
### Environment Verification
|
||||
|
||||
Verify your TPU setup before deploying:
|
||||
|
||||
```bash
|
||||
python -c "from sgl_jax import check_env; check_env.check_env()"
|
||||
```
|
||||
|
||||
This command checks:
|
||||
- Installed package versions
|
||||
- TPU device availability and specifications
|
||||
- System resources and configuration
|
||||
- Compatibility of settings
|
||||
|
||||
## Contributing
|
||||
|
||||
We welcome contributions to improve TPU support in SGLang-JAX!
|
||||
|
||||
### Areas for Contribution
|
||||
|
||||
**Check the [Development Roadmap](https://github.com/sgl-project/sglang-jax/issues/190)** to see planned features and find opportunities to contribute new functionality.
|
||||
|
||||
Current contribution areas include:
|
||||
|
||||
- Performance optimizations for specific TPU generations
|
||||
- Support for additional model architectures
|
||||
- Documentation improvements and examples
|
||||
- Bug reports and fixes
|
||||
- Benchmark results and performance analysis
|
||||
|
||||
### How to Contribute
|
||||
|
||||
1. Visit the [sglang-jax repository](https://github.com/sgl-project/sglang-jax)
|
||||
2. Read the [Contribution Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/contribution_guide.md)
|
||||
3. Join the [SGL-JAX Slack community](https://sgl-fru7574.slack.com/archives/C09EBE5HT5X) for discussions
|
||||
4. Report issues at [sglang-jax/issues](https://github.com/sgl-project/sglang-jax/issues)
|
||||
|
||||
### Testing on TPU
|
||||
|
||||
For contributors who need TPU access for testing:
|
||||
|
||||
- Refer to the [TPU Resources Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/tpu_resources_guide.md) for information on accessing TPU hardware
|
||||
- Use SkyPilot with spot instances for cost-effective testing
|
||||
- Follow the [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md) for performance validation
|
||||
|
||||
## References
|
||||
|
||||
### Documentation
|
||||
|
||||
- [SGLang-JAX Repository](https://github.com/sgl-project/sglang-jax)
|
||||
- [SGLang-JAX Installation Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/get_started/install.md)
|
||||
- [Qwen Models Quick Start](https://github.com/sgl-project/sglang-jax/blob/main/docs/basic_usage/qwen.md)
|
||||
- [Benchmark and Profiling Guide](https://github.com/sgl-project/sglang-jax/blob/main/docs/developer_guide/benchmark_and_profiling.md)
|
||||
- [Speculative Decoding](https://github.com/sgl-project/sglang-jax/blob/main/docs/features/speculative_decoding.md)
|
||||
|
||||
### External Resources
|
||||
|
||||
- [JAX Documentation](https://jax.readthedocs.io/)
|
||||
- [Google Cloud TPU Documentation](https://cloud.google.com/tpu/docs)
|
||||
- [SkyPilot Documentation](https://skypilot.readthedocs.io/)
|
||||
92
third_party/sglang/docs/platforms/xpu.md
vendored
Normal file
92
third_party/sglang/docs/platforms/xpu.md
vendored
Normal file
@@ -0,0 +1,92 @@
|
||||
# XPU
|
||||
|
||||
The document addresses how to set up the [SGLang](https://github.com/sgl-project/sglang) environment and run LLM inference on Intel GPU, [see more context about Intel GPU support within PyTorch ecosystem](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html).
|
||||
|
||||
Specifically, SGLang is optimized for [Intel® Arc™ Pro B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/242616/intel-arc-pro-b-series-graphics.html) and [
|
||||
Intel® Arc™ B-Series Graphics](https://www.intel.com/content/www/us/en/ark/products/series/240391/intel-arc-b-series-graphics.html).
|
||||
|
||||
## Optimized Model List
|
||||
|
||||
A list of LLMs have been optimized on Intel GPU, and more are on the way:
|
||||
|
||||
| Model Name | BF16 |
|
||||
|:---:|:---:|
|
||||
| Llama-3.2-3B | [meta-llama/Llama-3.2-3B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct) |
|
||||
| Llama-3.1-8B | [meta-llama/Llama-3.1-8B-Instruct](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct) |
|
||||
| Qwen2.5-1.5B | [Qwen/Qwen2.5-1.5B](https://huggingface.co/Qwen/Qwen2.5-1.5B) |
|
||||
|
||||
**Note:** The model identifiers listed in the table above
|
||||
have been verified on [Intel® Arc™ B580 Graphics](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html).
|
||||
|
||||
## Installation
|
||||
|
||||
### Install From Source
|
||||
|
||||
Currently SGLang XPU only supports installation from source. Please refer to ["Getting Started on Intel GPU"](https://docs.pytorch.org/docs/stable/notes/get_start_xpu.html) to install XPU dependency.
|
||||
|
||||
```bash
|
||||
# Create and activate a conda environment
|
||||
conda create -n sgl-xpu python=3.12 -y
|
||||
conda activate sgl-xpu
|
||||
|
||||
# Set PyTorch XPU as primary pip install channel to avoid installing the larger CUDA-enabled version and prevent potential runtime issues.
|
||||
pip3 install torch==2.10.0+xpu torchao torchvision torchaudio triton-xpu==3.6.0 --index-url https://download.pytorch.org/whl/xpu
|
||||
pip3 install xgrammar --no-deps # xgrammar will introduce CUDA-enabled triton which might conflict with XPU
|
||||
|
||||
# Clone the SGLang code
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
git checkout <YOUR-DESIRED-VERSION>
|
||||
|
||||
# Use dedicated toml file
|
||||
cd python
|
||||
cp pyproject_xpu.toml pyproject.toml
|
||||
# Install SGLang dependent libs, and build SGLang main package
|
||||
pip install --upgrade pip setuptools
|
||||
pip install -v . --extra-index-url https://download.pytorch.org/whl/xpu
|
||||
```
|
||||
|
||||
### Install Using Docker
|
||||
|
||||
The docker for XPU is under active development. Please stay tuned.
|
||||
|
||||
## Launch of the Serving Engine
|
||||
|
||||
Example command to launch SGLang serving:
|
||||
|
||||
```bash
|
||||
python -m sglang.launch_server \
|
||||
--model <MODEL_ID_OR_PATH> \
|
||||
--trust-remote-code \
|
||||
--disable-overlap-schedule \
|
||||
--device xpu \
|
||||
--host 0.0.0.0 \
|
||||
--tp 2 \ # using multi GPUs
|
||||
--attention-backend intel_xpu \ # using intel optimized XPU attention backend
|
||||
--page-size \ # intel_xpu attention backend supports [32, 64, 128]
|
||||
```
|
||||
|
||||
## Benchmarking with Requests
|
||||
|
||||
You can benchmark the performance via the `bench_serving` script.
|
||||
Run the command in another terminal.
|
||||
|
||||
```bash
|
||||
python -m sglang.bench_serving \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 1 \
|
||||
--request-rate inf \
|
||||
--random-range-ratio 1.0
|
||||
```
|
||||
|
||||
The detail explanations of the parameters can be looked up by the command:
|
||||
|
||||
```bash
|
||||
python -m sglang.bench_serving -h
|
||||
```
|
||||
|
||||
Additionally, the requests can be formed with
|
||||
[OpenAI Completions API](https://docs.sglang.io/basic_usage/openai_api_completions.html)
|
||||
and sent via the command line (e.g. using `curl`) or via your own script.
|
||||
Reference in New Issue
Block a user