chore: vendor sglang v0.5.10 snapshot
This commit is contained in:
163
third_party/sglang/docs/platforms/ascend/ascend_contribution_guide.md
vendored
Normal file
163
third_party/sglang/docs/platforms/ascend/ascend_contribution_guide.md
vendored
Normal file
@@ -0,0 +1,163 @@
|
||||
# Contribution Guide
|
||||
|
||||
Welcome to **SGLang**! We appreciate your interest in contributing. This guide provides a concise overview of how to set up your environment, run tests, build documentation, and open a Pull Request (PR). Whether you’re fixing a small bug or developing a major feature, we encourage following these steps for a smooth contribution process.
|
||||
|
||||
## Install SGLang from Source
|
||||
|
||||
### Prepare Environment
|
||||
|
||||
Before contributing, please ensure that your environment is set up correctly. Follow the steps in the [Installation Guide](ascend_npu.md) to install the necessary dependencies. We recommend [using docker](ascend_npu.md#method-2-using-docker-image) to build the environment.
|
||||
|
||||
### Fork and clone the repository
|
||||
|
||||
**Note**: New contributors do **not** have the write permission to push to the official SGLang repo. Please fork the repository under your GitHub account, then clone your fork locally.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/<your_user_name>/sglang.git
|
||||
# if you are using docker, the environment is already set up.
|
||||
cd sglang
|
||||
export PYTHONPATH=$PWD/python:$PYTHONPATH
|
||||
```
|
||||
|
||||
## Format code with pre-commit
|
||||
|
||||
We use [pre-commit](https://pre-commit.com/) to maintain consistent code style checks. Before pushing your changes, please run:
|
||||
|
||||
```bash
|
||||
pip3 install pre-commit
|
||||
pre-commit install
|
||||
pre-commit run --all-files
|
||||
```
|
||||
|
||||
- **`pre-commit run --all-files`** manually runs all configured checks, applying fixes if possible. If it fails the first time, re-run it to ensure lint errors are fully resolved. Make sure your code passes all checks **before** creating a Pull Request.
|
||||
- **Do not commit** directly to the `main` branch. Always create a new branch (e.g., `feature/my-new-feature`), push your changes, and open a PR from that branch.
|
||||
|
||||
## Run and add unit tests
|
||||
|
||||
If you add a new feature or fix a bug, please add corresponding unit tests to ensure coverage and prevent regression.
|
||||
SGLang uses Python's built-in [unittest](https://docs.python.org/3/library/unittest.html) framework.
|
||||
For detailed instructions on running tests and integrating them into CI, refer to [test/README.md](https://github.com/sgl-project/sglang/tree/main/test/README.md).
|
||||
|
||||
If you need to use model which is not in ```python/sglang/test/ascend/test_ascend_utils.py`` list. Follow these steps:
|
||||
1. Register account and upload your model to [modelscope](https://modelscope.cn/models).
|
||||
2. Make sure your model is pre-cached on the CI server and is on the way "/data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}".
|
||||
If this is not the case, use following command on CI server:
|
||||
```bash
|
||||
modelscope download
|
||||
--model {your_model_repo}/{your_model}
|
||||
--local_dir /data/ascend-ci-share-pkking-sglang/modelscope/hub/models/{your_model_repo}/{your_model}
|
||||
```
|
||||
> Note: If you don’t have access to CI server, please ask maintainers (zl19940307@163.com) to download your model.
|
||||
4. Add model to ```python/sglang/test/ascend/test_ascend_utils.py``` (use docker ```"/root/.cache/modelscope/hub/models/{your_model_repo}/{your_model}"``` path).
|
||||
|
||||
## Write documentations
|
||||
|
||||
We recommend new contributors start from writing documentation, which helps you quickly understand SGLang codebase.
|
||||
For more details, please refer to [docs/README.md](https://github.com/sgl-project/sglang/tree/main/docs/README.md).
|
||||
|
||||
## Test the accuracy
|
||||
If your code changes the model output, please run the accuracy tests. A quick sanity check is the few-shot GSM8K.
|
||||
|
||||
```
|
||||
# Launch a server
|
||||
python3 -m sglang.launch_server --model Qwen/Qwen2-7B-Instruct
|
||||
|
||||
# Evaluate
|
||||
python3 -m sglang.test.few_shot_gsm8k --num-questions 200
|
||||
```
|
||||
|
||||
Please note that the above script is primarily a sanity check, not a rigorous accuracy or speed test.
|
||||
This test can have significant variance (1%–5%) in accuracy due to batching and the non-deterministic nature of the inference engine.
|
||||
Also, do not rely on the "Latency/Output throughput" from this script, as it is not a proper speed test.
|
||||
|
||||
GSM8K is too easy for state-of-the-art models nowadays. Please try your own more challenging accuracy tests.
|
||||
You can find additional accuracy eval examples in:
|
||||
- [test_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_eval_accuracy_large.py)
|
||||
- [test_moe_eval_accuracy_large.py](https://github.com/sgl-project/sglang/blob/main/test/registered/eval/test_moe_eval_accuracy_large.py)
|
||||
|
||||
## Benchmark the speed
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md).
|
||||
|
||||
## Requesting a review for merge
|
||||
You can follow the pull request merge process described in [MAINTAINER.md](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md).
|
||||
You will need to work with the Merge Oncall, Codeowner, and other reviewers to get their approvals.
|
||||
Then your PR can be merged.
|
||||
|
||||
## How to Trigger CI Tests
|
||||
|
||||
We have a lot of open PRs but limited CI machines, so only top and trusted contributors have permission to trigger CI tests.
|
||||
Users with permission are listed in the [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json)
|
||||
|
||||
For CI to run on a pull request, it must have the "run-ci" label. Authorized users can add the label or rerun failed tests by commenting on the PR with one of these commands:
|
||||
|
||||
- `/tag-run-ci-label`: Adds the "run-ci" label. Every future commit will trigger CI.
|
||||
- `/rerun-failed-ci`: Reruns the failed or flaky tests from the most recent commit.
|
||||
- `/tag-and-rerun-ci`: A single command that performs both `/tag-run-ci-label` and `/rerun-failed-ci`.
|
||||
- `/rerun-stage <stage-name>`: Reruns a specific test stage without waiting for its dependencies. This is useful when you want to quickly validate a fix for a specific test failure instead of waiting ~30 minutes for preceding stages to complete.
|
||||
|
||||
If you have permission, the [Slash Command Handler](https://github.com/sgl-project/sglang/actions/workflows/slash-command-handler.yml) will run your command and react with a 👍 to your comment. It may take up to a few minutes for the reaction to appear. Here’s a usage [example](https://github.com/sgl-project/sglang/pull/14253#issuecomment-3599509302).
|
||||
|
||||
To avoid spamming a PR with too many `/rerun-failed-ci` comments, you can also trigger the command by editing an existing comment and adding any suffix (e.g., `/rerun-failed-ci try again`).
|
||||
|
||||
Example of rerunning a single test stage: `/rerun-stage unit-test-backend-4-gpu`.
|
||||
|
||||
If you don’t have permission, please ask maintainers to trigger CI for you.
|
||||
|
||||
### CI rate limits
|
||||
|
||||
Due to CI scheduling and limited resources, higher-priority PRs may preempt running jobs. In such cases, you may need to rerun the tests.
|
||||
|
||||
We apply CI rate limits to prevent abuse and ensure fair usage of our CI resources.
|
||||
|
||||
Each CI workflow has a default limit defined in its workflow configuration file. For example, in [pr-gate.yml](https://github.com/sgl-project/sglang/blob/main/.github/workflows/pr-gate.yml), the default cooldown period is 120 minutes, and each workflow can override it via the `cool-down-minutes` input parameter:
|
||||
|
||||
```yaml
|
||||
cool-down-minutes:
|
||||
description: "Default cooldown period in minutes; 0 disables rate limiting"
|
||||
type: number
|
||||
default: 120
|
||||
```
|
||||
|
||||
Users listed in [CI_PERMISSIONS.json](https://github.com/sgl-project/sglang/blob/main/.github/CI_PERMISSIONS.json) may have a per-user cooldown interval. In practice, we use the minimum of the workflow’s default window and the user-specific interval.
|
||||
|
||||
## Code style guidance
|
||||
- Avoid code duplication. If the same code snippet (more than five lines) appears multiple times, extract it into a shared function.
|
||||
- Minimize device synchronization. Reduce expensive CPU-GPU synchronization operations, such as `tensor.item()` or `tensor.cpu()`, whenever possible. Use vectorized code.
|
||||
- Prioritize extreme efficiency. SGLang is a runtime, and most of your code runs on the critical path for every request. Optimize all minor overheads as much as possible, especially in the model forward code.
|
||||
- A common pattern is some runtime checks in the model forward pass (e.g., [this](https://github.com/sgl-project/sglang/blob/f1b0eda55c2c4838e8ab90a0fac7fb1e3d7064ab/python/sglang/srt/models/deepseek_v2.py#L486-L491)). These are very likely the same for every layer. Please cache the result as a single boolean value whenever possible.
|
||||
- Make functions as pure as possible. Avoid in-place modification of arguments.
|
||||
- Keep files concise. If a file exceeds 2,000 lines of code, split it into multiple smaller files. (e.g., `scheduler.py`, `scheduler_output_processor_mixin.py`)
|
||||
- Keep tests run fast.
|
||||
- If a single test file run longer than 500 seconds, split it into multiple smaller files (e.g., `test_eagle_infer_a.py`, `test_eagle_infer_b.py`).
|
||||
- If a single job in a github workflow runs longer than 30 mins, split it into smaller jobs/steps.
|
||||
- Reuse server launches in your unit tests to make tests run faster.
|
||||
- When supporting new hardware or features, follow these guidelines:
|
||||
- Do not drastically change existing code.
|
||||
- Always prefer new files to introduce specific components for your new hardware (e.g., `allocator_ascend.py`).
|
||||
- If you write multiple if/else blocks for new features, ensure the common path (e.g., NVIDIA hardware or the existing code path) is the first branch.
|
||||
|
||||
## How to update sgl-kernel
|
||||
Since sglang and sgl-kernel are separate Python packages, our current GitHub CI infrastructure does not support updating a kernel and using it immediately within the same pull request (PR).
|
||||
To add a new kernel or modify an existing one in the `sgl-kernel/` source tree, you must use multiple PRs.
|
||||
|
||||
Follow these steps:
|
||||
|
||||
1. Submit a PR to update the sgl-kernel source code without using it in sglang python package (e.g., [#8884](https://github.com/sgl-project/sglang/pull/8884/files)).
|
||||
2. Bump the version of the kernel package (e.g., [#9220](https://github.com/sgl-project/sglang/pull/9220/files)).
|
||||
- Once merged, this will trigger an automatic release of the `sglang-kernel` wheel to PyPI.
|
||||
- If not urgent, you can wait for other people to release the wheel. A new version will typically be released within one week.
|
||||
3. Apply the changes:
|
||||
- Update the `sglang-kernel` version in `sglang/python/pyproject.toml` to use the modified kernels.
|
||||
- Update the related caller code in the sglang to use the new kernel.
|
||||
|
||||
## How to update sgl-kernel-npu
|
||||
|
||||
Sgl-kernel-npu is the kernel package for Ascend NPU and is maintained in the [sgl-kernel-npu](https://github.com/sgl-project/sgl-kernel-npu) repository. if you want to add a new kernel and want to use it in sglang, please follow the steps in [Contribution Guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/docs/developer_guide/contribution_guide.md).
|
||||
|
||||
## Tips for newcomers
|
||||
|
||||
If you want to contribute but don’t have a specific idea in mind, pick issues labeled [“good first issue” or “help wanted”](https://github.com/sgl-project/sglang/issues?q=is%3Aissue+label%3A%22good+first+issue%22%2C%22help+wanted%22). These tasks typically have lower complexity and provide an excellent introduction to the codebase. Also check out this [code walk-through](https://github.com/zhaochenyang20/Awesome-ML-SYS-Tutorial/tree/main/sglang/code-walk-through) for a deeper look into SGLang’s workflow.
|
||||
|
||||
If you have any questions or want to start a discussion, please feel free to ask in our [Slack channel](https://slack.sglang.io).
|
||||
|
||||
Thank you for your interest in SGLang. Happy coding!
|
||||
244
third_party/sglang/docs/platforms/ascend/ascend_npu.md
vendored
Normal file
244
third_party/sglang/docs/platforms/ascend/ascend_npu.md
vendored
Normal file
@@ -0,0 +1,244 @@
|
||||
|
||||
# SGLang installation with NPUs support
|
||||
|
||||
You can install SGLang using any of the methods below. Please go through `System Settings` section to ensure the clusters are roaring at max performance. Feel free to leave an issue [here at sglang](https://github.com/sgl-project/sglang/issues) if you encounter any issues or have any problems.
|
||||
|
||||
## Component Version Mapping For SGLang
|
||||
| Component | Version | Obtain Way |
|
||||
|-------------------|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| HDK | 25.3.RC1 | [link](https://www.hiascend.com/hardware/firmware-drivers/commercial?product=7&model=33) |
|
||||
| CANN | 8.5.0 | [Obtain Images](#obtain-cann-image) |
|
||||
| Pytorch Adapter | 7.3.0 | [link](https://gitcode.com/Ascend/pytorch/releases) |
|
||||
| MemFabric | 1.0.5 | `pip install memfabric-hybrid==1.0.5` |
|
||||
| Triton | 3.2.0 | `pip install triton-ascend`|
|
||||
| SGLang NPU Kernel | NA | [link](https://github.com/sgl-project/sgl-kernel-npu/releases) |
|
||||
|
||||
<a id="obtain-cann-image"></a>
|
||||
### Obtain CANN Image
|
||||
You can obtain the dependency of a specified version of CANN through an image.
|
||||
```shell
|
||||
# for Atlas 800I A3 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-a3-ubuntu22.04-py3.11
|
||||
# for Atlas 800I A2 and Ubuntu OS
|
||||
docker pull quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
|
||||
```
|
||||
|
||||
## Preparing the Running Environment
|
||||
|
||||
### Method 1: Installing from source with prerequisites
|
||||
|
||||
#### Python Version
|
||||
|
||||
Only `python==3.11` is supported currently. If you don't want to break system pre-installed python, try installing with [conda](https://github.com/conda/conda).
|
||||
|
||||
```shell
|
||||
conda create --name sglang_npu python=3.11
|
||||
conda activate sglang_npu
|
||||
```
|
||||
|
||||
#### CANN
|
||||
|
||||
Prior to start work with SGLang on Ascend you need to install CANN Toolkit, Kernels operator package and NNAL version 8.3.RC2 or higher, check the [installation guide](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/83RC1/softwareinst/instg/instg_0008.html?Mode=PmIns&InstallType=local&OS=openEuler&Software=cannToolKit)
|
||||
|
||||
#### MemFabric-Hybrid
|
||||
|
||||
If you want to use PD disaggregation mode, you need to install MemFabric-Hybrid. MemFabric-Hybrid is a drop-in replacement of Mooncake Transfer Engine that enables KV cache transfer on Ascend NPU clusters.
|
||||
|
||||
```shell
|
||||
pip install memfabric-hybrid==1.0.5
|
||||
```
|
||||
|
||||
#### Pytorch and Pytorch Framework Adaptor on Ascend
|
||||
|
||||
```shell
|
||||
PYTORCH_VERSION=2.8.0
|
||||
TORCHVISION_VERSION=0.23.0
|
||||
TORCH_NPU_VERSION=2.8.0
|
||||
pip install torch==$PYTORCH_VERSION torchvision==$TORCHVISION_VERSION --index-url https://download.pytorch.org/whl/cpu
|
||||
pip install torch_npu==$TORCH_NPU_VERSION
|
||||
```
|
||||
|
||||
If you are using other versions of `torch` and install `torch_npu`, check [installation guide](https://github.com/Ascend/pytorch/blob/master/README.md)
|
||||
|
||||
#### Triton on Ascend
|
||||
|
||||
We provide our own implementation of Triton for Ascend.
|
||||
|
||||
```shell
|
||||
pip install triton-ascend
|
||||
```
|
||||
For installation of Triton on Ascend nightly builds or from sources, follow [installation guide](https://gitcode.com/Ascend/triton-ascend/blob/master/docs/sources/getting-started/installation.md)
|
||||
|
||||
#### SGLang Kernels NPU
|
||||
We provide SGL kernels for Ascend NPU, check [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/sgl_kernel_npu/README.md).
|
||||
|
||||
#### DeepEP-compatible Library
|
||||
We provide a DeepEP-compatible Library as a drop-in replacement of deepseek-ai's DeepEP library, check the [installation guide](https://github.com/sgl-project/sgl-kernel-npu/blob/main/python/deep_ep/README.md).
|
||||
|
||||
#### Installing SGLang from source
|
||||
|
||||
```shell
|
||||
# Use the last release branch
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang
|
||||
mv python/pyproject_npu.toml python/pyproject.toml
|
||||
pip install -e python[all_npu]
|
||||
```
|
||||
|
||||
### Method 2: Using Docker Image
|
||||
#### Obtain Image
|
||||
You can download the SGLang image or build an image based on Dockerfile to obtain the Ascend NPU image.
|
||||
1. Download SGLang image
|
||||
```angular2html
|
||||
dockerhub: docker.io/lmsysorg/sglang:$tag
|
||||
# Main-based tag, change main to specific version like v0.5.6,
|
||||
# you can get image for specific version
|
||||
Atlas 800I A3 : {main}-cann8.5.0-a3
|
||||
Atlas 800I A2: {main}-cann8.5.0-910b
|
||||
```
|
||||
2. Build an image based on Dockerfile
|
||||
```shell
|
||||
# Clone the SGLang repository
|
||||
git clone https://github.com/sgl-project/sglang.git
|
||||
cd sglang/docker
|
||||
|
||||
# Build the docker image
|
||||
# If there are network errors, please modify the Dockerfile to use offline dependencies or use a proxy
|
||||
docker build -t <image_name> -f npu.Dockerfile .
|
||||
```
|
||||
|
||||
#### Create Docker
|
||||
__Notice:__ `--privileged` and `--network=host` are required by RDMA, which is typically needed by Ascend NPU clusters.
|
||||
|
||||
__Notice:__ The following docker command is based on Atlas 800I A3 machines. If you are using Atlas 800I A2, make sure only `davinci[0-7]` are mapped into container.
|
||||
|
||||
```shell
|
||||
|
||||
alias drun='docker run -it --rm --privileged --network=host --ipc=host --shm-size=16g \
|
||||
--device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \
|
||||
--device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \
|
||||
--device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \
|
||||
--device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \
|
||||
--device=/dev/davinci_manager --device=/dev/hisi_hdc \
|
||||
--volume /usr/local/sbin:/usr/local/sbin --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
--volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--volume /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
--volume /var/queue_schedule:/var/queue_schedule --volume ~/.cache/:/root/.cache/'
|
||||
|
||||
# Add HF_TOKEN env for download model by SGLang.
|
||||
drun --env "HF_TOKEN=<secret>" \
|
||||
<image_name> \
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
## System Settings
|
||||
|
||||
### CPU performance power scheme
|
||||
|
||||
The default power scheme on Ascend hardware is `ondemand` which could affect performance, changing it to `performance` is recommended.
|
||||
|
||||
```shell
|
||||
echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
|
||||
# Make sure changes are applied successfully
|
||||
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # shows performance
|
||||
```
|
||||
|
||||
### Disable NUMA balancing
|
||||
|
||||
```shell
|
||||
sudo sysctl -w kernel.numa_balancing=0
|
||||
# Check
|
||||
cat /proc/sys/kernel/numa_balancing # shows 0
|
||||
```
|
||||
|
||||
### Prevent swapping out system memory
|
||||
|
||||
```shell
|
||||
sudo sysctl -w vm.swappiness=10
|
||||
|
||||
# Check
|
||||
cat /proc/sys/vm/swappiness # shows 10
|
||||
```
|
||||
|
||||
## Running SGLang Service
|
||||
### Running Service For Large Language Models
|
||||
#### PD Mixed Scene
|
||||
```shell
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
python3 -m sglang.launch_server --model-path meta-llama/Llama-3.1-8B-Instruct --attention-backend ascend
|
||||
```
|
||||
|
||||
#### PD Disaggregation Scene
|
||||
1. Launch Prefill Server
|
||||
```shell
|
||||
# Enabling CPU Affinity
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 0 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
2. Launch Decode Server
|
||||
```shell
|
||||
# PIP: recommended to config first Prefill Server IP
|
||||
# PORT: one free port
|
||||
# all sglang servers need to be config the same PIP and PORT,
|
||||
export ASCEND_MF_STORE_URL="tcp://PIP:PORT"
|
||||
# if you are Atlas 800I A2 hardware and use rdma for kv cache transfer, add this parameter
|
||||
export ASCEND_MF_TRANSFER_PROTOCOL="device_rdma"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path meta-llama/Llama-3.1-8B-Instruct \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--base-gpu-id 1 \
|
||||
--tp-size 1 \
|
||||
--host 127.0.0.1 \
|
||||
--port 8001
|
||||
```
|
||||
|
||||
3. Launch Router
|
||||
```shell
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://127.0.0.1:8000 8995 \
|
||||
--decode http://127.0.0.1:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
|
||||
### Running Service For Multimodal Language Models
|
||||
#### PD Mixed Scene
|
||||
```shell
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path Qwen3-VL-30B-A3B-Instruct \
|
||||
--host 127.0.0.1 \
|
||||
--port 8000 \
|
||||
--tp 4 \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--enable-multimodal \
|
||||
--sampling-backend ascend
|
||||
```
|
||||
2485
third_party/sglang/docs/platforms/ascend/ascend_npu_best_practice.md
vendored
Normal file
2485
third_party/sglang/docs/platforms/ascend/ascend_npu_best_practice.md
vendored
Normal file
File diff suppressed because it is too large
Load Diff
297
third_party/sglang/docs/platforms/ascend/ascend_npu_deepseek_example.md
vendored
Normal file
297
third_party/sglang/docs/platforms/ascend/ascend_npu_deepseek_example.md
vendored
Normal file
@@ -0,0 +1,297 @@
|
||||
## DeepSeek examples
|
||||
|
||||
### Running DeepSeek-V3
|
||||
|
||||
#### Running DeepSeek in PD mixed mode on 1 x Atlas 800I A3.
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
```shell
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export HCCL_BUFFSIZE=1600
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--tp 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--watchdog-timeout 9000 \
|
||||
--cuda-graph-bs 8 16 24 28 32 \
|
||||
--mem-fraction-static 0.68 \
|
||||
--max-running-requests 128 \
|
||||
--context-length 8188 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 16384 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode auto \
|
||||
--enable-dp-attention \
|
||||
--dp-size 4 \
|
||||
--enable-dp-lm-head \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Running DeepSeek with PD disaggregation mode on 2 x Atlas 800I A3.
|
||||
|
||||
W4A8 Model weights could be found [here](https://modelers.cn/models/Modelers_Park/DeepSeek-R1-0528-w4a8).
|
||||
|
||||
1. Prefill:
|
||||
|
||||
```shell
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export HCCL_BUFFSIZE=1536
|
||||
|
||||
#npu acceleration operator
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--host $PREFILL_HOST_IP \
|
||||
--port 8000 \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-bootstrap-port 8996 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.6 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--load-balance-method round_robin \
|
||||
--max-running-requests 8 \
|
||||
--context-length 8192 \
|
||||
--disable-radix-cache \
|
||||
--chunked-prefill-size -1 \
|
||||
--max-prefill-tokens 28680 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--dp-size 2 \
|
||||
--enable-dp-attention \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
2. Decode:
|
||||
|
||||
```shell
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
#memfabric config store
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:<PORT>"
|
||||
|
||||
#Deepep communication settings
|
||||
export HCCL_BUFFSIZE=720
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=88
|
||||
|
||||
#spec overlap
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
#npu acceleration operator
|
||||
unset TASK_QUEUE_ENABLE
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
# suggest max-running-requests <= max-cuda-graph-bs * dp_size, Because when this value is exceeded, performance will significantly degrade.
|
||||
python -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode decode \
|
||||
--host $DECODE_HOST_IP \
|
||||
--port 8001 \
|
||||
--trust-remote-code \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--tp-size 16 \
|
||||
--dp-size 16 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--max-running-requests 352 \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep \
|
||||
--enable-dp-attention \
|
||||
--deepep-mode low_latency \
|
||||
--enable-dp-lm-head \
|
||||
--cuda-graph-bs 8 10 12 14 16 18 20 22 \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--watchdog-timeout 9000 \
|
||||
--context-length 8192 \
|
||||
--speculative-algorithm NEXTN \
|
||||
--speculative-num-steps 3 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 4 \
|
||||
--disable-shared-experts-fusion \
|
||||
--dtype bfloat16 \
|
||||
--tokenizer-worker-num 4
|
||||
```
|
||||
|
||||
3. SGLang Model Gateway (former Router)
|
||||
|
||||
```shell
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://<PREFILL_HOST_IP>:8000 8996 \
|
||||
--decode http://<DECODE_HOST_IP>:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688
|
||||
```
|
||||
|
||||
#### Running DeepSeek with PD disaggregation on 4 x Atlas 800I A3.
|
||||
|
||||
W8A8 Model weights could be found [here](https://modelers.cn/models/State_Cloud/Deepseek-R1-bf16-hfd-w8a8).
|
||||
|
||||
1. Prefill & Decode:
|
||||
|
||||
```shell
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
export PATH=/usr/local/Ascend/8.5.0/compiler/bishengir/bin:$PATH
|
||||
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
|
||||
export ASCEND_MF_STORE_URL="tcp://your prefill ip1:24669"
|
||||
|
||||
P_IP=('your prefill ip1' 'your prefill ip2')
|
||||
|
||||
D_IP=('your decode ip1' 'your decode ip2')
|
||||
|
||||
MODEL_PATH=xxx
|
||||
|
||||
export SGLANG_NPU_USE_MLAPO=1
|
||||
export SGLANG_USE_FIA_NZ=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
echo "${LOCAL_HOST1}"
|
||||
echo "${LOCAL_HOST2}"
|
||||
# prefill
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export TASK_QUEUE_ENABLE=2
|
||||
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode prefill --host ${P_IP[$i]} \
|
||||
--port 8000 --disaggregation-bootstrap-port $((8998+$i)) --trust-remote-code --nnodes 1 --node-rank 0 \
|
||||
--tp-size 16 --mem-fraction-static 0.81 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--disaggregation-transfer-backend ascend --max-running-requests 8 --context-length 8192 --disable-radix-cache \
|
||||
--chunked-prefill-size -1 --max-prefill-tokens 28680 --moe-a2a-backend deepep --deepep-mode normal \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 1 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2 \
|
||||
--dp-size 2 --enable-dp-attention --disable-shared-experts-fusion --dtype bfloat16 --enable-attn-tp-input-scattered
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
# decode
|
||||
for i in "${!D_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${D_IP[$i]}" || "$LOCAL_HOST2" == "${D_IP[$i]}" ]];
|
||||
then
|
||||
echo "${D_IP[$i]}"
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export HCCL_BUFFSIZE=650
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=78
|
||||
export TASK_QUEUE_ENABLE=1
|
||||
export SGLANG_SCHEDULER_SKIP_ALL_GATHER=1
|
||||
export HCCL_SOCKET_IFNAME=xxx
|
||||
export GLOO_SOCKET_IFNAME=xxx
|
||||
python -m sglang.launch_server --model-path ${MODEL_PATH} --disaggregation-mode decode --host ${D_IP[$i]} \
|
||||
--port 8001 --trust-remote-code --dist-init-addr ${D_IP[0]}:5000 --nnodes 2 --node-rank $i --tp-size 32 --dp-size 32 \
|
||||
--mem-fraction-static 0.815 --max-running-requests 832 --attention-backend ascend --device npu --quantization modelslim \
|
||||
--moe-a2a-backend deepep --enable-dp-attention --deepep-mode low_latency --enable-dp-lm-head --moe-dense-tp 1 \
|
||||
--cuda-graph-bs 12 14 16 18 20 22 24 26 --disaggregation-transfer-backend ascend --watchdog-timeout 9000 --context-length 8192 \
|
||||
--speculative-algorithm NEXTN --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 3 \
|
||||
--tokenizer-worker-num 4 --disable-shared-experts-fusion --dtype bfloat16 \
|
||||
--load-balance-method decode_round_robin
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
```
|
||||
|
||||
2. SGLang Model Gateway (former Router):
|
||||
|
||||
```shell
|
||||
python -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://P_IP:8000 8998 \
|
||||
--prefill http://P_IP:8000 8999 \
|
||||
--decode http://D_IP:8001 \
|
||||
--host 127.0.0.1 \
|
||||
--port 6688 \
|
||||
--mini-lb
|
||||
```
|
||||
|
||||
#### test gsm8k
|
||||
|
||||
```python
|
||||
from types import SimpleNamespace
|
||||
from sglang.test.few_shot_gsm8k import run_eval
|
||||
|
||||
def gsm8k():
|
||||
args = SimpleNamespace(
|
||||
num_shots=5,
|
||||
data_path=None,
|
||||
num_questions=200,
|
||||
max_new_tokens=512,
|
||||
parallel=32,
|
||||
host=f"http://127.0.0.1",
|
||||
port=6688,
|
||||
)
|
||||
metrics = run_eval(args)
|
||||
print(f"{metrics=}")
|
||||
print(f"{metrics['accuracy']=}")
|
||||
if __name__ == "__main__":
|
||||
gsm8k()
|
||||
```
|
||||
39
third_party/sglang/docs/platforms/ascend/ascend_npu_environment_variables.md
vendored
Normal file
39
third_party/sglang/docs/platforms/ascend/ascend_npu_environment_variables.md
vendored
Normal file
@@ -0,0 +1,39 @@
|
||||
# Environment Variables
|
||||
|
||||
SGLang supports various environment variables related to Ascend NPU that can be used to configure its runtime behavior.
|
||||
This document provides a list of commonly used environment variables and aims to stay updated over time.
|
||||
|
||||
## Directly Used in SGLang
|
||||
|
||||
| Environment Variable | Description | Default Value |
|
||||
|--------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
|
||||
| `SGLANG_NPU_USE_MLAPO` | Adopts the `MLAPO` fusion operator in attention <br/> preprocessing stage of the MLA model. | `false` |
|
||||
| `SGLANG_USE_FIA_NZ` | Reshapes KV Cache for FIA NZ format.<br/> `SGLANG_USE_FIA_NZ` must be enabled with `SGLANG_NPU_USE_MLAPO` | `false` |
|
||||
| `SGLANG_NPU_USE_MULTI_STREAM` | Enable dual-stream computation of shared experts <br/> and routing experts in DeepSeek models.<br/> Enable dual-stream computation in DeepSeek NSA Indexer. | `false` |
|
||||
| `SGLANG_NPU_DISABLE_ACL_FORMAT_WEIGHT` | Disable cast model weight tensor to a specific NPU <br/> ACL format. | `false` |
|
||||
| `SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK` | The maximum number of dispatched tokens on each rank. | `128` |
|
||||
|
||||
## Used in DeepEP Ascend
|
||||
|
||||
| Environment Variable | Description | Default Value |
|
||||
|-------------------------------------------|------------------------------------------------------------------------------------------------------------------------|---------------|
|
||||
| `DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS` | Enable ant-moving function in dispatch stage. Indicates <br/> the number of tokens transmitted per round on each rank. | `8192` |
|
||||
| `DEEPEP_NORMAL_LONG_SEQ_ROUND` | Enable ant-moving function in dispatch stage. Indicates <br/> the number of rounds transmitted on each rank. | `1` |
|
||||
| `DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ` | Enable ant-moving function in combine stage. <br/> The value `0` means disabled. | `0` |
|
||||
| `MOE_ENABLE_TOPK_NEG_ONE` | Needs to be enabled when the expert ID to be processed by <br/> DEEPEP contains -1. | `0` |
|
||||
| `DEEP_NORMAL_MODE_USE_INT8_QUANT` | Quantizes x to int8 and returns (tensor, scales) in dispatch operator. | `0` |
|
||||
|
||||
## Others
|
||||
|
||||
| Environment Variable | Description | Default Value |
|
||||
|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|
|
||||
| `TASK_QUEUE_ENABLE` | Used to control the optimization level of the dispatch queue<br/> about the task_queue operator. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/730/comref/Envvariables/docs/zh/environment_variable_reference/TASK_QUEUE_ENABLE.md) | `1` |
|
||||
| `INF_NAN_MODE_ENABLE` | Controls whether the chip uses saturation mode or INF_NAN mode. [Detail](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0056.html) | `1` |
|
||||
| `STREAMS_PER_DEVICE` | Configures the maximum number of streams for the stream pool. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/720/comref/Envvariables/Envir_041.html) | `32` |
|
||||
| `PYTORCH_NPU_ALLOC_CONF` | Controls the behavior of the cache allocator. <br/>This variable changes memory usage and may cause performance fluctuations. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/700/comref/Envvariables/Envir_012.html) | |
|
||||
| `ASCEND_MF_STORE_URL` | The address of config store in MemFabric during PD separation, <br/>which is generally set to the IP address of the P primary node<br/> with an arbitrary port number. | |
|
||||
| `ASCEND_LAUNCH_BLOCKING` | Controls whether synchronous mode is enabled during operator execution. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/710/comref/Envvariables/Envir_006.html) | `0` |
|
||||
| `HCCL_OP_EXPANSION_MODE` | Configures the expansion position for communication algorithm scheduling. [Detail](https://www.hiascend.com/document/detail/zh/CANNCommunityEdition/800alpha001/apiref/envref/envref_07_0094.html) | |
|
||||
| `HCCL_BUFFSIZE` | Controls the size of the buffer area for shared data between two NPUs. <br/>The unit is MB, and the value must be greater than or equal to 1. [Detail](https://www.hiascend.com/document/detail/zh/Pytorch/60RC3/ptmoddevg/trainingmigrguide/performance_tuning_0047.html) | `200` |
|
||||
| `HCCL_SOCKET_IFNAME` | Configures the name of the network card used by the Host <br/>during HCCL initialization. [Detail](https://www.hiascend.com/document/detail/zh/canncommercial/81RC1/apiref/envvar/envref_07_0075.html) | |
|
||||
| `GLOO_SOCKET_IFNAME` | Configures the network interface name for GLOO communication. | |
|
||||
194
third_party/sglang/docs/platforms/ascend/ascend_npu_glm5_examples.md
vendored
Normal file
194
third_party/sglang/docs/platforms/ascend/ascend_npu_glm5_examples.md
vendored
Normal file
@@ -0,0 +1,194 @@
|
||||
# GLM-5 examples
|
||||
|
||||
## Introduction
|
||||
|
||||
The GLM (General Language Model) series is an open-source bilingual large language model family jointly developed by the KEG Laboratory of Tsinghua University and Zhipu AI. This series of models has performed outstandingly in the field of Chinese NLP with its unique unified pre-training framework and bilingual capabilities. [GLM-5](https://huggingface.co/zai-org/GLM-5) adopts the DeepSeek-V3/V3.2 architecture, including the sparse attention (DSA) and multi-token prediction (MTP). Ascend supports GLM-5 with 0Day based on the SGLang inference framework, achieving low-code seamless enablement and compatibility with the mainstream distributed parallel capabilities within the current SGLang framework. We welcome developers to download and experience it.
|
||||
|
||||
## Environment Preparation
|
||||
|
||||
### Model Weight
|
||||
|
||||
- `GLM-5.0`(BF16 version): [Download model weight](https://www.modelscope.cn/models/ZhipuAI/GLM-5).
|
||||
- `GLM-5.0-w4a8`(Quantized version without mtp): [Download model weight](https://modelers.cn/models/Eco-Tech/GLM-5-w4a8).
|
||||
- You can use [msmodelslim](https://gitcode.com/Ascend/msmodelslim) to quantify the model naively.
|
||||
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the online platform. You can directly pull it.
|
||||
|
||||
```{code-block} bash
|
||||
#Atlas 800 A3
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-a3-glm5
|
||||
#Atlas 800 A2
|
||||
docker pull swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:cann8.5.0-910b-glm5
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
swr.cn-southwest-2.myhuaweicloud.com/base_image/dockerhub/lmsysorg/sglang:${TAG}
|
||||
```
|
||||
|
||||
Note: Using this image, you need to update transformers to main branch
|
||||
``` shell
|
||||
# reinstall transformers
|
||||
pip install git+https://github.com/huggingface/transformers.git
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
- Quantized model `glm5_w4a8` can be deployed on 1 Atlas 800 A3 (64G × 16) .
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 280000 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--moe-a2a-backend deepep --deepep-mode auto
|
||||
```
|
||||
|
||||
### Multi-node Deployment
|
||||
|
||||
- `GLM-5-bf16`: require at least 2 Atlas 800 A3 (64G × 16).
|
||||
|
||||
**A3 series**
|
||||
|
||||
Modify the IP of 2 nodes, then run the same scripts on two nodes.
|
||||
|
||||
**node 0/1**
|
||||
|
||||
```shell
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_NPU_USE_MULTI_STREAM=1
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
# Run command ifconfig on two nodes, find out which inet addr has same IP with your node IP. That is your public interface, which should be added here
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
|
||||
P_IP=('your ip1' 'your ip2')
|
||||
P_MASTER="${P_IP[0]}:your port"
|
||||
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
|
||||
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
|
||||
LOCAL_HOST1=`hostname -I|awk -F " " '{print$1}'`
|
||||
LOCAL_HOST2=`hostname -I|awk -F " " '{print$2}'`
|
||||
for i in "${!P_IP[@]}";
|
||||
do
|
||||
if [[ "$LOCAL_HOST1" == "${P_IP[$i]}" || "$LOCAL_HOST2" == "${P_IP[$i]}" ]];
|
||||
then
|
||||
echo "${P_IP[$i]}"
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 32 --nnodes 2 --node-rank $i --dist-init-addr $P_MASTER \
|
||||
--chunked-prefill-size 16384 --max-prefill-tokens 131072 \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.8\
|
||||
--port 8000 \
|
||||
--served-model-name glm-5 \
|
||||
--cuda-graph-max-bs 16 \
|
||||
--disable-radix-cache
|
||||
NODE_RANK=$i
|
||||
break
|
||||
fi
|
||||
done
|
||||
|
||||
```
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
||||
52
third_party/sglang/docs/platforms/ascend/ascend_npu_quantization.md
vendored
Normal file
52
third_party/sglang/docs/platforms/ascend/ascend_npu_quantization.md
vendored
Normal file
@@ -0,0 +1,52 @@
|
||||
# Quantization on Ascend
|
||||
|
||||
To load already quantized models, simply load the model weights and config. Again, if the model has been quantized offline, there's no need to add `--quantization` argument when starting the engine. The quantization method will be automatically parsed from the downloaded `quant_model_description.json` or `config.json` config.
|
||||
|
||||
SGLang support **mix-bits** quantization (independently defines and loads each layer depending on the type of quantification specified in the `quant_model_description'.json`). [Advanced mix-bits for MoE](https://github.com/sgl-project/sglang/pull/17361) in progress, will add independent quantization determination for the w13 (up-gate) and w2 (down) layers).
|
||||
|
||||
[ModelSlim on Ascend support](https://github.com/sgl-project/sglang/pull/14504)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported | Diffusion models |
|
||||
|-----------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|:------------------------------------------:|
|
||||
| W4A4 dynamic | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| W8A8 static | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| W8A8 dynamic | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: green;">√</span>** |
|
||||
| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | Linear | **<span style="color: red;">x</span>** | **<span style="color: red;">x</span>** | **<span style="color: blue;">WIP</span>** | **<span style="color: blue;">WIP</span>** |
|
||||
| W4A4 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| W4A8 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| W8A8 dynamic | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** | **<span style="color: red;">x</span>** |
|
||||
| [MXFP8](https://github.com/sgl-project/sglang/pull/20922) | MoE | **<span style="color: red;">x</span>** | **<span style="color: red;">x</span>** | **<span style="color: blue;">WIP</span>** | **<span style="color: red;">x</span>** |
|
||||
|
||||
[AWQ on Ascend support](https://github.com/sgl-project/sglang/pull/10158):
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:------------------------------------------:|
|
||||
| W4A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W4A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
GPTQ on Ascend support
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|----------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| [W4A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A16](https://github.com/sgl-project/sglang/pull/15203) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A16 MOE](https://github.com/sgl-project/sglang/pull/16364) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
[Auto-round on Ascend support](https://github.com/sgl-project/sglang/pull/16699)
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|--------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| W4A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W4A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| W8A16 | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
Compressed-tensors (LLM Compressor) on Ascend support:
|
||||
| Quantization scheme | Layer type | A2 Supported | A3 Supported | A5 Supported |
|
||||
|-----------------------------------------------------------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|:-----------------------------------------:|
|
||||
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | Linear | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A8 dynamic with/without activation clip](https://github.com/sgl-project/sglang/pull/14736) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W4A16 MOE](https://github.com/sgl-project/sglang/pull/12759) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
| [W8A8 dynamic](https://github.com/sgl-project/sglang/pull/14504) | MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** | **<span style="color: yellow;">TBD</span>** |
|
||||
|
||||
[GGUF on Ascend support](https://github.com/sgl-project/sglang/pull/17883)
|
||||
|
||||
in progress
|
||||
231
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_5_examples.md
vendored
Normal file
231
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_5_examples.md
vendored
Normal file
@@ -0,0 +1,231 @@
|
||||
# Qwen3.5 examples
|
||||
|
||||
## Environment Preparation
|
||||
|
||||
### Installation
|
||||
|
||||
The dependencies required for the NPU runtime environment have been integrated into a Docker image and uploaded to the quay.io platform. You can directly pull it.
|
||||
|
||||
```{code-block} bash
|
||||
#Atlas 800 A3
|
||||
docker pull quay.io/ascend/sglang:main-cann8.5.0-a3
|
||||
#Atlas 800 A2
|
||||
docker pull quay.io/ascend/sglang:main-cann8.5.0-910b
|
||||
|
||||
#start container
|
||||
docker run -itd --shm-size=16g --privileged=true --name ${NAME} \
|
||||
--privileged=true --net=host \
|
||||
-v /var/queue_schedule:/var/queue_schedule \
|
||||
-v /etc/ascend_install.info:/etc/ascend_install.info \
|
||||
-v /usr/local/sbin:/usr/local/sbin \
|
||||
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
|
||||
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
|
||||
--device=/dev/davinci0:/dev/davinci0 \
|
||||
--device=/dev/davinci1:/dev/davinci1 \
|
||||
--device=/dev/davinci2:/dev/davinci2 \
|
||||
--device=/dev/davinci3:/dev/davinci3 \
|
||||
--device=/dev/davinci4:/dev/davinci4 \
|
||||
--device=/dev/davinci5:/dev/davinci5 \
|
||||
--device=/dev/davinci6:/dev/davinci6 \
|
||||
--device=/dev/davinci7:/dev/davinci7 \
|
||||
--device=/dev/davinci8:/dev/davinci8 \
|
||||
--device=/dev/davinci9:/dev/davinci9 \
|
||||
--device=/dev/davinci10:/dev/davinci10 \
|
||||
--device=/dev/davinci11:/dev/davinci11 \
|
||||
--device=/dev/davinci12:/dev/davinci12 \
|
||||
--device=/dev/davinci13:/dev/davinci13 \
|
||||
--device=/dev/davinci14:/dev/davinci14 \
|
||||
--device=/dev/davinci15:/dev/davinci15 \
|
||||
--device=/dev/davinci_manager:/dev/davinci_manager \
|
||||
--device=/dev/hisi_hdc:/dev/hisi_hdc \
|
||||
--entrypoint=bash \
|
||||
quay.io/ascend/sglang:${tag}
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single-node Deployment
|
||||
|
||||
Run the following script to execute online inference.
|
||||
|
||||
#### Qwen3.5 397B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 16 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Qwen3.5 122B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 8 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Qwen3.5 35B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 2 --nnodes 1 --node-rank 0 \
|
||||
--chunked-prefill-size 4096 --max-prefill-tokens 280000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.7 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 16 \
|
||||
--quantization modelslim \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--dtype bfloat16
|
||||
```
|
||||
|
||||
#### Qwen3.5 27B
|
||||
|
||||
```shell
|
||||
# high performance cpu
|
||||
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
|
||||
sysctl -w vm.swappiness=0
|
||||
sysctl -w kernel.numa_balancing=0
|
||||
sysctl -w kernel.sched_migration_cost_ns=50000
|
||||
# bind cpu
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
|
||||
unset https_proxy
|
||||
unset http_proxy
|
||||
unset HTTPS_PROXY
|
||||
unset HTTP_PROXY
|
||||
unset ASCEND_LAUNCH_BLOCKING
|
||||
# cann
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
source /usr/local/Ascend/nnal/atb/set_env.sh
|
||||
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1000
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export HCCL_SOCKET_IFNAME=lo
|
||||
export GLOO_SOCKET_IFNAME=lo
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path $MODEL_PATH \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--tp-size 2 \
|
||||
--chunked-prefill-size -1 --max-prefill-tokens 120000 \
|
||||
--disable-radix-cache \
|
||||
--trust-remote-code \
|
||||
--host 127.0.0.1 \
|
||||
--mem-fraction-static 0.8 \
|
||||
--port 8000 \
|
||||
--cuda-graph-bs 32 \
|
||||
--enable-multimodal \
|
||||
--mm-attention-backend ascend_attn
|
||||
```
|
||||
|
||||
### Prefill-Decode Disaggregation
|
||||
|
||||
Not test yet.
|
||||
|
||||
### Using Benchmark
|
||||
|
||||
Refer to [Benchmark and Profiling](../../developer_guide/benchmark_and_profiling.md) for details.
|
||||
207
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_examples.md
vendored
Normal file
207
third_party/sglang/docs/platforms/ascend/ascend_npu_qwen3_examples.md
vendored
Normal file
@@ -0,0 +1,207 @@
|
||||
## Qwen3 examples
|
||||
|
||||
### Running Qwen3
|
||||
|
||||
#### Running Qwen3-32B on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
|
||||
#### Running Qwen3-32B on 1 x Atlas 800I A3 with Qwen3-32B-Eagle3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
|
||||
Speculative model weights could be found [here](https://huggingface.co/Zhihu-ai/Zhi-Create-Qwen3-32B-Eagle3)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
|
||||
export SGLANG_ENABLE_SPEC_V2=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-32B \
|
||||
--mem-fraction-static 0.8 \
|
||||
--speculative-algorithm EAGLE3 \
|
||||
--speculative-draft-model-path Qwen/Qwen3-32B-Eagle3 \
|
||||
--speculative-num-steps 1 \
|
||||
--speculative-eagle-topk 1 \
|
||||
--speculative-num-draft-tokens 2
|
||||
```
|
||||
|
||||
#### Running Qwen3-30B-A3B MOE on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-30B-A3B)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--device npu \
|
||||
--attention-backend ascend \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-30B-A3B \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
|
||||
#### Running Qwen3-235B-A22B-Instruct-2507 MOE on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export SGLANG_DEEPEP_NUM_MAX_DISPATCH_TOKENS_PER_RANK=32
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=1
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--model-path Qwen/Qwen3-235B-A22B-Instruct-2507 \
|
||||
--tp-size 16 \
|
||||
--trust-remote-code \
|
||||
--attention-backend ascend \
|
||||
--device npu \
|
||||
--watchdog-timeout 9000 \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
|
||||
#### Running Qwen3-235B-A22B-Instruct-2507 with 256K long sequence on 2 x Atlas 800I A3 without CP
|
||||
|
||||
This example uses **PD disaggregation** for long-sequence inference and keeps **context parallel disabled**.
|
||||
|
||||
Set the shared environment variables on both nodes first:
|
||||
|
||||
```shell
|
||||
export ASCEND_USE_FIA=1
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export ASCEND_MF_STORE_URL="tcp://<PREFILL_HOST_IP>:12345"
|
||||
export HCCL_SOCKET_IFNAME=<NETWORK_IFACE>
|
||||
export GLOO_SOCKET_IFNAME=<NETWORK_IFACE>
|
||||
|
||||
MODEL_PATH=/root/.cache/modelscope/hub/models/zcgy26/Qwen3-235B-A22B-Instruct-2507-w8a8
|
||||
```
|
||||
|
||||
**Prefill node:**
|
||||
|
||||
```shell
|
||||
export ASCEND_LAUNCH_BLOCKING=1
|
||||
export DEEP_NORMAL_MODE_USE_INT8_QUANT=1
|
||||
export HCCL_BUFFSIZE=1500
|
||||
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=1024
|
||||
export DEEPEP_NORMAL_LONG_SEQ_ROUND=128
|
||||
export DEEPEP_NORMAL_COMBINE_ENABLE_LONG_SEQ=1
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode prefill \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--disaggregation-bootstrap-port 8995 \
|
||||
--attention-backend ascend \
|
||||
--disable-radix-cache \
|
||||
--quantization modelslim \
|
||||
--chunked-prefill-size -1 \
|
||||
--skip-server-warmup \
|
||||
--device npu \
|
||||
--tp-size 16 \
|
||||
--mem-fraction-static 0.45 \
|
||||
--max-running-requests 1 \
|
||||
--host <PREFILL_HOST_IP> \
|
||||
--port 8000 \
|
||||
--dist-init-addr <PREFILL_HOST_IP>:5000 \
|
||||
--nnodes 1 \
|
||||
--node-rank 0 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode normal
|
||||
```
|
||||
|
||||
**Decode node:**
|
||||
|
||||
```shell
|
||||
export SGLANG_DEEPEP_BF16_DISPATCH=0
|
||||
export HCCL_BUFFSIZE=4000
|
||||
export DEEPEP_NORMAL_LONG_SEQ_PER_ROUND_TOKENS=4096
|
||||
export DEEPEP_NORMAL_LONG_SEQ_ROUND=16
|
||||
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path ${MODEL_PATH} \
|
||||
--disaggregation-mode decode \
|
||||
--disaggregation-transfer-backend ascend \
|
||||
--attention-backend ascend \
|
||||
--mem-fraction-static 0.8 \
|
||||
--disable-cuda-graph \
|
||||
--device npu \
|
||||
--disable-radix-cache \
|
||||
--quantization modelslim \
|
||||
--chunked-prefill-size 8192 \
|
||||
--skip-server-warmup \
|
||||
--tp-size 16 \
|
||||
--max-running-requests 1 \
|
||||
--host <DECODE_HOST_IP> \
|
||||
--port 8232 \
|
||||
--moe-a2a-backend deepep \
|
||||
--deepep-mode low_latency \
|
||||
--disable-overlap-schedule
|
||||
```
|
||||
|
||||
**Router:**
|
||||
|
||||
```shell
|
||||
python3 -m sglang_router.launch_router \
|
||||
--pd-disaggregation \
|
||||
--policy cache_aware \
|
||||
--prefill http://<PREFILL_HOST_IP>:8000 8995 \
|
||||
--decode http://<DECODE_HOST_IP>:8232 \
|
||||
--host <ROUTER_HOST_IP> \
|
||||
--port 6689 \
|
||||
--prometheus-port 29010
|
||||
```
|
||||
|
||||
#### Running Qwen3-VL-8B-Instruct on 1 x Atlas 800I A3.
|
||||
|
||||
Model weights could be found [here](https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct)
|
||||
|
||||
```shell
|
||||
export SGLANG_SET_CPU_AFFINITY=1
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export STREAMS_PER_DEVICE=32
|
||||
export HCCL_BUFFSIZE=1536
|
||||
export HCCL_OP_EXPANSION_MODE=AIV
|
||||
|
||||
python -m sglang.launch_server \
|
||||
--enable-multimodal \
|
||||
--attention-backend ascend \
|
||||
--mm-attention-backend ascend_attn \
|
||||
--trust-remote-code \
|
||||
--tp-size 4 \
|
||||
--model-path Qwen/Qwen3-VL-8B-Instruct \
|
||||
--mem-fraction-static 0.8
|
||||
```
|
||||
19
third_party/sglang/docs/platforms/ascend/ascend_npu_support.rst
vendored
Normal file
19
third_party/sglang/docs/platforms/ascend/ascend_npu_support.rst
vendored
Normal file
@@ -0,0 +1,19 @@
|
||||
Ascend NPUs
|
||||
===============================================================
|
||||
|
||||
.. toctree::
|
||||
:maxdepth: 1
|
||||
|
||||
ascend_npu.md
|
||||
ascend_npu_support_features.md
|
||||
ascend_npu_support_models.md
|
||||
ascend_npu_quantization.md
|
||||
ascend_npu_deepseek_example.md
|
||||
ascend_npu_qwen3_examples.md
|
||||
mindspore_backend.md
|
||||
ascend_contribution_guide.md
|
||||
ascend_npu_best_practice.md
|
||||
ascend_npu_ring_sp_performance.md
|
||||
ascend_npu_qwen3_5_examples.md
|
||||
ascend_npu_glm5_examples.md
|
||||
ascend_npu_environment_variables.md
|
||||
488
third_party/sglang/docs/platforms/ascend/ascend_npu_support_features.md
vendored
Normal file
488
third_party/sglang/docs/platforms/ascend/ascend_npu_support_features.md
vendored
Normal file
@@ -0,0 +1,488 @@
|
||||
# Support Features on Ascend NPU
|
||||
|
||||
This section describes the basic functions and features supported by the Ascend NPU.If you encounter issues or have any
|
||||
questions, please [open an issue](https://github.com/sgl-project/sglang/issues).
|
||||
|
||||
If you want to know the meaning and usage of each parameter,
|
||||
click [Server Arguments](https://docs.sglang.io/advanced_features/server_arguments.html).
|
||||
|
||||
## Model and tokenizer
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------|----------|---------------------------------------|:----------------:|
|
||||
| `--model-path`<br/>`--model` | `None` | Type: str | A2, A3 |
|
||||
| `--tokenizer-path` | `None` | Type: str | A2, A3 |
|
||||
| `--tokenizer-mode` | `auto` | `auto`, `slow` | A2, A3 |
|
||||
| `--tokenizer-worker-num` | `1` | Type: int | A2, A3 |
|
||||
| `--skip-tokenizer-init` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--load-format` | `auto` | `auto`, `safetensors` | A2, A3 |
|
||||
| `--model-loader-` <br/> `extra-config` | `{}` | Type: str | A2, A3 |
|
||||
| `--trust-remote-code` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--context-length` | `None` | Type: int | A2, A3 |
|
||||
| `--is-embedding` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--enable-multimodal` | `None` | bool flag (set to enable) | A2, A3 |
|
||||
| `--revision` | `None` | Type: str | A2, A3 |
|
||||
| `--model-impl` | `auto` | `auto`, `sglang`,<br/> `transformers` | A2, A3 |
|
||||
|
||||
## HTTP server
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------|-------------|---------------------------|:----------------:|
|
||||
| `--host` | `127.0.0.1` | Type: str | A2, A3 |
|
||||
| `--port` | `30000` | Type: int | A2, A3 |
|
||||
| `--skip-server-warmup` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--warmups` | `None` | Type: str | A2, A3 |
|
||||
| `--nccl-port` | `None` | Type: int | A2, A3 |
|
||||
| `--fastapi-root-path` | `None` | Type: str | A2, A3 |
|
||||
| `--grpc-mode` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
|
||||
## Quantization and data type
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------------|----------|-----------------------------------------|:----------------:|
|
||||
| `--dtype` | `auto` | `auto`,<br/> `float16`,<br/> `bfloat16` | A2, A3 |
|
||||
| `--quantization` | `None` | `modelslim` | A2, A3 |
|
||||
| `--quantization-param-path` | `None` | Type: str | Special For GPU |
|
||||
| `--kv-cache-dtype` | `auto` | `auto` | A2, A3 |
|
||||
| `--enable-fp32-lm-head` | `False` | bool flag <br/> (set to enable) | A2, A3 |
|
||||
| `--modelopt-quant` | `None` | Type: str | Special For GPU |
|
||||
| `--modelopt-checkpoint-`<br/>`restore-path` | `None` | Type: str | Special For GPU |
|
||||
| `--modelopt-checkpoint-`<br/>`save-path` | `None` | Type: str | Special For GPU |
|
||||
| `--modelopt-export-path` | `None` | Type: str | Special For GPU |
|
||||
| `--quantize-and-serve` | `False` | bool flag <br/> (set to enable) | Special For GPU |
|
||||
| `--rl-quant-profile` | `None` | Type: str | Special For GPU |
|
||||
|
||||
## Memory and scheduling
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-----------------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--mem-fraction-static` | `None` | Type: float | A2, A3 |
|
||||
| `--max-running-requests` | `None` | Type: int | A2, A3 |
|
||||
| `--prefill-max-requests` | `None` | Type: int | A2, A3 |
|
||||
| `--max-queued-requests` | `None` | Type: int | A2, A3 |
|
||||
| `--max-total-tokens` | `None` | Type: int | A2, A3 |
|
||||
| `--chunked-prefill-size` | `None` | Type: int | A2, A3 |
|
||||
| `--max-prefill-tokens` | `16384` | Type: int | A2, A3 |
|
||||
| `--schedule-policy` | `fcfs` | `lpm`, `fcfs` | A2, A3 |
|
||||
| `--enable-priority-`<br/>`scheduling` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--schedule-low-priority-`<br/>`values-first` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--priority-scheduling-`<br/>`preemption-threshold` | `10` | Type: int | A2, A3 |
|
||||
| `--schedule-conservativeness` | `1.0` | Type: float | A2, A3 |
|
||||
| `--page-size` | `128` | Type: int | A2, A3 |
|
||||
| `--swa-full-tokens-ratio` | `0.8` | Type: float | A2, A3 |
|
||||
| `--disable-hybrid-swa-memory` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--radix-eviction-policy` | `lru` | `lru`,<br/>`lfu` | A2, A3 |
|
||||
| `--enable-prefill-delayer` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--prefill-delayer-max-delay-passes` | `30` | Type: int | A2, A3 |
|
||||
| `--prefill-delayer-token-usage-low-watermark` | `None` | Type: float | A2, A3 |
|
||||
| `--prefill-delayer-forward-passes-buckets` | `None` | List[float] | A2, A3 |
|
||||
| `--prefill-delayer-wait-seconds-buckets` | `None` | List[float] | A2, A3 |
|
||||
| `--abort-on-priority-`<br/>`when-disabled` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-dynamic-chunking` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
|
||||
## Runtime options
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------------------|----------|---------------------------|:----------------:|
|
||||
| `--device` | `None` | Type: str | A2, A3 |
|
||||
| `--tensor-parallel-size`<br/>`--tp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pipeline-parallel-size`<br/>`--pp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--attention-context-parallel-size`<br/>`--attn-cp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--moe-data-parallel-size`<br/>`--moe-dp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--pp-max-micro-batch-size` | `None` | Type: int | A2, A3 |
|
||||
| `--pp-async-batch-depth` | `None` | Type: int | A2, A3 |
|
||||
| `--stream-interval` | `1` | Type: int | A2, A3 |
|
||||
| `--incremental-streaming-output` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--random-seed` | `None` | Type: int | A2, A3 |
|
||||
| `--constrained-json-`<br/>`whitespace-pattern` | `None` | Type: str | A2, A3 |
|
||||
| `--constrained-json-`<br/>`disable-any-whitespace` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--watchdog-timeout` | `300` | Type: float | A2, A3 |
|
||||
| `--soft-watchdog-timeout` | `300` | Type: float | A2, A3 |
|
||||
| `--dist-timeout` | `None` | Type: int | A2, A3 |
|
||||
| `--download-dir` | `None` | Type: str | A2, A3 |
|
||||
| `--model-checksum` | `None` | Type: str | A2, A3 |
|
||||
| `--base-gpu-id` | `0` | Type: int | A2, A3 |
|
||||
| `--gpu-id-step` | `1` | Type: int | A2, A3 |
|
||||
| `--sleep-on-idle` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
|
||||
## Logging
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------------------|-------------------|--------------------------------|:----------------:|
|
||||
| `--log-level` | `info` | Type: str | A2, A3 |
|
||||
| `--log-level-http` | `None` | Type: str | A2, A3 |
|
||||
| `--log-requests` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--log-requests-level` | `2` | `0`, `1`, `2`, `3` | A2, A3 |
|
||||
| `--log-requests-format` | `text` | `text`, `json` | A2, A3 |
|
||||
| `--crash-dump-folder` | `None` | Type: str | A2, A3 |
|
||||
| `--enable-metrics` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-metrics-for-`<br/>`all-schedulers` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--tokenizer-metrics-`<br/>`custom-labels-header` | `x-custom-labels` | Type: str | A2, A3 |
|
||||
| `--tokenizer-metrics-`<br/>`allowed-custom-labels` | `None` | List[str] | A2, A3 |
|
||||
| `--bucket-time-to-`<br/>`first-token` | `None` | List[float] | A2, A3 |
|
||||
| `--bucket-inter-token-`<br/>`latency` | `None` | List[float] | A2, A3 |
|
||||
| `--bucket-e2e-request-`<br/>`latency` | `None` | List[float] | A2, A3 |
|
||||
| `--collect-tokens-`<br/>`histogram` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--prompt-tokens-buckets` | `None` | List[str] | A2, A3 |
|
||||
| `--generation-tokens-buckets` | `None` | List[str] | A2, A3 |
|
||||
| `--gc-warning-threshold-secs` | `0.0` | Type: float | A2, A3 |
|
||||
| `--decode-log-interval` | `40` | Type: int | A2, A3 |
|
||||
| `--enable-request-time-`<br/>`stats-logging` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--kv-events-config` | `None` | Type: str | Special for GPU |
|
||||
| `--enable-trace` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--oltp-traces-endpoint` | `localhost:4317` | Type: str | A2, A3 |
|
||||
| `--log-requests-target` | `None` | Type: str | A2, A3 |
|
||||
| `--uvicorn-access-log-exclude-prefixes` | `[]` | List[str] | A2, A3 |
|
||||
|
||||
## RequestMetricsExporter configuration
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--export-metrics-to-`<br/>`file` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--export-metrics-to-`<br/>`file-dir` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## API related
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------|-----------|--------------------------------|:----------------:|
|
||||
| `--api-key` | `None` | Type: str | A2, A3 |
|
||||
| `--admin-api-key` | `None` | Type: str | A2, A3 |
|
||||
| `--served-model-name` | `None` | Type: str | A2, A3 |
|
||||
| `--weight-version` | `default` | Type: str | A2, A3 |
|
||||
| `--chat-template` | `None` | Type: str | A2, A3 |
|
||||
| `--hf-chat-template-name` | `None` | Type: str | A2, A3 |
|
||||
| `--completion-template` | `None` | Type: str | A2, A3 |
|
||||
| `--enable-cache-report` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--reasoning-parser` | `None` | `deepseek-r1`<br/>`deepseek-v3`<br/>`glm45`<br/>`gpt-oss`<br/>`kimi`<br/>`qwen3`<br/>`qwen3-thinking`<br/>`step3` | A2, A3 |
|
||||
| `--tool-call-parser` | `None` | `deepseekv3`<br/>`deepseekv31`<br/>`glm`<br/>`glm45`<br/>`glm47`<br/>`gpt-oss`<br/>`kimi_k2`<br/>`llama3`<br/>`mistral`<br/>`pythonic`<br/>`qwen`<br/>`qwen25`<br/>`qwen3_coder`<br/>`step3`<br/>`gigachat3` | A2, A3 |
|
||||
| `--sampling-defaults` | `model` | `openai`, `model` | A2, A3 |
|
||||
|
||||
## Data parallelism
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------|---------------|-----------------------------------------------------------|:----------------:|
|
||||
| `--data-parallel-size`<br/>`--dp-size` | `1` | Type: int | A2, A3 |
|
||||
| `--load-balance-method` | `auto` | `auto`,<br/> `round_robin`,<br/> `follow_bootstrap_room`,<br/> `total_requests`,<br/> `total_tokens` | A2, A3 |
|
||||
|
||||
## Multi-node distributed serving
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------|----------|-----------|:----------------:|
|
||||
| `--dist-init-addr`<br/>`--nccl-init-addr` | `None` | Type: str | A2, A3 |
|
||||
| `--nnodes` | `1` | Type: int | A2, A3 |
|
||||
| `--node-rank` | `0` | Type: int | A2, A3 |
|
||||
|
||||
## Model override args
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------------------|----------|-----------|:----------------:|
|
||||
| `--json-model-override-`<br/>`args` | `{}` | Type: str | A2, A3 |
|
||||
| `--preferred-sampling-`<br/>`params` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## LoRA
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------|----------|-------------------------------------|:----------------:|
|
||||
| `--enable-lora` | `False` | Bool flag <br/>(set to enable) | A2, A3 |
|
||||
| `--max-lora-rank` | `None` | Type: int | A2, A3 |
|
||||
| `--lora-target-modules` | `None` | `all` | A2, A3 |
|
||||
| `--lora-paths` | `None` | Type: List[str] /<br/> JSON objects | A2, A3 |
|
||||
| `--max-loras-per-batch` | `8` | Type: int | A2, A3 |
|
||||
| `--max-loaded-loras` | `None` | Type: int | A2, A3 |
|
||||
| `--lora-eviction-policy` | `lru` | `lru`,<br/> `fifo` | A2, A3 |
|
||||
| `--lora-backend` | `csgmv` | `triton`,<br/>`csgmv`,<br/>`ascend`,<br/>`torch_native` | A2, A3 |
|
||||
| `--max-lora-chunk-size` | `16` | `16`, `32`,<br/> `64`, `128` | Special for GPU |
|
||||
|
||||
## Kernel Backends (Attention, Sampling, Grammar, GEMM)
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------|-------------------|------------------------------------------------------------------------------------------------|:----------------:|
|
||||
| `--attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--prefill-attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--decode-attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--sampling-backend` | `None` | `pytorch`,<br/>`ascend` | A2, A3 |
|
||||
| `--grammar-backend` | `None` | `xgrammar` | A2, A3 |
|
||||
| `--mm-attention-backend` | `None` | `ascend_attn` | A2, A3 |
|
||||
| `--nsa-prefill-backend` | `flashmla_sparse` | `flashmla_sparse`,<br/> `flashmla_decode`,<br/>`fa3`,<br/> `tilelang`,<br/> `aiter` | Special for GPU |
|
||||
| `--nsa-decode-backend` | `fa3` | `flashmla_prefill`,<br/> `flashmla_kv`,<br/> `fa3`,<br/>`tilelang`,<br/> `aiter` | Special for GPU |
|
||||
| `--fp8-gemm-backend` | `auto` | `auto`,<br/> `deep_gemm`,<br/> `flashinfer_trtllm`,<br/>`flashinfer_cutlass`,<br/>`flashinfer_deepgemm`,<br/>`cutlass`,<br/> `triton`,<br/> `aiter` | Special for GPU |
|
||||
| `--disable-flashinfer-`<br/>`autotune` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## Speculative decoding
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------------------------------------------------|-----------|--------------------------|:----------------:|
|
||||
| `--speculative-algorithm` | `None` | `EAGLE3`,<br/> `NEXTN` | A2, A3 |
|
||||
| `--speculative-draft-model-path`<br/>`--speculative-draft-model` | `None` | Type: str | A2, A3 |
|
||||
| `--speculative-draft-model-`<br/>`revision` | `None` | Type: str | A2, A3 |
|
||||
| `--speculative-draft-load-format` | `None` | `auto` | A2, A3 |
|
||||
| `--speculative-num-steps` | `None` | Type: int | A2, A3 |
|
||||
| `--speculative-eagle-topk` | `None` | Type: int | A2, A3 |
|
||||
| `--speculative-num-draft-tokens` | `None` | Type: int | A2, A3 |
|
||||
| `--speculative-accept-`<br/>`threshold-single` | `1.0` | Type: float | Special for GPU |
|
||||
| `--speculative-accept-`<br/>`threshold-acc` | `1.0` | Type: float | Special for GPU |
|
||||
| `--speculative-token-map` | `None` | Type: str | A2, A3 |
|
||||
| `--speculative-attention-`<br/>`mode` | `prefill` | `prefill`,<br/> `decode` | A2, A3 |
|
||||
| `--speculative-moe-runner-`<br/>`backend` | `None` | `auto` | A2, A3 |
|
||||
| `--speculative-moe-a2a-`<br/>`backend` | `None` | `ascend_fuseep` | A2, A3 |
|
||||
| `--speculative-draft-attention-backend` | `None` | `ascend` | A2, A3 |
|
||||
| `--speculative-draft-model-quantization` | `None` | `unquant` | A2, A3 |
|
||||
|
||||
## Ngram speculative decoding
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------------------------|------------|--------------------|:----------------:|
|
||||
| `--speculative-ngram-`<br/>`min-match-window-size` | `1` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`max-match-window-size` | `12` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`min-bfs-breadth` | `1` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`max-bfs-breadth` | `10` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`match-type` | `BFS` | `BFS`,<br/> `PROB` | Experimental. `BFS` uses recency-based expansion; `PROB` uses frequency-based expansion. |
|
||||
| `--speculative-ngram-`<br/>`max-trie-depth` | `18` | Type: int | Experimental |
|
||||
| `--speculative-ngram-`<br/>`capacity` | `10000000` | Type: int | Experimental |
|
||||
|
||||
## Expert parallelism
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------------------|-----------|---------------------------------------------|:----------------:|
|
||||
| `--expert-parallel-size`<br/>`--ep-size`<br/>`--ep` | `1` | Type: int | A2, A3 |
|
||||
| `--moe-a2a-backend` | `none` | `none`,<br/> `deepep`,<br/> `ascend_fuseep` | A2, A3 |
|
||||
| `--moe-runner-backend` | `auto` | `auto`, `triton` | A2, A3 |
|
||||
| `--flashinfer-mxfp4-`<br/>`moe-precision` | `default` | `default`,<br/> `bf16` | Special for GPU |
|
||||
| `--enable-flashinfer-`<br/>`allreduce-fusion` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--deepep-mode` | `auto` | `normal`, <br/>`low_latency`,<br/> `auto` | A2, A3 |
|
||||
| `--deepep-config` | `None` | Type: str | Special for GPU |
|
||||
| `--ep-num-redundant-experts` | `0` | Type: int | A2, A3 |
|
||||
| `--ep-dispatch-algorithm` | `None` | Type: str | A2, A3 |
|
||||
| `--init-expert-location` | `trivial` | Type: str | A2, A3 |
|
||||
| `--enable-eplb` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--eplb-algorithm` | `auto` | Type: str | A2, A3 |
|
||||
| `--eplb-rebalance-layers-`<br/>`per-chunk` | `None` | Type: int | A2, A3 |
|
||||
| `--eplb-min-rebalancing-`<br/>`utilization-threshold` | `1.0` | Type: float | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-mode` | `None` | Type: str | A2, A3 |
|
||||
| `--expert-distribution-`<br/>`recorder-buffer-size` | `None` | Type: int | A2, A3 |
|
||||
| `--enable-expert-distribution-`<br/>`metrics` | `False` | bool flag (set to enable) | A2, A3 |
|
||||
| `--moe-dense-tp-size` | `None` | Type: int | A2, A3 |
|
||||
| `--elastic-ep-backend` | `None` | `none`, `mooncake` | Special for GPU |
|
||||
| `--mooncake-ib-device` | `None` | Type: str | Special for GPU |
|
||||
|
||||
## Mamba Cache
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------------|-----------|-----------------------------------------------|:----------------:|
|
||||
| `--max-mamba-cache-size` | `None` | Type: int | A2, A3 |
|
||||
| `--mamba-ssm-dtype` | `float32` | `float32`,<br/>`bfloat16`,<br/>`float16` | A2, A3 |
|
||||
| `--mamba-full-memory-ratio` | `0.9` | Type: float | A2, A3 |
|
||||
| `--mamba-scheduler-strategy` | `auto` | Only `auto`, `no_buffer` supported | A2, A3 |
|
||||
| `--mamba-track-interval` | `256` | Type: int | A2, A3 |
|
||||
|
||||
## Hierarchical cache
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------------|-----------------|---------------------------------------------------------------------|:----------------:|
|
||||
| `--enable-hierarchical-`<br/>`cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--hicache-ratio` | `2.0` | Type: float | A2, A3 |
|
||||
| `--hicache-size` | `0` | Type: int | A2, A3 |
|
||||
| `--hicache-write-policy` | `write_through` | Currently only `write_back` supported | A2, A3 |
|
||||
| `--hicache-io-backend` | `kernel` | `kernel_ascend`,<br/> `direct` | A2, A3 |
|
||||
| `--hicache-mem-layout` | `layer_first` | `page_first_direct`,<br/> `page_first_kv_split` | A2, A3 |
|
||||
| `--hicache-storage-`<br/>`backend` | `None` | `file` | A2, A3 |
|
||||
| `--hicache-storage-`<br/>`prefetch-policy` | `best_effort` | `best_effort`,<br/> `wait_complete`,<br/> `timeout` | Special for GPU |
|
||||
| `--hicache-storage-`<br/>`backend-extra-config` | `None` | Type: str | Special for GPU |
|
||||
|
||||
## LMCache
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------|----------|--------------------------------|:----------------:|
|
||||
| `--enable-lmcache` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## Offloading
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------|----------|-----------|:----------------:|
|
||||
| `--cpu-offload-gb` | `0` | Type: int | A2, A3 |
|
||||
| `--offload-group-size` | `-1` | Type: int | Planned |
|
||||
| `--offload-num-in-group` | `1` | Type: int | Planned |
|
||||
| `--offload-prefetch-step` | `1` | Type: int | Planned |
|
||||
| `--offload-mode` | `cpu` | Type: str | Planned |
|
||||
|
||||
## Args for multi-item scoring
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|----------------------------------|----------|-----------|:----------------:|
|
||||
| `--multi-item-scoring-delimiter` | `None` | Type: int | A2, A3 |
|
||||
|
||||
## Optimization/debug options
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--disable-radix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--cuda-graph-max-bs` | `None` | Type: int | A2, A3 |
|
||||
| `--cuda-graph-bs` | `None` | List[int] | A2, A3 |
|
||||
| `--disable-cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-cuda-graph-`<br/>`padding` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-profile-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-cudagraph-gc` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-nccl-nvls` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-flashinfer-`<br/>`cutlass-moe-fp4-allgather` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-tokenizer-`<br/>`batch-encode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-tokenizer-`<br/>`batch-decode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-custom-`<br/>`all-reduce` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-mscclpp` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-torch-`<br/>`symm-mem` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-overlap`<br/>`-schedule` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-mixed-`<br/>`chunk` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-dp-attention` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-dp-lm-head` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-two-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | Planned |
|
||||
| `--enable-single-`<br/>`batch-overlap` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--tbo-token-`<br/>`distribution-threshold` | `0.48` | Type: float | Planned |
|
||||
| `--enable-torch-`<br/>`compile` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-torch-`<br/>`compile-debug-mode` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-piecewise-`<br/>`cuda-graph` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--piecewise-cuda-`<br/>`graph-tokens` | `None` | Type: JSON<br/> list | A2, A3 |
|
||||
| `--piecewise-cuda-`<br/>`graph-compiler` | `eager` | ["eager", "inductor"] | A2, A3 |
|
||||
| `--torch-compile-max-bs` | `32` | Type: int | A2, A3 |
|
||||
| `--piecewise-cuda-`<br/>`graph-max-tokens` | `None` | Type: int | A2, A3 |
|
||||
| `--torchao-config` | `` | Type: str | Special for GPU |
|
||||
| `--enable-nan-detection` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-p2p-check` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--triton-attention-`<br/>`reduce-in-fp32` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--triton-attention-`<br/>`num-kv-splits` | `8` | Type: int | Special for GPU |
|
||||
| `--triton-attention-`<br/>`split-tile-size` | `None` | Type: int | Special for GPU |
|
||||
| `--delete-ckpt-`<br/>`after-loading` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-memory-saver` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-draft-weights-`<br/>`cpu-backup` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--allow-auto-truncate` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-custom-`<br/>`logit-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--flashinfer-mla-`<br/>`disable-ragged` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--disable-shared-`<br/>`experts-fusion` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-chunked-`<br/>`prefix-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--disable-fast-`<br/>`image-processor` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--keep-mm-feature-`<br/>`on-device` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-return-`<br/>`hidden-states` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-return-`<br/>`routed-experts` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--scheduler-recv-`<br/>`interval` | `1` | Type: int | A2, A3 |
|
||||
| `--numa-node` | `None` | List[int] | A2, A3 |
|
||||
| `--enable-deterministic-`<br/>`inference` | `False` | bool flag<br/> (set to enable) | Planned |
|
||||
| `--rl-on-policy-target` | `None` | `fsdp` | Planned |
|
||||
| `--enable-layerwise-`<br/>`nvtx-marker` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--enable-attn-tp-`<br/>`input-scattered` | `False` | bool flag<br/> (set to enable) | Experimental |
|
||||
| `--enable-nsa-prefill-`<br/>`context-parallel` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--enable-fused-qk-`<br/>`norm-rope` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## Dynamic batch tokenizer
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--enable-dynamic-`<br/>`batch-tokenizer` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--dynamic-batch-`<br/>`tokenizer-batch-size` | `32` | Type: int | A2, A3 |
|
||||
| `--dynamic-batch-`<br/>`tokenizer-batch-timeout` | `0.002` | Type: float | A2, A3 |
|
||||
|
||||
## Debug tensor dumps
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|--------------------------------------------|----------|-----------|:----------------:|
|
||||
| `--debug-tensor-dump-`<br/>`output-folder` | `None` | Type: str | A2, A3 |
|
||||
| `--debug-tensor-dump-`<br/>`layers` | `None` | List[int] | A2, A3 |
|
||||
| `--debug-tensor-dump-`<br/>`input-file` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## PD disaggregation
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------------------------------|------------|---------------------------------------|:----------------:|
|
||||
| `--disaggregation-mode` | `null` | `null`,<br/> `prefill`,<br/> `decode` | A2, A3 |
|
||||
| `--disaggregation-transfer-backend` | `mooncake` | `ascend` | A2, A3 |
|
||||
| `--disaggregation-bootstrap-port` | `8998` | Type: int | A2, A3 |
|
||||
| `--disaggregation-ib-device` | `None` | Type: str | Special for GPU |
|
||||
| `--disaggregation-decode-`<br/>`enable-offload-kvcache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--num-reserved-decode-tokens` | `512` | Type: int | A2, A3 |
|
||||
| `--disaggregation-decode-`<br/>`polling-interval` | `1` | Type: int | A2, A3 |
|
||||
|
||||
## Encode prefill disaggregation
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------------------------|--------------------|----------------------------------------------------------------|:----------------:|
|
||||
| `--encoder-only` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--language-only` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--encoder-transfer-backend` | `zmq_to_scheduler` | `zmq_to_scheduler`, <br/> `zmq_to_tokenizer`,<br/> `mooncake` | A2, A3 |
|
||||
| `--encoder-urls` | `[]` | List[str] | A2, A3 |
|
||||
|
||||
## Custom weight loader
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------------------------------------------------------------|----------|---------------------------------|:----------------:|
|
||||
| `--custom-weight-loader` | `None` | List[str] | A2, A3 |
|
||||
| `--weight-loader-disable-`<br/>`mmap` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-ip` | `None` | Type: str | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-seed-instance-service-port` | `None` | Type: int | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-send-weights-group-ports` | `None` | Type: JSON<br/> list | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-backend` | `nccl` | `transfer_engine`, <br/> `nccl` | A2, A3 |
|
||||
| `--remote-instance-weight-`<br/>`loader-start-seed-via-transfer-engine` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
|
||||
## For PD-Multiplexing
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-----------------------|----------|--------------------------------|:----------------:|
|
||||
| `--enable-pdmux` | `False` | bool flag<br/> (set to enable) | Special for GPU |
|
||||
| `--pdmux-config-path` | `None` | Type: str | Special for GPU |
|
||||
| `--sm-group-num` | `8` | Type: int | Special for GPU |
|
||||
|
||||
## For Multi-Modal
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-----------------------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--mm-max-concurrent-calls` | `32` | Type: int | A2, A3 |
|
||||
| `--mm-per-request-timeout` | `10.0` | Type: float | A2, A3 |
|
||||
| `--enable-broadcast-mm-`<br/>`inputs-process` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--mm-process-config` | `None` | Type: JSON / Dict | A2, A3 |
|
||||
| `--mm-enable-dp-encoder` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
| `--limit-mm-data-per-request` | `None` | Type: JSON / Dict | A2, A3 |
|
||||
|
||||
## For checkpoint decryption
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|---------------------------------|----------|--------------------------------|:----------------:|
|
||||
| `--decrypted-config-file` | `None` | Type: str | A2, A3 |
|
||||
| `--decrypted-draft-config-file` | `None` | Type: str | A2, A3 |
|
||||
| `--enable-prefix-mm-cache` | `False` | bool flag<br/> (set to enable) | A2, A3 |
|
||||
|
||||
## Forward hooks
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|-------------------|----------|-----------------|:----------------:|
|
||||
| `--forward-hooks` | `None` | Type: JSON list | A2, A3 |
|
||||
|
||||
## Configuration file support
|
||||
|
||||
| Argument | Defaults | Options | Server supported |
|
||||
|------------|----------|-----------|:----------------:|
|
||||
| `--config` | `None` | Type: str | A2, A3 |
|
||||
|
||||
## Other Params
|
||||
|
||||
The following parameters are not supported because the third-party components that depend on are not compatible with the
|
||||
NPU, like Ktransformer, checkpoint-engine etc.
|
||||
|
||||
| Argument | Defaults | Options |
|
||||
|-------------------------------------------------------------------|-----------|---------------------------|
|
||||
| `--checkpoint-engine-` <br/> `wait-weights-` <br/> `before-ready` | `False` | bool flag (set to enable) |
|
||||
| `--kt-weight-path` | `None` | Type: str |
|
||||
| `--kt-method` | `AMXINT4` | Type: str |
|
||||
| `--kt-cpuinfer` | `None` | Type: int |
|
||||
| `--kt-threadpool-count` | `2` | Type: int |
|
||||
| `--kt-num-gpu-experts` | `None` | Type: int |
|
||||
| `--kt-max-deferred-`<br/>`experts-per-token` | `None` | Type: int |
|
||||
|
||||
The following parameters have some functional deficiencies on community
|
||||
|
||||
| Argument | Defaults | Options |
|
||||
|---------------------------------------|----------|--------------------------------|
|
||||
| `--enable-double-sparsity` | `False` | bool flag<br/> (set to enable) |
|
||||
| `--ds-channel-config-path` | `None` | Type: str |
|
||||
| `--ds-heavy-channel-num` | `32` | Type: int |
|
||||
| `--ds-heavy-token-num` | `256` | Type: int |
|
||||
| `--ds-heavy-channel-type` | `qk` | Type: str |
|
||||
| `--ds-sparse-decode-`<br/>`threshold` | `4096` | Type: int |
|
||||
| `--tool-server` | `None` | Type: str |
|
||||
110
third_party/sglang/docs/platforms/ascend/ascend_npu_support_models.md
vendored
Normal file
110
third_party/sglang/docs/platforms/ascend/ascend_npu_support_models.md
vendored
Normal file
@@ -0,0 +1,110 @@
|
||||
# Support Models on Ascend NPU
|
||||
|
||||
This section describes the models supported on the Ascend NPU, including Large Language Models, Multimodal Language
|
||||
Models, Embedding Models, Reward Models and Rerank Models. Mainstream DeepSeek/Qwen/GLM series are included.
|
||||
You are welcome to enable various models based on your business requirements.
|
||||
|
||||
## Large Language Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|--------------------------------------------|--------------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| DeepSeek V3/V3.1 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek-V3.2-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek-R1-0528-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| DeepSeek-V2-Lite-W8A8 | DeepSeek | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-30B-A3B-Instruct-2507 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-32B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-0.6B | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-235B-A22B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-Next-80B-A3B-Instruct | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen3-Coder-480B-A35B-Instruct-w8a8-QuaRot | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-7B-Instruct | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| QWQ-32B-W8A8 | Qwen | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/Llama-3.1-8B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/llama-2-7b | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/Llama-3.2-1B-Instruct | Llama | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-7B-Instruct-v0.2 | Mistral | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| allenai/OLMoE-1B-7B-0924 | OLMoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| stabilityai/stablelm-2-1_6b | StableLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| CohereForAI/c4ai-command-r-v01 | Command-R | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| huihui-ai/grok-2 | Grok | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/chatglm2-6b | ChatGLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b | InternLM 2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LGAI-EXAONE/EXAONE-3.5-7.8B-Instruct | ExaONE 3 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| xverse/XVERSE-MoE-A36B | XVERSE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| HuggingFaceTB/SmolLM-1.7B | SmolLM | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/glm-4-9b-chat | GLM-4 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| XiaomiMiMo/MiMo-7B-RL | MiMo | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| arcee-ai/AFM-4.5B-Base | Arcee AFM-4.5B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Howeee/persimmon-8b-chat | Persimmon | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| inclusionAI/Ling-lite | Ling | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ibm-granite/granite-3.1-8b-instruct | Granite | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ibm-granite/granite-3.0-3b-a800m-instruct | Granite MoE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/dbrx-instruct | DBRX (Databricks) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| baichuan-inc/Baichuan2-13B-Chat | Baichuan 2 (7B, 13B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| baidu/ERNIE-4.5-21B-A3B-PT | ERNIE-4.5 (4.5, 4.5MoE series) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| OpenBMB/MiniCPM3-4B | MiniCPM (v3, 4B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-K2-Thinking | Kimi | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| eigen-ai-labs/gpt-oss-120b-bf16 | GPTOSS | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| allenai/OLMo-2-1124-7B-Instruct | OLMo | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| cyankiwi/MiniMax-M2-BF16 | MiniMax-M2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| upstage/SOLAR-10.7B-Instruct-v1.0 | Solar | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| bigcode/starcoder2-7b | StarCoder2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| arcee-ai/Trinity-Mini | Trinity (Nano, Mini) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Multimodal Language Models
|
||||
|
||||
| Models | Model Family (Variants) | A2 Supported | A3 Supported |
|
||||
|-----------------------------------------------|---------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| Qwen/Qwen2.5-VL-3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-VL-72B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-30B-A3B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-8B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-4B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-VL-235B-A22B-Instruct | Qwen-VL | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/deepseek-vl2 | DeepSeek-VL2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-1B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| deepseek-ai/Janus-Pro-7B | Janus-Pro (1B, 7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-V-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| openbmb/MiniCPM-o-2_6 | MiniCPM-V / MiniCPM-o | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| google/gemma-3-4b-it | Gemma 3 (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| mistralai/Mistral-Small-3.1-24B-Instruct-2503 | Mistral-Small-3.1-24B | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| microsoft/Phi-4-multimodal-instruct | Phi-4-multimodal-instruct | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| XiaomiMiMo/MiMo-VL-7B-RL | MiMo-VL (7B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/llava-v1.6-34b | LLaVA (v1.5 & v1.6) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-next-72b | LLaVA-NeXT (8B, 72B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| lmms-lab/llava-onevision-qwen2-7b-ov | LLaVA-OneVision | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Kimi/Kimi-VL-A3B-Instruct | Kimi-VL (A3B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| ZhipuAI/GLM-4.5V | GLM-4.5V (106B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| LLM-Research/Llama-3.2-11B-Vision-Instruct | Llama 3.2 Vision (11B) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| rednote-hilab/dots.ocr | DotsVLM-OCR | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Embedding Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|-------------------------------------------|--------------------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| intfloat/e5-mistral-7b-instruct | E5 (Llama/Mistral based) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| iic/gte_Qwen2-1.5B-instruct | GTE-Qwen2 | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen3-Embedding-8B | Qwen3-Embedding | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Alibaba-NLP/gme-Qwen2-VL-2B-Instruct | GME (Multimodal) | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/clip-vit-large-patch14-336 | CLIP | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| BAAI/bge-large-en-v1.5 | BGE | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Reward Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|------------------------------------------------|---------------------------|------------------------------------------|:----------------------------------------:|
|
||||
| Skywork/Skywork-Reward-Llama-3.1-8B-v0.2 | Llama3.1 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Shanghai_AI_Laboratory/internlm2-7b-reward | InternLM 2 Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Qwen/Qwen2.5-Math-RM-72B | Qwen2.5 Reward - Math | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| Howeee/Qwen2.5-1.5B-apeach | Qwen2.5 Reward - Sequence | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
| AI-ModelScope/Skywork-Reward-Gemma-2-27B-v0.2 | Gemma 2-27B Reward | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
|
||||
## Rerank Models
|
||||
|
||||
| Models | Model Family | A2 Supported | A3 Supported |
|
||||
|-------------------------|--------------|:----------------------------------------:|:----------------------------------------:|
|
||||
| BAAI/bge-reranker-v2-m3 | BGE-Reranker | **<span style="color: green;">√</span>** | **<span style="color: green;">√</span>** |
|
||||
151
third_party/sglang/docs/platforms/ascend/mindspore_backend.md
vendored
Normal file
151
third_party/sglang/docs/platforms/ascend/mindspore_backend.md
vendored
Normal file
@@ -0,0 +1,151 @@
|
||||
# MindSpore Models
|
||||
|
||||
## Introduction
|
||||
|
||||
MindSpore is a high-performance AI framework optimized for Ascend NPUs. This doc guides users to run MindSpore models in SGLang.
|
||||
|
||||
## Requirements
|
||||
|
||||
MindSpore currently only supports Ascend NPU devices. Users need to first install Ascend CANN software packages.
|
||||
The CANN software packages can be downloaded from the [Ascend Official Website](https://www.hiascend.com). The recommended version is 8.3.RC2.
|
||||
|
||||
## Supported Models
|
||||
|
||||
Currently, the following models are supported:
|
||||
|
||||
- **Qwen3**: Dense and MoE models
|
||||
- **DeepSeek V3/R1**
|
||||
- *More models coming soon...*
|
||||
|
||||
## Installation
|
||||
|
||||
> **Note**: Currently, MindSpore models are provided by an independent package `sgl-mindspore`. Support for MindSpore is built upon current SGLang support for Ascend NPU platform. Please first [install SGLang for Ascend NPU](ascend_npu.md) and then install `sgl-mindspore`:
|
||||
|
||||
```shell
|
||||
git clone https://github.com/mindspore-lab/sgl-mindspore.git
|
||||
cd sgl-mindspore
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
|
||||
## Run Model
|
||||
|
||||
Current SGLang-MindSpore supports Qwen3 and DeepSeek V3/R1 models. This doc uses Qwen3-8B as an example.
|
||||
|
||||
### Offline infer
|
||||
|
||||
Use the following script for offline infer:
|
||||
|
||||
```python
|
||||
import sglang as sgl
|
||||
|
||||
# Initialize the engine with MindSpore backend
|
||||
llm = sgl.Engine(
|
||||
model_path="/path/to/your/model", # Local model path
|
||||
device="npu", # Use NPU device
|
||||
model_impl="mindspore", # MindSpore implementation
|
||||
attention_backend="ascend", # Attention backend
|
||||
tp_size=1, # Tensor parallelism size
|
||||
dp_size=1 # Data parallelism size
|
||||
)
|
||||
|
||||
# Generate text
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The capital of France is",
|
||||
"The future of AI is"
|
||||
]
|
||||
|
||||
sampling_params = {"temperature": 0, "top_p": 0.9}
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for prompt, output in zip(prompts, outputs):
|
||||
print(f"Prompt: {prompt}")
|
||||
print(f"Generated: {output['text']}")
|
||||
print("---")
|
||||
```
|
||||
|
||||
### Start server
|
||||
|
||||
Launch a server with MindSpore backend:
|
||||
|
||||
```bash
|
||||
# Basic server startup
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--tp-size 1 \
|
||||
--dp-size 1
|
||||
```
|
||||
|
||||
For distributed server with multiple nodes:
|
||||
|
||||
```bash
|
||||
# Multi-node distributed server
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--dist-init-addr 127.0.0.1:29500 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--tp-size 4 \
|
||||
--dp-size 2
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
#### Debug Mode
|
||||
|
||||
Enable sglang debug logging by log-level argument.
|
||||
|
||||
```bash
|
||||
python3 -m sglang.launch_server \
|
||||
--model-path /path/to/your/model \
|
||||
--host 0.0.0.0 \
|
||||
--device npu \
|
||||
--model-impl mindspore \
|
||||
--attention-backend ascend \
|
||||
--log-level DEBUG
|
||||
```
|
||||
|
||||
Enable mindspore info and debug logging by setting environments.
|
||||
|
||||
```bash
|
||||
export GLOG_v=1 # INFO
|
||||
export GLOG_v=0 # DEBUG
|
||||
```
|
||||
|
||||
#### Explicitly select devices
|
||||
|
||||
Use the following environment variable to explicitly select the devices to use.
|
||||
|
||||
```shell
|
||||
export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7 # to set device
|
||||
```
|
||||
|
||||
#### Some communication environment issues
|
||||
|
||||
In case of some environment with special communication environment, users need set some environment variables.
|
||||
|
||||
```shell
|
||||
export MS_ENABLE_LCCL=off # current not support LCCL communication mode in SGLang-MindSpore
|
||||
```
|
||||
|
||||
#### Some dependencies of protobuf
|
||||
|
||||
In case of some environment with special protobuf version, users need set some environment variables to avoid binary version mismatch.
|
||||
|
||||
```shell
|
||||
export PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python # to avoid protobuf binary version mismatch
|
||||
```
|
||||
|
||||
## Support
|
||||
For MindSpore-specific issues:
|
||||
|
||||
- Refer to the [MindSpore documentation](https://www.mindspore.cn/)
|
||||
Reference in New Issue
Block a user