Gahow Wang
445e491123
third_party/vllm/ now tracked in git for direct patch management.
Based on vLLM v0.18.1 release with one patch applied:
vllm/v1/core/sched/scheduler.py:
Replace fatal assert with graceful skip when KV transfer callback
arrives for an already-aborted request during PD disaggregated serving.
Future vLLM modifications should be made directly in third_party/vllm/
and committed normally. The patches/ directory is kept as documentation
of what changed from upstream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1007 B
1007 B
Server Arguments
The vllm serve command is used to launch the OpenAI-compatible server.
CLI Arguments
The vllm serve command is used to launch the OpenAI-compatible server.
To see the available options, take a look at the CLI Reference!
Configuration file
You can load CLI arguments via a YAML config file. The argument names must be the long form of those outlined above.
For example:
# config.yaml
model: meta-llama/Llama-3.1-8B-Instruct
host: "127.0.0.1"
port: 6379
uvicorn-log-level: "info"
To use the above config file:
vllm serve --config config.yaml
!!! note
In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
The order of priorities is command line > config file values > defaults.
e.g. vllm serve SOME_MODEL --config config.yaml, SOME_MODEL takes precedence over model in config file.