Gahow Wang
445e491123
third_party/vllm/ now tracked in git for direct patch management.
Based on vLLM v0.18.1 release with one patch applied:
vllm/v1/core/sched/scheduler.py:
Replace fatal assert with graceful skip when KV transfer callback
arrives for an already-aborted request during PD disaggregated serving.
Future vLLM modifications should be made directly in third_party/vllm/
and committed normally. The patches/ directory is kept as documentation
of what changed from upstream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
36 lines
1007 B
Markdown
36 lines
1007 B
Markdown
# Server Arguments
|
|
|
|
The `vllm serve` command is used to launch the OpenAI-compatible server.
|
|
|
|
## CLI Arguments
|
|
|
|
The `vllm serve` command is used to launch the OpenAI-compatible server.
|
|
To see the available options, take a look at the [CLI Reference](../cli/README.md)!
|
|
|
|
## Configuration file
|
|
|
|
You can load CLI arguments via a [YAML](https://yaml.org/) config file.
|
|
The argument names must be the long form of those outlined [above](serve_args.md).
|
|
|
|
For example:
|
|
|
|
```yaml
|
|
# config.yaml
|
|
|
|
model: meta-llama/Llama-3.1-8B-Instruct
|
|
host: "127.0.0.1"
|
|
port: 6379
|
|
uvicorn-log-level: "info"
|
|
```
|
|
|
|
To use the above config file:
|
|
|
|
```bash
|
|
vllm serve --config config.yaml
|
|
```
|
|
|
|
!!! note
|
|
In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
|
|
The order of priorities is `command line > config file values > defaults`.
|
|
e.g. `vllm serve SOME_MODEL --config config.yaml`, SOME_MODEL takes precedence over `model` in config file.
|