Gahow Wang
445e491123
third_party/vllm/ now tracked in git for direct patch management.
Based on vLLM v0.18.1 release with one patch applied:
vllm/v1/core/sched/scheduler.py:
Replace fatal assert with graceful skip when KV transfer callback
arrives for an already-aborted request during PD disaggregated serving.
Future vLLM modifications should be made directly in third_party/vllm/
and committed normally. The patches/ directory is kept as documentation
of what changed from upstream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
949 B
949 B
Llama Stack
vLLM is also available via Llama Stack.
To install Llama Stack, run
pip install llama-stack -q
Inference using OpenAI-Compatible API
Then start the Llama Stack server and configure it to point to your vLLM server with the following settings:
inference:
- provider_id: vllm0
provider_type: remote::vllm
config:
url: http://127.0.0.1:8000
Please refer to this guide for more details on this remote vLLM provider.
Inference using Embedded vLLM
An inline provider is also available. This is a sample of configuration using that method:
inference:
- provider_type: vllm
config:
model: Llama3.1-8B-Instruct
tensor_parallel_size: 4