Gahow Wang
445e491123
third_party/vllm/ now tracked in git for direct patch management.
Based on vLLM v0.18.1 release with one patch applied:
vllm/v1/core/sched/scheduler.py:
Replace fatal assert with graceful skip when KV transfer callback
arrives for an already-aborted request during PD disaggregated serving.
Future vLLM modifications should be made directly in third_party/vllm/
and committed normally. The patches/ directory is kept as documentation
of what changed from upstream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
780 B
780 B
LangChain
vLLM is also available via LangChain .
To install LangChain, run
pip install langchain langchain_community -q
To run inference on a single or multiple GPUs, use VLLM class from langchain.
??? code
```python
from langchain_community.llms import VLLM
llm = VLLM(
model="Qwen/Qwen3-4B",
trust_remote_code=True, # mandatory for hf models
max_new_tokens=128,
top_k=10,
top_p=0.95,
temperature=0.8,
# for distributed inference
# tensor_parallel_size=...,
)
print(llm("What is the capital of France ?"))
```
Please refer to this Tutorial for more details.