Gahow Wang
445e491123
third_party/vllm/ now tracked in git for direct patch management.
Based on vLLM v0.18.1 release with one patch applied:
vllm/v1/core/sched/scheduler.py:
Replace fatal assert with graceful skip when KV transfer callback
arrives for an already-aborted request during PD disaggregated serving.
Future vLLM modifications should be made directly in third_party/vllm/
and committed normally. The patches/ directory is kept as documentation
of what changed from upstream.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2.3 KiB
2.3 KiB
MLP Draft Models
The following code configures vLLM to use speculative decoding where proposals are generated by draft models that condition draft predictions on both context vectors and sampled tokens. For more information see The Hitchhiker's Guide to Speculative Decoding and IBM Research's Technical Report.
MLP Drafter Example
from vllm import LLM, SamplingParams
prompts = ["The future of AI is"]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(
model="meta-llama/Meta-Llama-3.1-8B-Instruct",
tensor_parallel_size=1,
speculative_config={
"model": "ibm-ai-platform/llama3-8b-accelerator",
"draft_tensor_parallel_size": 1,
"method": "mlp_speculator",
},
)
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
!!! warning "Known issue"
ibm-ai-platform/llama3-70b-accelerator can fail with:
AttributeError: 'MLPSpeculatorConfig' object has no attribute 'num_attention_heads'.
Track status in #34106
and #34163.
Pre-Trained MLP Drafter Models
A variety of speculative models of this type are available on HF hub: