gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

chore: vendor sglang v0.5.10 snapshot

Browse Source
This commit is contained in:
Gahow Wang
2026-04-24 12:29:36 +00:00
parent 78f0d15221
commit bded08301f
4308 changed files with 1200894 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

162
third_party/sglang/docs/supported_models/retrieval_ranking/classify_models.md vendored Normal file
View File

@@ -0,0 +1,162 @@
# Classification API
This document describes the `/v1/classify` API endpoint implementation in SGLang, which is compatible with vLLM's classification API format.
## Overview
The classification API allows you to classify text inputs using classification models. This implementation follows the same format as vLLM's 0.7.0 classification API.
## API Endpoint
```
POST /v1/classify
```
## Request Format
```json
{
"model": "model_name",
"input": "text to classify"
}
```
### Parameters
- `model` (string, required): The name of the classification model to use
- `input` (string, required): The text to classify
- `user` (string, optional): User identifier for tracking
- `rid` (string, optional): Request ID for tracking
- `priority` (integer, optional): Request priority
## Response Format
```json
{
"id": "classify-9bf17f2847b046c7b2d5495f4b4f9682",
"object": "list",
"created": 1745383213,
"model": "jason9693/Qwen2.5-1.5B-apeach",
"data": [
{
"index": 0,
"label": "Default",
"probs": [0.565970778465271, 0.4340292513370514],
"num_classes": 2
}
],
"usage": {
"prompt_tokens": 10,
"total_tokens": 10,
"completion_tokens": 0,
"prompt_tokens_details": null
}
}
```
### Response Fields
- `id`: Unique identifier for the classification request
- `object`: Always "list"
- `created`: Unix timestamp when the request was created
- `model`: The model used for classification
- `data`: Array of classification results
- `index`: Index of the result
- `label`: Predicted class label
- `probs`: Array of probabilities for each class
- `num_classes`: Total number of classes
- `usage`: Token usage information
- `prompt_tokens`: Number of input tokens
- `total_tokens`: Total number of tokens
- `completion_tokens`: Number of completion tokens (always 0 for classification)
- `prompt_tokens_details`: Additional token details (optional)
## Example Usage
### Using curl
```bash
curl -v "http://127.0.0.1:8000/v1/classify" \
-H "Content-Type: application/json" \
-d '{
"model": "jason9693/Qwen2.5-1.5B-apeach",
"input": "Loved the new café—coffee was great."
}'
```
### Using Python
```python
import requests
import json
# Make classification request
response = requests.post(
"http://127.0.0.1:8000/v1/classify",
headers={"Content-Type": "application/json"},
json={
"model": "jason9693/Qwen2.5-1.5B-apeach",
"input": "Loved the new café—coffee was great."
}
)
# Parse response
result = response.json()
print(json.dumps(result, indent=2))
```
## Supported Models
The classification API works with any classification model supported by SGLang, including:
### Classification Models (Multi-class)
- `LlamaForSequenceClassification` - Multi-class classification
- `Qwen2ForSequenceClassification` - Multi-class classification
- `Qwen3ForSequenceClassification` - Multi-class classification
- `BertForSequenceClassification` - Multi-class classification
- `Gemma2ForSequenceClassification` - Multi-class classification
**Label Mapping**: The API automatically uses the `id2label` mapping from the model's `config.json` file to provide meaningful label names instead of generic class names. If `id2label` is not available, it falls back to `LABEL_0`, `LABEL_1`, etc., or `Class_0`, `Class_1` as a last resort.
### Reward Models (Single score)
- `InternLM2ForRewardModel` - Single reward score
- `Qwen2ForRewardModel` - Single reward score
- `LlamaForSequenceClassificationWithNormal_Weights` - Special reward model
**Note**: The `/classify` endpoint in SGLang was originally designed for reward models but now supports all non-generative models. Our `/v1/classify` endpoint provides a standardized vLLM-compatible interface for classification tasks.
## Error Handling
The API returns appropriate HTTP status codes and error messages:
- `400 Bad Request`: Invalid request format or missing required fields
- `500 Internal Server Error`: Server-side processing error
Error response format:
```json
{
"error": "Error message",
"type": "error_type",
"code": 400
}
```
## Implementation Details
The classification API is implemented using:
1. **Rust Model Gateway**: Handles routing and request/response models in `sgl-model-gateway/src/protocols/spec.rs`
2. **Python HTTP Server**: Implements the actual endpoint in `python/sglang/srt/entrypoints/http_server.py`
3. **Classification Service**: Handles the classification logic in `python/sglang/srt/entrypoints/openai/serving_classify.py`
## Testing
Use the provided test script to verify the implementation:
```bash
python test_classify_api.py
```
## Compatibility
This implementation is compatible with vLLM's classification API format, allowing seamless migration from vLLM to SGLang for classification tasks.

126
third_party/sglang/docs/supported_models/retrieval_ranking/embedding_models.md vendored Normal file
View File

@@ -0,0 +1,126 @@
# Embedding Models
SGLang provides robust support for embedding models by integrating efficient serving mechanisms with its flexible programming interface. This integration allows for streamlined handling of embedding tasks, facilitating faster and more accurate retrieval and semantic search operations. SGLang's architecture enables better resource utilization and reduced latency in embedding model deployment.
```{important}
Embedding models are executed with `--is-embedding` flag and some may require `--trust-remote-code`
```
## Quick Start
### Launch Server
```shell
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-Embedding-4B \
--is-embedding \
--host 0.0.0.0 \
--port 30000
```
### Client Request
```python
import requests
url = "http://127.0.0.1:30000"
payload = {
"model": "Qwen/Qwen3-Embedding-4B",
"input": "What is the capital of France?",
"encoding_format": "float"
}
response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embedding:", response["data"][0]["embedding"])
```
## Multimodal Embedding Example
For multimodal models like GME that support both text and images:
```shell
python3 -m sglang.launch_server \
--model-path Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
--is-embedding \
--chat-template gme-qwen2-vl \
--host 0.0.0.0 \
--port 30000
```
```python
import requests
url = "http://127.0.0.1:30000"
text_input = "Represent this image in embedding space."
image_path = "https://huggingface.co/datasets/liuhaotian/llava-bench-in-the-wild/resolve/main/images/023.jpg"
payload = {
"model": "gme-qwen2-vl",
"input": [
{
"text": text_input
},
{
"image": image_path
}
],
}
response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embeddings:", [x.get("embedding") for x in response.get("data", [])])
```
## Matryoshka Embedding Example
[Matryoshka Embeddings](https://sbert.net/examples/sentence_transformer/training/matryoshka/README.html#matryoshka-embeddings) or [Matryoshka Representation Learning (MRL)](https://arxiv.org/abs/2205.13147) is a technique used in training embedding models. It allows user to trade off between performance and cost.
### 1. Launch a Matryoshka‑capable model
If the model config already includes `matryoshka_dimensions` or `is_matryoshka` then no override is needed. Otherwise, you can use `--json-model-override-args` as below:
```shell
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-Embedding-0.6B \
--is-embedding \
--host 0.0.0.0 \
--port 30000 \
--json-model-override-args '{"matryoshka_dimensions": [128, 256, 512, 1024, 1536]}'
```
1. Setting `"is_matryoshka": true` allows truncating to any dimension. Otherwise, the server will validate that the specified dimension in the request is one of `matryoshka_dimensions`.
2. Omitting `dimensions` in a request returns the full vector.
### 2. Make requests with different output dimensions
```python
import requests
url = "http://127.0.0.1:30000"
# Request a truncated (Matryoshka) embedding by specifying a supported dimension.
payload = {
"model": "Qwen/Qwen3-Embedding-0.6B",
"input": "Explain diffusion models simply.",
"dimensions": 512 # change to 128 / 1024 / omit for full size
}
response = requests.post(url + "/v1/embeddings", json=payload).json()
print("Embedding:", response["data"][0]["embedding"])
```
## Supported Models
| Model Family | Example Model | Chat Template | Description |
| ------------------------------------------ | -------------------------------------- | ------------- | --------------------------------------------------------------------------- |
| **E5 (Llama/Mistral based)** | `intfloat/e5-mistral-7b-instruct` | N/A | High-quality text embeddings based on Mistral/Llama architectures |
| **GTE-Qwen2** | `Alibaba-NLP/gte-Qwen2-7B-instruct` | N/A | Alibaba's text embedding model with multilingual support |
| **Qwen3-Embedding** | `Qwen/Qwen3-Embedding-4B` | N/A | Latest Qwen3-based text embedding model for semantic representation |
| **BGE** | `BAAI/bge-large-en-v1.5` | N/A | BAAI's text embeddings (requires `attention-backend` triton/torch_native) |
| **GME (Multimodal)** | `Alibaba-NLP/gme-Qwen2-VL-2B-Instruct`| `gme-qwen2-vl`| Multimodal embedding for text and image cross-modal tasks |
| **CLIP** | `openai/clip-vit-large-patch14-336` | N/A | OpenAI's CLIP for image and text embeddings |

11
third_party/sglang/docs/supported_models/retrieval_ranking/index.rst vendored Normal file
View File

@@ -0,0 +1,11 @@
Retrieval & Ranking
===================
Models for embeddings, reranking, and classification.
.. toctree::
:maxdepth: 1
embedding_models.md
rerank_models.md
classify_models.md

314
third_party/sglang/docs/supported_models/retrieval_ranking/rerank_models.md vendored Normal file
View File

@@ -0,0 +1,314 @@
# Rerank Models
SGLang offers comprehensive support for rerank models by incorporating optimized serving frameworks with a flexible programming interface. This setup enables efficient processing of cross-encoder reranking tasks, improving the accuracy and relevance of search result ordering. SGLang’s design ensures high throughput and low latency during reranker model deployment, making it ideal for semantic-based result refinement in large-scale retrieval systems.
```{important}
Rerank models in SGLang fall into two categories:
- **Cross-encoder rerank models**: run with `--is-embedding` (embedding runner).
- **Decoder-only rerank models**: run **without** `--is-embedding` and use next-token logprob scoring (yes/no).
- Text-only (e.g. Qwen3-Reranker)
- Multimodal (e.g. Qwen3-VL-Reranker): also supports image/video content
Some models may require `--trust-remote-code`.
```
## Supported rerank models
| Model Family (Rerank) | Example HuggingFace Identifier | Chat Template | Description |
|------------------------------------------------|--------------------------------------|---------------|----------------------------------------------------------------------------------------------------------------------------------|
| **BGE-Reranker (BgeRerankModel)** | `BAAI/bge-reranker-v2-m3` | N/A | Currently only support `attention-backend` `triton` and `torch_native`. High-performance cross-encoder reranker model from BAAI. Suitable for reranking search results based on semantic relevance. |
| **Qwen3-Reranker (decoder-only yes/no)** | `Qwen/Qwen3-Reranker-8B` | `examples/chat_template/qwen3_reranker.jinja` | Decoder-only reranker using next-token logprob scoring for labels (yes/no). Launch **without** `--is-embedding`. |
| **Qwen3-VL-Reranker (multimodal yes/no)** | `Qwen/Qwen3-VL-Reranker-2B` | `examples/chat_template/qwen3_vl_reranker.jinja` | Multimodal decoder-only reranker supporting text, images, and videos. Uses yes/no logprob scoring. Launch **without** `--is-embedding`. |
## Cross-Encoder Rerank (embedding runner)
### Launch Command
```shell
python3 -m sglang.launch_server \
--model-path BAAI/bge-reranker-v2-m3 \
--host 0.0.0.0 \
--disable-radix-cache \
--chunked-prefill-size -1 \
--attention-backend triton \
--is-embedding \
--port 30000
```
### Example Client Request
```python
import requests
url = "http://127.0.0.1:30000/v1/rerank"
payload = {
"model": "BAAI/bge-reranker-v2-m3",
"query": "what is panda?",
"documents": [
"hi",
"The giant panda (Ailuropoda melanoleuca), sometimes called a panda bear or simply panda, is a bear species endemic to China."
],
"top_n": 1,
"return_documents": True
}
response = requests.post(url, json=payload)
response_json = response.json()
for item in response_json:
if item.get("document"):
print(f"Score: {item['score']:.2f} - Document: '{item['document']}'")
else:
print(f"Score: {item['score']:.2f} - Index: {item['index']}")
```
**Request Parameters:**
- `query` (required): The query text to rank documents against
- `documents` (required): List of documents to be ranked
- `model` (required): Model to use for reranking
- `top_n` (optional): Maximum number of documents to return. Defaults to returning all documents. If specified value is greater than the total number of documents, all documents will be returned.
- `return_documents` (optional): Whether to return documents in the response. Defaults to `True`.
## Qwen3-Reranker (decoder-only yes/no rerank)
### Launch Command
```shell
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-Reranker-0.6B \
--trust-remote-code \
--disable-radix-cache \
--host 0.0.0.0 \
--port 8001 \
--chat-template examples/chat_template/qwen3_reranker.jinja
```
```{note}
Qwen3-Reranker uses decoder-only logprob scoring (yes/no). Do NOT launch it with `--is-embedding`.
```
### Example Client Request (supports optional instruct, top_n, and return_documents)
```shell
curl -X POST http://127.0.0.1:8001/v1/rerank \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen3-Reranker-0.6B",
"query": "法国首都是哪里？",
"documents": [
"法国的首都是巴黎。",
"德国的首都是柏林。",
"香蕉是黄色的水果。"
],
"instruct": "Given a web search query, retrieve relevant passages that answer the query.",
"top_n": 2,
"return_documents": true
}'
```
**Request Parameters:**
- `query` (required): The query text to rank documents against
- `documents` (required): List of documents to be ranked
- `model` (required): Model to use for reranking
- `instruct` (optional): Instruction text for the reranker
- `top_n` (optional): Maximum number of documents to return. Defaults to returning all documents. If specified value is greater than the total number of documents, all documents will be returned.
- `return_documents` (optional): Whether to return documents in the response. Defaults to `True`.
### Response Format
`/v1/rerank` returns a list of objects (sorted by descending score):
- `score`: float, higher means more relevant
- `document`: the original document string (only included when `return_documents` is `true`)
- `index`: the original index in the input `documents`
- `meta_info`: optional debug/usage info (may be present for some models)
The number of returned results is controlled by the `top_n` parameter. If `top_n` is not specified or is greater than the total number of documents, all documents are returned.
Example (with `return_documents: true`):
```json
[
{"score": 0.99, "document": "法国的首都是巴黎。", "index": 0},
{"score": 0.01, "document": "德国的首都是柏林。", "index": 1},
{"score": 0.00, "document": "香蕉是黄色的水果。", "index": 2}
]
```
Example (with `return_documents: false`):
```json
[
{"score": 0.99, "index": 0},
{"score": 0.01, "index": 1},
{"score": 0.00, "index": 2}
]
```
Example (with `top_n: 2`):
```json
[
{"score": 0.99, "document": "法国的首都是巴黎。", "index": 0},
{"score": 0.01, "document": "德国的首都是柏林。", "index": 1}
]
```
### Common Pitfalls
- **`--chat-template` is required.** Without `--chat-template examples/chat_template/qwen3_reranker.jinja`, the server does not recognize the model as a decoder-only reranker and returns a 400 error: `"This model does not appear to be an embedding model by default. Please add `--is-embedding`..."`. The fix is to add the chat template flag, NOT `--is-embedding`.
- If you launch Qwen3-Reranker with `--is-embedding`, `/v1/rerank` cannot compute yes/no logprob scores. Relaunch **without** `--is-embedding`.
- If you see a validation error like "score should be a valid number" and the backend returned a list, upgrade to a version that coerces `embedding[0]` into `score` for rerank responses.
## Qwen3-VL-Reranker (multimodal decoder-only rerank)
Qwen3-VL-Reranker extends the Qwen3-Reranker to support multimodal content, allowing reranking of documents containing text, images, and videos.
### Launch Command
```shell
python3 -m sglang.launch_server \
--model-path Qwen/Qwen3-VL-Reranker-2B \
--trust-remote-code \
--disable-radix-cache \
--host 0.0.0.0 \
--port 30000 \
--chat-template examples/chat_template/qwen3_vl_reranker.jinja
```
```{note}
Qwen3-VL-Reranker uses decoder-only logprob scoring (yes/no) like Qwen3-Reranker. Do NOT launch it with `--is-embedding`.
```
### Text-Only Reranking (backward compatible)
```python
import requests
url = "http://127.0.0.1:30000/v1/rerank"
payload = {
"model": "Qwen3-VL-Reranker-2B",
"query": "What is machine learning?",
"documents": [
"Machine learning is a branch of artificial intelligence that enables computers to learn from data.",
"The weather in Paris is usually mild with occasional rain.",
"Deep learning is a subset of machine learning using neural networks with many layers.",
],
"instruct": "Retrieve passages that answer the question.",
"return_documents": True
}
response = requests.post(url, json=payload)
results = response.json()
for item in results:
print(f"Score: {item['score']:.4f} - {item['document'][:60]}...")
```
### Image Reranking (text query, image/mixed documents)
```python
import requests
url = "http://127.0.0.1:30000/v1/rerank"
payload = {
"query": "A woman playing with her dog on a beach at sunset.",
"documents": [
# Document 1: Text description
"A woman shares a joyful moment with her golden retriever on a sun-drenched beach at sunset.",
# Document 2: Image URL
[
{
"type": "image_url",
"image_url": {
"url": "https://example.com/beach_dog.jpeg"
}
}
],
# Document 3: Text + Image (mixed)
[
{"type": "text", "text": "A joyful scene at the beach:"},
{
"type": "image_url",
"image_url": {
"url": "https://example.com/beach_dog.jpeg"
}
}
]
],
"instruct": "Retrieve images or text relevant to the user's query.",
"return_documents": False
}
response = requests.post(url, json=payload)
results = response.json()
for item in results:
print(f"Index: {item['index']}, Score: {item['score']:.4f}")
```
### Multimodal Query Reranking (query with image)
```python
import requests
url = "http://127.0.0.1:30000/v1/rerank"
payload = {
# Query with text and image
"query": [
{"type": "text", "text": "Find similar images to this:"},
{
"type": "image_url",
"image_url": {
"url": "https://example.com/reference_image.jpeg"
}
}
],
"documents": [
"A cat sleeping on a couch.",
"A woman and her dog enjoying the sunset at the beach.",
"A busy city street with cars and pedestrians.",
[
{
"type": "image_url",
"image_url": {
"url": "https://example.com/similar_image.jpeg"
}
}
]
],
"instruct": "Find images or descriptions similar to the query image."
}
response = requests.post(url, json=payload)
results = response.json()
for item in results:
print(f"Index: {item['index']}, Score: {item['score']:.4f}")
```
### Request Parameters (Multimodal)
- `query` (required): Can be a string (text-only) or a list of content parts:
- `{"type": "text", "text": "..."}` for text
- `{"type": "image_url", "image_url": {"url": "..."}}` for images
- `{"type": "video_url", "video_url": {"url": "..."}}` for videos
- `documents` (required): List where each document can be a string or list of content parts (same format as query)
- `instruct` (optional): Instruction text for the reranker
- `top_n` (optional): Maximum number of documents to return
- `return_documents` (optional): Whether to return documents in the response (default: `false`)
### Common Pitfalls
- Always use `--chat-template examples/chat_template/qwen3_vl_reranker.jinja` for Qwen3-VL-Reranker.
- Do NOT launch with `--is-embedding`.
- For best results, use `--disable-radix-cache` to avoid caching issues with multimodal content.
- **Note**: Currently only `Qwen3-VL-Reranker-2B` is tested and supported. The 8B model may have different behavior and is not guaranteed to work with this template.