167 lines
6.1 KiB
Markdown
167 lines
6.1 KiB
Markdown
# ASR Benchmark
|
|
|
|
This benchmark evaluates the performance and accuracy (Word Error Rate - WER) of Automatic Speech Recognition (ASR) models served via SGLang.
|
|
|
|
## Supported Models
|
|
|
|
- `openai/whisper-large-v3`
|
|
- `openai/whisper-large-v3-turbo`
|
|
|
|
## Setup
|
|
|
|
Install the required dependencies:
|
|
|
|
```bash
|
|
apt install ffmpeg
|
|
pip install librosa soundfile datasets evaluate jiwer transformers openai torchcodec torch
|
|
```
|
|
|
|
## Running the Benchmark
|
|
|
|
### 1. Start SGLang Server
|
|
|
|
Launch the SGLang server with a Whisper model:
|
|
|
|
```bash
|
|
python -m sglang.launch_server --model-path openai/whisper-large-v3 --port 30000
|
|
```
|
|
|
|
### 2. Run the Benchmark Script
|
|
|
|
Basic usage (using chat completions API):
|
|
|
|
```bash
|
|
python bench_sglang.py --base-url http://localhost:30000 --model openai/whisper-large-v3 --n-examples 10
|
|
```
|
|
|
|
Using the OpenAI-compatible transcription API:
|
|
|
|
```bash
|
|
python bench_sglang.py \
|
|
--base-url http://localhost:30000 \
|
|
--model openai/whisper-large-v3 \
|
|
--api-type transcription \
|
|
--language English \
|
|
--n-examples 10
|
|
```
|
|
|
|
Run with streaming and show real-time output:
|
|
|
|
```bash
|
|
python bench_sglang.py \
|
|
--base-url http://localhost:30000 \
|
|
--model openai/whisper-large-v3 \
|
|
--api-type transcription \
|
|
--stream \
|
|
--show-predictions \
|
|
--concurrency 1
|
|
```
|
|
|
|
Run with higher concurrency and save results:
|
|
|
|
```bash
|
|
python bench_sglang.py \
|
|
--base-url http://localhost:30000 \
|
|
--model openai/whisper-large-v3 \
|
|
--concurrency 8 \
|
|
--n-examples 100 \
|
|
--output results.json \
|
|
--show-predictions
|
|
```
|
|
|
|
## Arguments
|
|
|
|
| Argument | Description | Default |
|
|
|----------|-------------|---------|
|
|
| `--base-url` | SGLang server URL | `http://localhost:30000` |
|
|
| `--model` | Model name on the server | `openai/whisper-large-v3` |
|
|
| `--dataset` | HuggingFace dataset for evaluation | `D4nt3/esb-datasets-earnings22-validation-tiny-filtered` |
|
|
| `--split` | Dataset split to use | `validation` |
|
|
| `--concurrency` | Number of concurrent requests | `4` |
|
|
| `--n-examples` | Number of examples to process (`-1` for all) | `-1` |
|
|
| `--output` | Path to save results as JSON | `None` |
|
|
| `--show-predictions` | Display sample predictions | `False` |
|
|
| `--print-n` | Number of samples to display | `5` |
|
|
| `--api-type` | API to use: `chat` (chat completions) or `transcription` (audio transcriptions) | `chat` |
|
|
| `--language` | Language for transcription API (e.g., `English`, `en`) | `None` |
|
|
| `--stream` | Enable streaming mode for transcription API | `False` |
|
|
|
|
## Metrics
|
|
|
|
The benchmark outputs:
|
|
|
|
| Metric | Description |
|
|
|--------|-------------|
|
|
| **Total Requests** | Number of successful ASR requests processed |
|
|
| **WER** | Word Error Rate (lower is better), computed using the `evaluate` library |
|
|
| **Average Latency** | Mean time per request (seconds) |
|
|
| **Median Latency** | 50th percentile latency (seconds) |
|
|
| **95th Latency** | 95th percentile latency (seconds) |
|
|
| **Throughput** | Requests processed per second |
|
|
| **Token Throughput** | Output tokens per second |
|
|
|
|
## Example Output
|
|
|
|
```bash
|
|
python bench_sglang.py --api-type transcription --concurrency 128 --model openai/whisper-large-v3 --show-predictions
|
|
|
|
Loading dataset: D4nt3/esb-datasets-earnings22-validation-tiny-filtered...
|
|
Using API type: transcription
|
|
Repo card metadata block was not found. Setting CardData to empty.
|
|
WARNING:huggingface_hub.repocard:Repo card metadata block was not found. Setting CardData to empty.
|
|
Performing warmup...
|
|
Processing 511 samples...
|
|
------------------------------
|
|
Results for openai/whisper-large-v3:
|
|
Total Requests: 511
|
|
WER: 12.7690
|
|
Average Latency: 1.3602s
|
|
Median Latency: 1.2090s
|
|
95th Latency: 2.9986s
|
|
Throughput: 19.02 req/s
|
|
Token Throughput: 354.19 tok/s
|
|
Total Test Time: 26.8726s
|
|
------------------------------
|
|
|
|
==================== Sample Predictions ====================
|
|
Sample 1:
|
|
REF: on the use of taxonomy i you know i think it is it is early days for us to to make any clear indications to the market about the proportion that would fall under that requirement
|
|
PRED: on the eu taxonomy i think it is early days for us to make any clear indications to the market about the proportion that would fall under that requirement
|
|
----------------------------------------
|
|
Sample 2:
|
|
REF: so within fiscal year 2021 say 120 a 100 depending on what the micro will do and next year it is not necessarily payable in q one is we will look at what the cash flows for 2022 look like
|
|
PRED: so within fiscal year 2021 say $120000 $100000 depending on what the macro will do and next year it is not necessarily payable in q one is we will look at what the cash flows for 2022 look like
|
|
----------------------------------------
|
|
Sample 3:
|
|
REF: we talked about 4.7 gigawatts
|
|
PRED: we talked about 4.7 gigawatts
|
|
----------------------------------------
|
|
Sample 4:
|
|
REF: and you know depending on that working capital build we will we will see what that yields
|
|
PRED: and depending on that working capital build we will see what that yields what
|
|
----------------------------------------
|
|
Sample 5:
|
|
REF: so on on sinopec what we have agreed with sinopec way back then is that free cash flows after paying all capexs are distributed out 30 70%
|
|
PRED: so on sinopec what we have agreed with sinopec way back then is that free cash flows after paying all capexes are distributed out 30% 70%
|
|
----------------------------------------
|
|
============================================================
|
|
```
|
|
|
|
## Notes
|
|
|
|
- Audio samples longer than 30 seconds are automatically filtered out (Whisper limitation)
|
|
- The benchmark performs a warmup request before measuring performance
|
|
- Results are normalized using the model's tokenizer when available
|
|
- When using `--stream` with `--show-predictions`, use `--concurrency 1` for clean sequential output
|
|
- The `--language` option accepts both full names (e.g., `English`) and ISO 639-1 codes (e.g., `en`)
|
|
|
|
## Troubleshooting
|
|
|
|
**Server connection refused**
|
|
- Ensure the SGLang server is running and accessible at the specified `--base-url`
|
|
- Check that the port is not blocked by a firewall
|
|
|
|
**Out of memory errors**
|
|
- Reduce `--concurrency` to lower GPU memory usage
|
|
- Use a smaller Whisper model variant
|