bench: run one server at a time, match thinking mode, fix tools package
Refinements from end-to-end bring-up on the GPU host:
- Run each system start→suites→stop in sequence. Two BF16 8B models don't
co-reside on one 32GB GPU, and a resident idle engine would distort the
other's latency/throughput.
- Match generation mode: xserv hardcodes Qwen3 thinking off, so send
chat_template_kwargs={enable_thinking:false} to llama.cpp via a per-endpoint
extra_body. --enable-thinking opts back into thinking mode.
- Add tools/__init__.py so `python3 -m tools.bench.runner` resolves our package
instead of a site-packages `tools` (nvfuser ships one that shadowed it).
- Document offline-GPU-host workflow, thinking-match, and the xserv 8192 OOM
finding that the bench surfaced.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -90,6 +90,7 @@ async def run_single_stream(
|
||||
api_key=ep.api_key,
|
||||
timeout=cfg.request_timeout_s,
|
||||
concurrency=1,
|
||||
extra_body=ep.extra_body,
|
||||
)
|
||||
rows.append(_summarize(ep.name, f"single/{bucket}", results, wall))
|
||||
for i, r in enumerate(results):
|
||||
@@ -122,6 +123,7 @@ async def run_concurrent(
|
||||
api_key=ep.api_key,
|
||||
timeout=cfg.request_timeout_s,
|
||||
concurrency=c,
|
||||
extra_body=ep.extra_body,
|
||||
)
|
||||
rows.append(_summarize(ep.name, f"concurrent-{c}", results, wall))
|
||||
for i, r in enumerate(results):
|
||||
@@ -147,7 +149,7 @@ def run_speed(
|
||||
asyncio.run(chat_concurrent(
|
||||
ep.base_url, ep.model_id, warm_messages,
|
||||
max_tokens=8, temperature=0.0, api_key=ep.api_key,
|
||||
timeout=120, concurrency=1,
|
||||
timeout=120, concurrency=1, extra_body=ep.extra_body,
|
||||
))
|
||||
|
||||
rows1, raw1 = asyncio.run(run_single_stream(ep, cfg))
|
||||
|
||||
Reference in New Issue
Block a user