bench: run one server at a time, match thinking mode, fix tools package

Refinements from end-to-end bring-up on the GPU host:

- Run each system start→suites→stop in sequence. Two BF16 8B models don't
  co-reside on one 32GB GPU, and a resident idle engine would distort the
  other's latency/throughput.
- Match generation mode: xserv hardcodes Qwen3 thinking off, so send
  chat_template_kwargs={enable_thinking:false} to llama.cpp via a per-endpoint
  extra_body. --enable-thinking opts back into thinking mode.
- Add tools/__init__.py so `python3 -m tools.bench.runner` resolves our package
  instead of a site-packages `tools` (nvfuser ships one that shadowed it).
- Document offline-GPU-host workflow, thinking-match, and the xserv 8192 OOM
  finding that the bench surfaced.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-28 11:40:07 +08:00
parent 49c7653222
commit 7cb9ee3870
7 changed files with 102 additions and 28 deletions

View File

@@ -90,6 +90,7 @@ async def run_single_stream(
api_key=ep.api_key,
timeout=cfg.request_timeout_s,
concurrency=1,
extra_body=ep.extra_body,
)
rows.append(_summarize(ep.name, f"single/{bucket}", results, wall))
for i, r in enumerate(results):
@@ -122,6 +123,7 @@ async def run_concurrent(
api_key=ep.api_key,
timeout=cfg.request_timeout_s,
concurrency=c,
extra_body=ep.extra_body,
)
rows.append(_summarize(ep.name, f"concurrent-{c}", results, wall))
for i, r in enumerate(results):
@@ -147,7 +149,7 @@ def run_speed(
asyncio.run(chat_concurrent(
ep.base_url, ep.model_id, warm_messages,
max_tokens=8, temperature=0.0, api_key=ep.api_key,
timeout=120, concurrency=1,
timeout=120, concurrency=1, extra_body=ep.extra_body,
))
rows1, raw1 = asyncio.run(run_single_stream(ep, cfg))