bench: run one server at a time, match thinking mode, fix tools package
Refinements from end-to-end bring-up on the GPU host:
- Run each system start→suites→stop in sequence. Two BF16 8B models don't
co-reside on one 32GB GPU, and a resident idle engine would distort the
other's latency/throughput.
- Match generation mode: xserv hardcodes Qwen3 thinking off, so send
chat_template_kwargs={enable_thinking:false} to llama.cpp via a per-endpoint
extra_body. --enable-thinking opts back into thinking mode.
- Add tools/__init__.py so `python3 -m tools.bench.runner` resolves our package
instead of a site-packages `tools` (nvfuser ships one that shadowed it).
- Document offline-GPU-host workflow, thinking-match, and the xserv 8192 OOM
finding that the bench surfaced.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -81,6 +81,7 @@ async def _run_one_task(
|
||||
temperature=cfg.quality_temperature,
|
||||
api_key=ep.api_key,
|
||||
timeout=cfg.request_timeout_s,
|
||||
extra_body=ep.extra_body,
|
||||
)
|
||||
pred = task_mod.extract_answer(r.text) if r.error is None else None
|
||||
correct = task_mod.score(pred, prob["answer"]) if r.error is None else False
|
||||
|
||||
Reference in New Issue
Block a user