bench: run one server at a time, match thinking mode, fix tools package
Refinements from end-to-end bring-up on the GPU host:
- Run each system start→suites→stop in sequence. Two BF16 8B models don't
co-reside on one 32GB GPU, and a resident idle engine would distort the
other's latency/throughput.
- Match generation mode: xserv hardcodes Qwen3 thinking off, so send
chat_template_kwargs={enable_thinking:false} to llama.cpp via a per-endpoint
extra_body. --enable-thinking opts back into thinking mode.
- Add tools/__init__.py so `python3 -m tools.bench.runner` resolves our package
instead of a site-packages `tools` (nvfuser ships one that shadowed it).
- Document offline-GPU-host workflow, thinking-match, and the xserv 8192 OOM
finding that the bench surfaced.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -24,6 +24,12 @@ class SystemEndpoint:
|
||||
base_url: str # http://host:port (OpenAI-compatible root, no /v1)
|
||||
model_id: str # what to put in the request body's "model" field
|
||||
api_key: str | None = None # llama-server doesn't need one; xserv ignores it
|
||||
# Extra fields merged into every request body for this system. Used to keep
|
||||
# the two engines in the SAME generation mode — xserv hardcodes Qwen3
|
||||
# thinking OFF (empty <think></think> in its prompt builder), so we disable
|
||||
# thinking on llama-server via chat_template_kwargs to match. Both engines
|
||||
# ignore unknown fields, so this is safe.
|
||||
extra_body: dict | None = None
|
||||
# Process supervision is optional — if base_url is already serving, we skip launch.
|
||||
launch_cmd: list[str] | None = None
|
||||
launch_env: dict[str, str] = field(default_factory=dict)
|
||||
|
||||
Reference in New Issue
Block a user