bench: fix llama.cpp per-slot context (was 1/parallel of intended)
llama.cpp divides total -c across --parallel slots, so -c 4096 --parallel 4 gave each request only 1024 tokens — truncating long AIME generations before the boxed answer and making xserv look artificially better (20% vs 3.3%). Set total -c = max_seq_len * n_parallel so per-slot context equals xserv's per-sequence max_seq_len. Also drop --log-disable; its startup log reports the per-slot n_ctx that catches exactly this misconfiguration. After the fix, AIME is at parity (xserv 23.3% vs llama.cpp 20.0%), matching the GSM8K parity and confirming the gap was a config artifact, not engine quality. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -134,7 +134,7 @@ def build_endpoints(args) -> list[SystemEndpoint]:
|
||||
model_id=args.llama_model_id,
|
||||
launch_cmd=llama_cpp_launch_cmd(
|
||||
args.llama_bin, gguf, args.llama_port,
|
||||
n_parallel=args.max_batch, ctx_size=args.max_seq_len,
|
||||
n_parallel=args.max_batch, ctx_per_slot=args.max_seq_len,
|
||||
),
|
||||
# llama-server's health endpoint also returns 200 only when model is loaded.
|
||||
health_path="/health",
|
||||
|
||||
Reference in New Issue
Block a user