xserv

gahow/xserv

Fork 0

Commit Graph

Author	SHA1	Message	Date
Gahow Wang	63f5599717	server: serve gpt-oss on a single GPU via the TP engine (world=1) gpt-oss has no single-GPU engine path, so --tp 1 fell through to the Qwen3-only engine and every request 503'd. Route gpt_oss to run_tp even at tp=1: NCCL world-1 init works and all_reduce already no-ops (bench-gpt-oss --tp 1 exercised this path). Quantized gpt-oss (22 GB FP8 / 13 GB MXFP4) now serves on one 32 GB 5090. Also fix eval_gsm8k_fast.py --gpu to accept a device list ("2,3"): it was type=int, so any --tp 2 run pinned CUDA_VISIBLE_DEVICES to one GPU and rank 1's set_device panicked while rank 0 spun in NCCL init. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>	2026-06-12 16:29:10 +08:00
Gahow Wang	3a530956af	tools: add FP8 vs BF16 benchmark and GSM8K eval harness bench_fp8.py — head-to-head comparison of FP8 and BF16 models on GSM8K / AIME2025 accuracy plus TTFT/TPOT performance measurement. eval_gsm8k_batch.sh — lightweight GSM8K accuracy evaluator that pipes one problem per xserv-chat invocation and scores with \boxed{} / last-number extraction. Benchmark results (gpt-oss-20b, 50-problem GSM8K): FP8 W8A8 TP1 : 94.0% (single RTX 5090, 25 GB) FP8 W8A16 TP1: 94.0% BF16 TP2 : 94.0% (requires 2× RTX 5090) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>	2026-06-08 15:43:04 +08:00

Author

SHA1

Message

Date

Gahow Wang

63f5599717

server: serve gpt-oss on a single GPU via the TP engine (world=1)

gpt-oss has no single-GPU engine path, so --tp 1 fell through to the
Qwen3-only engine and every request 503'd. Route gpt_oss to run_tp
even at tp=1: NCCL world-1 init works and all_reduce already no-ops
(bench-gpt-oss --tp 1 exercised this path). Quantized gpt-oss (22 GB
FP8 / 13 GB MXFP4) now serves on one 32 GB 5090.

Also fix eval_gsm8k_fast.py --gpu to accept a device list ("2,3"):
it was type=int, so any --tp 2 run pinned CUDA_VISIBLE_DEVICES to one
GPU and rank 1's set_device panicked while rank 0 spun in NCCL init.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-06-12 16:29:10 +08:00

Gahow Wang

3a530956af

tools: add FP8 vs BF16 benchmark and GSM8K eval harness

bench_fp8.py — head-to-head comparison of FP8 and BF16 models on
  GSM8K / AIME2025 accuracy plus TTFT/TPOT performance measurement.

eval_gsm8k_batch.sh — lightweight GSM8K accuracy evaluator that
  pipes one problem per xserv-chat invocation and scores with
  \boxed{} / last-number extraction.

Benchmark results (gpt-oss-20b, 50-problem GSM8K):
  FP8 W8A8 TP1 : 94.0%  (single RTX 5090, 25 GB)
  FP8 W8A16 TP1: 94.0%
  BF16 TP2     : 94.0%  (requires 2× RTX 5090)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-06-08 15:43:04 +08:00

2 Commits