server: serve gpt-oss on a single GPU via the TP engine (world=1)
gpt-oss has no single-GPU engine path, so --tp 1 fell through to the
Qwen3-only engine and every request 503'd. Route gpt_oss to run_tp
even at tp=1: NCCL world-1 init works and all_reduce already no-ops
(bench-gpt-oss --tp 1 exercised this path). Quantized gpt-oss (22 GB
FP8 / 13 GB MXFP4) now serves on one 32 GB 5090.
Also fix eval_gsm8k_fast.py --gpu to accept a device list ("2,3"):
it was type=int, so any --tp 2 run pinned CUDA_VISIBLE_DEVICES to one
GPU and rank 1's set_device panicked while rank 0 spun in NCCL init.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -120,7 +120,9 @@ fn worker_loop(
|
||||
/// Run the TP coordinator (rank 0) on the calling thread. Spawns worker ranks
|
||||
/// internally and consumes generation requests from `rx`.
|
||||
pub fn run_tp(model_dir: &Path, world: usize, max_seq_len: usize, rx: mpsc::Receiver<GenerateRequest>) {
|
||||
assert!(world >= 2, "run_tp requires world >= 2");
|
||||
// world=1 is a valid single-rank configuration (gpt-oss has no
|
||||
// single-GPU engine path; NCCL init and all_reduce no-op at world=1).
|
||||
assert!(world >= 1, "run_tp requires world >= 1");
|
||||
let config = ModelConfig::from_file(&model_dir.join("config.json"));
|
||||
assert!(
|
||||
config.num_kv_heads() % world == 0,
|
||||
|
||||
Reference in New Issue
Block a user