xserv/docs/benchmarks/tensor-parallelism.md

Benchmark: Tensor Parallelism (TP=1/2/4) — xserv vs llama.cpp

Setup. Qwen3-8B BF16 on 8× RTX 5090 (PCIe Gen5, no NVLink; GPUs grouped 0-3 / 4-7 by PHB). Both engines driven over the same OpenAI HTTP harness, same scorers, thinking-off, greedy (temp 0), max_tokens 2048. Datasets: AIME 2025 (30) + GSM8K (30). The two engines run concurrently on disjoint groups — xserv on GPU 0..N-1, llama.cpp (--split-mode row) on GPU 4..4+N-1 (tools/bench/run_tp_parallel.sh).

Correctness — on par across engines and TP

TP	task	xserv	llama.cpp
1	AIME 2025	16.7% (5/30)	13.3% (4/30)
1	GSM8K	96.7% (29/30)	96.7% (29/30)
2	AIME 2025	13.3% (4/30)	13.3% (4/30)
2	GSM8K	93.3% (28/30)	96.7% (29/30)
4	AIME 2025	16.7% (5/30)	13.3% (4/30)
4	GSM8K	96.7% (29/30)	96.7% (29/30)

Within ±1 problem everywhere — TP changes nothing about quality on either engine, and the two engines agree. (AIME is low for both: Qwen3-8B thinking-off, capped at 2048 tokens.)

Performance — TPOT (ms/token, lower is better)

TP	xserv AIME / GSM8K	llama.cpp AIME / GSM8K
1	21.0 / 17.8	10.4 / 10.3
2	17.2 / 13.9	19.0 / 18.9
4	15.2 / 12.1	20.2 / 20.2

Opposite TP scaling, with a crossover:

• xserv TP scales positively: TPOT 21.0 → 17.2 → 15.2 ms (AIME), 17.8 → 13.9 → 12.1 ms (GSM8K) — TP=4 is ~1.4–1.5× faster than TP=1. GPU 0-3 all ~82% utilized. (Sublinear because of the 72 PCIe AllReduces/token.)
• llama.cpp row-split regresses: TPOT 10.4 → 19.0 → 20.2 ms — TP=1 is its best; TP=2/4 nearly double the latency. GPU 4-7 only ~24% utilized (communication-bound). Row-split's per-layer cross-GPU traffic over PCIe without NVLink dominates.
• Crossover: llama.cpp is ~2× faster at TP=1, still ahead at TP=2, and xserv is ~1.3× faster at TP=4 (15.2 vs 20.2 ms AIME). AIME-30 wall clock: xserv 1046 → 846 → 730 s (falling), llama.cpp 520 → 952 → 1012 s (rising).

On a NVLink-less PCIe box, xserv's TP is a genuine win and llama.cpp's tensor-split is counterproductive — exactly what the topology predicts.

Clean same-path xserv scaling (bench-tp)

The HTTP numbers above mix engines (xserv TP=1 uses the production continuous- batching engine; TP≥2 uses the serial TP coordinator). The single-stream, same-code-path scaling from bench-tp (greedy, 8 prompts × 64 tokens):

TP	xserv decode tok/s	speedup	TTFT
1	58.5	1.00×	18.0 ms
2	75.7	1.29×	13.4 ms
4	86.1	1.47×	11.5 ms

Caveats

• xserv TP=1 uses the production Engine, TP≥2 the serial tp_engine coordinator — different per-token paths, so the HTTP TP=1→2 step has an engine confound. The clean same-path scaling (bench-tp, above) confirms the trend.
• xserv TTFT is weaker on long AIME prompts (~460–500 ms vs llama ~100–190 ms) — prefill is a known optimization target.
• llama.cpp uses --split-mode row (its tensor-parallel mode); the default layer split only memory-splits, without parallel compute.
• The TP HTTP server processes requests serially (sufficient for this serial quality benchmark); continuous-batching TP is future work.

Raw artifacts: bench-out/tp{1,2,4}-{xserv,llama}/comparison-*.{md,json}.