model: tensor-parallel Qwen3 (sharded weights + AllReduce)

from_weights_tp shards each rank's weights (column-split q/k/v/gate/up,
row-split o/down; replicate norms/embed/lm_head) and the paged forward uses
local head counts + AllReduces after o_proj and down_proj. PagedKVCache::new_tp
sizes the pool for the rank's local KV heads (KV is sharded too). TP=1 is the
identity path. New bench-tp binary runs E2E multi-GPU generation per TP degree.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-29 11:10:24 +08:00
parent 453520d622
commit f17011129e
4 changed files with 336 additions and 28 deletions

View File

@@ -8,6 +8,7 @@ xserv-cuda = { path = "../xserv-cuda" }
xserv-tensor = { path = "../xserv-tensor" }
xserv-kernels = { path = "../xserv-kernels" }
xserv-tokenizer = { path = "../xserv-tokenizer" }
xserv-distributed = { path = "../xserv-distributed" }
half.workspace = true
smallvec.workspace = true
serde.workspace = true