Community Qwen235B / Frontier smoke report

Bottom line

The community-vLLM TP4 feasibility smoke passed, but the unmodified Frontier profiles are not valid inputs for a formal simulator-versus-runtime ranking test yet. The smoke exposed an execution-semantic mismatch inside the MoE profiler: serving selects vLLM's tuned H20 block-FP8 Triton config, while the standalone Frontier profiler omits the FP8 dtype key and silently falls back to a default config. At the tested TP4/EP1/16-token point, fixing this lookup reduces measured grouped-GEMM time by about 19%, with the direction consistent across five paired routing seeds.

This is precisely why merely installing the same vLLM release on both sides is not a sufficient alignment contract.

Change

Built an isolated Python 3.12 environment around community vLLM 0.10.2 and Frontier commit d9cfeb6d8791fbf2f295dd9744c56a666171776e.
Pinned Transformers 4.55.2 and tokenizers 0.21.4 after reproducing an incompatibility with the unconstrained Transformers 5.13.1 resolver result.
Added Frontier metadata for the local block-FP8 checkpoint without changing the original Hugging Face config.
Collected representative FP8 linear, FlashInfer prefill-attention, FP8 MoE, and TP4 NCCL all-reduce measurements.
Loaded the full Qwen3-235B-A22B-FP8 checkpoint on four H20s through community vLLM and completed one real request.
Ran a 2x2 MoE diagnostic that independently toggles the runtime FP8 tuning key and BF16 compute type.

Expected effect

The smoke was intended to determine whether community vLLM can provide a shared, reproducible profiler/serving substrate before paying for the complete response surface. It was not intended to establish Frontier ranking accuracy.

Success required all representative operator and collective paths to execute, the TP4 server to become ready, one request to return successfully, and all GPU processes to be released. A discovered profiler/runtime semantic mismatch was treated as a gate on the later simulator comparison rather than calibrated away with end-to-end serving data.

Frozen environment

Host: dash0; 8 NVIDIA H20 GPUs.
Model: /home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8.
Original model-config SHA256: 702c46d431bb984db9035a1225186bbfdb52c0d19c82104df4a37cd005e0369e.
Python 3.12.3, torch 2.8.0+cu128, vLLM 0.10.2, flashinfer-python 0.3.1.post1, Transformers 4.55.2, tokenizers 0.21.4.
Ubuntu 24.04.2, kernel 5.10.134, CUDA runtime 12.8, CUDA toolkit 12.9, NVIDIA driver 580.95.05, GCC 13.3.0, glibc 2.39.
Two 80-core Intel Xeon sockets (160 online CPUs), 8 H20 GPUs with 97,871 MiB each and all-to-all NV18 links; GPU0--3 share NUMA node 0.
Serving: TP4, FlashInfer, eager execution, custom all-reduce disabled, FP8 weights, BF16 KV cache, MNS=64, MBT=8192, max model length 40,960, prefix caching and speculative decoding disabled.

The first environment resolved Transformers 5.13.1 because vLLM 0.10.2 only declares transformers>=4.55.2. That environment failed before model loading because vLLM accesses all_special_tokens_extended, an API absent from the new tokenizer implementation. Pinning the declared minimum restored the API; the original and corrected freezes are both retained.

Verification

End-to-end smoke

Stage	Result
FP8 linear, TP4, 16 tokens	Passed; QKV projection mean 0.2232 ms and output projection mean 0.1105 ms
FlashInfer prefill attention, TP4	Passed for sequence lengths 32/64/96/128; attention mean 0.0571--0.0665 ms
FP8 MoE, TP4/EP1, 16 tokens	Passed; unmodified grouped-GEMM mean 0.3075 ms in the initial point
TP4 NCCL all-reduce, 128 KiB/rank	Passed; mean 0.03694 ms, p50 0.03138 ms, p95 0.05843 ms
Community-vLLM TP4 model load	Passed; 55.1328 GiB of weights/rank, 627 seconds
KV-cache sizing	18.72 GiB/rank, 417,616 tokens; reported 10.20x concurrency at length 40,960
API readiness and one-token completion	Passed; `/v1/models` and `/v1/completions` returned HTTP 200
Cleanup	Passed; all eight GPUs returned to zero compute processes and zero MiB used

The successful serving-only run started at 09:09:52 UTC and completed at 09:22:29 UTC (12 minutes 37 seconds). The one-request response contains one completion choice and one completion token. The copied artifacts match the remote SHA256 manifests.

FP8 path audit

The Frontier confirmation UI prints FP8 Quantization: N/A and labels the operations BF16 because it displays the unset CLI override rather than the model-config-derived operation precision. This is misleading, not evidence of a BF16-weight profile:

the quantization manager configured attn_pre_proj, attn_post_proj, and moe_grouped_gemm as FP8 from the model config;
the linear implementation invokes apply_w8a8_block_fp8_linear with a 128x128 weight block;
the MoE implementation quantizes weights and activations and invokes vLLM's fused kernel with use_fp8_w8a8=True;
FRONTIER_FP8_GEMM_SURROGATE was not enabled;
each CSV carries method=fp8|act=dynamic|serialized=True|block=128x128 separately from its BF16 output/compute dtype metadata.

MoE runtime-alignment factorial

Point: Qwen3-235B-A22B-FP8, TP4, EP1, 16 input tokens, uniform routing seeds 0--4. Values are the mean of each row's CUDA-event samples, then averaged across the five paired routing seeds.

Variant	Grouped GEMM mean (ms)	Paired delta vs original	Routing-seed 95% CI
Original: default config + FP16 compute type	0.3100	0.00%	--
FP8 config key only + FP16 compute type	0.2508	-19.12%	[-21.26%, -16.98%]
Default config + BF16 compute type only	0.3088	-0.40%	[-1.31%, 0.51%]
FP8 config key + BF16 compute type	0.2512	-18.99%	[-21.14%, -16.85%]

These intervals describe variation across the five routing seeds, not independent process/server-run uncertainty. Even with that limitation, the factorial localizes the dominant error at this point to kernel tuning-config selection rather than FP16-versus-BF16 compute type.

The concrete lookup difference is:

Frontier calls get_config_dtype_str(base_dtype). For BF16 this returns None, so it searches for a config without an FP8 dtype component and uses the default when that file is absent.
vLLM serving calls the same helper with use_fp8_w8a8=True, obtains fp8_w8a8, and loads E=128,N=384,device_name=NVIDIA_H20,dtype=fp8_w8a8,block_shape=[128,128].json.

Frontier also hard-codes tl.float16 as the FP8 kernel compute type, while the serving path uses BF16 for this model. That difference was not material at this single point, but it should still be corrected to make the counterfactual execution semantics identical.

Validity audit

Headline claims and benchmark surface

Claim	Verdict	Scope
Community vLLM 0.10.2 can load and serve this model on TP4 H20	PASS	One frozen TP4 configuration and one real request
Unmodified Frontier chooses a different MoE tuning config, causing a material operator-time error at the tested point	PASS	TP4/EP1, 16 tokens, five paired routing seeds
Frontier is or is not sufficient as a Qwen235B config ranker	NEEDS EVIDENCE	No aligned full profiles, frozen simulator surface, or real config surface yet

The mechanism diagnostic uses the unmodified Frontier wrapper as baseline, grouped-GEMM CUDA-event latency as its direct metric, and the same model, hardware, vLLM binary, token count, TP/EP point, and routing seeds in all four cells. Absolute times, paired relative differences, across-seed standard deviations, and a paired routing-seed interval are retained. The data range is deliberately narrow and cannot support an end-to-end or cross-config claim.

Benchmark issue	Verdict	Severity	Evidence / required action
Microbenchmark presented as end-to-end performance	PASS	--	The 19% number is reported only as mechanism evidence; ranking remains blocked.
Simplified profiler matches the real execution path	FAIL	Blocking	Tuning-key and compute-type semantics differ. Apply the alignment patch and test exact config dictionaries before simulation.
Statistical significance / repeat protocol	NEEDS EVIDENCE	Major	Five paired routing seeds are not independent process repeats. Repeat in reverse/randomized variant order and at separated times.
Selective parameter range	NEEDS EVIDENCE	Major	Only TP4/EP1/16 tokens is measured. Cover the full decision-relevant token and TP range, including boundary values.
Calibration set equals evaluation set	PASS	--	No serving result calibrated the profile; the protocol freezes simulator outputs before ground truth.
Proper diagnostic baseline	PASS	--	Each ablation is compared with the exact unmodified Frontier commit on the same point.
Platform and absolute values missing	PASS	--	OS, kernel, CPU, GPU topology, driver/toolkit/runtime versions, commands, absolute metrics, logs, and hashes are retained.
Full ranking baseline/SOTA comparison	N/A	--	This smoke makes no ranking or tuner-superiority claim; it becomes required in the formal evaluation.

Overall audit decision: the feasibility smoke and localized mismatch result are reportable, but any Frontier-sufficiency or config-ranking claim is Block until the required reruns and end-to-end comparison are complete.

Result

The shared community stack is feasible: profiles can be collected and the 235B checkpoint can serve on TP4 within memory. The stronger result, however, is a falsification of the current alignment assumption. Same model, hardware, vLLM package, quantization signature, and nominal backend did not imply the same kernel configuration. A small wrapper-level omission produced a roughly 19% operator-time error before any scheduling approximation was involved.

Therefore no Frontier config ranking should be generated from the original MoE CSV and presented as a fair simulator-versus-real comparison. The correct next boundary is: first make the profiler reproduce the serving kernel selection, then freeze profile-only simulator outputs, and only then collect the real response surface.

Remaining risk and next gate

The 19% result is one TP4/EP1/16-token point. It must be checked across the decision-relevant token counts and TP4/TP8 before claiming its effect on config ordering.
Frontier's shuffling helper still performs a separate BF16/full-expert-width config lookup and reports a missing E=128,N=1536 H20 file. At 16 tokens its default BLOCK_SIZE_M matches the tuned path, but this must be validated over the full token grid.
The derived Frontier model config adds is_checkpoint_fp8_serialized=true, which Frontier requires to parse the checkpoint metadata. This adaptation must remain explicit and hash-tracked.
The representative attention and collective smokes do not provide the profile coverage needed by the prefill candidate grid. Decode-dominant EP8 remains blocked on all-to-all measurement/consumption.
A TP4 cold start spends about 10.5 minutes reading weights. Using that time as a lower-bound proxy, eight TP4/TP8 cells already cost roughly 8.4 H20-GPU-hours in weight loading alone; a real one-pass response surface will exceed 10 H20-GPU-hours after initialization and load probes. It requires a separate launch approval after simulator outputs are frozen.

Recommended next step: upstream the two semantic fixes with regression tests that compare the exact vLLM config dictionary selected by profiler and serving for each (M, TP, EP, quantization) point; align the shuffling block-size lookup; regenerate TP4/TP8 profile closure; then run and hash the Frontier prefill predictions. Do not start the eight-cell real serving sweep before that gate passes.

12 KiB Raw Blame History