Frontier multi-case sufficiency: current finding

Bottom line

The existing evidence does not show that Frontier already solves config selection. The strongest Qwen30B match is an action-conditioned calibrated result, while the zero-shot/profile-only result selects the wrong TP family. The Qwen235B surfaces are useful follow-up cases, but decode ground truth and Frontier's execution semantics must be repaired before a match claim is valid.

Qwen30B: calibration is decision-bearing

Under the aligned maximum-SLO-feasible-throughput objective:

reading	Frontier selection	worst real regret	Kendall tau-b
profile-only	TP4/MNS32 or TP4/MNS64	25.63%	0.0000
per-TP calibrated	TP2/MNS32 or TP2/MNS64	0.76%	0.9668

The calibrated run multiplies all simulated execution times by a separately fitted factor for each TP: TP1=0.723481, TP2=0.468089, TP4=0.352137. These factors were fitted to real end-to-end throughput on the same model/workload family. The large, TP-dependent residual is therefore evidence that the profile-only simulator misses action-dependent execution behavior; it is not evidence that the unmodified simulator predicts the action correctly.

Qwen235B real response surfaces

Prefill-only

8/8 cells are complete primary searches.
The point-estimate top set contains all four TP8 cells, so a random cell has a 50% top-set hit probability.
Capacity brackets still separate all TP8 cells from all TP4 cells. Thus this case can test the TP4 versus TP8 decision, but the current six probes do not distinguish MNS/MBT within TP8.

Decode-only

Only 7/8 cells are fully valid; TP2/DP4/EP8, MNS128, MBT384 is marked partial_probe_before_failure after an engine restart/port failure.
There are only two point-estimate score levels, and the top set contains 5/8 cells (62.5% random hit probability).
More importantly, all eight binary-search capacity brackets overlap the best observed lower bound. The current data cannot rule out any cell as optimal.

Consequently, a decode top-set hit on these results is not a match. After the simulator selects a cell, the cheapest rigorous next step is to refine only that cell and one competing topology/batching cell until their capacity intervals separate or remain statistically indistinguishable.

Frontier semantic coverage for Qwen235B

The topology itself is expressible: Frontier exposes attention TP/DP and MoE TP/EP separately, so TP4/DP2/EP8 and TP2/DP4/EP8 need not be decomposed into independent scheduling and execution problems.

The current execution model is not yet aligned, however:

no checked-in H20 Qwen235B linear_op.csv, attention.csv, or moe.csv;
checked-in Qwen235B config is BF16 with max position 40960, whereas the real run uses FP8 weights, FP8 KV, and max model length 262144;
real prefill uses FlashAttention and internal BLADNN kernels, while Frontier's attention profiler exposes only FlashInfer and NO-OP backends;
real decode starts with a dummy-filled external KV cache through DecodeBenchConnector; Frontier's trace request generator requires positive prefill tokens and has no equivalent initial-KV trace contract;
real decode combines DeepEP/NVSHMEM, EAGLE3, and FULL_DECODE_ONLY CUDA graphs; Frontier explicitly treats speculative decode plus decode CUDA graphs as a conflicting/diagnostic combination.

Running dummy profiles or silently substituting FlashInfer/eager decode would produce a number, but it would not test whether Frontier matches this system.

Community-vLLM Qwen235B smoke

The controlled community stack is feasible: vLLM 0.10.2 loaded the Qwen3-235B-A22B-FP8 checkpoint on TP4, allocated a BF16 KV cache, and completed a real request. Representative FP8 linear/MoE, FlashInfer attention, and TP4 NCCL paths also executed successfully.

The smoke nevertheless found a profiler/runtime mismatch before any scheduler model was involved. Frontier's MoE wrapper calls get_config_dtype_str without use_fp8_w8a8=True, so it misses vLLM's tuned H20 block-FP8 Triton config and uses a default. At TP4/EP1 with 16 tokens, a paired five-routing-seed factorial measured:

variant	grouped-GEMM mean	paired delta vs original
original default config + FP16 compute type	0.3100 ms	0.00%
FP8 config key only	0.2508 ms	-19.12%
BF16 compute type only	0.3088 ms	-0.40%
both aligned	0.2512 ms	-18.99%

The config-key-only routing-seed 95% interval is [-21.26%, -16.98%]; the compute-type-only interval crosses zero. This localizes the dominant error at this point to kernel tuning-config selection. The original MoE CSV must not be used for a formal Frontier ranking until this path and the related shuffling block-size lookup are aligned and re-profiled. Full evidence and hashes are in results/community-qwen235b-smoke/.

Research implication

The most interesting observation is already visible: Frontier's error is not a single global time bias. It changes strongly with the execution action (TP in Qwen30B), and a per-action residual can reverse the selected config. Qwen235B prefill and decode offer held-out tests of whether those residuals are explained by measurable execution-state features (kernel family, communication mode, graph mode, speculative width, and KV initial state) or require case-specific end-to-end fitting.

That is a systems question: which execution-state transitions make operator profile composition non-invariant across configurations, and what is the minimum real evidence needed to recover the counterfactual ordering?

5.5 KiB Raw Blame History