115 lines
5.5 KiB
Markdown
115 lines
5.5 KiB
Markdown
# Frontier multi-case sufficiency: current finding
|
|
|
|
## Bottom line
|
|
|
|
The existing evidence does **not** show that Frontier already solves config
|
|
selection. The strongest Qwen30B match is an action-conditioned calibrated
|
|
result, while the zero-shot/profile-only result selects the wrong TP family.
|
|
The Qwen235B surfaces are useful follow-up cases, but decode ground truth and
|
|
Frontier's execution semantics must be repaired before a match claim is valid.
|
|
|
|
## Qwen30B: calibration is decision-bearing
|
|
|
|
Under the aligned maximum-SLO-feasible-throughput objective:
|
|
|
|
| reading | Frontier selection | worst real regret | Kendall tau-b |
|
|
|---|---|---:|---:|
|
|
| profile-only | TP4/MNS32 or TP4/MNS64 | 25.63% | 0.0000 |
|
|
| per-TP calibrated | TP2/MNS32 or TP2/MNS64 | 0.76% | 0.9668 |
|
|
|
|
The calibrated run multiplies all simulated execution times by a separately
|
|
fitted factor for each TP: TP1=0.723481, TP2=0.468089, TP4=0.352137. These
|
|
factors were fitted to real end-to-end throughput on the same model/workload
|
|
family. The large, TP-dependent residual is therefore evidence that the
|
|
profile-only simulator misses action-dependent execution behavior; it is not
|
|
evidence that the unmodified simulator predicts the action correctly.
|
|
|
|
## Qwen235B real response surfaces
|
|
|
|
### Prefill-only
|
|
|
|
- 8/8 cells are complete primary searches.
|
|
- The point-estimate top set contains all four TP8 cells, so a random cell has a
|
|
50% top-set hit probability.
|
|
- Capacity brackets still separate all TP8 cells from all TP4 cells. Thus this
|
|
case can test the **TP4 versus TP8** decision, but the current six probes do
|
|
not distinguish MNS/MBT within TP8.
|
|
|
|
### Decode-only
|
|
|
|
- Only 7/8 cells are fully valid; `TP2/DP4/EP8, MNS128, MBT384` is marked
|
|
`partial_probe_before_failure` after an engine restart/port failure.
|
|
- There are only two point-estimate score levels, and the top set contains 5/8
|
|
cells (62.5% random hit probability).
|
|
- More importantly, all eight binary-search capacity brackets overlap the best
|
|
observed lower bound. The current data cannot rule out any cell as optimal.
|
|
|
|
Consequently, a decode top-set hit on these results is not a match. After the
|
|
simulator selects a cell, the cheapest rigorous next step is to refine only
|
|
that cell and one competing topology/batching cell until their capacity
|
|
intervals separate or remain statistically indistinguishable.
|
|
|
|
## Frontier semantic coverage for Qwen235B
|
|
|
|
The topology itself is expressible: Frontier exposes attention TP/DP and MoE
|
|
TP/EP separately, so TP4/DP2/EP8 and TP2/DP4/EP8 need not be decomposed into
|
|
independent scheduling and execution problems.
|
|
|
|
The current execution model is not yet aligned, however:
|
|
|
|
- no checked-in H20 Qwen235B `linear_op.csv`, `attention.csv`, or `moe.csv`;
|
|
- checked-in Qwen235B config is BF16 with max position 40960, whereas the real
|
|
run uses FP8 weights, FP8 KV, and max model length 262144;
|
|
- real prefill uses FlashAttention and internal BLADNN kernels, while Frontier's
|
|
attention profiler exposes only FlashInfer and NO-OP backends;
|
|
- real decode starts with a dummy-filled external KV cache through
|
|
`DecodeBenchConnector`; Frontier's trace request generator requires positive
|
|
prefill tokens and has no equivalent initial-KV trace contract;
|
|
- real decode combines DeepEP/NVSHMEM, EAGLE3, and FULL_DECODE_ONLY CUDA graphs;
|
|
Frontier explicitly treats speculative decode plus decode CUDA graphs as a
|
|
conflicting/diagnostic combination.
|
|
|
|
Running dummy profiles or silently substituting FlashInfer/eager decode would
|
|
produce a number, but it would not test whether Frontier matches this system.
|
|
|
|
## Community-vLLM Qwen235B smoke
|
|
|
|
The controlled community stack is feasible: vLLM 0.10.2 loaded the
|
|
Qwen3-235B-A22B-FP8 checkpoint on TP4, allocated a BF16 KV cache, and completed
|
|
a real request. Representative FP8 linear/MoE, FlashInfer attention, and TP4
|
|
NCCL paths also executed successfully.
|
|
|
|
The smoke nevertheless found a profiler/runtime mismatch before any scheduler
|
|
model was involved. Frontier's MoE wrapper calls `get_config_dtype_str` without
|
|
`use_fp8_w8a8=True`, so it misses vLLM's tuned H20 block-FP8 Triton config and
|
|
uses a default. At TP4/EP1 with 16 tokens, a paired five-routing-seed factorial
|
|
measured:
|
|
|
|
| variant | grouped-GEMM mean | paired delta vs original |
|
|
|---|---:|---:|
|
|
| original default config + FP16 compute type | 0.3100 ms | 0.00% |
|
|
| FP8 config key only | 0.2508 ms | -19.12% |
|
|
| BF16 compute type only | 0.3088 ms | -0.40% |
|
|
| both aligned | 0.2512 ms | -18.99% |
|
|
|
|
The config-key-only routing-seed 95% interval is [-21.26%, -16.98%]; the
|
|
compute-type-only interval crosses zero. This localizes the dominant error at
|
|
this point to kernel tuning-config selection. The original MoE CSV must not be
|
|
used for a formal Frontier ranking until this path and the related shuffling
|
|
block-size lookup are aligned and re-profiled. Full evidence and hashes are in
|
|
`results/community-qwen235b-smoke/`.
|
|
|
|
## Research implication
|
|
|
|
The most interesting observation is already visible: Frontier's error is not a
|
|
single global time bias. It changes strongly with the execution action (TP in
|
|
Qwen30B), and a per-action residual can reverse the selected config. Qwen235B
|
|
prefill and decode offer held-out tests of whether those residuals are explained
|
|
by measurable execution-state features (kernel family, communication mode,
|
|
graph mode, speculative width, and KV initial state) or require case-specific
|
|
end-to-end fitting.
|
|
|
|
That is a systems question: **which execution-state transitions make operator
|
|
profile composition non-invariant across configurations, and what is the
|
|
minimum real evidence needed to recover the counterfactual ordering?**
|