167 lines
7.3 KiB
Markdown
167 lines
7.3 KiB
Markdown
# Community vLLM Qwen235B versus Frontier protocol v0
|
|
|
|
## Hypothesis
|
|
|
|
I believe that collecting compute and communication profiles from the same
|
|
community vLLM execution stack used for serving removes the current
|
|
internal-runtime confounder. I will verify this by freezing Frontier's
|
|
profile-only predictions before collecting the corresponding serving response
|
|
surface, then measuring config-selection regret and rank agreement.
|
|
|
|
This experiment tests Frontier inside a declared compatibility envelope. It
|
|
does not claim fidelity for the previous internal vLLM, EAGLE3, DeepEP, or
|
|
external-KV setup.
|
|
|
|
## Frozen system boundary
|
|
|
|
- Host: `dash0`, 8 NVIDIA H20 GPUs, driver `580.95.05`.
|
|
- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8`.
|
|
- Model config SHA256:
|
|
`702c46d431bb984db9035a1225186bbfdb52c0d19c82104df4a37cd005e0369e`.
|
|
- Model index SHA256:
|
|
`e03d4abec9611fca05844d2c8b5a08318ddc3c86acfb9481a0ef5692bf76e6d6`.
|
|
- Frontier: NetX-lab/Frontier commit
|
|
`d9cfeb6d8791fbf2f295dd9744c56a666171776e`.
|
|
- vLLM target: exact community release `0.10.2`, installed in a new isolated
|
|
environment with wheel/source hash recorded. Frontier declares
|
|
`vllm>=0.10,<0.11`; the model README declares `vllm>=0.8.5`.
|
|
- Transformers is pinned to `4.55.2` and tokenizers resolves to `0.21.4`.
|
|
vLLM 0.10.2 only declares a lower bound on Transformers, while current
|
|
Transformers 5.x removes a tokenizer API used by this vLLM release.
|
|
- Do not use the shared `/usr/local` vLLM. Its imported version and package
|
|
metadata disagree, so it is not a reproducible community baseline.
|
|
|
|
The profiler and serving process must import the same vLLM installation. The
|
|
following execution choices are fixed for the first validation pass:
|
|
|
|
- FlashInfer attention in both profiler and serving;
|
|
- eager execution, with CUDA graphs disabled;
|
|
- no speculative decoding;
|
|
- no external KV connector;
|
|
- no prefix reuse;
|
|
- chunked prefill enabled;
|
|
- BF16 KV cache (`auto` for this model), while the checkpoint's block-wise FP8
|
|
weight quantization remains enabled;
|
|
- community vLLM default MoE implementation unless a backend is explicitly
|
|
frozen and supported on both sides.
|
|
|
|
The SM90 path cannot use `CutlassBlockScaledGroupedGemm` (the vLLM 0.10.2
|
|
implementation gates that path on SM100), so this system uses Triton block-FP8
|
|
MoE. However, the two sides do not yet select the same Triton tuning config.
|
|
Serving resolves the checked-in H20 config named with
|
|
`dtype=fp8_w8a8`, while Frontier's standalone MoE profiler requests a filename
|
|
without that FP8 dtype component and falls back to a default config. Formal
|
|
comparison is gated on repairing this profiler/runtime tuning-config mismatch
|
|
and re-profiling; recording the same vLLM package version is not sufficient.
|
|
|
|
These controls isolate operator composition and scheduling. CUDA graphs,
|
|
FlashAttention, speculative decoding, external KV, and optimized expert
|
|
communication become separate stress cases after the controlled pass.
|
|
|
|
## Case P: prefill-only
|
|
|
|
Reuse the original trace window and SLO because the community model's 40,960
|
|
token limit covers the filtered input plus the one-token completion.
|
|
|
|
- Window: `thinking_w20260327_1000`.
|
|
- Input filter: 0--32,768 tokens.
|
|
- Output override: 1 token.
|
|
- Replay scale: 1.0.
|
|
- SLO pass rate: at least 0.95.
|
|
- TTFT SLO: 1 s for input <=8,191; 2 s otherwise.
|
|
- Objective: maximum SLO-feasible offered requests/s/GPU.
|
|
- Candidate grid: TP `{4,8}` x MNS `{64,128}` x MBT `{8192,16384}`.
|
|
- DP=1 and expert parallel disabled.
|
|
|
|
This case can test the TP4-versus-TP8 decision and batching effects. It must not
|
|
reuse performance values from the internal 256k model as ground truth.
|
|
|
|
## Case D: decode-dominant
|
|
|
|
Do not initially reproduce the previous strict decode-only case. It depends on
|
|
`DecodeBenchConnector`, EAGLE3, FP8 KV, DeepEP/NVSHMEM, and decode CUDA graphs,
|
|
which are outside the controlled Frontier profile contract.
|
|
|
|
Construct a community-only decode-dominant case that both systems can express:
|
|
|
|
- same trace window and timestamp/sampling fields;
|
|
- input filter: 1--512 tokens;
|
|
- output override: 512 tokens with EOS ignored;
|
|
- SLO pass rate: at least 0.95;
|
|
- TPOT SLO: 40 ms;
|
|
- objective: maximum SLO-feasible offered requests/s/GPU;
|
|
- topology grid: `(TP=4, DP=2, EP=8)` and `(TP=2, DP=4, EP=8)`;
|
|
- batching grid: MNS `{64,128}` x MBT `{256,384}`.
|
|
|
|
This is deliberately named decode-dominant, not decode-only. A strict
|
|
decode-only claim requires an initial-KV state contract in Frontier.
|
|
|
|
## Required profile closure
|
|
|
|
Profile data are measurement inputs, not end-to-end calibration. No serving
|
|
throughput or latency from either case may scale the profiles.
|
|
|
|
1. Linear/operator profiles for the TP degrees consumed by the two grids.
|
|
2. FlashInfer attention profiles covering the observed prefill/decode batch,
|
|
context-length, and chunk-size ranges.
|
|
3. FP8 MoE profiles for the actually consumed parallel pairs:
|
|
`(MoE TP=4, EP=1)`, `(MoE TP=8, EP=1)`, and `(MoE TP=1, EP=8)`.
|
|
4. H20 intra-node collective profiles for TP all-reduce at world sizes 2, 4,
|
|
and 8 and the EP8 all-to-all path.
|
|
|
|
Frontier already provides an H20 device description but no checked-in H20
|
|
network profiles. Its public collective profiler covers all-reduce and
|
|
send/recv, not all-to-all. Therefore Case P may proceed after H20 all-reduce
|
|
closure; Case D remains blocked until EP8 all-to-all is either measured and
|
|
consumed or the selected communication model is independently validated
|
|
against those measurements.
|
|
|
|
## Blind run order
|
|
|
|
1. Build the isolated environment and record package/binary hashes.
|
|
2. Run one-row compute and collective smokes, then one TP4 server-load/request
|
|
smoke. A smoke failure stops the campaign.
|
|
3. Collect profiles and validate CSV metadata/coverage.
|
|
4. Run Frontier for every candidate and offered-load anchor.
|
|
5. Freeze simulator outputs and their SHA256 checksums.
|
|
6. Only then collect community-vLLM serving ground truth. Randomize the first
|
|
trial order and reverse it for the second trial.
|
|
7. Refine only decision-relevant capacity intervals that still overlap.
|
|
|
|
## Metrics and decision rule
|
|
|
|
Report per case:
|
|
|
|
- absolute simulated and real TTFT/TPOT/throughput values;
|
|
- anchor-level SLO feasibility confusion;
|
|
- selected-config real regret, including interval-robust regret;
|
|
- Kendall tau-b with ties preserved;
|
|
- informative-pair direction accuracy;
|
|
- top-set hit and random top-set hit probability;
|
|
- profile and real-GPU measurement cost.
|
|
|
|
Frontier is sufficient as a config ranker for this controlled family only if
|
|
every completed case has:
|
|
|
|
- worst selected-config real regret <=5%;
|
|
- Kendall tau-b >=0.8 on enough informative pairs;
|
|
- no unresolved ground-truth interval capable of reversing the decision;
|
|
- no per-case or per-action end-to-end calibration.
|
|
|
|
If the controlled pass succeeds, enable one omitted mechanism at a time. If it
|
|
fails, use operator/communication/stage residuals to localize which composition
|
|
assumption reverses the ranking before proposing a new tuner mechanism.
|
|
|
|
## Launch gates and initial cost cap
|
|
|
|
The first authorized GPU action should be smoke-only:
|
|
|
|
- isolated vLLM import and Qwen235B TP4 load;
|
|
- one request through the community server;
|
|
- one representative FP8 linear, attention, MoE, and all-reduce profile point;
|
|
- expected wall time: 20--40 minutes;
|
|
- hard GPU budget: 2 H20-GPU-hours.
|
|
|
|
No full response-surface sweep is authorized by this protocol. Its cost and
|
|
anchor count must be resolved from the smoke timings and echoed separately.
|