Track simulator fidelity experiment artifacts

runs/frontier-multicase-sufficiency-v0/community-qwen235b-protocol.md

@@ -0,0 +1,166 @@
+# Community vLLM Qwen235B versus Frontier protocol v0
+
+## Hypothesis
+
+I believe that collecting compute and communication profiles from the same
+community vLLM execution stack used for serving removes the current
+internal-runtime confounder. I will verify this by freezing Frontier's
+profile-only predictions before collecting the corresponding serving response
+surface, then measuring config-selection regret and rank agreement.
+
+This experiment tests Frontier inside a declared compatibility envelope. It
+does not claim fidelity for the previous internal vLLM, EAGLE3, DeepEP, or
+external-KV setup.
+
+## Frozen system boundary
+
+- Host: `dash0`, 8 NVIDIA H20 GPUs, driver `580.95.05`.
+- Model: `/home/admin/cpfs/wjh/models/Qwen/Qwen3-235B-A22B-FP8`.
+- Model config SHA256:
+  `702c46d431bb984db9035a1225186bbfdb52c0d19c82104df4a37cd005e0369e`.
+- Model index SHA256:
+  `e03d4abec9611fca05844d2c8b5a08318ddc3c86acfb9481a0ef5692bf76e6d6`.
+- Frontier: NetX-lab/Frontier commit
+  `d9cfeb6d8791fbf2f295dd9744c56a666171776e`.
+- vLLM target: exact community release `0.10.2`, installed in a new isolated
+  environment with wheel/source hash recorded. Frontier declares
+  `vllm>=0.10,<0.11`; the model README declares `vllm>=0.8.5`.
+- Transformers is pinned to `4.55.2` and tokenizers resolves to `0.21.4`.
+  vLLM 0.10.2 only declares a lower bound on Transformers, while current
+  Transformers 5.x removes a tokenizer API used by this vLLM release.
+- Do not use the shared `/usr/local` vLLM. Its imported version and package
+  metadata disagree, so it is not a reproducible community baseline.
+
+The profiler and serving process must import the same vLLM installation. The
+following execution choices are fixed for the first validation pass:
+
+- FlashInfer attention in both profiler and serving;
+- eager execution, with CUDA graphs disabled;
+- no speculative decoding;
+- no external KV connector;
+- no prefix reuse;
+- chunked prefill enabled;
+- BF16 KV cache (`auto` for this model), while the checkpoint's block-wise FP8
+  weight quantization remains enabled;
+- community vLLM default MoE implementation unless a backend is explicitly
+  frozen and supported on both sides.
+
+The SM90 path cannot use `CutlassBlockScaledGroupedGemm` (the vLLM 0.10.2
+implementation gates that path on SM100), so this system uses Triton block-FP8
+MoE. However, the two sides do not yet select the same Triton tuning config.
+Serving resolves the checked-in H20 config named with
+`dtype=fp8_w8a8`, while Frontier's standalone MoE profiler requests a filename
+without that FP8 dtype component and falls back to a default config. Formal
+comparison is gated on repairing this profiler/runtime tuning-config mismatch
+and re-profiling; recording the same vLLM package version is not sufficient.
+
+These controls isolate operator composition and scheduling. CUDA graphs,
+FlashAttention, speculative decoding, external KV, and optimized expert
+communication become separate stress cases after the controlled pass.
+
+## Case P: prefill-only
+
+Reuse the original trace window and SLO because the community model's 40,960
+token limit covers the filtered input plus the one-token completion.
+
+- Window: `thinking_w20260327_1000`.
+- Input filter: 0--32,768 tokens.
+- Output override: 1 token.
+- Replay scale: 1.0.
+- SLO pass rate: at least 0.95.
+- TTFT SLO: 1 s for input <=8,191; 2 s otherwise.
+- Objective: maximum SLO-feasible offered requests/s/GPU.
+- Candidate grid: TP `{4,8}` x MNS `{64,128}` x MBT `{8192,16384}`.
+- DP=1 and expert parallel disabled.
+
+This case can test the TP4-versus-TP8 decision and batching effects. It must not
+reuse performance values from the internal 256k model as ground truth.
+
+## Case D: decode-dominant
+
+Do not initially reproduce the previous strict decode-only case. It depends on
+`DecodeBenchConnector`, EAGLE3, FP8 KV, DeepEP/NVSHMEM, and decode CUDA graphs,
+which are outside the controlled Frontier profile contract.
+
+Construct a community-only decode-dominant case that both systems can express:
+
+- same trace window and timestamp/sampling fields;
+- input filter: 1--512 tokens;
+- output override: 512 tokens with EOS ignored;
+- SLO pass rate: at least 0.95;
+- TPOT SLO: 40 ms;
+- objective: maximum SLO-feasible offered requests/s/GPU;
+- topology grid: `(TP=4, DP=2, EP=8)` and `(TP=2, DP=4, EP=8)`;
+- batching grid: MNS `{64,128}` x MBT `{256,384}`.
+
+This is deliberately named decode-dominant, not decode-only. A strict
+decode-only claim requires an initial-KV state contract in Frontier.
+
+## Required profile closure
+
+Profile data are measurement inputs, not end-to-end calibration. No serving
+throughput or latency from either case may scale the profiles.
+
+1. Linear/operator profiles for the TP degrees consumed by the two grids.
+2. FlashInfer attention profiles covering the observed prefill/decode batch,
+   context-length, and chunk-size ranges.
+3. FP8 MoE profiles for the actually consumed parallel pairs:
+   `(MoE TP=4, EP=1)`, `(MoE TP=8, EP=1)`, and `(MoE TP=1, EP=8)`.
+4. H20 intra-node collective profiles for TP all-reduce at world sizes 2, 4,
+   and 8 and the EP8 all-to-all path.
+
+Frontier already provides an H20 device description but no checked-in H20
+network profiles. Its public collective profiler covers all-reduce and
+send/recv, not all-to-all. Therefore Case P may proceed after H20 all-reduce
+closure; Case D remains blocked until EP8 all-to-all is either measured and
+consumed or the selected communication model is independently validated
+against those measurements.
+
+## Blind run order
+
+1. Build the isolated environment and record package/binary hashes.
+2. Run one-row compute and collective smokes, then one TP4 server-load/request
+   smoke. A smoke failure stops the campaign.
+3. Collect profiles and validate CSV metadata/coverage.
+4. Run Frontier for every candidate and offered-load anchor.
+5. Freeze simulator outputs and their SHA256 checksums.
+6. Only then collect community-vLLM serving ground truth. Randomize the first
+   trial order and reverse it for the second trial.
+7. Refine only decision-relevant capacity intervals that still overlap.
+
+## Metrics and decision rule
+
+Report per case:
+
+- absolute simulated and real TTFT/TPOT/throughput values;
+- anchor-level SLO feasibility confusion;
+- selected-config real regret, including interval-robust regret;
+- Kendall tau-b with ties preserved;
+- informative-pair direction accuracy;
+- top-set hit and random top-set hit probability;
+- profile and real-GPU measurement cost.
+
+Frontier is sufficient as a config ranker for this controlled family only if
+every completed case has:
+
+- worst selected-config real regret <=5%;
+- Kendall tau-b >=0.8 on enough informative pairs;
+- no unresolved ground-truth interval capable of reversing the decision;
+- no per-case or per-action end-to-end calibration.
+
+If the controlled pass succeeds, enable one omitted mechanism at a time. If it
+fails, use operator/communication/stage residuals to localize which composition
+assumption reverses the ranking before proposing a new tuner mechanism.
+
+## Launch gates and initial cost cap
+
+The first authorized GPU action should be smoke-only:
+
+- isolated vLLM import and Qwen235B TP4 load;
+- one request through the community server;
+- one representative FP8 linear, attention, MoE, and all-reduce profile point;
+- expected wall time: 20--40 minutes;
+- hard GPU budget: 2 H20-GPU-hours.
+
+No full response-surface sweep is authorized by this protocol. Its cost and
+anchor count must be resolved from the smoke timings and echoed separately.