experiment: add workload regime taxonomy

This commit is contained in:
2026-07-20 17:36:02 +08:00
parent 39766141fb
commit 75946d9d73
7 changed files with 713 additions and 14 deletions

View File

@@ -0,0 +1,160 @@
# Frontier workload-regime taxonomy
- Date: 2026-07-20
- Status: proposed; awaiting review before workload generation or GPU runs
- Scope: explain when Frontier preserves the real-system config ranking, rather than merely comparing Fixed with Trace
## Claim under test
Frontier reliability is controlled by three quantities:
1. the latency-model residual between simulator and real execution;
2. the closed-loop gain from timing to scheduler state (batch, MoE routing, CUDA-graph bucket, MNS occupancy, admission/KV pressure);
3. the real decision margin between configurations.
For a config pair `a,b`, define
```text
D_real(a,b) = log L_real(a) - log L_real(b)
delta(a,b) = [log L_sim(a)-log L_real(a)]
- [log L_sim(b)-log L_real(b)]
slack(a,b) = sign(D_real) * [D_real + delta]
```
`slack < 0` means the simulator reverses the real pairwise ordering. The primary hypothesis is that reversals occur when simulator and real execution land on different sides of a scheduler-state knee, or when the real decision margin is too small to absorb the differential residual. `Fixed` and `Trace` are not themselves the causal classes.
## Existing evidence motivating the experiment
- Q30 Trace-PD preserves all six objective winners, but many pairwise residuals oppose the real winner. Its success is therefore often margin protection, not zero residual.
- Q235 Trace-PD preserves TTFT/TPOT winners but misses E2E p90 by 6.2%; Trace is not universally safe.
- Q30/Q235 Fixed-PD decode objectives show negative minimum signed slack and 13--37% regret.
- In Q30 low-load Fixed-PD, Frontier's batch-1 TP ordering is correct, while the closed-loop simulator increases TP4's effective batch and changes the MoE cost enough to reverse the ordering. This identifies a concrete state knee, but does not yet establish a general rule.
## Workload families
All comparisons use the same request multiset where applicable, the same total observation window, and the same normalized offered decode load
```text
rho = request_rate * E[output_tokens] / measured_reference_decode_capacity.
```
This avoids equating equal request rates with equal load.
| ID | Shape / request lengths | Arrival process | Prefix/session state | Isolated effect |
|---|---|---|---|---|
| W0 | short fixed `2048 -> 128` | uniform | off | known low-residence failure anchor |
| W1 | trace-mean fixed ISL/OSL | uniform | off | homogeneous baseline |
| W2 | trace-mean fixed ISL/OSL | trace timestamps | off | arrival burst only |
| W3 | exact trace ISL/OSL multiset | uniform | off | length heterogeneity only |
| W4 | exact trace ISL/OSL multiset | trace timestamps | off | length + burst |
| W5 | exact trace prompts/ISL/OSL | uniform | exact prefix/session identity | prefix state without burst |
| W6 | exact trace prompts/ISL/OSL | trace timestamps | exact prefix/session identity | full production trace |
Prefix is intentionally a nested factor: enabling a synthetic prefix graph on fixed identical requests would introduce a different workload rather than isolate production prefix reuse. Therefore this is not presented as a full `2^3` factorial.
## Load sweep and expected patterns
Simulator discovery sweep: `rho in {0.05, 0.25, 0.50, 0.90, 1.20}`. The points mean deep low load, light batching, moderate batching, capacity knee, and overload; their request rates are derived independently for every workload family.
| Pattern | Observable state | Prediction for Frontier |
|---|---|---|
| P1 singleton-linear | real and sim stay below the first batch/graph knee | works if the batch-1 operator ordering is correct |
| P2 knee-straddling | real and sim occupy opposite sides of a batch/MoE/graph/MNS knee | fails systematically; Fixed-PD is the current example |
| P3 same-side batched | both systems cross the same knee and remain below admission pressure | works if batch-conditioned operator ordering is correct |
| P4 capacity/admission aligned | both systems are governed by the same capacity bottleneck | TTFT/config winner may work despite large absolute error; E2E/MNS can remain fragile |
| P5 heterogeneity-smoothed | broad lengths reduce coherent threshold occupancy at matched `rho` | may work; this is a hypothesis, not an established explanation |
| P6 burst-sensitive | same request multiset, but transient queue/MNS occupancy differs | mean ranking may work while TTFT/E2E tail ranking fails |
| P7 prefix-state-sensitive | hit/eviction and reused-token distributions differ | TTFT ranking fails unless prefix-state transitions are modeled; decode TPOT may remain stable |
| P8 decision-boundary | real config margin is comparable to run variance/residual | fragile; an exact winner match is not reliable evidence |
## Hypotheses and distinguishing tests
### H1: state-regime hypothesis (primary)
I believe config-ranking failures occur when the latency residual moves a workload across a scheduler-state knee, because the residual is then amplified into a different batch/resource trajectory. I will verify this by checking whether signed-slack zero crossings co-locate with measured real/simulator state-knee crossings.
### H2: heterogeneity-smoothing hypothesis
I believe length heterogeneity can reduce coherent threshold amplification, because requests reach scheduler boundaries at dispersed times. I will verify it with W1 vs W3 and W2 vs W4 at matched `rho`, requiring a smaller real/sim state-distribution gap rather than merely a correct winner.
### H3: bottleneck/margin-protection alternative
Trace success may instead be explained entirely by a large real decision margin or a shared capacity bottleneck. This hypothesis wins over H2 if W3/W4 do not reduce state-distribution error after matching load and margin, while ranking correctness remains predicted by margin alone.
### H4: burst and prefix are independent failure channels
I believe arrival bursts primarily affect waiting/admission and tail TTFT/E2E, whereas prefix mismatch primarily affects prefill/TTFT state. I will verify this with W1/W2, W3/W4, and W3/W5 paired comparisons.
## Configuration and model scope
Discovery uses Qwen30B because its 12-cell `TP x MNS` surface already has simulator and real anchors:
- TP: `{1, 2, 4}`
- MNS: `{8, 32, 64, 128}`
- objectives: mean/p90 TTFT, TPOT, E2E
Qwen235B is a held-out confirmation, not pooled into discovery:
- existing four feasible TP/MNS configurations;
- only the workload/load patterns that discriminate H1--H4 after Q30 converges.
## Measurements
End-to-end:
- completed/failed requests and achieved request/token rate;
- TTFT, TPOT, E2E mean/p50/p90/p95;
- config regret, pairwise agreement, signed decision slack;
- run-to-run winner stability.
Closed-loop state:
- prefill/decode batch-size histograms and time-weighted batch;
- Running/Waiting distributions and admission delay;
- MNS active-token occupancy and KV/context pressure;
- CUDA-graph bucket residency and fallback frequency;
- prefix hit/reused-token/eviction distributions for W5/W6.
## Decision rules
A workload/load region is:
- **reliable** if regret is at most 5%, pairwise agreement is at least 0.8 at two adjacent load points, and the winner is stable across confirmation trials;
- **fragile** if regret is at most 5% but the real margin overlaps run uncertainty, or a small rate/timing perturbation changes the winner;
- **failed** if regret exceeds 5% or a decision-critical pair has negative signed slack;
- **mechanistically explained by H1** only if the ranking transition co-locates with an observed state-regime transition. Correlation with the Fixed/Trace label is insufficient.
H2 is supported only if the heterogeneous member of a matched pair reduces state-distribution error and shifts the failure boundary in repeated trials. A correct winner alone does not support smoothing.
## Execution plan after review
1. Materialize W0--W6 with one manifest recording request multiset, arrival timestamps, prefix identity, rate contract, and hashes.
2. Run the simulator sweep across `rho` and the Q30 surface; emit a per-stage state ledger.
3. Select real-machine pilot points only around the predicted knees plus one safe-side control. Use guard configs `TP1/MNS64`, `TP4/MNS8`, and `TP4/MNS64`; add `TP2/MNS32` only if the transition is not bracketed.
4. Use only `dash1`, `dash2`, `dash3`, and `dash4`, each verified as an 8×H20 host. `dash0` is excluded from probing, synchronization, and execution. Pin one independent experiment group to each host so at most four groups run in parallel; do not split one trial across hosts.
5. Run one pilot trial per selected point. Confirm only hypothesis-discriminating points with three fresh-server trials and rotated order.
6. Apply the resulting classifier unchanged to the Q235 held-out cases.
Provisional four-way allocation after the simulator identifies the discriminating points:
| Host | Experiment group | Primary contrast |
|---|---|---|
| dash1 | homogeneous controls | W0/W1 across safe side and first knee |
| dash2 | arrival effect | W1 vs W2 and W3 vs W4 |
| dash3 | length heterogeneity | W1 vs W3 and W2 vs W4 |
| dash4 | prefix/full trace | W3 vs W5 and W4 vs W6 |
The groups are logical queues, not permanent ownership: if a host probe fails, that host is excluded and its group waits or moves to another permitted idle host. Cross-host latency values are not pooled until a common canary config verifies that host effects are within run uncertainty.
No GPU run is authorized by this card yet. The review decision is whether the workload decomposition and decision rules are sufficient to implement the materializer and launch Phase 1.
## Expected figure
The accompanying mock figure is schematic, not data. Panel A shows the state knee that real and simulator trajectories may cross at different loads. Panel B shows the corresponding minimum signed decision slack; a negative value denotes a ranking reversal. The claim is supported only if measured zero crossings and state knees align across workload families.
## Risks and controls
- Equal `rho` does not guarantee equal prefill pressure; report both prefill and decode offered work and stratify if necessary.
- Full-trace overload can collapse all configs to similarly poor latency. Such points identify a capacity-limited region but cannot validate fine-grained ranking.
- MNS ties and censored/failed requests can create false winners; exclude invalid cells before calculating regret and report the exclusion.
- One trace cannot establish generality. The initial result is a mechanism boundary for this trace/model/hardware, followed by held-out Q235 validation.

View File

@@ -5,14 +5,18 @@
> **2026-07-19 update** Qwen235 Fixed-PD 的错误排序在 exact real state composition 下已经翻正,主因是 simulator closed-loop batch state而不是 collective。Qwen30 Fixed-PD 的 56--58% TTFT regret 也已定位Frontier 将 decode service time 高估 4--8×使 TP4 的 modeled concurrency 越过 MNS admission cap并产生虚假排队去掉该等待后 Frontier 与真机都判定 TP4 topology 更快。详见 [`experiments/qwen30-fixed-pd-ttft-admission-diagnosis-20260719.md`](experiments/qwen30-fixed-pd-ttft-admission-diagnosis-20260719.md)。
>
> **2026-07-20 update** 对全部 14 个 frozen case surface70 个 case×objective做了统一的 margin-vs-residual 分解与方向化机制普查([`experiments/frontier-split-rootcause-s0-20260720.md`](experiments/frontier-split-rootcause-s0-20260720.md))。三个要点:(1) 「residual 超过 margin」是失败的必要条件但远非充分——good/bad 分裂不能用无方向误差量解释;(2) 23 个 material failure 的 winner-deciding pair 中 21 个落在 TP 轴或 mixed其余 2 个是 6.2% regret 的边缘 mns-axis casetrace 面的 TP 反序为零A1 measured collective 把 Qwen235 两个 prefill-only 面的 TP 反序清零trace-PO p90 regret 21.2%→0.3%)却对 Fixed-PD 完全无效——prefill 路径的 TP 差异化误差源是 collective profile可修decode 耦合的 TP 差异化误差是当前所有 material failure 的载体;(3) 「Fixed-PD 失败因为高压」被否证:失败 Fixed-PD 的真机 in-flight14.05)低于全对的 Trace-PD38.69),且低压 Fixed-PD 同样失败、失败 objective 随负载切换。另有次要缺陷14 个 winner 错位来自 simulator 对 MNS 逐位不敏感的精确 tie。
>
> **2026-07-20 root-cause update** Q30 低压 Fixed-PD 的 exact stage ledger 关闭了最后一个未解释的 material failure。相同 batch=1 state 下 Frontier full predictor 给 TP4 `18.3515 ms/step`、TP1 `19.3506 ms/step`,方向正确;但 per-GPU 固定到达率使 cluster arrival 随 TP 增长,叠加 decode residence 高估后TP4 在 simulator 内自激到 time-weighted batch `3.0437`96.13% decode 时间 batch≥3own-state step 变为 `28.1712 ms`。其中相对 batch=1 的 `+9.8197 ms` 有 `+8.9297 ms` 来自 batch-conditioned MoEcollective 仅 `+0.0121 ms`。因此 Fixed-PD 的根因不是“固定 workload”或“高压力”本身而是 **execution-time residual 进入离散事件时钟后改变 future scheduler state该 state 再通过 MoE/profile/graph 或 MNS admission 非线性放大,形成 action-dependent signed residual 并穿过 decision margin**。Q30 低压是平滑 state-feedbackQ30 高压是跨 MNS cap 的 threshold amplificationQ235 是 composition drift三者为同一闭环机制族。
>
> **2026-07-20 load-audit update** Trace-PD overload 不是 Fixed/Trace good-bad 分裂的统一解释。旧 Q30 Trace-PD decode offered/observed-peak throughput≈`1.00×`、peak Running/Waiting=`47/0`;降到 `0.10 req/s/GPU` 后 TTFT `245.95/685.51 → 228.14/835.38 ms`mean/p90不出现 tail collapseTPOT `13.18/15.39 → 7.91/8.90 ms`。旧 Q235 则是 `3.44×` 明确过载、peak=`116/3`;降到 `0.035 req/s/GPU` 后 TTFT `1141.54/2616.69 → 478.14/1347.75 ms`TPOT `61.89/78.62 → 24.00/28.49 ms`。旧 surface 仍有 `417×/32.6×` mean-TTFT spread否定“所有配置一样差”。八 case baseline 与 claim boundary 见 [`experiments/frontier-eightcase-load-audit-20260720.md`](experiments/frontier-eightcase-load-audit-20260720.md)。
## 一眼看懂
- **Topic / problem** LLM serving 的自动、低成本配置调优AITuner。当前主线问题用 simulator 给部署配置(并行度、批量上限等)排序,什么时候可信?需要补多少真机证据?算上这些成本还划算吗?
- **Central claim** simulator 要能帮助配置调优,必须先满足 scheduler transition 的 liveness/coverage再满足「配置相关残差小于真机 decision margin」前者决定 capacity 是否有定义,后者决定排序是否正确。(ID: C0)
- **当前结论:** 早先 35 个 trace stall 不是 Frontier scheduler liveness failureadapter 为不满 16-token 的 prefix block 错误生成了 cache identityFrontier 又没有 fail-fast。修正为完整 block、使用真实 graph buckets/KV blocks 和 `piecewise`/`KERNEL_ONLY` profile 后Qwen30 Trace-PD 的全部 12 个 cell 完成 129/129 requestFrontier 对 TTFT/TPOT/E2E 的 6 个 argmin 均与三次 fresh-server 真机一致;但绝对 latency 仍高估 4--511×。这只证明一个 MoE Trace-PD surface 的 selection fidelity不能外推到 prefill-only、fixed workload 或 235B。
- **最大 uncertainty / risk** trace 面成功的原因未判——是 decode 耦合的 TP 差异化误差在 trace 状态分布下真的变小,还是只是被宽 TP margin 掩盖。这决定「sim 剪枝可信域」的边界怎么画,也决定加压 trace 是否会失败
- **下一项 critical action** 判别两个失败机制假设(离散阈值转换 vs 闭环 state 漂移):先做零 GPU 的 sim-only jitter 反事实与 Q30 低压 Fixed-PD TPOT 反转分量定位,再按结果决定 dash14 上的加压 Trace-PD / jittered Fixed-PD 真机判别面
- **最大 uncertainty / risk** 根因已收敛,且 overload 已被排除为统一解释但可信域边界仍未画清trace 面的 heterogeneity 是否让 closed-loop state residual 变小,还是当前 success 主要由 capacity/MNS margin 保护?两个降载点只建立 reference-config latency baseline不能证明新负载下全 surface 仍选对
- **下一项 critical action** 不再做无锚点的 jitter 猜测;保持 request shape 不变,在预测的 MoE/MNS knee 两侧做小规模 rate sweep并用少量真机 state/batch anchor 验证 `λR(B)` fixed point。成功标准是同时预测 state-regime、排名与 knee而不只是某个点的 regret
- **停止条件:** T1 出 verdict 且成本账本建立后pass 且摊销论证成立 → 转向「sim 剪枝 + 真机终选」的 hybrid 机制设计fail → 转入失败机制归因;两条路都无 insight 增量 → 收敛写作。
## 核心概念
@@ -39,7 +43,7 @@
- **Hypothesis机制active** 误差机制是 action-conditioned residual——执行状态的转移并行拓扑、kernel family、graph mode、batch 组成)使按算子 profile 的组合预测跨配置不可复合;残差大于 margin 时排序失败。(ID: H2supported已细化)
- **Supporting** 三个 TP 档的端到端校准系数为 0.72/0.47/0.35残差确实随配置剧烈变化235B 的批量上限交互预测错误但被 2× margin 容忍30B prefill-only 在低负载近似对齐、饱和后按 TP 反向放大,最终 τ-b=1。
- **细化2026-07-20 统一普查):** 决策相关的残差分量集中在 TP 轴且由 decode 状态耦合产生——prefill-only 面的绝对 scale 仅 0.961.37× 且 measured collective 即可清除其 TP 反序,而含 decode 的面 scale 4.3130×、全部 material failure 都由 TP/mixed pair 决定。「residual>margin」只是必要条件失败还需要残差对准 winner-deciding pair。
- **下一步:** 判别两个 competing 机制假设——离散阈值转换admission cap/profile cliff 把均匀高估转成差异化误差;打破 fixed workload 的同步性应恢复排序vs 闭环 state 漂移打破同步性也救不了。最便宜路径sim-only jitter 反事实 + Q30 低压 Fixed-PD TPOT 反转的分量定位(唯一无机制解释的 material failure
- **机制 verdict2026-07-20** closed-loop state drift 是根因,离散阈值是其放大器而非 competing explanation。Q30 低压 exact ledger 显示同 state 的 TP 方向正确,但 TP4 被模拟 residence 反馈推到 batch 3--4MoE step 增长后反序Q30 高压进一步跨过 MNS admission capQ235 换成 exact real composition 后排序翻正。下一步从“找根因”转为测量 state-regime/knee 的可信边界
- **Subclaim** 成本论证只有在摊销前提下成立。(ID: C3)
- **Hypothesisactive** 每个 model×硬件×runtime 的一次性对齐成本,摊销到大配置面、频繁重调(引擎版本 churn 的频率证据见 claim map或禁止在线实验的场景后低于重复真机调优。(ID: H3untested——分母已实测分子未入账)
- **下一步:** 建 cost ledger见「下一步」
@@ -48,7 +52,7 @@
- **Question** 生产 trace 忠实回放prefix 打开、原始到达时间与会话结构best-effort Frontier 能否满足 low-regret gate
- **为什么现在做:** 这是 H1 的判决实验;所有已完成的机制分解都在人工 workload 上,不能替代这个 verdict。
- **当前状态:** Trace-PD 的 graph-aligned surface 已通过 selection gate但绝对 latency 不通过 calibration。现在以不共享结果的 Fixed-PD、Trace-P、Fixed-P surface 检验它的泛化边界
- **当前状态:** Trace-PD 的 graph-aligned surface 已通过原负载 selection gate但绝对 latency 不通过 calibrationFixed-PD 的 failure 已定位为 closed-loop state drift。两个降载 Trace-PD anchor 已通过完成率/admission/backlog gate下一步需要 full surface rate sweep 才能检验 ranking 是否跨 load regime 保持
- **Result → decision** 若其它 surface 排序失败,保留 Trace-PD success 为条件化 envelope并按 fixed/trace/prefill/decode 的差异定位 state composition若都通过才扩大到 Q235 或寻找 simulator 已解决范围之外的新问题。
- **Experiment card** [`../runs/frontier-fidelity-envelope-v1/experiment-card.md`](../runs/frontier-fidelity-envelope-v1/experiment-card.md)
@@ -56,19 +60,19 @@
- **E1否证「prefill-only 是充分 easy condition」支持 H2** 30B BF16、去掉 decode/prefix/混合 batch 后,真机最优是 TP48 vs 7 req/s/GPUsimulator 却把 TP4 排最差6 vs 8top set 无交集regret 12.5%,τ-b=1。产物`../runs/frontier-phase-factorial-v0/results/final/`dash012.07 H20-GPUh
- **E2统一机制普查material failure 全部由 decode 耦合的 TP 差异化误差决定,支持 H2 细化):** 对 14 个 frozen surface、70 个 case×objective 的方向化分解显示23 个 material failure 中 21 个由 TP/mixed pair 决定(仅 2 个 6.2% 边缘 mns-axis case、trace 面 TP 反序为零measured collectiveA1把 Qwen235 两个 prefill-only 面的 TP 反序清零trace-PO p90 regret 21.2%→0.3%)但对 Fixed-PD 的 33% 无效「residual>margin」仅为失败的必要条件。产物[`../runs/frontier-split-rootcause-v0/results/`](../runs/frontier-split-rootcause-v0/results/decomposition.md)(实验 card[`experiments/frontier-split-rootcause-s0-20260720.md`](experiments/frontier-split-rootcause-s0-20260720.md))。
- **E3selection 与 calibration 分开):** 用完整 block projection、`piecewise` 和 graph-compatible KERNEL_ONLY profile 后Qwen30 Trace-PD 的 12/12 sim cells 完成6 个 mean/p90 latency objective 的 argmin 都与真机一致;但 sim/real latency ratio 仍为 4--511×。产物[`../runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md`](../runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md)。
- **E3closed-loop state 是 Fixed-PD 根因,而非同 state predictor 反序):** Q30 低压相同 batch=1 state 下 TP4 比 TP1 快约 1.00 ms/step但 TP4 own state 的 time-weighted batch=3.0437,使 step 增加 9.8197 ms其中 MoE +8.9297 ms并反序Q235 用 exact real composition 重放也把 TP8TP4 从错向 20.07 ms 翻为正确 +10.90 ms。Q30 高压再由 MNS cap 将同族 state/residence 误差放大成约 27 s 排队。产物:[`experiments/frontier-split-rootcause-s1-20260720.md`](experiments/frontier-split-rootcause-s1-20260720.md)。
## 下一步(最多 3 项)
- [ ] **完成 Qwen30 remaining matrixdirect progress** Fixed-PD、Trace-P、Fixed-P 分别冻结 Frontier/真机 12-cell surfaceP-only 的 TPOT 一律为 N/A
- [ ] **画可信域边界:** 固定 request shape在预测的 MoE/MNS knee 两侧做最小 rate sweep只在判别点补真机 batch/state anchor验证 `B≈min(MNS, λR(B))` 是否同时解释 state 与 ranking
- [ ] **Q235 portability gate** 先验证 vLLM0.20 TP4/TP8 FP8 runtime 和 deadc4a profile provenance再决定是否允许其 Fixed-P sweep。
- [ ] **建 cost ledger** parent H3完成标准 = 每 case 一行profiling GPU-h、补丁工时、校准探测、sim CPU-h与已实测的真机调优成本同表随每个 case 更新。
## Blocker 或 anomaly
- **下一启动已准备:** dash0 8×H20 当前空闲graph-compatible attention/linear/MoE/router kernel-only profile 会先在一 GPU smoke成功后以 3 张 GPU 并行完成 attention shards。完整 replay 仍为 CPU-only
- **当前运行状态:** 八 case load audit 的新增真机 run 已完成;未启动 full-surface rate sweep避免把两个 single-config anchor 外推成 ranking claim。自 2026-07-20 起,本任务只允许使用 `dash1`--`dash4`(每台 8×H20、最多四组并行`dash0` 保留给其他同事,不做 probe、同步或运行
- **Anomaly保留** 235B pilot 中 simulator 把 10/34 个 anchor 误判为不可行——false-infeasible 是 H1 的主要威胁模式T1 分析时须单独报告。
- **勘误(已固定):** 统一平台为 dash0-only早期 dash1 出处是文档错误;fixed-shape pilot 的主 SLOTPOT 40ms无判别力150ms 是事后明示的敏感性分析,不得写成盲选的 primary。
- **平台边界(已更新):** 历史结果仍来自其各自 card 记录的平台,不改写 provenance后续实验平台切换为 `dash1`--`dash4`。跨主机比较前必须跑相同 canary 并量化 host effect。fixed-shape pilot 的主 SLOTPOT 40ms无判别力150ms 是事后明示的敏感性分析,不得写成盲选的 primary。
## Related work