experiment: add workload regime taxonomy
This commit is contained in:
@@ -0,0 +1,160 @@
|
||||
# Frontier workload-regime taxonomy
|
||||
|
||||
- Date: 2026-07-20
|
||||
- Status: proposed; awaiting review before workload generation or GPU runs
|
||||
- Scope: explain when Frontier preserves the real-system config ranking, rather than merely comparing Fixed with Trace
|
||||
|
||||
## Claim under test
|
||||
|
||||
Frontier reliability is controlled by three quantities:
|
||||
|
||||
1. the latency-model residual between simulator and real execution;
|
||||
2. the closed-loop gain from timing to scheduler state (batch, MoE routing, CUDA-graph bucket, MNS occupancy, admission/KV pressure);
|
||||
3. the real decision margin between configurations.
|
||||
|
||||
For a config pair `a,b`, define
|
||||
|
||||
```text
|
||||
D_real(a,b) = log L_real(a) - log L_real(b)
|
||||
delta(a,b) = [log L_sim(a)-log L_real(a)]
|
||||
- [log L_sim(b)-log L_real(b)]
|
||||
slack(a,b) = sign(D_real) * [D_real + delta]
|
||||
```
|
||||
|
||||
`slack < 0` means the simulator reverses the real pairwise ordering. The primary hypothesis is that reversals occur when simulator and real execution land on different sides of a scheduler-state knee, or when the real decision margin is too small to absorb the differential residual. `Fixed` and `Trace` are not themselves the causal classes.
|
||||
|
||||
## Existing evidence motivating the experiment
|
||||
|
||||
- Q30 Trace-PD preserves all six objective winners, but many pairwise residuals oppose the real winner. Its success is therefore often margin protection, not zero residual.
|
||||
- Q235 Trace-PD preserves TTFT/TPOT winners but misses E2E p90 by 6.2%; Trace is not universally safe.
|
||||
- Q30/Q235 Fixed-PD decode objectives show negative minimum signed slack and 13--37% regret.
|
||||
- In Q30 low-load Fixed-PD, Frontier's batch-1 TP ordering is correct, while the closed-loop simulator increases TP4's effective batch and changes the MoE cost enough to reverse the ordering. This identifies a concrete state knee, but does not yet establish a general rule.
|
||||
|
||||
## Workload families
|
||||
|
||||
All comparisons use the same request multiset where applicable, the same total observation window, and the same normalized offered decode load
|
||||
|
||||
```text
|
||||
rho = request_rate * E[output_tokens] / measured_reference_decode_capacity.
|
||||
```
|
||||
|
||||
This avoids equating equal request rates with equal load.
|
||||
|
||||
| ID | Shape / request lengths | Arrival process | Prefix/session state | Isolated effect |
|
||||
|---|---|---|---|---|
|
||||
| W0 | short fixed `2048 -> 128` | uniform | off | known low-residence failure anchor |
|
||||
| W1 | trace-mean fixed ISL/OSL | uniform | off | homogeneous baseline |
|
||||
| W2 | trace-mean fixed ISL/OSL | trace timestamps | off | arrival burst only |
|
||||
| W3 | exact trace ISL/OSL multiset | uniform | off | length heterogeneity only |
|
||||
| W4 | exact trace ISL/OSL multiset | trace timestamps | off | length + burst |
|
||||
| W5 | exact trace prompts/ISL/OSL | uniform | exact prefix/session identity | prefix state without burst |
|
||||
| W6 | exact trace prompts/ISL/OSL | trace timestamps | exact prefix/session identity | full production trace |
|
||||
|
||||
Prefix is intentionally a nested factor: enabling a synthetic prefix graph on fixed identical requests would introduce a different workload rather than isolate production prefix reuse. Therefore this is not presented as a full `2^3` factorial.
|
||||
|
||||
## Load sweep and expected patterns
|
||||
|
||||
Simulator discovery sweep: `rho in {0.05, 0.25, 0.50, 0.90, 1.20}`. The points mean deep low load, light batching, moderate batching, capacity knee, and overload; their request rates are derived independently for every workload family.
|
||||
|
||||
| Pattern | Observable state | Prediction for Frontier |
|
||||
|---|---|---|
|
||||
| P1 singleton-linear | real and sim stay below the first batch/graph knee | works if the batch-1 operator ordering is correct |
|
||||
| P2 knee-straddling | real and sim occupy opposite sides of a batch/MoE/graph/MNS knee | fails systematically; Fixed-PD is the current example |
|
||||
| P3 same-side batched | both systems cross the same knee and remain below admission pressure | works if batch-conditioned operator ordering is correct |
|
||||
| P4 capacity/admission aligned | both systems are governed by the same capacity bottleneck | TTFT/config winner may work despite large absolute error; E2E/MNS can remain fragile |
|
||||
| P5 heterogeneity-smoothed | broad lengths reduce coherent threshold occupancy at matched `rho` | may work; this is a hypothesis, not an established explanation |
|
||||
| P6 burst-sensitive | same request multiset, but transient queue/MNS occupancy differs | mean ranking may work while TTFT/E2E tail ranking fails |
|
||||
| P7 prefix-state-sensitive | hit/eviction and reused-token distributions differ | TTFT ranking fails unless prefix-state transitions are modeled; decode TPOT may remain stable |
|
||||
| P8 decision-boundary | real config margin is comparable to run variance/residual | fragile; an exact winner match is not reliable evidence |
|
||||
|
||||
## Hypotheses and distinguishing tests
|
||||
|
||||
### H1: state-regime hypothesis (primary)
|
||||
|
||||
I believe config-ranking failures occur when the latency residual moves a workload across a scheduler-state knee, because the residual is then amplified into a different batch/resource trajectory. I will verify this by checking whether signed-slack zero crossings co-locate with measured real/simulator state-knee crossings.
|
||||
|
||||
### H2: heterogeneity-smoothing hypothesis
|
||||
|
||||
I believe length heterogeneity can reduce coherent threshold amplification, because requests reach scheduler boundaries at dispersed times. I will verify it with W1 vs W3 and W2 vs W4 at matched `rho`, requiring a smaller real/sim state-distribution gap rather than merely a correct winner.
|
||||
|
||||
### H3: bottleneck/margin-protection alternative
|
||||
|
||||
Trace success may instead be explained entirely by a large real decision margin or a shared capacity bottleneck. This hypothesis wins over H2 if W3/W4 do not reduce state-distribution error after matching load and margin, while ranking correctness remains predicted by margin alone.
|
||||
|
||||
### H4: burst and prefix are independent failure channels
|
||||
|
||||
I believe arrival bursts primarily affect waiting/admission and tail TTFT/E2E, whereas prefix mismatch primarily affects prefill/TTFT state. I will verify this with W1/W2, W3/W4, and W3/W5 paired comparisons.
|
||||
|
||||
## Configuration and model scope
|
||||
|
||||
Discovery uses Qwen30B because its 12-cell `TP x MNS` surface already has simulator and real anchors:
|
||||
|
||||
- TP: `{1, 2, 4}`
|
||||
- MNS: `{8, 32, 64, 128}`
|
||||
- objectives: mean/p90 TTFT, TPOT, E2E
|
||||
|
||||
Qwen235B is a held-out confirmation, not pooled into discovery:
|
||||
|
||||
- existing four feasible TP/MNS configurations;
|
||||
- only the workload/load patterns that discriminate H1--H4 after Q30 converges.
|
||||
|
||||
## Measurements
|
||||
|
||||
End-to-end:
|
||||
|
||||
- completed/failed requests and achieved request/token rate;
|
||||
- TTFT, TPOT, E2E mean/p50/p90/p95;
|
||||
- config regret, pairwise agreement, signed decision slack;
|
||||
- run-to-run winner stability.
|
||||
|
||||
Closed-loop state:
|
||||
|
||||
- prefill/decode batch-size histograms and time-weighted batch;
|
||||
- Running/Waiting distributions and admission delay;
|
||||
- MNS active-token occupancy and KV/context pressure;
|
||||
- CUDA-graph bucket residency and fallback frequency;
|
||||
- prefix hit/reused-token/eviction distributions for W5/W6.
|
||||
|
||||
## Decision rules
|
||||
|
||||
A workload/load region is:
|
||||
|
||||
- **reliable** if regret is at most 5%, pairwise agreement is at least 0.8 at two adjacent load points, and the winner is stable across confirmation trials;
|
||||
- **fragile** if regret is at most 5% but the real margin overlaps run uncertainty, or a small rate/timing perturbation changes the winner;
|
||||
- **failed** if regret exceeds 5% or a decision-critical pair has negative signed slack;
|
||||
- **mechanistically explained by H1** only if the ranking transition co-locates with an observed state-regime transition. Correlation with the Fixed/Trace label is insufficient.
|
||||
|
||||
H2 is supported only if the heterogeneous member of a matched pair reduces state-distribution error and shifts the failure boundary in repeated trials. A correct winner alone does not support smoothing.
|
||||
|
||||
## Execution plan after review
|
||||
|
||||
1. Materialize W0--W6 with one manifest recording request multiset, arrival timestamps, prefix identity, rate contract, and hashes.
|
||||
2. Run the simulator sweep across `rho` and the Q30 surface; emit a per-stage state ledger.
|
||||
3. Select real-machine pilot points only around the predicted knees plus one safe-side control. Use guard configs `TP1/MNS64`, `TP4/MNS8`, and `TP4/MNS64`; add `TP2/MNS32` only if the transition is not bracketed.
|
||||
4. Use only `dash1`, `dash2`, `dash3`, and `dash4`, each verified as an 8×H20 host. `dash0` is excluded from probing, synchronization, and execution. Pin one independent experiment group to each host so at most four groups run in parallel; do not split one trial across hosts.
|
||||
5. Run one pilot trial per selected point. Confirm only hypothesis-discriminating points with three fresh-server trials and rotated order.
|
||||
6. Apply the resulting classifier unchanged to the Q235 held-out cases.
|
||||
|
||||
Provisional four-way allocation after the simulator identifies the discriminating points:
|
||||
|
||||
| Host | Experiment group | Primary contrast |
|
||||
|---|---|---|
|
||||
| dash1 | homogeneous controls | W0/W1 across safe side and first knee |
|
||||
| dash2 | arrival effect | W1 vs W2 and W3 vs W4 |
|
||||
| dash3 | length heterogeneity | W1 vs W3 and W2 vs W4 |
|
||||
| dash4 | prefix/full trace | W3 vs W5 and W4 vs W6 |
|
||||
|
||||
The groups are logical queues, not permanent ownership: if a host probe fails, that host is excluded and its group waits or moves to another permitted idle host. Cross-host latency values are not pooled until a common canary config verifies that host effects are within run uncertainty.
|
||||
|
||||
No GPU run is authorized by this card yet. The review decision is whether the workload decomposition and decision rules are sufficient to implement the materializer and launch Phase 1.
|
||||
|
||||
## Expected figure
|
||||
|
||||
The accompanying mock figure is schematic, not data. Panel A shows the state knee that real and simulator trajectories may cross at different loads. Panel B shows the corresponding minimum signed decision slack; a negative value denotes a ranking reversal. The claim is supported only if measured zero crossings and state knees align across workload families.
|
||||
|
||||
## Risks and controls
|
||||
|
||||
- Equal `rho` does not guarantee equal prefill pressure; report both prefill and decode offered work and stratify if necessary.
|
||||
- Full-trace overload can collapse all configs to similarly poor latency. Such points identify a capacity-limited region but cannot validate fine-grained ranking.
|
||||
- MNS ties and censored/failed requests can create false winners; exclude invalid cells before calculating regret and report the exclusion.
|
||||
- One trace cannot establish generality. The initial result is a mechanism boundary for this trace/model/hardware, followed by held-out Q235 validation.
|
||||
@@ -5,14 +5,18 @@
|
||||
> **2026-07-19 update：** Qwen235 Fixed-PD 的错误排序在 exact real state composition 下已经翻正，主因是 simulator closed-loop batch state，而不是 collective。Qwen30 Fixed-PD 的 56--58% TTFT regret 也已定位：Frontier 将 decode service time 高估 4--8×，使 TP4 的 modeled concurrency 越过 MNS admission cap并产生虚假排队；去掉该等待后 Frontier 与真机都判定 TP4 topology 更快。详见 [`experiments/qwen30-fixed-pd-ttft-admission-diagnosis-20260719.md`](experiments/qwen30-fixed-pd-ttft-admission-diagnosis-20260719.md)。
|
||||
>
|
||||
> **2026-07-20 update：** 对全部 14 个 frozen case surface（70 个 case×objective）做了统一的 margin-vs-residual 分解与方向化机制普查（[`experiments/frontier-split-rootcause-s0-20260720.md`](experiments/frontier-split-rootcause-s0-20260720.md)）。三个要点：(1) 「residual 超过 margin」是失败的必要条件但远非充分——good/bad 分裂不能用无方向误差量解释；(2) 23 个 material failure 的 winner-deciding pair 中 21 个落在 TP 轴或 mixed（其余 2 个是 6.2% regret 的边缘 mns-axis case），trace 面的 TP 反序为零；A1 measured collective 把 Qwen235 两个 prefill-only 面的 TP 反序清零（trace-PO p90 regret 21.2%→0.3%）却对 Fixed-PD 完全无效——prefill 路径的 TP 差异化误差源是 collective profile（可修），decode 耦合的 TP 差异化误差是当前所有 material failure 的载体；(3) 「Fixed-PD 失败因为高压」被否证：失败 Fixed-PD 的真机 in-flight（14.05）低于全对的 Trace-PD（38.69），且低压 Fixed-PD 同样失败、失败 objective 随负载切换。另有次要缺陷：14 个 winner 错位来自 simulator 对 MNS 逐位不敏感的精确 tie。
|
||||
>
|
||||
> **2026-07-20 root-cause update：** Q30 低压 Fixed-PD 的 exact stage ledger 关闭了最后一个未解释的 material failure。相同 batch=1 state 下 Frontier full predictor 给 TP4 `18.3515 ms/step`、TP1 `19.3506 ms/step`，方向正确；但 per-GPU 固定到达率使 cluster arrival 随 TP 增长，叠加 decode residence 高估后，TP4 在 simulator 内自激到 time-weighted batch `3.0437`（96.13% decode 时间 batch≥3），own-state step 变为 `28.1712 ms`。其中相对 batch=1 的 `+9.8197 ms` 有 `+8.9297 ms` 来自 batch-conditioned MoE，collective 仅 `+0.0121 ms`。因此 Fixed-PD 的根因不是“固定 workload”或“高压力”本身，而是 **execution-time residual 进入离散事件时钟后改变 future scheduler state；该 state 再通过 MoE/profile/graph 或 MNS admission 非线性放大，形成 action-dependent signed residual 并穿过 decision margin**。Q30 低压是平滑 state-feedback，Q30 高压是跨 MNS cap 的 threshold amplification，Q235 是 composition drift；三者为同一闭环机制族。
|
||||
>
|
||||
> **2026-07-20 load-audit update：** Trace-PD overload 不是 Fixed/Trace good-bad 分裂的统一解释。旧 Q30 Trace-PD decode offered/observed-peak throughput≈`1.00×`、peak Running/Waiting=`47/0`；降到 `0.10 req/s/GPU` 后 TTFT `245.95/685.51 → 228.14/835.38 ms`（mean/p90，不出现 tail collapse），TPOT `13.18/15.39 → 7.91/8.90 ms`。旧 Q235 则是 `3.44×` 明确过载、peak=`116/3`；降到 `0.035 req/s/GPU` 后 TTFT `1141.54/2616.69 → 478.14/1347.75 ms`，TPOT `61.89/78.62 → 24.00/28.49 ms`。旧 surface 仍有 `417×/32.6×` mean-TTFT spread，否定“所有配置一样差”。八 case baseline 与 claim boundary 见 [`experiments/frontier-eightcase-load-audit-20260720.md`](experiments/frontier-eightcase-load-audit-20260720.md)。
|
||||
|
||||
## 一眼看懂
|
||||
|
||||
- **Topic / problem：** LLM serving 的自动、低成本配置调优（AITuner）。当前主线问题：用 simulator 给部署配置（并行度、批量上限等）排序，什么时候可信？需要补多少真机证据？算上这些成本还划算吗？
|
||||
- **Central claim：** simulator 要能帮助配置调优，必须先满足 scheduler transition 的 liveness/coverage，再满足「配置相关残差小于真机 decision margin」；前者决定 capacity 是否有定义，后者决定排序是否正确。(ID: C0)
|
||||
- **当前结论：** 早先 35 个 trace stall 不是 Frontier scheduler liveness failure：adapter 为不满 16-token 的 prefix block 错误生成了 cache identity，Frontier 又没有 fail-fast。修正为完整 block、使用真实 graph buckets/KV blocks 和 `piecewise`/`KERNEL_ONLY` profile 后，Qwen30 Trace-PD 的全部 12 个 cell 完成 129/129 request，Frontier 对 TTFT/TPOT/E2E 的 6 个 argmin 均与三次 fresh-server 真机一致；但绝对 latency 仍高估 4--511×。这只证明一个 MoE Trace-PD surface 的 selection fidelity，不能外推到 prefill-only、fixed workload 或 235B。
|
||||
- **最大 uncertainty / risk：** trace 面成功的原因未判——是 decode 耦合的 TP 差异化误差在 trace 状态分布下真的变小，还是只是被宽 TP margin 掩盖。这决定「sim 剪枝可信域」的边界怎么画，也决定加压 trace 是否会失败。
|
||||
- **下一项 critical action：** 判别两个失败机制假设（离散阈值转换 vs 闭环 state 漂移）：先做零 GPU 的 sim-only jitter 反事实与 Q30 低压 Fixed-PD TPOT 反转分量定位，再按结果决定 dash1–4 上的加压 Trace-PD / jittered Fixed-PD 真机判别面。
|
||||
- **最大 uncertainty / risk：** 根因已收敛，且 overload 已被排除为统一解释，但可信域边界仍未画清：trace 面的 heterogeneity 是否让 closed-loop state residual 变小，还是当前 success 主要由 capacity/MNS margin 保护？两个降载点只建立 reference-config latency baseline，不能证明新负载下全 surface 仍选对。
|
||||
- **下一项 critical action：** 不再做无锚点的 jitter 猜测；保持 request shape 不变，在预测的 MoE/MNS knee 两侧做小规模 rate sweep，并用少量真机 state/batch anchor 验证 `λR(B)` fixed point。成功标准是同时预测 state-regime、排名与 knee，而不只是某个点的 regret。
|
||||
- **停止条件：** T1 出 verdict 且成本账本建立后：pass 且摊销论证成立 → 转向「sim 剪枝 + 真机终选」的 hybrid 机制设计；fail → 转入失败机制归因；两条路都无 insight 增量 → 收敛写作。
|
||||
|
||||
## 核心概念
|
||||
@@ -39,7 +43,7 @@
|
||||
- **Hypothesis（机制，active）：** 误差机制是 action-conditioned residual——执行状态的转移（并行拓扑、kernel family、graph mode、batch 组成）使按算子 profile 的组合预测跨配置不可复合；残差大于 margin 时排序失败。(ID: H2；supported，已细化)
|
||||
- **Supporting：** 三个 TP 档的端到端校准系数为 0.72/0.47/0.35，残差确实随配置剧烈变化；235B 的批量上限交互预测错误但被 2× margin 容忍；30B prefill-only 在低负载近似对齐、饱和后按 TP 反向放大，最终 τ-b=−1。
|
||||
- **细化（2026-07-20 统一普查）：** 决策相关的残差分量集中在 TP 轴且由 decode 状态耦合产生——prefill-only 面的绝对 scale 仅 0.96–1.37× 且 measured collective 即可清除其 TP 反序，而含 decode 的面 scale 4.3–130×、全部 material failure 都由 TP/mixed pair 决定。「residual>margin」只是必要条件；失败还需要残差对准 winner-deciding pair。
|
||||
- **下一步：** 判别两个 competing 机制假设——离散阈值转换（admission cap/profile cliff 把均匀高估转成差异化误差；打破 fixed workload 的同步性应恢复排序）vs 闭环 state 漂移（打破同步性也救不了）。最便宜路径：sim-only jitter 反事实 + Q30 低压 Fixed-PD TPOT 反转的分量定位（唯一无机制解释的 material failure）。
|
||||
- **机制 verdict（2026-07-20）：** closed-loop state drift 是根因，离散阈值是其放大器而非 competing explanation。Q30 低压 exact ledger 显示同 state 的 TP 方向正确，但 TP4 被模拟 residence 反馈推到 batch 3--4，MoE step 增长后反序；Q30 高压进一步跨过 MNS admission cap；Q235 换成 exact real composition 后排序翻正。下一步从“找根因”转为测量 state-regime/knee 的可信边界。
|
||||
- **Subclaim：** 成本论证只有在摊销前提下成立。(ID: C3)
|
||||
- **Hypothesis（active）：** 每个 model×硬件×runtime 的一次性对齐成本，摊销到大配置面、频繁重调（引擎版本 churn 的频率证据见 claim map）或禁止在线实验的场景后，低于重复真机调优。(ID: H3；untested——分母已实测，分子未入账)
|
||||
- **下一步：** 建 cost ledger（见「下一步」）。
|
||||
@@ -48,7 +52,7 @@
|
||||
|
||||
- **Question：** 生产 trace 忠实回放（prefix 打开、原始到达时间与会话结构）下，best-effort Frontier 能否满足 low-regret gate？
|
||||
- **为什么现在做：** 这是 H1 的判决实验；所有已完成的机制分解都在人工 workload 上，不能替代这个 verdict。
|
||||
- **当前状态：** Trace-PD 的 graph-aligned surface 已通过 selection gate，但绝对 latency 不通过 calibration。现在以不共享结果的 Fixed-PD、Trace-P、Fixed-P surface 检验它的泛化边界。
|
||||
- **当前状态：** Trace-PD 的 graph-aligned surface 已通过原负载 selection gate，但绝对 latency 不通过 calibration；Fixed-PD 的 failure 已定位为 closed-loop state drift。两个降载 Trace-PD anchor 已通过完成率/admission/backlog gate，下一步需要 full surface rate sweep 才能检验 ranking 是否跨 load regime 保持。
|
||||
- **Result → decision：** 若其它 surface 排序失败，保留 Trace-PD success 为条件化 envelope，并按 fixed/trace/prefill/decode 的差异定位 state composition；若都通过，才扩大到 Q235 或寻找 simulator 已解决范围之外的新问题。
|
||||
- **Experiment card：** [`../runs/frontier-fidelity-envelope-v1/experiment-card.md`](../runs/frontier-fidelity-envelope-v1/experiment-card.md)
|
||||
|
||||
@@ -56,19 +60,19 @@
|
||||
|
||||
- **E1（否证「prefill-only 是充分 easy condition」，支持 H2）：** 30B BF16、去掉 decode/prefix/混合 batch 后，真机最优是 TP4（8 vs 7 req/s/GPU），simulator 却把 TP4 排最差（6 vs 8）：top set 无交集，regret 12.5%，τ-b=−1。产物：`../runs/frontier-phase-factorial-v0/results/final/`（dash0，12.07 H20-GPUh）。
|
||||
- **E2（统一机制普查：material failure 全部由 decode 耦合的 TP 差异化误差决定，支持 H2 细化）：** 对 14 个 frozen surface、70 个 case×objective 的方向化分解显示：23 个 material failure 中 21 个由 TP/mixed pair 决定（仅 2 个 6.2% 边缘 mns-axis case）、trace 面 TP 反序为零；measured collective（A1）把 Qwen235 两个 prefill-only 面的 TP 反序清零（trace-PO p90 regret 21.2%→0.3%）但对 Fixed-PD 的 33% 无效；「residual>margin」仅为失败的必要条件。产物：[`../runs/frontier-split-rootcause-v0/results/`](../runs/frontier-split-rootcause-v0/results/decomposition.md)（实验 card：[`experiments/frontier-split-rootcause-s0-20260720.md`](experiments/frontier-split-rootcause-s0-20260720.md)）。
|
||||
- **E3（selection 与 calibration 分开）：** 用完整 block projection、`piecewise` 和 graph-compatible KERNEL_ONLY profile 后，Qwen30 Trace-PD 的 12/12 sim cells 完成，6 个 mean/p90 latency objective 的 argmin 都与真机一致；但 sim/real latency ratio 仍为 4--511×。产物：[`../runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md`](../runs/frontier-fidelity-envelope-v1/graph-piecewise-experiment-card.md)。
|
||||
- **E3（closed-loop state 是 Fixed-PD 根因，而非同 state predictor 反序）：** Q30 低压相同 batch=1 state 下 TP4 比 TP1 快约 1.00 ms/step，但 TP4 own state 的 time-weighted batch=3.0437，使 step 增加 9.8197 ms（其中 MoE +8.9297 ms）并反序；Q235 用 exact real composition 重放也把 TP8−TP4 从错向 −20.07 ms 翻为正确 +10.90 ms。Q30 高压再由 MNS cap 将同族 state/residence 误差放大成约 27 s 排队。产物：[`experiments/frontier-split-rootcause-s1-20260720.md`](experiments/frontier-split-rootcause-s1-20260720.md)。
|
||||
|
||||
## 下一步（最多 3 项）
|
||||
|
||||
- [ ] **完成 Qwen30 remaining matrix（direct progress）：** Fixed-PD、Trace-P、Fixed-P 分别冻结 Frontier/真机 12-cell surface；P-only 的 TPOT 一律为 N/A。
|
||||
- [ ] **画可信域边界：** 固定 request shape，在预测的 MoE/MNS knee 两侧做最小 rate sweep；只在判别点补真机 batch/state anchor，验证 `B≈min(MNS, λR(B))` 是否同时解释 state 与 ranking。
|
||||
- [ ] **Q235 portability gate：** 先验证 vLLM0.20 TP4/TP8 FP8 runtime 和 deadc4a profile provenance，再决定是否允许其 Fixed-P sweep。
|
||||
- [ ] **建 cost ledger：** parent H3；完成标准 = 每 case 一行（profiling GPU-h、补丁工时、校准探测、sim CPU-h），与已实测的真机调优成本同表，随每个 case 更新。
|
||||
|
||||
## Blocker 或 anomaly
|
||||
|
||||
- **下一启动已准备：** dash0 8×H20 当前空闲；graph-compatible attention/linear/MoE/router kernel-only profile 会先在一 GPU smoke，成功后以 3 张 GPU 并行完成 attention shards。完整 replay 仍为 CPU-only。
|
||||
- **当前运行状态：** 八 case load audit 的新增真机 run 已完成；未启动 full-surface rate sweep，避免把两个 single-config anchor 外推成 ranking claim。自 2026-07-20 起，本任务只允许使用 `dash1`--`dash4`（每台 8×H20、最多四组并行）；`dash0` 保留给其他同事，不做 probe、同步或运行。
|
||||
- **Anomaly（保留）：** 235B pilot 中 simulator 把 10/34 个 anchor 误判为不可行——false-infeasible 是 H1 的主要威胁模式，T1 分析时须单独报告。
|
||||
- **勘误（已固定）：** 统一平台为 dash0-only，早期 dash1 出处是文档错误；fixed-shape pilot 的主 SLO（TPOT 40ms）无判别力，150ms 是事后明示的敏感性分析，不得写成盲选的 primary。
|
||||
- **平台边界（已更新）：** 历史结果仍来自其各自 card 记录的平台，不改写 provenance；后续实验平台切换为 `dash1`--`dash4`。跨主机比较前必须跑相同 canary 并量化 host effect。fixed-shape pilot 的主 SLO（TPOT 40ms）无判别力，150ms 是事后明示的敏感性分析，不得写成盲选的 primary。
|
||||
|
||||
## Related work
|
||||
|
||||
|
||||
14
AGENTS.md
14
AGENTS.md
@@ -1,10 +1,11 @@
|
||||
# Project Operating Notes
|
||||
|
||||
## Remote experiment host
|
||||
## Remote experiment hosts
|
||||
|
||||
- Default experiment machine: `dash0`.
|
||||
- Hardware expectation: 8 NVIDIA H20 GPUs.
|
||||
- SSH check: use `ssh dash0` before scheduling or debugging remote runs.
|
||||
- Experiment machines: `dash1`, `dash2`, `dash3`, and `dash4`.
|
||||
- Do not use or probe `dash0`; it is reserved for other users.
|
||||
- Hardware expectation: 8 NVIDIA H20 GPUs per host.
|
||||
- Before scheduling, probe only `dash1`--`dash4` and confirm all eight GPUs are idle and healthy.
|
||||
- Remote project path: `/home/admin/cpfs/wjh/aituner/aituner`.
|
||||
- If remote downloads are slow or fail, start the proxy from the remote `wjh`
|
||||
home directory with `./auto_proxy.sh`, then run downloads in a shell where
|
||||
@@ -13,7 +14,8 @@
|
||||
|
||||
## Local/remote sync workflow
|
||||
|
||||
- Treat this local repository and the `dash0` repository as the same project checkout.
|
||||
- Treat this local repository and the `dash1`--`dash4` repositories as the same project checkout.
|
||||
- Synchronize code through Git using `commit`, `push`, and `pull`.
|
||||
- For remote experiments, commit local changes, push to `origin`, then pull on `dash0` in `/home/admin/cpfs/wjh/aituner/aituner` before running.
|
||||
- For remote experiments, commit local changes, push to `origin`, then pull on each assigned host in `/home/admin/cpfs/wjh/aituner/aituner` before running.
|
||||
- Up to four independent 8-GPU experiment groups may run in parallel, one group per host; pin every job explicitly to one of `dash1`--`dash4`.
|
||||
- Do not ask for the remote host or project path again unless the user explicitly changes them.
|
||||
|
||||
@@ -0,0 +1,376 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Materialize controlled Qwen30 workload families from one exact trace cohort."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
|
||||
FIELDS = (
|
||||
"arrived_at",
|
||||
"num_prefill_tokens",
|
||||
"num_decode_tokens",
|
||||
"session_id",
|
||||
"block_hash_ids",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Family:
|
||||
name: str
|
||||
shape: str
|
||||
arrival: str
|
||||
prefix: bool
|
||||
|
||||
|
||||
FAMILIES = (
|
||||
Family("w0-short-fixed-uniform-none", "short-fixed", "uniform", False),
|
||||
Family("w1-mean-fixed-uniform-none", "mean-fixed", "uniform", False),
|
||||
Family("w2-mean-fixed-trace-none", "mean-fixed", "trace", False),
|
||||
Family("w3-heterogeneous-uniform-none", "heterogeneous", "uniform", False),
|
||||
Family("w4-heterogeneous-trace-none", "heterogeneous", "trace", False),
|
||||
Family("w5-heterogeneous-uniform-prefix", "heterogeneous", "uniform", True),
|
||||
Family("w6-heterogeneous-trace-prefix", "heterogeneous", "trace", True),
|
||||
)
|
||||
|
||||
|
||||
def parse_args() -> argparse.Namespace:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--source-public", type=Path, required=True)
|
||||
parser.add_argument("--source-private", type=Path, required=True)
|
||||
parser.add_argument("--model", type=Path, required=True)
|
||||
parser.add_argument("--output-root", type=Path, required=True)
|
||||
parser.add_argument(
|
||||
"--reference-decode-tokens-per-second", type=float, default=3064.0
|
||||
)
|
||||
parser.add_argument(
|
||||
"--rho", type=float, action="append", default=None,
|
||||
help="Normalized decode offered load; repeat for multiple levels.",
|
||||
)
|
||||
parser.add_argument("--requests", type=int)
|
||||
return parser.parse_args()
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as source:
|
||||
for chunk in iter(lambda: source.read(1 << 20), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def vector_sha256(rows: Iterable[dict[str, Any]]) -> str:
|
||||
digest = hashlib.sha256()
|
||||
for row in rows:
|
||||
digest.update(
|
||||
json.dumps(
|
||||
[
|
||||
row["source_index"],
|
||||
row["arrived_at"],
|
||||
row["input_length"],
|
||||
row["output_length"],
|
||||
row["session_id"],
|
||||
row["runtime_block_ids"],
|
||||
],
|
||||
separators=(",", ":"),
|
||||
).encode()
|
||||
)
|
||||
digest.update(b"\n")
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def load_source(public_path: Path, private_path: Path) -> list[dict[str, Any]]:
|
||||
with public_path.open(newline="") as source:
|
||||
public_rows = list(csv.DictReader(source))
|
||||
private_rows = [json.loads(line) for line in private_path.open() if line.strip()]
|
||||
if not public_rows or len(public_rows) != len(private_rows):
|
||||
raise ValueError("source public/private request count mismatch")
|
||||
|
||||
rows = []
|
||||
for index, (public, private) in enumerate(
|
||||
zip(public_rows, private_rows, strict=True)
|
||||
):
|
||||
input_length = int(public["num_prefill_tokens"])
|
||||
output_length = int(public["num_decode_tokens"])
|
||||
if input_length != int(private["input_length"]):
|
||||
raise ValueError(f"source input-length drift at row {index}")
|
||||
if output_length != int(private["output_length"]):
|
||||
raise ValueError(f"source output-length drift at row {index}")
|
||||
runtime_ids = [
|
||||
int(value) for value in public["block_hash_ids"].split("|") if value
|
||||
]
|
||||
if len(runtime_ids) != input_length // 16:
|
||||
raise ValueError(f"incomplete prefix-block projection at row {index}")
|
||||
rows.append(
|
||||
{
|
||||
"source_index": int(private["source_index"]),
|
||||
"source_arrived_at": float(public["arrived_at"]),
|
||||
"input_length": input_length,
|
||||
"output_length": output_length,
|
||||
"session_id": int(private["session_id"]),
|
||||
"runtime_block_ids": runtime_ids,
|
||||
"body": copy.deepcopy(private["body"]),
|
||||
}
|
||||
)
|
||||
if any(
|
||||
right["source_arrived_at"] < left["source_arrived_at"]
|
||||
for left, right in zip(rows, rows[1:])
|
||||
):
|
||||
raise ValueError("source arrival order drift")
|
||||
return rows
|
||||
|
||||
|
||||
def arrivals_for(
|
||||
source_rows: list[dict[str, Any]], arrival: str, target_rate: float
|
||||
) -> list[float]:
|
||||
if target_rate <= 0 or not math.isfinite(target_rate):
|
||||
raise ValueError("target request rate must be finite and positive")
|
||||
if arrival == "uniform":
|
||||
return [index / target_rate for index in range(len(source_rows))]
|
||||
if arrival != "trace":
|
||||
raise ValueError(f"unknown arrival family: {arrival}")
|
||||
if len(source_rows) < 2:
|
||||
raise ValueError("trace arrivals require at least two requests")
|
||||
first = source_rows[0]["source_arrived_at"]
|
||||
relative = [row["source_arrived_at"] - first for row in source_rows]
|
||||
if relative[-1] <= 0:
|
||||
raise ValueError("trace arrival window must be positive")
|
||||
source_rate = (len(relative) - 1) / relative[-1]
|
||||
scale = source_rate / target_rate
|
||||
return [value * scale for value in relative]
|
||||
|
||||
|
||||
def non_special_token_ids(model: Path, requests: int) -> list[int]:
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
|
||||
special = set(tokenizer.all_special_ids)
|
||||
candidates = [token for token in range(tokenizer.vocab_size) if token not in special]
|
||||
if len(candidates) < requests + 1:
|
||||
raise ValueError("tokenizer lacks enough non-special token IDs")
|
||||
return candidates[: requests + 1]
|
||||
|
||||
|
||||
def build_family_rows(
|
||||
source_rows: list[dict[str, Any]],
|
||||
family: Family,
|
||||
*,
|
||||
target_rate: float,
|
||||
fixed_token_ids: list[int],
|
||||
) -> list[dict[str, Any]]:
|
||||
arrivals = arrivals_for(source_rows, family.arrival, target_rate)
|
||||
mean_input = round(sum(row["input_length"] for row in source_rows) / len(source_rows))
|
||||
mean_output = round(sum(row["output_length"] for row in source_rows) / len(source_rows))
|
||||
if family.shape == "short-fixed":
|
||||
fixed_shape = (2048, 128)
|
||||
elif family.shape == "mean-fixed":
|
||||
fixed_shape = (mean_input, mean_output)
|
||||
elif family.shape == "heterogeneous":
|
||||
fixed_shape = None
|
||||
else:
|
||||
raise ValueError(f"unknown shape family: {family.shape}")
|
||||
|
||||
rows = []
|
||||
for index, (source, arrived_at) in enumerate(
|
||||
zip(source_rows, arrivals, strict=True)
|
||||
):
|
||||
if fixed_shape is None:
|
||||
input_length = source["input_length"]
|
||||
output_length = source["output_length"]
|
||||
body = copy.deepcopy(source["body"])
|
||||
body.update(
|
||||
{
|
||||
"min_tokens": output_length,
|
||||
"max_tokens": output_length,
|
||||
"ignore_eos": True,
|
||||
}
|
||||
)
|
||||
else:
|
||||
input_length, output_length = fixed_shape
|
||||
body = copy.deepcopy(source["body"])
|
||||
body.update(
|
||||
{
|
||||
"prompt": [
|
||||
fixed_token_ids[index + 1],
|
||||
*([fixed_token_ids[0]] * (input_length - 1)),
|
||||
],
|
||||
"min_tokens": output_length,
|
||||
"max_tokens": output_length,
|
||||
"ignore_eos": True,
|
||||
}
|
||||
)
|
||||
if input_length + output_length > 40960:
|
||||
raise ValueError(f"shape exceeds model limit at row {index}")
|
||||
rows.append(
|
||||
{
|
||||
"source_index": source["source_index"],
|
||||
"arrived_at": arrived_at,
|
||||
"input_length": input_length,
|
||||
"output_length": output_length,
|
||||
"session_id": source["session_id"] if family.prefix else index,
|
||||
"runtime_block_ids": (
|
||||
list(source["runtime_block_ids"]) if family.prefix else []
|
||||
),
|
||||
"body": body,
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def write_case(
|
||||
root: Path,
|
||||
rows: list[dict[str, Any]],
|
||||
*,
|
||||
family: Family,
|
||||
rho: float,
|
||||
target_rate: float,
|
||||
reference_capacity: float,
|
||||
source_public: Path,
|
||||
source_private: Path,
|
||||
) -> dict[str, Any]:
|
||||
public_root = root / "public"
|
||||
private_root = root / "private"
|
||||
public_root.mkdir(parents=True, exist_ok=True)
|
||||
private_root.mkdir(parents=True, exist_ok=True)
|
||||
public_path = public_root / "frontier.csv"
|
||||
private_path = private_root / "real_requests.jsonl"
|
||||
|
||||
with public_path.open("w", newline="") as output:
|
||||
writer = csv.DictWriter(output, fieldnames=FIELDS, lineterminator="\n")
|
||||
writer.writeheader()
|
||||
for row in rows:
|
||||
writer.writerow(
|
||||
{
|
||||
"arrived_at": f"{row['arrived_at']:.12f}",
|
||||
"num_prefill_tokens": row["input_length"],
|
||||
"num_decode_tokens": row["output_length"],
|
||||
"session_id": row["session_id"],
|
||||
"block_hash_ids": "|".join(
|
||||
str(value) for value in row["runtime_block_ids"]
|
||||
),
|
||||
}
|
||||
)
|
||||
with private_path.open("w") as output:
|
||||
for row in rows:
|
||||
output.write(json.dumps(row, separators=(",", ":")) + "\n")
|
||||
|
||||
input_lengths = [row["input_length"] for row in rows]
|
||||
output_lengths = [row["output_length"] for row in rows]
|
||||
arrivals = [row["arrived_at"] for row in rows]
|
||||
empirical_rate = (len(rows) - 1) / (arrivals[-1] - arrivals[0])
|
||||
manifest = {
|
||||
"schema": "frontier-workload-regime-v1",
|
||||
"family": family.name,
|
||||
"shape_contract": family.shape,
|
||||
"arrival_contract": family.arrival,
|
||||
"prefix_caching": family.prefix,
|
||||
"requests": len(rows),
|
||||
"rho": rho,
|
||||
"reference_decode_tokens_per_second": reference_capacity,
|
||||
"global_offered_request_rate": target_rate,
|
||||
"empirical_interarrival_rate": empirical_rate,
|
||||
"decode_offered_tokens_per_second": target_rate
|
||||
* sum(output_lengths)
|
||||
/ len(output_lengths),
|
||||
"input_tokens": {
|
||||
"mean": sum(input_lengths) / len(input_lengths),
|
||||
"min": min(input_lengths),
|
||||
"max": max(input_lengths),
|
||||
},
|
||||
"output_tokens": {
|
||||
"mean": sum(output_lengths) / len(output_lengths),
|
||||
"min": min(output_lengths),
|
||||
"max": max(output_lengths),
|
||||
},
|
||||
"first_arrival_s": arrivals[0],
|
||||
"last_arrival_s": arrivals[-1],
|
||||
"source_public": str(source_public.resolve()),
|
||||
"source_public_sha256": sha256(source_public),
|
||||
"source_private": str(source_private.resolve()),
|
||||
"source_private_sha256": sha256(source_private),
|
||||
"public_csv": str(public_path.resolve()),
|
||||
"public_csv_sha256": sha256(public_path),
|
||||
"private_jsonl": str(private_path.resolve()),
|
||||
"private_jsonl_sha256": sha256(private_path),
|
||||
"row_vector_sha256": vector_sha256(rows),
|
||||
}
|
||||
manifest_path = public_root / "manifest.json"
|
||||
manifest_path.write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
|
||||
return manifest
|
||||
|
||||
|
||||
def main() -> None:
|
||||
args = parse_args()
|
||||
rhos = args.rho or [0.05, 0.25, 0.50, 0.90, 1.20]
|
||||
if any(not math.isfinite(rho) or rho <= 0 for rho in rhos):
|
||||
raise ValueError("rho values must be finite and positive")
|
||||
if args.reference_decode_tokens_per_second <= 0:
|
||||
raise ValueError("reference decode capacity must be positive")
|
||||
|
||||
source_rows = load_source(args.source_public, args.source_private)
|
||||
if args.requests is not None:
|
||||
if args.requests < 2 or args.requests > len(source_rows):
|
||||
raise ValueError("requests must be between 2 and the source cohort size")
|
||||
source_rows = source_rows[: args.requests]
|
||||
fixed_token_ids = non_special_token_ids(args.model, len(source_rows))
|
||||
|
||||
cases = []
|
||||
for family in FAMILIES:
|
||||
if family.shape == "short-fixed":
|
||||
mean_output = 128.0
|
||||
elif family.shape == "mean-fixed":
|
||||
mean_output = round(
|
||||
sum(row["output_length"] for row in source_rows) / len(source_rows)
|
||||
)
|
||||
else:
|
||||
mean_output = sum(row["output_length"] for row in source_rows) / len(
|
||||
source_rows
|
||||
)
|
||||
for rho in rhos:
|
||||
target_rate = (
|
||||
rho * args.reference_decode_tokens_per_second / mean_output
|
||||
)
|
||||
rows = build_family_rows(
|
||||
source_rows,
|
||||
family,
|
||||
target_rate=target_rate,
|
||||
fixed_token_ids=fixed_token_ids,
|
||||
)
|
||||
rho_label = f"rho{rho:.2f}".replace(".", "p")
|
||||
root = args.output_root / family.name / rho_label
|
||||
cases.append(
|
||||
write_case(
|
||||
root,
|
||||
rows,
|
||||
family=family,
|
||||
rho=rho,
|
||||
target_rate=target_rate,
|
||||
reference_capacity=args.reference_decode_tokens_per_second,
|
||||
source_public=args.source_public,
|
||||
source_private=args.source_private,
|
||||
)
|
||||
)
|
||||
|
||||
experiment_manifest = {
|
||||
"schema": "frontier-workload-regime-suite-v1",
|
||||
"reference_decode_tokens_per_second": args.reference_decode_tokens_per_second,
|
||||
"rhos": rhos,
|
||||
"families": [family.__dict__ for family in FAMILIES],
|
||||
"cases": cases,
|
||||
}
|
||||
args.output_root.mkdir(parents=True, exist_ok=True)
|
||||
output = args.output_root / "manifest.json"
|
||||
output.write_text(json.dumps(experiment_manifest, indent=2, sort_keys=True) + "\n")
|
||||
print(output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 192 KiB |
@@ -0,0 +1,73 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Render the review-only schematic for the workload-regime experiment card."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
|
||||
|
||||
def logistic(x: np.ndarray, center: float, width: float) -> np.ndarray:
|
||||
return 1.0 / (1.0 + np.exp(-(x - center) / width))
|
||||
|
||||
|
||||
rho = np.linspace(0.02, 1.25, 300)
|
||||
families = {
|
||||
"homogeneous / real": (0.54, "#1f77b4", "-"),
|
||||
"homogeneous / simulator": (0.36, "#1f77b4", "--"),
|
||||
"heterogeneous / real": (0.76, "#d95f02", "-"),
|
||||
"heterogeneous / simulator": (0.68, "#d95f02", "--"),
|
||||
}
|
||||
|
||||
fig, axes = plt.subplots(1, 2, figsize=(11.5, 4.6), constrained_layout=True)
|
||||
|
||||
for label, (center, color, style) in families.items():
|
||||
state = 0.1 + 0.82 * logistic(rho, center, 0.055)
|
||||
axes[0].plot(rho, state, color=color, linestyle=style, linewidth=2.2, label=label)
|
||||
|
||||
axes[0].axhline(0.5, color="0.35", linewidth=1, linestyle=":")
|
||||
axes[0].set(
|
||||
title="A. Closed-loop scheduler-state knee",
|
||||
xlabel="Normalized decode offered load, ρ",
|
||||
ylabel="State index (batch / occupancy)",
|
||||
xlim=(0, 1.25),
|
||||
ylim=(0, 1.02),
|
||||
)
|
||||
axes[0].legend(frameon=False, fontsize=8, loc="lower right")
|
||||
|
||||
homogeneous_slack = 0.32 - 0.82 * np.exp(-((rho - 0.45) / 0.18) ** 2) - 0.12 * rho
|
||||
heterogeneous_slack = 0.38 - 0.43 * np.exp(-((rho - 0.72) / 0.20) ** 2) - 0.10 * rho
|
||||
capacity_aligned = 0.18 + 0.18 * logistic(rho, 0.83, 0.07)
|
||||
|
||||
axes[1].plot(rho, homogeneous_slack, color="#1f77b4", linewidth=2.2, label="homogeneous")
|
||||
axes[1].plot(rho, heterogeneous_slack, color="#d95f02", linewidth=2.2, label="heterogeneous")
|
||||
axes[1].plot(rho, capacity_aligned, color="#2a9d8f", linewidth=2.2, label="capacity-aligned")
|
||||
axes[1].axhline(0, color="black", linewidth=1.2)
|
||||
axes[1].fill_between(rho, -0.55, 0, color="#d62728", alpha=0.09, label="ranking reversal")
|
||||
axes[1].set(
|
||||
title="B. Config-ranking decision boundary",
|
||||
xlabel="Normalized decode offered load, ρ",
|
||||
ylabel="Minimum signed decision slack",
|
||||
xlim=(0, 1.25),
|
||||
ylim=(-0.55, 0.55),
|
||||
)
|
||||
axes[1].legend(frameon=False, fontsize=8, loc="lower right")
|
||||
|
||||
for ax in axes:
|
||||
ax.grid(alpha=0.18)
|
||||
ax.text(
|
||||
0.5,
|
||||
0.94,
|
||||
"MOCK / SCHEMATIC — NOT DATA",
|
||||
transform=ax.transAxes,
|
||||
ha="center",
|
||||
va="top",
|
||||
fontsize=10,
|
||||
fontweight="bold",
|
||||
color="#9d0208",
|
||||
bbox={"facecolor": "white", "edgecolor": "#9d0208", "alpha": 0.86},
|
||||
)
|
||||
|
||||
out = Path(__file__).with_name("mock_workload_regime_boundary.png")
|
||||
fig.savefig(out, dpi=180)
|
||||
print(out)
|
||||
84
tests/test_materialize_workload_regimes.py
Normal file
84
tests/test_materialize_workload_regimes.py
Normal file
@@ -0,0 +1,84 @@
|
||||
import importlib.util
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
SCRIPT = (
|
||||
Path(__file__).resolve().parents[1]
|
||||
/ "runs/frontier-workload-regime-taxonomy-v0/materialize_workload_regimes.py"
|
||||
)
|
||||
SPEC = importlib.util.spec_from_file_location("workload_regimes", SCRIPT)
|
||||
MODULE = importlib.util.module_from_spec(SPEC)
|
||||
assert SPEC.loader is not None
|
||||
sys.modules[SPEC.name] = MODULE
|
||||
SPEC.loader.exec_module(MODULE)
|
||||
|
||||
|
||||
def source_rows():
|
||||
return [
|
||||
{
|
||||
"source_index": index,
|
||||
"source_arrived_at": arrival,
|
||||
"input_length": input_length,
|
||||
"output_length": output_length,
|
||||
"session_id": 100 + index,
|
||||
"runtime_block_ids": list(range(input_length // 16)),
|
||||
"body": {"prompt": f"prompt-{index}", "model": "fixture"},
|
||||
}
|
||||
for index, (arrival, input_length, output_length) in enumerate(
|
||||
[(10.0, 16, 4), (10.5, 32, 8), (14.0, 48, 12)]
|
||||
)
|
||||
]
|
||||
|
||||
|
||||
def test_arrival_contracts_preserve_rate_and_burst_pattern():
|
||||
rows = source_rows()
|
||||
uniform = MODULE.arrivals_for(rows, "uniform", 2.0)
|
||||
traced = MODULE.arrivals_for(rows, "trace", 2.0)
|
||||
|
||||
assert uniform == [0.0, 0.5, 1.0]
|
||||
assert (len(traced) - 1) / (traced[-1] - traced[0]) == 2.0
|
||||
assert traced[1] - traced[0] < traced[2] - traced[1]
|
||||
|
||||
|
||||
def test_family_axes_change_only_the_declared_contract():
|
||||
rows = source_rows()
|
||||
token_ids = [10, 11, 12, 13]
|
||||
families = {family.name: family for family in MODULE.FAMILIES}
|
||||
|
||||
w1 = MODULE.build_family_rows(
|
||||
rows,
|
||||
families["w1-mean-fixed-uniform-none"],
|
||||
target_rate=1.0,
|
||||
fixed_token_ids=token_ids,
|
||||
)
|
||||
w3 = MODULE.build_family_rows(
|
||||
rows,
|
||||
families["w3-heterogeneous-uniform-none"],
|
||||
target_rate=1.0,
|
||||
fixed_token_ids=token_ids,
|
||||
)
|
||||
w5 = MODULE.build_family_rows(
|
||||
rows,
|
||||
families["w5-heterogeneous-uniform-prefix"],
|
||||
target_rate=1.0,
|
||||
fixed_token_ids=token_ids,
|
||||
)
|
||||
|
||||
assert {(row["input_length"], row["output_length"]) for row in w1} == {(32, 8)}
|
||||
assert [(row["input_length"], row["output_length"]) for row in w3] == [
|
||||
(16, 4),
|
||||
(32, 8),
|
||||
(48, 12),
|
||||
]
|
||||
assert all(row["runtime_block_ids"] == [] for row in w1 + w3)
|
||||
assert [row["runtime_block_ids"] for row in w5] == [
|
||||
[0],
|
||||
[0, 1],
|
||||
[0, 1, 2],
|
||||
]
|
||||
assert [row["body"]["prompt"] for row in w3] == [
|
||||
"prompt-0",
|
||||
"prompt-1",
|
||||
"prompt-2",
|
||||
]
|
||||
Reference in New Issue
Block a user