161 lines
10 KiB
Markdown
161 lines
10 KiB
Markdown
# Frontier workload-regime taxonomy
|
||
|
||
- Date: 2026-07-20
|
||
- Status: proposed; awaiting review before workload generation or GPU runs
|
||
- Scope: explain when Frontier preserves the real-system config ranking, rather than merely comparing Fixed with Trace
|
||
|
||
## Claim under test
|
||
|
||
Frontier reliability is controlled by three quantities:
|
||
|
||
1. the latency-model residual between simulator and real execution;
|
||
2. the closed-loop gain from timing to scheduler state (batch, MoE routing, CUDA-graph bucket, MNS occupancy, admission/KV pressure);
|
||
3. the real decision margin between configurations.
|
||
|
||
For a config pair `a,b`, define
|
||
|
||
```text
|
||
D_real(a,b) = log L_real(a) - log L_real(b)
|
||
delta(a,b) = [log L_sim(a)-log L_real(a)]
|
||
- [log L_sim(b)-log L_real(b)]
|
||
slack(a,b) = sign(D_real) * [D_real + delta]
|
||
```
|
||
|
||
`slack < 0` means the simulator reverses the real pairwise ordering. The primary hypothesis is that reversals occur when simulator and real execution land on different sides of a scheduler-state knee, or when the real decision margin is too small to absorb the differential residual. `Fixed` and `Trace` are not themselves the causal classes.
|
||
|
||
## Existing evidence motivating the experiment
|
||
|
||
- Q30 Trace-PD preserves all six objective winners, but many pairwise residuals oppose the real winner. Its success is therefore often margin protection, not zero residual.
|
||
- Q235 Trace-PD preserves TTFT/TPOT winners but misses E2E p90 by 6.2%; Trace is not universally safe.
|
||
- Q30/Q235 Fixed-PD decode objectives show negative minimum signed slack and 13--37% regret.
|
||
- In Q30 low-load Fixed-PD, Frontier's batch-1 TP ordering is correct, while the closed-loop simulator increases TP4's effective batch and changes the MoE cost enough to reverse the ordering. This identifies a concrete state knee, but does not yet establish a general rule.
|
||
|
||
## Workload families
|
||
|
||
All comparisons use the same request multiset where applicable, the same total observation window, and the same normalized offered decode load
|
||
|
||
```text
|
||
rho = request_rate * E[output_tokens] / measured_reference_decode_capacity.
|
||
```
|
||
|
||
This avoids equating equal request rates with equal load.
|
||
|
||
| ID | Shape / request lengths | Arrival process | Prefix/session state | Isolated effect |
|
||
|---|---|---|---|---|
|
||
| W0 | short fixed `2048 -> 128` | uniform | off | known low-residence failure anchor |
|
||
| W1 | trace-mean fixed ISL/OSL | uniform | off | homogeneous baseline |
|
||
| W2 | trace-mean fixed ISL/OSL | trace timestamps | off | arrival burst only |
|
||
| W3 | exact trace ISL/OSL multiset | uniform | off | length heterogeneity only |
|
||
| W4 | exact trace ISL/OSL multiset | trace timestamps | off | length + burst |
|
||
| W5 | exact trace prompts/ISL/OSL | uniform | exact prefix/session identity | prefix state without burst |
|
||
| W6 | exact trace prompts/ISL/OSL | trace timestamps | exact prefix/session identity | full production trace |
|
||
|
||
Prefix is intentionally a nested factor: enabling a synthetic prefix graph on fixed identical requests would introduce a different workload rather than isolate production prefix reuse. Therefore this is not presented as a full `2^3` factorial.
|
||
|
||
## Load sweep and expected patterns
|
||
|
||
Simulator discovery sweep: `rho in {0.05, 0.25, 0.50, 0.90, 1.20}`. The points mean deep low load, light batching, moderate batching, capacity knee, and overload; their request rates are derived independently for every workload family.
|
||
|
||
| Pattern | Observable state | Prediction for Frontier |
|
||
|---|---|---|
|
||
| P1 singleton-linear | real and sim stay below the first batch/graph knee | works if the batch-1 operator ordering is correct |
|
||
| P2 knee-straddling | real and sim occupy opposite sides of a batch/MoE/graph/MNS knee | fails systematically; Fixed-PD is the current example |
|
||
| P3 same-side batched | both systems cross the same knee and remain below admission pressure | works if batch-conditioned operator ordering is correct |
|
||
| P4 capacity/admission aligned | both systems are governed by the same capacity bottleneck | TTFT/config winner may work despite large absolute error; E2E/MNS can remain fragile |
|
||
| P5 heterogeneity-smoothed | broad lengths reduce coherent threshold occupancy at matched `rho` | may work; this is a hypothesis, not an established explanation |
|
||
| P6 burst-sensitive | same request multiset, but transient queue/MNS occupancy differs | mean ranking may work while TTFT/E2E tail ranking fails |
|
||
| P7 prefix-state-sensitive | hit/eviction and reused-token distributions differ | TTFT ranking fails unless prefix-state transitions are modeled; decode TPOT may remain stable |
|
||
| P8 decision-boundary | real config margin is comparable to run variance/residual | fragile; an exact winner match is not reliable evidence |
|
||
|
||
## Hypotheses and distinguishing tests
|
||
|
||
### H1: state-regime hypothesis (primary)
|
||
|
||
I believe config-ranking failures occur when the latency residual moves a workload across a scheduler-state knee, because the residual is then amplified into a different batch/resource trajectory. I will verify this by checking whether signed-slack zero crossings co-locate with measured real/simulator state-knee crossings.
|
||
|
||
### H2: heterogeneity-smoothing hypothesis
|
||
|
||
I believe length heterogeneity can reduce coherent threshold amplification, because requests reach scheduler boundaries at dispersed times. I will verify it with W1 vs W3 and W2 vs W4 at matched `rho`, requiring a smaller real/sim state-distribution gap rather than merely a correct winner.
|
||
|
||
### H3: bottleneck/margin-protection alternative
|
||
|
||
Trace success may instead be explained entirely by a large real decision margin or a shared capacity bottleneck. This hypothesis wins over H2 if W3/W4 do not reduce state-distribution error after matching load and margin, while ranking correctness remains predicted by margin alone.
|
||
|
||
### H4: burst and prefix are independent failure channels
|
||
|
||
I believe arrival bursts primarily affect waiting/admission and tail TTFT/E2E, whereas prefix mismatch primarily affects prefill/TTFT state. I will verify this with W1/W2, W3/W4, and W3/W5 paired comparisons.
|
||
|
||
## Configuration and model scope
|
||
|
||
Discovery uses Qwen30B because its 12-cell `TP x MNS` surface already has simulator and real anchors:
|
||
|
||
- TP: `{1, 2, 4}`
|
||
- MNS: `{8, 16, 32, 64}`
|
||
- objectives: mean/p90 TTFT, TPOT, E2E
|
||
|
||
Qwen235B is a held-out confirmation, not pooled into discovery:
|
||
|
||
- existing four feasible TP/MNS configurations;
|
||
- only the workload/load patterns that discriminate H1--H4 after Q30 converges.
|
||
|
||
## Measurements
|
||
|
||
End-to-end:
|
||
|
||
- completed/failed requests and achieved request/token rate;
|
||
- TTFT, TPOT, E2E mean/p50/p90/p95;
|
||
- config regret, pairwise agreement, signed decision slack;
|
||
- run-to-run winner stability.
|
||
|
||
Closed-loop state:
|
||
|
||
- prefill/decode batch-size histograms and time-weighted batch;
|
||
- Running/Waiting distributions and admission delay;
|
||
- MNS active-token occupancy and KV/context pressure;
|
||
- CUDA-graph bucket residency and fallback frequency;
|
||
- prefix hit/reused-token/eviction distributions for W5/W6.
|
||
|
||
## Decision rules
|
||
|
||
A workload/load region is:
|
||
|
||
- **reliable** if regret is at most 5%, pairwise agreement is at least 0.8 at two adjacent load points, and the winner is stable across confirmation trials;
|
||
- **fragile** if regret is at most 5% but the real margin overlaps run uncertainty, or a small rate/timing perturbation changes the winner;
|
||
- **failed** if regret exceeds 5% or a decision-critical pair has negative signed slack;
|
||
- **mechanistically explained by H1** only if the ranking transition co-locates with an observed state-regime transition. Correlation with the Fixed/Trace label is insufficient.
|
||
|
||
H2 is supported only if the heterogeneous member of a matched pair reduces state-distribution error and shifts the failure boundary in repeated trials. A correct winner alone does not support smoothing.
|
||
|
||
## Execution plan after review
|
||
|
||
1. Materialize W0--W6 with one manifest recording request multiset, arrival timestamps, prefix identity, rate contract, and hashes.
|
||
2. Run the simulator sweep across `rho` and the Q30 surface; emit a per-stage state ledger.
|
||
3. Select real-machine pilot points only around the predicted knees plus one safe-side control. Use guard configs `TP1/MNS64`, `TP4/MNS8`, and `TP4/MNS64`; add `TP2/MNS32` only if the transition is not bracketed.
|
||
4. Use only `dash1`, `dash2`, `dash3`, and `dash4`, each verified as an 8×H20 host. `dash0` is excluded from probing, synchronization, and execution. Pin one independent experiment group to each host so at most four groups run in parallel; do not split one trial across hosts.
|
||
5. Run one pilot trial per selected point. Confirm only hypothesis-discriminating points with three fresh-server trials and rotated order.
|
||
6. Apply the resulting classifier unchanged to the Q235 held-out cases.
|
||
|
||
Provisional four-way allocation after the simulator identifies the discriminating points:
|
||
|
||
| Host | Experiment group | Primary contrast |
|
||
|---|---|---|
|
||
| dash1 | homogeneous controls | W0/W1 across safe side and first knee |
|
||
| dash2 | arrival effect | W1 vs W2 and W3 vs W4 |
|
||
| dash3 | length heterogeneity | W1 vs W3 and W2 vs W4 |
|
||
| dash4 | prefix/full trace | W3 vs W5 and W4 vs W6 |
|
||
|
||
The groups are logical queues, not permanent ownership: if a host probe fails, that host is excluded and its group waits or moves to another permitted idle host. Cross-host latency values are not pooled until a common canary config verifies that host effects are within run uncertainty.
|
||
|
||
No GPU run is authorized by this card yet. The review decision is whether the workload decomposition and decision rules are sufficient to implement the materializer and launch Phase 1.
|
||
|
||
## Expected figure
|
||
|
||
The accompanying mock figure is schematic, not data. Panel A shows the state knee that real and simulator trajectories may cross at different loads. Panel B shows the corresponding minimum signed decision slack; a negative value denotes a ranking reversal. The claim is supported only if measured zero crossings and state knees align across workload families.
|
||
|
||
## Risks and controls
|
||
|
||
- Equal `rho` does not guarantee equal prefill pressure; report both prefill and decode offered work and stratify if necessary.
|
||
- Full-trace overload can collapse all configs to similarly poor latency. Such points identify a capacity-limited region but cannot validate fine-grained ranking.
|
||
- MNS ties and censored/failed requests can create false winners; exclude invalid cells before calculating regret and report the exclusion.
|
||
- One trace cannot establish generality. The initial result is a mechanism boundary for this trace/model/hardware, followed by held-out Q235 validation.
|