# Frontier workload-regime taxonomy

- Date: 2026-07-20
- Status: proposed; awaiting review before workload generation or GPU runs
- Scope: explain when Frontier preserves the real-system config ranking, rather than merely comparing Fixed with Trace

## Claim under test

Frontier reliability is controlled by three quantities:

1. the latency-model residual between simulator and real execution;
2. the closed-loop gain from timing to scheduler state (batch, MoE routing, CUDA-graph bucket, MNS occupancy, admission/KV pressure);
3. the real decision margin between configurations.

For a config pair `a,b`, define

```text
D_real(a,b) = log L_real(a) - log L_real(b)
delta(a,b)  = [log L_sim(a)-log L_real(a)]
              - [log L_sim(b)-log L_real(b)]
slack(a,b)  = sign(D_real) * [D_real + delta]
```

`slack < 0` means the simulator reverses the real pairwise ordering. The primary hypothesis is that reversals occur when simulator and real execution land on different sides of a scheduler-state knee, or when the real decision margin is too small to absorb the differential residual. `Fixed` and `Trace` are not themselves the causal classes.

## Existing evidence motivating the experiment

- Q30 Trace-PD preserves all six objective winners, but many pairwise residuals oppose the real winner. Its success is therefore often margin protection, not zero residual.
- Q235 Trace-PD preserves TTFT/TPOT winners but misses E2E p90 by 6.2%; Trace is not universally safe.
- Q30/Q235 Fixed-PD decode objectives show negative minimum signed slack and 13--37% regret.
- In Q30 low-load Fixed-PD, Frontier's batch-1 TP ordering is correct, while the closed-loop simulator increases TP4's effective batch and changes the MoE cost enough to reverse the ordering. This identifies a concrete state knee, but does not yet establish a general rule.

## Workload families

All comparisons use the same request multiset where applicable, the same total observation window, and the same normalized offered decode load

```text
rho = request_rate * E[output_tokens] / measured_reference_decode_capacity.
```

This avoids equating equal request rates with equal load.

| ID | Shape / request lengths | Arrival process | Prefix/session state | Isolated effect |
|---|---|---|---|---|
| W0 | short fixed `2048 -> 128` | uniform | off | known low-residence failure anchor |
| W1 | trace-mean fixed ISL/OSL | uniform | off | homogeneous baseline |
| W2 | trace-mean fixed ISL/OSL | trace timestamps | off | arrival burst only |
| W3 | exact trace ISL/OSL multiset | uniform | off | length heterogeneity only |
| W4 | exact trace ISL/OSL multiset | trace timestamps | off | length + burst |
| W5 | exact trace prompts/ISL/OSL | uniform | exact prefix/session identity | prefix state without burst |
| W6 | exact trace prompts/ISL/OSL | trace timestamps | exact prefix/session identity | full production trace |

Prefix is intentionally a nested factor: enabling a synthetic prefix graph on fixed identical requests would introduce a different workload rather than isolate production prefix reuse. Therefore this is not presented as a full `2^3` factorial.

## Load sweep and expected patterns

Simulator discovery sweep: `rho in {0.05, 0.25, 0.50, 0.90, 1.20}`. The points mean deep low load, light batching, moderate batching, capacity knee, and overload; their request rates are derived independently for every workload family.

| Pattern | Observable state | Prediction for Frontier |
|---|---|---|
| P1 singleton-linear | real and sim stay below the first batch/graph knee | works if the batch-1 operator ordering is correct |
| P2 knee-straddling | real and sim occupy opposite sides of a batch/MoE/graph/MNS knee | fails systematically; Fixed-PD is the current example |
| P3 same-side batched | both systems cross the same knee and remain below admission pressure | works if batch-conditioned operator ordering is correct |
| P4 capacity/admission aligned | both systems are governed by the same capacity bottleneck | TTFT/config winner may work despite large absolute error; E2E/MNS can remain fragile |
| P5 heterogeneity-smoothed | broad lengths reduce coherent threshold occupancy at matched `rho` | may work; this is a hypothesis, not an established explanation |
| P6 burst-sensitive | same request multiset, but transient queue/MNS occupancy differs | mean ranking may work while TTFT/E2E tail ranking fails |
| P7 prefix-state-sensitive | hit/eviction and reused-token distributions differ | TTFT ranking fails unless prefix-state transitions are modeled; decode TPOT may remain stable |
| P8 decision-boundary | real config margin is comparable to run variance/residual | fragile; an exact winner match is not reliable evidence |

## Hypotheses and distinguishing tests

### H1: state-regime hypothesis (primary)

I believe config-ranking failures occur when the latency residual moves a workload across a scheduler-state knee, because the residual is then amplified into a different batch/resource trajectory. I will verify this by checking whether signed-slack zero crossings co-locate with measured real/simulator state-knee crossings.

### H2: heterogeneity-smoothing hypothesis

I believe length heterogeneity can reduce coherent threshold amplification, because requests reach scheduler boundaries at dispersed times. I will verify it with W1 vs W3 and W2 vs W4 at matched `rho`, requiring a smaller real/sim state-distribution gap rather than merely a correct winner.

### H3: bottleneck/margin-protection alternative

Trace success may instead be explained entirely by a large real decision margin or a shared capacity bottleneck. This hypothesis wins over H2 if W3/W4 do not reduce state-distribution error after matching load and margin, while ranking correctness remains predicted by margin alone.

### H4: burst and prefix are independent failure channels

I believe arrival bursts primarily affect waiting/admission and tail TTFT/E2E, whereas prefix mismatch primarily affects prefill/TTFT state. I will verify this with W1/W2, W3/W4, and W3/W5 paired comparisons.

## Configuration and model scope

Discovery uses Qwen30B because its 12-cell `TP x MNS` surface already has simulator and real anchors:

- TP: `{1, 2, 4}`
- MNS: `{8, 16, 32, 64}`
- objectives: mean/p90 TTFT, TPOT, E2E

Qwen235B is a held-out confirmation, not pooled into discovery:

- existing four feasible TP/MNS configurations;
- only the workload/load patterns that discriminate H1--H4 after Q30 converges.

## Measurements

End-to-end:

- completed/failed requests and achieved request/token rate;
- TTFT, TPOT, E2E mean/p50/p90/p95;
- config regret, pairwise agreement, signed decision slack;
- run-to-run winner stability.

Closed-loop state:

- prefill/decode batch-size histograms and time-weighted batch;
- Running/Waiting distributions and admission delay;
- MNS active-token occupancy and KV/context pressure;
- CUDA-graph bucket residency and fallback frequency;
- prefix hit/reused-token/eviction distributions for W5/W6.

## Decision rules

A workload/load region is:

- **reliable** if regret is at most 5%, pairwise agreement is at least 0.8 at two adjacent load points, and the winner is stable across confirmation trials;
- **fragile** if regret is at most 5% but the real margin overlaps run uncertainty, or a small rate/timing perturbation changes the winner;
- **failed** if regret exceeds 5% or a decision-critical pair has negative signed slack;
- **mechanistically explained by H1** only if the ranking transition co-locates with an observed state-regime transition. Correlation with the Fixed/Trace label is insufficient.

H2 is supported only if the heterogeneous member of a matched pair reduces state-distribution error and shifts the failure boundary in repeated trials. A correct winner alone does not support smoothing.

## Execution plan after review

1. Materialize W0--W6 with one manifest recording request multiset, arrival timestamps, prefix identity, rate contract, and hashes.
2. Run the simulator sweep across `rho` and the Q30 surface; emit a per-stage state ledger.
3. Select real-machine pilot points only around the predicted knees plus one safe-side control. Use guard configs `TP1/MNS64`, `TP4/MNS8`, and `TP4/MNS64`; add `TP2/MNS32` only if the transition is not bracketed.
4. Use only `dash1`, `dash2`, `dash3`, and `dash4`, each verified as an 8×H20 host. `dash0` is excluded from probing, synchronization, and execution. Pin one independent experiment group to each host so at most four groups run in parallel; do not split one trial across hosts.
5. Run one pilot trial per selected point. Confirm only hypothesis-discriminating points with three fresh-server trials and rotated order.
6. Apply the resulting classifier unchanged to the Q235 held-out cases.

Provisional four-way allocation after the simulator identifies the discriminating points:

| Host | Experiment group | Primary contrast |
|---|---|---|
| dash1 | homogeneous controls | W0/W1 across safe side and first knee |
| dash2 | arrival effect | W1 vs W2 and W3 vs W4 |
| dash3 | length heterogeneity | W1 vs W3 and W2 vs W4 |
| dash4 | prefix/full trace | W3 vs W5 and W4 vs W6 |

The groups are logical queues, not permanent ownership: if a host probe fails, that host is excluded and its group waits or moves to another permitted idle host. Cross-host latency values are not pooled until a common canary config verifies that host effects are within run uncertainty.

No GPU run is authorized by this card yet. The review decision is whether the workload decomposition and decision rules are sufficient to implement the materializer and launch Phase 1.

## Expected figure

The accompanying mock figure is schematic, not data. Panel A shows the state knee that real and simulator trajectories may cross at different loads. Panel B shows the corresponding minimum signed decision slack; a negative value denotes a ranking reversal. The claim is supported only if measured zero crossings and state knees align across workload families.

## Risks and controls

- Equal `rho` does not guarantee equal prefill pressure; report both prefill and decode offered work and stratify if necessary.
- Full-trace overload can collapse all configs to similarly poor latency. Such points identify a capacity-limited region but cannot validate fine-grained ranking.
- MNS ties and censored/failed requests can create false winners; exclude invalid cells before calculating regret and report the exclusion.
- One trace cannot establish generality. The initial result is a mechanism boundary for this trace/model/hardware, followed by held-out Q235 validation.

## Execution log

### 2026-07-20: materialization and simulator launch

- Code baseline: `feature/sim@157bf36` for the valid v4 sweep.
- Hosts probed: `dash1`, `dash2`, `dash3`, `dash4`; each exposed 8 NVIDIA H20 GPUs with 0 MiB used at probe time. `dash0` was not probed or used.
- Source cohort: 129 Q30 Trace-PD requests. The private artifact supplies exact prompts, lengths, outputs, timestamps, sessions, and runtime block identities; the simulator projection retains only the first `floor(ISL/16)` complete block identities.
- Materialized: 35 cases = W0--W6 × `rho {0.05,0.25,0.50,0.90,1.20}`. Audit passed request count, exact decode offered load, empirical arrival rate, prefix block count, and prefix-off empty identity vectors.
- Simulator smoke: W0 / `rho=0.05` / TP4-MNS64 completed 129/129. Simulator TTFT mean/p90 was 109.81/124.16 ms and TPOT mean/p90 was 36.26/36.79 ms. This is a harness check, not real-system fidelity evidence.
- Invalid attempts retained for audit: v1 had a Bash argument-expansion error; v2 mixed multiple workload families into a runner that requires strictly increasing anchors from one family; v3 exposed a scikit-learn cache-version mismatch. None is used as scientific evidence.
- Valid v4 controls: isolated output/predictor cache per TP/prefix group; scikit-learn 1.9.0 matching the predictor cache format; per-family five-point runner invocations; stage batch ledger enabled; TP1 exempted from the collective fallback gate because a single rank has no all-reduce.
- Active v4 allocation: dash1=TP1 prefix off/on, dash2=TP2 prefix off/on, dash3=TP4 prefix off, dash4=TP4 prefix on. The four fleet jobs are running from fresh `sim-v4` output roots. First-process audit found the explicit isolated `--metrics_config_cache_dir` on all hosts and zero cross-version warnings.
- First valid v4 tranche: 16/16 observed cells completed, each with 129 requests, request metrics, and a stage-batch ledger; no traceback, fallback, or version warning was found. The tranche covers all five W0 load points at TP1/TP2/TP4-MNS8 plus the first W5 prefix points at TP4-MNS8.
- Early load-boundary observation: W0 at `rho=0.05` is low-latency for TP4-MNS8 (simulator TTFT mean 109.25 ms) but already queues for TP1-MNS8 (25.70 s); at `rho=0.25`, even TP4-MNS8 reaches 27.13 s mean TTFT. Because `rho` normalizes decode tokens only, high-rate short-output W0 also raises prefill and active-sequence pressure. These points map the overload boundary and are not eligible as reasonable-latency real pilots.
- Real-runtime gate: a stock vLLM 0.20.0 environment passed import/H20 checks but used CUDA 13.0, so it is excluded from comparison with the historical CUDA 12.9 baseline. The replacement environment `vllm-0.20.0-cu129-workload-regime-v2` passes `vllm CLI=0.20.0+cu129`, torch `2.11.0+cu129`, CUDA runtime 12.9, H20 visibility, and all 179 package dependency checks. The first CPFS install used file copies and was stopped after download because it was still copying roughly 7 GB after 12 minutes; its incomplete directory is retained with an `invalid-copy-incomplete` suffix, while v2 uses same-filesystem hardlinks from the validated cache.
- Load-contract correction: the original Fixed-PD surface held request rate per GPU constant, so global arrival rate scaled with TP. The v4 sweep holds global arrival rate constant and is retained as the control that isolates service-topology changes. A matched per-GPU sweep is now required to reproduce the original closed-loop intervention: TP1/TP2/TP4 receive `1x/2x/4x` global arrival rate at the same per-GPU `rho`.
- Per-GPU low-load materialization: 105 cases = W0--W6 × `rho {0.0025,0.005,0.01,0.02,0.05}` × TP `{1,2,4}` were generated under `traces-per-gpu-low`. Audit passed 105 unique paths, 129 public/private rows per case, digests, arrival alignment, and exact `global_rate / TP = per_gpu_rate`. W0 `rho=0.01` is 0.239375 req/s/GPU, bracketing the original 0.215 req/s/GPU Fixed-PD point with `rho=0.005`.
- The per-GPU sweep writes to a separate `sim-per-gpu-v1` result root but reuses the completed v4 predictor cache for the same TP/prefix/config. Predictor cache provenance is explicit in every surface manifest; workload results and state ledgers are never shared.
- `wait_and_dispatch_per_gpu.sh` is active locally as a serial gate. It requires all four exact v4 run directories to contain `finished_at` and exit code zero before probing dash1--dash4 and dispatching the four per-GPU jobs; it does not launch a second sweep while v4 is still consuming CPU.
- A first materialization attempt rounded both `rho=0.005` and `rho=0.01` to the same `rho0p01` directory. Digest validation stopped before simulator launch; the invalid directories were retained with an `invalid-rho-label-collision` suffix. The label function now preserves up to 12 significant digits and has a regression test.

Current decision: finish the v4 fixed-global-rate control, then reuse its trained predictors for the low-load per-GPU sweep before selecting discriminating real-machine pilot points. No real latency result from vLLM 0.20.2 will be compared with the historical vLLM 0.20.0 baseline until the runtime-version gate is resolved.