160 lines
7.3 KiB
Markdown
160 lines
7.3 KiB
Markdown
# Static-policy oracle-gap protocol
|
|
|
|
Status: **FROZEN BEFORE NEW GPU EXECUTION**.
|
|
|
|
Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were
|
|
inspected only to choose the workload pair and rate brackets. They are
|
|
exploratory calibration data, not primary observations in this protocol.
|
|
|
|
## Question and decision gate
|
|
|
|
The candidate motivation is:
|
|
|
|
> A single global static batching policy leaves at least 10% end-to-end
|
|
> SLO-goodput on the table when serving temporally heterogeneous phases; a
|
|
> phase-aware runtime policy can recover that gap without changing hardware,
|
|
> model, precision, or tensor-parallel topology.
|
|
|
|
This experiment tests a necessary condition in the existing TP1 policy space
|
|
`{C00,C10,C01,C11}`. The optimistic oracle knows the phase and switches with
|
|
zero delay, zero state-transfer cost, and no prediction error. If even this
|
|
oracle cannot beat the best one-config-for-all-phases policy by 10%, an online
|
|
controller over these MNS/MBT choices cannot do so either.
|
|
|
|
The primary gate uses a conservative capacity bracket:
|
|
|
|
- `L[p,c]`: highest offered rate accepted as SLO-feasible for phase `p` and
|
|
config `c`;
|
|
- `U[p,c]`: lowest higher offered rate accepted as SLO-infeasible;
|
|
- oracle upper bound at phase-time weights `w`:
|
|
`sum_p w[p] * max_c U[p,c]`;
|
|
- best-static lower bound:
|
|
`max_c sum_p w[p] * L[p,c]`.
|
|
|
|
We scan every P01/P06 time mixture, including pure endpoints. The current
|
|
motivation is **REFUTED** if the maximum conservative ratio
|
|
`oracle_upper / static_lower - 1` is below 10%. It is **NOT ESTABLISHED** if the
|
|
bound crosses 10% but the observed point estimate does not. A positive result
|
|
requires a point-estimate gap of at least 10% and then a separately
|
|
pre-registered interleaved-trace validation; this frontier experiment alone
|
|
cannot establish a positive E2E contribution.
|
|
|
|
The conclusion is scoped to the measured MNS/MBT policy family and the chosen
|
|
strongest-conflict phase pair. It does not rule out new scheduling mechanisms,
|
|
KV-state policies, topology changes, or other workload phases.
|
|
|
|
## Fixed system boundary
|
|
|
|
| Item | Frozen value |
|
|
|---|---|
|
|
| Host | `dash0`, one run at a time on physical GPU0 |
|
|
| GPU | NVIDIA H20; no other GPU process anywhere on the host |
|
|
| Model | `/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16 |
|
|
| Runtime | `/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2` |
|
|
| vLLM source | `/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0` |
|
|
| Topology | TP1, one server, no data/pipeline parallelism |
|
|
| Fixed mechanisms | chunked prefill on; prefix caching on |
|
|
| Client | Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client |
|
|
| Seeds | workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config |
|
|
|
|
SLO co-location results in Phase 6 showed pass-rate flips despite small
|
|
throughput deltas. Therefore unused H20s remain idle: parallel placement is not
|
|
authoritative for this experiment.
|
|
|
|
## Workloads and policies
|
|
|
|
The pair is chosen before new measurements because Phase 3 showed the strongest
|
|
opposing static preference:
|
|
|
|
- **P01:** input `U[128,512]`, output exactly 64 tokens, deterministic steady
|
|
arrivals. C10 lost 24.27% saturation throughput relative to C00.
|
|
- **P06:** 50/50 input mixture `U[128,512]`/`U[4096,8192]`, output exactly 512
|
|
tokens, deterministic bursts of eight. C10 gained 3.37% over C00.
|
|
|
|
Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a
|
|
derived manifest preserves request order, lengths, outputs, and arrival class,
|
|
but applies a trial-specific token-seed offset. The same derived manifest is
|
|
used for all four configs. This prevents prefix-cache carry-over when a hot
|
|
server executes several anchors without changing the logical workload.
|
|
|
|
| Config | Effective MNS | Effective MBT | Extra flags |
|
|
|---|---:|---:|---|
|
|
| C00 | 1024 | 8192 | none |
|
|
| C10 | 64 | 8192 | `--max-num-seqs 64` |
|
|
| C01 | 1024 | 2048 | `--max-num-batched-tokens 2048` |
|
|
| C11 | 64 | 2048 | both flags |
|
|
|
|
Startup logs must confirm these values. A default drift is a stop condition.
|
|
|
|
## Load grid, order, and repetitions
|
|
|
|
Primary grids:
|
|
|
|
- P01: `{26,28,30,32,34,36}` requests/s; execution order
|
|
`32,26,36,28,34,30`.
|
|
- P06: `{1.4,1.5,1.6,1.7,1.8,1.9,2.0}` requests/s; execution order
|
|
`1.7,1.4,2.0,1.5,1.9,1.6,1.8`.
|
|
|
|
Every primary anchor runs once. For each phase/config, the highest primary
|
|
feasible anchor and its next higher primary anchor are then run two more times,
|
|
giving three trials at both sides of the boundary. If all primary anchors are
|
|
feasible, extend upward in the fixed order P01 `38,40,42` or P06 `2.1,2.2,2.3`.
|
|
If all are infeasible, extend downward in the fixed order P01 `24,22,20` or P06
|
|
`1.3,1.2,1.1`. Stop extending at the first bracket.
|
|
|
|
One primary server is launched per config in order `C11,C00,C01,C10`.
|
|
Confirmation servers are fresh and launch in reverse order
|
|
`C10,C01,C00,C11`; their boundary anchors run high-to-low. This balances
|
|
machine-time drift and makes confirmation independent of the primary server's
|
|
cache/compiler state.
|
|
|
|
Timelines:
|
|
|
|
- P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
|
|
- P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
|
|
- no Kineto profiling; exact greedy output with `ignore_eos`; maximum client
|
|
concurrency 256.
|
|
|
|
A trial is SLO-feasible when at least 95% of requests admitted during the clean
|
|
interval eventually finish successfully and individually satisfy both:
|
|
|
|
- TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s
|
|
otherwise;
|
|
- TPOT <= 50 ms, computed as `(completion - first_token)/(output_tokens - 1)`.
|
|
|
|
SLO-goodput is the number of those passing clean-admission requests divided by
|
|
clean seconds. Client schedule lag must stay <=1 s and achieved clean offered
|
|
rate must be within 5% of target. Failure of either condition makes the anchor
|
|
infeasible; its admitted-only latency is not used to rescue it.
|
|
|
|
At a repeated boundary, feasibility is the majority of three trial verdicts.
|
|
All accepted anchor verdicts must be monotone in offered rate. A persistent
|
|
non-monotone result after the registered repeats is a red flag and stops the
|
|
oracle-gap inference.
|
|
|
|
## Validity and stopping rules
|
|
|
|
Before every server launch record host, GPU, driver, clocks, runtime package
|
|
versions, git/source hashes, manifest hashes, exact commands, and process
|
|
contamination. Stop on another GPU process, request/output mismatch, manifest
|
|
drift, server crash, non-finite latency, ratio outside `[0,1]`, negative
|
|
counter, or discontinuous/non-monotone accepted frontier.
|
|
|
|
The controller is detached and resumable. It kills only process groups it
|
|
created, checks zero GPU memory after every server, never overwrites a complete
|
|
trial, and writes state atomically. The hard budget is 6 H20-hours; expected
|
|
cost is 3.0--4.0 H20-hours and approximately the same wall time because runs
|
|
are serialized.
|
|
|
|
## Required report
|
|
|
|
The report includes every trial's target/achieved rate, clean cohort size,
|
|
pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure
|
|
reasons, accepted frontier brackets, per-phase oracle choices, best static
|
|
choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.
|
|
|
|
The final statistics section ends with a data-sanity block containing `n`,
|
|
min/max, distinct-value counts, and checks for non-negative counters, ratios in
|
|
`[0,1]`, non-identical per-config results, exact output work, monotone
|
|
frontiers, and continuous rate brackets.
|