Static-policy oracle-gap protocol

Status: FROZEN WITH A-OG-1 AMENDMENT.

Date frozen: 2026-07-13 (Asia/Singapore). Existing Phase-3 measurements were inspected only to choose the workload pair and rate brackets. They are exploratory calibration data, not primary observations in this protocol.

A-OG-1 — extend the P06 upper bracket (after trial 33)

The controller stopped as registered after C00/P06 remained SLO-feasible at every original and upward-extension anchor through 2.3 requests/s. At that point 33 trials and 1.519475 H20-hours were complete, all GPU memory had been released, and no C00/P06 infeasible upper bound existed. No oracle inference was performed.

This amendment changes only the P06 upward-extension list from 2.1,2.2,2.3 to 2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0. The controller stops at the first bracket exactly as before. Existing trials are immutable and are reused; config order, P01 rates, SLO, timelines, repetitions, placement, metrics, and decision threshold do not change. The resumable controller may accept the new code/protocol fingerprint only when all immutable runtime, client, model, manifest, config, and base-grid fields match the pre-amendment state, and it records both fingerprints under A-OG-1.

Question and decision gate

The candidate motivation is:

A single global static batching policy leaves at least 10% end-to-end SLO-goodput on the table when serving temporally heterogeneous phases; a phase-aware runtime policy can recover that gap without changing hardware, model, precision, or tensor-parallel topology.

This experiment tests a necessary condition in the existing TP1 policy space {C00,C10,C01,C11}. The optimistic oracle knows the phase and switches with zero delay, zero state-transfer cost, and no prediction error. If even this oracle cannot beat the best one-config-for-all-phases policy by 10%, an online controller over these MNS/MBT choices cannot do so either.

The primary gate uses a conservative capacity bracket:

L[p,c]: highest offered rate accepted as SLO-feasible for phase p and config c;
U[p,c]: lowest higher offered rate accepted as SLO-infeasible;
oracle upper bound at phase-time weights w: sum_p w[p] * max_c U[p,c];
best-static lower bound: max_c sum_p w[p] * L[p,c].

We scan every P01/P06 time mixture, including pure endpoints. The current motivation is REFUTED if the maximum conservative ratio oracle_upper / static_lower - 1 is below 10%. It is NOT ESTABLISHED if the bound crosses 10% but the observed point estimate does not. A positive result requires a point-estimate gap of at least 10% and then a separately pre-registered interleaved-trace validation; this frontier experiment alone cannot establish a positive E2E contribution.

The conclusion is scoped to the measured MNS/MBT policy family and the chosen strongest-conflict phase pair. It does not rule out new scheduling mechanisms, KV-state policies, topology changes, or other workload phases.

Fixed system boundary

Item	Frozen value
Host	`dash0`, one run at a time on physical GPU0
GPU	NVIDIA H20; no other GPU process anywhere on the host
Model	`/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B`, BF16
Runtime	`/tmp/wjh-opprof-phase2-dash0-20260711/.venv`, vLLM `0.24.1.dev3+g668cfb7e2`
vLLM source	`/home/admin/cpfs/wjh/opprof-phase2-dash0-20260711/vllm-v0.24.0`
Topology	TP1, one server, no data/pipeline parallelism
Fixed mechanisms	chunked prefill on; prefix caching on
Client	Phase-5 timestamp/fixed-rate wrapper over the Phase-3 exact-token client
Seeds	workload `20260712`; trial token-domain seed derived only from phase/rate/repetition, never config

SLO co-location results in Phase 6 showed pass-rate flips despite small throughput deltas. Therefore unused H20s remain idle: parallel placement is not authoritative for this experiment.

Workloads and policies

The pair is chosen before new measurements because Phase 3 showed the strongest opposing static preference:

P01: input U[128,512], output exactly 64 tokens, deterministic steady arrivals. C10 lost 24.27% saturation throughput relative to C00.
P06: 50/50 input mixture U[128,512]/U[4096,8192], output exactly 512 tokens, deterministic bursts of eight. C10 gained 3.37% over C00.

Both reuse the immutable 32,768-row Phase-3 manifests. For every trial a derived manifest preserves request order, lengths, outputs, and arrival class, but applies a trial-specific token-seed offset. The same derived manifest is used for all four configs. This prevents prefix-cache carry-over when a hot server executes several anchors without changing the logical workload.

Config	Effective MNS	Effective MBT	Extra flags
C00	1024	8192	none
C10	64	8192	`--max-num-seqs 64`
C01	1024	2048	`--max-num-batched-tokens 2048`
C11	64	2048	both flags

Startup logs must confirm these values. A default drift is a stop condition.

Load grid, order, and repetitions

Primary grids:

P01: {26,28,30,32,34,36} requests/s; execution order 32,26,36,28,34,30.
P06: {1.4,1.5,1.6,1.7,1.8,1.9,2.0} requests/s; execution order 1.7,1.4,2.0,1.5,1.9,1.6,1.8.

Every primary anchor runs once. For each phase/config, the highest primary feasible anchor and its next higher primary anchor are then run two more times, giving three trials at both sides of the boundary. If all primary anchors are feasible, extend upward in the fixed order P01 38,40,42 or P06 2.1,2.2,2.3,2.4,2.5,2.6,2.8,3.0. If all are infeasible, extend downward in the fixed order P01 24,22,20 or P06 1.3,1.2,1.1. Stop extending at the first bracket.

One primary server is launched per config in order C11,C00,C01,C10. Confirmation servers are fresh and launch in reverse order C10,C01,C00,C11; their boundary anchors run high-to-low. This balances machine-time drift and makes confirmation independent of the primary server's cache/compiler state.

Timelines:

P01: 60 s warm-up + 60 s clean measurement; drain cap 120 s.
P06: 60 s warm-up + 120 s clean measurement; drain cap 240 s.
no Kineto profiling; exact greedy output with ignore_eos; maximum client concurrency 256.

A trial is SLO-feasible when at least 95% of requests admitted during the clean interval eventually finish successfully and individually satisfy both:

TTFT <= 2 s for input <= 4,096 tokens; <= 4 s for input <= 32,768; <= 6 s otherwise;
TPOT <= 50 ms, computed as (completion - first_token)/(output_tokens - 1).

SLO-goodput is the number of those passing clean-admission requests divided by clean seconds. Client schedule lag must stay <=1 s and achieved clean offered rate must be within 5% of target. Failure of either condition makes the anchor infeasible; its admitted-only latency is not used to rescue it.

At a repeated boundary, feasibility is the majority of three trial verdicts. All accepted anchor verdicts must be monotone in offered rate. A persistent non-monotone result after the registered repeats is a red flag and stops the oracle-gap inference.

Validity and stopping rules

Before every server launch record host, GPU, driver, clocks, runtime package versions, git/source hashes, manifest hashes, exact commands, and process contamination. Stop on another GPU process, request/output mismatch, manifest drift, server crash, non-finite latency, ratio outside [0,1], negative counter, or discontinuous/non-monotone accepted frontier.

The controller is detached and resumable. It kills only process groups it created, checks zero GPU memory after every server, never overwrites a complete trial, and writes state atomically. The hard budget is 6 H20-hours; expected cost is 3.0--4.0 H20-hours and approximately the same wall time because runs are serialized.

Required report

The report includes every trial's target/achieved rate, clean cohort size, pass count/rate, SLO-goodput, TTFT/TPOT percentiles, schedule lag, failure reasons, accepted frontier brackets, per-phase oracle choices, best static choice, equal-time gap, worst-mixture conservative gap, and GPU-hours.

The final statistics section ends with a data-sanity block containing n, min/max, distinct-value counts, and checks for non-negative counters, ratios in [0,1], non-identical per-config results, exact output work, monotone frontiers, and continuous rate brackets.

8.3 KiB Raw Blame History