gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

experiment: add per-gpu workload control

Browse Source
This commit is contained in:
Gahow Wang
2026-07-20 18:48:22 +08:00
parent a033a72195
commit 809ad9ffef
8 changed files with 330 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
.research/experiments/frontier-workload-regime-taxonomy-20260720.md
View File

@@ -171,5 +171,11 @@ The accompanying mock figure is schematic, not data. Panel A shows the state kne
- Invalid attempts retained for audit: v1 had a Bash argument-expansion error; v2 mixed multiple workload families into a runner that requires strictly increasing anchors from one family; v3 exposed a scikit-learn cache-version mismatch. None is used as scientific evidence.
- Valid v4 controls: isolated output/predictor cache per TP/prefix group; scikit-learn 1.9.0 matching the predictor cache format; per-family five-point runner invocations; stage batch ledger enabled; TP1 exempted from the collective fallback gate because a single rank has no all-reduce.
- Active v4 allocation: dash1=TP1 prefix off/on, dash2=TP2 prefix off/on, dash3=TP4 prefix off, dash4=TP4 prefix on. The four fleet jobs are running from fresh `sim-v4` output roots. First-process audit found the explicit isolated `--metrics_config_cache_dir` on all hosts and zero cross-version warnings.
- First valid v4 tranche: 16/16 observed cells completed, each with 129 requests, request metrics, and a stage-batch ledger; no traceback, fallback, or version warning was found. The tranche covers all five W0 load points at TP1/TP2/TP4-MNS8 plus the first W5 prefix points at TP4-MNS8.
- Early load-boundary observation: W0 at `rho=0.05` is low-latency for TP4-MNS8 (simulator TTFT mean 109.25 ms) but already queues for TP1-MNS8 (25.70 s); at `rho=0.25`, even TP4-MNS8 reaches 27.13 s mean TTFT. Because `rho` normalizes decode tokens only, high-rate short-output W0 also raises prefill and active-sequence pressure. These points map the overload boundary and are not eligible as reasonable-latency real pilots.
- Real-runtime gate: a stock vLLM 0.20.0 environment passed import/H20 checks but used CUDA 13.0, so it is excluded from comparison with the historical CUDA 12.9 baseline. A separate official vLLM `0.20.0+cu129` environment is being materialized from the shared package cache before any real pilot is launched.
- Load-contract correction: the original Fixed-PD surface held request rate per GPU constant, so global arrival rate scaled with TP. The v4 sweep holds global arrival rate constant and is retained as the control that isolates service-topology changes. A matched per-GPU sweep is now required to reproduce the original closed-loop intervention: TP1/TP2/TP4 receive `1x/2x/4x` global arrival rate at the same per-GPU `rho`.
- Per-GPU low-load materialization: 105 cases = W0--W6 × `rho {0.0025,0.005,0.01,0.02,0.05}` × TP `{1,2,4}` were generated under `traces-per-gpu-low`. Audit passed 105 unique paths, 129 public/private rows per case, digests, arrival alignment, and exact `global_rate / TP = per_gpu_rate`. W0 `rho=0.01` is 0.239375 req/s/GPU, bracketing the original 0.215 req/s/GPU Fixed-PD point with `rho=0.005`.
- A first materialization attempt rounded both `rho=0.005` and `rho=0.01` to the same `rho0p01` directory. Digest validation stopped before simulator launch; the invalid directories were retained with an `invalid-rho-label-collision` suffix. The label function now preserves up to 12 significant digits and has a regression test.
Current decision: wait for the v4 simulator state/ranking knees, then select only discriminating real-machine pilot points. No real latency result from vLLM 0.20.2 will be compared with the historical vLLM 0.20.0 baseline until the runtime-version gate is resolved.
Current decision: finish the v4 fixed-global-rate control, then reuse its trained predictors for the low-load per-GPU sweep before selecting discriminating real-machine pilot points. No real latency result from vLLM 0.20.2 will be compared with the historical vLLM 0.20.0 baseline until the runtime-version gate is resolved.

33
runs/frontier-workload-regime-taxonomy-v0/jobs_simulator_per_gpu_v1.toml Normal file
View File

@@ -0,0 +1,33 @@
version = 1
[[jobs]]
name = "workload-regime-per-gpu-sim-tp1-v1-20260720"
gpus = 1
gpu_model = "H20"
hosts = ["dash1"]
command = "TRACES_ROOT=/home/admin/cpfs/wjh/aituner/workload-regime-20260720/traces-per-gpu-low SIM_GENERATION=per-gpu-v1 bash runs/frontier-workload-regime-taxonomy-v0/run_simulator_group.sh 1 both"
artifacts = []
[[jobs]]
name = "workload-regime-per-gpu-sim-tp2-v1-20260720"
gpus = 1
gpu_model = "H20"
hosts = ["dash2"]
command = "TRACES_ROOT=/home/admin/cpfs/wjh/aituner/workload-regime-20260720/traces-per-gpu-low SIM_GENERATION=per-gpu-v1 bash runs/frontier-workload-regime-taxonomy-v0/run_simulator_group.sh 2 both"
artifacts = []
[[jobs]]
name = "workload-regime-per-gpu-sim-tp4-noprefix-v1-20260720"
gpus = 1
gpu_model = "H20"
hosts = ["dash3"]
command = "TRACES_ROOT=/home/admin/cpfs/wjh/aituner/workload-regime-20260720/traces-per-gpu-low SIM_GENERATION=per-gpu-v1 bash runs/frontier-workload-regime-taxonomy-v0/run_simulator_group.sh 4 false"
artifacts = []
[[jobs]]
name = "workload-regime-per-gpu-sim-tp4-prefix-v1-20260720"
gpus = 1
gpu_model = "H20"
hosts = ["dash4"]
command = "TRACES_ROOT=/home/admin/cpfs/wjh/aituner/workload-regime-20260720/traces-per-gpu-low SIM_GENERATION=per-gpu-v1 bash runs/frontier-workload-regime-taxonomy-v0/run_simulator_group.sh 4 true"
artifacts = []

7
runs/frontier-workload-regime-taxonomy-v0/materialize_workload_regimes.py
View File

@@ -91,6 +91,10 @@ def vector_sha256(rows: Iterable[dict[str, Any]]) -> str:
return digest.hexdigest()
def rho_label(rho: float) -> str:
return f"rho{format(rho, '.12g').replace('.', 'p')}"
def load_source(
public_path: Path | None, private_path: Path
) -> list[dict[str, Any]]:
@@ -377,8 +381,7 @@ def main() -> None:
target_rate=target_rate,
fixed_token_ids=fixed_token_ids,
)
rho_label = f"rho{rho:.2f}".replace(".", "p")
root = args.output_root / family.name / rho_label
root = args.output_root / family.name / rho_label(rho)
cases.append(
write_case(
root,

134
runs/frontier-workload-regime-taxonomy-v0/project_per_gpu_load.py Normal file
View File

@@ -0,0 +1,134 @@
#!/usr/bin/env python3
"""Project base workload cases to a constant per-GPU request-rate contract."""
from __future__ import annotations
import argparse
import csv
import hashlib
import json
from pathlib import Path
from typing import Any
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser()
parser.add_argument("--source-manifest", type=Path, required=True)
parser.add_argument("--output-root", type=Path, required=True)
parser.add_argument("--tp", type=int, action="append", default=None)
return parser.parse_args()
def sha256(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as source:
for chunk in iter(lambda: source.read(1 << 20), b""):
digest.update(chunk)
return digest.hexdigest()
def rho_label(rho: float) -> str:
return format(rho, ".12g").replace(".", "p")
def project_case(
case: dict[str, Any], output_root: Path, tp: int
) -> dict[str, Any]:
source_public = Path(case["public_csv"])
source_private = Path(case["private_jsonl"])
if sha256(source_public) != case["public_csv_sha256"]:
raise ValueError(f"source public digest mismatch: {source_public}")
if sha256(source_private) != case["private_jsonl_sha256"]:
raise ValueError(f"source private digest mismatch: {source_private}")
case_root = output_root / f"tp{tp}" / case["family"] / f"rho{rho_label(case['rho'])}"
public_root = case_root / "public"
private_root = case_root / "private"
public_root.mkdir(parents=True, exist_ok=True)
private_root.mkdir(parents=True, exist_ok=True)
public_path = public_root / "frontier.csv"
private_path = private_root / "real_requests.jsonl"
with source_public.open(newline="") as source:
reader = csv.DictReader(source)
rows = list(reader)
fieldnames = reader.fieldnames
if not rows or fieldnames is None:
raise ValueError(f"empty public source: {source_public}")
with public_path.open("w", newline="") as output:
writer = csv.DictWriter(output, fieldnames=fieldnames, lineterminator="\n")
writer.writeheader()
for row in rows:
row["arrived_at"] = f"{float(row['arrived_at']) / tp:.12f}"
writer.writerow(row)
private_rows = [
json.loads(line) for line in source_private.open() if line.strip()
]
if len(private_rows) != len(rows):
raise ValueError(f"public/private row mismatch: {source_public}")
with private_path.open("w") as output:
for row in private_rows:
row["arrived_at"] = float(row["arrived_at"]) / tp
output.write(json.dumps(row, separators=(",", ":")) + "\n")
projected = dict(case)
projected.update(
{
"load_contract": "constant_per_gpu_request_rate",
"tp": tp,
"rho_per_gpu": case["rho"],
"per_gpu_offered_request_rate": case["global_offered_request_rate"],
"global_offered_request_rate": case["global_offered_request_rate"] * tp,
"empirical_interarrival_rate": case["empirical_interarrival_rate"] * tp,
"decode_offered_tokens_per_second_per_gpu": case[
"decode_offered_tokens_per_second"
],
"decode_offered_tokens_per_second": case[
"decode_offered_tokens_per_second"
]
* tp,
"last_arrival_s": case["last_arrival_s"] / tp,
"source_case_public_csv": str(source_public),
"source_case_public_csv_sha256": case["public_csv_sha256"],
"source_case_private_jsonl": str(source_private),
"source_case_private_jsonl_sha256": case["private_jsonl_sha256"],
"public_csv": str(public_path.resolve()),
"public_csv_sha256": sha256(public_path),
"private_jsonl": str(private_path.resolve()),
"private_jsonl_sha256": sha256(private_path),
}
)
(public_root / "manifest.json").write_text(
json.dumps(projected, indent=2, sort_keys=True) + "\n"
)
return projected
def main() -> None:
args = parse_args()
tps = args.tp or [1, 2, 4]
if any(tp not in (1, 2, 4) for tp in tps):
raise ValueError("TP must be 1, 2, or 4")
suite = json.loads(args.source_manifest.read_text())
projected = [
project_case(case, args.output_root, tp)
for tp in tps
for case in suite["cases"]
]
output = {
"schema": "frontier-workload-regime-per-gpu-v1",
"load_contract": "constant_per_gpu_request_rate",
"source_manifest": str(args.source_manifest.resolve()),
"source_manifest_sha256": sha256(args.source_manifest),
"tensor_parallel_sizes": tps,
"cases": projected,
}
args.output_root.mkdir(parents=True, exist_ok=True)
path = args.output_root / "manifest.json"
path.write_text(json.dumps(output, indent=2, sort_keys=True) + "\n")
print(path)
if __name__ == "__main__":
main()

74
runs/frontier-workload-regime-taxonomy-v0/run_real_pilot.sh Executable file
View File

@@ -0,0 +1,74 @@
#!/usr/bin/env bash
set -euo pipefail
FAMILY="${1:?usage: run_real_pilot.sh FAMILY RHO TP MNS SERVER_PORT}"
RHO="${2:?usage: run_real_pilot.sh FAMILY RHO TP MNS SERVER_PORT}"
TP="${3:?usage: run_real_pilot.sh FAMILY RHO TP MNS SERVER_PORT}"
MNS="${4:?usage: run_real_pilot.sh FAMILY RHO TP MNS SERVER_PORT}"
SERVER_PORT="${5:?usage: run_real_pilot.sh FAMILY RHO TP MNS SERVER_PORT}"
case "${TP}" in
1|2|4) ;;
*) echo "ERROR: TP must be 1, 2, or 4" >&2; exit 2 ;;
esac
case "${MNS}" in
8|16|32|64) ;;
*) echo "ERROR: MNS must be 8, 16, 32, or 64" >&2; exit 2 ;;
esac
EXPERIMENT_ROOT="${EXPERIMENT_ROOT:-/home/admin/cpfs/wjh/aituner/workload-regime-20260720}"
TRACES_MANIFEST="${TRACES_MANIFEST:-${EXPERIMENT_ROOT}/traces/manifest.json}"
VENV_ROOT="${VENV_ROOT:-/home/admin/cpfs/wjh/venvs/vllm-0.20.0-cu129-workload-regime}"
RUNNER_ROOT="${RUNNER_ROOT:-/home/admin/cpfs/wjh/aituner/aituner-graph-piecewise-bdc357d/runs/frontier-fidelity-envelope-v1}"
RUNNER="${RUNNER_ROOT}/run_qwen30_exact_trace_real_anchor.sh"
MODEL_ROOT="${MODEL_ROOT:-/home/admin/cpfs/wjh/models/Qwen/Qwen3-30B-A3B}"
OUTPUT_GENERATION="${OUTPUT_GENERATION:-v1}"
for path in "${TRACES_MANIFEST}" "${VENV_ROOT}/bin/vllm" "${RUNNER}" \
"${MODEL_ROOT}/config.json"; do
[[ -e "${path}" ]] || { echo "ERROR: missing ${path}" >&2; exit 1; }
done
mapfile -t CASE_FIELDS < <(
jq -r --arg family "${FAMILY}" --argjson rho "${RHO}" '
.cases
| map(select(.family == $family and ((.rho - $rho) | fabs) < 1e-12))
| .[]
| [.private_jsonl, (.prefix_caching | tostring), .private_jsonl_sha256]
| @tsv
' "${TRACES_MANIFEST}"
)
[[ "${#CASE_FIELDS[@]}" -eq 1 ]] || {
echo "ERROR: expected one manifest case for family=${FAMILY} rho=${RHO}, got ${#CASE_FIELDS[@]}" >&2
exit 1
}
IFS=$'\t' read -r REQUESTS_FILE PREFIX_CACHING EXPECTED_REQUESTS_SHA \
<<< "${CASE_FIELDS[0]}"
[[ -f "${REQUESTS_FILE}" ]] || { echo "ERROR: missing ${REQUESTS_FILE}" >&2; exit 1; }
ACTUAL_REQUESTS_SHA="$(sha256sum "${REQUESTS_FILE}" | cut -d' ' -f1)"
[[ "${ACTUAL_REQUESTS_SHA}" == "${EXPECTED_REQUESTS_SHA}" ]] || {
echo "ERROR: private trace digest mismatch" >&2
exit 1
}
RHO_LABEL="${RHO//./p}"
TRACE_LABEL="${FAMILY}-rho${RHO_LABEL}-tp${TP}-mns${MNS}"
OUTPUT_ROOT="${EXPERIMENT_ROOT}/real-pilot-${OUTPUT_GENERATION}/${TRACE_LABEL}"
[[ ! -e "${OUTPUT_ROOT}/results/result.json" ]] || {
echo "ERROR: refusing to overwrite completed ${OUTPUT_ROOT}" >&2
exit 1
}
FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE:-/tmp/wjh/flashinfer-workload-regime-vllm020-cu129}"
mkdir -p "${FLASHINFER_WORKSPACE_BASE}"
echo "WORKLOAD_REGIME_REAL_PILOT family=${FAMILY} rho=${RHO} tp=${TP} mns=${MNS} prefix=${PREFIX_CACHING} requests_sha256=${ACTUAL_REQUESTS_SHA} venv=${VENV_ROOT} output=${OUTPUT_ROOT}"
cd "${RUNNER_ROOT}"
OUTPUT_ROOT="${OUTPUT_ROOT}" REQUESTS_FILE="${REQUESTS_FILE}" \
TP="${TP}" MNS="${MNS}" TRACE_LABEL="${TRACE_LABEL}" \
SERVER_PORT="${SERVER_PORT}" PREFIX_CACHING="${PREFIX_CACHING}" \
VENV_ROOT="${VENV_ROOT}" MODEL_ROOT="${MODEL_ROOT}" \
FLASHINFER_WORKSPACE_BASE="${FLASHINFER_WORKSPACE_BASE}" \
SERVER_READY_ATTEMPTS="${SERVER_READY_ATTEMPTS:-300}" \
bash "${RUNNER}"

8
runs/frontier-workload-regime-taxonomy-v0/run_simulator_group.sh
View File

@@ -47,9 +47,9 @@ run_mode() {
fi
mapfile -t families < <(
jq -r --argjson prefix "${prefix_mode}" '
jq -r --argjson prefix "${prefix_mode}" --argjson tp "${TP}" '
.cases
| map(select(.prefix_caching == $prefix))
| map(select(.prefix_caching == $prefix and ((.tp // $tp) == $tp)))
| map(.family)
| unique
| .[]
@@ -75,9 +75,9 @@ run_mode() {
traces=()
command=()
mapfile -t traces < <(
jq -r --arg family "${family}" '
jq -r --arg family "${family}" --argjson tp "${TP}" '
.cases
| map(select(.family == $family))
| map(select(.family == $family and ((.tp // $tp) == $tp)))
| sort_by(.global_offered_request_rate)
| .[]
| "\(.family)-rho\(.rho | tostring | gsub("\\."; "p"))=\(.public_csv)"

6
tests/test_materialize_workload_regimes.py
View File

@@ -102,3 +102,9 @@ def test_private_source_projects_only_complete_prefix_blocks(tmp_path):
assert rows[0]["runtime_block_ids"] == [101]
assert rows[0]["source_arrived_at"] == 1.5
def test_rho_labels_do_not_collide_below_one_percent():
assert MODULE.rho_label(0.0025) == "rho0p0025"
assert MODULE.rho_label(0.005) == "rho0p005"
assert MODULE.rho_label(0.01) == "rho0p01"

67
tests/test_project_per_gpu_load.py Normal file
View File

@@ -0,0 +1,67 @@
import csv
import importlib.util
import json
import sys
from pathlib import Path
SCRIPT = (
Path(__file__).resolve().parents[1]
/ "runs/frontier-workload-regime-taxonomy-v0/project_per_gpu_load.py"
)
SPEC = importlib.util.spec_from_file_location("project_per_gpu_load", SCRIPT)
MODULE = importlib.util.module_from_spec(SPEC)
assert SPEC.loader is not None
sys.modules[SPEC.name] = MODULE
SPEC.loader.exec_module(MODULE)
def test_project_case_scales_only_arrivals_and_load_metadata(tmp_path):
public_path = tmp_path / "source.csv"
with public_path.open("w", newline="") as output:
writer = csv.DictWriter(
output,
fieldnames=("arrived_at", "num_prefill_tokens", "num_decode_tokens"),
)
writer.writeheader()
writer.writerow(
{"arrived_at": "0", "num_prefill_tokens": "16", "num_decode_tokens": "8"}
)
writer.writerow(
{"arrived_at": "4", "num_prefill_tokens": "32", "num_decode_tokens": "12"}
)
private_path = tmp_path / "source.jsonl"
private_path.write_text(
"\n".join(
json.dumps({"arrived_at": arrival, "body": {"prompt": index}})
for index, arrival in enumerate((0.0, 4.0))
)
+ "\n"
)
case = {
"family": "fixture",
"rho": 0.01,
"global_offered_request_rate": 0.25,
"empirical_interarrival_rate": 0.25,
"decode_offered_tokens_per_second": 2.5,
"last_arrival_s": 4.0,
"public_csv": str(public_path),
"public_csv_sha256": MODULE.sha256(public_path),
"private_jsonl": str(private_path),
"private_jsonl_sha256": MODULE.sha256(private_path),
}
projected = MODULE.project_case(case, tmp_path / "output", 4)
with Path(projected["public_csv"]).open(newline="") as source:
public_rows = list(csv.DictReader(source))
private_rows = [
json.loads(line) for line in Path(projected["private_jsonl"]).open()
]
assert [float(row["arrived_at"]) for row in public_rows] == [0.0, 1.0]
assert [row["arrived_at"] for row in private_rows] == [0.0, 1.0]
assert [row["num_decode_tokens"] for row in public_rows] == ["8", "12"]
assert projected["per_gpu_offered_request_rate"] == 0.25
assert projected["global_offered_request_rate"] == 1.0
assert projected["decode_offered_tokens_per_second"] == 10.0
assert projected["last_arrival_s"] == 1.0