H7 OVERLOAD_FACTOR sweep: negative result + H4 GPU profiling
H7: Sweeping OVERLOAD_FACTOR (2.0/1.5/1.3/1.0) has no effect on GPU imbalance (~3.5-4x across all settings). Root cause: imbalance is from workload skew at session placement (turn 1), not from routing at turn 2+. H4 GPU profiling confirms: GPU balance improvement IS real (4.0x→2.0x), and it directly improves HEAVY_COLO TTFT by 10.5%. But RDMA-offloaded requests have bimodal transfer times (0.6s or 18-31s) that negate the routing benefit. Updated elastic_hypotheses.md with H7 results and next directions: higher load experiments where contention amplifies routing differences. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -28,7 +28,7 @@ from fastapi.responses import StreamingResponse
|
||||
BLOCK_SIZE = 512
|
||||
CACHE_HIT_ALPHA = 1.0
|
||||
HEAVY_THRESHOLD = 20000 # default; overridden by --heavy-threshold
|
||||
OVERLOAD_FACTOR = 2.0
|
||||
OVERLOAD_FACTOR = 2.0 # default; overridden by --overload-factor
|
||||
|
||||
|
||||
class InstanceState:
|
||||
@@ -627,6 +627,8 @@ def parse_args():
|
||||
help="Comma-separated bootstrap ports for combined instances (for offload mode)")
|
||||
p.add_argument("--policy", type=str, default="linear", choices=["linear", "lmetric"],
|
||||
help="Routing policy: linear (default) or lmetric (P_tokens × BS, OSDI'26)")
|
||||
p.add_argument("--overload-factor", type=float, default=2.0,
|
||||
help="Break session affinity when instance load > factor * avg")
|
||||
args = p.parse_args()
|
||||
|
||||
args.prefill = []
|
||||
@@ -645,4 +647,5 @@ def parse_args():
|
||||
if __name__ == "__main__":
|
||||
global_args = parse_args()
|
||||
HEAVY_THRESHOLD = global_args.heavy_threshold
|
||||
OVERLOAD_FACTOR = global_args.overload_factor
|
||||
uvicorn.run(app, host=global_args.host, port=global_args.port)
|
||||
|
||||
Reference in New Issue
Block a user