feat(policy): load-floor bonus for KvAwarePolicy (Q2.B)
Implements the design proposed and approved in
docs/E1_E2_FIX_DESIGN_ZH.md §Q2.B.
KvAwarePolicy gains a `load_floor_bonus: int = 0` knob. When > 0:
mean_assigned = sum(assigned[*]) / len(D)
for each D candidate:
if not sticky and mean_assigned > 0:
deficit = max(0, mean_assigned - assigned[D])
floor_bonus = K * deficit / mean_assigned
else:
floor_bonus = 0
score = (overlap + sticky*α + floor_bonus, sticky, -inflight, -assigned)
Properties (verified by unit-style probe in commit message):
- Default 0 = old behavior preserved
- Sticky-gated: turn-1+ requests of an existing session keep going
to their original D (cache locality preserved)
- Graduated: bonus magnitude scales with the D's deficit ratio,
approaches K as deficit/mean → 1, drops to 0 when balanced
- Set above max expected boilerplate overlap (Inferact ~50 → 200)
so cross-session shared-prefix overlap doesn't pin cold D's idle,
but real per-session prefix overlap (>K blocks) still wins
Plumbed through ReplayConfig, BenchmarkConfig, and CLI flag
--kvcache-load-floor-bonus on both `replay` and `benchmark-live`.
Empirical verification on synthetic state (same conditions as the
E2 cold-D pathology):
- OFF (K=0): route fresh session → decode-0 (boilerplate winner)
- ON (K=200): route fresh session → decode-1 (cold D rebalanced)
Validation pass next: scripts/sweep_e3_kvc_v2_loadfloor_rdma.sh
(committed separately).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -111,6 +111,11 @@ class ReplayConfig:
|
||||
# KvAwarePolicy skips that D for the session (forcing migration). Default 3.
|
||||
# Set 0 to disable. See REFACTOR_PLAN_V1 §6.2.
|
||||
kvcache_migration_reject_threshold: int = 3
|
||||
# Load-floor bonus magnitude for KvAwarePolicy: graduated boost added to
|
||||
# under-loaded D workers to break overlap-pinning imbalance on workloads
|
||||
# with shared cross-session prefix. 0 disables. See
|
||||
# docs/E1_E2_FIX_DESIGN_ZH.md §Q2.
|
||||
kvcache_load_floor_bonus: int = 0
|
||||
structural_log_dir: Path | None = None
|
||||
|
||||
|
||||
@@ -198,6 +203,7 @@ async def replay_trace(config: ReplayConfig) -> list[RequestMetrics]:
|
||||
policy = create_policy(
|
||||
config.policy_name,
|
||||
migration_reject_threshold=config.kvcache_migration_reject_threshold,
|
||||
load_floor_bonus=config.kvcache_load_floor_bonus,
|
||||
)
|
||||
state = RoutingState.create(config.topology)
|
||||
state_lock = asyncio.Lock()
|
||||
|
||||
Reference in New Issue
Block a user