feat(policy): load-floor bonus for KvAwarePolicy (Q2.B)
Implements the design proposed and approved in
docs/E1_E2_FIX_DESIGN_ZH.md §Q2.B.
KvAwarePolicy gains a `load_floor_bonus: int = 0` knob. When > 0:
mean_assigned = sum(assigned[*]) / len(D)
for each D candidate:
if not sticky and mean_assigned > 0:
deficit = max(0, mean_assigned - assigned[D])
floor_bonus = K * deficit / mean_assigned
else:
floor_bonus = 0
score = (overlap + sticky*α + floor_bonus, sticky, -inflight, -assigned)
Properties (verified by unit-style probe in commit message):
- Default 0 = old behavior preserved
- Sticky-gated: turn-1+ requests of an existing session keep going
to their original D (cache locality preserved)
- Graduated: bonus magnitude scales with the D's deficit ratio,
approaches K as deficit/mean → 1, drops to 0 when balanced
- Set above max expected boilerplate overlap (Inferact ~50 → 200)
so cross-session shared-prefix overlap doesn't pin cold D's idle,
but real per-session prefix overlap (>K blocks) still wins
Plumbed through ReplayConfig, BenchmarkConfig, and CLI flag
--kvcache-load-floor-bonus on both `replay` and `benchmark-live`.
Empirical verification on synthetic state (same conditions as the
E2 cold-D pathology):
- OFF (K=0): route fresh session → decode-0 (boilerplate winner)
- ON (K=200): route fresh session → decode-1 (cold D rebalanced)
Validation pass next: scripts/sweep_e3_kvc_v2_loadfloor_rdma.sh
(committed separately).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -48,6 +48,7 @@ class BenchmarkConfig:
|
||||
enable_backpressure: bool = False
|
||||
backpressure_max_pause_s: float = 2.0
|
||||
kvcache_migration_reject_threshold: int = 3
|
||||
kvcache_load_floor_bonus: int = 0
|
||||
sample_profile: str = "default"
|
||||
min_initial_input_tokens: int | None = None
|
||||
max_initial_input_tokens: int | None = None
|
||||
@@ -200,6 +201,7 @@ def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
|
||||
enable_backpressure=config.enable_backpressure,
|
||||
backpressure_max_pause_s=config.backpressure_max_pause_s,
|
||||
kvcache_migration_reject_threshold=config.kvcache_migration_reject_threshold,
|
||||
kvcache_load_floor_bonus=config.kvcache_load_floor_bonus,
|
||||
)
|
||||
if config.request_timeout_s is not None:
|
||||
replay_config = replace(
|
||||
@@ -261,6 +263,7 @@ def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
|
||||
"enable_backpressure": config.enable_backpressure,
|
||||
"backpressure_max_pause_s": config.backpressure_max_pause_s,
|
||||
"kvcache_migration_reject_threshold": config.kvcache_migration_reject_threshold,
|
||||
"kvcache_load_floor_bonus": config.kvcache_load_floor_bonus,
|
||||
"sample_profile": config.sample_profile,
|
||||
"min_initial_input_tokens": config.min_initial_input_tokens,
|
||||
"max_initial_input_tokens": config.max_initial_input_tokens,
|
||||
|
||||
Reference in New Issue
Block a user