feat(policy): load-floor bonus for KvAwarePolicy (Q2.B)

Implements the design proposed and approved in
docs/E1_E2_FIX_DESIGN_ZH.md §Q2.B.

KvAwarePolicy gains a `load_floor_bonus: int = 0` knob. When > 0:

  mean_assigned = sum(assigned[*]) / len(D)
  for each D candidate:
    if not sticky and mean_assigned > 0:
      deficit = max(0, mean_assigned - assigned[D])
      floor_bonus = K * deficit / mean_assigned
    else:
      floor_bonus = 0
    score = (overlap + sticky*α + floor_bonus, sticky, -inflight, -assigned)

Properties (verified by unit-style probe in commit message):
- Default 0 = old behavior preserved
- Sticky-gated: turn-1+ requests of an existing session keep going
  to their original D (cache locality preserved)
- Graduated: bonus magnitude scales with the D's deficit ratio,
  approaches K as deficit/mean → 1, drops to 0 when balanced
- Set above max expected boilerplate overlap (Inferact ~50 → 200)
  so cross-session shared-prefix overlap doesn't pin cold D's idle,
  but real per-session prefix overlap (>K blocks) still wins

Plumbed through ReplayConfig, BenchmarkConfig, and CLI flag
--kvcache-load-floor-bonus on both `replay` and `benchmark-live`.

Empirical verification on synthetic state (same conditions as the
E2 cold-D pathology):
  - OFF (K=0):   route fresh session → decode-0 (boilerplate winner)
  - ON  (K=200): route fresh session → decode-1 (cold D rebalanced)

Validation pass next: scripts/sweep_e3_kvc_v2_loadfloor_rdma.sh
(committed separately).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
tim
2026-05-12 11:45:09 +08:00
parent 905d671135
commit 93fce42747
4 changed files with 86 additions and 4 deletions

View File

@@ -48,6 +48,7 @@ class BenchmarkConfig:
enable_backpressure: bool = False
backpressure_max_pause_s: float = 2.0
kvcache_migration_reject_threshold: int = 3
kvcache_load_floor_bonus: int = 0
sample_profile: str = "default"
min_initial_input_tokens: int | None = None
max_initial_input_tokens: int | None = None
@@ -200,6 +201,7 @@ def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
enable_backpressure=config.enable_backpressure,
backpressure_max_pause_s=config.backpressure_max_pause_s,
kvcache_migration_reject_threshold=config.kvcache_migration_reject_threshold,
kvcache_load_floor_bonus=config.kvcache_load_floor_bonus,
)
if config.request_timeout_s is not None:
replay_config = replace(
@@ -261,6 +263,7 @@ def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
"enable_backpressure": config.enable_backpressure,
"backpressure_max_pause_s": config.backpressure_max_pause_s,
"kvcache_migration_reject_threshold": config.kvcache_migration_reject_threshold,
"kvcache_load_floor_bonus": config.kvcache_load_floor_bonus,
"sample_profile": config.sample_profile,
"min_initial_input_tokens": config.min_initial_input_tokens,
"max_initial_input_tokens": config.max_initial_input_tokens,