feat(policy): load-floor bonus for KvAwarePolicy (Q2.B)

Implements the design proposed and approved in
docs/E1_E2_FIX_DESIGN_ZH.md §Q2.B.

KvAwarePolicy gains a `load_floor_bonus: int = 0` knob. When > 0:

  mean_assigned = sum(assigned[*]) / len(D)
  for each D candidate:
    if not sticky and mean_assigned > 0:
      deficit = max(0, mean_assigned - assigned[D])
      floor_bonus = K * deficit / mean_assigned
    else:
      floor_bonus = 0
    score = (overlap + sticky*α + floor_bonus, sticky, -inflight, -assigned)

Properties (verified by unit-style probe in commit message):
- Default 0 = old behavior preserved
- Sticky-gated: turn-1+ requests of an existing session keep going
  to their original D (cache locality preserved)
- Graduated: bonus magnitude scales with the D's deficit ratio,
  approaches K as deficit/mean → 1, drops to 0 when balanced
- Set above max expected boilerplate overlap (Inferact ~50 → 200)
  so cross-session shared-prefix overlap doesn't pin cold D's idle,
  but real per-session prefix overlap (>K blocks) still wins

Plumbed through ReplayConfig, BenchmarkConfig, and CLI flag
--kvcache-load-floor-bonus on both `replay` and `benchmark-live`.

Empirical verification on synthetic state (same conditions as the
E2 cold-D pathology):
  - OFF (K=0):   route fresh session → decode-0 (boilerplate winner)
  - ON  (K=200): route fresh session → decode-1 (cold D rebalanced)

Validation pass next: scripts/sweep_e3_kvc_v2_loadfloor_rdma.sh
(committed separately).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
tim
2026-05-12 11:45:09 +08:00
parent 905d671135
commit 93fce42747
4 changed files with 86 additions and 4 deletions

View File

@@ -111,6 +111,11 @@ class ReplayConfig:
# KvAwarePolicy skips that D for the session (forcing migration). Default 3.
# Set 0 to disable. See REFACTOR_PLAN_V1 §6.2.
kvcache_migration_reject_threshold: int = 3
# Load-floor bonus magnitude for KvAwarePolicy: graduated boost added to
# under-loaded D workers to break overlap-pinning imbalance on workloads
# with shared cross-session prefix. 0 disables. See
# docs/E1_E2_FIX_DESIGN_ZH.md §Q2.
kvcache_load_floor_bonus: int = 0
structural_log_dir: Path | None = None
@@ -198,6 +203,7 @@ async def replay_trace(config: ReplayConfig) -> list[RequestMetrics]:
policy = create_policy(
config.policy_name,
migration_reject_threshold=config.kvcache_migration_reject_threshold,
load_floor_bonus=config.kvcache_load_floor_bonus,
)
state = RoutingState.create(config.topology)
state_lock = asyncio.Lock()