feat(policy): cold-D bonus to break overlap-pinning death spiral
KvAwarePolicy now accepts an optional cold_d_bonus int. When > 0, fresh requests (sticky=0, i.e. no prior D for this session) receive the bonus added to lex-score position 0 (overlap+sticky_bonus) for any D worker that has never been assigned a session yet (decode_assignment_counts == 0). This breaks the pathology documented in docs/E1_E2_RESULTS_ZH.md §5d where workloads with shared cross-session prefix (e.g. Inferact's "permissions instructions" boilerplate) cause every D that has hosted any session to dominate the overlap term against any cold D, leaving the cold D permanently unused. Sticky behavior is preserved: turn 1+ requests of an existing session continue to stick to their original D because the bonus is gated on `not sticky`. Plumbed through ReplayConfig.kvcache_cold_d_bonus (default 0, keeping current behavior unchanged), BenchmarkConfig, and CLI flag --kvcache-cold-d-bonus on both `replay` and `benchmark-live` subcommands. Set above max expected boilerplate overlap (Inferact's ~50 24-token blocks → 1000 is safe). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -111,6 +111,10 @@ class ReplayConfig:
|
||||
# KvAwarePolicy skips that D for the session (forcing migration). Default 3.
|
||||
# Set 0 to disable. See REFACTOR_PLAN_V1 §6.2.
|
||||
kvcache_migration_reject_threshold: int = 3
|
||||
# Cold-D bonus: synthetic boost to lex-score position 0 for any D worker
|
||||
# that has never been assigned a session yet, applied only when the request
|
||||
# has no sticky preference. 0 disables. See docs/E1_E2_RESULTS_ZH.md §5d.
|
||||
kvcache_cold_d_bonus: int = 0
|
||||
structural_log_dir: Path | None = None
|
||||
|
||||
|
||||
@@ -198,6 +202,7 @@ async def replay_trace(config: ReplayConfig) -> list[RequestMetrics]:
|
||||
policy = create_policy(
|
||||
config.policy_name,
|
||||
migration_reject_threshold=config.kvcache_migration_reject_threshold,
|
||||
cold_d_bonus=config.kvcache_cold_d_bonus,
|
||||
)
|
||||
state = RoutingState.create(config.topology)
|
||||
state_lock = asyncio.Lock()
|
||||
|
||||
Reference in New Issue
Block a user