feat(policy): cold-D bonus to break overlap-pinning death spiral

KvAwarePolicy now accepts an optional cold_d_bonus int. When > 0,
fresh requests (sticky=0, i.e. no prior D for this session) receive
the bonus added to lex-score position 0 (overlap+sticky_bonus) for
any D worker that has never been assigned a session yet
(decode_assignment_counts == 0). This breaks the pathology
documented in docs/E1_E2_RESULTS_ZH.md §5d where workloads with
shared cross-session prefix (e.g. Inferact's "permissions
instructions" boilerplate) cause every D that has hosted any session
to dominate the overlap term against any cold D, leaving the cold D
permanently unused.

Sticky behavior is preserved: turn 1+ requests of an existing
session continue to stick to their original D because the bonus is
gated on `not sticky`.

Plumbed through ReplayConfig.kvcache_cold_d_bonus (default 0,
keeping current behavior unchanged), BenchmarkConfig, and CLI flag
--kvcache-cold-d-bonus on both `replay` and `benchmark-live`
subcommands. Set above max expected boilerplate overlap (Inferact's
~50 24-token blocks → 1000 is safe).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
tim
2026-05-12 11:14:00 +08:00
parent bf4da281c0
commit 786cbb8d91
4 changed files with 67 additions and 3 deletions

View File

@@ -111,6 +111,10 @@ class ReplayConfig:
# KvAwarePolicy skips that D for the session (forcing migration). Default 3.
# Set 0 to disable. See REFACTOR_PLAN_V1 §6.2.
kvcache_migration_reject_threshold: int = 3
# Cold-D bonus: synthetic boost to lex-score position 0 for any D worker
# that has never been assigned a session yet, applied only when the request
# has no sticky preference. 0 disables. See docs/E1_E2_RESULTS_ZH.md §5d.
kvcache_cold_d_bonus: int = 0
structural_log_dir: Path | None = None
@@ -198,6 +202,7 @@ async def replay_trace(config: ReplayConfig) -> list[RequestMetrics]:
policy = create_policy(
config.policy_name,
migration_reject_threshold=config.kvcache_migration_reject_threshold,
cold_d_bonus=config.kvcache_cold_d_bonus,
)
state = RoutingState.create(config.topology)
state_lock = asyncio.Lock()