gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(policy): cold-D bonus to break overlap-pinning death spiral

Browse Source 
KvAwarePolicy now accepts an optional cold_d_bonus int. When > 0,
fresh requests (sticky=0, i.e. no prior D for this session) receive
the bonus added to lex-score position 0 (overlap+sticky_bonus) for
any D worker that has never been assigned a session yet
(decode_assignment_counts == 0). This breaks the pathology
documented in docs/E1_E2_RESULTS_ZH.md §5d where workloads with
shared cross-session prefix (e.g. Inferact's "permissions
instructions" boilerplate) cause every D that has hosted any session
to dominate the overlap term against any cold D, leaving the cold D
permanently unused.

Sticky behavior is preserved: turn 1+ requests of an existing
session continue to stick to their original D because the bonus is
gated on `not sticky`.

Plumbed through ReplayConfig.kvcache_cold_d_bonus (default 0,
keeping current behavior unchanged), BenchmarkConfig, and CLI flag
--kvcache-cold-d-bonus on both `replay` and `benchmark-live`
subcommands. Set above max expected boilerplate overlap (Inferact's
~50 24-token blocks → 1000 is safe).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
tim
2026-05-12 11:14:00 +08:00
parent bf4da281c0
commit 786cbb8d91
4 changed files with 67 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
src/agentic_pd_hybrid/benchmark.py
View File

@@ -48,6 +48,7 @@ class BenchmarkConfig:
enable_backpressure: bool = False
backpressure_max_pause_s: float = 2.0
kvcache_migration_reject_threshold: int = 3
kvcache_cold_d_bonus: int = 0
sample_profile: str = "default"
min_initial_input_tokens: int | None = None
max_initial_input_tokens: int | None = None
@@ -200,6 +201,7 @@ def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
enable_backpressure=config.enable_backpressure,
backpressure_max_pause_s=config.backpressure_max_pause_s,
kvcache_migration_reject_threshold=config.kvcache_migration_reject_threshold,
kvcache_cold_d_bonus=config.kvcache_cold_d_bonus,
)
if config.request_timeout_s is not None:
replay_config = replace(
@@ -261,6 +263,7 @@ def run_live_benchmark(config: BenchmarkConfig) -> BenchmarkArtifacts:
"enable_backpressure": config.enable_backpressure,
"backpressure_max_pause_s": config.backpressure_max_pause_s,
"kvcache_migration_reject_threshold": config.kvcache_migration_reject_threshold,
"kvcache_cold_d_bonus": config.kvcache_cold_d_bonus,
"sample_profile": config.sample_profile,
"min_initial_input_tokens": config.min_initial_input_tokens,
"max_initial_input_tokens": config.max_initial_input_tokens,