feat(kvc): honor admission backpressure hints + structural event logging
Replay-side changes paired with the SGLang admission hint: - DecodeResidencyState gains pause_until_s; admission probe parses recommended_pause_ms and updates the per-D pause window. - _wait_for_decode_pause is invoked at request entry points (_invoke_router, _invoke_session_direct) so requests stall before hitting a saturated D instead of timing out via mooncake. - New CLI flags: --enable-backpressure (default off, baseline preserved), --backpressure-max-pause-s (cap on per-request sleep, default 2s). Structural instrumentation written under <run_dir>/structural/: - admission-events.jsonl: every admission probe (RTT, queue_depth, pause_ms, available_tokens, evicted_count) - backpressure-events.jsonl: every actual pause sleep - session-d-binding.jsonl: per-request policy decision Used to validate the structural claims documented separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -245,6 +245,21 @@ def main() -> None:
|
||||
"Disable per-session detail in the pool timeseries (smaller files)."
|
||||
),
|
||||
)
|
||||
replay.add_argument(
|
||||
"--enable-backpressure",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Honor recommended_pause_ms hints from D's admission endpoint. "
|
||||
"When set, replay sleeps before issuing requests to a saturated D. "
|
||||
"Default off — preserves baseline behavior."
|
||||
),
|
||||
)
|
||||
replay.add_argument(
|
||||
"--backpressure-max-pause-s",
|
||||
type=float,
|
||||
default=2.0,
|
||||
help="Cap on per-request backpressure sleep, regardless of D hint.",
|
||||
)
|
||||
|
||||
sample = subparsers.add_parser(
|
||||
"sample-sessions",
|
||||
@@ -473,6 +488,19 @@ def main() -> None:
|
||||
"Disable per-session detail in the pool timeseries (smaller files)."
|
||||
),
|
||||
)
|
||||
benchmark.add_argument(
|
||||
"--enable-backpressure",
|
||||
action="store_true",
|
||||
help=(
|
||||
"Honor recommended_pause_ms hints from D's admission endpoint."
|
||||
),
|
||||
)
|
||||
benchmark.add_argument(
|
||||
"--backpressure-max-pause-s",
|
||||
type=float,
|
||||
default=2.0,
|
||||
help="Cap on per-request backpressure sleep, regardless of D hint.",
|
||||
)
|
||||
benchmark.add_argument(
|
||||
"--sample-profile",
|
||||
choices=["default", "small-append"],
|
||||
@@ -556,6 +584,8 @@ def main() -> None:
|
||||
kvcache_prefill_normal_priority=args.kvcache_prefill_normal_priority,
|
||||
pool_poll_interval_s=args.pool_poll_interval_s,
|
||||
pool_poll_include_sessions=not args.pool_poll_no_sessions,
|
||||
enable_backpressure=args.enable_backpressure,
|
||||
backpressure_max_pause_s=args.backpressure_max_pause_s,
|
||||
)
|
||||
results = asyncio.run(replay_trace(config))
|
||||
print(
|
||||
@@ -700,6 +730,8 @@ def main() -> None:
|
||||
),
|
||||
pool_poll_interval_s=args.pool_poll_interval_s,
|
||||
pool_poll_include_sessions=not args.pool_poll_no_sessions,
|
||||
enable_backpressure=args.enable_backpressure,
|
||||
backpressure_max_pause_s=args.backpressure_max_pause_s,
|
||||
sample_profile=args.sample_profile,
|
||||
min_initial_input_tokens=args.min_initial_input_tokens,
|
||||
max_initial_input_tokens=args.max_initial_input_tokens,
|
||||
|
||||
Reference in New Issue
Block a user