feat(kvc): Option D - delegate seed/reseed admission to D worker

v4 (cap=16) saw 35% session-cap fallback because the local soft_cap
min(16, usable / target) evaluates to 1-2 for large agentic inputs.
The cap was hit not because D was full but because replay's heuristic
underestimated capacity.

This change makes worker admission_mode authoritative for ALL paths:

SGLang side:
- io_struct.py: DirectAppendAdmissionReqInput gains a `mode` field
  ("direct_append" | "seed", default "direct_append" preserves prior
  behavior).
- scheduler.py:admit_direct_append: when mode == "seed", skip the
  resident-on-D requirement and run the same capacity check + LRU
  eviction (maybe_trim_decode_session_cache) that direct_append uses.
  This lets D atomically decide if a new session can be admitted based
  on actual token_to_kv_pool_allocator state.

Replay side (replay.py):
- _query_decode_direct_admission gains a `mode` parameter.
- _reserve_decode_session_capacity: in worker admission_mode, the
  seed/reseed branch now queries D with mode="seed" and trusts the
  result, instead of estimating capacity from the residency snapshot.
- _should_admit_new_decode_session: in worker mode, skip the local
  soft_cap pre-check and let D decide. Same-D session fast-path is
  preserved.

Effects:
- Local hardcoded cap of 16 is bypassed under worker mode; D's real
  KV pool size is the only constraint.
- LRU eviction runs in D's process atomically with admission, so
  starvation (the v3 bimodal "lucky vs starved sessions" pattern)
  should resolve.

scripts/sweep_tp1_v5_optD.sh added to run the same 1P7D / 2P6D
configs as v4 with the new admission path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kzlin
2026-04-28 23:40:03 +08:00
parent 74194e660a
commit 6e5ed8da80
4 changed files with 173 additions and 2 deletions

View File

@@ -1602,6 +1602,9 @@ class DirectAppendAdmissionReqInput(BaseReq):
session_id: str
uncached_input_tokens: int
output_tokens: int
# "direct_append": existing behavior — require session resident on this D
# "seed": new admission for session not yet resident; do capacity check + LRU eviction
mode: str = "direct_append"
@dataclass

View File

@@ -3508,6 +3508,9 @@ class Scheduler(
reason="unsupported",
)
mode = getattr(recv_req, "mode", "direct_append") or "direct_append"
is_seed = mode == "seed"
session_cache_status = self.session_controller.get_streaming_session_cache_status(
recv_req.session_id
)
@@ -3515,7 +3518,9 @@ class Scheduler(
resident = bool(
isinstance(target_session, dict) and target_session.get("resident")
)
if not resident:
if not resident and not is_seed:
# direct_append requires the session already resident on this D.
# For seed we skip this check and let capacity decide.
return DirectAppendAdmissionReqOutput(
can_admit=False,
resident=False,
@@ -3543,10 +3548,13 @@ class Scheduler(
0, recv_req.output_tokens
)
available_tokens_before = int(self.token_to_kv_pool_allocator.available_size())
# Don't evict the session itself when it's already resident; for seed
# of a fresh session there is nothing to exclude.
exclude_ids = {recv_req.session_id} if resident else set()
trim_result = self.maybe_trim_decode_session_cache(
required_tokens=required_tokens,
force=available_tokens_before < required_tokens,
exclude_session_ids={recv_req.session_id},
exclude_session_ids=exclude_ids,
)
available_tokens_after = int(self.token_to_kv_pool_allocator.available_size())
decode_retracted_queue_reqs = len(self.disagg_decode_prealloc_queue.retracted_queue)