feat(kvc): Option D - delegate seed/reseed admission to D worker
v4 (cap=16) saw 35% session-cap fallback because the local soft_cap
min(16, usable / target) evaluates to 1-2 for large agentic inputs.
The cap was hit not because D was full but because replay's heuristic
underestimated capacity.
This change makes worker admission_mode authoritative for ALL paths:
SGLang side:
- io_struct.py: DirectAppendAdmissionReqInput gains a `mode` field
("direct_append" | "seed", default "direct_append" preserves prior
behavior).
- scheduler.py:admit_direct_append: when mode == "seed", skip the
resident-on-D requirement and run the same capacity check + LRU
eviction (maybe_trim_decode_session_cache) that direct_append uses.
This lets D atomically decide if a new session can be admitted based
on actual token_to_kv_pool_allocator state.
Replay side (replay.py):
- _query_decode_direct_admission gains a `mode` parameter.
- _reserve_decode_session_capacity: in worker admission_mode, the
seed/reseed branch now queries D with mode="seed" and trusts the
result, instead of estimating capacity from the residency snapshot.
- _should_admit_new_decode_session: in worker mode, skip the local
soft_cap pre-check and let D decide. Same-D session fast-path is
preserved.
Effects:
- Local hardcoded cap of 16 is bypassed under worker mode; D's real
KV pool size is the only constraint.
- LRU eviction runs in D's process atomically with admission, so
starvation (the v3 bimodal "lucky vs starved sessions" pattern)
should resolve.
scripts/sweep_tp1_v5_optD.sh added to run the same 1P7D / 2P6D
configs as v4 with the new admission path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1602,6 +1602,9 @@ class DirectAppendAdmissionReqInput(BaseReq):
|
||||
session_id: str
|
||||
uncached_input_tokens: int
|
||||
output_tokens: int
|
||||
# "direct_append": existing behavior — require session resident on this D
|
||||
# "seed": new admission for session not yet resident; do capacity check + LRU eviction
|
||||
mode: str = "direct_append"
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
@@ -3508,6 +3508,9 @@ class Scheduler(
|
||||
reason="unsupported",
|
||||
)
|
||||
|
||||
mode = getattr(recv_req, "mode", "direct_append") or "direct_append"
|
||||
is_seed = mode == "seed"
|
||||
|
||||
session_cache_status = self.session_controller.get_streaming_session_cache_status(
|
||||
recv_req.session_id
|
||||
)
|
||||
@@ -3515,7 +3518,9 @@ class Scheduler(
|
||||
resident = bool(
|
||||
isinstance(target_session, dict) and target_session.get("resident")
|
||||
)
|
||||
if not resident:
|
||||
if not resident and not is_seed:
|
||||
# direct_append requires the session already resident on this D.
|
||||
# For seed we skip this check and let capacity decide.
|
||||
return DirectAppendAdmissionReqOutput(
|
||||
can_admit=False,
|
||||
resident=False,
|
||||
@@ -3543,10 +3548,13 @@ class Scheduler(
|
||||
0, recv_req.output_tokens
|
||||
)
|
||||
available_tokens_before = int(self.token_to_kv_pool_allocator.available_size())
|
||||
# Don't evict the session itself when it's already resident; for seed
|
||||
# of a fresh session there is nothing to exclude.
|
||||
exclude_ids = {recv_req.session_id} if resident else set()
|
||||
trim_result = self.maybe_trim_decode_session_cache(
|
||||
required_tokens=required_tokens,
|
||||
force=available_tokens_before < required_tokens,
|
||||
exclude_session_ids={recv_req.session_id},
|
||||
exclude_session_ids=exclude_ids,
|
||||
)
|
||||
available_tokens_after = int(self.token_to_kv_pool_allocator.available_size())
|
||||
decode_retracted_queue_reqs = len(self.disagg_decode_prealloc_queue.retracted_queue)
|
||||
|
||||
Reference in New Issue
Block a user