Q1: Mooncake "is not alive" is hair-trigger — a single
send_kvcache_slice ret != 0 in
third_party/sglang/python/sglang/srt/disaggregation/mooncake/conn.py
:1270 permanently adds the D's mooncake_session_id to failed_sessions
and blacklists it for the rest of the process lifetime. The D worker
process is alive (D1 keeps serving admit_direct_append OK seconds
after), but every subsequent P→D transfer for that session
short-circuits at conn.py:1184. The "Failures should never happen if
the session is not dead" comment encodes the wrong assumption for the
saturation regime we hit.
Q2: KVC v2's migration mechanism IS sound but its trigger is gated
by replay.py:1379 _ADMISSION_REJECTION_SUBSTRINGS = ("session-cap",
"no-d-capacity", "d-backpressure"). All 1054 failures have
execution_mode="kvcache-centric" (generic fallback bucket) which
contains none of those substrings, so session_d_rejects is never
incremented. Empirically 46 of 49 (sess, D) pairs that the worker
RPC rejected would have qualified for blacklist (most-rejected
pair: 25 rejects), but policy never saw them. Result: D0 reject
→ next-bind D0 (253×), D1 reject → next-bind D1 (329×), D0/D1 reject
→ next-bind D2 (0×).
Fix paths documented for both, shortest path is widening the
substring filter to include the failure-fallback bucket, but the
right fix is to call record_admission_reject directly from the
actual rejection signal site instead of string-matching execution_mode.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
7f2ebf3d87