gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(env): MC_TRANSFER_TIMEOUT=1800s default in setup_env + stack

Browse Source 
Mooncake C++ batch_transfer_sync defaults to 30s timeout; on
saturated D scheduler threads doing LRU eviction, that fires as a
false positive and the SGLang hair-trigger in conn.py:1270
permanently blacklists the D's mooncake_session_id (E2 forensic in
docs/E1_E2_RESULTS_ZH.md §5c). Bump to 1800s in setup_env.sh and
mirror to subprocess env in stack.py so SGLang workers get it too.
30-min envelope still detects genuinely broken peers eventually.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
tim
2026-05-12 11:45:09 +08:00
parent 9a166ac43b
commit 905d671135
3 changed files with 20 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
scripts/setup_env.sh
View File

@@ -30,6 +30,15 @@ export CUDA_HOME="$HOME/cuda-12.8"
export PATH="$HOME/cuda-12.8/bin:$PATH"
export LD_LIBRARY_PATH="$REPO_ROOT/.venv/lib/python3.12/site-packages/nvidia/cuda_runtime/lib:$HOME/cuda-12.8/lib64${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
# Mooncake batch_transfer_sync C++ timeout (seconds). Default in mooncake is
# 30 s; a single LRU eviction sweep on a saturated D scheduler can exceed
# that and cause the hair-trigger blacklist in conn.py:1270 to permanently
# mark the D's mooncake_session_id "failed". 1800 s = 30 min gives us
# headroom while still detecting genuinely broken peers eventually.
# See docs/E1_E2_RESULTS_ZH.md §5c and docs/E1_E2_FIX_DESIGN_ZH.md Q1.C.
export MC_TRANSFER_TIMEOUT="${MC_TRANSFER_TIMEOUT:-1800}"
echo "agentic-pd-hybrid env ready:"
echo " CUDA_HOME=$CUDA_HOME ($(nvcc --version | grep release | sed 's/.*release //'))"
echo " libcudart.so.12 at $REPO_ROOT/.venv/lib/python3.12/site-packages/nvidia/cuda_runtime/lib"
echo " MC_TRANSFER_TIMEOUT=${MC_TRANSFER_TIMEOUT}s"