gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Proxy/runner support for Nixl connector + unified_v3 (offload-decode) policy

Browse Source 
scripts/b3_isolated_policy.sh:
  Recognize unified_v3 as a kv_both-requiring policy; respect explicit
  KV_CONNECTOR=Nixl override (so unified_v2 / unified_v3 / unified_kv_both
  can run against either Mooncake or Nixl back-end). When Nixl is
  selected, skip the bootstrap-ports plumbing — Nixl uses its own UCX
  side-channel and the proxy forwards kv_transfer_params from the src
  response body instead of pre-baking engine_id/bootstrap_addr.

scripts/cache_aware_proxy.py:
  - New unified_v3 policy (~250 lines): prefill stays on session-affinity
    host (preserves intra-session prefix-cache reuse), decode is migrated
    to a lower-load target when the affinity host is busy with concurrent
    decodes. KV transfer flows prefill_host → decode_target, opposite of
    v2. Knobs: v3_min_new_tokens, v3_min_prefill_decode_busy,
    v3_target_load_ratio, v3_min_load_gap, v3_rotate_affinity,
    v3_prefer_cache_target. cache_miss_audit found rotation hurts cross-
    turn locality (9.5% hit with vs ~80% without) so default
    v3_rotate_affinity=False.
  - New connector_type setting ("mooncake" | "nixl") gating the PD-sep
    handshake form: mooncake uses pre-baked kv_transfer_params,
    nixl forwards them from the response body.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-27 22:05:19 +08:00
parent da39ab6804
commit f739f7d461
2 changed files with 283 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
scripts/b3_isolated_policy.sh
View File

@@ -32,10 +32,14 @@ RUNDIR="${3:?usage: $0 <policy> <trace> <rundir>}"
# Auto-enable kv_both when the policy requires it.
# KV_CONNECTOR (Mooncake|Nixl) selects the underlying connector when KV_BOTH=1.
_KV_CONNECTOR_EXPLICIT="${KV_CONNECTOR:-}"
KV_CONNECTOR="${KV_CONNECTOR:-Mooncake}"
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_kv_both" ]; then
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_v3" ] || [ "$POLICY" = "unified_kv_both" ]; then
ENABLE_KV_BOTH=1
KV_CONNECTOR="Mooncake"
# honor explicit KV_CONNECTOR override (e.g. =Nixl); otherwise default Mooncake.
if [ -z "$_KV_CONNECTOR_EXPLICIT" ]; then
KV_CONNECTOR="Mooncake"
fi
fi
if [ "$POLICY" = "unified_nixl_both" ]; then
ENABLE_KV_BOTH=1
@@ -143,7 +147,10 @@ for i in $(seq 0 $((N_INSTANCES - 1))); do
combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))"
done
proxy_extra=""
if [ "$ENABLE_KV_BOTH" = "1" ]; then
# Bootstrap ports only needed for Mooncake handshake. Nixl uses its own
# UCX side-channel and the proxy forwards kv_transfer_params from src's
# response body instead of pre-baking engine_id/bootstrap_addr.
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Mooncake" ]; then
bp_list=""
for i in $(seq 0 $((N_INSTANCES - 1))); do
if [ -z "$bp_list" ]; then
@@ -154,11 +161,15 @@ if [ "$ENABLE_KV_BOTH" = "1" ]; then
done
proxy_extra="--bootstrap-ports $bp_list"
fi
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Nixl" ]; then
proxy_extra="--connector-type nixl"
fi
nohup "$VENV/python" "$ROOT/scripts/cache_aware_proxy.py" \
--port "$PROXY_PORT" \
--combined $combined_args \
--policy "$POLICY" \
$proxy_extra \
${EXTRA_PROXY_ARGS:-} \
> "$RUNDIR/proxy.log" 2>&1 &
disown
tries=0