Correct PD-disagg cost/benefit framing across repo
The §3.2 cost-vs-benefit math in commits029821c(MB1 plot + pd_cost_vs_benefit.png) andabde010(RESULTS_SUMMARY.md) was wrong. What was wrong: I framed PD-disagg's max phase-isolation benefit as "≤ decode duration of the new request (~50–200 ms)" — implicitly treating the benefit as per-request and bounded by that request's own decode. The correct accounting is per-prefill-event across all stalled streams: benefit_per_prefill = D × T_prefill × (1 − TPOT_baseline/TPOT_during) ≈ D × T_prefill which the user pointed out follows from the chunked-prefill math (each of L/N chunks slows D ongoing decode steps from ~10 ms to t ms, summing to D × T_prefill). Plug MB1 + MB2 numbers in: prefill size | T_prefill | T_transfer | D=8 benefit | cost/benefit 2k tok | 0.14 s | 8 ms | 1.1 s | 0.7 % 33k tok | 4.5 s | 320 ms | 36 s | 0.9 % 125k tok | 57 s | 1.9 s | 456 s | 0.4 % On the phase-isolation axis alone, PD-disagg WINS by 100×–250× — the opposite of what the deleted figure showed. The actual dominant reason static PD-disagg fails in agentic is the D-side KV pool capacity wall (figs/f4b_pdsep_kv_wall.png) — p99 single-request KV is 11.5 GiB, per-D-instance pool is 38 GiB, so 4P+4D halves system decode capacity. Colleague's 4P+4D experiment showed TTFT p50 62× worse and success rate 99.5% → 52%, driven by pool overflow + queueing, not by transfer latency. Changes: - figs/pd_cost_vs_benefit.png : DELETED (figure built on wrong math) - microbench/fresh_setup/plot_mb1.py : drop the pd_cost_vs_benefit function; keep mb1_interference.png and update its title to note per-prefill aggregate stall = D × T_prefill (not capped by decode) - figs/mb1_interference.png : regenerated, no misleading band annotation - analysis/mb1/README.md : Summary block rewritten ("what MB1 measures"; no more "max benefit = decode duration" claim); §3.2 implications section replaced with the corrected per-prefill-event table; explicit ⚠ Correction note documents what was wrong - analysis/mb2/README.md : Summary block + §3.2 implications section rewritten the same way; ⚠ Correction note links to RESULTS_SUMMARY §4 - RESULTS_SUMMARY.md §4 + §6 : §4 reordered to lead with the D-side capacity argument (the real failure mode), MB1/MB2 demoted from "kill-shot for PD-disagg" to "supporting context inputs to a cost-benefit table that actually favors PD-disagg on this axis"; §6 paper-claims list reordered to remove the wrong "PD-disagg loses on cost-vs-benefit" claim and replace with the corrected ones PAPER_OUTLINE.md and MEETING.md were checked and never picked up this specific wrong claim — they already (correctly) frame §3.2 around the D-side KV memory wall. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -32,10 +32,14 @@ RUNDIR="${3:?usage: $0 <policy> <trace> <rundir>}"
|
||||
|
||||
# Auto-enable kv_both when the policy requires it.
|
||||
# KV_CONNECTOR (Mooncake|Nixl) selects the underlying connector when KV_BOTH=1.
|
||||
_KV_CONNECTOR_EXPLICIT="${KV_CONNECTOR:-}"
|
||||
KV_CONNECTOR="${KV_CONNECTOR:-Mooncake}"
|
||||
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_kv_both" ]; then
|
||||
if [ "$POLICY" = "unified_v2" ] || [ "$POLICY" = "unified_v3" ] || [ "$POLICY" = "unified_kv_both" ]; then
|
||||
ENABLE_KV_BOTH=1
|
||||
KV_CONNECTOR="Mooncake"
|
||||
# honor explicit KV_CONNECTOR override (e.g. =Nixl); otherwise default Mooncake.
|
||||
if [ -z "$_KV_CONNECTOR_EXPLICIT" ]; then
|
||||
KV_CONNECTOR="Mooncake"
|
||||
fi
|
||||
fi
|
||||
if [ "$POLICY" = "unified_nixl_both" ]; then
|
||||
ENABLE_KV_BOTH=1
|
||||
@@ -143,7 +147,10 @@ for i in $(seq 0 $((N_INSTANCES - 1))); do
|
||||
combined_args="$combined_args http://127.0.0.1:$((BASE_PORT + i))"
|
||||
done
|
||||
proxy_extra=""
|
||||
if [ "$ENABLE_KV_BOTH" = "1" ]; then
|
||||
# Bootstrap ports only needed for Mooncake handshake. Nixl uses its own
|
||||
# UCX side-channel and the proxy forwards kv_transfer_params from src's
|
||||
# response body instead of pre-baking engine_id/bootstrap_addr.
|
||||
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Mooncake" ]; then
|
||||
bp_list=""
|
||||
for i in $(seq 0 $((N_INSTANCES - 1))); do
|
||||
if [ -z "$bp_list" ]; then
|
||||
@@ -154,11 +161,15 @@ if [ "$ENABLE_KV_BOTH" = "1" ]; then
|
||||
done
|
||||
proxy_extra="--bootstrap-ports $bp_list"
|
||||
fi
|
||||
if [ "$ENABLE_KV_BOTH" = "1" ] && [ "$KV_CONNECTOR" = "Nixl" ]; then
|
||||
proxy_extra="--connector-type nixl"
|
||||
fi
|
||||
nohup "$VENV/python" "$ROOT/scripts/cache_aware_proxy.py" \
|
||||
--port "$PROXY_PORT" \
|
||||
--combined $combined_args \
|
||||
--policy "$POLICY" \
|
||||
$proxy_extra \
|
||||
${EXTRA_PROXY_ARGS:-} \
|
||||
> "$RUNDIR/proxy.log" 2>&1 &
|
||||
disown
|
||||
tries=0
|
||||
|
||||
Reference in New Issue
Block a user