gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

P2P prefill offload: TTFT p50 -13% but p90 +59% (median-vs-tail tradeoff)

Browse Source 
Fixed race condition in P instance selection (all going to inst_0).
P2P design: HEAVY requests prefill on least-loaded OTHER instance,
KV transfer via Mooncake, decode on session-sticky instance.

Result (200 req, fresh restart, vs baseline):
  TTFT p50: 1.080 -> 0.939 (-13%)   <- median improves (decode not disrupted)
  TTFT p90: 9.410 -> 14.987 (+59%)  <- tail worsens (KV transfer on large req)
  TPOT p90: 0.076 -> 0.075 (-1%)    <- unchanged (not the bottleneck)
  E2E p50: 5.306 -> 5.565 (+5%)     <- slightly worse overall

The P2P offload helps the common case (WARM/MEDIUM get lower TTFT because
their instance isn't blocked by a heavy prefill) but hurts HEAVY requests
(extra KV transfer latency). This is a median-vs-tail tradeoff.

For SLOs targeting p50: P2P offload helps.
For SLOs targeting p90/p99: baseline combined is better.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-22 12:28:24 +08:00
parent 7f93d36970
commit 1b9268ba4c
4 changed files with 290 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
scripts/cache_aware_proxy.py
View File

@@ -229,23 +229,29 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
"t_proxy_recv": _time.monotonic(),
}
use_offload = (estimated_new >= HEAVY_THRESHOLD and global_args.offload
and len(combined_instances) >= 2)
offload_enabled = getattr(global_args, 'offload', False) if global_args else False
use_offload = (estimated_new >= HEAVY_THRESHOLD and offload_enabled
and len(combined_instances) >= 2
and any(inst.bootstrap_port for inst in combined_instances))
if use_offload:
# HEAVY with offload: P on least-loaded, D on session-sticky (best_inst)
p_inst = min(combined_instances, key=lambda x: x.ongoing_tokens)
# HEAVY P2P OFFLOAD: D on session-sticky instance, P on a DIFFERENT
# least-loaded instance (any instance can serve as P for others).
d_inst = best_inst
if p_inst is d_inst:
# Pick second-least-loaded for P
sorted_by_load = sorted(combined_instances, key=lambda x: x.ongoing_tokens)
p_inst = sorted_by_load[0] if sorted_by_load[0] is not d_inst else sorted_by_load[1]
d_idx = best_idx
breakdown["route_class"] = "HEAVY_OFFLOAD"
# P instance: least ongoing_tokens EXCLUDING D.
# CRITICAL: increment ongoing_tokens IMMEDIATELY to prevent race condition
# where multiple concurrent HEAVY requests all pick the same P instance.
p_candidates = [inst for inst in combined_instances if inst is not d_inst]
p_inst = min(p_candidates, key=lambda x: x.ongoing_tokens)
p_inst.ongoing_tokens += input_length # reserve immediately
breakdown["route_class"] = "HEAVY_P2P"
breakdown["p_inst"] = p_inst.url
breakdown["d_inst"] = d_inst.url
if session_id:
session_affinity[session_id] = combined_instances.index(d_inst)
session_affinity[session_id] = d_idx
return await _handle_heavy_offload(api, req_data, headers, token_ids,
input_length, p_inst, d_inst, breakdown)
@@ -285,8 +291,7 @@ async def _handle_heavy_offload(api, req_data, headers, token_ids, input_length,
"""HEAVY request: prefill on p_inst, KV via Mooncake, decode on d_inst."""
request_id = headers.get("X-Request-Id", "")
# Step 1: Await prefill on p_inst
p_inst.ongoing_tokens += input_length
# Step 1: Await prefill on p_inst (ongoing_tokens already reserved by caller)
breakdown["t_prefill_sent"] = _time.monotonic()
try:
prefill_data = req_data.copy()