gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Connector tax: trace-replay confirms +45% kv_both penalty is gone; DR-fix adds 22% more

Browse Source 
Re-runs the elastic_migration_v2 trace (w600 r0.0015 st30, 1214 reqs,
274 sessions, 8×TP1 vLLM + cache_aware_proxy) with three configs:
- plain unified
- unified + Mooncake kv_both
- unified + Mooncake kv_both + DR-fix (env-gated O(|cache|) hash sync removal)

TTFT p90: 11.97 s → 9.74 s (−18.6%) → 7.58 s (−36.6% vs plain)
E2E p90:  23.48 s → 21.25 s (−9.5%) → 17.93 s (−23.6% vs plain)

Two findings:
1. The "+45% kv_both penalty" claim from elastic_migration_v2 is OBSOLETE
   on current codebase — kv_both is now *faster* than plain at p90.
   Likely fixed by e3a1d70 (RDMA-READ → bootstrap PUSH refactor) and
   the connector-mode delay_free_blocks extending cross-turn prefix
   cache hits on a 93%-intra-session-reuse trace.
2. DR-fix removes another 22% from TTFT p90 by skipping the
   O(|cache|) hash sync in build_connector_meta. Cache-sweep with
   DR-fix shows slope drops from +94.5 to +2.3 μs/1k blocks.

Adds:
- run_trace_replay_drfix.sh: A/B/C harness (env CT_DR_FIX gates patch)
- analyze_trace_replay.py: TTFT/TPOT/E2E delta analysis
- REPORT_TRACE_REPLAY.md: summary + reproduction
- results/20260526_1627_drfix/: cache-sweep with DR-fix
- results/trace_replay_20260526_1652/: full trace-replay A/B/C

Implication for EAR paper: the kv_both substrate is no longer the
bottleneck blocking session migration. The prior 4 migration reverts
were dominated by transfer overhead that has now been characterized
and (partially) removed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-27 09:13:50 +08:00
parent df0ee5a02b
commit ef9e0102ec
34 changed files with 38636 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

51
microbench/connector_tax/cache_sweep/run_cache_sweep.py
View File

@@ -56,21 +56,25 @@ class ReqMetric:
error: str | None = None
def make_random_prompt(target_tokens: int) -> str:
def make_random_prompt(target_tokens: int, rng: random.Random) -> str:
"""Same calibration as the bench_loop.py used elsewhere:
'Block N: <32-hex>' tokenizes to ~35 tokens on Qwen3-Coder."""
'Block N: <32-hex>' tokenizes to ~35 tokens on Qwen3-Coder.
Deterministic given `rng` — same RNG state produces the same prompt.
Used so two runs with the same --seed get bit-identical request streams.
"""
n_parts = max(1, target_tokens // 35)
seed = uuid.uuid4().hex
# 32 random hex chars from rng (uuid.uuid4 would use os.urandom, unseedable)
seed = "".join(f"{rng.randrange(16):x}" for _ in range(32))
parts = []
for i in range(n_parts):
h = hashlib.md5(f"{seed}_{i}_{time.time_ns()}".encode()).hexdigest()
h = hashlib.md5(f"{seed}_{i}".encode()).hexdigest()
parts.append(f"Block {i}: {h}")
return " ".join(parts)
async def send_one(client, url, model, inp_tokens, out_tokens,
rate, inflight, inflight_cap, fh):
rid = uuid.uuid4().hex[:16]
async def send_one(client, url, model, prompt, inp_tokens, out_tokens,
rate, inflight, inflight_cap, fh, rid):
if inflight[0] >= inflight_cap:
m = ReqMetric(req_id=rid, rate_target=rate,
input_tokens_target=inp_tokens,
@@ -88,7 +92,6 @@ async def send_one(client, url, model, inp_tokens, out_tokens,
t_send_ns=time.perf_counter_ns(),
inflight_at_send=inflight[0])
try:
prompt = make_random_prompt(inp_tokens)
payload = {
"model": model,
"messages": [{"role": "user", "content": prompt}],
@@ -141,25 +144,39 @@ async def main_async(args):
inflight = [0]
pending: list[asyncio.Task] = []
interval_mean = 1.0 / args.rate
rng = random.Random(int(time.time_ns()) & 0xFFFFFFFF)
print(f"[bench] rate={args.rate} shape=({args.input_tokens},{args.output_tokens}) "
f"duration={args.duration}s output={out_dir}")
# Shared seed across configs gives bit-identical arrival times AND prompt
# content. arrival_rng feeds expovariate, content_rng feeds make_random_prompt,
# rid_rng feeds the per-request id. All three derive from --seed so two runs
# with the same seed are bit-identical from the producer side; only server
# response timing differs (which is what we want to measure).
if args.seed is not None:
seed = args.seed
else:
seed = int(time.time_ns()) & 0xFFFFFFFF
print(f"[bench] seed={seed} rate={args.rate} shape=({args.input_tokens},"
f"{args.output_tokens}) duration={args.duration}s output={out_dir}")
arrival_rng = random.Random(seed)
content_rng = random.Random(seed ^ 0xC0FFEE)
rid_rng = random.Random(seed ^ 0xDEADBEEF)
fh = open(req_path, "a", buffering=1)
t0 = time.perf_counter()
last_print = t0
async with httpx.AsyncClient(timeout=httpx.Timeout(600.0)) as client:
# producer
# producer — prompts generated here in order so async scheduling can't
# reorder the content RNG draw across requests.
async def producer():
while time.perf_counter() - t0 < args.duration:
prompt = make_random_prompt(args.input_tokens, content_rng)
rid = f"{rid_rng.randrange(1 << 64):016x}"
pending.append(asyncio.create_task(
send_one(client, args.url, args.model,
send_one(client, args.url, args.model, prompt,
args.input_tokens, args.output_tokens,
args.rate, inflight, args.inflight_cap, fh)
args.rate, inflight, args.inflight_cap, fh, rid)
))
await asyncio.sleep(rng.expovariate(1.0 / interval_mean))
await asyncio.sleep(arrival_rng.expovariate(1.0 / interval_mean))
# heartbeat
async def heartbeat():
@@ -210,6 +227,10 @@ def main():
ap.add_argument("--output-tokens", type=int, default=256)
ap.add_argument("--duration", type=float, default=480.0,
help="Total run duration in seconds (default 8 min)")
ap.add_argument("--seed", type=int, default=None,
help="Master seed; same value across configs gives "
"bit-identical Poisson arrivals and prompt content. "
"Default: time-based (different each run).")
ap.add_argument("--inflight-cap", type=int, default=256)
ap.add_argument("--output-dir", required=True)
args = ap.parse_args()