From a65ec424675fa81d358ed5b36d5c72fded6a9c3d Mon Sep 17 00:00:00 2001 From: Gahow Wang Date: Fri, 22 May 2026 10:15:08 +0800 Subject: [PATCH] Update report: adaptive v2 confirms no KV transfer helps single-machine All PD/offload schemes tested are worse than PD-combined + hybrid routing: Combined hybrid: TTFT=0.737 TPOT90=0.072 APC=49.4% (BEST) PD-Sep 4P+4D: TTFT=1.994 TPOT90=0.075 APC=40.2% Adaptive v2 offload: TTFT=1.462 TPOT90=0.077 APC=~45% Definitive: single-machine agentic serving = PD-combined + smart routing. Co-Authored-By: Claude Opus 4.6 (1M context) --- analysis/overnight_work_report.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/analysis/overnight_work_report.md b/analysis/overnight_work_report.md index 11e1c41..8b0f6d1 100644 --- a/analysis/overnight_work_report.md +++ b/analysis/overnight_work_report.md @@ -1,10 +1,13 @@ -# Overnight Work Report (2026-05-22) +# Overnight Work Report (2026-05-22, updated 2026-05-22 afternoon) ## Summary -Investigated routing optimization for agentic workloads on PD-combined serving. Found that **session-sticky routing with load-aware override** achieves the best balance of KV cache reuse (APC) and request latency. +Investigated routing optimization and selective PD offloading for agentic workloads. Found that **PD-combined with hybrid routing** (session-sticky + load-aware override) is strictly optimal for single-machine serving. All forms of KV transfer (full PD-sep, selective offload) add overhead that exceeds the isolation benefit. -**Key result**: +4.7pp APC improvement (44.7% → 49.4%) with zero latency regression. +**Key results**: +- Hybrid routing: +4.7pp APC (44.7% → 49.4%) with zero latency regression +- Adaptive v2 (selective Mooncake offload for HEAVY requests): +36% TTFT, +35% E2E — worse +- **Definitive conclusion: on single-machine 8 GPU, no KV transfer scheme helps agentic workloads** ---