gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update report: adaptive v2 confirms no KV transfer helps single-machine

Browse Source 
All PD/offload schemes tested are worse than PD-combined + hybrid routing:
  Combined hybrid:    TTFT=0.737  TPOT90=0.072  APC=49.4%  (BEST)
  PD-Sep 4P+4D:       TTFT=1.994  TPOT90=0.075  APC=40.2%
  Adaptive v2 offload: TTFT=1.462  TPOT90=0.077  APC=~45%

Definitive: single-machine agentic serving = PD-combined + smart routing.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-22 10:15:08 +08:00
parent 2fee355626
commit a65ec42467
1 changed files with 6 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
analysis/overnight_work_report.md
View File

@@ -1,10 +1,13 @@
# Overnight Work Report (2026-05-22)
# Overnight Work Report (2026-05-22, updated 2026-05-22 afternoon)
## Summary
Investigated routing optimization for agentic workloads on PD-combined serving. Found that **session-sticky routing with load-aware override** achieves the best balance of KV cache reuse (APC) and request latency.
Investigated routing optimization and selective PD offloading for agentic workloads. Found that **PD-combined with hybrid routing** (session-sticky + load-aware override) is strictly optimal for single-machine serving. All forms of KV transfer (full PD-sep, selective offload) add overhead that exceeds the isolation benefit.
**Key result**: +4.7pp APC improvement (44.7% → 49.4%) with zero latency regression.
**Key results**:
- Hybrid routing: +4.7pp APC (44.7% → 49.4%) with zero latency regression
- Adaptive v2 (selective Mooncake offload for HEAVY requests): +36% TTFT, +35% E2E — worse
- **Definitive conclusion: on single-machine 8 GPU, no KV transfer scheme helps agentic workloads**
---