gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add leastwork_kappa decode-aware ablation (net-negative, documented)

Browse Source 
--policy leastwork_kappa + --kappa (default 2.5e-6, derived from KV ~100KB/tok
/ HBM 4TB/s / TPOT 10ms on H20+Qwen3-30B-A3B): score = prefill_work * (1 +
kappa * ongoing_decode_tokens), modelling decode as a fractional throughput tax
on a new prefill.

Result on the 600s trace: NET-NEGATIVE vs plain leastwork — TTFT p90 +18%,
E2E p90 +14%, balance 1.55x->1.97x, and it does NOT fix the E2E-p99 it targeted.
Decode is too cheap in agentic (output p50~80) for the term to help; it just
bounces heavy reqs off their cache-owner into cold re-prefill. The E2E-p99 tail
is the structural HEAVY+>50k floor (per-class p99 ~51-52k for ALL policies), not
decode interference. Kept in-tree as a documented ablation justifying LPWL's
omission of any decode term; do not revive without a decode-heavy regime.
See analysis/lpwl_5policy_600s.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-29 17:07:23 +08:00
parent 71b0747b3b
commit a0db3cbe77
2 changed files with 94 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
analysis/lpwl_5policy_600s.md
View File

@@ -82,6 +82,33 @@ On the warmer full trace, LPWL vs unified+A+B is a wash: LPWL wins TTFT p90
(−14%) but loses TPOT (+38%) and per-worker balance. Combined claim across both
regimes: **LPWL ∈ [tied, clearly-better] vs a tuned baseline, at zero knobs.**
## Ablation: derived-κ decode term (`leastwork_kappa`) — NET-NEGATIVE
Tested the proposed knob-free fix for LPWL's E2E-p99: `--policy leastwork_kappa`,
`score = (pending_prefill + new_uncached) × (1 + κ·ongoing_decode_tokens)`, with
κ = 2.5e-6 *derived* from hardware (KV ~100 KB/tok ÷ HBM 4 TB/s ÷ TPOT 10 ms on
H20+Qwen3-30B-A3B), not trace-tuned. Same 600s trace, fresh vLLM, cold APC.
| metric | leastwork | leastwork_kappa | Δ |
|---|---:|---:|---:|
| TTFT p90 | 7983 | 9390 | +18% (worse) |
| TTFT p99 | 44891 | 42370 | −6% |
| E2E p90 | 19014 | 21674 | +14% (worse) |
| E2E p99 | 87024 | 90155 | +4% (did NOT fix) |
| APC | 0.648 | 0.647 | tie |
| req-balance | 1.55× | 1.97× | worse |
**Verdict: decode-awareness is the wrong lever for agentic.** The κ term is
correct physics aimed at a negligible effect (decode is cheap, output p50≈80),
so it mostly bounces heavy requests off their cache-owner → cold re-prefill
elsewhere → new hotspots (balance degrades 1.55×→1.97×). It does NOT fix E2E-p99
because that tail is the **structural HEAVY+>50k floor** (per-class p99 ≈51–52k
for *all* policies), not decode interference — i.e. not routing-fixable. This is
a negative result that *justifies* LPWL's omission of any decode term. The policy
is kept in-tree as a documented ablation; do not revive without a decode-heavy
regime. (First run on the GPU-capturing harness: per-worker GPU util mean 42–83%,
1.95× spread — it even shows the κ-induced imbalance.)
## Caveats / open work
- n=1 per arm. The 600s −31% TTFT p90 is corroborated by mean/p50/per-class, but