gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Remove second push_new gate that caused downgrade-to-cold-LOCAL

Browse Source 
After _push_allowed was relaxed, the cost model correctly chose push
for high-cache sessions on overloaded instances. But a second gate at
execution time (push_new < heavy_threshold) blocked the actual offload,
downgrading to LOCAL on the target instance — which had no cache.
Worse, session affinity was already updated to the target, so all
subsequent turns also hit cold prefill.

This was the root cause of relaxed gate's performance regression:
affinity broken + push blocked = worst of both worlds.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-25 00:42:31 +08:00
parent 3ae99293fd
commit 1d2148cf65
1 changed files with 0 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
scripts/cache_aware_proxy.py
View File

@@ -613,16 +613,6 @@ async def _handle_combined(api, req_data, token_ids, input_length, session_id, h
return await _handle_local_request(
api, req_data, headers, token_ids, input_length,
chosen, estimated_new, breakdown)
if push_new < SETTINGS.heavy_threshold:
breakdown["push_downgraded"] = "below_heavy_threshold"
return await _handle_local_request(
api, req_data, headers, token_ids, input_length,
chosen, estimated_new, breakdown)
if SETTINGS.cache_gate_ratio > 0 and cache_ratio < SETTINGS.cache_gate_ratio:
breakdown["push_downgraded"] = "cache_gate"
return await _handle_local_request(
api, req_data, headers, token_ids, input_length,
chosen, estimated_new, breakdown)
offload_mode = getattr(global_args, 'offload_mode', 'cached_prefill')
breakdown["c_inst"] = c_inst.url