Gahow Wang
020be9f444
M1: cached_blocks was a plain set with a "trim half via list slicing" eviction. CPython does not guarantee set iteration order, so the trim discarded an arbitrary half of the entries — completely unlike vLLM's LRU and a known contributor to the router's cache_hit estimate diverging from real APC. Replace with an OrderedDict-backed LRU: move_to_end on hits, popitem(last=False) on overflow. Capacity exposed as CACHE_CAPACITY_BLOCKS module constant (200000 by default). M5: streamed responses decrement load counters in their generator's finally block. If a client disconnects before consuming the body the generator is never entered and the decrement is lost, causing ongoing_tokens / num_requests / pending_prefill_tokens to drift negative under load. Add a 60s background reconcile_loop that clamps those counters at zero as a safety net. Started in lifespan, cancelled on shutdown. Does not replace proper vLLM exact-state syncing.