Microbench 3 (connector_tax): infrastructure for KV connector substrate tax

Validates the elastic_migration_v2 finding that kv_role=kv_both adds
TTFT p90 +45% even when PD-sep never fires. Replicates under
single-instance, synthetic, open-loop workload to disambiguate
mechanism cost from 8-instance feedback amplification.

Configurations (8):
  plain, noop_connector, mooncake_{producer,consumer,both},
  nixl_both, lmcache_only, multi_mooncake_lmcache.

Pre-flight verification gates risky configs (kv_consumer needs dummy
bootstrap, multi-connector composition, NoOp custom class loading).

Workload: two-phase sweep
  Phase A: rate {0.5..32} req/s × shape (4096, 256), saturation criteria
  Phase B: ref_safe rate × cartesian (input ∈ {512,4k,32k}, output ∈ {64,256,1024})

Step-timing patch enriches vLLM's existing AGENTIC_STEP_LOG_PATH emit
with step_duration_us and build_meta_us — directly measures per-step
substrate cost, not just user-visible TTFT/TPOT.

run_all.sh runs as 5-stage barrier:
  0 pre-flight + apply patch
  1 Phase A all configs
  2 pick ref_safe / ref_load
  3 Phase B all configs
  4 revert patch + analyze + plot

Outputs aggregate.{json,csv}, MANIFEST.tsv, and 5 figures.
Estimated runtime: 4-5.5 hours on idle dash0 H20.

microbench/connector_tax/tools/dummy_bootstrap.py

@@ -0,0 +1,84 @@
+"""Dummy Mooncake bootstrap server for kv_consumer pre-flight.
+
+Exposes the same HTTP routes as MooncakeBootstrapServer but returns
+empty / accepting responses. Allows a kv_consumer vLLM to start up
+without a real prefiller behind it.
+
+Usage:
+  python dummy_bootstrap.py --port 8997
+"""
+
+import argparse
+import logging
+import threading
+
+from fastapi import FastAPI, Request
+from fastapi.responses import JSONResponse
+import uvicorn
+
+logging.basicConfig(level=logging.INFO)
+log = logging.getLogger("dummy_bootstrap")
+
+
+def make_app() -> FastAPI:
+    app = FastAPI()
+    state = {"workers": {}, "hash_table": {}}
+
+    @app.post("/register")
+    async def register_worker(req: Request):
+        body = await req.json()
+        log.info("register_worker: %s", body)
+        # Pretend success
+        dp_rank = int(body.get("dp_rank", 0))
+        engine_id = body.get("engine_id", "dummy-engine")
+        state["workers"][dp_rank] = {
+            "engine_id": engine_id,
+            "worker_addr": body.get("worker_addr", {}),
+        }
+        return JSONResponse({"status": "ok"})
+
+    @app.get("/query")
+    async def query():
+        # Return whatever we have. Empty {} is acceptable for the consumer
+        # because no PD-sep request will actually trigger a pull.
+        return JSONResponse(state["workers"])
+
+    @app.post("/query_blocks")
+    async def query_blocks(req: Request):
+        return JSONResponse({"matched_blocks": []})
+
+    @app.post("/unpin_blocks")
+    async def unpin_blocks(req: Request):
+        return JSONResponse({"status": "ok"})
+
+    @app.post("/push_blocks")
+    async def push_blocks(req: Request):
+        return JSONResponse({"status": "ok"})
+
+    @app.post("/estimate_hit")
+    async def estimate_hit(req: Request):
+        return JSONResponse({"hit_tokens": 0})
+
+    @app.get("/health")
+    async def health():
+        return JSONResponse({"status": "ok"})
+
+    return app
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--host", default="127.0.0.1")
+    ap.add_argument("--port", type=int, default=8997)
+    args = ap.parse_args()
+
+    app = make_app()
+    config = uvicorn.Config(app=app, host=args.host, port=args.port,
+                            log_level="info")
+    server = uvicorn.Server(config)
+    log.info("Dummy Mooncake bootstrap listening on %s:%d", args.host, args.port)
+    server.run()
+
+
+if __name__ == "__main__":
+    main()

microbench/connector_tax/tools/noop_connector.py

@@ -0,0 +1,90 @@
+"""Pure no-op KV connector for measuring vLLM v1 framework overhead.
+
+This connector implements every abstract hook of KVConnectorBase_V1 with
+the cheapest possible no-op return. Loaded via:
+
+  --kv-transfer-config '{
+    "kv_connector_module_path":
+        "microbench.connector_tax.tools.noop_connector:NoOpConnector",
+    "kv_role": "kv_both"
+  }'
+
+It does:
+  - no I/O
+  - no per-step cache key walk
+  - no per-layer save/load
+  - no metadata serialization beyond an empty dataclass
+
+So `tax(NoOpConnector) ≈ pure vLLM v1 framework overhead`.
+"""
+
+from typing import TYPE_CHECKING, Any
+
+from vllm.distributed.kv_transfer.kv_connector.v1.base import (
+    KVConnectorBase_V1,
+    KVConnectorMetadata,
+)
+
+if TYPE_CHECKING:
+    import torch
+    from vllm.attention.backends.abstract import AttentionMetadata
+    from vllm.forward_context import ForwardContext
+    from vllm.v1.core.kv_cache_manager import KVCacheBlocks
+    from vllm.v1.core.sched.output import SchedulerOutput
+    from vllm.v1.request import Request
+
+
+class NoOpConnector(KVConnectorBase_V1):
+    """Empty connector — every hook is a no-op.
+
+    Used as a control to isolate vLLM v1 framework dispatch cost
+    (build_connector_meta walking SchedulerOutput, mixin hooks, etc.)
+    from any specific connector implementation work (RDMA setup,
+    per-layer save, hash table walks).
+    """
+
+    # ---- scheduler-side abstract methods ------------------------------
+    def get_num_new_matched_tokens(
+        self,
+        request: "Request",
+        num_computed_tokens: int,
+    ) -> tuple[int | None, bool]:
+        # Never advertises any external cache hits.
+        return 0, False
+
+    def update_state_after_alloc(
+        self,
+        request: "Request",
+        blocks: "KVCacheBlocks",
+        num_external_tokens: int,
+    ) -> None:
+        return None
+
+    def build_connector_meta(
+        self,
+        scheduler_output: "SchedulerOutput",
+    ) -> KVConnectorMetadata:
+        return KVConnectorMetadata()
+
+    # ---- worker-side abstract methods ---------------------------------
+    def start_load_kv(
+        self,
+        forward_context: "ForwardContext",
+        **kwargs: Any,
+    ) -> None:
+        return None
+
+    def wait_for_layer_load(self, layer_name: str) -> None:
+        return None
+
+    def save_kv_layer(
+        self,
+        layer_name: str,
+        kv_layer: "torch.Tensor",
+        attn_metadata: "AttentionMetadata",
+        **kwargs: Any,
+    ) -> None:
+        return None
+
+    def wait_for_save(self) -> None:
+        return None

microbench/connector_tax/tools/verify_kv_consumer.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+# Pre-flight: verify Mooncake kv_consumer + dummy bootstrap can start
+# and answer at least a trivial request.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${RUN_DIR:-$HERE/../results/preflight/kv_consumer}"
+mkdir -p "$RUN_DIR"
+
+DUMMY_BOOTSTRAP_PORT="${DUMMY_BOOTSTRAP_PORT:-8997}"
+PORT="${PORT:-8000}"
+
+bash "$HERE/../launch/launch_mooncake_consumer.sh" || {
+    echo "SKIP: kv_consumer launch failed (likely Mooncake bootstrap incompatible with dummy)" >&2
+    exit 42
+}
+
+# kv_consumer can be sent a regular (non-PD-sep) request — it will just
+# do local prefill+decode. It should succeed.
+MODEL="$HOME/models/Qwen/$(ls $HOME/models/Qwen | grep Qwen3-Coder-30B | head -1)"
+
+for i in 1 2 3 4 5; do
+    code=$(curl -s -o "$RUN_DIR/req_$i.json" -w "%{http_code}" \
+        -X POST "http://127.0.0.1:$PORT/v1/chat/completions" \
+        -H 'Content-Type: application/json' \
+        -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"hi $i\"}],\"max_tokens\":4,\"temperature\":0}" \
+        --max-time 30 || echo "000")
+    if [[ "$code" != "200" ]]; then
+        echo "FAIL: req $i status=$code" >&2
+        exit 1
+    fi
+done
+
+echo "OK: kv_consumer reachable, all 5 requests succeeded"

microbench/connector_tax/tools/verify_multi_connector.sh

@@ -0,0 +1,35 @@
+#!/bin/bash
+# Pre-flight: verify MultiConnector(Mooncake kv_both, LMCacheConnectorV1).
+# Exits 42 if LMCache missing, 1 on crash, 0 on OK.
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${RUN_DIR:-$HERE/../results/preflight/multi}"
+mkdir -p "$RUN_DIR"
+
+PORT="${PORT:-8000}"
+
+if ! "$(dirname "$HERE")/launch/launch_multi_mooncake_lmcache.sh"; then
+    rc=$?
+    if [[ $rc == 42 ]]; then
+        echo "SKIP: lmcache missing" >&2
+        exit 42
+    fi
+    echo "FAIL: launch failed with code $rc" >&2
+    exit 1
+fi
+
+MODEL="$HOME/models/Qwen/$(ls $HOME/models/Qwen | grep Qwen3-Coder-30B | head -1)"
+
+for i in 1 2 3 4 5; do
+    code=$(curl -s -o "$RUN_DIR/req_$i.json" -w "%{http_code}" \
+        -X POST "http://127.0.0.1:$PORT/v1/chat/completions" \
+        -H 'Content-Type: application/json' \
+        -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"multi $i\"}],\"max_tokens\":32,\"temperature\":0}" \
+        --max-time 60 || echo "000")
+    if [[ "$code" != "200" ]]; then
+        echo "FAIL: req $i status=$code" >&2
+        exit 1
+    fi
+done
+
+echo "OK: MultiConnector reachable, all 5 requests succeeded"

microbench/connector_tax/tools/verify_noop_connector.sh

@@ -0,0 +1,26 @@
+#!/bin/bash
+# Pre-flight: verify NoOpConnector loads and serves requests.
+# Exits 0 = OK, 42 = SKIP (dependency missing), nonzero = fail.
+
+set -euo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+RUN_DIR="${RUN_DIR:-$HERE/../results/preflight/noop}"
+mkdir -p "$RUN_DIR"
+
+bash "$HERE/../launch/launch_noop_connector.sh"
+PORT="${PORT:-8000}"
+
+# Issue 5 short requests
+for i in 1 2 3 4 5; do
+    code=$(curl -s -o "$RUN_DIR/req_$i.json" -w "%{http_code}" \
+        -X POST "http://127.0.0.1:$PORT/v1/chat/completions" \
+        -H 'Content-Type: application/json' \
+        -d "{\"model\":\"$(ls $HOME/models/Qwen | grep Qwen3-Coder-30B | head -1 | xargs -I{} echo $HOME/models/Qwen/{})\",\"messages\":[{\"role\":\"user\",\"content\":\"hello $i\"}],\"max_tokens\":4,\"temperature\":0}" \
+        --max-time 60 || echo "000")
+    if [[ "$code" != "200" ]]; then
+        echo "FAIL: req $i status=$code" >&2
+        exit 1
+    fi
+done
+
+echo "OK: all 5 requests succeeded"