gahow/agentic-pd-hybrid
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: KVC v1-v4 debug journey + raise session soft_cap to 16

Browse Source 
Document the iterative debugging from v1 (broken KVC) through v4
(routing fixed + session cap raised), with code-level analysis of
the two main bugs encountered:

1. v2 root cause (mis-diagnosed previously as `allow_local_prefill`):
   `--policy default` for KVC mechanism caused replay's round-robin
   policy and the PD router's round-robin to diverge, sending requests
   with `session_params` to a D worker that did not have the session
   open. Resulted in 56-61% truncation with finish_reason
   "session id X does not exist".
   Fix: use `--policy kv-aware` (sweep_tp1_v3_kvaware.sh) so replay
   emits `x-smg-target-worker` and PD router uses consistent_hashing.

2. v3 new bottleneck: `pd-router-fallback-large-append-session-cap`
   dominated 52-65% of requests. Root cause was hardcoded
   `min(4, ...)` in `_decode_session_soft_cap`. With 7 D workers x 4
   sessions = 28 slots for 52 trace sessions, ~24 sessions starved
   permanently (bimodal direct-to-D rate of 0% or 99%).
   Fix: raise the cap to 16 (replay.py).

Also includes the v3 finding that direct-to-d-session path P50=0.495s
and TTFT P50=0.043s already beats the 8-way DP baseline (0.65s/0.093s)
- the KVC core mechanism works when fallback paths are avoided.

Files:
- docs/KVC_DEBUG_JOURNEY_V1_TO_V4.md: full journey + code location index
- docs/SWEBENCH_EXPERIMENT_{PROGRESS,RESULTS}.md: prior session notes
- scripts/sweep_tp1_v{2,3,4}*.sh: experiment driver scripts
- src/agentic_pd_hybrid/replay.py: cap 4 -> 16, audit fields
- src/agentic_pd_hybrid/pd_router.py: strip session_params from prefill
- src/agentic_pd_hybrid/metrics.py: truncated_request_count

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
kzlin
2026-04-28 21:10:41 +08:00
parent e9062b1d6e
commit c9d350b372
24 changed files with 1672 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files

110
scripts/convert_audit_to_trace.py Normal file
View File

@@ -0,0 +1,110 @@
#!/usr/bin/env python3
"""Convert sibench audit.jsonl to agentic-pd-hybrid trace format.
Source format (sibench audit.jsonl):
{"instance_id": "...", "ts": float, "messages": [...],
"audit": {"prompt_tokens": int, "completion_tokens": int, ...}}
Target format (agentic-pd-hybrid trace JSONL):
{"chat_id": int, "parent_chat_id": int, "timestamp": float,
"turn": int, "input_length": int, "output_length": int,
"type": str, "hash_ids": [int, ...]}
"""
import json
import sys
from collections import defaultdict
from pathlib import Path
BLOCK_TOKEN_BUDGET = 24 # tokens per block, matching trace.py default
def convert(src: Path, dst: Path) -> None:
# Group lines by instance_id, preserving order within each instance
instances: dict[str, list[dict]] = defaultdict(list)
with src.open() as f:
for line in f:
line = line.strip()
if not line:
continue
rec = json.loads(line)
instances[rec["instance_id"]].append(rec)
# Sort each instance's turns by timestamp
for iid in instances:
instances[iid].sort(key=lambda r: r["ts"])
# Assign stable chat_id bases: each instance gets a block of IDs
# Max turns across all instances determines the spacing
max_turns = max(len(turns) for turns in instances.values())
spacing = max_turns + 10 # extra headroom
total_written = 0
with dst.open("w") as out:
for inst_idx, (iid, turns) in enumerate(instances.items()):
base_chat_id = (inst_idx + 1) * spacing # start from spacing to avoid 0
# Track cumulative hash_ids for prefix cache simulation
cumulative_hash_ids: list[int] = []
global_block_counter = inst_idx * 100_000 # unique block namespace per instance
for turn_idx, rec in enumerate(turns):
audit = rec.get("audit", {})
input_length = audit.get("prompt_tokens", 0)
output_length = audit.get("completion_tokens", 0)
if input_length <= 0:
# Fallback: estimate from message content
total_chars = sum(len(m.get("content", "")) for m in rec.get("messages", []))
input_length = max(1, total_chars // 4)
if output_length <= 0:
output_length = 128 # reasonable default
chat_id = base_chat_id + turn_idx
if turn_idx == 0:
parent_chat_id = -1
else:
parent_chat_id = base_chat_id + turn_idx - 1
# Build hash_ids: for turn 0, generate blocks for full input
# For turn N>0, keep previous blocks and add new ones for the delta
if turn_idx == 0:
num_blocks = input_length // BLOCK_TOKEN_BUDGET
cumulative_hash_ids = list(
range(global_block_counter, global_block_counter + num_blocks)
)
global_block_counter += num_blocks
else:
# The new input is the full prompt (cumulative), so the delta
# is the new tokens beyond what was in the previous turn's prompt
prev_input = audit.get("prompt_tokens", 0)
prev_rec_audit = turns[turn_idx - 1].get("audit", {})
prev_input_length = prev_rec_audit.get("prompt_tokens", 0)
delta = max(0, prev_input - prev_input_length) if prev_input_length > 0 else 0
new_blocks = delta // BLOCK_TOKEN_BUDGET
new_ids = list(
range(global_block_counter, global_block_counter + new_blocks)
)
global_block_counter += new_blocks
cumulative_hash_ids = cumulative_hash_ids + new_ids
trace_line = {
"chat_id": chat_id,
"parent_chat_id": parent_chat_id,
"timestamp": rec["ts"],
"turn": turn_idx,
"input_length": input_length,
"output_length": output_length,
"type": "chat",
"hash_ids": cumulative_hash_ids,
}
out.write(json.dumps(trace_line, separators=(",", ":")) + "\n")
total_written += 1
print(f"Converted {total_written} lines from {len(instances)} instances -> {dst}")
if __name__ == "__main__":
if len(sys.argv) != 3:
print(f"Usage: {sys.argv[0]} <input_audit.jsonl> <output_trace.jsonl>")
sys.exit(1)
convert(Path(sys.argv[1]), Path(sys.argv[2]))

73
scripts/run_all_experiments.sh Executable file
View File

@@ -0,0 +1,73 @@
#!/bin/bash
# Run all 3 PD hybrid experiments sequentially
# Uses 52 sessions / 4,449 requests (10% sample of 497 sessions)
# Each experiment takes ~30-40 min
set -euo pipefail
cd "$(dirname "$0")/.."
TRACE="outputs/qwen35-swebench-50sess.jsonl"
MODEL="/mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B"
OUTPUT="outputs/swebench-exps"
echo "=== Experiment A: pd-disaggregation ==="
uv run agentic-pd-hybrid benchmark-live \
--trace "$TRACE" \
--output-root "$OUTPUT" \
--mechanism pd-disaggregation \
--policy default \
--model-path "$MODEL" \
--prefill-workers 1 --decode-workers 1 \
--prefill-tp-size 4 --decode-tp-size 4 \
--prefill-gpu-ids 0,1,2,3 --decode-gpu-ids 4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300
echo "=== Experiment B: pd-colo ==="
uv run agentic-pd-hybrid benchmark-live \
--trace "$TRACE" \
--output-root "$OUTPUT" \
--mechanism pd-colo \
--policy default \
--model-path "$MODEL" \
--prefill-workers 0 --decode-workers 0 \
--direct-workers 2 --direct-tp-size 4 \
--direct-gpu-ids 0,1,2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300
echo "=== Experiment C: kvcache-centric ==="
uv run agentic-pd-hybrid benchmark-live \
--trace "$TRACE" \
--output-root "$OUTPUT" \
--mechanism kvcache-centric \
--policy default \
--model-path "$MODEL" \
--prefill-workers 1 --decode-workers 1 \
--prefill-tp-size 4 --decode-tp-size 4 \
--prefill-gpu-ids 0,1,2,3 --decode-gpu-ids 4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 2 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
echo "=== All experiments complete ==="

24
scripts/run_exp_a_pd_disagg.sh Executable file
View File

@@ -0,0 +1,24 @@
#!/bin/bash
# Experiment A: pd-disaggregation baseline
# 1P(GPU 0-3) + 1D(GPU 4-7), TP4, mooncake TCP
# Full 39K trace from SWE-Bench 500 instances
set -euo pipefail
cd "$(dirname "$0")/.."
uv run agentic-pd-hybrid benchmark-live \
--trace outputs/qwen35-swebench-500.jsonl \
--output-root outputs/swebench-exps \
--mechanism pd-disaggregation \
--policy default \
--model-path /mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B \
--prefill-workers 1 --decode-workers 1 \
--prefill-tp-size 4 --decode-tp-size 4 \
--prefill-gpu-ids 0,1,2,3 --decode-gpu-ids 4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 64 \
--timeout-s 900 \
--request-timeout-s 300

23
scripts/run_exp_b1_dp_colo_rr.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/bin/bash
# Experiment B1: Naive DP colocation — round-robin policy
# 2 direct workers (GPU 0-3, 4-7), TP4, DP router with round-robin
# No disaggregation — each worker does prefill+decode locally
set -euo pipefail
cd "$(dirname "$0")/.."
uv run agentic-pd-hybrid benchmark-live \
--trace outputs/qwen35-swebench-50sess.jsonl \
--output-root outputs/swebench-exps \
--mechanism pd-colo \
--policy default \
--model-path /mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B \
--prefill-workers 0 --decode-workers 0 \
--direct-workers 2 --direct-tp-size 4 \
--direct-gpu-ids 0,1,2,3,4,5,6,7 \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300

23
scripts/run_exp_b2_dp_colo_cache_aware.sh Executable file
View File

@@ -0,0 +1,23 @@
#!/bin/bash
# Experiment B2: Naive DP colocation — cache-aware (kv-aware) policy
# 2 direct workers (GPU 0-3, 4-7), TP4, DP router with consistent-hashing
# Replay kv-aware policy picks the worker with most prefix overlap
set -euo pipefail
cd "$(dirname "$0")/.."
uv run agentic-pd-hybrid benchmark-live \
--trace outputs/qwen35-swebench-50sess.jsonl \
--output-root outputs/swebench-exps \
--mechanism pd-colo \
--policy kv-aware \
--model-path /mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B \
--prefill-workers 0 --decode-workers 0 \
--direct-workers 2 --direct-tp-size 4 \
--direct-gpu-ids 0,1,2,3,4,5,6,7 \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300

24
scripts/run_exp_b_pd_colo.sh Executable file
View File

@@ -0,0 +1,24 @@
#!/bin/bash
# Experiment B: pd-colo (direct/colocation)
# 2 direct workers (GPU 0-3, 4-7), TP4, no router
# Full 39K trace from SWE-Bench 500 instances
set -euo pipefail
cd "$(dirname "$0")/.."
uv run agentic-pd-hybrid benchmark-live \
--trace outputs/qwen35-swebench-500.jsonl \
--output-root outputs/swebench-exps \
--mechanism pd-colo \
--policy default \
--model-path /mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B \
--prefill-workers 0 --decode-workers 0 \
--direct-workers 2 --direct-tp-size 4 \
--direct-gpu-ids 0,1,2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 64 \
--timeout-s 900 \
--request-timeout-s 300

28
scripts/run_exp_c_kvcache_centric.sh Executable file
View File

@@ -0,0 +1,28 @@
#!/bin/bash
# Experiment C: kvcache-centric (session-aware PD)
# 1P(GPU 0-3) + 1D(GPU 4-7), TP4, mooncake TCP
# Full 39K trace from SWE-Bench 500 instances
set -euo pipefail
cd "$(dirname "$0")/.."
uv run agentic-pd-hybrid benchmark-live \
--trace outputs/qwen35-swebench-500.jsonl \
--output-root outputs/swebench-exps \
--mechanism kvcache-centric \
--policy default \
--model-path /mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B \
--prefill-workers 1 --decode-workers 1 \
--prefill-tp-size 4 --decode-tp-size 4 \
--prefill-gpu-ids 0,1,2,3 --decode-gpu-ids 4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 64 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 2 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction

30
scripts/smoke_test.sh Executable file
View File

@@ -0,0 +1,30 @@
#!/bin/bash
# Smoke test: pd-disaggregation with mooncake TCP, 100 requests
set -euo pipefail
cd "$(dirname "$0")/.."
# Sample a small trace for smoke testing
uv run agentic-pd-hybrid sample-sessions \
--trace outputs/qwen35-swebench-500.jsonl \
--output outputs/qwen35-smoke-3sess.jsonl \
--session-sample-rate 0.02 \
--min-turns 5 \
--target-duration-s 300 \
--max-requests 100
# Run smoke test
uv run agentic-pd-hybrid benchmark-live \
--trace outputs/qwen35-smoke-3sess.jsonl \
--output-root outputs/smoke \
--mechanism pd-disaggregation \
--policy default \
--model-path /mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3.5-35B-A3B \
--prefill-workers 1 --decode-workers 1 \
--prefill-tp-size 4 --decode-tp-size 4 \
--prefill-gpu-ids 0,1,2,3 --decode-gpu-ids 4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300

60
scripts/sweep_kvc_qwen3_30b.sh Executable file
View File

@@ -0,0 +1,60 @@
#!/bin/bash
# KVC admission control parameter sweep on Qwen3-30B
# 5 experiments, ~35 min each, ~3 hours total
set -euo pipefail
cd "$(dirname "$0")/.."
MODEL=/mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3-30B-A3B-Instruct-2507
TRACE=outputs/qwen35-swebench-50sess.jsonl
OUTPUT=outputs/qwen3-30b-exps
VENV_PYTHON=.venv/bin/python
run_kvc() {
local label=$1
local inflight=$2
local min_turn=$3
echo "=== [$label] inflight=$inflight min_turn=$min_turn === $(date)"
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy default \
--model-path $MODEL \
--prefill-workers 1 --decode-workers 1 \
--prefill-tp-size 4 --decode-tp-size 4 \
--prefill-gpu-ids 0,1,2,3 --decode-gpu-ids 4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id $min_turn \
--kvcache-seed-max-inflight-decode $inflight \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
echo "=== [$label] DONE === $(date)"
echo ""
}
# C1: inflight=8, min-turn=2
run_kvc "C1" 8 2
# C2: inflight=16, min-turn=2
run_kvc "C2" 16 2
# C3: inflight=-1 (disabled), min-turn=2
run_kvc "C3" -1 2
# C4: inflight=8, min-turn=1
run_kvc "C4" 8 1
# C5: inflight=-1 (disabled), min-turn=1
run_kvc "C5" -1 1
echo "=== ALL SWEEP EXPERIMENTS DONE === $(date)"

133
scripts/sweep_tp1_configs.sh Executable file
View File

@@ -0,0 +1,133 @@
#!/bin/bash
# TP1 configuration sweep: 8-way DP, 1P7D KVC, 2P6D KVC
# Qwen3-30B-A3B TP=1, single GPU per worker
# Most aggressive KVC admission: inflight=-1 (off), seed-min-turn=1
set -euo pipefail
cd "$(dirname "$0")/.."
MODEL=/mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3-30B-A3B-Instruct-2507
TRACE=outputs/qwen35-swebench-50sess.jsonl
OUTPUT=outputs/qwen3-30b-tp1-exps
VENV_PYTHON=.venv/bin/python
RESULTS_FILE=$OUTPUT/sweep_results.txt
mkdir -p $OUTPUT
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a $RESULTS_FILE
}
save_result() {
local label=$1
local run_dir=$2
log "=== $label COMPLETED ==="
if [ -f "$run_dir/request-metrics.jsonl.summary.json" ]; then
log "Summary:"
cat "$run_dir/request-metrics.jsonl.summary.json" >> $RESULTS_FILE
echo "" >> $RESULTS_FILE
# Also copy summary to a named file for easy access
cp "$run_dir/request-metrics.jsonl.summary.json" "$OUTPUT/${label}_summary.json"
log "Saved to $OUTPUT/${label}_summary.json"
else
log "WARNING: No summary file found in $run_dir"
fi
}
log "Starting TP1 configuration sweep"
log "Model: $MODEL"
log "Trace: $TRACE (4449 requests, 52 sessions)"
########################################
# Experiment 1: 8-way DP cache-aware
########################################
log ""
log "=== [EXP1] 8-way DP cache-aware (8 direct × TP1) ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism pd-colo \
--policy kv-aware \
--model-path $MODEL \
--prefill-workers 0 --decode-workers 0 \
--direct-workers 8 --direct-tp-size 1 \
--direct-gpu-ids 0,1,2,3,4,5,6,7 \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300
# Find latest run dir for this experiment
EXP1_DIR=$(ls -td $OUTPUT/pd-colo-kv-aware-*/ 2>/dev/null | head -1)
save_result "exp1_8way_dp_cache_aware" "$EXP1_DIR"
########################################
# Experiment 2: 1P + 7D KVC (most aggressive)
########################################
log ""
log "=== [EXP2] 1P7D KVC (inflight=off, min-turn=1) ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy default \
--model-path $MODEL \
--prefill-workers 1 --decode-workers 7 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0 --decode-gpu-ids 1,2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP2_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp2_1p7d_kvc_aggressive" "$EXP2_DIR"
########################################
# Experiment 3: 2P + 6D KVC (most aggressive)
########################################
log ""
log "=== [EXP3] 2P6D KVC (inflight=off, min-turn=1) ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy default \
--model-path $MODEL \
--prefill-workers 2 --decode-workers 6 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0,1 --decode-gpu-ids 2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP3_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp3_2p6d_kvc_aggressive" "$EXP3_DIR"
########################################
log ""
log "=== ALL TP1 SWEEP EXPERIMENTS DONE ==="

131
scripts/sweep_tp1_v2_fixed.sh Executable file
View File

@@ -0,0 +1,131 @@
#!/bin/bash
# TP1 configuration sweep v2 — after session_params fix + audit fields
# Qwen3-30B-A3B TP=1, single GPU per worker
set -euo pipefail
cd "$(dirname "$0")/.."
MODEL=/mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3-30B-A3B-Instruct-2507
TRACE=outputs/qwen35-swebench-50sess.jsonl
OUTPUT=outputs/qwen3-30b-tp1-v2-fixed
VENV_PYTHON=.venv/bin/python
RESULTS_FILE=$OUTPUT/sweep_results.txt
mkdir -p $OUTPUT
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a $RESULTS_FILE
}
save_result() {
local label=$1
local run_dir=$2
log "=== $label COMPLETED ==="
if [ -f "$run_dir/request-metrics.jsonl.summary.json" ]; then
log "Summary:"
cat "$run_dir/request-metrics.jsonl.summary.json" >> $RESULTS_FILE
echo "" >> $RESULTS_FILE
cp "$run_dir/request-metrics.jsonl.summary.json" "$OUTPUT/${label}_summary.json"
cp "$run_dir/request-metrics.jsonl" "$OUTPUT/${label}_metrics.jsonl"
log "Saved to $OUTPUT/${label}_summary.json + ${label}_metrics.jsonl"
else
log "WARNING: No summary file found in $run_dir"
fi
}
log "Starting TP1 v2 sweep (session_params fix + audit fields)"
log "Model: $MODEL"
log "Trace: $TRACE (4449 requests, 52 sessions)"
########################################
# Experiment 1: 8-way DP cache-aware
########################################
log ""
log "=== [EXP1] 8-way DP cache-aware (8 direct × TP1) ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism pd-colo \
--policy kv-aware \
--model-path $MODEL \
--prefill-workers 0 --decode-workers 0 \
--direct-workers 8 --direct-tp-size 1 \
--direct-gpu-ids 0,1,2,3,4,5,6,7 \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300
EXP1_DIR=$(ls -td $OUTPUT/pd-colo-kv-aware-*/ 2>/dev/null | head -1)
save_result "exp1_8way_dp_cache_aware" "$EXP1_DIR"
########################################
# Experiment 2: 1P + 7D KVC (aggressive)
########################################
log ""
log "=== [EXP2] 1P7D KVC (inflight=off, min-turn=1) ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy default \
--model-path $MODEL \
--prefill-workers 1 --decode-workers 7 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0 --decode-gpu-ids 1,2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP2_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp2_1p7d_kvc_aggressive" "$EXP2_DIR"
########################################
# Experiment 3: 2P + 6D KVC (aggressive)
########################################
log ""
log "=== [EXP3] 2P6D KVC (inflight=off, min-turn=1) ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy default \
--model-path $MODEL \
--prefill-workers 2 --decode-workers 6 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0,1 --decode-gpu-ids 2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP3_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp3_2p6d_kvc_aggressive" "$EXP3_DIR"
########################################
log ""
log "=== ALL TP1 V2 SWEEP EXPERIMENTS DONE ==="

108
scripts/sweep_tp1_v3_kvaware.sh Executable file
View File

@@ -0,0 +1,108 @@
#!/bin/bash
# TP1 v3 sweep — KVC with kv-aware policy (fix routing mismatch)
# v2 used --policy default for KVC experiments, causing session routing
# mismatch: replay round-robin ≠ router round-robin → "session not found".
# v3 uses --policy kv-aware for KVC to ensure session affinity.
set -euo pipefail
cd "$(dirname "$0")/.."
MODEL=/mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3-30B-A3B-Instruct-2507
TRACE=outputs/qwen35-swebench-50sess.jsonl
OUTPUT=outputs/qwen3-30b-tp1-v3-kvaware
VENV_PYTHON=.venv/bin/python
RESULTS_FILE=$OUTPUT/sweep_results.txt
mkdir -p $OUTPUT
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a $RESULTS_FILE
}
save_result() {
local label=$1
local run_dir=$2
log "=== $label COMPLETED ==="
if [ -f "$run_dir/request-metrics.jsonl.summary.json" ]; then
log "Summary:"
cat "$run_dir/request-metrics.jsonl.summary.json" >> $RESULTS_FILE
echo "" >> $RESULTS_FILE
cp "$run_dir/request-metrics.jsonl.summary.json" "$OUTPUT/${label}_summary.json"
cp "$run_dir/request-metrics.jsonl" "$OUTPUT/${label}_metrics.jsonl"
log "Saved to $OUTPUT/${label}_summary.json + ${label}_metrics.jsonl"
else
log "WARNING: No summary file found in $run_dir"
fi
}
log "Starting TP1 v3 sweep (KVC with kv-aware policy)"
log "Model: $MODEL"
log "Trace: $TRACE (4449 requests, 52 sessions)"
log "Key change: --policy kv-aware for KVC (was --policy default in v2)"
########################################
# Experiment 1: 1P + 7D KVC kv-aware
########################################
log ""
log "=== [EXP1] 1P7D KVC kv-aware ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy kv-aware \
--model-path $MODEL \
--prefill-workers 1 --decode-workers 7 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0 --decode-gpu-ids 1,2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP1_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp1_1p7d_kvc_kvaware" "$EXP1_DIR"
########################################
# Experiment 2: 2P + 6D KVC kv-aware
########################################
log ""
log "=== [EXP2] 2P6D KVC kv-aware ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy kv-aware \
--model-path $MODEL \
--prefill-workers 2 --decode-workers 6 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0,1 --decode-gpu-ids 2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP2_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp2_2p6d_kvc_kvaware" "$EXP2_DIR"
########################################
log ""
log "=== ALL TP1 V3 SWEEP EXPERIMENTS DONE ==="

108
scripts/sweep_tp1_v4_cap16.sh Executable file
View File

@@ -0,0 +1,108 @@
#!/bin/bash
# TP1 v4 sweep — KVC with kv-aware policy + soft_cap raised from 4 to 16
# v3 (kv-aware) fixed routing but session-cap fallback still dominated 52-65%
# of requests. Hardcoded min(4, ...) in _decode_session_soft_cap was the
# bottleneck — only 4*7=28 session slots for 52 trace sessions.
# v4 raises the cap to 16 (4*7=28 -> 16*7=112 slots).
set -euo pipefail
cd "$(dirname "$0")/.."
MODEL=/mnt/kzlin/workflow/pd-hybrid/simm-swe-bench/models/Qwen3-30B-A3B-Instruct-2507
TRACE=outputs/qwen35-swebench-50sess.jsonl
OUTPUT=outputs/qwen3-30b-tp1-v4-cap16
VENV_PYTHON=.venv/bin/python
RESULTS_FILE=$OUTPUT/sweep_results.txt
mkdir -p $OUTPUT
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a $RESULTS_FILE
}
save_result() {
local label=$1
local run_dir=$2
log "=== $label COMPLETED ==="
if [ -f "$run_dir/request-metrics.jsonl.summary.json" ]; then
log "Summary:"
cat "$run_dir/request-metrics.jsonl.summary.json" >> $RESULTS_FILE
echo "" >> $RESULTS_FILE
cp "$run_dir/request-metrics.jsonl.summary.json" "$OUTPUT/${label}_summary.json"
cp "$run_dir/request-metrics.jsonl" "$OUTPUT/${label}_metrics.jsonl"
log "Saved to $OUTPUT/${label}_summary.json + ${label}_metrics.jsonl"
else
log "WARNING: No summary file found in $run_dir"
fi
}
log "Starting TP1 v4 sweep (KVC kv-aware, session soft_cap raised 4->16)"
log "Model: $MODEL"
log "Trace: $TRACE (4449 requests, 52 sessions)"
log "Key change: _decode_session_soft_cap now min(16, ...) instead of min(4, ...)"
########################################
# Experiment 1: 1P + 7D KVC kv-aware (cap=16)
########################################
log ""
log "=== [EXP1] 1P7D KVC kv-aware cap=16 ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy kv-aware \
--model-path $MODEL \
--prefill-workers 1 --decode-workers 7 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0 --decode-gpu-ids 1,2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP1_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp1_1p7d_kvc_cap16" "$EXP1_DIR"
########################################
# Experiment 2: 2P + 6D KVC kv-aware (cap=16)
########################################
log ""
log "=== [EXP2] 2P6D KVC kv-aware cap=16 ==="
PYTHONPATH=src:third_party/sglang/python \
$VENV_PYTHON -m agentic_pd_hybrid.cli benchmark-live \
--trace $TRACE \
--output-root $OUTPUT \
--mechanism kvcache-centric \
--policy kv-aware \
--model-path $MODEL \
--prefill-workers 2 --decode-workers 6 \
--prefill-tp-size 1 --decode-tp-size 1 \
--prefill-gpu-ids 0,1 --decode-gpu-ids 2,3,4,5,6,7 \
--transfer-backend mooncake \
--gpu-budget 8 \
--time-scale 10 \
--session-sample-rate 1.0 \
--target-duration-s 100000 \
--concurrency-limit 32 \
--timeout-s 900 \
--request-timeout-s 300 \
--kvcache-admission-mode worker \
--kvcache-seed-min-turn-id 1 \
--kvcache-seed-max-inflight-decode -1 \
--kvcache-prefill-backup-policy release-after-transfer \
--kvcache-prefill-priority-eviction
EXP2_DIR=$(ls -td $OUTPUT/kvcache-centric-*/ 2>/dev/null | head -1)
save_result "exp2_2p6d_kvc_cap16" "$EXP2_DIR"
log ""
log "=== ALL TP1 V4 SWEEP EXPERIMENTS DONE ==="