replaysim/docs/comparison.md

# RS2 Simulator Comparison

Checked on 2026-06-24. RS2 compares simulator capabilities and first local
ReplayServe results. It does not start the RS3 sweep and does not make
performance-quality claims.

## Sources

| Source | Local path | Commit / HEAD | RS2 use |
|---|---|---:|---|
| ReplayServe | `/home/gahow/phd/replayserve` | local RS0/RS1/RS1B artifacts | Adapter, fixtures, runs, postprocess summaries |
| Qwen trace | `/home/gahow/phd/qwen-bailian-usagetraces-anon` | `5f7439c51ec248a0c585f7d90a41a6f57773b912` | Source `qwen_coder_blksz_16.jsonl` |
| Frontier canonical | `/tmp/toc-llm-sim-research/Frontier` | `d9cfeb6d8791fbf2f295dd9744c56a666171776e` | RS1 fixed config and source inspection |
| Frontier patched scratch | `/tmp/replayserve-frontier-rs1b` | base `d9cfeb6...` plus `patches/frontier-vllm-v1-prefix-cache-chunked-prefill.patch` | RS1B unblock verification |
| Vidur | `/tmp/toc-llm-sim-research/vidur` | `8383d2935bc62723a212090baa9f98ada206fc14` | Source inspection for baseline capability |
| AIConfigurator | `/tmp/toc-llm-sim-research/aiconfigurator` | `e46ece7510e727fafefb8212e5846172145a30ea` | Source/docs inspection for config-estimator capability |

Key local evidence:

- Frontier trace replay: `/tmp/toc-llm-sim-research/Frontier/frontier/request_generator/trace_replay_request_generator.py`
- Frontier prefix-cache validation: `/tmp/toc-llm-sim-research/Frontier/frontier/scheduler/cluster_scheduler/base_cluster_scheduler.py`
- Frontier prefix-cache request metrics: `/tmp/toc-llm-sim-research/Frontier/frontier/metrics/metrics_store.py`
- Vidur trace replay: `/tmp/toc-llm-sim-research/vidur/vidur/request_generator/trace_replay_request_generator.py`
- Vidur request entity: `/tmp/toc-llm-sim-research/vidur/vidur/entities/request.py`
- AIConfigurator CLI/docs: `/tmp/toc-llm-sim-research/aiconfigurator/README.md` and `/tmp/toc-llm-sim-research/aiconfigurator/src/aiconfigurator/cli/main.py`

## Capability Matrix

| Capability | Frontier | Vidur | AIConfigurator |
|---|---|---|---|
| Per-request timestamp replay | Yes. `trace_replay` consumes `arrived_at` and RS1 runs `simulation_mode=online`. | Yes. `TraceReplayRequestGenerator` consumes `arrived_at`. | No per-request replay. CLI consumes workload summaries such as `--isl`, `--osl`, and SLA targets. |
| Input/output length replay | Yes. Consumes `num_prefill_tokens` and `num_decode_tokens`. Frontier can clip overflows internally, so ReplayServe adapter validates before run. | Yes. Consumes `num_prefill_tokens` and `num_decode_tokens`; current code clips prefill length if total exceeds max tokens. | Only summary lengths, not per-request traces. |
| Explicit `block_hash_ids` / prefix KV reuse replay | Yes. Current Frontier parses `session_id` and `block_hash_ids`, validates they are present when prefix caching is enabled, and applies prefix-cache accounting in vLLM v1. RS1B needs a patch for prefix cache plus chunked prefill under pressure. | No in this checkout. Request objects carry arrival/prefill/decode lengths and processed-token state, but no `session_id`, block hashes, or explicit prefix-reuse replay. README says prefix-caching work lives on a canary branch with sharp edges, not this main checkout. | No. `--prefix` is an aggregate prefix length/workload parameter, not an explicit hash/session replay model. |
| Online arrival pattern | Yes. RS1 fixed config uses online mode and trace replay. | Yes for trace replay baseline. | No event-level online replay. It estimates candidate deployments from summary workload/SLA inputs. |
| Prefix-cache hit-ratio output | Yes. Frontier emits request metrics including cached prefill tokens, query blocks, and hit blocks when present. ReplayServe postprocess adds token-weighted hit ratio using sidecar partial-block counts. | No native prefix-hit ratio in current main because no explicit prefix replay. | No prefix-hit replay metric. |
| TTFT / TPOT / E2E / throughput output | Yes. Request and system metrics are emitted under Frontier metrics dirs. RS1 uses dummy execution predictor, so values are plumbing-only. | Yes. Vidur metrics include request E2E, prefill/TTFT-style, decode-normalized, and system metrics. Fidelity depends on matching profiles. | Yes as estimates: best throughput, per-GPU throughput, per-user throughput, TTFT, TPOT, and request latency. These are configuration-search outputs, not replay observations. |
| TP / EP / DP / config knobs | Yes. Frontier has model, device, network device, attention TP/DP, MoE TP/EP, PP, scheduler, block, batch, prefix-cache, chunked-prefill, and memory-planner knobs. | Partial. Vidur exposes model, device, network device, tensor parallel size, pipeline stages, scheduler and batch/KV knobs. The inspected checkout is not a faithful EP/DP prefix-replay candidate. | Strong for config search. Supports TP/PP/DP and expert TP/EP style search across supported backends/systems. |
| Arbitrary model/hardware/config boundary | Not arbitrary. Model/device configs may exist, but reliable latency/throughput requires compute/network profiles, scheduler support, matching parallel semantics, and bug-free code paths. RS1 Qwen3-32B on A800 uses dummy predictor because public A800 dense Qwen3-32B compute profiles are absent. | Not arbitrary. README lists supported model/device/profile combinations; docs say profiling on actual GPUs is needed for new model/hardware fidelity. Current public device/profile coverage does not match A800 Qwen3-32B. | Not arbitrary. It depends on supported model families, backend/system databases, and estimate modes. The support matrix includes H100/H200/B200/GB200/A100 variants; no A800 built-in silicon database was found. |
| Needs profile/calibration | Yes for performance claims. Dummy predictor plus analytical comm is only a smoke. | Yes for performance claims. New model/device requires compute, network, and CPU-overhead style profiling. | Yes for production-quality estimates. It relies on collected silicon/perf databases or rough estimate modes; README warns memory/results need validation. |

## First Results

### Frontier Canonical

Fixed RS1 config:

- `simulation_mode=online`
- `sys_arch=co-location`
- `replica_scheduler=vllm_v1`
- `device=a800`
- `network_device=a800_dgx`
- `model_name=Qwen/Qwen3-32B`
- `attn_tensor_parallel_size=2`
- dummy execution predictor
- analytical communication backend
- `trace_request_generator_config_max_tokens=32768`
- prefix caching enabled
- block size 16
- chunked prefill enabled
- batch cap 128
- max batch tokens 32768
- KV capacity from Frontier memory planner with `gpu_memory_utilization=0.9` and `non_kv_cache_overhead_bytes=0`

Results:

| Run | Result | Evidence | Notes |
|---|---|---|---|
| `coder_100` | Pass | `runs/rs1/coder_100/` | Frontier block hit ratio `0.04948661841440835`; ReplayServe token-weighted hit ratio `0.04956232588915065`; no preemptions. |
| `coder_2000` | Fail | `runs/rs1/coder_2000/` | Exit code 1 after 4 seconds with `ValueError: Request 194 already scheduled.` Traceback ends at Frontier vLLM v1 waiting scheduling calling `request.on_cache_hit(prefix_cached_tokens)`. |

The canonical failure was minimized in `docs/rs1_frontier_blocker.md`: first-N
`N=192` passes, `N=193` fails as `Request 192 already scheduled`, and larger
fixed-config probes fail around the same preempted prefix-cache path. Prefix off,
chunked-prefill off, or a high long-prefill threshold avoids the failure, so this
is not a bad Qwen trace row.

### Frontier Patched Scratch

Patch:

- File: `patches/frontier-vllm-v1-prefix-cache-chunked-prefill.patch`
- Documentation: `docs/rs1_frontier_patch.md`
- Scratch checkout only: `/tmp/replayserve-frontier-rs1b`

The patch resets preempted request scheduler/cache-hit admission state before
the request re-enters the waiting path. As of 2026-06-25 it also replays
decode-phase preemption by moving already-produced tokens into the next prefill
segment, preserves user-facing lengths for metrics, and fails fast if
sequential simulation drains before all generated requests complete. It keeps
the canonical Frontier checkout clean.

| Run | Result | Evidence | Hit ratios | Preemption | Memory planner facts |
|---|---|---|---|---|---|
| `N=193` fixed config | Pass | `runs/rs1b/patched/n193_fixed_v2/` | Frontier block `0.12458971786194112`; ReplayServe token-weighted `0.12476981408429115` | 5 total events, 1 request | `num_blocks=36902`, `gpu_memory_utilization=0.9`, non-KV overhead `0`, weight shard estimate `26.953125 GiB` |
| `coder_100` fixed config | Pass | `runs/rs1b/patched/coder_100/` | Frontier block `0.04948661841440835`; ReplayServe token-weighted `0.04956232588915065` | 0 | same derived memory planner point |
| `coder_2000` fixed config | Pass | `runs/rs1b/patched/coder_2000/` | Frontier block `0.12318930248025924`; ReplayServe token-weighted `0.12332978217090633` | 35940 total events, 1061 requests | same derived memory planner point |

Metrics caveats:

- These are plumbing-smoke metrics. The run uses dummy 1 ms execution time and
  analytical communication, not calibrated Qwen3-32B A800 compute profiles.
- `coder_2000` produced `request_metrics.csv` with 2000 rows, but 745 rows have
  blank request-level prefix-cache fields. ReplayServe token-weighted hit ratio
  therefore uses the 1255 rows with complete cache metrics. Frontier's aggregate
  prefix-cache statistics in the same summary also report 1255 requests with
  cache metrics. This is acceptable for blocker removal evidence, but it is not
  a final metric-quality result.
- No allocation/OOM pressure log lines were found in the postprocess summaries.

### Vidur

No Vidur baseline run was executed for RS2. Based on source inspection, Vidur is
useful as an arrival-and-length baseline candidate, but it cannot faithfully
compare ReplayServe prefix reuse without additional code:

- `vidur/request_generator/trace_replay_request_generator.py` consumes
  `arrived_at`, `num_prefill_tokens`, and `num_decode_tokens`.
- `vidur/entities/request.py` stores arrival, prefill length, decode length,
  processed tokens, schedule/completion timestamps, and preemption state.
- The inspected request path does not carry `session_id`, `block_hash_ids`, or
  sidecar block-token accounting.
- Current Vidur trace replay clips prefill lengths when total tokens exceed
  `max_tokens`; ReplayServe must keep its own hard-fail validation if Vidur is
  used later as a length-only baseline.

Conclusion for Vidur in RS2: it can likely replay `coder_100`/`coder_2000`
arrival and length after a simple CSV compatibility conversion, but it would
measure a different workload because prefix KV reuse is absent.

### AIConfigurator

No AIConfigurator run was executed for RS2 because it is not a per-request
replay simulator. Source/docs show it is a deployment/config search estimator:

- CLI examples take workload summaries such as `--isl`, `--osl`, `--prefix`,
  `--ttft`, `--tpot`, `--total-gpus`, `--system`, and model path.
- Outputs are best throughput, per-GPU throughput, per-user throughput, TTFT,
  TPOT, request latency, concurrency, and parallel deployment choices.
- It models operations and searches aggregated/disaggregated serving
  configurations using collected or estimated performance data.

Conclusion for AIConfigurator in RS2: it is useful for config candidates and
reference sizing assumptions. It cannot directly compare faithful per-request
prefix-hit replay on Qwen trace fixtures.

## Metric Definitions

`TTFT`:
Time from request arrival to first generated token / prefill completion. Frontier
and Vidur both have request-level prefill/first-token style timing fields, but
RS1 Frontier values are not performance claims because the execution predictor is
dummy.

`TPOT`:
Decode time per output token. Tools differ on whether they report total decode
normalized by output tokens, inter-token latency, or a configured SLA target.
Use each tool's native field only within that tool unless calibrated against the
same serving definition.

`E2E latency`:
Completion time minus arrival time for one request.

`Throughput`:
Completed tokens or requests per unit time. AIConfigurator reports estimated
tokens/s style capacity; Frontier/Vidur report simulated metrics. RS1 Frontier
throughput is plumbing-only because compute is dummy.

`KV-cache hit ratio`:

- Frontier native block-level ratio:
  `sum(request_prefix_cache_hit_blocks) / sum(request_prefix_cache_query_blocks)`.
- ReplayServe token-weighted ratio:
  use sidecar `block_token_counts` and count the first
  `request_prefix_cache_hit_blocks` blocks by true token count, so a partial
  final block contributes its actual token count instead of always 16.

For `coder_2000` patched, both ratios are computed only for request rows with
complete cache fields because 745 request metric rows have blank cache fields.
This is a metrics completeness caveat, not evidence that the trace has invalid
hashes.

## Non-Comparable Items

- Frontier canonical and patched scratch are not equivalent artifacts. The
  patched result demonstrates an RS1 unblock path; it is not an upstream
  Frontier release.
- Frontier/Vidur simulator timings and AIConfigurator estimator timings are not
  directly comparable without shared profiles, calibration, and metric
  definitions.
- Prefix-reuse fidelity is not comparable across all three tools. Only Frontier
  currently consumes explicit block hash traces in the inspected checkouts.
- AIConfigurator's `prefix` workload parameter is not the same as ReplayServe
  `block_hash_ids`; it cannot recover session-level sharing or partial-block
  token accounting.

## Conclusions

There is no best open-source implementation that satisfies ReplayServe's target
out of the box.

Frontier is the closest because it supports online trace replay, prefix-cache
metadata, vLLM v1 style scheduler controls, memory planning, and request/system
metrics. It is still not out of the box for ReplayServe: RS0 needed an adapter,
RS1B needed a local patch or upstream fix for prefix cache plus chunked prefill,
and performance-quality claims need Qwen3-32B A800 profiles/calibration.

Vidur can be a useful arrival-plus-length baseline, but not a faithful prefix KV
reuse replay engine in the inspected checkout.

AIConfigurator can guide candidate deployment/config choices, but it is a
workload-summary estimator rather than a per-request simulator.

Frontier also does not support arbitrary model plus arbitrary hardware plus
arbitrary config in a performance-reliable sense. A model/device config may be
accepted syntactically, but fidelity depends on compute profiles, network
profiles, scheduler support, parallelism semantics, memory-planner assumptions,
and bug surface. For RS1, A800 network profiles exist, but the public checkout
does not provide dense Qwen3-32B A800 compute profiles, so latency/throughput
remain plumbing smoke.

## Next Steps

RS3 sweep prerequisites:

- Decide whether RS3 uses the local RS1B patch, waits for upstream Frontier, or
  carries both canonical and patched modes explicitly.
- Keep fixed-config smoke and any sweep configs separate from performance claims.
- Add a small run manifest/check script that records Frontier commit, patch
  status, fixture, command, and metric completeness.
- Treat the `coder_2000` blank cache fields as a metrics issue to investigate
  before using request-level hit ratios as a headline metric.

RS4 calibration prerequisites:

- Collect or obtain dense `Qwen/Qwen3-32B` A800 compute profiles for the Frontier
  predictor path.
- Verify the A800 network profile and node SKU semantics match the target
  deployment.
- Add non-KV memory overhead assumptions from a real serving stack instead of
  using `0`.
- Validate simulator TTFT/TPOT/E2E/throughput against measured vLLM runs before
  making performance conclusions.

Patch path recommendation:

- Keep `patches/frontier-vllm-v1-prefix-cache-chunked-prefill.patch` pinned in
  ReplayServe until an upstream Frontier fix is available; it now covers both
  the original prefix-cache/chunked-prefill preemption bug and the RS10
  decode-phase preemption lifecycle bug.
- Open an upstream issue or PR with the RS1B minimal repro (`N=193`) and
  evidence from `docs/rs1_frontier_blocker.md`.
- Re-run `coder_100`, `N=193`, and `coder_2000` when changing Frontier commit or
  patch status.