agentic-kvc/analysis/characterization/agentic_dispatch_coupling.md

Agentic Dispatch Coupling: Why Session-Sequential Replay is the Realistic Mode

Date: 2026-05-26 Owner: characterization Status: methodology note for paper framing

The observation

In replayer/replay.py:282-287, turn N of a session fires at:

t_fire(N) = max(
    turn_N_trace_timestamp,         # what the trace asked for
    turn_{N-1}_finish_wall_clock,   # but turn N-1 must complete first
)

When the system is fast, the second term loses → turn N fires at its trace timestamp → the replay matches the captured trace. When the system is slow, the second term dominates → turn N fires immediately after turn N-1 completes, with the trace timestamp ignored. The session's "inter-turn think time" collapses to zero.

A first reading flags this as a benchmark concern: under saturation the arrival process becomes policy-dependent, so cross-policy latency comparisons are confounded by a feedback loop (slow policy → longer sessions → more concurrent in-flight → harder system → slower latency).

This note argues the opposite: the trace-replayer's behavior is the correct model of agentic workloads, and the feedback loop is a real property of production systems, not a methodology artifact.

Why agentic workloads do not have user think-time

In chat workloads, the turn N+1 message is composed by a human reading the turn N response. The inter-turn gap is dominated by human reading + typing speed, which is independent of how fast the server replied. The trace's timestamp captures the human cadence and is a faithful arrival process.

In agentic workloads, turn N+1 is generated by:

• A tool-call response feeding back into the model context
• An autonomous loop deciding the next action
• A planner / executor stepping to the next subgoal

None of these wait for a human. Turn N+1 fires as soon as the infrastructure can hand the previous turn's output back to the next inference step. There is no think-time floor.

This means: in a real agentic system, the wall-clock time between turn N finish and turn N+1 dispatch is essentially zero. If the model serving infrastructure slows down (high TTFT or E2E for turn N), turn N+1's dispatch slips by the same amount — exactly the behavior the replayer exhibits.

What B3's session-sequential dispatch is actually measuring

B3's trace replayer drives a workload that:

• preserves the causal structure of the original trace (which turns belong to which session and in what order),
• uses the original timestamps as a lower bound (turn N+1 cannot fire before its trace timestamp),
• but lets turn N+1 fire immediately when the system has fallen behind.

For an agentic workload, this is the right model:

1. The captured trace's timestamps reflect the production system's actual response speed at capture time — they already encode the round-trip time the model + tool stack delivered.
2. When we replay against a different policy, what we want to measure is "what wall-clock would this session take under policy X" — which includes the same tool-call-driven cadence: each next turn fires as soon as it can.
3. The "inter-turn gap" is not a fixed delay we should respect; it is an artifact of the captured system's speed that we are explicitly trying to replace.

So the replayer's behavior is not "broken under saturation"; it is modeling the agentic semantic correctly: no think-time, sequential within session, fire-immediately when ready.

The feedback loop is a real production phenomenon

Once we accept the agentic semantic, the so-called "dispatch slip artifact" is not an artifact — it is a real system behavior:

slow policy
    → each turn takes longer
    → each session lives in the system longer
    → at any moment, more sessions are concurrently in-flight
    → 8 workers' KV / queue pressure is higher
    → each request gets less per-worker capacity
    → each turn takes even longer
    → ...

By Little's Law: N_concurrent ≈ session_arrival_rate × mean_session_lifetime.

In our B3 data:

• lmetric: mean session lifetime is much longer than the original trace's ~600 s span (lmetric's 1214-request replay took 49 min wall clock — sessions stayed alive ~8× longer than the trace captured).
• unified: sessions drain ~3× faster than lmetric.

So under unified, the 8-worker pool sees fewer concurrent sessions than under lmetric — and this is what production would also see if the operator switched routing policies on the same incoming agentic load.

This is not a fairness violation. It is a faithful reflection of: "a faster routing policy is faster both because of its per-request behavior AND because it reduces the steady-state concurrent load it inflicts on itself".

A user running an agentic system does benefit from both effects when they pick a better policy. The combined "policy × system-feedback" gain is what the user actually experiences.

What this means for B3 and B4 in the paper

	B3 trace-driven replay	B4 open-loop Poisson
Arrival process	original trace timestamps with session-sequential "fire-on-finish"	Poisson session inter-arrival at fixed λ
Inter-turn think-time	none (matches agentic)	none (matches agentic)
Session lifetime under load	grows with policy slowness (feedback)	fixed by trace template plus per-turn latency
What latency at p90 measures	end-user latency under agentic feedback amplification	per-request behavior at the operator-chosen load level
What "fair across policies" means	same trace, same total session set; arrival process is policy-dependent on purpose	same λ, decoupled from policy throughput
When to use it	"if we run this real captured load through our system, what does the user see"	"what is the max sustainable rate (SRR) before SLO breaks, per policy"

The two are complementary, not "B3 is unfair and B4 fixes it":

• B3 answers the "in-production replay" question — including feedback amplification, which agentic users will actually experience.
• B4 answers the "saturation envelope" question — what's the policy's intrinsic throughput at fixed load.

A paper that drops B3 in favor of B4 would understate how much the combined effect (policy + feedback) actually helps the user. A paper that drops B4 in favor of B3 would conflate the two effects and prevent a "policy X sustains higher λ" statement.

Recommended paper framing

1. B3 is the production-replay experiment. Report latency percentiles as "end-to-end under captured agentic load with no-think-time sequencing". Acknowledge that the combined gap (e.g. unified TTFT p90 = 7.24 s vs lmetric 15.6 s) reflects both policy and feedback; call this out, do not hide it.

2. B4 is the controlled-load experiment. Report SRR_max per policy under per-class SLO. This is the experiment that decouples policy from feedback and gives a sustainable-rate ranking.

3. The feedback amplification itself is a finding to call out. It is the reason why a "marginally better" routing policy (e.g. unified over lmetric in microbenchmarks) can deliver a much bigger gap in production (here ~2×): the feedback halves the in-flight count which compounds on top of the per-request improvement.

4. The contrast with chat workloads is a paper section (or at least a paragraph). Chat workloads have human think-time bounded by reading speed, so the feedback loop is partially broken: even if the server slows down, the user-driven inter-turn delay still puts a floor on how concentrated the load can become. Agentic workloads remove that floor.

Open questions

• Is the feedback amplification quantifiable from B3 alone? We have total wall-clock per policy and per-session lifetime distributions; we can in principle attribute the policy-vs-feedback split by comparing B3's saturated-replay p90 to B4's at-fixed-λ p90 (when B4 runs).
• Does it matter that the original trace was captured under one policy's behavior? The trace's timestamps were the production system's output at capture time. When we replay against a slower policy, we are asking "what if this same set of session+turns ran on a worse policy" — and the answer is "the sessions would live longer". This is precisely the counterfactual we want.
• What happens if real tools have variable per-call latency? Our replayer assumes turn N+1 fires the instant turn N finishes. Real agentic systems have some tool-execution time between turns. This is a quantitative correction (raises the floor on inter-turn gap), not a qualitative one — the feedback loop still applies, just with a higher baseline.

Cross-reference

• replayer/replay.py:282-287 — the dispatch rule
• analysis/characterization/window_1_results.md §"What Window 1 does not answer" — current treatment as caveat
• analysis/claude_characterization_work_plan.md §B4 — open-loop Poisson loadgen as the orthogonal measurement