gahow/agentic-kvc
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Window 1 results: recompute with fixed metrics + reframe limitations

Browse Source 
After the B3 audit bug fixes (joined_analysis hotspot median +
b3_analyze percentile interp), regenerate b3_policy_comparison.json
and the per-policy hotspot_index.json from the same raw run on
dash0 and re-render the three affected figures (apc-vs-hotspot,
latency-bars, per-worker TTFT).

Key number changes in window_1_results.md:
- hotspot_index magnitudes corrected (all five policies; lmetric
  smallest delta at +0.7%, sticky largest at +16.1%)
- "capped reduces hotspot 13%" -> "~10% (2.253 -> 2.020)"
- TTFT/E2E/TPOT percentiles shift by <1% from floor->interp
  (unified TTFT p90 7.24 -> 7.35 s)

Restructured "Caveats" into "Limitations (read this before quoting
B3 numbers)":
1. Agentic dispatch coupling is by design — promoted from caveat
   to top-level methodology framing, tied to
   agentic_dispatch_coupling.md
2. B3 interference_index is binary (not size-graded) — added
3. Hot-sweep cache contamination (<1%) — kept
4. Unified interference unrecoverable — kept with explicit warning
   not to read unified's failure attribution as causal
5. w600 is a sample, not full trace — kept
6. Reuse decomposition is per-token in expectation — added

current_results/characterization_claim_matrix.md updates:
- The "heavy-tail not sole cause" claim now cites the corrected
  ~10% drop with the median bug noted
- New supported claim: "B3 saturated-replay latency gaps include an
  agentic dispatch-coupling feedback term, which is intentional and
  matches production"; cited against agentic_dispatch_coupling.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Gahow Wang
2026-05-26 01:08:55 +08:00
parent 0e82612100
commit 0881942cf3
11 changed files with 131 additions and 72 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
analysis/characterization/window_1_results.md
View File

@@ -15,9 +15,10 @@ Per-policy artifacts under `window_1_results/`. Figures under `window_1_results/
| LMetric leaves 23 pp of APC on the table | **supported** | lmetric achieved 56.9% vs intra-session ceiling 79.6% (theoretical) |
| Hard session affinity recovers the locality lost by LMetric | **supported** | sticky APC 77.2% = 97% of theoretical ceiling |
| Hard affinity inflates same-worker prefill-decode interference | **supported** | sticky interference_index 13.65 vs lmetric 6.53 |
| Hybrid affinity (Unified) breaks the locality-vs-latency tradeoff | **supported** | unified hits 79.4% APC and TTFT p90 7.24 s (lmetric 15.6 s) simultaneously |
| Hybrid affinity (Unified) breaks the locality-vs-latency tradeoff | **supported** | unified hits 79.4% APC and TTFT p90 7.35 s (lmetric 15.67 s) simultaneously |
| Same-worker prefill-decode interference is causal, not correlation | **supported** | different-worker control idx≈1.0; same-worker idx scales monotonically with prefill size |
| Heavy-tail sessions are *a* contributor to hot-spot, not the sole cause | **supported** | cap=8 truncated trace cuts 37% of work; hotspot drops only 13% (2.24→1.94) |
| Heavy-tail sessions are *a* contributor to hot-spot, not the sole cause | **supported** | cap=8 truncated trace cuts 37% of work; hotspot drops only ~10% (2.253→2.020) |
| The agentic dispatch coupling amplifies policy gaps under saturation | **supported, framed as feature** | Slow policy → longer session lifetime → more concurrent in-flight → harder system. B3 measures the combined policy + feedback effect, which is what an agentic operator experiences. See `agentic_dispatch_coupling.md`. |
## B1' Workload characterization
@@ -66,14 +67,26 @@ Gap "any − intra" is 0.7 pp → no meaningful cross-session sharing in this tr
| policy | TTFT p50/p90/p99 | TPOT p50/p90/p99 ms | E2E p50/p90/p99 | **APC** | interference | **hotspot** | n_slow |
|---|---|---|---|---:|---:|---:|---:|
| lmetric | 0.94 / 15.59 / 52.95 | 8.9 / 21.2 / 175.9 | 2.75 / 24.75 / 79.62 | 56.9% | 6.53 | 2.24 | 295 |
| load_only | 1.25 / 20.15 / 52.65 | 9.2 / 26.7 / 320.7 | 3.58 / 33.43 / 93.92 | 54.1% | 9.16 | **1.14** | 379 |
| sticky | 0.54 / 18.02 / 71.37 | 8.9 / 36.1 / 345.2 | 2.08 / 34.61 / 133.58 | 77.2% | **13.65** | 2.35 | 234 |
| **unified** | **0.50 / 7.24 / 42.02** | 8.1 / 17.1 / 118.1 | **1.75 / 17.89 / 68.18** | **79.4%** | n/a* | 3.35 | **189** |
| capped | 1.20 / 12.76 / 46.05 | 7.2 / 16.0 / 101.5 | 2.59 / 21.24 / 73.39 | 31.6% | 6.33 | 1.94 | 185 |
| lmetric | 0.94 / 15.67 / 53.57 | 8.9 / 21.2 / 176.9 | 2.75 / 24.82 / 79.83 | 56.9% | 6.53 | 2.253 | 295 |
| load_only | 1.26 / 20.20 / 52.84 | 9.2 / 26.9 / 320.7 | 3.59 / 33.46 / 93.93 | 54.1% | 9.16 | **1.294** | 379 |
| sticky | 0.54 / 18.02 / 74.09 | 8.9 / 36.4 / 357.2 | 2.08 / 34.63 / 134.36 | 77.2% | **13.65** | 2.728 | 234 |
| **unified** | **0.50 / 7.35 / 42.34** | 8.1 / 17.1 / 118.3 | **1.75 / 18.03 / 68.43** | **79.4%** | n/a* | **3.667** | **189** |
| capped | 1.20 / 12.83 / 46.62 | 7.2 / 16.0 / 101.7 | 2.59 / 21.25 / 73.79 | 31.6% | 6.33 | 2.020 | 185 |
\*unified `engine_state` was overwritten by my analyzer's slice step before the `b3_analyze.sh` fix landed; vLLM and the patch worked correctly. The B2 microbench provides a cleaner interference proof.
> **Methodology note (read before interpreting latency comparisons)**: B3 uses
> session-sequential trace dispatch — turn N+1 fires the instant turn N
> completes when the trace timestamp has already passed. This is the right
> model of agentic workloads (tool-call driven, no user think-time), but it
> means under saturation each policy's effective in-flight session count is
> a function of its own per-turn latency (slower policy → longer mean
> session lifetime → more concurrent in-flight). The reported gaps are
> therefore "policy + agentic-feedback-amplification", which is what a
> production agentic operator would experience when switching policies.
> See `agentic_dispatch_coupling.md` for the full argument. B4 will report
> the orthogonal "fixed-λ open-loop" measurement.
**Mechanism indices**
- `interference_index` = TPOT_p90(decode overlapping same-worker prefill) / TPOT_p90(clean)
- `hotspot_index` = max(worker TTFT p90) / median(worker TTFT p90)
@@ -85,10 +98,10 @@ Figures: `fig_b3_latency_bars.png`, `fig_b3_apc_vs_upper.png`,
### Per-policy reading
- **lmetric** is the cache-aware baseline. APC 56.9% achieves only 71% of the intra-session ceiling — the missing 23 pp is the locality opportunity unified picks up.
- **load_only** strips cache awareness. Hot-spot drops to 1.14 (best), but APC only drops 3 pp because the picker's `min(num_requests)` tie-break to instance 0 creates accidental stickiness at low concurrency.
- **load_only** strips cache awareness. Hot-spot drops to 1.294 (best), but APC only drops 3 pp because the picker's `min(num_requests)` tie-break to instance 0 creates accidental stickiness at low concurrency.
- **sticky** locks each session to one worker. APC climbs to 77.2% (97% of ceiling) but interference doubles to 13.65 and TPOT p99 hits 345 ms.
- **unified** is the hybrid — affinity gate `(cache_ratio>0.5 AND num_req ≤ 2×avg)` keeps locality where it pays and drops it where it would hurt. The result is APC 79.4% **and** TTFT p90 cut in half from lmetric. The one bad worker (engine_4 at 37.7s p90) drives `hotspot_index=3.35`, but the other seven workers are all under 18 s.
- **capped** runs lmetric on a turn-capped trace (max 8 turns/session). Removes 37% of requests but APC also crashes to 31.6% and hotspot only improves by 13%. This is the session-mass ablation: heavy sessions are *a* contributor to hot-spot but not the sole cause.
- **unified** is the hybrid — affinity gate `(cache_ratio>0.5 AND num_req ≤ 2×avg)` keeps locality where it pays and drops it where it would hurt. The result is APC 79.4% **and** TTFT p90 cut in half from lmetric. The one bad worker (engine_4 at 37.7s p90) drives `hotspot_index=3.667`, but the other seven workers are all under 18 s.
- **capped** runs lmetric on a turn-capped trace (max 8 turns/session). Removes 37% of requests but APC also crashes to 31.6% and hotspot only improves by ~10% (2.253 → 2.020). This is the session-mass ablation: heavy sessions are *a* contributor to hot-spot but not the sole cause.
### Slow-request cause breakdown (from `joined_analysis.label_slow_requests`)
@@ -168,11 +181,56 @@ Optional / paper-polish runs (not blocking the story):
4. B2 with the proxy in path — measure whether the production cache_aware routing actually pushes prefill and decode onto different workers in practice.
5. KV-occupancy timeline per worker — needs polling `vllm:gpu_cache_usage` during B3 reruns; useful for "KV pressure drives cache miss" subsection.
## Caveats and known data hygiene issues
## Limitations (read this before quoting B3 numbers)
- **APC contamination across B3 hot-sweep**: `lmetric` ran from cold; `load_only` and `sticky` ran on the same 8 vLLMs without restart. Empirical contamination is < 1% (verified by first-turn cached_tokens distribution), but `unified` and `capped` were rerun cold-start specifically to remove any residual concern.
- **Unified's `interference_index` is missing** because the original `b3_analyze.sh` unconditionally truncate-wrote sliced engine_state files; isolated runs that wrote engine_state into their own per-policy directory were overwritten. Fixed in commit `df32499`; capped was the first run to benefit and survived with intact 86 MB engine_state.
- **w600 is not the full GLM-5.1 trace** (1214 req vs 2.11 M). All B3/B2 percentiles are on the sample. The full-trace KV-footprint stats are on the full trace.
1. **Agentic dispatch coupling is by design**. B3 is the
"production-replay under captured agentic load" experiment, not the
"controlled-load envelope" experiment. Latency p90 reflects both
per-request policy effect AND the agentic feedback amplification
(slow policy → longer mean session lifetime → more concurrent
in-flight). Both contributions are real and visible to a production
operator; **the paper must report both, not subtract one**. See
`agentic_dispatch_coupling.md`. The orthogonal "fixed-λ Poisson"
measurement is B4.
2. **B3 `interference_index` is a binary indicator**. A decode is
labeled "overlap" iff *any* other request's prefill exists on the
chosen worker during `[t_first_token, t_finish]`, regardless of
prefill size. B2's per-prefill-size cells (2k = 1.16×, 65k = 2.26×)
cannot be directly read off B3's aggregate numbers (lmetric 6.53,
sticky 13.65). The B3 numbers are size-weighted averages of the
per-cell signal, valid for *within-B3 cross-policy* comparison but
not for direct cross-batch numerical comparison with B2.
3. **Hot-sweep cache contamination (low)**: `lmetric` ran from cold;
`load_only` and `sticky` ran on the same 8 vLLMs without restart.
First-turn cached_tokens verification puts empirical contamination
at < 1% APC, well below the cross-policy gaps. `unified` and
`capped` were rerun cold-start specifically to remove any residual
concern.
4. **Unified's `interference_index` is missing**. The original
`b3_analyze.sh` unconditionally truncate-wrote sliced engine_state
files; isolated runs that wrote engine_state into their own
per-policy directory were overwritten. Fixed in commit `df32499`;
capped was the first run to benefit and survived. **Implication**:
unified's slow-request mechanism breakdown (rows 0 / 116 / 18 / 55
for same-worker overlap / hot worker queue / cache miss / unknown)
has the "same-worker overlap" label *unrecoverable* and forced into
the catch-all buckets — do not read unified's failure attribution
as causal.
5. **w600 is not the full GLM-5.1 trace** (1214 req vs 2.11 M). All
B3/B2 percentiles are on the sample. The full-trace KV-footprint
stats are on the full trace.
6. **Reuse decomposition (intra/cross/shared/unclassified) is
per-cached-token only in expectation** — `joined_analysis.py`
distributes a request's `cached_tokens` count uniformly across its
`hash_ids` and classifies block-by-block. For the w600 trace with
<1% cross-session sharing the qualitative split is robust; for
workloads with mixed-class hashes within a single request the
classifier should be revisited.
## Reproduction commands