Window 1 results: recompute with fixed metrics + reframe limitations

After the B3 audit bug fixes (joined_analysis hotspot median +
b3_analyze percentile interp), regenerate b3_policy_comparison.json
and the per-policy hotspot_index.json from the same raw run on
dash0 and re-render the three affected figures (apc-vs-hotspot,
latency-bars, per-worker TTFT).

Key number changes in window_1_results.md:
- hotspot_index magnitudes corrected (all five policies; lmetric
  smallest delta at +0.7%, sticky largest at +16.1%)
- "capped reduces hotspot 13%" -> "~10% (2.253 -> 2.020)"
- TTFT/E2E/TPOT percentiles shift by <1% from floor->interp
  (unified TTFT p90 7.24 -> 7.35 s)

Restructured "Caveats" into "Limitations (read this before quoting
B3 numbers)":
1. Agentic dispatch coupling is by design — promoted from caveat
   to top-level methodology framing, tied to
   agentic_dispatch_coupling.md
2. B3 interference_index is binary (not size-graded) — added
3. Hot-sweep cache contamination (<1%) — kept
4. Unified interference unrecoverable — kept with explicit warning
   not to read unified's failure attribution as causal
5. w600 is a sample, not full trace — kept
6. Reuse decomposition is per-token in expectation — added

current_results/characterization_claim_matrix.md updates:
- The "heavy-tail not sole cause" claim now cites the corrected
  ~10% drop with the median bug noted
- New supported claim: "B3 saturated-replay latency gaps include an
  agentic dispatch-coupling feedback term, which is intentional and
  matches production"; cited against agentic_dispatch_coupling.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
2026-05-26 01:08:55 +08:00
parent 0e82612100
commit 0881942cf3
11 changed files with 131 additions and 72 deletions

View File

@@ -15,6 +15,7 @@ sweep, B2 PD-colo interference microbench).
| Same-worker prefill-decode interference is causal, not correlation. | `supported` | B2 microbench: different-worker control idx 0.92-1.02 across 32× prefill-size variation; same-worker TTFT idx scales 2.15× (2k) → 218× (65k). window_1_results/b2_sweep_summary.json. | — | Synthetic decode load (256-token prompts at 4 req/s) bounds the realism; production behavior is layered on top of B3. |
| The cost of same-worker prefill interference migrates from TPOT to TTFT as prefill size grows past the chunked-prefill horizon. | `supported` | B2 same-worker TPOT p90 idx peaks at 32k (7.89×) and *drops* at 65k (2.26×), while TTFT idx grows monotonically (94.6× → 218×) and TPOT p99 grows monotonically (59 → 169.5 ms). See window_1_results.md "TPOT idx peaks at 32k, not 65k". | — | SLO thresholds for TTFT and TPOT cannot be the same under PD-colo; this should be reflected in B4 SRR sweep design. |
| Hard session affinity (`sticky`) inflates same-worker prefill-decode interference. | `supported` | sticky interference_index 13.65 vs lmetric 6.53; sticky's slow-request breakdown 57% same-worker overlap vs lmetric 23%. | — | Confirms the B2 causal claim observed at the system level. |
| Heavy-tail sessions are a contributor to hot-spot but not the sole cause. | `supported` | Cap-8 trace (37% requests dropped) reduces hotspot_index only 13% (2.241.94). | Run capped under unified to see whether unified's hotspot also persists. | Reviewer might counter that cap=8 is too soft; a stricter cap could be tried. |
| Heavy-tail sessions are a contributor to hot-spot but not the sole cause. | `supported` | Cap-8 trace (37% requests dropped) reduces hotspot_index only ~10% (2.2532.020 after fixing the `joined_analysis.hotspot_index` median bug). | Run capped under unified to see whether unified's hotspot also persists. | Reviewer might counter that cap=8 is too soft; a stricter cap could be tried. |
| B3 saturated-replay latency gaps include an agentic dispatch-coupling feedback term, which is intentional and matches production. | `supported, framed as feature` | `replayer/replay.py:282-287` fires turn N+1 immediately when turn N is behind schedule (no human think-time). Under saturation, slow policies have longer mean session lifetime, more concurrent in-flight, higher worker pressure — so B3 latency gaps reflect "policy + feedback amplification", which is what a production operator switching policies on agentic workload experiences. See `analysis/characterization/agentic_dispatch_coupling.md`. | Run B4 open-loop Poisson at fixed λ to get the orthogonal "controlled-load" measurement; both are needed, not "B4 fixes B3". | Some reviewers will read "non-Poisson arrivals" as benchmark crime; the rebuttal is the agentic-vs-chat workload distinction. |
| SRR per policy under SLO is not yet measured. | `not_yet_supported` | B3 was driven by trace timestamps with strict session sequentiality; saturation is reached but not parameterized. | Run B4 with the A4 open-loop Poisson loadgen, per-class SLO, 5 policies × λ binary search. | Without B4 the paper cannot claim "policy X sustains higher load than Y". |
| Failure attribution near SRR boundary is not yet measured. | `not_yet_supported` | B5 protocol exists; no runs. | After B4, rerun each policy at 0.9× / 1.0× / 1.1× of its SRR_max with the same instrumentation, label slow requests. | The current `joined_analysis.label_slow_requests` is the labeler; needs SRR boundaries to point at. |