gahow/aituner
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
e0811e95b4c645e275bb00a5c75c82e4e4110fec
aituner/runs/frontier-split-rootcause-v0/frozen-inputs/q30-trace-pd/comparison.md
Gahow Wang 39766141fb Decompose good/bad selection split across frozen surfaces 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 12:05:20 +08:00

1.6 KiB
Raw Blame History

Frontier piecewise graph-profile vs. real vLLM

Each cell uses its TP-normalized, 129-request trace. Frontier values are one deterministic simulation; real values pool three fresh-server trials (387 requests/cell).

Config TTFT sim / real mean (ms) TPOT sim / real mean (ms) E2E sim / real mean (ms)
tp1_mns8 1669003.8 / 88479.9 76.0 / 14.3 1930772.9 / 137996.4
tp1_mns16 870562.2 / 17378.2 99.1 / 20.6 1209178.7 / 87355.2
tp1_mns32 316980.6 / 620.5 119.6 / 23.8 722298.9 / 81051.2
tp1_mns64 105003.3 / 619.8 137.8 / 24.1 549486.0 / 82253.7
tp2_mns8 1171654.6 / 95920.1 51.1 / 9.2 1348614.2 / 128296.9
tp2_mns16 606885.2 / 34443.0 64.4 / 14.1 828082.5 / 82936.7
tp2_mns32 224495.4 / 1050.6 74.8 / 18.0 479340.1 / 62267.5
tp2_mns64 30524.5 / 375.8 82.6 / 18.2 311160.1 / 62353.6
tp4_mns8 834500.2 / 102605.9 35.4 / 6.8 957838.5 / 126405.4
tp4_mns16 429098.7 / 35042.6 43.2 / 8.8 578209.7 / 65779.1
tp4_mns32 166734.5 / 6296.2 49.4 / 12.2 336309.9 / 47944.5
tp4_mns64 26688.5 / 246.0 53.7 / 13.2 209788.3 / 44985.1

Selection agreement

Target Frontier winner Real winner Match Pairwise agreement
ttft_ms:mean tp4_mns64 tp4_mns64 yes 84.8% (56/66)
ttft_ms:p90 tp4_mns64 tp4_mns64 yes 84.8% (56/66)
tpot_ms:mean tp4_mns8 tp4_mns8 yes 97.0% (64/66)
tpot_ms:p90 tp4_mns8 tp4_mns8 yes 97.0% (64/66)
e2e_ms:mean tp4_mns64 tp4_mns64 yes 92.4% (61/66)
e2e_ms:p90 tp4_mns64 tp4_mns64 yes 83.3% (55/66)