# Phase 14 Benchmark: Flash Attention 2

**Date**: 2026-05-22
**Hardware**: RTX 5090 (32GB GDDR7, SM120 CC 12.0, 170 SMs)
**Model**: Qwen3-8B (BF16, 36 layers, 4096 hidden, 32 Q / 8 KV GQA heads, head_dim=128)
**Config**: greedy decoding (temperature=0), max_tokens=64, single-request serial

## Correctness

Logits comparison with HuggingFace transformers (10 prompts, raw text without ChatML):

| Metric | Result |
|--------|--------|
| Prefill Top-1 match vs HF | **9/10 (90%)** |
| Avg Top-5 overlap vs HF | **4.0/5** |
| Result vs pre-FA2 naive attention | **Identical** (same 9/10 top-1, same 4.0/5 overlap) |

The single top-1 mismatch ("Explain quantum computing.") has logits differing by 0.125
(22.000 vs 21.875) — within BF16 precision. The top-5 sets are identical (5/5 overlap).

FA2 introduces no precision degradation compared to the naive attention path.

## API Generation

52 diverse prompts (English, Chinese, code) via `/v1/chat/completions`:

| Metric | Result |
|--------|--------|
| Success rate | **52/52 (100%)** |
| SSE streaming | **Working** (role chunk, content chunks, finish_reason, [DONE]) |
| Usage stats | Correct (prompt_tokens + completion_tokens = total_tokens) |

## Performance

### xserv vs HuggingFace transformers

8 prompts (short/medium/long) × max_tokens=64, greedy:

| Category | Prompt Tokens | xserv (tok/s) | HF (tok/s) | Ratio |
|----------|--------------|---------------|------------|-------|
| Short (~12 tok) | 12-14 | 12.5 | 38.5 | 0.32x |
| Medium (~28 tok) | 27-28 | 13.6 | 44.1 | 0.31x |
| Long (~60 tok) | 58-64 | 13.0 | 36.0 | 0.36x |
| **Overall** | — | **12.9** | **36.6** | **0.35x** |

### Phase-over-Phase Improvement

| Phase | Attention | repeat_kv | tok/s | vs HF |
|-------|-----------|-----------|-------|-------|
| 10 | Naive (O(S²), cuBLAS batched) | CPU round-trip | 6.9 | 15% |
| 11 | Naive + GPU KV cache | GPU repeat_kv | 10.3 | 30% |
| **14** | **FA2 (O(1), fused kernel)** | **None (GQA in kernel)** | **12.9** | **35%** |

Phase 14 vs Phase 11: **+25% throughput** (10.3 → 12.9 tok/s).

### Improvement Breakdown (estimated)

| Factor | Contribution |
|--------|-------------|
| Eliminating repeat_kv GPU alloc + copy (per layer) | ~10% |
| Eliminating K^T transpose + contiguous | ~5% |
| Eliminating S×S score matrix alloc | ~5% |
| Fused kernel (1 launch vs 6) | ~5% |

### Concurrent Requests

8 concurrent requests, max_batch=4:

| Metric | Result |
|--------|--------|
| Wall clock | 22.5s |
| Sum of individual latencies | 135.0s |
| Scheduling speedup | **6.0x** |
| Throughput | 11.4 tok/s |

Continuous batching scheduling confirmed working (decode batch_size=4 in logs).

## Remaining Performance Gap

35% of HF throughput. Main bottlenecks:

| Bottleneck | Impact | Fix |
|-----------|--------|-----|
| **Decode Q_len=1 inefficiency** | FA2 kernel: 64 threads, only 1 active (owns_row=true for single query) | Specialized decode attention kernel (vector-dot against KV, parallel reduction along S) |
| **No kernel fusion** | RMSNorm+residual, SiLU*up: separate kernels, redundant HBM reads/writes | Fused kernels (Phase 15) |
| **No CUDA Graphs** | ~100+ kernel launches per decode step, each has host-side overhead | Capture decode iteration as CUDA Graph (Phase 15) |
| **Per-seq forward (no batched decode)** | With batch=4, 4 serial forward passes per iteration | Batched projections + per-seq attention (Phase 15, depends on FA2 decode kernel) |
| **No vectorized loads in FA2** | Scalar bf16→f32 conversion in dot product loop | float4 / bfloat162 vectorized loads |

## Memory Usage

| Component | Naive (Phase 11) | FA2 (Phase 14) |
|-----------|-----------------|----------------|
| Score matrix [1, 32, S, S] | S² × 32 × 2B | **0** |
| repeat_kv K/V [1, 32, S, 128] | 2 × S × 32 × 128 × 2B per layer | **0** |
| K^T contiguous copy | S × 32 × 128 × 2B per layer | **0** |

For S=256 (current max): savings ~6 MB per layer × 36 layers ≈ 216 MB.
For S=2048: savings ~384 MB per layer × 36 layers ≈ 13.5 GB (naive would OOM).

## Tracking

| Phase | Attention | tok/s | vs HF | Correctness |
|-------|-----------|-------|-------|-------------|
| 8 | Naive (no cache) | 2.5 | 5% | 50/50 vs HF |
| 9 | Naive + CPU KV cache | 44.3 (GPT-2) | — | 50/50 self |
| 10 | Naive + CPU KV cache | 6.9 (Qwen3-8B) | 15% | 100% top-5 |
| 11 | Naive + GPU KV cache | 10.3 | 30% | 9/10 top-1 |
| **14** | **FA2 + GQA in kernel** | **12.9** | **35%** | **9/10 top-1** |