phase 15: Tensor::empty + CUDA Graph infra — 50.3 tok/s (140% of HF, 45% roofline)

Two optimizations:

1. Tensor::empty() — skip cudaMemset for output tensors
   All kernel wrappers that fully overwrite their output now use
   Tensor::empty() instead of Tensor::zeros(). Eliminates ~756
   cudaMemset calls per decode step (21 per layer × 36 layers).
   Improvement: 46.6 → 50.3 tok/s (+8%).

2. CUDA Graph infrastructure (for future use)
   Added FFI bindings (cudaStreamBeginCapture, cudaGraphInstantiate,
   cudaGraphLaunch) and RAII CudaGraph wrapper. Not yet used in the
   forward pass due to variable kv_len, but provides foundation for
   future graph-based decode optimization.

Ablation (dash5, RTX 5090, Qwen3-8B BF16, serial decode):

| Optimization | tok/s | vs HF | Roofline |
|-------------|-------|-------|----------|
| Phase 14 baseline | 12.9 | 36% | 12% |
| + Fused kernels | 13.2 | 37% | 12% |
| + Batched decode | 13.2 (serial) | 37% | 12% |
| + Custom GEMV | 46.6 | 130% | 42% |
| + Tensor::empty | 50.3 | 140% | 45% |

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-05-22 23:57:34 +08:00
parent e207523e21
commit d5532ef209
13 changed files with 170 additions and 20 deletions

View File

@@ -116,4 +116,18 @@ impl Storage {
}
}
}
/// Allocate storage **without zeroing** on the given device.
/// The buffer may contain stale data from the caching allocator's pool.
/// Only use when the caller guarantees the kernel will fully overwrite
/// every element before any read.
pub fn empty(len_bytes: usize, device: Device) -> CudaResult<Self> {
match device {
Device::Cpu => Ok(Storage::cpu(vec![0u8; len_bytes])), // CPU still zeros (cheap)
Device::Cuda(dev) => {
let buf = xserv_cuda::allocator::cached_alloc(len_bytes)?;
Ok(Storage::cuda(buf, dev))
}
}
}
}

View File

@@ -65,6 +65,22 @@ impl Tensor {
}
}
/// Allocate a tensor **without zeroing** the backing memory.
/// The buffer may contain stale data. Only use when the calling kernel
/// will fully overwrite every element before any read.
pub fn empty(shape: &[usize], dtype: DType, device: Device) -> Self {
let numel = shape::num_elements(shape);
let len_bytes = numel * dtype.size_bytes();
let storage = Storage::empty(len_bytes, device).expect("alloc failed");
Self {
storage,
shape: Dims::from_slice(shape),
strides: shape::contiguous_strides(shape),
offset: 0,
dtype,
}
}
pub fn ones(shape: &[usize], dtype: DType) -> Self {
let numel = shape::num_elements(shape);
match dtype {