perf: streams / drop per-op sync

Default-stream kernels run in order and every host read goes through a
stream-ordered cudaMemcpy (to_device), so the per-op cudaDeviceSynchronize
after each kernel was pure overhead — remove all 21 in tensor.rs. Host
data is still correctly ordered by the D2H memcpy that reads it.

Also zero op-output buffers with cudaMemset (device-side, async) instead of
a blocking H2D memcpy of a host zero buffer on every allocation — that
copy was itself a hidden per-op sync point.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-06-15 16:56:17 +08:00
parent 8070c1949a
commit a842e432b5
4 changed files with 11 additions and 23 deletions

View File

@@ -99,9 +99,11 @@ impl Storage {
match device {
Device::Cpu => Ok(Storage::cpu(vec![0u8; len_bytes])),
Device::Cuda(dev) => {
// No device memset in T2: stage zeros from the host.
// Device-side memset (Phase T7): avoids a blocking H2D memcpy of a
// host zero buffer on every op-output allocation. cudaMemset is
// async on the default stream, so it doesn't serialize the stream.
let mut buf = GpuBuffer::alloc(len_bytes)?;
buf.copy_from_host(&vec![0u8; len_bytes])?;
buf.memset(0)?;
Ok(Storage::cuda(buf, dev))
}
}