perf: streams / drop per-op sync
Default-stream kernels run in order and every host read goes through a stream-ordered cudaMemcpy (to_device), so the per-op cudaDeviceSynchronize after each kernel was pure overhead — remove all 21 in tensor.rs. Host data is still correctly ordered by the D2H memcpy that reads it. Also zero op-output buffers with cudaMemset (device-side, async) instead of a blocking H2D memcpy of a host zero buffer on every allocation — that copy was itself a hidden per-op sync point. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -99,9 +99,11 @@ impl Storage {
|
||||
match device {
|
||||
Device::Cpu => Ok(Storage::cpu(vec![0u8; len_bytes])),
|
||||
Device::Cuda(dev) => {
|
||||
// No device memset in T2: stage zeros from the host.
|
||||
// Device-side memset (Phase T7): avoids a blocking H2D memcpy of a
|
||||
// host zero buffer on every op-output allocation. cudaMemset is
|
||||
// async on the default stream, so it doesn't serialize the stream.
|
||||
let mut buf = GpuBuffer::alloc(len_bytes)?;
|
||||
buf.copy_from_host(&vec![0u8; len_bytes])?;
|
||||
buf.memset(0)?;
|
||||
Ok(Storage::cuda(buf, dev))
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user