cuda/kernels.cu:709 and :773:
CUDA_CHECK(cudaMalloc(&d_shape, ndim * sizeof(int)));
...
cudaFree(d_shape);
cudaFree is synchronising, it drains the device. So every sum_axis (which
means every broadcast backward, since reduce_to goes through it) and every min
costs a full device sync plus a driver round trip. That's precisely the cost the
CudaBuffer caching allocator was built to avoid.
fastnn_cuda_min also allocates a full n element scratch buffer per call.
Pass the shape through a by value struct the way permute_nd already does (up to
8 dims), and put scratch through CudaBuffer.
Minor, in the same function: int axis_size = 1; is dead and the first two
assignments to output_size are never read.
cuda/kernels.cu:709and:773:cudaFreeis synchronising, it drains the device. So everysum_axis(whichmeans every broadcast backward, since
reduce_togoes through it) and everymincosts a full device sync plus a driver round trip. That's precisely the cost the
CudaBuffercaching allocator was built to avoid.fastnn_cuda_minalso allocates a full n element scratch buffer per call.Pass the shape through a by value struct the way
permute_ndalready does (up to8 dims), and put scratch through
CudaBuffer.Minor, in the same function:
int axis_size = 1;is dead and the first twoassignments to
output_sizeare never read.