Module cuda 

CUDA memory pool for efficient device memory allocation in hot paths.

This module provides a safe wrapper around CUDA’s memory pool APIs, enabling fast async allocations that avoid the overhead of cudaMalloc/cudaFree per call. Memory is returned to the pool on free and reused for subsequent allocations.

Thread Safety

CudaMemPool uses internal locking to serialize host-side calls to the CUDA driver. This is required because cuMemAllocFromPoolAsync is not host-thread reentrant. The GPU-side operations remain stream-ordered and asynchronous.

Structs

CudaMemPool

Safe wrapper around a CUDA memory pool.

CudaMemPoolBuilder

Builder for creating a CUDA memory pool with configurable parameters.