pub struct StreamMemoryPool { /* private fields */ }Expand description
A memory pool for stream-ordered allocations.
Every pool drives a faithful CPU model of the stream-ordered allocator (the
source of truth for byte accounting, block reuse, and per-stream ordering).
On platforms with a real CUDA driver (Linux, Windows),
StreamMemoryPool::new additionally creates a driver-side pool via
cuMemPoolCreate. StreamMemoryPool::cpu_pool builds a pool backed only
by the CPU model and never touches the driver, so the API can be exercised
on any platform.
§Allocation tracking
The pool maintains running allocation counts and byte totals (mirrored from
the CPU model) for diagnostics; these are available everywhere via
StreamMemoryPool::stats.
Implementations§
Source§impl StreamMemoryPool
impl StreamMemoryPool
Sourcepub fn new(config: StreamOrderedAllocConfig) -> CudaResult<Self>
pub fn new(config: StreamOrderedAllocConfig) -> CudaResult<Self>
Create a new memory pool for the given device.
The configuration is validated and the CPU model is initialised. On
platforms with a real CUDA driver, cuMemPoolCreate is also invoked and
its handle stored; without a driver this fails cleanly.
To obtain a pool that never touches the driver (e.g. for CPU-only use of
the stream-ordered API), use StreamMemoryPool::cpu_pool.
§Errors
CudaError::InvalidValueif the config fails validation.- On non-macOS, any
CudaErrorfromcuMemPoolCreate(e.g.CudaError::NotInitializedwhen no driver is loadable).
Sourcepub fn cpu_pool(config: StreamOrderedAllocConfig) -> CudaResult<Self>
pub fn cpu_pool(config: StreamOrderedAllocConfig) -> CudaResult<Self>
Create a pool backed solely by the faithful CPU model, without touching the CUDA driver.
This always succeeds (given a valid configuration) on every platform and is the recommended entry point for using the stream-ordered allocation semantics on a CPU.
§Errors
CudaError::InvalidValueif the config fails validation.
Sourcepub fn stream_id(stream: &Stream) -> StreamOrderId
pub fn stream_id(stream: &Stream) -> StreamOrderId
Derive a stream-ordering identifier from a genuine Stream.
The stream’s raw handle is a stable, unique token for the lifetime of the stream, which the CPU model uses as the stream’s ordering identity.
Sourcepub fn alloc_async(
&mut self,
size: usize,
stream: u64,
) -> CudaResult<StreamAllocation>
pub fn alloc_async( &mut self, size: usize, stream: u64, ) -> CudaResult<StreamAllocation>
Allocate memory on a stream (stream-ordered), identified by a raw ordering token.
The allocation becomes valid on the stream once the stream reaches the allocation point. A previously-freed block of the same-or-larger size is reused when available; otherwise a fresh block is carved from the pool.
§Errors
CudaError::InvalidValueifsizeis zero.CudaError::OutOfMemoryifmax_pool_sizewould be exceeded.
Sourcepub fn alloc_async_on_stream(
&mut self,
size: usize,
stream: &Stream,
) -> CudaResult<StreamAllocation>
pub fn alloc_async_on_stream( &mut self, size: usize, stream: &Stream, ) -> CudaResult<StreamAllocation>
Allocate memory ordered on a genuine Stream.
This is the recommended entry point when a real CUDA Stream is
available: the allocation is sequenced against that exact stream in the
pool’s StreamOrderModel (see StreamMemoryPool::stream_id).
§Errors
CudaError::InvalidValueifsizeis zero.CudaError::OutOfMemoryifmax_pool_sizewould be exceeded.
Sourcepub fn alloc_on(
&mut self,
size: usize,
stream: StreamOrderId,
) -> CudaResult<StreamAllocation>
pub fn alloc_on( &mut self, size: usize, stream: StreamOrderId, ) -> CudaResult<StreamAllocation>
Allocate memory ordered on the stream identified by stream.
The block is carved from the pool — reusing a previously-freed block of
the same-or-larger size when one is available — and sequenced on the
stream so that it only becomes valid once the stream reaches the
allocation point (queryable via StreamMemoryPool::is_ready).
§Errors
CudaError::InvalidValueifsizeis zero.CudaError::OutOfMemoryifmax_pool_sizewould be exceeded.
Sourcepub fn free_async(&mut self, alloc: &mut StreamAllocation) -> CudaResult<()>
pub fn free_async(&mut self, alloc: &mut StreamAllocation) -> CudaResult<()>
Free memory on a stream (stream-ordered).
The memory is returned to the pool once all prior work on the allocation’s stream has completed. The allocation is marked freed and cannot be freed again.
§Errors
CudaError::InvalidValueif the allocation is already freed, or its pointer is not live in this pool (foreign-pointer free).
Sourcepub fn free_async_on_stream(
&mut self,
alloc: &mut StreamAllocation,
stream: &Stream,
) -> CudaResult<()>
pub fn free_async_on_stream( &mut self, alloc: &mut StreamAllocation, stream: &Stream, ) -> CudaResult<()>
Free alloc ordered on a genuine Stream.
CUDA permits freeing on a stream different from the one the allocation was made on; the free still completes only once that stream reaches the free point.
§Errors
CudaError::InvalidValueif the allocation is already freed or its pointer is not live in this pool.
Sourcepub fn free_on(
&mut self,
alloc: &mut StreamAllocation,
stream: StreamOrderId,
) -> CudaResult<()>
pub fn free_on( &mut self, alloc: &mut StreamAllocation, stream: StreamOrderId, ) -> CudaResult<()>
Free alloc ordered on the stream identified by stream.
§Errors
CudaError::InvalidValueif the allocation is already freed or its pointer is not live in this pool.
Sourcepub fn synchronize_stream(&mut self, stream: StreamOrderId)
pub fn synchronize_stream(&mut self, stream: StreamOrderId)
Advance a stream to its head (model of cuStreamSynchronize),
completing every operation submitted on it so far and reclaiming any
completed stream-ordered frees into the pool for reuse.
Sourcepub fn is_ready(&self, alloc: &StreamAllocation) -> bool
pub fn is_ready(&self, alloc: &StreamAllocation) -> bool
Returns true if alloc is valid to read on its own ordering stream,
i.e. that stream has executed past the allocation point.
Sourcepub fn is_ready_on(
&self,
alloc: &StreamAllocation,
consumer: StreamOrderId,
wait_seq: u64,
) -> bool
pub fn is_ready_on( &self, alloc: &StreamAllocation, consumer: StreamOrderId, wait_seq: u64, ) -> bool
Returns true if alloc (made on its own stream) is safe to use on
consumer given that consumer was ordered after wait_seq on the
allocation’s stream (the sequence captured by an event it waited on).
Use StreamMemoryPool::record_event on the producing stream to obtain
a wait_seq that captures the allocation.
Sourcepub fn record_event(&mut self, stream: StreamOrderId) -> u64
pub fn record_event(&mut self, stream: StreamOrderId) -> u64
Record an event on stream, returning the sequence number it captures.
A later cross-stream wait on this value orders the waiting stream after
every operation submitted on stream before this point.
Sourcepub fn model_trim(&mut self, min_bytes_to_keep: usize)
pub fn model_trim(&mut self, min_bytes_to_keep: usize)
Trim the CPU model’s pool, releasing free-list bytes above
min_bytes_to_keep back to the (virtual) device.
This is the CPU-model analogue of cuMemPoolTrimTo; it always succeeds
and is available on every platform. For the raw driver trim, see the
platform-gated StreamMemoryPool::trim.
Sourcepub fn trim(&mut self, min_bytes_to_keep: usize) -> CudaResult<()>
pub fn trim(&mut self, min_bytes_to_keep: usize) -> CudaResult<()>
Trim the driver-side pool, releasing unused memory back to the OS.
At least min_bytes_to_keep bytes of reserved memory remain in the
pool. This drives the real cuMemPoolTrimTo binding; for the
always-available CPU-model trim, use StreamMemoryPool::model_trim.
§Errors
CudaError::NotSupportedon macOS.- Any
CudaErrorfromcuMemPoolTrimTo.
Sourcepub fn stats(&self) -> PoolUsageStats
pub fn stats(&self) -> PoolUsageStats
Get pool usage statistics from the CPU model.
reserved_* reflects everything the pool has carved from the (virtual)
device (live + reusable + pending-free bytes), whereas used_* reflects
only currently-live allocations.
Sourcepub fn set_attribute(&mut self, attr: PoolAttribute) -> CudaResult<()>
pub fn set_attribute(&mut self, attr: PoolAttribute) -> CudaResult<()>
Set a pool attribute.
Only attributes that carry a value (e.g. PoolAttribute::ReleaseThreshold)
modify pool state. Read-only attributes (e.g. ReservedMemCurrent)
return CudaError::InvalidValue.
The release threshold is applied to the CPU model as well as (on non-macOS) the driver pool.
§Errors
CudaError::InvalidValuefor read-only attributes.CudaError::NotSupportedon macOS for non-threshold attributes.
Sourcepub fn enable_peer_access(&self, peer_device: i32) -> CudaResult<()>
pub fn enable_peer_access(&self, peer_device: i32) -> CudaResult<()>
Enable peer access from another device to allocations in this pool.
After this call, kernels running on peer_device can access memory
allocated from this pool.
§Errors
CudaError::InvalidDeviceifpeer_deviceequals this pool’s device.CudaError::NotSupportedon macOS.
Sourcepub fn disable_peer_access(&self, peer_device: i32) -> CudaResult<()>
pub fn disable_peer_access(&self, peer_device: i32) -> CudaResult<()>
Disable peer access from another device to allocations in this pool.
§Errors
CudaError::InvalidDeviceifpeer_deviceequals this pool’s device.CudaError::NotSupportedon macOS.
Sourcepub fn reset_peak_stats(&mut self)
pub fn reset_peak_stats(&mut self)
Reset peak statistics (peak used bytes and peak allocation count).
Sourcepub fn default_pool(device: i32) -> CudaResult<Self>
pub fn default_pool(device: i32) -> CudaResult<Self>
Get the default memory pool for a device.
CUDA provides a default pool per device, queried via
cuDeviceGetDefaultMemPool. The returned pool is owned by the
driver and is not destroyed when the StreamMemoryPool wrapper
is dropped. On macOS, this returns a local-only pool with default
configuration. In all cases the CPU model is initialised.
§Errors
CudaError::InvalidValueifdeviceis negative.CudaError::NotInitializedif the CUDA driver is not loaded.- Any
CudaErrormapped fromcuDeviceGetDefaultMemPool.
Sourcepub fn config(&self) -> &StreamOrderedAllocConfig
pub fn config(&self) -> &StreamOrderedAllocConfig
Returns the pool configuration.