pub struct IGpuAllocator { /* private fields */ }Implementations§
Source§impl IGpuAllocator
impl IGpuAllocator
Sourcepub fn allocate(
self: Pin<&mut IGpuAllocator>,
size: u64,
alignment: u64,
flags: u32,
) -> *mut c_void
pub fn allocate( self: Pin<&mut IGpuAllocator>, size: u64, alignment: u64, flags: u32, ) -> *mut c_void
A thread-safe callback implemented by the application to handle acquisition of GPU memory.
sizeThe size of the memory block required (in bytes).alignmentThe required alignment of memory. Alignment will be zero or a power of 2 not exceeding the alignment guaranteed by cudaMalloc. Thus this allocator can be safely implemented with cudaMalloc/cudaFree. An alignment value of zero indicates any alignment is acceptable.flagsReserved for future use. In the current release, 0 will be passed.
If the allocation was successful, the start address of a device memory block of the requested size. If an allocation request of size 0 is made, nullptr must be returned. If an allocation request cannot be satisfied, nullptr must be returned. If a non-null address is returned, it is guaranteed to have the specified alignment.
The implementation must guarantee thread safety for concurrent allocate/reallocate/deallocate requests.
- Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
Deprecated in TensorRT 10.0. Superseded by allocateAsync
Sourcepub unsafe fn reallocate(
self: Pin<&mut IGpuAllocator>,
arg1: *mut c_void,
arg2: u64,
arg3: u64,
) -> *mut c_void
pub unsafe fn reallocate( self: Pin<&mut IGpuAllocator>, arg1: *mut c_void, arg2: u64, arg3: u64, ) -> *mut c_void
A thread-safe callback implemented by the application to resize an existing allocation.
Only allocations which were allocated with AllocatorFlag::kRESIZABLE will be resized.
Options are one of:
- resize in place leaving min(oldSize, newSize) bytes unchanged and return the original address
- move min(oldSize, newSize) bytes to a new location of sufficient size and return its address
- return nullptr, to indicate that the request could not be fulfilled.
If nullptr is returned, TensorRT will assume that resize() is not implemented, and that the allocation at baseAddr is still valid.
This method is made available for use cases where delegating the resize strategy to the application provides an opportunity to improve memory management. One possible implementation is to allocate a large virtual device buffer and progressively commit physical memory with cuMemMap. CU_MEM_ALLOC_GRANULARITY_RECOMMENDED is suggested in this case.
TensorRT may call realloc to increase the buffer by relatively small amounts.
baseAddrthe address of the original allocation, which will have been returned by previously calling allocate() or reallocate() on the same object.alignmentThe alignment used by the original allocation. This will be the same value that was previously passed to the allocate() or reallocate() call that returned baseAddr.newSizeThe new memory size required (in bytes).
The address of the reallocated memory, or nullptr. If a non-null address is returned, it is guaranteed to have the specified alignment.
The implementation must guarantee thread safety for concurrent allocate/reallocate/deallocate requests.
- Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
Sourcepub unsafe fn deallocate(
self: Pin<&mut IGpuAllocator>,
memory: *mut c_void,
) -> bool
pub unsafe fn deallocate( self: Pin<&mut IGpuAllocator>, memory: *mut c_void, ) -> bool
A thread-safe callback implemented by the application to handle release of GPU memory.
TensorRT may pass a nullptr to this function if it was previously returned by allocate().
memoryA memory address that was previously returned by an allocate() or reallocate() call of the same allocator object.
True if the acquired memory is released successfully.
The implementation must guarantee thread safety for concurrent allocate/reallocate/deallocate requests.
- Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads. Deprecated in TensorRT 10.0. Superseded by deallocateAsync
Sourcepub unsafe fn allocateAsync(
self: Pin<&mut IGpuAllocator>,
size: u64,
alignment: u64,
flags: u32,
arg1: *mut CUstream_st,
) -> *mut c_void
pub unsafe fn allocateAsync( self: Pin<&mut IGpuAllocator>, size: u64, alignment: u64, flags: u32, arg1: *mut CUstream_st, ) -> *mut c_void
A thread-safe callback implemented by the application to handle stream-ordered acquisition of GPU memory.
The default behavior is to call method allocate(), which is synchronous and thus loses any performance benefits of asynchronous allocation. If you want the benefits of asynchronous allocation, see discussion of IGpuAsyncAllocator vs. IGpuAllocator in the documentation for nvinfer1::IGpuAllocator.
sizeThe size of the memory block required (in bytes).alignmentThe required alignment of memory. Alignment will be zero or a power of 2 not exceeding the alignment guaranteed by cudaMalloc. Thus this allocator can be safely implemented with cudaMalloc/cudaFree. An alignment value of zero indicates any alignment is acceptable.flagsReserved for future use. In the current release, 0 will be passed.streamspecifies the cudaStream for asynchronous usage.
If the allocation was successful, the start address of a device memory block of the requested size. If an allocation request of size 0 is made, nullptr must be returned. If an allocation request cannot be satisfied, nullptr must be returned. If a non-null address is returned, it is guaranteed to have the specified alignment.
The implementation must guarantee thread safety for concurrent allocate/reallocate/deallocate requests.
- Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
Sourcepub unsafe fn deallocateAsync(
self: Pin<&mut IGpuAllocator>,
memory: *mut c_void,
arg1: *mut CUstream_st,
) -> bool
pub unsafe fn deallocateAsync( self: Pin<&mut IGpuAllocator>, memory: *mut c_void, arg1: *mut CUstream_st, ) -> bool
A thread-safe callback implemented by the application to handle stream-ordered release of GPU memory.
The default behavior is to call method deallocate(), which is synchronous and thus loses any performance benefits of asynchronous deallocation. If you want the benefits of asynchronous deallocation, see discussion of IGpuAsyncAllocator vs. IGpuAllocator in the documentation for nvinfer1::IGpuAllocator.
TensorRT may pass a nullptr to this function if it was previously returned by allocate().
memoryA memory address that was previously returned by an allocate() or reallocate() call of the same allocator object.streamspecifies the cudaStream for asynchronous usage.
True if the acquired memory is released successfully.
The implementation must guarantee thread safety for concurrent allocate/reallocate/deallocate requests.
The implementation is not required to be asynchronous. It is permitted to synchronize, albeit doing so will lose the performance advantage of asynchronous deallocation. Either way, it is critical that it not actually free the memory until the current stream position is reached.
- Allowed context for the API call
- Thread-safe: Yes, this method is required to be thread-safe and may be called from multiple threads.
Sourcepub fn getInterfaceInfo<'a>(
self: &'a IGpuAllocator,
) -> impl New<Output = InterfaceInfo> + 'a
pub fn getInterfaceInfo<'a>( self: &'a IGpuAllocator, ) -> impl New<Output = InterfaceInfo> + 'a
Return version information associated with this interface. Applications must not override this method.