Struct IRefitter 

pub struct IRefitter { /* private fields */ }

IRefitter

Updates weights in an engine.

Do not inherit from this class, as doing so will break forward-compatibility of the API and ABI.

Implementations

impl IRefitter

pub unsafe fn setNamedWeights1( self: Pin<&mut Self>, name: *const c_char, weights: Weights, location: TensorLocation, ) -> bool

Specify new weights on a specified device of given name.

• name The name of the weights to be refitted.
• weights The new weights on the specified device.
• location The location (host vs. device) of the new weights.

True on success, or false if new weights are rejected. Possible reasons for rejection are:

• The name of the weights is nullptr or does not correspond to any refittable weights.
• The count of the weights is inconsistent with the count returned from calling getWeightsPrototype() with the same name.
• The type of the weights is inconsistent with the type returned from calling getWeightsPrototype() with the same name.

It is allowed to provide some weights on CPU and others on GPU. Modifying the weights before the method refitCudaEngine() or refitCudaEngineAsync() completes will result in undefined behavior.

The string name must be null-terminated, and be at most 4096 bytes including the terminator.

impl IRefitter

pub unsafe fn setWeights( self: Pin<&mut IRefitter>, layerName: *const c_char, role: WeightsRole, weights: Weights, ) -> bool

Specify new weights for a layer of given name. Returns true on success, or false if new weights are rejected. Possible reasons for rejection are:

• There is no such layer by that name.
• The layer does not have weights with the specified role.
• The count of weights is inconsistent with the layer’s original specification.
• The type of weights is inconsistent with the layer’s original specification.

Modifying the weights before method refitCudaEngine or refitCudaEngineAsync returns will result in undefined behavior.

The string layerName must be null-terminated, and be at most 4096 bytes including the terminator.

pub fn refitCudaEngine(self: Pin<&mut IRefitter>) -> bool

Refits associated engine.

True on success, or false if new weights validation fails or getMissingWeights() != 0 before the call. If false is returned, a subset of weights may have been refitted.

The behavior is undefined if the engine has pending enqueued work. Provided weights on CPU or GPU can be unset and released, or updated after refitCudaEngine returns.

IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead.

pub unsafe fn setErrorRecorder( self: Pin<&mut IRefitter>, recorder: *mut IErrorRecorder, )

See [getErrorRecorder()]

pub fn getErrorRecorder(self: &IRefitter) -> *mut IErrorRecorder

Get the ErrorRecorder assigned to this interface.

Retrieves the assigned error recorder object for the given class. A nullptr will be returned if an error handler has not been set.

A pointer to the IErrorRecorder object that has been registered.

See [setErrorRecorder()]

pub unsafe fn setNamedWeights( self: Pin<&mut IRefitter>, name: *const c_char, weights: Weights, ) -> bool

Specify new weights of given name.

• name The name of the weights to be refit.
• weights The new weights to associate with the name.

Returns true on success, or false if new weights are rejected. Possible reasons for rejection are:

• The name of weights is nullptr or does not correspond to any refittable weights.
• The count of the weights is inconsistent with the count returned from calling getWeightsPrototype() with the same name.
• The type of the weights is inconsistent with the type returned from calling getWeightsPrototype() with the same name.

Modifying the weights before method refitCudaEngine or refitCudaEngineAsync returns will result in undefined behavior.

The string name must be null-terminated, and be at most 4096 bytes including the terminator.

pub fn getLogger(self: &IRefitter) -> *mut ILogger

get the logger with which the refitter was created

the logger

pub fn setMaxThreads(self: Pin<&mut IRefitter>, maxThreads: i32) -> bool

Set the maximum number of threads.

• maxThreads The maximum number of threads that can be used by the refitter.

True if successful, false otherwise.

The default value is 1 and includes the current thread. A value greater than 1 permits TensorRT to use multi-threaded algorithms. A value less than 1 triggers a kINVALID_ARGUMENT error.

pub fn getMaxThreads(self: &IRefitter) -> i32

get the maximum number of threads that can be used by the refitter.

Retrieves the maximum number of threads that can be used by the refitter.

The maximum number of threads that can be used by the refitter.

See [setMaxThreads()]

pub unsafe fn getNamedWeights( self: &IRefitter, weightsName: *const c_char, ) -> Weights

Get weights associated with the given name.

• weightsName The name of the weights to be refitted.

Weights associated with the given name.

If the weights were never set, returns null weights and reports an error to the refitter errorRecorder.

The string weightsName must be null-terminated, and be at most 4096 bytes including the terminator.

pub unsafe fn getWeightsLocation( self: &IRefitter, weightsName: *const c_char, ) -> TensorLocation

Get location for the weights associated with the given name.

• weightsName The name of the weights to be refitted.

Location for the weights associated with the given name.

If the weights were never set, returns TensorLocation::kHOST and reports an error to the refitter errorRecorder.

The string weightsName must be null-terminated, and be at most 4096 bytes including the terminator.

pub unsafe fn unsetNamedWeights( self: Pin<&mut IRefitter>, weightsName: *const c_char, ) -> bool

Unset weights associated with the given name.

• weightsName The name of the weights to be refitted.

False if the weights were never set, returns true otherwise.

Unset weights before releasing them.

The string weightsName must be null-terminated, and be at most 4096 bytes including the terminator.

pub fn setWeightsValidation(self: Pin<&mut IRefitter>, weightsValidation: bool)

Set whether to validate weights during refitting.

• weightsValidation Indicate whether to validate weights during refitting.

When set to true, TensorRT will validate weights during FP32 to FP16/BF16 weights conversions or sparsifying weights in the refit call. If provided weights are not proper for some weights transformations, TensorRT will issue a warning and continue the transformation for minor issues (such as overflow during narrowing conversion), or issue an error and stop the refitting process for severe issues (such as sparsifying dense weights). By default the flag is true. Set the flag to false for faster refitting performance.

pub fn getWeightsValidation(self: &IRefitter) -> bool

Get whether to validate weights values during refitting.

pub unsafe fn refitCudaEngineAsync( self: Pin<&mut IRefitter>, stream: *mut CUstream_st, ) -> bool

Enqueue weights refitting of the associated engine on the given stream.

• stream The stream to enqueue the weights updating task.

True on success, or false if new weights validation fails or getMissingWeights() != 0 before the call. If false is returned, a subset of weights may have been refitted.

The behavior is undefined if the engine has pending enqueued work on a different stream from the provided one. Provided weights on CPU can be unset and released, or updated after refitCudaEngineAsync returns. Freeing or updating of the provided weights on GPU can be enqueued on the same stream after refitCudaEngineAsync returns.

IExecutionContexts associated with the engine remain valid for use afterwards. There is no need to set the same weights repeatedly for multiple refit calls as the weights memory can be updated directly instead. The weights updating task should use the same stream as the one used for the refit call.

pub unsafe fn getWeightsPrototype( self: &IRefitter, weightsName: *const c_char, ) -> Weights

Get the Weights prototype associated with the given name.

• weightsName The name of the weights to be refitted.

Weights prototype associated with the given name.

The type and count of weights prototype is the same as weights used for engine building. The values property is nullptr for weights prototypes. The count of the weights prototype is -1 when the name of the weights is nullptr or does not correspond to any refittable weights.

The string weightsName must be null-terminated, and be at most 4096 bytes including the terminator.

Trait Implementations

impl Drop for IRefitter

fn drop(self: &mut IRefitter)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.

impl ExternType for IRefitter

type Id = (n, v, i, n, f, e, r, _1, (), I, R, e, f, i, t, t, e, r)

A type-level representation of the type’s C++ namespace and type name.

type Kind = Opaque

Either cxx::kind::Opaque or cxx::kind::Trivial.

impl MakeCppStorage for IRefitter

unsafe fn allocate_uninitialized_cpp_storage() -> *mut IRefitter

Allocates heap space for this type in C++ and return a pointer to that space, but do not initialize that space (i.e. do not yet call a constructor).

unsafe fn free_uninitialized_cpp_storage(arg0: *mut IRefitter)

Frees a C++ allocation which has not yet had a constructor called.

impl SharedPtrTarget for IRefitter

impl UniquePtrTarget for IRefitter

impl WeakPtrTarget for IRefitter