Crate trtx_sys

Expand description

Raw FFI bindings to NVIDIA TensorRT-RTX using autocxx

⚠️ EXPERIMENTAL - NOT FOR PRODUCTION USE

This crate is in early experimental development. The API is unstable and will change. This is NOT production-ready software. Use at your own risk.

This crate provides low-level, unsafe bindings to the TensorRT-RTX C++ library. For safe, ergonomic Rust API, use the trtx crate instead.

§Architecture

This crate uses a hybrid approach:

autocxx for direct C++ bindings to TensorRT classes
Minimal C wrapper for Logger callbacks (virtual methods)

§Safety

All functions in this crate are unsafe as they directly call into C++ code and perform no safety checks. Callers must ensure:

Pointers are valid and properly aligned
Lifetimes are managed correctly
Thread safety requirements are met
CUDA context is properly initialized

Re-exports§

pub use nvinfer1::Dims64;
pub use nvinfer1::Weights;

Modules§

nvinfer1

Structs§

RustLoggerBridge

Enums§

ActivationType: ! ! \enum ActivationType ! ! \brief Enumerates the types of activation to perform in an activation layer. !
AttentionNormalizationOp: ! ! \enum AttentionNormalizationOp ! ! \brief Enumerates the operations that may be performed by the normalization in the attention subgraph. !
BuilderFlag: ! ! \enum BuilderFlag ! ! \brief List of valid modes that the builder can enable when creating an engine from a network definition. ! ! \see IBuilderConfig::setFlags(), IBuilderConfig::getFlags() !
CollectiveOperation: ! ! \enum CollectiveOperation ! ! \brief Enumerates the collective operations that may be performed by a DistCollective layer. ! ! \see IDistCollectiveLayer !
ComputeCapability: ! ! \enum ComputeCapability ! ! \brief Describes compute capability that an engine will be built for. !
CumulativeOperation: ! ! \enum CumulativeOperation ! ! \brief Enumerates the cumulative operations that may be performed by a Cumulative layer. ! ! The table shows the initial value of each Cumulative operation. ! ! Operation | kFLOAT, kHALF, kBF16 | kINT32, kINT64 | ! ——— | –––––––––– | ––––––– | ! kSUM | +0.0 | 0 | !
DataType: ! ! \enum DataType ! \brief The type of weights and tensors. ! The datatypes other than kBOOL, kINT32, and kINT64 are “activation datatypes,” ! as they often represent values corresponding to inference results. !
DeviceType: ! ! \enum DeviceType ! \brief The device that this layer/network will execute on. ! !
ElementWiseOperation: ! ! \enum ElementWiseOperation ! ! \brief Enumerates the binary operations that may be performed by an ElementWise layer. ! ! Operations kAND, kOR, and kXOR must have inputs of DataType::kBOOL. ! ! All other operations must have inputs of floating-point type, DataType::kINT8, DataType::kINT32, or ! DataType::kINT64. ! ! \see IElementWiseLayer !
EngineCapability: ! ! \enum EngineCapability ! ! \brief List of supported engine capability flows. ! ! \details The EngineCapability determines the restrictions of a network during build time and what runtime ! it targets. EngineCapability::kSTANDARD does not provide any restrictions on functionality and the resulting ! serialized engine can be executed with TensorRT’s standard runtime APIs in the nvinfer1 namespace. ! EngineCapability::kSAFETY provides a restricted subset of network operations that are safety certified and the ! resulting serialized engine can be executed with TensorRT’s safe runtime APIs in the nvinfer2::safe namespace. ! EngineCapability::kDLA_STANDALONE provides a restricted subset of network operations that are DLA compatible and the ! resulting serialized engine can be executed using standalone DLA runtime APIs. See sampleCudla for an example of ! integrating cuDLA APIs with TensorRT APIs. !
ErrorCode: ! ! \enum ErrorCode ! ! \brief Error codes that can be returned by TensorRT during execution. !
GatherMode: ! ! \brief Control form of IGatherLayer ! ! \see IGatherLayer !
HardwareCompatibilityLevel: ! ! \enum HardwareCompatibilityLevel ! ! \brief Describes requirements of compatibility with GPU architectures other than that of the GPU on which the engine ! was built. ! ! \warning Note that compatibility with future hardware depends on CUDA forward compatibility support. !
InterpolationMode: ! \enum InterpolationMode ! ! \brief Enumerates various modes of interpolation ! !
LayerInformationFormat: ! ! \enum LayerInformationFormat ! ! \brief The format in which the IEngineInspector prints the layer information. ! ! \see IEngineInspector::getLayerInformation(), IEngineInspector::getEngineInformation() !
LayerType: ! ! \enum LayerType ! ! \brief The type values of layer classes. ! ! \see ILayer::getType() !
LoopOutput: ! ! \enum LoopOutput ! ! \brief Enum that describes kinds of loop outputs. !
MatrixOperation: ! ! \enum MatrixOperation ! ! \brief Enumerates the operations that may be performed on a tensor ! by IMatrixMultiplyLayer before multiplication. !
MemoryPoolType: ! ! \enum MemoryPoolType ! ! \brief The type for memory pools used by TensorRT. ! ! \see IBuilderConfig::setMemoryPoolLimit, IBuilderConfig::getMemoryPoolLimit !
MoEActType: ! ! \enum MoEActType ! ! \brief Enumerates the activation type for the MoE layer. !
OptProfileSelector: ! ! \enum OptProfileSelector ! ! \brief When setting or querying optimization profile parameters (such as shape tensor inputs or dynamic dimensions), ! select whether we are interested in the minimum, optimum, or maximum values for these parameters. ! The minimum and maximum specify the permitted range that is supported at runtime, while the optimum value ! is used for the kernel selection. This should be the “typical” value that is expected to occur at runtime. ! ! \see IOptimizationProfile::setDimensions(), IOptimizationProfile::setShapeValuesV2(), IOptimizationProfile::setShapeValues() !
PaddingMode: ! ! \enum PaddingMode ! ! \brief Enumerates the modes of padding to perform in convolution, deconvolution and pooling layer, ! padding mode takes precedence if setPaddingMode() and setPrePadding() are also used. ! ! There are two padding styles, EXPLICIT and SAME with each style having two variants. ! The EXPLICIT style determine if the final sampling location is used or not. ! The SAME style determine if the asymmetry in the padding is on the pre or post padding. ! ! \code ! Shorthand: ! I = dimensions of input image. ! B = prePadding, before the image data. ! A = postPadding, after the image data. ! P = delta between input and output ! S = stride ! F = filter ! O = output ! D = dilation ! M = I + B + A ; The image data plus any padding ! DK = 1 + D * (F - 1) ! \endcode ! ! Formulas for Convolution: ! - EXPLICIT_ROUND_DOWN: ! \code ! O = floor((M - DK) / S) + 1 ! \endcode ! - EXPLICIT_ROUND_UP: ! \code ! O = ceil((M - DK) / S) + 1 ! \endcode ! - SAME_UPPER: ! \code ! O = ceil(I / S) ! P = floor((I - 1) / S) * S + DK - I; ! B = floor(P / 2) ! A = P - B ! \endcode ! - SAME_LOWER: ! \code ! O = ceil(I / S) ! P = floor((I - 1) / S) * S + DK - I; ! A = floor(P / 2) ! B = P - A ! \endcode ! ! Formulas for Deconvolution: ! - EXPLICIT_ROUND_DOWN: ! - EXPLICIT_ROUND_UP: ! \code ! O = (I - 1) * S + DK - (B + A) ! \endcode ! - SAME_UPPER: ! \code ! O = min(I * S, (I - 1) * S + DK) ! P = max(DK - S, 0) ! B = floor(P / 2) ! A = P - B ! \endcode ! - SAME_LOWER: ! \code ! O = min(I * S, (I - 1) * S + DK) ! P = max(DK - S, 0) ! A = floor(P / 2) ! B = P - A ! \endcode ! ! Formulas for Pooling: ! - EXPLICIT_ROUND_DOWN: ! \code ! O = floor((M - F) / S) + 1 ! \endcode ! - EXPLICIT_ROUND_UP: ! \code ! O = ceil((M - F) / S) + 1 ! \endcode ! - SAME_UPPER: ! \code ! O = ceil(I / S) ! P = floor((I - 1) / S) * S + F - I; ! B = floor(P / 2) ! A = P - B ! \endcode ! - SAME_LOWER: ! \code ! O = ceil(I / S) ! P = floor((I - 1) / S) * S + F - I; ! A = floor(P / 2) ! B = P - A ! \endcode ! ! Pooling Example 1: ! \code ! Given I = {6, 6}, B = {3, 3}, A = {2, 2}, S = {2, 2}, F = {3, 3}. What is O? ! (B, A can be calculated for SAME_UPPER and SAME_LOWER mode) ! \endcode ! ! - EXPLICIT_ROUND_DOWN: ! \code ! Computation: ! M = {6, 6} + {3, 3} + {2, 2} ==> {11, 11} ! O ==> floor((M - F) / S) + 1 ! ==> floor(({11, 11} - {3, 3}) / {2, 2}) + {1, 1} ! ==> floor({8, 8} / {2, 2}) + {1, 1} ! ==> {5, 5} ! \endcode ! - EXPLICIT_ROUND_UP: ! \code ! Computation: ! M = {6, 6} + {3, 3} + {2, 2} ==> {11, 11} ! O ==> ceil((M - F) / S) + 1 ! ==> ceil(({11, 11} - {3, 3}) / {2, 2}) + {1, 1} ! ==> ceil({8, 8} / {2, 2}) + {1, 1} ! ==> {5, 5} ! \endcode ! The sample points are {0, 2, 4, 6, 8} in each dimension. ! ! - SAME_UPPER: ! \code ! Computation: ! I = {6, 6} ! S = {2, 2} ! O = ceil(I / S) = {3, 3} ! P = floor((I - 1) / S) * S + F - I ! ==> floor(({6, 6} - {1, 1}) / {2, 2}) * {2, 2} + {3, 3} - {6, 6} ! ==> {4, 4} + {3, 3} - {6, 6} ! ==> {1, 1} ! B = floor({1, 1} / {2, 2}) ! ==> {0, 0} ! A = {1, 1} - {0, 0} ! ==> {1, 1} ! \endcode ! - SAME_LOWER: ! \code ! Computation: ! I = {6, 6} ! S = {2, 2} ! O = ceil(I / S) = {3, 3} ! P = floor((I - 1) / S) * S + F - I ! ==> {1, 1} ! A = floor({1, 1} / {2, 2}) ! ==> {0, 0} ! B = {1, 1} - {0, 0} ! ==> {1, 1} ! \endcode ! The sample pointers are {0, 2, 4} in each dimension. ! SAMPLE_UPPER has {O0, O1, O2, pad} in output in each dimension. ! SAMPLE_LOWER has {pad, O0, O1, O2} in output in each dimension. ! ! Pooling Example 2: ! \code ! Given I = {6, 6}, B = {3, 3}, A = {3, 3}, S = {2, 2}, F = {3, 3}. What is O? ! \endcode !
PoolingType: ! ! \enum PoolingType ! ! \brief The type of pooling to perform in a pooling layer. !
PreviewFeature: ! ! \enum PreviewFeature ! ! \brief Define preview features ! ! Preview Features have been fully tested but are not yet as stable as other features in TensorRT. ! They are provided as opt-in features for at least one release. !
ProfilingVerbosity: ! ! \enum ProfilingVerbosity ! ! \brief List of verbosity levels of layer information exposed in NVTX annotations and in IEngineInspector. ! ! \see IBuilderConfig::setProfilingVerbosity(), ! IBuilderConfig::getProfilingVerbosity(), ! IEngineInspector !
ReduceOperation: ! ! \enum ReduceOperation ! ! \brief Enumerates the reduce operations that may be performed by a Reduce layer. ! ! The table shows the result of reducing across an empty volume of a given type. ! ! Operation | kFLOAT and kHALF | kINT32 | kINT8 ! ——— | —————– | —–– | —– ! kSUM | 0 | 0 | 0 ! kPROD | 1 | 1 | 1 ! kMAX | negative infinity | INT_MIN | -128 ! kMIN | positive infinity | INT_MAX | 127 ! kAVG | NaN | 0 | -128 ! kNONE | Undefined | Undefined | Undefined ! ! The current version of TensorRT usually performs reduction for kINT8 via kFLOAT or kHALF. ! The kINT8 values show the quantized representations of the floating-point values. ! \note kNONE is a reduce operation which does not modify the input tensor. ! This is applicable to Multi-Device mode only, ! as a reduce operation is not mandatory for certain collective operations. ! See \ref INetworkDefinition::addDistCollective for more details. !
ResizeCoordinateTransformation: ! ! \enum ResizeCoordinateTransformation ! ! \brief The resize coordinate transformation function. ! ! \see IResizeLayer::setCoordinateTransformation() !
ResizeRoundMode: ! ! \enum ResizeRoundMode ! ! \brief The rounding mode for nearest neighbor resize. ! ! \see IResizeLayer::setNearestRounding() !
ResizeSelector: ! ! \enum ResizeSelector ! ! \brief The coordinate selector when resize to single pixel output. ! ! \see IResizeLayer::setSelectorForSinglePixel() !
RuntimePlatform: ! ! \enum RuntimePlatform ! ! \brief Describes the intended runtime platform (operating system and CPU architecture) for the execution of the ! TensorRT engine. TensorRT provides support for cross-platform engine compatibility when the target runtime ! platform is different from the build platform. ! ! \note The cross-platform engine will not be able to run on the host platform it was built on. ! ! \note When building a cross-platform engine that also requires version forward compatibility, ! kEXCLUDE_LEAN_RUNTIME must be set to exclude the target platform lean runtime. ! ! \note The cross-platform engine might have performance differences compared to the natively built engine on the ! target platform. ! ! \see IBuilderConfig::setRuntimePlatform(), IBuilderConfig::getRuntimePlatform() !
SampleMode: ! ! \brief Controls how ISliceLayer and IGridSample handle out-of-bounds coordinates. ! ! \see ISliceLayer and IGridSample !
ScaleMode: ! ! \brief Controls how shift, scale and power are applied in a Scale layer. ! ! \see IScaleLayer !
ScatterMode: ! ! \enum ScatterMode ! ! \brief Control form of IScatterLayer ! ! \see IScatterLayer !
SeekPosition: ! ! \enum SeekPosition ! \brief Controls the seek mode of IStreamReaderV2. !
SerializationFlag: ! ! \enum SerializationFlag ! ! \brief List of valid flags that the engine can enable when serializing the bytes. ! ! \see ISerializationConfig::setFlags(), ISerializationConfig::getFlags() !
TensorFormat: ! ! \enum TensorFormat ! ! \brief Format of the input/output tensors. ! ! This enum is used by both plugins and network I/O tensors. ! ! \see IPluginV2::supportsFormat(), safe::ICudaEngine::getBindingFormat() ! ! Many of the formats are vector-major or vector-minor. These formats specify ! a vector dimension and scalars per vector. ! For example, suppose that the tensor has has dimensions [M,N,C,H,W], ! the vector dimension is C and there are V scalars per vector. ! ! * A vector-major format splits the vectorized dimension into two axes in the ! memory layout. The vectorized dimension is replaced by an axis of length ceil(C/V) ! and a new dimension of length V is appended. For the example tensor, the memory layout ! is equivalent to an array with dimensions [M][N][ceil(C/V)][H][W][V]. ! Tensor coordinate (m,n,c,h,w) maps to array location [m][n][c/V][h][w][c%V]. ! ! * A vector-minor format moves the vectorized dimension to become the last axis ! in the memory layout. For the example tensor, the memory layout is equivalent to an ! array with dimensions [M][N][H][W][ceil(C/V)*V]. Tensor coordinate (m,n,c,h,w) maps ! array location subscript [m][n][h][w][c]. ! ! In interfaces that refer to “components per element”, that’s the value of V above. ! ! For more information about data formats, see the topic “Data Format Description” located in the ! TensorRT Developer Guide. ! https://docs.nvidia.com/deeplearning/tensorrt/latest/inference-library/advanced.html#i-o-formats !
TensorIOMode: ! ! \enum TensorIOMode ! ! \brief Definition of tensor IO Mode. !
TensorLocation: ! ! \enum TensorLocation ! ! \brief The location for tensor data storage, device or host. !
TilingOptimizationLevel: ! ! \enum TilingOptimizationLevel ! ! \brief Define the optimization levels for Tiling ! ! TensorRT will try tiling optimization for on-chip caching if non-zero level is set. ! This level determines how much effort TensorRT would take to find a better solution for performance. !
TopKOperation: ! ! \enum TopKOperation ! ! \brief Enumerates the operations that may be performed by a TopK layer. !
TripLimit: ! ! \enum TripLimit ! ! \brief Enum that describes kinds of trip limits. !
UnaryOperation: ! ! \enum UnaryOperation ! ! \brief Enumerates the unary operations that may be performed by a Unary layer. ! ! Operations kNOT must have inputs of DataType::kBOOL. ! ! Operation kSIGN and kABS must have inputs of floating-point type, DataType::kINT8, DataType::kINT32 or ! DataType::kINT64. ! ! Operation kISINF must have inputs of floating-point type. ! ! All other operations must have inputs of floating-point type. ! ! \see IUnaryLayer !
WeightsRole: ! ! \enum WeightsRole ! ! \brief How a layer uses particular Weights. ! ! The power weights of an IScaleLayer are omitted. Refitting those is not supported. !

Crate trtx_sys

Crate trtx_sys

§Architecture

§Safety

Re-exports§

Modules§

Structs§

Enums§

Traits§

Functions§

Type Aliases§

Crate trtx_sys

Crate trtx_sys Copy item path

§Architecture

§Safety

Re-exports§

Modules§

Structs§

Enums§

Traits§

Functions§

Type Aliases§

Crate trtx_sys