Skip to main content

Crate trt_runner

Crate trt_runner 

Source
Expand description

Inference against a serialized TensorRT engine, in as few layers as possible.

Engine deserializes bytes, Context binds device buffers to the engine’s I/O tensors and runs enqueueV3. Nothing else: no preprocessing, no model-specific decoding, no bindings generator, and no third-party runtime dependencies.

§Targets TensorRT 10

Only the inference runtime (libnvinfer) and the CUDA runtime (libcudart) are linked. In particular libnvparsers is not, because NVIDIA removed the NvParser API in TensorRT 10; bindings that still link it cannot be built on JetPack 6. Building against TensorRT 8 headers fails at compile time with a message rather than at run time with a missing symbol.

Set TENSORRT_INCLUDE_PATH, TENSORRT_LIB_PATH, CUDA_INCLUDE_PATH and CUDA_LIB_PATH when the toolchain is in an unusual place. On Jetson the headers live in /usr/include/aarch64-linux-gnu.

§Two rules this crate is designed around

A context has to be current on the calling thread. Use cuda::set_device rather than driver-API context retention: cudaSetDevice activates the primary context for the thread, while cuDevicePrimaryCtxRetain only takes a reference to it and leaves later calls failing with CUDA_ERROR_INVALID_CONTEXT.

Asynchronous copies are not finished when they return. DeviceBuffer::copy_to_host waits for the stream and is the safe default; DeviceBuffer::copy_to_host_async exists for pipelines that want to overlap several transfers, and reading the destination before synchronising the stream that carried them yields zeroes. That failure mode looks exactly like a preprocessing bug, so it is worth the extra method name.

§Example

use trt_runner::{cuda, Context, DeviceBuffer, Engine, IoMode, Stream};

cuda::set_device(0)?;
let engine = Engine::from_file("model_fp16.engine")?;
let stream = Stream::new()?;
let mut context = engine.context()?;

// one device buffer per I/O tensor, sized from the engine's own metadata
let mut buffers = Vec::new();
for info in engine.io() {
    let bytes = info.size_bytes().expect("a static shape");
    let mut buffer = DeviceBuffer::new(bytes)?;
    // SAFETY: the buffers live until after the last synchronised enqueue.
    unsafe { context.bind_buffer(&info.name, &buffer)? };
    buffers.push((info.name.clone(), info.mode, buffer));
}

context.run(&stream)?; // enqueue plus synchronize

for (name, mode, buffer) in &buffers {
    if *mode == IoMode::Output {
        let mut head = vec![0u8; 16.min(buffer.size_bytes())];
        buffer.copy_to_host(&mut head, &stream)?;
        println!("{name}: {} bytes", buffer.size_bytes());
    }
}

Re-exports§

pub use cuda::DeviceBuffer;
pub use cuda::PinnedBuffer;
pub use cuda::Stream;

Modules§

cuda
CUDA plumbing: the device context, a stream, and the two buffer kinds a runner needs.

Structs§

Context
A bound execution context. Borrows the engine so it can never outlive it.
Engine
A deserialized engine plus the runtime that produced it.
OwnedContext
An execution context that owns the engine and runtime behind it, so it can be stored in a struct without a lifetime parameter.
TensorInfo
One I/O tensor of an engine: name, direction, element type and declared shape.

Enums§

DataType
Element type of a tensor, as reported by the engine.
Error
Everything this crate can fail at, carrying the TensorRT or CUDA diagnostic when there is one.
IoMode
Whether the engine reads or writes a tensor.

Constants§

MAX_DIMS
Maximum dimensions TensorRT allows for one tensor.

Functions§

header_version
The TensorRT header version this crate was compiled against.

Type Aliases§

Result
Result alias with this crate’s error type.