Expand description
Inference against a serialized TensorRT engine, in as few layers as possible.
Engine deserializes bytes, Context binds device buffers to the engine’s I/O tensors and
runs enqueueV3. Nothing else: no preprocessing, no model-specific decoding,
no bindings generator, and no third-party runtime dependencies.
§Targets TensorRT 10
Only the inference runtime (libnvinfer) and the CUDA runtime (libcudart) are linked. In
particular libnvparsers is not, because NVIDIA removed the NvParser API in TensorRT 10; bindings
that still link it cannot be built on JetPack 6. Building against TensorRT 8 headers fails at
compile time with a message rather than at run time with a missing symbol.
Set TENSORRT_INCLUDE_PATH, TENSORRT_LIB_PATH, CUDA_INCLUDE_PATH and CUDA_LIB_PATH when the
toolchain is in an unusual place. On Jetson the headers live in /usr/include/aarch64-linux-gnu.
§Two rules this crate is designed around
A context has to be current on the calling thread. Use cuda::set_device rather than
driver-API context retention: cudaSetDevice activates the primary context for the thread, while
cuDevicePrimaryCtxRetain only takes a reference to it and leaves later calls failing with
CUDA_ERROR_INVALID_CONTEXT.
Asynchronous copies are not finished when they return. DeviceBuffer::copy_to_host waits
for the stream and is the safe default; DeviceBuffer::copy_to_host_async exists for pipelines
that want to overlap several transfers, and reading the destination before synchronising the
stream that carried them yields zeroes. That failure mode looks exactly like a preprocessing bug,
so it is worth the extra method name.
§Example
use trt_runner::{cuda, Context, DeviceBuffer, Engine, IoMode, Stream};
cuda::set_device(0)?;
let engine = Engine::from_file("model_fp16.engine")?;
let stream = Stream::new()?;
let mut context = engine.context()?;
// one device buffer per I/O tensor, sized from the engine's own metadata
let mut buffers = Vec::new();
for info in engine.io() {
let bytes = info.size_bytes().expect("a static shape");
let mut buffer = DeviceBuffer::new(bytes)?;
// SAFETY: the buffers live until after the last synchronised enqueue.
unsafe { context.bind_buffer(&info.name, &buffer)? };
buffers.push((info.name.clone(), info.mode, buffer));
}
context.run(&stream)?; // enqueue plus synchronize
for (name, mode, buffer) in &buffers {
if *mode == IoMode::Output {
let mut head = vec![0u8; 16.min(buffer.size_bytes())];
buffer.copy_to_host(&mut head, &stream)?;
println!("{name}: {} bytes", buffer.size_bytes());
}
}Re-exports§
pub use cuda::DeviceBuffer;pub use cuda::PinnedBuffer;pub use cuda::Stream;
Modules§
- cuda
- CUDA plumbing: the device context, a stream, and the two buffer kinds a runner needs.
Structs§
- Context
- A bound execution context. Borrows the engine so it can never outlive it.
- Engine
- A deserialized engine plus the runtime that produced it.
- Owned
Context - An execution context that owns the engine and runtime behind it, so it can be stored in a struct without a lifetime parameter.
- Tensor
Info - One I/O tensor of an engine: name, direction, element type and declared shape.
Enums§
- Data
Type - Element type of a tensor, as reported by the engine.
- Error
- Everything this crate can fail at, carrying the TensorRT or CUDA diagnostic when there is one.
- IoMode
- Whether the engine reads or writes a tensor.
Constants§
- MAX_
DIMS - Maximum dimensions TensorRT allows for one tensor.
Functions§
- header_
version - The TensorRT header version this crate was compiled against.
Type Aliases§
- Result
- Result alias with this crate’s error type.