trt-runner
Inference against a serialized TensorRT engine, in as few layers as possible.
Engine deserializes bytes, Context binds device buffers to the engine's I/O tensors and calls
enqueueV3. Nothing else: no preprocessing, no model-specific decoding, no bindings generator,
and no third-party runtime dependencies.
The crate exists because the Rust to TensorRT glue on Jetson keeps breaking in the same three
places, and because most existing wrappers are either unmaintained or reach TensorRT through ONNX
Runtime, so they cannot open a .engine file produced by trtexec.
Why this instead of a full binding
- TensorRT 10 only. Only
libnvinferandlibcudartare linked.libnvparsersis not, because NVIDIA removed the NvParser API in TensorRT 10, so bindings that still name it fail to link on JetPack 6. Building against TensorRT 8 headers stops at compile time with a message instead of at run time with a missing symbol. - No bindgen, no cxx. A ~340 line C shim and a hand written
extern "C"block. The build needsccand nothing else. - Context currentness is handled the way CUDA actually wants.
cuda::set_devicecallscudaSetDevice, which activates the primary context for the thread. The driver-API route (cuDevicePrimaryCtxRetain) only takes a reference and leaves the following TensorRT calls withCUDA_ERROR_INVALID_CONTEXT. - Readback has a safe default.
DeviceBuffer::copy_to_hostwaits for the stream.copy_to_host_asyncexists when overlapping transfers matter. Reading the destination before synchronising yields zeroes, which looks exactly like a preprocessing bug, so the two variants are deliberately named differently. - Buffer sizes come from the engine.
bind_bufferrejects a buffer whose byte count differs from what the tensor declares, rather than corrupting whichever allocation happens to be adjacent.
Install
[]
= "0.1"
Requirements: TensorRT 10 headers, libnvinfer, the CUDA runtime, a C++ compiler.
On Jetson (JetPack 6) the headers live in a triplet directory, which the build script already prefers:
All four are optional. TENSORRT_INCLUDE_PATH, TENSORRT_LIB_PATH, CUDA_INCLUDE_PATH and
CUDA_LIB_PATH (plus _INC_DIR / _LIB_DIR aliases) override the defaults wherever the toolchain
actually lives.
Usage
use ;
engine.context() borrows the engine and carries a lifetime. When the runner has to live inside a
struct, engine.into_context() returns an OwnedContext that owns the engine and runtime, is Send,
and has no lifetime parameter. One execution context per thread.
Dynamic shapes are supported but explicit: context.set_shape("images", &[1, 3, 640, 640]), then size
the buffers from the resolved shape.
The bundled example loads any engine, feeds it a raw little-endian float32 blob, times it, and prints per-output statistics, which makes it easy to check two runtimes against the same input:
examples/quickstart.rs is the snippet above as a runnable file, and
examples/run_engine.rs adds timing and output statistics.
Measured
Jetson Orin Nano 8GB, JetPack 6, CUDA 12.6, TensorRT 10.3.0, aarch64. A YOLO26s-seg model exported to
ONNX and built with trtexec --fp16, static 1x3x640x640 input.
| workload | time | fps |
|---|---|---|
enqueueV3 plus sync, 300 runs |
8.97 ms mean | 111.5 |
examples/run_engine, including 3.2 MB of output readback |
9.25 ms mean, p50 8.29, p95 9.57 | 108.2 |
Outputs match a pycuda reference run on the same engine (output mean 0.183709 on both sides, head
tensor score 0.5737 against 0.57373), so the plumbing is not quietly changing values.
Non-goals
No preprocessing or letterboxing, no YOLO decode or NMS, no mask rasterization, no ONNX parsing, no
engine builder, no plugins, no INT8 calibration, no multi-stream scheduling. Context and
OwnedContext are Send but not Sync.
Those exclusions are deliberate. Keep the decoder in your own crate, where changing an output layout does not mean waiting on a dependency release.
License
Dual licensed under either of Apache License 2.0 or MIT license, at your option.