trt-runner 0.1.0

Minimal TensorRT 10 inference runner: deserialize an engine, bind I/O tensors, enqueue V3. No bindings generator, no runtime dependencies.
Documentation
  • Coverage
  • 100%
    78 out of 78 items documented1 out of 25 items with examples
  • Size
  • Source code size: 86.1 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 1.1 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 4s Average build duration of successful builds.
  • all releases: 4s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Homepage
  • jlu-lujing/trt-runner
    0 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • jlu-lujing

trt-runner

Inference against a serialized TensorRT engine, in as few layers as possible.

Engine deserializes bytes, Context binds device buffers to the engine's I/O tensors and calls enqueueV3. Nothing else: no preprocessing, no model-specific decoding, no bindings generator, and no third-party runtime dependencies.

The crate exists because the Rust to TensorRT glue on Jetson keeps breaking in the same three places, and because most existing wrappers are either unmaintained or reach TensorRT through ONNX Runtime, so they cannot open a .engine file produced by trtexec.

Why this instead of a full binding

  • TensorRT 10 only. Only libnvinfer and libcudart are linked. libnvparsers is not, because NVIDIA removed the NvParser API in TensorRT 10, so bindings that still name it fail to link on JetPack 6. Building against TensorRT 8 headers stops at compile time with a message instead of at run time with a missing symbol.
  • No bindgen, no cxx. A ~340 line C shim and a hand written extern "C" block. The build needs cc and nothing else.
  • Context currentness is handled the way CUDA actually wants. cuda::set_device calls cudaSetDevice, which activates the primary context for the thread. The driver-API route (cuDevicePrimaryCtxRetain) only takes a reference and leaves the following TensorRT calls with CUDA_ERROR_INVALID_CONTEXT.
  • Readback has a safe default. DeviceBuffer::copy_to_host waits for the stream. copy_to_host_async exists when overlapping transfers matter. Reading the destination before synchronising yields zeroes, which looks exactly like a preprocessing bug, so the two variants are deliberately named differently.
  • Buffer sizes come from the engine. bind_buffer rejects a buffer whose byte count differs from what the tensor declares, rather than corrupting whichever allocation happens to be adjacent.

Install

[dependencies]
trt-runner = "0.1"

Requirements: TensorRT 10 headers, libnvinfer, the CUDA runtime, a C++ compiler.

On Jetson (JetPack 6) the headers live in a triplet directory, which the build script already prefers:

export CUDA_HOME=/usr/local/cuda
export CUDA_INCLUDE_PATH=/usr/local/cuda/include
export TENSORRT_INCLUDE_PATH=/usr/include/aarch64-linux-gnu
export TENSORRT_LIB_PATH=/usr/lib/aarch64-linux-gnu

All four are optional. TENSORRT_INCLUDE_PATH, TENSORRT_LIB_PATH, CUDA_INCLUDE_PATH and CUDA_LIB_PATH (plus _INC_DIR / _LIB_DIR aliases) override the defaults wherever the toolchain actually lives.

Usage

use trt_runner::{cuda, DeviceBuffer, Engine, IoMode, PinnedBuffer, Stream};

fn run() -> trt_runner::Result<()> {
    cuda::set_device(0)?;
    let engine = Engine::from_file("model_fp16.engine")?;
    let stream = Stream::new()?;
    let mut context = engine.context()?;

    // One device buffer per I/O tensor, sized from the engine's own metadata.
    let mut buffers = Vec::new();
    for info in engine.io() {
        let mut buffer = DeviceBuffer::zeroed(info.size_bytes().unwrap(), &stream)?;
        if info.mode == IoMode::Input {
            let host = PinnedBuffer::new(buffer.size_bytes())?; // fill me
            buffer.copy_from_host(host.bytes(), &stream)?;
        }
        // SAFETY: `buffers` outlives the last synchronised run below.
        unsafe { context.bind_buffer(&info.name, &buffer)? };
        buffers.push((info.name.clone(), info.mode, buffer));
    }

    context.run(&stream)?; // enqueue plus synchronize

    for (name, mode, buffer) in &buffers {
        if *mode == IoMode::Output {
            let mut out = vec![0u8; buffer.size_bytes()];
            buffer.copy_to_host(&mut out, &stream)?; // waits for the stream
        }
    }
    Ok(())
}

engine.context() borrows the engine and carries a lifetime. When the runner has to live inside a struct, engine.into_context() returns an OwnedContext that owns the engine and runtime, is Send, and has no lifetime parameter. One execution context per thread.

Dynamic shapes are supported but explicit: context.set_shape("images", &[1, 3, 640, 640]), then size the buffers from the resolved shape.

The bundled example loads any engine, feeds it a raw little-endian float32 blob, times it, and prints per-output statistics, which makes it easy to check two runtimes against the same input:

cargo run --release --example run_engine -- model.engine --blob input.bin --repeat 200

examples/quickstart.rs is the snippet above as a runnable file, and examples/run_engine.rs adds timing and output statistics.

Measured

Jetson Orin Nano 8GB, JetPack 6, CUDA 12.6, TensorRT 10.3.0, aarch64. A YOLO26s-seg model exported to ONNX and built with trtexec --fp16, static 1x3x640x640 input.

workload time fps
enqueueV3 plus sync, 300 runs 8.97 ms mean 111.5
examples/run_engine, including 3.2 MB of output readback 9.25 ms mean, p50 8.29, p95 9.57 108.2

Outputs match a pycuda reference run on the same engine (output mean 0.183709 on both sides, head tensor score 0.5737 against 0.57373), so the plumbing is not quietly changing values.

Non-goals

No preprocessing or letterboxing, no YOLO decode or NMS, no mask rasterization, no ONNX parsing, no engine builder, no plugins, no INT8 calibration, no multi-stream scheduling. Context and OwnedContext are Send but not Sync.

Those exclusions are deliberate. Keep the decoder in your own crate, where changing an output layout does not mean waiting on a dependency release.

License

Dual licensed under either of Apache License 2.0 or MIT license, at your option.