trt-runner 0.1.0

Minimal TensorRT 10 inference runner: deserialize an engine, bind I/O tensors, enqueue V3. No bindings generator, no runtime dependencies.
Documentation
# trt-runner

Inference against a serialized TensorRT engine, in as few layers as possible.

`Engine` deserializes bytes, `Context` binds device buffers to the engine's I/O tensors and calls
`enqueueV3`. Nothing else: no preprocessing, no model-specific decoding, no bindings generator,
and no third-party runtime dependencies.

The crate exists because the Rust to TensorRT glue on Jetson keeps breaking in the same three
places, and because most existing wrappers are either unmaintained or reach TensorRT through ONNX
Runtime, so they cannot open a `.engine` file produced by `trtexec`.

## Why this instead of a full binding

* **TensorRT 10 only.** Only `libnvinfer` and `libcudart` are linked. `libnvparsers` is not, because
  NVIDIA removed the NvParser API in TensorRT 10, so bindings that still name it fail to link on
  JetPack 6. Building against TensorRT 8 headers stops at compile time with a message instead of at
  run time with a missing symbol.
* **No bindgen, no cxx.** A ~340 line C shim and a hand written `extern "C"` block. The build needs
  `cc` and nothing else.
* **Context currentness is handled the way CUDA actually wants.** `cuda::set_device` calls
  `cudaSetDevice`, which activates the primary context for the thread. The driver-API route
  (`cuDevicePrimaryCtxRetain`) only takes a reference and leaves the following TensorRT calls with
  `CUDA_ERROR_INVALID_CONTEXT`.
* **Readback has a safe default.** `DeviceBuffer::copy_to_host` waits for the stream.
  `copy_to_host_async` exists when overlapping transfers matter. Reading the destination before
  synchronising yields zeroes, which looks exactly like a preprocessing bug, so the two variants are
  deliberately named differently.
* **Buffer sizes come from the engine.** `bind_buffer` rejects a buffer whose byte count differs from
  what the tensor declares, rather than corrupting whichever allocation happens to be adjacent.

## Install

```toml
[dependencies]
trt-runner = "0.1"
```

Requirements: TensorRT 10 headers, `libnvinfer`, the CUDA runtime, a C++ compiler.

On Jetson (JetPack 6) the headers live in a triplet directory, which the build script already prefers:

```bash
export CUDA_HOME=/usr/local/cuda
export CUDA_INCLUDE_PATH=/usr/local/cuda/include
export TENSORRT_INCLUDE_PATH=/usr/include/aarch64-linux-gnu
export TENSORRT_LIB_PATH=/usr/lib/aarch64-linux-gnu
```

All four are optional. `TENSORRT_INCLUDE_PATH`, `TENSORRT_LIB_PATH`, `CUDA_INCLUDE_PATH` and
`CUDA_LIB_PATH` (plus `_INC_DIR` / `_LIB_DIR` aliases) override the defaults wherever the toolchain
actually lives.

## Usage

```rust
use trt_runner::{cuda, DeviceBuffer, Engine, IoMode, PinnedBuffer, Stream};

fn run() -> trt_runner::Result<()> {
    cuda::set_device(0)?;
    let engine = Engine::from_file("model_fp16.engine")?;
    let stream = Stream::new()?;
    let mut context = engine.context()?;

    // One device buffer per I/O tensor, sized from the engine's own metadata.
    let mut buffers = Vec::new();
    for info in engine.io() {
        let mut buffer = DeviceBuffer::zeroed(info.size_bytes().unwrap(), &stream)?;
        if info.mode == IoMode::Input {
            let host = PinnedBuffer::new(buffer.size_bytes())?; // fill me
            buffer.copy_from_host(host.bytes(), &stream)?;
        }
        // SAFETY: `buffers` outlives the last synchronised run below.
        unsafe { context.bind_buffer(&info.name, &buffer)? };
        buffers.push((info.name.clone(), info.mode, buffer));
    }

    context.run(&stream)?; // enqueue plus synchronize

    for (name, mode, buffer) in &buffers {
        if *mode == IoMode::Output {
            let mut out = vec![0u8; buffer.size_bytes()];
            buffer.copy_to_host(&mut out, &stream)?; // waits for the stream
        }
    }
    Ok(())
}
```

`engine.context()` borrows the engine and carries a lifetime. When the runner has to live inside a
struct, `engine.into_context()` returns an `OwnedContext` that owns the engine and runtime, is `Send`,
and has no lifetime parameter. One execution context per thread.

Dynamic shapes are supported but explicit: `context.set_shape("images", &[1, 3, 640, 640])`, then size
the buffers from the resolved shape.

The bundled example loads any engine, feeds it a raw little-endian float32 blob, times it, and prints
per-output statistics, which makes it easy to check two runtimes against the same input:

```bash
cargo run --release --example run_engine -- model.engine --blob input.bin --repeat 200
```

[`examples/quickstart.rs`](examples/quickstart.rs) is the snippet above as a runnable file, and
[`examples/run_engine.rs`](examples/run_engine.rs) adds timing and output statistics.

## Measured

Jetson Orin Nano 8GB, JetPack 6, CUDA 12.6, TensorRT 10.3.0, aarch64. A YOLO26s-seg model exported to
ONNX and built with `trtexec --fp16`, static 1x3x640x640 input.

| workload | time | fps |
| --- | --- | --- |
| `enqueueV3` plus sync, 300 runs | 8.97 ms mean | 111.5 |
| `examples/run_engine`, including 3.2 MB of output readback | 9.25 ms mean, p50 8.29, p95 9.57 | 108.2 |

Outputs match a `pycuda` reference run on the same engine (output mean `0.183709` on both sides, head
tensor score `0.5737` against `0.57373`), so the plumbing is not quietly changing values.

## Non-goals

No preprocessing or letterboxing, no YOLO decode or NMS, no mask rasterization, no ONNX parsing, no
engine builder, no plugins, no INT8 calibration, no multi-stream scheduling. `Context` and
`OwnedContext` are `Send` but not `Sync`.

Those exclusions are deliberate. Keep the decoder in your own crate, where changing an output layout
does not mean waiting on a dependency release.

## License

Dual licensed under either of [Apache License 2.0](LICENSE-APACHE) or [MIT license](LICENSE-MIT), at
your option.