trt-runner 0.1.0

Minimal TensorRT 10 inference runner: deserialize an engine, bind I/O tensors, enqueue V3. No bindings generator, no runtime dependencies.
Documentation
//! Inference against a serialized [TensorRT] engine, in as few layers as possible.
//!
//! [`Engine`] deserializes bytes, [`Context`] binds device buffers to the engine's I/O tensors and
//! runs [`enqueueV3`][Context::enqueue]. Nothing else: no preprocessing, no model-specific decoding,
//! no bindings generator, and no third-party runtime dependencies.
//!
//! # Targets TensorRT 10
//!
//! Only the inference runtime (`libnvinfer`) and the CUDA runtime (`libcudart`) are linked. In
//! particular `libnvparsers` is not, because NVIDIA removed the NvParser API in TensorRT 10; bindings
//! that still link it cannot be built on JetPack 6. Building against TensorRT 8 headers fails at
//! compile time with a message rather than at run time with a missing symbol.
//!
//! Set `TENSORRT_INCLUDE_PATH`, `TENSORRT_LIB_PATH`, `CUDA_INCLUDE_PATH` and `CUDA_LIB_PATH` when the
//! toolchain is in an unusual place. On Jetson the headers live in `/usr/include/aarch64-linux-gnu`.
//!
//! # Two rules this crate is designed around
//!
//! **A context has to be current on the calling thread.** Use [`cuda::set_device`] rather than
//! driver-API context retention: `cudaSetDevice` activates the primary context for the thread, while
//! `cuDevicePrimaryCtxRetain` only takes a reference to it and leaves later calls failing with
//! `CUDA_ERROR_INVALID_CONTEXT`.
//!
//! **Asynchronous copies are not finished when they return.** [`DeviceBuffer::copy_to_host`] waits
//! for the stream and is the safe default; [`DeviceBuffer::copy_to_host_async`] exists for pipelines
//! that want to overlap several transfers, and reading the destination before synchronising the
//! stream that carried them yields zeroes. That failure mode looks exactly like a preprocessing bug,
//! so it is worth the extra method name.
//!
//! # Example
//!
//! ```no_run
//! use trt_runner::{cuda, Context, DeviceBuffer, Engine, IoMode, Stream};
//!
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
//! cuda::set_device(0)?;
//! let engine = Engine::from_file("model_fp16.engine")?;
//! let stream = Stream::new()?;
//! let mut context = engine.context()?;
//!
//! // one device buffer per I/O tensor, sized from the engine's own metadata
//! let mut buffers = Vec::new();
//! for info in engine.io() {
//!     let bytes = info.size_bytes().expect("a static shape");
//!     let mut buffer = DeviceBuffer::new(bytes)?;
//!     // SAFETY: the buffers live until after the last synchronised enqueue.
//!     unsafe { context.bind_buffer(&info.name, &buffer)? };
//!     buffers.push((info.name.clone(), info.mode, buffer));
//! }
//!
//! context.run(&stream)?; // enqueue plus synchronize
//!
//! for (name, mode, buffer) in &buffers {
//!     if *mode == IoMode::Output {
//!         let mut head = vec![0u8; 16.min(buffer.size_bytes())];
//!         buffer.copy_to_host(&mut head, &stream)?;
//!         println!("{name}: {} bytes", buffer.size_bytes());
//!     }
//! }
//! # Ok(())
//! # }
//! ```
//!
//! [TensorRT]: https://docs.nvidia.com/deeplearning/tensorrt/

#![deny(missing_docs)]

pub mod cuda;
mod engine;
mod error;
mod ffi;
mod util;

pub use cuda::{DeviceBuffer, PinnedBuffer, Stream};
pub use engine::{
    header_version, Context, DataType, Engine, IoMode, OwnedContext, TensorInfo, MAX_DIMS,
};
pub use error::Error;

/// Result alias with this crate's error type.
pub type Result<T> = std::result::Result<T, Error>;