1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
//! Inference against a serialized [TensorRT] engine, in as few layers as possible.
//!
//! [`Engine`] deserializes bytes, [`Context`] binds device buffers to the engine's I/O tensors and
//! runs [`enqueueV3`][Context::enqueue]. Nothing else: no preprocessing, no model-specific decoding,
//! no bindings generator, and no third-party runtime dependencies.
//!
//! # Targets TensorRT 10
//!
//! Only the inference runtime (`libnvinfer`) and the CUDA runtime (`libcudart`) are linked. In
//! particular `libnvparsers` is not, because NVIDIA removed the NvParser API in TensorRT 10; bindings
//! that still link it cannot be built on JetPack 6. Building against TensorRT 8 headers fails at
//! compile time with a message rather than at run time with a missing symbol.
//!
//! Set `TENSORRT_INCLUDE_PATH`, `TENSORRT_LIB_PATH`, `CUDA_INCLUDE_PATH` and `CUDA_LIB_PATH` when the
//! toolchain is in an unusual place. On Jetson the headers live in `/usr/include/aarch64-linux-gnu`.
//!
//! # Two rules this crate is designed around
//!
//! **A context has to be current on the calling thread.** Use [`cuda::set_device`] rather than
//! driver-API context retention: `cudaSetDevice` activates the primary context for the thread, while
//! `cuDevicePrimaryCtxRetain` only takes a reference to it and leaves later calls failing with
//! `CUDA_ERROR_INVALID_CONTEXT`.
//!
//! **Asynchronous copies are not finished when they return.** [`DeviceBuffer::copy_to_host`] waits
//! for the stream and is the safe default; [`DeviceBuffer::copy_to_host_async`] exists for pipelines
//! that want to overlap several transfers, and reading the destination before synchronising the
//! stream that carried them yields zeroes. That failure mode looks exactly like a preprocessing bug,
//! so it is worth the extra method name.
//!
//! # Example
//!
//! ```no_run
//! use trt_runner::{cuda, Context, DeviceBuffer, Engine, IoMode, Stream};
//!
//! # fn main() -> Result<(), Box<dyn std::error::Error>> {
//! cuda::set_device(0)?;
//! let engine = Engine::from_file("model_fp16.engine")?;
//! let stream = Stream::new()?;
//! let mut context = engine.context()?;
//!
//! // one device buffer per I/O tensor, sized from the engine's own metadata
//! let mut buffers = Vec::new();
//! for info in engine.io() {
//! let bytes = info.size_bytes().expect("a static shape");
//! let mut buffer = DeviceBuffer::new(bytes)?;
//! // SAFETY: the buffers live until after the last synchronised enqueue.
//! unsafe { context.bind_buffer(&info.name, &buffer)? };
//! buffers.push((info.name.clone(), info.mode, buffer));
//! }
//!
//! context.run(&stream)?; // enqueue plus synchronize
//!
//! for (name, mode, buffer) in &buffers {
//! if *mode == IoMode::Output {
//! let mut head = vec![0u8; 16.min(buffer.size_bytes())];
//! buffer.copy_to_host(&mut head, &stream)?;
//! println!("{name}: {} bytes", buffer.size_bytes());
//! }
//! }
//! # Ok(())
//! # }
//! ```
//!
//! [TensorRT]: https://docs.nvidia.com/deeplearning/tensorrt/
pub use ;
pub use ;
pub use Error;
/// Result alias with this crate's error type.
pub type Result<T> = Result;