Crate native_onnx 

native-onnx

Rust ONNX inference using your system’s ONNX Runtime, CUDA, and TensorRT. Load a model, supply typed tensors, and choose a required backend or ordered fallback. No Python runtime or automatic native-library downloads.

Why use it?

• Less repeated work: reuse input/output buffers; GPU paths reuse pinned host and device storage. Eligible fixed-shape models can use CUDA graph replay.
• Faster startup after compilation: compile a TensorRT model once, then load its embedded engine on later launches.
• Explicit execution: require CUDA/TensorRT or allow fallback to CPU, with recorded failure reasons and checked tensor names, types, shapes, and lengths.

Compute speed comes from ONNX Runtime and its native providers. This crate reduces buffer setup and allocation work; it does not promise a universal speedup over ort, C++, or Python. Benchmark your model and hardware.

Install the native libraries first

Requires Rust 1.88+ and a matching native runtime for your process architecture. You must install these system libraries yourself:

Execution	Required installation
CPU	ONNX Runtime 1.27+ shared library (C API 27); no NVIDIA libraries needed
CUDA	ONNX Runtime 1.27+ with CUDA EP; standard GPU builds require CUDA 13.x (13.0+), cuDNN 9.x for CUDA 13, and an NVIDIA driver supporting CUDA 13 (R580+)
TensorRT	The GPU stack above plus the TensorRT version required by your ORT build. The validated baseline is TensorRT 10.15.1 (ABI 10); a newer incompatible major version will not work
CoreML	macOS 12+ and ONNX Runtime 1.27+ with CoreML EP

GPU dependency versions must match the installed ORT build, not merely exceed a number. Custom CUDA 12 builds need their matching CUDA/cuDNN/TensorRT libraries. Validated on Linux with ORT 1.29.1, CUDA 13, cuDNN 9.20, and TensorRT 10.15.1; macOS and Windows still need native validation on those platforms. For that Linux stack, the cuDNN driver minimum is 580.65.06. See ORT installation, CUDA requirements, and TensorRT compatibility.

On Linux, set the paths before launching your application:

export ORT_DYLIB_PATH=/opt/onnxruntime/lib/libonnxruntime.so
# GPU only: replace these with your actual native-library directories.
export LD_LIBRARY_PATH=/opt/onnxruntime/lib:/usr/local/cuda/lib64:/opt/cudnn/lib:/opt/tensorrt/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}

On Windows, point ORT_DYLIB_PATH to onnxruntime.dll and add the dependency folders to PATH. On macOS, use libonnxruntime.dylib. You can also set OnnxOptions::runtime_path; the first ORT initialization selects the library for the entire process. Missing/incompatible libraries return installation guidance; print errors with eprintln!("{error:#}") to include the native loader’s cause.

Basic example

[dependencies]
native-onnx = "0.1.0"

For a model with an FP32 input named x, shape [1, 16]:

use native_onnx::{OnnxOptions, OnnxRuntime, Result, TensorView};

fn main() -> Result<()> {
    let mut model = OnnxRuntime::load("model.onnx", OnnxOptions::cpu())?;
    let values = [1.0_f32; 16];
    let input = TensorView::f32("x", &[1, 16], &values);

    for output in model.inference(&[input])? {
        println!("{}: {:?}", output.name, output.shape);
    }
    Ok(())
}

Replace the path, name, shape, and values for your model. model.info() exposes its specifications. Input storage is contiguous and row-major; preprocessing and postprocessing belong in your application.

Select a GPU and compile once

use native_onnx::{
    Backend, BackendSelection, CompileOptions, CompileTarget, OnnxOptions,
    OnnxRuntime, TensorRtOptions, TensorView,
};

let config = TensorRtOptions {
    fp16: true,
    ..TensorRtOptions::default()
};
let compile = CompileOptions::new(CompileTarget::TensorRt(config.clone()));
let values = [1.0_f32; 16];
let input = TensorView::f32("x", &[1, 16], &values);

// Run once. The destination must not already exist.
OnnxRuntime::compile("model.onnx", "model.ctx.onnx", compile, &[input])?;

// Later launches start here; the original model is not needed.
let options = OnnxOptions {
    backend: BackendSelection::Require(Backend::TensorRt),
    tensorrt: Some(config),
    ..OnnxOptions::default()
};
let mut model = OnnxRuntime::load("model.ctx.onnx", options)?;
let outputs = model.inference(&[input])?;

Compilation requires an explicit target and its settings: CompileTarget::TensorRt(config), CompileTarget::Cuda(config), or CompileTarget::Cpu. CompileOptions::new(target) also exposes threading, runtime path, and dimension overrides. It has no default target or fallback; the artifact is validated with your inputs before saving.

For loading/inference, use Require(Backend::Cuda) for CUDA. Use Auto(vec![Backend::TensorRt, Backend::Cuda, Backend::Cpu]) to permit fallback. backend() reports the primary provider; in automatic mode some nodes can run on CPU. fallback_events() reports whole-session failures. Require rejects CPU node fallback.

TensorRT export currently requires one partition. CPU/CUDA compilation produces an optimized ONNX graph. Compiled contexts require their native provider; they cannot fall back to a CPU engine. Recompile after changing the model, GPU, or incompatible runtime/provider settings. Raw TensorRT .engine files are not accepted.

Reuse buffers and configure execution

API / option	Purpose
inference(inputs)	Borrow inputs and return owned CPU outputs
inference_into(inputs, outputs)	Fill caller-owned outputs
input_mut(name), run(), output(name)	Reuse fixed-shape buffers; inputs start at zero
CudaOptions, TensorRtOptions	Device, precision, CUDA graphs, and TensorRT profiles
dimension_overrides	Resolve named dynamic dimensions
parallel_execution, inter_threads	Enable/configure inter-operator CPU parallelism

CUDA graphs require eligible fixed-shape I/O and sequential execution. CUDA’s capture option defaults off; TensorRT’s defaults on. TensorRT defaults to FP32 with TF32 allowed; unset NVIDIA_TF32_OVERRIDE, or set it to 0 with tf32: false. Dynamic or unknown-rank I/O uses ordinary inference. Dense numeric and boolean tensors are supported; strings, sequences, and sparse I/O are not.

Calls are synchronous. OnnxRuntime is neither Send nor Sync: construct one inside each inference worker. Native provider crashes cannot be recovered as Rust errors. Additional providers use Backend::Custom; enable the openvino feature for OpenVINO configuration and install its matching native provider separately.

The pinned ort rc.13 dependency has an exit-time CUDA teardown bug. This crate retains one global environment until process exit as a workaround; model sessions and buffers are still released when dropped.

Development

With ORT_DYLIB_PATH configured:

cargo fmt --all -- --check
cargo clippy --all-targets --all-features -- -D warnings
cargo test --all-features
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features
cargo publish --dry-run
# On a GPU host with the matching native libraries:
cargo test --tests -- --ignored

The crate archive includes only engine source, build script, README, license, and Cargo metadata. Tests stay in Git; local examples, experiments, and model assets are ignored.

Licensed under Apache-2.0. See LICENSE-APACHE for the full text.

Modules

ep

Configure additional installed ONNX Runtime providers through Backend::Custom. ExecutionProviders provide hardware acceleration to Sessions.

Structs

Compilation

Artifact compiled and validated for the explicitly requested target.

CompileOptions

Compilation configuration with a mandatory target and no automatic fallback. Construct with Self::new, then adjust shared settings as needed.

CudaOptions

Options used only by the built-in CUDA execution provider.

Error

The Error type, a wrapper around a dynamic error type.

FallbackEvent

A provider failure recorded before trying the next configured backend.

ModelInfo

ONNX input and output names, data types, and shapes.

OnnxOptions

Runtime library, backend selection, threading, and model-shape configuration.

OnnxRuntime

An ONNX model session with ordered provider fallback and reusable inference buffers. Methods are synchronous. Use one instance per concurrent inference worker.

Tensor

Owned, named tensor with contiguous, row-major CPU storage.

TensorRtOptions

Build and execution options used only by TensorRT.

TensorSpec

Dense, contiguous, row-major tensor metadata.

TensorView

Named tensor borrowing contiguous, row-major CPU storage. Creating a view does not validate it; inference validates it against the model.

TensorViewMut

Named tensor exclusively borrowing contiguous, row-major CPU storage.

bf16

A 16-bit floating point type implementing the bfloat16 format.

f16

A 16-bit floating point type implementing the IEEE 754-2008 standard binary16 a.k.a “half” format.

Enums

Backend

An installed ONNX Runtime execution provider.

BackendSelection

Select one required provider, or try candidates in order with fallback.

CompileTarget

One compilation target together with its provider-specific configuration. Compilation never selects a target automatically or falls back to another provider.

CompiledFormat

Format produced by a successful compilation.

DType

Supported dense tensor element types.

TensorBuffer

Owned, contiguous tensor elements stored on the CPU.

TensorData

Borrowed, contiguous tensor elements.

TensorDataMut

Exclusively borrowed, contiguous tensor elements.

Functions

element_count

Checked product of a tensor’s dimensions. Scalars ([]) have one element; any zero dimension makes the tensor empty. Returns an error when a nonempty tensor’s element count overflows usize.

Type Aliases

Result

Result<T, Error>