Struct OnnxSession 

pub struct OnnxSession { /* private fields */ }

A loaded ONNX model with one active ONNX Runtime session.

Adds provider fallback, model metadata, and reusable I/O buffers around the native session. Methods are synchronous; load one session inside each inference worker.

This type is neither Send nor Sync. Create, use, and drop it on the same thread. Independent workers can own graph-enabled sessions on the same GPU. Native setup, first-run capture and teardown are coordinated; subsequent inference/replay can run concurrently on separate session-owned streams. CUDA graph state in ONNX Runtime is per host thread, so moving a session after capture would require renewed capture coordination, even without concurrent calls.

Start with Self::inference for ordinary or dynamic inputs. For fixed tensor shapes, Self::input_mut, Self::run, and Self::output reuse storage. Public inputs and outputs always use CPU memory, including on GPU backends.

Implementations

impl OnnxSession

pub fn load(path: impl AsRef<Path>, options: OnnxOptions) -> Result<Self>

Load an ONNX graph or an EPContext engine file (detected from its contents). EPContext files need a compatible configured provider; their original graph is not rebuilt. The file extension does not determine the format. External ONNX tensor files must be available relative to the model file when the model references them. Native libraries must already be installed; see runtime discovery.

Errors

Returns an error for missing/incompatible libraries, invalid model/configuration, unsupported dense tensor metadata, or when every selected provider fails. BackendSelection::Auto may continue to a later candidate; inspect Self::fallback_events and Self::backend after loading.

Example

use native_onnx::{OnnxOptions, OnnxSession};
let model = OnnxSession::load("model.onnx", OnnxOptions::cpu())?;
for input in &model.info().inputs {
    println!("{}: {:?} {:?}", input.name, input.dtype, input.shape);
}

pub fn compile( source: impl AsRef<Path>, destination: impl AsRef<Path>, options: CompileOptions, inputs: &[TensorView<'_>], ) -> Result<CompileReport>

Compile for an explicit target and validate the saved model with representative inputs. CompileOptions requires a target with its provider configuration. Any target failure is returned directly; compilation never falls back to another provider. The result reports an optimized ONNX graph or an embedded provider context. Existing output files are never overwritten; correspondence checks are the caller’s job.

TensorRT compilation currently requires one exportable partition and produces an ONNX EPContext containing its engine. CPU/CUDA produce optimized ONNX. The exported artifact is reopened and executed using inputs before the destination is created. TensorRT artifacts depend on compatible hardware and native libraries; loading with different builder settings does not rebuild them.

Errors

Returns an error for an existing destination, a source that is already an EPContext, invalid inputs/options, compilation or validation failure, CUDA stream/capture errors, or file I/O failures. No partial destination is retained on failure. Native compilation may use temporary disk space.

Inference options with automatic backend selection cannot be used to compile:

use native_onnx::{OnnxOptions, OnnxSession};
OnnxSession::compile("model.onnx", "compiled.onnx", OnnxOptions::default(), &[]);

pub fn backend(&self) -> Option<&str>

Primary registered provider, not a claim that every node runs on that provider. Auto permits node partitioning across providers. Returns None if an inference failure exhausted all fallback candidates and left no active session.

pub fn fallback_events(&self) -> &[FallbackEvent]

Whole-session provider failures recorded during loading or inference. Entries are chronological and retain formatted native error context. Successful node partitioning inside ORT does not add an event here.

pub fn is_prepared(&self) -> bool

Whether the active session has reusable buffers for fixed-shape tensor I/O. All inputs and outputs must have known ranks and positive, fixed dimensions. Custom providers use ordinary inference. This does not indicate whether CUDA graphs are enabled, or whether every model operation runs on a GPU.

pub fn cuda_graph_enabled(&self) -> bool

Whether the selected session is configured for CUDA graph capture/replay. Requires fixed, positive I/O, sequential execution and strict GPU placement (or a compiled EPContext). Capture completes during the first successful run.

pub fn end_profiling(&mut self) -> Result<String>

Finish ONNX Runtime profiling and return the profile’s file path. Enable profiling with OnnxOptions::profiling before loading the model. This ends profiling for the active native session. If fallback has occurred, earlier sessions may have separate profile files under the configured prefix. Native profiling or filesystem failures are returned with their context.

impl OnnxSession

pub fn info(&self) -> &ModelInfo

Return the ONNX model input/output metadata. Metadata reflects the active provider’s resolved shapes. A failed fallback leaves the last known metadata available even if Self::backend is None.

pub fn inference(&mut self, inputs: &[TensorView<'_>]) -> Result<Vec<Tensor>>

Run inference from borrowed inputs and return owned, CPU-ready output tensors.

Supply every input exactly once, matched by name; input order is arbitrary. Outputs are in ModelInfo::outputs order and remain valid after the next inference or after dropping the session. Fixed-shape models reuse internal buffers but copy results into the returned tensors. Dynamic models use ordinary ORT execution. No image preprocessing or output decoding is applied.

Errors

Names, element types, ranks, concrete dimensions, and buffer lengths must match the model. Provider errors are returned directly in required mode; automatic mode can retry the same inputs on a later backend. A native crash cannot be converted to a Rust error.

pub fn inference_into( &mut self, inputs: &[TensorView<'_>], outputs: &mut [TensorViewMut<'_>], ) -> Result<()>

Run inference into caller-owned output buffers. Fixed tensor I/O reuses prepared storage.

Inputs and outputs are matched by name and must each cover their full model interface without duplicates. Output shapes, types, and lengths must match the actual results. The dynamic path first creates owned results and then copies them; this method is not a guarantee of allocation-free execution.

Errors

Returns the input/provider errors of Self::inference, or an output name/shape/type/length mismatch. All output buffers are validated before result data is copied into them.

impl OnnxSession

pub fn input_mut(&mut self, name: &str) -> Result<TensorViewMut<'_>>

Borrow a persistent input buffer for fixed-shape tensor I/O. Inputs are initialized to zero when the session is loaded. Borrowing an input invalidates the previous prepared outputs, even if no elements are changed. The borrow must end before another mutable operation on the session. Other input buffers retain their values until updated.

Errors

Returns an error for an unknown name, unavailable prepared buffers, or a native tensor access failure. Use Self::inference for dynamic I/O.

pub fn output(&self, name: &str) -> Result<TensorView<'_>>

Borrow the last successful prepared output. The borrow expires before the next run. The data is CPU-accessible even when inference ran on a GPU. To retain it, call TensorView::to_owned before changing inputs or running again.

Errors

Returns an error before a successful run, after an input was borrowed for mutation, after a failed run, for an unknown name, or without prepared buffers.

pub fn run(&mut self) -> Result<()>

Run the fixed-shape model using its persistent input and output buffers. Unchanged inputs retain their previous values. Successful results become available through Self::output. This call completes GPU-to-host transfers before returning; it does not launch background inference.

Errors

Returns an error without prepared buffers or on an unrecoverable provider failure. In automatic mode, input data is preserved across fallback. If the fallback provider has no prepared output API, use Self::inference instead.