Struct InferenceSession 

pub struct InferenceSession { /* private fields */ }

A loaded model ready to run inference (§20.2).

Implementations

impl InferenceSession

pub fn load(path: impl AsRef<Path>) -> Result<Self, SessionError>

Primary entry point: load a model with auto device detection.

pub fn load_bytes(bytes: &[u8]) -> Result<Self, SessionError>

Load a model from an in-memory buffer.

pub fn from_graph(graph: Graph) -> Result<Self, SessionError>

Build a session directly from an in-memory IR Graph.

Initializer bytes are read from the graph’s inline [WeightRef]s, so no on-disk model or weight store is required. Useful for programmatically constructed graphs and tests.

pub fn builder() -> SessionBuilder

Start a configuration builder.

pub fn run( &mut self, inputs: &[(&str, &Tensor)], ) -> Result<Vec<Tensor>, SessionError>

Run inference with named inputs, returning the graph outputs in order.

pub fn set_trace_context(&mut self, trace: TraceContext)

Attach the shared runtime trace context. When enabled, the executor opens one span per executed op so kernels can attach kernel-variant and capture-rejection reasons to a live span. Defaults to a disabled no-op context, so untraced runs pay only a single relaxed atomic load per op.

pub fn run_outputs( &mut self, inputs: &[(&str, &Tensor)], ) -> Result<Vec<SessionOutput>, SessionError>

Run inference and preserve tensor or sequence graph-output types.

pub fn decode_memo_counts(&self) -> (u64, u64, u64, u64)

F5 Stage 1 decode-plan memo activity counters (primed, rebuilt, replayed, ineligible) over this session’s lifetime. replayed > 0 after a decode run proves the memo actually engaged on the real (persistent-KV-binding) path; the coordinator’s on-model A/B reads this to reject a vacuous pass.

pub fn decode_view_plan_counts(&self) -> (u64, u64)

F5 Stage 2 view-plan activity counters (views_reused, dispatch_elided) over this session’s lifetime. Both > 0 after a decode run prove the invariant zero-copy view reuse and pure-view dispatch elision actually fired on the real path (not a vacuous pass); an on-model A/B reads this alongside Self::decode_memo_counts.

pub fn run_with_device_bindings( &mut self, inputs: &[(&str, &Tensor)], bindings: &mut [DeviceIoBinding], ) -> Result<Vec<Option<Tensor>>, SessionError>

Run with persistent device allocations supplying graph inputs and, optionally, aliasing graph outputs. Bound outputs are returned as None because their bytes remain resident in the caller-owned allocation.

pub fn allocate_device_binding( &self, input_name: impl Into<String>, output_name: Option<impl Into<String>>, dtype: DataType, physical_shape: Vec<usize>, logical_shape: Vec<usize>, ) -> Result<DeviceIoBinding, SessionError>

Allocate a persistent buffer on this session’s execution device.

pub fn allocate_device_output_binding( &self, output_name: impl Into<String>, dtype: DataType, physical_shape: Vec<usize>, logical_shape: Vec<usize>, ) -> Result<DeviceIoBinding, SessionError>

Allocate a persistent buffer for a graph output without also binding it as an input.

pub fn try_capture_with_device_bindings( &mut self, inputs: &[(&str, &Tensor)], bindings: &mut [DeviceIoBinding], ) -> Result<DeviceGraphCaptureResult, SessionError>

Execute once while recording the kernel launches into a device graph.

NotCapturable means the mandatory all-kernel audit rejected the run before stream capture began, so callers may safely retry eagerly.

pub fn replay_device_graph( &mut self, bindings: &mut [DeviceIoBinding], ) -> Result<bool, SessionError>

Replay the installed device graph after the caller has refreshed any persistent scalar inputs. Returns true when the graph is still valid for the next step, or false when a control-flow branch flip retired it this step (the token was produced correctly via eager fallback) and the caller should re-warm and re-capture.

pub fn reset_device_graph(&mut self) -> Result<bool, SessionError>

Invalidate the installed device graph before reset, rewind, shape change, or binding destruction.

pub fn captured_graph_segment_count(&self) -> usize

Number of captured device-graph segments installed by the most recent Self::try_capture_with_device_bindings call.

1 for a whole-subgraph capture; >= 2 when the CUDA EP claimed the subgraph but split it into segments around non-capturable seam nodes.

pub fn capture_segmentation(&self) -> &[CaptureDecline]

Structured, transparent segment boundaries from the most recent capture: one entry per non-capturable seam node the EP ran eagerly between captured segments (with its structural seam kind and CaptureSupport decline reason). Empty for a whole-subgraph capture.

pub fn check_device_capture_error(&self) -> Result<u32, SessionError>

Read (without clearing) any latching device capture-safety error recorded during graph replay, as a raw violation bitmask (zero when none). Callers poll this at the per-step logits sync to fail before consuming a token produced from an out-of-range captured replay.

pub fn device_allocation_counts(&self) -> Option<DeviceAllocationCounts>

pub fn device_id(&self) -> DeviceId

pub fn execution_provider_fallback_report( &self, ) -> Option<&ExecutionProviderFallbackReport>

Report why an explicitly requested accelerator session was assigned to CPU instead. None means the requested EP serves the whole graph.

pub fn inputs(&self) -> &[IoMeta]

Input metadata.

pub fn outputs(&self) -> &[IoMeta]

Output metadata.

pub fn model_metadata(&self) -> &ModelMetadata

Model-level metadata from the source ModelProto.

pub fn cache_stats(&self) -> CacheStats

Kernel-cache statistics (§11.1); useful to observe warmup/run reuse.

pub fn control_flow_stats(&self) -> ControlFlowStats

Control-flow subgraph build/run statistics. A Loop or Scan body with a stable input-shape signature should build once and run many times.

pub fn warmup(&mut self, shapes: &[WarmupShape]) -> Result<(), SessionError>

Pre-compile kernels for common shapes to avoid first-inference latency (§11.3). Phase-1 minimal: the compiled plan’s shapes already key the cache, so this repopulates it for the plan; shapes are validated to name real inputs.

pub fn ep_context_config(&self) -> &EpContextDumpConfig

The EPContext dump configuration parsed from the ep.context_* session options (§21.4). Disabled by default.

pub fn graph(&self) -> &Graph

The session’s (post-optimize) compiled graph.

This is the graph the executor runs and the same one Self::export_ep_context serialises — a caller identifying the NodeIds of a compiled partition (the CompiledPartition::covered_nodes) must read them from here so they reference the exact nodes the exporter will splice out. This is the compiler-integration seam: a real compiling EP inspects this graph to choose the subgraphs it claims.

pub fn export_ep_context( &self, orig_path: &Path, partitions: &[CompiledPartition<'_>], ) -> Result<PathBuf, SessionError>

Export a com.microsoft::EPContext context-cache model for this session (§55.4 dump path), driven by the ep.context_* session options (Self::ep_context_config).

orig_path is the source model path the default output location (<orig>_ctx.onnx) is derived from when ep.context_file_path is unset. partitions are the EP-compiled partitions to serialise — each names the ExecutionProvider that compiled it, so the driver pulls the blob + SDK version via save_context and the source key via context_source_keys (§55.6 — nothing is hardcoded).

When ep.context_enable is false (the default) this is a no-op: no EP save_context is called and no files are written; it returns the path it would have written to.

Compiler-integration seam

The Phase-1 CPU EP has no compile step, so no real EP yet yields CompiledPartitions — partitions is therefore supplied by the caller (proven end-to-end with a mock compiling EP in the crate tests). TODO(compiler): when a real compiling EP lands, collect its partitions from the compile/placement stage and call this internally at build time so a session created with ep.context_enable=1 dumps automatically.