pub struct InferenceSession { /* private fields */ }Expand description
A loaded model ready to run inference (§20.2).
Implementations§
Source§impl InferenceSession
impl InferenceSession
Sourcepub fn load(path: impl AsRef<Path>) -> Result<Self, SessionError>
pub fn load(path: impl AsRef<Path>) -> Result<Self, SessionError>
Primary entry point: load a model with auto device detection.
Sourcepub fn load_bytes(bytes: &[u8]) -> Result<Self, SessionError>
pub fn load_bytes(bytes: &[u8]) -> Result<Self, SessionError>
Load a model from an in-memory buffer.
Sourcepub fn from_graph(graph: Graph) -> Result<Self, SessionError>
pub fn from_graph(graph: Graph) -> Result<Self, SessionError>
Build a session directly from an in-memory IR Graph.
Initializer bytes are read from the graph’s inline [WeightRef]s, so no
on-disk model or weight store is required. Useful for programmatically
constructed graphs and tests.
Sourcepub fn builder() -> SessionBuilder
pub fn builder() -> SessionBuilder
Start a configuration builder.
Sourcepub fn run(
&mut self,
inputs: &[(&str, &Tensor)],
) -> Result<Vec<Tensor>, SessionError>
pub fn run( &mut self, inputs: &[(&str, &Tensor)], ) -> Result<Vec<Tensor>, SessionError>
Run inference with named inputs, returning the graph outputs in order.
Sourcepub fn cache_stats(&self) -> CacheStats
pub fn cache_stats(&self) -> CacheStats
Kernel-cache statistics (§11.1); useful to observe warmup/run reuse.
Sourcepub fn warmup(&mut self, shapes: &[WarmupShape]) -> Result<(), SessionError>
pub fn warmup(&mut self, shapes: &[WarmupShape]) -> Result<(), SessionError>
Pre-compile kernels for common shapes to avoid first-inference latency
(§11.3). Phase-1 minimal: the compiled plan’s shapes already key the
cache, so this repopulates it for the plan; shapes are validated to
name real inputs.
Sourcepub fn ep_context_config(&self) -> &EpContextDumpConfig
pub fn ep_context_config(&self) -> &EpContextDumpConfig
The EPContext dump configuration parsed from the ep.context_* session
options (§21.4). Disabled by default.
Sourcepub fn graph(&self) -> &Graph
pub fn graph(&self) -> &Graph
The session’s (post-optimize) compiled graph.
This is the graph the executor runs and the same one
Self::export_ep_context serialises — a caller identifying the
NodeIds of a compiled partition (the
CompiledPartition::covered_nodes) must read them from here so they
reference the exact nodes the exporter will splice out. This is the
compiler-integration seam: a real compiling EP inspects this graph to
choose the subgraphs it claims.
Sourcepub fn export_ep_context(
&self,
orig_path: &Path,
partitions: &[CompiledPartition<'_>],
) -> Result<PathBuf, SessionError>
pub fn export_ep_context( &self, orig_path: &Path, partitions: &[CompiledPartition<'_>], ) -> Result<PathBuf, SessionError>
Export a com.microsoft::EPContext context-cache model for this session
(§55.4 dump path), driven by the ep.context_* session options
(Self::ep_context_config).
orig_path is the source model path the default output location
(<orig>_ctx.onnx) is derived from when ep.context_file_path is unset.
partitions are the EP-compiled partitions to serialise — each names the
ExecutionProvider that
compiled it, so the driver pulls the blob + SDK version via
save_context and
the source key via
context_source_keys
(§55.6 — nothing is hardcoded).
When ep.context_enable is false (the default) this is a no-op: no
EP save_context is called and no files are written; it returns the path
it would have written to.
§Compiler-integration seam
The Phase-1 CPU EP has no compile step, so no real EP yet yields
CompiledPartitions — partitions is therefore supplied by the
caller (proven end-to-end with a mock compiling EP in the crate tests).
TODO(compiler): when a real compiling EP lands, collect its partitions
from the compile/placement stage and call this internally at build time
so a session created with ep.context_enable=1 dumps automatically.