Struct ort::session::SessionBuilder

pub struct SessionBuilder { /* private fields */ }

Type used to create a session using the builder pattern. Once created, you can use the different methods to configure the session.

Once configured, use the SessionBuilder::with_model_from_file method to “commit” the builder configuration into a Session.

Example

let environment = Environment::builder()
	.with_name("test")
	.with_log_level(LoggingLevel::Verbose)
	.build()?
	.into_arc();
let mut session = SessionBuilder::new(&environment)?
	.with_optimization_level(GraphOptimizationLevel::Level1)?
	.with_intra_threads(1)?
	.with_model_from_file("squeezenet.onnx")?;

Implementations

impl SessionBuilder

pub fn new(env: &Arc<Environment>) -> OrtResult<Self>

Creates a new session builder in the given environment.

pub fn with_execution_providers( self, execution_providers: impl AsRef<[ExecutionProvider]> ) -> OrtResult<Self>

Configures a list of execution providers to attempt to use for the session.

Execution providers are loaded in the order they are provided until a suitable execution provider is found. Most execution providers will silently fail if they are unavailable or misconfigured (see notes below), however, some may log to the console, which is sadly unavoidable. The CPU execution provider is always available, so always put it last in the list (though it is not required).

Execution providers will only work if the corresponding Cargo feature is enabled and ONNX Runtime was built with support for the corresponding execution provider. Execution providers that do not have their corresponding feature enabled are currently ignored.

Execution provider options can be specified in the second argument. Refer to ONNX Runtime’s execution provider docs for configuration options. In most cases, passing None to configure with no options is suitable.

It is recommended to enable the cuda EP for x86 platforms and the acl EP for ARM platforms for the best performance, though this does mean you’ll have to build ONNX Runtime for these targets. Microsoft’s prebuilt binaries are built with CUDA and TensorRT support, if you built ort with the cuda or tensorrt features enabled.

Supported execution providers:

• cpu: Default CPU/MLAS execution provider. Available on all platforms.
• acl: Arm Compute Library
• cuda: NVIDIA CUDA/cuDNN
• tensorrt: NVIDIA TensorRT

Notes

• Use of SessionBuilder::with_execution_providers in a library is discouraged. Execution providers should always be configurable by the user, in case an execution provider is misconfigured and causes the application to crash (see notes below). Instead, your library should accept an Environment from the user rather than creating its own. This way, the user can configure execution providers for all modules that use it.
• Using the CUDA/TensorRT execution providers can terminate the process if the CUDA/TensorRT installation is misconfigured. Configuring the execution provider will seem to work, but when you attempt to run a session, it will hard crash the process with a “stack buffer overrun” error. This can occur when CUDA/TensorRT is missing a DLL such as zlibwapi.dll. To prevent your app from crashing, you can check to see if you can load zlibwapi.dll before enabling the CUDA/TensorRT execution providers.

pub fn with_intra_threads(self, num_threads: i16) -> OrtResult<Self>

Configure the session to use a number of threads to parallelize the execution within nodes. If ONNX Runtime was built with OpenMP (as is the case with Microsoft’s prebuilt binaries), this will have no effect on the number of threads used. Instead, you can configure the number of threads OpenMP uses via the OMP_NUM_THREADS environment variable.

For configuring the number of threads used when the session execution mode is set to Parallel, see SessionBuilder::with_inter_threads().

pub fn with_disable_per_session_threads(self) -> OrtResult<Self>

Configure the session to disable per-session thread pool, instead using the environment’s global thread pool. This must be used with an environment created with EnvBuilder::with_global_thread_pool enabled.

pub fn with_inter_threads(self, num_threads: i16) -> OrtResult<Self>

Configure the session to use a number of threads to parallelize the execution of the graph. If nodes can be run in parallel, this sets the maximum number of threads to use to run them in parallel.

This has no effect when the session execution mode is set to Sequential.

For configuring the number of threads used to parallelize the execution within nodes, see SessionBuilder::with_intra_threads().

pub fn with_parallel_execution( self, parallel_execution: bool ) -> OrtResult<Self>

Enable/disable the parallel execution mode for this session. By default, this is disabled.

Parallel execution can improve performance for models with many branches, at the cost of higher memory usage. You can configure the amount of threads used to parallelize the execution of the graph via SessionBuilder::with_inter_threads().

pub fn with_optimization_level( self, opt_level: GraphOptimizationLevel ) -> OrtResult<Self>

Set the session’s optimization level. See GraphOptimizationLevel for more information on the different optimization levels.

pub fn with_memory_pattern(self, enable: bool) -> OrtResult<Self>

Enables/disables memory pattern optimization. Disable it if the input size varies, i.e., dynamic batch

pub fn with_allocator(self, allocator: AllocatorType) -> OrtResult<Self>

Set the session’s allocator. Defaults to AllocatorType::Device.

pub fn with_memory_type(self, memory_type: MemType) -> OrtResult<Self>

Set the session’s memory type. Defaults to MemType::Default.

pub fn with_custom_op_lib(self, lib_path: &str) -> OrtResult<Self>

Registers a custom operator library with the given library path in the session.

pub fn with_model_downloaded<M>(self, model: M) -> OrtResult<Session>where M: ModelUrl,

Downloads a pre-trained ONNX model from the ONNX Model Zoo and builds the session.

pub fn with_model_from_file<P>( self, model_filepath_ref: P ) -> OrtResult<Session>where P: AsRef<Path>,

Loads an ONNX model from a file and builds the session.

pub fn with_model_from_memory( self, model_bytes: &[u8] ) -> OrtResult<InMemorySession<'_>>

Load an ONNX graph from memory and commit the session.

Trait Implementations

impl Debug for SessionBuilder

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Drop for SessionBuilder

fn drop(&mut self)

Executes the destructor for this type.