Struct ort::session::SessionBuilder
source · pub struct SessionBuilder { /* private fields */ }
Expand description
Type used to create a session using the builder pattern. Once created, you can use the different methods to configure the session.
Once configured, use the SessionBuilder::with_model_from_file
method to “commit” the builder configuration into a Session
.
Example
let environment = Environment::builder()
.with_name("test")
.with_log_level(LoggingLevel::Verbose)
.build()?
.into_arc();
let mut session = SessionBuilder::new(&environment)?
.with_optimization_level(GraphOptimizationLevel::Level1)?
.with_intra_threads(1)?
.with_model_from_file("squeezenet.onnx")?;
Implementations§
source§impl SessionBuilder
impl SessionBuilder
sourcepub fn new(env: &Arc<Environment>) -> OrtResult<Self>
pub fn new(env: &Arc<Environment>) -> OrtResult<Self>
Creates a new session builder in the given environment.
sourcepub fn with_execution_providers(
self,
execution_providers: impl AsRef<[ExecutionProvider]>
) -> OrtResult<Self>
pub fn with_execution_providers( self, execution_providers: impl AsRef<[ExecutionProvider]> ) -> OrtResult<Self>
Configures a list of execution providers to attempt to use for the session.
Execution providers are loaded in the order they are provided until a suitable execution provider is found. Most execution providers will silently fail if they are unavailable or misconfigured (see notes below), however, some may log to the console, which is sadly unavoidable. The CPU execution provider is always available, so always put it last in the list (though it is not required).
Execution providers will only work if the corresponding Cargo feature is enabled and ONNX Runtime was built with support for the corresponding execution provider. Execution providers that do not have their corresponding feature enabled are currently ignored.
Execution provider options can be specified in the second argument. Refer to ONNX Runtime’s
execution provider docs for configuration options. In most
cases, passing None
to configure with no options is suitable.
It is recommended to enable the cuda
EP for x86 platforms and the acl
EP for ARM platforms for the best
performance, though this does mean you’ll have to build ONNX Runtime for these targets. Microsoft’s prebuilt
binaries are built with CUDA and TensorRT support, if you built ort
with the cuda
or tensorrt
features
enabled.
Supported execution providers:
cpu
: Default CPU/MLAS execution provider. Available on all platforms.acl
: Arm Compute Librarycuda
: NVIDIA CUDA/cuDNNtensorrt
: NVIDIA TensorRT
Notes
- Use of
SessionBuilder::with_execution_providers
in a library is discouraged. Execution providers should always be configurable by the user, in case an execution provider is misconfigured and causes the application to crash (see notes below). Instead, your library should accept anEnvironment
from the user rather than creating its own. This way, the user can configure execution providers for all modules that use it. - Using the CUDA/TensorRT execution providers can terminate the process if the CUDA/TensorRT installation is
misconfigured. Configuring the execution provider will seem to work, but when you attempt to run a session,
it will hard crash the process with a “stack buffer overrun” error. This can occur when CUDA/TensorRT is
missing a DLL such as
zlibwapi.dll
. To prevent your app from crashing, you can check to see if you can loadzlibwapi.dll
before enabling the CUDA/TensorRT execution providers.
sourcepub fn with_intra_threads(self, num_threads: i16) -> OrtResult<Self>
pub fn with_intra_threads(self, num_threads: i16) -> OrtResult<Self>
Configure the session to use a number of threads to parallelize the execution within nodes. If ONNX Runtime was
built with OpenMP (as is the case with Microsoft’s prebuilt binaries), this will have no effect on the number of
threads used. Instead, you can configure the number of threads OpenMP uses via the OMP_NUM_THREADS
environment
variable.
For configuring the number of threads used when the session execution mode is set to Parallel
, see
SessionBuilder::with_inter_threads()
.
sourcepub fn with_disable_per_session_threads(self) -> OrtResult<Self>
pub fn with_disable_per_session_threads(self) -> OrtResult<Self>
Configure the session to disable per-session thread pool, instead using the environment’s global thread pool.
This must be used with an environment created with
EnvBuilder::with_global_thread_pool
enabled.
sourcepub fn with_inter_threads(self, num_threads: i16) -> OrtResult<Self>
pub fn with_inter_threads(self, num_threads: i16) -> OrtResult<Self>
Configure the session to use a number of threads to parallelize the execution of the graph. If nodes can be run in parallel, this sets the maximum number of threads to use to run them in parallel.
This has no effect when the session execution mode is set to Sequential
.
For configuring the number of threads used to parallelize the execution within nodes, see
SessionBuilder::with_intra_threads()
.
sourcepub fn with_parallel_execution(
self,
parallel_execution: bool
) -> OrtResult<Self>
pub fn with_parallel_execution( self, parallel_execution: bool ) -> OrtResult<Self>
Enable/disable the parallel execution mode for this session. By default, this is disabled.
Parallel execution can improve performance for models with many branches, at the cost of higher memory usage.
You can configure the amount of threads used to parallelize the execution of the graph via
SessionBuilder::with_inter_threads()
.
sourcepub fn with_optimization_level(
self,
opt_level: GraphOptimizationLevel
) -> OrtResult<Self>
pub fn with_optimization_level( self, opt_level: GraphOptimizationLevel ) -> OrtResult<Self>
Set the session’s optimization level. See GraphOptimizationLevel
for more information on the different
optimization levels.
sourcepub fn with_memory_pattern(self, enable: bool) -> OrtResult<Self>
pub fn with_memory_pattern(self, enable: bool) -> OrtResult<Self>
Enables/disables memory pattern optimization. Disable it if the input size varies, i.e., dynamic batch
sourcepub fn with_allocator(self, allocator: AllocatorType) -> OrtResult<Self>
pub fn with_allocator(self, allocator: AllocatorType) -> OrtResult<Self>
Set the session’s allocator. Defaults to AllocatorType::Device
.
sourcepub fn with_memory_type(self, memory_type: MemType) -> OrtResult<Self>
pub fn with_memory_type(self, memory_type: MemType) -> OrtResult<Self>
Set the session’s memory type. Defaults to MemType::Default
.
sourcepub fn with_custom_op_lib(self, lib_path: &str) -> OrtResult<Self>
pub fn with_custom_op_lib(self, lib_path: &str) -> OrtResult<Self>
Registers a custom operator library with the given library path in the session.
sourcepub fn with_model_downloaded<M>(self, model: M) -> OrtResult<Session>where
M: ModelUrl,
pub fn with_model_downloaded<M>(self, model: M) -> OrtResult<Session>where M: ModelUrl,
Downloads a pre-trained ONNX model from the ONNX Model Zoo and builds the session.
sourcepub fn with_model_from_file<P>(
self,
model_filepath_ref: P
) -> OrtResult<Session>where
P: AsRef<Path>,
pub fn with_model_from_file<P>( self, model_filepath_ref: P ) -> OrtResult<Session>where P: AsRef<Path>,
Loads an ONNX model from a file and builds the session.
sourcepub fn with_model_from_file_and_external_initializers<'v, 'i, P>(
self,
model_filepath_ref: P,
initializers: &'i [(String, Value<'v>)]
) -> OrtResult<Session>where
P: AsRef<Path>,
'i: 'v,
pub fn with_model_from_file_and_external_initializers<'v, 'i, P>( self, model_filepath_ref: P, initializers: &'i [(String, Value<'v>)] ) -> OrtResult<Session>where P: AsRef<Path>, 'i: 'v,
Loads an ONNX model from a file, replacing external data with data provided in initializers.
This will find initialized tensors with external data in the graph with the provided names and replace them with the provided tensors. The replacement will occur before any optimizations take place, and the data will be copied into the graph. Tensors replaced by this function must be using external data. (you cannot replace a non-external tensor)
sourcepub fn with_model_from_memory(
self,
model_bytes: &[u8]
) -> OrtResult<InMemorySession<'_>>
pub fn with_model_from_memory( self, model_bytes: &[u8] ) -> OrtResult<InMemorySession<'_>>
Load an ONNX graph from memory and commit the session.