Struct async_tensorrt::engine::ExecutionContext

pub struct ExecutionContext<'engine> { /* private fields */ }

Context for executing inference using an engine.

TensorRT documentation

Implementations

impl ExecutionContext<'static>

pub async fn from_engine(engine: Engine) -> Result<Self, Error>

Create an execution context from an Engine.

This is the owned version of ExecutionContext::new(). It consumes the engine. In exchange, it produces an execution context with a 'static lifetime.

TensorRT documentation

Arguments

• engine - Parent engine.

pub async fn from_engine_many( engine: Engine, num: usize ) -> Result<Vec<Self>, Error>

Create multiple execution contexts from an Engine.

This is the owned version of ExecutionContext::new(). It consumes the engine. In exchange, it produces a set of execution contexts with a 'static lifetime.

TensorRT documentation

Arguments

• engine - Parent engine.
• num - Number of execution contexsts to produce.

impl<'engine> ExecutionContext<'engine>

pub async fn new(engine: &mut Engine) -> Result<ExecutionContext<'_>, Error>

Create an execution context from an Engine.

TensorRT documentation

Arguments

• engine - Parent engine.

pub async fn enqueue<T: Copy>( &mut self, io_buffers: &mut HashMap<&str, &mut DeviceBuffer<T>>, stream: &Stream ) -> Result<(), Error>

Asynchronously execute inference.

TensorRT documentation

Stream ordered semantics

This function exhibits stream ordered semantics. This means that it is only guaranteed to complete serially with respect to other operations on the same stream.

Thread-safety

Calling this function from the same context with a different CUDA stream concurrently results in undefined behavior. To perform inference concurrently in multiple streams, use one execution context per stream.

Arguments

• io_buffers - Input and output buffers.
• stream - CUDA stream to execute on.