Struct FixedBatchInferer 

pub struct FixedBatchInferer { /* private fields */ }

A reliable batched inferer that is a good fit if you know how much data you’ll have and want stable performance.

As an added bonus, it’ll subdivide your data into minibatches if the batching doesn’t fit perfectly. To make this work, it’ll add a single-element mode as well to ensure all data is consumed - such as if you feed it 9 elements with a configured batch size of 8.

You can configure a wide number of different batch sizes, and the largest one will be used. Note that the overhead for execution still is fairly large, but this helps amortize some of that cost away. For example; if you use a setup of [1, 2, 4, 8] as your supported batch sizes a batch of 15 elements would run each plan once.

Pros

• Good and predictable performance if you know amount of data
• Flexible if you sometimes get extra data to deal with

Cons

• Mini-batches add overhead
• Diminishing returns on each supported batch size.

Implementations

impl FixedBatchInferer

pub fn from_model(model: InferenceModel, sizes: &[usize]) -> TractResult<Self>

Create an inferer for the provided inference model.

Errors

Will only forward errors from the tract_core::model::Graph optimization and graph building steps.

pub fn from_typed(model: TypedModel, sizes: &[usize]) -> TractResult<Self>

Create an inferer for the provided typed model.

Errors

Will only forward errors from the tract_core::model::Graph optimization and graph building steps.

Trait Implementations

impl Inferer for FixedBatchInferer

fn infer_raw(&self, batch: &mut ScratchPadView<'_>) -> Result<(), Error>

Execute the model on the provided pre-batched data.

fn select_batch_size(&self, max_count: usize) -> usize

Query the inferer for how many elements it can deal with in a single batch.

fn raw_input_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model inputs.

fn raw_output_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model outputs.

fn begin_agent(&self, _id: u64)

fn end_agent(&self, _id: u64)

fn input_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model inputs. This API only contains the external API, so code-based transforms outside the model are hidden.

fn output_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model outputs. This API only contains the external API, so code-based transforms outside the model are hidden.

impl IntoStateful for FixedBatchInferer

fn into_stateful<WrapStack: InfererWrapper>( self, wrapper_stack: WrapStack, ) -> StatefulInferer<WrapStack, Self>

Construct a StatefulInferer by wrapping this concrete inferer with the given wrapper stack.