Struct MemoizingDynamicInferer 

pub struct MemoizingDynamicInferer { /* private fields */ }

The dynamic memoizing batch inferer generates execution plans to fit each batch perfectly, achieving near-perfect performance no matter how much data you have - with a hefty up-front cost for each new batch size.

The dynamic batcher has the highest potential throughput when the amount of data isn’t known. By dynamically generating execution plans to fit the exact amount of elements in each batch, it will give tract optimal knowledge for execution each time. The downside of this is that setting up a new plan is fairly costly, so doing this for a batch size that is only seen once will waste memory and compute resources.

While plans are cached; this still means that if your expected batch size is can vary greatly, you’ll end up with noticeable spikes each time a new plan is generated. If you know you’ll have one or a few batch sizes - but not the exact size - this batcher will end up providing good value and inform tuning for a fixed batcher later.

If you know some batch sizes but not all, you can preload the batcher with those plans to avoid having to build them at runtime.

Pros

• Optimal amortized performance without tuning
• Requires no tuning for good results

Cons

• For small amounts of data and large models the spikes can offset amortized gains significantly

Implementations

impl MemoizingDynamicInferer

pub fn from_model( model: InferenceModel, preloaded_sizes: &[usize], ) -> TractResult<Self>

Create an inferer for the provided inference model.

Errors

Will only forward errors from the tract_core::model::Graph optimization and graph building steps.

pub fn from_typed( model: TypedModel, preloaded_sizes: &[usize], ) -> TractResult<Self>

Create an inferer for the provided typed model.

Errors

Will only forward errors from the tract_core::model::Graph optimization and graph building steps.

Trait Implementations

impl Inferer for MemoizingDynamicInferer

fn select_batch_size(&self, max_count: usize) -> usize

Query the inferer for how many elements it can deal with in a single batch.

fn infer_raw(&self, pad: &mut ScratchPadView<'_>) -> Result<(), Error>

Execute the model on the provided pre-batched data.

fn raw_input_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model inputs.

fn raw_output_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model outputs.

fn begin_agent(&self, _id: u64)

fn end_agent(&self, _id: u64)

fn input_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model inputs. This API only contains the external API, so code-based transforms outside the model are hidden.

fn output_shapes(&self) -> &[(String, Vec<usize>)]

Retrieve the name and shapes of the model outputs. This API only contains the external API, so code-based transforms outside the model are hidden.

impl IntoStateful for MemoizingDynamicInferer

fn into_stateful<WrapStack: InfererWrapper>( self, wrapper_stack: WrapStack, ) -> StatefulInferer<WrapStack, Self>

Construct a StatefulInferer by wrapping this concrete inferer with the given wrapper stack.