Struct Batcher 

pub struct Batcher { /* private fields */ }

Low-level batch builder to help transition from per-entity code to batched inference. Consider using the Batched wrapper instead to avoid tracking two objects.

Reusing this across frames will have a noticeable performance impact for large model inputs or outputs, and reduce memory pressure.

Note that Batchers are specific to the inferer used for initialization.

Implementations

impl Batcher

pub fn new(inferer: &dyn Inferer) -> Self

Create a new batcher for the provided inferer.

pub fn new_sized(inferer: &dyn Inferer, size: usize) -> Self

Create a new batcher for the provided inferer with space for the specified batch size.

pub fn push(&mut self, id: u64, state: State<'_>) -> Result<()>

Insert a single element into the batch to include in the next execution.

pub fn extend<'a, Iter: IntoIterator<Item = (u64, State<'a>)>>( &mut self, states: Iter, ) -> Result<()>

Insert a sequence of elements into the batch to include in the next execution.

pub fn execute<'b>( &mut self, inferer: &'b dyn Inferer, ) -> Result<HashMap<u64, Response<'b>>>

Run the provided inferer on the data that has been enqueued previously.

pub fn is_empty(&self) -> bool

Check if there is any data to run on here.

pub fn len(&self) -> usize

Amount of elements to run on in the batch here.