pub struct DynamicInferer { /* private fields */ }Expand description
The dynamic inferer hits a spot between the raw simplicity of a crate::prelude::BasicInferer and the spikiness
of a crate::prelude::MemoizingDynamicInferer. Instead of explicitly concretizing models and caching them, it
relies on tracts internal concretization which leads to worse performance overall; but beating out the
crate::prelude::BasicInferer.
§Pros
- Requires no tuning for OK results
- Fixed memory and fairly linear performance scaling
§Cons
- Small extra overhead for small extra performance
- Worst option for small batch sizes
Implementations§
Source§impl DynamicInferer
impl DynamicInferer
Sourcepub fn from_model(model: InferenceModel) -> TractResult<Self>
pub fn from_model(model: InferenceModel) -> TractResult<Self>
Create an inferer for the provided inference model.
§Errors
Will only forward errors from the tract_core::model::Graph optimization and graph building steps.
Sourcepub fn from_typed(model: TypedModel) -> TractResult<Self>
pub fn from_typed(model: TypedModel) -> TractResult<Self>
Create an inferer for the provided typed model.
§Errors
Will only forward errors from the tract_core::model::Graph optimization and graph building steps.
Trait Implementations§
Source§impl Inferer for DynamicInferer
impl Inferer for DynamicInferer
Source§fn select_batch_size(&self, max_count: usize) -> usize
fn select_batch_size(&self, max_count: usize) -> usize
Query the inferer for how many elements it can deal with in a single batch.
Source§fn infer_raw(&self, pad: &mut ScratchPadView<'_>) -> Result<(), Error>
fn infer_raw(&self, pad: &mut ScratchPadView<'_>) -> Result<(), Error>
Execute the model on the provided pre-batched data.
Source§fn raw_input_shapes(&self) -> &[(String, Vec<usize>)]
fn raw_input_shapes(&self) -> &[(String, Vec<usize>)]
Retrieve the name and shapes of the model inputs.
Source§fn raw_output_shapes(&self) -> &[(String, Vec<usize>)]
fn raw_output_shapes(&self) -> &[(String, Vec<usize>)]
Retrieve the name and shapes of the model outputs.
fn begin_agent(&self, _id: u64)
fn end_agent(&self, _id: u64)
Source§impl IntoStateful for DynamicInferer
impl IntoStateful for DynamicInferer
Source§fn into_stateful<WrapStack: InfererWrapper>(
self,
wrapper_stack: WrapStack,
) -> StatefulInferer<WrapStack, Self>
fn into_stateful<WrapStack: InfererWrapper>( self, wrapper_stack: WrapStack, ) -> StatefulInferer<WrapStack, Self>
Construct a
StatefulInferer by wrapping this concrete
inferer with the given wrapper stack.Auto Trait Implementations§
impl Freeze for DynamicInferer
impl !RefUnwindSafe for DynamicInferer
impl Send for DynamicInferer
impl Sync for DynamicInferer
impl Unpin for DynamicInferer
impl !UnwindSafe for DynamicInferer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> Downcast for Twhere
T: Any,
impl<T> Downcast for Twhere
T: Any,
Source§fn into_any(self: Box<T>) -> Box<dyn Any>
fn into_any(self: Box<T>) -> Box<dyn Any>
Convert
Box<dyn Trait> (where Trait: Downcast) to Box<dyn Any>. Box<dyn Any> can
then be further downcast into Box<ConcreteType> where ConcreteType implements Trait.Source§fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
fn into_any_rc(self: Rc<T>) -> Rc<dyn Any>
Convert
Rc<Trait> (where Trait: Downcast) to Rc<Any>. Rc<Any> can then be
further downcast into Rc<ConcreteType> where ConcreteType implements Trait.Source§fn as_any(&self) -> &(dyn Any + 'static)
fn as_any(&self) -> &(dyn Any + 'static)
Convert
&Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot
generate &Any’s vtable from &Trait’s.Source§fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
fn as_any_mut(&mut self) -> &mut (dyn Any + 'static)
Convert
&mut Trait (where Trait: Downcast) to &Any. This is needed since Rust cannot
generate &mut Any’s vtable from &mut Trait’s.Source§impl<T> DowncastSync for T
impl<T> DowncastSync for T
Source§impl<T> InfererExt for Twhere
T: Inferer,
impl<T> InfererExt for Twhere
T: Inferer,
Source§fn with_default_epsilon(self, key: &str) -> Result<EpsilonInjector<Self>>
fn with_default_epsilon(self, key: &str) -> Result<EpsilonInjector<Self>>
Add an epsilon injector using the default noise kind.
Source§fn with_epsilon<G: NoiseGenerator>(
self,
generator: G,
key: &str,
) -> Result<EpsilonInjector<Self, G>>
fn with_epsilon<G: NoiseGenerator>( self, generator: G, key: &str, ) -> Result<EpsilonInjector<Self, G>>
Add an epsilon injector with a specific noise generator.
Source§fn into_batched(self) -> Batched<Self>
fn into_batched(self) -> Batched<Self>
Wrap in a batching interface.
Source§fn infer(
&mut self,
observations: HashMap<u64, State<'_>>,
) -> Result<HashMap<u64, Response<'_>>, Error>
fn infer( &mut self, observations: HashMap<u64, State<'_>>, ) -> Result<HashMap<u64, Response<'_>>, Error>
👎Deprecated since 0.3.0: Please use the more explicit ‘infer_batch’ instead.
Execute the model on the provided batch of elements.
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more