Struct InferenceEngine 

pub struct InferenceEngine {
    pub model: Model,
    pub device: Device,
    pub stats: InferenceStats,
}

Inference engine wrapping a model and target device.

Fields

model: Model

device: Device

stats: InferenceStats

Implementations

impl InferenceEngine

pub fn new(model: Model, device: Device) -> Self

pub fn infer(&mut self, input: &Tensor) -> Tensor

Run a single inference pass.

pub fn batch_infer(&mut self, inputs: &[Tensor]) -> Vec<Tensor>

Batched inference: run each input through the model.

pub fn warm_up(&mut self, input_shape: Vec<usize>, runs: usize)

Warm up the inference pipeline by running dummy inputs.