Struct Inference 

pub struct Inference<B: Backend, M, I, O, S = ()> {
    pub id: String,
    /* private fields */
}

Inference instance wrapping a single model and a handler routine.

An Inference can create multiple jobs (sequentially or concurrently) without re-loading the model.

Fields

id: String

Implementations

impl<B, M, I, O, S> Inference<B, M, I, O, S>

where B: Backend, M: Send + 'static, I: RoutineInput + 'static, O: Send + 'static, S: Send + Sync + 'static,

pub fn infer( &self, input: I::Inner<'static>, ) -> StrappedInferenceJobBuilder<'_, B, M, I, O, S, StateMissing>

Start building an inference job for the given input payload.

pub fn run(&self, job: InferenceJob<B, I, S>) -> Result<Vec<O>, InferenceError>

Execute the provided job synchronously and collect all emitted outputs.

pub fn spawn(&self, job: InferenceJob<B, I, S>) -> JobHandle<O>

where <I as RoutineInput>::Inner<'static>: Send,

Spawn the job on a background thread returning a JobHandle. Outputs can be read from the handle’s stream.

pub fn into_model(self) -> M

Consume the inference instance and retrieve ownership of the underlying model.