Trait InferBackend

Source

pub trait InferBackend: Send + Sync {
    // Required methods
    fn infer<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        req: InferRequest<'life1>,
        on_token: &'life2 mut (dyn for<'t> FnMut(&'t str) -> bool + Send),
    ) -> Pin<Box<dyn Future<Output = Result<InferResult>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait;
    fn model_name(&self) -> String;

    // Provided methods
    fn supports_images(&self) -> bool { ... }
    fn embed<'life0, 'life1, 'async_trait>(
        &'life0 self,
        _texts: &'life1 [String],
    ) -> Pin<Box<dyn Future<Output = Result<Vec<Vec<f32>>>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait { ... }
    fn supports_embeddings(&self) -> bool { ... }
}

Expand description

Anything that can serve an inference request.

Required Methods§

Source

fn infer<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, req: InferRequest<'life1>, on_token: &'life2 mut (dyn for<'t> FnMut(&'t str) -> bool + Send), ) -> Pin<Box<dyn Future<Output = Result<InferResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Generate from req.prompt, invoking on_token once per token.

on_token returns whether to keep going; returning false ends generation early, which is how a guest’s Stop verdict is honoured.

The for<'t> is load-bearing. #[async_trait] rewrites elided lifetimes into named ones, which would make this closure non-generic over the token’s lifetime and leave implementations unable to hand it a local &str — an E0597 that appears only in the implementor, with a message that does not obviously point back here.

Source

fn model_name(&self) -> String

Identifier recorded in the job’s usage accounting.

Provided Methods§

Source

fn supports_images(&self) -> bool

Whether this backend can accept images.

Defaults to false, so a backend added without thinking about vision refuses images rather than quietly discarding them.

Source

fn embed<'life0, 'life1, 'async_trait>( &'life0 self, _texts: &'life1 [String], ) -> Pin<Box<dyn Future<Output = Result<Vec<Vec<f32>>>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Embed each of texts, returning one vector per input, in order.

Batched rather than one call per text, and that is the whole point: embedding a corpus means tens of thousands of chunks, and a round trip each turns minutes into hours. Ollama’s /api/embed accepts an array natively, so the batch is real rather than a loop wearing a batch’s clothes.

The default refuses. A backend that cannot embed must say so rather than return empty vectors, which downstream would store as a valid row and quietly poison every similarity search made against it.

Source