Trait InferBackend

Source

pub trait InferBackend: Send + Sync {
    // Required methods
    fn infer<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        req: InferRequest<'life1>,
        on_token: &'life2 mut (dyn for<'t> FnMut(&'t str) -> bool + Send),
    ) -> Pin<Box<dyn Future<Output = Result<InferResult>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait;
    fn model_name(&self) -> String;

    // Provided method
    fn supports_images(&self) -> bool { ... }
}

Expand description

Anything that can serve an inference request.

Required Methods§

Source

fn infer<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, req: InferRequest<'life1>, on_token: &'life2 mut (dyn for<'t> FnMut(&'t str) -> bool + Send), ) -> Pin<Box<dyn Future<Output = Result<InferResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Generate from req.prompt, invoking on_token once per token.

on_token returns whether to keep going; returning false ends generation early, which is how a guest’s Stop verdict is honoured.

The for<'t> is load-bearing. #[async_trait] rewrites elided lifetimes into named ones, which would make this closure non-generic over the token’s lifetime and leave implementations unable to hand it a local &str — an E0597 that appears only in the implementor, with a message that does not obviously point back here.

Source