Trait InferBackend

Source

pub trait InferBackend: Send + Sync {
    // Required methods
    fn infer<'life0, 'life1, 'life2, 'async_trait>(
        &'life0 self,
        prompt: &'life1 str,
        max_tokens: u32,
        on_token: &'life2 mut (dyn for<'t> FnMut(&'t str) -> bool + Send),
    ) -> Pin<Box<dyn Future<Output = Result<InferResult>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait,
             'life1: 'async_trait,
             'life2: 'async_trait;
    fn model_name(&self) -> String;
}

Expand description

Anything that can serve an inference request.

Required Methods§

Source

fn infer<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, prompt: &'life1 str, max_tokens: u32, on_token: &'life2 mut (dyn for<'t> FnMut(&'t str) -> bool + Send), ) -> Pin<Box<dyn Future<Output = Result<InferResult>> + Send + 'async_trait>>
where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Generate from prompt, invoking on_token once per token.

on_token returns whether to keep going; returning false ends generation early, which is how a guest’s Stop verdict is honoured.

The for<'t> is load-bearing. #[async_trait] rewrites elided lifetimes into named ones, which would make this closure non-generic over the token’s lifetime and leave implementations unable to hand it a local &str — an E0597 that appears only in the implementor, with a message that does not obviously point back here.

Source