Trait AsyncEngine

pub trait AsyncEngine<Req: Data, Resp: Data + AsyncEngineContextProvider, E: Data>: Send + Sync {
    // Required method
    fn generate<'life0, 'async_trait>(
        &'life0 self,
        request: Req,
    ) -> Pin<Box<dyn Future<Output = Result<Resp, E>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
}

Engine is a trait that defines the interface for a steaming LLM completion engine. The synchronous Engine version is does not need to be awaited.

Required Methods

fn generate<'life0, 'async_trait>( &'life0 self, request: Req, ) -> Pin<Box<dyn Future<Output = Result<Resp, E>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Generate a stream of completion responses.

Implementors

impl<In: PipelineIO, Out: PipelineIO> AsyncEngine<In, Out, Error> for SegmentSource<In, Out>

impl<In: PipelineIO, Out: PipelineIO> AsyncEngine<In, Out, Error> for ServiceFrontend<In, Out>

impl<T, U> AsyncEngine<Context<AddressedRequest<T>>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for AddressedPushRouter

where T: Data + Serialize, U: Data + for<'de> Deserialize<'de>,

impl<T, U> AsyncEngine<Context<T>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for Client<T, U>

where T: Data + Serialize, U: Data + for<'de> Deserialize<'de>,

impl<T, U> AsyncEngine<Context<T>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for Egress<SingleIn<T>, ManyOut<U>>

where T: Data + Serialize, U: for<'de> Deserialize<'de> + Data,

impl<UpIn, UpOut, DownIn, DownOut> AsyncEngine<UpIn, UpOut, Error> for PipelineOperator<UpIn, UpOut, DownIn, DownOut>

where UpIn: PipelineIO, DownIn: PipelineIO, DownOut: PipelineIO, UpOut: PipelineIO,

A PipelineOperator is an AsyncEngine for the upstream AsyncEngine<UpIn, UpOut, Error>.