Trait AsyncEngine 

pub trait AsyncEngine<Req: Send + 'static, Resp: AsyncEngineContextProvider, E: Data>: Send + Sync {
    // Required method
    fn generate<'life0, 'async_trait>(
        &'life0 self,
        request: Req,
    ) -> Pin<Box<dyn Future<Output = Result<Resp, E>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
}

Engine is a trait that defines the interface for a streaming engine. The synchronous Engine version is does not need to be awaited.

This is the core trait for all async engine implementations. It provides:

• Generic type parameters for request, response, and error types
• Async generation capabilities with proper error handling
• Thread-safe design with Send + Sync bounds

Type Parameters

• Req: The request type — required to be Send + 'static. The Sync bound was removed from Req for convenience: forcing Sync on Req propagates a + Sync constraint onto every type that flows in (in particular, every input-side trait-object alias), and no existing implementation of AsyncEngine relies on the Sync nature of the request. Revisit if a future implementation genuinely needs shared-reference access to a request value across threads.
• Resp: The response type that implements AsyncEngineContextProvider
• E: The error type that implements Data

Implementation Notes

Implementations should ensure proper error handling and resource management. The generate method should be cancellable via the response’s context provider.

Required Methods

fn generate<'life0, 'async_trait>( &'life0 self, request: Req, ) -> Pin<Box<dyn Future<Output = Result<Resp, E>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Generate a stream of completion responses.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors

impl<In: PipelineIO + Sync, Out: PipelineIO> AsyncEngine<In, Out, Error> for SegmentSource<In, Out>

impl<In: PipelineIO + Sync, Out: PipelineIO> AsyncEngine<In, Out, Error> for ServiceFrontend<In, Out>

impl<T, U> AsyncEngine<Context<AddressedRequest<T>>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for AddressedPushRouter

where T: Data + Serialize, U: Data + for<'de> Deserialize<'de> + MaybeError,

impl<T, U> AsyncEngine<Context<RequestStream<T>>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for PushRouter<T, U>

where T: Data + Serialize, U: Data + for<'de> Deserialize<'de> + MaybeError,

Bidirectional AsyncEngine impl for streaming-input workloads (e.g. the OpenAI Realtime API). Reserves a sticky worker up front — before any inbound frame is observed — and binds the whole input stream to that worker. KV and Direct modes inherit the same bail! invariants as the unary impl.

Reserve-before-observe rationale. The router-mode strategies (RoundRobin, Random, PowerOfTwoChoices, LeastLoaded, DeviceAwareWeighted) don’t depend on frame contents, so selection runs immediately and connection setup proceeds in parallel with the client producing its first frame. A client that connects but never sends one still releases the slot via the response-stream-drop path; the dispatch-side cancel_both cleanup covers the early-bail case.

impl<T, U> AsyncEngine<Context<T>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for Egress<SingleIn<T>, ManyOut<U>>

where T: Data + Serialize, U: for<'de> Deserialize<'de> + Data,

impl<T, U> AsyncEngine<Context<T>, Pin<Box<dyn AsyncEngineStream<U, Item = U>>>, Error> for PushRouter<T, U>

where T: Data + Serialize, U: Data + for<'de> Deserialize<'de> + MaybeError,

impl<UpIn, UpOut, DownIn, DownOut> AsyncEngine<UpIn, UpOut, Error> for PipelineOperator<UpIn, UpOut, DownIn, DownOut>

where UpIn: PipelineIO + Sync, DownIn: PipelineIO + Sync, DownOut: PipelineIO, UpOut: PipelineIO,

A PipelineOperator is an AsyncEngine for the upstream AsyncEngine<UpIn, UpOut, Error>.