Trait NativeLlmPort 

pub trait NativeLlmPort: Send + Sync {
    // Required methods
    fn complete(
        &self,
        req: LlmRequest,
    ) -> impl Future<Output = Result<LlmResponse, LlmError>> + Send;
    fn complete_stream(
        &self,
        req: LlmRequest,
    ) -> impl Future<Output = Result<LlmStream, LlmError>> + Send;
}

Native async LLM port — implement this for the cleanest developer experience.

The framework provides a blanket implementation that bridges any NativeLlmPort to the dyn-compatible LlmPort.

Required Methods

fn complete( &self, req: LlmRequest, ) -> impl Future<Output = Result<LlmResponse, LlmError>> + Send

Run a non-streaming inference call.

fn complete_stream( &self, req: LlmRequest, ) -> impl Future<Output = Result<LlmStream, LlmError>> + Send

Run a streaming inference call.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors