Trait LlmInferenceEngine 

pub trait LlmInferenceEngine: InferenceEngine {
    // Required methods
    fn infer<'life0, 'async_trait>(
        &'life0 self,
        request: InferenceRequest,
    ) -> Pin<Box<dyn Future<Output = Result<InferenceResponse>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;
    fn infer_stream<'life0, 'async_trait>(
        &'life0 self,
        request: InferenceRequest,
    ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamChunk>> + Send>>>> + Send + 'async_trait>>
       where Self: 'async_trait,
             'life0: 'async_trait;

    // Provided method
    fn context_capacity(&self) -> Option<usize> { ... }
}

LLM text-generation engine.

Implemented by ContinuousBatchEngine (the production path) and DefaultInferenceEngine (legacy reference path). Backs /v1/chat/completions and /v1/completions.

Required Methods

fn infer<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<InferenceResponse>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Execute single inference request.

fn infer_stream<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamChunk>> + Send>>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Execute streaming inference request.

Provided Methods

fn context_capacity(&self) -> Option<usize>

Effective per-request capacity in tokens, including input and output. Implementations must report the limit used by request admission, not the model weights’ nominal context window. None means unreported.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors