Skip to main content

LlmInferenceEngine

ferrum_engine

Trait LlmInferenceEngine 

Source 
pub trait LlmInferenceEngine: InferenceEngine {
    // Required methods
    fn infer<'life0, 'async_trait>(
        &'life0 self,
        request: InferenceRequest,
    ) -> Pin<Box<dyn Future<Output = Result<InferenceResponse, FerrumError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait;
    fn infer_stream<'life0, 'async_trait>(
        &'life0 self,
        request: InferenceRequest,
    ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamChunk, FerrumError>> + Send>>, FerrumError>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait;

    // Provided method
    fn context_capacity(&self) -> Option<usize> { ... }
}
Expand description

LLM text-generation engine.

Implemented by ContinuousBatchEngine (the production path) and DefaultInferenceEngine (legacy reference path). Backs /v1/chat/completions and /v1/completions.

Required Methods§

Source

fn infer<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<InferenceResponse, FerrumError>> + Send + 'async_trait>>
where 'life0: 'async_trait, Self: 'async_trait,

Execute single inference request.

Source

fn infer_stream<'life0, 'async_trait>( &'life0 self, request: InferenceRequest, ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamChunk, FerrumError>> + Send>>, FerrumError>> + Send + 'async_trait>>
where 'life0: 'async_trait, Self: 'async_trait,

Execute streaming inference request.

Provided Methods§

Source

fn context_capacity(&self) -> Option<usize>

Effective per-request capacity in tokens, including input and output. Implementations must report the limit used by request admission, not the model weights’ nominal context window. None means unreported.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§