StreamingProvider

ferrous_llm

Trait StreamingProvider 

Source 
pub trait StreamingProvider: ChatProvider {
    type StreamItem: Send + 'static;
    type Stream: Stream<Item = Result<Self::StreamItem, Self::Error>> + Send + 'static;

    // Required method
    fn chat_stream<'life0, 'async_trait>(
        &'life0 self,
        request: ChatRequest,
    ) -> Pin<Box<dyn Future<Output = Result<Self::Stream, Self::Error>> + Send + 'async_trait>>
       where 'life0: 'async_trait,
             Self: 'async_trait;
}
Expand description

Optional trait for providers that support streaming responses.

This extends ChatProvider to add streaming capabilities.

Required Associated Types§

Source

type StreamItem: Send + 'static

Stream item type for incremental responses

Source

type Stream: Stream<Item = Result<Self::StreamItem, Self::Error>> + Send + 'static

Stream type for the response

Required Methods§

Source

fn chat_stream<'life0, 'async_trait>( &'life0 self, request: ChatRequest, ) -> Pin<Box<dyn Future<Output = Result<Self::Stream, Self::Error>> + Send + 'async_trait>>
where 'life0: 'async_trait, Self: 'async_trait,

Send a chat request and receive a streaming response.

§Arguments
  • request - The chat request containing messages and parameters
§Returns

A result containing a stream of response chunks or an error

Implementors§