Struct CachedLLM 

pub struct CachedLLM { /* private fields */ }

Wraps an LLM and answers repeated requests from an LLMCache instead of calling the model again.

The cache key is the namespace, the messages and the call options set through add_options (temperature, stop words, tools, …). The wrapper can’t see settings inside the model itself, such as the model name, so give each model configuration its own namespace when they share a cache. By default the namespace is the wrapped type’s name.

• A hit returns the stored result with tokens: None: it cost nothing. If a streaming_func is set, it gets the whole text at once.
• Errors are never cached.
• generate_stream caches the final result of a streamed response and replays hits as one Text event plus Done. The legacy stream replays hits but doesn’t store misses.
• Concurrent identical misses each call the model.

let cache = Arc::new(InMemoryLLMCache::new().with_ttl(Duration::from_secs(3600)));
let llm = CachedLLM::new(OpenAI::default(), cache).with_namespace("gpt-4o-mini");
let first = llm.invoke("Say hi").await?; // calls OpenAI
let again = llm.invoke("Say hi").await?; // from the cache

Implementations

impl CachedLLM

pub fn new<L: LLM + 'static>(llm: L, cache: Arc<dyn LLMCache>) -> Self

pub fn with_namespace<S: Into<String>>(self, namespace: S) -> Self

Separates this model’s entries from others in a shared cache, e.g. the model name.

pub fn cache_key(&self, messages: &[Message]) -> Result<String, LLMError>

The key a request is cached under.

Trait Implementations

impl Clone for CachedLLM

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl LLM for CachedLLM

fn generate<'life0, 'life1, 'async_trait>( &'life0 self, messages: &'life1 [Message], ) -> Pin<Box<dyn Future<Output = Result<GenerateResult, LLMError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

fn stream<'life0, 'life1, 'async_trait>( &'life0 self, messages: &'life1 [Message], ) -> Pin<Box<dyn Future<Output = Result<Pin<Box<dyn Stream<Item = Result<StreamData, LLMError>> + Send>>, LLMError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

fn generate_stream<'a>(&'a self, messages: &'a [Message]) -> LLMEventStream<'a>

Streams the response as LLMEvents: Text chunks while the model writes, then one Done with the complete GenerateResult (including typed tool_calls). Errors, including the initial request failing, come as stream items.

fn add_options(&mut self, options: CallOptions)

This is usefull when you want to create a chain and override LLM options

fn messages_to_string(&self, messages: &[Message]) -> String

fn invoke<'life0, 'life1, 'async_trait>( &'life0 self, prompt: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<String, LLMError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,