Struct InMemoryLLMCache 

pub struct InMemoryLLMCache { /* private fields */ }

An in-process LLMCache. Unbounded by default; with_capacity evicts the least recently used response and with_ttl expires responses after a while.

Implementations

impl InMemoryLLMCache

pub fn new() -> Self

pub fn with_capacity(self, capacity: usize) -> Self

Keeps at most capacity responses, evicting the least recently used one.

pub fn with_ttl(self, ttl: Duration) -> Self

Forgets responses older than ttl.

pub fn len(&self) -> usize

Number of stored responses (expired ones included until they are looked up or evicted).

pub fn is_empty(&self) -> bool

pub fn clear(&self)

Trait Implementations

impl Default for InMemoryLLMCache

fn default() -> Self

Returns the “default value” for a type.

impl LLMCache for InMemoryLLMCache

fn get<'life0, 'life1, 'async_trait>( &'life0 self, key: &'life1 str, ) -> Pin<Box<dyn Future<Output = Option<GenerateResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

The cached response for key, if any.

fn put<'life0, 'life1, 'async_trait>( &'life0 self, key: &'life1 str, value: GenerateResult, ) -> Pin<Box<dyn Future<Output = ()> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Stores a response.