Struct CacheClient 

pub struct CacheClient<C> { /* private fields */ }

An LLMClient wrapper that caches complete responses in a KvStore.

On complete, the cache is checked first; a fresh (non-expired) hit returns the stored response without calling the inner client. On a miss the inner client is called and a successful response is stored. Cache read/write errors are non-fatal — a failed read falls through to the inner client, a failed write is dropped, and the response is still returned.

stream_complete is always delegated to the inner client (never cached).

Implementations

impl<C> CacheClient<C>

pub fn new(inner: C, store: Arc<dyn KvStore>) -> Self

Wrap inner with a response cache backed by store (no expiry).

pub fn with_ttl(inner: C, store: Arc<dyn KvStore>, ttl: Duration) -> Self

Wrap inner with a response cache whose entries expire after ttl.

Expired entries are lazily evicted on read (treated as a miss). A TTL bounds staleness; to bound total size, wrap the KvStore with an evicting implementation — CacheClient does not enforce a size cap.

pub fn inner(&self) -> &C

Access the underlying (uncached) client.

Trait Implementations

impl<C: LLMClient> LLMClient for CacheClient<C>

fn complete<'life0, 'async_trait>( &'life0 self, request: LLMRequest, ) -> Pin<Box<dyn Future<Output = Result<LLMResponse>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Send a chat completion request and return the full response.

fn model_name(&self) -> &str

Return the model name this client is configured to use.

fn stream_complete<'life0, 'async_trait>( &'life0 self, request: LLMRequest, ) -> Pin<Box<dyn Future<Output = Result<LLMStream>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Stream a chat completion, yielding events as they arrive.