Module cache 

Response caching infrastructure for LLM API calls.

This module provides a flexible caching system to reduce API costs and improve response times by caching identical requests.

Example

use llmkit::{CacheConfig, CachingProvider, InMemoryCache, OpenAIProvider};

// Create a caching provider
let inner = OpenAIProvider::from_env()?;
let cache = InMemoryCache::new(CacheConfig::default());
let provider = CachingProvider::new(inner, cache);

// First request hits the API
let response1 = provider.complete(request.clone()).await?;

// Second identical request hits the cache
let response2 = provider.complete(request).await?;

Cache Key Computation

Cache keys are computed from:

• Model name
• Messages content
• Tools (if any)
• System prompt

By default, non-deterministic parameters (temperature, top_p) are excluded from the cache key to allow caching regardless of sampling settings.

Structs

CacheConfig

Configuration for the caching system.

CacheKeyBuilder

Cache key builder for custom cache key computation.

CacheStats

Statistics about cache performance.

CachedResponse

A cached response with metadata.

CachingProvider

A provider wrapper that caches responses.

InMemoryCache

In-memory cache backend using DashMap.

Traits

CacheBackend

Trait for cache backends.