Module cache 

Response caching decorator

Response Caching

CacheProvider wraps an inner Box<dyn LlmProvider> and caches responses for identical requests. Cache keys are computed from the serialized request parameters (messages, model, temperature, max_tokens).

Behavior

• Only complete() results are cached; complete_stream() always delegates.
• Requests with temperature > Some(0.0) bypass the cache by default (configurable via cache_nonzero_temperature).
• Entries expire after ttl and are evicted on access.
• When at capacity, the oldest entry is evicted to make room.

Structs

CacheConfig

Configuration for the response cache

CacheProvider

Caching decorator for any LlmProvider.

CacheStats

Cache usage statistics