Struct SemanticCache 

pub struct SemanticCache { /* private fields */ }

Semantic cache using TF-IDF embeddings and cosine similarity.

The cache embeds every incoming prompt with a refittable TF-IDF model and performs a brute-force cosine search over stored entries. When a result above SemanticCacheConfig::similarity_threshold is found and has not expired, the stored response is returned without running inference.

Thread-safety: all fields are guarded by Mutex. The cache is Send + Sync and can be shared across threads via Arc<SemanticCache>.

Implementations

impl SemanticCache

pub fn new(config: SemanticCacheConfig) -> Self

Create a new SemanticCache with the given configuration.

The TF-IDF embedder is bootstrapped with synthetic vocabulary so that lookup calls before any insert return gracefully.

pub fn lookup(&self, prompt: &str) -> Option<CachedResponse>

Check whether a semantically similar response is cached.

Returns None on a miss, or when the best-matching entry has expired. On a hit, the entry’s hit_count and the global access clock are updated.

pub fn insert(&self, prompt: &str, response: &str)

Store a new prompt→response mapping in the cache.

If the cache is at capacity, the least-recently-used entry is evicted. The TF-IDF embedder is refitted periodically as new prompts accumulate.

pub fn evict_expired(&self) -> usize

Remove all expired entries from the cache.

Returns the number of entries that were removed.

pub fn clear(&self)

Remove all entries and reset statistics.

pub fn len(&self) -> usize

Current number of entries in the cache.

pub fn is_empty(&self) -> bool

Returns true if the cache contains no entries.

pub fn stats(&self) -> SemanticCacheStats

Snapshot of current cache statistics.