Struct SemanticCache 

pub struct SemanticCache { /* private fields */ }

Cache that uses embedding similarity to match semantically equivalent queries.

When a cache lookup is performed, the key is embedded and compared against all stored entries using cosine similarity. If any entry exceeds the similarity threshold, its cached response is returned.

Implementations

impl SemanticCache

pub fn new(embeddings: Arc<dyn Embeddings>, similarity_threshold: f32) -> Self

Create a new semantic cache with the given embeddings provider and similarity threshold.

The threshold should be between 0.0 and 1.0. A typical value is 0.95, meaning only very similar queries will match.

Trait Implementations

impl LlmCache for SemanticCache

fn get<'life0, 'life1, 'async_trait>( &'life0 self, key: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<Option<ChatResponse>, SynapticError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Look up a cached response by cache key.

fn put<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, key: &'life1 str, response: &'life2 ChatResponse, ) -> Pin<Box<dyn Future<Output = Result<(), SynapticError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

Store a response in the cache.

fn clear<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Result<(), SynapticError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Clear all entries from the cache.