Expand description
L2 semantic cache: prompts embedded with bge-small-en-v1.5 and matched
per-tenant by cosine similarity against a threshold.
The index is a flat in-memory scan. At the expected <10k entries per tenant
a full 384-dim pass costs ~3ms; HNSW (instant-distance) only pays off
past ~50k.
Structsยง
- L2Cache
- Semantic L2 cache. Constructing this downloads the bge model on first run
(cached by fastembed under
~/.cache/fastembed). - Semantic
Stats