Expand description
Optional semantic-retrieval seam for durable-memory recall (“L1” of the
memory redesign): the MemoryEmbedder interface a backend implements, plus
the hybrid-score building blocks (cosine, HYBRID_COSINE_WEIGHT,
SEMANTIC_FLOOR) that lexical_bm25 combines as bm25 + β·cosine.
Embedding is OPTIONAL. With no embedder configured, recall stays pure
BM25 — this seam is entirely inert (query vectors are None, so the cosine
term drops out and scoring is byte-identical to L0). Only the interface and
the hybrid-scoring path land here; the concrete backend (a local multilingual
ONNX embedder, provider-independent — or an optional hosted embedding
endpoint) is a deferred follow-up that just implements this trait, populates
LexicalIndexItem.embedding at index-build time, and embeds the query at
recall time. BM25 stays the PRIMARY term so the #61 cache-stable ordering and
the CJK/lexical guarantees survive; the vector term only re-ranks within that
and surfaces paraphrase matches lexical scoring would miss.
Constants§
- HYBRID_
COSINE_ WEIGHT - Weight (β) on the cosine term in the hybrid score
bm25 + β·cosine. - SEMANTIC_
FLOOR - Minimum cosine for a NON-lexical (pure-semantic) match to be recalled — stops the vector term from surfacing every embedded doc as a weak match. A doc still recalls on lexical BM25 alone below this; the floor only gates cosine-only hits.
Traits§
- Memory
Embedder - A text embedder. Backends SHOULD return a unit-normalized vector; an empty vec means “no embedding available” (recall falls back to pure BM25 for that item/query). Must be cheap enough to call once per query at recall time (document vectors are precomputed at write/index time).
Functions§
- cosine
- Cosine similarity in [-1, 1]. Returns 0 for empty, mismatched-length, degenerate (zero-norm), or non-finite (NaN/inf) vectors, so a missing/absent or malformed embedding is a no-op that can never leak NaN/inf into a score.