Module embedding_index 

Embedding index: mmap-backed pre-computed embedding vectors.

Sparse format — only blocks that actually received an embedding are stored (depth <= max_depth, non-trivial text, non-zero vector). Previously every block held a full vector with a NaN sentinel for the ~99% un-embedded ones, which blew the file up to gigabytes for no benefit. Search now scans only the embedded subset.

Layout: [u32 embedded_count][u32 dim][u32 max_depth] [u32 block_idx × embedded_count] (ascending, for binary search) [f32 × dim × embedded_count]

embedding(block_idx) returns None for blocks without a stored vector.

Structs

AppendEmbeddings

EmbeddingIndex

Mmap-backed embedding index for fast semantic lookup.

Enums

EmbedVerdict

Why a text is not worth embedding. Deliberately about what the text is, never about how it scores: a cosine-threshold dedup would merge contradictory facts – of 40,000 sampled high-cosine pairs, 1,666 differ in numbers and 245 differ in negation.

Constants

APPEND_EMBEDDINGS_FILE

Sidecar holding vectors for memories that are still in the append log.

Functions

build_embedding_index

Build a sparse embedding index file from a provider and reader. Only blocks at depth 0..=max_depth with non-trivial text get embedded; failed or zero embeddings are omitted (search treats them as absent).

min_embed_chars

Minimum text length, in characters, for a block to be embedded. Overridable with MICROSCOPE_MIN_EMBED_CHARS.

open_append_cached

AppendEmbeddings::open through the process-wide cache.

open_embedding_cached

EmbeddingIndex::open through the process-wide cache.

quality_gate

The embedding quality gate, shared by the index build and the store path so a freshly stored memory is admitted under exactly the policy that built the index – a second, looser admission rule would re-introduce what this removes.

similarity_floor

The cosine below which a stored vector is not offered as a candidate.