semantic-memory

Local-first hybrid retrieval for Rust, with SQLite as authoritative state and receipts for execution evidence.

semantic-memory stores facts, documents and chunks, conversations, episodes, embeddings, temporal state, authority ledgers, and search receipts in SQLite. FTS indexes, vector sidecars, sparse representations, and compressed candidate artifacts accelerate retrieval; they do not replace canonical state and can be reconciled from SQLite.

Status: research-grade library with a tested default retrieval contract. Feature-gated research and orchestration modules are not implicit guarantees of MemoryStore::search() behavior.

What the crate owns

What the crate does not own

• Claim truth. A search receipt records retrieval execution; it is not a claim-ledger verification decision.
• Agent action permission. Recall authority does not grant permission to assert a memory or act on it.
• MCP transport policy. See semantic-memory-mcp for transport, tool profiles, and application-level authority composition.
• Automatic activation of every feature. Compiling routing, graph, topology, community, decoder, or compression modules does not make the normal search path invoke them.
• Native SPLADE or native ColBERT. Dense-derived sparse values and the current late-interaction proxy must not be presented as those systems.

Install

[dependencies]
semantic-memory = "0.5.10"
tokio = { version = "1", features = ["macros", "rt"] }

The default build enables usearch-backend. For an exact pure-Rust backend without the C++ bridge:

[dependencies]
semantic-memory = { version = "0.5.10", default-features = false, features = ["brute-force"] }

Quick start

This example uses the deterministic MockEmbedder, so it needs no network service or downloaded model.

use semantic_memory::{EmbeddingConfig, MemoryConfig, MemoryStore, MockEmbedder};
use std::path::PathBuf;

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), semantic_memory::MemoryError> {
    let config = MemoryConfig {
        base_dir: PathBuf::from("memory-example"),
        embedding: EmbeddingConfig {
            dimensions: 768,
            ..Default::default()
        },
        ..Default::default()
    };

    let store = MemoryStore::open_with_embedder(
        config,
        Box::new(MockEmbedder::new(768)),
    )?;

    store
        .add_fact("general", "Rust was first released in 2015", None, None)
        .await?;

    let results = store
        .search(
            "when was Rust released",
            Some(5),
            Some(&["general"]),
            None,
        )
        .await?;

    for result in results {
        println!("{:.4} {}", result.score, result.content);
    }

    Ok(())
}

MemoryStore::open() selects OllamaEmbedder unless the crate is compiled with candle-embedder, in which case it selects CandleEmbedder. Use open_with_embedder to inject another provider or a deterministic fixture.

Choose the retrieval API deliberately

Messages are not part of the normal hybrid source set unless selected with SearchSourceType::Messages. Conversation search is available separately.

Default hybrid retrieval contract

The baseline lanes are SQLite FTS5/BM25 and dense-vector retrieval. Their ranked lists are fused with weighted Reciprocal Rank Fusion (RRF). SearchConfig controls lane weights, RRF constant, candidate pool size, minimum similarity, result limits, recency behavior, and optional stages.

Dense retrieval and exactness

The active backend is a runtime fact, not a README promise. When receipts are enabled, candidate_backend, fallback, exact_rerank, candidate counts, and degradations report what actually happened.

• DerivedVectorBackendPolicy::Disabled selects the non-derived vector policy. With hnsw compiled, ordinary search can use HNSW candidates; without that candidate path it uses authoritative SQLite-backed f32 scoring.
• ExactnessProfile::PreferExact requests the exact reference path.
• HNSW can supply candidates when the hnsw feature is compiled and exactness does not require the reference path.
• TurboQuant and proveKV/poly-kv policies are candidate-only. Their final ranking requires authoritative f32 reranking.
• The default usearch-backend feature supplies the public VectorIndex implementation; enabling it does not by itself make ordinary MemoryStore::search() approximate.
• The brute-force feature satisfies the crate's backend feature guard for ordinary exact search; it is not a separate implementation of VectorIndex::new().

Optional lanes and stages

Cache boundary

Result caching is intentionally narrow. Only ordinary, unfiltered, current-state searches with default exactness, no receipt, no replay, no request/trace/budget identity, and no recency half-life are cache eligible. Historical, explained, exact, replay-bearing, receipt-bearing, filtered, or governed requests execute independently. Cache entries are also bound to the authority retrieval epoch.

Durability, recovery, and replay privacy

The store uses SQLite WAL with pooled readers and a serialized writer. Under the hnsw feature, pending sidecar operations are journaled and replayed so a committed SQLite write remains authoritative while HNSW synchronization is pending. The default standalone usearch VectorIndex does not share that MemoryStore HNSW lifecycle.

• verify_integrity(VerifyMode::Quick | VerifyMode::Full) reports malformed or drifting state.
• reconcile(ReconcileAction::ReportOnly | ReconcileAction::RebuildFts | ReconcileAction::ReEmbed) reports or rebuilds derived state from SQLite.
• Sidecar and derived-artifact failures must not silently become canonical state.
• Search receipts receive canonical BLAKE3 digests when persisted.

Receipts and replay

use semantic_memory::{MemoryStore, ReceiptMode, ReplayMode, SearchContext};

async fn replayable_search(store: &MemoryStore) -> Result<(), semantic_memory::MemoryError> {
    let context = SearchContext {
        receipt_mode: ReceiptMode::ReturnReceipt,
        replay_mode: ReplayMode::StoreInputs,
        ..SearchContext::default()
    };

    let response = store
        .search_with_context(
            "when was Rust released",
            Some(5),
            None,
            None,
            context,
        )
        .await?;

    if let Some(receipt) = response.receipt {
        let report = store
            .replay_search_from_stored_inputs(&receipt.receipt_id)
            .await?;
        println!("same result IDs: {}", report.result_ids_match);
    }

    Ok(())
}

ReplayMode::NoReplay is the default privacy boundary. Query text, namespace filters, and source-type filters are not retained; receipts keep their digests. ReplayMode::StoreInputs explicitly retains those inputs in the V35 replay_inputs table and enables replay_search_from_stored_inputs.

A receipt can establish that a named backend, exactness route, fallback, degradation set, and pre-visibility result identity were recorded. Receipt result IDs are assembled before later StateView and forgetting filters, so do not equate them universally with the final governed caller-visible list. A receipt does not establish that a result is true, safe to assert, or authorized for downstream action.

Temporal truth and governed mutation

Normal fact reads use StateView::Current, which excludes superseded facts. Historical fact reconstruction is explicit through StateView and bitemporal fields. Chunks and messages do not receive an equivalent authority-state filter in the normal hybrid result pass, while episodes and imported projections use their own validity/invalidation handling. If the fact store cannot establish one coherent current head, state-aware resolution fails closed rather than selecting an arbitrary version.

MemoryStore::authority() exposes a capability-gated surface for trusted integrations. Governed mutations require an authority permit issued inside the trust boundary; ordinary downstream code is not intended to mint its own authority. The surface covers:

• append and supersession;
• redaction and selective forgetting;
• governed direct reads, search, and graph traversal;
• export and replay;
• immutable origin-authority labels and revocations;
• transition verification, quarantine, and rollback references where declared by the operation.

The compatibility API remains available for ordinary local use. Raw compatibility reads and writes are not equivalent to governed reads: they do not apply the same origin, revocation, scope, purpose, and state decisions. Do not infer a typed transition receipt from a method unless its return contract declares one. Graph utility operations return domain objects and do not universally produce governance receipts.

Hard delete and in-place truth mutation are disabled by default. The admin-ops feature exposes administrative operations; use supersession for normal correction workflows.

Embedding identity and purpose

The committed embedding contract distinguishes EmbeddingPurpose::Query from EmbeddingPurpose::Document. Query and document text receive different role prefixes, and purpose/profile identity participates in embedding caches and durable dirty-metadata checks. Custom embedders, imported vectors, re-embedding workflows, and benchmark fixtures must preserve that asymmetry or explicitly disclose a different profile.

The SciFact fixture records its own supplied vectors and refuses unknown query text so evaluation cannot silently cross into another embedding provider.

Cargo features

At least one vector backend (usearch-backend, hnsw, or brute-force) must be enabled.

Storage, retrieval, and embedding

Governance and operations

Research and orchestration

A feature flag proves that code is compiled, not that a default search request exercised it. Use receipts, explained results, tests, or application-level traces for runtime claims.

Selected schema milestones, V18–V36

MAX_SCHEMA_VERSION is 36. Migrations are monotonic; compatibility columns can exist even when the corresponding feature-gated Rust API is disabled. This table is intentionally partial. Earlier schema versions remain in the migration ledger, and some versions use procedural Rust migrations rather than a non-empty SQL constant.

Projection-import compatibility APIs are migration surfaces. New integrations should use the current batch-import seam and preserve source export time, transformation time, importer commit time, scope, and temporal fields separately.

Evaluation

The BEIR SciFact evaluation guide runs the checked-in retrieval APIs in three frozen modes:

1. FTS-only;
2. vector-only with PreferExact;
3. baseline hybrid with explained results.

The repository includes the builder, runner, independent validator, and tests. It does not check in a frozen score artifact, so this README intentionally makes no numeric quality or latency claim. Publish measurements only with the raw per-query JSONL, aggregate receipt, corpus hashes, executable identity, configuration, split definition, and validator result from the same run.

The SciFact harness does not evaluate general-domain quality, graph retrieval, native sparse/SPLADE retrieval, token-level late interaction, Matryoshka quality, or model quality.

Verification

From the workspace root:

cargo fmt --package semantic-memory -- --check
cargo check -p semantic-memory --all-targets
cargo test -p semantic-memory --all-targets
cargo clippy -p semantic-memory --all-targets -- -D warnings

Backend-specific examples:

cargo test -p semantic-memory --no-default-features --features brute-force --all-targets
cargo test -p semantic-memory --no-default-features --features hnsw --all-targets --no-run

Examples

Public claim boundary

Defensible from current source and tests:

• local-first SQLite-backed authoritative state;
• rebuildable lexical/vector/sparse/derived projections;
• default hybrid retrieval with explained score components;
• explicit current/historical fact views;
• optional durable search receipts and opt-in replay inputs;
• capability-gated authority APIs and administrative-operation feature gates;
• experimental and feature-gated retrieval/governance modules.

Not established merely by this README or by compiling the crate:

• production certification or secure autonomous administration;
• universal truth verification;
• permission to act on recalled content;
• superiority over other retrieval systems;
• native SPLADE, native ColBERT, or universal lossless compression;
• performance outside a named, reproducible benchmark receipt.