semantic-memory 0.5.0

# semantic-memory

Hybrid semantic search and durable local memory for Rust applications, backed by
SQLite, FTS5, vector embeddings, and an optional recoverable HNSW sidecar.

`semantic-memory` is built for agents and local-first applications that need more
than a key-value cache. It stores facts, chunked documents, conversation
messages, causal episodes, and imported projection rows in SQLite, then exposes
retrieval APIs that combine lexical search, vector similarity, recency, graph
traversal, integrity checks, and repair workflows.

## Contents

- [What it provides](#what-it-provides)
- [Current package status](#current-package-status)
- [Install](#install)
- [Quick start](#quick-start)
- [Opening a store](#opening-a-store)
- [Embedders](#embedders)
- [Facts](#facts)
- [Documents](#documents)
- [Conversations](#conversations)
- [Episodes](#episodes)
- [Search](#search)
- [Explainable ranking](#explainable-ranking)
- [Graph view](#graph-view)
- [Projection imports](#projection-imports)
- [Configuration](#configuration)
- [Feature flags](#feature-flags)
- [Storage layout](#storage-layout)
- [Integrity, repair, and re-embedding](#integrity-repair-and-re-embedding)
- [Concurrency model](#concurrency-model)
- [Operational notes](#operational-notes)
- [Examples](#examples)
- [Minimum supported Rust version](#minimum-supported-rust-version)
- [License](#license)

## What it provides

- Durable SQLite storage for facts, documents, chunks, sessions, messages,
  episodes, embeddings, projection imports, and sidecar journals.
- SQLite FTS5 lexical search over facts, document chunks, messages, and
  episodes.
- Vector search over f32 embeddings, with optional quantized q8 storage for
  compact derived vectors.
- Hybrid ranking using Reciprocal Rank Fusion (RRF), configurable BM25/vector
  weights, optional recency weighting, and source filtering.
- Optional HNSW approximate nearest-neighbor acceleration through `hnsw_rs`.
- Brute-force vector search mode for exact cosine similarity or simpler
  deployments.
- Explainable search through `search_explained()`, including BM25 rank, vector
  rank, exact/vector-rerank details, RRF contribution, and configured weights.
- Conversation memory with session CRUD, message insertion, token budgets, and
  conversation-specific search.
- Document ingestion that chunks text, embeds all chunks, and commits the
  document plus chunks atomically.
- Causal episode storage with outcomes, confidence, verification status, and
  searchable episode text.
- Deterministic graph traversal over namespaces, facts, documents, chunks,
  sessions, messages, episodes, and semantic/temporal/causal/entity edges.
- Canonical projection import support for verified cross-crate data from
  `forge-memory-bridge`, with scope-aware query surfaces for claims, relations,
  episodes, aliases, and evidence references.
- Integrity verification and repair workflows for FTS state, malformed JSON,
  invalid enum values, embedding blob shape, quantized vectors, and HNSW sidecar
  drift.
- Local-first operation: SQLite is the source of truth; HNSW is only a
  recoverable acceleration sidecar.

## Current package status

| Property | Value |
| --- | --- |
| Crate | `semantic-memory` |
| Current local version | `0.5.0` |
| Rust edition | 2021 |
| MSRV | Rust 1.75 |
| License | Apache-2.0 |
| Default vector backend | `hnsw` feature |
| Default embedder | Ollama `/api/embed` with `nomic-embed-text` |
| Default dimensions | 768 |
| Default storage directory | `memory/` |

## Install

Use the default HNSW-backed search path:

```toml
[dependencies]
semantic-memory = "0.5"
```

Use exact brute-force vector search without HNSW:

```toml
[dependencies]
semantic-memory = { version = "0.5", default-features = false, features = ["brute-force"] }
```

At least one backend feature must be enabled:

- `hnsw`
- `brute-force`

The default runtime embedder talks to Ollama. For the default configuration,
start Ollama and pull the default embedding model:

```bash
ollama pull nomic-embed-text
ollama serve
```

For tests or custom providers, use `MemoryStore::open_with_embedder()` with an
implementation of the `Embedder` trait.

## Quick start

```rust
use semantic_memory::{MemoryConfig, MemoryStore};

#[tokio::main]
async fn main() -> Result<(), semantic_memory::MemoryError> {
    let store = MemoryStore::open(MemoryConfig::default())?;

    store
        .add_fact(
            "general",
            "Rust was first released in 2015",
            Some("release-notes"),
            None,
        )
        .await?;

    let results = store
        .search("when was Rust released?", Some(5), None, None)
        .await?;

    for result in results {
        println!("{:.4} {}", result.score, result.content);
    }

    Ok(())
}
```

A no-network test setup can use the deterministic mock embedder:

```rust
use semantic_memory::{MemoryConfig, MemoryStore, MockEmbedder};

fn test_store() -> Result<MemoryStore, semantic_memory::MemoryError> {
    let config = MemoryConfig::default();
    MemoryStore::open_with_embedder(config, Box::new(MockEmbedder::new(768)))
}
```

## Opening a store

`MemoryStore::open(config)` creates the storage directory, opens SQLite, runs
migrations, validates embedding metadata, and initializes the configured vector
backend.

```rust
use semantic_memory::{MemoryConfig, MemoryStore};
use std::path::PathBuf;

let config = MemoryConfig {
    base_dir: PathBuf::from("./agent-memory"),
    ..Default::default()
};

let store = MemoryStore::open(config)?;
```

`MemoryStore` is cheap to clone. Clones share the same internal SQLite pool,
embedder, token counter, configuration, and HNSW sidecar state.

## Embedders

The public `Embedder` trait uses boxed futures so callers can provide custom
async embedding providers without adding a crate-level async trait dependency:

```rust
pub trait Embedder: Send + Sync {
    fn embed<'a>(&'a self, text: &'a str) -> EmbedFuture<'a>;
    fn embed_batch<'a>(&'a self, texts: Vec<String>) -> EmbedBatchFuture<'a>;
    fn model_name(&self) -> &str;
    fn dimensions(&self) -> usize;
}
```

Built-in embedders:

- `OllamaEmbedder`: production default, calls `POST /api/embed`, batches
  according to `EmbeddingConfig::batch_size`, validates HTTP status before
  parsing, and rejects non-numeric embedding values.
- `MockEmbedder`: deterministic, hash-seeded, normalized vectors for tests and
  offline development.

The embedder dimensions must match `config.embedding.dimensions`; store opening
fails with `MemoryError::DimensionMismatch` if they differ.

## Facts

Facts are short durable knowledge records grouped by namespace.

```rust
let fact_id = store
    .add_fact(
        "project-alpha",
        "The ingestion worker owns transcript normalization.",
        Some("architecture-notes"),
        None,
    )
    .await?;

let fact = store.get_fact(&fact_id).await?;
let facts = store.list_facts("project-alpha", 50, 0).await?;

store
    .update_fact(&fact_id, "The ingestion worker owns transcript normalization and chunk hints.")
    .await?;

store.delete_fact(&fact_id).await?;
```

Fact writes update SQLite, FTS, embeddings, optional quantized vectors, and HNSW
sidecar journal entries transactionally. SQLite remains authoritative even if a
sidecar flush is delayed.

## Documents

Documents are split into chunks, embedded in batches, and committed atomically.
If embedding any chunk fails, no document or chunk rows are written.

```rust
let doc_id = store
    .ingest_document(
        "Runbook",
        "Long text to chunk, embed, and search...",
        "ops",
        Some("/docs/runbook.md"),
        None,
    )
    .await?;

let docs = store.list_documents("ops", 20, 0).await?;
let chunk_count = store.count_chunks_for_document(&doc_id).await?;

store.delete_document(&doc_id).await?;
```

Chunking uses the configured `ChunkingConfig` and the configured `TokenCounter`.
The default token counter is an estimate (`text.len() / 4`, minimum 1 for
non-empty text). Applications can supply their own tokenizer by setting
`MemoryConfig::token_counter`.

## Conversations

Conversation memory is organized into sessions and messages.

```rust
use semantic_memory::Role;

let session_id = store.create_session("repl").await?;

store
    .add_message(
        &session_id,
        Role::User,
        "What should we do next?",
        None,
        None,
    )
    .await?;

store
    .add_message_embedded(
        &session_id,
        Role::Assistant,
        "Prioritize the SQLite repair path before UI work.",
        None,
        None,
    )
    .await?;

let recent = store.get_recent_messages(&session_id, 16).await?;
let budgeted = store.get_messages_within_budget(&session_id, 2_000).await?;
let total_tokens = store.session_token_count(&session_id).await?;
let hits = store.search_conversations("SQLite repair", Some(5), None).await?;
```

Message insertion can be plain, FTS-only, or embedded:

- `add_message()`: stores the message and updates the session timestamp.
- `add_message_fts()`: stores the message and indexes it for FTS.
- `add_message_embedded()`: stores the message, FTS entry, embedding, q8 vector,
  and sidecar journal entry.

When `token_count` is `None`, the facade computes a token estimate with the
configured token counter before writing.

## Episodes

Episodes represent causal or outcome-oriented records attached to memory state.
They include cause IDs, effect type, outcome, confidence, verification status,
optional experiment ID, and searchable episode text.

```rust
use semantic_memory::{EpisodeMeta, EpisodeOutcome, VerificationStatus};

let meta = EpisodeMeta {
    cause_ids: vec!["fact:abc".to_string()],
    effect_type: "regression_fix".to_string(),
    outcome: EpisodeOutcome::Confirmed,
    confidence: 0.92,
    verification_status: VerificationStatus::Verified {
        method: "integration-test".to_string(),
        at: "2026-05-07T00:00:00Z".to_string(),
    },
    experiment_id: Some("run-42".to_string()),
};

let doc_id = store
    .ingest_document(
        "Regression note",
        "The fix held under the regression suite.",
        "engineering",
        None,
        None,
    )
    .await?;

let episode_id = store
    .create_episode("episode-1", &doc_id, &meta)
    .await?;

let episodes = store
    .search_episodes(Some("regression_fix"), Some(&EpisodeOutcome::Confirmed), 5)
    .await?;
```

Episode search participates in the same FTS/vector machinery as other search
sources when `SearchSourceType::Episodes` is included.

## Search

The main search APIs are:

| Method | Embedding required | Searches | Use when |
| --- | --- | --- | --- |
| `search()` | Yes | Facts, chunks, episodes by default | General hybrid retrieval |
| `search_explained()` | Yes | Same as `search()` | You need ranking diagnostics |
| `search_fts_only()` | No | FTS-backed sources | Ollama is offline or lexical search is enough |
| `search_vector_only()` | Yes | Embedded sources | Pure semantic similarity is desired |
| `search_conversations()` | Yes | Messages | Conversation recall |
| `search_episodes()` | No | Episode metadata | Filtered causal/outcome inspection |

Search supports namespace filtering and source-type filtering:

```rust
use semantic_memory::SearchSourceType;

let namespaces = ["project-alpha", "project-beta"];
let sources = [SearchSourceType::Facts, SearchSourceType::Chunks];

let results = store
    .search("deployment rollback checklist", Some(10), Some(&namespaces), Some(&sources))
    .await?;
```

FTS query strings are sanitized before they reach SQLite FTS5. Empty sanitized
queries simply skip FTS instead of issuing an invalid FTS query.

## Explainable ranking

`search_explained()` returns `ExplainedResult`, which contains the normal
`SearchResult` plus `ScoreBreakdown`.

Useful fields include:

- `rrf_score`: final fused score.
- `bm25_score`: raw SQLite FTS5 BM25 score when present.
- `vector_score`: vector similarity used for final vector ordering.
- `recency_score`: optional recency contribution.
- `bm25_rank` and `vector_rank`: 1-based ranks before fusion.
- `bm25_contribution` and `vector_contribution`: weighted RRF components.
- `vector_source_rank` and `vector_source_score`: source vector retrieval data.
- `vector_reranked_from_f32`: whether HNSW hits were reranked with exact f32
  cosine similarity from SQLite.

This is useful for tuning `SearchConfig`, explaining agent recall, debugging
stale embeddings, and verifying that source filters are doing what you expect.

## Graph view

`store.graph_view()` exposes a deterministic graph traversal API derived from
SQLite state:

```rust
use semantic_memory::GraphDirection;

let graph = store.graph_view();

let edges = graph.neighbors("namespace:project-alpha", GraphDirection::Outgoing, 2)?;
let path = graph.path("fact:abc", "episode:def", 4)?;
```

Node IDs follow stable string forms such as:

- `namespace:<namespace>`
- `fact:<fact-id>`
- `document:<document-id>`
- `chunk:<chunk-id>`
- `session:<session-id>`
- `message:<message-id>`
- `episode:<episode-id>`

Edge families include semantic, temporal, causal, and entity relationships.

## Projection imports

The crate owns queryable projected truth for imported batches. Upstream crates
own source truth and transformation:

- `semantic-memory-forge`: evidence bundles and source export truth.
- `forge-memory-bridge`: transformation into projection import batches.
- `semantic-memory`: atomic import, authoritative importer `recorded_at`, query
  storage, scope filtering, and temporal filtering.

Canonical import path:

```rust
let receipt = store.import_projection_batch(batch).await?;
```

Supported projection read APIs:

- `query_projection_imports()`
- `query_projection_import_failures()`
- `latest_rebuildable_kernel_projection_import_for_scope()`
- `query_claim_versions()`
- `query_relation_versions()`
- `query_episodes()`
- `query_entity_aliases()`
- `query_evidence_refs()`

`ProjectionQuery` carries the common filter surface:

- `scope`
- optional free-text query
- optional valid-time filter
- optional transaction-time cutoff
- optional subject/canonical entity filters
- optional claim filters
- result limit

Legacy V10 import surfaces are retained only under hidden, deprecated
compatibility modules. New integrations should use the canonical projection
batch import path.

## Configuration

`MemoryConfig` is the top-level configuration:

```rust
use semantic_memory::{
    ChunkingConfig, EmbeddingConfig, MemoryConfig, MemoryLimits, PoolConfig, SearchConfig,
};
use std::path::PathBuf;
use std::time::Duration;

let config = MemoryConfig {
    base_dir: PathBuf::from("./memory"),
    embedding: EmbeddingConfig {
        ollama_url: "http://localhost:11434".to_string(),
        model: "nomic-embed-text".to_string(),
        dimensions: 768,
        batch_size: 32,
        timeout_secs: 30,
    },
    search: SearchConfig {
        default_top_k: 8,
        candidate_pool_size: 80,
        bm25_weight: 1.0,
        vector_weight: 1.0,
        rrf_k: 60.0,
        min_similarity: 0.3,
        recency_half_life_days: Some(30.0),
        recency_weight: 0.5,
        rerank_from_f32: true,
    },
    chunking: ChunkingConfig {
        target_size: 1000,
        min_size: 100,
        max_size: 2000,
        overlap: 200,
    },
    pool: PoolConfig {
        busy_timeout_ms: 5000,
        wal_autocheckpoint: 1000,
        enable_wal: true,
        max_read_connections: 4,
        reader_timeout_secs: 30,
    },
    limits: MemoryLimits {
        max_facts_per_namespace: 100_000,
        max_chunks_per_document: 1_000,
        max_content_bytes: 1_048_576,
        max_embedding_concurrency: 8,
        max_db_size_bytes: 0,
        embedding_timeout: Duration::from_secs(30),
    },
    token_counter: None,
    ..Default::default()
};
```

Configuration is normalized and validated on open. Examples of normalization:

- `embedding.batch_size == 0` becomes 1.
- `embedding.timeout_secs == 0` becomes 1.
- `search.candidate_pool_size` is raised to at least `default_top_k`.
- `chunking.target_size` is clamped into `[min_size, max_size]`.
- `chunking.overlap` is kept below `min_size`.
- `limits.max_embedding_concurrency` is hard-capped at 32.
- `pool.reader_timeout_secs` is capped at 300 seconds.

## Feature flags

| Feature | Default | Description |
| --- | --- | --- |
| `hnsw` | Yes | Enables the `hnsw_rs` sidecar for approximate nearest-neighbor retrieval. |
| `brute-force` | No | Enables exact brute-force vector retrieval. |
| `testing` | No | Exposes test-only helpers such as `raw_execute()`. |

Compile-time rule: at least one of `hnsw` or `brute-force` must be enabled.

Common build modes:

```bash
cargo test
cargo test --no-default-features --features brute-force
cargo test --all-features
```

## Storage layout

Given `base_dir = "./memory"`, the crate creates and uses paths under that
directory for SQLite and, when enabled, HNSW sidecar files.

SQLite contains the durable source of truth:

- `sessions`, `messages`, `messages_fts`, `messages_rowid_map`
- `facts`, `facts_fts`, `facts_rowid_map`
- `documents`, `chunks`, `chunks_fts`, `chunks_rowid_map`
- `episodes`, `episodes_fts`, `episodes_rowid_map`
- `embedding_metadata`
- `hnsw_metadata`, `hnsw_keymap`, `pending_index_ops`
- projection import log, failures, claim versions, relation versions, entity
  aliases, evidence refs, and supporting projection tables

SQLite pragmas are configured for local concurrent use:

- WAL journal mode
- foreign keys enabled
- busy timeout
- normal synchronous mode

## Integrity, repair, and re-embedding

Run verification:

```rust
use semantic_memory::{VerifyMode};

let report = store.verify_integrity(VerifyMode::Full).await?;
if !report.ok {
    for issue in report.issues {
        eprintln!("{issue}");
    }
}
```

Repair FTS state:

```rust
use semantic_memory::ReconcileAction;

let report = store.reconcile(ReconcileAction::RebuildFts).await?;
```

Re-embed all embedded records after changing models or dimensions:

```rust
let dirty = store.embeddings_are_dirty().await?;
if dirty {
    let updated_rows = store.reembed_all().await?;
    println!("re-embedded {updated_rows} rows");
}
```

`reembed_all()` covers facts, chunks, messages, and episodes. On completion it
clears the dirty flag and rebuilds HNSW when the `hnsw` feature is enabled.

HNSW maintenance APIs:

```rust
store.flush_hnsw()?;
store.rebuild_hnsw_index().await?;
store.compact_hnsw().await?;
```

## Concurrency model

The store uses:

- one writer connection for serialized SQLite writes,
- a configurable pool of reader connections,
- WAL mode for concurrent reads,
- `spawn_blocking` for SQLite work so database I/O does not block the async
  executor,
- a semaphore around embedding calls to cap concurrent embedding requests,
- SQLite-journaled HNSW sidecar mutations so committed writes can be replayed
  after crashes or sidecar flush failures.

All public methods that touch SQLite route through the facade's connection
helpers.

## Operational notes

- SQLite is authoritative. HNSW can be rebuilt from SQLite.
- HNSW files are an acceleration sidecar, not the durable truth store.
- If the embedding model or dimensions change, the store marks embeddings dirty.
  Search still works, but quality is degraded until `reembed_all()` completes.
- `search_fts_only()` does not require Ollama or another embedding provider.
- `search()` and vector APIs require an embedder to be available.
- Document ingestion is all-or-nothing for a document and its chunks.
- Fact, document, and message FTS updates are transactional with their source
  rows.
- Evidence payloads imported through projection paths remain opaque in normal
  retrieval; explicit query/audit paths can access evidence references.
- Compatibility import APIs are deprecated and hidden. They are retained for
  migration windows only.

## Examples

Run the basic search example with Ollama:

```bash
ollama pull nomic-embed-text
cargo run --example basic_search
```

Run the conversation example:

```bash
cargo run --example conversation_memory
```

Run tests:

```bash
cargo test
```

Run tests with the exact brute-force backend:

```bash
cargo test --no-default-features --features brute-force
```

## Minimum supported Rust version

`semantic-memory` declares `rust-version = "1.75"` and uses Rust 2021.

## License

Apache-2.0. See [LICENSE](LICENSE).