This directory is the Memory Nemo Rust crate: encrypted single-file storage, the mnemo CLI, examples, and tests.

The repository overview and links to the landing page and Python bindings are in the root README.

Quick start

cargo run --example quickstart
cargo run --bin mnemo -- demo

For AI agents

The CLI is designed so an agent can become productive against a .mnemo file in two commands, with no external documentation:

# Handed an existing file? The file introduces itself.
mnemo about path/to/agent.mnemo

# Starting fresh? `init` auto-inserts a scaffold manifest so the new file
# is self-describing from creation. Replace the scaffold once you know your
# embedder, agent_id convention, and project metadata.
mnemo init my-agent.mnemo --dimensions 768
mnemo about my-agent.mnemo

Working on this codebase (not just using mnemo)? Start at ../AGENTS.md — repo layout, build/test commands, conventions, and the dogfood workflow.

Serve as an MCP server

mnemo can expose a .mnemo file as a Model Context Protocol server over stdio, so any MCP-compatible agent (Claude Desktop, Cursor, custom MCP clients) can drive it as a tool:

mnemo serve --mcp path/to/agent.mnemo

The passphrase is read from MNEMO_PASSPHRASE (never a CLI flag — that would leak into shell history and process listings). Tools exposed: about, remember, recall, forget, list, snapshot_list, stats, cache_put, cache_get, cache_stats, cache_put_semantic, cache_get_semantic. Every mutation flushes before returning.

Text-in via `--embed-cmd` (Phase 3)

remember, recall, cache_put_semantic, and cache_get_semantic accept either a caller-supplied vector/query or a text string that the server embeds for them. Enable the text path by passing an embedder command at serve start:

MNEMO_PASSPHRASE=... mnemo serve --mcp agent.mnemo \
  --embed-cmd "python3 mnemo/examples/embedder_server.py" \
  --embed-name "sentence-transformers/all-MiniLM-L6-v2"

The embedder is a persistent child process spawned once at serve start. Protocol: it reads one JSON-encoded string per line from stdin and writes one JSON array of floats per line to stdout, flushed after every reply. See mnemo/examples/embedder_server.py for a 10-line reference implementation (sentence-transformers + MiniLM, 384-dim).

Stdout is for replies only. Any line the embedder prints to stdout outside a reply — startup banners, model-load progress, warnings, "model loaded" messages — is a protocol violation and would shift every subsequent reply by one line. EmbedCmd detects it, kills the child, and surfaces a clear error ("stdout is reserved for replies only; log to stderr instead"); the next call respawns the child. Log to stderr. Python: print("...", file=sys.stderr, flush=True), or configure logging with stream=sys.stderr. Stderr is inherited to the parent and is safe for anything you want to see while the server runs.

Per-request timeout defaults to 60 seconds; override with --embed-timeout <secs> on serve, remember, and recall. Set it longer than your slowest cold-start reply (sentence-transformers' first encode() on CPU can be 3–5s). On timeout — or any other protocol error (malformed JSON, wrong dimensionality, zero vector, unexpected EOF) — the child is killed and the next call respawns a fresh child with the same argv/env/timeout. The guarantee is asymmetric and deliberate: an error is allowed; a mismatched vector never is.

The --embed-name string is matched against (and persisted on) the scaffold manifest's metadata.embedder = { name, dimensions, normalize } object. Starting a server with a different --embed-name on an existing file fails loudly — mixing embedders in one file silently corrupts recall quality.

Claude Desktop config snippet (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "mnemo": {
      "command": "mnemo",
      "args": [
        "serve", "--mcp", "/absolute/path/to/agent.mnemo",
        "--embed-cmd", "python3 /absolute/path/to/embedder_server.py",
        "--embed-name", "sentence-transformers/all-MiniLM-L6-v2"
      ],
      "env": { "MNEMO_PASSPHRASE": "your-passphrase-here" }
    }
  }
}

Omit --embed-cmd/--embed-name to keep the pre-Phase-3 embedder-agnostic behavior (vector/query only).

CLI text path

The same --embed-cmd + --embed-name pair works on the one-shot CLI commands:

mnemo remember agent.mnemo --text "the user prefers tea" \
  --type semantic --importance 0.8 --agent assistant \
  --embed-cmd "python3 embedder_server.py" \
  --embed-name "sentence-transformers/all-MiniLM-L6-v2"

mnemo recall agent.mnemo --text "what drink?" --top-k 5 \
  --embed-cmd "python3 embedder_server.py" \
  --embed-name "sentence-transformers/all-MiniLM-L6-v2" \
  --no-track

--text is mutually exclusive with --query/--query-file.

The server holds an exclusive write lock on the file for its lifetime (see Concurrency), so two MCP servers on the same file will conflict — one will fail to start with Locked. That's intentional.

Result caching

The same .mnemo file is also a durable exact-key cache (Phase 10.1). One file, two live surfaces: agent memories and result cache, sharing the same encrypted-page storage and the same single-fsync WAL commit. Useful for memoizing LLM tool-call outputs, prompt→completion pairs, expensive HTTP responses — anything reconstructible on cache miss:

use mnemo_engine::{Mnemo, CachePutOpts};

let mut db = Mnemo::open("agent.mnemo", "correct horse battery")?;

// Insert (key is SHA-256-hashed by the engine — pass the raw input).
db.cache_put(
    "llm",
    "prompt-42",
    br#"{"model":"gpt-4","cost":0.03}"#,
    CachePutOpts { content_type: "json".into(), ttl_secs: Some(3600), cost_hint_ms: Some(250) },
)?;
db.flush()?;

// Lookup — hit updates access stats catalog-only (no full record rewrite).
if let Some(hit) = db.cache_get("llm", "prompt-42")? {
    println!("cached: {} bytes, {} accesses", hit.value.len(), hit.access_count);
}

Namespaces ("llm", "http", "tool", ...) isolate independent caches inside one file. Per-namespace [CacheBudget] caps entry count and byte total; overflow evicts LRU. TTL is per-entry.

Flush policy (Mnemo::set_cache_flush_policy) is per-handle:

Strict (default) — every cache_put dirties the cache directory; next flush() persists it.
Batched { max_dirty, max_age } — cache mutations accumulate; when either threshold trips, the engine internally calls flush(). Memory writes stay strict — only cache mutations get the relaxed lane.

Losing unflushed batched entries on a crash is a cache miss, never corruption — commits still ride the same single-fsync WAL transaction.

CLI:

mnemo cache put   agent.mnemo --ns llm  --key prompt-42 --value '{"...":"..."}' --content-type json --ttl 3600
mnemo cache get   agent.mnemo --ns llm  --key prompt-42
mnemo cache stats agent.mnemo --ns llm         # or omit --ns for the whole cache
mnemo cache purge agent.mnemo --ns llm --expired
mnemo cache delete agent.mnemo --ns llm --key prompt-42

MCP: mnemo serve --mcp exposes cache_put, cache_get, cache_stats, cache_put_semantic, and cache_get_semantic as tools (Phase 10.4) — any MCP-compatible agent can drive the cache alongside the memory surface. Python: the Mnemo class carries matching cache_* methods; the ~90-line @db.cached(...) decorator recipe (with OpenAI, Anthropic, and semantic-mode examples) lives in mnemo-python/README.md — copy it into your project.

Semantic cache (Phase 10.2)

For LLM workloads where prompts vary slightly but map to the same completion, the exact-key hash-of-string layer isn't enough — "please summarize this doc" and "summarize the following doc please" would miss. Semantic caching bridges the gap by keying on an embedding vector and returning the closest hit above a similarity threshold:

use mnemo_engine::{Mnemo, SemanticCachePutOpts, DEFAULT_SEMANTIC_THRESHOLD};

let mut db = Mnemo::open("agent.mnemo", "pw")?;

// Store: bring your own embedding vector + the model that produced it.
db.cache_put_semantic(
    "llm",
    "prompt-abc",
    embed("please summarize this doc"),          // Vec<f32>, caller-supplied
    b"...LLM response bytes...",
    SemanticCachePutOpts::new("bge-large-en-v1.5"),
)?;
db.flush()?;

// Look up: top-1 cosine over the namespace's vectored entries whose
// model matches. Returns Some((value, similarity)) iff sim >= τ.
if let Some((hit, sim)) = db.cache_get_semantic(
    "llm",
    &embed("summarize the following doc please"),
    DEFAULT_SEMANTIC_THRESHOLD,                  // 0.97
    "bge-large-en-v1.5",
)? {
    println!("cache hit at similarity {sim:.3}: {} bytes", hit.value.len());
}

Key contracts:

Exact-key and semantic entries coexist in the same namespace. cache_get (exact-key) and cache_get_semantic (vector) each ignore the other kind, so you can freely mix.
Model is matched exactly. A hit under "text-embedding-3-small" is invisible to a query under "bge-large-en-v1.5". Different embedders produce vectors in incompatible spaces — a cross-model hit would be a semantics bug.
Vector dimensionality is enforced against the database's configured dimensions (same as remember) — a mismatch returns MnemoError::DimensionMismatch.
DEFAULT_SEMANTIC_THRESHOLD = 0.97 is a conservative default. Model-dependent — with a lossier embedder consider raising it; with dense models on well-separated tasks you can drop it.

Text-in/text-out (cache_get_semantic_text(ns, "prompt", &embedder, τ)) lands with Phase 3's embedder integration. Until then callers own the embedding step, same as remember / recall.

Memory Nemo (MNemo)

An encrypted, single-file, portable agent-memory engine, written in Rust.

A whole memory store — vectors, content, structured metadata, and the multi-signal recall machinery an agent needs — lives in one file you can copy, back up, or hand to another process. The file is encrypted at rest, so the memory of an agent is as portable and as private as a SQLite database.

This repository implements a real, compiling, tested core. See Scope for what is deliberately left as roadmap.

The product is Memory Nemo (MNemo). The crate and Python package are distributed as mnemo-engine (both mnemo and mnemo-db were already taken by unrelated projects — see the "Breaking: import name" section in CHANGELOG.md for the full rationale). Starting in v0.7.0, the Rust library name and Python import name are mnemo_engine: use mnemo_engine::... and import mnemo_engine (or import mnemo_engine as mnemo if you want call sites unchanged). Rust code that wants to keep writing use mnemo::... can rename the dependency locally via mnemo = { package = "mnemo-engine", version = "0.7" } — that stays supported indefinitely because it's a Cargo-side alias. Python has no import mnemo shim: the old top-level mnemo module would still clash with mnemo-db and PyPI mnemo, which is exactly what the rename exists to end. The CLI command (mnemo) and the .mnemo file extension are unchanged.

Why

Agent frameworks usually bolt memory onto an external vector service: a network hop, a second thing to deploy, a second place secrets can leak. MNemo takes the SQLite position instead — memory is a file:

Single file. No server, no daemon, no schema migration dance.
Encrypted at rest. Passphrase in, ciphertext on disk.
Portable. cp agent.mnemo backup.mnemo is a complete, consistent backup.
Agent-native. First-class notions of memory type, importance, recency, access frequency, TTL, and per-agent scoping — recall ranks on all of them, not similarity alone.

Architecture

passphrase ──Argon2id──▶ KEK ──AES-256-GCM──▶ wraps DEK (random 256-bit)
                                                   │
                                                   ▼
                              every 8 KiB page ──AES-256-GCM──▶ ciphertext

Two-tier key hierarchy: a passphrase derives a key-encryption key (KEK) via Argon2id; the KEK wraps a random data-encryption key (DEK); the DEK encrypts every data page with AES-256-GCM. Changing the passphrase (rekey) only re-wraps the DEK — the bulk pages are never rewritten.

The file is a sequence of fixed 8 KiB pages:

Page 0 — the header (plaintext): magic, version, KDF parameters, salt, the wrapped DEK, pointers to the current catalog and ANN index runs, the location of the WAL region, and a CRC-32 over the header itself.
Pages 1..W — the write-ahead log region (see below).
Pages W+ — encrypted runs. Each record (a memory, MessagePack-encoded), each catalog snapshot, the optional ANN index, and the snapshot manifest occupy runs of consecutive pages. The page allocator is append-only.

Every page is sealed with a unique nonce derived from page_number and a monotonic write_counter, so a nonce is never reused under the DEK. From format v6 onwards, each page's home page_no is also bound into the GCM authentication tag as Associated Authenticated Data (AAD) — an attacker who swaps two valid encrypted pages between slots makes them un-decryptable at the new addresses (the auth check fails with PageAuthFailed). The v5→v6 migration re-encrypts every live data page in place under the new AAD.

Format v7 adds an AES-GCM seal at the tail of the header page that authenticates every mutable header field (write counter, next page, catalog/index/manifest pointers, version) under the DEK. The CRC at byte 238 stays as a pre-passphrase torn-write check; the seal is the keyed integrity layer that catches an attacker rewriting, say, catalog_start to point at an older catalog run. Open after a tampered flush errors with HeaderTampered instead of silently loading the stale state. The seal does not prevent rollback to a previous valid sealed state (replay of an old header byte-block); detecting that would need monotonic counters tracked outside the file.

Durability — write-ahead log

flush() is one atomic transaction, committed through a write-ahead log:

Record (vector) data pages are written copy-on-write to fresh pages and fsynced. Being unreferenced until the catalog below commits, they are crash-safe by construction.
The transaction's control plane — the new catalog, the ANN index, and the new header — is written into the WAL region as a run of (txn_id, page_no, page_image) frames followed by a checksummed COMMIT frame.
One fsync of the WAL is the commit point. Before it the transaction does not exist; after it the transaction is durable even though no home page has changed.
A checkpoint copies each frame to its home page and rewrites page 0.

A crash before the commit fsync leaves the previous state untouched. A crash after it is repaired on open: recover replays the committed transaction; a torn, never-committed tail is discarded. Because page 0 carries a CRC, even a torn header write is detected and healed from the WAL. The WAL region grows automatically when a transaction's control plane outgrows it. (tests/integration.rs exercises crash recovery, the torn-header heal, a discarded uncommitted log, and WAL growth.)

Torn-header recovery (DUR-1). Every post-create page-0 change — WAL relocation when the control plane outgrows the current region, and rekey when the passphrase moves — now goes through the WAL first: commit a one-frame (0, new_header) transaction into the new region, checkpoint page 0, then invalidate the predecessor region's first frame header. Combined with a streaming file-wide scan on open, this means a torn page-0 write at any WAL location is recoverable: open walks every 8 KiB-aligned MWAL candidate in the file, picks the highest-txn_id committed header frame whose wal_start + wal_seq agree with the frame's region, whose DEK unwraps under the passphrase, and whose AES-GCM seal validates, then replays it. Only the initial Mnemo::create writes page 0 unlogged. A header that fails its seal is reported as HeaderTampered and never auto-healed — the entire 294-byte header lies inside one 512-byte sector that storage writes atomically, so a surviving CRC with a failing seal cannot be a torn write, and silently loading a stale authenticated header from the WAL would erase the tamper-evidence the seal exists to give.

What this buys over a plain copy-on-write header swap: a single-fsync commit and an explicit, replayable transaction boundary.

Concurrency & read-only opens

Every read-write open (Mnemo::create, Mnemo::open) takes an exclusive OS advisory lock on the file for the lifetime of the handle (flock(2) on Unix, LockFileEx on Windows). A second concurrent open — from any process — returns MnemoError::Locked instead of silently interleaving WAL frames and corrupting the file. The lock releases when the handle drops.

Mnemo::open_read_only(path, passphrase) opens the file with a shared lock. Multiple read-only handles coexist — this is what the shared-lock tier buys you. A read-only handle does not coexist with a writer holding the exclusive lock: flock(2) on Unix and LockFileEx on Windows both refuse a shared-lock request while an exclusive lock is held. So "long-lived reader alongside a live writer" needs the writer to release first (drop the write handle, then reopen read-only). Every mutating method on a read-only handle returns MnemoError::ReadOnly; recall with the default track_access(true) counts as a mutation (it bumps catalog access stats) and requires track_access(false) on a read-only handle. The matching CLI knob is mnemo recall --no-track.

Two conditions block a read-only open and return MnemoError::NeedsWriteOpen: a committed-but-uncheckpointed WAL transaction waiting to be replayed on open, and an on-disk format version older than the current build. Both cases require write access to fix. The remedy is the same either way: open read-write once (which recovers / migrates in place), close cleanly, and reopen read-only. We chose this fail-loud policy over an in-memory WAL overlay so read-only reads can never return stale pre-recovery state.

Snapshots & point-in-time recovery

Because record, catalog, and index pages are only ever appended, every past flush's pages are still on disk. MNemo exploits this: each flush appends a small entry to a snapshot manifest recording where that transaction's catalog and index runs live. The manifest turns the append-only file into a navigable history.

snapshots() lists every committed transaction — its id, timestamp, and memory count.
restore_to(txn_id) reinstates the database exactly as that transaction left it; restore_to_time(unix_secs) picks the latest snapshot at or before an instant.

A restore is itself an ordinary committed transaction (and a new snapshot), so it is crash-safe and reversible — having rewound, you can roll forward again. History reaches back to the last compact_file, which reclaims space by rewriting the file and so collapses the manifest to a single snapshot. That compaction boundary is the one limit: there is no external log archiving, so you cannot restore to an instant older than the last compaction.

Page cache

Decrypted page payloads are held in a bounded LRU cache so repeated reads skip a decrypt. Eviction targets the least-recently-used clean page; a dirty page — the only copy of an un-flushed write — is never evicted, so the cap bounds retained clean pages without ever risking data. The default cap is 8192 pages (~64 MiB); Mnemo::set_cache_capacity tunes it, and Mnemo::cache_stats reports occupancy. The LRU order is an intrusive linked list over a slab of indices, so it stays within the crate's #![forbid(unsafe_code)].

Recall

recall scores each candidate with the v0.7.0 scoring-v2 formula:

score = similarity
      + max(similarity, 0)
        · ( β · recency
          + γ · (importance − 0.5)
          + δ · tanh(access_count / 5) )

recency = exp( -ln(2) · age_secs / half_life_secs )
age_secs = now - memory.created_at      # not accessed_at

Four deliberate shape changes from the pre-v0.7.0 additive formula (α·sim + β·recency + γ·importance + δ·ln(1 + uses)):

Similarity-gated bonuses. Recency / importance / usage are multiplied by max(similarity, 0), so a stale-but-off-topic memory can't ride recency past a semantically-on-point one.
Importance centered at 0.5. A default-importance memory contributes zero importance bonus; importance only pushes a score up or down from that reference point.
Saturating usage. tanh(access_count / 5) caps the feedback bonus so a heavily-returned memory stops accumulating weight past ~5 uses.
Recency from created_at. A recall() no longer bumps its own results' recency on later queries.

recency still decays exponentially (default 7-day half-life). recall also filters by memory type and by agent scope (an agent sees its own memories plus any marked Shared), and skips TTL-expired entries. Weights and the similarity metric (cosine / dot / L2) are per-request. The raw RecallResult.score sits on a different scale than pre-v0.7.0 — callers comparing raw scores to a fixed threshold should re-pick the threshold.

Sessions

A Session wraps the database for the span of one conversation. db.session(agent) opens it with a fresh session id; add_turn records each conversation turn as a Working memory tagged with that session and agent; recall retrieves context scoped to the agent. Closing the session consolidates its turns — close() promotes them from working memory to durable Episodic memory ("what happened"), while discard() throws them away. The session borrows the database mutably for its lifetime, so the single-writer rule is enforced by the compiler. See examples/session.rs.

Approximate index (IVF + PQ)

Exact scan is O(n) — fine for thousands of memories. Past that, build an IVF + PQ index and recall becomes sub-linear:

IVF (inverted file): k-means groups vectors into ≈√n partitions; a query only scans the n_probe partitions nearest its centroid.
PQ (product quantization): each vector is split into m subspaces, each quantized to one of ≤256 learned codewords — a vector becomes m bytes. Candidate distances come from precomputed per-subspace lookup tables, so the scan never touches a full float vector.
Rerank: the closest n_rerank candidates are loaded at full precision and ranked exactly with the caller's metric.

db.build_index()?;                 // or build_index_with(IndexConfig { .. })
db.flush()?;                       // the index is persisted in the file

// recall now runs IVF → PQ → rerank automatically; tune per query:
let req = RecallRequest::new(query).top_k(10).n_probe(16).n_rerank(128);

n_probe and n_rerank are the accuracy/speed dials — higher is more accurate, lower is faster. The index is stored inside the encrypted file (its own page run), maintained incrementally on insert, and rebuilt fresh by compact. build_index re-clusters; drop_index reverts to exact scans. Internally the index ranks by squared-L2; the exact rerank uses the requested metric, so for cosine queries a generous n_rerank is advisable.

Default auto-formulas (PR ANN-1 + ANN-1.1, pending v0.6.0). IndexConfig::default() now picks pq_subspaces = dims/4 at build time (was dims/8), plus query-time formulas:

n_probe = max(ceil(partitions / 2), min(partitions, 100)) (was fixed 8) — full probe up to ~10K-vector files (≤100 partitions), half probe above
n_rerank = (n_vectors / 200).clamp(100, 1000) (ANN-1 shipped a fixed 500; ANN-1.1 made it size-aware — see index::effective_auto_n_rerank). Hits 100 at 10K, 500 at 100K, caps at 1000.

Measured on the Phase 6 synthetic corpus: 10K hits recall@10 ≈ 0.98 at ~1.4 ms/query; 100K hits ~0.98 at ~10.6 ms/query. The old pre-ANN-1 n_probe=8, n_rerank=64 defaults capped 100K recall@10 at ~0.40.

Upgrade path for existing files. The n_probe / n_rerank fix takes effect at query time (ANN-1.1), so a v0.5.x-built index gets the new defaults immediately on first open under v0.6.0+ — no rebuild needed. The pq_subspaces fix is build-time (baked into the PQ codebook), so you'll stay on the old 48-subspace codebook until you run mnemo index --rebuild (CLI) or db.rebuild_index() (Rust / Python). With pq_subspaces=48 + new query-time formulas, 100K tops out at ~0.92; rebuilding to 96 subspaces lifts it to ~0.98. See test/scripts/perf_v0.5.0.json keys ann_recall_grid_v2 (grid) and ann_defaults_v0_6_0 (measurements at the shipped defaults).

Any explicit n_probe / n_rerank passed per call still overrides everything above.

Mnemo::search always stays exact — it is the brute-force ground truth.

Security model

MNemo's threat model is data at rest plus tamper-evidence. The file on disk is opaque without the passphrase, and an attacker who modifies bytes in the file is detected — not allowed to silently substitute or roll back to a previous valid state.

What's protected:

Confidentiality of page contents. Every 8 KiB page (records, catalog, ANN index, snapshot manifest, WAL frames) is encrypted under the random 256-bit DEK using AES-256-GCM. The header's KDF parameters and salt are plaintext — they're the bootstrap; everything past the passphrase derivation is ciphertext.
Page placement integrity (v6+). Each page's home page_no is bound into the GCM authentication tag as AAD. An attacker who transplants a valid encrypted page to a different slot makes it un-decryptable at the new address — open errors with MnemoError::PageAuthFailed.
Header field integrity (v7+). A small AES-GCM seal at the tail of the header page authenticates every mutable header field (write_counter, next_page, catalog/index/manifest pointers, version) under the DEK. Rewriting any of them invalidates the GCM tag and open errors with MnemoError::HeaderTampered — not a silent stale-data load.
Passphrase secrecy. Wrong passphrase fails authenticated decryption on the wrapped DEK and is reported as MnemoError::WrongPassphrase — indistinguishable from a tampered wrapped-DEK blob. No oracle.
Crash-safety nonce uniqueness. flush leases write-counter and page slots in the header before any data page hits the disk, so a crash mid-flush can't lead a subsequent flush to reuse a (page_no, write_counter) nonce on different plaintext under the same DEK.

What's not protected:

Availability. An attacker with write access can always destroy data — truncate the file, zero pages, delete it outright. Tamper-evidence means modifications are detected, not prevented.
Rollback to a previously valid sealed state. The v7 seal catches a point-in-time tamper, but it can't tell that the entire header block has been replaced with an older valid snapshot of itself (a "replay" attack). Detecting that needs monotonic counters tracked outside the file — out of scope.
agent_id scoping. recall(agent_id=...) is cooperative filtering, not isolation. Anyone holding the passphrase can read every agent's memories; the scoping helps an honest agent stay in its lane, not protect one agent's data from another.
Side channels. Encryption is not constant-time at the engine level; page-cache behavior, file-size growth, and access patterns can leak metadata. Mnemo trusts the OS and the AES-GCM implementation for primitive-level side-channel hardness (AES-NI on x86, equivalent on ARM).
In-memory secrecy. Decrypted page payloads and the DEK live in process memory while the database is open. Use [Mnemo::close] (or drop the handle) promptly when the database isn't needed; do not swap-out without mlock if the host's swap policy is a concern.
External attackers with the passphrase. Mnemo authenticates page content under the DEK, not the user — anyone with the passphrase IS the user as far as the format is concerned.

The v0.2.0 format (v7) closes every "silent stale-data load" path flagged in the security review; rollback-replay and availability are acknowledged limitations.

v0.5.2 (SEC-1): AES-GCM nonces are now per-session random. Up to v0.5.1, each page's encryption nonce was (page_no, write_counter) — unique within a single file, but shared across copies of a file under the same DEK (the data key never changes after create). If you had two copies of the same .mnemo file and both received writes (two laptops syncing through Dropbox and the file "conflict-copies" diverge; a restored older backup resuming alongside the current file; anything that leads to cp agent.mnemo backup.mnemo + writes to both), the two copies' writers would produce pages with the same encryption nonce under the same key but different plaintexts — breaking AES-GCM confidentiality and tamper-evidence. v0.5.2 moves to a random 64-bit per-session prefix + 32-bit per-session counter (NIST SP 800-38D §8.2.1), drawn fresh on every writable open. Cross-copy collisions now require two sessions to independently draw the same 64-bit value — astronomically unlikely. If you ran a pre-v0.5.2 build on diverged copies, rotate your DEK by running compact_file — it rewrites the file through a fresh random_dek, so the prior nonce reuse under the old key stops mattering for the rewritten pages. No format change; existing files open on v0.5.2 unchanged.

Quick start (library)

use mnemo_engine::{Mnemo, MnemoConfig, Memory, MemoryType, RecallRequest};

fn main() -> mnemo::Result<()> {
    let cfg = MnemoConfig { dimensions: 3, ..Default::default() };
    let mut db = Mnemo::create("agent.mnemo", "correct horse battery", cfg)?;

    db.remember(
        Memory::new("the user prefers dark mode", MemoryType::Semantic, vec![0.1, 0.2, 0.9])
            .with_agent("assistant-1")
            .with_importance(0.8),
    )?;
    db.flush()?; // durable

    for hit in db.recall(&RecallRequest::new(vec![0.1, 0.2, 0.9]).top_k(5))? {
        println!("{:.3}  {}", hit.score, hit.memory.content);
    }
    Ok(())
}

Run the bundled example: cargo run --example quickstart.

Command-line tool

cargo run --bin mnemo -- <command>

Command	Purpose
`init`	Create a new encrypted database (auto-adds scaffold manifest)
`info`	Print statistics (including ANN index shape)
`about`	Self-describing briefing — print onboarding memories
`import`	Bulk-load memories from a JSON Lines file
`index`	Build, rebuild, or drop the IVF+PQ index
`list`	Browse live memories (table / json / jsonl, filters)
`get`	Fetch one memory by its ULID
`search`	Exact nearest-neighbour search (similarity only)
`recall`	Multi-signal ranked retrieval (sim + recency + …)
`verify`	Decrypt and re-validate every live record
`rekey`	Re-encrypt the data key under a new passphrase
`compact`	Rebuild the file, dropping tombstones and expired
`snapshots`	List the restorable snapshots (one per flush)
`restore`	Roll the database back to a past snapshot
`demo`	Self-contained end-to-end demonstration

Self-describing databases

A .mnemo file should be able to introduce itself. The single-file philosophy is that everything an agent needs to use this database lives in the file — not in a sibling README, not in environment configuration, not in tribal knowledge. The about command surfaces that introduction:

cargo run --bin mnemo -- about agent.mnemo
# Prints a one-line stats header, then every memory tagged
# metadata.area = "onboarding" sorted with the canonical manifest first,
# then by importance descending, ending with a quick-start footer.

The convention is two metadata keys on ordinary memories — no new schema, no extra file:

metadata.area = "onboarding" marks a memory as part of the orientation briefing. Returned by Mnemo::about() (Rust) and db.about() (Python).
metadata.topic = "manifest" marks the one canonical "I am this file" entry. Hoisted to the top of about regardless of importance, and the only entry returned by mnemo about --manifest-only.

The manifest is the headline orientation point — every database should have one. mnemo init auto-inserts a scaffold manifest (a placeholder with metadata.scaffold = true) so a new file is self-describing from the moment it's created; mnemo about tags it as (scaffold — please replace) until you overwrite it. Pass --no-manifest to mnemo init for an entirely empty file.

Replace the scaffold with one that records your project's actual values. Recommended fields inside metadata on the manifest itself:

embedder.name, embedder.dimensions, embedder.normalize — which embedding model produces vectors compatible with this file. A receiving agent uses this to pick the right embedder or fail loudly when it can't.
agent_id_default — the agent id convention used when writing here.
project.name, project.repo — what project this database serves.
conventions.* — any project-specific metadata schemas (e.g. perf entries always carry version, metric, value, units, build, corpus).

seed.json in test/scripts/ shows a working example. With a manifest in place, an agent receiving a .mnemo file plus its passphrase needs no external docs: mnemo about <file> (or db.about()) tells them what the file is and how to use it.

Exploring a database

The exploration commands compose into a quick read-only workflow:

# orient — what is this file? (manifest + onboarding briefing)
cargo run --bin mnemo -- about agent.mnemo

# overview — size, agents, index shape, snapshot count
cargo run --bin mnemo -- info agent.mnemo

# browse — table by default; --format json|jsonl for pipelines
cargo run --bin mnemo -- list agent.mnemo --type semantic --limit 20

# fetch one — copy a ULID from `list`
cargo run --bin mnemo -- get agent.mnemo 01HXYZ... --verbose

# rank — multi-signal recall (uses the ANN index if built)
cargo run --bin mnemo -- recall agent.mnemo --query 0.1,0.2,0.3 --top-k 5

list decrypts every live record (O(n)); it is meant for human-scale exploration, not for serving requests. recall updates each returned memory's accessed_at and access_count — but as of v5 these live on the catalog entry, not in the record body, so the update touches one catalog page per flush, not the full vector of every result. Pre-v5, a top-K recall rewrote K full records (vector + content) at the next flush; v5 makes recall effectively a write-once-on-catalog operation. Set RecallRequest::track_access(false) for a fully read-only recall — useful for batch scoring, dry-runs, or tooling that shouldn't perturb the database.

recall needs a vector in the database's dimensionality; for dogfooded runs over real embeddings, pull a query vector from your embedding model (or from project-memory.jsonl in the test/ sandbox).

The passphrase is resolved in this priority order:

--passphrase flag if given. The CLI prints a warning to stderr because the value lands in shell history and process listings.
MNEMO_PASSPHRASE environment variable if set. Recommended for scripts and CI.
Interactive TTY prompt without echo. The fallback when neither the flag nor the env var is set. init and rekey prompt twice and verify a match so a typo doesn't lock the file. rekey's new passphrase follows the same chain via --new-passphrase, MNEMO_NEW_PASSPHRASE, or a double-prompt.

For real applications, prefer the library API and supply the passphrase in-process.

export MNEMO_PASSPHRASE=hunter2
cargo run --bin mnemo -- init agent.mnemo --dimensions 768
cargo run --bin mnemo -- demo            # try it without any setup

Build and test

cargo build --release
cargo test            # 66 integration + 26 CLI smoke + proptest state machine + unit tests
cargo run --example quickstart

Minimum supported Rust version: 1.75. All dependency versions are pinned exactly for reproducibility.

Performance

A running log of measured performance, so future versions can be compared against earlier baselines. Each entry records the corpus, the build, and the numbers — append a new entry rather than overwriting one when a change is expected to move them.

ARM users: enable hardware AES

This affects everyone running mnemo on Apple Silicon, AWS Graviton, or any other aarch64 CPU, if you build mnemo from crates.io.

The aes and polyval crates (used by aes-gcm, which encrypts every page in a .mnemo file) only emit ARMv8 hardware AES + PMULL instructions when built with two cfg flags. Without them, every aarch64 build runs a constant-time software fallback at roughly 5× the cost per page decrypted or encrypted, dominating the exact-scan recall path and the per-page WAL/flush overhead on an M1.

If you installed from crates.io (cargo install mnemo-engine), set RUSTFLAGS before installing:

RUSTFLAGS="--cfg aes_armv8 --cfg polyval_armv8" cargo install mnemo-engine

Or add to your own ~/.cargo/config.toml so every build picks it up:

[target.'cfg(target_arch = "aarch64")']
rustflags = ["--cfg", "aes_armv8", "--cfg", "polyval_armv8"]

To verify which backend your build picked up, run mnemo info (CLI) or mnemo.build_info() (Python) — the output names each backend explicitly:

aes:         armv8 hardware
polyval:     armv8 PMULL hardware

If you see armv8 software (constant-time fallback) instead, your RUSTFLAGS didn't reach the build.

x86_64 users don't need to do anything — the same crates runtime-detect AES-NI + CLMUL. The flags above are a no-op on x86_64.

Mnemo's own PyPI wheels (pip install mnemo-engine) and GitHub Release binaries carry the hardware path baked in — this section only matters if you build from source.

Sizing tips

Three knobs cover most of the small-file size questions; the fourth is a modelling choice that compounds with the others.

1. Right-size the WAL reservation. The default initial WAL is 8 pages (64 KiB), down from 64 (512 KiB) in v0.1.0 — the v0.1.0 reservation dominated small-file size (~62% of a 31-memory dogfood file). The WAL auto-grows, so the default is safe even for large catalogs; raise it only if you know each transaction routinely commits a large catalog or ANN index and you want to skip the first grow event:

let cfg = MnemoConfig {
    dimensions: 768,
    wal_pages_initial: 64,     // 512 KiB up front
    ..Default::default()
};

2. Cap the snapshot manifest. Every flush appends one snapshot entry; without a cap, long-running services accumulate them forever — manifest serialize-cost and the manifest run on disk grow O(total flushes). The default cap is 256 retained snapshots (the most recent N are kept; older ones are pruned at flush time, with their data pages left on disk until the next compact_file). Override per-database:

let cfg = MnemoConfig {
    dimensions: 768,
    max_snapshots: 1024,   // keep more history
    ..Default::default()
};
// or 0 to disable the cap entirely (pre-v0.3 behavior):
let cfg = MnemoConfig { max_snapshots: 0, ..Default::default() };

Mnemo::set_max_snapshots(usize) overrides on an already-open handle. restore_to a pruned txn_id returns MnemoError::NotFound.

3. Pick dimensions wisely. Vector storage is dimensions × 4 bytes per memory and dominates the data line for everything but tiny corpora. Cut dimensions and you cut that line proportionally — a 1024-dim model costs 4 KiB/memory; a 256-dim model costs 1 KiB/memory.

4. Use a Matryoshka (MRL) embedder, then truncate. Matryoshka Representation Learning trains a model so the most important information is front-loaded into the early dimensions. With an MRL-trained embedder (OpenAI text-embedding-3-*, Nomic nomic-embed-text-v1.5, Snowflake arctic-embed-l-v2.0, and several others) you can store the first k dims of a d-dim vector and lose only 1–2% recall — a 4× storage win before any quantization. Concretely: a 1024-dim MRL model truncated to 256 dims fits in 1 KiB/memory instead of 4 KiB. Slice the vector on the client before passing it to remember; mnemo just sees a 256-dim vector and uses it as the database's native dimensionality.

These knobs combine: a 256-dim MRL model in a fresh-default file gives roughly 8× less on-disk footprint at small N versus a 1024-dim model in a v0.1.0 default file, and the gap widens as the corpus grows. Future versions may add page-level compression (Zstd) and binary quantization (32× on vectors); when they land they will compose on top of these choices, not replace them.

Baseline — v0.1.0, May 2026 (dogfood DB)

Corpus. test/project.mnemo, 31 live memories, 384-dim MiniLM embeddings, encrypted, no ANN index, 1.41 MB on disk (844 KB at 28 memories — the file grew under repeated bootstraps via append-only history).

Builds. CLI numbers are from the debug cargo run --bin mnemo; Python numbers are from the release maturin develop wheel. Argon2id parameters are KdfParams::secure() (m_cost=19456, t_cost=2). Latencies are medians over a handful of runs, not rigorous percentiles.

File-size breakdown at 844 KB / 28 memories:

Region	Pages	Size	Share
Header (plaintext, KDF + salt)	1	8 KB	~1%
WAL region (reserved)	64	512 KB	~62%
Data, catalog, manifest, history	38	311 KB	~37%

Per-page crypto overhead is 28 B (12-byte nonce + 16-byte GCM tag) ≈ 0.3% — negligible. The dominant costs at small N are the 512 KiB WAL reservation, 8 KiB page rounding, and the append-only snapshot history left behind by repeated flushes. Logical payload (UTF-8 text + f32 vectors) is ~50 KB, ≈ 5.9% of the file; effective on-disk cost is ~45 KB per memory at N=31.

Operation latency (debug CLI unless noted):

Operation	Median	Notes
`info`	~330 ms	KDF + read header/catalog
`list` (31 memories)	~330 ms	KDF + decrypt/decode all records
`get` (one memory)	~290 ms	KDF + decrypt one record
`verify`	~330 ms	KDF + decrypt/validate all
Python `open` + `recall` top-10	~35 ms	release wheel, same crypto
Python `remember` + `flush`	~31 ms	includes MiniLM embed + WAL fsync

Verdict at this scale. Argon2id dominates every CLI invocation (~250–300 ms of the ~330 ms). Per-record decrypt is cheap; reopening the file is not. Long-lived processes (an agent server, the Python binding held across calls) avoid the KDF tax and land in the tens-of-milliseconds range. For a 31-memory dogfood file, on-disk overhead is ~17× the raw payload — this is the format's worst case (fixed WAL reservation + page rounding + append history dominate everything else). The README design targets (<5 ms recall at 100 K, <15 ms at 1 M, <4 KB/memory) are aspirational goals at scale, not yet measured here.

Historical note (updated 2026-10). The CLI timings in this baseline were taken on a debug cargo run --bin mnemo build. The v0.5.x Phase 6 harness re-measures the same operations with release builds and reports them in the Performance history row + the test/scripts/perf_v0.5.0.json artifact — on a release build with production KDF the open latency settles at ~15 ms, not ~330 ms. Keeping this row for history, not as a current baseline.

Knobs that should help small files. Running compact collapses snapshot history back to a single page run; the default 64-page WAL reservation is fixed today and would need a config change to shrink. ANN index builds are not worth it below ~thousands of memories.

Performance history

Append a row whenever a release changes how the format behaves. Earlier rows stay as-is so improvements (or regressions) are visible.

Version	Date	Corpus	File size	CLI `info`	Python recall	Notes
v0.1.0	2026-05	31 mems, 384-dim, no index	1.41 MB	~330 ms	~35 ms	Initial dogfood baseline
v0.5.0	2026-10	10K mems, 384-dim, IVF+PQ	82.9 MB	— (open 14.9 ms secure KDF)	374 µs (IVF+PQ†) / 623 ms (exact‡)	Phase 6: criterion benches + ranking harness + ANN recall@10 grid + file-size breakdown. IVF+PQ ≈ 1,660× faster than exact at 10K; 10,400× at 100K. Default `IndexConfig` recovers only 40% of exact top-10 — index-quality defect, fixed in ANN-1 below (originally mis-filed as a scoring issue). Full numbers in `test/scripts/perf_v0.5.0.json`.
v0.5.1	2026-10	same corpus, hardware AES on ARM	—	—	105 ms (exact 10K) / 1.05 s (exact 100K); IVF+PQ unchanged	PR 11.0: enable `--cfg aes_armv8 --cfg polyval_armv8` so aarch64 builds use hardware AES + PMULL instead of the constant-time software fallback. 6.0× faster exact scan at 10K, 6.1× at 100K; 1.3× on cache_put; IVF+PQ and Argon2id-bound open unchanged. See `test/scripts/perf_v0.5.0.json`'s `speed_hw_aes` block for the full set. No API or format change. If you build from crates.io see the "ARM users" section above.

† IVF+PQ speed at the old defaults; recall@10 was ~0.40 at those defaults, not 1.0. PR ANN-1 raises the default to ≥0.95 recall@10 at ~10 ms/query at 100K; see the ann_recall_grid_v2 block in perf_v0.5.0.json. ‡ Both v0.5.0 timings were measured on software AES (the fix landed in v0.5.1). On hardware AES, exact scan at 10K drops to 105 ms; IVF+PQ was never AES-bound so it's unchanged.

Scope: what is and isn't here

This crate is a faithful build of the MNemo plan, built to actually compile, run, and pass tests. The storage engine, encryption, agent-memory model, multi-signal recall, the IVF+PQ approximate index (Phase 2), the write-ahead log (Phase 3), snapshot-based point-in-time recovery, the bounded LRU page cache, and the Session lifecycle wrapper (Phase 5) are all built and tested. Search and recall scale from exact brute force to sub-linear retrieval; flush is a single-fsync atomic transaction repaired by WAL replay on a crash; every transaction is a restorable snapshot back to the last compaction; and a conversation runs through a Session that consolidates its turns into episodic memory. See Durability, Snapshots, Sessions, and Approximate index.

One item is deliberately left as a roadmap item and is not in this build:

TypeScript bindings. Phase 6 of the plan also calls for a Node/WASM package via napi-rs; only the Python bindings are built so far. They live in the sibling mnemo-python/ crate — a PyO3 wrapper exposing mnemo.open(...) and a Mnemo class (pip install maturin && maturin build). The Rust core a TypeScript binding would wrap is complete.

License

Apache-2.0.

mnemo-engine 0.7.0