Crate mnemo_engine

Expand description

§Mnemo

Mnemo is an encrypted, single-file, portable agent-memory engine.

A whole memory store — vectors, content, metadata, and the multi-signal recall machinery an agent needs — lives in one file you can copy, back up, or hand to another process. The file is encrypted at rest with a two-tier key hierarchy: an Argon2id key-encryption key (KEK) derived from a passphrase wraps a random data-encryption key (DEK), and the DEK encrypts every page with AES-256-GCM.

§Quick start

use mnemo_engine::{Mnemo, MnemoConfig, Memory, MemoryType, RecallRequest};

let cfg = MnemoConfig { dimensions: 3, ..Default::default() };
let mut db = Mnemo::create("agent.mnemo", "correct horse battery", cfg)?;

db.remember(
    Memory::new("the user prefers dark mode", MemoryType::Semantic, vec![0.1, 0.2, 0.9])
        .with_agent("assistant-1")
        .with_importance(0.8),
)?;
db.flush()?;

let hits = db.recall(&RecallRequest::new(vec![0.1, 0.2, 0.9]).top_k(5))?;
for h in hits {
    println!("{:.3}  {}", h.score, h.memory.content);
}

§What is and is not built

This crate implements a real, tested core: the encrypted single-file storage engine, the crypto layer, the agent-memory model, multi-signal recall, an IVF+PQ approximate-nearest-neighbour index, a write-ahead log, snapshot-based point-in-time recovery, a bounded LRU page cache, and the Session conversation wrapper. Exact brute-force search remains available as the ground-truth baseline; a built index makes Mnemo::recall sub-linear. Each Mnemo::flush is one atomic, WAL-committed transaction and a restorable snapshot — Mnemo::restore_to rewinds the database to any past transaction. Python and TypeScript language bindings are the one documented roadmap item — see the README.

Modules§

essence_status: Status values the essence section supports (Phase 12.7b). More kinds (tasks: queued → active → ready → done | blocked; findings: open → accepted | deferred | rejected | needs-user → verified) ship with Phase 13 reusing the same status-record core; the transition tables for those are declared as constants in this crate but currently unused.
mcp: Hand-rolled Model Context Protocol (MCP) stdio server for MNemo.

Structs§

BuildInfo: Build-time cryptographic backend information.
CacheBudget: Per-namespace eviction budget. cache_put enforces both caps: on insert, the LRU is evicted until both live-entry count and live byte total fit under the caps.
CachePutOpts: Options accepted by Mnemo::cache_put. Defaults describe an untyped, never-expiring, non-cost-hinted entry.
CacheStats: Summary statistics for a namespace (or the whole cache).
CachedValue: A returned cache hit. accessed_at / access_count are updated on the directory before this is constructed, so the values here reflect the post-touch state (mirroring the recall-time convention).
CommitBatch: Atomic write batch for crate::Mnemo::commit_deltas (Phase 12.7c). The planner’s “one call that composes facts + essence + turns” shape: everything validates up front, then every write runs under a single flush().
CommitReport: Result of crate::Mnemo::commit_deltas (Phase 12.7c).
CompactReport: Result of a Mnemo::compact_file run.
EmbedCmd: Persistent-child-process adapter for a user-supplied embedder command. See the module docs for the protocol.
EmbedderIdentity: Describe an embedder’s observable identity. Used both in-memory on an open [Mnemo] handle and persisted at metadata.embedder on the scaffold manifest so that reopening with a different embedder is caught at set_embedder time instead of silently polluting the index.
EssenceOp: One essence op inside a CommitBatch (Phase 12.7c). Validation (missing/extra fields per kind) runs up front in crate::Mnemo::commit_deltas — a single bad op aborts the whole batch before any write lands.
ExportedTurn: One turn of a thread’s archive in the ingest shape (Phase 12.7d), as returned by crate::Mnemo::thread_export_turns and by Python db.thread_export, MCP thread_export and mnemo thread export.
Filter: Pre-scoring filter applied by crate::Mnemo::recall (Phase 12.6).
IndexConfig: Tuning knobs for building and querying an [IvfPqIndex].
IndexInfo: A read-only snapshot of an index’s shape, surfaced via Mnemo::stats.
IngestReport: Result of one crate::Mnemo::ingest_thread call (Phase 12.7a).
KdfParams: Argon2id cost parameters. Stored (unencrypted) in the file header so the KEK can be re-derived on open.
KeyedMemory: One keyed fact inside a CommitBatch (Phase 12.7c).
Memory: A single memory entry.
MemoryPatch: Patch body for crate::Mnemo::update (Phase 12.1): every field is optional, and only the Some(...) fields overwrite the corresponding field on the target memory. All fields can be set independently.
Mnemo: An encrypted, single-file agent memory database.
MnemoConfig: Configuration for creating a new database.
RecallRequest: A retrieval request for Mnemo::recall.
RecallResponse: Phase 12.8 budgeted-recall envelope returned by Mnemo::recall_budgeted. Wraps the Vec<RecallResult> the legacy Mnemo::recall returns so callers can tell whether RecallRequest::max_bytes cut anything.
RecallResult: One scored result from Mnemo::recall.
ScoreWeights: Weights for the multi-signal recall score (scoring v2, shipped in v0.7.0).
SemanticCachePutOpts: Options for crate::Mnemo::cache_put_semantic. Same as CachePutOpts plus a required model string — a hit from a different embedding model’s cache is a semantics bug, not a win, so the engine forces callers to declare the model and matches it on lookup.
Session: A scoped conversation session over a Mnemo database.
SnapshotInfo: A restorable point in a database’s history — see Mnemo::snapshots.
Stats: Summary statistics for a database.
ThreadTurn: One turn fed into crate::Mnemo::ingest_thread (Phase 12.7a).
Turn: One conversation turn, written to working memory by Session::add_turn.
Ulid: Re-export of ulid::Ulid, the identifier type used for memories. A Ulid is a unique 128-bit lexicographically sortable identifier

Enums§

CacheFlushPolicy: Per-handle flush policy for cache writes. Memory writes ignore this and always behave as Strict.
EssenceOpKind: Which op a EssenceOp carries. Mirrors the per-method Mnemo::essence_* surface so one batch can mix section + timeline edits in one call.
EssenceScope: Scope selector for an essence item (Phase 12.7b). Thread-scoped items belong to one _mnemo.thread; project-scoped items are shared across every thread in the file.
EssenceSection: Which essence section an item belongs to.
FilterTerm: One AND-ed condition applied to a candidate memory’s metadata during filtered recall (Phase 12.6). Terms are combined conjunctively — a memory passes the filter only if every term evaluates true against its metadata map. See Filter for the surface API.
KeyScope: Scope selector for a keyed slot (Phase 12.1.1). A keyed remember’s slot is always identified by (scope_owner, key): Private(agent_id) slots are per-agent, Shared slots are per-file. history_in(key, scope) uses the same selector so the lookup matches the write exactly.
MemoryType: The four kinds of agent memory.
Metric: Distance/similarity metric for vector comparison.
MnemoError: All errors the Mnemo engine can produce.
Role: Who produced a conversation turn.
Scope: Visibility of a memory across agents sharing one .mnemo file.
TranscriptFormat: The supported transcript shapes for read_transcript / mnemo thread ingest --format ….
TurnPolicy: What crate::Mnemo::commit_deltas does with the turns carried in the batch (Phase 12.7c).
WriteOutcome: Outcome of a 12.1 write operation (remember with key, or update).

Constants§

CONTINUATION_TIMELINE_ENTRIES: How many timeline entries from the predecessor thread the continuation render pulls in (Phase 12.7c). Small enough to keep the render under the default 6 KB budget, large enough to carry a handful of milestones.
DEFAULT_BATCH_MAX_AGE: Default max_age for CacheFlushPolicy::Batched.
DEFAULT_BATCH_MAX_DIRTY: Default max_dirty for CacheFlushPolicy::Batched.
DEFAULT_EMBED_TIMEOUT: Default per-request timeout. Set generously because a cold-start sentence-transformers call can genuinely take a few seconds the first time; configurable via EmbedCmd::with_timeout (Rust API) or the --embed-timeout CLI flag.
DEFAULT_MAX_BYTES: Default per-namespace byte budget (values + directory overhead). Once exceeded, cache_put evicts LRU until the total falls back under the cap.
DEFAULT_MAX_ENTRIES: Default per-namespace entry-count budget. Once a namespace holds this many live entries, cache_put evicts the LRU entry before inserting.
DEFAULT_SEMANTIC_THRESHOLD: Default similarity threshold for crate::Mnemo::cache_get_semantic. Lookups below this are treated as misses. Chosen conservatively to avoid false hits on subtly-different prompts; callers can override per-lookup. Model-dependent — text-embedding-ada-002 and BGE at this threshold rarely surface unrelated content, but if you’re working with a lossier embedder consider raising it.
ESSENCE_DEFAULT_RENDER_BYTES: Default byte budget for crate::Mnemo::essence rendering (Phase 12.7b). Matches the planner’s “max_bytes = 6144” spec, which sits comfortably below a 4K-token context window.
ESSENCE_SECTION_MAX_BYTES: Maximum content length for a section item (Phase 12.7b). intent / conventions / decisions / state / open / next / lessons all share the same single-line cap. Items over this are rejected.
ESSENCE_TIMELINE_MAX_BYTES: Maximum content length for a timeline entry (Phase 12.7b). Entries over this are rejected, not truncated — the planner’s spec is explicit about this because truncation would lose the narrative, and splitting across records would lose ordering.
ESSENCE_TRANSITIONS: Essence transition table for Phase 12.7b. Only open → resolved is allowed. Timeline entries don’t use this (they’re append-only and have no status).
FINDING_TRANSITIONS: Finding transition table (declared for Phase 13; currently unused in the engine). open → {accepted | deferred | rejected | needs-user} → verified.
INDEX_PERSIST_EVERY_CHANGES: PERF-1: …or once this many vectors were added, changed or removed since the last persist. Records written after the persisted index are re-assigned when the index is next loaded, so this bounds that work.
INDEX_PERSIST_EVERY_COMMITS: PERF-1: the ANN index is derived data, so flush persists it only every this many commits that changed it (or see INDEX_PERSIST_EVERY_CHANGES, Mnemo::flush_index, Mnemo::close). Between persists, only the catalog is rewritten.
MNEMO_META_CONTINUES: Reserved _mnemo.continues key (Phase 12.7c). Set on a thread-start marker record to point at the previous thread whose context the new thread continues from. Value: the prior thread id (string). Rendering a continued thread pulls the prior thread’s open items, next and last-N timeline entries ahead of the new thread’s own essence.
MNEMO_META_INVALID_AT: Set on the earlier record when superseded. Value: unix seconds (i64). Not interpreted by the engine’s scorer — purely for audit / history.
MNEMO_META_KEY: Upsert slot name on a remember(…, key = K) write (12.1). Combined with the memory’s scope (and agent_id for Private scope) to identify a single live memory per slot: a newer remember(key=K) supersedes the prior one.
MNEMO_META_KIND: Record kind for thread-archive turns (Phase 12.7a). Value: "turn".
MNEMO_META_KIND_ESSENCE: Marker value for _mnemo.kind on essence records (Phase 12.7b).
MNEMO_META_KIND_THREAD_LINK: Marker value for _mnemo.kind on a thread-start record (one per thread_start(new, --from old) call, carrying the link in _mnemo.continues).
MNEMO_META_KIND_TURN: Marker value for _mnemo.kind on thread-turn records.
MNEMO_META_LENGTH: Byte length of the original tool-result content (Phase 12.7a). Set alongside _mnemo.sha256 when truncation happened; absent otherwise.
MNEMO_META_NS: Namespace prefix used for every v0.8.0+ reserved metadata key (R-001.10). Keys outside this namespace are untouched user metadata.
MNEMO_META_PART: Zero-based part number for a turn that was chunked at the 8 KB boundary during ingest (Phase 12.7a). Absent → the turn fits in a single record.
MNEMO_META_REFS: Reserved _mnemo.refs key — optional array of free-form reference strings (turn ids, memory ids, commits, paths) attached to an essence record (Phase 12.7b).
MNEMO_META_ROLE: Role of the turn’s author (Phase 12.7a). Value: "user", "assistant", "system", "tool", or any other string the source transcript used.
MNEMO_META_SECTION: Reserved _mnemo.section key — which essence section an item belongs to (Phase 12.7b). Values: "timeline", "intent", "conventions", "decisions", "state", "open", "next", "lessons". Timeline entries are append-only.
MNEMO_META_SEQ: Monotonically increasing sequence number within the thread (Phase 12.7a). Value: unsigned integer, 0-based. Lets thread export reconstruct the transcript in order.
MNEMO_META_SHA256: Hex-encoded SHA-256 of the original tool-result content (Phase 12.7a). Set when the content exceeded the 16 KB truncation cap during ingest; absent otherwise.
MNEMO_META_STATUS: Reserved _mnemo.status key — current status of a status-record. Values are per-kind (Phase 12.7b: essence uses "open" and "resolved"). Timeline entries never carry this.
MNEMO_META_STATUS_HISTORY: Reserved _mnemo.status_history key — array of status transitions, each {"status": <str>, "at": <unix_secs>}. Append-only — the first entry records creation, each later entry records a transition.
MNEMO_META_SUPERSEDED_BY: Set on the earlier record when a newer remember(key=K) has replaced it. Value: the new record’s ULID string. Together with _mnemo.invalid_at this marks the record as “superseded, not current” — default recall, about and the key index skip it; history(key) walks the chain.
MNEMO_META_SUPERSEDES: Set on the newly-inserted record when a remember(key=K) supersedes an earlier record. Value: the superseded record’s ULID string.
MNEMO_META_THREAD: Thread identifier the record belongs to (Phase 12.7a). Value: a caller-supplied string, typically the Claude Code session uuid or the planner-chosen thread slug.
MNEMO_META_TURN_ID: Opaque per-turn identifier from the source transcript (Phase 12.7a). Value: typically a uuid4. Used for idempotence — ingest_thread skips any turn whose turn_id already exists on a record in the target thread.
MNEMO_META_UPDATED_AT: Set on a record each time Mnemo::update succeeds. Value: unix seconds. Lets callers distinguish an unchanged-since-creation record from one that has been edited in place. Not interpreted by the scorer.
TASK_TRANSITIONS: Task transition table (declared for Phase 13; currently unused in the engine). queued → active → ready → done | blocked, with blocked → active as the only back-edge.
THREAD_TOOL_RESULT_TRUNCATE_BYTES: Maximum content size for a tool-result turn before truncation (Phase 12.7a). Tool results frequently run tens of kilobytes; stored truncated with _mnemo.sha256 + _mnemo.length so the original can be recovered (or verified) out-of-band.
THREAD_TURN_CHUNK_BYTES: Maximum content size for a single turn record before chunking kicks in (Phase 12.7a). 8 KB per chunk keeps each record small enough that the per-record page cost dominates — and so that a thread of mostly short messages ingests as one record each.

Traits§

Embedder: Minimal trait any embedder implementation must satisfy. Returns vectors in the same order as the input texts.

Functions§

build_info: Report this build’s cryptographic backend configuration.
read_transcript: Read a transcript from any Read in the given format and return the turns in file order. Non-matching / unrecognized line types are skipped silently (Claude Code adds new type values over time).

Type Aliases§

Result: Convenience result alias used throughout the crate.