memvid-core

A crash-safe, deterministic, single-file AI memory system.

About

memvid-core is the Rust library powering Memvid - a memory system for AI applications that stores everything in a single .mv2 file. No databases, no sidecar files, no external dependencies. Each memory file is completely self-contained with embedded WAL, full-text search indices, vector embeddings, and time-indexed storage.

Why memvid-core?

• Single File - Everything in one portable .mv2 file
• Crash-Safe - Embedded WAL ensures durability and automatic recovery
• Hybrid Search - Full-text (Tantivy BM25) + semantic vector search (HNSW)
• Deterministic - Same operations produce identical bytes
• Fast - Sub-millisecond search across 50K documents

Performance

Installation

[dependencies]
memvid-core = "2.0.1"

Feature Flags

Enable features based on your needs:

[dependencies]
memvid-core = { version = "2.0.1", features = ["lex", "vec", "parallel_segments"] }

Quick Start

use memvid_core::{Memvid, MemvidResult};

fn main() -> MemvidResult<()> {
    // Create a new memory file
    let mut mem = Memvid::create("knowledge.mv2")?;

    // Add content
    mem.put_text("Important meeting notes from Q4 planning session.")?;
    mem.put_text("Project roadmap includes AI features for 2025.")?;

    // Search
    let results = mem.find("AI features", 5)?;
    for result in results {
        println!("Score: {:.2} - {}", result.score, result.snippet);
    }

    // Seal when done
    mem.seal()?;

    Ok(())
}

Feature Flag Compatibility

When opening an existing .mv2 file, you must enable the same features that were used during creation. Mismatched features cause deserialization errors.

// If the file was created with the CLI (which enables all features),
// you need all features to open it:

[dependencies]
memvid-core = { version = "2.0.1", features = [
    "lex",
    "vec",
    "temporal_track",
    "parallel_segments"
]}

Check what features a file needs:

memvid stats file.mv2 --json | jq '.indexes'
# Shows: lexical, vector, time index presence

Architecture

┌─────────────────────────────────────┐
│      .mv2 File (Single File)        │
├─────────────────────────────────────┤
│ Header (4KB)                        │
│ - Magic bytes, version, metadata    │
├─────────────────────────────────────┤
│ Embedded WAL (1-64MB)               │
│ - Crash recovery, append-only log   │
├─────────────────────────────────────┤
│ Data Segments                       │
│ - Compressed frames, embeddings     │
├─────────────────────────────────────┤
│ Tantivy Index (Full-Text)           │
├─────────────────────────────────────┤
│ HNSW Vector Index (Semantic)        │
├─────────────────────────────────────┤
│ Time Index                          │
├─────────────────────────────────────┤
│ Table of Contents (Footer)          │
└─────────────────────────────────────┘

Logging

memvid-core uses the log and tracing crates. Configure logging in your application:

// Using env_logger
env_logger::Builder::from_env(
    env_logger::Env::default().default_filter_or("warn")
).init();

// Using tracing
tracing_subscriber::fmt()
    .with_env_filter("memvid_core=warn")
    .init();

Recommended levels:

• error - Production (critical failures only)
• warn - Production with warnings
• info - Development
• debug/trace - Debugging

Examples

Hybrid Search

use memvid_core::{Memvid, SearchMode};

let mem = Memvid::open("knowledge.mv2")?;

// Auto mode (lexical + semantic reranking)
let results = mem.search("machine learning", SearchMode::Auto, 10)?;

// Lexical only (BM25)
let results = mem.search("error", SearchMode::Lex, 10)?;

// Semantic only (vector similarity)
let results = mem.search("AI concepts", SearchMode::Sem, 10)?;

Timeline Queries

use memvid_core::Memvid;

let mem = Memvid::open("journal.mv2")?;

// Get recent entries
let entries = mem.timeline(None, None, 20)?;

// Filter by time range
let entries = mem.timeline(
    Some(1706745600),  // start timestamp
    Some(1706832000),  // end timestamp
    100
)?;

Parallel Ingestion

use memvid_core::{Memvid, BuildOpts};

let mut mem = Memvid::create("large-dataset.mv2")?;

let opts = BuildOpts {
    parallel: true,
    threads: 8,
    embeddings: true,
    compression: true,
};

mem.put_file("large-document.pdf", opts)?;

Documentation

• API Documentation
• GitHub Repository
• Full Documentation

Security

• Runs fully offline by default
• OS-level file locking prevents concurrent writes
• Ed25519 signed tickets for access control
• Single-file design reduces attack surface

License

Dual-licensed under MIT or Apache 2.0. Choose either for your use.

• MIT License
• Apache License 2.0

Acknowledgments

Built with:

• Tantivy - Full-text search
• HNSW - Vector similarity
• ONNX Runtime - ML inference