shodh-redb

Embedded vector database for edge AI. IVF-PQ search, blob storage, TTL, causal tracking, CDC -- ACID transactions, single file, no_std core, zero dependencies.

95% recall@1 at 1,000+ QPS on SIFT1M. Single-threaded. 8 GB RAM. No GPU.

[dependencies]

shodh-redb = "0.3"

Why shodh-redb?

Every edge vector database (sqlite-vec, LanceDB, Qdrant) bolts vector search onto a storage engine that wasn't designed for it. shodh-redb builds both from the same B-tree:

• Vector search -- IVF-PQ with integer ADC, SIMD distance, and ACID-transactional index updates
• Blob storage -- content-addressed dedup, chunked writes, causal lineage graph
• TTL tables -- per-key expiration for sessions, caches, and ephemeral agent state
• CDC -- row-level change capture for replication and reactive pipelines
• Composite queries -- ranked fusion across semantic, temporal, and causal signals
• Runs anywhere -- no_std core compiles to wasm32, ARM Cortex, RISC-V. No allocator, no filesystem required

One binary. One file. ACID crash-safe.

Benchmarks (SIFT1M)

1M vectors, 128 dimensions, Euclidean distance. Single-threaded on Intel i7-1355U, 8 GB RAM.

Index build: 30.8s (single-threaded, 256 clusters, M=16 subquantizers).

Full results and reproduction steps: BENCHMARKS.md

Quick start

use shodh_redb::{Database, TableDefinition, ReadableDatabase, ReadableTable};

const TABLE: TableDefinition<&str, u64> = TableDefinition::new("agent_state");

let db = Database::create("agent.redb")?;

let wtxn = db.begin_write()?;
{
    let mut table = wtxn.open_table(TABLE)?;
    table.insert("task_count", &42u64)?;
}
wtxn.commit()?;

let rtxn = db.begin_read()?;
let table = rtxn.open_table(TABLE)?;
assert_eq!(table.get("task_count")?.unwrap().value(), 42);

Vector search

Native IVF-PQ index with integer ADC, residual encoding, and per-vector metadata filtering.

use shodh_redb::{FixedVec, DistanceMetric, nearest_k, TableDefinition};

// Brute-force top-k (small datasets)
const EMBEDDINGS: TableDefinition<u64, FixedVec<384>> = TableDefinition::new("memory");
let results = nearest_k(
    table.iter()?.map(|r| { let (k, v) = r.unwrap(); (k.value(), v.value().to_vec()) }),
    &query, 10, |a, b| DistanceMetric::Cosine.compute(a, b),
);

Three quantization levels:

For datasets beyond brute-force range, use the IVF-PQ index -- inverted file with product quantization, integer asymmetric distance tables, and optional reranking from stored raw vectors.

Blob store

Content-addressed storage with SHA-256 dedup, chunked streaming writes, and causal lineage tracking.

let blob_id = wtxn.store_blob(data, ContentType::ImagePng, "screenshot", &opts)?;
let reader = rtxn.blob_reader(blob_id)?;

Per-blob tags, namespaces, temporal indexing, and causal graph with BFS traversal.

TTL tables

Per-key expiration. Lazy filtering on read, bulk purge on demand.

let mut table = wtxn.open_table(CACHE)?;
table.insert_with_ttl(&"result_abc", data, Duration::from_secs(1800))?;
table.purge_expired()?;

Merge operators

Atomic read-modify-write without manual locking.

table.merge(&"tool_calls", &1u64.to_le_bytes(), &NumericAdd)?;

Built-in: NumericAdd, SaturatingAdd, FloatAdd, NumericMax, NumericMin, BitwiseOr, BytesAppend. Implement MergeOperator for custom logic.

CDC (change data capture)

Row-level change streaming with cursors, retention pruning, and HLC timestamps.

Composite queries

Multi-signal ranked retrieval fusing semantic similarity, temporal recency, causal proximity, and namespace/tag filtering into a single scored result set.

Multimap tables

Multiple values per key for tag indices, inverted lookups, and many-to-many relationships.

Group commit

Batched fsync for write-heavy workloads. Multiple operations share a single transaction and disk sync.

Performance

SIMD acceleration

Distance functions use hand-written AVX2 intrinsics on x86_64 with runtime feature detection and scalar fallback.

On no_std or non-x86 targets, scalar code is structured for LLVM auto-vectorization.

Zero-copy serialization

On little-endian targets (x86, ARM LE, RISC-V LE), FixedVec and DynVec use bulk copy_nonoverlapping -- the f32 memory layout matches the on-disk LE format directly.

no_std support

The core library compiles with #![no_std] (disable the std feature). Vector types, distance functions, quantization, IVF-PQ indexing, and the B-tree engine all work without the standard library. Targets: wasm32, ARM Cortex-M, RISC-V bare metal.

The std feature adds file backends, TTL tables, group commit, and runtime SIMD dispatch.

Feature flags

Architecture

                    +--------------------+
                    |   Database API     |
                    |  (typed, ACID)     |
                    +--------+-----------+
                             |
              +--------------+--------------+
              |              |              |
        +-----+------+ +----+-----+ +------+------+
        | Key-Value  | | Blob     | | Vector      |
        | Tables     | | Store    | | Index       |
        | TTL, Merge | | CDC,     | | IVF-PQ,     |
        | Multimap   | | dedup,   | | int ADC,    |
        |            | | causal   | | SIMD        |
        +-----+------+ +----+-----+ +------+------+
              |              |              |
              +--------------+--------------+
                             |
                   +---------+---------+
                   | B-tree Engine     |
                   | (COW pages, MVCC, |
                   |  crash-safe,      |
                   |  no_std core)     |
                   +-------------------+

Credits

Core B-tree page store and crash recovery derived from redb. All extensions -- vector indexing, blob store, TTL, merge operators, HLC, CDC, composite queries, group commit, SIMD distance, quantization -- are original work.

License

Apache License, Version 2.0

Copyright 2025-2026 Varun Sharma