emdb 0.7.1

Why emdb

Bitcask-style architecture: one mmap-backed append-only file, sharded in-memory hash index, single-writer with multi-reader. Same shape that LMDB and redb use for reads; same shape that Riak/HaloDB use for writes.

Performance vs. peers

5 M records, 24-byte random keys, 150-byte random values — same workload shape as redb's published bench. Lower is better; numbers in milliseconds. Run on a Windows 11 NVMe consumer box. Reproduce with cargo bench --bench lmdb_style --features ttl,bench-compare.

emdb wins every aggregate-throughput column at 5 M scale, often by order-of-magnitude margins. Two honest caveats:

1. individual writes is fsync-bound. On Windows FlushFileBuffers is slow regardless of dirty-page count, and emdb pays it on every db.flush(). redb / sled batch their syncs differently and win this column. For per-record durability-on-every-write workloads, prefer db.transaction(|tx| ...) (one fsync per transaction) or db.insert_many(...) (one fsync per batch).
2. Range reads are not enabled by default. emdb's primary index is hash-keyed, so unsorted iteration is the default. Set EmdbBuilder::enable_range_scans(true) to maintain a parallel BTreeMap secondary index — see the Range scans section below.

Read scaling under fan-out

The MT random-read columns above show emdb scaling to 7.66 M reads/sec aggregate at 8 threads on a 4-core consumer box, while redb stalls near 290 K/sec past one thread. The lock-free Arc<Mmap> read path plus the 64-shard hash index keep the hot path contention-free; past core count, shared memory bandwidth is the only cap.

For more thread-count granularity, run cargo bench --bench concurrent_reads.

See docs/BENCH.md for full run instructions and tuning notes.

Status

v0.7.1. The storage engine is a Bitcask-style mmap-backed append-only log with a sharded in-memory hash index. Single-writer, multi-reader. Optional at-rest encryption (AES-256-GCM or ChaCha20-Poly1305, raw key or Argon2id passphrase). Optional sorted-iteration secondary index via EmdbBuilder::enable_range_scans(true). Pre-1.0; the API may still change before 1.0.

Installation

[dependencies]
emdb = "0.7"

Quick start

use emdb::Emdb;

let db = Emdb::open_in_memory();
db.insert("name", "emdb")?;
assert_eq!(db.get("name")?, Some(b"emdb".to_vec()));
# Ok::<(), emdb::Error>(())

Persistence

use emdb::Emdb;

let path = std::env::temp_dir().join("app.emdb");

{
    let db = Emdb::open(&path)?;
    db.insert("user:1", "james")?;
    db.flush()?;
}

let reopened = Emdb::open(&path)?;
assert_eq!(reopened.get("user:1")?, Some(b"james".to_vec()));
# let _cleanup = std::fs::remove_file(path);
# Ok::<(), emdb::Error>(())

Storage path resolution

Emdb::open(path) is the simplest entry point. For library / app authors who want platform-aware path resolution, set both app_name and database_name so your project gets a clearly-scoped subdirectory under the platform data root.

use emdb::Emdb;

// Resolves to:
//   Linux:   $XDG_DATA_HOME/hivedb-kv/sessions.emdb
//   macOS:   ~/Library/Application Support/hivedb-kv/sessions.emdb
//   Windows: %LOCALAPPDATA%\hivedb-kv\sessions.emdb
let db = Emdb::builder()
    .app_name("hivedb-kv")
    .database_name("sessions.emdb")
    .build()?;
# Ok::<(), emdb::Error>(())

app_name is a single folder name by design — path separators (/, \), .. components, and the empty string are rejected at build time. Mixing path() with any of the OS-resolution methods returns Error::InvalidConfig.

Bulk loading

For high-volume inserts, prefer insert_many — it packs every record into a single buffer and does one pwrite, which is the path that beats redb 2.4× in the bench above.

use emdb::Emdb;

let db = Emdb::open_in_memory();
let items: Vec<(String, String)> = (0..1000)
    .map(|i| (format!("k{i}"), format!("v{i}")))
    .collect();
db.insert_many(items.iter().map(|(k, v)| (k.as_str(), v.as_str())))?;
db.flush()?;
# Ok::<(), emdb::Error>(())

Transactions

use emdb::Emdb;

let db = Emdb::open_in_memory();
db.transaction(|tx| {
    tx.insert("user:1", "james")?;
    tx.insert("user:2", "alex")?;
    Ok(())
})?;

assert_eq!(db.get("user:1")?, Some(b"james".to_vec()));
# Ok::<(), emdb::Error>(())

Transactions buffer writes and commit them as one bulk insert on success. Err from the closure drops the buffered writes — nothing hits disk.

use emdb::{Emdb, Error};

let db = Emdb::open_in_memory();
let failed = db.transaction::<_, ()>(|tx| {
    tx.insert("temp", "value")?;
    Err(Error::TransactionAborted("rollback"))
});

assert!(failed.is_err());
assert_eq!(db.get("temp")?, None);
# Ok::<(), emdb::Error>(())

Durability model

Each record is framed with a CRC32. On crash recovery the engine walks records from header.tail_hint and treats the first bad CRC as the truncation point. Per-record atomicity is guaranteed; batch atomicity across a transaction is not — a crash mid-commit leaves a prefix of the batch durable. Callers that need true all-or-nothing across N records must layer that on top.

Compaction

The append-only log accumulates tombstoned and superseded records over time. Emdb::compact() rewrites the live records into a sibling file, truncates to logical size, and atomically swaps it in.

use emdb::Emdb;

let path = std::env::temp_dir().join("compact.emdb");
let db = Emdb::open(&path)?;
db.insert("k", "v")?;
db.remove("k")?;            // tombstone added to log
db.compact()?;              // log now holds only the live records
db.flush()?;
# let _cleanup = std::fs::remove_file(&path);
# let _cleanup2 = std::fs::remove_file(format!("{}.lock", path.display()));
# Ok::<(), emdb::Error>(())

Compaction is a heavier operation than flush — call it on maintenance windows, not on every write. Existing readers holding Arc<Mmap> snapshots from before the compaction continue reading from the old inode until they release; new reads see the compacted layout.

Range scans

emdb's primary index is a sharded hash, so unsorted iteration is the default. To support range / prefix queries, opt in at open time with EmdbBuilder::enable_range_scans(true). The engine maintains a parallel BTreeMap<Vec<u8>, u64> secondary index per namespace; range queries hit the BTreeMap and resolve values through the mmap.

use emdb::Emdb;

let db = Emdb::builder()
    .enable_range_scans(true)
    .build()?;

db.insert("user:001", "alice")?;
db.insert("user:002", "bob")?;
db.insert("session:abc", "token")?;

// Half-open range: ["user:", "user;").
let users = db.range(b"user:".to_vec()..b"user;".to_vec())?;
assert_eq!(users.len(), 2);
assert_eq!(users[0].0, b"user:001");
assert_eq!(users[1].0, b"user:002");

// Prefix shorthand: builds the half-open `[prefix, prefix++)` range.
let same = db.range_prefix(b"user:")?;
assert_eq!(users.len(), same.len());
# Ok::<(), emdb::Error>(())

Cost: one Vec<u8> clone of the key per insert plus the BTreeMap node overhead — roughly doubles in-memory index size for a typical workload. Calling db.range(...) without enabling this at open time returns Error::InvalidConfig.

Namespace::range and Namespace::range_prefix give the same view scoped to a named namespace.

Cargo features

• ttl (default) — per-record expiration and default_ttl.
• nested — dotted-prefix group operations and Focus handles.
• encrypt — AES-256-GCM + ChaCha20-Poly1305 at-rest encryption with raw-key or Argon2id-derived passphrase. Pulls in aes-gcm, chacha20poly1305, argon2, rand_core.
• bench-compare — pulls in redb and sled for the comparative bench (dev-only; not for production builds).
• bench-rocksdb / bench-redis — additional comparative bench peers.

Concurrency

Emdb is Send + Sync and cheap to clone — clones share the same underlying engine via Arc. Pass clones across threads instead of synchronising access to a single handle.

Reads scale. A 64-shard sharded RwLock<HashMap> index plus zero-copy slices from a shared Arc<Mmap> keep the hot path contention-free: the comparative bench above hits 7.66 M reads/sec aggregate at 8 threads on a 4-core consumer box.

Writes are single-writer. All writers serialise on one mutex that covers the encode-and-pwrite step. This matches the model used by LMDB, redb, BoltDB, and most of the embedded-KV ecosystem (multi-writer concurrency requires either a recovery model with sentinel records or per-thread log segments — both queued for v1.0). High-throughput producer workloads should batch through db.insert_many(...) or db.transaction(|tx| ...), which amortise the writer-mutex acquire across many records.

use std::sync::Arc;
use std::thread;

use emdb::Emdb;

let db = Arc::new(Emdb::open_in_memory());
db.insert("counter", "0")?;

let mut workers = Vec::new();
for i in 0_u32..4 {
    let db = Arc::clone(&db);
    workers.push(thread::spawn(move || {
        let _ = db.insert(format!("k{i}"), format!("v{i}"));
    }));
}

for worker in workers {
    let _ = worker.join();
}

assert!(db.len()? >= 4);
# Ok::<(), emdb::Error>(())

TTL example

# #[cfg(feature = "ttl")]
# {
use std::time::Duration;

use emdb::{Emdb, Ttl};

let db = Emdb::builder()
    .default_ttl(Duration::from_secs(30))
    .build()?;
db.insert_with_ttl("session", "token", Ttl::Default)?;
assert!(db.ttl("session")?.is_some());
# }
# Ok::<(), emdb::Error>(())

Nested example

# #[cfg(feature = "nested")]
# {
use emdb::Emdb;

let db = Emdb::open_in_memory();
let product = db.focus("product");
product.set("name", "phone")?;
product.set("price", "799")?;

assert_eq!(product.get("name")?, Some(b"phone".to_vec()));
assert_eq!(db.group("product")?.count(), 2);
# }
# Ok::<(), emdb::Error>(())

Encryption

# #[cfg(feature = "encrypt")]
# {
use emdb::Emdb;

let path = std::env::temp_dir().join("encrypted.emdb");
let _ = std::fs::remove_file(&path);
let _ = std::fs::remove_file(format!("{}.lock", path.display()));

let db = Emdb::builder()
    .path(path.clone())
    .encryption_passphrase("correct horse battery staple")
    .build()?;
db.insert("k", "v")?;
db.flush()?;
drop(db);

let reopened = Emdb::builder()
    .path(path.clone())
    .encryption_passphrase("correct horse battery staple")
    .build()?;
assert_eq!(reopened.get("k")?, Some(b"v".to_vec()));

# drop(reopened);
# let _ = std::fs::remove_file(&path);
# let _ = std::fs::remove_file(format!("{}.lock", path.display()));
# }
# Ok::<(), emdb::Error>(())

The cipher is creation-time-fixed and stored in the header — reopens auto-dispatch. Wrong passphrase surfaces as Error::EncryptionKeyMismatch from a verification block check, not from a corrupted-data read. Three offline admin functions (Emdb::enable_encryption, disable_encryption, rotate_encryption_key) let you toggle encryption or rotate keys on an existing file via atomic rewrite-then-rename, leaving an .encbak backup.

Goals

• Embedded-first — runs in-process; no separate server, no network.
• High performance — zero-copy reads, allocation-free hot paths, cache-friendly layout, batched writes amortise lock and syscall costs.
• Safe — strict clippy profile, no unwrap in library code, every unsafe block documented with its invariant.
• Small footprint — minimal dependency graph, fast compile times.
• Portable — Linux, macOS, Windows on x86_64 and ARM64.

Non-goals

• Client/server operation (use a dedicated DBMS for that).
• SQL.
• Distributed replication.
• Range scans on a single namespace (the index is hash-based; insert a prefix-sorted secondary structure on top if you need ranges).

Benchmarking

emdb ships Criterion benches. The comparative bench can include redb, sled, optionally RocksDB, and optionally Redis.

• Core: benches/kv.rs
• Comparative: benches/comparative.rs

# Just emdb
cargo bench --bench kv --features ttl

# emdb vs sled vs redb
cargo bench --bench comparative --features ttl,bench-compare

# Add RocksDB
cargo bench --bench comparative --features ttl,bench-compare,bench-rocksdb

# Add Redis (set EMDB_REDIS_URL first)
$env:EMDB_REDIS_URL = "redis://127.0.0.1/"
cargo bench --bench comparative --features ttl,bench-compare,bench-redis

Full bench workflow and tuning notes: docs/BENCH.md.

Related projects

emdb is the Rust implementation. Implementations in other languages (Go, C, etc.) are planned and will live under their own repositories.

License

Licensed under the Apache License, Version 2.0.

Copyright © 2026 James Gober.