Crate armdb

Expand description

Embedded key-value storage engine optimized for NVMe.

Single process, multi-threaded. Sync read/write API. Each tree/map owns its storage — one tree = one database directory.

§Durability backends

Collections are generic over D: Durability (default: Bitcask):

Bitcask — append-only log + compaction. Good for general workloads.
FixedStore (Fixed) — fixed-slot pwrite, no compaction. Optimized for frequent updates of fixed-size values (counters, metrics, sessions).

FixedTree and FixedMap are type aliases for ConstTree<K, V, Fixed> and ConstMap<K, V, Fixed>. ZeroTree and ZeroMap also accept Fixed as backend.

§Collection types

Type	Index	Values	Ordered	FixedStore	Feature
`ConstTree`	SkipList	inline `[u8; V]`	yes	yes	(default)
`ConstMap`	HashMap	inline `[u8; V]`	no	yes	(default)
`TypedTree`	SkipList	typed `T` (in-memory, `TypedRef`)	yes	no	`typed-tree`
`TypedMap`	HashMap	typed `T` (in-memory, `TypedRef`)	no	no	`typed-tree`
`ZeroTree`	SkipList	zerocopy `T` (in-memory)	yes	yes	`typed-tree`
`ZeroMap`	HashMap	zerocopy `T` (in-memory)	no	yes	`typed-tree`
`VarTree`	SkipList	`ByteView` (disk + cache)	yes	no	`var-collections`
`VarMap`	HashMap	`ByteView` (disk + cache)	no	no	`var-collections`
`VarTypedTree`	SkipList	typed `T` (disk + cache)	yes	no	`typed-tree` + `var-collections`
`VarTypedMap`	HashMap	typed `T` (disk + cache)	no	no	`typed-tree` + `var-collections`

§Usage

// Bitcask backend (default)
let tree = ConstTree::<[u8; 16], 64>::open("data/users", Config::balanced().build())?;

// FixedStore backend
let tree = FixedTree::<[u8; 16], 64>::open("data/counters", FixedConfig::balanced().build())?;

tree.put(&key, &value)?;
let val = tree.get(&key);
tree.close()?;

§Durability

§Bitcask

Writes are buffered in memory (write_buffer_size per shard, default 1 MB). put() returns as soon as the entry is copied into the buffer and the in-memory index is updated — no disk I/O on the write path.

This means unflushed data is lost on crash. To control durability:

flush_buffers() — flush write buffers to disk (no fsync)
close() — flush + fsync + write hint files

§FixedStore

Each put() does a pwrite through the page cache (~200ns). fdatasync is called in batches (configurable interval/count), after releasing the shard lock. With enable_fsync = true, an additional per-write fdatasync runs inside the shard lock for immediate durability. No compaction, no write amplification, 1x space usage.

§Thread safety

All tree/map types are Send + Sync. Share via Arc for concurrent access:

let tree = Arc::new(ConstTree::<[u8; 8], 8>::open("data/mydb", Config::balanced().build())?);
let t = tree.clone();
std::thread::spawn(move || { t.put(&key, &value).unwrap(); });

Reads:

ConstTree / ZeroTree / TypedTree — lock-free: the index is a lock-free SkipList (TypedTree resolves the value through a seize RCU guard, no mutex).
VarTree — lock-free SkipList lookup; on a block-cache miss a brief shard lock fetches the file handles, then the pread runs outside the lock.
ConstMap / ZeroMap / TypedMap — a brief per-shard mutex guards the HashMap index. Values are inline for Const/Zero; TypedMap reads a pointer under the lock, then accesses the value lock-free via a seize guard.
VarMap / VarTypedMap — brief per-shard mutex on the index; a pread on a cache miss runs outside the lock.

Writes acquire the target key’s per-shard mutex (write buffer + index update), so writes to different shards never contend on the buffer or durability I/O. Tree writes additionally link the new node into the shared SkipList under its internal linking lock — briefly contended across shards, but the critical section is O(node height), not the full O(log n) search.

§Encryption at rest (feature `encryption`)

Page-level AES-256-GCM encryption. Key from env or direct config:

let key = PageCipher::key_from_env("ARMDB_KEY")?;
let config = Config::balanced().encryption_key(Some(key)).build();
let tree = ConstTree::<[u8; 16], 64>::open("data/encrypted", config)?;
// all data is transparently encrypted on disk

§Write hooks (secondary indexes)

Generic WriteHook<K> / TypedWriteHook<K, T> parameter for synchronous write and init notifications. Zero overhead when unused (NoHook default).

on_write — fires on every put/insert/delete/cas/compare_delete/update/fetch_update, including mutations inside atomic() (buffered under the shard lock and replayed once it is released, in application order). If the closure returns Err, hooks still fire for operations that were applied. On Const*/Zero* collections replay happens after the sync()? seam; Var/Typed collections have no per-write sync.
on_init — fires once per live entry during collection open (after recovery). Enable via NEEDS_INIT = true. See [armdb/docs/hooks.md] for details.
NEEDS_OLD_VALUE — only affects VarTree/VarMap (skips disk I/O for old value when false). In-memory collections (Const/Typed/Zero) always provide the old value at zero cost.

impl WriteHook<[u8; 16]> for MyIndex {
    const NEEDS_OLD_VALUE: bool = true;
    const NEEDS_INIT: bool = true;

    fn on_write(&self, key: &[u8; 16], old: Option<&[u8]>, new: Option<&[u8]>) {
        // update secondary index incrementally
    }

    fn on_init(&self, key: &[u8; 16], value: &[u8]) {
        // populate secondary index at startup
    }
}

let tree = ConstTree::<[u8; 16], 64, MyIndex>::open_hooked("data/indexed", config, my_index)?;
tree.migrate(|_, _| MigrateAction::Keep)?; // triggers on_init for all entries

§Compaction

Compaction is not automatic. Dead bytes accumulate as entries are overwritten or deleted. Use Compactor to run it in the background:

use std::sync::Arc;
use std::time::Duration;

let tree = Arc::new(ConstTree::<[u8; 16], 64>::open("data/mydb", Config::balanced().build())?);
let t = tree.clone();
let _compactor = Compactor::start(move || t.compact(), Duration::from_secs(60));

Or call tree.compact() manually when needed.

§Iteration

All Tree types provide iter(), range(), and prefix_iter() methods. Map types (HashMap index) do not support iteration.

Each method returns a dedicated iterator implementing Iterator + DoubleEndedIterator:

Tree	Iterator	Item
`ConstTree`	`ConstIter`	`(K, [u8; V])` — copy
`VarTree`	`VarIter`	`(K, ByteView)` — RC, possible disk I/O
`TypedTree`	`TypedIter`	`(K, &T)` — reference, zero I/O
`ZeroTree`	`ZeroIter`	`(K, T)` — copy, zero I/O

Method	Description
`iter()`	All entries in index order
`range(start, end)`	Entries in `[start, end)` — start inclusive, end exclusive
`range_bounds(start, end)`	Entries with custom `Bound` — `Included`, `Excluded`, or `Unbounded`
`prefix_iter(prefix)`	Entries whose key starts with `prefix`

for (key, value) in tree.iter() { /* ... */ }

let latest = tree.prefix_iter(&user_id).take(20).collect::<Vec<_>>();

for (key, value) in tree.range(&start_key, &end_key) { /* ... */ }

// Custom bounds: (5, 10] — exclude 5, include 10
use std::ops::Bound;
let entries: Vec<_> = tree.range_bounds(
    Bound::Excluded(&5u64.to_be_bytes()),
    Bound::Included(&10u64.to_be_bytes()),
).collect();

// DoubleEndedIterator — .rev() or .next_back()
let oldest = tree.prefix_iter(&user_id).rev().take(10);

§Complexity

Operation	Complexity	Notes
`next()`	O(1)	follows SkipList level-0 forward pointer
`next_back()`	O(log n)	calls `find_last_lt()` — SkipList search from top
`iter()` / `range()` / `prefix_iter()` setup	O(log n)	initial SkipList search

VarIter: both next() and next_back() may additionally perform a pread on block-cache miss. Use warmup() to pre-populate the cache.

§Weakly-consistent semantics

Iterators do not create a snapshot. They are weakly-consistent:

Concurrent inserts/updates may be visible during iteration
Deleted entries (marked nodes) are automatically skipped
The seize guard prevents memory reclamation for the lifetime of the iterator — no use-after-free, but the index is not frozen

§Ordering: `Config::reversed`

Config::reversed controls the SkipList comparator direction.

`reversed`	`iter()` / `prefix_iter()`	`.rev()` / `next_back()`
`true` (default)	DESC (newest first)	ASC (oldest first)
`false`	ASC (oldest first)	DESC (newest first)

// reversed=true (default) — DESC: идеально для "newest first" пагинации
let tree = ConstTree::<[u8; 16], 64>::open("data/posts", Config::balanced().build())?;
let latest = tree.prefix_iter(&user_id).take(20);    // newest 20
let oldest = tree.prefix_iter(&user_id).rev().take(5); // oldest 5

// reversed=false — ASC: естественный порядок ключей
let config = Config::balanced().reversed(false).build();
let tree = ConstTree::<[u8; 16], 64>::open("data/logs", config)?;
for (key, value) in tree.iter() { /* ascending order */ }

Keys are stored on disk as-is. reversed can be changed between restarts without migration — it only affects in-memory index ordering.

FixedStore collections take the same knob via FixedConfig::reversed (default true → DESC); it behaves identically and is likewise in-memory only.

§Prefix sharding

let config = Config::balanced().shard_prefix_bits(32).build();
let tree = ConstTree::<[u8; 16], 64>::open("data/users", config)?;

§Caveats

Iterators are weakly-consistent, not snapshot. Concurrent inserts and updates may be visible during iteration. Deleted entries are skipped. The seize guard prevents use-after-free but does not freeze the index.
CAS/update holds shard lock during possible disk I/O. VarTree::cas, VarTree::update, VarMap::cas, and VarMap::update read the current value under the shard mutex. On a block-cache miss this issues a pread — blocking all writes to that shard until the read completes. Pre-warm the cache or size it to cover the working set.
migrate() on HashMap trees allocates O(keys/shards) memory. ConstMap::migrate and VarMap::migrate collect all shard keys into a Vec before iterating. For very large shards this causes a transient memory spike. SkipList trees (ConstTree, VarTree) are not affected.

§Shutdown

// 1. Stop background tasks first
compactor.stop();
// 2. Close the tree (writes hint files, flushes, fsyncs)
Arc::try_unwrap(tree).expect("no other references").close()?;

If close() is not called (e.g. the tree is dropped via Arc::drop), Shard::Drop still flushes write buffers, fsyncs, and writes hint files automatically — no data loss and no slow recovery on next startup.

§Features

typed-tree — enables TypedTree, TypedMap, ZeroTree, ZeroMap, Codec and codec implementations
var-collections — enables VarTree, VarMap (variable-length values with LRU cache)
rapira-codec — RapiraCodec for rapira serialization (implies typed-tree)
bytemuck-codec — BytemuckCodec / BytemuckSliceCodec (implies typed-tree)
bitcode-codec — [BitcodeCodec] (implies typed-tree)
postcard-codec — PostcardCodec for postcard/serde serialization (implies typed-tree)
encryption — AES-256-GCM page-level encryption at rest
replication — leader/follower log-shipping replication
armour — integration with armour ecosystem: Db, schema-versioned migrations, binary RPC server (TCP/UDS). See armour module docs.
rpc — TCP/UDS RPC server (implies armour)
solana — Solana pubkey/signature key types (implies armour)
hot-path-tracing — per-operation tracing::trace! calls

Internal/test-only: parking_lot, loom, proptest.

Re-exports§

pub use codec::Codec;
pub use codec::RapiraCodec;
pub use codec::ZerocopyCodec;
pub use codec::BytemuckCodec;
pub use codec::BytemuckSliceCodec;
pub use codec::BytemuckVec;
pub use compaction::Compactor;
pub use fixed::FixedConfig;
pub use fixed::FixedMap;
pub use fixed::FixedTree;
pub use fixed::FixedWorkload;
pub use armour_core;

Modules§

armour: Integration with the armour ecosystem (feature armour).
codec
compaction
docs: In-depth documentation for armdb.
durability: Durability trait — abstracts Bitcask (append-only log) and FixedStore (fixed-slot pwrite) backends behind a common interface.
fixed
fixed_replication: Replication for the FixedStore backend.
replication
schema: Schema metadata, relation registry, CI validator and ER export. Spec: docs/superpowers/specs/26-06-11-armdb-schema-metadata.md

Macros§

const_map: Expands to ConstMap<<V as CollectionMeta>::SelfId, { size_of::<V>() } [, H, D]>.
const_tree: Expands to ConstTree<<V as CollectionMeta>::SelfId, { size_of::<V>() } [, H, D]>.
impl_key_bytemuck: Implement Key for a type that derives bytemuck::{Pod, Zeroable}.
impl_key_zerocopy: Implement Key for a type that derives zerocopy::{FromBytes, IntoBytes, Immutable}.
typed_map: Expands to TypedMap<<V as CollectionMeta>::SelfId, V, C [, H]>.
typed_tree: Expands to TypedTree<<V as CollectionMeta>::SelfId, V, C [, H]>.
var_map: Expands to VarMap<<V as CollectionMeta>::SelfId [, H]>.
var_tree: Expands to VarTree<<V as CollectionMeta>::SelfId [, H]>.
var_typed_map: Expands to VarTypedMap<<V as CollectionMeta>::SelfId, V, C [, H]>.
var_typed_tree: Expands to VarTypedTree<<V as CollectionMeta>::SelfId, V, C [, H]>.
zero_map: Expands to ZeroMap<<V as CollectionMeta>::SelfId, { size_of::<V>() }, V [, H, D]>.
zero_tree: Expands to ZeroTree<<V as CollectionMeta>::SelfId, { size_of::<V>() }, V [, H, D]>.

Structs§

ByteView: Immutable byte slice: inline up to 20 bytes, heap-allocated with reference counting for larger. Total size: 24 bytes on 64-bit systems, alignment: 8.
CacheConfig: Configuration for the value cache used by VarTree.
Config: Database configuration.
ConstIter: Iterator over entries in a ConstTree. Returned by iter(), range(), and prefix_iter().
ConstMap: A map with fixed-size keys and values. All values are stored inline in a per-shard HashMap. Reads never touch disk — zero I/O reads. O(1) lookup instead of O(log n) SkipList. Ordered iteration is available when opened with Config::iterable(true) via iter_view; otherwise use ConstTree for prefix/range scans.
ConstMapIterView: View over an iterable ConstMap; exposes ordered iteration over (K, [u8; V]) entries and cheap keys-only iteration. Obtained via ConstMap::iter_view.
ConstMapShard: Handle for atomic multi-key operations within a single shard. Obtained via ConstMap::atomic. The shard + index locks are held for the lifetime of this struct — keep the closure short.
ConstShard: Handle for atomic multi-key operations within a single shard. Obtained via ConstTree::atomic. The shard lock is held for the lifetime of this struct — keep the closure short.
ConstTree: A tree with fixed-size keys and values. All values are stored inline in SkipList nodes. Reads never touch disk — zero I/O reads.
Flusher: Background periodic-flush handle. Stops and joins its thread on stop() or Drop.
KeyPage: Keys-only pagination result, returned by keys_paginate. Carries no values, so it is cheap to produce even for VarMap/VarTypedMap (no off-disk read).
NoHook: Default no-op hook. All branches are eliminated at compile time.
Page: One page of key/value entries returned by paginate.
ShutdownSignal: Shared shutdown signal for coordinating graceful termination of background workers (compactor, replication, RPC).
TreeMeta: Metadata about a named tree/map collection.
TypedIter: Iterator over entries in a TypedTree. Returned by iter(), range(), and prefix_iter().
TypedMap: A map with fixed-size keys and typed values T. Values are encoded via a Codec for disk persistence but stored as T in memory — reads never touch disk and return TypedRef<T> (guard-protected reference).
TypedMapIterView: View over an iterable TypedMap; exposes ordered iteration yielding guard-protected TypedRef values.
TypedMapShard: Handle for atomic multi-key operations within a single shard. Obtained via TypedMap::atomic. The shard + index locks are held for the lifetime of this struct — keep the closure short.
TypedRef: Guard-protected reference to a typed value inside a TypedTree.
TypedShard: Handle for atomic multi-key operations within a single shard. Obtained via TypedTree::atomic. The shard lock is held for the lifetime of this struct — keep the closure short.
TypedTree: A tree with fixed-size keys and typed values T. Values are encoded via a Codec for disk persistence but stored as T in memory — reads never touch disk and return TypedRef<T> (guard-protected reference).
VarIter: Iterator over entries in a VarTree. Returned by iter(), range(), and prefix_iter().
VarMap
VarMapShard: Handle for atomic multi-key operations within a single shard. Obtained via VarMap::atomic. The shard + index locks are held for the lifetime of this struct — keep the closure short.
VarShard: Handle for atomic multi-key operations within a single shard. Obtained via VarTree::atomic. The shard lock is held for the lifetime of this struct — keep the closure short.
VarTree: A tree with fixed-size keys and variable-length values. Values are stored as ByteView (inline ≤20 bytes, heap with ref counting for larger). Disk reads are cached at 4096-byte block granularity via BlockCache.
VarTypedIter: Iterator over entries in a VarTypedTree. Returned by iter(), range(), range_bounds(), and prefix_iter(). Wraps VarIter and decodes each value via the codec. Entries with decode errors are skipped (see tracing::debug!).
VarTypedMap: A map with fixed-size keys and typed values T. Values are encoded via a Codec and stored on disk (variable length), with a block cache (≤ 8 KB values) and a value cache (> 8 KB values) for reads. Uses per-shard HashMap for O(1) lookup. Ordered scans are available when opened with iterable(true) via iter_view. Use VarTypedTree for prefix/range scans without the iterable flag.
VarTypedMapIterView: View over an iterable VarTypedMap; exposes ordered iteration yielding freshly decoded T values.
VarTypedMapShard: Handle for atomic multi-key operations on a single shard of a VarTypedMap. Obtained via VarTypedMap::atomic. The shard + index locks are held for the lifetime of this struct — keep the closure short.
VarTypedShard: Handle for atomic multi-key operations within a single shard. Obtained via VarTypedTree::atomic. The shard lock is held for the lifetime of this struct — keep the closure short.
VarTypedTree: A tree with fixed-size keys and typed values T. Values are encoded via a Codec and stored on disk (variable length), with a block cache (≤ 8 KB values) and a value cache (> 8 KB values) for reads — unlike TypedTree which keeps values in memory.
ZeroIter: Iterator over entries in a ZeroTree. Wraps ConstIter and converts [u8; V] values to T via zerocopy.
ZeroMap: A map with fixed-size keys and zerocopy-compatible typed values.
ZeroMapIterView: View over an iterable ZeroMap; exposes ordered iteration over (K, T) entries and cheap keys-only iteration. Obtained via ZeroMap::iter_view.
ZeroMapShard: Handle for atomic multi-key operations within a single shard. Obtained via ZeroMap::atomic.
ZeroShard: Handle for atomic multi-key operations within a single shard. Obtained via ZeroTree::atomic.
ZeroTree: A tree with fixed-size keys and zerocopy-compatible values.

Enums§

Applied: What actually happened to one key — a self-sufficient application event channel (old + new are present here, so duplicating an on_write hook is optional).
BatchWrite: What a batch update closure decided for one key. The absence/deletion policy lives entirely in the closure: it receives Option<&T> (None = key absent) and returns one of these.
DbError
IoBackend: Bitcask flush-path I/O strategy. Tunable; in-memory effect only — the on-disk format is identical regardless. Default: Pwrite.
MigrateAction: Action returned by the migrate() callback for each entry.
SchemaMismatchKind: Specific kind of schema mismatch (see DbError::SchemaMismatch).
Workload: Bitcask deployment-intent presets for Config. Pick one as the builder start point: Config::for_workload(Workload::…) or a sugar method.

Traits§

CollectionMeta: Trait for types that carry enough metadata to describe an armdb collection.
Key: Key metadata trait — describes key encoding for armdb collections.
Location: Marker trait for disk location types stored in SkipList nodes. DiskLoc (12 bytes) for Bitcask, u32 (4 bytes) for FixedStore slot_id.
TypedWriteHook: Typed write hook for TypedTree.
WriteHook: Trait for receiving write notifications from tree/map operations.

Type Aliases§

DbResult

Crate armdb

Crate armdb Copy item path

§Durability backends

§Collection types

§Usage

§Durability

§Bitcask

§FixedStore

§Thread safety

§Encryption at rest (feature encryption)

§Write hooks (secondary indexes)

§Compaction

§Iteration

§Complexity

§Weakly-consistent semantics

§Ordering: Config::reversed

§Prefix sharding

§Caveats

§Shutdown

§Features

Re-exports§

Modules§

Macros§

Structs§

Enums§

Traits§

Type Aliases§

Crate armdb

§Encryption at rest (feature `encryption`)

§Ordering: `Config::reversed`