Crate armdb

Expand description

Embedded key-value storage engine optimized for NVMe.

Single process, multi-threaded. Sync read/write API. Each tree/map owns its storage — one tree = one database directory.

§Durability backends

Collections are generic over D: Durability (default: Bitcask):

Bitcask — append-only log + compaction. Good for general workloads.
FixedStore (Fixed) — fixed-slot pwrite, no compaction. Optimized for frequent updates of fixed-size values (counters, metrics, sessions).

FixedTree and FixedMap are type aliases for ConstTree<K, V, Fixed> and ConstMap<K, V, Fixed>. ZeroTree and ZeroMap also accept Fixed as backend.

§Collection types

Type	Index	Values	Ordered	FixedStore
`ConstTree`	SkipList	inline `[u8; V]`	yes	yes
[`VarTree`]	SkipList	[`ByteView`] (disk + cache)	yes	no
`TypedTree`	SkipList	typed `T` (in-memory, `TypedRef`)	yes	no
`ZeroTree`	SkipList	zerocopy `T` (in-memory)	yes	yes
`ConstMap`	HashMap	inline `[u8; V]`	no	yes
[`VarMap`]	HashMap	[`ByteView`] (disk + cache)	no	no
`TypedMap`	HashMap	typed `T` (in-memory, `TypedRef`)	no	no
`ZeroMap`	HashMap	zerocopy `T` (in-memory)	no	yes

§Usage

// Bitcask backend (default)
let tree = ConstTree::<[u8; 16], 64>::open("data/users", Config::default())?;

// FixedStore backend
let tree = FixedTree::<[u8; 16], 64>::open("data/counters", FixedConfig::default())?;

tree.put(&key, &value)?;
let val = tree.get(&key);
tree.close()?;

§Durability

§Bitcask

Writes are buffered in memory (write_buffer_size per shard, default 1 MB). put() returns as soon as the entry is copied into the buffer and the in-memory index is updated — no disk I/O on the write path.

This means unflushed data is lost on crash. To control durability:

flush_buffers() — flush write buffers to disk (no fsync)
close() — flush + fsync + write hint files
config.enable_fsync = true — fsync on every buffer flush

§FixedStore

Each put() does a pwrite through the page cache (~200ns). fdatasync is called in batches (configurable interval/count), after releasing the shard lock — no latency spikes from sync. No compaction, no write amplification, 1x space usage.

§Thread safety

All tree/map types are Send + Sync. Share via Arc for concurrent access:

let tree = Arc::new(ConstTree::<[u8; 8], 8>::open(config)?);
let t = tree.clone();
std::thread::spawn(move || { t.put(&key, &value).unwrap(); });

Reads on ConstTree/ConstMap/ZeroTree/ZeroMap are lock-free (values inline in index). Reads on TypedTree/TypedMap are lock-free (seize RCU guard, no mutex). Reads on VarTree/VarMap are lock-free on cache hit, brief shard lock on miss. Writes acquire a per-shard mutex — different shards never contend.

§Encryption at rest (feature `encryption`)

Page-level AES-256-GCM encryption. Key from env or direct config:

let key = PageCipher::key_from_env("ARMDB_KEY")?;
let mut config = Config::default();
config.encryption_key = Some(key);
let tree = ConstTree::<[u8; 16], 64>::open("data/encrypted", config)?;
// all data is transparently encrypted on disk

§Write hooks (secondary indexes)

Generic WriteHook<K> / TypedWriteHook<K, T> parameter for synchronous write and init notifications. Zero overhead when unused (NoHook default).

on_write — fires on every put/insert/delete/cas/update. Does not fire inside atomic() blocks.
on_init — fires once per live entry during collection open (after recovery). Enable via NEEDS_INIT = true. See [armdb/docs/hooks.md] for details.
NEEDS_OLD_VALUE — only affects VarTree/VarMap (skips disk I/O for old value when false). In-memory collections (Const/Typed/Zero) always provide the old value at zero cost.

impl WriteHook<[u8; 16]> for MyIndex {
    const NEEDS_OLD_VALUE: bool = true;
    const NEEDS_INIT: bool = true;

    fn on_write(&self, key: &[u8; 16], old: Option<&[u8]>, new: Option<&[u8]>) {
        // update secondary index incrementally
    }

    fn on_init(&self, key: &[u8; 16], value: &[u8]) {
        // populate secondary index at startup
    }
}

let tree = ConstTree::<[u8; 16], 64, MyIndex>::open_hooked("data/indexed", config, my_index)?;
tree.migrate(|_, _| MigrateAction::Keep)?; // triggers on_init for all entries

§Compaction

Compaction is not automatic. Dead bytes accumulate as entries are overwritten or deleted. Use Compactor to run it in the background:

use std::sync::Arc;
use std::time::Duration;

let tree = Arc::new(ConstTree::<[u8; 16], 64>::open(config)?);
let t = tree.clone();
let _compactor = Compactor::start(move || t.compact(), Duration::from_secs(60));

Or call tree.compact() manually when needed.

§Iteration

All Tree types provide iter(), range(), and prefix_iter() methods. Map types (HashMap index) do not support iteration.

Each method returns a dedicated iterator implementing Iterator + DoubleEndedIterator:

Tree	Iterator	Item
`ConstTree`	`ConstIter`	`(K, [u8; V])` — copy
[`VarTree`]	[`VarIter`]	`(K, ByteView)` — RC, possible disk I/O
`TypedTree`	`TypedIter`	`(K, &T)` — reference, zero I/O
`ZeroTree`	`ZeroIter`	`(K, T)` — copy, zero I/O

Method	Description
`iter()`	All entries in index order
`range(start, end)`	Entries in `[start, end)` — start inclusive, end exclusive
`range_bounds(start, end)`	Entries with custom `Bound` — `Included`, `Excluded`, or `Unbounded`
`prefix_iter(prefix)`	Entries whose key starts with `prefix`

for (key, value) in tree.iter() { /* ... */ }

let latest = tree.prefix_iter(&user_id).take(20).collect::<Vec<_>>();

for (key, value) in tree.range(&start_key, &end_key) { /* ... */ }

// Custom bounds: (5, 10] — exclude 5, include 10
use std::ops::Bound;
let entries: Vec<_> = tree.range_bounds(
    Bound::Excluded(&5u64.to_be_bytes()),
    Bound::Included(&10u64.to_be_bytes()),
).collect();

// DoubleEndedIterator — .rev() or .next_back()
let oldest = tree.prefix_iter(&user_id).rev().take(10);

§Complexity

Operation	Complexity	Notes
`next()`	O(1)	follows SkipList level-0 forward pointer
`next_back()`	O(log n)	calls `find_last_lt()` — SkipList search from top
`iter()` / `range()` / `prefix_iter()` setup	O(log n)	initial SkipList search

VarIter: both next() and next_back() may additionally perform a pread on block-cache miss. Use warmup() to pre-populate the cache.

§Weakly-consistent semantics

Iterators do not create a snapshot. They are weakly-consistent:

Concurrent inserts/updates may be visible during iteration
Deleted entries (marked nodes) are automatically skipped
The seize guard prevents memory reclamation for the lifetime of the iterator — no use-after-free, but the index is not frozen

§Ordering: `Config::reversed`

Config::reversed controls the SkipList comparator direction.

`reversed`	`iter()` / `prefix_iter()`	`.rev()` / `next_back()`
`true` (default)	DESC (newest first)	ASC (oldest first)
`false`	ASC (oldest first)	DESC (newest first)

// reversed=true (default) — DESC: идеально для "newest first" пагинации
let tree = ConstTree::<[u8; 16], 64>::open("data/posts", Config::default())?;
let latest = tree.prefix_iter(&user_id).take(20);    // newest 20
let oldest = tree.prefix_iter(&user_id).rev().take(5); // oldest 5

// reversed=false — ASC: естественный порядок ключей
let mut config = Config::default();
config.reversed = false;
let tree = ConstTree::<[u8; 16], 64>::open("data/logs", config)?;
for (key, value) in tree.iter() { /* ascending order */ }

Keys are stored on disk as-is. reversed can be changed between restarts without migration — it only affects in-memory index ordering.

§Prefix sharding

let mut config = Config::default();
config.shard_prefix_bits = 32;
let tree = ConstTree::<[u8; 16], 64>::open("data/users", config)?;

§Caveats

Iterators are weakly-consistent, not snapshot. Concurrent inserts and updates may be visible during iteration. Deleted entries are skipped. The seize guard prevents use-after-free but does not freeze the index.
CAS/update holds shard lock during possible disk I/O. VarTree::cas, VarTree::update, VarMap::cas, and VarMap::update read the current value under the shard mutex. On a block-cache miss this issues a pread — blocking all writes to that shard until the read completes. Pre-warm the cache or size it to cover the working set.
migrate() on HashMap trees allocates O(keys/shards) memory. ConstMap::migrate and VarMap::migrate collect all shard keys into a Vec before iterating. For very large shards this causes a transient memory spike. SkipList trees (ConstTree, VarTree) are not affected.

§Shutdown

// 1. Stop background tasks first
compactor.stop();
// 2. Close the tree (writes hint files, flushes, fsyncs)
Arc::try_unwrap(tree).expect("no other references").close()?;

If close() is not called (e.g. the tree is dropped via Arc::drop), Shard::Drop still flushes write buffers, fsyncs, and writes hint files automatically — no data loss and no slow recovery on next startup.

§Features

typed-tree — enables TypedTree, TypedMap, Codec and codec implementations
rapira-codec — RapiraCodec for rapira serialization (implies typed-tree)
bytemuck-codec — BytemuckCodec / BytemuckSliceCodec (implies typed-tree)
bitcode-codec — [BitcodeCodec] (implies typed-tree)
encryption — AES-256-GCM page-level encryption at rest
replication — leader/follower log-shipping replication
armour — integration with armour ecosystem: Db, schema-versioned migrations, binary RPC server (TCP/UDS). See armour module docs.
hot-path-tracing — per-operation tracing::trace! calls

Re-exports§

pub use codec::Codec;
pub use codec::RapiraCodec;
pub use codec::ZerocopyCodec;
pub use codec::BytemuckCodec;
pub use codec::BytemuckSliceCodec;
pub use codec::BytemuckVec;
pub use compaction::Compactor;
pub use fixed::FixedConfig;
pub use fixed::FixedMap;
pub use fixed::FixedTree;
pub use armour_core as core;

Modules§

armour: Integration with the armour ecosystem (feature armour).
codec
compaction
docs: In-depth documentation for armdb.
durability: Durability trait — abstracts Bitcask (append-only log) and FixedStore (fixed-slot pwrite) backends behind a common interface.
fixed
replication

Macros§

const_map: Expands to ConstMap<<V as CollectionMeta>::SelfId, { size_of::<V>() } [, H, Bitcask]>.
const_tree: Expands to ConstTree<<V as CollectionMeta>::SelfId, { size_of::<V>() } [, H, Bitcask]>.
impl_key_bytemuck: Implement Key for a type that derives bytemuck::{Pod, Zeroable}.
impl_key_zerocopy: Implement Key for a type that derives zerocopy::{FromBytes, IntoBytes, Immutable}.
typed_map: Expands to TypedMap<<V as CollectionMeta>::SelfId, V, C [, H]>.
typed_tree: Expands to TypedTree<<V as CollectionMeta>::SelfId, V, C [, H]>.
var_map: Expands to VarMap<<V as CollectionMeta>::SelfId [, H]>.
var_tree: Expands to VarTree<<V as CollectionMeta>::SelfId [, H]>.
zero_map: Expands to ZeroMap<<V as CollectionMeta>::SelfId, { size_of::<V>() }, V [, H, Bitcask]>.
zero_tree: Expands to ZeroTree<<V as CollectionMeta>::SelfId, { size_of::<V>() }, V [, H, Bitcask]>.

Structs§

Config: Database configuration.
ConstIter: Iterator over entries in a ConstTree. Returned by iter(), range(), and prefix_iter().
ConstMap: A map with fixed-size keys and values. All values are stored inline in a per-shard HashMap. Reads never touch disk — zero I/O reads. O(1) lookup instead of O(log n) SkipList. No ordered iteration — use ConstTree if you need prefix/range scans.
ConstMapShard: Handle for atomic multi-key operations within a single shard. Obtained via ConstMap::atomic. The shard + index locks are held for the lifetime of this struct — keep the closure short.
ConstShard: Handle for atomic multi-key operations within a single shard. Obtained via ConstTree::atomic. The shard lock is held for the lifetime of this struct — keep the closure short.
ConstTree: A tree with fixed-size keys and values. All values are stored inline in SkipList nodes. Reads never touch disk — zero I/O reads.
DiskLoc: Position of an entry’s value on disk.
EntryHeader: On-disk entry header. 16 bytes, 8-byte aligned, no padding.
HintEntry: A parsed hint entry.
NoHook: Default no-op hook. All branches are eliminated at compile time.
TreeMeta: Metadata about a named tree/map collection.
TypedIter: Iterator over entries in a TypedTree. Returned by iter(), range(), and prefix_iter().
TypedMap: A map with fixed-size keys and typed values T. Values are encoded via a Codec for disk persistence but stored as T in memory — reads never touch disk and return TypedRef<T> (guard-protected reference).
TypedMapShard: Handle for atomic multi-key operations within a single shard. Obtained via TypedMap::atomic. The shard + index locks are held for the lifetime of this struct — keep the closure short.
TypedRef: Guard-protected reference to a typed value inside a TypedTree.
TypedShard: Handle for atomic multi-key operations within a single shard. Obtained via TypedTree::atomic. The shard lock is held for the lifetime of this struct — keep the closure short.
TypedTree: A tree with fixed-size keys and typed values T. Values are encoded via a Codec for disk persistence but stored as T in memory — reads never touch disk and return TypedRef<T> (guard-protected reference).
ZeroIter: Iterator over entries in a ZeroTree. Wraps ConstIter and converts [u8; V] values to T via zerocopy.
ZeroMap: A map with fixed-size keys and zerocopy-compatible typed values.
ZeroMapShard: Handle for atomic multi-key operations within a single shard. Obtained via ZeroMap::atomic.
ZeroShard: Handle for atomic multi-key operations within a single shard. Obtained via ZeroTree::atomic.
ZeroTree: A tree with fixed-size keys and zerocopy-compatible values.

Enums§

DbError
MigrateAction: Action returned by the migrate() callback for each entry.

Constants§

TOMBSTONE_BIT

Traits§

CollectionMeta: Trait for types that carry enough metadata to describe an armdb collection.
Key: Key metadata trait — describes key encoding for armdb collections.
Location: Marker trait for disk location types stored in SkipList nodes. DiskLoc (12 bytes) for Bitcask, u32 (4 bytes) for FixedStore slot_id.
TypedWriteHook: Typed write hook for TypedTree.
WriteHook: Trait for receiving write notifications from tree/map operations.

Functions§

compute_crc32: Compute CRC32 over gsn || value_len || key || value.
entry_size: Compute the total on-disk size of an entry including padding to 8-byte alignment.
hint_entry_size: Size of a single hint entry: GSN(8) + Key(key_len) + Offset(8) + Len(4).
parse_hint_entries: Parse hint entries from raw hint file bytes.
serialize_entry: Serialize a complete entry (header + key + value + padding) into a Vec<u8>.

Type Aliases§

DbResult

Crate armdb

Crate armdb Copy item path

§Durability backends

§Collection types

§Usage

§Durability

§Bitcask

§FixedStore

§Thread safety

§Encryption at rest (feature encryption)

§Write hooks (secondary indexes)

§Compaction

§Iteration

§Complexity

§Weakly-consistent semantics

§Ordering: Config::reversed

§Prefix sharding

§Caveats

§Shutdown

§Features

Re-exports§

Modules§

Macros§

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Crate armdb

§Encryption at rest (feature `encryption`)

§Ordering: `Config::reversed`