ember-persistence

durability layer for ember. handles append-only file logging, point-in-time snapshots, and crash recovery.

each shard gets its own persistence files (shard-{id}.aof and shard-{id}.snap), keeping the shared-nothing architecture all the way down to disk.

what's in here

• aof — append-only file writer/reader with CRC32 integrity checks. binary TLV format, configurable fsync (always, every-second, OS-managed). gracefully handles truncated records from mid-write crashes
• snapshot — point-in-time serialization of an entire shard's keyspace. writes to a .tmp file first, then atomic rename to prevent partial snapshots from corrupting existing data
• recovery — startup sequence: load snapshot, replay AOF tail, skip expired entries. handles corrupt files gracefully (logs warning, starts empty)
• encryption — optional AES-256-GCM encryption for AOF records and snapshot files (enabled with --features encryption). accepts a 32-byte key file (raw bytes or hex). existing plaintext files are read transparently and migrated on the next rewrite/snapshot
• format — low-level binary serialization helpers: length-prefixed bytes, integers, floats, checksums, header validation

file formats

AOF — [EAOF magic][version][record...] where each record is [tag][payload][crc32]

supported record types: SET, DEL, EXPIRE, PERSIST, PEXPIRE, INCR, DECR, INCRBY, DECRBY, APPEND, RENAME, COPY, LPUSH, RPUSH, LPOP, RPOP, ZADD, ZREM, HSET, HDEL, HINCRBY, SADD, SREM (plus VADD, VREM, PROTO.SET, PROTO.REGISTER when compiled with those features)

snapshot (v2) — [ESNP magic][version][shard_id][entry_count][entries...][footer_crc32] where entries are type-tagged (string=0, list=1, sorted set=2, hash=3, set=4). v1 snapshots (no type tags) are still readable.

encrypted formats (v3) — when an encryption key is configured, AOF records and snapshot entries are wrapped with AES-256-GCM: [nonce (12 bytes)][ciphertext][auth tag (16 bytes)]. the file headers and version bytes remain plaintext so the reader can detect encrypted files and select the right decoding path. plaintext v2 files are still readable even with a key configured.

usage

use ember_persistence::aof::{AofWriter, AofRecord};
use ember_persistence::snapshot::{SnapshotWriter, SnapshotReader, SnapEntry, SnapValue};
use ember_persistence::recovery::recover_shard;

// recovery is typically called by the shard on startup
let result = recover_shard(data_dir, shard_id);
for entry in result.entries {
    // insert into keyspace...
}

related crates