Crate nodedb_wal 

nodedb-wal

Deterministic, O_DIRECT write-ahead log with group commit.

This crate bypasses the Linux page cache entirely. Every WAL write goes directly to NVMe via O_DIRECT (and eventually io_uring). This is non-negotiable: if AI agents dump 10 GB of telemetry logs, the OS must NOT evict hot HNSW vector indexes from RAM to cache WAL pages.

Design

• O_DIRECT: All writes bypass the page cache. Aligned to 4 KiB.
• Group commit: Thousands of concurrent writes are batched into a single fsync, maximizing NVMe IOPS.
• CRC32C: Every record has a checksum for silent bit-rot detection.
• Deterministic replay: WAL replay is idempotent — crash at any point, recover to a consistent prefix.

Validation target

Sustain 100,000+ async writes/sec with sub-millisecond p99 latency. free -m cached memory must not move during the benchmark.

Targets

On wasm32 the crate builds only crypto, the key and envelope code it needs, error, and the record-header constants in record::header. The writer, readers, segments, double-write buffer, replay, and recovery build only on native targets.

Re-exports

pub use double_write::DoubleWriteBuffer;

pub use double_write::DwbDegradation;

pub use double_write::DwbMirror;

pub use double_write::DwbMode;

pub use double_write::DwbProtection;

pub use double_write::DwbSkipReason;

pub use double_write::wal_dwb_bytes_written_total;

pub use double_write::wal_dwb_degradations_total;

pub use double_write::wal_dwb_unprotected_records_total;

pub use error::Result;

pub use error::WalError;

pub use lazy_reader::LazyWalReader;

pub use preamble::CIPHER_AES_256_GCM;

pub use preamble::PREAMBLE_SIZE;

pub use preamble::PREAMBLE_VERSION;

pub use preamble::SEG_PREAMBLE_MAGIC;

pub use preamble::SegmentPreamble;

pub use preamble::WAL_PREAMBLE_MAGIC;

pub use reader::StopReason;

pub use reader::WalReader;

pub use record::FtsDeletePayload;

pub use record::FtsIndexPayload;

pub use record::RecordStamp;

pub use record::RecordTarget;

pub use record::RecordType;

pub use record::SpatialDeletePayload;

pub use record::SpatialPutPayload;

pub use record::WalRecord;

pub use record::WalRecordArgs;

pub use record::WriteAbortedPayload;

pub use record::NO_EVENT_SOURCE;

pub use record::RecordHeader;

pub use recovery::RecoveryInfo;

pub use recovery::recover;

pub use replay::AbortedWrites;

pub use replay::DatabaseTombstones;

pub use replay::ReplayFilters;

pub use replay::TombstoneSet;

pub use replay::drop_aborted_records;

pub use replay::extract_replay_filters;

pub use replay::extract_tombstones;

pub use secure_mem::SecureKey;

pub use segmented::SegmentedWal;

pub use segmented::SegmentedWalConfig;

pub use temporal_purge::TemporalPurgeEngine;

pub use temporal_purge::TemporalPurgePayload;

pub use time_anchors::TimeAnchors;

pub use tombstone::CollectionTombstonePayload;

pub use tombstone::MAX_COLLECTION_NAME_LEN;

pub use torn_tail::TailVerdict;

pub use torn_tail::holds_only_crash_residue;

pub use torn_tail::verify_committed_prefix;

pub use writer::WalWriter;

Modules

align

O_DIRECT alignment utilities.

crypto

WAL payload encryption using AES-256-GCM.

diag

Black-box recorder wiring for the WAL.

double_write

Double-write buffer for torn write protection.

error

lazy_reader

Lazy WAL reader: reads headers without payload for selective replay.

mmap_reader

Memory-mapped WAL segment reading for Event Plane catchup.

preamble

Segment preamble: 16-byte plaintext header written at offset 0 of every WAL segment file and every storage segment file.

reader

WAL reader for crash recovery and replay.

record

recovery

WAL recovery: scan an existing WAL file to determine the last committed LSN and file offset, enabling safe reopening for continued writes.

replay

Replay-time utilities layered over raw [WalRecord] streams.

secure_mem

Secure memory utilities for key material.

segment

WAL segment management.

segmented

Segmented WAL: manages a directory of segment files with automatic rollover and truncation.

temporal_purge

TemporalPurge record payload codec.

time_anchors

The live LSN ↔ commit-time map of one WAL.

tombstone

CollectionTombstoned record payload codec.

torn_tail

Telling a torn tail apart from mid-file corruption.

writer

WAL writer with O_DIRECT and group commit.