IronWal

IronWal is a high-durability, deterministic Write-Ahead Log (WAL) designed for systems of record, such as financial ledgers, game state synchronization, and event stores. It prioritizes data integrity, precise replication control, and predictable resource usage over raw, unsafe throughput, providing a robust foundation for building crash-safe applications.

Installation

Add ironwal to your Cargo.toml dependencies:

[dependencies]
ironwal = "0.6"

LZ4 compression support is included in the default features. To be explicit:

[dependencies]
ironwal = { version = "0.6", features = ["compression"] }

Quick Start

The core type is Wal: you append binary entries to named streams and read them back by ID or by iteration.

use ironwal::{Wal, WalOptions};

fn main() -> ironwal::Result<()> {
    // All data lives under this directory, one subdirectory per stream.
    let wal = Wal::new(WalOptions::new("./my_wal_data"))?;

    // Append an entry; it is assigned a monotonically increasing ID.
    let id = wal.append("orders", b"order_created")?;

    // Random access by ID...
    let entry = wal.get("orders", id)?;

    // ...or sequential scans from any starting ID.
    for entry in wal.iter("orders", 0)? {
        let data = entry?;
        // process data...
    }

    Ok(())
}

Atomic batch writes (append_batch, TxnWriter) and disk reclamation (truncate) are covered in the Usage Guide.

Core Concepts

Stream — an independent, append-only sequence of records, addressed by name. Each stream maps to its own directory on disk.
Entry — a single binary record. Every entry in a stream gets a monotonically increasing 64-bit sequence ID.
Segment — streams are stored as segment files that rotate automatically at configurable size/count thresholds. Truncation reclaims disk space by deleting whole segments below a safe ID.

Key Features

Deterministic Storage

Stream names map explicitly to physical directories on disk. This deterministic structure simplifies external backups, replication strategies, and manual auditing of the log files.

Data Integrity & Safety

Every frame written to disk is protected by a CRC32 checksum to detect bit rot and disk corruption. The library includes a rigorous recovery system capable of handling partial writes, power failures, and corrupted indices without data loss.

Hybrid Compression

To balance storage efficiency and latency, IronWal employs a hybrid compression strategy. Frames can be stored as raw binary or compressed using the LZ4 format, automatically decided based on configurable size thresholds.

Per-Stream Concurrency

Wal is cheaply cloneable and thread-safe. Locking is per stream, so appending to one stream never blocks writers on other streams — throughput scales with the number of independent streams being written.

Resource Protection

Open file descriptors are held in bounded LRU caches, for both writing and reading. This prevents the application from exhausting system file handle limits, even when managing thousands of active streams.

Tunable Durability

Developers can choose between three synchronization modes: Strict (fsync on every write), BatchOnly (fsync only on batches/transactions), and Async (rely on OS page cache), allowing for precise trade-offs between latency and safety.

Sharded WAL (optional `sharded` feature)

Everything above is the core, always-available API. For write volumes beyond what a single stream can absorb, the optional ShardedWal builds on top of Wal: it partitions writes across a fixed number of internal streams (shard_00, shard_01, …) by hashing a key you supply. The key only routes the write — it is not stored — and each entry is addressed by (shard_id, seq_id). On top of this partitioning, ShardedWal adds checkpoints — a consistent cut of every shard's durable offset, tagged with an ID you choose (a Raft index, transaction ID, ULID, …) — and pruning of segments older than a checkpoint.

Note that this is a different mechanism from the per-stream concurrency above: the core Wal already lets independent streams proceed in parallel. Reach for ShardedWal only when the throughput of a single logical stream is the bottleneck. In exchange you give up per-key get (reads are per-shard iteration), atomicity of batches that span shards, and the ability to change the shard count without a migration.

[dependencies]
ironwal = { version = "0.6", features = ["sharded"] }

use ironwal::sharded::ShardedWal;
use ironwal::WalOptions;

// 16 shards, fixed at creation.
let wal = ShardedWal::new(WalOptions::new("./data"), 16)?;

// The key picks the shard; it is not stored with the entry.
let (shard_id, seq_id) = wal.append(b"user_123", b"profile_data")?;

// Record every shard's durable offset under an ID you choose.
wal.create_checkpoint(b"raft_index_5000")?;

// Reclaim disk space for segments entirely before that checkpoint.
wal.prune_before_checkpoint(b"raft_index_5000")?;

Restoring from checkpoints, batch writes, durability caveats under BatchOnly/Async, and shard-count sizing are covered in the Usage Guide.

Notable Users

Hi Stakes Markets Game - The worlds most advanced financial simulator, available on iPhone and Android.

Documentation

For a detailed guide on configuration, API usage, and architectural concepts, please see the Usage Guide.

You can also view the full API reference documentation on docs.rs.

License

This library is distributed under the terms of the Mozilla Public License 2.0 (MPL-2.0).

ironwal 0.6.5