laminar-storage 0.18.3

Storage layer for LaminarDB - WAL, checkpointing, and lakehouse integration
Documentation

laminar-storage

Storage layer for LaminarDB -- WAL, checkpointing, and recovery.

Overview

Ring 1 durability: write-ahead logging with per-core segments, incremental checkpointing with directory-based snapshots, and checkpoint manifests. Never blocks Ring 0.

Note: Lakehouse sinks (Delta Lake, Iceberg) are in laminar-connectors, not here. This crate handles LaminarDB's internal durability.

Key Modules

Module Purpose
wal Write-ahead log with CRC32C checksums, torn write detection, fdatasync
per_core_wal Per-core WAL segments for lock-free per-core writers, epoch ordering
incremental Incremental checkpointing with directory-based delta snapshots
checkpoint Checkpoint manager, object-store checkpointer, checkpoint layout
checkpoint_manifest CheckpointManifest, ConnectorCheckpoint, OperatorCheckpoint
checkpoint_store CheckpointStore trait, FileSystemCheckpointStore (atomic writes)
changelog_drainer Ring 1 SPSC changelog consumer
wal_state_store Combines state store with WAL for durability
io_uring_wal io_uring-backed WAL for Linux 5.10+ (feature-gated)

Key Types

  • WriteAheadLog -- Core WAL with CRC32C checksums and torn write detection
  • PerCoreWalManager / CoreWalWriter -- Per-core WAL segments (one per CPU core)
  • IncrementalCheckpointManager -- Directory-based incremental checkpointing
  • ObjectStoreCheckpointer -- Cloud object store checkpoint backend
  • CheckpointManifest -- Serializable snapshot of all operator and connector state
  • RecoveryManager -- Restores state from checkpoint + WAL replay
  • ChangelogDrainer -- Consumes Ring 0 changelog entries via SPSC queues
  • CheckpointCoordinator -- Coordinates per-core WAL checkpoint epochs

Checkpoint Architecture

Ring 0                          Ring 1
+------------------+            +------------------+
| ChangelogAware   |  SPSC -->  | Changelog        |
| StateStore       |            | Drainer          |
+------------------+            +--------+---------+
                                         |
                                         v
                                +--------+---------+
                                | Per-Core WAL     |
                                | (CRC32C, epoch)  |
                                +--------+---------+
                                         |
                                         v
                                +--------+---------+
                                | Incremental      |
                                | Checkpoint Mgr   |
                                +--------+---------+
                                         |
                                         v
                                +--------+---------+
                                | Checkpoint Store  |
                                | (FS / Object)    |
                                +------------------+

Benchmarks

cargo bench -p laminar-storage --bench wal_bench          # WAL write throughput
cargo bench -p laminar-storage --bench checkpoint_bench   # Checkpoint/recovery time

Feature Flags

Flag Purpose
io-uring io_uring-backed WAL (Linux 5.10+ only)

Related Crates

  • laminar-core -- Ring 0 state stores that produce changelog entries
  • laminar-db -- Checkpoint coordinator and recovery manager