laminar-storage 0.18.5

Storage layer for LaminarDB - WAL, checkpointing, and lakehouse integration
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80

# laminar-storage

Storage layer for LaminarDB -- WAL, checkpointing, and recovery.

## Overview

Ring 1 durability: write-ahead logging with per-core segments, incremental checkpointing with directory-based snapshots, and checkpoint manifests. Never blocks Ring 0.

Note: Lakehouse sinks (Delta Lake, Iceberg) are in `laminar-connectors`, not here. This crate handles LaminarDB's internal durability.

## Key Modules

| Module | Purpose |
|--------|---------|
| `wal` | Write-ahead log with CRC32C checksums, torn write detection, fdatasync |
| `per_core_wal` | Per-core WAL segments for lock-free per-core writers, epoch ordering |
| `incremental` | Incremental checkpointing with directory-based delta snapshots |
| `checkpoint` | Checkpoint manager, object-store checkpointer, checkpoint layout |
| `checkpoint_manifest` | `CheckpointManifest`, `ConnectorCheckpoint`, `OperatorCheckpoint` |
| `checkpoint_store` | `CheckpointStore` trait, `FileSystemCheckpointStore` (atomic writes) |
| `changelog_drainer` | Ring 1 SPSC changelog consumer |
| `wal_state_store` | Combines state store with WAL for durability |
| `io_uring_wal` | io_uring-backed WAL for Linux 5.10+ (feature-gated) |

## Key Types

- **`WriteAheadLog`** -- Core WAL with CRC32C checksums and torn write detection
- **`PerCoreWalManager`** / **`CoreWalWriter`** -- Per-core WAL segments (one per CPU core)
- **`IncrementalCheckpointManager`** -- Directory-based incremental checkpointing
- **`ObjectStoreCheckpointer`** -- Cloud object store checkpoint backend
- **`CheckpointManifest`** -- Serializable snapshot of all operator and connector state
- **`RecoveryManager`** -- Restores state from checkpoint + WAL replay
- **`ChangelogDrainer`** -- Consumes Ring 0 changelog entries via SPSC queues
- **`CheckpointCoordinator`** -- Coordinates per-core WAL checkpoint epochs

## Checkpoint Architecture

```
Ring 0                          Ring 1
+------------------+            +------------------+
| ChangelogAware   |  SPSC -->  | Changelog        |
| StateStore       |            | Drainer          |
+------------------+            +--------+---------+
                                         |
                                         v
                                +--------+---------+
                                | Per-Core WAL     |
                                | (CRC32C, epoch)  |
                                +--------+---------+
                                         |
                                         v
                                +--------+---------+
                                | Incremental      |
                                | Checkpoint Mgr   |
                                +--------+---------+
                                         |
                                         v
                                +--------+---------+
                                | Checkpoint Store  |
                                | (FS / Object)    |
                                +------------------+
```

## Benchmarks

```bash
cargo bench -p laminar-storage --bench wal_bench          # WAL write throughput
cargo bench -p laminar-storage --bench checkpoint_bench   # Checkpoint/recovery time
```

## Feature Flags

| Flag | Purpose |
|------|---------|
| `io-uring` | io_uring-backed WAL (Linux 5.10+ only) |

## Related Crates

- [`laminar-core`](../laminar-core) -- Ring 0 state stores that produce changelog entries
- [`laminar-db`](../laminar-db) -- Checkpoint coordinator and recovery manager