fsqlite-wal 0.1.10

Write-ahead logging
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108

# fsqlite-wal

Write-ahead logging implementation for fsqlite. This crate handles WAL file I/O, frame checksumming, checkpointing, group commit consolidation, WAL index (shared-memory) management, crash recovery, and forward error correction (FEC) for WAL frames.

## Overview

`fsqlite-wal` implements the WAL protocol that enables concurrent readers alongside a writer without blocking. Committed pages are appended as frames to a WAL file; checkpoint operations transfer those frames back to the main database. The crate provides extensive integrity checking (five levels), torn-write detection, and an optional RaptorQ-based FEC sidecar that can reconstruct damaged WAL frames from repair symbols.

The WAL crate depends on `fsqlite-vfs` for file I/O but does not depend on `fsqlite-pager` directly. Instead, the pager defines a `WalBackend` trait that an adapter in `fsqlite-core` implements by wrapping `WalFile` from this crate. This breaks the circular dependency.

**Position in the dependency graph:**

```
   fsqlite-vfs
        |
   fsqlite-pager
      /    \
fsqlite-wal  fsqlite-btree    <-- you are here
      \    /
   fsqlite-mvcc
```

## Key Types

### WAL File

- `WalFile` -- Core WAL file handle. Manages frame append, read-back, header parsing, and sync.

### Checksum and Integrity

- `WalHeader` / `WalFrameHeader` -- On-disk WAL and frame header structures.
- `SqliteWalChecksum` / `Xxh3Checksum128` -- Checksum algorithms (SQLite-compatible and XXH3-128).
- `integrity_check_*` functions -- Five levels of integrity checking: L1 page checksums, L2 B-tree structure, L3 overflow chains, L4 cross-reference, L5 schema validation.
- `detect_torn_write_in_wal` -- Torn-write detection using sector-size analysis.
- `WalChainValidation` / `validate_wal_chain` -- End-to-end WAL chain validation.

### Checkpointing

- `CheckpointMode` -- `Passive`, `Full`, `Restart`, `Truncate`.
- `CheckpointPlan` / `CheckpointState` / `CheckpointProgress` -- Checkpoint planning and execution state.
- `plan_checkpoint` / `execute_checkpoint` -- Plan and execute a checkpoint operation.

### Group Commit

- `GroupCommitConsolidator` / `GroupCommitConfig` -- Batches multiple transaction frame submissions into consolidated WAL writes for throughput.
- `FrameSubmission` / `TransactionFrameBatch` -- Individual and batched frame submissions.
- `write_consolidated_frames` -- Writes a batch of consolidated frames to the WAL.

### WAL Index (Shared Memory)

- `WalIndexHdr` / `WalCkptInfo` -- Shared-memory header and checkpoint info structures.
- `WalIndexHashSegment` -- Hash table segments for fast page-to-frame lookup.
- `parse_shm_header` / `write_shm_header` -- Read/write the WAL-index header in shared memory.
- `wal_index_hash_slot` -- Hash function for WAL index page lookups.

### Forward Error Correction (FEC)

- `WalFecRepairPipeline` / `WalFecRepairPipelineConfig` -- Pipeline for detecting and repairing damaged WAL frames using RaptorQ erasure coding.
- `WalFecGroupMeta` / `WalFecGroupRecord` -- FEC group metadata and recovery records.
- `generate_wal_fec_repair_symbols` -- Generate RaptorQ repair symbols for a commit group.
- `recover_wal_fec_group_with_config` -- Attempt FEC-based recovery of a damaged commit group.

### Recovery

- `WalRecoveryDecision` / `RecoveryAction` -- Recovery logic for checksum mismatches and corrupted frames.
- `recovery_compaction` (module) -- WAL compaction during recovery.

### Metrics

- `WalMetrics` / `GroupCommitMetrics` / `WalFecRepairCounters` / `WalRecoveryCounters` -- Global atomic counters with snapshot export.

## Usage

```rust
use fsqlite_wal::{
    WalHeader, WalFrameHeader, SqliteWalChecksum,
    compute_wal_frame_checksum, WAL_HEADER_SIZE, WAL_FRAME_HEADER_SIZE,
};

// Parse a WAL header from raw bytes.
let header_bytes = [0u8; WAL_HEADER_SIZE];
// ... read from file ...

// Compute a frame checksum (SQLite-compatible big-endian).
let frame_header = [0u8; WAL_FRAME_HEADER_SIZE];
let page_data = vec![0u8; 4096];
let (s0, s1) = compute_wal_frame_checksum(
    &frame_header,
    &page_data,
    (0, 0), // running checksum from previous frame
    true,   // big-endian (WAL_MAGIC_BE)
);
```

## Dependencies

- `fsqlite-types` -- Shared type definitions.
- `fsqlite-error` -- Unified error/result types.
- `fsqlite-vfs` -- File I/O abstraction.
- `xxhash-rust` -- XXH3 hashing.
- `crc32c` -- CRC32C checksums.
- `blake3` -- BLAKE3 content-address hashing.
- `tracing` -- Structured logging.
- `asupersync` -- Async-compatible synchronization primitives.

## License

MIT