linkrs-storage 0.1.0

Storage engine with CSR and memory-mapped containers
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

//! Frozen group mmap snapshot file.
//!
//! Derived read-only cache beside the authoritative checkpoint: while the
//! checkpoint stays the source of truth, a frozen group can also serve
//! queries straight from a memory-mapped flat column file, paging entries in
//! on demand instead of decoding the whole group into the heap at open.
//!
//! File layout, everything little-endian:
//! - magic (u32)
//! - rows (u64), entries (u64), live edge count (u64), flags (u64)
//! - seven `(offset u64, length u64)` column descriptors: degrees, endpoints,
//!   ranks, edge ids, delete stamps, bundled values, validity bytes
//! - degrees: `rows` u32 values
//! - endpoints: `entries` u32 values
//! - ranks: `entries` i64 values
//! - edge ids, delete stamps, bundled values: `entries` u64 values each
//! - validity: `ceil(entries / 8)` bytes, one bit per slot
//!
//! The bundled value columns only occupy bytes when the flags word marks a
//! valued snapshot; otherwise both descriptors are empty ranges. Column
//! widths are fixed, so any slot is addressable by index with one
//! little-endian decode and no full-file decode. Row offsets are rebuilt in
//! memory on open and never persisted, mirroring the heap frozen form.
//!
//! The snapshot payload carries a trailing CRC32 covering every preceding
//! byte, using the same checksum pattern as the heap checkpoint dumps. Open
//! rejects structural mismatches (magic, out-of-range descriptors,
//! length disagreements, trailing bytes) and CRC mismatches alike, and the
//! caller falls back to the authoritative checkpoint, optionally
//! regenerating the snapshot file. A bad cache is discarded and rebuilt. Writers use a sibling temp file
//! plus atomic rename, so readers only ever observe complete files.
//!
//! The mapped handle is reference-counted (`MappedFrozen` clones share one
//! mapping), so readers holding a view keep the file alive across snapshot
//! file replacement. Single-writer discipline applies: the checkpoint flush
//! syncs base file and snapshot file together, and flushing a non-frozen
//! group removes its snapshot file. On Linux the mapping carries a transparent
//! huge page hint; a rejected hint falls back to base pages without failing
//! the open.
//!
//! # Snapshot cache state machine
//!
//! The snapshot file is a derived cache, never the truth. States and
//! transitions, all covered by the checkpoint tests:
//!
//! - Absent: no sidecar beside the base. Frozen or mapped bases backfill one
//!   on flush when missing; mutable bases stay absent.
//! - Valid: sidecar parses (magic, descriptors, lengths, CRC) and
//!   loads validate it against the base. Loads map it directly and skip the
//!   authoritative decode.
//! - Stale: the base was rewritten as mutable, or the group is gone. The
//!   flush removes the sidecar; readers never see it.
//! - Expired: the sidecar fails validation (corrupt bytes or
//!   a pending append delta the read-only view cannot absorb). Loads fall
//!   back to the authoritative base and regenerate the cache; the failure is
//!   counted in debug logs, never silent, and never fails the load.
//!
//! Generation, deletion, expiry and fallback are all decided in the
//! checkpoint snapshot helpers and the group load path; no other module
//! creates or removes sidecars.

use std::sync::Arc;

pub(crate) mod columns;
pub(crate) mod format;
pub(crate) mod iter;
pub(crate) mod open;
pub(crate) mod persistence;
pub(crate) mod read;
pub(crate) mod trait_impl;

#[cfg(test)]
mod tests;

use format::SnapshotColumns;
pub use format::{snapshot_path_for, write_snapshot_file};
pub use iter::{MappedFrozenIterator, MappedFrozenRowIter};

/// Memory-mapped read view of one frozen group snapshot file.
///
/// Clones share one mapping through the reference count, so snapshotting the
/// handle for a reader is cheap and the mapping outlives snapshot file
/// replacement while readers hold it. The query surface mirrors
/// [`ImmutableCsr`] so variant dispatch treats both identically; writes are
/// rejected the same way.
#[derive(Debug, Clone)]
pub struct MappedFrozen {
    map: Arc<memmap2::Mmap>,
    columns: SnapshotColumns,
    offsets: Arc<Vec<u32>>,
    rows: usize,
    entries: usize,
    edge_count: u64,
}