Skip to main content

Crate kevy_seg

Crate kevy_seg 

Source
Expand description

Immutable ordered-record segment files — build once, binary-search forever. The storage stone every cold tier shares: row segments, scalar-index cold segments and text bucket segments differ only in what their records MEAN; how records are laid out, located, checksummed and retired is identical, so it lives here once.

use kevy_seg::{Manifest, ManifestEntry, Seg, SegBuilder};

// keys go in strictly ascending; the footer seals the file
let mut b = SegBuilder::create(&dir.join("s1.seg"))?;
b.push(b"a", b"1")?;
b.push(b"b", b"2")?;
let meta = b.finish()?;

// the manifest makes the sealed segment part of the live set
Manifest::open(&dir)?.add(ManifestEntry::new("s1.seg", meta))?;

let seg = Seg::open(&dir.join("s1.seg"))?;
assert_eq!(seg.get(b"b")?, Some(b"2".to_vec()));
assert_eq!(seg.count_range(b"a", b"z")?, 2);

§Model

SegBuilder appends records in strictly ascending key order, pages them, checksums every page, and seals the file with a footer (record count, min/max key, and a fence table — the first key of every page). Seg opens the sealed file, keeps the fence table in memory, and answers get / range / count_range with one fence binary search plus one page read. Nothing ever mutates a sealed segment; deletion is the caller’s directory concern (tombstones live above this crate), and removal is unlink.

§Layout (v1)

Data pages are 4 KiB: a small header, cells packed forward, a slot directory (u16 cell offsets) packed backward from the tail, and a CRC32C over the page in the last 4 bytes. A record too large for one page spills its payload into a run of dedicated overflow pages (the cell keeps the key and points at the run — SQLite’s overflow idea, shorn of its freelist because nothing here is ever freed). The footer is written last: fence entries, min/max keys, counts, magic, and its own CRC; the final 16 bytes locate and size it.

§References

The read-only half of SQLite’s page format (slot directory growing backward, overflow chains) is the reference for the data pages. Its WAL, freelist, cursors and varint cell headers are deliberately absent: an immutable segment needs none of them.

Structs§

Manifest
The append-only ledger. One per segment directory.
ManifestEntry
One live segment as the manifest knows it.
RangeIter
Ascending (key, payload) iterator over a closed range.
Seg
An open segment. Cheap to clone-by-Arc above this crate; internally one file handle plus the in-memory fence table.
SegBuilder
Streaming builder. Records must arrive in strictly ascending key order (the reader’s binary search is the reason); a violation is a refusal, not a sort.
SegMeta
Sealed-segment summary, from the footer.

Enums§

SegError
Why a segment file was refused at open. Corruption is a refusal, never a silent partial read.