gnitz-zset 0.1.3

The Z-set kernel of the gnitz database: schema, columnar batches, cursors and operators
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

//! A Z-set as bytes — in memory, on the wire, on disk — and the cursor that
//! reads it back.
//!
//! The pure batch representation and the operations that work directly on it:
//! region layout (`batch`), wire serialization (`batch_wire`), TLS buffer
//! recycling (`batch_pool`), the OPK lower-bound search (`seek`), sort-merge
//! consolidation (`merge`), the German-string heap (`string_heap`), the direct
//! row writer (`writer`), the column-first row copy (`scatter`), the N-way min-merge tournament (`loser_tree`), the
//! shard PK-probe filter (`shard_filter`), the shard writer (`shard_file`), the
//! shard file's framing (`layout`) and its region encodings (`encoding`), and the
//! guard-routed shard merge a compaction runs (`compact`), the bounded external sort of
//! key records (`spill`), and the read-only file mapping (`mmap`). The low-level
//! WAL-block framer lives in `gnitz_wire::wal` (the one definition client and
//! engine share). The row-at-a-time system-table writer over a batch is
//! `batch_builder`.
//!
//! The shard *image* has one owner: `shard_file` writes it, `shard_reader`
//! mmaps and validates it, and `layout` and `encoding` hold the format rules
//! both call.
//!
//! Above the representation sit the sorted run, whatever backs it (`run`), and
//! the opaque read cursor that merges runs (`read_cursor`). Which runs a store
//! holds, and when they flush, compact or publish, is `gnitz-store`'s: this rung
//! is handed runs and never asks where they came from.
//!
//! A `pub use` below is named by another crate; a `pub(crate) use` is this
//! crate's own facade. The submodules stay private, so a `pub` item inside one
//! is still dead-code checked.

mod batch;
mod batch_builder;
mod batch_pool;
mod batch_wire;
mod compact;
mod encoding;
mod error;
mod layout;
mod loser_tree;
mod merge;
mod mmap;
mod read_cursor;
mod run;
mod scatter;
mod seek;
mod shard_file;
mod shard_filter;
mod shard_reader;
mod spill;
mod string_heap;
mod writer;

pub use batch::Batch;
pub use batch_builder::BatchBuilder;
pub use batch_pool::PooledBuf;
pub use batch_wire::WireRows;
pub use compact::{guard_slot, merge_and_route, merge_guard};
pub use error::StorageError;
pub use merge::{merge_consolidated, MemBatch};
pub use read_cursor::{
    empty_cursor, from_runs, from_runs_at, from_runs_in_band, BoundedIndexCursor, PkSetGather, ReadCursor,
    SkeletonKeys, SourceCursor,
};
pub use run::{first_live_payload_group, Run, StoredRow};
pub use seek::pk_group_end;
pub use shard_file::ShardWriteOpts;
pub use shard_reader::{MappedShard, ShardDirectory};
pub use spill::{KeyProducer, SpillSort};

pub(crate) use batch::{range_rows, runs_where, write_to_batch, RowMark};
pub(crate) use scatter::{copy_runs, materialize_carrying, width_dispatch};
pub(crate) use seek::pk_prefix_group_end;
pub(crate) use string_heap::{prorated_blob_cap, relocate_german_string_vec, should_relocate_blob};
pub(crate) use writer::DirectWriter;