Module format 

Columnar segment binary format definitions.

All multi-byte integers are little-endian. The segment is self-describing: the footer contains all metadata needed to read any column without scanning the entire file.

Structs

BlockStats

Per-block statistics for a single column. Enables predicate pushdown: skip blocks where WHERE price > 100 and block’s max_price < 100.

ColumnMeta

Metadata for a single column within the segment footer.

SegmentFooter

Segment footer: contains all metadata needed to read any column.

SegmentHeader

Segment header: identifies the file as a NodeDB columnar segment.

Constants

BLOCK_SIZE

Rows per block. Each block is independently compressed and decompressible. 1024 balances compression ratio (larger = better) vs random access granularity (smaller = less waste on filtered scans).

ENDIANNESS_LE

Endianness marker: 0x01 = little-endian (always LE for NodeDB).

HEADER_SIZE

Size of the segment header in bytes: magic(4) + major(1) + minor(1) + endianness(1) = 7.

MAGIC

Magic bytes identifying a NodeDB columnar segment.

VERSION_MAJOR

Current format major version. Readers reject segments with higher major.

VERSION_MINOR

Current format minor version. Readers tolerate segments with same major but higher minor (unknown footer fields are ignored).