Skip to main content

Module reader

Module reader 

Source
Expand description

Segment reader — reads .csx segment files.

The reader validates the file integrity (magic bytes, CRC32c checksum), parses header/footer/metadata, and provides methods to read individual columns or entire row groups as Apache Arrow arrays.

§I/O strategy — memory-mapped prefetch

The entire segment file is memory-mapped via memmap2::Mmap. No per-column seek()/read() system calls are issued; instead, column data is accessed by slicing the mapped byte range, and the OS page cache transparently pages data in from disk on first access.

Two advisory hints accelerate sequential scans:

  1. madvise(MADV_SEQUENTIAL) — set once at open time, tells the kernel the file will be read front-to-back so it can perform aggressive read-ahead (typically 128–256 KiB on Linux).
  2. madvise(MADV_WILLNEED) — issued per-column in SegmentReader::read_columns: while decoding column N, the reader hints the kernel to begin paging in column N+1 (and the same column in the next row group for multi-row-group scans). This overlaps I/O with CPU decoding.

Because the Mmap holds the entire file mapping, there is no separate read-ahead buffer or prefetch thread; the OS page cache is the buffer. This is the most efficient approach for columnar segment files that fit in the page cache — it avoids double-copying into user-space buffers and lets the kernel coalesce I/O for adjacent columns.

§Predicate pushdown

The reader supports multi-level predicate pushdown to minimise the amount of data that is read and decoded:

LevelMechanismMethod
SegmentCatalog-level time-range + tag-index + bloom filterChronix::prune_segments()
Row group — timemin_ts/max_ts from timestamp column statsread_projected_filtered
Row group — tagsvalue_count == 0 with a non-zero null_count (a genuinely all-null block → impossible match)read_projected_filtered_with_predicates
Columnprojection pushdown (only requested columns decoded)read_projected

Structs§

FieldPredicate
A field predicate that can be evaluated against per-row-group ColumnStats min/max to skip row groups that cannot possibly match.
SegmentReader
Reads data from a .csx segment file.

Enums§

StatsOp
A numeric comparison operator for zone-map row-group pruning.