Module overlay 

Data overlay files.

An overlay file supplies new values for a subset of (physical offset, field) cells within a fragment, without rewriting the fragment’s base data files. See the Data Overlay Files specification for the full rules; the invariants this module relies on are:

• Physical-offset coverage. Coverage bitmaps index physical row offsets (positions in the base data files, counting deleted rows), so they are stable across deletions, like deletion vectors.
• Rank-based values. The overlay’s data_file stores one value column per field, with no row-offset key column. Within a value column, a covered offset’s value sits at its rank — the 0-based count of set bits below it in that field’s coverage bitmap.
• Dense vs. sparse coverage. A dense overlay shares one bitmap across every field (OverlayCoverage::Shared); a sparse overlay carries one bitmap per field (OverlayCoverage::PerField).
• Parse once. Bitmaps are parsed from their 32-bit Roaring encoding a single time when the fragment loads and held behind an Arc, so cloning a fragment is cheap.
• Newest-last ordering. A fragment’s overlays are stored newest-last and stable-sorted by committed_version on load (see sort_overlays_newest_last), with list position breaking ties for equal versions. When two overlays cover the same (offset, field), the higher committed_version wins.
• Field tombstones. When new base values are written for a field (a DataReplacement, or an in-place column rewrite), any overlay value for that field is stale and must stop shadowing the fresh base. The field is marked obsolete in the overlay’s data_file.fields with TOMBSTONE_FIELD_ID (the same sentinel used for obsolete base columns) rather than physically removed, so the overlay’s other fields — and its coverage positions — stay intact (see tombstone_overlay_fields).

Structs

DataOverlayFile

An overlay file supplies new values for a subset of (physical offset, field) cells within a fragment, without rewriting the fragment’s base data files. See the module documentation for the coverage, rank, and versioning rules.

Enums

OverlayCoverage

Which (physical offset, field) cells a DataOverlayFile provides values for.

Constants

TOMBSTONE_FIELD_ID

Field-id sentinel marking a tombstoned (obsolete) field within an overlay’s data_file.fields. Matches the tombstone convention for obsolete columns in base data files; a tombstoned field’s values are ignored on read.

Functions

sort_overlays_newest_last

Stable-sort a fragment’s overlays newest-last by committed_version. The stable sort preserves list position as the tiebreak for equal versions, so resolution can rely on the ordering without re-checking. See the module documentation for the ordering invariant.

tombstone_overlay_fields

Tombstone fields across a fragment’s overlays, dropping any overlay left with no live fields.

verify_overlays_newest_last

Verify a fragment’s overlays are stored newest-last (non-decreasing committed_version), the ordering invariant readers rely on for resolution. Returns an error identifying the first out-of-order pair.