Skip to main content

Module chunk

Module chunk 

Source
Expand description

A worked Chunk: the columnar UpdatesTyped<U> trie, resident or paged.

Where vec backs a chunk with a flat Vec<((K,V),T,R)>, this backs it with the column-oriented trie from crate::columnar::updates — deduplicated keys, per-key val runs, per-val (time, diff) runs. It is a retargeting of the columnar trace pile at the Chunk abstraction: the storage (UpdatesTyped) and the trie-native merge (trie_merger) are reused verbatim, and the four transducers delegate to them. The harness (ChunkBatch, the straddle cursor, the batcher/builder/ spine aliases) is shared with vec.

This makes columnar trace merges trie-native (the old OrdValBatch-backed trace ran them through ord_neu’s row-oriented merger).

§Resident vs paged

A ColChunk is either Resident (the trie in memory) or Paged (resident bounds + a byte handle). Chunk::settle is the spill point: it pages committed chunks out via spill when a worker has installed a spiller. Reads fetch a paged chunk’s trie back, caching it in a OnceCell so repeated cursor access pays the fetch once; len and bounds read the resident metadata and never fetch.

Chunk::advance is trie-native: it melds its input in place and rewrites only the time column (keys/vals/diffs stay columnar refs), then re-consolidates the advanced time runs (a global pass a future per-(key,val) streaming consolidation could narrow).

Structs§

ChunkMeta
Resident bounds for a paged chunk: the first and last (key, val, time) as single-element columnar containers (so [Chunk::bounds] returns refs without fetching), plus the record count.
ColChunkCursor
A cursor over a ColChunk, tracking the current key and value as absolute indices into the trie’s flat keys.values / vals.values columns.
PagedChunk
A paged chunk: resident meta, a byte handle to fetch the trie, and a cache populated on first read. Opaque — held inside ColChunk::Paged.

Enums§

ColChunk
A sorted, consolidated columnar trie of ((key, val), time, diff): resident, or paged to backing storage.