Module spill 

Per-worker spill control for the columnar Chunk trace.

ColChunk is either resident (a trie in memory) or paged (resident bounds + a byte handle to fetch the trie back). Chunk::settle is where committed chunks may be paged out: it calls [try_page], which consults a per-worker [SpillState] and, when over budget, serializes the trie and hands the bytes to a pluggable BytesStore. Reads (Chunk::merge etc., and the cursor) fetch paged chunks back on demand.

The backend is pluggable: a worker calls install with its own BytesStore (e.g. file-backed). With nothing installed, no chunk is ever paged and ColChunk stays resident — zero overhead beyond a thread-local peek.

Known limitation

settle sees only its local committed output, not the whole batcher queue (timely’s MergeBatcher does not expose it, and the shared settle is generic so it can’t notify a columnar-specific accountant per consumed chunk). So the eviction policy is a simple per-worker high-water mark: keep the first budget_records worth of committed chunks resident, page the rest. It bounds the resident set and round-trips through the store correctly, but it favours keeping older data and can’t reclaim budget as data leaves — coarser than the exact head-reserve-over-the-queue policy the old bespoke batcher ran.

Structs

SpillStats

Cumulative spill counters, shared (via Arc) so a run can sum across workers.

Traits

BytesSource

A handle to bytes previously written via a BytesStore.

BytesStore

Append a chunk’s bytes to backing storage, returning a handle to read them back.

Functions

install

Install spill on this worker: page committed chunks once resident records exceed budget_records, writing to store. stats receives the counters.

uninstall

Remove this worker’s spill state (so later traces stay resident).