Struct SpillBuffer 

pub struct SpillBuffer { /* private fields */ }

Append-only batch buffer with an encoded-byte memory budget.

The budget is exact for the serialized representation and does not claim to be the Rust allocator’s resident-byte accounting. At most one incoming or decoded batch can itself be larger than the budget; successful pushes do not retain such an oversized batch in memory.

Implementations

impl SpillBuffer

pub fn new(budget_bytes: usize) -> Self

pub fn new_in(budget_bytes: usize, directory: impl Into<PathBuf>) -> Self

Create a buffer whose temporary spill file will be placed in directory.

File creation is deferred until the first spill. This is primarily useful when an engine has a dedicated temporary-data volume.

pub fn unbounded() -> Self

pub fn push(&mut self, batch: Batch) -> ExecResult<bool>

Append a batch, spilling automatically when required by the byte budget.

Returns true if this push wrote one or more batches to disk. If disk creation, encoding, or writing fails, the new batch and all earlier batches remain owned by the buffer and the error is returned.

pub fn encoded_size(batch: &Batch) -> ExecResult<usize>

Exact byte count used for budget accounting, including the record length prefix written to disk.

pub fn rows(&self) -> usize

Total buffered rows, including rows already written to disk.

pub fn in_memory_rows(&self) -> usize

Rows currently retained in memory.

pub fn in_memory_bytes(&self) -> usize

Exact encoded bytes currently retained in memory.

pub fn budget_bytes(&self) -> usize

pub fn over_budget(&self) -> bool

pub fn has_spilled(&self) -> bool

pub fn spilled_rows(&self) -> usize

pub fn spilled_batches(&self) -> usize

pub fn spilled_bytes(&self) -> usize

pub fn spill_path(&self) -> Option<&Path>

Path of the live spill file, if one has been created.

The path is diagnostic only and becomes invalid as soon as the buffer or the drain iterator that owns the file is dropped.

pub fn spill_if_over_budget(&mut self) -> ExecResult<bool>

Flush all pending in-memory batches when the byte budget is exceeded.

Returns true when batches were written. A failed append is rolled back to the previous file length and the pending batches remain in memory, so callers never observe a silent partial spill.

pub fn spill_pending(&mut self) -> ExecResult<bool>

Force all pending in-memory batches to disk regardless of the budget.

This is useful at a blocking-operator phase boundary. It returns false when there is nothing pending.

pub fn reader(&self) -> ExecResult<SpillReader<'_>>

Open a repeatable streaming reader without consuming this buffer.

Spilled batches are decoded one at a time. The in-memory tail is cloned one batch at a time only when the reader reaches it.

pub fn read_rows(&self) -> ExecResult<SpillRows<SpillReader<'_>>>

Open a repeatable physical-row stream without collecting all batches.

pub fn drain(&mut self) -> ExecResult<SpillDrain>

Drain buffered batches in their original input order.

The returned iterator owns the temporary file. Each disk read or decode failure is returned as a crate::physical::ExecError, and dropping the iterator early still removes the temporary file.

pub fn drain_all(&mut self) -> ExecResult<Vec<Batch>>

Drain and materialize every restored batch.

pub fn drain_rows(&mut self) -> ExecResult<SpillRows<SpillDrain>>

Consume the buffer as a physical-row stream without collecting batches.

pub fn clear(&mut self)

Discard all buffered data and remove any spill file.

pub fn into_shared( self, schema: impl Into<RowSchema>, ) -> ExecResult<SharedSpill>

Seal this buffer as an immutable, cheaply cloneable materialization. Batches that fit within the configured byte budget remain in memory; once spilling has started, every pending batch is flushed and readers reopen the file independently. Both forms support repeatable scans without collecting the complete input again.