Struct SpillPoolWriter 

pub struct SpillPoolWriter { /* private fields */ }

Writer for a spill pool. Provides coordinated write access with FIFO semantics.

Created by channel. See that function for architecture diagrams and usage examples.

The writer is Clone, allowing multiple writers to coordinate on the same pool. All clones share the same current write file and coordinate file rotation. The writer automatically manages file rotation based on the max_file_size_bytes configured in channel. When the last writer clone is dropped, it finalizes the current file so readers can access all written data.

Implementations

impl SpillPoolWriter

pub fn push_batch(&self, batch: &RecordBatch) -> Result<(), DataFusionError>

Spills a batch to the pool, rotating files when necessary.

If the current file would exceed max_file_size_bytes after adding this batch, the file is finalized and a new one is started.

See channel for overall architecture and examples.

File Rotation Logic

push_batch()
     │
     ▼
Current file exists?
     │
     ├─ No ──▶ Create new file ──▶ Add to shared queue
     │                               Wake readers
     ▼
Write batch to current file
     │
     ▼
estimated_size > max_file_size_bytes?
     │
     ├─ No ──▶ Keep current file for next batch
     │
     ▼
Yes: finish() current file
     Mark writer_finished = true
     Wake readers
     │
     ▼
Next push_batch() creates new file

Errors

Returns an error if disk I/O fails or disk quota is exceeded.

Trait Implementations

impl Clone for SpillPoolWriter

fn clone(&self) -> SpillPoolWriter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Drop for SpillPoolWriter

fn drop(&mut self)

Executes the destructor for this type.