Skip to main content

write_batch_chunked

Function write_batch_chunked 

Source
pub fn write_batch_chunked(
    batch: &RecordBatch,
    dir: impl AsRef<Path>,
    file_type: &FileType,
    chunk_rows: usize,
) -> Result<ExportReport>
Expand description

Slice one in-memory RecordBatch into chunk_rows-sized record batches and write each as chunk{i}.<ext> under dir. Split out from export_cypher_chunked so the chunking can be exercised (round-tripped) without a FalkorDB connection.

The slicing is zero-copy (RecordBatch::slice shares Arrow buffers); peak extra memory is one encoder’s worth of chunk_rows rows at a time.