Struct deltalake::datafusion::parquet::arrow::arrow_writer::ArrowWriter

pub struct ArrowWriter<W>
where
    W: Write,
{ /* private fields */ }

Encodes RecordBatch to parquet

Writes Arrow RecordBatches to a Parquet writer. Multiple RecordBatch will be encoded to the same row group, up to max_row_group_size rows. Any remaining rows will be flushed on close, leading the final row group in the output file to potentially contain fewer than max_row_group_size rows

let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

let mut buffer = Vec::new();
let mut writer = ArrowWriter::try_new(&mut buffer, to_write.schema(), None).unwrap();
writer.write(&to_write).unwrap();
writer.close().unwrap();

let mut reader = ParquetRecordBatchReader::try_new(Bytes::from(buffer), 1024).unwrap();
let read = reader.next().unwrap().unwrap();

assert_eq!(to_write, read);

Memory Limiting

The nature of parquet forces buffering of an entire row group before it can be flushed to the underlying writer. Data is buffered in its encoded form, to reduce memory usage, but if writing rows containing large strings or very nested data, this may still result in non-trivial memory usage.

ArrowWriter::in_progress_size can be used to track the size of the buffered row group, and potentially trigger an early flush of a row group based on a memory threshold and/or global memory pressure. However, users should be aware that smaller row groups will result in higher metadata overheads, and may worsen compression ratios and query performance.

writer.write(&batch).unwrap();
// Trigger an early flush if buffered size exceeds 1_000_000
if writer.in_progress_size() > 1_000_000 {
    writer.flush().unwrap();
}

Implementations

impl<W> ArrowWriter<W>

where W: Write + Send,

pub fn try_new( writer: W, arrow_schema: Arc<Schema>, props: Option<WriterProperties> ) -> Result<ArrowWriter<W>, ParquetError>

Try to create a new Arrow writer

The writer will fail if:

• a SerializedFileWriter cannot be created from the ParquetWriter
• the Arrow schema contains unsupported datatypes such as Unions

pub fn try_new_with_options( writer: W, arrow_schema: Arc<Schema>, options: ArrowWriterOptions ) -> Result<ArrowWriter<W>, ParquetError>

Try to create a new Arrow writer with ArrowWriterOptions.

The writer will fail if:

• a SerializedFileWriter cannot be created from the ParquetWriter
• the Arrow schema contains unsupported datatypes such as Unions

pub fn flushed_row_groups(&self) -> &[Arc<RowGroupMetaData>]

Returns metadata for any flushed row groups

pub fn in_progress_size(&self) -> usize

Returns the estimated length in bytes of the current in progress row group

pub fn in_progress_rows(&self) -> usize

Returns the number of rows buffered in the in progress row group

pub fn bytes_written(&self) -> usize

Returns the number of bytes written by this instance

pub fn write(&mut self, batch: &RecordBatch) -> Result<(), ParquetError>

Encodes the provided RecordBatch

If this would cause the current row group to exceed WriterProperties::max_row_group_size rows, the contents of batch will be written to one or more row groups such that all but the final row group in the file contain WriterProperties::max_row_group_size rows.

This will fail if the batch’s schema does not match the writer’s schema.

pub fn flush(&mut self) -> Result<(), ParquetError>

Flushes all buffered rows into a new row group

pub fn append_key_value_metadata(&mut self, kv_metadata: KeyValue)

Additional KeyValue metadata to be written in addition to those from WriterProperties

This method provide a way to append kv_metadata after write RecordBatch

pub fn inner(&self) -> &W

Returns a reference to the underlying writer.

pub fn inner_mut(&mut self) -> &mut W

Returns a mutable reference to the underlying writer.

It is inadvisable to directly write to the underlying writer, doing so will likely result in a corrupt parquet file

pub fn into_inner(self) -> Result<W, ParquetError>

Flushes any outstanding data and returns the underlying writer.

pub fn finish(&mut self) -> Result<FileMetaData, ParquetError>

Close and finalize the underlying Parquet writer

Unlike Self::close this does not consume self

Attempting to write after calling finish will result in an error

pub fn close(self) -> Result<FileMetaData, ParquetError>

Close and finalize the underlying Parquet writer

Trait Implementations

impl<W> Debug for ArrowWriter<W>

where W: Write + Send,

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<W> RecordBatchWriter for ArrowWriter<W>

where W: Write + Send,

fn write(&mut self, batch: &RecordBatch) -> Result<(), ArrowError>

Write a single batch to the writer.

fn close(self) -> Result<(), ArrowError>

Write footer or termination data, then mark the writer as done.