Struct parquet::arrow::async_writer::AsyncArrowWriter

pub struct AsyncArrowWriter<W> { /* private fields */ }

Encodes RecordBatch to parquet, outputting to an AsyncWrite

Memory Usage

This writer eagerly writes data as soon as possible to the underlying AsyncWrite, permitting fine-grained control over buffering and I/O scheduling. However, the columnar nature of parquet forces data for an entire row group to be buffered in memory, before it can be flushed. Depending on the data and the configured row group size, this buffering may be substantial.

Memory usage can be limited by calling Self::flush to flush the in progress row group, although this will likely increase overall file size and reduce query performance. See ArrowWriter for more information.

let mut writer: AsyncArrowWriter<File> = todo!();
let batch: RecordBatch = todo!();
writer.write(&batch).await.unwrap();
// Trigger an early flush if buffered size exceeds 1_000_000
if writer.in_progress_size() > 1_000_000 {
    writer.flush().await.unwrap()
}

Implementations

impl<W: AsyncWrite + Unpin + Send> AsyncArrowWriter<W>

pub fn try_new( writer: W, arrow_schema: SchemaRef, props: Option<WriterProperties> ) -> Result<Self>

Try to create a new Async Arrow Writer

pub fn try_new_with_options( writer: W, arrow_schema: SchemaRef, options: ArrowWriterOptions ) -> Result<Self>

Try to create a new Async Arrow Writer with ArrowWriterOptions

pub fn flushed_row_groups(&self) -> &[RowGroupMetaDataPtr]

Returns metadata for any flushed row groups

pub fn in_progress_size(&self) -> usize

Returns the estimated length in bytes of the current in progress row group

pub fn in_progress_rows(&self) -> usize

Returns the number of rows buffered in the in progress row group

pub fn bytes_written(&self) -> usize

Returns the number of bytes written by this instance

pub async fn write(&mut self, batch: &RecordBatch) -> Result<()>

Enqueues the provided RecordBatch to be written

After every sync write by the inner ArrowWriter, the inner buffer will be checked and flush if at least half full

pub async fn flush(&mut self) -> Result<()>

Flushes all buffered rows into a new row group

pub fn append_key_value_metadata(&mut self, kv_metadata: KeyValue)

Append KeyValue metadata in addition to those in WriterProperties

This method allows to append metadata after RecordBatches are written.

pub async fn close(self) -> Result<FileMetaData>

Close and finalize the writer.

All the data in the inner buffer will be force flushed.