Struct deltalake::datafusion::parquet::arrow::arrow_reader::ArrowReaderBuilder

pub struct ArrowReaderBuilder<T> { /* private fields */ }

A generic builder for constructing sync or async arrow parquet readers. This is not intended to be used directly, instead you should use the specialization for the type of reader you wish to use

• For a synchronous API - ParquetRecordBatchReaderBuilder
• For an asynchronous API - ParquetRecordBatchStreamBuilder

Implementations

impl<T> ArrowReaderBuilder<T>

pub fn metadata(&self) -> &Arc<ParquetMetaData>

Returns a reference to the ParquetMetaData for this parquet file

pub fn parquet_schema(&self) -> &SchemaDescriptor

Returns the parquet SchemaDescriptor for this parquet file

pub fn schema(&self) -> &Arc<Schema>

Returns the arrow SchemaRef for this parquet file

pub fn with_batch_size(self, batch_size: usize) -> ArrowReaderBuilder<T>

Set the size of RecordBatch to produce. Defaults to 1024 If the batch_size more than the file row count, use the file row count.

pub fn with_row_groups(self, row_groups: Vec<usize>) -> ArrowReaderBuilder<T>

Only read data from the provided row group indexes

pub fn with_projection(self, mask: ProjectionMask) -> ArrowReaderBuilder<T>

Only read data from the provided column indexes

pub fn with_row_selection( self, selection: RowSelection ) -> ArrowReaderBuilder<T>

Provide a RowSelection to filter out rows, and avoid fetching their data into memory.

Row group filtering is applied prior to this, and therefore rows from skipped row groups should not be included in the RowSelection

An example use case of this would be applying a selection determined by evaluating predicates against the Index

It is recommended to enable reading the page index if using this functionality, to allow more efficient skipping over data pages. See ArrowReaderOptions::with_page_index

pub fn with_row_filter(self, filter: RowFilter) -> ArrowReaderBuilder<T>

Provide a RowFilter to skip decoding rows

Row filters are applied after row group selection and row selection

It is recommended to enable reading the page index if using this functionality, to allow more efficient skipping over data pages. See ArrowReaderOptions::with_page_index.

pub fn with_limit(self, limit: usize) -> ArrowReaderBuilder<T>

Provide a limit to the number of rows to be read

The limit will be applied after any Self::with_row_selection and Self::with_row_filter allowing it to limit the final set of rows decoded after any pushed down predicates

It is recommended to enable reading the page index if using this functionality, to allow more efficient skipping over data pages. See ArrowReaderOptions::with_page_index

pub fn with_offset(self, offset: usize) -> ArrowReaderBuilder<T>

Provide an offset to skip over the given number of rows

The offset will be applied after any Self::with_row_selection and Self::with_row_filter allowing it to skip rows after any pushed down predicates

It is recommended to enable reading the page index if using this functionality, to allow more efficient skipping over data pages. See ArrowReaderOptions::with_page_index