Struct parquet::arrow::arrow_reader::RowFilter

pub struct RowFilter { /* private fields */ }

A RowFilter allows pushing down a filter predicate to skip IO and decode

This consists of a list of ArrowPredicate where only the rows that satisfy all of the predicates will be returned. Any RowSelection will be applied prior to the first predicate, and each predicate in turn will then be used to compute a more refined RowSelection to use when evaluating the subsequent predicates.

Once all predicates have been evaluated, the final RowSelection is applied to the top-level ProjectionMask to produce the final output RecordBatch.

This design has a couple of implications:

• RowFilter can be used to skip entire pages, and thus IO, in addition to CPU decode overheads
• Columns may be decoded multiple times if they appear in multiple ProjectionMask
• IO will be deferred until needed by a ProjectionMask

As such there is a trade-off between a single large predicate, or multiple predicates, that will depend on the shape of the data. Whilst multiple smaller predicates may minimise the amount of data scanned/decoded, it may not be faster overall.

For example, if a predicate that needs a single column of data filters out all but 1% of the rows, applying it as one of the early ArrowPredicateFn will likely significantly improve performance.

As a counter example, if a predicate needs several columns of data to evaluate but leaves 99% of the rows, it may be better to not filter the data from parquet and apply the filter after the RecordBatch has been fully decoded.

Additionally, even if a predicate eliminates a moderate number of rows, it may still be faster to filter the data after the RecordBatch has been fully decoded, if the eliminated rows are not contiguous.

Implementations

impl RowFilter

pub fn new(predicates: Vec<Box<dyn ArrowPredicate>>) -> Self

Create a new RowFilter from an array of ArrowPredicate