Struct datafusion::physical_plan::file_format::ParquetExec

pub struct ParquetExec { /* private fields */ }

Execution plan for scanning one or more Parquet partitions

Implementations

impl ParquetExec

pub fn new( base_config: FileScanConfig, predicate: Option<Expr>, metadata_size_hint: Option<usize> ) -> Self

Create a new Parquet reader execution plan provided file list and schema.

pub fn base_config(&self) -> &FileScanConfig

Ref to the base configs

pub fn pruning_predicate(&self) -> Option<&Arc<PruningPredicate>>

Optional reference to this parquet scan’s pruning predicate

pub fn with_parquet_file_reader_factory( self, parquet_file_reader_factory: Arc<dyn ParquetFileReaderFactory> ) -> Self

Optional user defined parquet file reader factory.

ParquetFileReaderFactory complements TableProvider, It enables users to provide custom implementation for data access operations.

If custom ParquetFileReaderFactory is provided, then data access operations will be routed to this factory instead of ObjectStore.

pub fn with_pushdown_filters(self, pushdown_filters: bool) -> Self

If true, any filter Exprs on the scan will converted to a RowFilter in the ParquetRecordBatchStream. These filters are applied by the parquet decoder to skip unecessairly decoding other columns which would not pass the predicate. Defaults to false

pub fn with_reorder_filters(self, reorder_filters: bool) -> Self

If true, the RowFilter made by pushdown_filters may try to minimize the cost of filter evaluation by reordering the predicate Exprs. If false, the predicates are applied in the same order as specified in the query. Defaults to false.

pub fn with_enable_page_index(self, enable_page_index: bool) -> Self

If enabled, the reader will read the page index This is used to optimise filter pushdown via RowSelector and RowFilter by eliminating unnecessary IO and decoding

pub fn get_repartitioned( &self, target_partitions: usize, repartition_file_min_size: usize ) -> Self

Redistribute files across partitions according to their size

Trait Implementations

impl Clone for ParquetExec

fn clone(&self) -> ParquetExec

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ParquetExec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ExecutionPlan for ParquetExec

fn as_any(&self) -> &dyn Any

Return a reference to Any that can be used for downcasting

fn output_partitioning(&self) -> Partitioning

Get the output partitioning of this plan

fn schema(&self) -> SchemaRef

Get the schema for this execution plan

fn children(&self) -> Vec<Arc<dyn ExecutionPlan>>

Get a list of child execution plans that provide the input for this plan. The returned list will be empty for leaf nodes, will contain a single value for unary nodes, or two values for binary nodes (such as joins).

fn output_ordering(&self) -> Option<&[PhysicalSortExpr]>

If the output of this operator within each partition is sorted, returns Some(keys) with the description of how it was sorted.

fn with_new_children( self: Arc<Self>, _: Vec<Arc<dyn ExecutionPlan>> ) -> Result<Arc<dyn ExecutionPlan>>

Returns a new plan where all children were replaced by new plans.

fn execute( &self, partition_index: usize, ctx: Arc<TaskContext> ) -> Result<SendableRecordBatchStream>

creates an iterator

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format this ExecutionPlan to f in the specified type.

fn metrics(&self) -> Option<MetricsSet>

Return a snapshot of the set of Metrics for this ExecutionPlan.

fn statistics(&self) -> Statistics

Returns the global output statistics for this ExecutionPlan node.

fn unbounded_output(&self, _children: &[bool]) -> Result<bool>

Specifies whether this plan generates an infinite stream of records. If the plan does not support pipelining, but it its input(s) are infinite, returns an error to indicate this.

fn required_input_distribution(&self) -> Vec<Distribution>

Specifies the data distribution requirements for all the children for this operator, By default it’s [Distribution::UnspecifiedDistribution] for each child,

fn required_input_ordering(&self) -> Vec<Option<&[PhysicalSortExpr]>>

Specifies the ordering requirements for all of the children For each child, it’s the local ordering requirement within each partition rather than the global ordering

fn maintains_input_order(&self) -> Vec<bool>

Returns false if this operator’s implementation may reorder rows within or between partitions.

fn benefits_from_input_partitioning(&self) -> bool

Returns true if this operator would benefit from partitioning its input (and thus from more parallelism). For operators that do very little work the overhead of extra parallelism may outweigh any benefits

fn equivalence_properties(&self) -> EquivalenceProperties

Get the EquivalenceProperties within the plan