Module datafusion::physical_optimizer

This module contains a query optimizer that operates against a physical plan and applies rules to a physical plan, such as “Repartition”.

Re-exports

pub use optimizer::PhysicalOptimizerRule;

Modules

aggregate_statistics

Utilizing exact statistics from sources to avoid scanning data

coalesce_batches

CoalesceBatches optimizer that groups batches together rows in bigger batches to avoid overhead with small batches

dist_enforcement

EnforceDistribution optimizer rule inspects the physical plan with respect to distribution requirements and adds RepartitionExecs to satisfy them when necessary.

global_sort_selection

Select the efficient global sort implementation based on sort details.

join_selection

Select the proper PartitionMode and build side based on the avaliable statistics for hash join.

optimizer

Physical optimizer traits

pipeline_checker

The PipelineChecker rule ensures that a given plan can accommodate its infinite sources, if there are any. It will reject non-runnable query plans that use pipeline-breaking operators on infinite input(s).

pipeline_fixer

The PipelineFixer rule tries to modify a given plan so that it can accommodate its infinite sources, if there are any. In other words, it tries to obtain a runnable query (with the given infinite sources) from an non-runnable query by transforming pipeline-breaking operations to pipeline-friendly ones. If this can not be done, the rule emits a diagnostic error message.

pruning

This module contains code to prune “containers” of row groups based on statistics prior to execution. This can lead to significant performance improvements by avoiding the need to evaluate a plan on entire containers (e.g. an entire file)

repartition

Repartition optimizer that introduces repartition nodes to increase the level of parallelism available

sort_enforcement

EnforceSorting optimizer rule inspects the physical plan with respect to local sorting requirements and does the following: