Module physical_optimizer

re-export of datafusion_physical_optimizer crate

Modules

aggregate_statistics

Utilizing exact statistics from sources to avoid scanning data

coalesce_batches

CoalesceBatches optimizer that groups batches together rows in bigger batches to avoid overhead with small batches

combine_partial_final_agg

CombinePartialFinalAggregate optimizer rule checks the adjacent Partial and Final AggregateExecs and try to combine them if necessary

enforce_distribution

EnforceDistribution optimizer rule inspects the physical plan with respect to distribution requirements and adds RepartitionExecs to satisfy them when necessary. If increasing parallelism is beneficial (and also desirable according to the configuration), this rule increases partition counts in the physical plan.

enforce_sorting

EnforceSorting optimizer rule inspects the physical plan with respect to local sorting requirements and does the following:

filter_pushdown

join_selection

The JoinSelection rule tries to modify a given plan so that it can accommodate infinite sources and utilize statistical information (if there is any) to obtain more performant plans. To achieve the first goal, it tries to transform a non-runnable query (with the given infinite sources) into a runnable query by replacing pipeline-breaking join operations with pipeline-friendly ones. To achieve the second goal, it selects the proper PartitionMode and the build side using the available statistics for hash joins.

limit_pushdown

LimitPushdown pushes LIMIT down through ExecutionPlans to reduce data transfer as much as possible.

limited_distinct_aggregation

A special-case optimizer rule that pushes limit into a grouped aggregation which has no aggregate expressions or sorting requirements

optimizer

Physical optimizer traits

output_requirements

The GlobalOrderRequire optimizer rule either:

projection_pushdown

This file implements the ProjectionPushdown physical optimization rule. The function remove_unnecessary_projections tries to push down all projections one by one if the operator below is amenable to this. If a projection reaches a source, it can even disappear from the plan entirely.

pruning

PruningPredicate to apply filter Expr to prune “containers” based on statistics (e.g. Parquet Row Groups)

sanity_checker

The SanityCheckPlan rule ensures that a given plan can accommodate its infinite sources, if there are any. It will reject non-runnable query plans that use pipeline-breaking operators on infinite input(s). In addition, it will check if all order and distribution requirements of a plan are satisfied by its children.

topk_aggregation

An optimizer rule that detects aggregate operations that could use a limited bucket count

update_aggr_exprs

An optimizer rule that checks ordering requirements of aggregate expressions and modifies the expressions to work more efficiently if possible.

utils

Traits

PhysicalOptimizerRule

PhysicalOptimizerRule transforms one [‘ExecutionPlan’] into another which computes the same results, but in a potentially more efficient way.