Struct datafusion_optimizer::push_down_filter::PushDownFilter

pub struct PushDownFilter {}

Optimizer rule for pushing (moving) filter expressions down in a plan so they are applied as early as possible.

Introduction

The goal of this rule is to improve query performance by eliminating redundant work.

For example, given a plan that sorts all values where a > 10:

 Filter (a > 10)
   Sort (a, b)

A better plan is to filter the data before the Sort, which sorts fewer rows and therefore does less work overall:

 Sort (a, b)
   Filter (a > 10)  <-- Filter is moved before the sort

However it is not always possible to push filters down. For example, given a plan that finds the top 3 values and then keeps only those that are greater than 10, if the filter is pushed below the limit it would produce a different result.

 Filter (a > 10)   <-- can not move this Filter before the limit
   Limit (fetch=3)
     Sort (a, b)

More formally, a filter-commutative operation is an operation op that satisfies filter(op(data)) = op(filter(data)).

The filter-commutative property is plan and column-specific. A filter on a can be pushed through a Aggregate(group_by = [a], agg=[SUM(b)). However, a filter on SUM(b) can not be pushed through the same aggregate.

Handling Conjunctions

It is possible to only push down part of a filter expression if is connected with ANDs (more formally if it is a “conjunction”).

For example, given the following plan:

Filter(a > 10 AND SUM(b) < 5)
  Aggregate(group_by = [a], agg = [SUM(b))

The a > 10 is commutative with the Aggregate but SUM(b) < 5 is not. Therefore it is possible to only push part of the expression, resulting in:

Filter(SUM(b) < 5)
  Aggregate(group_by = [a], agg = [SUM(b))
    Filter(a > 10)

Handling Column Aliases

This optimizer must sometimes handle re-writing filter expressions when they pushed, for example if there is a projection that aliases a+1 to "b":

Filter (b > 10)
    Projection: [a+1 AS "b"]  <-- changes the name of `a+1` to `b`

To apply the filter prior to the Projection, all references to b must be rewritten to a+1:

Projection: a AS "b"
    Filter: (a + 1 > 10)  <--- changed from b to a + 1

Implementation Notes

This implementation performs a single pass through the plan, “pushing” down filters. When it passes through a filter, it stores that filter, and when it reaches a plan node that does not commute with that filter, it adds the filter to that place. When it passes through a projection, it re-writes the filter’s expression taking into account that projection.

Implementations

impl PushDownFilter

pub fn new() -> Self

Trait Implementations

impl Default for PushDownFilter

fn default() -> PushDownFilter

Returns the “default value” for a type.

impl OptimizerRule for PushDownFilter

fn name(&self) -> &str

A human readable name for this optimizer rule

fn apply_order(&self) -> Option<ApplyOrder>

How should the rule be applied by the optimizer? See comments on ApplyOrder for details.

fn try_optimize( &self, plan: &LogicalPlan, _config: &dyn OptimizerConfig ) -> Result<Option<LogicalPlan>>

Try and rewrite plan to an optimized form, returning None if the plan cannot be optimized by this rule.