Module limit_pushdown 

LimitPushdown pushes LIMIT down through ExecutionPlans to reduce data transfer as much as possible.

Plan Limit Absorption

In addition to pushing down GlobalLimitExec and LocalLimitExec nodes in the plan, some operators can “absorb” a limit and stop early during execution.

Background: vectorized volcano execution model

DataFusion uses a batched volcano model. For most operators, output is produced in batches of datafusion.execution.batch_size (default 8192), so the batch sizes typically look like:

8192, 8192, ..., 8192, 100 (the final batch may be partial)

Example

For a join with an expensive, selective predicate:

GlobalLimitExec: skip=0, fetch=10
-- NestedLoopJoinExec(on=expr_expensive_and_selective)
--- DataSourceExec()
--- DataSourceExec()

Under this model, NestedLoopJoinExec would keep working until it can emit a full batch (8192 rows), even though the query only needs 10. If the limit cannot be pushed below the join, we can still embed it inside the join so it stops once the limit is satisfied. The transformed plan looks like:

NestedLoopJoinExec(on=expr_expensive_and_selective, fetch=10)
--- DataSourceExec()
--- DataSourceExec()

Implementation

The current optimizer rule optionally pushes fetch requirements into operators via ExecutionPlan::with_fetch.

To support early termination in operators, LimitedBatchCoalescer can help manage the output buffer.

Reference implementation in Hash Join: https://github.com/apache/datafusion/pull/20228

Structs

GlobalRequirements

This is a “data class” we use within the LimitPushdown rule to push down limits in the plan. GlobalRequirements are hold as a rule-wide state and holds the fetch and skip information. The struct also has a field named satisfied which means if the “current” plan is valid in terms of limits or not.

LimitPushdown

This rule inspects ExecutionPlan’s and pushes down the fetch limit from the parent to the child if applicable.

Functions

pushdown_limit_helper

This function is the main helper function of the LimitPushDown rule. The helper takes an ExecutionPlan and a global (algorithm) state which is an instance of GlobalRequirements and modifies these parameters while checking if the limits can be pushed down or not.