Module datafusion::physical_plan
source · Expand description
Traits for physical query plan, supporting parallel execution for partitioned relations.
Re-exports
pub use self::metrics::Metric;
pub use display::DisplayFormatType;
pub use self::planner::PhysicalPlanner;
Modules
Aggregates functionalities
Defines the ANALYZE operator
CoalesceBatchesExec combines small batches into larger batches for more efficient use of
vectorized processing by upstream operators.
Defines the merge plan for executing partitions in parallel and then merging the results
into a single partition
Defines common code used in execution plans
Implementation of physical plan display. See
crate::physical_plan::displayable
for examples of how to
formatEmptyRelation execution plan
Defines the EXPLAIN operator
Defines physical expressions that can evaluated at runtime during query execution
Execution plans that read file formats
FilterExec evaluates a boolean predicate against all input batches to determine which rows to
include in its output batches.
Declaration of built-in (scalar) functions.
This module contains built-in functions’ enumeration and metadata.
Functionality used both on logical and physical plans
DataFusion Join implementations
Defines the LIMIT plan
Execution plan for reading in-memory batches of data
Metrics for recording information about execution
Physical query planner
Defines the projection execution plan. A projection determines which columns or expressions
are returned from a query. The SQL statement
SELECT a, b, a+b FROM t1
is an example
of a projection on table t1
where the expressions a
, b
, and a+b
are the
projection expressions. SELECT
without FROM
will only evaluate expressions.The repartition operator maps N input partitions to M output partitions based on a
partitioning scheme.
Trait to make Executionplan rewritable
Sort functionalities
Stream wrappers for physical operators
Execution plan for streaming
PartitionStream
Type coercion rules for functions with multiple valid signatures
This module contains functions and structs supporting user-defined aggregate functions.
UDF support
The Union operator combines multiple inputs with the same schema
Defines the unnest column plan for unnesting values in a column that contains a list
type, conceptually is like joining each row with all the values in the list column.
Values execution plan
Physical expressions for window functions
Structs
Statistics for a column within a relation
EmptyRecordBatchStream can be used to create a RecordBatchStream
that will produce no results
Statistics for a relation
Fields are optional and can be inexact because the sources
sometimes provide approximate estimates for performance reasons
and the transformations output are not always predictable.
Enums
Represents the result of evaluating an expression: either a single
ScalarValue
or an [ArrayRef
].Distribution schemes
Partitioning schemes supported by operators.
Traits
An accumulator represents a stateful object that lives throughout the evaluation of multiple rows and
generically accumulates values.
An aggregate expression that:
ExecutionPlan
represent nodes in the DataFusion Physical Plan.Trait that implements the Visitor
pattern for a
depth first walk of
ExecutionPlan
nodes. pre_visit
is called
before any children are visited, and then post_visit
is called
after all children have been visited.
To use, define a struct that implements this trait and then invoke
[‘accept’].Expression that can be evaluated against a RecordBatch
A Physical expression knows its type, nullability and how to evaluate itself.
Trait for types that stream [arrow::record_batch::RecordBatch]
Row-based accumulator where the internal aggregate state(s) are stored using row format.
A window expression that:
Functions
Visit all children of this plan, according to the order defined on
ExecutionPlanVisitor
.Execute the ExecutionPlan and collect the results in memory
Execute the ExecutionPlan and collect the results in memory
Return a wrapper around an
ExecutionPlan
which can be displayed in various easier to
understand ways.Execute the ExecutionPlan and return a single stream of results
Execute the ExecutionPlan and return a vec with one stream per output partition
Indicate whether a data exchange is needed for the input of
plan
, which will be very helpful
especially for the distributed engine to judge whether need to deal with shuffling.
Currently there are 3 kinds of execution plan which needs data exchange
1. RepartitionExec for changing the partition number between two operators
2. CoalescePartitionsExec for collapsing all of the partitions into one without ordering guarantee
3. SortPreservingMergeExec for collapsing all of the sorted partitions into one with ordering guaranteeApplies an optional projection to a [
SchemaRef
], returning the
projected schemaRecursively calls
pre_visit
and post_visit
for this node and
all of its children, as described on ExecutionPlanVisitor
Returns a copy of this plan if we change any child according to the pointer comparison.
The size of
children
must be equal to the size of ExecutionPlan::children()
.
Allow the vtable address comparisons for ExecutionPlan Trait Objects,it is harmless even
in the case of ‘false-native’.Type Definitions
Trait for a stream of record batches.