Module datafusion::physical_plan

Traits for physical query plan, supporting parallel execution for partitioned relations.

Re-exports

• pub use self::metrics::Metric;
• pub use display::DefaultDisplay;
• pub use display::DisplayAs;
• pub use display::DisplayFormatType;
• pub use display::VerboseDisplay;

Modules

• aggregates
  Aggregates functionalities
• analyze
  Defines the ANALYZE operator
• coalesce_batches
  CoalesceBatchesExec combines small batches into larger batches for more efficient use of vectorized processing by upstream operators.
• coalesce_partitions
  Defines the merge plan for executing partitions in parallel and then merging the results into a single partition
• common
  Defines common code used in execution plans
• display
  Implementation of physical plan display. See crate::physical_plan::displayable for examples of how to format
• empty
  EmptyRelation execution plan
• explain
  Defines the EXPLAIN operator
• expressions
  Defines physical expressions that can evaluated at runtime during query execution
• filter
  FilterExec evaluates a boolean predicate against all input batches to determine which rows to include in its output batches.
• functions
  Declaration of built-in (scalar) functions. This module contains built-in functions’ enumeration and metadata.
• hash_utils
  Functionality used both on logical and physical plans
• insert
  Execution plan for writing data to DataSinks
• joins
  DataFusion Join implementations
• limit
  Defines the LIMIT plan
• memory
  Execution plan for reading in-memory batches of data
• metrics
  Metrics for recording information about execution
• projection
  Defines the projection execution plan. A projection determines which columns or expressions are returned from a query. The SQL statement SELECT a, b, a+b FROM t1 is an example of a projection on table t1 where the expressions a, b, and a+b are the projection expressions. SELECT without FROM will only evaluate expressions.
• repartition
  The repartition operator maps N input partitions to M output partitions based on a partitioning scheme (according to flag preserve_order ordering can be preserved during repartitioning if its input is ordered).
• sorts
  Sort functionalities
• stream
  Stream wrappers for physical operators
• streaming
  Execution plan for streaming PartitionStream
• tree_node
  This module provides common traits for visiting or rewriting tree nodes easily.
• udaf
  This module contains functions and structs supporting user-defined aggregate functions.
• udf
  UDF support
• union
  The Union operator combines multiple inputs with the same schema
• unnest
  Defines the unnest column plan for unnesting values in a column that contains a list type, conceptually is like joining each row with all the values in the list column.
• values
  Values execution plan
• windows
  Physical expressions for window functions

Structs

• ColumnStatistics
  Statistics for a column within a relation
• EmptyRecordBatchStream
  EmptyRecordBatchStream can be used to create a RecordBatchStream that will produce no results
• Statistics
  Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.

Enums

• ColumnarValue
  Represents the result of evaluating an expression: either a single ScalarValue or an ArrayRef.
• Distribution
  Distribution schemes
• Partitioning
  Partitioning schemes supported by operators.

Traits

• Accumulator
  Describes an aggregate functions’s state.
• AggregateExpr
  An aggregate expression that:
• ExecutionPlan
  ExecutionPlan represent nodes in the DataFusion Physical Plan.
• ExecutionPlanVisitor
  Trait that implements the Visitor pattern for a depth first walk of ExecutionPlan nodes. pre_visit is called before any children are visited, and then post_visit is called after all children have been visited. To use, define a struct that implements this trait and then invoke [‘accept’].
• PhysicalExpr
  Expression that can be evaluated against a RecordBatch A Physical expression knows its type, nullability and how to evaluate itself.
• RecordBatchStream
  Trait for types that stream arrow::record_batch::RecordBatch
• WindowExpr
  Common trait for window function implementations

Functions

• accept
  Visit all children of this plan, according to the order defined on ExecutionPlanVisitor.
• collect
  Execute the ExecutionPlan and collect the results in memory
• collect_partitioned
  Execute the ExecutionPlan and collect the results in memory
• displayable
  Return a wrapper around an ExecutionPlan which can be displayed in various easier to understand ways.
• execute_stream
  Execute the ExecutionPlan and return a single stream of results
• execute_stream_partitioned
  Execute the ExecutionPlan and return a vec with one stream per output partition
• need_data_exchange
  Indicate whether a data exchange is needed for the input of plan, which will be very helpful especially for the distributed engine to judge whether need to deal with shuffling. Currently there are 3 kinds of execution plan which needs data exchange 1. RepartitionExec for changing the partition number between two operators 2. CoalescePartitionsExec for collapsing all of the partitions into one without ordering guarantee 3. SortPreservingMergeExec for collapsing all of the sorted partitions into one with ordering guarantee
• ordering_equivalence_properties_helper
  Retrieves the ordering equivalence properties for a given schema and output ordering.
• project_schema
  Applies an optional projection to a SchemaRef, returning the projected schema
• visit_execution_plan
  Recursively calls pre_visit and post_visit for this node and all of its children, as described on ExecutionPlanVisitor
• with_new_children_if_necessary
  Returns a copy of this plan if we change any child according to the pointer comparison. The size of children must be equal to the size of ExecutionPlan::children().

Type Definitions

• SendableRecordBatchStream
  Trait for a [Stream] of RecordBatches