Crate datafusion_physical_plan

Traits for physical query plan, supporting parallel execution for partitioned relations.

Re-exports

• pub use crate::display::DefaultDisplay;
• pub use crate::display::DisplayAs;
• pub use crate::display::DisplayFormatType;
• pub use crate::display::VerboseDisplay;
• pub use crate::metrics::Metric;
• pub use crate::stream::EmptyRecordBatchStream;

Modules

• aggregates
  Aggregates functionalities
• analyze
  Defines the ANALYZE operator
• coalesce_batches
  CoalesceBatchesExec combines small batches into larger batches for more efficient use of vectorized processing by upstream operators.
• coalesce_partitions
  Defines the merge plan for executing partitions in parallel and then merging the results into a single partition
• common
  Defines common code used in execution plans
• display
  Implementation of physical plan display. See crate::displayable for examples of how to format
• empty
  EmptyRelation with produce_one_row=false execution plan
• explain
  Defines the EXPLAIN operator
• expressions
  Defines physical expressions that can evaluated at runtime during query execution
• filter
  FilterExec evaluates a boolean predicate against all input batches to determine which rows to include in its output batches.
• functions
  Declaration of built-in (scalar) functions. This module contains built-in functions’ enumeration and metadata.
• hash_utils
  Functionality used both on logical and physical plans
• insert
  Execution plan for writing data to DataSinks
• joins
  DataFusion Join implementations
• limit
  Defines the LIMIT plan
• memory
  Execution plan for reading in-memory batches of data
• metrics
  Metrics for recording information about execution
• placeholder_row
  EmptyRelation produce_one_row=true execution plan
• projection
  Defines the projection execution plan. A projection determines which columns or expressions are returned from a query. The SQL statement SELECT a, b, a+b FROM t1 is an example of a projection on table t1 where the expressions a, b, and a+b are the projection expressions. SELECT without FROM will only evaluate expressions.
• recursive_query
  Defines the recursive query plan
• repartition
  This file implements the RepartitionExec operator, which maps N input partitions to M output partitions based on a partitioning scheme, optionally maintaining the order of the input rows in the output.
• sorts
  Sort functionalities
• stream
  Stream wrappers for physical operators
• streaming
  Generic plans for deferred execution: StreamingTableExec and PartitionStream
• test
  Utilities for testing datafusion-physical-plan
• tree_node
  This module provides common traits for visiting or rewriting tree nodes easily.
• udaf
  This module contains functions and structs supporting user-defined aggregate functions.
• udf
  UDF support
• union
  The Union operator combines multiple inputs with the same schema
• unnest
  Defines the unnest column plan for unnesting values in a column that contains a list type, conceptually is like joining each row with all the values in the list column.
• values
  Values execution plan
• windows
  Physical expressions for window functions
• work_table
  Defines the work table query plan

Macros

• handle_async_state
  The handle_async_state macro adapts the handle_state macro for use in asynchronous operations, particularly when dealing with Poll results within async traits like EagerJoinStream. It polls the asynchronous state-changing function using poll_unpin and then passes the result to handle_state for further processing.
• handle_state
  The handle_state macro is designed to process the result of a state-changing operation, encountered e.g. in implementations of EagerJoinStream. It operates on a StatefulStreamResult by matching its variants and executing corresponding actions. This macro is used to streamline code that deals with state transitions, reducing boilerplate and improving readability.
• internal_err
  Macro wraps Err($ERR) to add backtrace feature

Structs

• ColumnStatistics
  Statistics for a column within a relation
• Statistics
  Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.
• TopK
  Global TopK

Enums

• ColumnarValue
  Represents the result of evaluating an expression: either a single ScalarValue or an ArrayRef.
• Distribution
  How data is distributed amongst partitions. See Partitioning for more details.
• InputOrderMode
  Specifies how the input to an aggregation or window operator is ordered relative to their GROUP BY or PARTITION BY expressions.
• Partitioning
  Output partitioning supported by ExecutionPlans.

Traits

• Accumulator
  Tracks an aggregate function’s state.
• AggregateExpr
  An aggregate expression that:
• ExecutionPlan
  Represent nodes in the DataFusion Physical Plan.
• ExecutionPlanVisitor
  Trait that implements the Visitor pattern for a depth first walk of ExecutionPlan nodes. pre_visit is called before any children are visited, and then post_visit is called after all children have been visited.
• PhysicalExpr
  PhysicalExpr evaluate DataFusion expressions such as A + 1, or CAST(c1 AS int).
• RecordBatchStream
  Trait for types that stream arrow::record_batch::RecordBatch
• WindowExpr
  Common trait for window function implementations

Functions

• accept
  Visit all children of this plan, according to the order defined on ExecutionPlanVisitor.
• collect
  Execute the ExecutionPlan and collect the results in memory
• collect_partitioned
  Execute the ExecutionPlan and collect the results in memory
• displayable
  Return a wrapper around an ExecutionPlan which can be displayed in various easier to understand ways.
• execute_stream
  Execute the ExecutionPlan and return a single stream of RecordBatches.
• execute_stream_partitioned
  Execute the ExecutionPlan and return a vec with one stream per output partition
• get_plan_string
  Utility function yielding a string representation of the given ExecutionPlan.
• need_data_exchange
  Indicate whether a data exchange is needed for the input of plan, which will be very helpful especially for the distributed engine to judge whether need to deal with shuffling. Currently there are 3 kinds of execution plan which needs data exchange 1. RepartitionExec for changing the partition number between two ExecutionPlans 2. CoalescePartitionsExec for collapsing all of the partitions into one without ordering guarantee 3. SortPreservingMergeExec for collapsing all of the sorted partitions into one with ordering guarantee
• project_schema
  Applies an optional projection to a SchemaRef, returning the projected schema
• unbounded_output
• visit_execution_plan
  Recursively calls pre_visit and post_visit for this node and all of its children, as described on ExecutionPlanVisitor
• with_new_children_if_necessary
  Returns a copy of this plan if we change any child according to the pointer comparison. The size of children must be equal to the size of ExecutionPlan::children().

Type Aliases

• SendableRecordBatchStream
  Trait for a [Stream] of RecordBatches