Struct Stage 

pub struct Stage {
    pub tasks: Vec<ExecutionTask>,
    /* private fields */
}

A unit of isolation for a portion of a physical execution plan that can be executed independently and across a network boundary. It implements ExecutionPlan and can be executed to produce a stream of record batches.

If a stage has input stages, then those input stages will be executed on remote resources and will be provided the remainder of the stage tree.

For example, if our stage tree looks like this:

                      ┌─────────┐
                      │ stage 1 │
                      └───┬─────┘
                          │
                   ┌──────┴────────┐
              ┌────┴────┐     ┌────┴────┐
              │ stage 2 │     │ stage 3 │
              └────┬────┘     └─────────┘
                   │
            ┌──────┴────────┐
       ┌────┴────┐     ┌────┴────┐
       │ stage 4 │     │ Stage 5 │
       └─────────┘     └─────────┘

Then executing Stage 1 will run its plan locally. Stage 1 has two inputs, Stage 2 and Stage 3. We know these will execute on remote resources. As such, the plan for Stage 1 must contain a NetworkShuffleExec node that will read the results of Stage 2 and Stage 3 and coalesce the results.

When Stage 1’s NetworkShuffleExec node is executed, it makes an ArrowFlightRequest to the host assigned in the Stage. It provides the following Stage tree serialized in the body of the Arrow Flight Ticket:

              ┌─────────┐
              │ Stage 2 │
              └────┬────┘
                   │
            ┌──────┴────────┐
       ┌────┴────┐     ┌────┴────┐
       │ Stage 4 │     │ Stage 5 │
       └─────────┘     └─────────┘

The receiving Worker will then execute Stage 2 and will repeat this process.

When Stage 4 is executed, it has no input tasks, so it is assumed that the plan included in that Stage can complete on its own; it’s likely holding a leaf node in the overall physical plan and producing data from a [DataSourceExec].

Fields

tasks: Vec<ExecutionTask>

Our tasks which tell us how finely grained to execute the partitions in the plan

Trait Implementations

impl Clone for Stage

fn clone(&self) -> Stage

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Stage

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.