Struct datafusion::physical_optimizer::repartition::Repartition
source · pub struct Repartition {}
Expand description
Optimizer that introduces repartition to introduce more parallelism in the plan
For example, given an input such as:
┌─────────────────────────────────┐
│ │
│ ExecutionPlan │
│ │
└─────────────────────────────────┘
▲ ▲
│ │
┌─────┘ └─────┐
│ │
│ │
│ │
┌───────────┐ ┌───────────┐
│ │ │ │
│ batch A1 │ │ batch B1 │
│ │ │ │
├───────────┤ ├───────────┤
│ │ │ │
│ batch A2 │ │ batch B2 │
│ │ │ │
├───────────┤ ├───────────┤
│ │ │ │
│ batch A3 │ │ batch B3 │
│ │ │ │
└───────────┘ └───────────┘
Input Input
A B
This optimizer will attempt to add a RepartitionExec
to increase
the parallelism (to 3 in this case)
┌─────────────────────────────────┐
│ │
│ ExecutionPlan │
│ │
└─────────────────────────────────┘
▲ ▲ ▲ Input now has 3
│ │ │ partitions
┌───────┘ │ └───────┐
│ │ │
│ │ │
┌───────────┐ ┌───────────┐ ┌───────────┐
│ │ │ │ │ │
│ batch A1 │ │ batch A3 │ │ batch B3 │
│ │ │ │ │ │
├───────────┤ ├───────────┤ ├───────────┤
│ │ │ │ │ │
│ batch B2 │ │ batch B1 │ │ batch A2 │
│ │ │ │ │ │
└───────────┘ └───────────┘ └───────────┘
▲ ▲ ▲
│ │ │
└─────────┐ │ ┌──────────┘
│ │ │
│ │ │
┌─────────────────────────────────┐ batches are
│ RepartitionExec(3) │ repartitioned
│ RoundRobin │
│ │
└─────────────────────────────────┘
▲ ▲
│ │
┌─────┘ └─────┐
│ │
│ │
│ │
┌───────────┐ ┌───────────┐
│ │ │ │
│ batch A1 │ │ batch B1 │
│ │ │ │
├───────────┤ ├───────────┤
│ │ │ │
│ batch A2 │ │ batch B2 │
│ │ │ │
├───────────┤ ├───────────┤
│ │ │ │
│ batch A3 │ │ batch B3 │
│ │ │ │
└───────────┘ └───────────┘
Input Input
A B
Implementations§
source§impl Repartition
impl Repartition
Trait Implementations§
source§impl Default for Repartition
impl Default for Repartition
source§fn default() -> Repartition
fn default() -> Repartition
Returns the “default value” for a type. Read more
source§impl PhysicalOptimizerRule for Repartition
impl PhysicalOptimizerRule for Repartition
source§fn optimize(
&self,
plan: Arc<dyn ExecutionPlan>,
config: &ConfigOptions
) -> Result<Arc<dyn ExecutionPlan>>
fn optimize( &self, plan: Arc<dyn ExecutionPlan>, config: &ConfigOptions ) -> Result<Arc<dyn ExecutionPlan>>
Rewrite
plan
to an optimized formsource§fn schema_check(&self) -> bool
fn schema_check(&self) -> bool
A flag to indicate whether the physical planner should valid the rule will not
change the schema of the plan after the rewriting.
Some of the optimization rules might change the nullable properties of the schema
and should disable the schema check.
Auto Trait Implementations§
impl RefUnwindSafe for Repartition
impl Send for Repartition
impl Sync for Repartition
impl Unpin for Repartition
impl UnwindSafe for Repartition
Blanket Implementations§
source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere T: ?Sized,
source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more