Struct SpillableJoinSelection 

pub struct SpillableJoinSelection { /* private fields */ }

Convert hash joins whose estimated build side cannot fit the per-task memory share into sort-merge joins, which can spill.

Implementations

impl SpillableJoinSelection

pub fn from_capacity() -> Self

Derive the threshold from the process’s capacity decision, honouring SPILL_JOIN_BUILD_BYTES_ENV.

pub fn without_broadcast_rescue(self) -> Self

Forbid the degenerate-broadcast rescue — for a plan that is about to be cut into stages, where an added exchange is an added stage boundary.

pub fn for_local_execution() -> Self

Same threshold as Self::from_capacity, but allowed to choose the grace hash join.

Only for plans that are already decoded and will not be serialized — see distributed_plan::apply_local_spill_strategy.

pub fn with_grace_where_plans_are_never_encoded(self) -> Self

Allow grace only in a process that never encodes a stage plan.

with_krishiv_optimizer_rules is shared by two callers with opposite requirements: the coordinator’s staging planner, whose output must survive datafusion-proto (a GraceHashJoinExec there fails to encode and the scheduler’s response is to run the whole query as a SINGLE TASK), and the one-shot CLI, whose plans never leave the process.

is_single_query_process() separates them exactly: stage building happens only in build_stages_for_parquet_tables, reached solely from krishiv-scheduler’s distributed_batch, i.e. the coordinator daemon, which never declares itself single-query. It is also the same predicate that gates rescue_degenerate_broadcast, which is the point: the rescue exists to make a degenerate broadcast spillable, and grace is the better way to spill one. Enabling them apart is what left the rescue handing every join to sort-merge while grace sat unreachable.

This does not turn grace on — grace_hash_join::enabled() still defaults off. It stops KRISHIV_GRACE_HASH_JOIN from being silently inert in the tier that has the rescue.

pub fn with_threshold(threshold_bytes: Option<u64>) -> Self

Explicit threshold, for tests. Keeps the sort-merge conversion.

pub fn with_threshold_and_grace( threshold_bytes: Option<u64>, grace: bool, ) -> Self

Explicit threshold and algorithm, for tests.

Trait Implementations

impl Debug for SpillableJoinSelection

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PhysicalOptimizerRule for SpillableJoinSelection

fn name(&self) -> &str

A human readable name for this optimizer rule

fn schema_check(&self) -> bool

A flag to indicate whether the physical planner should validate that the rule will not change the schema of the plan after the rewriting. Some of the optimization rules might change the nullable properties of the schema and should disable the schema check.

fn optimize( &self, plan: Arc<dyn ExecutionPlan>, config: &ConfigOptions, ) -> Result<Arc<dyn ExecutionPlan>>

Rewrite plan to an optimized form.

fn optimize_with_context( &self, plan: Arc<dyn ExecutionPlan>, context: &dyn PhysicalOptimizerContext, ) -> Result<Arc<dyn ExecutionPlan>, DataFusionError>

Rewrite plan with access to extended context (statistics registry, etc.).