Struct DistributedConfig 

#[non_exhaustive]
pub struct DistributedConfig {
    pub files_per_task: usize,
    pub cardinality_task_count_factor: f64,
    pub shuffle_batch_size: usize,
    pub children_isolator_unions: bool,
    pub collect_metrics: bool,
    pub broadcast_joins: bool,
    pub compression: String,
    pub max_tasks_per_stage: usize,
    /* private fields */
}

Configuration for the distributed planner.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

files_per_task: usize

Sets the maximum amount of files that will be assigned to each task. Reducing this number will spawn more tasks for the same number of files. This only applies when estimating tasks for stages containing DataSourceExec nodes with FileScanConfig implementations.

cardinality_task_count_factor: f64

Task multiplying factor for when a node declares that it changes the cardinality of the data:

• If a node is increasing the cardinality of the data, this factor will increase.
• If a node reduces the cardinality of the data, this factor will decrease.
• In any other situation, this factor is left intact.

shuffle_batch_size: usize

Upon shuffling over the network, data streams need to be disassembled in a lot of output partitions, which means the resulting streams might contain a lot of tiny record batches to be sent over the wire. This parameter controls the batch size in number of rows for the CoalesceBatchExec operator that is placed at the top of the stage for sending bigger batches over the wire. If set to 0, batch coalescing is disabled on network shuffle operations.

children_isolator_unions: bool

When encountering a UNION operation, isolate its children depending on the task context. For example, on a UNION operation with 3 children running in 3 distributed tasks, instead of executing the 3 children in each 3 tasks with a DistributedTaskContext of 1/3, 2/3, and 3/3 respectively, Execute:

• The first child in the first task with a DistributedTaskContext of 1/1
• The second child in the second task with a DistributedTaskContext of 1/1
• The third child in the third task with a DistributedTaskContext of 1/1

collect_metrics: bool

Propagate collected metrics from all nodes in the plan across network boundaries so that they can be reconstructed on the head node of the plan.

broadcast_joins: bool

Enable broadcast joins for CollectLeft hash joins. When enabled, the build side of a CollectLeft join is broadcast to all consumer tasks. TODO: This option exists temporarily until we become smarter about when to actually use broadcasting like checking build side size. For now, broadcasting all CollectLeft joins is not always beneficial.

compression: String

The compression used for sending data over the network between workers. It can be set to either zstd, lz4 or none.

max_tasks_per_stage: usize

Maximum tasks that will be assigned per stage during distributed planning. If set to 0, this value is the number of workers returned by the provided WorkerResolver. It defaults to 0.

Implementations

impl DistributedConfig

pub fn with_task_estimator( self, task_estimator: impl TaskEstimator + Send + Sync + 'static, ) -> Self

Appends a TaskEstimator to the list. TaskEstimator will be executed sequentially in order on leaf nodes, and the first one to provide a value is the one that gets to decide how many tasks are used for that [Stage].

pub fn from_config_options( cfg: &ConfigOptions, ) -> Result<&Self, DataFusionError>

Gets the DistributedConfig from the ConfigOptions’s extensions.

pub fn from_config_options_mut( cfg: &mut ConfigOptions, ) -> Result<&mut Self, DataFusionError>

Gets the DistributedConfig from the ConfigOptions’s extensions.

Trait Implementations

impl Clone for DistributedConfig

fn clone(&self) -> DistributedConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl ConfigExtension for DistributedConfig

const PREFIX: &'static str = "distributed"

Configuration namespace prefix to use

impl ConfigField for DistributedConfig

fn set(&mut self, key: &str, value: &str) -> Result<()>

fn visit<V: Visit>( &self, v: &mut V, _key_prefix: &str, _description: &'static str, )

fn reset(&mut self, key: &str) -> Result<(), DataFusionError>

impl Debug for DistributedConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DistributedConfig

fn default() -> Self

Returns the “default value” for a type.

impl ExtensionOptions for DistributedConfig

fn as_any(&self) -> &dyn Any

Return self as Any

fn as_any_mut(&mut self) -> &mut dyn Any

Return self as Any

fn cloned(&self) -> Box<dyn ExtensionOptions>

Return a deep clone of this ExtensionOptions

fn set(&mut self, key: &str, value: &str) -> Result<()>

Set the given key, value pair

fn entries(&self) -> Vec<ConfigEntry>

Returns the ConfigEntry stored in this ExtensionOptions