Skip to main content

ParallelSource

grafeo_core::execution::parallel

Trait ParallelSource 

Source 
pub trait ParallelSource:
    Source
    + Send
    + Sync {
    // Required methods
    fn total_rows(&self) -> Option<usize>;
    fn create_partition(&self, morsel: &Morsel) -> Box<dyn Source>;
    fn num_columns(&self) -> usize;

    // Provided methods
    fn is_partitionable(&self) -> bool { ... }
    fn generate_morsels(
        &self,
        morsel_size: usize,
        source_id: usize,
    ) -> Vec<Morsel> { ... }
}
Expand description

Trait for sources that support parallel partitioning.

Parallel sources can:

  • Report their total row count (if known)
  • Be partitioned into independent morsels
  • Create partition sources for specific morsels

Required Methods§

Source

fn total_rows(&self) -> Option<usize>

Returns the total number of rows in this source, if known.

Returns None if the total is unknown (e.g., for streaming sources).

Source

fn create_partition(&self, morsel: &Morsel) -> Box<dyn Source>

Creates a partition source for the given morsel.

The returned source produces data only for the row range specified in the morsel.

Source

fn num_columns(&self) -> usize

Returns the number of columns in this source.

Provided Methods§

Source

fn is_partitionable(&self) -> bool

Returns whether this source can be partitioned.

Some sources (like streaming or network sources) cannot be partitioned.

Source

fn generate_morsels(&self, morsel_size: usize, source_id: usize) -> Vec<Morsel>

Generates morsels that cover all rows in this source.

Returns an empty vector if the source has no rows or cannot be partitioned.

Implementors§