Struct ParallelIteratorSource 

pub struct ParallelIteratorSource<Source>
where
    Source: IntoParallelSource,
{ /* private fields */ }

Source that ingests items into a stream using the maximum parallelism. The items are from the iterators returned by a generating function.

Each replica (i.e. each core) will have a different iterator. The iterator are produced by a generating function passed to the ParallelIteratorSource::new method.

Implementations

impl<S> ParallelIteratorSource<S>

where S: IntoParallelSource,

pub fn new(generator: S) -> Self

Create a new source that ingest items into the stream using the maximum parallelism available.

The function passed as argument is cloned in each core, and called to get the iterator for that replica. The first parameter passed to the function is a 0-based index of the replica, while the second is the total number of replicas.

Example

// generate the numbers from 0 to 99 using multiple replicas
let n = 100;
let source = ParallelIteratorSource::new(move |id, instances| {
    let chunk_size = (n + instances - 1) / instances;
    let remaining = n - n.min(chunk_size * id);
    let range = remaining.min(chunk_size);
     
    let start = id * chunk_size;
    let stop = id * chunk_size + range;
    start..stop
});
let s = env.stream(source);

Trait Implementations

impl<S> Clone for ParallelIteratorSource<S>

where S: IntoParallelSource,

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<Source> Debug for ParallelIteratorSource<Source>

where Source: IntoParallelSource,

fn fmt(&self, __f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<Source> Display for ParallelIteratorSource<Source>

where Source: IntoParallelSource,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<S> Operator for ParallelIteratorSource<S>

where S: IntoParallelSource, S::Iter: Send, <S::Iter as Iterator>::Item: Send,

type Out = <<S as IntoParallelSource>::Iter as Iterator>::Item

fn setup(&mut self, metadata: &mut ExecutionMetadata<'_>)

Setup the operator chain. This is called before any call to next and it’s used to initialize the operator. When it’s called the operator has already been cloned and it will never be cloned again. Therefore it’s safe to store replica-specific metadata inside of it.

fn next(&mut self) -> StreamElement<Self::Out>

Take a value from the previous operator, process it and return it.

fn structure(&self) -> BlockStructure

A more refined representation of the operator and its predecessors.

impl<S> Source for ParallelIteratorSource<S>

where S: IntoParallelSource, S::Iter: Send, <S::Iter as Iterator>::Item: Send,

fn replication(&self) -> Replication

The maximum parallelism offered by this operator.