Trait polyester::Polyester

pub trait Polyester<T> {
    fn par_fold<Acc, InnerFold, OuterFold>(
        self, 
        seed: Acc, 
        inner_fold: InnerFold, 
        outer_fold: OuterFold
    ) -> Acc
    where
        Acc: Clone + Send + 'static,
        InnerFold: Fn(Acc, T) -> Acc + Send + Sync + 'static,
        OuterFold: Fn(Acc, Acc) -> Acc;
    fn par_map<Map, Out>(self, map: Map) -> ParMap<Self, Map, Out>
    where
        Self: Sized,
        Map: Fn(T) -> Out + Send + Sync + 'static,
        Out: Send + 'static;
}

A trait to extend Iterators with consumers that work in parallel.

This trait is applied to any iterator where it and its items are Send, allowing them to be processed by multiple threads. Importing the trait into your code allows these adaptors to be used like any other iterator adaptor - the only difference is that between the time they're started and when they finish, they'll have spawned a number of threads to perform their work.

Implementation Note

It's worth noting that even though this promises parallel processing, that's no guarantee that it will be faster than just doing it sequentially. The iterator itself is a bottleneck for the processing, since it needs an exclusive &mut self borrow to get each item. This library attempts to get around that by draining the items in a separate thread into a cache that the workers load from, but the synchronization costs for this mean that switching map for par_map (for example) is not a universal win. Because of this, these adapters are only expected to speed up processing if your source iterator is rather quick, and the closures you hand to the adapters are not.

Required Methods

fn par_fold<Acc, InnerFold, OuterFold>(
    self,
    seed: Acc,
    inner_fold: InnerFold,
    outer_fold: OuterFold
) -> Acc where
    Acc: Clone + Send + 'static,
    InnerFold: Fn(Acc, T) -> Acc + Send + Sync + 'static,
    OuterFold: Fn(Acc, Acc) -> Acc, 

Fold the iterator in parallel.

This method works in two parts:

1. Use a set of threads to fold items individually into a per-thread "sub-accumulator" using inner_fold. Each per-thread sub-accumulator begins with a clone of seed.
2. Once the source iterator is exhausted and has no more items, collect each intermediate sub-accumulator into a final accumulator, starting with the first thread's personal sub-accumulator and folding additional sub-accumulators using outer_fold.

If there are no items in the iterator, seed is returned untouched.

Example

use polyester::Polyester;

let my_results = (0..1_000_000).par_fold(
    vec![],
    |mut acc, it| {
        acc.push(some_expensive_computation(it));
        acc
    },
    |mut left, right| {
        left.extend(right);
        left
    }
);

fn par_map<Map, Out>(self, map: Map) -> ParMap<Self, Map, Out> where
    Self: Sized,
    Map: Fn(T) -> Out + Send + Sync + 'static,
    Out: Send + 'static, 

Maps the given closure onto each element in the iterator, in parallel.

The ParMap adaptor returned by this function starts up a thread pool to run map on each item of the iterator. The result of each map is then passed back to the calling thread, where it can then be returned by ParMap's Iterator implementation. Note that ParMap will yield items in the order the threads return them, which may not be the same as the order the source iterator does.

The ParMap adaptor does not start its thread pool until it is first polled, after which it will block for the next item until the iterator is exhausted.

Example

use polyester::Polyester;

let my_results = (0..1_000_000).par_map(|it| some_expensive_computation(it))
                               .collect::<Vec<_>>();

Implementors

• impl<T, I> Polyester<T> for I where
    I: Iterator<Item = T> + Send + 'static,
    T: Send + 'static,