Trait polyester::Polyester [] [src]

pub trait Polyester<T> {
    fn par_fold<Acc, InnerFold, OuterFold>(
        seed: Acc,
        inner_fold: InnerFold,
        outer_fold: OuterFold
    ) -> Acc
        Acc: Clone + Send + 'static,
        InnerFold: Fn(Acc, T) -> Acc + Send + Sync + 'static,
        OuterFold: Fn(Acc, Acc) -> Acc
fn par_map<Map, Out>(self, map: Map) -> ParMap<Self, Map, Out>
        Self: Sized,
        Map: Fn(T) -> Out + Send + Sync + 'static,
        Out: Send + 'static
; }

A trait to extend Iterators with consumers that work in parallel.

This trait is applied to any iterator where it and its items are Send, allowing them to be processed by multiple threads. Importing the trait into your code allows these adaptors to be used like any other iterator adaptor - the only difference is that between the time they're started and when they finish, they'll have spawned a number of threads to perform their work.

Implementation Note

It's worth noting that even though this promises parallel processing, that's no guarantee that it will be faster than just doing it sequentially. The iterator itself is a bottleneck for the processing, since it needs an exclusive &mut self borrow to get each item. This library attempts to get around that by draining the items in a separate thread into a cache that the workers load from, but the synchronization costs for this mean that switching map for par_map (for example) is not a universal win. Because of this, these adapters are only expected to speed up processing if your source iterator is rather quick, and the closures you hand to the adapters are not.

Required Methods

Fold the iterator in parallel.

This method works in two parts:

  1. Use a set of threads to fold items individually into a per-thread "sub-accumulator" using inner_fold. Each per-thread sub-accumulator begins with a clone of seed.
  2. Once the source iterator is exhausted and has no more items, collect each intermediate sub-accumulator into a final accumulator, starting with the first thread's personal sub-accumulator and folding additional sub-accumulators using outer_fold.

If there are no items in the iterator, seed is returned untouched.


use polyester::Polyester;

let my_results = (0..1_000_000).par_fold(
    |mut acc, it| {
    |mut left, right| {

Maps the given closure onto each element in the iterator, in parallel.

The ParMap adaptor returned by this function starts up a thread pool to run map on each item of the iterator. The result of each map is then passed back to the calling thread, where it can then be returned by ParMap's Iterator implementation. Note that ParMap will yield items in the order the threads return them, which may not be the same as the order the source iterator does.

The ParMap adaptor does not start its thread pool until it is first polled, after which it will block for the next item until the iterator is exhausted.


use polyester::Polyester;

let my_results = (0..1_000_000).par_map(|it| some_expensive_computation(it))