Trait polyester::Polyester
[−]
[src]
pub trait Polyester<T> { fn par_fold<Acc, InnerFold, OuterFold>(
self,
seed: Acc,
inner_fold: InnerFold,
outer_fold: OuterFold
) -> Acc
where
Acc: Clone + Send + 'static,
InnerFold: Fn(Acc, T) -> Acc + Send + Sync + 'static,
OuterFold: Fn(Acc, Acc) -> Acc; fn par_map<Map, Out>(self, map: Map) -> ParMap<Self, Map, Out>
where
Self: Sized,
Map: Fn(T) -> Out + Send + Sync + 'static,
Out: Send + 'static; }
A trait to extend Iterator
s with consumers that work in parallel.
This trait is applied to any iterator where it and its items are Send
, allowing them to be
processed by multiple threads. Importing the trait into your code allows these adaptors to be
used like any other iterator adaptor - the only difference is that between the time they're
started and when they finish, they'll have spawned a number of threads to perform their work.
Implementation Note
It's worth noting that even though this promises parallel processing, that's no guarantee that
it will be faster than just doing it sequentially. The iterator itself is a bottleneck for the
processing, since it needs an exclusive &mut self
borrow to get each item. This library
attempts to get around that by draining the items in a separate thread into a cache that the
workers load from, but the synchronization costs for this mean that switching map
for
par_map
(for example) is not a universal win. Because of this, these adapters are only
expected to speed up processing if your source iterator is rather quick, and the closures you
hand to the adapters are not.
Required Methods
fn par_fold<Acc, InnerFold, OuterFold>(
self,
seed: Acc,
inner_fold: InnerFold,
outer_fold: OuterFold
) -> Acc where
Acc: Clone + Send + 'static,
InnerFold: Fn(Acc, T) -> Acc + Send + Sync + 'static,
OuterFold: Fn(Acc, Acc) -> Acc,
self,
seed: Acc,
inner_fold: InnerFold,
outer_fold: OuterFold
) -> Acc where
Acc: Clone + Send + 'static,
InnerFold: Fn(Acc, T) -> Acc + Send + Sync + 'static,
OuterFold: Fn(Acc, Acc) -> Acc,
Fold the iterator in parallel.
This method works in two parts:
- Use a set of threads to fold items individually into a per-thread "sub-accumulator"
using
inner_fold
. Each per-thread sub-accumulator begins with a clone ofseed
. - Once the source iterator is exhausted and has no more items, collect each intermediate
sub-accumulator into a final accumulator, starting with the first thread's personal
sub-accumulator and folding additional sub-accumulators using
outer_fold
.
If there are no items in the iterator, seed
is returned untouched.
Example
use polyester::Polyester; let my_results = (0..1_000_000).par_fold( vec![], |mut acc, it| { acc.push(some_expensive_computation(it)); acc }, |mut left, right| { left.extend(right); left } );
fn par_map<Map, Out>(self, map: Map) -> ParMap<Self, Map, Out> where
Self: Sized,
Map: Fn(T) -> Out + Send + Sync + 'static,
Out: Send + 'static,
Self: Sized,
Map: Fn(T) -> Out + Send + Sync + 'static,
Out: Send + 'static,
Maps the given closure onto each element in the iterator, in parallel.
The ParMap
adaptor returned by this function starts up a thread pool to run map
on each
item of the iterator. The result of each map
is then passed back to the calling thread,
where it can then be returned by ParMap
's Iterator
implementation. Note that ParMap
will yield items in the order the threads return them, which may not be the same as the
order the source iterator does.
The ParMap
adaptor does not start its thread pool until it is first polled, after which
it will block for the next item until the iterator is exhausted.
Example
use polyester::Polyester; let my_results = (0..1_000_000).par_map(|it| some_expensive_computation(it)) .collect::<Vec<_>>();