Macro rayon_iter_concurrent_limit::iter_concurrent_limit

macro_rules! iter_concurrent_limit {
    ( $concurrent_limit:expr, $iterator:expr, for_each, $op:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, try_for_each, $op:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, map, $map_op:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, update, $update_op:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, filter, $filter_op:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, filter_map, $filter_op:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, any, $predicate:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, all, $predicate:expr ) => { ... };
    ( $concurrent_limit:expr, $iterator:expr, $method:ident, $predicate:expr ) => { ... };
}

Apply a method on a rayon::iter::IndexedParallelIterator with a limit on the number of concurrent executions of the function passed to the method.

Concurrent executions are limited by chunking the iterator to reduce the number of work items. The crate root documentation explains the motivation for this approach, provides further details on the underlying implementation of the macro, and details its limitations.

Arguments

The macro arguments are (concurrent_limit, iterator, method, function):

• concurrent_limit is a usize specifying the maximum concurrent executions of function.
  • A concurrent_limit of zero means no concurrent limit. Some methods will skip internal chunking in this case.
• iterator implements std::iter::IntoIterator and rayon::iter::IntoParallelIterator
  • The parallel iterator must implement rayon::iter::IndexedParallelIterator.
• method is the name of a supported iterator method:
  • Only methods which call a supplied function are supported.
  • Operations without a function (e.g. min, max) will not allocate and there is little benefit in setting a concurrent limit for such methods.
  • Not every iterator method matching this criteria is currently supported.
• function is a function compatible with method, such as an operation, predicate, etc.
  • The function is called sequentially on the items in each chunk, but in parallel over the chunks, with the number of concurrent executions upper bounded by the concurrent_limit.
  • Parallel rayon methods executed in the function will implicitly utilise the global thread pool unless an alternative thread pool has been installed (see [rayon::ThreadPool]).

Examples

for_each

let op = |i: usize| {
    let alloc = vec![i; 1000]; // limited concurrency
    alloc.into_par_iter().for_each(|_j| {}); // runs on all threads
};
iter_concurrent_limit!(2, (0..10), for_each, op);

try_for_each

let op = |i: usize| {
    let alloc = vec![i; 1000]; // limited concurrency
    alloc.into_par_iter().for_each(|_j| {}); // runs on all threads
    Ok::<(), std::io::Error>(())
};
iter_concurrent_limit!(2, (0..10), try_for_each, op)?;

map

let op = |i: usize| {
    let alloc = vec![i; 1000]; // limited concurrency
    alloc.into_par_iter().sum::<usize>() // runs on all threads
};
let sum =
    iter_concurrent_limit!(2, (0..100), map, op)
    .sum::<usize>();
assert_eq!(sum, (0..100).into_iter().map(op).sum::<usize>());

filter

let op = |i: &usize| -> bool {
    // ... do work with limited concurrency
    i % 2 == 0
};
let even =
    iter_concurrent_limit!(2, (0..100), filter, op)
    .collect::<Vec<usize>>();
assert_eq!(even, (0..100).into_iter().filter(op).collect::<Vec<usize>>());

filter_map

let op = |i: usize| -> Option<usize> {
    // ... do work with limited concurrency
    if i % 2 == 0 { Some(i * 2) } else { None }
};
let even_doubled =
    iter_concurrent_limit!(2, (0..100), filter_map, op)
    .collect::<Vec<usize>>();
assert_eq!(even_doubled, (0..100).into_iter().filter_map(op).collect::<Vec<usize>>());

any

let op = |i: usize| -> bool {
    // ... do work with limited concurrency
    i == 50
};
let any_eq_50 = iter_concurrent_limit!(2, (0..100), any, op);
assert_eq!(any_eq_50, (0..100).into_iter().any(op));

all

let op = |i: usize| -> bool {
    // ... do work with limited concurrency
    i == 50
};
let all_eq_50 = iter_concurrent_limit!(2, (0..100), all, op);
assert_eq!(all_eq_50, (0..100).into_iter().all(op));