Expand description
Crate for computing function over iterator in a streaming fashion using multi-threading while preserving order.
Examples
let xs: &[u64] = &[100, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5];
let mut ys = Vec::new();
let f = |x| x*x;
let res: Result<usize, ()> = parstream::run(
xs, 4,
// closure which is called for every x in xs
|x| {
std::thread::sleep(std::time::Duration::from_millis(*x));
Ok(f(x))
},
// closure which is called for every result with preserved order
|y| {
ys.push(y);
Ok(())
},
);
assert_eq!(res, Ok(xs.len()));
assert_eq!(ys, xs.iter().map(f).collect::<Vec<_>>());
If one of callbacks will return error, no new tasks will be started and
run
will end as soon as possible (after threads cleanup) to report this
error to caller.
#[derive(Eq, PartialEq, Debug)]
struct MyError(usize);
let xs: &[u64] = &[100, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5];
let mut ys = Vec::new();
let f = |x| x*x;
let res = parstream::run(xs.iter().enumerate(), 4,
|(i, x)| {
std::thread::sleep(std::time::Duration::from_millis(*x));
if *x == 0 { return Err(MyError(i)); }
Ok(f(x))
},
|y| {
ys.push(y);
Ok(())
},
);
assert_eq!(res, Err(MyError(5)));
assert_eq!(ys.len(), 0);
Warnings
The first closure in run
should not panic as it will lead to a deadlock!
Also report thread will not recover after the second closure panic, it will
not result in deadlock, but the second closure will not be called anymore.
Functions
Compute
f(x)
for every x
in xs
using thread pool and call report
for every result and preserve order of elements.