Crate parstream

Crate for computing function over iterator in a streaming fashion using multi-threading while preserving order.

Examples

let xs: &[u64] = &[100, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5];
let mut ys = Vec::new();
let f = |x| x*x;
let res: Result<usize, ()> = parstream::run(
    xs, 4,
    // closure which is called for every x in xs
    |x| {
        std::thread::sleep(std::time::Duration::from_millis(*x));
        Ok(f(x))
    },
    // closure which is called for every result with preserved order
    |y| {
        ys.push(y);
        Ok(())
    },
);

assert_eq!(res, Ok(xs.len()));
assert_eq!(ys, xs.iter().map(f).collect::<Vec<_>>());

If one of callbacks will return error, no new tasks will be started and run will end as soon as possible (after threads cleanup) to report this error to caller.

#[derive(Eq, PartialEq, Debug)]
struct MyError(usize);

let xs: &[u64] = &[100, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5];
let mut ys = Vec::new();
let f = |x| x*x;
let res = parstream::run(xs.iter().enumerate(), 4,
    |(i, x)| {
        std::thread::sleep(std::time::Duration::from_millis(*x));
        if *x == 0 { return Err(MyError(i)); }
        Ok(f(x))
    },
    |y| {
        ys.push(y);
        Ok(())
    },
);

assert_eq!(res, Err(MyError(5)));
assert_eq!(ys.len(), 0);

Warnings

The first closure in run should not panic as it will lead to a deadlock! Also report thread will not recover after the second closure panic, it will not result in deadlock, but the second closure will not be called anymore.

Functions

run

Compute f(x) for every x in xs using thread pool and call report for every result and preserve order of elements.