Crate pipeliner

This crate provides a high-level framework for parallel processing.

Main features:

• Accept input lazily from an Iterator.
• Performs work in a user-specified number of threads.
• Return all output via an Iterator.
• Optionally buffer output.
• panics in your worker threads are propagated out of the output Iterator. (No silent loss of data.)
• No unsafe code.

// Import the Pipeline trait to give all Iterators and IntoIterators the 
// .with_threads() method:
use pipeliner::Pipeline;

for result in (0..100).with_threads(10).map(|x| x + 1) {
    println!("result: {}", result);
}

And, since the output is also an iterator, you can easily create a pipeline with varying number of threads for each step of work:

use pipeliner::Pipeline;
// You might want a high number of threads for high-latency work:
let results = (0..100).with_threads(50).map(|x| {
    x + 1 // Let's pretend this is high latency. (ex: network access)
})
// But you might want lower thread usage for cpu-bound work:
.with_threads(4).out_buffer(100).map(|x| {
    x * x // ow my CPUs :p
}); 
for result in results {
    println!("result: {}", result);
}

Structs

PipelineBuilder

This is an intermediate data structure which allows you to configure how your pipeline should run.

Traits

Pipeline

Things which implement this can be used with the Pipeliner library.