Crate pipeliner [−] [src]
This crate provides a high-level framework for parallel processing.
Main features:
- Accept input lazily from an Iterator.
- Performs work in a user-specified number of threads.
- Return all output via an Iterator.
- Optionally buffer output.
panic
s in your worker threads are propagated out of the output Iterator. (No silent loss of data.)- No
unsafe
code.
Since IntoIterator
s implement Pipeline, you can, for example:
use pipeliner::Pipeline; for result in (0..100).with_threads(10).map(|x| x + 1) { println!("result: {}", result); }
And, since the output is also an iterator, you can easily create a pipeline with varying number of threads for each step of work:
use pipeliner::Pipeline; // You might want a high number of threads for high-latency work: let results = (0..100).with_threads(50).map(|x| { x + 1 // Let's pretend this is high latency. (ex: network access) }) // But you might want lower thread usage for cpu-bound work: .with_threads(4).out_buffer(100).map(|x| { x * x // ow my CPUs :p }); for result in results { println!("result: {}", result); }
Structs
PipelineBuilder |
This is an intermediate data structure which allows you to configure how your pipeline should run. |
PipelineIter |
Traits
Pipeline |
Things which implement this can be used with the Pipeliner library. |