Crate pipeliner [] [src]

This crate provides a high-level framework for parallel processing.

Main features:

  • Accept input lazily from an Iterator.
  • Performs work in a user-specified number of threads.
  • Return all output via an Iterator.
  • Optionally buffer output.
  • panics in your worker threads are propagated out of the output Iterator. (No silent loss of data.)
  • No unsafe code.

Since IntoIterators implement Pipeline, you can, for example:

use pipeliner::Pipeline;
for result in (0..100).with_threads(10).map(|x| x + 1) {
    println!("result: {}", result);

And, since the output is also an iterator, you can easily create a pipeline with varying number of threads for each step of work:

use pipeliner::Pipeline;
// You might want a high number of threads for high-latency work:
let results = (0..100).with_threads(50).map(|x| {
    x + 1 // Let's pretend this is high latency. (ex: network access)
// But you might want lower thread usage for cpu-bound work:
.with_threads(4).out_buffer(100).map(|x| {
    x * x // ow my CPUs :p
for result in results {
    println!("result: {}", result);



This is an intermediate data structure which allows you to configure how your pipeline should run.




Things which implement this can be used with the Pipeliner library.