Crate readahead_iterator 

Readahead adaptor for iterators.

Items are generated from the iterator in a separate thread, and returned to the caller as a regular iterator, in the same order.

This is useful when the wrapped iterator does significant work that can be parallelized with other work on the calling thread.

For example:

1. Both the iterator and its client are CPU-intensive: allowing the iterator to run ahead will let it do some work in parallel on a separate core.
2. The iterator generating work does blocking or lengthy IO such as opening and reading many files: opening the files can proceed in parallel with processing already-open files.

The wrapped iterator (and its items) must be Send so that they can be sent between threads.

The iterator must also have 'static lifetime, so that it lives long enough for the thread and wrapper. Often this can be accomplished by making sure the inner iterator is by-value, rather than iterating references through a collection: construct it with into_iter().

For example, to overlap opening files with reading from them:

use std::fs::File;
use std::io::{BufRead, BufReader};
use readahead_iterator::IntoReadahead;

let filenames = vec!["src/lib.rs", "examples/linecount.rs", "Cargo.toml"];
let total_lines: usize = filenames
    .into_iter()
    .filter_map(|filename| {
        File::open(filename.clone())
            .map_err(|err| eprintln!("failed to open {}: {:?}", filename, err))
            .map(|file| (filename, file))
            .ok()
    })
    .readahead(5)
    .map(|(filename, file)| {
        let line_count = BufReader::new(file).lines().count();
        println!("{:>8} {}", line_count, filename);
        line_count
    })
    .sum();
println!("{:>8} TOTAL", total_lines);

Potential future features:

1. A threaded map across a bounded readahead from the iterator, processing them out of order within a sliding window.

Modules

_changelog

readahead-iterator Release History

Structs

Readahead

An iterator adaptor that evaluates the iterator on a separate thread, and transports the items back to be consumed from the original thread.

Traits

IntoReadahead

Adds a .readahead(buffer_size) method to any iterator.