Crate readahead_iterator

Readahead adaptor for iterators.

Items are generated from the iterator in a separate thread, and returned to the caller as a regular iterator, in the same order.

This is useful when the wrapped iterator does significant work that can be parallelized with other work on the calling thread. For example, if both the iterator and its client are CPU-intensive, they utilize separate cores. Or if the iterator does blocking IO on multiple files, opening of later files can be overlapped with processing of earlier files.

The wrapped iterator (and its items) must be Send so that they can be sent between threads.

The iterator must also have 'static lifetime, so that it lives long enough for the thread and wrapper. Often this can be accomplished by making sure the inner iterator is by-value, rather than iterating references through a collection: construct it with into_iter().

For example, to overlap opening files with reading from them:

use std::fs::File;
use std::io::{BufRead, BufReader};
use readahead_iterator::IntoReadahead;

let filenames = vec!["src/lib.rs", "examples/linecount.rs", "Cargo.toml"];
let total_lines: usize = filenames
    .into_iter()
    .filter_map(|filename| {
        File::open(filename.clone())
            .map_err(|err| eprintln!("failed to open {}: {:?}", filename, err))
            .map(|file| (filename, file))
            .ok()
    })
    .readahead(5)
    .map(|(filename, file)| {
        let line_count = BufReader::new(file).lines().count();
        println!("{:>8} {}", line_count, filename);
        line_count
    })
    .sum();
println!("{:>8} TOTAL", total_lines);

Structs

Readahead

An iterator adaptor that evaluates the iterator on a separate thread, and transports the items back to be consumed from the original thread.

Traits

IntoReadahead

Adds a .readahead(buffer_size) method to any iterator.