readahead-iterator 0.1.1

Readahead from an iterator on a separate thread, returning items in order and synchronously.
Documentation
  • Coverage
  • 77.78%
    7 out of 9 items documented3 out of 7 items with examples
  • Size
  • Source code size: 12.69 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 2.19 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Links
  • sourcefrog/readahead-iterator
    1 0 0
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • sourcefrog

Readahead adaptor for iterators

Items are generated from the iterator in a separate thread, and returned to the caller as a regular iterator, in the same order.

This is useful when the wrapped iterator does significant work that can be parallelized with other work on the calling thread. For example, if both the iterator and its client are CPU-intensive, they utilize separate cores. Or if the iterator does blocking IO on multiple files, opening of later files can be overlapped with processing of earlier files.

Rayon offers much more powerful ways to parallelize iterators. This adaptor is useful for some simpler and complementary cases:

  • Both the producer and consumer are serial.

  • Items should be produced and consumed one at a time and in order, but production and consumption can be overlapped.

  • The work is potentially IO-bound, so a separate thread can be dedicated to this iterator, rather than using an NCPUS thread pool.