cue
cue is a very basic library for providing a "streaming" parallel pipeline for long-running tasks which need to limit memory usage. It's specifically intended for scenarios where:
- An expensive computation needs to be run in parallel on many inputs.
- Many worker threads are desired (for example, I commonly use this on 32 CPU machines).
- The number of worker inputs in memory must be limited.
- Aggregating the results has some overhead and may occasionally block several workers if it's handled in the computation threads (i.e. writing to a file, network socket, etc.).
- Accumulating all of the results in memory is not practical (either due to the size of individual result values, the number of items to process, or both).
In writing some long-running CLI tools, I found myself using a pattern for this frequently enough that I put it in a library.
Usage
Here's a basic usage example. This will:
- Spin up a threadpool of scoped threads (no need for
Arc<T>
). - Submit all of the items from the iterator to the worker pool, blocking on submissions which would overfill the work queue's small-ish buffer.
- In parallel, remove each work item from the queue, process it with the worker closure, and submit it to the lock-free "results" queue.
- The joiner closure will be applied to every item in the results queue, serializing/joining the results.
- Logging: the
debug!
log macro will be invoked every 10,000 work items that are processed (this can be disabled -- seeCargo.toml
for features).
extern crate cue;
For an example, see the test in src/lib.rs
. For documentation, see the currently somewhat sparse API docs.
License
MIT, see LICENSE
.