Module git_repository::parallel

Run computations in parallel, or not based the parallel feature toggle.

in_parallel(…)

The in_parallel(…) is the typical fan-out-fan-in mode of parallelism, with thread local storage made available to a consume(…) function to process input. The result is sent to the Reduce running in the calling thread to aggregate the results into a single output, which is returned by in_parallel().

Interruptions can be achieved by letting the reducers feed(…)` method fail.

It gets a boost in usability as it allows threads to borrow variables from the stack, most commonly the repository itself or the data to work on.

This mode of operation doesn’t lend itself perfectly to being wrapped for async as it appears like a single long-running operation which runs as fast as possible, which is cancellable only by merit of stopping the input or stopping the output aggregation.

reduce::Stepwise

The Stepwise iterator works exactly as in_parallel() except that the processing of the output produced by consume(I, &mut State) -> O is made accessible by the Iterator trait’s next() method. As produced work is not buffered, the owner of the iterator controls the progress made.

Getting the final output of the Reduce is achieved through the consuming Stepwise::finalize() method, which is functionally equivalent to calling in_parallel().

In an async context this means that progress is only made each time next() is called on the iterator, while merely dropping the iterator will wind down the computation without any result.

Maintaining Safety

In order to assure that threads don’t outlive the data they borrow because their handles are leaked, we enforce the 'static lifetime for its inputs, making it less intuitive to use. It is, however, possible to produce suitable input iterators as long as they can hold something on the heap.

Modules

reduce

Structs

EagerIter

Evaluate any iterator in their own thread.

InOrderIter

An iterator which olds iterated items with a sequential ID starting at 0 long enough to dispense them in order.

Enums

EagerIterIf

An conditional EagerIter, which may become a just-in-time iterator running in the main thread depending on a condition.

Traits

Reduce

An trait for aggregating items commonly produced in threads into a single result, without itself needing to be thread safe.

Functions

in_parallel

Read items from input and consume them in multiple threads, whose output output is collected by a reducer. Its task is to aggregate these outputs into the final result returned by this function with the benefit of not having to be thread-safe.

in_parallel_ifparallel

Run in_parallel() only if the given condition() returns true when eagerly evaluated.

in_parallel_with_slice

An experiment to have fine-grained per-item parallelization with built-in aggregation via thread state. This is only good for operations where near-random access isn’t detrimental, so it’s not usually great for file-io as it won’t make use of sorted inputs well. Note that periodic is not guaranteed to be called in case other threads come up first and finish too fast.

join

Runs left and right in parallel, returning their output when both are done.

num_threadsparallel

Returns the amount of threads the system can effectively use as the amount of its logical cores.

optimize_chunk_size_and_thread_limitparallel

Return the ‘optimal’ (size of chunks, amount of threads as Option, amount of threads) to use in in_parallel() for the given desired_chunk_size, num_items, thread_limit and available_threads.

threads

Runs f with a scope to be used for spawning threads that will not outlive the function call. That way it’s possible to handle threads without needing the ’static lifetime for data they interact with.

Type Definitions

SequenceId

A counter for items that are in sequence, to be able to put them back into original order later.