Module differential_dataflow::operators::consolidate

Aggregates the weights of equal records into at most one record.

As differential dataflow streams are unordered and taken to be the accumulation of all records, no semantic change happens via consolidate. However, there is a practical difference between a collection that aggregates down to zero records, and one that actually has no records. The underlying system can more clearly see that no work must be done in the later case, and we can drop out of, e.g. iterative computations.

Examples

This example performs a standard "word count", where each line of text is split into multiple words, each word is converted to a word with count 1, and consolidate then accumulates the counts of equivalent words.

stream.flat_map(|line| line.split_whitespace())
      .map(|word| (word, 1))
      .consolidate();

Traits

ConsolidateExt

An extension method for consolidating weighted streams.