Expand description
Runiq is a utility to filter unique lines from input.
It operates in a much faster pattern than either the Unix sort
or uniq utilities, and without the constraints the two provide
(either sorting input or only filtering sequential duplicates).
Runiq has a focus on memory space rather than throughput, simply because it comes from a need of filtering large streams of data. Having said this, it should be a goal to perform at least as fast as other tools of the same ilk.
Runiq is built mainly as a command line tool, although it can be
used as a library as the Filter trait is exposed publicly. If
you are using Runiq as a library, do not rely on any modules
hidden from the public documentation.
Structs
- Bitset filter backed by a scalable Bloom Filter.
- Digest filter implementation backed by a
HashSet. - Basic filter implementation backed by a
HashSet. - Uniq filter implementation to only remove consecutive duplicates.
Enums
- Enum to store all possible variants of filters.
Traits
- Trait for any type which can be used to filter unique values.