Crate hyperminhash
source ·Expand description
Very fast, constant memory-footprint cardinality approximation, including intersection and union operation. A straight port of Hyperminhash.
As with other cardinality estimators, Hyperminhash
has two advantages when counting very large
sets or streams of elements:
- It uses a single data structure that never grows while counting elements. The structure currently consumes 32kb of memory, allocated on the stack.
- Because there are no indirections due to allocations, the amount of work done for counting a marginal element stays constant.
Given that, a Vec
or HashSet
is usually faster when counting small sets. When counting
streams of millions of elements, Hyperminhash
is much faster and uses much less memory.
use hyperminhash::Sketch;
// A `Sketch` can approximate the unique count of elements it has seen over it's lifetime.
let mut sk = Sketch::default();
// After initialization, a `Sketch` will never allocate.
// Elements added to Sketch need to implement `std::hash::Hash`
sk.add("foobar");
sk.add(1);
sk.add(2);
sk.add(vec![3, 4, 5]);
for i in 0..5000 {
sk.add(i);
}
// There will be some error in the count
assert!(sk.cardinality() > 4_900.0 && sk.cardinality() < 5_100.0);
// Using `std::iter::FromIterator`
let sketch1 = (0..10_000).collect::<Sketch>();
let sketch2 = (5_000..15_000).collect::<Sketch>();
assert!(sketch1.cardinality() > 9_800.0 && sketch1.cardinality() < 10_200.0);
assert!(sketch2.cardinality() > 9_800.0 && sketch2.cardinality() < 10_200.0);
// The intersection of two sketches yields the approximate number of unique
// elements that are in both sets.
let i = sketch1.intersection(&sketch2);
assert!(i > 4_800.0 && i < 5_200.0);
// Merge sketch1 with sketch2
let mut sketch1 = sketch1;
sketch1.union(&sketch2);
assert!(sketch1.cardinality() > 14_800.0 && sketch1.cardinality() < 15_200.0);
// If the `serialize`-feature is used, Sketches can be serialized/deserialized
// from any Reader/Writer.
#[cfg(feature = "serialize")]
{
let mut buffer = Vec::new();
sketch1.save(&mut buffer).expect("Failed to write");
let sketch2 = Sketch::load(&buffer[..]).expect("Failed to read");
assert_eq!(sketch1.cardinality(), sketch2.cardinality());
}
Structs§
- Records the approximate number of unique elements it has seen over it’s lifetime.