pub struct Sketch { /* private fields */ }Expand description
A bottom-k sketch of the distinct values of a column.
The retained hashes are sorted and deduplicated, so the sketch is a function of the set of values and not of the order they were added in.
Implementations§
Source§impl Sketch
impl Sketch
Sourcepub fn new(k: usize) -> Result<Self>
pub fn new(k: usize) -> Result<Self>
An empty sketch that will keep the k smallest hashes.
§Errors
If k is zero, which would make every estimate a division by nothing.
Sourcepub fn add_hash(&mut self, hash: u64)
pub fn add_hash(&mut self, hash: u64)
Adds a value that has already been hashed, for a caller that is hashing anyway.
Sourcepub fn is_exact(&self) -> bool
pub fn is_exact(&self) -> bool
Whether the sketch saw at most k distinct values, in which case it holds all of them and every count it gives is exact rather than estimated.
Sourcepub fn distinct(&self) -> f64
pub fn distinct(&self) -> f64
The estimated number of distinct values, which is the exact number when Sketch::is_exact
holds.
Sourcepub fn union(&self, other: &Self) -> Result<Self>
pub fn union(&self, other: &Self) -> Result<Self>
The union of two sketches, which is the sketch the union of the two columns would have produced.
§Errors
If the two sketches keep a different number of hashes, because then neither one’s threshold applies to the other and no estimate over the pair means anything.
Sourcepub fn jaccard(&self, other: &Self) -> Result<f64>
pub fn jaccard(&self, other: &Self) -> Result<f64>
The estimated Jaccard similarity, which is the size of the intersection of the two value sets over the size of their union.
Section 6.4 wants this to decide whether two columns are drawn from the same universe and should share a dictionary. It is not a decision on its own, because two columns can overlap heavily and still be better off apart if one of them is tiny, but it is what prunes 5,460 pairs down to the handful worth measuring properly.
§Errors
As Sketch::union.