pub struct RowUniquenessTracker { /* private fields */ }Expand description
Full-stream row-duplicate tracking with bounded memory.
Every record’s fields are folded into a RowSignature and fed to a
CardinalityEstimator: duplicates are counted exactly while the
distinct-row signatures fit the estimator’s exact set, and estimated
(flagged approximate) once it spills to its HLL sketch. Unlike the
per-column reservoirs, this sees whole rows — including null-like values —
so the count is row-aligned by construction.
The exact set holds 64-bit fingerprints of the signatures. Two different
rows share one with probability about n^2 / 2^65 over n distinct rows:
3e-8 at the default limit of one million, 2e-5 at 27 million, 2e-3 at 300
million. A shared fingerprint counts one duplicate that is not there, so
on a large raised limit a max_duplicate_rows 0 failure on a count of one
carries that chance of being wrong (#1292).
Implementations§
Source§impl RowUniquenessTracker
impl RowUniquenessTracker
Sourcepub fn with_exact_limit(limit: Option<usize>) -> Self
pub fn with_exact_limit(limit: Option<usize>) -> Self
A tracker that counts exactly up to limit distinct rows, or up to
EXACT_CARDINALITY_THRESHOLD when None. A limit given here is
kept under memory pressure; see Self::spill.
Sourcepub fn exact_limit(&self) -> usize
pub fn exact_limit(&self) -> usize
Distinct rows this tracker counts exactly before estimating.
pub fn observe(&mut self, signature: String)
pub fn rows_seen(&self) -> usize
Sourcepub fn duplicate_rows(&self) -> usize
pub fn duplicate_rows(&self) -> usize
Rows minus distinct rows; exact until the estimator spills.
pub fn is_approximate(&self) -> bool
Sourcepub fn mark_incomparable(&mut self)
pub fn mark_incomparable(&mut self)
Record that the signatures folded in so far were not all made against the same schema, so they cannot be compared with each other.
pub fn merge(&mut self, other: &RowUniquenessTracker)
pub fn memory_usage_bytes(&self) -> usize
Sourcepub fn exact_bytes(&self) -> usize
pub fn exact_bytes(&self) -> usize
Heap held by the exact set of row signatures: what Self::spill frees.
Sourcepub fn budgeted_exact_bytes(&self) -> usize
pub fn budgeted_exact_bytes(&self) -> usize
The part of Self::exact_bytes a memory budget answers for: all of
it, unless the caller set the limit. That set is the caller’s choice
of memory and stays outside the budget, so it neither spills the
column sets nor shrinks the reservoirs on every chunk (#1292).
Sourcepub fn spill(&mut self) -> usize
pub fn spill(&mut self) -> usize
Answer the duplicate count from the sketch from here on, freeing the exact set. For memory pressure; the count is then reported approximate. Returns the bytes freed.
A tracker whose limit the caller set keeps its exact set and frees nothing: the limit is the caller’s choice of memory for an exact count, and a count that ended where the machine’s budget did would make the verdict depend on the machine (#1292).
Sourcepub fn summary(&self) -> Option<RowDuplicateSummary>
pub fn summary(&self) -> Option<RowDuplicateSummary>
Summary for quality metrics, or None when no rows were observed
(e.g. an engine that never fed whole records through this tracker) or
when the signatures seen are not comparable with one another.