Skip to main content

RowUniquenessTracker

Struct RowUniquenessTracker 

Source
pub struct RowUniquenessTracker { /* private fields */ }
Expand description

Full-stream row-duplicate tracking with bounded memory.

Every record’s fields are folded into a RowSignature and fed to a CardinalityEstimator: duplicates are counted exactly while the distinct-row signatures fit the estimator’s exact set, and estimated (flagged approximate) once it spills to its HLL sketch. Unlike the per-column reservoirs, this sees whole rows — including null-like values — so the count is row-aligned by construction.

The exact set holds 64-bit fingerprints of the signatures. Two different rows share one with probability about n^2 / 2^65 over n distinct rows: 3e-8 at the default limit of one million, 2e-5 at 27 million, 2e-3 at 300 million. A shared fingerprint counts one duplicate that is not there, so on a large raised limit a max_duplicate_rows 0 failure on a count of one carries that chance of being wrong (#1292).

Implementations§

Source§

impl RowUniquenessTracker

Source

pub fn with_exact_limit(limit: Option<usize>) -> Self

A tracker that counts exactly up to limit distinct rows, or up to EXACT_CARDINALITY_THRESHOLD when None. A limit given here is kept under memory pressure; see Self::spill.

Source

pub fn exact_limit(&self) -> usize

Distinct rows this tracker counts exactly before estimating.

Source

pub fn observe(&mut self, signature: String)

Source

pub fn rows_seen(&self) -> usize

Source

pub fn duplicate_rows(&self) -> usize

Rows minus distinct rows; exact until the estimator spills.

Source

pub fn is_approximate(&self) -> bool

Source

pub fn mark_incomparable(&mut self)

Record that the signatures folded in so far were not all made against the same schema, so they cannot be compared with each other.

Source

pub fn merge(&mut self, other: &RowUniquenessTracker)

Source

pub fn memory_usage_bytes(&self) -> usize

Source

pub fn exact_bytes(&self) -> usize

Heap held by the exact set of row signatures: what Self::spill frees.

Source

pub fn budgeted_exact_bytes(&self) -> usize

The part of Self::exact_bytes a memory budget answers for: all of it, unless the caller set the limit. That set is the caller’s choice of memory and stays outside the budget, so it neither spills the column sets nor shrinks the reservoirs on every chunk (#1292).

Source

pub fn spill(&mut self) -> usize

Answer the duplicate count from the sketch from here on, freeing the exact set. For memory pressure; the count is then reported approximate. Returns the bytes freed.

A tracker whose limit the caller set keeps its exact set and frees nothing: the limit is the caller’s choice of memory for an exact count, and a count that ended where the machine’s budget did would make the verdict depend on the machine (#1292).

Source

pub fn summary(&self) -> Option<RowDuplicateSummary>

Summary for quality metrics, or None when no rows were observed (e.g. an engine that never fed whole records through this tracker) or when the signatures seen are not comparable with one another.

Trait Implementations§

Source§

impl Clone for RowUniquenessTracker

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for RowUniquenessTracker

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for RowUniquenessTracker

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> DynClone for T
where T: Clone,

Source§

fn __clone_box(&self, _: Private) -> *mut ()

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V