Struct StreamReservoirSampler 

pub struct StreamReservoirSampler { /* private fields */ }

Incremental statistics computation for streaming data processing.

This module provides bounded-memory statistical computation using:

• NumericAccumulator for numerically stable mean/variance/stddev (O(1) memory), shared with the batch paths so every engine reports the same numbers
• CardinalityEstimator for distinct counts: exact up to EXACT_CARDINALITY_THRESHOLD distinct values, then a HyperLogLog estimate (~16 KB fixed registers)
• Reservoir sampling for unbiased samples (fixed capacity; total memory depends on the capacity and the length of sampled strings)
• Streaming text-length tracking with min/max/mean/histogram (O(1) memory)

Implementations

impl StreamReservoirSampler

pub fn new(capacity: usize) -> Self

pub fn offer(&mut self, value: &str)

pub fn shrink_to(&mut self, new_capacity: usize)

Keep a uniformly chosen new_capacity of the retained values.

A reservoir is a uniform sample as a set, not slot by slot: the values that filled it first keep their slots until replaced, so truncating kept the head of the stream. Under early memory pressure that made the quality sample the first rows of the source. A uniform subset of a uniform sample is a uniform sample, and later offers keep it one.

pub fn samples(&self) -> &[String]

pub fn memory_usage_bytes(&self) -> usize

pub fn merge(&mut self, other: &StreamReservoirSampler)

Fold another sampler in, leaving every value of the combined population equally likely to have survived.

The two reservoirs stand for populations of very different sizes: a partition of a million rows and one of a hundred each hand over at most capacity values. Pooling the reservoirs and subsampling uniformly would let the small partition supply half the merged sample, so sample values, and the patterns inferred from them, would depend on how the input happened to be split. The number of survivors drawn from each side follows the hypergeometric distribution over the two population counts instead, which is the split a single pass would have produced.

A sampler shrunk under memory pressure holds fewer values than its population earns it. Nothing can be conjured in their place, so the shortfall is filled from the other side rather than returning a sample shorter than the capacity allows.

Trait Implementations

impl Clone for StreamReservoirSampler

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StreamReservoirSampler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.