Skip to main content

Module streaming

datarust_profile

Module streaming 

Source
Expand description

Streaming single-pass CSV profiling (the streaming feature).

Profiles a CSV file in one forward pass with bounded memory: per-column Welford accumulators and raw central moments for exact mean/variance/ skewness/kurtosis, reservoir-sampled quantiles + histogram, online Pearson covariances, hashed Cramér’s V contingency tables, and point-biserial group sums — reusing the crate’s interpretation helpers (skewness, kurtosis, histogram, IQR outlier fences) and quality checks unchanged.

Fidelity contract:

  • Exact: row/cell counts, missing counts, mean, variance/std, min/max, categorical frequencies/imbalance, and duplicate rows (when StreamingOptions::compute_duplicates is enabled).
  • Pairwise-complete Pearson: correlations use every row where both columns hold finite values. The in-memory path instead propagates NaN (any missing cell on either side yields NaN for that pair), so values differ on data with missing numeric cells; the streaming definition is the more statistically useful of the two.
  • Approximate: quantiles and histogram (reservoir sampling with capacity StreamingOptions::reservoir_capacity — exact whenever the table has at most that many rows), IQR outlier counts/fractions (derived from the same reservoir sample, so the count refers to the sampled values, not the full column), skewness/kurtosis (single-pass raw-moment accumulation; last-ULP differences vs the two-pass path are possible on large inputs), and unique on categorical columns whose distinct count exceeds StreamingOptions::distinct_cap (a documented lower bound).
  • Column types are decided online: a numeric column permanently converts to categorical at its first non-parseable non-missing cell (parseable-but-non-finite cells such as inf stay numeric and are skipped like missing values, matching the in-memory path).

Memory is bounded per row, not globally: the accumulators are O(rows × columns × constant), but Cramér’s V tables hold up to min(rows, distinct_cap²) entries per column pair, opt-in duplicate tracking stores one key per unique row, and a single physical line or field is buffered at its natural size.

Structs§

StreamingOptions
Options for profile_csv_streaming_with.

Functions§

profile_csv_streaming
Profiles a CSV file in a single streaming pass with StreamingOptions::default.
profile_csv_streaming_with
Profiles a CSV file in a single streaming pass with explicit options.