Module profiling 

Learned column profiles with drift detection (#708).

A run’s written records are folded into one RunProfile by a Profiler — per top-level column: null rate, type mix, distinct-count estimate, numeric / string summaries and the most frequent values, all in bounded memory (see sketch). detect_drift then compares the profile against a rolling baseline of earlier runs and reports statistically significant changes per column, with no thresholds to write. ProfilingSink is the decorator that feeds a pipeline’s sink writes into the profiler.

The baseline store, the post-run evaluation, the catalog surfaces and the faucet profiling verbs live in the CLI; this module is the pure core.

Vocabulary: profiling is this feature. A config profile (profiles:, --profile) is an unrelated config overlay.

Re-exports

pub use column::ColumnProfile;

pub use column::NumericSummary;

pub use column::Profiler;

pub use column::RunProfile;

pub use column::StringSummary;

pub use column::TopValue;

pub use column::TypeCounts;

pub use drift::DriftMetric;

pub use drift::ProfileDrift;

pub use drift::detect_drift;

pub use drift::population_stability_index;

pub use sink::ProfilingSink;

pub use spec::OnProfileDrift;

pub use spec::ProfilingSpec;

Modules

column

Per-run column profiling (#708): the accumulator that turns written pages into one RunProfile — a ColumnProfile per top-level column with null rate, type mix, distinct estimate, numeric / string summaries and the most frequent values. Nested objects and arrays count as one column (their canonical JSON is the value), matching schema-drift semantics.

drift

Pure profile-drift detection (#708): compare one run’s RunProfile against the rolling baseline of earlier runs and report per-column findings. Numeric metrics (null rate, distinct count, mean, min, max, string length) go through the shared z-score / IQR test in crate::anomaly; categorical columns are checked for new and vanished frequent values and for a population-stability-index shift.

sink

ProfilingSink — a sink decorator that profiles every record the inner sink accepted (#708). Installed by the CLI executor outermost on the sink, so it observes records after transforms and masking and never sees the _faucet_* metadata columns a lower decorator adds — the profile describes exactly what landed in the destination, and a masked column’s values are its masked values.

sketch

Bounded-memory sketches behind column profiling (#708). Each is a few KB per column regardless of row count, so a 10M-row run profiles in the same memory as a 10-row one.

spec

Config types for the top-level profiling: block (#708).