Skip to main content

QualityPolicy

Struct QualityPolicy 

Source
#[non_exhaustive]
pub struct QualityPolicy { /* private fields */ }
Expand description

What a report must satisfy.

Build one with the chainable setters, then evaluate it against a report. Requirements are evaluated in a fixed order that does not depend on the order they were added, so two callers stating the same policy get the same document out.

Implementations§

Source§

impl QualityPolicy

Source

pub fn new() -> Self

An empty policy. It states no requirement, so evaluating it as-is is a PolicyError::NoRequirements.

Source

pub fn min_rows(self, min: usize) -> Self

Require the scan to have read at least min rows, so an empty or truncated extract fails instead of passing every other check.

Source

pub fn min_quality_score(self, min: f64) -> Self

Require the overall quality score to be at least min (0..=100).

Source

pub fn min_dimension_score(self, dimension: QualityDimension, min: f64) -> Self

Require one dimension’s score to be at least min (0..=100).

Source

pub fn max_null_percentage(self, column: impl Into<String>, max: f64) -> Self

Require the named column’s null percentage to be at most max (0..=100). Takes precedence over max_null_percentage_any for that column.

Source

pub fn max_null_percentage_any(self, max: f64) -> Self

Require every other profiled column’s null percentage to be at most max (0..=100).

Source

pub fn max_null_count(self, column: impl Into<String>, max: usize) -> Self

Require the named column to hold at most max nulls, decided on the exact count. max_null_count(column, 0) states “no nulls” at any row count, where a zero null percentage is decided at the two decimals it is shown at. Takes precedence over max_null_count_any for that column.

Source

pub fn max_null_count_any(self, max: usize) -> Self

Require every other profiled column to hold at most max nulls.

Source

pub fn max_duplicate_rows(self, max: usize) -> Self

Require the report to count at most max duplicate rows.

Source

pub fn unique_columns<I, S>(self, columns: I) -> Self
where I: IntoIterator<Item = S>, S: Into<String>,

Require no non-null value of each named column to occur more than once, as a business key must not repeat (#904). Whole-row duplicates miss a reused key whenever another column differs, and always when a loader adds a row id.

The rule reads the column’s distinct count: its non-null values minus its distinct values are the values that repeat, and the check observes that number against an allowance of zero. Nulls are not values, so they never repeat; max_null_count rules them out. A repeat the scan witnessed stays one however much was not read, so it fails under any scope, and none passes under PolicyScope::FullSource only when the scan covered the source, as with max_duplicate_rows. A distinct count that is an estimate (past one million distinct values, or sooner under memory pressure) or not recorded as exact witnesses nothing, so the check is NotEvaluated::ApproximateCount either way. A column with no distinct count (a nested column) is NotEvaluated::NotAssessed.

This is a requirement on the columns named, independent of the identifier_columns profiling option, which picks the key the uniqueness score and the key_duplicates finding read. Declaring one does not imply the other.

Source

pub fn max_date_age(self, column: impl Into<String>, days: u32) -> Self

Require the named column’s newest date to lie at most days days before the timeliness reference date (#1290), so a feed that stopped moving fails while every other rule still passes it.

The newest date is the column’s max_datetime, a calendar date, and the reference date is the one the report’s timeliness clock recorded (reference_date, or the run’s date on a rolling clock), so the age is counted in calendar days and a saved report evaluated later decides the same. A date after the reference date is fresh. A newest date only gets newer as more values are read, so under PolicyScope::FullSource a fresh date passes whatever the scan missed, provided the column stays a date: on a truncated or sampled scan that holds only for a date type the source declared (NativeType::declares_date), since the rows not read may hold text that types an inferred column as text (#1333). A stale one fails only when the values behind it cover the column: every row read, and the date statistics computed over every value read. A report that records metric_semantics.date_statistics computed them so (#1297); for an older one the retained sample they came from must hold every non-null value. The exception is a column of slash dates whose day and month order the unread values could still change (assumed_day_first or month_first): there a fresh date read on part of the column is NotEvaluated::EvidenceIncomplete too. A column with no newest date (not typed as a date, all null, or statistics not computed) is NotEvaluated::NotAssessed, and a report without a recorded clock is NotEvaluated::SettingUnrecorded on timeliness_clock.

Source

pub fn max_date_gap(self, column: impl Into<String>, days: u32) -> Self

Require the largest gap between the named column’s consecutive distinct days to be at most days (#1300), so a daily file that lost a period from the middle of its range fails while its newest date, schema and most of its rows stay normal. 1 allows no missing day.

The gap is the column’s largest_gap, counted in whole calendar days over every value the profiler read. Under PolicyScope::FullSource the rule decides only when those values cover the column: the values that were not read can fill a hole the read ones show, or open a new one at either edge of the range, so a partial scan is NotEvaluated::EvidenceIncomplete whichever way the read values point. A column with no gap recorded (not typed as a date, all null, a report from before 0.14, or dates spanning more than DATE_GAP_YEAR_LIMIT distinct years) is NotEvaluated::NotAssessed.

Source

pub fn require_quality(self) -> Self

Require the report to carry a quality assessment at all.

Without this, a report with no assessment leaves quality requirements unevaluated and the verdict inconclusive. With it, the absence is itself a violation — which is what a pipeline wants when quality is supposed to be configured on.

Source

pub fn require_dimension(self, dimension: QualityDimension) -> Self

Require a specific dimension to have been assessed.

Source

pub fn max_quality_score_drop(self, points: f64) -> Self

Allow the overall quality score to drop at most points (0..=100) below the baseline’s. Needs evaluate_against.

Under PolicyScope::FullSource, a score computed over a retained sample on either side is read as its recorded whole-source interval, and the drop is decided when every change those intervals allow lies on one side of points (#1035). Each side’s intervals hold together with that side’s confidence_level; when both sides are sampled, the two hold together with probability at least the sum of the two levels minus one.

Source

pub fn max_dimension_score_drop( self, dimension: QualityDimension, points: f64, ) -> Self

Allow one dimension’s score to drop at most points (0..=100) below the baseline’s. Needs evaluate_against. A sampled score is decided on its interval, as in max_quality_score_drop.

Source

pub fn max_null_percentage_increase( self, column: impl Into<String>, points: f64, ) -> Self

Allow the named column’s null percentage to rise at most points (0..=100) above the baseline’s. Takes precedence over max_null_percentage_increase_any for that column. Needs evaluate_against.

Source

pub fn max_null_percentage_increase_any(self, points: f64) -> Self

Allow every other column both reports profiled to have its null percentage rise at most points (0..=100) above the baseline’s. A column only the evaluated report has is not covered: it has no baseline, and gets a rule only when the policy names it. Needs evaluate_against.

Source

pub fn max_null_count_increase( self, column: impl Into<String>, nulls: usize, ) -> Self

Allow the named column’s null count to rise at most nulls above the baseline’s (#1346). Decided on the exact null_count of both sides, so one new null fails an allowance of 0 at any row count, where the percentage rule rounds it away from about 30,000 rows. Takes precedence over max_null_count_increase_any for that column. Needs evaluate_against.

Source

pub fn max_null_count_increase_any(self, nulls: usize) -> Self

max_null_count_increase for every other column both reports profiled. A column only the evaluated report has is not covered. Needs evaluate_against.

Source

pub fn max_distinct_drop(self, column: impl Into<String>, percent: f64) -> Self

Allow the named column’s distinct count to fall at most percent (0..=100) of the baseline’s (#1026). Fewer rows alone do not make a collapse, so the check fails only when the column’s distinct share of non-null values fell by more than percent of the baseline’s share as well. Decided on exact counts only: an estimate on either side leaves it NotEvaluated::ApproximateCount. Takes precedence over max_distinct_drop_any for that column. Needs evaluate_against.

Source

pub fn max_distinct_drop_any(self, percent: f64) -> Self

max_distinct_drop for every other column both reports profiled. A column only the evaluated report has is not covered. Needs evaluate_against.

Source

pub fn max_row_count_drop(self, percent: f64) -> Self

Allow the row count to fall at most percent (0..=100) of the baseline’s rows_processed (#1291), so a truncated extract that keeps its header and its shares fails. Needs evaluate_against.

A scan that stopped early read a lower bound of its source’s rows. So under PolicyScope::FullSource, an incomplete scan of this report still decides a pass and an incomplete baseline still decides a failure, and the other outcome is unevaluated: the rows a scan did not read could make up the difference. With both scans incomplete the rule is unevaluated. An allowance of 100% passes whatever the scans missed, since a count cannot fall by more. A different column set leaves it NotEvaluated::SchemaChanged, as for a dataset-level score.

Source

pub fn max_row_count_growth(self, percent: f64) -> Self

Allow the row count to grow at most percent (0 or more) of the baseline’s rows_processed (#1291), so a join that fans out or a file published twice fails. The mirror of max_row_count_drop: an incomplete scan of this report still decides a failure, and an incomplete baseline a pass. Needs evaluate_against.

Source

pub fn require_columns<I, S>(self, columns: I) -> Self
where I: IntoIterator<Item = S>, S: Into<String>,

Require each named column to be profiled. A column missing from the report fails only when the report shows its source lacks it: the run’s column selection included the column, and the column set covers the whole source (see the module docs).

Source

pub fn forbid_removed_columns(self) -> Self

Require every column the baseline profiled to be profiled still. The rule is about the columns the baseline recorded: a column its source held but it did not profile (a projection, or a key past a partial read) is not one it can require. Needs evaluate_against.

Source

pub fn forbid_added_columns(self) -> Self

Require every column the report profiled to be one the baseline profiled: a closed schema. Under PolicyScope::FullSource, a report whose profiled columns may not be all of its source’s also gets one check with no column, left unevaluated: a column selection that left columns out, one that was not recorded, or a column set covering part of the source. Needs evaluate_against.

Source

pub fn forbid_findings<I>(self, codes: I) -> Self
where I: IntoIterator<Item = FindingCode>,

Fail when the report holds a finding with one of these codes (#1066), derived with the default FindingPolicy thresholds, as ProfileReport::findings derives them. One check per code, in code order. See the module docs for when a partial scan decides one.

Source

pub fn timeliness_clock(self, rule: TimelinessClockRule) -> Self

Declare how the timeliness clocks of the baseline and the report may differ, for the rules about change that read the timeliness score: the overall score drop, when both sides assessed timeliness, and the timeliness score drop. Defaults to TimelinessClockRule::Fixed. A difference the rule does not allow, or a clock either side did not record, leaves those rules unevaluated on the setting timeliness_clock.

Source

pub fn scope(self, scope: PolicyScope) -> Self

Set what the requirements are statements about. Defaults to PolicyScope::FullSource.

Source

pub fn validate(&self) -> Result<(), PolicyError>

Check that the policy is evaluable, without needing a report.

Callers that read a policy out of configuration should do this at startup rather than discovering a bad threshold on the first dataset.

Source

pub fn evaluate( &self, report: &ProfileReport, ) -> Result<GateResult, PolicyError>

Evaluate the policy against a report.

Returns the structured outcome. The report is not modified, nothing is printed, and the process is not exited: what to do about a failing gate is the caller’s decision. A policy with a rule about change is a PolicyError::BaselineRequired here; use evaluate_against.

Source

pub fn evaluate_first_run( &self, report: &ProfileReport, ) -> Result<GateResult, PolicyError>

Evaluate the policy when the caller explicitly expects no baseline. Change rules are recorded as not applicable. The absolute rules alone decide the verdict, which passes if there are none.

Source

pub fn evaluate_against( &self, report: &ProfileReport, baseline: &ReportSide, ) -> Result<GateResult, PolicyError>

Evaluate the policy against a report and a baseline it is compared with.

Every requirement is evaluated as evaluate does, then each rule about change, in a fixed order: the overall score, the dimensions in QualityDimension::all order, the row count drop and then growth, then for the null percentage increase, the null count increase and then the distinct drop rules the named columns in name order and every other shared column in the baseline’s order. The baseline may come from a flat summary, which leaves what it did not record unrecorded. See the module docs for when a change is decided.

Trait Implementations§

Source§

impl Clone for QualityPolicy

Source§

fn clone(&self) -> Self

Returns a duplicate of the value. Read more
1.0.0 (const: unstable) · Source§

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source. Read more
Source§

impl Debug for QualityPolicy

Source§

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter. Read more
Source§

impl Default for QualityPolicy

Source§

fn default() -> Self

Returns the “default value” for a type. Read more

Auto Trait Implementations§

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
where ST: ?Sized, DT: ?Sized,

Source§

impl<T> CloneToUninit for T
where T: Clone,

Source§

unsafe fn clone_to_uninit(&self, dest: *mut u8)

🔬This is a nightly-only experimental API. (clone_to_uninit)
Performs copy-assignment from self to dest. Read more
Source§

impl<T> DynClone for T
where T: Clone,

Source§

fn __clone_box(&self, _: Private) -> *mut ()

Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T> Read<Exclusive, BecauseExclusive> for T
where T: ?Sized,

Source§

impl<T> ToOwned for T
where T: Clone,

Source§

type Owned = T

The resulting type after obtaining ownership.
Source§

fn to_owned(&self) -> T

Creates owned data from borrowed data, usually by cloning. Read more
Source§

fn clone_into(&self, target: &mut T)

Uses borrowed data to replace owned data, usually by cloning. Read more
Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.
Source§

impl<V, T> VZip<V> for T
where V: MultiLane<T>,

Source§

fn vzip(self) -> V