#[non_exhaustive]pub struct QualityPolicy { /* private fields */ }Expand description
What a report must satisfy.
Build one with the chainable setters, then evaluate it
against a report. Requirements are evaluated in a fixed order that does not
depend on the order they were added, so two callers stating the same policy
get the same document out.
Implementations§
Source§impl QualityPolicy
impl QualityPolicy
Sourcepub fn new() -> Self
pub fn new() -> Self
An empty policy. It states no requirement, so evaluating it as-is is a
PolicyError::NoRequirements.
Sourcepub fn min_rows(self, min: usize) -> Self
pub fn min_rows(self, min: usize) -> Self
Require the scan to have read at least min rows, so an empty or
truncated extract fails instead of passing every other check.
Sourcepub fn min_quality_score(self, min: f64) -> Self
pub fn min_quality_score(self, min: f64) -> Self
Require the overall quality score to be at least min (0..=100).
Sourcepub fn min_dimension_score(self, dimension: QualityDimension, min: f64) -> Self
pub fn min_dimension_score(self, dimension: QualityDimension, min: f64) -> Self
Require one dimension’s score to be at least min (0..=100).
Sourcepub fn max_null_percentage(self, column: impl Into<String>, max: f64) -> Self
pub fn max_null_percentage(self, column: impl Into<String>, max: f64) -> Self
Require the named column’s null percentage to be at most max
(0..=100). Takes precedence over
max_null_percentage_any for that
column.
Sourcepub fn max_null_percentage_any(self, max: f64) -> Self
pub fn max_null_percentage_any(self, max: f64) -> Self
Require every other profiled column’s null percentage to be at most
max (0..=100).
Sourcepub fn max_null_count(self, column: impl Into<String>, max: usize) -> Self
pub fn max_null_count(self, column: impl Into<String>, max: usize) -> Self
Require the named column to hold at most max nulls, decided on the
exact count. max_null_count(column, 0) states “no nulls” at any row
count, where a zero null percentage is decided at the two decimals it
is shown at. Takes precedence over
max_null_count_any for that column.
Sourcepub fn max_null_count_any(self, max: usize) -> Self
pub fn max_null_count_any(self, max: usize) -> Self
Require every other profiled column to hold at most max nulls.
Sourcepub fn max_duplicate_rows(self, max: usize) -> Self
pub fn max_duplicate_rows(self, max: usize) -> Self
Require the report to count at most max duplicate rows.
Sourcepub fn unique_columns<I, S>(self, columns: I) -> Self
pub fn unique_columns<I, S>(self, columns: I) -> Self
Require no non-null value of each named column to occur more than once, as a business key must not repeat (#904). Whole-row duplicates miss a reused key whenever another column differs, and always when a loader adds a row id.
The rule reads the column’s distinct count: its non-null values minus
its distinct values are the values that repeat, and the check observes
that number against an allowance of zero. Nulls are not values, so they
never repeat; max_null_count rules them out.
A repeat the scan witnessed stays one however much was not read, so it
fails under any scope, and none passes under
PolicyScope::FullSource only when the scan covered the source, as
with max_duplicate_rows. A distinct count
that is an estimate (past one million distinct values, or sooner under
memory pressure) or not recorded as exact witnesses nothing, so the
check is NotEvaluated::ApproximateCount either way. A column with
no distinct count (a nested column) is NotEvaluated::NotAssessed.
This is a requirement on the columns named, independent of the
identifier_columns profiling option, which picks the key the
uniqueness score and the key_duplicates finding read. Declaring one
does not imply the other.
Sourcepub fn max_date_age(self, column: impl Into<String>, days: u32) -> Self
pub fn max_date_age(self, column: impl Into<String>, days: u32) -> Self
Require the named column’s newest date to lie at most days days
before the timeliness reference date (#1290), so a feed that stopped
moving fails while every other rule still passes it.
The newest date is the column’s max_datetime, a calendar date, and
the reference date is the one the report’s timeliness clock recorded
(reference_date, or the run’s date on a rolling clock), so the age is
counted in calendar days and a saved report evaluated later decides
the same. A date after the reference date is fresh. A newest date only
gets newer as more values are read, so under PolicyScope::FullSource
a fresh date passes whatever the scan missed, provided the column
stays a date: on a truncated or sampled scan that holds only for a
date type the source declared (NativeType::declares_date), since
the rows not read may hold text that types an inferred column as text
(#1333). A stale one fails only when the values behind it cover the
column: every row read, and
the date statistics computed over every value read. A report that
records metric_semantics.date_statistics computed them so (#1297);
for an older one the retained sample they came from must hold every
non-null value. The exception is a column of slash dates whose
day and month order the unread values could still change
(assumed_day_first or month_first): there a fresh date read on
part of the column is NotEvaluated::EvidenceIncomplete too. A column with no newest date (not typed as a
date, all null, or statistics not computed) is
NotEvaluated::NotAssessed, and a report without a recorded clock is
NotEvaluated::SettingUnrecorded on timeliness_clock.
Sourcepub fn max_date_gap(self, column: impl Into<String>, days: u32) -> Self
pub fn max_date_gap(self, column: impl Into<String>, days: u32) -> Self
Require the largest gap between the named column’s consecutive
distinct days to be at most days (#1300), so a daily file that lost
a period from the middle of its range fails while its newest date,
schema and most of its rows stay normal. 1 allows no missing day.
The gap is the column’s largest_gap, counted in whole calendar days
over every value the profiler read. Under PolicyScope::FullSource
the rule decides only when those values cover the column: the values
that were not read can fill a hole the read ones show, or open a new
one at either edge of the range, so a partial scan is
NotEvaluated::EvidenceIncomplete whichever way the read values
point. A column with no gap recorded (not typed as a date, all null, a
report from before 0.14, or dates spanning more than
DATE_GAP_YEAR_LIMIT distinct
years) is NotEvaluated::NotAssessed.
Sourcepub fn require_quality(self) -> Self
pub fn require_quality(self) -> Self
Require the report to carry a quality assessment at all.
Without this, a report with no assessment leaves quality requirements unevaluated and the verdict inconclusive. With it, the absence is itself a violation — which is what a pipeline wants when quality is supposed to be configured on.
Sourcepub fn require_dimension(self, dimension: QualityDimension) -> Self
pub fn require_dimension(self, dimension: QualityDimension) -> Self
Require a specific dimension to have been assessed.
Sourcepub fn max_quality_score_drop(self, points: f64) -> Self
pub fn max_quality_score_drop(self, points: f64) -> Self
Allow the overall quality score to drop at most points (0..=100)
below the baseline’s. Needs evaluate_against.
Under PolicyScope::FullSource, a score computed over a retained
sample on either side is read as its recorded whole-source interval,
and the drop is decided when every change those intervals allow lies
on one side of points (#1035). Each side’s intervals hold together
with that side’s confidence_level; when both sides are sampled, the
two hold together with probability at least the sum of the two levels
minus one.
Sourcepub fn max_dimension_score_drop(
self,
dimension: QualityDimension,
points: f64,
) -> Self
pub fn max_dimension_score_drop( self, dimension: QualityDimension, points: f64, ) -> Self
Allow one dimension’s score to drop at most points (0..=100) below
the baseline’s. Needs evaluate_against.
A sampled score is decided on its interval, as in
max_quality_score_drop.
Sourcepub fn max_null_percentage_increase(
self,
column: impl Into<String>,
points: f64,
) -> Self
pub fn max_null_percentage_increase( self, column: impl Into<String>, points: f64, ) -> Self
Allow the named column’s null percentage to rise at most points
(0..=100) above the baseline’s. Takes precedence over
max_null_percentage_increase_any
for that column. Needs evaluate_against.
Sourcepub fn max_null_percentage_increase_any(self, points: f64) -> Self
pub fn max_null_percentage_increase_any(self, points: f64) -> Self
Allow every other column both reports profiled to have its null
percentage rise at most points (0..=100) above the baseline’s. A
column only the evaluated report has is not covered: it has no
baseline, and gets a rule only when the policy names it. Needs
evaluate_against.
Sourcepub fn max_distinct_drop(self, column: impl Into<String>, percent: f64) -> Self
pub fn max_distinct_drop(self, column: impl Into<String>, percent: f64) -> Self
Allow the named column’s distinct count to fall at most percent
(0..=100) of the baseline’s (#1026). Fewer rows alone do not make a
collapse, so the check fails only when the column’s distinct share of
non-null values fell by more than percent of the baseline’s share as
well. Decided on exact counts only: an estimate on either side leaves
it NotEvaluated::ApproximateCount. Takes precedence over
max_distinct_drop_any for that
column. Needs evaluate_against.
Sourcepub fn max_distinct_drop_any(self, percent: f64) -> Self
pub fn max_distinct_drop_any(self, percent: f64) -> Self
max_distinct_drop for every other column
both reports profiled. A column only the evaluated report has is not
covered. Needs evaluate_against.
Sourcepub fn max_row_count_drop(self, percent: f64) -> Self
pub fn max_row_count_drop(self, percent: f64) -> Self
Allow the row count to fall at most percent (0..=100) of the
baseline’s rows_processed (#1291), so a truncated extract that keeps
its header and its shares fails. Needs
evaluate_against.
A scan that stopped early read a lower bound of its source’s rows. So
under PolicyScope::FullSource, an incomplete scan of this report
still decides a pass and an incomplete baseline still decides a
failure, and the other outcome is unevaluated: the rows a scan did not
read could make up the difference. With both scans incomplete the rule
is unevaluated. An allowance of 100% passes whatever the scans missed,
since a count cannot fall by more. A different column set leaves it
NotEvaluated::SchemaChanged, as for a dataset-level score.
Sourcepub fn max_row_count_growth(self, percent: f64) -> Self
pub fn max_row_count_growth(self, percent: f64) -> Self
Allow the row count to grow at most percent (0 or more) of the
baseline’s rows_processed (#1291), so a join that fans out or a file
published twice fails. The mirror of
max_row_count_drop: an incomplete scan
of this report still decides a failure, and an incomplete baseline a
pass. Needs evaluate_against.
Sourcepub fn require_columns<I, S>(self, columns: I) -> Self
pub fn require_columns<I, S>(self, columns: I) -> Self
Require each named column to be profiled. A column missing from the report fails only when the report shows its source lacks it: the run’s column selection included the column, and the column set covers the whole source (see the module docs).
Sourcepub fn forbid_removed_columns(self) -> Self
pub fn forbid_removed_columns(self) -> Self
Require every column the baseline profiled to be profiled still. The
rule is about the columns the baseline recorded: a column its source
held but it did not profile (a projection, or a key past a partial
read) is not one it can require. Needs
evaluate_against.
Sourcepub fn forbid_added_columns(self) -> Self
pub fn forbid_added_columns(self) -> Self
Require every column the report profiled to be one the baseline
profiled: a closed schema. Under PolicyScope::FullSource, a report
whose profiled columns may not be all of its source’s also gets one
check with no column, left unevaluated: a column selection that left
columns out, one that was not recorded, or a column set covering part
of the source. Needs evaluate_against.
Sourcepub fn forbid_findings<I>(self, codes: I) -> Selfwhere
I: IntoIterator<Item = FindingCode>,
pub fn forbid_findings<I>(self, codes: I) -> Selfwhere
I: IntoIterator<Item = FindingCode>,
Fail when the report holds a finding with one of these codes (#1066),
derived with the default FindingPolicy thresholds, as
ProfileReport::findings derives them. One check per code, in code
order. See the module docs for when a partial scan decides
one.
Sourcepub fn timeliness_clock(self, rule: TimelinessClockRule) -> Self
pub fn timeliness_clock(self, rule: TimelinessClockRule) -> Self
Declare how the timeliness clocks of the baseline and the report may
differ, for the rules about change that read the timeliness score: the
overall score drop, when both sides assessed timeliness, and the
timeliness score drop. Defaults to TimelinessClockRule::Fixed. A
difference the rule does not allow, or a clock either side did not
record, leaves those rules unevaluated on the setting
timeliness_clock.
Sourcepub fn scope(self, scope: PolicyScope) -> Self
pub fn scope(self, scope: PolicyScope) -> Self
Set what the requirements are statements about. Defaults to
PolicyScope::FullSource.
Sourcepub fn validate(&self) -> Result<(), PolicyError>
pub fn validate(&self) -> Result<(), PolicyError>
Check that the policy is evaluable, without needing a report.
Callers that read a policy out of configuration should do this at startup rather than discovering a bad threshold on the first dataset.
Sourcepub fn evaluate(
&self,
report: &ProfileReport,
) -> Result<GateResult, PolicyError>
pub fn evaluate( &self, report: &ProfileReport, ) -> Result<GateResult, PolicyError>
Evaluate the policy against a report.
Returns the structured outcome. The report is not modified, nothing is
printed, and the process is not exited: what to do about a failing gate
is the caller’s decision. A policy with a rule about change is a
PolicyError::BaselineRequired here; use
evaluate_against.
Sourcepub fn evaluate_against(
&self,
report: &ProfileReport,
baseline: &ReportSide,
) -> Result<GateResult, PolicyError>
pub fn evaluate_against( &self, report: &ProfileReport, baseline: &ReportSide, ) -> Result<GateResult, PolicyError>
Evaluate the policy against a report and a baseline it is compared with.
Every requirement is evaluated as evaluate does,
then each rule about change, in a fixed order: the overall score, the
dimensions in QualityDimension::all order, the row count drop and
then growth, then for the null increase and then the distinct drop
rules the named columns in name order and every other shared column in
the baseline’s order.
The baseline may come from a flat summary, which leaves what it did
not record unrecorded. See the module docs for when a change
is decided.