# Python API Guide
Complete reference for the `dataprof` Python package (v0.13.1).
For upgrade-sensitive changes to sampling, execution controls, quality scores,
parser behavior, semantic hints, and exception types, read the
[release notes](../release-notes.md): 0.13.1, then the 0.13.0 migration guide.
The Python API is built for quick inspection and follow-up analysis: point it at a file, DataFrame, Arrow batch, ad-hoc notebook data, or database query and get back a report you can slice, export, and wire into notebooks or checks.
## Installation
```bash
uv pip install dataprof
# or
pip install dataprof
```
Supports standard, GIL-enabled CPython 3.11β3.15. PyPy, preview Python and
free-threaded builds are outside the supported wheel set; see the
[interpreter policy](../CONTRIBUTING.md#python-interpreters-and-release-wheels).
The package ships pre-built wheels for Linux, macOS, and Windows, and declares **no Python dependencies**. The base API needs nothing else: local file profiling, DataFrame and Arrow inputs, ad-hoc dict/bytes inputs, and report exports. Install the `pandas` extra only for pandas-typed exports (`to_dataframe()`, `describe()` as a DataFrame).
The wheel also carries the async API: `dataprof.asyncio`, HTTP URL profiling,
and remote Parquet all work on a bare `pip install dataprof`.
Database profiling is the one documented feature the wheel does not contain.
Connectors are a compile-time feature rather than a Python dependency, so no
extra can install them; they need a build from source, either with pip from the
published sdist or from a checkout:
```bash
pip install dataprof --no-binary dataprof --config-settings="build-args=--features python,python-async,async-streaming,parquet-async,database,sqlite"
```
The feature list has to be complete, not just the extra parts. See
[Database Profiling](#database-profiling-source-build-only) for why, and for
the checkout equivalent.
Inspect the current installation without importing optional packages or trying
a network/database operation:
```python
import dataprof as dp
features = dp.capabilities()
print(features)
if features.database and "sqlite" in features.database_connectors:
# Database helpers are available in this build.
...
if features.pandas_interop and features.pandas_installed:
# Both compiled interoperability and the optional Python package are present.
...
```
`features.commit` is the git hash the installed build was compiled from, and
`features.dirty` whether its Rust sources had uncommitted changes; both are
`None` where the build did not record them, as from an sdist. A development
build reports the version of the release it follows, so key anything that
depends on the exact build to the commit. Every report records the same in
`report.generator`, and `python -m dataprof.check --version` prints it.
## Quick Start
```python
import dataprof as dp
# Profile a file
report = dp.profile("data.csv")
print(f"{report.rows} rows, {report.columns} columns")
print(f"Quality score: {report.quality_score}")
# Access columns directly
col = report["age"]
print(f"mean={col.mean}, nulls={col.null_percentage}%")
# Profile a pandas DataFrame
import pandas as pd
df = pd.read_csv("data.csv")
report = dp.profile(df)
# Profile ad-hoc notebook data
report = dp.profile({"age": [31, 42, 29], "city": ["Rome", "Milan", "Rome"]})
report = dp.profile([{"age": 31, "city": "Rome"}, {"age": 42, "city": "Milan"}])
report = dp.profile(b"age,city\n31,Rome\n", format="csv")
# Profile a PyArrow table
import pyarrow.parquet as pq
table = pq.read_table("data.parquet")
report = dp.profile(table)
```
## `profile()` -- Primary Entry Point
```python
dp.profile(
source, # str, Path, DataFrame, Arrow, dict, rows, or bytes
*,
engine="auto", # "auto", "incremental", "columnar"
chunk_size=None, # int -- bytes per streaming chunk
memory_limit_mb=None, # int -- memory cap
format=None, # str -- file format override
max_rows=None, # int -- row cap; Parquet samples across the file
name=None, # str -- label for DataFrame/Arrow sources
csv_delimiter=None, # str -- override auto-detection (e.g. ";")
csv_flexible=None, # bool -- allow variable column counts
sampling=None, # SamplingStrategy
stop_condition=None, # StopCondition
on_progress=None, # Callable[[ProgressEvent], None]
progress_interval_ms=None, # int -- ms between progress events
metrics=None, # list[str] -- "schema", "statistics", "patterns", "quality"
quality_dimensions=None, # list[str] -- subset of dimensions to compute
columns=None, # list[str] -- profile only these columns
locale=None, # str -- "CA"|"DE"|"FR"|"GB"|"IT"|"US"
positive_columns=None, # list[str] -- columns expected to be non-negative
identifier_columns=None, # list[str] -- semantic IDs, not measures; the first is the key
temporal_columns=None, # list[str] -- columns assessed for timeliness
temporal_pairs=None, # list[tuple[str, str]] -- start/end pairs compared as given
) -> ProfileReport
```
For file profiling, `engine` accepts the case-insensitive names `auto`,
`incremental` (compatibility alias: `streaming`), and `columnar` (compatibility
alias: `arrow`). This applies to `dp.profile(..., engine=...)`,
`dp.profile_file(..., engine=...)`, and `dp.Profiler().engine(...).profile(...)`.
CSV reports use the selected canonical name, `incremental` or `columnar`, in
`report.engine` and `report.to_dict()["execution"]["engine"]`; aliases are never
emitted, and `auto` reports the engine it selected.
**Source types:**
| Type | Description |
|---|---|
| `str` or `Path` | File path (CSV, JSON, JSONL, Parquet) |
| pandas `DataFrame` | In-memory DataFrame |
| polars `DataFrame` | In-memory Polars DataFrame |
| PyArrow `Table` or `RecordBatch` | Zero-copy via PyCapsule interface |
| Arrow `RecordBatchReader` / `__arrow_c_stream__` producer | One-shot, incremental batch profiling; includes DuckDB relations |
| `dict[str, list]` | Columns of cells; profiled natively, no dependencies |
| `list[dict]` | Row-oriented notebook data; rows may omit keys, which read as nulls |
| `bytes` or `io.BytesIO` | In-memory file contents; requires `format="csv"`, `"json"`, `"jsonl"`, or `"parquet"` |
Dict, row-dict, and byte inputs are profiled by the Rust core directly, so they
need no third-party package. That includes `format="parquet"`: byte buffers go
through the same compiled Arrow reader as Parquet files, so the two report the
same types, column order, and statistics, and `capabilities().local_parquet`
predicts both.
A cell is missing when it is `None`, NaN, or a null-like token -- the same rule
the CSV, JSON, Parquet and Arrow paths use. The tokens are empty or whitespace,
`"null"` and `"nan"` in any case, and exactly `"NA"`, `"N/A"`, `"n/a"`, `"#N/A"`,
`"\N"` and `"None"`; `metric_semantics["null_tokens"]` records this vocabulary.
Note that a `dict` is *not* round-tripped through pandas, so an integer column
containing a null stays `integer` rather than being widened to `float`.
Dicts, row lists, pandas, polars and Arrow inputs apply `StopCondition.max_rows`
and row-only `|`/`&` compositions. They accept `StopCondition.never()` and
`SamplingStrategy.none()` as no-ops. Other stop conditions and sampling strategies
raise `ValueError` because those routes cannot apply them. `max_rows` and
`stop_condition` remain mutually exclusive.
`StopCondition.confidence_threshold(f)` means the fraction of the estimated
record count read. CSV files and materialized async CSV buffers use an exact
logical-record count from a bounded-memory scan before profiling. This preflight
also supplies progress percentages; scans without fraction stops or progress
skip it. The preflight reads the whole CSV before a profiling row or byte cap
takes effect; `bytes_consumed` describes the profiling pass, excluding this
preflight. A true async stream with a size hint estimates its total from complete
records and bytes
already read, excluding CSV header bytes; changing row widths can bias that
estimate. For a true stream without a size hint the condition is inert. Schema
stability starts after the first chunk establishes a baseline, and each leaf in an `|`/`&` tree
retains its own threshold.
**Column order** follows the source on every input and transport: the CSV
header, the Parquet/Arrow schema, the dict or DataFrame key order, and for
JSON/JSONL the field order of the first record, with fields that only appear in
later records appended where they were first seen. Converting a dataset between
formats therefore does not reshuffle the report.
**Column projection** narrows a profile to the columns you name. `columns`
takes a list of source column names and is accepted by `dp.profile()`,
`dp.profile_file()`, `dp.Profiler().columns([...])`, and every
`dataprof.asyncio` entry point β `profile_file()`, `profile_bytes()` and
`profile_url()` forward it through `ProfilerConfig`, which also carries it into
`analyze_database_async()`:
```python
report = dp.profile("orders.csv", columns=["amount_eur", "city"])
list(report) # ['city', 'amount_eur']
report.rows # every row is still read
```
Projection selects columns, never rows: `report.rows` is unchanged, and each
retained column reports exactly the values it reports in an unprojected profile
of the same source β same types, statistics, patterns, and lengths. It applies
identically on every input and transport (CSV, JSON, JSONL, Parquet, bytes,
dicts, row dicts, DataFrames, and Arrow) and on both engines.
The projected report keeps **source order**, not the order you listed the names
in, so a projection is a filter on the report rather than a reordering of it.
Names are validated rather than silently ignored. A name absent from the source
raises `ValueError` listing the unmatched names, and so does a repeated name.
`columns=[]` is a projection onto no columns, which is distinct from
`columns=None` (no projection): it reports `columns == 0` while `rows` still
counts the rows that were read.
Quality is assessed over the projected columns only, so `quality_score`
generally differs from the score for the whole dataset. Two dimensions are
**withheld** rather than narrowed: `completeness` and `uniqueness` both report
row-level measurements (`complete_records_ratio`, `total_cells`,
`duplicate_rows`, `rows_checked`) that mean something different once a row has
been projected, and the report schema cannot mark only those fields as
projected. Rather than publish plausible numbers under full-row names, a
projected report leaves both dimensions `None` and out of
`assessed_dimensions()`:
```python
dp.profile("orders.csv").quality.completeness
# {'complete_records_ratio': 73.8, 'total_cells': 5000, ...}
dp.profile("orders.csv", columns=["amount_eur", "city"]).quality.completeness
# None
```
The overall score is therefore an average over fewer dimensions, and is often
*higher* than the unprojected score even for the same columns β dropping
`completeness` drops the dimension that penalizes nulls. Compare projected
scores against other projected scores, not against whole-dataset ones.
If a projection leaves no assessable dimension β `quality_dimensions` selecting
only `completeness` and/or `uniqueness` β the report carries no quality object
at all: `report.quality` is `None` and so is `quality_score`, which is the
"not analyzed" answer rather than a zero.
Semantic hints are resolved against the projection too β a `positive_columns`
hint naming a column that projection excluded raises, since that column is not
in the report it would describe.
Synchronous byte inputs use the in-memory columnar path. They support
`max_rows`, metric/quality selection, semantic hints, CSV delimiters, and JSONL
error policy, but reject streaming-only controls (`chunk_size`,
`memory_limit_mb`, `stop_condition`, progress callbacks, and flexible CSV
recovery) instead of silently ignoring them. For those controls, use
`dataprof.asyncio.profile_bytes()`. JSON and JSONL byte buffers follow RFC 8259:
the non-standard `NaN` and `Infinity` constants are malformed input, and so are
an integer outside the 64-bit range, a number that overflows a float, such as
`1e400`, and a string escape for a lone surrogate, such as `"\ud800"`, as they
are in a file.
**JSON record policy.** Only JSON objects are profileable records β they are the
only JSON value with named fields to become columns. A record that is valid JSON
but not an object (a scalar, an array, `null`) is never silently dropped: with
`jsonl_on_error="skip"` (the default) it is counted in `error_count` and the
records after it still profile, and with `jsonl_on_error="strict"` the first one
raises `ValueError` naming its position and JSON kind. Input whose every record
is non-object fails, exactly as all-malformed input does. File, byte, and async
byte transports apply this identically.
**Records with no fields.** A JSON object with no fields (`{}`) is a record: it
was read and analysed, and nothing was found in it. It profiles as a row against
zero columns, so `[{}, {}]` reports `rows == 2, columns == 0` on every transport
β file, bytes, async bytes, URL, and a Python list of dicts. That keeps three
shapes distinct:
| Shape | Meaning | Example |
|---|---|---|
| `rows > 0`, `columns == 0` | rows were read; none of them had fields | `[{}, {}]` |
| `rows == 0`, `columns == 0` | the input held no records | `[]` |
| `rows == 0`, `columns > 0` | a known schema with no rows under it | a CSV header line |
A zero-column report is a normal report: it serialises, round-trips, and
renders. `len(report)` counts columns, so it is `0` while `report.rows` is not.
A record with no fields is well-formed, so it is clean under both error
policies and never counted in `error_count`.
**Engine options:**
| Engine | When to use |
|---|---|
| `"auto"` | Let dataprof choose based on file size and format (recommended) |
| `"incremental"` | True streaming with bounded memory -- large files, streams |
| `"columnar"` | Arrow-based batch processing -- Parquet, in-memory data |
## `ProfileReport`
Returned by `profile()` and all analysis functions.
**Properties:**
| Property | Type | Description |
|---|---|---|
| `id` | `str \| None` | The report's identifier, as `compare()` and `check()` name it; `None` for a report loaded from a flat `to_dict()` summary, which does not record one |
| `timestamp` | `str \| None` | The report's creation timestamp, as `compare()` and `check()` record it; `None` for a report loaded from a flat `to_dict()` summary, which does not record one |
| `source` | `str` | Source identifier (file path, table name, etc.) |
| `source_type` | `str` | `"file"`, `"bytes"`, `"query"`, `"dataframe"`, `"stream"` |
| `engine` | `str \| None` | Engine or parser that produced the report |
| `rows` | `int` | Number of rows processed |
| `columns` | `int` | Number of columns detected |
| `column_profiles` | `dict[str, ColumnProfile]` | Per-column statistics (by name) |
| `quality_score` | `float \| None` | Overall quality score (0--100) |
| `quality` | `DataQualityMetrics \| None` | Detailed quality breakdown |
| `quality_status` | `str` | Why `quality` is or is not there (see below) |
| `quality_error` | `str \| None` | The error a failed quality computation reported |
| `quality_sampled_dimensions` | `list[str] \| None` | Metric components computed from a retained sample rather than every scanned row; `None` when there is no assessment or a loaded document does not record it |
| `quality_score_bounds` | `dict \| None` | Where the scores over every scanned row lie when some were computed over a retained sample: `confidence_level`, `overall_score` and `dimension_scores`, each interval `{"lower", "upper"}` or `None` when unbounded. `None` when every score is exact (see [quality gates](#check----quality-gates)) |
| `analysis_settings` | `dict[str, Any] \| None` | The settings it was profiled with that change what a score measures and are recorded nowhere else: `{"pattern_locale": "IT", "columns": {"selection": "all"}}`, where `"none"` means no locale and a `columns=` projection reads `{"selection": "listed", "names": [...]}`; `None` for a report written before 0.13, whose settings are unknown |
| `metric_semantics` | `dict[str, str] \| None` | How the measurements were defined, e.g. `{"text_length_unit": "unicode_scalar", "null_tokens": "common_markers", "pattern_match_count": "validated", "shape_moments": "adjusted_sample", "range_rule_count": "violating_values", "key_uniqueness_units": "non_null_values", "outlier_units": "tested_values", "key_column_source": "declared"}`; `None` for a report written before 0.12, whose definitions are unknown. A 0.12 report records only `text_length_unit`: it counted only empty, `null` and `nan` as missing, counted every regex match as a pattern match, and used another skewness and kurtosis estimator |
| `generator` | `dict[str, Any] \| None` | The dataprof build that produced it: `{"version": "0.13.0", "commit": "<40-hex>", "dirty": False}`. A development build carries the version of the release it follows; `commit` and `dirty` (uncommitted changes in the compiled Rust sources) tell them apart, and are absent where the build did not record them, as from an sdist. `None` for a report written before 0.13 |
| `execution_time_ms` | `int` | Total processing time |
| `throughput` | `float \| None` | Rows per second |
| `memory_peak_mb` | `float \| None` | Peak memory usage |
| `truncation_reason` | `str \| None` | Why processing stopped early |
| `source_exhausted` | `bool` | Whether the entire source was read |
| `ragged_row_count` | `int` | Rows whose field count differed from the header (`0` = no ragged rows). Reported for file, async and columnar CSV inputs |
| `unterminated_quote` | `bool \| None` | `True` when a CSV source ended inside a quoted field: a quote was never closed, so the last record holds every row after it. `None` for non-CSV input and for a scan stopped before the end of the source. `csv_flexible=False` refuses such a source, and CSV bytes passed to `profile()` always do |
| `recovery_events` | `list[dict[str, str]] \| None` | Ordered failed attempts and retries; `[]` means no recovery, `None` means history was not recorded |
| `sampling_applied` | `bool` | Whether sampling was used |
| `sampling_ratio` | `float \| None` | Fraction of data sampled |
| `sampled_row_ranges` | `list[list[int]] \| None` | Exact zero-based, half-open source row intervals when recorded; `[]` means zero selected rows |
For Parquet files, byte buffers and HTTP URLs, `max_rows=N` selects up to 32
contiguous ranges spread across the file, with exactly N rows when the file
is larger than the cap. The first and last rows are included for N β₯ 2;
N = 1 selects the middle row and N = 0 selects none. Selection is deterministic
and independent of row-group sizes, batch sizes and transport. The exact ranges
appear in `sampled_row_ranges` and `to_dict()["execution"]["sampled_row_ranges"]`.
`sampling_applied` is true, `sampling_ratio` is selected rows / file rows, and
`source_exhausted` is false with the usual `max_rows(N)` truncation reason.
A cap at or above the file row count reads the whole file without sampling.
Statistics and row-level quality metrics describe the selected population.
Duplicate detection includes matches across separate ranges. This deterministic
sample does not establish full-source uniqueness or provide an unbiased
estimate of full-source ratios. Before 0.12, local and byte-buffer Parquet caps
selected a prefix, while HTTP reads ignored the cap. Older reports have no
recorded ranges and load with `sampled_row_ranges=None`. Re-baseline capped
Parquet comparisons when upgrading.
**What happened to the quality computation:**
`quality is None` answers two different questions, so every report carries
`quality_status`. It is `computed` exactly when `quality` holds an assessment;
the other five states are the reasons it does not:
| `quality_status` | meaning |
|---|---|
| `computed` | quality was computed; `quality` holds the assessment |
| `not_requested` | the quality pack was deselected (`metrics=["schema"]`) |
| `no_data` | requested, but no quality sample was supplied to the assembler |
| `withheld_by_projection` | requested, but every requested dimension measures whole rows and `columns=` selected a subset |
| `failed` | requested and attempted; the computation failed, and `quality_error` says how |
| `unrecorded` | loaded from a report saved before 0.12 that carried no assessment |
An analyzed empty source reports `computed` with an empty assessment and
`quality_score=None`, whether or not it declares columns. This is distinct from
an unavailable quality sample (`no_data`).
A gate deciding on a report should branch on this before reading
`quality_score`: a failed computation and a run that never asked for quality
both leave the score `None`, and they are not the same verdict.
```python
report = dataprof.profile("data.csv")
if report.quality_status != "computed":
raise SystemExit(f"no quality to gate on: {report.quality_status}")
```
The same object is in `to_dict()["quality_status"]`, as `{"state": ...}` plus
an `"error"` key on `failed` only.
**Dict-like column access:**
```python
col = report["column_name"] # -> ColumnProfile
"column_name" in report # -> True/False
for name in report: print(name) # iterate column names
len(report) # number of columns
```
**Export methods:**
```python
report.to_dict() # historical flat summary (rounded values)
report.to_json(indent=2) # complete canonical Rust report as JSON
report.to_dataframe() # pandas DataFrame -- all stats (requires pandas)
report.to_polars() # polars DataFrame -- all stats (requires polars)
report.to_arrow() # PyArrow Table -- all stats (requires pyarrow)
report.describe() # transposed summary like pandas describe()
report.quality_summary() # single-row dict for quality tracking
report.to_html() # embeddable HTML (same as the notebook display)
report.to_markdown() # GitHub-flavored markdown table
report.compare(other) # dict of quality/schema/null deltas vs another report
report.save("report.json") # save to JSON
report.save("report.csv") # save column profiles to CSV
report.save("report.parquet") # save column profiles to Parquet (requires pyarrow)
# Round-trip a saved report without re-profiling (read-only view)
reloaded = dp.ProfileReport.load("report.json") # from a saved .json file
reloaded = dp.ProfileReport.from_json(report.to_json()) # from a JSON string
reloaded = dp.ProfileReport.from_dict(report.to_dict()) # from a dict
```
### Rounding
All floating-point values in exported data are rounded. The precision is chosen
by what the number *is*, not by which object it lives on:
| Kind | Precision | Examples |
| --- | --- | --- |
| `0..100` percentage | 2dp | `null_percentage`, `coefficient_of_variation`, every float in the quality dimension dicts |
| statistic | 4dp | `mean`, `std_dev`, `variance`, `skewness`, `kurtosis`, `avg_length` |
| data value | 4dp | `min`, `max`, `median`, `mode` |
| `0..1` ratio | 4dp | `uniqueness_ratio`, `true_ratio` |
A `0..1` ratio takes 4dp so it carries the same resolution as the equivalent
percentage at 2dp. Data values take 4dp because a profiler must not report a
`min` the column never contained. Quartiles are the deliberate exception: they
stay at 2dp, being distribution landmarks rather than exact values.
**Ties** round the stored float, away from zero. This is the value actually
held, not the shortest decimal string that prints for it -- `23 / 4000 * 100`
prints as `0.575` but is stored just below it, and so rounds to `0.57`.
The Rust and Python layers implement the same convention and are held to shared
fixtures (`tests/fixtures/rounding_parity.json` and
`report_rounding_parity.json`), so the same data profiled through either gives
the same numbers. Raw property access on `ColumnProfile` returns unrounded Rust
values; use the export methods for rounded output. Cross-engine numeric
equality is defined on those serialized metrics, with no extra tolerance.
Native attributes may differ in their final digits with accumulation order;
loading a report retains the precision that was saved. Compare the same data,
schema semantics, analysis options and analyzed population, and exclude source
and execution provenance such as timing and memory from equality checks. See
the [numeric contract](../schema/README.md#numeric-equality-contract) for scope,
sampling and absence semantics.
**Round-trip fidelity:** a report reloaded with `from_dict`, `from_json`, or
`load` reports the same values as the report it was saved from, at the precision
above. Live and reloaded reports use the same read-only column, pattern, and
quality accessors. They read native values or the saved document respectively;
reloading does not recompute metrics or invent values for missing fields.
Both expose `dataprof.ColumnProfile` and `dataprof.DataQualityMetrics` objects,
so type checks and accessor behavior no longer depend on how a report was loaded.
## `ColumnProfile`
Per-column profiling statistics.
| Field | Type | Description |
|---|---|---|
| `name` | `str` | Column name |
| `data_type` | `str` | Inferred type: `"string"`, `"identifier"`, `"integer"`, `"float"`, `"date"`, `"boolean"`, `"nested"`. A `nested` column (struct, list or map) reports its counts only: `unique_count`, `type_homogeneity`, `locale_number_count`, `stats` and `patterns` are `None` |
| `total_count` | `int` | Total number of values |
| `null_count` | `int` | Number of null/missing values |
| `unique_count` | `int \| None` | Distinct value count |
| `invalid_count` | `int \| None` | Non-null values that did not parse as a finite number (parse failures and non-finite tokens like `inf`/`NaN`) and are excluded from the statistics. `None` = check did not run (non-numeric column, or statistics skipped); `0` = every non-null value parsed |
| `type_homogeneity` | `dict[str, int] \| None` | Non-null values counted by lexical class: `{"numeric", "date", "boolean", "text"}`. Tells a `string` column of ordinary text from one that defeated type inference, which `data_type` cannot. All four keys are always present; all-zero = classified with nothing to classify (all-null or no rows); `None` = the classification did not run. Counted over the values the profiler retained, so sum them against `total_count - null_count` to tell an exact count from one bounded by the 10k reservoir sample |
| `locale_number_count` | `int \| None` | Of the `text` values in `type_homogeneity`, how many are numbers written with a decimal comma or digit-group separators (`10,50`, `1.234,56`, `1,234.56`, `1'234.56`, `1 234,56`). dataprof does not parse them as numbers, so they are in no numeric statistic: a column of them is typed `string`, and a numeric column counts them in `invalid_count`. `None` = the count did not run (a `nested` column, or a report saved before 0.12); `0` = none found |
| `null_percentage` | `float` | Null ratio (0.0--100.0) |
| `uniqueness_ratio` | `float` | Unique values / total values |
| `min` | `float \| None` | Minimum (numeric columns) |
| `max` | `float \| None` | Maximum (numeric columns) |
| `mean` | `float \| None` | Mean (numeric columns) |
| `std_dev` | `float \| None` | Standard deviation |
| `variance` | `float \| None` | Variance |
| `median` | `float \| None` | Median |
| `mode` | `float \| None` | Mode |
| `skewness` | `float \| None` | Skewness |
| `kurtosis` | `float \| None` | Kurtosis |
| `coefficient_of_variation` | `float \| None` | CV |
| `quartiles` | `dict \| None` | `{"q1", "q2", "q3", "iqr"}`; `iqr` is `None` when `q3 - q1` overflows `f64` |
| `is_approximate` | `bool \| None` | Whether stats were estimated from a sample |
| `min_length` | `int \| None` | Shortest value, in Unicode scalar values (0.11: UTF-8 bytes) |
| `max_length` | `int \| None` | Longest value, in Unicode scalar values (0.11: UTF-8 bytes) |
| `avg_length` | `float \| None` | Mean length, in Unicode scalar values (0.11: UTF-8 bytes) |
| `true_count` | `int \| None` | Number of `True` values (boolean columns) |
| `false_count` | `int \| None` | Number of `False` values (boolean columns) |
| `true_ratio` | `float \| None` | Ratio of `True` values (0.0--1.0) |
| `patterns` | `list[Pattern] \| None` | List of detected value patterns |
**Changed in 0.12 (#667).** Numeric columns with no finite parsed values and
boolean columns with no parsed booleans have no statistics block. Their statistic
accessors return `None`, including boolean counts and `true_ratio`. The declared
column type, row/null counts, and numeric `invalid_count` remain available. A
measured numeric zero or an all-false column still reports `0.0`. CSV columns
containing only nulls infer as text because CSV carries no declared type. A
column with no non-null value has no statistics block whatever its type, so an
all-null text column's `min_length`, `max_length` and `avg_length` are `None`,
not `0`.
This uses the existing absent-statistics representation (`ColumnStats::None` in
Rust); numeric fields do not become optional within `NumericStats`, and the report
schema version is unchanged. Saved reports remain readable as written: loading an
older report does not replace its recorded zeros. Re-profile the source to obtain
the corrected absence semantics.
**Changed in 0.12.** Text lengths count Unicode scalar values, not UTF-8 bytes
and not grapheme clusters. `"ζ±δΊ¬"` has a length of 2 and `"π"` a length of 1.
**In 0.11 and earlier the same values reported 6 and 4**, because every
accumulator used Rust's byte-valued `str::len()` under a field named `length`
(#627). ASCII text is unaffected, and reports written by different releases are
not comparable for non-ASCII text.
A combining sequence counts each scalar, so the decomposed spelling of e-acute
(`e` followed by U+0301) has a length of 2 while the precomposed spelling
(U+00E9) has a length of 1. The two render identically and profile differently.
Segmenting grapheme clusters instead would need a Unicode segmentation table and
a policy for which version of it; encoded size is a property of the encoding
rather than of the value, and is reported at the source level instead.
## `Pattern`
Represents a pattern found in a column. Matches pass both the regular expression
and the semantic validator, when the detector has one. The counts and percentages
describe the analyzed non-null values, which may be a sample of the column.
| Field | Type | Description |
| --- | --- | --- |
| `name` | `str` | Name of the pattern (e.g. `"Email"`, `"URL"`) |
| `regex` | `str` | Regular expression used for matching |
| `match_count` | `int` | Number of analyzed non-null values passing validation |
| `match_percentage` | `float` | Percentage of analyzed non-null values passing validation (0.0--100.0) |
## Configuration
### `ProfilerConfig`
Reusable configuration object:
```python
config = dp.ProfilerConfig(
engine="incremental",
chunk_size=65536, # bytes per chunk
memory_limit_mb=512,
max_rows=100000,
csv_delimiter=";",
quality_dimensions=["completeness", "uniqueness"],
positive_columns=["pressure"],
identifier_columns=["order_id", "customer_id"],
)
```
### `SamplingStrategy`
Controls how data is sampled during profiling:
```python
from dataprof import SamplingStrategy
SamplingStrategy.none() # process everything
SamplingStrategy.random(size=10000) # uniform sample of 10000 rows
SamplingStrategy.reservoir(size=10000) # same guarantee, Algorithm R
SamplingStrategy.systematic(interval=10) # every Nth row
```
Sampling applies to CSV sources on the `auto` and `incremental` engines and to
every `dataprof.asyncio` entry point. The columnar engine and the JSON/Parquet
readers cannot sample row by row and raise `ValueError` rather than silently
returning a full profile.
`random` and `reservoir` give the same guarantee β a uniform sample of exactly
`size` rows β and both hold `size` rows in memory, because which rows belong in
the sample is not settled until the source ends. The other strategies decide
each row as it arrives and add no memory. Sampling bounds the cost of
*analysis*, not of reading: use a `StopCondition` to stop early.
### `StopCondition`
Composable early-termination conditions. Combine with `|` (any) or `&` (all):
```python
from dataprof import StopCondition
# Stop after 10k rows or 50 MB
stop = StopCondition.max_rows(10000) | StopCondition.max_bytes(50_000_000)
# Stop when schema stabilizes AND confidence exceeds 95%
stop = StopCondition.schema_stable(500) & StopCondition.confidence_threshold(0.95)
# Built-in presets
StopCondition.schema_inference() # fast schema-only mode
StopCondition.quality_sample() # enough rows for quality assessment
StopCondition.never() # process everything (default)
report = dp.profile("huge.csv", stop_condition=stop)
```
### `ProgressEvent`
Track progress with a callback:
```python
def on_progress(event):
if event.percentage is not None:
print(f"{event.percentage:.1f}% ({event.rows_processed} rows)")
report = dp.profile("data.csv", on_progress=on_progress)
```
Every file route reports at least a `started` and a `finished` event. A CSV on
the default or incremental engine also reports `schema_detected` and
`chunk_processed` events while it reads; the columnar engine, JSON and Parquet
read without reporting, so they send the two bracketing events only.
Event fields: `kind`, `rows_processed`, `bytes_consumed`, `elapsed_ms`, `processing_speed`, `percentage`, `column_names`, `total_rows`, `total_bytes`, `truncated`, `message`, `estimated_total_rows`, `estimated_total_bytes`.
## `DataQualityMetrics`
Quality metrics informed by ISO 8000 and ISO/IEC 25012, accessible via
`report.quality`. The aggregate score is dataprof's formula, not an ISO-defined
or certified score:
```python
q = report.quality
# Overall score. None when no dimension was assessable β a header-only file, or
# one whose every dimension had a zero denominator. `assessed_dimensions()` is
# empty for exactly those reports, and `report.quality_score` agrees.
print(q.overall_quality_score())
print(q.assessed_dimensions())
print(q.score_weights) # relative weights used by dataprof's aggregate formula
# Nested dimension evidence (None when the dimension was not assessed)
print(q.completeness) # {"missing_values_ratio": ..., "complete_records_ratio": ..., "null_columns": [...]}
print(q.consistency) # {"data_type_consistency": ..., "format_violations": ..., "encoding_issues": ...}
print(q.uniqueness) # {"duplicate_rows": ..., "key_uniqueness": ..., "high_cardinality_warning": ...}
print(q.accuracy) # {"outlier_ratio": ..., "range_violations": ..., "rule_violating_values": ..., "rule_values_checked": ...}
print(q.timeliness) # {"future_dates_count": ..., "stale_data_ratio": ..., "temporal_violations": ...}
print(q.validity) # {"valid_values_ratio": ..., "invalid_values": ..., "values_checked": ...}
print(q.precision) # {"decimal_places_consistency": ..., "inconsistent_precision_values": ..., "numeric_values_checked": ...}
```
A dimension's evidence is present only when it assessed something. `None` covers
both "not computed" and "computed with nothing to divide by": no cells, no
non-null values, no numeric values, no dates, no confidently detected pattern.
The dimensions holding evidence are exactly the ones whose `dimension_scores()`
entry is a number.
`assessed_dimensions()` is that set narrowed once more, to the dimensions
carrying weight in the overall score. The two coincide under the default
weights; a dimension given a weight of `0.0` still holds evidence and a score
while sitting outside `assessed_dimensions()`, because it contributes nothing to
the aggregate. Branch on `dimension_scores()` to ask "was this measured", and on
`assessed_dimensions()` to ask "is this behind the overall score".
That distinction is the point rather than a technicality. `validity` reporting
`valid_values_ratio: 100.0` from `values_checked: 0` reads as a clean bill of
health for something nobody looked at, and any file without a pattern-bearing
column used to report exactly that. Serialized reports follow the same rule: an
unassessed dimension has no key, so a stored report cannot be read as a perfect
score either.
Flat `DataQualityMetrics` accessors, deprecated in 0.9, are removed in 0.12.
The old names raise `AttributeError` naming the replacement key. Use nested
dimensions so skipped dimensions are explicit (see the
[migration table](https://github.com/AndreaBozzo/dataprof/blob/v0.12.0/docs/release-notes.md#removed-flat-quality-accessors)):
```python
if q.completeness is not None:
q.completeness["missing_values_ratio"]
```
`negative_values_in_positive` is driven by explicit `positive_columns`; dataprof
does not infer positive-only domains from column names. `identifier_columns`
marks numeric-looking IDs as semantic strings so numeric stats and outlier
metrics do not treat them as measures. Its first entry the source has is also
the key `key_uniqueness` measures; with none declared there is no key
component, since a name cannot tell a table's key from a foreign key. List
the table key first: `identifier_columns=["order_id", "customer_id"]`.
Timeliness scoring assesses confidently inferred date columns by default.
`temporal_columns` adds columns that inference cannot identify confidently, such
as mixed-format date strings. `temporal_pairs=[("valid_from", "valid_until")]`
declares start/end pairs that timeliness compares row by row, for names no role
word pairs; the remaining temporal columns are still paired by name.
Hints are validated, never silently dropped. A hint that names a missing column
raises `ValueError` listing the unmatched names and the available columns; a
`positive_columns` hint on a column with no numeric values, or a
`temporal_columns` hint on a column with no dates, is likewise rejected. A
`temporal_pairs` entry is checked the same way: a missing column, or a column
with no dates, raises `ValueError` naming `temporal_pairs`, as does a pair that
names one column twice or that is also declared in the reverse order.
`report.semantic_hint_bindings` records how each hint bound β `column`, `kind`,
`checked_values`, `matched_values`, and `exact` (whether the counts covered
every row or a sample). Each column of a declared pair is recorded once, with
kind `temporal_pair`:
```python
report = dp.profile("readings.csv", positive_columns=["pressure"])
report.semantic_hint_bindings
# [{"column": "pressure", "kind": "positive",
# "checked_values": 1000, "matched_values": 1000, "exact": True}]
```
**Selective dimensions** -- compute only what you need:
```python
report = dp.profile("data.csv", quality_dimensions=["completeness", "uniqueness"])
# report.quality.consistency will be None
# report.quality.completeness will have values
```
## Export Methods
### `to_dataframe()` / `to_polars()` / `to_arrow()`
All three return an enriched table of column profiles with rounded values:
```python
# pandas DataFrame
df = report.to_dataframe()
# polars DataFrame (no pandas dependency needed)
pl_df = report.to_polars()
# PyArrow Table (no pandas dependency needed)
table = report.to_arrow()
```
Columns included: `name`, `data_type`, `total_count`, `null_count`, `null_percentage`,
`unique_count`, `uniqueness_ratio`, `locale_number_count`, `dominant_type`,
`dominant_type_share`,
`min`, `max`, `mean`, `std_dev`, `variance`,
`median`, `mode`, `skewness`, `kurtosis`, `coefficient_of_variation`, `q1`, `q2`,
`q3`, `iqr`, `is_approximate`, `min_length`, `max_length`, `avg_length`,
`top_pattern`, `top_pattern_pct`.
### `describe()`
Transposed summary similar to `pandas.DataFrame.describe()`:
```python
desc = report.describe()
# col_a col_b col_c
# count 1000 1000 1000
# null% 0.0 2.1 0.0
# unique 45 800 3
# mean 34.5 None None
# std 12.1 None None
# min 1.0 None None
# 25% 25.0 None None
# 50% 33.0 None None
# 75% 44.0 None None
# max 99.0 None None
```
Returns a pandas DataFrame if available, otherwise a dict-of-dicts.
### `quality_summary()`
Single-row dict for easy aggregation across multiple reports:
```python
qs = report.quality_summary()
# {"source": "data.csv", "rows": 1000, "quality_score": 92.3,
# "completeness": 98.0, "consistency": 95.0, ...}
# Track quality over time
import pandas as pd
rows = [dp.profile(f).quality_summary() for f in files]
history = pd.DataFrame(rows)
```
### `to_html()` / `to_markdown()`
Render the report for sharing outside a notebook:
```python
html = report.to_html() # same rich table Jupyter shows, as a string
open("report.html", "w").write(html)
md = report.to_markdown() # GitHub-flavored markdown table
# Paste straight into a PR comment, issue, or Slack message
```
### `load()` / `from_json()` / `from_dict()`
Rebuild a report from previously exported data without re-profiling. The
reconstructed report is a **read-only view** backed by the exported values, but
all export methods (`to_json`, `to_markdown`, `to_dataframe`, `describe`,
`quality_summary`, mapping access, β¦) work as usual.
`load(path)` is the path-based entry point β the natural counterpart to
`save()`. Only `.json` files carry a full report; `.csv` / `.parquet` store
column profiles only and cannot round-trip:
```python
report.save("report.json")
# ...later...
reloaded = dp.ProfileReport.load("report.json")
reloaded.quality_score # == the original report's quality_score
reloaded["email"].null_percentage
```
`from_json(text)` and `from_dict(data)` take an in-memory JSON string or dict
instead of a file path:
```python
reloaded = dp.ProfileReport.from_json(report.to_json())
reloaded = dp.ProfileReport.from_dict(report.to_dict())
```
#### Report schema versioning
Saved reports are durable artifacts β CI baselines, drift references, agent
inputs β so the document carries its own schema version, independent of the
package version:
```python
report.to_dict()["schema_version"] # == dp.REPORT_SCHEMA_VERSION
```
The compatibility policy when loading:
| Document | Behavior |
|---|---|
| No `schema_version` field | Legacy pre-0.10 report; loads through a compatibility path |
| `schema_version` β€ `dp.REPORT_SCHEMA_VERSION` | Loads normally; unknown additive fields from newer writers are ignored |
| `schema_version` > `dp.REPORT_SCHEMA_VERSION` | Raises `ValueError` immediately β an incompatible report is never partially decoded |
The version only increments when the document format itself changes
incompatibly, not on every dataprof release. The same field with the same
semantics appears in reports serialized from Rust (`serde`), where readers
enforce the identical policy.
The committed [JSON Schema 2020-12 contract](../schema/profile-report.v1.schema.json)
can validate `to_dict()`, `to_json()`, and JSON `save()` output in CI or another
consumer. Version 1 includes both the high-level Python export shape and the
complete Rust serialization shape; see the
[schema notes](../schema/README.md) for compatibility and regeneration rules.
In 0.12, `to_json()` and JSON `save()` write the complete Rust document, with
`data_source`, `column_profiles`, `id`, `timestamp`, and `quality.metrics` /
`quality.confidence`. `to_dict()` keeps the historical summary keys `source`,
`source_type`, `columns`, and flattened quality scores. It is a convenience
projection, so `json.loads(report.to_json())` is the way to obtain the complete
document as a dictionary. All three loaders accept both layouts. An old flat
document stays flat on resave because its missing provenance cannot be inferred.
When quality metrics are present, the summary's `quality` block always carries a
`low_sample_warning` boolean (`true` when the profiled sample was below the
recommended minimum of 10 rows, `false` otherwise). It round-trips through
`to_dict()`/`from_dict()`; treat `quality_score` and the per-dimension ratios
as directional rather than reliable whenever it is `true`.
### `findings()` -- what deserves attention
A report holds every metric and says nothing about which ones matter.
`findings()` turns the metrics already computed into a short, deterministic
list, so the call site does not have to invent thresholds:
```python
report = dp.profile("orders.csv")
result = report.findings()
for finding in result:
print(finding.severity, finding.code, finding.column, finding.evidence)
# warning null_heavy amount {'null_percentage': 25.0, 'threshold': 20.0}
# info constant_column channel {'non_null_count': 4, 'unique_count': 1}
# info sensitive_pattern email {'category': 'contact', 'match_percentage': 100.0, 'pattern': 'Email'}
```
Each finding has a stable `code`, a `severity` (`"warning"` or `"info"`), the
`column` it concerns (`None` for the whole report), the `evidence` that caused
it, and a fixed `summary` sentence. Evidence is metric values, thresholds, and
names; no finding carries a raw cell value.
| Code | Severity | Reported when |
|---|---|---|
| `all_null` | warning | Every value of a column is null |
| `null_heavy` | warning | A column's null percentage is at least `null_heavy_percentage` (default 20) |
| `mixed_types` | warning | Values outside a column's dominant lexical type are at least `mixed_types_percentage` (default 5) of those classified. `dominant_type` in the evidence says which way the mix leans. Identifier columns are exempt |
| `locale_numbers` | warning | At least half of a column's text values are numbers written with a decimal comma or digit grouping (`locale_number_count`), which no numeric statistic includes. Identifier columns are exempt |
| `duplicate_rows` | warning | The source holds exact duplicate rows |
| `future_dates` | warning | Date values lie after the time the report was produced |
| `temporal_order_violations` | warning | Start dates fall after their paired end dates |
| `ragged_rows` | warning | Rows had a different field count from the header and were recovered |
| `unterminated_quote` | warning | A CSV source ended inside a quoted field, so its last record may hold several source rows |
| `records_skipped` | warning | Errors were counted while reading, e.g. JSONL lines skipped under `jsonl_on_error="skip"` |
| `constant_column` | info | Every non-null value of a column is the same one, seen more than once |
| `sensitive_pattern` | info | A column confidently matches a contact or financial pattern, a US SSN, or an Italian codice fiscale |
| `partial_scan` | info | The scan stopped early or sampled rows (`reason` is `"truncated"` or `"sampled"`) |
Findings sort by severity, then code, then report-level before column-level,
then column position. The default thresholds are the ones `to_llm_context()`
flags at. Findings compare at the report's 2dp precision and the flags do not,
so a share within rounding of a threshold (4.9992% against 5) can be a
finding without being a flag. Both thresholds take a percentage above 0 and at
most 100, and are applied at 2dp, so the evidence states exactly the threshold
that was compared. Anything else, including a value that rounds to 0, raises
`ValueError`:
```python
report.findings(null_heavy_percentage=50, mixed_types_percentage=10)
```
**Absence is not a clean result.** A rule whose input the report does not
carry produces no finding and is listed in `result.not_evaluated`, so an empty
`result.findings` means "looked, found nothing" only for the rules not listed
there. The result has no `len()`, and `bool(result)` raises `TypeError`, so
`if not report.findings():` cannot silently read an unevaluated rule as clean.
```python
dp.profile("orders.csv", metrics=["schema"]).findings().not_evaluated
# ({'code': 'duplicate_rows', 'reason': 'quality_unavailable', 'quality_status': 'not_requested'},
# ...
# {'code': 'sensitive_pattern', 'reason': 'not_computed', 'columns': ['order_id', 'email', ...]})
```
| `reason` | Meaning |
|---|---|
| `quality_unavailable` | The report carries no quality assessment; `quality_status` says why |
| `not_assessed` | Quality was computed, but the dimension the rule reads had nothing to assess |
| `estimated` | The duplicate count is an estimate, which witnesses nothing |
| `sampled` | The count is zero, but it came from the retained quality sample, so it rules nothing out for the rows the sample left behind. This is the ordinary state for timeliness on a source larger than the sample. A nonzero count is still reported |
| `unrecorded` | The document was written before dataprof recorded what the rule needs: ragged rows before 0.10, quality sample coverage and locale-formatted numbers before 0.12 |
| `not_computed` | The metric was not computed for the listed `columns` (pack not selected, or a nested column) |
| `no_values` | The listed `columns` had no values to look at |
**Findings are derived, not stored.** They are not part of the saved report,
and a report loaded with `ProfileReport.load()` or `from_dict()` yields the
same findings as the one that was saved. Findings describe the rows the report
read: a truncated or sampled scan adds `partial_scan` rather than withholding
the rest. For a pass/fail decision about the whole source, use `check()`.
`result.to_dict()` / `to_json()` serialize the whole result, identical to what
the Rust `FindingPolicy` writes for the same report and thresholds.
### `check()` -- quality gates
State a policy as data and get a structured verdict back. Nothing is printed,
the process is not exited, and the report is not modified, so this composes
into Airflow, Dagster, a GitHub Action, or a shell script -- no CLI involved.
```python
report = dp.profile("daily_drop.csv")
result = report.check(
min_quality_score=90,
max_null_count={"customer_id": 0},
max_null_percentage={"*": 20},
max_duplicate_rows=0,
require_metrics=["quality"],
)
if not result.passed:
for check in result.violations:
print(check.code, check.column, check.message, check.observed)
```
| Keyword | Type | Meaning |
|---|---|---|
| `min_rows` | `int` | Floor for the number of rows the scan read |
| `min_quality_score` | `float` | Floor for the overall score, 0--100 |
| `min_dimension_scores` | `dict[str, float]` | Floor per ISO 25012 dimension, 0--100 |
| `max_null_percentage` | `dict[str, float] \| float` | Ceiling for a column's null percentage, 0--100. `"*"` sets the limit for every column without its own entry; a bare number means the same as `{"*": number}` |
| `max_null_count` | `dict[str, int] \| int` | Ceiling for a column's null count, decided on the exact count (0.14+). Same shape as `max_null_percentage`; `{"customer_id": 0}` means no nulls at any row count |
| `max_duplicate_rows` | `int` | Ceiling for the duplicate-row count |
| `require_metrics` | `list[str]` | Metrics that must have been analyzed: `"quality"`, or a dimension name |
| `scope` | `str` | `"full_source"` (default) or `"observed"` |
| `baseline` | `ProfileReport` | A saved report to measure change from; the three rules below need one |
| `max_quality_score_drop` | `float` | How many points the overall score may fall below the baseline's, 0--100 |
| `max_dimension_score_drops` | `dict[str, float]` | The same per ISO 25012 dimension |
| `max_null_percentage_increase` | `dict[str, float] \| float` | How many points a column's null percentage may rise above the baseline's, 0--100. `"*"` covers every other column both reports profiled |
| `require_columns` | `list[str]` | Columns this report must have; needs no baseline |
| `forbid_removed_columns` | `bool` | Every column the baseline profiled must still be profiled |
| `forbid_added_columns` | `bool` | No column the baseline did not profile: a closed schema |
Thresholds are percentages on the same 0--100 scale the report reports, not
0--1 ratios. A score, its bounds, a null percentage and the threshold are
compared at the 2-decimal precision the check shows them at, so a score shown
as `66.67` meets a floor of `66.67`, and a saved report reaches the verdict the
live one did (0.13+). The price is that a zero null-percentage limit is a
rounded endpoint: one null in 20,001 rows shows as `0.0` and passes it. For a
no-null rule on a key or required column, use `max_null_count={"column": 0}`,
which compares the exact count and needs no rounding. Like `max_duplicate_rows`,
it fails a truncated or sampled scan that already counted more nulls than
allowed, and otherwise leaves a full-source rule unevaluated on such a scan.
A policy that cannot be evaluated as written -- a threshold
outside the range, an unknown dimension, no requirement at all -- raises
`ValueError` rather than failing the dataset: a misconfigured gate is not a
bad extract. A threshold must be a real number: text is refused even when it
spells one (`"90"`), and so is a number too large for a float. Numpy scalars,
`Decimal` and `Fraction` are accepted.
The gate is evaluated in Rust, the same code a Rust caller uses. It reads a
profiled report or one loaded from the JSON `save()` and `to_json()` write. A
report loaded from a flat `to_dict()` summary raises `ValueError`: the summary
does not record the provenance the gate decides on.
**The verdict has three values, not two.**
| `result.verdict` | Meaning |
|---|---|
| `"fail"` | At least one requirement was conclusively violated |
| `"inconclusive"` | Nothing was violated, and something could not be checked |
| `"pass"` | Every requirement was evaluated and met |
`result.passed` is True only for `"pass"`, so an unanswerable gate never reads
as a green one. `"fail"` wins over `"inconclusive"`: a witnessed violation is a
decision.
A requirement goes unevaluated -- and lands in `result.unevaluated` -- when:
- the report carries no quality assessment (`reason: quality_unavailable`,
carrying the report's own `quality_status`, so "you did not ask for this" is
distinguishable from "this broke");
- the dimension or column had nothing to measure (`reason: not_assessed`);
- the report holds no profile for a named column (`reason:
column_not_profiled`) -- it was projected away or it is not in the source,
and a report does not record which, so it is not decided either way;
- the evidence does not reach as far as the requirement does (`reason:
evidence_incomplete`).
`require_metrics` is how a caller turns the first of those into a failure,
which is usually what a pipeline wants when quality is supposed to be
configured on.
**Scope and evidence.** `scope="full_source"` states the requirements about the
entire source. A ratio or an average computed over a truncated scan, a sampled
scan, or a bounded quality sample bounds nothing about the rows that were not
read, so such a requirement is left unevaluated rather than passed. The
exception is a witnessed count: rows already seen as duplicates do not stop
being duplicates when more rows are read, so an exact duplicate count above the
allowance fails conclusively even on a partial scan -- while a count at or
below it is still not a pass, and an estimated count settles neither direction.
`scope="observed"` states the requirements about whatever the metrics actually
measured, which is always evaluable and says nothing about the rest of the
source.
Each check records its own `evidence`, which can be weaker than the scan's:
`report.quality_sampled_dimensions` names the components computed from a
retained sample, and a fully read file can still have them. Provenance is
resolved per component, so a completeness score from exact column counters is
decided on its value, and the overall score only counts as sampled when a
sampled component actually reaches it (a dimension the score weights exclude
cannot move the aggregate).
**Sampled scores on a fully read source (0.12+).** Most quality dimensions are
computed over a uniform sample of up to 10,000 values per column, so on any
larger source the scores are estimates. The report bounds them:
`report.quality_score_bounds` gives an interval per score that holds, all at
once, with probability `confidence_level` (0.999). The sample fixes each check
(a column's type, dominant form, detected pattern, decimal scale and outlier
fences); the interval covers what those same checks give over every value the
scan read. When the scan read the whole source and the only gap is the
quality sample, a `min_quality_score` or `min_dimension_scores` requirement is
decided on that interval: it passes when the whole interval meets the minimum,
fails when none of it does, and stays `evidence_incomplete` only when the
minimum falls inside it. The check records the interval it used as `bounds`
next to `evidence`, which still says `quality_sampled`. On a clean column of
10,000 sampled values the interval is about 0.15 points wide; at 5% failures,
about 2 points.
Past a million distinct values the key count and the duplicate-row count are
estimates rather than samples. They are bounded with certainty instead of at
0.999: when an exact count is dropped, the distinct values it held are a floor
the true count cannot go under. Uniqueness then gets an interval from those
floors, wide on a source far past a million rows (the floor is a million of
its rows), and the overall score is bounded again. A `max_duplicate_rows`
requirement on an estimated count stays unevaluated, since the estimate can
neither witness duplicates nor rule them out.
`min_rows` reads the report's `execution.rows_processed`. Rows already read do
not go away when more are read, so a scan that read at least `min_rows` rows
passes even when it stopped early or sampled. One that read fewer fails when it
covered the whole source, or under `scope="observed"`; under `"full_source"` a
scan that stopped early leaves it `evidence_incomplete`, since the rows it did
not read may make up the difference.
A duplicate-row scan over a sample, on engines without a row tracker, has no
bound, and a requirement that reads it stays unevaluated. Start/end date ordering is
compared only between columns whose samples hold the same rows, so a pair
where either date column has nulls is not compared. When no pair is compared,
the `temporal_order_violations` finding reads `not_assessed`.
`scope="observed"` is unchanged and decides on the sampled score itself.
A report loaded from a document written before dataprof recorded this says
nothing about how its numbers were obtained. Unknown coverage is a third answer
rather than "nothing was sampled", so `quality_sampled_dimensions` is `None`
and a full-source check reports `coverage_unrecorded`. Both language layers
answer the same way for such a report. The same holds for the overall score
of a report written by a later release that sampled a dimension this release
does not know: that dimension's weight is in the stored score, so
`min_quality_score` and `compare()` both read it as `coverage_unrecorded`,
while each known dimension keeps its own coverage. A component the report
computed but lists neither as exact nor as sampled has no recorded coverage
either: `check()`, `compare()` and `findings()` read it, and any score that
weighs it, as `coverage_unrecorded`. No release writes such a report; a
flat `to_dict()` summary records no exact list, so it keeps reading a
component that is not sampled as exact.
**Rules about change (0.13+).** With `baseline=`, a gate also judges how far
this report moved from a saved one, read through the same delta `compare()`
returns, at the serialized precision on both sides:
```python
baseline = dp.ProfileReport.load("baseline.json")
result = dp.profile("daily_drop.csv").check(
baseline=baseline,
max_quality_score_drop=2,
max_null_percentage_increase={"customer_id": 0, "*": 5},
)
```
The allowance is compared at the same 2-decimal precision as the change, so
the decision matches the numbers the result shows. A change is decided only
when both reports measured the same thing the same way. Otherwise the rule is
unevaluated, with one of these reasons:
- `not_in_baseline`: the baseline did not assess the score, did not profile
the column, or read no values for it;
- `schema_changed`, for a score, when the two reports profiled different
columns. A dataset-level score has the column set as its subject, so a
column that appeared or vanished changes what is measured. A column's own
rule is not affected and still decides the verdict, and a new column gets a
rule only when the policy names it: `"*"` covers the columns both reports
have;
- `dimensions_changed`, for the overall score, when a dimension was assessed
on one side only, since the overall score averages the assessed ones;
- `definition_not_comparable`, with the `metric_semantics` key the rule
depends on (`null_tokens` for every rule, `pattern_match_count` also for
validity and the overall score, `range_rule_count` also for accuracy and the
overall score, `key_uniqueness_units` also for uniqueness and the overall
score, `outlier_units` also for accuracy and the overall score,
`key_column_source` also for uniqueness and the overall score), when the
two reports do not both record it
identically. A baseline saved before 0.13 records neither;
- `setting_changed` or `setting_unrecorded`, with the setting, when a setting
that moves the score on unchanged data differs or is not recorded on a side:
the score weights for the overall score, `identifier_columns` (in the order
given, since the first one present is the uniqueness key) for every score
but completeness, `positive_columns` for accuracy, `temporal_columns` and
`temporal_pairs` for timeliness, and `pattern_locale` for validity. The
overall score reads them all. A report records which columns the pairs named
but not which column pairs with which, so a non-empty `temporal_pairs` reads
as `setting_unrecorded`, and so does the locale on a report saved before
0.13, which did not record it;
- `evidence_incomplete`, under `scope="full_source"`, when either side does
not cover its whole source.
Each such check carries `baseline`, holding the baseline's `observed` value,
its `evidence`, and `change` (this report minus the baseline, in points), and
its `expected` is `{"comparison": "change_at_least", "value": -2.0}` for a
drop or `change_at_most` for an increase. `result.baseline` names both reports
(`a` is the baseline) and the `schema` change between them. The baseline may be
loaded from a flat `to_dict()` summary; what it did not record reads as
unrecorded.
**Rules about the column set (0.13+).** `require_columns` names columns the
report must have, and needs no baseline. With a baseline,
`forbid_removed_columns=True` requires every baseline column and
`forbid_added_columns=True` allows no new one:
```python
result = dp.profile("orders.csv").check(
baseline=baseline,
require_columns=["order_id", "customer_id"],
forbid_removed_columns=True,
forbid_added_columns=True,
)
```
Each makes one check per column: per name for `require_columns`, per baseline
column for `forbid_removed_columns`, per column of this report for
`forbid_added_columns`, with `expected` `{"comparison": "present"}` or
`{"comparison": "in_baseline"}`. A missing column fails only when the side
without it shows its source lacks it: it recorded its column selection
(`analysis_settings["columns"]`, from 0.13), the selection included the column,
and its column set covers the whole source. A CSV header, a Parquet schema, a
DataFrame or a query result lists every column however few rows were read, so
a capped CSV run still decides; for JSON, JSONL and `dataprof.asyncio` streams
the columns are the keys of the records read, so only a scan of every record
decides under `scope="full_source"`. A flat `to_dict()` summary records the
kind of source but not a file's format, so a summary baseline of a file
decides only when its scan was complete. Otherwise the check is unevaluated,
with `setting_changed` or `setting_unrecorded` on the setting `columns`,
`column_not_profiled` for a required column the run did not select, or
`evidence_incomplete`.
`forbid_added_columns` checks the columns this report profiled, so under
`scope="full_source"` a report whose profiled columns may not be all of its
source's gets one more check, with no `column`, left unevaluated: a `columns=`
projection (`column_not_profiled`), a report saved before the selection was
recorded (`setting_unrecorded`), or a column set covering part of the source,
such as a JSON or JSONL run stopped by `max_rows` (`evidence_incomplete`). The
verdict is then inconclusive rather than a pass.
`forbid_removed_columns` is about the columns the baseline recorded. A column
its source held but it did not profile, because of a projection or a key past
a partial read, is not one it can require; profile the baseline in full when
that matters.
`result.baseline["diagnostics"]` lists the schema changes from the baseline,
whatever the policy states, at most one per column: the baseline's columns in
order, then the added ones. A diagnostic is a warning and never moves the
verdict; it is `[]` when nothing changed.
| `code` | When |
| --- | --- |
| `column_added` | The report profiled a column the baseline did not. A rule about change that names it is `not_in_baseline`; absolute rules still read it |
| `column_removed` | The baseline profiled a column the report has no profile for |
| `column_type_changed` | The column holds values on both sides and its inferred type differs; `data_type` gives both, `a` first. A column whose values are all null establishes no type |
| `column_all_null` | The column held values in the baseline and every value read for it now is null. Decided on the counts, so one value in 100,000, shown as `100.0` percent null, is not all null |
Column order is not compared. A flat summary that does not record a column's
counts reports neither of the last two for it.
For an added or removed column, the side without it says why, from its
`analysis_settings["columns"]`. With no `reason`, that side profiled every
column it selected and this one was not in what it read: for CSV and Parquet
the source's header or schema, for JSON and JSONL the keys of the records read,
so a scan that did not reach the end (the `evidence` of `reports.a` or
`reports.b`) may have missed a key that appears later. The async reader
(`dataprof.asyncio`) also drops a key first seen after its first chunk, even
on a complete scan, until #1020 is fixed. With `"reason":
"setting_changed"` and `"setting": "columns"`, its `columns=` selection left
the column out; `setting_unrecorded` when it was saved before 0.13, which did
not record the selection.
The type is the inferred one, so a setting can change it on unchanged data.
`identifier_columns` gives a column the `identifier` type, and a type change
to or from `identifier` under a hint that differs carries `"reason":
"setting_changed"` and `"setting": "identifier_columns"`
(`setting_unrecorded` when a side did not record its hints). An input path
that types the same data differently is not recorded, so it is not named.
```python
{"code": "column_type_changed", "column": "amount",
"data_type": {"a": "integer", "b": "string"},
"message": "this column holds values on both sides, and its inferred type differs"}
```
**`result.to_dict()` / `to_json()`** serialize the whole result, identical to
what the Rust `QualityPolicy` writes for the same report and policy:
```python
{
"verdict": "fail",
"scope": "full_source",
"evidence": {"coverage": "complete"},
"checks": [
{
"code": "max_null_percentage",
"column": "customer_id",
"expected": {"comparison": "at_most", "value": 0.0},
"observed": 16.67,
"scope": "full_source",
"evidence": {"coverage": "complete"},
"status": "failed",
"message": "this column's null percentage is above the allowance",
}
],
}
```
A score decided on its interval also carries `bounds`:
```python
{
"code": "min_quality_score",
"expected": {"comparison": "at_least", "value": 90.0},
"observed": 97.41,
"scope": "full_source",
"evidence": {"coverage": "incomplete", "reason": "quality_sampled"},
"bounds": {"lower": 96.87, "upper": 97.93, "confidence_level": 0.999},
"status": "passed",
"message": "the overall quality score was computed over a sample, and its whole-source interval meets the required minimum",
}
```
`message` carries no numbers on purpose: `observed` and `expected` hold those,
so nothing depends on how a float was formatted into a sentence.
See [`python/examples/etl_quality_gate.py`](../../python/examples/etl_quality_gate.py)
for a runnable gate over four drops, and
[`python/examples/baseline_checkpoint.py`](../../python/examples/baseline_checkpoint.py)
for one that judges each drop against the last accepted report, promotes only
on a pass, and shows a schema change, a metric that was not analyzed and a
baseline written by an older release reaching their verdicts.
### `python -m dataprof.check` -- CI entrypoint (0.12+)
Evaluate the same policy from a shell or CI job:
```bash
python -m dataprof.check daily_drop.csv --min-quality 90 --max-null-count 'customer_id=0' --json
python -m dataprof.check daily_drop.csv --policy quality-policy.json --json > verdict.json
```
To keep the full profile behind the verdict, add `--report PATH` (0.13+):
```bash
python -m dataprof.check daily_drop.csv --min-quality 90 --report profile.json --json > verdict.json
```
The path must end in `.json`; reload it with `dp.ProfileReport.load()`. It is the
report the verdict was computed from, written on `pass`, `fail` and
`inconclusive`, and not when profiling or the comparison with a baseline fails. An existing file is overwritten; a
write failure exits `3` and names the path. `--json` stdout is unchanged.
The report path must not refer to the source, policy or baseline file, including
through symbolic links, hard links or case-insensitive path aliases.
The policy file is a UTF-8 JSON object containing the `check()` keywords above;
a leading byte-order mark (BOM) is accepted.
Thresholds are JSON numbers; a quoted one such as `"90"` is a policy error
(exit 3). Commit it alongside the pipeline so threshold changes can be reviewed:
```json
{
"min_rows": 1,
"min_quality_score": 90,
"max_null_count": {"customer_id": 0},
"max_null_percentage": {"*": 20},
"max_duplicate_rows": 0,
"require_metrics": ["quality"],
"scope": "full_source"
}
```
Flags replace the corresponding policy-file key in full. For example,
`--max-null '*=10'` replaces the entire `max_null_percentage` mapping, including
any column-specific entries. Repeat `--max-null COLUMN=PERCENT`,
`--max-null-count COLUMN=COUNT` (0.14+), `--min-dimension NAME=PERCENT`, `--require-metric NAME`, or `--require-column NAME`
to supply multiple entries; the last occurrence of a repeated mapping key wins.
`--min-rows`, `--min-quality`, `--max-duplicate-rows`, and `--scope` cover the
other policy keywords. Empty policies, unknown keywords, and duplicate JSON keys (including
inside dimension or column mappings) are policy errors (exit 3).
`--engine`, `--format`, `--max-rows`, and repeatable `--metric PACK` pass through
to `profile_file()`. As with the library, capped or sampled evidence may leave a
full-source requirement inconclusive. `--scope observed` explicitly limits the
claim to the analyzed population. A threshold on an unavailable metric stays
unevaluated; an explicit `--require-metric` requirement fails if it is absent.
| Exit code | Meaning |
|---|---|
| `0` | Every requirement passed; also used by `--help` and `--version` |
| `1` | The gate found a proven violation, even if other checks are unevaluated |
| `2` | The gate is inconclusive: a threshold could not be evaluated |
| `3` | An argument, policy, source, or baseline could not be read or used, or the report could not be written |
Human summaries go to stderr: the verdict, each check that did not pass, and
with `--baseline` each schema-change diagnostic, which a pass can carry. With
`--json`, stdout contains only the existing `QualityGateResult.to_json()`
document, including inconclusive results. Errors
(exit `3`) write an error object instead of a result document:
`{"error": {"kind": "argument" | "policy" | "input" | "baseline" | "output", "message": "...", "path": "..."}}`,
where `path` is present when the error names a file. A `policy` error without
`path` comes from a command-line flag (for example `--min-quality 150`), not
from the policy file. Without `--json`, errors
go to stderr and stdout stays empty. `--help` prints usage to stdout and lists
every flag and code. `--version` prints `dataprof <version>` to stdout and exits 0.
#### Against a baseline (0.13+)
`--baseline PATH` compares the run with a report an earlier run saved, through
`ProfileReport.check(baseline=...)`. The rules about change need it:
| Flag | Policy key |
|---|---|
| `--max-quality-drop POINTS` | `max_quality_score_drop` |
| `--max-dimension-drop NAME=POINTS`, repeatable | `max_dimension_score_drops` |
| `--max-null-increase COLUMN=POINTS`, repeatable, `*=POINTS` for every column both reports profiled | `max_null_percentage_increase` |
| `--forbid-removed-columns` | `forbid_removed_columns` |
| `--forbid-added-columns` | `forbid_added_columns` |
A rule about change without `--baseline` is a policy error (exit `3`). The
baseline is a `.json` report that `--report` or `save()` wrote, or a flat `to_dict()` summary,
which leaves what it did not record unrecorded. A baseline that cannot be read,
is not a report, or cannot be compared exits `3` with an error of kind `baseline`
naming the path.
When the two reports did not measure the same thing the same way (a changed
metric definition or setting, a schema change behind a score, or incomplete
evidence under `full_source`), the rule is unevaluated and the run exits `2`, with
the reason in the check. `--json` output then carries `baseline`: both reports,
how their columns differ, and diagnostics that never move the verdict.
`--report` cannot write over the baseline, because a report is written whatever
the verdict and a failing run would become the next baseline. Write it elsewhere
and promote it on a pass, the first run included:
```bash
# First run: no baseline yet, so absolute rules only; promote the report on a pass.
python -m dataprof.check orders.csv --min-quality 90 --report candidate.json \
&& mv candidate.json baseline.json
# Later runs: compare with the last accepted report, promote today's on a pass.
python -m dataprof.check orders.csv --min-quality 90 \
--baseline baseline.json --max-quality-drop 5 --max-null-increase '*=2' \
--forbid-removed-columns --report candidate.json --json > verdict.json \
&& mv candidate.json baseline.json
```
The [baseline checkpoint example](../../examples/README.md#4-baseline-checkpoint)
runs this cycle over seven drops, including the ones that must not become the
baseline, and shows it as a CI job that keeps the baseline in the repository.
For a project with `daily_drop.csv` and `quality-policy.json` checked in, this
GitHub Actions job runs the gate and retains its result even when it fails:
```yaml
name: Data quality
on: [push, pull_request]
jobs:
quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7.0.1
- uses: actions/setup-python@v7
with:
python-version: '3.12'
- run: python -m pip install dataprof==0.13.1
- name: Check incoming data
run: python -m dataprof.check daily_drop.csv --policy quality-policy.json --json > verdict.json
- uses: actions/upload-artifact@v7
if: always()
with:
name: quality-verdict
path: verdict.json
```
This example targets the 0.12 release. Before it is published, build this
checkout with `uv sync` and `uv run maturin develop`, then invoke the module with
`uv run --no-sync python -m dataprof.check ...`.
### `compare()`
`a.compare(b)` returns what changed from `a` to `b`, such as a saved baseline
and today's run. It is computed by the Rust `ReportDelta`, so both languages
return the same document:
```python
baseline = dp.ProfileReport.load("baseline.json")
delta = baseline.compare(dp.profile("orders.csv"))
delta["quality_score"] # {"a": 91.2, "b": 88.75, "abs": -2.45, "rel_pct": -2.69, "evidence": {...}}
delta["dimensions"]["completeness"] # the same shape, for each of the seven dimensions
delta["columns"]["email"] # {"null_pct_a": 1.0, "null_pct_b": 4.5, "null_pct_delta": 3.5}
delta["schema"] # {"added": [...], "removed": [...], "common": [...]}
delta["metric_semantics"]["comparable"] # True, False, or None when either side predates the record
delta["reports"]["b"] # {"id", "timestamp", "source", "rows_processed", "evidence"}
```
- **Signs.** Every change is `b - a`. A positive score change is an
improvement; a positive `null_pct_delta` is more nulls.
- **Precision.** Both sides are compared at their serialized 2-decimal
precision, so a report and its saved file compare alike.
- **No invented numbers.** A score either side did not assess, or a column one
side lacks or that holds no values, leaves the change `None`. `rel_pct` is
`None` when `a` is 0.
- **Evidence.** Each score carries the coverage behind it on both sides, with
the reasons `check()` uses (`truncated`, `sampled`, `records_skipped`,
`quality_sampled`, `coverage_unrecorded`). A change between an exact
baseline and a sampled or truncated run can be a sampling difference.
- **Definitions.** Only `comparable: True` means a change in a measurement
is a change in the data. Text lengths counted UTF-8 bytes through 0.11, so
across that boundary a changed `max_length` on non-ASCII text can be the
unit rather than the data. `comparable` covers how dataprof measures, not
how each report was configured.
- **Configuration.** `configuration` lists the recorded settings that move a
score on unchanged data: `score_weights`, `semantic_hints` (the columns
named by `positive_columns`, `identifier_columns`, `temporal_columns` and
`temporal_pairs`), and `pattern_locale` (see `analysis_settings`). Each has
`a`, `b` and `same`, which is `None` when a side did not record the
setting: weights on a summary saved before 0.12, hints on one saved before
0.10, the locale on one saved before 0.13. A score that moved while `same` is `False` may
reflect the setting rather than the data. Settings not recorded here, such
as `metrics` and the timeliness reference date, still need the same
options on both sides.
Either side may be a report loaded from a flat `to_dict()` summary, including
one saved before 0.12; its `id` and `timestamp` are `None`, and coverage it
did not record reads as `coverage_unrecorded`. The shape is stable: later
releases may add keys, but do not remove or rename them.
### `save()`
```python
report.save("report.json") # full report as JSON
report.save("profiles.csv") # column profiles as CSV (no extra deps)
report.save("profiles.parquet") # column profiles as Parquet (requires pyarrow)
report.save("report.html") # HTML fragment (same as to_html())
report.save("report.md") # markdown table (same as to_markdown())
```
## Partial Analysis
Fast operations that don't require a full profile:
### `infer_schema()`
```python
result = dp.infer_schema("data.csv")
print(f"{result.num_columns} columns, {result.rows_sampled} rows sampled")
for col in result.columns:
print(f" {col['name']}: {col['data_type']}")
```
Every format runs the same inference `profile()` does. On a Parquet file that
means the metadata answers for every column it types, and text columns are
sampled: only their values say whether they hold dates, integers or booleans,
which is what a writer that did not type its input leaves behind.
`rows_sampled` reports what that cost -- `0` for a fully typed file.
The sample is bounded in every format, so the type matches a full `profile()`
whenever `schema_stable` is true -- nothing was left unread that could move it,
either because the scan reached the end of the file or because the Parquet
metadata typed every column and no scan was needed. Once inference stops at the
cap, a later row can still move it.
### `analyze_structure()`
```python
structure = dp.analyze_structure("data.parquet")
for column in structure.columns:
print(f"{column.name}: {column.data_type} nulls={column.null_count} ({column.provenance})")
```
Every counter it reports describes the whole file or is absent -- it never
returns a partial count. For CSV and JSON that means checking `truncated`; for
Parquet the counts come from the file footer, so they cover every row even when
the type sample stopped at `max_rows`. Distinct counts are absent for Parquet,
which has no whole-file source for one.
### `quick_row_count()`
```python
result = dp.quick_row_count("data.parquet")
print(f"{result.count} rows ({'exact' if result.exact else 'estimated'})")
print(f"Method: {result.method}, took {result.count_time_ms}ms")
```
CSV counts scan every logical record, including quoted multiline fields, and
return `exact=True`, method `full_scan`, and no relative-error estimate. This
costs a complete read even for large CSV files. Parquet uses footer metadata;
large JSONL files retain their sampled density estimate.
## Async API
The `dataprof.asyncio` module provides async variants for use in web frameworks, stream processors, and other async contexts. These helpers ship in the published wheels.
```python
from dataprof.asyncio import profile_file, profile_bytes, profile_url
# Async file profiling
report = await profile_file("data.csv", max_rows=10000)
# Profile raw bytes (e.g. from an HTTP request body)
report = await profile_bytes(csv_bytes, format="csv")
# Profile a remote file over HTTP
report = await profile_url("https://example.com/data.parquet")
```
Remote Parquet is read with several ranged requests, all tied to the `ETag`
the server sent first. If the object is replaced while it is being read,
`profile_url` raises `OSError` naming the URL instead of profiling a mix of the
two versions. A server that sends no `ETag` leaves nothing to check against.
Async byte streams use the incremental engine; `engine="columnar"` is rejected.
Local async Parquet profiling honors `max_rows` and row-limit stop conditions,
and rejects stop conditions or sampling strategies the Parquet reader cannot
apply.
Additional async utilities:
```python
from dataprof.asyncio import infer_schema_stream, quick_row_count_stream
schema = await infer_schema_stream(csv_bytes, format="csv")
count = await quick_row_count_stream(csv_bytes, format="csv")
```
## Database Profiling (Source Build Only)
Async database functions for PostgreSQL, MySQL, and SQLite are **not** in the
published wheels, and there is no extra that installs them. On a wheel install
they exist as stubs that raise `ImportError` when called, and
`dp.capabilities().database` is `False`. They need a source build with
`python-async`, `database`, and the connector features you want.
### From the published sdist, with pip
No checkout required:
```bash
pip install dataprof --no-binary dataprof --config-settings="build-args=--features python,python-async,async-streaming,parquet-async,database,sqlite"
```
### From a checkout
```bash
uv run maturin develop --features "python,python-async,async-streaming,parquet-async,database,sqlite"
```
### List every feature you want, not only the extra ones
`--features` **replaces** the `[tool.maturin] features` list in
`pyproject.toml`; it does not add to it. This is the part that bites, because
the result looks like it worked:
```bash
# Wrong: builds database support and silently drops async, URL profiling,
# and remote Parquet β surface that a plain `pip install dataprof` includes.
--features "python,python-async,database,sqlite"
```
That command yields `capabilities().database == True` alongside
`async_streaming == False`, `url_profiling == False`, and
`remote_parquet == False`. Nothing errors; the package is simply smaller than
the published wheel. The lists above are the shipped set plus the database
features, which is why they are long.
`database` on its own is also not enough: the connectors are reported only when
`python-async` is compiled in as well.
A source build needs a Rust toolchain (1.96 or later; CI compiles the extension
crate on exactly 1.96, so that floor is tested rather than assumed).
`postgres` and `mysql` are pure Rust; `sqlite` compiles `libsqlite3-sys`, which
is C.
Check what you actually got before relying on it:
```python
import dataprof as dp
caps = dp.capabilities()
assert caps.database and "sqlite" in caps.database_connectors
assert caps.async_streaming # still present, because the list above kept it
```
Then the following APIs become available:
```python
import asyncio
import dataprof as dp
async def main():
# Test connection
ok = await dp.test_connection_async("postgres://user:pass@localhost/mydb")
# Profile a query
report = await dp.analyze_database_async(
"postgres://user:pass@localhost/mydb",
"SELECT * FROM users",
batch_size=10000,
)
print(f"{report.rows} rows, quality: {report.quality_score}")
# Get table schema
columns = await dp.get_table_schema_async(
"postgres://user:pass@localhost/mydb", "users"
)
# Count rows
count = await dp.count_table_rows_async(
"postgres://user:pass@localhost/mydb", "users"
)
asyncio.run(main())
```
`batch_size` must be greater than zero. Query result column names must be
unique; duplicate aliases are rejected before values can be merged.
## Arrow Interop
`profile()` also accepts one-shot producers implementing the
[Arrow C Stream capsule protocol](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html#arrowstream-export):
```python
import pyarrow as pa
import dataprof
batches = [pa.record_batch({"id": [1, 2]}), pa.record_batch({"id": [3, 4]})]
reader = pa.RecordBatchReader.from_batches(batches[0].schema, iter(batches))
report = dataprof.profile(reader, max_rows=3)
assert report.rows == 3
assert report.source_type == "stream"
# With DuckDB installed, pass a relation directly:
# report = dataprof.profile(duckdb.sql("select * from my_table"), max_rows=1000)
```
dataprof consumes and releases each batch before fetching another; it does not
call `to_arrow_table()` or collect the stream. Producer batch buffers and the
bounded profiling accumulators determine memory use. A producer may itself
materialize data when exporting; dataprof cannot control that allocation.
PyArrow and DuckDB remain optional producer dependencies.
`max_rows` stops at the requested row boundary, including zero, without asking
for a subsequent batch. Reaching the cap sets `truncation_reason` even if that
row happened to be the stream's last: end-of-stream was not checked. An uncapped
or shorter stream is complete only after its producer signals the end. The
report uses engine `columnar`, source type `stream`, and source system
`arrow_c_stream`; the name identifies one profiling operation (`batch_id="0"`).
The stream is one-shot: use a fresh reader for another profile.
Metric packs, quality dimensions, locale, semantic hints, and column selection
work as on Arrow tables. Unsupported file/transport controls raise `ValueError`
before capsule export. Duplicate names are rejected before any batch is read,
even with column selection. Selected nested types (structs, lists, maps, unions)
raise `TypeError`; select flat columns before exporting, or with `columns=`.
An empty stream with a schema retains its columns and unassessed quality.
Batch/schema failures abort the call; no partial report is returned. The C
interface carries producer error text, which is retained as `__cause__`, rather
than the original Python exception object. Existing Table, C Array, pandas, and
polars adapters retain their behavior.
The `RecordBatch` class supports zero-copy exchange via the [Arrow PyCapsule interface](https://arrow.apache.org/docs/format/CDataInterface/PyCapsuleInterface.html):
```python
import dataprof as dp
import pyarrow as pa
# Profile a PyArrow table directly
report = dp.profile(table)
# RecordBatch properties
batch.num_rows
batch.num_columns
batch.column_names
# Convert to other formats
df = batch.to_pandas()
pl_df = batch.to_polars()
# PyCapsule protocol for zero-copy exchange
schema_capsule = batch.__arrow_c_schema__()
array_capsule = batch.__arrow_c_array__()
```