Crate dataprof

Expand description

High-performance data profiling and quality assessment.

The dataprof crate is the user-facing facade for profiling CSV, JSON, JSONL, Parquet, database, DataFrame, and Arrow sources. Implementation details live in the workspace crates under crates/; this package keeps the public API compact and oriented around Profiler.

§Quick Start

use dataprof::Profiler;

let report = Profiler::new().analyze_file("data.csv")?;
println!("Rows: {}", report.execution.rows_processed);
println!("Quality: {:?}", report.quality_score());

Structs§

AccuracyMetrics: Accuracy metrics (ISO 25012).
AnalysisOptions: What to analyze, and how, for a single profiling run.
AnalysisSettings: The analysis settings a report was profiled with that change what its scores measure on unchanged data and are recorded nowhere else in it.
AsyncSourceInfo: Metadata about an async data source for report construction and progress tracking.
AsyncStreamingProfiler: Async streaming profiler that accepts AsyncDataSource instead of file paths.
BaselineComparison: The two reports a baseline-relative evaluation compared, and how their column sets differ.
BaselineReading: What the baseline said about the metric a rule about change reads.
BooleanStats: Statistics for boolean columns.
BytesSource: An in-memory byte buffer that implements AsyncDataSource.
Check: One requirement and what the report said about it.
CheckBounds: The interval a sampled score was decided on.
ColumnChange: One column’s change. Serialized as an entry of the columns object, keyed by the column’s name. Each side’s number covers what that side’s scan read; see ReportReference::evidence.
ColumnProfile: Profiling statistics for a single column.
ColumnSchema: A single column’s name and inferred data type.
CompletenessMetrics: Completeness metrics (ISO 8000-8).
ConfigurationComparison: The recorded settings that move a score on unchanged data, on each side.
ConsistencyMetrics: Consistency metrics (ISO 8000-61).
CsvParserConfig: Configuration for CSV parsing and analysis.
DatabaseConfig: Database configuration for connection strings and settings
DatabaseCredentials: Database credentials management with environment variable support
DateTimeStats: Statistics for date/datetime columns.
Diagnostic: A schema change between the baseline and the report.
DimensionChange: One quality dimension’s change. Serialized as an entry of the dimensions object, keyed by the dimension’s name.
ExecutionMetadata: Metadata about the profiling execution.
Finding: Something in the report that deserves attention.
FindingPolicy: The thresholds findings apply.
FindingsResult: What FindingPolicy::evaluate found.
FrequencyItem: A value and its frequency count within a column.
GateResult: The structured result of evaluating a QualityPolicy.
HttpParquetReader: An asynchronous reader that fetches byte ranges from an HTTP server using HTTP Range requests. Designed specifically for remote Parquet parsing.
IsoQualityConfig: Quality thresholds and score settings informed by ISO 8000/25012 concepts. These values are dataprof configuration, not ISO-mandated parameters.
JsonParserConfig: Configuration for JSON/JSONL parsing and scanning.
MetricSemantics: How the measurements in a report were defined.
MetricsCalculator: Engine for calculating comprehensive data quality metrics.
MySqlConnector: MySQL/MariaDB connector
NumericStats: Statistics for numeric (integer or float) columns.
ParquetConfig: Configuration options for Parquet analysis.
ParquetMetadata: Metadata specific to Parquet files
Pattern: A detected value pattern within a column (e.g. email, phone, UUID).
PatternMetadata: Public metadata for a pattern detector.
PostgresConnector: PostgreSQL connector with connection pooling support
PrecisionMetrics: Precision metrics for effective decimal-scale consistency.
ProfileReport: Complete profiling report for a data source.
Profiler: Unified profiler with builder pattern
ProfilerConfig: Plain-data configuration for a profiler
QualityAssessment: Wraps quality metrics with confidence information.
QualityMetrics: Comprehensive data quality metrics following industry standards.
QualityPolicy: What a report must satisfy.
QualityScoreWeights: Relative weights used to aggregate assessed quality dimensions.
QualityScores: Aggregate scores retained alongside rounded metric inputs in saved reports. Recomputing these from rounded ratios can change a threshold decision.
Quartiles: Quartile statistics for numeric distributions.
RecordedHints: The semantic hints a report recorded, by the configuration field that carries each, read from its hint bindings.
RecoveryEvent: A failed execution attempt followed by a retry, in chronological order.
ReportDelta: What changed between two reports. See the module docs for the contract.
ReportGenerator: The dataprof build that produced a report.
ReportReference: Which report a side is, and how much of its source it read.
ReportSide: What a comparison reads from one report.
ReqwestSource: An HTTP response body that implements AsyncDataSource.
RowCountEstimate: Result of a quick row count operation.
SchemaChange: Which columns each side has.
SchemaResult: Result of fast schema inference — column names paired with inferred data types.
ScoreBounds: Where the scores over every scanned row lie, for an assessment whose scores were partly computed over a retained sample of those rows.
ScoreChange: One score on both sides and how it moved.
ScoreInterval: A range a score lies in, on the 0-100 scale.
SemanticHintBinding: Per-column evidence of whether a semantic hint actually bound to any value.
SemanticHints: User-supplied semantic hints that affect profiling and quality metrics.
SemanticsComparison: Each side’s recorded measurement definitions, and whether they agree.
SettingComparison: One recorded setting on each side, and whether the two agree.
Sides: The same fact about each side of a comparison.
SqliteConnector: SQLite embedded database connector
SslConfig: SSL/TLS configuration for database connections
StopEvaluator: Runtime evaluator that checks a StopCondition against accumulated counters.
StructureColumnSummary: Sample-derived or metadata-derived structural summary for one column.
StructureReport: Lightweight structural summary for a file.
TextStats: Statistics for text/string columns.
TimelinessMetrics: Timeliness metrics (ISO 8000-8).
TypeHomogeneity: How a column’s non-null values distribute across LexicalClasses.
UnevaluatedRule: A rule that could not be evaluated, and why.
UniquenessMetrics: Uniqueness metrics (ISO 8000-110).
ValidityMetrics: Validity metrics derived from confidently detected semantic patterns.

Enums§

CheckCode: Stable identifier for the requirement a check came from.
CheckStatus: What happened to one requirement.
ChunkSize: How much data a streaming engine reads per chunk.
ColumnSelection: Which of the source’s columns a run profiled.
ColumnStats: Type-specific statistics for a column, determined by the inferred data type.
CountMethod: Method used to obtain the row count.
DataFrameLibrary: Source library for in-memory DataFrame profiling
DataProfilerError: Enhanced error types with more descriptive messages for DataProfiler
DataSource: Source-agnostic data source metadata.
DataType: Inferred column data type.
DiagnosticCode: Stable identifier for a kind of schema change.
EngineType: Which engine to use for profiling
Evidence: Whether the data behind a number covers the whole source.
EvidenceGap: Why the data behind a number falls short of the whole source.
EvidenceValue: One value a finding read from the report, or a threshold it applied.
Expectation: The constraint a check applied.
FileFormat: Supported file formats for data profiling
FindingCode: Stable identifier for the rule a finding came from.
FindingPolicyError: A finding threshold that cannot be applied as written.
JsonErrorPolicy: How JSON/JSONL parsing reacts to a malformed record.
JsonFormat: Which grammar a JSON source is read with.
KeyColumnSource: Which column key_uniqueness may measure, as recorded in a report’s metric semantics.
KeyUniquenessUnits: What key_uniqueness divides the key column’s distinct values by, as recorded in a report’s metric semantics.
LexicalClass: Mutually exclusive lexical form of a single non-null value.
Locale: A locale the pattern catalogue carries locale-specific detectors for.
MetricConfidence: Confidence level for quality metrics.
MetricPack: High-level categories of analysis that can be selectively enabled.
MetricValue: A number a check read, or a threshold it compared against.
NotEvaluated: Why a requirement was not evaluated.
NotEvaluatedReason: Why a rule produced no finding without having looked.
NullTokenSet: The null vocabulary a report was measured with, as recorded in its metric semantics.
OutlierUnits: What outlier_ratio is measured over, as recorded in a report’s metric semantics.
PatternCategory: Semantic category for a detected pattern.
PatternLocale: The locale a report’s detected patterns were ranked with: none, or the ISO 3166-1 alpha-2 code of a Locale the pattern catalogue supports.
PatternMatchCount: What a pattern’s match_count counts, as recorded in a report’s metric semantics.
PolicyError: A policy that cannot be evaluated as written.
PolicyScope: What the requirements in a policy are statements about.
ProgressEvent: Structured progress events emitted by profiling engines.
ProgressSink: How progress events are delivered to the consumer.
QualityAnalysisStatus: Why a report does or does not carry a quality assessment.
QualityDimension: ISO 25012 quality dimensions that can be selectively requested.
QueryEngine: Supported query engines for SQL-based profiling
RangeRuleCount: What the range and sign rules of the accuracy score count, as recorded in a report’s metric semantics.
RecoveryKind: Recovery boundaries recorded by dataprof.
RequiredMetric: A metric a policy requires to have been analyzed.
SamplingStrategy: How rows are selected for profiling.
SemanticHintKind: The kind of a semantic hint, used for diagnostics and binding evidence.
SettingDifference: How a setting that can produce a diagnostic’s change compares across the two reports. A diagnostic carrying one may not reflect a change in the data.
Severity: How much attention a finding asks for.
ShapeMomentEstimator: The estimator behind skewness and kurtosis, as recorded in a report’s metric semantics.
SlashDateOrder: The day/month order a column’s slash dates were read in.
StopCondition: A composable condition that can trigger early termination of profiling.
TextLengthUnit: The unit a report’s text lengths were measured in, as recorded in its metric semantics.
TruncationReason: Reason why profiling was truncated before exhausting the source.
Verdict: The overall outcome of a policy.

Constants§

DEFAULT_MIXED_TYPES_PERCENTAGE: Default for FindingPolicy::mixed_types_percentage. Matches the mixed types flag in the Python binding’s to_llm_context().
DEFAULT_NULL_HEAVY_PERCENTAGE: Default for FindingPolicy::null_heavy_percentage. Matches the null-heavy flag in the Python binding’s to_llm_context().
NULL_MARKERS: Missing-value markers counted as null in exactly this spelling.
REPORT_SCHEMA_VERSION: Version of the serialized ProfileReport schema written by this build.

Traits§

AsyncDataSource: A source of raw bytes that can be consumed asynchronously.
DatabaseConnector: Trait that all database connectors must implement

Functions§

analyze_column: Analyze a column with full profiling (includes pattern detection and unique counts)
analyze_column_fast: Analyze a column in fast mode (skips expensive operations)
analyze_column_with_analysis_options: Analyze a column, honouring the caller’s analysis selection.
analyze_csv_file: Analyze a CSV file, returning a full ProfileReport.
analyze_csv_from_reader: Analyze CSV data from any Read source using streaming statistics.
analyze_database: High-level function to analyze a database table or query.
analyze_database_with_options: Analyze a database table or query, honouring the caller’s full analysis selection.
analyze_json_file: Analyze a JSON or JSONL file, returning a full ProfileReport.
analyze_json_file_with_options: Analyze a JSON or JSONL file, honouring the caller’s full analysis selection.
analyze_json_from_reader: Analyze JSON/JSONL data from a buffered reader using streaming statistics.
analyze_parquet_async_http: Analyzes a remote Parquet file by fetching it via HTTP Range requests.
analyze_parquet_bytes: Profile Parquet bytes held in memory, without touching the filesystem.
analyze_parquet_bytes_with_options: Like analyze_parquet_bytes, honouring the caller’s full analysis selection.
analyze_parquet_with_config: Analyze a Parquet file with an explicit ParquetConfig.
analyze_parquet_with_options: Analyze a Parquet file, honouring the caller’s full analysis selection.
analyze_parquet_with_quality: Analyze a Parquet file with default settings and every quality dimension.
analyze_parquet_with_quality_dims
analyze_structure: Analyze a file’s structure with a bounded, lightweight pass.
calculate_datetime_stats
calculate_numeric_stats
calculate_text_stats
classify_lexical_forms: Distribute values across the lexical classes, skipping null-like tokens.
create_connector: Factory function to create appropriate database connector
detect_patterns: Detect common data patterns in a column.
infer_schema: Infer the schema (column names + data types) of a file.
infer_schema_async
infer_schema_stream: Infer schema from any async byte stream.
infer_type
is_null_like_token: Whether a text value is null under NullTokenSet::CommonMarkers.
is_parquet_file: Check if a file is a valid Parquet file by examining its magic number.
lexical_class: Classify one trimmed, non-null value into its LexicalClass.
list_patterns: List supported pattern detectors in detector order.
quick_quality_check: One-liner API for quick profiling with intelligent engine selection.
quick_quality_check_source: One-liner API for quick profiling from a DataSource.
quick_row_count: Quick row count (exact or estimated) for a file.
quick_row_count_async
quick_row_count_stream: Quick row count from any async byte stream.
validate_unique_column_names: Reject repeated names in an ordered column-name list, before any per-column state is built.

Crate dataprof

Crate dataprof Copy item path

§Quick Start

Structs§

Enums§

Constants§

Traits§

Functions§

Crate dataprof