ProofFrame

Arrow-native data contracts, canonical fingerprints, and proof receipts for Rust.

ProofFrame is a Rust crate for checking Arrow RecordBatchReader streams and producing deterministic evidence:

versioned canonical BLAKE3 dataset fingerprints (pf-fp-v1);
typed validation reports with valid, violation_count, truncated, and bounded row findings;
exact keyed diffs with added, removed, and changed-column evidence;
high-signal PII scanning and train/test leakage checks;
Ed25519 signed proof receipts;
#![forbid(unsafe_code)];
optional Python bindings behind the python feature, not enabled by default.

The default crates.io build is a normal Rust library. It does not require PyO3, does not enable pyo3/extension-module, and does not package Python source into the crate artifact.

Install

cargo add proofframe@0.4.0-alpha.4

Quick use

use proofframe::{profile_reader, validate_reader, ColumnContract, Contract};
use std::collections::HashMap;

let profile = profile_reader(reader)?;

let contract = Contract {
    columns: HashMap::from([
        ("id".to_string(), ColumnContract {
            required: true,
            unique: true,
            not_null: true,
            ..ColumnContract::default()
        }),
    ]),
    max_findings: 100,
};

let report = validate_reader(reader_again, &contract)?;

assert!(profile.fingerprint.starts_with("pf-fp-v1:"));
assert_eq!(report.valid, report.violation_count == 0);

Public Rust API

The core functions accept Arrow readers and return typed Rust structs:

profile_reader(reader) -> Result<Profile, ProofFrameError>
profile_reader_with_distinct(reader, DistinctMode::None|Exact) -> Result<Profile, ProofFrameError>
fingerprint_reader(reader) -> Result<String, ProofFrameError>
validate_reader(reader, &contract) -> Result<ValidationReport, ProofFrameError>
validate_fast_reader(reader, &contract) -> Result<FastValidationReport, ProofFrameError>
diff_readers(before, after, &keys) -> Result<DiffReport, ProofFrameError>
scan_pii_reader(reader, max_findings) -> Result<PiiReport, ProofFrameError>
detect_leakage_readers(train, test, &keys, max_samples) -> Result<LeakageReport, ProofFrameError>
receipt::generate_keypair_json()
receipt::sign_json(report_json, private_key)
receipt::verify_json(receipt_json)

Features

[dependencies]
proofframe = "0.4.0-alpha.4"

The default feature set is intentionally empty. Enable python only when building the Python extension path:

proofframe = { version = "0.4.0-alpha.4", features = ["python"] }

The PyPI package enables python plus pyo3/extension-module through maturin. Plain Rust tests and cargo package should not need Python linker symbols, including on macOS.

Validation semantics

Null values are ignored by unique; combine unique with not_null when nulls must be rejected.
Float uniqueness uses IEEE bit semantics via to_bits(): -0.0 and 0.0 are distinct, and NaN payload differences are distinct.
Timestamp uniqueness and min/max use the native integer value in the declared Arrow time unit; different timestamp units or timezones are different schema types and are not normalized together.
Integer min/max and uniqueness use native Arrow integer values, not display strings.
Decimal values are fingerprinted canonically, but validation min/max does not coerce decimals through display text; add an explicit decimal rule in a future contract schema if needed.
Duplicate findings render only the offending duplicate value when user-facing evidence is emitted; clean rows are not stringified for numeric hot paths.

Why ProofFrame

Most data checks answer one narrow question: did this table pass? ProofFrame is built for the production question: what exactly was checked, why did it fail, and can we prove that later?

It keeps validation evidence bounded while still counting every violation, fingerprints the ordered Arrow stream with canonical bytes rather than display formatting, and emits deterministic reports that can be stored in CI, data contracts, and release artifacts.

Python

Python users should install the wheel from PyPI:

pip install proofframe==0.4.0a4

The Python README and CLI documentation live in the repository README.md.

proofframe 0.4.0-alpha.4