Parqkit — Fast Parquet CLI and Rust library
Inspect, preview, convert, and merge Parquet files with pretty interactive output and predictable formats for scripts. The typed, streaming library API is the foundation for future jq-like querying.
Installation
Parqkit requires Rust 1.85 or newer.
From a source checkout:
Usage
parqkit <COMMAND>
Commands:
schema Show schema (column names, types, nullability)
head Show first N rows (default 10)
tail Show last N rows (default 10)
count Count total rows
stats Column statistics (min, max, nulls)
convert Convert to CSV, JSON, or JSONL
merge Merge multiple parquet files
info File metadata (row groups, compression, size)
Common command options
schema,head,tail,stats, andinfosupport-o, --output <table|json|jsonl|csv>headandtailsupport-n, --rows <N>schema,head,tail,count,stats, andinfosupport-q, --quietconvertinfers the output format from the destination file extension:.csv,.json, or.jsonl
Examples
View schema
| | | |
+================================+
| | | |
| | | |
| | | |
Preview data
| | | |
+=========================+
| | | |
| | | |
| | | |
Count rows
Column statistics
| | | | | | |
+=====================================================+
| | | | | | |
| | | | | | |
File info
| | |
+================================================+
| | |
| | |
| | |
| | |
| | |
| | |
Convert formats
Merge files
Output formats
Read-oriented commands support multiple output formats:
Schema and stats JSON output include display type plus explicit physical/logical type
metadata. Nested Parquet columns use their full dotted paths. Stats output includes a
statistics_complete field; unavailable null counts and bounds are emitted as null
instead of misleading zeroes or partial values. Signed and unsigned integers, finite
floats, and booleans use native JSON types. Decimals use exact scaled strings,
non-finite floats use "NaN", "Infinity", or "-Infinity", and non-logical binary
values use deterministic hexadecimal strings. Parqkit ignores bounds whose column order
is legacy, unknown, or undefined rather than presenting them as authoritative.
Empty Parquet scans retain their schema, so CSV output still includes headers and multi-file structured scans still validate schema compatibility.
count prints plain text counts, convert writes the format implied by the output file extension, and merge writes a Parquet file.
Glob support
Quote patterns to let Parqkit expand, sort, and deduplicate glob matches itself. Explicitly repeated file arguments remain repeated.
An existing literal path takes precedence over pattern syntax, so names containing *, ?, or [ remain usable.
Features
- Fast startup
- Batch-oriented reads and conversions
- Multiple output formats
- Glob pattern support
- Snappy compression for merge output
Development
- Core contracts captures the foundation invariants for input handling, output rendering, safe writes, and error behavior.
- Release process documents the tag-driven crates.io and GitHub release pipeline.
- Stress fixtures and the
parqkit-generatehelper are opt-in:cargo test --all-featuresorcargo build --features stress-tools --bin parqkit-generate. - CI audits dependencies, builds, formats, lints, tests, documents, and packages the crate with the committed lockfile.
Streaming library queries
The library API can project and stream Arrow batches without loading complete scan results:
let dataset = dataset_from_inputs?;
let options = QueryOptions ;
let stream = execute_query?;
let _output_schema = stream.compatible_schema?;
for result in stream
Planning resolves every projected source schema before iteration. Projection, limits, and batching are pushed into the Parquet reader; consumers validate multi-source compatibility before output, and existing output code remains responsible for atomic file replacement.