Expand description
Typed Parquet inspection, streaming, conversion, and merge operations.
The library keeps input expansion and validation in Dataset, returns
typed metadata and Arrow results, and leaves presentation to callers. Use
execute_query when rows should be decoded lazily in bounded batches.
§Example
use parqkit::{dataset_from_inputs, execute_query, Projection, QueryOptions, Result};
use std::path::PathBuf;
fn main() -> Result<()> {
let dataset = dataset_from_inputs(vec![PathBuf::from("data.parquet")])?;
let options = QueryOptions {
projection: Projection::Columns(vec!["name".into(), "id".into()]),
limit: Some(1_000),
batch_size: 256,
};
let stream = execute_query(&dataset, options)?;
let _schema = stream.compatible_schema()?;
for result in stream {
let batch = result?.batch;
// Consume this bounded Arrow record batch.
let _row_count = batch.num_rows();
}
Ok(())
}Structs§
- Column
Info - Typed schema metadata for one Parquet leaf column.
- Column
Stats - Aggregated Parquet metadata statistics for one leaf column.
- Column
Type - Physical and optional logical type metadata for a Parquet leaf column.
- Count
Entry - Metadata row count for one source occurrence.
- Count
Result - Per-source and aggregate row counts for a dataset.
- Dataset
- Ordered, validated Parquet input paths.
- File
Info - File-level Parquet metadata.
- Query
Batch - One decoded batch from a streaming query.
- Query
Options - Controls streaming query execution for each source in a dataset.
- Query
Source - Metadata produced while planning one source in a streaming query.
- Query
Stream - A pull-based stream of projected Arrow record batches.
- Scan
Options - Options for an eager row scan.
- Scan
Result - Eager scan output for one source.
- Schema
Result - Leaf-column schema information for one source.
- Stats
Result - Column-statistics rows for one source.
Enums§
- Compression
Codec - Parquet column-chunk compression codec.
- Compression
Summary - File-level summary of column-chunk compression codecs.
- Logical
Type Kind - Supported Parquet logical type annotation.
- Parqkit
Error - User-facing error with context
- Physical
Type - Parquet physical storage type.
- Projection
- Columns decoded by a streaming query.
- Scan
Kind - Direction used by an eager row scan.
- Stat
Value - Typed value read from Parquet statistics metadata.
- Time
Unit - Resolution for Parquet time and timestamp logical types.
Constants§
- DEFAULT_
QUERY_ BATCH_ SIZE - Default maximum rows yielded in one streaming query batch.
Functions§
- count
- Read row counts from Parquet metadata for every dataset source.
- dataset_
from_ inputs - Build a validated dataset from explicit paths and glob patterns.
- execute_
query - Plan a streaming query over every source in
dataset. - info
- Read file-level Parquet metadata for every dataset source.
- merge
- Merge all dataset sources into one Snappy-compressed Parquet file.
- scan
- Eagerly read the first or last rows from every dataset source.
- schema
- Read the Parquet leaf-column schema for every dataset source.
- stats
- Read and aggregate column statistics from Parquet row-group metadata.
Type Aliases§
- Result
- Result type returned by Parqkit library operations.