Skip to main content

Crate parqkit

Crate parqkit 

Source
Expand description

Typed Parquet inspection, streaming, conversion, and merge operations.

The library keeps input expansion and validation in Dataset, returns typed metadata and Arrow results, and leaves presentation to callers. Use execute_query when rows should be decoded lazily in bounded batches.

§Example

use parqkit::{dataset_from_inputs, execute_query, Projection, QueryOptions, Result};
use std::path::PathBuf;

fn main() -> Result<()> {
    let dataset = dataset_from_inputs(vec![PathBuf::from("data.parquet")])?;
    let options = QueryOptions {
        projection: Projection::Columns(vec!["name".into(), "id".into()]),
        limit: Some(1_000),
        batch_size: 256,
    };

    let stream = execute_query(&dataset, options)?;
    let _schema = stream.compatible_schema()?;
    for result in stream {
        let batch = result?.batch;
        // Consume this bounded Arrow record batch.
        let _row_count = batch.num_rows();
    }
    Ok(())
}

Structs§

ColumnInfo
Typed schema metadata for one Parquet leaf column.
ColumnStats
Aggregated Parquet metadata statistics for one leaf column.
ColumnType
Physical and optional logical type metadata for a Parquet leaf column.
CountEntry
Metadata row count for one source occurrence.
CountResult
Per-source and aggregate row counts for a dataset.
Dataset
Ordered, validated Parquet input paths.
FileInfo
File-level Parquet metadata.
QueryBatch
One decoded batch from a streaming query.
QueryOptions
Controls streaming query execution for each source in a dataset.
QuerySource
Metadata produced while planning one source in a streaming query.
QueryStream
A pull-based stream of projected Arrow record batches.
ScanOptions
Options for an eager row scan.
ScanResult
Eager scan output for one source.
SchemaResult
Leaf-column schema information for one source.
StatsResult
Column-statistics rows for one source.

Enums§

CompressionCodec
Parquet column-chunk compression codec.
CompressionSummary
File-level summary of column-chunk compression codecs.
LogicalTypeKind
Supported Parquet logical type annotation.
ParqkitError
User-facing error with context
PhysicalType
Parquet physical storage type.
Projection
Columns decoded by a streaming query.
ScanKind
Direction used by an eager row scan.
StatValue
Typed value read from Parquet statistics metadata.
TimeUnit
Resolution for Parquet time and timestamp logical types.

Constants§

DEFAULT_QUERY_BATCH_SIZE
Default maximum rows yielded in one streaming query batch.

Functions§

count
Read row counts from Parquet metadata for every dataset source.
dataset_from_inputs
Build a validated dataset from explicit paths and glob patterns.
execute_query
Plan a streaming query over every source in dataset.
info
Read file-level Parquet metadata for every dataset source.
merge
Merge all dataset sources into one Snappy-compressed Parquet file.
scan
Eagerly read the first or last rows from every dataset source.
schema
Read the Parquet leaf-column schema for every dataset source.
stats
Read and aggregate column statistics from Parquet row-group metadata.

Type Aliases§

Result
Result type returned by Parqkit library operations.