nc2parquet

High-performance NetCDF to Parquet converter with advanced filtering, cloud storage, and post-processing.

Features

• Multi-variable extraction — extract one or many variables per pass into a single Parquet file
• Batch processing — convert entire directory trees with a single glob pattern
• Four filter types — range, list, 2D spatial point, and 3D spatiotemporal point with intersection logic
• Post-processing pipeline — rename columns, convert units, apply mathematical formulas, and parse datetime values
• Amazon S3 support — read from and write to S3 buckets with standard AWS credential chains
• Parquet output control — choose compression codec, compression level, row group size, and column statistics
• Multi-source configuration — compose settings from CLI flags, environment variables, and JSON/YAML config files

Installation

Build from Source

git clone https://github.com/rjmalves/nc2parquet.git
cd nc2parquet
cargo build --release
# Binary is at target/release/nc2parquet

The netcdf crate is compiled with features = ["static"], so HDF5/NetCDF system headers are only required at build time. The resulting binary is self-contained.

Library Dependency

[dependencies]
nc2parquet = "0.1"

Quick Start

Basic CLI Conversion

# Convert a single variable to Parquet
nc2parquet convert examples/data/simple_xy.nc output.parquet -n data

# Extract multiple variables into one file
nc2parquet convert examples/data/pres_temp_4D.nc output.parquet \
  -N temperature,pressure

# Batch convert all .nc files under a directory
nc2parquet convert "data/**/*.nc" output/ --glob "data/**/*.nc" -n temperature

CLI with Filters and Post-Processing

# Range + list filters, then rename and convert units
nc2parquet convert examples/data/pres_temp_4D.nc output.parquet \
  -n temperature \
  --range "latitude:30:50" \
  --list "level:1000,850,500" \
  --rename "temperature:temp_k" \
  --kelvin-to-celsius temp_k \
  --formula "temp_f:temp_k*1.8+32:temp_k"

Inspect a NetCDF File

# Human-readable file summary
nc2parquet info examples/data/pres_temp_4D.nc

# JSON output for scripting
nc2parquet info examples/data/pres_temp_4D.nc --format json

# Variable-specific detail
nc2parquet info examples/data/pres_temp_4D.nc -n temperature --detailed

Library Usage

use nc2parquet::input::JobConfig;
use nc2parquet::process_netcdf_job_async;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = JobConfig::from_json(r#"{
        "nc_key": "examples/data/simple_xy.nc",
        "variable_name": "data",
        "parquet_key": "output.parquet",
        "filters": []
    }"#)?;

    process_netcdf_job_async(&config).await?;
    Ok(())
}

CLI Reference

Global flags available on every subcommand:

Key convert flags:

Configuration

Settings are resolved in the following priority order (highest wins):

1. CLI arguments
2. Environment variables (NC2PARQUET_*)
3. Configuration file (-c / --config)

Both JSON and YAML formats are supported. Generate a starter template with:

nc2parquet template basic -o config.json
nc2parquet template s3 --format yaml -o s3.yaml

See examples/configs/ for complete annotated examples covering single-file, multi-filter, and S3 scenarios.

Filter Types

All specified filters are intersected — a data point must satisfy every filter to be included.

See examples/configs/ for JSON/YAML filter configuration examples.

Post-Processing

Processors run sequentially after extraction. Results from earlier steps are available to later steps.

See examples/postprocessing/ for complete pipeline examples.

Storage

Both local paths and S3 URIs are accepted anywhere a file path is expected:

# Local to local
nc2parquet convert data/input.nc output/result.parquet -n temperature

# S3 to S3
nc2parquet convert s3://input-bucket/data.nc s3://output-bucket/result.parquet -n temperature

# Mixed
nc2parquet convert s3://input-bucket/data.nc local_result.parquet -n temperature

AWS credentials are resolved automatically via the standard AWS credential chain (environment variables, ~/.aws/credentials, IAM instance roles, etc.).

Contributing

See CONTRIBUTING.md for development setup, branch conventions, and the pull request process.

Changelog

See CHANGELOG.md for release history and upgrade notes.

License

MIT