ctddump

A Rust CLI tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF to Parquet (data) or YAML (metadata).

Links:

• 📦 crates.io
• 📖 Documentation
• 🌊 Live example report site: sample output of the summary_site.sh pipeline phase

Two data sources are supported:

Installation

Download a prebuilt binary (no Rust needed)

Grab an archive for your platform from the latest release. HDF5 and netCDF are built into the binary, so nothing else has to be installed:

VERSION=v0.26.0
TARGET=x86_64-unknown-linux-gnu    # or aarch64-unknown-linux-gnu,
                                   # aarch64-apple-darwin, x86_64-apple-darwin
curl -LO "https://github.com/AIQC-Hub/ctddump/releases/download/$VERSION/ctddump-$VERSION-$TARGET.tar.gz"
tar -xzf "ctddump-$VERSION-$TARGET.tar.gz"
cd "ctddump-$VERSION-$TARGET" && ./ctddump --version

The archive also carries the helper scripts, so the pipeline works as soon as ctddump is on your PATH.

Install from crates.io

Needs a Rust toolchain and the HDF5 development headers:

# Ubuntu / Debian
sudo apt-get install libhdf5-dev libnetcdf-dev
# macOS (Homebrew)
brew install hdf5

cargo install ctddump

The binary lands in ~/.cargo/bin. See the installation guide if the build cannot find libnetcdf.

Build from source

Needs the same system dependencies as above.

git clone https://github.com/AIQC-Hub/ctddump.git
cd ctddump
cargo build --release

The binary is placed at target/release/ctddump.

Commands

Every command and subcommand supports -h / --help.

convert: single NetCDF → Parquet

ctddump convert <subcommand> [OPTIONS] <src_file> <target_file>

ctddump convert nrt_ar   input.nc output.parquet
ctddump convert nrt_bo   input.nc output.parquet
ctddump convert nrt_mo   input.nc output.parquet
ctddump convert nrt_gl   input.nc output.parquet
ctddump convert cora     input.nc output.parquet
ctddump convert cora_legacy input.nc output.parquet

# Use a saved config preset
ctddump convert nrt_ar --config my_preset.toml input.nc output.parquet

# Override individual fields
ctddump convert nrt_bo --no-deph-source input.nc output.parquet

batch: directory tree NetCDF → Parquet or YAML (multi-threaded)

ctddump batch convert <subcommand> [OPTIONS] <src_dir>
ctddump batch header  <subcommand> [OPTIONS] <src_dir>

Recursively finds all matching .nc files and processes them in parallel. Output files keep the source stem with a new extension. If --output is omitted, each output is written alongside its source.

# Convert all NRT Arctic files to Parquet, flat into /output
ctddump batch convert nrt_ar --output /output /data/arctic

# Limit to 4 threads
ctddump batch convert nrt_ar --threads 4 --output /output /data/arctic

# Override the filename pattern
ctddump batch convert nrt_ar --pattern "AR_PR_CT_ITP-*.nc" --output /output /data

# Extract YAML metadata for all NRT files
ctddump batch header nrt --output /output /data/arctic

Default filename patterns:

header: single NetCDF → YAML metadata

ctddump header <subcommand> <src_file> <target_file>

ctddump header nrt  input.nc output.yaml
ctddump header cora input.nc output.yaml

concat: merge files from a directory tree into a single file

ctddump concat convert [OPTIONS] <src_dir> <output_file>
ctddump concat header  [OPTIONS] <src_dir> <output_file>

concat convert merges Parquet files and re-assigns profile_no and observation_no by default (pass --no-renumber to skip). Renumbering sorts rows by platform_code, profile_timestamp, longitude, latitude, pres; pass --no-pres-sort to sort without pres, keeping each profile's observations in their original source order instead of reordering them by pressure. Rows with missing (null/NaN) pres are dropped before merging by default (this keeps observation_no contiguous over the remaining rows); pass --keep-na-pres to retain them. Renumbering processes platform ranges in parallel across all CPU cores by default (via temporary files in the output folder); pass --threads N to cap the worker count, or --threads 1 for the sequential, lowest-memory path. Peak memory rises with the thread count, but the result is identical either way.

concat header merges YAML header files: each file contributes its top-level keys to the combined output. An error is raised if any two files share the same key.

# Merge all Parquet files with profile renumbering
ctddump concat convert /data/parquet merged.parquet

# Merge without renumbering
ctddump concat convert --no-renumber /data/parquet merged.parquet

# Merge, but keep each profile's observations in their original order (don't sort by pres)
ctddump concat convert --no-pres-sort /data/parquet merged.parquet

# Merge, keeping rows with missing pres (they are dropped by default)
ctddump concat convert --keep-na-pres /data/parquet merged.parquet

# Cap renumbering at 8 threads (defaults to all cores)
ctddump concat convert --threads 8 /data/parquet merged.parquet

# Sequential, lowest-memory merge
ctddump concat convert --threads 1 /data/parquet merged.parquet

# Merge only a subset
ctddump concat convert --pattern "AR_PR_CT_*.parquet" /data/parquet merged.parquet

# Merge YAML headers
ctddump concat header /data/yaml merged.yaml

report: summarise a Parquet or YAML file

ctddump report parquet [--level global|platform|profile] [--format tsv|text|json] <src.parquet> [dest]
ctddump report yaml    [--format tsv|text|json] <src.yaml> [dest]

Writes a text summary to dest, or to stdout when omitted. Default format is tsv.

report parquet aggregates a data file at one of three --levels (default platform): global (one row), platform (one row per platform_code), or profile (one row per profile). Each row reports profile / observation counts, per-profile "good" QC counts (time_qc/position_qc == "1"), missing-value counts, min / max / mean for temp, psal, and pres, and the geographic bounding box (longitude/latitude min/max; global and platform levels only).

report yaml summarises a merged header YAML: one row per source file with presence flags for the core columns (TEMP, PSAL, PRES, DEPH, TIME, position) and an extra_params list of the extra measurement parameters detected: biogeochemical/biological and other non-core variables (e.g. DOXY;FLU2;TUR3).

# Per-platform summary of a merged Parquet file
ctddump report parquet --level platform merged.parquet report.tsv

# Whole-file summary, human-readable, to stdout
ctddump report parquet --level global --format text merged.parquet

# YAML header summary as JSON
ctddump report yaml --format json merged.yaml report.json

filter: filter a Parquet file's profiles by a bounding box

ctddump filter [--mode include|exclude] --min-lon <W> --max-lon <E> --min-lat <S> --max-lat <N> <src.parquet> <dest.parquet>

filter keeps or drops whole profiles by a geographic bounding box (inclusive on all edges; an inverted box is rejected). --mode defaults to include (keep profiles inside the box); exclude keeps everything outside it. A profile with a NaN position counts as outside the box. The file is streamed one row group at a time, so peak memory stays bounded regardless of size.

# Keep only profiles inside a bounding box
ctddump filter --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 merged.parquet box.parquet

# Drop profiles inside that box, keep the rest
ctddump filter --mode exclude --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 merged.parquet outside.parquet

dropna: drop profiles with no usable data in a core parameter

ctddump dropna <src.parquet> <dest.parquet>

dropna keeps a profile only if each of temp, psal, and pres has at least one non-NA observation, and drops the whole profile if any one of them is entirely NA (null or NaN). Partial NAs are fine: a kept profile keeps all its rows. It runs in two streaming passes, so peak memory stays bounded regardless of file size and the output is independent of chunking.

# Drop profiles with an all-NA temp, psal, or pres
ctddump dropna merged.parquet cleaned.parquet

dropqc: drop profiles flagged bad by profile-level QC

ctddump dropqc <src.parquet> <dest.parquet>

dropqc keeps a profile only if both time_qc and position_qc are either "1" (OK) or missing (absent QC or the NA byte -128, both stored as ""). Any other flag ("0", "2"…"9", or a non-numeric code) drops the whole profile. Missing QC is kept on purpose: several source files ship no profile-level QC at all. Since these flags are constant within a profile, it is a plain per-row predicate; the file is streamed one row group at a time, so peak memory stays bounded regardless of size.

# Drop profiles flagged bad in time_qc or position_qc
ctddump dropqc merged.parquet cleaned.parquet

markdup: mark duplicate profiles

ctddump markdup [OPTIONS] <src.parquet> <dest.parquet> <dups.tsv>

markdup adds a Boolean is_dup column marking profiles that share a duplicate key with at least one other profile. The key is built from profile_timestamp (date only by default), longitude, and latitude (rounded to 3 decimals by default), across platforms. It also writes a TSV listing the duplicated profiles (dup_group, platform_code, profile_no, profile_time, profile_timestamp, longitude, latitude, n_obs). The timestamp format (--time-format), rounding precision (--decimals), and rounding mode (--round-mode = round/floor/ceil/trunc) are configurable.

ctddump markdup merged.parquet marked.parquet duplicates.tsv

dedup: remove duplicate profiles

ctddump dedup [OPTIONS] <src.parquet> <dest.parquet>

dedup re-derives the same key as markdup (pass the same options) and keeps one profile per duplicate group: the one with the most observation rows, ties broken by first appearance. Profiles with no key (NaN position / null timestamp) are always kept; an is_dup column, if present, is reset to false.

ctddump dedup marked.parquet deduped.parquet

compare: compare two Parquet files

ctddump compare [OPTIONS] <a.parquet> <b.parquet> [dest]

compare reports how far each of two files covers the other's platforms and profiles, writing one row per direction (the second file is the reference first) to dest or stdout. Profiles are matched on the same key as markdup, except that platform_code is part of the key by default; pass --no-platform-key to match on time and position alone. Coverage is reported both ways because it is not symmetric: a file contained in a larger one is fully covered while covering only part of it. For the profiles found in both files the report also says whether they carry the same number of observations (same_nobs / diff_nobs), which separates the same profiles cleaned differently from genuinely different profiles. The key column names (--platform-col, --time-col, --lon-col, --lat-col), the time format, the decimals, and the rounding mode are configurable, and the time column may be a datetime or a float of days since 1950. Output is --format tsv (default), text, or json.

ctddump compare --format text old.parquet new.parquet

Configuration

All convert and batch convert subcommands support a --config TOML file plus individual flag overrides. Priority order:

built-in default  <  --config file  <  individual CLI flags

NRT flags

CORA flags

TOML config file format

# NRT
has_deph_source    = true
has_profile_coords = false
pattern            = "AR_PR_CT_*.nc"  # optional

# CORA
time_var      = "TIME"
qc_type       = "int"    # "int" or "char"
has_time_qc   = true
has_deph_source = true
pattern       = "*.nc"   # optional

Output schema

All converters produce a uniform, observation-level flat table:

Development

# Run all tests
cargo test

# Run a single test
cargo test test_convert_nrt_ar_1

# Download test fixture files (requires gh CLI, authenticated)
scripts/fetch_test_data.sh

Note: HDF5-DIAG messages may appear in test output on systems with HDF5 ≤ 1.10. They are harmless: the data is read correctly and all tests pass.

Releases are versioned, changelogged, and published following RELEASING.md.

License

Licensed under the MIT License.