ctddump

A Rust CLI tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF to Parquet (data) or YAML (metadata).

Links:

• 📖 Documentation
• 🌊 Live example report site: sample output of the summary_site.sh pipeline phase

Two data sources are supported:

Installation

System dependencies

The netcdf crate links against the HDF5 C library:

# Ubuntu / Debian
sudo apt-get install libhdf5-dev libnetcdf-dev

# macOS (Homebrew)
brew install hdf5

Build

git clone https://github.com/AIQC-Hub/ctddump.git
cd ctddump
cargo build --release

The binary is placed at target/release/ctddump.

Commands

Every command and subcommand supports -h / --help.

convert: single NetCDF → Parquet

ctddump convert <subcommand> [OPTIONS] <src_file> <target_file>

ctddump convert nrt_ar   input.nc output.parquet
ctddump convert nrt_bo   input.nc output.parquet
ctddump convert nrt_mo   input.nc output.parquet
ctddump convert nrt_gl   input.nc output.parquet
ctddump convert cora     input.nc output.parquet
ctddump convert cora_legacy input.nc output.parquet

# Use a saved config preset
ctddump convert nrt_ar --config my_preset.toml input.nc output.parquet

# Override individual fields
ctddump convert nrt_bo --no-deph-source input.nc output.parquet

batch: directory tree NetCDF → Parquet or YAML (multi-threaded)

ctddump batch convert <subcommand> [OPTIONS] <src_dir>
ctddump batch header  <subcommand> [OPTIONS] <src_dir>

Recursively finds all matching .nc files and processes them in parallel. Output files keep the source stem with a new extension. If --output is omitted, each output is written alongside its source.

# Convert all NRT Arctic files to Parquet, flat into /output
ctddump batch convert nrt_ar --output /output /data/arctic

# Limit to 4 threads
ctddump batch convert nrt_ar --threads 4 --output /output /data/arctic

# Override the filename pattern
ctddump batch convert nrt_ar --pattern "AR_PR_CT_ITP-*.nc" --output /output /data

# Extract YAML metadata for all NRT files
ctddump batch header nrt --output /output /data/arctic

Default filename patterns:

header: single NetCDF → YAML metadata

ctddump header <subcommand> <src_file> <target_file>

ctddump header nrt  input.nc output.yaml
ctddump header cora input.nc output.yaml

concat: merge files from a directory tree into a single file

ctddump concat convert [OPTIONS] <src_dir> <output_file>
ctddump concat header  [OPTIONS] <src_dir> <output_file>

concat convert merges Parquet files and re-assigns profile_no and observation_no by default (pass --no-renumber to skip). Renumbering sorts rows by platform_code, profile_timestamp, longitude, latitude, pres; pass --no-pres-sort to sort without pres, keeping each profile's observations in their original source order instead of reordering them by pressure. Rows with missing (null/NaN) pres are dropped before merging by default (this keeps observation_no contiguous over the remaining rows); pass --keep-na-pres to retain them. Renumbering processes platform ranges in parallel across all CPU cores by default (via temporary files in the output folder); pass --threads N to cap the worker count, or --threads 1 for the sequential, lowest-memory path. Peak memory rises with the thread count, but the result is identical either way.

concat header merges YAML header files: each file contributes its top-level keys to the combined output. An error is raised if any two files share the same key.

# Merge all Parquet files with profile renumbering
ctddump concat convert /data/parquet merged.parquet

# Merge without renumbering
ctddump concat convert --no-renumber /data/parquet merged.parquet

# Merge, but keep each profile's observations in their original order (don't sort by pres)
ctddump concat convert --no-pres-sort /data/parquet merged.parquet

# Merge, keeping rows with missing pres (they are dropped by default)
ctddump concat convert --keep-na-pres /data/parquet merged.parquet

# Cap renumbering at 8 threads (defaults to all cores)
ctddump concat convert --threads 8 /data/parquet merged.parquet

# Sequential, lowest-memory merge
ctddump concat convert --threads 1 /data/parquet merged.parquet

# Merge only a subset
ctddump concat convert --pattern "AR_PR_CT_*.parquet" /data/parquet merged.parquet

# Merge YAML headers
ctddump concat header /data/yaml merged.yaml

report: summarise a Parquet or YAML file

ctddump report parquet [--level global|platform|profile] [--format tsv|text|json] <src.parquet> [dest]
ctddump report yaml    [--format tsv|text|json] <src.yaml> [dest]

Writes a text summary to dest, or to stdout when omitted. Default format is tsv.

report parquet aggregates a data file at one of three --levels (default platform): global (one row), platform (one row per platform_code), or profile (one row per profile). Each row reports profile / observation counts, per-profile "good" QC counts (time_qc/position_qc == "1"), missing-value counts, min / max / mean for temp, psal, and pres, and the geographic bounding box (longitude/latitude min/max; global and platform levels only).

report yaml summarises a merged header YAML: one row per source file with presence flags for the core columns (TEMP, PSAL, PRES, DEPH, TIME, position) and an extra_params list of the extra measurement parameters detected: biogeochemical/biological and other non-core variables (e.g. DOXY;FLU2;TUR3).

# Per-platform summary of a merged Parquet file
ctddump report parquet --level platform merged.parquet report.tsv

# Whole-file summary, human-readable, to stdout
ctddump report parquet --level global --format text merged.parquet

# YAML header summary as JSON
ctddump report yaml --format json merged.yaml report.json

filter: filter a Parquet file's profiles by a bounding box

ctddump filter [--mode include|exclude] --min-lon <W> --max-lon <E> --min-lat <S> --max-lat <N> <src.parquet> <dest.parquet>

filter keeps or drops whole profiles by a geographic bounding box (inclusive on all edges; an inverted box is rejected). --mode defaults to include (keep profiles inside the box); exclude keeps everything outside it. A profile with a NaN position counts as outside the box. The file is streamed one row group at a time, so peak memory stays bounded regardless of size.

# Keep only profiles inside a bounding box
ctddump filter --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 merged.parquet box.parquet

# Drop profiles inside that box, keep the rest
ctddump filter --mode exclude --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 merged.parquet outside.parquet

dropna: drop profiles with no usable data in a core parameter

ctddump dropna <src.parquet> <dest.parquet>

dropna keeps a profile only if each of temp, psal, and pres has at least one non-NA observation, and drops the whole profile if any one of them is entirely NA (null or NaN). Partial NAs are fine: a kept profile keeps all its rows. It runs in two streaming passes, so peak memory stays bounded regardless of file size and the output is independent of chunking.

# Drop profiles with an all-NA temp, psal, or pres
ctddump dropna merged.parquet cleaned.parquet

dropqc: drop profiles flagged bad by profile-level QC

ctddump dropqc <src.parquet> <dest.parquet>

dropqc keeps a profile only if both time_qc and position_qc are either "1" (OK) or missing (absent QC or the NA byte -128, both stored as ""). Any other flag ("0", "2"…"9", or a non-numeric code) drops the whole profile. Missing QC is kept on purpose: several source files ship no profile-level QC at all. Since these flags are constant within a profile, it is a plain per-row predicate; the file is streamed one row group at a time, so peak memory stays bounded regardless of size.

# Drop profiles flagged bad in time_qc or position_qc
ctddump dropqc merged.parquet cleaned.parquet

markdup: mark duplicate profiles

ctddump markdup [OPTIONS] <src.parquet> <dest.parquet> <dups.tsv>

markdup adds a Boolean is_dup column marking profiles that share a duplicate key with at least one other profile. The key is built from profile_timestamp (date only by default), longitude, and latitude (rounded to 3 decimals by default), across platforms. It also writes a TSV listing the duplicated profiles (dup_group, platform_code, profile_no, profile_time, profile_timestamp, longitude, latitude, n_obs). The timestamp format (--time-format), rounding precision (--decimals), and rounding mode (--round-mode = round/floor/ceil/trunc) are configurable.

ctddump markdup merged.parquet marked.parquet duplicates.tsv

dedup: remove duplicate profiles

ctddump dedup [OPTIONS] <src.parquet> <dest.parquet>

dedup re-derives the same key as markdup (pass the same options) and keeps one profile per duplicate group: the one with the most observation rows, ties broken by first appearance. Profiles with no key (NaN position / null timestamp) are always kept; an is_dup column, if present, is reset to false.

ctddump dedup marked.parquet deduped.parquet

Configuration

All convert and batch convert subcommands support a --config TOML file plus individual flag overrides. Priority order:

built-in default  <  --config file  <  individual CLI flags

NRT flags

CORA flags

TOML config file format

# NRT
has_deph_source    = true
has_profile_coords = false
pattern            = "AR_PR_CT_*.nc"  # optional

# CORA
time_var      = "TIME"
qc_type       = "int"    # "int" or "char"
has_time_qc   = true
has_deph_source = true
pattern       = "*.nc"   # optional

Output schema

All converters produce a uniform, observation-level flat table:

Development

# Run all tests
cargo test

# Run a single test
cargo test test_convert_nrt_ar_1

# Download test fixture files (requires gh CLI, authenticated)
scripts/fetch_test_data.sh

Note: HDF5-DIAG messages may appear in test output on systems with HDF5 ≤ 1.10. They are harmless: the data is read correctly and all tests pass.

Releases are versioned, changelogged, and published following RELEASING.md.

License

Licensed under the MIT License.