# ctddump
A Rust CLI tool for converting oceanographic CTD (Conductivity, Temperature, Depth) data from NetCDF to Parquet (data) or YAML (metadata).
**Links:**
- 📖 [Documentation](https://aiqc-hub.github.io/ctddump/)
- 🌊 [Live example report site](https://aiqc-hub.github.io/ctddump-report-example/): sample output of the `summary_site.sh` pipeline phase
Two data sources are supported:
| Source | Description |
|--------|-------------|
| **NRT** | Near Real Time: Arctic Sea (`nrt_ar`), Baltic Sea (`nrt_bo`), Mediterranean Sea (`nrt_mo`), Global (`nrt_gl`) |
| **CORA** | Copernicus Ocean Reanalysis: current format (`cora`), legacy format (`cora_legacy`) |
## Installation
### System dependencies
The `netcdf` crate links against the HDF5 C library:
```bash
# Ubuntu / Debian
sudo apt-get install libhdf5-dev libnetcdf-dev
# macOS (Homebrew)
brew install hdf5
```
### Build
```bash
git clone https://github.com/AIQC-Hub/ctddump.git
cd ctddump
cargo build --release
```
The binary is placed at `target/release/ctddump`.
## Commands
Every command and subcommand supports `-h` / `--help`.
### `convert`: single NetCDF → Parquet
```
ctddump convert <subcommand> [OPTIONS] <src_file> <target_file>
```
```bash
ctddump convert nrt_ar input.nc output.parquet
ctddump convert nrt_bo input.nc output.parquet
ctddump convert nrt_mo input.nc output.parquet
ctddump convert nrt_gl input.nc output.parquet
ctddump convert cora input.nc output.parquet
ctddump convert cora_legacy input.nc output.parquet
# Use a saved config preset
ctddump convert nrt_ar --config my_preset.toml input.nc output.parquet
# Override individual fields
ctddump convert nrt_bo --no-deph-source input.nc output.parquet
```
### `batch`: directory tree NetCDF → Parquet or YAML (multi-threaded)
```
ctddump batch convert <subcommand> [OPTIONS] <src_dir>
ctddump batch header <subcommand> [OPTIONS] <src_dir>
```
Recursively finds all matching `.nc` files and processes them in parallel. Output files keep the source stem with a new extension. If `--output` is omitted, each output is written alongside its source.
```bash
# Convert all NRT Arctic files to Parquet, flat into /output
ctddump batch convert nrt_ar --output /output /data/arctic
# Limit to 4 threads
ctddump batch convert nrt_ar --threads 4 --output /output /data/arctic
# Override the filename pattern
ctddump batch convert nrt_ar --pattern "AR_PR_CT_ITP-*.nc" --output /output /data
# Extract YAML metadata for all NRT files
ctddump batch header nrt --output /output /data/arctic
```
Default filename patterns:
| Subcommand | Pattern |
|------------|---------|
| `nrt_ar` | `AR_PR_CT_*.nc` |
| `nrt_bo` | `BO_PR_CT_*.nc` |
| `nrt_mo` | `MO_PR_CT_*.nc` |
| `nrt_gl` | `GL_PR_CT_*.nc` |
| `cora`, `cora_legacy` | `*.nc` |
| `batch header nrt`, `batch header cora` | `*.nc` |
### `header`: single NetCDF → YAML metadata
```
ctddump header <subcommand> <src_file> <target_file>
```
```bash
ctddump header nrt input.nc output.yaml
ctddump header cora input.nc output.yaml
```
### `concat`: merge files from a directory tree into a single file
```
ctddump concat convert [OPTIONS] <src_dir> <output_file>
ctddump concat header [OPTIONS] <src_dir> <output_file>
```
`concat convert` merges Parquet files and re-assigns `profile_no` and `observation_no` by default (pass `--no-renumber` to skip). Renumbering sorts rows by `platform_code, profile_timestamp, longitude, latitude, pres`; pass `--no-pres-sort` to sort without `pres`, keeping each profile's observations in their original source order instead of reordering them by pressure. Rows with missing (null/NaN) `pres` are dropped before merging by default (this keeps `observation_no` contiguous over the remaining rows); pass `--keep-na-pres` to retain them. Renumbering processes platform ranges in parallel across all CPU cores by default (via temporary files in the output folder); pass `--threads N` to cap the worker count, or `--threads 1` for the sequential, lowest-memory path. Peak memory rises with the thread count, but the result is identical either way.
`concat header` merges YAML header files: each file contributes its top-level keys to the combined output. An error is raised if any two files share the same key.
```bash
# Merge all Parquet files with profile renumbering
ctddump concat convert /data/parquet merged.parquet
# Merge without renumbering
ctddump concat convert --no-renumber /data/parquet merged.parquet
# Merge, but keep each profile's observations in their original order (don't sort by pres)
ctddump concat convert --no-pres-sort /data/parquet merged.parquet
# Merge, keeping rows with missing pres (they are dropped by default)
ctddump concat convert --keep-na-pres /data/parquet merged.parquet
# Cap renumbering at 8 threads (defaults to all cores)
ctddump concat convert --threads 8 /data/parquet merged.parquet
# Sequential, lowest-memory merge
ctddump concat convert --threads 1 /data/parquet merged.parquet
# Merge only a subset
ctddump concat convert --pattern "AR_PR_CT_*.parquet" /data/parquet merged.parquet
# Merge YAML headers
ctddump concat header /data/yaml merged.yaml
```
### `report`: summarise a Parquet or YAML file
```
ctddump report parquet [--level global|platform|profile] [--format tsv|text|json] <src.parquet> [dest]
ctddump report yaml [--format tsv|text|json] <src.yaml> [dest]
```
Writes a text summary to `dest`, or to stdout when omitted. Default format is `tsv`.
`report parquet` aggregates a data file at one of three `--level`s (default `platform`): `global` (one row), `platform` (one row per `platform_code`), or `profile` (one row per profile). Each row reports profile / observation counts, per-profile "good" QC counts (`time_qc`/`position_qc == "1"`), missing-value counts, min / max / mean for `temp`, `psal`, and `pres`, and the geographic bounding box (`longitude`/`latitude` min/max; `global` and `platform` levels only).
`report yaml` summarises a merged header YAML: one row per source file with presence flags for the core columns (`TEMP`, `PSAL`, `PRES`, `DEPH`, `TIME`, position) and an `extra_params` list of the extra measurement parameters detected: biogeochemical/biological and other non-core variables (e.g. `DOXY;FLU2;TUR3`).
```bash
# Per-platform summary of a merged Parquet file
ctddump report parquet --level platform merged.parquet report.tsv
# Whole-file summary, human-readable, to stdout
ctddump report parquet --level global --format text merged.parquet
# YAML header summary as JSON
ctddump report yaml --format json merged.yaml report.json
```
### `filter`: filter a Parquet file's profiles by a bounding box
```
ctddump filter [--mode include|exclude] --min-lon <W> --max-lon <E> --min-lat <S> --max-lat <N> <src.parquet> <dest.parquet>
```
`filter` keeps or drops **whole profiles** by a geographic bounding box (inclusive on all edges; an inverted box is rejected). `--mode` defaults to `include` (keep profiles inside the box); `exclude` keeps everything outside it. A profile with a NaN position counts as outside the box. The file is streamed one row group at a time, so peak memory stays bounded regardless of size.
```bash
# Keep only profiles inside a bounding box
ctddump filter --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 merged.parquet box.parquet
# Drop profiles inside that box, keep the rest
ctddump filter --mode exclude --min-lon 5 --max-lon 15 --min-lat 35 --max-lat 40 merged.parquet outside.parquet
```
### `dropna`: drop profiles with no usable data in a core parameter
```
ctddump dropna <src.parquet> <dest.parquet>
```
`dropna` keeps a profile only if **each** of `temp`, `psal`, and `pres` has at least one non-NA observation, and drops the whole profile if **any one** of them is entirely NA (null or NaN). Partial NAs are fine: a kept profile keeps all its rows. It runs in two streaming passes, so peak memory stays bounded regardless of file size and the output is independent of chunking.
```bash
# Drop profiles with an all-NA temp, psal, or pres
ctddump dropna merged.parquet cleaned.parquet
```
### `dropqc`: drop profiles flagged bad by profile-level QC
```
ctddump dropqc <src.parquet> <dest.parquet>
```
`dropqc` keeps a profile only if **both** `time_qc` and `position_qc` are either `"1"` (OK) or **missing** (absent QC or the NA byte `-128`, both stored as `""`). Any other flag (`"0"`, `"2"`…`"9"`, or a non-numeric code) drops the whole profile. Missing QC is kept on purpose: several source files ship no profile-level QC at all. Since these flags are constant within a profile, it is a plain per-row predicate; the file is streamed one row group at a time, so peak memory stays bounded regardless of size.
```bash
# Drop profiles flagged bad in time_qc or position_qc
ctddump dropqc merged.parquet cleaned.parquet
```
### `markdup`: mark duplicate profiles
```
ctddump markdup [OPTIONS] <src.parquet> <dest.parquet> <dups.tsv>
```
`markdup` adds a Boolean `is_dup` column marking profiles that share a duplicate key with at least one other profile. The key is built from `profile_timestamp` (date only by default), `longitude`, and `latitude` (rounded to 3 decimals by default), **across platforms**. It also writes a TSV listing the duplicated profiles (`dup_group`, `platform_code`, `profile_no`, `profile_time`, `profile_timestamp`, `longitude`, `latitude`, `n_obs`). The timestamp format (`--time-format`), rounding precision (`--decimals`), and rounding mode (`--round-mode` = `round`/`floor`/`ceil`/`trunc`) are configurable.
```bash
ctddump markdup merged.parquet marked.parquet duplicates.tsv
```
### `dedup`: remove duplicate profiles
```
ctddump dedup [OPTIONS] <src.parquet> <dest.parquet>
```
`dedup` re-derives the same key as `markdup` (pass the same options) and keeps one profile per duplicate group: the one with the most observation rows, ties broken by first appearance. Profiles with no key (NaN position / null timestamp) are always kept; an `is_dup` column, if present, is reset to `false`.
```bash
ctddump dedup marked.parquet deduped.parquet
```
## Configuration
All `convert` and `batch convert` subcommands support a `--config` TOML file plus individual flag overrides. Priority order:
```
built-in default < --config file < individual CLI flags
```
### NRT flags
| Flag | Field | Default |
|------|-------|---------|
| `--deph-source` / `--no-deph-source` | `has_deph_source` | `true` for BO/GL, `false` for AR/MO |
| `--profile-coords` / `--no-profile-coords` | `has_profile_coords` | `true` for BO, `false` otherwise |
| `--pattern <GLOB>` | `pattern` | see table above |
### CORA flags
| Flag | Field | `cora` default | `cora_legacy` default |
|------|-------|---------------|----------------------|
| `--time-var <VAR>` | `time_var` | `TIME` | `JULD` |
| `--qc-type <int\|char>` | `qc_type` | `int` | `char` |
| `--time-qc` / `--no-time-qc` | `has_time_qc` | `true` | `false` |
| `--deph-source` / `--no-deph-source` | `has_deph_source` | `true` | `false` |
| `--pattern <GLOB>` | `pattern` | `*.nc` | `*.nc` |
### TOML config file format
```toml
# NRT
has_deph_source = true
has_profile_coords = false
pattern = "AR_PR_CT_*.nc" # optional
# CORA
time_var = "TIME"
qc_type = "int" # "int" or "char"
has_time_qc = true
has_deph_source = true
pattern = "*.nc" # optional
```
## Output schema
All converters produce a uniform, observation-level flat table:
| Column | Type | Notes |
|--------|------|-------|
| `platform_code` | `String` | |
| `profile_no` | `u32` | |
| `profile_time` | `f64` | days since 1950-01-01 |
| `profile_timestamp` | `Datetime(ms)` | Unix milliseconds |
| `observation_no` | `u32` | |
| `longitude` / `latitude` | `f32` (NRT) / `f64` (CORA) | |
| `profile_longitude` / `profile_latitude` | `f32` (NRT) / `f64` (CORA) | from `PRECISE_*` or expanded `DEPLOY_*`; NaN when unavailable |
| `time_qc` / `position_qc` | `String` | `""` if absent |
| `filename` | `String` | source file stem |
| `temp`, `psal`, `pres`, `deph` | `f32` | NaN where missing |
| `temp_qc`, `psal_qc`, `pres_qc`, `deph_qc` | `String` | single-char flag; `""` if missing |
| `pres_conv`, `deph_conv` | `i8` | `1` = derived by conversion |
## Development
```bash
# Run all tests
cargo test
# Run a single test
cargo test test_convert_nrt_ar_1
# Download test fixture files (requires gh CLI, authenticated)
scripts/fetch_test_data.sh
```
> **Note:** HDF5-DIAG messages may appear in test output on systems with HDF5 ≤ 1.10. They are harmless: the data is read correctly and all tests pass.
Releases are versioned, changelogged, and published following [RELEASING.md](RELEASING.md).
## License
Licensed under the [MIT License](LICENSE).