pigeon

pigeon, a Rust CLI that imports, transcodes, and deduplicates personal data into S3-compatible remotes. It provides a job runner and a keyring for batch operations over data sets, with support for checkpointed resume, concurrent operations, and rate-limited transfers.

Structure

• src/ — the Rust CLI source.
• docs/adr/ — architecture decision records governing every change in this repo.
• docs/report/ — job-run log analysis reports produced by the analyze-job-run Claude Code skill.

Getting started

This repo uses mise as the single entry point for all Rust tooling:

mise run build              # build the pigeon binary
mise run pigeon -- <args>   # run it, e.g. `mise run pigeon -- keyring list`
mise run test                # run the test suite
mise run fmt                 # format
mise run fmt-check           # check formatting
mise run lint                 # clippy, warnings denied
mise run ci                   # the full local gate (fmt-check + lint + test)

Install (published crate)

cargo install pigeon-cli

This installs a binary named pigeon.

Run via Docker

An alternative to installing Rust/ffmpeg locally: build and run pigeon in a container that bundles everything it needs (see ADR-0087). Image targets linux/arm64 only.

mise run docker-build                       # docker build --platform linux/arm64 -t pigeon-cli .
mise run docker-run -- keyring list         # docker run ... pigeon-cli keyring list

/data inside the container is the single mount point for config, logs, and job output — docker-run's named volume (pigeon-data) persists it across runs. PIGEON_CONFIG_DIR and PIGEON_LOG_DIR are pre-set to /data/config//data/logs.

Because a container restart doesn't preserve the OS keyring (see ADR-0085's Linux caveat), non-interactive/restarted use works in two phases:

1. One-time setup: run pigeon keyring add <kind> <alias> once to populate keyring.toml's non-secret metadata — it persists in the /data volume.

2. Per-run: supply the actual secret via a PIGEON_SECRET_<ALIAS> environment variable (alias uppercased, non-alphanumeric characters replaced with _) — pigeon reads it directly instead of the OS keyring, e.g.:

   docker run --platform linux/arm64 --rm -v pigeon-data:/data \
     -e PIGEON_SECRET_MY_ALIAS=<secret> \
     pigeon-cli job run email import
   

How that env var gets populated (a secrets manager, CI variable, etc.) is left to your own infrastructure.

Local observability stack (Grafana/Prometheus/Loki)

Previously, pigeon job runs on Scaleway compute were observed via Scaleway's managed Cockpit (see ADR-0092/0093), fed by an on-host Grafana Alloy agent. Running pigeon bare-metal on your own machine instead (see ADR-0128) drops that path entirely — nothing ships /metrics or pigeon.jsonl anywhere by default. observability/ provides an equivalent local stack via Docker Compose.

mise run observability-up      # starts Prometheus, Loki, Promtail, Grafana

Then run pigeon commands normally, bare-metal, alongside the stack — e.g. pigeon job run transform .... Open http://localhost:3000 for Grafana (default credentials admin/admin on first login — change them) and open the pre-provisioned "pigeon-cli overview" dashboard.

mise run observability-down stops the stack; Prometheus/Grafana data persists in named Docker volumes across restarts. mise run observability-logs tails the stack's own container logs (not pigeon's), for debugging the stack itself.

Known v1 limitation: no host-wide metrics (CPU/mem/disk for the whole machine, not just pigeon's own process tree) — see ADR-0128.

Commands

• pigeon keyring add [email|bucket] — authenticate an email identity, or configure an S3-compatible bucket.
• pigeon keyring modify [alias] — edit an existing entry's fields.
• pigeon keyring delete <alias> — remove a configured entry and its secret.
• pigeon keyring list — list every configured entry.
• pigeon job run email import [flags] — deliver each message's raw .eml and unpacked attachments from an authenticated identity to a destination, at a chosen --messages-per-minute pace.
• pigeon job run deduplicate [flags] — inflate every zip, content-hash a bucket, keep one copy of each byte-identical file, and upload the result.
• pigeon job run transform [flags] — transcode images, video, or audio-only recordings between a source and destination bucket via ffmpeg.
• pigeon job run rclone copy [flags] — copy one configured bucket to another via rclone.
• pigeon job run rclone delete [flags] — recursively and permanently purge a configured bucket's contents.

Run pigeon --help, pigeon keyring --help, or pigeon job --help for the full command reference.

The full design rationale for every decision behind this crate lives in docs/adr/, as a sequence of architecture decision records.

License

Licensed under the GNU General Public License v3.0 or later — see LICENSE.md.