๐ช parachuter
Reliable, rate-limited file downlink over lossy unidirectional UDP links.
parachuter is a Rust crate and command-line tool inspired by, and first
deployed on, the SuperBIT balloon telescope. It chunks arbitrary files into
CRC-protected UDP datagrams, ships them at a configurable rate over a named
link, reassembles them durably on the ground, and provides a live control
socket to retune every parameter, including chunk size, destination IP, and
throughput cap, without restarting any process.
Everything ships as a single binary, parachuter, with mode subcommands.
A typical ground deployment runs parachuter receiver and parachuter cleaner as two systemd units pointing at the same binary; a flight
deployment runs parachuter sender as its own unit.
Table of Contents
- Overview
- Architecture
- Subcommands
- Quick Start
- How Files Enter the Sender
- Configuration Reference
- Runtime Control (
parachuter ctl) - Named Links
- Dedup and Re-request Logic
- Recovering Files Lost Whole
- Building
- Testing
- Deployment (systemd)
- Design Notes
Overview ๐ช
โโโโโโโโโโโโโโโโโโโโโโโโ UDP datagrams โโโโโโโโโโโโโโโโโโโโโโโโ
โ parachuter sender โ โโโโโโโโโโโโโโถ โ parachuter receiver โ
โ (payload / SBC) โ lossy one-way โ (ground station) โ
โโโโโโโโโโฌโโโโโโโโโโโโโโ โโโโโโโโโโฌโโโโโโโโโโโโโโ
โ Unix socket โ Unix socket
โโโโโโโโโโผโโโโโโโโโโโโโโ โโโโโโโโโโผโโโโโโโโโโโโโโ
โ parachuter ctl โ โ parachuter cleaner โ
โ (operator CLI) โ โ (gap filler / retx) โ
โโโโโโโโโโโโโโโโโโโโโโโโ โโโโโโโโโโโโโโโโโโโโโโโโ
The sender chunks files (from a watched priority queue or from explicit external submissions), ships every chunk over UDP, and waits. The receiver writes each chunk directly to disk using a CRC-protected bitmap manifest sidecar. When chunks are missing, the cleaner asks the sender to retransmit exactly the missing ranges, without re-sending anything that is already in flight.
Architecture
The whole project is one crate, parachuter. It ships both a library
(protocol, chunker, reassembler, ledger, config, control plane) and a
single binary, also called parachuter, whose modes are:
| Mode | Role |
|---|---|
parachuter sender |
Reads priority dirs, chunks files, ships UDP datagrams, accepts external submissions over its control socket |
parachuter receiver |
Listens for UDP, writes chunks to disk, finalises files |
parachuter cleaner |
Detects missing chunks, requests targeted retransmits |
parachuter ctl |
Pokes a running daemon over its Unix socket (status, set-link, submit, enqueue, flush, โฆ) |
parachuter monitor |
Live TTY table of in-progress reassemblies, polled from the receiver |
Subcommands
parachuter sender
Runs on the payload (flight computer / SBC). Watches the priority directories for new files and accepts file submissions over its Unix socket. Registers files in a SQLite ledger and ships them chunk by chunk. Config is hot-reloaded: edit the TOML and changes apply within ~1 second.
parachuter receiver
Runs on the ground station. Listens on a UDP port, validates every datagram (magic + CRC), writes payload bytes at the correct file offset, and moves completed files into the final directory. Also exposes a Unix socket for status.
parachuter cleaner
Runs on the ground station alongside the receiver. Periodically scans the
receiver's holding directory for incomplete assemblies, computes missing
chunk ranges, then asks the sender to retransmit only those ranges. Three
dedup layers prevent flooding the link. Can be triggered on demand with
parachuter ctl ... cleaner-run.
# or one-shot (cron / systemd timer):
parachuter ctl
Talk to any running daemon from the command line.
parachuter monitor
Live terminal table polled from the receiver socket every second.
Quick Start ๐ช
# 1. Build the single binary
# Binary lives at target/release/parachuter
# 2. Create directories
# 3. Install the sample config
# Edit [links.pilot] ip = "<ground-station-IP>"
# 4. Start the sender (on the payload)
# 5. Start the receiver (on the ground station)
# 6. Start the cleaner (on the ground station)
# 7. Hand the sender a file (either path works):
# 8. Watch it arrive ๐ช
How Files Enter the Sender
The sender accepts files from two completely independent ingest paths. Both end up in the same SQLite ledger and the same in-memory queue, so downstream code does not care which one a file came in through. Use whichever fits your producer.
1. Priority-directory scan (drop a file, walk away)
When the queue is empty, the sender periodically walks the directories
listed in [sender] priority_dirs in order, picks the oldest unsent file
from the first directory that has one, registers it in the ledger, and
queues it.
[]
= [
"/data/parachuter/queue/science",
"/data/parachuter/queue/dark",
"/data/parachuter/queue/flat",
]
# Walked instead when the sender is in `debug` state. Empty means no
# auto-ingest in debug.
= [
"/data/parachuter/queue/debug",
]
# Optional extension filters. See "Configuration Reference" for the full
# precedence rules โ short version: skip wins, both empty = allow all.
= ["fits", "fits.bz2"]
= ["log", "tmp"]
Anything that produces a regular file under one of those directories
triggers ingest: cp, rsync, a shell redirect, a Python script that
calls pathlib.Path.write_bytes, a C program that closes a write(),
a camera driver writing FITS, anything. The sender does not need to know
who put the file there. There is no IPC and no library binding to the
producer โ the contract is "a regular file appears under a watched path."
A .bz2 or .gz file is skipped while a sibling without the suffix
exists, so partially-written compressed files are not picked up
mid-compression.
The auto-scanner uses priority_dirs in auto state and
priority_dirs_debug in debug state. The two lists are completely
independent, so flight-day testing won't accidentally drain the live
science queue. Switch with parachuter ctl โฆ set-state debug /
set-state auto.
2. submit over the control socket (push by absolute path)
Any process can hand the sender a file by absolute path over its Unix
domain socket. The path does not need to live under a watched directory.
The reply is a JSON blob with the assigned file_id, total chunk count,
and file size, so the caller can correlate later progress reports.
The easiest way to call it is the bundled CLI wrapper:
Reply (printed to stdout as JSON):
Pass --interrupt to push the file to the head of the queue instead of
the back.
Driving it from Python
A producer that writes a file and then submits it:
=
=
return
=
Driving it from the raw socket
If you do not want a parachuter binary on the producer host, the wire
format is plain JSON over a Unix domain socket. Frame: 4-byte big-endian
length, then the JSON bytes; the same framing applies to the response.
=
=
=
This is the same path parachuter ctl submit takes internally, so any
language that can speak Unix sockets and JSON can act as a producer.
Choosing between them
Watched directories are the path of least resistance for batch producers
that already write files to disk (image pipelines, dump scripts, syncs).
The submit socket is right when you want immediate, ordered enqueueing,
need the file_id back synchronously, are submitting from a directory you
do not want the sender to scan, or want --interrupt semantics.
Note that priority_dirs_debug, include_extensions, and
skip_extensions only affect the auto-scanner. submit always honours
the path the caller hands it, regardless of state and regardless of
extension filters โ submission is an explicit instruction.
Configuration Reference
The single TOML file at /etc/parachuter/config.toml is shared by every
mode and is hot-reloaded by long-running daemons when it changes on disk.
Any value can also be overridden at runtime via parachuter ctl without
touching the file.
[links.<name>]
Define one or more named links. Every mode refers to links by name.
| Key | Type | Description |
|---|---|---|
ip |
string | Destination IP address (unicast or multicast). |
port |
u16 | Destination UDP port. |
max_kbps |
u64 | Sustained throughput cap in kilobits/second. |
note |
string (optional) | Human-readable label. |
[]
= "192.168.98.24"
= 41410
= 1000
= "Pilot (tethered) link"
[]
= "239.255.0.2"
= 41410
= 30
= "TDRSS satellite relay (multicast)"
[sender]
| Key | Default | Description |
|---|---|---|
bind_ip |
"0.0.0.0" |
Source IP for outbound datagrams. |
bind_port |
34646 |
Source port. |
chunk_size |
16192 |
Datagram size including the 32-byte header. Must be in [64, 65000]. |
active_link |
"pilot" |
Which [links.*] to send on. Change at runtime with set-link. |
ledger_path |
/var/lib/parachuter/ledger.sqlite |
SQLite ledger. |
priority_dirs |
(none) | Ordered list of directories the auto-scanner walks in auto state; earlier = higher priority. |
priority_dirs_debug |
[] |
Ordered list walked instead of priority_dirs when the sender is in debug state. Empty means no auto-ingest in debug. |
include_extensions |
[] |
Whitelist of file extensions the auto-scanner is allowed to pick up. See Extension filtering below. |
skip_extensions |
[] |
Blacklist of file extensions the auto-scanner must skip. Wins over include_extensions on clashes. |
control_socket |
/run/parachuter/sender.sock |
Unix socket path. |
initial_state |
"auto" |
One of auto, manual, paused, debug. |
recheck_period_secs |
100 |
How often to scan priority dirs when queue is empty. |
status_dump_period_secs |
21600 |
How often to write a ledger CSV snapshot into status_dumps/ under the first priority directory. It is downlinked like any other file and drives reconciliation. |
Auto vs. debug priority lists
auto state walks priority_dirs; debug state walks priority_dirs_debug.
This lets you carry a separate "things I want to test on the ground today"
queue without disturbing the live science queue. Switch at runtime:
If priority_dirs_debug is empty in debug state, auto-ingest just stops;
explicit submit / enqueue still work, which is usually what you want
for debug sessions where you're hand-feeding files.
Extension filtering
Both include_extensions and skip_extensions apply only to the
auto-scanner. Entries are matched case-insensitively against the file's
tail and may include or omit a leading dot. Compound extensions like
"fits.bz2" are supported and only match the full compound (so
"fits" does not match foo.fits.bz2 โ list both if you want
both).
Precedence:
skip_extensions |
include_extensions |
Result |
|---|---|---|
| empty | empty | Downlink every file the scanner finds. |
| non-empty | empty | Downlink everything except the skip list. |
| empty | non-empty | Downlink only the include list. |
| non-empty | non-empty | Downlink (include โ skip): a file must be in include and not in skip. |
Skip always wins on clashes, so listing the same extension in both
effectively just blacklists it. Explicit parachuter ctl submit --path โฆ
ignores both lists โ if you point at a file by hand, the sender takes
your word for it.
[receiver]
| Key | Default | Description |
|---|---|---|
bind_ip |
"0.0.0.0" |
IP to bind the receive socket. |
bind_port |
41410 |
UDP port to listen on. |
holding_dir |
/var/lib/parachuter/holding |
In-flight assemblies live here. |
final_dir |
/var/lib/parachuter/downloads |
Completed files move here, at final_dir + their full payload path (/data/qsc/m31.fits.bz2 โ <final_dir>/data/qsc/m31.fits.bz2). |
control_socket |
/run/parachuter/receiver.sock |
Unix socket path. |
[cleaner]
| Key | Default | Description |
|---|---|---|
holding_dir |
same as receiver | Must match receiver's holding_dir. |
final_dir |
same as receiver | Must match receiver's final_dir. |
active_link |
"pilot" |
Which link budget to apply. |
run_period_secs |
60 |
How often the cleaner wakes to scan. Can be triggered immediately with cleaner-run. |
reconcile_period_secs |
21600 |
How often to reconcile the newest downlinked ledger snapshot against final_dir (6 h). The old name checker_period_secs is still accepted. |
reconcile_grace_secs |
10800 |
How long after a file was queued before it can be called lost (3 h). |
dedup_ttl_secs |
300 |
How long to suppress a re-request for the same chunk. |
state_path |
/var/lib/parachuter/cleaner-state.json |
Persistent dedup table. |
control_socket |
/run/parachuter/cleaner.sock |
Unix socket path. |
[cleaner.links.<name>]
Per-link budget for the cleaner's retransmit throttle.
| Key | Description |
|---|---|
max_in_flight |
Maximum outstanding re-requests per cleaner pass. |
min_period_ms |
Minimum milliseconds between consecutive requests. |
Runtime Control (parachuter ctl) ๐๏ธ
All flags take effect immediately without restarting any daemon.
# Check liveness
# Full status (auto-detects sender / receiver / cleaner)
# Switch to a different named link
# Change chunk size on the fly (bytes, including header)
# Change throughput cap
# Override destination IP and port without changing the named link
# Pause / resume
# Submit any file by absolute path (returns assigned file_id as JSON)
# Manually queue a file already in the ledger
# Queue a chunk range only, at the head of the queue
# Resend only the filename (manifest) packet for a file
# Drop everything from the queue
# Trigger a cleaner scan immediately (does not wait for the next timer tick)
# Reconcile the downlinked ledger against downloads now (don't wait 6 h)
Named Links ๐ก
Links are declared in [links.<name>] and referenced by name throughout the
config. Changing the active link, even at runtime, switches the destination
IP, port, and throughput cap atomically. The sender keeps a single bound UDP
socket and changes only the send_to destination on each packet.
Multicast links (any IP in 224.0.0.0/4) are automatically joined by the
receiver at startup. No configuration change is needed on the receiver side
when the sender switches from unicast (pilot) to multicast (LOS / TDRSS).
Dedup and Re-request Logic ๐งน
The cleaner uses three independent dedup mechanisms to avoid flooding the link:
-
Live queue inspection โ On each pass the cleaner calls
SenderStatusand subtracts any chunk ranges already queued from its own request list. -
TTL dedup table โ Every request is written to
cleaner-state.jsonwith a TTL (default 5 minutes). If the sender control socket is unreachable, the TTL cache still prevents re-requesting something sent seconds ago. -
Per-link budget โ Each pass is hard-capped by
max_in_flight, and consecutive requests are spaced at leastmin_period_msapart.
The cleaner can be triggered on demand via parachuter ctl ... cleaner-run,
which wakes the main loop immediately (no waiting for the next timer tick).
Recovering Files Lost Whole ๐
The cleaner can only repair files the ground knows about. If an outage swallows every packet of a file, the ground never creates a manifest for it, while the sender's ledger says it was sent. Reconciliation closes that gap without any acknowledgements from the ground:
- The sender downlinks its own ledger. Every
status_dump_period_secs(6 h) it writesfiles_sent_list_<unix>.csvintostatus_dumps/under its first priority directory, so the snapshot is sent down like any other file. It lists every file's full path, size, whether it still exists, andqueued_at, when it was last queued for a full send. - Files land at their payload path.
/data/qsc/science/m31_002.fits.bz2lands at<final_dir>/data/qsc/science/m31_002.fits.bz2, and only once it is complete: partial files stay inholding/until an atomic rename. - The cleaner compares the two. Every
reconcile_period_secs(6 h), it reads the newest snapshot underfinal_dirand classifies each file:
| Verdict | Meaning | Action |
|---|---|---|
| complete | At its mirrored path with the ledger's size | none |
| in flight | Partially received, in holding/ |
the cleaner already chases its gaps; flagged if unchanged for longer than the grace period |
| queued | Still in the sender's queue | none |
| too recent | Queued less than reconcile_grace_secs (3 h) ago |
wait |
| gone on payload | Never arrived and deleted on the payload | reported; cannot be resent |
| size mismatch | Landed, but not at the ledger's size | reported, not resent (the payload copy likely changed after sending) |
| missing | Lost whole | resent in full, at the back of the queue |
A file is requested at most once per period; if the resend is lost too, the
next pass asks again. No ground-side log is needed: final_dir is the
record, so it must be treated as an archive (copy files out, don't move
them). Run a pass on demand with parachuter ctl -s โฆ reconcile-run; the
last result is in parachuter ctl -s /run/parachuter/cleaner.sock status.
This also recovers files that were queued but never sent before a sender restart, since the queue lives in memory but the ledger does not.
Building
# Debug (fast compile)
# Release (LTO, symbol-stripped, panic=abort)
# Run the test suite (requires no external services)
Minimum Rust version: 1.74 (declared as rust-version in the workspace).
Testing โ
The test suite is entirely self-contained: no real network sockets, no
running daemons required. Tests use tempfile for isolated temporary
directories.
The integration tests in crates/parachuter/tests/end_to_end.rs cover:
- Out-of-order chunk delivery
- Manifest-first and data-first delivery paths
- Single-chunk and empty files
- Exact-multiple payload size
- Large files spanning multiple bitmap bytes (> 64 chunks)
- Duplicate detection
- Missing-range computation
- Packet corruption detection (CRC and magic)
- Ledger idempotency and CSV export
- Config validation
Deployment (systemd)
Two systemd units on the ground both invoke the same binary with different subcommands; one unit on the payload runs the sender.
# /etc/systemd/system/parachuter-sender.service (on payload)
[Unit]
Description=parachuter sender
After=network-online.target
[Service]
ExecStart=/usr/local/bin/parachuter sender --config /etc/parachuter/config.toml
Restart=on-failure
User=parachuter
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/parachuter-receiver.service (on ground)
[Unit]
Description=parachuter receiver
After=network-online.target
[Service]
ExecStart=/usr/local/bin/parachuter receiver --config /etc/parachuter/config.toml
Restart=on-failure
User=parachuter
[Install]
WantedBy=multi-user.target
# /etc/systemd/system/parachuter-cleaner.service (on ground)
[Unit]
Description=parachuter cleaner
After=parachuter-receiver.service
[Service]
ExecStart=/usr/local/bin/parachuter cleaner --config /etc/parachuter/config.toml
Restart=on-failure
User=parachuter
[Install]
WantedBy=multi-user.target
All modes handle SIGTERM gracefully. The sender drains the current packet
before exiting; the receiver and cleaner flush their state files.
Design Notes
See docs/DESIGN.md for:
- ๐ช Wire protocol layout (full 32-byte header spec)
- ๐ Manifest sidecar format
- ๐ง Design decisions
- โ ๏ธ Known limitations and future work
License
Copyright (c) 2026 Emaad Paracha
This software is licensed for personal, academic, research, educational, and other non-commercial use only. Commercial use requires prior written permission from the copyright holder. See LICENSE for the full terms.