Courier

Async Rust framework for composable data pipelines:

Source → Transform* → Sink[]

Each stage runs as its own Tokio task and communicates through bounded tokio::mpsc channels using a shared Envelope type. Backpressure, graceful shutdown, retries, and dead-letter handling are built in.

Beta: APIs are provisional and may change without notice.

Documentation

Full documentation lives at https://gbpagano.github.io/courier/.

Jump in:

• Quickstart — first pipeline in a few minutes
• Configuration — config.toml reference
• Components — built-in sources, transforms, sinks
• Scripting — Rhai, Lua, and Python transforms
• Architecture — internals and design
• Observability — logs, metrics, and traces over OTLP (sample Collector config, Compose stack, and Grafana dashboard)

Install

cargo install data-courier --version 0.1.0-beta.4

This installs the courier binary. See the installation guide for prerequisites (Rust toolchain, C toolchain for rdkafka, optional python3 for the Python script transform) and build-from-source instructions.

Quick example

[[pipelines]]
name = "api->kafka"

[pipelines.source]
type = "api_poll"
url = "https://jsonplaceholder.typicode.com/posts/1"
interval_secs = 3

[[pipelines.sinks]]
type = "kafka"
brokers = "localhost:9092"
topic = "topic1"

Run it:

courier run

Pipelines are loaded at runtime from config.toml (override with COURIER_CONFIG). Restart the binary to pick up edits — no recompile required.

Contributing

See the contributing guide.