pipeflow

A lightweight, configuration-driven data pipeline framework for Rust.

Source → Transform → Sink

Features

Configuration-driven: Define pipelines in YAML, no code required
Lightweight: ~10MB binary, minimal resource footprint
Built-in DLQ: Dead Letter Queue for error handling out of the box
Backpressure: Bounded channels with configurable strategies (block/drop)
Disk Buffer: JSONL-based persistence for reliable delivery
Fan-out: One source/transform can feed multiple downstream nodes

Quick Example

# pipeline.yaml
name: "api-to-db"

sources:
  - id: api_poller
    type: http_client
    config:
      url: "https://api.example.com/data"
      interval_secs: 60

transforms:
  - id: field_mapper
    type: remap
    input: api_poller
    config:
      mappings:
        price: "$.data.price"
        symbol: "$.data.symbol"

sinks:
  - id: db_writer
    type: database
    input: field_mapper
    config:
      connection: "${DATABASE_URL}"
      table: metrics

Architecture

Sources

Type	Description
`http_client`	HTTP polling with configurable interval
`http_server`	HTTP push/webhook receiver
`websocket`	WebSocket streaming
`database`	Database polling/CDC
`file`	File watching (tail mode)
`internal`	Receive events from internal sinks

Transforms

Type	Description	I/O
`remap`	Field mapping (JSONPath-like)	1:1
`filter`	Conditional filtering	1:0/1
`aggregate`	Window-based aggregation	n:m

Sinks

Type	Description	Disk Buffer
`console`	Print to stdout	No
`file`	Write to file	No
`database`	Database insert	Yes
`redis`	Redis SET/PUBLISH	Yes
`http_api`	HTTP POST/PUT	Yes
`notify`	Email/Telegram/Webhook	Yes
`internal`	Send to internal source	No
`blackhole`	Discard	No

Dead Letter Queue (DLQ)

All errors are automatically captured and written to dead_letters.jsonl:

dead_letter:
  enabled: true
  default_sink:
    type: file
    path: "./dead_letters.jsonl"

Comparison with Vector

Feature	Vector	pipeflow
Sources/Sinks	100+	~10
Transform DSL	VRL (full language)	JSONPath (simple)
Built-in DLQ	No	Yes
Binary size	~50MB	~10MB
Learning curve	Steep	Easy

pipeflow is NOT a Vector replacement. Use pipeflow for:

Simple HTTP API data collection
Lightweight alerting pipelines
Resource-constrained environments

License

MIT OR Apache-2.0

pipeflow 0.0.1