Crate pipeflow 

pipeflow - A lightweight, configuration-driven data pipeline framework

Overview

pipeflow follows the classic ETL pattern:

Source → Transform → Sink

Pipelines are defined in YAML configuration files, requiring no code to set up common data processing workflows.

Quick Start

# pipeline.yaml

pipeline:
  sources:
    - id: api_poller
      type: http_client
      config:
        url: "https://httpbin.org/json"
        interval: "10s" # supports "30s", "5m", "1h 30m", etc.

  sinks:
    - id: console_out
      type: console
      inputs: [api_poller]
      config:
        format: pretty

Features

• Configuration-driven: Define pipelines in YAML
• Backpressure: Bounded channels with configurable capacity
• Fan-out: One source/transform can feed multiple downstream nodes

Re-exports

pub use error::Error;

pub use error::Result;

pub use message::DeadLetter;

pub use message::Message;

pub use message::MessageMeta;

pub use message::SharedMessage;

Modules

config

Configuration parsing

engine

Engine for DAG construction and execution

error

Error types for the pipeline

message

Message types for the pipeline

prelude

Prelude module for convenient imports

sink

Sink trait and implementations

source

Source trait and implementations

transform

Transform trait and implementations