Crate pipeflow 

pipeflow - A lightweight, configuration-driven data pipeline framework

Overview

pipeflow follows the classic ETL pattern:

Source → Transform → Sink

Pipelines are defined in YAML configuration files, requiring no code to set up common data processing workflows.

Quick Start

# pipeline.yaml

pipeline:
  sources:
    - id: api_poller
      type: http_client
      config:
        url: "https://httpbin.org/json"
        interval: "10s" # supports "30s", "5m", "1h 30m", etc.

  transforms:
    - id: pass_through
      inputs: [api_poller]
      outputs: [console_out]

  sinks:
    - id: console_out
      type: console
      config:
        format: pretty

Features

• Configuration-driven: Define pipelines in YAML
• Bounded fan-out buffers: tokio::sync::broadcast with configurable capacity (slow consumers may lag and drop messages)
• Fan-out: One source/transform can feed multiple downstream nodes

Re-exports

pub use common::message::Message;

pub use common::message::MessageMeta;

pub use common::message::NodeType;

pub use common::message::SharedMessage;

pub use engine::MetricsSnapshot;

pub use engine::PipelineMetrics;

pub use error::Error;

pub use error::Result;

Modules

common

Common types and utilities shared across pipeline components

config

Configuration parsing

engine

Engine for DAG construction and execution

error

Error types for the pipeline

prelude

Prelude module for convenient imports

sink

Sink trait and implementations

source

Source trait and implementations

transform

Transform trait and implementations