pipeflow 0.0.4

A lightweight, configuration-driven data pipeline framework
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71

//! pipeflow - A lightweight, configuration-driven data pipeline framework
//!
//! # Overview
//!
//! pipeflow follows the classic ETL pattern:
//!
//! ```text
//! Source → Transform → Sink
//! ```
//!
//! Pipelines are defined in YAML configuration files, requiring no code to set up
//! common data processing workflows.
//!
//! # Quick Start
//!
//! ```yaml
//! # pipeline.yaml
//!
//! pipeline:
//!   sources:
//!     - id: api_poller
//!       type: http_client
//!       config:
//!         url: "https://httpbin.org/json"
//!         interval: "10s" # supports "30s", "5m", "1h 30m", etc.
//!
//!   transforms:
//!     - id: pass_through
//!       inputs: [api_poller]
//!       outputs: [console_out]
//!
//!   sinks:
//!     - id: console_out
//!       type: console
//!       config:
//!         format: pretty
//! ```
//!
//! # Features
//!
//! - **Configuration-driven**: Define pipelines in YAML
//! - **Bounded fan-out buffers**: `tokio::sync::broadcast` with configurable capacity (slow consumers may lag and drop messages)
//! - **Fan-out**: One source/transform can feed multiple downstream nodes

#![warn(missing_docs)]
#![warn(clippy::all)]
#![deny(unsafe_code)]

pub mod common;
pub mod config;
pub mod engine;
pub mod error;
pub mod sink;
pub mod source;
pub mod transform;

// Re-exports
pub use common::message::{Message, MessageMeta, NodeType, SharedMessage};
pub use engine::{MetricsSnapshot, PipelineMetrics};
pub use error::{Error, Result};

/// Prelude module for convenient imports
pub mod prelude {
    pub use crate::common::message::{Message, MessageMeta, NodeType, SharedMessage};
    pub use crate::config::Config;
    pub use crate::engine::{Engine, MetricsSnapshot, PipelineMetrics};
    pub use crate::error::{Error, Result};
    pub use crate::sink::Sink;
    pub use crate::source::Source;
    pub use crate::transform::Transform;
}