pipeflow 0.0.1

A lightweight, configuration-driven data pipeline framework
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112

# pipeflow

A lightweight, configuration-driven data pipeline framework for Rust.

```
Source → Transform → Sink
```

## Features

- **Configuration-driven**: Define pipelines in YAML, no code required
- **Lightweight**: ~10MB binary, minimal resource footprint
- **Built-in DLQ**: Dead Letter Queue for error handling out of the box
- **Backpressure**: Bounded channels with configurable strategies (block/drop)
- **Disk Buffer**: JSONL-based persistence for reliable delivery
- **Fan-out**: One source/transform can feed multiple downstream nodes

## Quick Example

```yaml
# pipeline.yaml
name: "api-to-db"

sources:
  - id: api_poller
    type: http_client
    config:
      url: "https://api.example.com/data"
      interval_secs: 60

transforms:
  - id: field_mapper
    type: remap
    input: api_poller
    config:
      mappings:
        price: "$.data.price"
        symbol: "$.data.symbol"

sinks:
  - id: db_writer
    type: database
    input: field_mapper
    config:
      connection: "${DATABASE_URL}"
      table: metrics
```

## Architecture

### Sources

| Type | Description |
|------|-------------|
| `http_client` | HTTP polling with configurable interval |
| `http_server` | HTTP push/webhook receiver |
| `websocket` | WebSocket streaming |
| `database` | Database polling/CDC |
| `file` | File watching (tail mode) |
| `internal` | Receive events from internal sinks |

### Transforms

| Type | Description | I/O |
|------|-------------|-----|
| `remap` | Field mapping (JSONPath-like) | 1:1 |
| `filter` | Conditional filtering | 1:0/1 |
| `aggregate` | Window-based aggregation | n:m |

### Sinks

| Type | Description | Disk Buffer |
|------|-------------|-------------|
| `console` | Print to stdout | No |
| `file` | Write to file | No |
| `database` | Database insert | Yes |
| `redis` | Redis SET/PUBLISH | Yes |
| `http_api` | HTTP POST/PUT | Yes |
| `notify` | Email/Telegram/Webhook | Yes |
| `internal` | Send to internal source | No |
| `blackhole` | Discard | No |

## Dead Letter Queue (DLQ)

All errors are automatically captured and written to `dead_letters.jsonl`:

```yaml
dead_letter:
  enabled: true
  default_sink:
    type: file
    path: "./dead_letters.jsonl"
```

## Comparison with Vector

| Feature | Vector | pipeflow |
|---------|--------|----------|
| Sources/Sinks | 100+ | ~10 |
| Transform DSL | VRL (full language) | JSONPath (simple) |
| Built-in DLQ | No | **Yes** |
| Binary size | ~50MB | ~10MB |
| Learning curve | Steep | **Easy** |

**pipeflow is NOT a Vector replacement.** Use pipeflow for:
- Simple HTTP API data collection
- Lightweight alerting pipelines
- Resource-constrained environments

## License

MIT OR Apache-2.0