spring_batch_rs/
lib.rs

1#![cfg_attr(docsrs, feature(doc_cfg))]
2//#![warn(missing_docs)]
3
4/*!
5 <div align="center">
6   <h1>spring-batch-rs</h1>
7   <h3>Stop writing batch boilerplate. Start processing data.</h3>
8
9   [![crate](https://img.shields.io/crates/v/spring-batch-rs.svg)](https://crates.io/crates/spring-batch-rs)
10   [![docs](https://docs.rs/spring-batch-rs/badge.svg)](https://docs.rs/spring-batch-rs)
11   [![build status](https://github.com/sboussekeyt/spring-batch-rs/actions/workflows/test.yml/badge.svg)](https://github.com/sboussekeyt/spring-batch-rs/actions/workflows/test.yml)
12   [![Discord chat](https://img.shields.io/discord/1097536141617528966.svg?logo=discord&style=flat-square)](https://discord.gg/9FNhawNsG6)
13   [![CodeCov](https://codecov.io/gh/sboussekeyt/spring-batch-rs/branch/main/graph/badge.svg)](https://codecov.io/gh/sboussekeyt/spring-batch-rs)
14   ![license](https://shields.io/badge/license-MIT%2FApache--2.0-blue)
15
16  </div>
17
18Processing a large CSV into a database? You end up writing readers, chunk logic, error
19loops, retry handling — just to move data. **Spring Batch RS** handles the plumbing: you
20define what to read, what to transform, where to write. Skip policies, execution metrics,
21and fault tolerance come built-in.
22
23## Quick Start
24
25### 1. Add to `Cargo.toml`
26
27```toml
28[dependencies]
29spring-batch-rs = { version = "0.3", features = ["csv", "json"] }
30serde = { version = "1.0", features = ["derive"] }
31```
32
33### 2. Your first batch job (CSV → JSON)
34
35> **Note:** `rdbc-*` and `orm` features require `tokio = { version = "1", features = ["full"] }`.
36> See the [Getting Started guide](https://spring-batch-rs.boussekeyt.dev/getting-started/) for the async setup.
37
38```rust,no_run
39use spring_batch_rs::{
40    core::{job::{Job, JobBuilder}, step::StepBuilder, item::PassThroughProcessor},
41    item::{
42        csv::csv_reader::CsvItemReaderBuilder,
43        json::json_writer::JsonItemWriterBuilder,
44    },
45    BatchError,
46};
47use serde::{Deserialize, Serialize};
48use std::env::temp_dir;
49
50#[derive(Deserialize, Serialize, Clone)]
51struct Order {
52    id: u32,
53    amount: f64,
54    status: String,
55}
56
57fn main() -> Result<(), BatchError> {
58    let csv = "id,amount,status\n1,99.5,pending\n2,14.0,complete\n3,bad,pending";
59
60    // Read from CSV
61    let reader = CsvItemReaderBuilder::<Order>::new()
62        .has_headers(true)
63        .from_reader(csv.as_bytes());
64
65    // Write to JSON
66    let output = temp_dir().join("orders.json");
67    let writer = JsonItemWriterBuilder::<Order>::new()
68        .from_path(&output);
69
70    // Wire together: read 100 items at a time, tolerate up to 5 bad rows
71    let processor = PassThroughProcessor::<Order>::new();
72    let step = StepBuilder::new("csv-to-json")
73        .chunk::<Order, Order>(100)
74        .reader(&reader)
75        .processor(&processor)
76        .writer(&writer)
77        .skip_limit(5)
78        .build();
79
80    JobBuilder::new().start(&step).build().run().map(|_| ())?;
81    println!("Output: {}", output.display());
82    Ok(())
83}
84```
85
86## How It Works
87
88A **Job** contains one or more **Steps**. Each Step reads items one by one from a source,
89buffers them into a configurable chunk, then writes the whole chunk at once — balancing
90throughput with memory usage.
91
92```text
93Read item → Read item → ... → [chunk full] → Write chunk → repeat
94```
95
96## Why spring-batch-rs
97
98- **Chunk-oriented processing** — reads one item at a time, writes in batches. Memory usage stays constant regardless of dataset size.
99- **Fault tolerance built-in** — set a `skip_limit` to keep processing when bad rows appear. No manual try/catch loops.
100- **Type-safe pipelines** — reader, processor, and writer types are verified at compile time. Mismatched types don't compile.
101- **Modular by design** — enable only what you need via feature flags. No unused dependencies.
102
103## Features
104
105**Formats**
106
107| Feature | Description |
108| ------- | ----------- |
109| `csv`   | CSV `ItemReader` and `ItemWriter` |
110| `json`  | JSON `ItemReader` and `ItemWriter` |
111| `xml`   | XML `ItemReader` and `ItemWriter` |
112
113**Databases** *(require `tokio` — see [Getting Started](https://spring-batch-rs.boussekeyt.dev/getting-started/))*
114
115| Feature         | Description |
116| --------------- | ----------- |
117| `rdbc-postgres` | PostgreSQL `ItemReader` and `ItemWriter` |
118| `rdbc-mysql`    | MySQL / MariaDB `ItemReader` and `ItemWriter` |
119| `rdbc-sqlite`   | SQLite `ItemReader` and `ItemWriter` |
120| `mongodb`       | MongoDB `ItemReader` and `ItemWriter` (sync) |
121| `orm`           | SeaORM `ItemReader` and `ItemWriter` |
122
123**Utilities**
124
125| Feature  | Description |
126| -------- | ----------- |
127| `zip`    | ZIP compression `Tasklet` |
128| `ftp`    | FTP / FTPS `Tasklet` |
129| `fake`   | Fake data `ItemReader` for generating test datasets |
130| `logger` | Logger `ItemWriter` for debugging pipelines |
131| `full`   | All of the above |
132
133## Examples
134
135| Use case | Run |
136| -------- | --- |
137| CSV → JSON | `cargo run --example csv_processing --features csv,json` |
138| JSON processing | `cargo run --example json_processing --features json,csv,logger` |
139| XML processing | `cargo run --example xml_processing --features xml,json,csv` |
140| CSV → SQLite | `cargo run --example database_processing --features rdbc-sqlite,csv,json,logger` |
141| MongoDB | `cargo run --example mongodb_processing --features mongodb,csv,json` |
142| SeaORM | `cargo run --example orm_processing --features orm,csv,json` |
143| Advanced ETL pipeline | `cargo run --example advanced_patterns --features csv,json,logger` |
144| ZIP tasklet | `cargo run --example tasklet_zip --features zip` |
145| FTP tasklet | `cargo run --example tasklet_ftp --features ftp` |
146
147> Database examples require Docker. Browse the **[full examples gallery](https://spring-batch-rs.boussekeyt.dev/quick-examples/)** for tutorials and advanced patterns.
148
149## Documentation
150
151| Resource | Link |
152| -------- | ---- |
153| Getting Started | [spring-batch-rs.boussekeyt.dev/getting-started](https://spring-batch-rs.boussekeyt.dev/getting-started/) |
154| Item Readers & Writers | [spring-batch-rs.boussekeyt.dev/item-readers-writers](https://spring-batch-rs.boussekeyt.dev/item-readers-writers/overview/) |
155| API Reference | [docs.rs/spring-batch-rs](https://docs.rs/spring-batch-rs) |
156| Architecture | [spring-batch-rs.boussekeyt.dev/architecture](https://spring-batch-rs.boussekeyt.dev/architecture/) |
157
158## Community
159
160- [Discord](https://discord.gg/9FNhawNsG6) — Chat with the community
161- [GitHub Issues](https://github.com/sboussekeyt/spring-batch-rs/issues) — Bug reports and feature requests
162- [GitHub Discussions](https://github.com/sboussekeyt/spring-batch-rs/discussions) — Questions and ideas
163
164## License
165
166Licensed under [MIT](https://github.com/sboussekeyt/spring-batch-rs/blob/main/LICENSE-MIT) or [Apache-2.0](https://github.com/sboussekeyt/spring-batch-rs/blob/main/LICENSE-APACHE) at your option.
167
168*/
169
170/// Core module for batch operations
171pub mod core;
172
173/// Error types for batch operations
174pub mod error;
175
176#[doc(inline)]
177pub use error::*;
178
179/// Set of items readers / writers  (for exemple: csv reader and writer)
180pub mod item;
181
182/// Set of tasklets for common batch operations
183pub mod tasklet;
spring_batch_rs/lib.rs

spring_batch_rs/
lib.rs
