spring_batch_rs/
lib.rs

1#![cfg_attr(docsrs, feature(doc_cfg))]
2//#![warn(missing_docs)]
3
4/*!
5 <div align="center">
6   <h1>spring-batch-rs</h1>
7   <h3>Stop writing batch boilerplate. Start processing data.</h3>
8
9   [![crate](https://img.shields.io/crates/v/spring-batch-rs.svg)](https://crates.io/crates/spring-batch-rs)
10   [![docs](https://docs.rs/spring-batch-rs/badge.svg)](https://docs.rs/spring-batch-rs)
11   [![build status](https://github.com/sboussekeyt/spring-batch-rs/actions/workflows/test.yml/badge.svg)](https://github.com/sboussekeyt/spring-batch-rs/actions/workflows/test.yml)
12   [![Discord chat](https://img.shields.io/discord/1097536141617528966.svg?logo=discord&style=flat-square)](https://discord.gg/9FNhawNsG6)
13   [![CodeCov](https://codecov.io/gh/sboussekeyt/spring-batch-rs/branch/main/graph/badge.svg)](https://codecov.io/gh/sboussekeyt/spring-batch-rs)
14   ![license](https://shields.io/badge/license-MIT%2FApache--2.0-blue)
15
16  </div>
17
18Processing a large CSV into a database? You end up writing readers, chunk logic, error
19loops, retry handling — just to move data. **Spring Batch RS** handles the plumbing: you
20define what to read, what to transform, where to write. Skip policies, execution metrics,
21and fault tolerance come built-in.
22
23## Quick Start
24
25### 1. Add to `Cargo.toml`
26
27```toml
28[dependencies]
29spring-batch-rs = { version = "0.3", features = ["csv", "json"] }
30serde = { version = "1.0", features = ["derive"] }
31```
32
33### 2. Your first batch job (CSV → JSON)
34
35> **Note:** `rdbc-*` and `orm` features require `tokio = { version = "1", features = ["full"] }`.
36> See the [Getting Started guide](https://spring-batch-rs.boussekeyt.dev/getting-started/) for the async setup.
37
38```rust,no_run
39use spring_batch_rs::{
40    core::{job::{Job, JobBuilder}, step::StepBuilder},
41    item::{
42        csv::csv_reader::CsvItemReaderBuilder,
43        json::json_writer::JsonItemWriterBuilder,
44    },
45    BatchError,
46};
47use serde::{Deserialize, Serialize};
48use std::env::temp_dir;
49
50#[derive(Deserialize, Serialize, Clone)]
51struct Order {
52    id: u32,
53    amount: f64,
54    status: String,
55}
56
57fn main() -> Result<(), BatchError> {
58    let csv = "id,amount,status\n1,99.5,pending\n2,14.0,complete\n3,bad,pending";
59
60    // Read from CSV
61    let reader = CsvItemReaderBuilder::<Order>::new()
62        .has_headers(true)
63        .from_reader(csv.as_bytes());
64
65    // Write to JSON
66    let output = temp_dir().join("orders.json");
67    let writer = JsonItemWriterBuilder::<Order>::new()
68        .from_path(&output);
69
70    // Wire together: read 100 items at a time, tolerate up to 5 bad rows
71    // (no processor needed here since we're just moving Order -> Order)
72    let step = StepBuilder::new("csv-to-json")
73        .chunk::<Order, Order>(100)
74        .reader(&reader)
75        .writer(&writer)
76        .skip_limit(5)
77        .build();
78
79    JobBuilder::new().start(&step).build().run().map(|_| ())?;
80    println!("Output: {}", output.display());
81    Ok(())
82}
83```
84
85## How It Works
86
87A **Job** contains one or more **Steps**. Each Step reads items one by one from a source,
88buffers them into a configurable chunk, then writes the whole chunk at once — balancing
89throughput with memory usage.
90
91```text
92Read item → Read item → ... → [chunk full] → Write chunk → repeat
93```
94
95## Why spring-batch-rs
96
97- **Chunk-oriented processing** — reads one item at a time, writes in batches. Memory usage stays constant regardless of dataset size.
98- **Fault tolerance built-in** — set a `skip_limit` to keep processing when bad rows appear. No manual try/catch loops.
99- **Type-safe pipelines** — reader, processor, and writer types are verified at compile time. Mismatched types don't compile.
100- **Modular by design** — enable only what you need via feature flags. No unused dependencies.
101
102## Features
103
104**Formats**
105
106| Feature | Description |
107| ------- | ----------- |
108| `csv`   | CSV `ItemReader` and `ItemWriter` |
109| `json`  | JSON `ItemReader` and `ItemWriter` |
110| `xml`   | XML `ItemReader` and `ItemWriter` |
111
112**Databases** *(require `tokio` — see [Getting Started](https://spring-batch-rs.boussekeyt.dev/getting-started/))*
113
114| Feature         | Description |
115| --------------- | ----------- |
116| `rdbc-postgres` | PostgreSQL `ItemReader` and `ItemWriter` |
117| `rdbc-mysql`    | MySQL / MariaDB `ItemReader` and `ItemWriter` |
118| `rdbc-sqlite`   | SQLite `ItemReader` and `ItemWriter` |
119| `mongodb`       | MongoDB `ItemReader` and `ItemWriter` (sync) |
120| `orm`           | SeaORM `ItemReader` and `ItemWriter` |
121
122**Utilities**
123
124| Feature  | Description |
125| -------- | ----------- |
126| `zip`    | ZIP compression `Tasklet` |
127| `ftp`    | FTP / FTPS `Tasklet` |
128| `fake`   | Fake data `ItemReader` for generating test datasets |
129| `logger` | Logger `ItemWriter` for debugging pipelines |
130| `full`   | All of the above |
131
132## Examples
133
134| Use case | Run |
135| -------- | --- |
136| CSV → JSON | `cargo run --example csv_processing --features csv,json` |
137| JSON processing | `cargo run --example json_processing --features json,csv,logger` |
138| XML processing | `cargo run --example xml_processing --features xml,json,csv` |
139| CSV → SQLite | `cargo run --example database_processing --features rdbc-sqlite,csv,json,logger` |
140| MongoDB | `cargo run --example mongodb_processing --features mongodb,csv,json` |
141| SeaORM | `cargo run --example orm_processing --features orm,csv,json` |
142| Advanced ETL pipeline | `cargo run --example advanced_patterns --features csv,json,logger` |
143| ZIP tasklet | `cargo run --example tasklet_zip --features zip` |
144| FTP tasklet | `cargo run --example tasklet_ftp --features ftp` |
145
146> Database examples require Docker. Browse the **[full examples gallery](https://spring-batch-rs.boussekeyt.dev/quick-examples/)** for tutorials and advanced patterns.
147
148## Documentation
149
150| Resource | Link |
151| -------- | ---- |
152| Getting Started | [spring-batch-rs.boussekeyt.dev/getting-started](https://spring-batch-rs.boussekeyt.dev/getting-started/) |
153| Item Readers & Writers | [spring-batch-rs.boussekeyt.dev/item-readers-writers](https://spring-batch-rs.boussekeyt.dev/item-readers-writers/overview/) |
154| API Reference | [docs.rs/spring-batch-rs](https://docs.rs/spring-batch-rs) |
155| Architecture | [spring-batch-rs.boussekeyt.dev/architecture](https://spring-batch-rs.boussekeyt.dev/architecture/) |
156
157## Community
158
159- [Discord](https://discord.gg/9FNhawNsG6) — Chat with the community
160- [GitHub Issues](https://github.com/sboussekeyt/spring-batch-rs/issues) — Bug reports and feature requests
161- [GitHub Discussions](https://github.com/sboussekeyt/spring-batch-rs/discussions) — Questions and ideas
162
163## License
164
165Licensed under [MIT](https://github.com/sboussekeyt/spring-batch-rs/blob/main/LICENSE-MIT) or [Apache-2.0](https://github.com/sboussekeyt/spring-batch-rs/blob/main/LICENSE-APACHE) at your option.
166
167*/
168
169/// Core module for batch operations
170pub mod core;
171
172/// Error types for batch operations
173pub mod error;
174
175#[doc(inline)]
176pub use error::*;
177
178/// Set of items readers / writers  (for exemple: csv reader and writer)
179pub mod item;
180
181/// Set of tasklets for common batch operations
182pub mod tasklet;
spring_batch_rs/lib.rs

spring_batch_rs/
lib.rs
