spring_batch_rs/lib.rs
1#![cfg_attr(docsrs, feature(doc_cfg))]
2//#![warn(missing_docs)]
3
4/*!
5 <div align="center">
6 <h1>Spring-Batch for Rust</h1>
7 <h3>🐞 A toolkit for building enterprise-grade batch applications</h3>
8
9 [](https://crates.io/crates/spring-batch-rs)
10 [](https://docs.rs/spring-batch-rs)
11 [](https://github.com/sboussekeyt/spring-batch-rs/actions/workflows/test.yml)
12 [](https://discord.gg/9FNhawNsG6)
13 [](https://codecov.io/gh/sboussekeyt/spring-batch-rs)
14 
15
16 </div>
17
18 # Spring-Batch for Rust
19 Spring Batch for Rust, offers a robust and flexible framework for the development of batch processing applications, addressing the challenges of handling large-scale data processing tasks efficiently and reliably. It provides developers a comprehensive toolkit for building enterprise-grade batch applications.
20
21 ## Features
22| **Feature** | **Description** |
23|---------------|---------------------------------------------------------------|
24| mongodb | Enable reader and writer for Mongodb database |
25| rdbc-postgres | Enable rdbc reader and writer for Postgres database |
26| rdbc-mysql | Enable rdbc reader and writer for Mysql and MariaDb databases |
27| rdbc-sqlite | Enable rdbc reader and writer for Sqlite database |
28| json | Enable json reader and writer |
29| csv | Enable csv reader and writer |
30| fake | Enable fake reader. Useful for generate fake dataset |
31| logger | Enable logger writer. Useful for debugging |
32
33 ## Roadmap
34 + XML reader and writer
35 + SQL reader and writer
36 + Kafka reader and writer
37 + Pulsar reader and writer
38 + Retry/Skip policies
39 + Save execution data in database
40
41 ## Getting Started
42 Make sure you activated the suitable features crate on Cargo.toml:
43
44```toml
45[dependencies]
46spring-batch-rs = { version = "<version>", features = ["<full|json|csv|fake|logger>"] }
47```
48
49Then, on your main.rs:
50
51```rust
52# use serde::{Deserialize, Serialize};
53# use spring_batch_rs::{
54# core::{
55# item::{ItemProcessor, ItemProcessorResult},
56# job::{Job, JobBuilder},
57# step::{Step, StepBuilder, StepInstance, StepStatus},
58# },
59# error::BatchError,
60# item::csv::csv_reader::CsvItemReaderBuilder,
61# item::json::json_writer::JsonItemWriterBuilder,
62# };
63# use std::env::temp_dir;
64# #[derive(Deserialize, Serialize, Debug, Clone)]
65# struct Car {
66# year: u16,
67# make: String,
68# model: String,
69# description: String,
70# }
71# #[derive(Default)]
72# struct UpperCaseProcessor {}
73# impl ItemProcessor<Car, Car> for UpperCaseProcessor {
74# fn process(&self, item: &Car) -> ItemProcessorResult<Car> {
75# let car = Car {
76# year: item.year,
77# make: item.make.to_uppercase(),
78# model: item.model.to_uppercase(),
79# description: item.description.to_uppercase(),
80# };
81# Ok(car)
82# }
83# }
84
85fn main() -> Result<(), BatchError> {
86 let csv = "year,make,model,description
87 1948,Porsche,356,Luxury sports car
88 1995,Peugeot,205,City car
89 2021,Mazda,CX-30,SUV Compact
90 1967,Ford,Mustang fastback 1967,American car";
91
92 let reader = CsvItemReaderBuilder::new()
93 .delimiter(b',')
94 .has_headers(true)
95 .from_reader(csv.as_bytes());
96
97 let processor = UpperCaseProcessor::default();
98
99 let writer = JsonItemWriterBuilder::new().from_path(temp_dir().join("cars.json"));
100
101 let step: StepInstance<Car, Car> = StepBuilder::new()
102 .reader(&reader) // set csv reader
103 .writer(&writer) // set json writer
104 .processor(&processor) // set upper case processor
105 .chunk(2) // set commit interval
106 .skip_limit(2) // set fault tolerance
107 .build();
108
109 let job = JobBuilder::new().start(&step).build();
110 let result = job.run();
111
112 assert!(result.is_ok());
113 assert!(step.get_status() == StepStatus::Success);
114
115 Ok(())
116}
117```
118
119## Examples
120+ [Generate CSV file from JSON file with processor](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/generate_csv_file_from_json_file_with_processor.rs)
121+ [Generate JSON file from CSV string with fault tolerance](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/generate_json_file_from_csv_string_with_fault_tolerance.rs)
122+ [Generate JSON file from fake persons](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/generate_json_file_from_fake_persons.rs)
123+ [Generate CSV file without headers from fake persons](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/generate_csv_file_without_headers_from_fake_persons.rs)
124+ [Insert records into Mysql database](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/insert_records_into_mysql_database.rs)
125+ [Log records from Postgres database](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/log_records_from_postgres_database.rs)
126+ [Read records from MongoDb database](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/read_records_from_mongodb_database.rs)
127+ [Write records to MongoDb database](https://github.com/sboussekeyt/spring-batch-rs/blob/main/examples/write_records_to_mongodb_database.rs)
128
129 ## License
130 Licensed under either of
131
132 - Apache License, Version 2.0
133 ([LICENSE-APACHE](LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
134 - MIT license
135 ([LICENSE-MIT](LICENSE-MIT) or <http://opensource.org/licenses/MIT>)
136
137 at your option.
138
139 ## Contribution
140 Unless you explicitly state otherwise, any contribution intentionally submitted
141 for inclusion in the work by you, as defined in the Apache-2.0 license, shall be
142 dual licensed as above, without any additional terms or conditions
143
144 */
145
146/// Core module for batch operations
147pub mod core;
148
149/// Error types for batch operations
150pub mod error;
151
152#[doc(inline)]
153pub use error::*;
154
155/// Set of items readers / writers (for exemple: csv reader and writer)
156pub mod item;