processing-chain
processing-chain
provides a convenient way to seamlessly set up processing
chains for large amounts of data.
Please read the API documentation on docs.rs
or take a look at the examples
.
processing-chain
is based on the concept of Item which is an abstraction that is used to spawn all the processes in parallel. All the user needs to do is define:
- The Items to be processed
- The function that processes a single Item
processing-chain
will take care of spawning the process across all Items via parallelization.
The user can also provide some extra processing configuration information (e.g., overwrite).
Highlights
- Set-up generic data processing chains
Define the Items
Using a JSON file
Write the _process_item
function
In rust:
If your function is written in Python and you don't feel like converting it to Rust (yet), you could use the inline-python crate.
use python;
Some examples can be found here
.