Shepherd Framework
An scheduler framework that reliably undertakes data transformation and submission of results with perseverance.
Overview
The scheduler framework is designed to:
- Retry Mechanism: Ensure reliable transformation and submission of data by retrying tasks until completion.
- Non-blocking Architecture: Handle compute-heavy tasks efficiently, especially when data transforming units (DTUs) are hosted remotely (e.g., zkVM provers).
Features
- Flexible Integration: Supports various transformation units and consumer endpoints.
- Robustness: Guarantees task completion through persistent retries.
- Scalability: Optimized for compute-heavy workloads.
Requirements
Ensure you have rustup installed. Compile the system using the Rust nightly channel, as it utilizes nightly-only features such as associated_type_defaults.
Testing
PostgreSQL Database test requires docker to be installed on system & may pose intermittent failures if ran parallely. Use:
Documentation
Generate and view the documentation:
License
This project is licensed under the terms of the MIT License.