Expand description
§Shepherd Framework
This crate provides a general framework for writing schedulers that transform input data into output data by invoking workers.
§Features
- Flexible Worker Integration: Workers can be implemented as separate processes, remote services, or via Apache Kafka.
- Data Transformation: Input data is transformed into output data using configurable pipelines.
- Output Handling: Output data can be sent to remote services, written to files, or stored in databases.
§Modules
config: Handles composite configurations for the shepherd.consumer: Manages consumption attempts and results.database: Provides database abstractions for storing transformation data.emitter: Handles emission of transformation requests.instance: Represents an shepherd instance.instrumentation: Provides instrumentation for monitoring and debugging.processor: Processes transformation requests and manages attempts.test: Contains testing utilities and mock implementations.transform: Defines transformation requests and attempts.worker: Manages worker instances and their interactions.
Modules§
- config
- A module for managing configurations
- consumer
- A module for handling consumption attempts
- database
- A module for database abstractions
- emitter
- A module for emitting transformation requests
- instance
- Top-level module for an instance of the shepherd
- instrumentation
- A module for instrumenting the runtime
- processor
- A module for processing transformation requests
- transform
- A module for handling transformations
- worker
- A module for managing transformation work, and management of workers