Crate fiddler

Expand description

Fast and flexible data stream processor written in Rust

Provides a library for building data streaming pipelines using a declaritive yaml based configuration for data aggregation and transformation

Modules§

config: Contains configuration and module registration primitives for module development

BatchingPolicy: BatchingPolicy defines common configuration items for used in batching operations such as OutputBatch modules
Message: Message is the uniform struct utilized within all modules of fiddler. /// ```
Runtime: Represents a single data pipeline configuration Runtime to run

Closer: Closer trait utilized by input and output modules to optionally gracefully exit upon successful processing of the pipeline
Input: Input module trait to insert crate::Message into the processing pipeline.
InputBatch: BatchInput module trait to insert one to may crate::Message into the pipeline. InputBatch is currently not yet introduced into the runtime.
Output: Output module trait to write a single crate::Message to the output
OutputBatch: Batching output module utilized to write many messages to the output based on batch_size and provided interval. Defaults are batch_size: 500, interval: 10 seconds. The queuing mechanism is provided by the runtime and will call write_batch if the batch size has been reached, or the desired interval has passed, whichever comes first.
Processor: Processor is the processing module trait utilized to accept a single crate::Message and provide one to many additional messages through crate::MessageBatch

new_callback_chan: Helper function to generate transmitting and receiver pair that can be utilized for the crate::CallbackChan for input modules.

CallbackChan: Acknowledgement channel utilized to send feedback to input module on the successful or unsuccessful processing of event emited by input.
MessageBatch: Type alias for Vec<crate::Message>, a grouping of Messages being produced