Crate fiddler

Expand description

Fast and flexible data stream processor written in Rust

Provides a library for building data streaming pipelines using a declarative yaml based configuration for data aggregation and transformation

Modules§

config: Contains configuration and module registration primitives for module development

BatchingPolicy: BatchingPolicy defines common configuration items for used in batching operations such as OutputBatch modules.
Message: Message is the uniform struct utilized within all modules of fiddler.
MetricEntry: MetricEntry is the uniform struct utilized within metrics modules of fiddler.
Runtime: Represents a single data pipeline configuration Runtime to run

Error: Enum to capture errors occurred through the pipeline.
MessageType: MessageType is utilized by plugins to identiy which type of message are they sending to the runtime. MessageType::Default is utilized for processing data that will be sent to their configured outputs. MessageType::BeginStream and MessageType::EndStream will not be processed by the pipeline but are utilized to logically group messages together under a shared callback function.
Status: Status returned through the crate::CallbackChan to input modules

SHUTDOWN_MESSAGE_ID: Reserved message ID used internally for shutdown signaling. User messages should not use this ID.

Closer: Closer trait utilized by input and output modules to optionally gracefully exit upon successful processing of the pipeline
Input: Input module trait to insert crate::Message into the processing pipeline.
InputBatch: BatchInput module trait to insert one to many crate::Message into the pipeline.
Metrics: Trait for metrics backends.
Output: Output module trait to write a single crate::Message to the output
OutputBatch: Batching output module utilized to write many messages to the output based on batch_size and provided interval. Defaults are batch_size: 500, interval: 10 seconds. The queuing mechanism is provided by the runtime and will call write_batch if the batch size has been reached, or the desired interval has passed, whichever comes first.
Processor: Processor is the processing module trait utilized to accept a single crate::Message and provide one to many additional messages through crate::MessageBatch

new_callback_chan: Helper function to generate transmitting and receiver pair that can be utilized for the crate::CallbackChan for input modules.

CallbackChan: Channel for sending acknowledgment status back to input modules.
MessageBatch: A batch of messages being processed together.