Crate feedme 

FeedMe

FeedMe is a deterministic, linear, streaming ingest pipeline with mechanical guarantees around memory, ordering, and failure.

FeedMe provides a linear, deterministic processing model for Rust applications that need reliable data ingestion. It emphasizes bounded resource usage, explicit error handling, and comprehensive observability without affecting execution.

Key Features

• Streaming, bounded memory: Processes one event at a time; memory usage stays flat
• Deterministic processing: Same input + same config → same output
• Structured errors: Stage, code, and message for every failure
• Observability: Metrics exportable (Prometheus or JSON) without affecting execution
• Extensible: Add custom stages via a defined plugin contract

Guarantees

FeedMe provides these mechanical guarantees:

• Events are processed strictly in input order
• Memory usage is bounded and input-size independent
• Stages cannot observe shared or mutated state
• Validation failures cannot be silently ignored
• Metrics collection cannot influence execution

Example

use feedme::{
    Pipeline, FieldSelect, RequiredFields, StdoutOutput, Deadletter,
    PIIRedaction, Filter, InputSource, Stage
};
use std::path::PathBuf;

fn main() -> anyhow::Result<()> {
    // Create pipeline: select fields → redact PII → require fields → filter → output
    let mut pipeline = Pipeline::new();
    pipeline.add_stage(Box::new(FieldSelect::new(vec![
        "timestamp".into(), "level".into(), "message".into(), "email".into()
    ])));
    let email_pattern = regex::Regex::new(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b")?;
    pipeline.add_stage(Box::new(PIIRedaction::new(vec![email_pattern])));
    pipeline.add_stage(Box::new(RequiredFields::new(vec!["level".into()])));
    pipeline.add_stage(Box::new(Filter::new(Box::new(|event| {
        event.data.get("level").and_then(|v| v.as_str()) != Some("debug")
    }))));
    pipeline.add_stage(Box::new(StdoutOutput::new()));

    // Deadletter for errors
    let mut deadletter = Deadletter::new(PathBuf::from("samples/errors.ndjson"));

    // Process input file
    let mut input = InputSource::File(PathBuf::from("samples/input.ndjson"));
    input.process_input(&mut pipeline, &mut Some(&mut deadletter))?;

    // Export final metrics
    println!("Pipeline complete. Metrics:");
    for metric in pipeline.export_json_logs() {
        println!("{}", serde_json::to_string(&metric)?);
    }

    Ok(())
}

Modules

invariant_ppt

replay

Replay Harness (Testing & Debugging Only)

Macros

assert_invariant

Structs

Config

Configuration: ensure pipeline behavior is fully declared and validated before execution. YAML input, version required, schema validated, unknown field rejection, no runtime mutation.

Deadletter

DerivedFields

Event

Represents a structured event in the pipeline. Owned, mutable data, supports JSON-like types, typed field access, optional metadata.

FieldRemap

FieldSelect

FileOutput

Filter

JSONArrayParser

LatencyStats

Metrics

Metrics for observability: counters, latency summaries, drop reason codes. No execution feedback loops. Bounded storage.

NDJSONParser

OutputError

PIIRedaction

ParseError

Pipeline

Pipeline: linear, deterministic execution of stages. No distributed coordination, constant memory streaming.

PluginRegistry

Plugins: enable user-defined stages with explicit registration and isolation. No implicit discovery.

RequiredFields

StdoutOutput

SyslogParser

SystemError

TransformError

TypeChecking

ValidationError

ValueConstraints

Enums

DropReason

InputSource

Input sources: local, synchronous, stream-oriented, ordered read. No distributed offsets, no remote coordination.

OutputErrorCode

ParseErrorCode

PipelineError

Error taxonomy for pipeline failures. Explicit category, stage attribution, machine-readable code.

SystemErrorCode

TransformErrorCode

ValidationErrorCode

Traits

Output

Outputs: emit processed events to local or synchronous destinations with explicit failure semantics. Ordered write, bounded retry, no unbounded retry, no background flush.

Parser

Parsers: convert raw bytes to Event with explicit error handling. Best effort syslog, zero copy where possible, no implicit recovery.

Stage

Stage contract: ownership-based execution. Takes Event, returns Option<Event>, with explicit drop semantics.

Transform

Transforms: bounded, explicit modification or filtering of events. Deterministic, side-effect free, no network, no persistence.

Validator

Validators: enforce structural and semantic correctness of events before output. Schema enforced, fail closed, no silent acceptance.

Type Aliases

EventDerivationFn

Type aliases for complex function types to reduce clippy warnings

StageFactoryFn

ValueConstraintFn