Crate streamweave 

StreamWeave

StreamWeave is a composable, async, stream-first computation framework for Rust. It provides a fluent API for building data processing pipelines with producers, transformers, and consumers.

Core Concepts

• Producers: Generate data streams
• Transformers: Process and transform stream items
• Consumers: Consume stream data
• Pipelines: Compose producers, transformers, and consumers together

Example

use streamweave::prelude::*;

let pipeline = Pipeline::new()
    .with_producer(ArrayProducer::new(vec![1, 2, 3, 4, 5]))
    .with_transformer(MapTransformer::new(|x| x * 2))
    .with_consumer(VecConsumer::new());

let result = pipeline.run().await?;

Re-exports

pub use consumer::Consumer;

pub use error::ErrorAction;

pub use error::ErrorContext;

pub use error::ErrorStrategy;

pub use error::StreamError;

pub use input::Input;

pub use message::Message;

pub use message::MessageId;

pub use message::MessageMetadata;

pub use output::Output;

pub use pipeline::Complete;

pub use pipeline::Empty;

pub use pipeline::HasProducer;

pub use pipeline::HasTransformer;

pub use pipeline::Pipeline;

pub use producer::Producer;

pub use stateful_transformer::InMemoryStateStore;

pub use stateful_transformer::StateStore;

pub use stateful_transformer::StateStoreExt;

pub use stateful_transformer::StatefulTransformer;

pub use transformer::Transformer;

pub use consumers::csv::csv_consumer::CsvConsumer;

pub use consumers::jsonl::jsonl_consumer::JsonlConsumer;

pub use consumers::msgpack::msgpack_consumer::MsgPackConsumer;

pub use consumers::parquet::parquet_consumer::ParquetConsumer;

pub use producers::csv::csv_producer::CsvProducer;

pub use producers::jsonl::jsonl_producer::JsonlProducer;

pub use producers::msgpack::msgpack_producer::MsgPackProducer;

pub use producers::parquet::parquet_producer::ParquetProducer;

Modules

consumer

Consumer trait and related types.

consumers

Built-in consumer implementations. Built-in consumer implementations.

distributed

Distributed processing support. Distributed stream processing architecture.

error

Error handling types and strategies.

http_server

HTTP server integration for stream processing.

input

Input type definitions.

message

Message types for stream items. Message envelope types for exactly-once processing.

metrics

Metrics collection and reporting.

offset

Offset tracking for stream processing. Offset tracking for resumable pipeline processing.

output

Output type definitions.

pipeline

Pipeline builder and execution.

producer

Producer trait and related types.

producers

Built-in producer implementations.

stateful_transformer

Stateful transformer support. Stateful transformer trait for stream processing with persistent state.

transaction

Transaction management for exactly-once processing. Transaction support for exactly-once processing.

transformer

Transformer trait and related types.

transformers

Built-in transformer implementations. Transformers module.

visualization

Visualization utilities for stream monitoring.

window

Window-based processing utilities. Windowing operations for stream processing.