Crate pipe_io 

Typed source-transform-sink pipelines with backpressure, batching, and per-stage error isolation. A lightweight runtime-agnostic stream processor for in-process workloads. The missing middle ground between raw iterators and full distributed stream processing.

Quick start

use pipe_io::{Pipeline, sink::VecSink};

let sink = VecSink::<i64>::new();
let handle = sink.handle();

Pipeline::from_iter(1..=5)
    .map(|n: i32| i64::from(n) * 10)
    .filter(|n: &i64| *n > 20)
    .sink(sink)
    .run()
    .expect("pipeline run");

assert_eq!(handle.take(), vec![30, 40, 50]);

Features

• std (default): enables source::ChannelSource, source::ReaderSource, sink::ChannelSink, sink::WriterSink, and driver::ThreadedDriver. With std disabled the crate compiles in no_std mode (requires alloc) with the data primitives, closure-based adapters, batching, and driver::SyncDriver available.

Design notes

Sources are pull-based (Source::pull). Stages receive items one at a time and emit zero or more outputs via an Emit callback handed in by the driver. Sinks are push-based (Sink::write). The builder is fully typed; the carrier type of the pipeline is tracked at compile time across every stage transition. See REPS.md for the binding API contract and .dev/DESIGN.md for design notes.

Re-exports

pub use crate::batch::Batch;

pub use crate::batch::BatchPolicy;

pub use crate::batch::ByteSize;

pub use crate::driver::Driver;

pub use crate::driver::RunStats;

pub use crate::emit::Emit;

pub use crate::emit::EmitError;

pub use crate::error::BoxError;

pub use crate::error::Error;

pub use crate::error::ErrorPolicy;

pub use crate::error::Result;

pub use crate::error::StageError;

pub use crate::error::StageFailure;

pub use crate::sink::Sink;

pub use crate::source::Source;

pub use crate::stage::Stage;

pub use crate::window::Clock;std

pub use crate::window::SystemClock;std

pub use crate::window::Window;std

pub use crate::window::WindowPolicy;std

Modules

batch

Batching primitives: BatchPolicy, Batch, and ByteSize.

driver

Pipeline execution drivers.

emit

The Emit trait and its error type.

error

Error model for pipe-io.

sink

The Sink trait and built-in sink adapters.

source

The Source trait and built-in source adapters.

stage

The Stage trait.

windowstd

Windowing primitives: Clock, SystemClock, WindowPolicy, and Window<T>.

Structs

Pipeline

A built pipeline. Run with Pipeline::run (sync) or Pipeline::run_threaded (std).

PipelineBuilder

Typed builder. Each transformer changes the carrier type T.

StageId

Identifier for a pipeline stage. Carried by crate::Error variants and by crate::StageFailure so a consumer can route or log by stage name.

Constants

VERSION

Crate version string, populated by Cargo at build time.