pipe-io 1.0.0

Typed source-transform-sink pipelines with backpressure, batching, windowing, and per-stage error isolation. A lightweight runtime-agnostic stream processor for in-process workloads. The missing middle ground between raw iterators and full distributed stream processing.
Documentation
# pipe-io v0.3.0 - Minimum viable implementation

**Date:** 2026-05-19
**Compare:** `v0.1.0...v0.3.0`

## Headline

First substantive release after the `v0.1.0` reservation scaffold.
Lands the design lock that was drafted on `main` (and never tagged
on its own) and the minimum viable implementation of the locked
public surface: typed pipelines with `Source`, `Stage`, and `Sink`
traits; closure adapters; count- and byte-based batching;
fail-fast and continue error policies; and two built-in drivers
(`SyncDriver` for the calling thread, `ThreadedDriver` for a
spawned worker). 50 tests pass under `--all-features`.

The version skip from `0.1.0` to `0.3.0` is intentional: `0.2.0`
referred to a documentation-only design lock that never got its
own SemVer release; its content is folded into `0.3.0`.

## What changed since `v0.1.0`

### New - Public surface

The full public surface listed in `REPS.md` section 4 is now
implemented except for the deliberately deferred slices noted
below. See [`docs/API.md`](../API.md) for the offline reference.

### New - Core traits and adapters

- `Source` with adapters `IterSource`, `FnSource` (no_std-friendly)
  plus `ChannelSource` and `ReaderSource` under `std`.
- `Stage` with closure-based adapters exposed on `PipelineBuilder`:
  `map`, `filter`, `filter_map`, `flat_map`, `inspect`, `try_map`,
  and a generic `.stage(impl Stage)` for plugging in custom stages.
- `Sink` with adapters `NullSink`, `FnSink` (no_std-friendly) plus
  `VecSink` (with cloneable `SharedHandle`), `ChannelSink`, and
  `WriterSink` under `std`.
- `Emit` trait and `EmitError` enum for the inside of `Stage::process`.

### New - Error model

- `Error` enum with `Source` / `Stage` / `Sink` / `Buffer`
  variants carrying `StageId`, plus `Cancelled` and `Closed`.
- `StageError` trait blanket-implemented for every
  `Debug + Display + Send + Sync + 'static`. The
  `core::error::Error` super-trait was avoided so the bound holds
  at the crate's MSRV of 1.75.
- `BoxError` alias.
- `ErrorPolicy` (`FailFast`, `Continue`, `DeadLetter`); the first
  two are fully wired in `0.3.0`. `DeadLetter` is reserved and
  behaves as `Continue` until a future release lands the
  dead-letter sink wiring.
- `StageFailure` record for future dead-letter routing.
- `BufferErrorKind` (`Full`, `Closed`) for backpressure failures.

### New - Batching

- `Batch<T>` newtype with `Deref<Target = [T]>` and `IntoIterator`.
- `BatchPolicy` with `max_items`, `max_bytes`, and `max_age` (std)
  triggers; all OR together.
- `ByteSize` opt-in trait with blanket impls for `&str`, `String`,
  `Vec<u8>`, `&[u8]`.
- `PipelineBuilder::batch(policy)` for count and age triggers;
  `PipelineBuilder::batch_bytes(policy)` for byte-aware batching
  (requires `T: ByteSize`).

### New - Drivers

- `SyncDriver` (`no_std`-compatible) pumps the pipeline on the
  caller's thread.
- `ThreadedDriver` (std-only) spawns a worker thread and joins on
  it. Per-stage threading is a future enhancement; `0.3.0` runs
  the source plus every stage on one worker.
- `RunStats` returned by every successful run, carrying `items_in`
  (always) and `duration` (under `std`).

### New - Builder

- `Pipeline::from_source(source)` for arbitrary sources;
  `Pipeline::from_iter(iter)` convenience for any `IntoIterator`.
- `.stage_id("name")` labels the next stage; appears in
  `Error::*::stage` and `RunStats`.
- `.on_error(policy)` switches the active `ErrorPolicy`.
- `.sink(sink)` materializes the pipeline.
- `Pipeline::run()` for sync; `Pipeline::run_threaded()` (std) for
  the threaded driver.

### New - Documentation

- [`docs/API.md`]../API.md rewritten as the offline mirror of
  the public surface.
- Every public item has a rustdoc block. Doctests on the
  user-facing constructors, adapters, and policy types
  (8 doctests, all passing).

### Changed - Design-lock adjustments to `REPS.md`

Implementation revealed three spec adjustments worth recording:

- Section 4.7: `StageError` uses a
  `Debug + Display + Send + Sync + 'static` bound rather than
  `core::error::Error` (which stabilized in Rust 1.81, post-MSRV).
- Section 4.8: a unified `Driver` trait is deferred past `0.3.x`.
  `SyncDriver` and `ThreadedDriver` differ in their `Send` bounds,
  and a single trait would force the stricter bound on both.
  Consumers select via `Pipeline::run` / `Pipeline::run_threaded`.
- Section 4.6: `window` module deferred past `0.3.0`. The locked
  surface is preserved verbatim for the release that ships it.
- Section 4.4: `VecSink` is `std`-only because its shared handle
  uses `Arc<Mutex<_>>`. `WriterSink<W>` pins `Item = String`.

## Public surface

Full listing in [`docs/API.md`](../API.md). Summary by module:

### Always available (data primitives and traits)

- Traits: `pipe_io::{Source, Stage, Sink, Emit, ByteSize, StageError}`.
- Data types: `pipe_io::{Batch, BatchPolicy, Error, Result,
  BoxError, StageId, StageFailure, BufferErrorKind, ErrorPolicy,
  EmitError, RunStats, VERSION}`.
- Source adapters: `pipe_io::source::{IterSource, FnSource, Infallible}`.
- Sink adapters: `pipe_io::sink::{NullSink, FnSink}`.
- Drivers: `pipe_io::driver::SyncDriver`.
- Pipeline machinery: `pipe_io::{Pipeline, PipelineBuilder}` and
  all builder methods.

### `std` feature (default)

- Source adapters: `pipe_io::source::{ChannelSource, ReaderSource}`.
- Sink adapters: `pipe_io::sink::{VecSink, SharedHandle, ChannelSink,
  ChannelSinkError, WriterSink}`.
- Drivers: `pipe_io::driver::ThreadedDriver`.
- `RunStats::duration: Duration` field.
- `BatchPolicy::max_age` / `BatchPolicy::age_limit` triggers.
- `Pipeline::run_threaded`.

### Feature flags

| Flag    | Default | Effect                                                                 |
|---------|---------|------------------------------------------------------------------------|
| `std`   | yes     | `ThreadedDriver`, channel/reader/writer adapters, `VecSink`, batch age trigger. |

### MSRV

`1.75`, unchanged from `v0.1.0`.

### Runtime dependencies

Zero. The crate depends only on `core` and `alloc` (always) plus
`std` when the `std` feature is enabled.

## Verification

Local matrix (Windows x86_64-pc-windows-msvc, stable 1.95):

- `cargo build` (default features) [OK]
- `cargo build --no-default-features` [OK]
- `cargo build --all-features` [OK]
- `cargo +1.75 build --all-features` (MSRV) [OK]
- `cargo fmt --all -- --check` [OK]
- `cargo clippy --all-targets --all-features -- -D warnings` [OK]
- `cargo test --all-features` - **50 tests pass, 0 ignored**
  (27 unit + 15 integration + 8 doctest)
- `RUSTDOCFLAGS="-D warnings -D rustdoc::broken-intra-doc-links" cargo doc --all-features --no-deps` [OK]
- `cargo package --no-verify --allow-dirty` [OK]
  (29 files, 121.3 KiB; 30.8 KiB compressed)
- Banned-word and em-dash scan across `src/`, `tests/`, `docs/`,
  `README.md`, `CHANGELOG.md`, `REPS.md`: 0 hits

CI matrix continues to run on ubuntu-latest, macos-latest,
windows-latest.

## Migration

### From `v0.1.0`

The `v0.1.0` release was a placeholder scaffold with no public
items beyond `pipe_io::VERSION`. Upgrading to `v0.3.0` adds the
entire pipeline surface; no removals or renames apply because
there was nothing to rename.

A consumer upgrading from `v0.1.0` can start with:

```rust
use pipe_io::{Pipeline, sink::VecSink};

let sink = VecSink::<i64>::new();
let handle = sink.handle();
Pipeline::from_iter(0..5)
    .map(|n: i32| i64::from(n) * 10)
    .sink(sink)
    .run()
    .unwrap();
assert_eq!(handle.take(), vec![0, 10, 20, 30, 40]);
```

## Stability commitment

`pipe-io` follows SemVer: `0.x.y` releases are not API-stable
(see `REPS.md` section 8). `0.3.0` is a minor release adding the
core pipeline surface; further `0.x` releases may rename or
restructure items as the implementation matures. API freeze
happens at `1.0.0`; `cargo-semver-checks` gates the public
surface in CI from `0.9.0` onward.

Three slices of the locked `1.0.0` surface are not yet shipped
and may evolve before they land:

1. The `window` module (tumbling / sliding / session policies +
   `Clock` trait).
2. A unified `Driver` trait extensibility hook.
3. The dead-letter sink wiring for `ErrorPolicy::DeadLetter`.

## Release ceremony

```bash
git tag -a v0.3.0 -m "Release v0.3.0 - Minimum viable implementation"
git push origin main
git push origin v0.3.0

cargo publish

cargo search pipe-io | head -3
```

GitHub release title: `v0.3.0 - Minimum viable implementation`.
Not tagged as pre-release.

## Acknowledgements

`pipe-io` reached `0.3.0` through `v0.1.0` (scaffold), an untagged
design-lock pass on main, and this release. The implementation
landed in a single session against the locked specification:
50 tests pass, clippy is clean under `-D warnings`, the public
surface compiles on MSRV 1.75 with zero runtime dependencies, and
the threaded driver moves items between threads without any
`unsafe` code (the crate sets `#![forbid(unsafe_code)]`).

---

**Full Changelog:** https://github.com/jamesgober/pipe-io/compare/v0.1.0...v0.3.0