pipeline-toolkit 0.2.0

A small toolkit for building linear, async processing pipelines out of independently testable steps that communicate through a shared, type-keyed Context.
Documentation

pipeline-toolkit

crates.io docs.rs CI codecov MSRV license

A small toolkit for building linear, async processing pipelines out of independently testable steps that communicate through a shared, type-keyed Context.

Each step declares, via Step::access, exactly which types it reads, writes, or consumes from the Context. PipelineBuilder::build uses those declarations to catch wiring mistakes - a step reading a type nothing upstream produces - before the pipeline ever runs, and Context enforces them again at run time, so a step that reads or writes something it didn't declare fails fast with PipelineError::AccessViolation instead of silently drifting from its declared contract.

Installation

cargo add pipeline-toolkit

Requires Rust 1.85 or newer (edition 2024). No Cargo features to configure - the crate has a single, always-on API surface.

What's here

  • Context - a type-keyed store of at most one value per type, shared across the steps of a running pipeline. Enforces each step's declared AccessSpec at run time and reports violations as AccessViolation.
  • Step - the trait a unit of work implements: name, access (declared AccessSpec), and run, which returns a Flow telling the pipeline whether to continue or stop with a final output.
  • AccessSpec/AccessKind - declares which types a step reads, writes, or consumes, checked statically at build time and dynamically at run time.
  • Pipeline/PipelineBuilder - assembles steps into a validated, immutable sequence. Supports step (append), insert_before/ insert_after (anchor relative to another step by name), and build, which fails fast with BuildError on unresolved anchors, duplicate step names, or a read no earlier step (or the pipeline input) satisfies.
  • Branch/Arm - a built-in Step for match-style (N-way) branching: a condition function picks a key at run time, and the arm (an Arm, built the same way as a top-level Pipeline) registered under that key runs in its place. Branch::access is computed automatically from its arms - reads as their union, writes as their intersection, consumes as their union - so PipelineBuilder::build's static checks see through a branch exactly as if it were a single ordinary step. An arm may itself contain a Branch, which is how nested branching is expressed. A key with no matching arm fails at run time with PipelineError::NoMatchingBranch rather than requiring a mandatory default arm.
  • Loop/LoopBuilder - a built-in Step for do-while style iteration: the same Arm body runs at least once, then repeats as long as a condition function (given the Context and the 1-based iteration number just completed) returns true, up to a required max_iterations cap. Loop::access is computed automatically from the body's effective access plus the condition's own declared reads. A body Err propagates immediately, exactly like any other step's failure, without ever reaching the condition function. Exceeding max_iterations fails at run time with PipelineError::MaxIterationsExceeded. The body may itself contain a Branch or another Loop, and a Branch arm may contain a Loop, with no extra API needed.

Example

use pipeline_toolkit::{AccessSpec, Context, Flow, Pipeline, Step, StepError};

struct LoadName;

#[async_trait::async_trait]
impl Step<String> for LoadName {
    fn name(&self) -> &str {
        "load_name"
    }

    fn access(&self) -> AccessSpec {
        AccessSpec::new().writes::<String>()
    }

    async fn run(&self, ctx: &mut Context) -> Result<Flow<String>, StepError> {
        ctx.insert("Ada".to_string());
        Ok(Flow::Continue)
    }
}

struct Greet;

#[async_trait::async_trait]
impl Step<String> for Greet {
    fn name(&self) -> &str {
        "greet"
    }

    fn access(&self) -> AccessSpec {
        AccessSpec::new().reads::<String>()
    }

    async fn run(&self, ctx: &mut Context) -> Result<Flow<String>, StepError> {
        let name = ctx.get::<String>().expect("declared as read").clone();
        Ok(Flow::Done(format!("Hello, {name}!")))
    }
}

# // This crate has no async-runtime dependency of its own; swap in
# // `tokio::main`/`pollster::block_on`/etc. in a real project.
# fn block_on<F: std::future::Future>(fut: F) -> F::Output {
#     use std::task::{Context as TaskContext, Poll, Waker};
#     let mut fut = std::pin::pin!(fut);
#     let mut cx = TaskContext::from_waker(Waker::noop());
#     loop {
#         if let Poll::Ready(val) = fut.as_mut().poll(&mut cx) {
#             return val;
#         }
#     }
# }
#
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let pipeline = Pipeline::<(), String>::builder()
        .step(LoadName)
        .step(Greet)
        .build()?;

    let greeting = block_on(pipeline.run(()))?;
    assert_eq!(greeting, "Hello, Ada!");
    Ok(())
}

If Greet were registered before LoadName, or LoadName never declared writes::<String>(), build() would return BuildError::UnsatisfiedRead instead of letting the mistake reach run().

Branching

Branch lets a pipeline pick one of several arms at run time, without giving up build-time validation:

use pipeline_toolkit::{AccessSpec, Arm, Branch, Context, Flow, Step, StepError};

#[derive(Clone, Copy, PartialEq, Eq, Hash)]
enum Route {
    Small,
    Large,
}

// ...steps `MarkSmall` and `MarkLarge`, each declaring `reads::<i32>()`...
# struct MarkSmall;
# #[async_trait::async_trait]
# impl Step<i32> for MarkSmall {
#     fn name(&self) -> &str { "mark_small" }
#     fn access(&self) -> AccessSpec { AccessSpec::new().reads::<i32>() }
#     async fn run(&self, ctx: &mut Context) -> Result<Flow<i32>, StepError> {
#         Ok(Flow::Done(*ctx.get::<i32>().unwrap()))
#     }
# }
# struct MarkLarge;
# #[async_trait::async_trait]
# impl Step<i32> for MarkLarge {
#     fn name(&self) -> &str { "mark_large" }
#     fn access(&self) -> AccessSpec { AccessSpec::new().reads::<i32>() }
#     async fn run(&self, ctx: &mut Context) -> Result<Flow<i32>, StepError> {
#         Ok(Flow::Done(*ctx.get::<i32>().unwrap() * 100))
#     }
# }

let branch = Branch::builder("route", |ctx: &Context| {
    if *ctx.get::<i32>().expect("declared as read") < 10 {
        Route::Small
    } else {
        Route::Large
    }
})
.reads::<i32>()
.arm(Route::Small, Arm::builder().step(MarkSmall).build().unwrap())
.arm(Route::Large, Arm::builder().step(MarkLarge).build().unwrap())
.build()
.unwrap();

Branch::access() is derived from its arms rather than declared by hand: reads is the union across arms (whichever arm runs, its reads must already be available), writes is the intersection across arms (only what every arm guarantees is safe for a downstream step to rely on), and consumes is the union across arms (if any arm might remove a type, downstream can't assume it survives). This keeps PipelineBuilder::build's existing checks meaningful even though the actual arm isn't chosen until run time. An arm may itself contain a Branch, and this composition is recursive with no extra API needed. A key with no registered arm fails at run time with PipelineError::NoMatchingBranch instead of forcing a catch-all default arm.

Both ArmBuilder::build and BranchBuilder::build return a Result. Arm::builder().build() applies the same sequential validation PipelineBuilder::build applies to a whole pipeline, scoped to that one arm: a step reading a type an earlier step in the same arm already consumed (and nothing rewrote since) is a BuildError::UnsatisfiedRead, since no upstream write outside the arm could ever rescue a value the arm already threw away itself. Branch::builder().build() fails with BuildError::EmptyBranch if no arm was ever registered.

Iteration

Loop repeats the same Arm body, do-while style, until an external condition function says to stop:

use pipeline_toolkit::{AccessSpec, Arm, Context, Flow, Loop, Step, StepError};

struct Counter(i32);

// ...step `Increment`, declaring `reads_writes::<Counter>()`...
# struct Increment;
# #[async_trait::async_trait]
# impl Step<i32> for Increment {
#     fn name(&self) -> &str { "increment" }
#     fn access(&self) -> AccessSpec { AccessSpec::new().reads_writes::<Counter>() }
#     async fn run(&self, ctx: &mut Context) -> Result<Flow<i32>, StepError> {
#         ctx.get_mut::<Counter>().unwrap().0 += 1;
#         Ok(Flow::Continue)
#     }
# }

let body = Arm::builder().step(Increment).build().unwrap();
let count_to_three = Loop::builder(
    "count_to_three",
    body,
    |ctx: &Context, _iteration: u32| ctx.get::<Counter>().expect("declared as read").0 < 3,
    10, // max_iterations
)
.reads::<Counter>()
.build()
.unwrap();

The body always runs at least once. After each run, Flow::Done stops the whole pipeline exactly like a top-level step would; Flow::Continue hands off to the condition function, which is given the shared Context and the 1-based number of the iteration just completed. true runs the body again; false stops the loop, and Loop::run itself returns Flow::Continue so the enclosing pipeline moves on to whatever follows. If the condition never returns false within max_iterations runs - required at construction time, with no default - Loop::run fails with PipelineError::MaxIterationsExceeded.

A body Err is never passed to the condition function; it propagates immediately as an ordinary step failure. To retry only on a specific business-level failure, have the body write its outcome to the Context (e.g. as an AttemptResult type) and have the condition read that back, rather than trying to distinguish retryable failures from fatal ones through Err itself.

LoopBuilder::build() adds one check beyond what Arm::build() already does: since the same body runs more than once, any type it consumes (via AccessSpec::consumes) must also be rewritten by the time it finishes, or the second iteration would find that type already gone. A body that fails this is rejected with BuildError::UnrepeatableConsume when the Loop is built, rather than failing only on a later iteration at run time. The body may itself contain a Branch or another Loop, and a Branch arm may contain a Loop; both compose with no extra API.

More usage, including insert_before/insert_after and the full set of BuildError/PipelineError cases, is covered in the API docs.

Changelog

See CHANGELOG.md for release history.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.