pipeline-toolkit
A small toolkit for building linear, async processing pipelines out of
independently testable steps that communicate through a shared, type-keyed
Context.
Each step declares, via Step::access, exactly which types it reads,
writes, or consumes from the Context. PipelineBuilder::build uses those
declarations to catch wiring mistakes - a step reading a type nothing
upstream produces - before the pipeline ever runs, and Context enforces
them again at run time, so a step that reads or writes something it didn't
declare fails fast with PipelineError::AccessViolation instead of silently
drifting from its declared contract.
Installation
Requires Rust 1.85 or newer (edition 2024). No Cargo features to configure - the crate has a single, always-on API surface.
What's here
Context- a type-keyed store of at most one value per type, shared across the steps of a running pipeline. Enforces each step's declaredAccessSpecat run time and reports violations asAccessViolation.Step- the trait a unit of work implements:name,access(declaredAccessSpec), andrun, which returns aFlowtelling the pipeline whether to continue or stop with a final output.AccessSpec/AccessKind- declares which types a step reads, writes, or consumes, checked statically at build time and dynamically at run time.Pipeline/PipelineBuilder- assembles steps into a validated, immutable sequence. Supportsstep(append),insert_before/insert_after(anchor relative to another step by name), andbuild, which fails fast withBuildErroron unresolved anchors, duplicate step names, or a read no earlier step (or the pipeline input) satisfies.Branch/Arm- a built-inStepfor match-style (N-way) branching: a condition function picks a key at run time, and the arm (anArm, built the same way as a top-levelPipeline) registered under that key runs in its place.Branch::accessis computed automatically from its arms - reads as their union, writes as their intersection, consumes as their union - soPipelineBuilder::build's static checks see through a branch exactly as if it were a single ordinary step. An arm may itself contain aBranch, which is how nested branching is expressed. A key with no matching arm fails at run time withPipelineError::NoMatchingBranchrather than requiring a mandatory default arm.Loop/LoopBuilder- a built-inStepfor do-while style iteration: the sameArmbody runs at least once, then repeats as long as a condition function (given theContextand the 1-based iteration number just completed) returnstrue, up to a requiredmax_iterationscap.Loop::accessis computed automatically from the body's effective access plus the condition's own declared reads. A bodyErrpropagates immediately, exactly like any other step's failure, without ever reaching the condition function. Exceedingmax_iterationsfails at run time withPipelineError::MaxIterationsExceeded. The body may itself contain aBranchor anotherLoop, and aBrancharm may contain aLoop, with no extra API needed.
Example
use ;
;
;
# // This crate has no async-runtime dependency of its own; swap in
# // `tokio::main`/`pollster::block_on`/etc. in a real project.
#
#
If Greet were registered before LoadName, or LoadName never declared
writes::<String>(), build() would return
BuildError::UnsatisfiedRead instead of letting the mistake reach
run().
Branching
Branch lets a pipeline pick one of several arms at run time, without
giving up build-time validation:
use ;
// ...steps `MarkSmall` and `MarkLarge`, each declaring `reads::<i32>()`...
# ;
#
#
# ;
#
#
let branch = builder
.
.arm
.arm
.build
.unwrap;
Branch::access() is derived from its arms rather than declared by hand:
reads is the union across arms (whichever arm runs, its reads must
already be available), writes is the intersection across arms (only
what every arm guarantees is safe for a downstream step to rely on), and
consumes is the union across arms (if any arm might remove a type,
downstream can't assume it survives). This keeps PipelineBuilder::build's
existing checks meaningful even though the actual arm isn't chosen until
run time. An arm may itself contain a Branch, and this composition is
recursive with no extra API needed. A key with no registered arm fails at
run time with PipelineError::NoMatchingBranch instead of forcing a
catch-all default arm.
Both ArmBuilder::build and BranchBuilder::build return a Result.
Arm::builder().build() applies the same sequential validation
PipelineBuilder::build applies to a whole pipeline, scoped to that one
arm: a step reading a type an earlier step in the same arm already
consumed (and nothing rewrote since) is a BuildError::UnsatisfiedRead,
since no upstream write outside the arm could ever rescue a value the arm
already threw away itself. Branch::builder().build() fails with
BuildError::EmptyBranch if no arm was ever registered.
Iteration
Loop repeats the same Arm body, do-while style, until an external
condition function says to stop:
use ;
;
// ...step `Increment`, declaring `reads_writes::<Counter>()`...
# ;
#
#
let body = builder.step.build.unwrap;
let count_to_three = builder
.
.build
.unwrap;
The body always runs at least once. After each run, Flow::Done stops the
whole pipeline exactly like a top-level step would; Flow::Continue hands
off to the condition function, which is given the shared Context and the
1-based number of the iteration just completed. true runs the body
again; false stops the loop, and Loop::run itself returns
Flow::Continue so the enclosing pipeline moves on to whatever follows.
If the condition never returns false within max_iterations runs -
required at construction time, with no default - Loop::run fails with
PipelineError::MaxIterationsExceeded.
A body Err is never passed to the condition function; it propagates
immediately as an ordinary step failure. To retry only on a specific
business-level failure, have the body write its outcome to the Context
(e.g. as an AttemptResult type) and have the condition read that back,
rather than trying to distinguish retryable failures from fatal ones
through Err itself.
LoopBuilder::build() adds one check beyond what Arm::build() already
does: since the same body runs more than once, any type it consumes (via
AccessSpec::consumes) must also be rewritten by the time it finishes, or
the second iteration would find that type already gone. A body that fails
this is rejected with BuildError::UnrepeatableConsume when the Loop is
built, rather than failing only on a later iteration at run time. The
body may itself contain a Branch or another Loop, and a Branch arm
may contain a Loop; both compose with no extra API.
More usage, including insert_before/insert_after and the full set of
BuildError/PipelineError cases, is covered in the
API docs.
Changelog
See CHANGELOG.md for release history.
License
Licensed under either of Apache License, Version 2.0 or MIT license at your option.