Expand description
Pipelines as graphs (DAGs) of nodes whose meaning belongs to the application: a Kind
says what a node spec means (a source, a step or a sink) and how it reads and prints.
This crate parses ID=KIND+tee nodes and A>B[:NAME] edges, checks a graph
(Graph::check), orders it, and runs it (run::execute): each node once, fan-out to
every node downstream, fan-in as named frames, tees as side saves, and the nodes below a
failure skipped. Everything a node does happens in the application’s run::Host.
frame has helpers for the JSON column frames the nodes pass along.
Re-exports§
pub use run::Host;pub use run::NodeRun;pub use run::PipelineRun;pub use run::execute;
Modules§
- frame
- Frames: what flows between nodes, as JSON:
{model, fields, ids, columns, groups}withfieldsa list of{name, type},idsone per row andcolumnsan object of equally long arrays, one per field. These helpers only reshape frames; they never look anything up. - run
- Running a graph: each node once, in order, through the application’s
Host.
Structs§
Enums§
- Pipeline
Error - A graph, node, edge or frame input that is not valid.
- Problem
- Something wrong with a graph (
Graph::check). The texts are general; an application can word them its own way by matching on the variant. - Role
- Where a node sits: sources have no inputs, sinks are the leaves, steps are in between.
Traits§
- Kind
- What a node does, defined by the application: how its spec reads and prints, and its role. The crate gives no source, step or sink any meaning of its own.