Skip to main content

Crate weirdnet_pipelines

Crate weirdnet_pipelines 

Source
Expand description

Pipelines as graphs (DAGs) of nodes whose meaning belongs to the application: a Kind says what a node spec means (a source, a step or a sink) and how it reads and prints. This crate parses ID=KIND+tee nodes and A>B[:NAME] edges, checks a graph (Graph::check), orders it, and runs it (run::execute): each node once, fan-out to every node downstream, fan-in as named frames, tees as side saves, and the nodes below a failure skipped. Everything a node does happens in the application’s run::Host. frame has helpers for the JSON column frames the nodes pass along.

Re-exports§

pub use run::Host;
pub use run::NodeRun;
pub use run::PipelineRun;
pub use run::execute;

Modules§

frame
Frames: what flows between nodes, as JSON: {model, fields, ids, columns, groups} with fields a list of {name, type}, ids one per row and columns an object of equally long arrays, one per field. These helpers only reshape frames; they never look anything up.
run
Running a graph: each node once, in order, through the application’s Host.

Structs§

Edge
Graph
Node

Enums§

PipelineError
A graph, node, edge or frame input that is not valid.
Problem
Something wrong with a graph (Graph::check). The texts are general; an application can word them its own way by matching on the variant.
Role
Where a node sits: sources have no inputs, sinks are the leaves, steps are in between.

Traits§

Kind
What a node does, defined by the application: how its spec reads and prints, and its role. The crate gives no source, step or sink any meaning of its own.

Type Aliases§

Resolve
Says why a node’s names do not resolve in the application (None: they do).
Result