Expand description
A pipeline execution library for building data processing workflows.
Pipelines are composed of Nodes that read from and write to Datasets.
Nodes are grouped into Steps (tuples) and executed by a Runner.
The App struct ties everything together with CLI dispatch, YAML config
loading, and hook-based lifecycle events.
Works in no_std environments (with CellDataset + SequentialRunner)
and scales up to parallel execution with std.
Re-exports§
pub use app::App;pub use catalog_indexer::CatalogIndex;pub use catalog_indexer::index_catalog;pub use catalog_indexer::index_catalog_with_params;pub use datasets::Dataset;pub use datasets::DatasetMeta;pub use datasets::TemplatedCatalog;pub use error::CheckError;pub use error::PondError;pub use graph::PipelineGraph;pub use graph::build_pipeline_graph;pub use hooks::Hook;pub use hooks::Hooks;pub use pipeline::DatasetEvent;pub use pipeline::DatasetRef;pub use pipeline::Ident;pub use pipeline::IntoNodeResult;pub use pipeline::Node;pub use pipeline::Pipeline;pub use pipeline::PipelineFn;pub use pipeline::PipelineInfo;pub use pipeline::RunnableStep;pub use pipeline::StepInfo;pub use pipeline::Steps;pub use pipeline::StepVec;pub use pipeline::Split;pub use pipeline::Join;pub use runners::ParallelRunner;pub use runners::Runner;pub use runners::Runners;pub use runners::SequentialRunner;
Modules§
- app
- Application framework for building pipeline executables.
- catalog_
indexer - Catalog indexer: maps dataset pointer IDs to human-readable field names.
- datasets
- Dataset types for loading and saving data.
- error
- Error types for the pipeline framework.
- graph
- Pipeline graph: a pre-computed DAG representation of the pipeline.
- hooks
- Hooks for pipeline execution events.
- pipeline
- Core types and traits for pipelines.
- runners
- Pipeline runners.
- viz
- Pipeline visualization: axum server, serialization, and VizHook.