knut-thund 0.1.7

//! The Þund **intermediate representation** (IR) — one dataflow graph that
//! all three authoring surfaces (typed Rust builder, declarative DSL, facett
//! graph editor) produce, and that every [`crate::backend::ExecBackend`]
//! consumes.
//!
//! # Why one IR
//!
//! The hypothesis Þund is built on is **all-three-over-one-IR**: rather than
//! pick a single authoring model, every front-end lowers to *this* graph, and
//! every execution backend is a function *from* this graph. The IR is the
//! contract; the front-ends and back-ends are interchangeable around it. This
//! is what lets a pipeline authored as a Rust builder run on the native
//! Arrow/DataFusion engine in dev, then be lowered to Spark Declarative
//! Pipelines (SDP) over Spark Connect for a clustered run — unchanged.
//!
//! # Shape
//!
//! A [`Pipeline`] is a set of [`Dataset`] nodes (the *outputs* — what data
//! should exist) and [`Flow`] edges (how each dataset is computed from its
//! inputs). This is the **asset-graph** model (declare the artifact, infer the
//! DAG from declared inputs/outputs) that Dagster pioneered and that makes
//! incremental *reconciliation* — recompute only what is stale — possible,
//! rather than Airflow's blind task-order re-execution.
//!
//! Crucially the IR is a **superset** of the Spark-SDP model implemented by
//! [`knut_pipelines::DataflowGraph`]: every Thund concept that SDP also has
//! lowers 1:1 (see [`crate::backend::spark`]), and the streaming concepts SDP
//! cannot express yet (explicit watermarks, triggers, output modes,
//! windowing) lower onto the native backend or degrade gracefully.
//!
//! The IR is **Arrow-typed**: a [`Dataset`]'s schema is an
//! [`arrow_schema::Schema`], so the same type information flows from authoring
//! through lowering into the RecordBatch streams the native backend runs and
//! the Arrow-IPC/Flight transport that carries them.

mod dataset;
mod expectation;
mod flow;
mod pipeline;
mod schema;
mod streaming;

pub use dataset::{Dataset, Materialization, OutputType, SourceLocation, WriteMode};
pub use expectation::{Expectation, OnViolation};
pub use flow::{CdcSpec, Flow, FlowKind, ScdType};
pub use pipeline::Pipeline;
pub use schema::DatasetSchema;
pub use streaming::{OutputMode, SourceSpec, Trigger, Watermark, WindowSpec};