fv-plan
fv-plan executes FusionVault pipelines on Apache DataFusion. Give it a pipeline definition and the input rows; it runs every transform in dependency order, evaluates the output datasets' expectations, and hands back a build record with the promoted outputs. It reads no databases and writes none: the host feeds rows in and takes rows out, which is what lets the same executor serve batch builds, previews, and streams.
A build in twenty lines
use ;
use json;
// A pipeline: transforms over datasets, each a list of steps. This one keeps fast vehicles and
// adds a converted speed column.
let pipeline = json!;
let datasets = vec!;
let mut slow = new;
slow.set;
slow.set;
let mut fast = new;
fast.set;
fast.set;
let record = new.unwrap.block_on;
assert_eq!;
let out = &record.outputs;
assert_eq!;
assert_eq!;
A build never panics. Its record's status is SUCCEEDED, BLOCKED (an expectation failed, so
the previous output stays current) or FAILED (with the reason in error).
The model
- A dataset is a named table (
apiName). Inputs come from the host; outputs are produced by transforms. A dataset marked"materialized": falseis an intermediate that is never promoted. - A transform takes one or more input datasets and produces one output dataset through a list of steps.
- A row (
fv_plan::Row) is an ordered list of(column, value)pairs, with values from the value dialect: numbers, strings, booleans, null, lists and objects. Rows are dynamically typed so that a pipeline can be authored without a schema. - Expectations on an output dataset (row counts, non-null columns, freshness) are checked after its transform runs. Any failure blocks promotion.
The step families
Inline row steps run row by row and can be chained freely inside one transform:
Expressions are the value dialect. When the executor runs on DataFusion it translates each expression into a native DataFusion expression wherever the semantics are provably identical (so it vectorises and can be pushed down to a source), and evaluates it per row otherwise; the two paths are checked against each other row for row.
Reshapes change the shape of the data and run as relational operations on DataFusion. Their inputs are the transform's inputs in order (a join's left and right):
Spatial SQL (ST_* from geodatafusion) is available to reshapes, and an lttb aggregate
downsamples time series while keeping their shape.
Compute steps hand the rows to FusionVault Kinetics: a WebAssembly component or a container process, selected from a registry by id and version:
The executor types the input rows into Arrow batches using the transform's declared signature, runs it, and continues with its output rows. See Runtimes below for how to enable this.
One transform, one reshape or compute step. A transform runs its inline steps, then at most one reshape or compute step, then inline steps on that step's output — as a FusionVault Streams stage does, so the same pipeline means the same thing in both. A second reshape or compute step is refused (chain transforms instead), and so is an inline step before a reshape or compute step with several inputs, which could not say which input it applies to.
Sources read files into rows for the host: read_csv, read_ndjson, read_parquet.
Running steps without a build
The executors are plain functions, so a host can run a step over rows directly. This is how a pipeline builder previews a step, and how a streaming worker applies a transform to each message:
use ;
let mut row = new;
row.set;
row.set;
let step = Filter ;
let kept = apply_step.unwrap;
assert_eq!;
apply_steps chains steps; apply_steps_isolating does the same but drops only the rows that
fail rather than the whole batch, which is what a stream wants; validate_steps compiles every
expression up front so a bad one fails a deployment instead of every message. inline::parse
decodes a step from its JSON form, and reshape::run executes a reshape step over named inputs.
Runtimes
Two seams let a host decide where the heavy work happens. Both default to something that works on one machine.
Compute steps need a runtime for the transforms they reference. KineticsRunner wraps a
fv_compute::Runtime; the host registers whichever backends it ships, and this crate links none
of them:
use Arc;
use Runtime;
use WasmBackend;
use ;
let runtime = new;
let kinetics = new;
let record = new
.input
.compute
.run
.await;
KineticsRunner::ensure_loadable(&step) resolves and loads a step's transform without running
it, so a missing or broken transform is reported when a pipeline or stream is configured.
Reshapes generate SQL that a SqlRunner executes. The default LocalSqlRunner runs it on
an in-process DataFusion session; a host with a cluster implements the trait once (register the
inputs, run the SQL, return rows) and passes it with .sql(&runner). The op logic and the SQL
are the same either way.
Precision and conformance
Every step kind and every build outcome is checked against the published
fv-contract conformance vectors, shared with the
TypeScript control plane, so an executor change cannot silently diverge from what the builder
promises. The DataFusion expression path is checked row for row against the row-wise executor.
Develop
&& &&
Apache-2.0. Plain semver; tags vX.Y.Z at the published commit; see CHANGELOG.md.