# fv-plan
fv-plan executes FusionVault pipelines on [Apache DataFusion](https://datafusion.apache.org/).
Give it a pipeline definition and the input rows; it runs every transform in dependency order,
evaluates the output datasets' expectations, and hands back a build record with the promoted
outputs. It reads no databases and writes none: the host feeds rows in and takes rows out, which
is what lets the same executor serve batch builds, previews, and streams.
## A build in twenty lines
```rust
use fv_plan::{Build, Row};
use serde_json::json;
// A pipeline: transforms over datasets, each a list of steps. This one keeps fast vehicles and
// adds a converted speed column.
let pipeline = json!({
"transforms": [{
"inputs": ["positions"],
"output": "fast",
"steps": [
{ "op": "filter", "expression": "speed > 40" },
{ "op": "applyExpression", "column": "kmh", "expression": "round(speed * 1.609)" },
]
}]
});
let datasets = vec![json!({ "apiName": "positions" }), json!({ "apiName": "fast" })];
let mut slow = Row::new();
slow.set("id", "a".into());
slow.set("speed", 30.0.into());
let mut fast = Row::new();
fast.set("id", "b".into());
fast.set("speed", 60.0.into());
let record = tokio::runtime::Runtime::new().unwrap().block_on(async {
Build::new(&pipeline, &datasets)
.input("positions", vec![slow, fast])
.run()
.await
});
assert_eq!(record.status, "SUCCEEDED", "{:?}", record.error);
let out = &record.outputs["fast"];
assert_eq!(out.len(), 1);
assert_eq!(out[0].get("kmh"), 97.0.into());
```
A build never panics. Its record's `status` is `SUCCEEDED`, `BLOCKED` (an expectation failed, so
the previous output stays current) or `FAILED` (with the reason in `error`).
## The model
- A **dataset** is a named table (`apiName`). Inputs come from the host; outputs are produced by
transforms. A dataset marked `"materialized": false` is an intermediate that is never promoted.
- A **transform** takes one or more input datasets and produces one output dataset through a
list of **steps**.
- A **row** (`fv_plan::Row`) is an ordered list of `(column, value)` pairs, with values from the
[value dialect](https://crates.io/crates/fv-value): numbers, strings, booleans, null, lists and
objects. Rows are dynamically typed so that a pipeline can be authored without a schema.
- **Expectations** on an output dataset (row counts, non-null columns, freshness) are checked
after its transform runs. Any failure blocks promotion.
## The step families
**Inline row steps** run row by row and can be chained freely inside one transform:
```json
{ "op": "select", "columns": ["id", "speed"] }
{ "op": "rename", "mapping": { "speed": "velocity" } }
{ "op": "drop", "columns": ["debug"] }
{ "op": "filter", "expression": "status == 'ACTIVE' && speed > 40" }
{ "op": "applyExpression", "column": "kmh", "expression": "speed * 1.609" }
```
Expressions are the value dialect. When the executor runs on DataFusion it translates each
expression into a native DataFusion expression wherever the semantics are provably identical
(so it vectorises and can be pushed down to a source), and evaluates it per row otherwise; the
two paths are checked against each other row for row.
**Reshapes** change the shape of the data and run as relational operations on DataFusion. Their
inputs are the transform's inputs in order (a join's left and right):
```json
{ "op": "join", "on": ["customerId"], "how": "inner" }
{ "op": "aggregate", "groupBy": ["region"], "aggregations": [{ "op": "sum", "column": "amount", "as": "total" }] }
{ "op": "window", "partitionBy": ["vehicle"], "orderBy": ["ts"], "functions": [{ "op": "row_number", "as": "n" }] }
{ "op": "union" }
```
Spatial SQL (`ST_*` from geodatafusion) is available to reshapes, and an `lttb` aggregate
downsamples time series while keeping their shape.
**Compute steps** hand the rows to
[FusionVault Kinetics](https://github.com/FusionVault/kinetics): a WebAssembly component or a
container process, selected from a registry by id and version:
```json
{ "op": "wasm", "ref": "spikeTotal@0.1.0" }
```
The executor types the input rows into Arrow batches using the transform's declared signature,
runs it, and continues with its output rows. See *Runtimes* below for how to enable this.
**One transform, one reshape or compute step.** A transform runs its inline steps, then at most
one reshape or compute step, then inline steps on that step's output — as a FusionVault Streams
stage does, so the same pipeline means the same thing in both. A second reshape or compute step is
refused (chain transforms instead), and so is an inline step before a reshape or compute step with
several inputs, which could not say which input it applies to.
**Sources** read files into rows for the host: `read_csv`, `read_ndjson`, `read_parquet`.
## Running steps without a build
The executors are plain functions, so a host can run a step over rows directly. This is how a
pipeline builder previews a step, and how a streaming worker applies a transform to each message:
```rust
use fv_plan::{apply_step, Row, Step};
let mut row = Row::new();
row.set("speed", 60.0.into());
row.set("status", "ACTIVE".into());
let step = Step::Filter { expression: "status == 'ACTIVE' && speed > 40".into() };
let kept = apply_step(&step, &[row]).unwrap();
assert_eq!(kept.len(), 1);
```
`apply_steps` chains steps; `apply_steps_isolating` does the same but drops only the rows that
fail rather than the whole batch, which is what a stream wants; `validate_steps` compiles every
expression up front so a bad one fails a deployment instead of every message. `inline::parse`
decodes a step from its JSON form, and `reshape::run` executes a reshape step over named inputs.
## Runtimes
Two seams let a host decide where the heavy work happens. Both default to something that works
on one machine.
**Compute steps** need a runtime for the transforms they reference. `KineticsRunner` wraps a
`fv_compute::Runtime`; the host registers whichever backends it ships, and this crate links none
of them:
```rust,ignore
use std::sync::Arc;
use fv_compute::Runtime;
use fv_compute_wasm::WasmBackend;
use fv_plan::{Build, KineticsRunner};
let runtime = Arc::new(
Runtime::builder()
.root("./transforms")
.backend(WasmBackend::new()?)
.build()?,
);
let kinetics = KineticsRunner::new(runtime);
let record = Build::new(&pipeline, &datasets)
.input("raw", rows)
.compute(&kinetics)
.run()
.await;
```
`KineticsRunner::ensure_loadable(&step)` resolves and loads a step's transform without running
it, so a missing or broken transform is reported when a pipeline or stream is configured.
**Reshapes** generate SQL that a `SqlRunner` executes. The default `LocalSqlRunner` runs it on
an in-process DataFusion session; a host with a cluster implements the trait once (register the
inputs, run the SQL, return rows) and passes it with `.sql(&runner)`. The op logic and the SQL
are the same either way.
## Precision and conformance
Every step kind and every build outcome is checked against the published
[`fv-contract`](https://crates.io/crates/fv-contract) conformance vectors, shared with the
TypeScript control plane, so an executor change cannot silently diverge from what the builder
promises. The DataFusion expression path is checked row for row against the row-wise executor.
## Develop
```bash
cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test
cargo publish --dry-run
```
Apache-2.0. Plain semver; tags `vX.Y.Z` at the published commit; see [CHANGELOG.md](CHANGELOG.md).