Crate fv_plan 

The DataFusion-backed transform executor.

Two execution paths: • inline row ops (select/rename/drop/filter/applyExpression) — dynamic-typed, row-wise, carrying the bespoke value dialect → executed in Rust via the frozen-vectored fv-value. Vectored against the published contract’s step-kind conformance vectors. • reshapes (join/aggregate/window/union) — cardinality/shape changes that need a relational engine → executed on DataFusion (SQL planner over a MemTable), with Value↔Arrow at the edges.

Governance (gate, lineage, expectations semantics, labels) stays in the control plane; this is execution only.

Re-exports

pub use build::run_build;

pub use build::run_build_with;

pub use build::run_build_with_native;

pub use build::Build;

pub use build::BuildRecord;

pub use build::ComputeRunner;

pub use build::ExpResult;

pub use build::NoComputeRuntime;

pub use compute::KineticsRunner;

pub use inline::apply_step;

pub use inline::apply_steps;

pub use inline::Step;

pub use inline::StepError;

pub use reshape::aggregate;

pub use reshape::join;

pub use reshape::union;

pub use reshape::window;

pub use reshape::Aggregation;

pub use reshape::Input;

pub use reshape::LocalSqlRunner;

pub use reshape::SqlRunner;

pub use reshape::WindowFn;

pub use row::Row;

Modules

build

Build orchestration — runs a pipeline’s transforms in dependency order and records the result.

compute

Compute steps: running a Kinetics transform from a pipeline.

convert

Row ⇄ Arrow conversion for the DataFusion execution path (reshapes). Column types are inferred from the values (real pipelines will pass declared schemas later); the value dialect stays dynamic.

decimate_udaf

lttb(ts, value, target) — server-side time-series decimation as a DataFusion aggregate Largest-Triangle-Three-Buckets keeps the points that shape the line (spikes inflections), so a million-row series aggregates to a render-ready target points in ONE query instead of shipping the rows. The maths is fv-util’s index-returning core — the same code the OSS history endpoint uses, so both surfaces decimate identically.

inline

Inline row-op executor. Executes a chain of row ops (select/rename/drop/filter/applyExpression) row-wise, delegating value compute to fv-value. Vectored against the published contract’s step-kind conformance vectors.

reshape

Reshape steps on DataFusion — the executors that genuinely need a relational engine (they change cardinality/shape, unlike the row-wise inline ops). This crate lands aggregate (group-by) on DataFusion’s SQL planner/executor via a MemTable; join/window/union follow. Value↔Arrow via convert. Semantics are SQL’s (null semantics: sum/avg/min/max ignore nulls, all-null group → null; count(col) = non-null count, count(*) = rows).

row

Ordered pipeline rows. Column order is significant (it becomes the output schema), so a row is an ordered list of (name, value) — not a sorted map. Values are fv_value::Value.

session

Shared DataFusion SessionContext factory for pipeline SQL execution.

sources

File-format sources — CSV / NDJSON / Parquet read natively by DataFusion into engine rows. No extra dependencies: DataFusion ships these readers. Feeds the funnel file_lake source and connector snapshots. Types are inferred by DataFusion and collapsed to the dialect’s dynamic Value via convert::batch_to_rows.

udf

The value dialect inside a DataFusion plan.

Functions

datafusion_smoke

Smoke check that DataFusion is wired (used by the crate’s integration test).

is_whole_transform_op

True if op is a whole-transform step (a reshape, raw sql, or a container) — i.e. NOT an inline row op. Such a step must be a transform’s only step. The single source for this classification.

now_ms

Milliseconds since the Unix epoch (the wall clock a build record is stamped with).