fv-value-datafusion 0.1.0

The fv-value expression dialect as DataFusion expressions: exact translation for pushdown and vectorized evaluation, with a per-row UDF fallback.
Documentation

fv-value-datafusion

The fv-value expression dialect as DataFusion expressions.

A dialect filter such as status == 'ACTIVE' && speed > 40 becomes a DataFusion Expr that the planner can push down to the source (a Parquet row-group, an OpenSearch query), vectorize over Arrow batches, and reorder with the rest of the plan. Anything the translator cannot express with identical semantics runs as a per-row UDF fallback instead, so nothing changes meaning — only what gets pushed down.

use std::sync::Arc;
use datafusion::prelude::*;
use fv_value_datafusion::Translator;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("t", "rows.csv", CsvReadOptions::new()).await?;
    let df = ctx.table("t").await?;

    let expr = fv_value::compile("status == 'ACTIVE' && speed * 1.852 > 100").unwrap();
    let schema = Arc::new(df.schema().as_arrow().clone());
    let translated = Translator::new(schema).translate_or_fallback(&expr).expect("typeable");
    df.filter(translated.expr)?.show().await
}

The rule: exact or fallback

The dialect and SQL disagree in places — strict booleans versus three-valued logic, null < 1 is false not NULL, true == 1 is false, / 0 is an error, round is half-up, NaN never orders or equals, -0.0 == 0.0. The translator only emits a DataFusion expression when it reproduces the dialect's answer for every row, wrapping in CASE and guards where needed, and raising the dialect's run-time errors through an error-raising UDF inside a CASE branch (DataFusion evaluates a branch only for the rows that select it). Where that is not possible, that sub-expression becomes a per-row fv_eval_… UDF over its columns and the rest of the expression still translates.

The guarantee is tested: the contract's conformance vectors are evaluated both natively and through DataFusion, and every translated vector must agree.

API

  • Translator::new(schema) — column types decide what is translatable.
  • translate(&expr) -> Result<Translation, Unsupported> — exact or nothing.
  • translate_or_fallback(&expr) -> Translation — exact where possible, UDF where not; exact says which.
  • fallback(&expr, return_type) — the per-row UDF on its own.

Apache-2.0.