fv-value-datafusion 0.2.0

The fv-value expression dialect as DataFusion expressions: exact translation for pushdown and vectorized evaluation, with a per-row UDF fallback.
Documentation

fv-value-datafusion

The fv-value expression dialect as DataFusion expressions.

A dialect filter such as status == 'ACTIVE' && speed > 40 becomes a DataFusion Expr that the planner can push down to the source (a Parquet row-group, an OpenSearch query), vectorize over Arrow batches, and reorder with the rest of the plan. Anything the translator cannot express with identical semantics runs as a per-row UDF fallback instead, so nothing changes meaning — only what gets pushed down.

use std::sync::Arc;
use datafusion::prelude::*;
use fv_value_datafusion::Translator;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("t", "rows.csv", CsvReadOptions::new()).await?;
    let df = ctx.table("t").await?;

    let expr = fv_value::compile("status == 'ACTIVE' && speed * 1.852 > 100").unwrap();
    let schema = Arc::new(df.schema().as_arrow().clone());
    let translated = Translator::new(schema).translate_or_fallback(&expr).expect("typeable");
    df.filter(translated.expr)?.show().await
}

The rule: exact or fallback

The dialect and SQL disagree in places:

  • strict booleans versus three-valued logic;
  • null < 1 is false, not NULL;
  • true == 1 is false;
  • / 0 is an error, and so is a NaN or infinite computed number;
  • NaN never orders or equals;
  • -0.0 == 0.0, where Arrow's total order puts -0 below +0.

The translator only emits a DataFusion expression when it reproduces the dialect's answer for every row. It wraps in CASE and guards where needed. It raises the dialect's run-time errors through an error-raising UDF inside a CASE branch (DataFusion evaluates a branch only for the rows that select it), and through fv_finite, one pass that returns its input array unchanged or names what overflowed. Where that is not possible, that sub-expression becomes a per-row fv_eval_… UDF over its columns, and the rest of the expression still translates. A column whose type has no value in the dialect (binary, decimal, temporal) is refused by name when the plan is built.

The guarantee is tested:

  • The conformance vectors, replayed natively and through DataFusion with their literals bound as columns (every subset of them, so a translation that applies only to a literal argument runs both ways). Every replay must agree, in value or in error message.
  • Float columns holding NaN, ±∞ and −0, run through every operator and function.
  • A mutation check for every guard.

Filters: predicate mode

A filter is the case that matters most for pushdown, and it has an extra freedom: DataFusion drops a row whose predicate is NULL, which is exactly what the dialect does with a comparison involving null (null < 1 is false). translate_predicate uses that to emit comparisons and &&/|| chains plainly. Without further knowledge, a float comparison gains a NaN bound (speed > 40 AND speed <= +inf, since Arrow ranks NaN above every number), which a source can still prune on. Options { assume_finite: true } drops it. That is sound when the rows hold no NaN and no infinity: documents from OpenSearch, or a batch whose float columns all_finite has just checked. Together:

use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use fv_value_datafusion::{Options, Translator};

let schema = Arc::new(Schema::new(vec![
    Field::new("status", DataType::Utf8, true),
    Field::new("speed", DataType::Float64, true),
]));
let expr = fv_value::compile("status == 'ACTIVE' && speed > 40").unwrap();
let t = Translator::new(schema).with_options(Options { assume_finite: true });
let p = t.translate_predicate(&expr).unwrap();
assert_eq!(p.expr.to_string(), "status = Utf8(\"ACTIVE\") AND speed > Float64(40)");

That is the form datafusion-opensearch turns into a term and a range query on the server — proven end to end in tests/opensearch_pushdown.rs. The guards stay wherever NULL and false must differ (!, ?:, !=, == on a boolean result) or where a side can fail, and DataFusion splits ANDs into separate filters, so the pushable conjuncts still push down when one needs a guard or a fallback.

Speed

The same filter, status == 'ACTIVE' && roundTo(speed * 1.852, 1) > 100, over 1 000 000 rows on all cores, release build (cargo run --release --example bench_paths; three runs):

  • translated, vectorised Arrow kernels, nothing assumed: 5.2–5.9 ms. The NaN bound and the overflow checks a column that may hold NaN or ±∞ needs.
  • translated with assume_finite: 3.7–4.5 ms.
  • the per-row UDF fallback inside DataFusion: 15.7–17.1 ms.
  • native per-row evaluation over row slices (the path a streaming consumer takes): 161–163 ms.

Same rows out of all four. The translated path is also the one that pushes down, so on a real source it is faster still, by however much never leaves the store. Per shape, per batch: cargo bench --bench translated.

API

  • Translator::new(schema) — column types decide what is translatable.
  • translate(&expr) — exact or nothing; translate_or_fallback(&expr) — exact where possible, UDF where not; exact on the result says which.
  • translate_predicate(&expr) / translate_predicate_or_fallback(&expr) — the same for a filter.
  • with_options(Options { assume_finite }) — what the integrator knows about the rows.
  • all_finite(&array) — whether a float column holds only finite values (one pass), the check that makes assume_finite sound for a batch.
  • fallback(&expr, return_type, &schema) — the per-row UDF on its own; refuses a column with no dialect value.
  • cell_to_value(&array, row) — one Arrow cell as a dialect value, or an error for a type with no dialect value.

Apache-2.0.