fv-value-datafusion 0.3.0

The fv-value expression dialect as DataFusion expressions: exact translation for pushdown and vectorized evaluation, with a per-row UDF fallback.
Documentation

fv-value-datafusion

The fv-value expression dialect as DataFusion expressions.

A dialect filter such as status == 'ACTIVE' && speed > 40 becomes a DataFusion Expr that the planner can push down to the source (a Parquet row-group, an OpenSearch query), vectorize over Arrow batches, and reorder with the rest of the plan. Anything the translator cannot express with identical semantics runs as a per-row UDF fallback instead, so nothing changes meaning — only what gets pushed down.

use std::sync::Arc;
use datafusion::prelude::*;
use fv_value_datafusion::Translator;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("t", "rows.csv", CsvReadOptions::new()).await?;
    let df = ctx.table("t").await?;

    let expr = fv_value::compile("status == 'ACTIVE' && speed * 1.852 > 100").unwrap();
    let schema = Arc::new(df.schema().as_arrow().clone());
    let translated = Translator::new(schema).translate_or_fallback(&expr).expect("typeable");
    df.filter(translated.expr)?.show().await
}

The rule: exact or fallback

The dialect and SQL disagree in places:

  • strict booleans versus three-valued logic;
  • null < 1 is false, not NULL;
  • true == 1 is false;
  • / 0 is an error, and so is a NaN or infinite computed number;
  • NaN never orders or equals;
  • -0.0 == 0.0, where Arrow's total order puts -0 below +0.

The translator only emits a DataFusion expression when it reproduces the dialect's answer for every row. It wraps in CASE and guards where needed. It raises the dialect's run-time errors through an error-raising UDF inside a CASE branch (DataFusion evaluates a branch only for the rows that select it), and through fv_finite, one pass that returns its input array unchanged or names what overflowed. Where that is not possible, that sub-expression becomes a per-row fv_eval_… UDF over its columns, and the rest of the expression still translates. A column whose type has no value in the dialect (binary, decimal, temporal) is refused by name when the plan is built.

The guarantee is tested:

  • The conformance vectors, replayed natively and through DataFusion with their literals bound as columns (every subset of them, so a translation that applies only to a literal argument runs both ways). Every replay must agree, in value or in error message.
  • Float columns holding NaN, ±∞ and −0, run through every operator and function.
  • A mutation check for every guard.

Filters: predicate mode

A filter is the case that matters most for pushdown, and it has an extra freedom: DataFusion drops a row whose predicate is NULL, which is exactly what the dialect does with a comparison involving null (null < 1 is false). translate_predicate uses that to emit comparisons and &&/|| chains plainly. Without further knowledge, a float comparison gains a NaN bound (speed > 40 AND speed <= +inf, since Arrow ranks NaN above every number), which a source can still prune on. Options { assume_finite: true } drops it. That is sound when the rows hold no NaN and no infinity: documents from OpenSearch, or a batch whose float columns all_finite has just checked. Together:

use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use fv_value_datafusion::{Options, Translator};

let schema = Arc::new(Schema::new(vec![
    Field::new("status", DataType::Utf8, true),
    Field::new("speed", DataType::Float64, true),
]));
let expr = fv_value::compile("status == 'ACTIVE' && speed > 40").unwrap();
let t = Translator::new(schema).with_options(Options { assume_finite: true });
let p = t.translate_predicate(&expr).unwrap();
assert_eq!(p.expr.to_string(), "status = Utf8(\"ACTIVE\") AND speed > Float64(40)");

That is the form datafusion-opensearch turns into a term and a range query on the server — proven end to end in tests/opensearch_pushdown.rs. The guards stay wherever NULL and false must differ (!, ?:, !=, == on a boolean result) or where a side can fail, and DataFusion splits ANDs into separate filters, so the pushable conjuncts still push down when one needs a guard or a fallback.

Geometry columns

A GeoArrow column says what it holds in its field's ARROW:extension:name:

  • geoarrow.point, separated or interleaved: a row is the dialect's point [x, y], or [x, y, z] with an altitude. An m is not a coordinate.
  • geoarrow.wkb or ogc.wkb: a row is its geometry as GeoJSON, with its altitude kept.

A geo call whose other arguments are literals runs as one call over such a column:

  • geoWithin(points, polygon), geoContains(polygon, points), and geoIntersects either way;
  • geoContains(polygons, point), geoWithin(point, polygons), and geoIntersects either way, over a WKB column;
  • geoWithinVolume(footprint, points, alt, floor, ceiling);
  • distance3dM from the column to a literal point.

The literal is read once, and each row is read straight from the column: a point's coordinates, or a row's WKB in place. A row the dialect would not read as a plain point or polygon goes to the dialect, so its answer, or its error, is its own. A point is placed against a polygon by one exact rule, row by row and over the column alike.

use std::collections::HashMap;
use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Fields, Schema};
use fv_value_datafusion::Translator;

let xy = Fields::from(vec![
    Field::new("x", DataType::Float64, false),
    Field::new("y", DataType::Float64, false),
]);
let pos = Field::new("pos", DataType::Struct(xy), true).with_metadata(HashMap::from([(
    "ARROW:extension:name".to_string(),
    "geoarrow.point".to_string(),
)]));
let zone = r#"{"type":"Polygon","coordinates":[[[150,-35],[152,-35],[152,-33],[150,-33],[150,-35]]]}"#;
let expr = fv_value::compile(&format!("geoWithin(pos, '{zone}')")).unwrap();
let t = Translator::new(Arc::new(Schema::new(vec![pos]))).translate(&expr).unwrap();
assert!(t.exact && t.expr.to_string().starts_with("fv_geo_point_column_"));

Per row, over 65,536 rows (cargo bench --bench geo):

call over the column row by row
geoWithin, a GeoArrow point column 27 ns 158 ns (207 ns as text)
geoWithin, a WKB point column 30 ns 851 ns
geoContains, a WKB polygon column 28 ns 2,175 ns
geoWithinVolume, points with an altitude 19 ns 253 ns (297 ns as text)
distance3dM, points with an altitude 49 ns 355 ns (369 ns as text)

Speed

The same filter, status == 'ACTIVE' && roundTo(speed * 1.852, 1) > 100, over 1 000 000 rows on all cores, release build (cargo run --release --example bench_paths; three runs):

  • translated, vectorised Arrow kernels, nothing assumed: 5.2–5.9 ms. The NaN bound and the overflow checks a column that may hold NaN or ±∞ needs.
  • translated with assume_finite: 3.7–4.5 ms.
  • the per-row UDF fallback inside DataFusion: 15.7–17.1 ms.
  • native per-row evaluation over row slices (the path a streaming consumer takes): 161–163 ms.

Same rows out of all four. The translated path is also the one that pushes down, so on a real source it is faster still, by however much never leaves the store. Per shape, per batch: cargo bench --bench translated.

API

  • Translator::new(schema) — column types decide what is translatable.
  • translate(&expr) — exact or nothing; translate_or_fallback(&expr) — exact where possible, UDF where not; exact on the result says which.
  • translate_predicate(&expr) / translate_predicate_or_fallback(&expr) — the same for a filter.
  • with_options(Options { assume_finite }) — what the integrator knows about the rows.
  • all_finite(&array) — whether a float column holds only finite values (one pass), the check that makes assume_finite sound for a batch.
  • fallback(&expr, return_type, &schema) — the per-row UDF on its own; refuses a column with no dialect value.
  • cell_to_value(&array, row) — one Arrow cell as a dialect value, or an error for a type with no dialect value.
  • Cells::of_field(&field, &array) — a column read as dialect values, by its type or its GeoArrow extension; field_has_value_form(&field) says whether it has one.

Apache-2.0.