fv-value-datafusion
The fv-value expression dialect as
DataFusion expressions.
A dialect filter such as status == 'ACTIVE' && speed > 40 becomes a DataFusion Expr that the
planner can push down to the source (a Parquet row-group, an OpenSearch query), vectorize over Arrow
batches, and reorder with the rest of the plan. Anything the translator cannot express with
identical semantics runs as a per-row UDF fallback instead, so nothing changes meaning — only
what gets pushed down.
use Arc;
use *;
use Translator;
async
The rule: exact or fallback
The dialect and SQL disagree in places:
- strict booleans versus three-valued logic;
null < 1isfalse, notNULL;true == 1isfalse;/ 0is an error, and so is a NaN or infinite computed number;- NaN never orders or equals;
-0.0 == 0.0, where Arrow's total order puts-0below+0.
The translator only emits a DataFusion expression when it reproduces the dialect's answer for every
row. It wraps in CASE and guards where needed. It raises the dialect's run-time errors through an
error-raising UDF inside a CASE branch (DataFusion evaluates a branch only for the rows that
select it), and through fv_finite, one pass that returns its input array unchanged or names what
overflowed. Where that is not possible, that sub-expression becomes a per-row fv_eval_… UDF over
its columns, and the rest of the expression still translates. A column whose type has no value in
the dialect (binary, decimal, temporal) is refused by name when the plan is built.
The guarantee is tested:
- The conformance vectors, replayed natively and through DataFusion with their literals bound as columns (every subset of them, so a translation that applies only to a literal argument runs both ways). Every replay must agree, in value or in error message.
- Float columns holding NaN, ±∞ and −0, run through every operator and function.
- A mutation check for every guard.
Filters: predicate mode
A filter is the case that matters most for pushdown, and it has an extra freedom: DataFusion drops
a row whose predicate is NULL, which is exactly what the dialect does with a comparison involving
null (null < 1 is false). translate_predicate uses that to emit comparisons and &&/||
chains plainly. Without further knowledge, a float comparison gains a NaN bound (speed > 40 AND speed <= +inf, since Arrow ranks NaN above every number), which a source can still prune on.
Options { assume_finite: true } drops it. That is sound when the rows hold no NaN and no infinity:
documents from OpenSearch, or a batch whose float columns all_finite has just checked. Together:
use Arc;
use ;
use ;
let schema = new;
let expr = compile.unwrap;
let t = new.with_options;
let p = t.translate_predicate.unwrap;
assert_eq!;
That is the form datafusion-opensearch turns into a term and a range query on the server —
proven end to end in tests/opensearch_pushdown.rs. The guards stay wherever NULL and false must
differ (!, ?:, !=, == on a boolean result) or where a side can fail, and DataFusion splits
ANDs into separate filters, so the pushable conjuncts still push down when one needs a guard or
a fallback.
Geometry columns
A GeoArrow column says what it holds in its field's ARROW:extension:name:
geoarrow.point, separated or interleaved: a row is the dialect's point[x, y], or[x, y, z]with an altitude. Anmis not a coordinate.geoarrow.wkborogc.wkb: a row is its geometry as GeoJSON, with its altitude kept.
A geo call whose other arguments are literals runs as one call over such a column:
geoWithin(points, polygon),geoContains(polygon, points), andgeoIntersectseither way;geoContains(polygons, point),geoWithin(point, polygons), andgeoIntersectseither way, over a WKB column;geoWithinVolume(footprint, points, alt, floor, ceiling);distance3dMfrom the column to a literal point.
The literal is read once, and each row is read straight from the column: a point's coordinates, or a row's WKB in place. A row the dialect would not read as a plain point or polygon goes to the dialect, so its answer, or its error, is its own. A point is placed against a polygon by one exact rule, row by row and over the column alike.
use HashMap;
use Arc;
use ;
use Translator;
let xy = from;
let pos = new.with_metadata;
let zone = r#"{"type":"Polygon","coordinates":[[[150,-35],[152,-35],[152,-33],[150,-33],[150,-35]]]}"#;
let expr = compile.unwrap;
let t = new.translate.unwrap;
assert!;
Per row, over 65,536 rows (cargo bench --bench geo):
| call | over the column | row by row |
|---|---|---|
geoWithin, a GeoArrow point column |
27 ns | 158 ns (207 ns as text) |
geoWithin, a WKB point column |
30 ns | 851 ns |
geoContains, a WKB polygon column |
28 ns | 2,175 ns |
geoWithinVolume, points with an altitude |
19 ns | 253 ns (297 ns as text) |
distance3dM, points with an altitude |
49 ns | 355 ns (369 ns as text) |
Speed
The same filter, status == 'ACTIVE' && roundTo(speed * 1.852, 1) > 100, over 1 000 000 rows on
all cores, release build (cargo run --release --example bench_paths; three runs):
- translated, vectorised Arrow kernels, nothing assumed: 5.2–5.9 ms. The NaN bound and the overflow checks a column that may hold NaN or ±∞ needs.
- translated with
assume_finite: 3.7–4.5 ms. - the per-row UDF fallback inside DataFusion: 15.7–17.1 ms.
- native per-row evaluation over row slices (the path a streaming consumer takes): 161–163 ms.
Same rows out of all four. The translated path is also the one that pushes down, so on a real
source it is faster still, by however much never leaves the store. Per shape, per batch:
cargo bench --bench translated.
API
Translator::new(schema)— column types decide what is translatable.translate(&expr)— exact or nothing;translate_or_fallback(&expr)— exact where possible, UDF where not;exacton the result says which.translate_predicate(&expr)/translate_predicate_or_fallback(&expr)— the same for a filter.with_options(Options { assume_finite })— what the integrator knows about the rows.all_finite(&array)— whether a float column holds only finite values (one pass), the check that makesassume_finitesound for a batch.fallback(&expr, return_type, &schema)— the per-row UDF on its own; refuses a column with no dialect value.cell_to_value(&array, row)— one Arrow cell as a dialect value, or an error for a type with no dialect value.Cells::of_field(&field, &array)— a column read as dialect values, by its type or its GeoArrow extension;field_has_value_form(&field)says whether it has one.
Apache-2.0.