fv-value-datafusion
The fv-value expression dialect as
DataFusion expressions.
A dialect filter such as status == 'ACTIVE' && speed > 40 becomes a DataFusion Expr that the
planner can push down to the source (a Parquet row-group, an OpenSearch query), vectorize over Arrow
batches, and reorder with the rest of the plan. Anything the translator cannot express with
identical semantics runs as a per-row UDF fallback instead, so nothing changes meaning — only
what gets pushed down.
use Arc;
use *;
use Translator;
async
The rule: exact or fallback
The dialect and SQL disagree in places — strict booleans versus three-valued logic, null < 1 is
false not NULL, true == 1 is false, / 0 is an error, round is half-up, NaN never orders
or equals, -0.0 == 0.0. The translator only emits a DataFusion expression when it reproduces the
dialect's answer for every row, wrapping in CASE and guards where needed, and raising the
dialect's run-time errors through an error-raising UDF inside a CASE branch (DataFusion evaluates
a branch only for the rows that select it). Where that is not possible, that sub-expression becomes
a per-row fv_eval_… UDF over its columns and the rest of the expression still translates.
The guarantee is tested: the contract's conformance vectors are evaluated both natively and through DataFusion, and every translated vector must agree.
Filters: predicate mode
A filter is the case that matters most for pushdown, and it has an extra freedom: DataFusion drops
a row whose predicate is NULL, which is exactly what the dialect does with a comparison involving
null (null < 1 is false). translate_predicate uses that to emit comparisons and &&/||
chains plainly, and Options { assume_no_nan: true } — true for any JSON-derived source, since
JSON cannot express NaN — drops the NaN guards. Together:
use Arc;
use ;
use ;
let schema = new;
let expr = compile.unwrap;
let t = new.with_options;
let p = t.translate_predicate.unwrap;
assert_eq!;
That is the form datafusion-opensearch turns into a term and a range query on the server —
proven end to end in tests/opensearch_pushdown.rs. The guards stay wherever NULL and false must
differ (!, ?:, !=, == on a boolean result) or where a side can fail, and DataFusion splits
ANDs into separate filters, so the pushable conjuncts still push down when one needs a guard or
a fallback.
Speed
The same filter over 1 000 000 rows, release build (cargo run --release --example bench_paths):
- translated, vectorized Arrow kernels: 5 ms
- the per-row UDF fallback inside DataFusion: 19 ms
- native per-row evaluation over row slices (the streaming path): 157 ms
Same rows out of all three. The translated path is the one that also pushes down, so on a real source it is faster still by however much never leaves the store.
API
Translator::new(schema)— column types decide what is translatable.translate(&expr)— exact or nothing;translate_or_fallback(&expr)— exact where possible, UDF where not;exacton the result says which.translate_predicate(&expr)/translate_predicate_or_fallback(&expr)— the same for a filter.with_options(Options { assume_no_nan })— what the integrator knows about the rows.fallback(&expr, return_type)— the per-row UDF on its own.
Apache-2.0.