fv-value-datafusion 0.3.0

The fv-value expression dialect as DataFusion expressions: exact translation for pushdown and vectorized evaluation, with a per-row UDF fallback.
Documentation
# fv-value-datafusion

The [`fv-value`](https://crates.io/crates/fv-value) expression dialect as
[DataFusion](https://datafusion.apache.org/) expressions.

A dialect filter such as `status == 'ACTIVE' && speed > 40` becomes a DataFusion `Expr` that the
planner can push down to the source (a Parquet row-group, an OpenSearch query), vectorize over Arrow
batches, and reorder with the rest of the plan. Anything the translator cannot express with
*identical* semantics runs as a per-row UDF fallback instead, so nothing changes meaning — only
what gets pushed down.

```rust,no_run
use std::sync::Arc;
use datafusion::prelude::*;
use fv_value_datafusion::Translator;

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    ctx.register_csv("t", "rows.csv", CsvReadOptions::new()).await?;
    let df = ctx.table("t").await?;

    let expr = fv_value::compile("status == 'ACTIVE' && speed * 1.852 > 100").unwrap();
    let schema = Arc::new(df.schema().as_arrow().clone());
    let translated = Translator::new(schema).translate_or_fallback(&expr).expect("typeable");
    df.filter(translated.expr)?.show().await
}
```

## The rule: exact or fallback

The dialect and SQL disagree in places:
- strict booleans versus three-valued logic;
- `null < 1` is `false`, not `NULL`;
- `true == 1` is `false`;
- `/ 0` is an error, and so is a NaN or infinite computed number;
- NaN never orders or equals;
- `-0.0 == 0.0`, where Arrow's total order puts `-0` below `+0`.

The translator only emits a DataFusion expression when it reproduces the dialect's answer for every
row. It wraps in `CASE` and guards where needed. It raises the dialect's run-time errors through an
error-raising UDF inside a `CASE` branch (DataFusion evaluates a branch only for the rows that
select it), and through `fv_finite`, one pass that returns its input array unchanged or names what
overflowed. Where that is not possible, that sub-expression becomes a per-row `fv_eval_…` UDF over
its columns, and the rest of the expression still translates. A column whose type has no value in
the dialect (binary, decimal, temporal) is refused by name when the plan is built.

The guarantee is tested:
- **The conformance vectors**, replayed natively and through DataFusion with their literals bound
  as columns (every subset of them, so a translation that applies only to a literal argument runs
  both ways). Every replay must agree, in value or in error message.
- **Float columns holding NaN, ±∞ and −0**, run through every operator and function.
- **A mutation check for every guard.**

## Filters: predicate mode

A filter is the case that matters most for pushdown, and it has an extra freedom: DataFusion drops
a row whose predicate is NULL, which is exactly what the dialect does with a comparison involving
null (`null < 1` is `false`). `translate_predicate` uses that to emit comparisons and `&&`/`||`
chains plainly. Without further knowledge, a float comparison gains a NaN bound (`speed > 40 AND
speed <= +inf`, since Arrow ranks NaN above every number), which a source can still prune on.
`Options { assume_finite: true }` drops it. That is sound when the rows hold no NaN and no infinity:
documents from OpenSearch, or a batch whose float columns `all_finite` has just checked. Together:

```rust
use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Schema};
use fv_value_datafusion::{Options, Translator};

let schema = Arc::new(Schema::new(vec![
    Field::new("status", DataType::Utf8, true),
    Field::new("speed", DataType::Float64, true),
]));
let expr = fv_value::compile("status == 'ACTIVE' && speed > 40").unwrap();
let t = Translator::new(schema).with_options(Options { assume_finite: true });
let p = t.translate_predicate(&expr).unwrap();
assert_eq!(p.expr.to_string(), "status = Utf8(\"ACTIVE\") AND speed > Float64(40)");
```

That is the form `datafusion-opensearch` turns into a `term` and a `range` query on the server —
proven end to end in `tests/opensearch_pushdown.rs`. The guards stay wherever NULL and false must
differ (`!`, `?:`, `!=`, `==` on a boolean result) or where a side can fail, and DataFusion splits
`AND`s into separate filters, so the pushable conjuncts still push down when one needs a guard or
a fallback.

## Geometry columns

A GeoArrow column says what it holds in its field's `ARROW:extension:name`:
- `geoarrow.point`, separated or interleaved: a row is the dialect's point `[x, y]`, or
  `[x, y, z]` with an altitude. An `m` is not a coordinate.
- `geoarrow.wkb` or `ogc.wkb`: a row is its geometry as GeoJSON, with its altitude kept.

A geo call whose other arguments are literals runs as one call over such a column:
- `geoWithin(points, polygon)`, `geoContains(polygon, points)`, and `geoIntersects` either way;
- `geoContains(polygons, point)`, `geoWithin(point, polygons)`, and `geoIntersects` either way,
  over a WKB column;
- `geoWithinVolume(footprint, points, alt, floor, ceiling)`;
- `distance3dM` from the column to a literal point.

The literal is read once, and each row is read straight from the column: a point's coordinates,
or a row's WKB in place. A row the dialect would not read as a plain point or polygon goes to the
dialect, so its answer, or its error, is its own. A point is placed against a polygon by one exact
rule, row by row and over the column alike.

```rust
use std::collections::HashMap;
use std::sync::Arc;
use datafusion::arrow::datatypes::{DataType, Field, Fields, Schema};
use fv_value_datafusion::Translator;

let xy = Fields::from(vec![
    Field::new("x", DataType::Float64, false),
    Field::new("y", DataType::Float64, false),
]);
let pos = Field::new("pos", DataType::Struct(xy), true).with_metadata(HashMap::from([(
    "ARROW:extension:name".to_string(),
    "geoarrow.point".to_string(),
)]));
let zone = r#"{"type":"Polygon","coordinates":[[[150,-35],[152,-35],[152,-33],[150,-33],[150,-35]]]}"#;
let expr = fv_value::compile(&format!("geoWithin(pos, '{zone}')")).unwrap();
let t = Translator::new(Arc::new(Schema::new(vec![pos]))).translate(&expr).unwrap();
assert!(t.exact && t.expr.to_string().starts_with("fv_geo_point_column_"));
```

Per row, over 65,536 rows (`cargo bench --bench geo`):

| call | over the column | row by row |
|---|---|---|
| `geoWithin`, a GeoArrow point column | 27 ns | 158 ns (207 ns as text) |
| `geoWithin`, a WKB point column | 30 ns | 851 ns |
| `geoContains`, a WKB polygon column | 28 ns | 2,175 ns |
| `geoWithinVolume`, points with an altitude | 19 ns | 253 ns (297 ns as text) |
| `distance3dM`, points with an altitude | 49 ns | 355 ns (369 ns as text) |

## Speed

The same filter, `status == 'ACTIVE' && roundTo(speed * 1.852, 1) > 100`, over 1 000 000 rows on
all cores, release build (`cargo run --release --example bench_paths`; three runs):

- translated, vectorised Arrow kernels, nothing assumed: **5.2–5.9 ms**. The NaN bound and the
  overflow checks a column that may hold NaN or ±∞ needs.
- translated with `assume_finite`: **3.7–4.5 ms**.
- the per-row UDF fallback inside DataFusion: 15.7–17.1 ms.
- native per-row evaluation over row slices (the path a streaming consumer takes): 161–163 ms.

Same rows out of all four. The translated path is also the one that pushes down, so on a real
source it is faster still, by however much never leaves the store. Per shape, per batch:
`cargo bench --bench translated`.

## API

- `Translator::new(schema)` — column types decide what is translatable.
- `translate(&expr)` — exact or nothing; `translate_or_fallback(&expr)` — exact where possible,
  UDF where not; `exact` on the result says which.
- `translate_predicate(&expr)` / `translate_predicate_or_fallback(&expr)` — the same for a filter.
- `with_options(Options { assume_finite })` — what the integrator knows about the rows.
- `all_finite(&array)` — whether a float column holds only finite values (one pass), the check
  that makes `assume_finite` sound for a batch.
- `fallback(&expr, return_type, &schema)` — the per-row UDF on its own; refuses a column with no
  dialect value.
- `cell_to_value(&array, row)` — one Arrow cell as a dialect value, or an error for a type with
  no dialect value.
- `Cells::of_field(&field, &array)` — a column read as dialect values, by its type or its GeoArrow
  extension; `field_has_value_form(&field)` says whether it has one.

Apache-2.0.