arrow only.Expand description
Opt-in Apache Arrow columnar record path (feature arrow, RFC 0002 / #375).
This is the escape hatch described in #324: a columnar representation a
connector may produce or consume at the page boundary, additive to and
coexisting with the default serde_json::Value row model. It touches neither
StreamPage nor any existing connector — an Arrow-native
source overrides Source::stream_batches and
an Arrow-native sink overrides
Sink::write_batch_columnar; the pipeline
uses the columnar path only when both sides advertise support (and no
Value-shaped stage needs to observe the records), so a
parquet → parquet chain never materializes Value.
The RecordBatch ↔ Value conversions here are the single source of truth for
the shim (they match faucet-transform-sql’s shovel byte-for-byte, incl.
with_explicit_nulls(true) so an explicit-null field round-trips as
"key": null rather than being silently dropped — audit #321 H6).
Structs§
- Columnar
Page - A page of records in columnar (Arrow) form, the columnar analogue of
StreamPage.
Functions§
- infer_
arrow_ schema - Infer an Arrow
Schemafrom a slice of JSON records (each a JSON object). - record_
batch_ to_ values - Decode a
RecordBatchinto JSON objects (one per row). - schema_
eq - Compare two schemas for field-level equality (name + data-type + nullability).
- values_
to_ record_ batch - Encode a slice of JSON records into a single
RecordBatchagainstschema. - values_
to_ record_ batch_ inferred - Convenience: infer the schema from
recordsand encode them into a batch.