Skip to main content

Module columnar

Module columnar 

Source
Available on crate feature arrow only.
Expand description

Opt-in Apache Arrow columnar record path (feature arrow, RFC 0002 / #375).

This is the escape hatch described in #324: a columnar representation a connector may produce or consume at the page boundary, additive to and coexisting with the default serde_json::Value row model. It touches neither StreamPage nor any existing connector — an Arrow-native source overrides Source::stream_batches and an Arrow-native sink overrides Sink::write_batch_columnar; the pipeline uses the columnar path only when both sides advertise support (and no Value-shaped stage needs to observe the records), so a parquet → parquet chain never materializes Value.

The RecordBatch ↔ Value conversions here are the single source of truth for the shim (they match faucet-transform-sql’s shovel byte-for-byte, incl. with_explicit_nulls(true) so an explicit-null field round-trips as "key": null rather than being silently dropped — audit #321 H6).

Structs§

ColumnarPage
A page of records in columnar (Arrow) form, the columnar analogue of StreamPage.

Functions§

infer_arrow_schema
Infer an Arrow Schema from a slice of JSON records (each a JSON object).
record_batch_to_values
Decode a RecordBatch into JSON objects (one per row).
schema_eq
Compare two schemas for field-level equality (name + data-type + nullability).
values_to_record_batch
Encode a slice of JSON records into a single RecordBatch against schema.
values_to_record_batch_inferred
Convenience: infer the schema from records and encode them into a batch.