Module arrow 

Arrow RecordBatches as rows, and Arrow schemas in the format-neutral model (the arrow feature).

schema maps an Arrow schema into Schema: integers of every width are integer, Decimal*(p, s) is decimal(p,s), timestamps keep their time zone, dictionaries (and run-end encoded arrays) take their value type, and lists, structs and maps nest. Each field keeps Arrow’s own type name as its native type (Int32, Timestamp(ms, "UTC"), Dictionary(Int32, Utf8); List, Struct and Map for nested types, whose children say the rest), is required when it is not nullable, and carries its metadata, sorted by key. Types the model has no name for (times, durations, intervals, unions) are unknown, with the native type saying what they are.

The schema explorer (#342): tree draws an Arrow schema as a SchemaTree, and diff compares two as a SchemaDiff, fields added, removed and changed with breaking changes marked.

read_schema reads the schema of an Arrow IPC file (Feather v2, .arrow) or stream (.arrows) without reading its record batches.

rows reads one batch, and BatchSource streams many. Cells are Value::Int and Value::Float for numbers (a UInt64 past i64::MAX becomes a float), Value::Null for nulls, and text for the rest: booleans as true/false, decimals exactly, dates as YYYY-MM-DD, timestamps as YYYY-MM-DDTHH:MM:SS[.fff] in UTC (with a Z when the type has a time zone), binary as 0x… hex, and nested values as compact JSON.

use std::sync::Arc;
use arrow_array::{ArrayRef, Int32Array, RecordBatch, StringArray};
use rich_data::{arrow, Value};

let batch = RecordBatch::try_from_iter([
    ("id", Arc::new(Int32Array::from(vec![Some(1), None])) as ArrayRef),
    ("name", Arc::new(StringArray::from(vec!["ada", "alan"])) as ArrayRef),
])
.unwrap();
let rows = arrow::rows(&batch);
assert_eq!(rows.columns(), ["id", "name"]);
assert_eq!(rows.rows()[1], [Value::Null, Value::from("alan")]);
let id = &rows.schema().unwrap().fields()[0];
assert_eq!((id.data_type().to_string(), id.native_type()), ("integer".into(), Some("Int32")));

Structs

BatchSource

A RowSource over record batches that share a schema: the first batch’s schema is the source’s, and a later batch with different columns is an error.

Functions

cell

The cell at index of array.

data_type

One Arrow type in the model.

diff

What changed between two Arrow schemas: fields added and removed, type changes, fields that became (or stopped being) nullable, and metadata, with breaking changes marked.

field

One Arrow field in the model.

native_type

Arrow’s name for a type: its own for scalars, and only the kind for nested types (List, FixedSizeList(3), Struct, Map), whose children are fields of their own.

read_schema

The schema of an Arrow IPC file (ARROW1…, Feather v2) or stream, whichever reader holds: the file’s footer, or the stream’s first message. Neither record batches nor dictionary batches are read: only the footer’s schema, its length checked against the input’s, so a hostile footer is an error rather than a panic or a large allocation.

rows

One batch as rows, with its schema.

schema

An Arrow schema in the format-neutral model, with its metadata.

tree

An Arrow schema drawn as a tree: each field with its Arrow type, (required) when it is not nullable, and its metadata; nested fields under their parents.