Module extension 

The nd_arrow.array Arrow extension type.

An nd column is an Arrow Struct with this storage layout. Each row holds one complete N-dimensional array:

Struct {
  values:    List<T>,        // flat C-order values
  dim_names: List<Utf8>,     // one name per axis
  dim_sizes: List<UInt32>,   // one size per axis
  attrs:     Struct { .. },  // optional: one nullable child per attribute key
}

The field carries ARROW:extension:name = nd_arrow.array and ARROW:extension:metadata = {"version":1}, so the intent survives IPC, Parquet and any engine that keeps field metadata.

Rules

FORMAT.md in the repository is the full, language-neutral specification. This section is a summary.

• The field order is fixed: values, dim_names, dim_sizes, then attrs if present.
• A column without attributes leaves out the attrs field, so no storage format needs a struct with zero fields. Readers also accept an attrs struct with zero fields and treat it as no attributes.
• dim_names and dim_sizes have the same length. This length is the rank.
• The length of values is the product of dim_sizes.
• Rank 0 (empty dimension lists) is a scalar with one value.
• Dimension names are not empty and are unique in a row.
• Validity of elements is in the null buffer of values. There is no fill value.
• An attribute can be any Arrow type. The attrs struct holds the union of the keys of all rows. A null child means that the row does not have that key.
• A null row decodes to a rank-0 null scalar.

Example

A (time=2, lat=3) array of Float64 with units = "K":

values:    [0.0, 0.1, 0.2, 1.0, 1.1, 1.2]
dim_names: ["time", "lat"]
dim_sizes: [2, 3]
attrs:     { units: "K" }

Global attributes

An encoded record batch (see encode_batches) stores the global attributes of each nd batch in one extra column, GLOBAL_ATTRIBUTES_FIELD, tagged with GLOBAL_ATTRIBUTES_EXTENSION. It has one row per nd batch and the union of keys, with the same null rule as attrs. The column is left out when no batch has global attributes.

Structs

NdArrayMetadata

Extension metadata, serialized as JSON in ARROW:extension:metadata.

NdArrayType

The nd extension type: the element field of the values, plus the attribute fields.

Constants

ATTRS_FIELD

Optional storage field with the attributes.

DIM_NAMES_FIELD

Storage field with the dimension names.

DIM_SIZES_FIELD

Storage field with the dimension sizes.

ENCODING_VERSION

The encoding version this crate writes and reads.

GLOBAL_ATTRIBUTES_EXTENSION

Extension name of the column that holds the global attributes of each nd batch.

GLOBAL_ATTRIBUTES_FIELD

Name of the global attributes column in an encoded record batch.

ITEM_FIELD

Name of the list item field in values, dim_names and dim_sizes.

VALUES_FIELD

Storage field with the flat values.

Functions

encoded_field

The nd field for a logical field: same name and metadata, values of the logical type, no attributes. The field is nullable.

encoded_schema

Every field of logical as an nd field, see encoded_field.

global_attributes_field

The global attributes column: a Struct with one nullable child per key, tagged with GLOBAL_ATTRIBUTES_EXTENSION.

is_global_attributes_field

True when field is a global attributes column.

is_nd_field

True when field carries the nd extension tag.

logical_field

The logical field of an nd field: the element type of the values, with the field metadata minus the extension keys.

logical_schema

Every field of encoded as its logical field, see logical_field.