Crate serde_arrow

Expand description

§`serde_arrow` - convert sequences of Rust objects to and from Arrow arrays

The Arrow in-memory format is a powerful way to work with data-frame-like structures. However, the API of the underlying Rust crates can be at times cumbersome to use due to the statically typed nature of Rust. serde_arrow offers a simple way to convert Rust objects into Arrow arrays and back. serde_arrow relies on Serde to interpret Rust objects. Therefore, adding support for serde_arrow to custom types is as easy as using Serde’s derive macros.

serde_arrow supports the arrow crate. The Arrow version can be selected via features.

serde_arrow relies on a schema to translate between Rust and Arrow as their type systems do not directly match. The schema is expressed as a collection of Arrow fields with additional metadata describing the arrays. For example, to convert a vector of Rust strings representing timestamps to an Arrow Timestamp array, the schema should contain a field with data type Timestamp. serde_arrow can derive the schema from the data or the Rust types themselves via schema tracing, but does not require it. It is always possible to specify the schema manually. See the schema module and SchemaLike for further details.

§Overview

Operation	`arrow-*`	`marrow`
Rust to Arrow	`to_record_batch`, `to_arrow`	`to_marrow`
Arrow to Rust	`from_record_batch`, `from_arrow`	`from_marrow`
`ArrayBuilder`	`ArrayBuilder::from_arrow`	`ArrayBuilder::from_marrow`
`Serializer`	`ArrayBuilder::from_arrow` + `Serializer::new`	`ArrayBuilder::from_marrow` + `Serializer::new`
`Deserializer`	`Deserializer::from_record_batch`, `Deserializer::from_arrow`	`Deserializer::from_marrow`

§Example

use arrow::datatypes::FieldRef;
use serde_arrow::schema::{SchemaLike, TracingOptions};

#[derive(Serialize, Deserialize)]
struct Record {
    a: f32,
    b: i32,
}

let records = vec![
    Record { a: 1.0, b: 1 },
    Record { a: 2.0, b: 2 },
    Record { a: 3.0, b: 3 },
];

// Determine Arrow schema
let fields = Vec::<FieldRef>::from_type::<Record>(TracingOptions::default())?;

// Build the record batch
let batch = serde_arrow::to_record_batch(&fields, &records)?;

The RecordBatch can then be written to disk, e.g., as parquet using the ArrowWriter from the parquet crate.

§Features:

The version of arrow used can be selected via features. By default, no Arrow implementation is used. In that case only the base features of serde_arrow are available.

The highest selected arrow-* version is used if multiple features are activated.

Note that because the highest version is selected, the features are not additive. In particular, it is not possible to use serde_arrow::to_arrow for multiple different arrow versions at the same time. See the next section for how to use serde_arrow in library code.

Available features:

Arrow Feature	Arrow Version
`arrow-59`	`arrow=59`
`arrow-58`	`arrow=58`
`arrow-57`	`arrow=57`
`arrow-56`	`arrow=56`
`arrow-55`	`arrow=55`
`arrow-54`	`arrow=54`
`arrow-53`	`arrow=53`

§Usage in libraries

In libraries, it is not recommended to use the arrow functions directly. Rather it is recommended to rely on the marrow-based functionality, as the features of marrow are designed to be strictly additive.

For example, to build a record batch, first build the corresponding marrow types and then use them to build the record batch:

// Determine Arrow schema
let fields = Vec::<marrow::datatypes::Field>::from_type::<Record>(TracingOptions::default())?;

// Build the marrow arrays
let arrays = serde_arrow::to_marrow(&fields, &records)?;

// Build the record batch
let arrow_fields = fields.iter()
    .map(arrow::datatypes::Field::try_from)
    .collect::<Result<Vec<_>, _>>()?;

let arrow_arrays = arrays.into_iter()
    .map(arrow::array::ArrayRef::try_from)
    .collect::<Result<Vec<_>, _>>()?;

let record_batch = arrow::array::RecordBatch::try_new(
    Arc::new(arrow::datatypes::Schema::new(arrow_fields)),
    arrow_arrays,
);

Re-exports§

pub use marrow;

Modules§

_impl: Internal. Do not use
deserializer: Deserialization of items
schema: Type mappings between Rust, Serde, and Arrow
utils: Helpers that may be useful when using serde_arrow

Structs§

ArrayBuilder: Construct arrays by pushing individual records
Deserializer: A structure to deserialize Arrow arrays into Rust objects
Error: Common errors during serde_arrow’s usage
Serializer: Wrap an ArrayBuilder as a serializer

Enums§

ErrorKind: Classifies an error for pattern matching

Functions§

from_arrow: Deserialize items from arrow arrays (requires one of the arrow-* features)
from_marrow: Deserialize items from marrow views
from_record_batch: Deserialize items from a record batch (requires one of the arrow-* features)
to_arrow: Build arrow arrays from the given items (requires one of the arrow-* features)
to_marrow: Build marrow arrays from the given items
to_record_batch: Build a record batch from the given items (requires one of the arrow-* features)

Type Aliases§

Result: A Result type that defaults to serde_arrow’s Error type