Module arrow_array::builder
source · Expand description
Defines builders that can be used to safely build arrays
Basic Usage
Builders can be used to build simple, non-nested arrays
let mut a = Int32Builder::new();
a.append_value(1);
a.append_null();
a.append_value(2);
let a = a.finish();
assert_eq!(a, PrimitiveArray::from(vec![Some(1), None, Some(2)]));
let mut a = StringBuilder::new();
a.append_value("foo");
a.append_value("bar");
a.append_null();
let a = a.finish();
assert_eq!(a, StringArray::from_iter([Some("foo"), Some("bar"), None]));
Nested Usage
Builders can also be used to build more complex nested arrays, such as lists
let mut a = ListBuilder::new(Int32Builder::new());
// [1, 2]
a.values().append_value(1);
a.values().append_value(2);
a.append(true);
// null
a.append(false);
// []
a.append(true);
// [3, null]
a.values().append_value(3);
a.values().append_null();
a.append(true);
// [[1, 2], null, [], [3, null]]
let a = a.finish();
assert_eq!(a, ListArray::from_iter_primitive::<Int32Type, _, _>([
Some(vec![Some(1), Some(2)]),
None,
Some(vec![]),
Some(vec![Some(3), None])]
))
Custom Builders
It is common to have a collection of statically defined Rust types that you want to convert to Arrow arrays. An example of doing so is below
/// A custom row representation
struct MyRow {
i32: i32,
optional_i32: Option<i32>,
string: Option<String>,
i32_list: Option<Vec<Option<i32>>>,
}
/// Converts `Vec<Row>` into `StructArray`
#[derive(Debug, Default)]
struct MyRowBuilder {
i32: Int32Builder,
string: StringBuilder,
i32_list: ListBuilder<Int32Builder>,
}
impl MyRowBuilder {
fn append(&mut self, row: &MyRow) {
self.i32.append_value(row.i32);
self.string.append_option(row.string.as_ref());
self.i32_list.append_option(row.i32_list.as_ref().map(|x| x.iter().copied()));
}
/// Note: returns StructArray to allow nesting within another array if desired
fn finish(&mut self) -> StructArray {
let i32 = Arc::new(self.i32.finish()) as ArrayRef;
let i32_field = Field::new("i32", DataType::Int32, false);
let string = Arc::new(self.string.finish()) as ArrayRef;
let string_field = Field::new("i32", DataType::Utf8, false);
let i32_list = Arc::new(self.i32_list.finish()) as ArrayRef;
let value_field = Arc::new(Field::new("item", DataType::Int32, true));
let i32_list_field = Field::new("i32_list", DataType::List(value_field), true);
StructArray::from(vec![
(i32_field, i32),
(string_field, string),
(i32_list_field, i32_list),
])
}
}
impl<'a> Extend<&'a MyRow> for MyRowBuilder {
fn extend<T: IntoIterator<Item = &'a MyRow>>(&mut self, iter: T) {
iter.into_iter().for_each(|row| self.append(row));
}
}
/// Converts a slice of [`MyRow`] to a [`RecordBatch`]
fn rows_to_batch(rows: &[MyRow]) -> RecordBatch {
let mut builder = MyRowBuilder::default();
builder.extend(rows);
RecordBatch::from(&builder.finish())
}
Structs
- A builder for creating a boolean
Buffer
- Array builder for fixed-width primitive types
- Builder for creating a
Buffer
object. - A fixed size binary array builder
- Array builder for
FixedSizeListArray
- Array builder for
GenericByteArray
- Generic array builder for
DictionaryArray
that stores generic byte values. For example to map a set of byte indices to String values. Note that the use of aHashMap
here will not scale to very large arrays or result in an ordered dictionary. - Array builder for
RunArray
for String and Binary types. - Array builder for
GenericListArray
s. - Creates a new
MapBuilder
- Contains details of the mapping
- Array builder for fixed-width primitive types
- Array builder for
DictionaryArray
. For example to map a set of byte indices to f32 values. Note that the use of aHashMap
here will not scale to very large arrays or result in an ordered dictionary. - Array builder for
RunArray
that encodes primitive values. - Array builder for Struct types.
- Builder type for creating a new
UnionArray
.
Traits
- Trait for dealing with different array builders at runtime
Functions
- Returns a builder with capacity
capacity
that corresponds to the datatypeDataType
This function is useful to construct arrays from an arbitrary vectors with known/expected schema.
Type Definitions
- A binary array builder with i32 offsets
- Array builder for
DictionaryArray
that stores binary. For example to map a set of byte indices to binary values. Note that the use of aHashMap
here will not scale to very large arrays or result in an ordered dictionary. - Array builder for
RunArray
that encodes binary values(BinaryType
). - Buffer builder for 32-bit date type.
- A 32-bit date array builder.
- Buffer builder for 64-bit date type.
- A 64-bit date array builder.
- A decimal 128 array builder
- A decimal 256 array builder
- Buffer builder for elaspsed time of microseconds unit.
- An elapsed time in microseconds array builder.
- Buffer builder for elaspsed time of milliseconds unit.
- An elapsed time in milliseconds array builder.
- Buffer builder for elaspsed time of nanoseconds unit.
- An elapsed time in nanoseconds array builder.
- Buffer builder for elaspsed time of second unit.
- An elapsed time in seconds array builder.
- Buffer builder for 32-bit floating point type.
- A 32-bit floating point array builder.
- Buffer builder for 64-bit floating point type.
- A 64-bit floating point array builder.
- Array builder for
GenericBinaryArray
- Array builder for
GenericStringArray
- Buffer builder for signed 8-bit integer type.
- A signed 8-bit integer array builder.
- Buffer builder for signed 16-bit integer type.
- A signed 16-bit integer array builder.
- Buffer builder for signed 32-bit integer type.
- A signed 32-bit integer array builder.
- Buffer builder for signed 64-bit integer type.
- A signed 64-bit integer array builder.
- Buffer builder for “calendar” interval in days and milliseconds.
- A “calendar” interval in days and milliseconds array builder.
- Buffer builder “calendar” interval in months, days, and nanoseconds.
- A “calendar” interval in months, days, and nanoseconds array builder.
- Buffer builder for “calendar” interval in months.
- A “calendar” interval in months array builder.
- A binary array builder with i64 offsets
- Array builder for
DictionaryArray
that stores large binary. For example to map a set of byte indices to binary values. Note that the use of aHashMap
here will not scale to very large arrays or result in an ordered dictionary. - Array builder for
RunArray
that encodes large binary values(LargeBinaryType
). See documentation ofBinaryRunBuilder
for an example. - Builder for
LargeListArray
s (i64 offsets) - A string array builder with i64 offsets
- Array builder for
DictionaryArray
that stores large Strings. For example to map a set of byte indices to String values. Note that the use of aHashMap
here will not scale to very large arrays or result in an ordered dictionary. - Array builder for
RunArray
that encodes large strings (LargeUtf8Type
). SeeStringRunBuilder
for an example. - Builder for
ListArray
s (i32 offsets) - A string array builder with i32 offsets
- Array builder for
DictionaryArray
that stores Strings. For example to map a set of byte indices to String values. Note that the use of aHashMap
here will not scale to very large arrays or result in an ordered dictionary. - Buffer builder for 32-bit elaspsed time since midnight of millisecond unit.
- A 32-bit elaspsed time in milliseconds array builder.
- Buffer builder for 32-bit elaspsed time since midnight of second unit.
- A 32-bit elaspsed time in seconds array builder.
- Buffer builder for 64-bit elaspsed time since midnight of microsecond unit.
- A 64-bit elaspsed time in microseconds array builder.
- Buffer builder for 64-bit elaspsed time since midnight of nanosecond unit.
- A 64-bit elaspsed time in nanoseconds array builder.
- Buffer builder for timestamp type of microsecond unit.
- A timestamp microsecond array builder.
- Buffer builder for timestamp type of millisecond unit.
- A timestamp millisecond array builder.
- Buffer builder for timestamp type of nanosecond unit.
- A timestamp nanosecond array builder.
- Buffer builder for timestamp type of second unit.
- A timestamp second array builder.
- Buffer builder for usigned 8-bit integer type.
- An usigned 8-bit integer array builder.
- Buffer builder for usigned 16-bit integer type.
- An usigned 16-bit integer array builder.
- Buffer builder for usigned 32-bit integer type.
- An usigned 32-bit integer array builder.
- Buffer builder for usigned 64-bit integer type.
- An usigned 64-bit integer array builder.