Struct arrow::record_batch::RecordBatch
source · [−]pub struct RecordBatch { /* private fields */ }
Expand description
A two-dimensional batch of column-oriented data with a defined schema.
A RecordBatch
is a two-dimensional dataset of a number of
contiguous arrays, each the same length.
A record batch has a schema which must match its arrays’
datatypes.
Record batches are a convenient unit of work for various serialization and computation functions, possibly incremental. See also CSV reader and JSON reader.
Implementations
sourceimpl RecordBatch
impl RecordBatch
sourcepub fn try_new(schema: SchemaRef, columns: Vec<ArrayRef>) -> Result<Self>
pub fn try_new(schema: SchemaRef, columns: Vec<ArrayRef>) -> Result<Self>
Creates a RecordBatch
from a schema and columns.
Expects the following:
- the vec of columns to not be empty
- the schema and column data types to have equal lengths and match
- each array in columns to have the same length
If the conditions are not met, an error is returned.
Example
use std::sync::Arc;
use arrow::array::Int32Array;
use arrow::datatypes::{Schema, Field, DataType};
use arrow::record_batch::RecordBatch;
let id_array = Int32Array::from(vec![1, 2, 3, 4, 5]);
let schema = Schema::new(vec![
Field::new("id", DataType::Int32, false)
]);
let batch = RecordBatch::try_new(
Arc::new(schema),
vec![Arc::new(id_array)]
)?;
sourcepub fn try_new_with_options(
schema: SchemaRef,
columns: Vec<ArrayRef>,
options: &RecordBatchOptions
) -> Result<Self>
pub fn try_new_with_options(
schema: SchemaRef,
columns: Vec<ArrayRef>,
options: &RecordBatchOptions
) -> Result<Self>
Creates a RecordBatch
from a schema and columns, with additional options,
such as whether to strictly validate field names.
See RecordBatch::try_new
for the expected conditions.
sourcepub fn new_empty(schema: SchemaRef) -> Self
pub fn new_empty(schema: SchemaRef) -> Self
Creates a new empty RecordBatch
.
sourcepub fn project(&self, indices: &[usize]) -> Result<RecordBatch>
pub fn project(&self, indices: &[usize]) -> Result<RecordBatch>
Projects the schema onto the specified columns
sourcepub fn num_columns(&self) -> usize
pub fn num_columns(&self) -> usize
Returns the number of columns in the record batch.
Example
use std::sync::Arc;
use arrow::array::Int32Array;
use arrow::datatypes::{Schema, Field, DataType};
use arrow::record_batch::RecordBatch;
let id_array = Int32Array::from(vec![1, 2, 3, 4, 5]);
let schema = Schema::new(vec![
Field::new("id", DataType::Int32, false)
]);
let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(id_array)])?;
assert_eq!(batch.num_columns(), 1);
sourcepub fn num_rows(&self) -> usize
pub fn num_rows(&self) -> usize
Returns the number of rows in each column.
Example
use std::sync::Arc;
use arrow::array::Int32Array;
use arrow::datatypes::{Schema, Field, DataType};
use arrow::record_batch::RecordBatch;
let id_array = Int32Array::from(vec![1, 2, 3, 4, 5]);
let schema = Schema::new(vec![
Field::new("id", DataType::Int32, false)
]);
let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(id_array)])?;
assert_eq!(batch.num_rows(), 5);
sourcepub fn slice(&self, offset: usize, length: usize) -> RecordBatch
pub fn slice(&self, offset: usize, length: usize) -> RecordBatch
Return a new RecordBatch where each column is sliced
according to offset
and length
Panics
Panics if offset
with length
is greater than column length.
sourcepub fn try_from_iter<I, F>(value: I) -> Result<Self> where
I: IntoIterator<Item = (F, ArrayRef)>,
F: AsRef<str>,
pub fn try_from_iter<I, F>(value: I) -> Result<Self> where
I: IntoIterator<Item = (F, ArrayRef)>,
F: AsRef<str>,
Create a RecordBatch
from an iterable list of pairs of the
form (field_name, array)
, with the same requirements on
fields and arrays as RecordBatch::try_new
. This method is
often used to create a single RecordBatch
from arrays,
e.g. for testing.
The resulting schema is marked as nullable for each column if
the array for that column is has any nulls. To explicitly
specify nullibility, use RecordBatch::try_from_iter_with_nullable
Example:
use std::sync::Arc;
use arrow::array::{ArrayRef, Int32Array, StringArray};
use arrow::datatypes::{Schema, Field, DataType};
use arrow::record_batch::RecordBatch;
let a: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
let b: ArrayRef = Arc::new(StringArray::from(vec!["a", "b"]));
let record_batch = RecordBatch::try_from_iter(vec![
("a", a),
("b", b),
]);
sourcepub fn try_from_iter_with_nullable<I, F>(value: I) -> Result<Self> where
I: IntoIterator<Item = (F, ArrayRef, bool)>,
F: AsRef<str>,
pub fn try_from_iter_with_nullable<I, F>(value: I) -> Result<Self> where
I: IntoIterator<Item = (F, ArrayRef, bool)>,
F: AsRef<str>,
Create a RecordBatch
from an iterable list of tuples of the
form (field_name, array, nullable)
, with the same requirements on
fields and arrays as RecordBatch::try_new
. This method is often
used to create a single RecordBatch
from arrays, e.g. for
testing.
Example:
use std::sync::Arc;
use arrow::array::{ArrayRef, Int32Array, StringArray};
use arrow::datatypes::{Schema, Field, DataType};
use arrow::record_batch::RecordBatch;
let a: ArrayRef = Arc::new(Int32Array::from(vec![1, 2]));
let b: ArrayRef = Arc::new(StringArray::from(vec![Some("a"), Some("b")]));
// Note neither `a` nor `b` has any actual nulls, but we mark
// b an nullable
let record_batch = RecordBatch::try_from_iter_with_nullable(vec![
("a", a, false),
("b", b, true),
]);
Trait Implementations
sourceimpl Clone for RecordBatch
impl Clone for RecordBatch
sourcefn clone(&self) -> RecordBatch
fn clone(&self) -> RecordBatch
Returns a copy of the value. Read more
1.0.0 · sourcefn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
Performs copy-assignment from source
. Read more
sourceimpl Debug for RecordBatch
impl Debug for RecordBatch
sourceimpl From<&'_ StructArray> for RecordBatch
impl From<&'_ StructArray> for RecordBatch
sourcefn from(struct_array: &StructArray) -> Self
fn from(struct_array: &StructArray) -> Self
Create a record batch from struct array, where each field of
the StructArray
becomes a Field
in the schema.
This currently does not flatten and nested struct types
sourceimpl From<RecordBatch> for StructArray
impl From<RecordBatch> for StructArray
sourcefn from(batch: RecordBatch) -> Self
fn from(batch: RecordBatch) -> Self
Converts to this type from the input type.
sourceimpl PartialEq<RecordBatch> for RecordBatch
impl PartialEq<RecordBatch> for RecordBatch
sourcefn eq(&self, other: &RecordBatch) -> bool
fn eq(&self, other: &RecordBatch) -> bool
This method tests for self
and other
values to be equal, and is used
by ==
. Read more
sourcefn ne(&self, other: &RecordBatch) -> bool
fn ne(&self, other: &RecordBatch) -> bool
This method tests for !=
.
impl StructuralPartialEq for RecordBatch
Auto Trait Implementations
impl !RefUnwindSafe for RecordBatch
impl Send for RecordBatch
impl Sync for RecordBatch
impl Unpin for RecordBatch
impl !UnwindSafe for RecordBatch
Blanket Implementations
sourceimpl<T> BorrowMut<T> for T where
T: ?Sized,
impl<T> BorrowMut<T> for T where
T: ?Sized,
const: unstable · sourcefn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
sourceimpl<T> ToOwned for T where
T: Clone,
impl<T> ToOwned for T where
T: Clone,
type Owned = T
type Owned = T
The resulting type after obtaining ownership.
sourcefn clone_into(&self, target: &mut T)
fn clone_into(&self, target: &mut T)
toowned_clone_into
)Uses borrowed data to replace owned data, usually by cloning. Read more