Struct PrimitiveArray

pub struct PrimitiveArray<T: NativeType> { /* private fields */ }

A PrimitiveArray is Arrow’s semantically equivalent of an immutable Vec<Option<T>> where T is NativeType (e.g. i32). It implements Array.

One way to think about a PrimitiveArray is (DataType, Arc<Vec<T>>, Option<Arc<Vec<u8>>>) where:

• the first item is the array’s logical type
• the second is the immutable values
• the third is the immutable validity (whether a value is null or not as a bitmap).

The size of this struct is O(1), as all data is stored behind an std::sync::Arc.

Example

use polars_arrow::array::PrimitiveArray;
use polars_arrow::bitmap::Bitmap;
use polars_arrow::buffer::Buffer;

let array = PrimitiveArray::from([Some(1i32), None, Some(10)]);
assert_eq!(array.value(0), 1);
assert_eq!(array.iter().collect::<Vec<_>>(), vec![Some(&1i32), None, Some(&10)]);
assert_eq!(array.values_iter().copied().collect::<Vec<_>>(), vec![1, 0, 10]);
// the underlying representation
assert_eq!(array.values(), &Buffer::from(vec![1i32, 0, 10]));
assert_eq!(array.validity(), Some(&Bitmap::from([true, false, true])));

Implementations

impl<T: NativeType> PrimitiveArray<T>

pub fn try_new( dtype: ArrowDataType, values: Buffer<T>, validity: Option<Bitmap>, ) -> PolarsResult<Self>

The canonical method to create a PrimitiveArray out of its internal components.

Implementation

This function is O(1).

Errors

This function errors iff:

• The validity is not None and its length is different from values’s length
• The dtype’s PhysicalType is not equal to [PhysicalType::Primitive(T::PRIMITIVE)]

pub unsafe fn new_unchecked( dtype: ArrowDataType, values: Buffer<T>, validity: Option<Bitmap>, ) -> Self

Safety

Doesn’t check invariants

pub fn to(self, dtype: ArrowDataType) -> Self

Returns a new PrimitiveArray with a different logical type.

This function is useful to assign a different ArrowDataType to the array. Used to change the arrays’ logical type (see example).

Example

use polars_arrow::array::Int32Array;
use polars_arrow::datatypes::ArrowDataType;

let array = Int32Array::from(&[Some(1), None, Some(2)]).to(ArrowDataType::Date32);
assert_eq!(
   format!("{:?}", array),
   "Date32[1970-01-02, None, 1970-01-03]"
);

Panics

Panics iff the dtype’s PhysicalType is not equal to [PhysicalType::Primitive(T::PRIMITIVE)]

pub fn from_vec(values: Vec<T>) -> Self

Creates a (non-null) PrimitiveArray from a vector of values. This function is O(1).

Examples

use polars_arrow::array::PrimitiveArray;

let array = PrimitiveArray::from_vec(vec![1, 2, 3]);
assert_eq!(format!("{:?}", array), "Int32[1, 2, 3]");

pub fn iter(&self) -> ZipValidity<&T, Iter<'_, T>, BitmapIter<'_>>

Returns an iterator over the values and validity, Option<&T>.

pub fn values_iter(&self) -> Iter<'_, T>

Returns an iterator of the values, &T, ignoring the arrays’ validity.

pub fn non_null_values_iter(&self) -> NonNullValuesIter<'_, [T]>

Returns an iterator of the non-null values T.

pub fn len(&self) -> usize

Returns the length of this array

pub fn values(&self) -> &Buffer<T>

The values Buffer. Values on null slots are undetermined (they can be anything).

pub fn validity(&self) -> Option<&Bitmap>

Returns the optional validity.

pub fn dtype(&self) -> &ArrowDataType

Returns the arrays’ ArrowDataType.

pub fn value(&self, i: usize) -> T

Returns the value at slot i.

Equivalent to self.values()[i]. The value of a null slot is undetermined (it can be anything).

Panic

This function panics iff i >= self.len.

pub unsafe fn value_unchecked(&self, i: usize) -> T

Returns the value at index i. The value on null slots is undetermined (it can be anything).

Safety

Caller must be sure that i < self.len()

pub fn slice(&mut self, offset: usize, length: usize)

Slices this PrimitiveArray by an offset and length.

Implementation

This operation is O(1).

pub unsafe fn slice_unchecked(&mut self, offset: usize, length: usize)

Slices this PrimitiveArray by an offset and length.

Implementation

This operation is O(1).

Safety

The caller must ensure that offset + length <= self.len().

pub fn sliced(self, offset: usize, length: usize) -> Self

Returns this array sliced.

Implementation

This function is O(1).

Panics

iff offset + length > self.len().

pub unsafe fn sliced_unchecked(self, offset: usize, length: usize) -> Self

Returns this array sliced.

Implementation

This function is O(1).

Safety

The caller must ensure that offset + length <= self.len().

pub fn with_validity(self, validity: Option<Bitmap>) -> Self

Returns this array with a new validity.

Panic

Panics iff validity.len() != self.len().

pub fn set_validity(&mut self, validity: Option<Bitmap>)

Sets the validity of this array.

Panics

This function panics iff values.len() != self.len().

pub fn take_validity(&mut self) -> Option<Bitmap>

Takes the validity of this array, leaving it without a validity mask.

pub fn boxed(self) -> Box<dyn Array>

Boxes this array into a Box<dyn Array>.

pub fn arced(self) -> Arc<dyn Array>

Arcs this array into a std::sync::Arc<dyn Array>.

pub fn with_values(self, values: Buffer<T>) -> Self

Returns this PrimitiveArray with new values.

Panics

This function panics iff values.len() != self.len().

pub fn set_values(&mut self, values: Buffer<T>)

Update the values of this PrimitiveArray.

Panics

This function panics iff values.len() != self.len().

pub fn apply_validity<F: FnOnce(Bitmap) -> Bitmap>(&mut self, f: F)

Applies a function f to the validity of this array.

This is an API to leverage clone-on-write

Panics

This function panics if the function f modifies the length of the Bitmap.

pub fn get_mut_values(&mut self) -> Option<&mut [T]>

Returns an option of a mutable reference to the values of this PrimitiveArray.

pub fn into_inner(self) -> (ArrowDataType, Buffer<T>, Option<Bitmap>)

Returns its internal representation

pub fn from_inner( dtype: ArrowDataType, values: Buffer<T>, validity: Option<Bitmap>, ) -> PolarsResult<Self>

Creates a PrimitiveArray from its internal representation. This is the inverted from PrimitiveArray::into_inner

pub unsafe fn from_inner_unchecked( dtype: ArrowDataType, values: Buffer<T>, validity: Option<Bitmap>, ) -> Self

Creates a PrimitiveArray from its internal representation. This is the inverted from PrimitiveArray::into_inner

Safety

Callers must ensure all invariants of this struct are upheld.

pub fn into_mut(self) -> Either<Self, MutablePrimitiveArray<T>>

Try to convert this PrimitiveArray to a MutablePrimitiveArray via copy-on-write semantics.

A PrimitiveArray is backed by a Buffer and Bitmap which are essentially Arc<Vec<_>>. This function returns a MutablePrimitiveArray (via std::sync::Arc::get_mut) iff both values and validity have not been cloned / are unique references to their underlying vectors.

This function is primarily used to reuse memory regions.

pub fn new_empty(dtype: ArrowDataType) -> Self

Returns a new empty (zero-length) PrimitiveArray.

pub fn new_null(dtype: ArrowDataType, length: usize) -> Self

Returns a new PrimitiveArray where all slots are null / None.

pub fn from_values<I: IntoIterator<Item = T>>(iter: I) -> Self

Creates a (non-null) PrimitiveArray from an iterator of values.

Implementation

This does not assume that the iterator has a known length.

pub fn from_slice<P: AsRef<[T]>>(slice: P) -> Self

Creates a (non-null) PrimitiveArray from a slice of values.

Implementation

This is essentially a memcopy and is thus O(N)

pub fn from_trusted_len_values_iter<I: TrustedLen<Item = T>>(iter: I) -> Self

Creates a (non-null) PrimitiveArray from a TrustedLen of values.

Implementation

This does not assume that the iterator has a known length.

pub unsafe fn from_trusted_len_values_iter_unchecked<I: Iterator<Item = T>>( iter: I, ) -> Self

Creates a new PrimitiveArray from an iterator over values

Safety

The iterator must be TrustedLen. I.e. that size_hint().1 correctly reports its length.

pub fn from_trusted_len_iter<I: TrustedLen<Item = Option<T>>>(iter: I) -> Self

Creates a PrimitiveArray from a TrustedLen of optional values.

pub unsafe fn from_trusted_len_iter_unchecked<I: Iterator<Item = Option<T>>>( iter: I, ) -> Self

Creates a PrimitiveArray from an iterator of optional values.

Safety

The iterator must be TrustedLen. I.e. that size_hint().1 correctly reports its length.

pub fn new( dtype: ArrowDataType, values: Buffer<T>, validity: Option<Bitmap>, ) -> Self

Alias for Self::try_new(..).unwrap().

Panics

This function errors iff:

• The validity is not None and its length is different from values’s length
• The dtype’s PhysicalType is not equal to PhysicalType::Primitive.

pub fn transmute<U: NativeType>(self) -> PrimitiveArray<U>

Transmute this PrimitiveArray into another PrimitiveArray.

T and U must have the same size and alignment.

pub fn fill_with(self, value: T) -> Self

Fills this entire array with the given value, leaving the validity mask intact.

Reuses the memory of the PrimitiveArray if possible.

Trait Implementations

impl<T: NativeType> Array for PrimitiveArray<T>

fn as_any(&self) -> &dyn Any

Converts itself to a reference of Any, which enables downcasting to concrete types.

fn as_any_mut(&mut self) -> &mut dyn Any

Converts itself to a mutable reference of Any, which enables mutable downcasting to concrete types.

fn len(&self) -> usize

The length of the Array. Every array has a length corresponding to the number of elements (slots).

fn dtype(&self) -> &ArrowDataType

The ArrowDataType of the Array. In combination with Array::as_any, this can be used to downcast trait objects (dyn Array) to concrete arrays.

fn split_at_boxed(&self, offset: usize) -> (Box<dyn Array>, Box<dyn Array>)

Split Self at offset into two boxed Arrays where offset <= self.len().

unsafe fn split_at_boxed_unchecked( &self, offset: usize, ) -> (Box<dyn Array>, Box<dyn Array>)

Split Self at offset into two boxed Arrays without checking offset <= self.len().

fn slice(&mut self, offset: usize, length: usize)

Slices this Array.

unsafe fn slice_unchecked(&mut self, offset: usize, length: usize)

Slices the Array.

fn to_boxed(&self) -> Box<dyn Array>

Clone a &dyn Array to an owned Box<dyn Array>.

fn validity(&self) -> Option<&Bitmap>

The validity of the Array: every array has an optional Bitmap that, when available specifies whether the array slot is valid or not (null). When the validity is None, all slots are valid.

fn with_validity(&self, validity: Option<Bitmap>) -> Box<dyn Array>

Clones this Array with a new assigned bitmap.

fn is_empty(&self) -> bool

whether the array is empty

fn null_count(&self) -> usize

The number of null slots on this Array.

fn has_nulls(&self) -> bool

fn is_null(&self, i: usize) -> bool

Returns whether slot i is null.

unsafe fn is_null_unchecked(&self, i: usize) -> bool

Returns whether slot i is null.

fn is_valid(&self, i: usize) -> bool

Returns whether slot i is valid.

fn sliced(&self, offset: usize, length: usize) -> Box<dyn Array>

Returns a slice of this Array.

unsafe fn sliced_unchecked( &self, offset: usize, length: usize, ) -> Box<dyn Array>

Returns a slice of this Array.

impl<T: NativeType> ArrayFromIter<Option<T>> for PrimitiveArray<T>

fn arr_from_iter<I: IntoIterator<Item = Option<T>>>(iter: I) -> Self

fn arr_from_iter_trusted<I>(iter: I) -> Self

where I: IntoIterator<Item = Option<T>>, I::IntoIter: TrustedLen,

fn try_arr_from_iter<E, I: IntoIterator<Item = Result<Option<T>, E>>>( iter: I, ) -> Result<Self, E>

fn try_arr_from_iter_trusted<E, I>(iter: I) -> Result<Self, E>

where I: IntoIterator<Item = Result<Option<T>, E>>, I::IntoIter: TrustedLen,

impl<T: NativeType> ArrayFromIter<T> for PrimitiveArray<T>

fn arr_from_iter<I: IntoIterator<Item = T>>(iter: I) -> Self

fn arr_from_iter_trusted<I>(iter: I) -> Self

where I: IntoIterator<Item = T>, I::IntoIter: TrustedLen,

fn try_arr_from_iter<E, I: IntoIterator<Item = Result<T, E>>>( iter: I, ) -> Result<Self, E>

fn try_arr_from_iter_trusted<E, I>(iter: I) -> Result<Self, E>

where I: IntoIterator<Item = Result<T, E>>, I::IntoIter: TrustedLen,

impl<T: NativeType> Bounded for PrimitiveArray<T>

fn len(&self) -> usize

fn is_empty(&self) -> bool

impl<T: Clone + NativeType> Clone for PrimitiveArray<T>

fn clone(&self) -> PrimitiveArray<T>

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: NativeType> Debug for PrimitiveArray<T>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<T: NativeType> Default for PrimitiveArray<T>

fn default() -> Self

Returns the “default value” for a type.

impl<'a, T: NativeType> From<GrowablePrimitive<'a, T>> for PrimitiveArray<T>

fn from(val: GrowablePrimitive<'a, T>) -> Self

Converts to this type from the input type.

impl<T: NativeType> From<MutablePrimitiveArray<T>> for PrimitiveArray<T>

fn from(other: MutablePrimitiveArray<T>) -> Self

Converts to this type from the input type.

impl<T: NativeType, P: AsRef<[Option<T>]>> From<P> for PrimitiveArray<T>

fn from(slice: P) -> Self

Converts to this type from the input type.

impl<T: NativeType> FromData<Buffer<T>> for PrimitiveArray<T>

fn from_data_default(values: Buffer<T>, validity: Option<Bitmap>) -> Self

impl<T: NativeType, Ptr: Borrow<Option<T>>> FromIterator<Ptr> for PrimitiveArray<T>

fn from_iter<I: IntoIterator<Item = Ptr>>(iter: I) -> Self

Creates a value from an iterator.

impl<T: NativeType> FromIteratorReversed<Option<T>> for PrimitiveArray<T>

fn from_trusted_len_iter_rev<I: TrustedLen<Item = Option<T>>>(iter: I) -> Self

impl<T: NativeType> FromIteratorReversed<T> for PrimitiveArray<T>

fn from_trusted_len_iter_rev<I: TrustedLen<Item = T>>(iter: I) -> Self

impl<T: NativeType> FromTrustedLenIterator<Option<T>> for PrimitiveArray<T>

fn from_iter_trusted_length<I: IntoIterator<Item = Option<T>>>(iter: I) -> Self

where I::IntoIter: TrustedLen,

impl<T: NativeType> FromTrustedLenIterator<T> for PrimitiveArray<T>

fn from_iter_trusted_length<I: IntoIterator<Item = T>>(iter: I) -> Self

where I::IntoIter: TrustedLen,

impl<T: NativeType> Indexable for PrimitiveArray<T>

type Item = Option<T>

fn get(&self, i: usize) -> Self::Item

unsafe fn get_unchecked(&self, i: usize) -> Self::Item

Safety

impl<'a, T: NativeType> IntoIterator for &'a PrimitiveArray<T>

type Item = Option<&'a T>

The type of the elements being iterated over.

type IntoIter = ZipValidity<&'a T, Iter<'a, T>, BitmapIter<'a>>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl<T: NativeType> IntoIterator for PrimitiveArray<T>

type Item = Option<T>

The type of the elements being iterated over.

type IntoIter = ZipValidity<T, IntoIter<T>, IntoIter>

Which kind of iterator are we turning this into?

fn into_iter(self) -> Self::IntoIter

Creates an iterator from a value.

impl<T: NativeType> NullCount for PrimitiveArray<T>

fn null_count(&self) -> usize

impl<T: NativeType> ParameterFreeDtypeStaticArray for PrimitiveArray<T>

fn get_dtype() -> ArrowDataType

impl<T: NativeType> PartialEq<&(dyn Array + 'static)> for PrimitiveArray<T>

fn eq(&self, other: &&dyn Array) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: NativeType> PartialEq<PrimitiveArray<T>> for &dyn Array

fn eq(&self, other: &PrimitiveArray<T>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: NativeType> PartialEq for PrimitiveArray<T>

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl<T: NativeType> SliceAble for PrimitiveArray<T>

unsafe fn slice_unchecked(&self, range: Range<usize>) -> Self

Safety

fn slice(&self, range: Range<usize>) -> Self

impl<T: NativeType> Splitable for PrimitiveArray<T>

fn check_bound(&self, offset: usize) -> bool

unsafe fn _split_at_unchecked(&self, offset: usize) -> (Self, Self)

Internal implementation of split_at_unchecked. For any usage, prefer the using split_at or split_at_unchecked.

fn split_at(&self, offset: usize) -> (Self, Self)

Split Self at offset where offset <= self.len().

unsafe fn split_at_unchecked(&self, offset: usize) -> (Self, Self)

Split Self at offset without checking offset <= self.len().

impl<T: NativeType> StaticArray for PrimitiveArray<T>

type ValueT<'a> = T

type ZeroableValueT<'a> = T

type ValueIterT<'a> = Copied<Iter<'a, T>>

unsafe fn value_unchecked(&self, idx: usize) -> Self::ValueT<'_>

Safety

fn values_iter(&self) -> Self::ValueIterT<'_>

fn as_slice(&self) -> Option<&[Self::ValueT<'_>]>

fn iter( &self, ) -> ZipValidity<Self::ValueT<'_>, Self::ValueIterT<'_>, BitmapIter<'_>>

fn with_validity_typed(self, validity: Option<Bitmap>) -> Self

fn from_vec(v: Vec<Self::ValueT<'_>>, _dtype: ArrowDataType) -> Self

fn from_zeroable_vec( v: Vec<Self::ZeroableValueT<'_>>, _dtype: ArrowDataType, ) -> Self

fn full_null(length: usize, dtype: ArrowDataType) -> Self

fn full(length: usize, value: Self::ValueT<'_>, _dtype: ArrowDataType) -> Self

fn get(&self, idx: usize) -> Option<Self::ValueT<'_>>

unsafe fn get_unchecked(&self, idx: usize) -> Option<Self::ValueT<'_>>

Safety

fn last(&self) -> Option<Self::ValueT<'_>>

fn value(&self, idx: usize) -> Self::ValueT<'_>

impl<T: NativeType> ArrowArray for PrimitiveArray<T>