Struct polars_core::chunked_array::ChunkedArray[−][src]

pub struct ChunkedArray<T> { /* fields omitted */ }

Expand description

ChunkedArray

Every Series contains a ChunkedArray<T>. Unlike Series, ChunkedArray’s are typed. This allows us to apply closures to the data and collect the results to a ChunkedArray of the same type T. Below we use an apply to use the cosine function to the values of a ChunkedArray.

fn apply_cosine(ca: &Float32Chunked) -> Float32Chunked {
    ca.apply(|v| v.cos())
}

If we would like to cast the result we could use a Rust Iterator instead of an apply method. Note that Iterators are slightly slower as the null values aren’t ignored implicitly.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float64Chunked {
    ca.into_iter()
        .map(|opt_v| {
        opt_v.map(|v| v.cos() as f64)
    }).collect()
}

Another option is to first cast and then use an apply.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float64Chunked {
    ca.cast::<Float64Type>()
        .unwrap()
        .apply(|v| v.cos())
}

Conversion between Series and ChunkedArray’s

Conversion from a Series to a ChunkedArray is effortless.

fn to_chunked_array(series: &Series) -> Result<&Int32Chunked>{
    series.i32()
}

fn to_series(ca: Int32Chunked) -> Series {
    ca.into_series()
}

Iterators

ChunkedArrays fully support Rust native Iterator and DoubleEndedIterator traits, thereby giving access to all the excellent methods available for Iterators.


fn iter_forward(ca: &Float32Chunked) {
    ca.into_iter()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

fn iter_backward(ca: &Float32Chunked) {
    ca.into_iter()
        .rev()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

Memory layout

ChunkedArray’s use Apache Arrow as backend for the memory layout. Arrows memory is immutable which makes it possible to make multiple zero copy (sub)-views from a single array.

To be able to append data, Polars uses chunks to append new memory locations, hence the ChunkedArray<T> data structure. Appends are cheap, because it will not lead to a full reallocation of the whole array (as could be the case with a Rust Vec).

However, multiple chunks in a ChunkArray will slow down the Iterators, arithmetic and other operations. When multiplying two ChunkArray's with different chunk sizes they cannot utilize SIMD for instance. However, when chunk size don’t match, Iterators will be used to do the operation (instead of arrows upstream implementation, which may utilize SIMD) and the result will be a single chunked array.

The key takeaway is that by applying operations on a ChunkArray of multiple chunks, the results will converge to a ChunkArray of a single chunk! It is recommended to leave them as is. If you want to have predictable performance (no unexpected re-allocation of memory), it is advised to call the rechunk after multiple append operations.

Implementations

`impl<T> ChunkedArray<T> where T: PolarsNumericType,` [src]

`pub fn downcast_iter( &self ) -> impl Iterator<Item = &PrimitiveArray<T>> + DoubleEndedIterator`[src]

`pub fn downcast_chunks(&self) -> Chunks<'_, PrimitiveArray<T>>`[src]

`impl ChunkedArray<BooleanType>`[src]

`pub fn downcast_iter( &self ) -> impl Iterator<Item = &BooleanArray> + DoubleEndedIterator`[src]

`pub fn downcast_chunks(&self) -> Chunks<'_, BooleanArray>`[src]

`impl ChunkedArray<Utf8Type>`[src]

`pub fn downcast_iter( &self ) -> impl Iterator<Item = &LargeStringArray> + DoubleEndedIterator`[src]

`pub fn downcast_chunks(&self) -> Chunks<'_, LargeStringArray>`[src]

`impl ChunkedArray<ListType>`[src]

`pub fn downcast_iter( &self ) -> impl Iterator<Item = &LargeListArray> + DoubleEndedIterator`[src]

`pub fn downcast_chunks(&self) -> Chunks<'_, LargeListArray>`[src]

`impl<'a, T> ChunkedArray<ObjectType<T>> where T: PolarsObject,` [src]

`pub fn downcast_iter( &self ) -> impl Iterator<Item = &ObjectArray<T>> + DoubleEndedIterator`[src]

`pub fn downcast_chunks(&self) -> Chunks<'_, ObjectArray<T>>`[src]

`impl ChunkedArray<BooleanType>`[src]

`pub fn arg_true(&self) -> UInt32Chunked`[src]

`impl ChunkedArray<BooleanType>`[src]

`pub fn all_true(&self) -> bool`[src]

Check if all values are true

`impl ChunkedArray<BooleanType>`[src]

`pub fn all_false(&self) -> bool`[src]

Check if all values are false

`impl ChunkedArray<BooleanType>`[src]

`pub fn into_no_null_iter( &self ) -> impl Iterator<Item = bool> + '_ + Send + Sync + ExactSizeIterator + DoubleEndedIterator`[src]

`impl ChunkedArray<Utf8Type>`[src]

`pub fn into_no_null_iter<'a>( &'a self ) -> impl Iterator<Item = &'a str> + '_ + Send + Sync + ExactSizeIterator + DoubleEndedIterator`[src]

`impl ChunkedArray<ListType>`[src]

`pub fn into_no_null_iter( &self ) -> impl Iterator<Item = Series> + '_ + Send + Sync + ExactSizeIterator + DoubleEndedIterator`[src]

`impl<T: PolarsObject> ChunkedArray<ObjectType<T>>`[src]

`pub fn into_no_null_iter( &self ) -> impl Iterator<Item = &T> + '_ + Send + Sync + ExactSizeIterator + DoubleEndedIterator`[src]

`impl ChunkedArray<CategoricalType>`[src]

`pub fn into_no_null_iter( &self ) -> impl Iterator<Item = u32> + '_ + Send + Sync + ExactSizeIterator + DoubleEndedIterator`[src]

`impl<T> ChunkedArray<T> where T: PolarsNumericType,` [src]

`pub fn to_ndarray(&self) -> Result<ArrayView1<'_, T::Native>>`[src]

This is supported on crate feature ndarray only.

If data is aligned in a single chunk and has no Null values a zero copy view is returned as an ndarray

`impl ChunkedArray<ListType>`[src]

`pub fn to_ndarray<N>(&self) -> Result<Array2<N::Native>> where N: PolarsNumericType,` [src]

This is supported on crate feature ndarray only.

If all nested Series have the same length, a 2 dimensional ndarray::Array is returned.

`impl<T> ChunkedArray<ObjectType<T>> where T: PolarsObject,` [src]

`pub fn new_from_vec(name: &str, v: Vec<T>) -> Self`[src]

This is supported on crate feature object only.

`impl<T> ChunkedArray<ObjectType<T>> where T: PolarsObject,` [src]

`pub unsafe fn get_as_any(&self, index: usize) -> &dyn Any`[src]

This is supported on crate feature object only.

Safety

No bounds checks

`impl<T> ChunkedArray<T> where ChunkedArray<T>: ChunkTake,` [src]

`pub fn sample_n(&self, n: usize, with_replacement: bool) -> Result<Self>`[src]

This is supported on crate feature random only.

Sample n datapoints from this ChunkedArray.

`pub fn sample_frac(&self, frac: f64, with_replacement: bool) -> Result<Self>`[src]

This is supported on crate feature random only.

Sample a fraction between 0.0-1.0 of this ChunkedArray.

`impl<T> ChunkedArray<T> where T: PolarsNumericType, T::Native: Float + NumCast,` [src]

`pub fn rand_normal( name: &str, length: usize, mean: f64, std_dev: f64 ) -> Result<Self>`[src]

This is supported on crate feature random only.

Create ChunkedArray with samples from a Normal distribution.

`pub fn rand_standard_normal(name: &str, length: usize) -> Self`[src]

This is supported on crate feature random only.

Create ChunkedArray with samples from a Standard Normal distribution.

`pub fn rand_uniform(name: &str, length: usize, low: f64, high: f64) -> Self`[src]

This is supported on crate feature random only.

Create ChunkedArray with samples from a Uniform distribution.

`impl ChunkedArray<BooleanType>`[src]

`pub fn rand_bernoulli(name: &str, length: usize, p: f64) -> Result<Self>`[src]

This is supported on crate feature random only.

Create ChunkedArray with samples from a Bernoulli distribution.

`impl ChunkedArray<Utf8Type>`[src]

`pub fn str_lengths(&self) -> UInt32Chunked`[src]

This is supported on crate feature strings only.

Get the length of the string values.

`pub fn contains(&self, pat: &str) -> Result<BooleanChunked>`[src]

This is supported on crate feature strings only.

Check if strings contain a regex pattern

`pub fn replace(&self, pat: &str, val: &str) -> Result<Utf8Chunked>`[src]

This is supported on crate feature strings only.

Replace the leftmost (sub)string by a regex pattern

`pub fn replace_all(&self, pat: &str, val: &str) -> Result<Utf8Chunked>`[src]

This is supported on crate feature strings only.

Replace all (sub)strings by a regex pattern

`pub fn to_lowercase(&self) -> Utf8Chunked`[src]

This is supported on crate feature strings only.

Modify the strings to their lowercase equivalent

`pub fn to_uppercase(&self) -> Utf8Chunked`[src]

This is supported on crate feature strings only.

Modify the strings to their uppercase equivalent

`pub fn concat(&self, other: &Utf8Chunked) -> Self`[src]

This is supported on crate feature strings only.

Concat with the values from a second Utf8Chunked

`pub fn str_slice(&self, start: i64, length: Option<u64>) -> Result<Self>`[src]

This is supported on crate feature strings only.

Slice the string values Determines a substring starting from start and with optional length length of each of the elements in array. start can be negative, in which case the start counts from the end of the string.