Struct polars_core::chunked_array::ChunkedArray[−][src]

pub struct ChunkedArray<T> { /* fields omitted */ }

ChunkedArray

Every Series contains a ChunkedArray<T>. Unlike Series, ChunkedArray's are typed. This allows us to apply closures to the data and collect the results to a ChunkedArray of te same type T. Below we use an apply to use the cosine function to the values of a ChunkedArray.

fn apply_cosine(ca: &Float32Chunked) -> Float32Chunked {
    ca.apply(|v| v.cos())
}

If we would like to cast the result we could use a Rust Iterator instead of an apply method. Note that Iterators are slightly slower as the null values aren't ignored implicitly.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float64Chunked {
    ca.into_iter()
        .map(|opt_v| {
        opt_v.map(|v| v.cos() as f64)
    }).collect()
}

Another option is to first cast and then use an apply.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float64Chunked {
    ca.cast::<Float64Type>()
        .unwrap()
        .apply(|v| v.cos())
}

Conversion between Series and ChunkedArray's

Conversion from a Series to a ChunkedArray is effortless.

fn to_chunked_array(series: &Series) -> Result<&Int32Chunked>{
    series.i32()
}

fn to_series(ca: Int32Chunked) -> Series {
    ca.into_series()
}

Iterators

ChunkedArrays fully support Rust native Iterator and DoubleEndedIterator traits, thereby giving access to all the excelent methods available for Iterators.


fn iter_forward(ca: &Float32Chunked) {
    ca.into_iter()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

fn iter_backward(ca: &Float32Chunked) {
    ca.into_iter()
        .rev()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

Memory layout

ChunkedArray's use Apache Arrow as backend for the memory layout. Arrows memory is immutable which makes it possible to make mutliple zero copy (sub)-views from a single array.

To be able to append data, Polars uses chunks to append new memory locations, hence the ChunkedArray<T> data structure. Appends are cheap, because it will not lead to a full reallocation of the whole array (as could be the case with a Rust Vec).

However, multiple chunks in a ChunkArray will slow down the Iterators, arithmetic and other operations. When multiplying two ChunkArray's with different chunk sizes they cannot utilize SIMD for instance. However, when chunk size don't match, Iterators will be used to do the operation (instead of arrows upstream implementation, which may utilize SIMD) and the result will be a single chunked array.

The key takeaway is that by applying operations on a ChunkArray of multiple chunks, the results will converge to a ChunkArray of a single chunk! It is recommended to leave them as is. If you want to have predictable performance (no unexpected re-allocation of memory), it is adviced to call the rechunk after multiple append operations.

Implementations

`impl ChunkedArray<BooleanType>`[src]

`pub fn arg_true(&self) -> UInt32Chunked`[src]

`impl ChunkedArray<BooleanType>`[src]

`pub fn all_true(&self) -> bool`[src]