Struct polars::prelude::ChunkedArray[−][src]

pub struct ChunkedArray<T> { /* fields omitted */ }

ChunkedArray

Every Series contains a ChunkedArray<T>. Unlike Series, ChunkedArray's are typed. This allows us to apply closures to the data and collect the results to a ChunkedArray of te same type T. Below we use an apply to use the cosine function to the values of a ChunkedArray.

fn apply_cosine(ca: &Float32Chunked) -> Float32Chunked {
    ca.apply(|v| v.cos())
}

If we would like to cast the result we could use a Rust Iterator instead of an apply method. Note that Iterators are slightly slower as the null values aren't ignored implicitly.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float64Chunked {
    ca.into_iter()
        .map(|opt_v| {
        opt_v.map(|v| v.cos() as f64)
    }).collect()
}

Another option is to first cast and then use an apply.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float64Chunked {
    ca.cast::<Float64Type>()
        .unwrap()
        .apply(|v| v.cos())
}

Conversion between Series and ChunkedArray's

Conversion from a Series to a ChunkedArray is effortless.

fn to_chunked_array(series: &Series) -> Result<&Int32Chunked>{
    series.i32()
}

fn to_series(ca: Int32Chunked) -> Series {
    ca.into_series()
}

Iterators

ChunkedArrays fully support Rust native Iterator and DoubleEndedIterator traits, thereby giving access to all the excelent methods available for Iterators.


fn iter_forward(ca: &Float32Chunked) {
    ca.into_iter()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

fn iter_backward(ca: &Float32Chunked) {
    ca.into_iter()
        .rev()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

Memory layout

ChunkedArray's use Apache Arrow as backend for the memory layout. Arrows memory is immutable which makes it possible to make mutliple zero copy (sub)-views from a single array.

To be able to append data, Polars uses chunks to append new memory locations, hence the ChunkedArray<T> data structure. Appends are cheap, because it will not lead to a full reallocation of the whole array (as could be the case with a Rust Vec).

However, multiple chunks in a ChunkArray will slow down the Iterators, arithmetic and other operations. When multiplying two ChunkArray's with different chunk sizes they cannot utilize SIMD for instance. However, when chunk size don't match, Iterators will be used to do the operation (instead of arrows upstream implementation, which may utilize SIMD) and the result will be a single chunked array.

The key takeaway is that by applying operations on a ChunkArray of multiple chunks, the results will converge to a ChunkArray of a single chunk! It is recommended to leave them as is. If you want to have predictable performance (no unexpected re-allocation of memory), it is adviced to call the rechunk after multiple append operations.

Implementations

`impl ChunkedArray<BooleanType>`[src]

`pub fn arg_true(&self) -> ChunkedArray<UInt32Type>`[src]

`impl ChunkedArray<BooleanType>`[src]

`pub fn all_true(&self) -> bool`[src]

`impl ChunkedArray<BooleanType>`[src]

`pub fn all_false(&self) -> bool`[src]

`impl<T> ChunkedArray<T> where T: PolarsNumericType,` [src]

`pub fn to_ndarray( &self ) -> Result<ArrayBase<ViewRepr<&<T as ArrowPrimitiveType>::Native>, Dim<[usize; 1]>>, PolarsError>`[src]

If data is aligned in a single chunk and has no Null values a zero copy view is returned as an ndarray

`impl ChunkedArray<ListType>`[src]

`pub fn to_ndarray<N>( &self ) -> Result<ArrayBase<OwnedRepr<<N as ArrowPrimitiveType>::Native>, Dim<[usize; 2]>>, PolarsError> where N: PolarsNumericType,` [src]

If all nested Series have the same length, a 2 dimensional ndarray::Array is returned.

`impl<T> ChunkedArray<ObjectType<T>> where T: Any + Debug + Clone + Send + Sync + Default,` [src]

`pub fn new_from_vec( name: &str, v: Vec<T, Global> ) -> ChunkedArray<ObjectType<T>>`[src]

`impl<T> ChunkedArray<ObjectType<T>> where T: Any + Debug + Clone + Send + Sync + Default,` [src]

`pub fn get_as_any(&self, index: usize) -> &(dyn Any + 'static)`[src]

`impl<T> ChunkedArray<T> where ChunkedArray<T>: ChunkTake,` [src]

`pub fn sample_n( &self, n: usize, with_replacement: bool ) -> Result<ChunkedArray<T>, PolarsError>`[src]

Sample n datapoints from this ChunkedArray.

`pub fn sample_frac( &self, frac: f64, with_replacement: bool ) -> Result<ChunkedArray<T>, PolarsError>`[src]

Sample a fraction between 0.0-1.0 of this ChunkedArray.

`impl<T> ChunkedArray<T> where T: PolarsNumericType, <T as ArrowPrimitiveType>::Native: Float, <T as ArrowPrimitiveType>::Native: NumCast,` [src]

`pub fn rand_normal( name: &str, length: usize, mean: f64, std_dev: f64 ) -> Result<ChunkedArray<T>, PolarsError>`[src]

Create ChunkedArray with samples from a Normal distribution.

`pub fn rand_standard_normal(name: &str, length: usize) -> ChunkedArray<T>`[src]

Create ChunkedArray with samples from a Standard Normal distribution.

`pub fn rand_uniform( name: &str, length: usize, low: f64, high: f64 ) -> ChunkedArray<T>`[src]

Create ChunkedArray with samples from a Uniform distribution.

`impl ChunkedArray<BooleanType>`[src]

`pub fn rand_bernoulli( name: &str, length: usize, p: f64 ) -> Result<ChunkedArray<BooleanType>, PolarsError>`[src]

Create ChunkedArray with samples from a Bernoulli distribution.

`impl ChunkedArray<Utf8Type>`[src]

`pub fn str_lengths(&self) -> ChunkedArray<UInt32Type>`[src]

Get the length of the string values.

`pub fn contains( &self, pat: &str ) -> Result<ChunkedArray<BooleanType>, PolarsError>`[src]

Check if strings contain a regex pattern

`pub fn replace( &self, pat: &str, val: &str ) -> Result<ChunkedArray<Utf8Type>, PolarsError>`[src]

Replace the leftmost (sub)string by a regex pattern

`pub fn replace_all( &self, pat: &str, val: &str ) -> Result<ChunkedArray<Utf8Type>, PolarsError>`[src]

Replace all (sub)strings by a regex pattern

`pub fn to_lowercase(&self) -> ChunkedArray<Utf8Type>`[src]

Modify the strings to their lowercase equivalent

`pub fn to_uppercase(&self) -> ChunkedArray<Utf8Type>`[src]

Modify the strings to their uppercase equivalent

`pub fn concat(&self, other: &ChunkedArray<Utf8Type>) -> ChunkedArray<Utf8Type>`[src]

Concat with the values from a second Utf8Chunked

`impl ChunkedArray<Utf8Type>`[src]

`pub fn as_date32( &self, fmt: Option<&str> ) -> Result<ChunkedArray<Date32Type>, PolarsError>`[src]

`pub fn as_date64( &self, fmt: Option<&str> ) -> Result<ChunkedArray<Date64Type>, PolarsError>`[src]

`impl ChunkedArray<Date64Type>`[src]

`pub fn year(&self) -> ChunkedArray<Int32Type>`[src]

Extract month from underlying NaiveDateTime representation. Returns the year number in the calendar date.

`pub fn month(&self) -> ChunkedArray<UInt32Type>`[src]

Extract month from underlying NaiveDateTime representation. Returns the month number starting from 1.

The return value ranges from 1 to 12.

`pub fn day(&self) -> ChunkedArray<UInt32Type>`[src]

Extract day from underlying NaiveDateTime representation. Returns the day of month starting from 1.

The return value ranges from 1 to 31. (The last day of month differs by months.)

`pub fn hour(&self) -> ChunkedArray<UInt32Type>`[src]

Extract hour from underlying NaiveDateTime representation. Returns the hour number from 0 to 23.

`pub fn minute(&self) -> ChunkedArray<UInt32Type>`[src]

Extract minute from underlying NaiveDateTime representation. Returns the minute number from 0 to 59.

`pub fn second(&self) -> ChunkedArray<UInt32Type>`[src]

Extract second from underlying NaiveDateTime representation. Returns the second number from 0 to 59.

`pub fn nanosecond(&self) -> ChunkedArray<UInt32Type>`[src]

Extract second from underlying NaiveDateTime representation. Returns the number of nanoseconds since the whole non-leap second. The range from 1,000,000,000 to 1,999,999,999 represents the leap second.

`pub fn ordinal(&self) -> ChunkedArray<UInt32Type>`[src]

Returns the day of year starting from 1.

The return value ranges from 1 to 366. (The last day of year differs by years.)

`pub fn str_fmt(&self, fmt: &str) -> ChunkedArray<Utf8Type>`[src]

Format Date64 with a fmt rule. See chrono strftime/strptime.

`impl ChunkedArray<Date32Type>`[src]

`pub fn year(&self) -> ChunkedArray<Int32Type>`[src]

Extract month from underlying NaiveDateTime representation. Returns the year number in the calendar date.

`pub fn month(&self) -> ChunkedArray<UInt32Type>`[src]

Extract month from underlying NaiveDateTime representation. Returns the month number starting from 1.

The return value ranges from 1 to 12.

`pub fn day(&self) -> ChunkedArray<UInt32Type>`[src]

Extract day from underlying NaiveDateTime representation. Returns the day of month starting from 1.

The return value ranges from 1 to 31. (The last day of month differs by months.)

`pub fn ordinal(&self) -> ChunkedArray<UInt32Type>`[src]

Returns the day of year starting from 1.

The return value ranges from 1 to 366. (The last day of year differs by years.)

`pub fn str_fmt(&self, fmt: &str) -> ChunkedArray<Utf8Type>`[src]

Format Date32 with a fmt rule. See chrono strftime/strptime.

`impl<T> ChunkedArray<T>`[src]

`pub fn array_data(&self) -> Vec<Arc<ArrayData>, Global>`[src]

Get Arrow ArrayData

`pub fn get_categorical_map( &self ) -> Option<&Arc<AHashMap<u32, String, RandomState>>>`[src]

Get a reference to the mapping of categorical types to the string values.

`pub fn first_non_null(&self) -> Option<usize>`[src]

Get the index of the first non null value in this ChunkedArray.

`pub fn null_bits(&self) -> Vec<(usize, Option<Buffer>), Global>`[src]

Get the null count and the buffer of bits representing null values

`pub fn unpack_series_matching_type( &self, series: &Series ) -> Result<&ChunkedArray<T>, PolarsError>`[src]

Series to ChunkedArray

`pub fn len(&self) -> usize`[src]

Combined length of all the chunks.

`pub fn is_empty(&self) -> bool`[src]

Check if ChunkedArray is empty.

`pub fn chunk_id(&self) -> &Vec<usize, Global>`[src]

Unique id representing the number of chunks

`pub fn chunks(&self) -> &Vec<Arc<dyn Array + 'static>, Global>`[src]

A reference to the chunks

`pub fn is_optimal_aligned(&self) -> bool`[src]

Returns true if contains a single chunk and has no null values

`pub fn null_count(&self) -> usize`[src]

Count the null values.

`pub fn limit(&self, num_elements: usize) -> Result<ChunkedArray<T>, PolarsError>`[src]

Take a view of top n elements

`pub fn append_array( &mut self, other: Arc<dyn Array + 'static> ) -> Result<(), PolarsError>`[src]

Append arrow array in place.

let mut array = Int32Chunked::new_from_slice("array", &[1, 2]);
let array_2 = Int32Chunked::new_from_slice("2nd", &[3]);

array.append(&array_2);
assert_eq!(Vec::from(&array), [Some(1), Some(2), Some(3)])

`pub fn slice( &self, offset: usize, length: usize ) -> Result<ChunkedArray<T>, PolarsError>`[src]

Slice the array. The chunks are reallocated the underlying data slices are zero copy.

`pub fn is_null(&self) -> ChunkedArray<BooleanType>`[src]

Get a mask of the null values.

`pub fn is_not_null(&self) -> ChunkedArray<BooleanType>`[src]

Get a mask of the null values.

`pub fn dtype(&self) -> &DataType`[src]

Get data type of ChunkedArray.

`pub fn head(&self, length: Option<usize>) -> ChunkedArray<T>`[src]

Get the head of the ChunkedArray

`pub fn tail(&self, length: Option<usize>) -> ChunkedArray<T>`[src]

Get the tail of the ChunkedArray

`pub fn append(&mut self, other: &ChunkedArray<T>) where ChunkedArray<T>: Sized,` [src]

Append in place.

`pub fn name(&self) -> &str`[src]

Name of the ChunkedArray.

`pub fn ref_field(&self) -> &Field`[src]

Get a reference to the field.

`pub fn rename(&mut self, name: &str)`[src]

Rename this ChunkedArray.

`impl<T> ChunkedArray<T> where T: PolarsDataType,` [src]

`pub fn new_from_chunks( name: &str, chunks: Vec<Arc<dyn Array + 'static>, Global> ) -> ChunkedArray<T>`[src]

Create a new ChunkedArray from existing chunks.

`impl<T> ChunkedArray<T> where T: PolarsPrimitiveType,` [src]

`pub fn new_from_aligned_vec( name: &str, v: AlignedVec<<T as ArrowPrimitiveType>::Native> ) -> ChunkedArray<T>`[src]

Create a new ChunkedArray by taking ownership of the AlignedVec. This operation is zero copy.

`pub fn new_with_null_bitmap( name: &str, values: &[<T as ArrowPrimitiveType>::Native], buffer: Option<Buffer>, null_count: usize ) -> ChunkedArray<T>`[src]

Nullify values in slice with an existing null bitmap

`pub fn new_from_owned_with_null_bitmap( name: &str, values: AlignedVec<<T as ArrowPrimitiveType>::Native>, buffer: Option<Buffer>, null_count: usize ) -> ChunkedArray<T>`[src]

Nullify values in slice with an existing null bitmap

`impl<T> ChunkedArray<T> where T: PolarsNumericType,` [src]

`pub fn cont_slice( &self ) -> Result<&[<T as ArrowPrimitiveType>::Native], PolarsError>`[src]

Contiguous slice

`pub fn data_views(&self) -> Vec<&[<T as ArrowPrimitiveType>::Native], Global>`[src]

Get slices of the underlying arrow data. NOTE: null values should be taken into account by the user of these slices as they are handled separately

`pub fn map<B, F>( &self, f: F ) -> Result<Map<Copied<Iter<'_, <T as ArrowPrimitiveType>::Native>>, F>, PolarsError> where F: Fn(<T as ArrowPrimitiveType>::Native) -> B,` [src]

If cont_slice is successful a closure is mapped over the elements.

Example

use polars_core::prelude::*;
fn multiply(ca: &UInt32Chunked) -> Result<Series> {
    let mapped = ca.map(|v| v * 2)?;
    Ok(mapped.collect())
}

`pub fn map_null_checks<B, F>( &'a self, f: F ) -> Map<Box<dyn PolarsIterator<Item = Option<<T as ArrowPrimitiveType>::Native>> + 'a, Global>, F> where F: Fn(Option<<T as ArrowPrimitiveType>::Native>) -> B,` [src]

If cont_slice fails we can fallback on an iterator with null checks and map a closure over the elements.

Example

use polars_core::prelude::*;
use itertools::Itertools;
fn multiply(ca: &UInt32Chunked) -> Series {
    let mapped_result = ca.map(|v| v * 2);

    if let Ok(mapped) = mapped_result {
        mapped.collect()
    } else {
        ca
        .map_null_checks(|opt_v| opt_v.map(|v |v * 2)).collect()
    }
}

`pub fn fold<F, B>(&self, init: B, f: F) -> Result<B, PolarsError> where F: Fn(B, <T as ArrowPrimitiveType>::Native) -> B,` [src]

If cont_slice is successful a closure can be applied as aggregation

Example

use polars_core::prelude::*;
fn compute_sum(ca: &UInt32Chunked) -> Result<u32> {
    ca.fold(0, |acc, value| acc + value)
}

`pub fn fold_null_checks<F, B>(&self, init: B, f: F) -> B where F: Fn(B, Option<<T as ArrowPrimitiveType>::Native>) -> B,` [src]

If cont_slice fails we can fallback on an iterator with null checks and a closure for aggregation

Example

use polars_core::prelude::*;
fn compute_sum(ca: &UInt32Chunked) -> u32 {
    match ca.fold(0, |acc, value| acc + value) {
        // faster sum without null checks was successful
        Ok(sum) => sum,
        // Null values or multiple chunks in ChunkedArray, we need to do more bounds checking
        Err(_) => ca.fold_null_checks(0, |acc, opt_value| {
            match opt_value {
                Some(v) => acc + v,
                None => acc
            }
        })
    }
}