Crate runlength_arrow 

Run-length encoded sequences over Apache Arrow arrays.

This crate builds on runlength and uses its plans and merge rules, but stores values as an Arrow array, so values can be of any Arrow type: numbers, strings, dictionaries, lists, structs, and nulls in any of them.

• Combining sequences gathers values through a runlength::Plan and applies any Arrow kernel (kernels has the usual ones).
• Merging uses the same rule as the core crate for every type (identical_neighbours).
• ArrowRleBatch keys sequences by a table of key columns of any types, and converts to one long table for polars or pandas.

use std::sync::Arc;
use arrow_array::{ArrayRef, StringArray};
use runlength_arrow::{ArrowRle, kernels};

let names: ArrayRef = Arc::new(StringArray::from(vec!["a", "a", "b", "c", "c"]));
let rle = ArrowRle::from_dense(names).unwrap();
assert_eq!(rle.runs(), vec![2, 1, 2]);

let other: ArrayRef = Arc::new(StringArray::from(vec!["a", "x", "c", "c", "c"]));
let other = ArrowRle::from_dense(other).unwrap();
let same = rle.zip_with(&other, None, None, kernels::eq).unwrap();
assert_eq!(same.runs(), vec![1, 2, 2]);

Modules

kernels

Element-wise kernels in the shape zip_with expects.

Structs

ArrowRle

A run-length encoded sequence whose values are an Arrow array of any type.

ArrowRleBatch

Many Arrow-backed sequences, each identified by a row of a key table.

OverlapPlan

Which intervals cover each segment, for aggregating interval values in any way.

PlanMany

How to combine k sequences, computed from their run ends alone: the k-input version of Plan.

Enums

Agg

How coverage_agg combines the intervals covering a position.

Error

Errors from Arrow-backed sequences.

How

Which keys a combination of two batches keeps, as in a DataFrame join.

Keep

Which value a merged group keeps when its runs hold different values.

SliceAgg

How ArrowRle::slice_agg reduces each slice. Null values are skipped, as in polars and pandas.

Functions

common_numeric_type

The type two numeric types are converted to before combining them: themselves if equal, Float64 if either is a float, Int64 otherwise. None if either is not numeric.

concat

Joins sequences end to end and merges identical neighbours, including at the joins. All sequences must have the same value type.

coverage

Coverage of half-open intervals [start, end).

coverage_agg

Coverage of half-open intervals [start, end) with any aggregation.

coverage_agg_by_key

coverage_agg per key, computed in parallel; see coverage_by_key for the keys.

coverage_by_key

coverage per key, computed in parallel.

default_fill

The fill value operators use for a value type: zero for numbers, false for booleans, and None for other types (which then require equal lengths).

identical_neighbours

For each pair of neighbouring values, whether they are identical by the same rule as the core crate’s Identical: nulls are identical to nulls, NaN to NaN, and -0.0 is not identical to 0.0.

overlap_plan

Computes the OverlapPlan for intervals [start, end).

overlap_plan_by_key

runlength::overlap_plan per key, as one long table: the key columns, then start and end of each segment and ids, a list of the intervals covering it (positions in the input, sorted). Keys appear in order of first appearance; each key’s segments cover 0..max(end) of its own intervals.

plan_many

Computes the PlanMany for sequences with the given run ends.

zip_many

Combines several sequences position by position, and merges the result.

Type Aliases

Result