Crate vortex_array 

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays.

Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Modules

accessor

arrays

All the built-in encoding schemes and arrays.

arrow

Utilities to work with Arrow data and types.

buffer

builders

Builders for Vortex arrays.

builtins

A collection of built-in common scalar functions.

compute

Compute kernels on top of Vortex Arrays.

display

expr

Vortex’s expression language.

flatbuffers

Re-exported autogenerated code from the core Vortex flatbuffer definitions.

iter

Iterator over slices of an array, and related utilities.

kernel

mask

matcher

normalize

optimizer

patches

scalar

Scalar values and types for the Vortex system.

search_sorted

serde

session

stats

Traits and utilities to compute and access array statistics.

stream

test_harness

validity

Array validity and nullability behavior, used by arrays and compute functions.

variants

This module defines extension functionality specific to each Vortex DType.

vtable

This module contains the VTable definitions for a Vortex encoding.

Macros

assert_arrays_eq

assert_nth_scalar

Asserts that the scalar at position $n in array $arr equals $expected.

assert_nth_scalar_is_null

Asserts that the scalar at position $n in array $arr is null.

match_each_pvalue

Utility macro that makes it easy to write expressions generic over the different PValue variants.

register_kernel

Register a kernel for a compute function. See each compute function for the correct type of kernel to register.

search_sorted_conformance_9014284834973034448

Apply #macro_name template to given body

vtable

Structs

AnyCanonical

A matcher for any canonical array type.

AnyColumnar

ArrayAdapter

Adapter struct used to lift the VTable trait into an object-safe Array implementation.

CanonicalValidity

Recursively execute the array until it reaches canonical form along with its validity.

EmptyMetadata

Empty array metadata

ExecutionCtx

Execution context for batch CPU compute.

MaskFuture

A future that resolves to a mask.

ProstMetadata

A utility wrapper for Prost metadata serialization.

RawMetadata

A utility wrapper for raw metadata serialization. This delegates the serialiation step to the arrays’ vtable.

RecursiveCanonical

Recursively execute the array until all of its children are canonical.

Enums

Canonical

An enum capturing the default uncompressed encodings for each Vortex type.

CanonicalView

A view into a canonical array type.

Columnar

Represents a columnnar array of data, either in canonical form or as a constant array.

ColumnarView

Precision

The precision level for structural equality and hashing of arrays.

Statics

LEGACY_SESSION

Traits

Array

The public API trait for all Vortex arrays.

ArrayBufferVisitor

ArrayChildVisitor

ArrayChildVisitorUnnamed

A visitor for array children that does not require names.

ArrayEq

An equality trait for arrays that represents structural equality with a configurable level of precision. This trait is used primarily to implement common subtree elimination and other array-based caching mechanisms.

ArrayHash

A hash trait for arrays that represents structural equality with a configurable level of precision. This trait is used primarily to implement common subtree elimination and other array-based caching mechanisms.

ArrayVisitor

ArrayVisitorExt

DeserializeMetadata

Trait for deserializing Vortex metadata from a vector of unaligned bytes.

DynArrayEq

A dynamic version of ArrayEq.

DynArrayHash

A dynamic version of ArrayHash.

Executable

Marker trait for types that an ArrayRef can be executed into.

IntoArray

Trait for converting a type into a Vortex ArrayRef.

SerializeMetadata

Trait for serializing Vortex metadata to a vector of unaligned bytes.

ToCanonical

Trait for types that can be converted from an owned type into an owned array variant.

VortexSessionExecute

Extension trait for creating an execution context from a session.

Type Aliases

ArrayContext

ArrayRef

A reference counted pointer to a dynamic Array trait object.