Crate vortex_array 

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays and encodings. Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Modules

accessor

arrays

All the built-in encoding schemes and arrays.

arrow

Utilities to work with Arrow data and types.

builders

Builders for Vortex arrays.

compute

Compute kernels on top of Vortex Arrays.

display

execution

expr

Vortex’s expression language.

flatbuffers

Re-exported autogenerated code from the core Vortex flatbuffer definitions.

iter

Iterator over slices of an array, and related utilities.

optimizer

patches

pipeline

search_sorted

serde

stats

Traits and utilities to compute and access array statistics.

stream

test_harness

validity

Array validity and nullability behavior, used by arrays and compute functions.

variants

This module defines extension functionality specific to each Vortex DType.

vectors

vtable

This module contains the VTable definitions for a Vortex encoding.

Macros

assert_arrays_eq

register_kernel

Register a kernel for a compute function. See each compute function for the correct type of kernel to register.

search_sorted_conformance_6200121742319254681

Apply #macro_name template to given body

vtable

Structs

ArrayAdapter

Adapter struct used to lift the VTable trait into an object-safe Array implementation.

ArraySession

EmptyMetadata

Empty array metadata

EncodingAdapter

Adapter struct used to lift the VTable trait into an object-safe Encoding implementation.

MaskFuture

A future that resolves to a mask.

ProstMetadata

A utility wrapper for Prost metadata serialization.

RawMetadata

A utility wrapper for raw metadata serialization. This delegates the serialiation step to the arrays’ vtable.

VTableContext

A collection of encodings that can be addressed by a u16 positional index. This is used to map array encodings and layout encodings when reading from a file.

Enums

Canonical

An enum capturing the default uncompressed encodings for each Vortex type.

Precision

The precision level for structural equality and hashing of arrays.

Traits

Array

The public API trait for all Vortex arrays.

ArrayBufferVisitor

ArrayChildVisitor

ArrayEq

An equality trait for arrays that represents structural equality with a configurable level of precision. This trait is used primarily to implement common subtree elimination and other array-based caching mechanisms.

ArrayHash

A hash trait for arrays that represents structural equality with a configurable level of precision. This trait is used primarily to implement common subtree elimination and other array-based caching mechanisms.

ArrayOperator

Array functions as provided by the OperatorVTable.

ArraySessionExt

Session data for Vortex arrays.

ArrayVisitor

ArrayVisitorExt

DeserializeMetadata

Trait for deserializing Vortex metadata from a vector of unaligned bytes.

DynArrayEq

A dynamic version of ArrayEq.

DynArrayHash

A dynamic version of ArrayHash.

Encoding

Marker trait for array encodings with their associated Array type.

IntoArray

Trait for converting a type into a Vortex ArrayRef.

SerializeMetadata

Trait for serializing Vortex metadata to a vector of unaligned bytes.

ToCanonical

Trait for types that can be converted from an owned type into an owned array variant.

Type Aliases

ArrayContext

ArrayRef

A reference counted pointer to a dynamic Array trait object.

ArrayRegistry

EncodingId

EncodingId is a globally unique name of the array’s encoding.

EncodingRef