Crate vortex_array

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays and encodings. Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Re-exports

pub use paste;

pub use partial_ord::*;

Modules

accessor

aliases

array

All the built-in encoding schemes and arrays.

arrow

Utilities to work with Arrow data and types.

builders

compress

compute

Compute kernels on top of Vortex Arrays.

encoding_ids

Encoding ID constants for all Vortex-provided encodings

flatbuffers

Re-exported autogenerated code from the core Vortex flatbuffer definitions.

iter

nbytes

opaque

partial_ord

parts

patches

stats

Traits and utilities to compute and access array statistics.

stream

tree

validity

Array validity and nullability behavior, used by arrays and compute functions.

variants

This module defines array traits for each Vortex DType.

visitor

vtable

This module contains the VTable definitions for a Vortex Array.

Macros

impl_encoding

Macro to generate all the necessary code for a new type of array encoding. Including:

Structs

Array

A central type for all Vortex arrays, which are known length sequences of typed and possibly compressed data.

ArrayChildrenIterator

A depth-first pre-order iterator over a Array.

ChildrenCollector

Context

A mapping between an encoding’s ID to an VTableRef, used to have a shared view of all available encoding schemes.

EmptyMetadata

EncodingId

EncodingId is a unique name and numerical code of the array

NamedChildrenCollector

RkyvMetadata

A utility wrapper for automating the serialization of metadata using rkyv.

SerdeMetadata

Enums

Canonical

The set of canonical array encodings, also the set of encodings that can be transferred to Arrow with zero-copy.

Traits

ArrayMetadata

DeserializeMetadata

Encoding

Marker trait for array encodings with their associated Array type.

IntoArray

Consume self and turn it into an Array infallibly.

IntoArrayVariant

Trait for types that can be converted from an owned type into an owned array variant.

IntoCanonical

Canonicalize an Array into one of the Canonical array forms.

SerializeMetadata

Type Aliases

ContextRef

An atomic shared reference to a Context.