Crate vortex_array

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays and encodings. Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Modules

accessor

aliases

Re-exports of third-party crates we use in macros exported from vortex-array.

arrays

All the built-in encoding schemes and arrays.

arrow

Utilities to work with Arrow data and types.

builders

Builders for Vortex arrays.

compress

compute

Compute kernels on top of Vortex Arrays.

data

flatbuffers

Re-exported autogenerated code from the core Vortex flatbuffer definitions.

iter

Iterator over slices of an array, and related utilities.

patches

search_sorted

serde

stats

Traits and utilities to compute and access array statistics.

stream

validity

Array validity and nullability behavior, used by arrays and compute functions.

variants

This module defines extension functionality specific to each Vortex DType.

vtable

This module contains the VTable definitions for a Vortex encoding.

Macros

register_kernel

Register a kernel for a compute function. See each compute function for the correct type of kernel to register.

vtable

Structs

ArrayAdapter

Adapter struct used to lift the VTable trait into an object-safe Array implementation.

EmptyMetadata

Empty array metadata

EncodingAdapter

Adapter struct used to lift the VTable trait into an object-safe Encoding implementation.

ProstMetadata

A utility wrapper for Prost metadata serialization.

RawMetadata

A utility wrapper for raw metadata serialization. This delegates the serialiation step to the arrays’ vtable.

VTableContext

A collection of encodings that can be addressed by a u16 positional index. This is used to map array encodings and layout encodings when reading from a file.

VTableRegistry

A registry of encodings that can be used to construct a context for serde.

Enums

Canonical

An enum capturing the default uncompressed encodings for each Vortex type.

Traits

Array

The public API trait for all Vortex arrays.

ArrayBufferVisitor

ArrayChildVisitor

ArrayVisitor

ArrayVisitorExt

DeserializeMetadata

Trait for deserializing Vortex metadata from a vector of unaligned bytes.

Encoding

Marker trait for array encodings with their associated Array type.

IntoArray

Trait for converting a type into a Vortex ArrayRef.

SerializeMetadata

Trait for serializing Vortex metadata to a vector of unaligned bytes.

ToCanonical

Trait for types that can be converted from an owned type into an owned array variant.

TryIntoArray

Trait for converting a type into a Vortex ArrayRef, returning an error if the conversion fails.

Type Aliases

ArrayContext

A collection of array encodings.

ArrayRef

A reference counted pointer to a dynamic Array trait object.

ArrayRegistry

EncodingId

EncodingId is a globally unique name of the array’s encoding.

EncodingRef