Crate vortex_array

Expand description

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays.

Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Modules§

accessor
arrays: All the built-in encoding schemes and arrays.
arrow: Utilities to work with Arrow data and types.
buffer
builders: Builders for Vortex arrays.
builtins: A collection of built-in common scalar functions.
compute: Compute kernels on top of Vortex Arrays.
display
expr: Vortex’s expression language.
flatbuffers: Re-exported autogenerated code from the core Vortex flatbuffer definitions.
iter: Iterator over slices of an array, and related utilities.
kernel
mask
matcher
normalize
optimizer
patches
scalar: Scalar values and types for the Vortex system.
search_sorted
serde
session
stats: Traits and utilities to compute and access array statistics.
stream
test_harness
validity: Array validity and nullability behavior, used by arrays and compute functions.
variants: This module defines extension functionality specific to each Vortex DType.
vtable: This module contains the VTable definitions for a Vortex encoding.

Macros§

assert_arrays_eq
assert_nth_scalar: Asserts that the scalar at position $n in array $arr equals $expected.
assert_nth_scalar_is_null: Asserts that the scalar at position $n in array $arr is null.
match_each_pvalue: Utility macro that makes it easy to write expressions generic over the different PValue variants.
register_kernel: Register a kernel for a compute function. See each compute function for the correct type of kernel to register.
search_sorted_conformance_6161102565324939617: Apply #macro_name template to given body
vtable

Structs§

AnyCanonical: A matcher for any canonical array type.
AnyColumnar
ArrayAdapter: Adapter struct used to lift the VTable trait into an object-safe Array implementation.
CanonicalValidity: Recursively execute the array until it reaches canonical form along with its validity.
EmptyMetadata: Empty array metadata
ExecutionCtx: Execution context for batch CPU compute.
MaskFuture: A future that resolves to a mask.
ProstMetadata: A utility wrapper for Prost metadata serialization.
RawMetadata: A utility wrapper for raw metadata serialization. This delegates the serialiation step to the arrays’ vtable.
RecursiveCanonical: Recursively execute the array until all of its children are canonical.

Enums§

Canonical: An enum capturing the default uncompressed encodings for each Vortex type.
CanonicalView: A view into a canonical array type.
Columnar: Represents a columnnar array of data, either in canonical form or as a constant array.
ColumnarView
Precision: The precision level for structural equality and hashing of arrays.

Statics§

LEGACY_SESSION

Traits§

Array: The public API trait for all Vortex arrays.
ArrayBufferVisitor
ArrayChildVisitor
ArrayChildVisitorUnnamed: A visitor for array children that does not require names.
ArrayEq: An equality trait for arrays that represents structural equality with a configurable level of precision. This trait is used primarily to implement common subtree elimination and other array-based caching mechanisms.
ArrayHash: A hash trait for arrays that represents structural equality with a configurable level of precision. This trait is used primarily to implement common subtree elimination and other array-based caching mechanisms.
ArrayVisitor
ArrayVisitorExt
DeserializeMetadata: Trait for deserializing Vortex metadata from a vector of unaligned bytes.
DynArrayEq: A dynamic version of ArrayEq.
DynArrayHash: A dynamic version of ArrayHash.
Executable: Marker trait for types that an ArrayRef can be executed into.
IntoArray: Trait for converting a type into a Vortex ArrayRef.
SerializeMetadata: Trait for serializing Vortex metadata to a vector of unaligned bytes.
ToCanonical: Trait for types that can be converted from an owned type into an owned array variant.
VortexSessionExecute: Extension trait for creating an execution context from a session.

Type Aliases§

ArrayContext
ArrayRef: A reference counted pointer to a dynamic Array trait object.