Crate vortex_array

Vortex crate containing core logic for encoding and memory representation of arrays.

At the heart of Vortex are arrays and encodings. Arrays are typed views of memory buffers that hold scalars. These buffers can be held in a number of physical encodings to perform lightweight compression that exploits the particular data distribution of the array’s values.

Every data type recognized by Vortex also has a canonical physical encoding format, which arrays can be canonicalized into for ease of access in compute functions.

Re-exports

• pub use paste;

Modules

• accessor
• aliases
• array
  All the built-in encoding schemes and arrays.
• arrow
  Utilities to work with Arrow data and types.
• compress
• compute
  Compute kernels on top of Vortex Arrays.
• elementwise
• encoding
  Traits and types to define shared unique encoding identifiers.
• flatbuffers
  Re-exported autogenerated code from the core Vortex flatbuffer definitions.
• iter
• stats
  Traits and utilities to compute and access array statistics.
• stream
• validity
  Array validity and nullability behavior, used by arrays and compute functions.
• variants
  This module defines array traits for each Vortex DType.

Macros

• impl_encoding
  Macro to generate all the necessary code for a new type of array encoding. Including:
• iterate_integer_array
  Iterate over an array of integers by dispatching at run-time on the array type.
• iterate_primitive_array
  Iterate over an array of primitives by dispatching at run-time on the array type.

Structs

• Array
  A central type for all Vortex arrays, which are known length sequences of typed and possibly compressed data.
• ArrayChildrenIterator
  A depth-first pre-order iterator over a ArrayData.
• ArrayData
  Owned Array with serialized metadata, backed by heap-allocated memory.
• ArrayView
  Zero-copy view over flatbuffer-encoded array data, created without eager serialization.
• Context
  A mapping between an encoding’s ID to an EncodingRef, used to have a shared view of all available encoding schemes.
• TypedArray
  Container for an array with all the associated implementation type information (encoding reference and ID, actual array type, metadata type).

Enums

• Canonical
  The set of canonical array encodings, also the set of encodings that can be transferred to Arrow with zero-copy.

Traits

• ArrayDType
• ArrayDef
  Trait the defines the set of types relating to an array. Because it has associated types it can’t be used as a trait object.
• ArrayLen
• ArrayMetadata
  Dynamic trait used to represent opaque owned Array metadata
• ArrayTrait
  Collects together the behavior of an array.
• GetArrayMetadata
• IntoArray
  Consume self and turn it into an Array infallibly.
• IntoArrayVariant
  Trait for types that can be converted from an owned type into an owned array variant.
• IntoCanonical
  Support trait for transmuting an array into its vortex_dtype::DType’s canonical encoding.
• ToArray
• ToArrayData
• TryDeserializeArrayMetadata
• TrySerializeArrayMetadata