Enum Canonical

pub enum Canonical {
    Null(NullArray),
    Bool(BoolArray),
    Primitive(PrimitiveArray),
    Decimal(DecimalArray),
    Struct(StructArray),
    List(ListArray),
    VarBinView(VarBinViewArray),
    Extension(ExtensionArray),
}

An enum capturing the default uncompressed encodings for each Vortex type.

Any array can be decoded into canonical form via the to_canonical trait method. This is the simplest encoding for a type, and will not be compressed but may contain compressed child arrays.

Canonical form is useful for doing type-specific compute where you need to know that all elements are laid out decompressed and contiguous in memory.

Laziness

Canonical form is not recursive, so while a StructArray is the canonical format for any Struct type, individual column child arrays may still be compressed. This allows compute over Vortex arrays to push decoding as late as possible, and ideally many child arrays never need to be decoded into canonical form at all depending on the compute.

Arrow interoperability

All of the Vortex canonical encodings have an equivalent Arrow encoding that can be built zero-copy, and the corresponding Arrow array types can also be built directly.

The full list of canonical types and their equivalent Arrow array types are:

• NullArray: arrow_array::NullArray
• BoolArray: arrow_array::BooleanArray
• PrimitiveArray: arrow_array::PrimitiveArray
• DecimalArray: arrow_array::Decimal128Array and arrow_array::Decimal256Array
• StructArray: arrow_array::StructArray
• ListArray: arrow_array::ListArray
• VarBinViewArray: arrow_array::GenericByteViewArray

Vortex uses a logical type system, unlike Arrow which uses physical encodings for its types. As an example, there are at least six valid physical encodings for a Utf8 array. This can create ambiguity. Thus, if you receive an Arrow array, compress it using Vortex, and then decompress it later to pass to a compute kernel, there are multiple suitable Arrow array variants to hold the data.

To disambiguate, we choose a canonical physical encoding for every Vortex DType, which will correspond to an arrow-rs arrow_schema::DataType.

Views support

Binary and String views, also known as “German strings” are a better encoding format for nearly all use-cases. Variable-length binary views are part of the Apache Arrow spec, and are fully supported by the Datafusion query engine. We use them as our canonical string encoding for all Utf8 and Binary typed arrays in Vortex. They provide considerably faster filter execution than the core StringArray and BinaryArray types, at the expense of potentially needing garbage collection to clear unreferenced items from memory.

Variants

Null(NullArray)

Bool(BoolArray)

Primitive(PrimitiveArray)

Decimal(DecimalArray)

Struct(StructArray)

List(ListArray)

VarBinView(VarBinViewArray)

Extension(ExtensionArray)

Implementations

impl Canonical

pub fn empty(dtype: &DType) -> Canonical

Create an empty canonical array of the given dtype.

impl Canonical

pub fn into_null(self) -> VortexResult<NullArray>

pub fn into_bool(self) -> VortexResult<BoolArray>

pub fn into_primitive(self) -> VortexResult<PrimitiveArray>

pub fn into_decimal(self) -> VortexResult<DecimalArray>

pub fn into_struct(self) -> VortexResult<StructArray>

pub fn into_list(self) -> VortexResult<ListArray>

pub fn into_varbinview(self) -> VortexResult<VarBinViewArray>

pub fn into_extension(self) -> VortexResult<ExtensionArray>

Trait Implementations

impl AsRef<dyn Array> for Canonical

fn as_ref(&self) -> &(dyn Array + 'static)

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for Canonical

fn clone(&self) -> Canonical

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Canonical

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Canonical> for ArrayRef

fn from(value: Canonical) -> Self

Converts to this type from the input type.

impl IntoArray for Canonical

fn into_array(self) -> ArrayRef