Enum Canonical

pub enum Canonical {
    Null(NullArray),
    Bool(BoolArray),
    Primitive(PrimitiveArray),
    Struct(StructArray),
    List(ListArray),
    VarBinView(VarBinViewArray),
    Extension(ExtensionArray),
}

The set of canonical array encodings, also the set of encodings that can be transferred to Arrow with zero-copy.

Note that a canonical form is not recursive, i.e. a StructArray may contain non-canonical child arrays, which may themselves need to be canonicalized.

Logical vs. Physical encodings

Vortex separates logical and physical types, however this creates ambiguity with Arrow, there is no separation. Thus, if you receive an Arrow array, compress it using Vortex, and then decompress it later to pass to a compute kernel, there are multiple suitable Arrow array variants to hold the data.

To disambiguate, we choose a canonical physical encoding for every Vortex DType, which will correspond to an arrow-rs arrow_schema::DataType.

Views support

Binary and String views, also known as “German strings” are a better encoding format for nearly all use-cases. Variable-length binary views are part of the Apache Arrow spec, and are fully supported by the Datafusion query engine. We use them as our canonical string encoding for all Utf8 and Binary typed arrays in Vortex.

Variants

Null(NullArray)

Bool(BoolArray)

Primitive(PrimitiveArray)

Struct(StructArray)

List(ListArray)

VarBinView(VarBinViewArray)

Extension(ExtensionArray)

Implementations

impl Canonical

pub fn empty(dtype: &DType) -> Canonical

pub fn try_empty(dtype: &DType) -> VortexResult<Canonical>

impl Canonical

pub fn into_null(self) -> VortexResult<NullArray>

pub fn into_bool(self) -> VortexResult<BoolArray>

pub fn into_primitive(self) -> VortexResult<PrimitiveArray>

pub fn into_struct(self) -> VortexResult<StructArray>

pub fn into_list(self) -> VortexResult<ListArray>

pub fn into_varbinview(self) -> VortexResult<VarBinViewArray>

pub fn into_extension(self) -> VortexResult<ExtensionArray>

Methods from Deref<Target = Array>

pub fn vtable(&self) -> &VTableRef

Return the array’s encoding VTable.

pub fn encoding(&self) -> EncodingId

Return the array’s encoding ID.

pub fn len(&self) -> usize

Returns the number of logical elements in the array.

pub fn is_empty(&self) -> bool

Check whether the array has any data

pub fn dtype(&self) -> &DType

Return the array’s dtype

pub fn is_canonical(&self) -> bool

Whether the array is of a canonical encoding.

pub fn is_arrow(&self) -> bool

Whether the array is fully zero-copy to Arrow (including children). This means any nested types, like Structs, Lists, and Extensions are not present.

pub fn is_constant(&self) -> bool

Return whether the array is constant.

pub fn as_constant(&self) -> Option<Scalar>

Return scalar value of this array if the array is constant

pub fn child<'a>( &'a self, idx: usize, dtype: &'a DType, len: usize, ) -> VortexResult<Self>

pub fn children(&self) -> Vec<Array>

Returns a Vec of Arrays with all the array’s child arrays.

pub fn named_children(&self) -> Vec<(String, Array)>

Returns a Vec of Arrays with all the array’s child arrays.

pub fn nchildren(&self) -> usize

Returns the number of child arrays

pub fn depth_first_traversal(&self) -> ArrayChildrenIterator

pub fn cumulative_nbuffers(&self) -> usize

Count the number of cumulative buffers encoded by self.

pub fn all_buffer_offsets(&self, alignment: usize) -> Vec<u64>

Return the buffer offsets and the total length of all buffers, assuming the given alignment. This includes all child buffers.

pub fn metadata_bytes(&self) -> Option<&[u8]>

pub fn nbuffers(&self) -> usize

pub fn byte_buffer(&self, index: usize) -> Option<&ByteBuffer>

pub fn byte_buffers(&self) -> impl Iterator<Item = ByteBuffer> + '_

pub fn is_encoding(&self, id: EncodingId) -> bool

Checks whether array is of a given encoding.

pub fn try_downcast_ref<E: Encoding>(&self) -> VortexResult<(&E::Array, &E)>

where for<'a> &'a E::Array: TryFrom<&'a Array, Error = VortexError>,

pub fn nbytes(&self) -> usize

Total size of the array in bytes, including all children and buffers.

pub fn statistics(&self) -> &(dyn Statistics + '_)

pub fn inherit_statistics(&self, parent: &dyn Statistics)

pub fn tree_display(&self) -> TreeDisplayWrapper<'_>

pub fn is_valid(&self, index: usize) -> VortexResult<bool>

Return whether the element at the given index is valid (true) or null (false).

pub fn null_count(&self) -> VortexResult<usize>

Return the number of null elements in the array.

pub fn logical_validity(&self) -> VortexResult<Mask>

Return the logical validity of the array if nullable, and None if non-nullable.

pub fn as_null_array(&self) -> Option<&dyn NullArrayTrait>

pub fn as_bool_array(&self) -> Option<&dyn BoolArrayTrait>

pub fn as_primitive_array(&self) -> Option<&dyn PrimitiveArrayTrait>

pub fn as_utf8_array(&self) -> Option<&dyn Utf8ArrayTrait>

pub fn as_binary_array(&self) -> Option<&dyn BinaryArrayTrait>

pub fn as_struct_array(&self) -> Option<&dyn StructArrayTrait>

pub fn as_list_array(&self) -> Option<&dyn ListArrayTrait>

pub fn as_extension_array(&self) -> Option<&dyn ExtensionArrayTrait>

Trait Implementations

impl AsRef<Array> for Canonical

fn as_ref(&self) -> &Array

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for Canonical

fn clone(&self) -> Canonical

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Canonical

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Deref for Canonical

type Target = Array

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl From<Canonical> for Array

This conversion is always “free” and should not touch underlying data. All it does is create an owned pointer to the underlying concrete array type.

This combined with the above IntoCanonical impl for Array allows simple two-way conversions between arbitrary Vortex encodings and canonical Arrow-compatible encodings.

fn from(value: Canonical) -> Self

Converts to this type from the input type.

impl IntoArray for Canonical

fn into_array(self) -> Array