Struct ExtensionArray

pub struct ExtensionArray { /* private fields */ }

An extension array that wraps another array with additional type information.

⚠️ Unstable API: This is an experimental feature that may change significantly in future versions. The extension type system is still evolving.

Unlike Apache Arrow’s extension arrays, Vortex extension arrays provide a more flexible mechanism for adding semantic meaning to existing array types without requiring changes to the core type system.

Design Philosophy

Extension arrays serve as a type-safe wrapper that:

• Preserves the underlying storage format and operations
• Adds semantic type information via ExtDType
• Enables custom serialization and deserialization logic
• Allows domain-specific interpretations of generic data

Storage and Type Relationship

The extension array maintains a strict contract:

• Storage array: Contains the actual data in a standard Vortex encoding
• Extension type: Defines how to interpret the storage data semantically
• Type safety: The storage array’s dtype must match the extension type’s storage dtype

Use Cases

Extension arrays are ideal for:

• Custom numeric types: Units of measurement, currencies
• Temporal types: Custom date/time formats, time zones, calendars
• Domain-specific types: UUIDs, IP addresses, geographic coordinates
• Encoded types: Base64 strings, compressed data, encrypted values

Validity and Operations

Extension arrays delegate validity and most operations to their storage array:

• Validity is inherited from the underlying storage
• Slicing preserves the extension type
• Scalar access wraps storage scalars with extension metadata

Examples

use std::sync::Arc;
use vortex_array::arrays::{ExtensionArray, PrimitiveArray};
use vortex_dtype::{ExtDType, ExtID, DType, Nullability, PType};
use vortex_array::validity::Validity;
use vortex_array::IntoArray;
use vortex_buffer::buffer;

// Define a custom extension type for representing currency values
let currency_id = ExtID::from("example.currency");
let currency_dtype = Arc::new(ExtDType::new(
    currency_id,
    Arc::new(DType::Primitive(PType::I64, Nullability::NonNullable)), // Storage as i64 cents
    None, // No additional metadata needed
));

// Create storage array with currency values in cents
let cents_storage = PrimitiveArray::new(
    buffer![12345i64, 67890, 99999], // $123.45, $678.90, $999.99
    Validity::NonNullable
);

// Wrap with extension type
let currency_array = ExtensionArray::new(
    currency_dtype.clone(),
    cents_storage.into_array()
);

assert_eq!(currency_array.len(), 3);
assert_eq!(currency_array.id().as_ref(), "example.currency");

// Access maintains extension type information
let first_value = currency_array.scalar_at(0);
assert!(first_value.as_extension_opt().is_some());

Implementations

impl ExtensionArray

pub fn new(ext_dtype: Arc<ExtDType>, storage: ArrayRef) -> Self

pub fn ext_dtype(&self) -> &Arc<ExtDType>

pub fn storage(&self) -> &ArrayRef

pub fn id(&self) -> &ExtID

Methods from Deref<Target = dyn Array>

pub fn display_values(&self) -> impl Display

Display logical values of the array

For example, an i16 typed array containing the first five non-negative integers is displayed as: [0i16, 1i16, 2i16, 3i16, 4i16].

Examples

let array = buffer![0_i16, 1, 2, 3, 4].into_array();
assert_eq!(
    format!("{}", array.display_values()),
    "[0i16, 1i16, 2i16, 3i16, 4i16]",
)

See also: Array::display_as, DisplayArrayAs, and DisplayOptions.

pub fn display_as(&self, options: DisplayOptions) -> impl Display

Display the array as specified by the options.

See DisplayOptions for examples.

pub fn display_tree(&self) -> impl Display

Display the tree of encodings of this array as an indented lists.

While some metadata (such as length, bytes and validity-rate) are included, the logical values of the array are not displayed. To view the logical values see Array::display_as and DisplayOptions.

Examples

let array = buffer![0_i16, 1, 2, 3, 4].into_array();
let expected = "root: vortex.primitive(i16, len=5) nbytes=10 B (100.00%)
  metadata: EmptyMetadata
  buffer (align=2): 10 B (100.00%)
";
assert_eq!(format!("{}", array.display_tree()), expected);

pub fn as_<V: VTable>(&self) -> &V::Array

Returns the array downcast to the given A.

pub fn as_opt<V: VTable>(&self) -> Option<&V::Array>

Returns the array downcast to the given A.

pub fn is<V: VTable>(&self) -> bool

Is self an array with encoding from vtable V.

pub fn is_constant(&self) -> bool

pub fn is_constant_opts(&self, cost: Cost) -> bool

pub fn as_constant(&self) -> Option<Scalar>

pub fn nbytes(&self) -> u64

Total size of the array in bytes, including all children and buffers.

pub fn to_array_iterator(&self) -> impl ArrayIterator + 'static

Create an ArrayIterator over the array.

pub fn serialize( &self, ctx: &ArrayContext, options: &SerializeOptions, ) -> VortexResult<Vec<ByteBuffer>>

Serialize the array into a sequence of byte buffers that should be written contiguously. This function returns a vec to avoid copying data buffers.

Optionally, padding can be included to guarantee buffer alignment and ensure zero-copy reads within the context of an external file or stream. In this case, the alignment of the first byte buffer should be respected when writing the buffers to the stream or file.

The format of this blob is a sequence of data buffers, possible with prefixed padding, followed by a flatbuffer containing an fba::Array message, and ending with a little-endian u32 describing the length of the flatbuffer message.

pub fn to_array_stream(&self) -> impl ArrayStream + 'static

Create an ArrayStream over the array.

pub fn as_null_typed(&self) -> NullTyped<'_>

Downcasts the array for null-specific behavior.

pub fn as_bool_typed(&self) -> BoolTyped<'_>

Downcasts the array for bool-specific behavior.

pub fn as_primitive_typed(&self) -> PrimitiveTyped<'_>

Downcasts the array for primitive-specific behavior.

pub fn as_decimal_typed(&self) -> DecimalTyped<'_>

Downcasts the array for decimal-specific behavior.

pub fn as_utf8_typed(&self) -> Utf8Typed<'_>

Downcasts the array for utf8-specific behavior.

pub fn as_binary_typed(&self) -> BinaryTyped<'_>

Downcasts the array for binary-specific behavior.

pub fn as_struct_typed(&self) -> StructTyped<'_>

Downcasts the array for struct-specific behavior.

pub fn as_list_typed(&self) -> ListTyped<'_>

Downcasts the array for list-specific behavior.

pub fn as_extension_typed(&self) -> ExtensionTyped<'_>

Downcasts the array for extension-specific behavior.

Trait Implementations

impl AsRef<dyn Array> for ExtensionArray

fn as_ref(&self) -> &dyn Array

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for ExtensionArray

fn clone(&self) -> ExtensionArray

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ExtensionArray

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Deref for ExtensionArray

type Target = dyn Array

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl From<&TemporalArray> for ExtensionArray

fn from(value: &TemporalArray) -> Self

Converts to this type from the input type.

impl From<ExtensionArray> for ArrayRef

fn from(value: ExtensionArray) -> ArrayRef

Converts to this type from the input type.

impl From<TemporalArray> for ExtensionArray

fn from(value: TemporalArray) -> Self

Converts to this type from the input type.

impl IntoArray for ExtensionArray

fn into_array(self) -> ArrayRef

impl TryFrom<ExtensionArray> for TemporalArray

type Error = VortexError

The type returned in the event of a conversion error.

fn try_from(ext: ExtensionArray) -> Result<Self, Self::Error>

Performs the conversion.