Struct ListViewData 

pub struct ListViewData { /* private fields */ }

The canonical encoding for variable-length list arrays.

The ListViewArray encoding differs from ListArray in that it stores a child sizes array in addition to a child offsets array (which is the only child in ListArray).

In the past, we used ListArray as the canonical encoding for DType::List, but we have since migrated to ListViewArray for a few reasons:

• Enables better SIMD vectorization (no sequential dependency when reading offsets)
• Allows out-of-order offsets for better compression (we can shuffle the buffers)
• Supports different integer types for offsets vs sizes

It is worth mentioning that this encoding mirrors Apache Arrow’s ListView array type, but does not exactly mirror the similar type found in DuckDB and Velox, which stores the pair of offset and size in a row-major fashion rather than column-major. More specifically, the row-major layout has a single child array with alternating offset and size next to each other.

We choose the column-major layout as it allows better compressability, as well as using different (logical) integer widths for our offsets and sizes buffers (note that the compressor will likely compress to a different bit-packed width, but this is speaking strictly about flexibility in the logcial type).

Examples

// Create a list view array representing [[3, 4], [1], [2, 3]].
// Note: Unlike `ListArray`, offsets don't need to be monotonic.

let elements = buffer![1i32, 2, 3, 4, 5].into_array();
let offsets = buffer![2u32, 0, 1].into_array();  // Out-of-order offsets
let sizes = buffer![2u32, 1, 2].into_array();  // The sizes cause overlaps

let list_view = ListViewArray::new(
    elements.into_array(),
    offsets.into_array(),
    sizes.into_array(),
    Validity::NonNullable,
);

assert_eq!(list_view.len(), 3);

// Access individual lists
let first_list = list_view.list_elements_at(0)?;
assert_eq!(first_list.len(), 2);
// First list contains elements[2..4] = [3, 4]

let first_offset = list_view.offset_at(0);
let first_size = list_view.size_at(0);
assert_eq!(first_offset, 2);
assert_eq!(first_size, 2);

Implementations

impl ListViewData

pub fn new() -> Self

Creates a new ListViewArray.

Panics

Panics if the provided components do not satisfy the invariants documented in ListViewArray::new_unchecked.

pub fn try_new() -> VortexResult<Self>

Constructs a new ListViewArray.

Errors

Returns an error if the provided components do not satisfy the invariants documented in ListViewArray::new_unchecked.

pub unsafe fn new_unchecked() -> Self

Creates a new ListViewArray without validation.

This unsafe function does not check the validity of the data. Prefer calling new() or try_new() over this function, as they will check the validity of the data.

Safety

The caller must ensure all of the following invariants are satisfied:

• offsets and sizes must be non-nullable integer arrays.
• offsets and sizes must have the same length.
• Size integer width must be smaller than or equal to offset type (to prevent overflow).
• For each i, offsets[i] + sizes[i] must not overflow and must be <= elements.len() (even if the corresponding view is defined as null by the validity array).
• If validity is an array, its length must equal offsets.len().

pub fn validate( elements: &ArrayRef, offsets: &ArrayRef, sizes: &ArrayRef, validity: &Validity, ) -> VortexResult<()>

Validates the components that would be used to create a ListViewArray.

pub unsafe fn with_zero_copy_to_list(self, is_zctl: bool) -> Self

Sets whether this ListViewArray is zero-copyable to a ListArray.

This is an optimization flag that enables more efficient conversion to ListArray without needing to copy or reorganize the data.

Safety

When setting is_zctl to true, the caller must ensure that the ListViewArray is actually zero-copyable to a ListArray. This means:

• Offsets must be sorted (but not strictly sorted, zero-length lists are allowed).
• offsets[i] + sizes[i] == offsets[i + 1] for all i.
• No gaps in elements between first and last referenced elements.
• No overlapping list views (each element referenced at most once).

Note that leading and trailing unreferenced elements ARE allowed.

pub fn is_zero_copy_to_list(&self) -> bool

Returns true if the ListViewArray is zero-copyable to a ListArray.

Trait Implementations

impl ArrayEq for ListViewData

fn array_eq(&self, other: &Self, _precision: Precision) -> bool

impl ArrayHash for ListViewData

fn array_hash<H: Hasher>(&self, state: &mut H, _precision: Precision)

impl Clone for ListViewData

fn clone(&self) -> ListViewData

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ListViewData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ListViewData

fn default() -> Self

Returns the “default value” for a type.

impl Display for ListViewData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.