Struct VarBinViewArray

pub struct VarBinViewArray { /* private fields */ }

A variable-length binary view array that stores strings and binary data efficiently.

This mirrors the Apache Arrow StringView/BinaryView array encoding and provides an optimized representation for variable-length data with excellent performance characteristics for both short and long strings.

Data Layout

The array uses a hybrid storage approach with two main components:

• Views buffer: Array of 16-byte BinaryView entries (one per logical element)
• Data buffers: Shared backing storage for strings longer than 12 bytes

View Structure

Commonly referred to as “German Strings”, each 16-byte view entry contains either:

• Inlined data: For strings ≤ 12 bytes, the entire string is stored directly in the view
• Reference data: For strings > 12 bytes, contains:
  • String length (4 bytes)
  • First 4 bytes of string as prefix (4 bytes)
  • Buffer index and offset (8 bytes total)

The following ASCII graphic is reproduced verbatim from the Arrow documentation:

                        ┌──────┬────────────────────────┐
                        │length│      string value      │
   Strings (len <= 12)  │      │    (padded with 0)     │
                        └──────┴────────────────────────┘
                         0    31                      127

                        ┌───────┬───────┬───────┬───────┐
                        │length │prefix │  buf  │offset │
   Strings (len > 12)   │       │       │ index │       │
                        └───────┴───────┴───────┴───────┘
                         0    31       63      95    127

Examples

use vortex_array::arrays::VarBinViewArray;
use vortex_dtype::{DType, Nullability};
use vortex_array::IntoArray;

// Create from an Iterator<Item = &str>
let array = VarBinViewArray::from_iter_str([
        "inlined",
        "this string is outlined"
]);

assert_eq!(array.len(), 2);

// Access individual strings
let first = array.bytes_at(0);
assert_eq!(first.as_slice(), b"inlined"); // "short"

let second = array.bytes_at(1);
assert_eq!(second.as_slice(), b"this string is outlined"); // Long string

Implementations

impl VarBinViewArray

pub fn compact_buffers(&self) -> VortexResult<VarBinViewArray>

Returns a compacted copy of the input array, where all wasted space has been cleaned up. This operation can be very expensive, in the worst cast copying all existing string data into a new allocation.

After slicing/taking operations VarBinViewArrays can continue to hold references to buffers that are no longer visible. We detect when there is wasted space in any of the buffers, and if so, will aggressively compact all visile outlined string data into a single new buffer.

impl VarBinViewArray

pub unsafe fn new_unchecked( views: Buffer<BinaryView>, buffers: Arc<[ByteBuffer]>, dtype: DType, validity: Validity, ) -> Self

Build a new VarBinViewArray from components with validation.

Safety

This should only be used when you know for certain that all components are already validated, for example during array operations that preserve the invariants of the encoding.

See VarBinViewArray::try_new for a safe constructor that does validation.

pub fn new( views: Buffer<BinaryView>, buffers: Arc<[ByteBuffer]>, dtype: DType, validity: Validity, ) -> Self

pub fn try_new( views: Buffer<BinaryView>, buffers: Arc<[ByteBuffer]>, dtype: DType, validity: Validity, ) -> VortexResult<Self>

pub fn nbuffers(&self) -> usize

Number of raw string data buffers held by this array.

pub fn views(&self) -> &Buffer<BinaryView>

Access to the primitive views buffer.

Variable-sized binary view buffer contain a “view” child array, with 16-byte entries that contain either a pointer into one of the array’s owned buffers OR an inlined copy of the string (if the string has 12 bytes or fewer).

pub fn bytes_at(&self, index: usize) -> ByteBuffer

Access value bytes at a given index

Will return a ByteBuffer containing the data without performing a copy.

pub fn buffer(&self, idx: usize) -> &ByteBuffer

Access one of the backing data buffers.

Panics

This method panics if the provided index is out of bounds for the set of buffers provided at construction time.

pub fn buffers(&self) -> &Arc<[ByteBuffer]>

Iterate over the underlying raw data buffers, not including the views buffer.

pub fn from_iter<T: AsRef<[u8]>, I: IntoIterator<Item = Option<T>>>( iter: I, dtype: DType, ) -> Self

Accumulate an iterable set of values into our type here.

pub fn from_iter_str<T: AsRef<str>, I: IntoIterator<Item = T>>(iter: I) -> Self

pub fn from_iter_nullable_str<T: AsRef<str>, I: IntoIterator<Item = Option<T>>>( iter: I, ) -> Self

pub fn from_iter_bin<T: AsRef<[u8]>, I: IntoIterator<Item = T>>(iter: I) -> Self

pub fn from_iter_nullable_bin<T: AsRef<[u8]>, I: IntoIterator<Item = Option<T>>>( iter: I, ) -> Self

Methods from Deref<Target = dyn Array>

pub fn display_values(&self) -> impl Display

Display logical values of the array

For example, an i16 typed array containing the first five non-negative integers is displayed as: [0i16, 1i16, 2i16, 3i16, 4i16].

Examples

let array = buffer![0_i16, 1, 2, 3, 4].into_array();
assert_eq!(
    format!("{}", array.display_values()),
    "[0i16, 1i16, 2i16, 3i16, 4i16]",
)

See also: Array::display_as, DisplayArrayAs, and DisplayOptions.

pub fn display_as(&self, options: DisplayOptions) -> impl Display

Display the array as specified by the options.

See DisplayOptions for examples.

pub fn display_tree(&self) -> impl Display

Display the tree of encodings of this array as an indented lists.

While some metadata (such as length, bytes and validity-rate) are included, the logical values of the array are not displayed. To view the logical values see Array::display_as and DisplayOptions.

Examples

let array = buffer![0_i16, 1, 2, 3, 4].into_array();
let expected = "root: vortex.primitive(i16, len=5) nbytes=10 B (100.00%)
  metadata: EmptyMetadata
  buffer (align=2): 10 B (100.00%)
";
assert_eq!(format!("{}", array.display_tree()), expected);

pub fn as_<V: VTable>(&self) -> &V::Array

Returns the array downcast to the given A.

pub fn as_opt<V: VTable>(&self) -> Option<&V::Array>

Returns the array downcast to the given A.

pub fn is<V: VTable>(&self) -> bool

Is self an array with encoding from vtable V.

pub fn is_constant(&self) -> bool

pub fn is_constant_opts(&self, cost: Cost) -> bool

pub fn as_constant(&self) -> Option<Scalar>

pub fn nbytes(&self) -> u64

Total size of the array in bytes, including all children and buffers.

pub fn to_array_iterator(&self) -> impl ArrayIterator + 'static

Create an ArrayIterator over the array.

pub fn serialize( &self, ctx: &ArrayContext, options: &SerializeOptions, ) -> VortexResult<Vec<ByteBuffer>>

Serialize the array into a sequence of byte buffers that should be written contiguously. This function returns a vec to avoid copying data buffers.

Optionally, padding can be included to guarantee buffer alignment and ensure zero-copy reads within the context of an external file or stream. In this case, the alignment of the first byte buffer should be respected when writing the buffers to the stream or file.

The format of this blob is a sequence of data buffers, possible with prefixed padding, followed by a flatbuffer containing an fba::Array message, and ending with a little-endian u32 describing the length of the flatbuffer message.

pub fn to_array_stream(&self) -> impl ArrayStream + 'static

Create an ArrayStream over the array.

pub fn as_null_typed(&self) -> NullTyped<'_>

Downcasts the array for null-specific behavior.

pub fn as_bool_typed(&self) -> BoolTyped<'_>

Downcasts the array for bool-specific behavior.

pub fn as_primitive_typed(&self) -> PrimitiveTyped<'_>

Downcasts the array for primitive-specific behavior.

pub fn as_decimal_typed(&self) -> DecimalTyped<'_>

Downcasts the array for decimal-specific behavior.

pub fn as_utf8_typed(&self) -> Utf8Typed<'_>

Downcasts the array for utf8-specific behavior.

pub fn as_binary_typed(&self) -> BinaryTyped<'_>

Downcasts the array for binary-specific behavior.

pub fn as_struct_typed(&self) -> StructTyped<'_>

Downcasts the array for struct-specific behavior.

pub fn as_list_typed(&self) -> ListTyped<'_>

Downcasts the array for list-specific behavior.

pub fn as_extension_typed(&self) -> ExtensionTyped<'_>

Downcasts the array for extension-specific behavior.

Trait Implementations

impl ArrayAccessor<[u8]> for VarBinViewArray

fn with_iterator<F: for<'a> FnOnce(&mut dyn Iterator<Item = Option<&'a [u8]>>) -> R, R>( &self, f: F, ) -> VortexResult<R>

Iterate over each element of the array, in-order.

impl AsRef<dyn Array> for VarBinViewArray

fn as_ref(&self) -> &dyn Array

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for VarBinViewArray

fn clone(&self) -> VarBinViewArray

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for VarBinViewArray

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Deref for VarBinViewArray

type Target = dyn Array

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl From<VarBinViewArray> for ArrayRef

fn from(value: VarBinViewArray) -> ArrayRef

Converts to this type from the input type.

impl<'a> FromIterator<Option<&'a [u8]>> for VarBinViewArray

fn from_iter<T: IntoIterator<Item = Option<&'a [u8]>>>(iter: T) -> Self

Creates a value from an iterator.

impl<'a> FromIterator<Option<&'a str>> for VarBinViewArray

fn from_iter<T: IntoIterator<Item = Option<&'a str>>>(iter: T) -> Self

Creates a value from an iterator.

impl FromIterator<Option<String>> for VarBinViewArray

fn from_iter<T: IntoIterator<Item = Option<String>>>(iter: T) -> Self

Creates a value from an iterator.

impl FromIterator<Option<Vec<u8>>> for VarBinViewArray

fn from_iter<T: IntoIterator<Item = Option<Vec<u8>>>>(iter: T) -> Self

Creates a value from an iterator.

impl IntoArray for VarBinViewArray

fn into_array(self) -> ArrayRef

impl ValidityHelper for VarBinViewArray

fn validity(&self) -> &Validity