Struct VarBinViewData 

pub struct VarBinViewData { /* private fields */ }

A variable-length binary view array that stores strings and binary data efficiently.

This mirrors the Apache Arrow StringView/BinaryView array encoding and provides an optimized representation for variable-length data with excellent performance characteristics for both short and long strings.

Data Layout

The array uses a hybrid storage approach with two main components:

• Views buffer: Array of 16-byte BinaryView entries (one per logical element)
• Data buffers: Shared backing storage for strings longer than 12 bytes

View Structure

Commonly referred to as “German Strings”, each 16-byte view entry contains either:

• Inlined data: For strings ≤ 12 bytes, the entire string is stored directly in the view
• Reference data: For strings > 12 bytes, contains:
  • String length (4 bytes)
  • First 4 bytes of string as prefix (4 bytes)
  • Buffer index and offset (8 bytes total)

The following ASCII graphic is reproduced verbatim from the Arrow documentation:

                        ┌──────┬────────────────────────┐
                        │length│      string value      │
   Strings (len <= 12)  │      │    (padded with 0)     │
                        └──────┴────────────────────────┘
                         0    31                      127

                        ┌───────┬───────┬───────┬───────┐
                        │length │prefix │  buf  │offset │
   Strings (len > 12)   │       │       │ index │       │
                        └───────┴───────┴───────┴───────┘
                         0    31       63      95    127

Examples

use vortex_array::arrays::VarBinViewArray;
use vortex_array::dtype::{DType, Nullability};
use vortex_array::IntoArray;

// Create from an Iterator<Item = &str>
let array = VarBinViewArray::from_iter_str([
        "inlined",
        "this string is outlined"
]);

assert_eq!(array.len(), 2);

// Access individual strings
let first = array.bytes_at(0);
assert_eq!(first.as_slice(), b"inlined"); // "short"

let second = array.bytes_at(1);
assert_eq!(second.as_slice(), b"this string is outlined"); // Long string

Implementations

impl VarBinViewData

pub fn new( views: Buffer<BinaryView>, buffers: Arc<[ByteBuffer]>, dtype: DType, validity: Validity, ) -> Self

Creates a new VarBinViewArray.

Panics

Panics if the provided components do not satisfy the invariants documented in VarBinViewArray::new_unchecked.

pub fn new_handle( views: BufferHandle, buffers: Arc<[BufferHandle]>, dtype: DType, validity: Validity, ) -> Self

Creates a new VarBinViewArray with device or host memory.

Panics

Panics if the provided components do not satisfy the invariants documented in VarBinViewArray::new_unchecked.

pub fn try_new( views: Buffer<BinaryView>, buffers: Arc<[ByteBuffer]>, dtype: DType, validity: Validity, ) -> VortexResult<Self>

Constructs a new VarBinViewArray.

See VarBinViewArray::new_unchecked for more information.

Errors

Returns an error if the provided components do not satisfy the invariants documented in VarBinViewArray::new_unchecked.

pub fn try_new_handle( views: BufferHandle, buffers: Arc<[BufferHandle]>, dtype: DType, validity: Validity, ) -> VortexResult<Self>

Constructs a new VarBinViewArray.

See VarBinViewArray::new_unchecked for more information.

Errors

Returns an error if the provided components do not satisfy the invariants documented in VarBinViewArray::new_unchecked.

pub unsafe fn new_unchecked( views: Buffer<BinaryView>, buffers: Arc<[ByteBuffer]>, dtype: DType, validity: Validity, ) -> Self

Creates a new VarBinViewArray without validation from these components:

• views is a buffer of 16-byte view entries (one per logical element).
• buffers contains the backing storage for strings longer than 12 bytes.
• dtype specifies whether this contains UTF-8 strings or binary data.
• validity holds the null values.

Safety

The caller must ensure all of the following invariants are satisfied:

View Requirements

• Views must be properly formatted 16-byte BinaryView entries.
• Inlined views (length ≤ 12) must have valid data in the first length bytes.
• Reference views (length > 12) must:
  • Have a valid buffer index < buffers.len().
  • Have valid offsets that don’t exceed the referenced buffer’s bounds.
  • Have a 4-byte prefix that matches the actual data at the referenced location.

Type Requirements

• dtype must be either DType::Utf8 or DType::Binary.
• For DType::Utf8, all string data (both inlined and referenced) must be valid UTF-8.

Validity Requirements

• The validity must have the same nullability as the dtype.
• If validity is an array, its length must match views.len().

pub unsafe fn new_handle_unchecked( views: BufferHandle, buffers: Arc<[BufferHandle]>, dtype: DType, _validity: Validity, ) -> Self

Construct a new array from BufferHandles without validation.

Safety

See documentation in new_unchecked.

pub fn validate( views: &Buffer<BinaryView>, buffers: &Arc<[ByteBuffer]>, dtype: &DType, validity: &Validity, ) -> VortexResult<()>

Validates the components that would be used to create a VarBinViewArray.

This function checks all the invariants required by VarBinViewArray::new_unchecked.

pub fn len(&self) -> usize

Returns the length of this array.

pub fn is_empty(&self) -> bool

Returns true if this array is empty.

pub fn views(&self) -> &[BinaryView]

Access to the primitive views buffer.

Variable-sized binary view buffer contain a “view” child array, with 16-byte entries that contain either a pointer into one of the array’s owned buffers OR an inlined copy of the string (if the string has 12 bytes or fewer).

pub fn views_handle(&self) -> &BufferHandle

Return the buffer handle backing the views.

pub fn bytes_at(&self, index: usize) -> ByteBuffer

Access value bytes at a given index

Will return a ByteBuffer containing the data without performing a copy.

pub fn buffer(&self, idx: usize) -> &ByteBuffer

Access one of the backing data buffers.

Panics

This method panics if the provided index is out of bounds for the set of buffers provided at construction time.

pub fn data_buffers(&self) -> &Arc<[BufferHandle]>

The underlying raw data buffers, not including the views buffer.

pub fn from_iter<T: AsRef<[u8]>, I: IntoIterator<Item = Option<T>>>( iter: I, dtype: DType, ) -> Self

Accumulate an iterable set of values into our type here.

pub fn from_iter_str<T: AsRef<str>, I: IntoIterator<Item = T>>(iter: I) -> Self

pub fn from_iter_nullable_str<T: AsRef<str>, I: IntoIterator<Item = Option<T>>>( iter: I, ) -> Self

pub fn from_iter_bin<T: AsRef<[u8]>, I: IntoIterator<Item = T>>(iter: I) -> Self

pub fn from_iter_nullable_bin<T: AsRef<[u8]>, I: IntoIterator<Item = Option<T>>>( iter: I, ) -> Self

Trait Implementations

impl ArrayEq for VarBinViewData

fn array_eq(&self, other: &Self, precision: Precision) -> bool

impl ArrayHash for VarBinViewData

fn array_hash<H: Hasher>(&self, state: &mut H, precision: Precision)

impl Clone for VarBinViewData

fn clone(&self) -> VarBinViewData

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for VarBinViewData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for VarBinViewData

fn fmt(&self, _f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'a> FromIterator<Option<&'a [u8]>> for VarBinViewData

fn from_iter<T: IntoIterator<Item = Option<&'a [u8]>>>(iter: T) -> Self

Creates a value from an iterator.

impl<'a> FromIterator<Option<&'a str>> for VarBinViewData

fn from_iter<T: IntoIterator<Item = Option<&'a str>>>(iter: T) -> Self

Creates a value from an iterator.

impl FromIterator<Option<String>> for VarBinViewData

fn from_iter<T: IntoIterator<Item = Option<String>>>(iter: T) -> Self

Creates a value from an iterator.

impl FromIterator<Option<Vec<u8>>> for VarBinViewData

fn from_iter<T: IntoIterator<Item = Option<Vec<u8>>>>(iter: T) -> Self

Creates a value from an iterator.