Struct GenericListArray 

pub struct GenericListArray<OffsetSize>
where
    OffsetSize: OffsetSizeTrait,
{ /* private fields */ }

An array of variable length lists, similar to JSON arrays (e.g. ["A", "B", "C"]). This struct specifically represents the list layout. Refer to GenericListViewArray for the list-view layout.

Lists are represented using offsets into a values child array. Offsets are stored in two adjacent entries of an OffsetBuffer.

Arrow defines ListArray with i32 offsets and LargeListArray with i64 offsets.

Use GenericListBuilder to construct a GenericListArray.

Representation

A ListArray can represent a list of values of any other supported Arrow type. Each element of the ListArray itself is a list which may be empty, may contain NULL and non-null values, or may itself be NULL.

For example, the ListArray shown in the following diagram stores lists of strings. Note that [] represents an empty (length 0), but non NULL list.

┌─────────────┐
│   [A,B,C]   │
├─────────────┤
│     []      │
├─────────────┤
│    NULL     │
├─────────────┤
│     [D]     │
├─────────────┤
│  [NULL, F]  │
└─────────────┘

The values are stored in a child StringArray and the offsets are stored in an OffsetBuffer as shown in the following diagram. The logical values and offsets are shown on the left, and the actual ListArray encoding on the right.

                                        ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                                                                ┌ ─ ─ ─ ─ ─ ─ ┐    │
 ┌─────────────┐  ┌───────┐             │     ┌───┐   ┌───┐       ┌───┐ ┌───┐
 │   [A,B,C]   │  │ (0,3) │                   │ 1 │   │ 0 │     │ │ 1 │ │ A │ │ 0  │
 ├─────────────┤  ├───────┤             │     ├───┤   ├───┤       ├───┤ ├───┤
 │      []     │  │ (3,3) │                   │ 1 │   │ 3 │     │ │ 1 │ │ B │ │ 1  │
 ├─────────────┤  ├───────┤             │     ├───┤   ├───┤       ├───┤ ├───┤
 │    NULL     │  │ (3,4) │                   │ 0 │   │ 3 │     │ │ 1 │ │ C │ │ 2  │
 ├─────────────┤  ├───────┤             │     ├───┤   ├───┤       ├───┤ ├───┤
 │     [D]     │  │ (4,5) │                   │ 1 │   │ 4 │     │ │ ? │ │ ? │ │ 3  │
 ├─────────────┤  ├───────┤             │     ├───┤   ├───┤       ├───┤ ├───┤
 │  [NULL, F]  │  │ (5,7) │                   │ 1 │   │ 5 │     │ │ 1 │ │ D │ │ 4  │
 └─────────────┘  └───────┘             │     └───┘   ├───┤       ├───┤ ├───┤
                                                      │ 7 │     │ │ 0 │ │ ? │ │ 5  │
                                        │  Validity   └───┘       ├───┤ ├───┤
    Logical       Logical                  (nulls)   Offsets    │ │ 1 │ │ F │ │ 6  │
     Values       Offsets               │                         └───┘ └───┘
                                                                │    Values   │    │
                (offsets[i],            │   ListArray               (Array)
               offsets[i+1])                                    └ ─ ─ ─ ─ ─ ─ ┘    │
                                        └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

Slicing

Slicing a ListArray creates a new ListArray without copying any data, but this means the Self::values and Self::offsets may have “unused” data

For example, calling slice(1, 3) on the ListArray in the above example would result in the following. Note

1. Values array is unchanged
2. Offsets do not start at 0, nor cover all values in the Values array.

                                ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
                                                        ┌ ─ ─ ─ ─ ─ ─ ┐    │  ╔═══╗
                                │                         ╔═══╗ ╔═══╗         ║   ║  Not used
                                                        │ ║ 1 ║ ║ A ║ │ 0  │  ╚═══╝
 ┌─────────────┐  ┌───────┐     │     ┌───┐   ┌───┐       ╠═══╣ ╠═══╣
 │ [] (empty)  │  │ (3,3) │           │ 1 │   │ 3 │     │ ║ 1 ║ ║ B ║ │ 1  │
 ├─────────────┤  ├───────┤     │     ├───┤   ├───┤       ╠═══╣ ╠═══╣
 │    NULL     │  │ (3,4) │           │ 0 │   │ 3 │     │ ║ 1 ║ ║ C ║ │ 2  │
 ├─────────────┤  ├───────┤     │     ├───┤   ├───┤       ╠───╣ ╠───╣
 │     [D]     │  │ (4,5) │           │ 1 │   │ 4 │     │ │ 0 │ │ ? │ │ 3  │
 └─────────────┘  └───────┘     │     └───┘   ├───┤       ├───┤ ├───┤
                                              │ 5 │     │ │ 1 │ │ D │ │ 4  │
                                │             └───┘       ├───┤ ├───┤
                                                        │ │ 0 │ │ ? │ │ 5  │
                                │  Validity               ╠═══╣ ╠═══╣
    Logical       Logical          (nulls)   Offsets    │ ║ 1 ║ ║ F ║ │ 6  │
     Values       Offsets       │                         ╚═══╝ ╚═══╝
                                                        │    Values   │    │
                (offsets[i],    │   ListArray               (Array)
               offsets[i+1])                            └ ─ ─ ─ ─ ─ ─ ┘    │
                                └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─

Implementations

impl<OffsetSize> GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

pub const DATA_TYPE_CONSTRUCTOR: fn(Arc<Field>) -> DataType

The data type constructor of list array. The input is the schema of the child array and the output is the DataType, List or LargeList.

pub fn try_new( field: Arc<Field>, offsets: OffsetBuffer<OffsetSize>, values: Arc<dyn Array>, nulls: Option<NullBuffer>, ) -> Result<GenericListArray<OffsetSize>, ArrowError>

Create a new GenericListArray from the provided parts

Errors

Errors if

• offsets.len() - 1 != nulls.len()
• offsets.last() > values.len()
• !field.is_nullable() && values.is_nullable()
• field.data_type() != values.data_type()

pub fn new( field: Arc<Field>, offsets: OffsetBuffer<OffsetSize>, values: Arc<dyn Array>, nulls: Option<NullBuffer>, ) -> GenericListArray<OffsetSize>

Create a new GenericListArray from the provided parts

Panics

Panics if Self::try_new returns an error

pub fn new_null(field: Arc<Field>, len: usize) -> GenericListArray<OffsetSize>

Create a new GenericListArray of length len where all values are null

pub fn into_parts( self, ) -> (Arc<Field>, OffsetBuffer<OffsetSize>, Arc<dyn Array>, Option<NullBuffer>)

Deconstruct this array into its constituent parts

pub fn offsets(&self) -> &OffsetBuffer<OffsetSize>

Returns a reference to the offsets of this list

Unlike Self::value_offsets this returns the OffsetBuffer allowing for zero-copy cloning.

Notes: The offsets may not start at 0 and may not cover all values in Self::values. This can happen when the list array was sliced via Self::slice. See documentation for Self for more details.

pub fn values(&self) -> &Arc<dyn Array>

Returns a reference to the values of this list

Note: The list array may not refer to all values in the values array. For example if the list array was sliced via Self::slice values will still contain values both before and after the slice. See documentation for Self for more details.

pub fn value_type(&self) -> DataType

Returns a clone of the value type of this list.

pub unsafe fn value_unchecked(&self, i: usize) -> Arc<dyn Array>

Returns ith value of this list array.

Note: This method does not check for nulls and the value is arbitrary if is_null returns true for the index.

Safety

Caller must ensure that the index is within the array bounds

pub fn value(&self, i: usize) -> Arc<dyn Array>

Returns ith value of this list array.

Note: This method does not check for nulls and the value is arbitrary (but still well-defined) if is_null returns true for the index.

Panics

Panics if index i is out of bounds

pub fn value_offsets(&self) -> &[OffsetSize]

Returns the offset values in the offsets buffer.

See Self::offsets for more details.

pub fn value_length(&self, i: usize) -> OffsetSize

Returns the length for value at index i.

pub fn iter<'a>(&'a self) -> ArrayIter<&'a GenericListArray<OffsetSize>>

constructs a new iterator

pub fn slice( &self, offset: usize, length: usize, ) -> GenericListArray<OffsetSize>

Returns a zero-copy slice of this array with the indicated offset and length.

Notes: this method does NOT slice the underlying values array or modify the values in the offsets buffer. See Self::values and Self::offsets for more information.

pub fn from_iter_primitive<T, P, I>(iter: I) -> GenericListArray<OffsetSize>

where T: ArrowPrimitiveType, P: IntoIterator<Item = Option<<T as ArrowPrimitiveType>::Native>>, I: IntoIterator<Item = Option<P>>,

Creates a GenericListArray from an iterator of primitive values

Example

let data = vec![
   Some(vec![Some(0), Some(1), Some(2)]),
   None,
   Some(vec![Some(3), None, Some(5)]),
   Some(vec![Some(6), Some(7)]),
];
let list_array = ListArray::from_iter_primitive::<Int32Type, _, _>(data);
println!("{:?}", list_array);

Trait Implementations

impl<OffsetSize> ArrayAccessor for &GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

type Item = Arc<dyn Array>

The Arrow type of the element being accessed.

fn value( &self, index: usize, ) -> <&GenericListArray<OffsetSize> as ArrayAccessor>::Item

Returns the element at index i

unsafe fn value_unchecked( &self, index: usize, ) -> <&GenericListArray<OffsetSize> as ArrayAccessor>::Item

Returns the element at index i

impl<OffsetSize> Array for GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

fn as_any(&self) -> &(dyn Any + 'static)

Returns the array as Any so that it can be downcasted to a specific implementation.

fn to_data(&self) -> ArrayData

Returns the underlying data of this array

fn into_data(self) -> ArrayData

Returns the underlying data of this array

fn data_type(&self) -> &DataType

Returns a reference to the DataType of this array.

fn slice(&self, offset: usize, length: usize) -> Arc<dyn Array>

Returns a zero-copy slice of this array with the indicated offset and length.

fn len(&self) -> usize

Returns the length (i.e., number of elements) of this array.

fn is_empty(&self) -> bool

Returns whether this array is empty.

fn shrink_to_fit(&mut self)

Shrinks the capacity of any exclusively owned buffer as much as possible

fn offset(&self) -> usize

Returns the offset into the underlying data used by this array(-slice). Note that the underlying data can be shared by many arrays. This defaults to 0.

fn nulls(&self) -> Option<&NullBuffer>

Returns the null buffer of this array if any.

fn logical_null_count(&self) -> usize

Returns the total number of logical null values in this array.

fn get_buffer_memory_size(&self) -> usize

Returns the total number of bytes of memory pointed to by this array. The buffers store bytes in the Arrow memory format, and include the data as well as the validity map. Note that this does not always correspond to the exact memory usage of an array, since multiple arrays can share the same buffers or slices thereof.

fn get_array_memory_size(&self) -> usize

Returns the total number of bytes of memory occupied physically by this array. This value will always be greater than returned by get_buffer_memory_size() and includes the overhead of the data structures that contain the pointers to the various buffers.

fn logical_nulls(&self) -> Option<NullBuffer>

Returns a potentially computed NullBuffer that represents the logical null values of this array, if any.

fn is_null(&self, index: usize) -> bool

Returns whether the element at index is null according to Array::nulls

fn is_valid(&self, index: usize) -> bool

Returns whether the element at index is not null, the opposite of Self::is_null.

fn null_count(&self) -> usize

Returns the total number of physical null values in this array.

fn is_nullable(&self) -> bool

Returns false if the array is guaranteed to not contain any logical nulls

impl<OffsetSize> Clone for GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

fn clone(&self) -> GenericListArray<OffsetSize>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<OffsetSize> Debug for GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<OffsetSize> From<ArrayData> for GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

fn from(data: ArrayData) -> GenericListArray<OffsetSize>

Converts to this type from the input type.

impl<OffsetSize> From<FixedSizeListArray> for GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

fn from(value: FixedSizeListArray) -> GenericListArray<OffsetSize>

Converts to this type from the input type.

impl<OffsetSize> From<GenericListArray<OffsetSize>> for ArrayData

where OffsetSize: OffsetSizeTrait,

fn from(array: GenericListArray<OffsetSize>) -> ArrayData

Converts to this type from the input type.

impl<OffsetSize> From<GenericListArray<OffsetSize>> for GenericByteArray<GenericStringType<OffsetSize>>

where OffsetSize: OffsetSizeTrait,

fn from( v: GenericListArray<OffsetSize>, ) -> GenericByteArray<GenericStringType<OffsetSize>>

Converts to this type from the input type.

impl<T> From<GenericListArray<T>> for GenericByteArray<GenericBinaryType<T>>

where T: OffsetSizeTrait,

fn from(v: GenericListArray<T>) -> GenericByteArray<GenericBinaryType<T>>

Converts to this type from the input type.

impl From<MapArray> for GenericListArray<i32>

fn from(value: MapArray) -> GenericListArray<i32>

Converts to this type from the input type.

impl<OffsetSize> PartialEq for GenericListArray<OffsetSize>

where OffsetSize: OffsetSizeTrait,

fn eq(&self, other: &GenericListArray<OffsetSize>) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl SizeBytes for GenericListArray<i32>

fn heap_size_bytes(&self) -> u64

Returns how many bytes self uses on the heap.

fn total_size_bytes(&self) -> u64

Returns the total size of self in bytes, accounting for both stack and heap space.

fn stack_size_bytes(&self) -> u64

Returns the total size of self on the stack, in bytes.

fn is_pod() -> bool

Is Self just plain old data?