Enum DataBlock

pub enum DataBlock {
    AllNull(AllNullDataBlock),
    Nullable(NullableDataBlock),
    FixedWidth(FixedWidthDataBlock),
    FixedSizeList(FixedSizeListBlock),
    VariableWidth(VariableWidthBlock),
    Opaque(OpaqueBlock),
    Struct(StructDataBlock),
    Dictionary(DictionaryDataBlock),
}

A DataBlock is a collection of buffers that represents an “array” of data in very generic terms

The output of each decoder is a DataBlock. Decoders can be chained together to transform one DataBlock into a different kind of DataBlock.

The DataBlock is somewhere in between Arrow’s ArrayData and Array and represents a physical layout of the data.

A DataBlock can be converted into an Arrow ArrayData (and then Array) for a given array type. For example, a FixedWidthDataBlock can be converted into any primitive type or a fixed size list of a primitive type. This is a zero-copy operation.

In addition, a DataBlock can be created from an Arrow array or arrays. This is not a zero-copy operation as some normalization may be required.

Variants

AllNull(AllNullDataBlock)

Nullable(NullableDataBlock)

FixedWidth(FixedWidthDataBlock)

FixedSizeList(FixedSizeListBlock)

VariableWidth(VariableWidthBlock)

Opaque(OpaqueBlock)

Struct(StructDataBlock)

Dictionary(DictionaryDataBlock)

Implementations

impl DataBlock

pub fn into_arrow( self, data_type: DataType, validate: bool, ) -> Result<ArrayData>

Convert self into an Arrow ArrayData

pub fn into_buffers(self) -> Vec<LanceBuffer>

Convert the data block into a collection of buffers for serialization

The order matters and will be used to reconstruct the data block at read time.

pub fn borrow_and_clone(&mut self) -> Self

Converts the data buffers into borrowed mode and clones the block

This is a zero-copy operation but requires a mutable reference to self and, afterwards, all buffers will be in Borrowed mode.

pub fn try_clone(&self) -> Result<Self>

Try and clone the block

This will fail if any buffers are in owned mode. You can call borrow_and_clone() to ensure that all buffers are in borrowed mode before calling this method.

pub fn name(&self) -> &'static str

pub fn num_values(&self) -> u64

impl DataBlock

pub fn as_all_null(self) -> Option<AllNullDataBlock>

pub fn as_nullable(self) -> Option<NullableDataBlock>

pub fn as_fixed_width(self) -> Option<FixedWidthDataBlock>

pub fn as_fixed_size_list(self) -> Option<FixedSizeListBlock>

pub fn as_variable_width(self) -> Option<VariableWidthBlock>

pub fn as_struct(self) -> Option<StructDataBlock>

pub fn as_dictionary(self) -> Option<DictionaryDataBlock>

impl DataBlock

pub fn from_arrays(arrays: &[ArrayRef], num_values: u64) -> Self

pub fn from_array<T: Array + 'static>(array: T) -> Self

Trait Implementations

impl Debug for DataBlock

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Arc<dyn Array>> for DataBlock

fn from(array: ArrayRef) -> Self

Converts to this type from the input type.