Struct LiquidByteArray 

pub struct LiquidByteArray { /* private fields */ }

An array that stores strings in a dictionary format, with a bit-packed array for the keys and a FSST array for the values.

Implementations

impl LiquidByteArray

pub fn from_bytes(bytes: Bytes, compressor: Arc<Compressor>) -> Self

Deserialize a LiquidByteArray from bytes, using zero-copy where possible.

impl LiquidByteArray

pub fn from_string_view_array( array: &StringViewArray, compressor: Arc<Compressor>, ) -> Self

Create a LiquidByteArray from an Arrow StringViewArray.

pub fn from_binary_view_array( array: &BinaryViewArray, compressor: Arc<Compressor>, ) -> Self

Create a LiquidByteArray from an Arrow BinaryViewArray.

pub fn train_compressor_bytes<'a, T: ArrayAccessor<Item = &'a [u8]>>( array: ArrayIter<T>, ) -> Arc<Compressor>

Train a compressor from an iterator of byte arrays.

pub fn train_compressor<'a, T: ArrayAccessor<Item = &'a str>>( array: ArrayIter<T>, ) -> Arc<Compressor>

Train a compressor from an iterator of strings.

pub fn from_string_array( array: &StringArray, compressor: Arc<Compressor>, ) -> Self

Create a LiquidByteArray from an Arrow StringArray.

pub fn from_byte_array<T: ByteArrayType>( array: &GenericByteArray<T>, compressor: Arc<Compressor>, ) -> Self

Create a LiquidByteArray from an Arrow ByteArray.

pub fn train_from_string_view( array: &StringViewArray, ) -> (Arc<Compressor>, Self)

Train a compressor from an Arrow StringViewArray.

pub fn train_from_binary_view( array: &BinaryViewArray, ) -> (Arc<Compressor>, Self)

Train a compressor from an Arrow BinaryViewArray.

pub fn train_from_arrow<T: ByteArrayType>( array: &GenericByteArray<T>, ) -> (Arc<Compressor>, Self)

Train a compressor from an Arrow ByteArray.

pub fn train_from_arrow_dict( array: &DictionaryArray<UInt16Type>, ) -> (Arc<Compressor>, Self)

Train a compressor from an Arrow DictionaryArray.

pub unsafe fn from_unique_dict_array( array: &DictionaryArray<UInt16Type>, compressor: Arc<Compressor>, ) -> Self

Only used when the dictionary is read from a trusted parquet reader, which reads a trusted parquet file, written by a trusted writer.

Safety

The caller must ensure that the values in the dictionary are unique.

pub fn from_dict_array( array: &DictionaryArray<UInt16Type>, compressor: Arc<Compressor>, ) -> Self

Create a LiquidByteArray from an Arrow DictionaryArray.

pub fn decompressor(&self) -> Decompressor<'_>

Get the decompressor of the LiquidStringArray.

pub fn to_dict_arrow(&self) -> DictionaryArray<UInt16Type>

Convert the LiquidStringArray to arrow’s DictionaryArray.

pub fn to_dict_arrow_with_selection( &self, selection: &BooleanArray, ) -> DictionaryArray<UInt16Type>

Convert the LiquidStringArray to a DictionaryArray with a selection.

pub fn to_arrow_array(&self) -> ArrayRef

Convert the LiquidStringArray to a StringArray.

pub fn compare_not_equals(&self, needle: &str) -> BooleanArray

Compare the values of the LiquidStringArray with a given string and return a BooleanArray of the result.

pub fn nulls(&self) -> Option<&NullBuffer>

Get the nulls of the LiquidStringArray.

pub fn compare_equals(&self, needle: &str) -> BooleanArray

Compare the values of the LiquidStringArray with a given string. Leverage the distinct values to speed up the comparison. TODO: We can further optimize this by vectorizing the comparison.

Trait Implementations

impl Clone for LiquidByteArray

fn clone(&self) -> LiquidByteArray

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LiquidByteArray

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl LiquidArray for LiquidByteArray

fn as_any(&self) -> &dyn Any

Get the underlying any type.

fn get_array_memory_size(&self) -> usize

Get the memory size of the Liquid array.

fn len(&self) -> usize

Get the length of the Liquid array.

fn to_arrow_array(&self) -> ArrayRef

Convert the Liquid array to an Arrow array.

fn to_best_arrow_array(&self) -> ArrayRef

Convert the Liquid array to an Arrow array. Except that it will pick the best encoding for the arrow array. Meaning that it may not obey the data type of the original arrow array.

fn try_eval_predicate( &self, expr: &Arc<dyn PhysicalExpr>, filter: &BooleanBuffer, ) -> Option<BooleanArray>

Try to evaluate a predicate on the Liquid array with a filter. Returns None if the predicate is not supported.

fn to_bytes(&self) -> Vec<u8>

Serialize the Liquid array to a byte array.

fn original_arrow_data_type(&self) -> DataType

Get the original arrow data type of the Liquid array.

fn data_type(&self) -> LiquidDataType

Get the logical data type of the Liquid array.

fn is_empty(&self) -> bool

Check if the Liquid array is empty.

fn filter(&self, selection: &BooleanBuffer) -> ArrayRef

Filter the Liquid array with a boolean array and return an arrow array.

fn squeeze( &self, _io: Arc<dyn SqueezeIoHandler>, _expression_hint: Option<&CacheExpression>, ) -> Option<(LiquidSqueezedArrayRef, Bytes)>

Squeeze the Liquid array to a LiquidHybridArrayRef and a bytes::Bytes. Return None if the Liquid array cannot be squeezed.