Struct FSSTData 

pub struct FSSTData { /* private fields */ }

The inner data for an FSST-compressed array.

Holds the FSST symbol table (symbols + symbol_lengths) and the raw compressed codes bytes buffer. The codes offsets and validity live in the outer array’s slots (slots 1 and 2 respectively).

A full VarBinArray representing the codes can be reconstructed on demand via FSSTArrayExt::codes(), combining this buffer with the offsets/validity from slots.

Implementations

impl FSSTData

pub fn try_new( symbols: Buffer<Symbol>, symbol_lengths: Buffer<u8>, codes_bytes: BufferHandle, len: usize, ) -> VortexResult<Self>

Build FSST data from a set of symbols, symbol_lengths, and compressed codes bytes.

Symbols are 8-bytes and can represent short strings, each of which is assigned a code.

The codes_bytes buffer contains the concatenated compressed bytecodes for all elements. Each element’s compressed bytecodes are a sequence of 8-bit codes, where each code corresponds either to a symbol or to the “escape code” (which tells the decoder to emit the following byte without doing a table lookup).

symbols and symbol_lengths hold only the populated entries; they are padded out to FSST_SYMBOL_TABLE_LEN entries so that Self::decompressor can borrow them without copying.

The offsets and validity for the codes are stored in the array’s slots, not here. Use FSSTArrayExt::codes() to reconstruct a full VarBinArray.

pub fn validate( &self, dtype: &DType, len: usize, slots: &[Option<ArrayRef>], ) -> VortexResult<()>

Check the structure of the array without executing any of its children.

The bounds of the codes offsets are checked by the constructors and on deserialization, which have an execution context.

pub fn len(&self) -> usize

Returns the number of elements in the array.

pub fn is_empty(&self) -> bool

Returns true if the array contains no elements.

pub fn symbols(&self) -> &[Symbol]

Access the symbol table array.

This is the populated prefix of the symbol table; the padding added on construction to reach FSST_SYMBOL_TABLE_LEN is not included. Use Self::padded_symbols to get the whole buffer.

pub fn symbol_lengths(&self) -> &[u8]

Access the symbol lengths array.

As with Self::symbols, this excludes the padding added on construction.

pub fn padded_symbols(&self) -> &Buffer<Symbol>

The whole symbols buffer, padded to exactly FSST_SYMBOL_TABLE_LEN entries with Symbol::ZERO. Entries at or past Self::n_symbols are padding.

pub fn padded_symbol_lengths(&self) -> &Buffer<u8>

The whole symbol lengths buffer, padded to exactly FSST_SYMBOL_TABLE_LEN entries with zeros. Entries at or past Self::n_symbols are padding.

pub fn n_symbols(&self) -> usize

The number of populated entries in the symbol table.

pub fn symbol_table(&self) -> Arc<FSSTSymbolTable>

pub fn codes_bytes_handle(&self) -> &BufferHandle

Access the compressed codes bytes buffer handle (may be on host or device).

pub fn codes_bytes(&self) -> &ByteBuffer

Access the compressed codes bytes on the host.

pub fn decompressor(&self) -> Decompressor<'_>

Build a Decompressor that can be used to decompress values from this array.

pub fn compressor(&self) -> &Compressor

Retrieves the FSST compressor.

Trait Implementations

impl ArrayEq for FSSTData

fn array_eq(&self, other: &Self, precision: EqMode) -> bool

impl ArrayHash for FSSTData

fn array_hash<H: Hasher>(&self, state: &mut H, precision: EqMode)

impl Clone for FSSTData

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FSSTData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for FSSTData

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.