Module compression 

Compression traits and definitions for Lance 2.1

In 2.1 the first step of encoding is structural encoding, where we shred inputs into leaf arrays and take care of the validity / offsets structure. Then we pick a structural encoding (mini-block or full-zip) and then we compress the data.

This module defines the traits for the compression step. Each structural encoding has its own compression strategy.

Miniblock compression is a block based approach for small data. Since we introduce some read amplification and decompress entire blocks we are able to use opaque compression.

Fullzip compression is a per-value approach where we require that values are transparently compressed so that we can locate them later.

Structs

DefaultDecompressionStrategy

Traits

BlockCompressor

Trait for compression algorithms that compress an entire block of data into one opaque and self-described chunk.

BlockDecompressor

CompressionStrategy

A trait to pick which compression to use for given data

DecompressionStrategy

FixedPerValueDecompressor

MiniBlockDecompressor

VariablePerValueDecompressor

Functions

field_metadata_params

Parse field-level compression metadata without applying format-specific constraints.

finalize_miniblock_compressor

Apply general-purpose compression requested for a fixed-width miniblock.

reject_packed_struct_per_value

Reject variable-width packed structs while preserving the fixed-width error.

try_bitpacking_block

Select block bitpacking for applicable fixed-width values.

try_bitpacking_miniblock

Select inline bitpacking for applicable fixed-width miniblocks.

try_byte_stream_split_miniblock

Select byte-stream-split compression for an applicable fixed-width miniblock.

try_child_rle_miniblock

Select variable-width RLE with independently encoded children.

try_fixed_packed_struct_miniblock

Encode fixed-width packed structs as miniblocks.

try_fixed_u8_rle_block

Select fixed-u8 RLE for block compression.

try_fixed_u8_rle_miniblock

Select the original fixed-u8 RLE miniblock grammar.

try_general_block

Select explicitly requested or automatic general-purpose block compression.

try_raw_block

Store fixed- and variable-width block values without block compression.

try_raw_fixed_size_list_miniblock

Store fixed-size-list miniblocks without a value codec.

try_raw_fixed_width_miniblock

Store fixed-width miniblock values without a value codec.

try_raw_per_value

Store fixed-width and fixed-size-list values directly in full-zip pages.

try_uncompressed_fixed_width_miniblock

Honor an explicit compression = none request for fixed-width miniblocks.

try_variable_packed_struct_per_value

Encode variable-width packed structs with the exact strategy recursively.

try_variable_rle_block

Select variable-width RLE for block compression.

try_variable_width_miniblock

Encode variable-width miniblocks with binary or FSST encoding.

try_variable_width_per_value

Encode variable-width values directly, with FSST or per-value compression when applicable.