Skip to main content

Crate vortex_btrblocks

Crate vortex_btrblocks 

Source
Expand description

Vortex’s BtrBlocks-inspired adaptive compression framework.

This crate provides a sophisticated multi-level compression system that adaptively selects optimal compression schemes based on data characteristics. The compressor analyzes arrays to determine the best encoding strategy, supporting cascaded compression with multiple encoding layers for maximum efficiency.

§Key Features

  • Adaptive Compression: Automatically selects the best compression scheme based on data patterns
  • Type-Specific Compressors: Specialized compression for integers, floats, strings, and temporal data
  • Cascaded Encoding: Multiple compression layers can be applied for optimal results
  • Statistical Analysis: Uses data sampling and statistics to predict compression ratios
  • Recursive Structure Handling: Compresses nested structures like structs and lists

§How It Works

BtrBlocksCompressor::compress() takes an &ArrayRef and returns an ArrayRef that may use a different encoding. It first canonicalizes the input, then dispatches by type. Primitives go to a type-specific Compressor (integer, float, or string). Compound types like structs and lists recurse into their fields and elements.

Each type-specific compressor holds a static list of Scheme implementations (e.g. BitPacking, ALP, Dict). There is no dynamic registry. The compressor evaluates each scheme by compressing a ~1% sample and measuring the ratio, then picks the best. See SchemeExt for details on how sampling works.

Schemes can produce arrays that are themselves further compressed (e.g. FoR then BitPacking), up to MAX_CASCADE (3) layers deep. An Excludes set prevents the same scheme from being applied twice in a chain.

§Example

use vortex_btrblocks::{BtrBlocksCompressor, BtrBlocksCompressorBuilder, IntCode};
use vortex_array::DynArray;

// Default compressor with all schemes enabled
let compressor = BtrBlocksCompressor::default();

// Configure with builder to exclude specific schemes
let compressor = BtrBlocksCompressorBuilder::default()
    .exclude_int([IntCode::Dict])
    .build();

Structs§

BtrBlocksCompressor
The main compressor type implementing BtrBlocks-inspired compression.
BtrBlocksCompressorBuilder
Builder for creating configured BtrBlocksCompressor instances.
GenerateStatsOptions
Configures how stats are generated.
IntegerStats
Array of integers and relevant stats for compression.

Enums§

FloatCode
Unique identifier for float compression schemes.
IntCode
Unique identifier for integer compression schemes.
StringCode
Unique identifier for string compression schemes.

Traits§

CanonicalCompressor
Trait for compressors that can compress canonical arrays.
CompressorStats
Stats for the compressor.

Functions§

integer_dictionary_encode
Compresses an integer array into a dictionary arrays according to attached stats.