Crate vortex_btrblocks 

Vortex’s BtrBlocks-inspired adaptive compression framework.

This crate provides a sophisticated multi-level compression system that adaptively selects optimal compression schemes based on data characteristics. The compressor analyzes arrays to determine the best encoding strategy, supporting cascaded compression with multiple encoding layers for maximum efficiency.

Key Features

• Adaptive Compression: Automatically selects the best compression scheme based on data patterns.
• Unified Scheme Trait: A single Scheme trait covers all data types (integers, floats, strings, etc.) with a SchemeId for identity.
• Cascaded Encoding: Multiple compression layers can be applied for optimal results.
• Statistical Analysis: Uses data sampling and statistics to predict compression ratios.
• Recursive Structure Handling: Compresses nested structures like structs and lists.

How It Works

BtrBlocksCompressor::compress() takes an &ArrayRef plus a mutable execution context and returns an ArrayRef that may use a different encoding. It first canonicalizes the input, then dispatches by type. Primitives and strings go through choose_and_compress, which evaluates every enabled Scheme and picks the one with the best compression ratio. Compound types like structs and lists recurse into their fields and elements.

Each Scheme implementation declares whether it matches a given canonical form and, if so, estimates the compression ratio (often by compressing a ~1% sample). There is no dynamic registry — the set of schemes is fixed at build time via ALL_SCHEMES.

Schemes can produce arrays that are themselves further compressed (e.g. FoR then BitPacking), up to MAX_CASCADE (3) layers deep. Descendant exclusion rules for of SchemeId prevents the same scheme from being applied twice in a chain.

Example

use vortex_btrblocks::{BtrBlocksCompressor, BtrBlocksCompressorBuilder, Scheme, SchemeExt};
use vortex_btrblocks::schemes::integer::IntDictScheme;

// Default compressor with all schemes enabled.
let compressor = BtrBlocksCompressor::default();

// Remove specific schemes using the builder.
let compressor = BtrBlocksCompressorBuilder::default()
    .exclude_schemes([IntDictScheme.id()])
    .build();

Modules

schemes

Compression scheme implementations. Compression scheme implementations.

Structs

ArrayAndStats

An array bundled with its lazily-computed statistics cache.

BoolStats

Array of booleans and relevant stats for compression.

BtrBlocksCompressor

The BtrBlocks-style compressor with all built-in schemes pre-registered.

BtrBlocksCompressorBuilder

Builder for creating configured BtrBlocksCompressor instances.

CascadingCompressor

The main compressor type implementing cascading adaptive compression.

CompressorContext

Context passed through recursive compression calls.

FloatStats

Array of floating-point numbers and relevant stats for compression.

GenerateStatsOptions

Configures how stats are generated.

IntegerStats

Array of integers and relevant stats for compression.

SchemeId

Unique identifier for a compression scheme.

StringStats

Array of variable-length byte arrays, and relevant stats for compression.

Constants

ALL_SCHEMES

All available compression schemes.

MAX_CASCADE

Maximum cascade depth for compression.

Traits

Scheme

A single compression encoding that the CascadingCompressor can select from.

SchemeExt

Extension trait providing id for all Scheme implementors.

Functions

compress_patches

Compresses the given patches by downscaling integers and checking for constant values.