Skip to main content

Crate vortex_btrblocks

Crate vortex_btrblocks 

Source
Expand description

Vortex’s BtrBlocks-inspired adaptive compression framework.

This crate provides a sophisticated multi-level compression system that adaptively selects optimal compression schemes based on data characteristics. The compressor analyzes arrays to determine the best encoding strategy, supporting cascaded compression with multiple encoding layers for maximum efficiency.

§Key Features

  • Adaptive Compression: Automatically selects the best compression scheme based on data patterns.
  • Unified Scheme Trait: A single Scheme trait covers all data types (integers, floats, strings, etc.) with a SchemeId for identity.
  • Cascaded Encoding: Multiple compression layers can be applied for optimal results.
  • Statistical Analysis: Uses data sampling and statistics to predict compression ratios.
  • Recursive Structure Handling: Compresses nested structures like structs and lists.

§How It Works

BtrBlocksCompressor::compress() takes an &ArrayRef and returns an ArrayRef that may use a different encoding. It first canonicalizes the input, then dispatches by type. Primitives and strings go through choose_and_compress, which evaluates every enabled Scheme and picks the one with the best compression ratio. Compound types like structs and lists recurse into their fields and elements.

Each Scheme implementation declares whether it matches a given canonical form and, if so, estimates the compression ratio (often by compressing a ~1% sample). There is no dynamic registry — the set of schemes is fixed at build time via ALL_SCHEMES.

Schemes can produce arrays that are themselves further compressed (e.g. FoR then BitPacking), up to MAX_CASCADE (3) layers deep. Descendant exclusion rules for of SchemeId prevents the same scheme from being applied twice in a chain.

§Example

use vortex_btrblocks::{BtrBlocksCompressor, BtrBlocksCompressorBuilder, Scheme, SchemeExt};
use vortex_btrblocks::schemes::integer::IntDictScheme;

// Default compressor with all schemes enabled.
let compressor = BtrBlocksCompressor::default();

// Remove specific schemes using the builder.
let compressor = BtrBlocksCompressorBuilder::default()
    .exclude_schemes([IntDictScheme.id()])
    .build();

Modules§

schemes
Compression scheme implementations. Compression scheme implementations.

Structs§

ArrayAndStats
An array bundled with its lazily-computed statistics cache.
BoolStats
Array of booleans and relevant stats for compression.
BtrBlocksCompressor
The BtrBlocks-style compressor with all built-in schemes pre-registered.
BtrBlocksCompressorBuilder
Builder for creating configured BtrBlocksCompressor instances.
CascadingCompressor
The main compressor type implementing cascading adaptive compression.
CompressorContext
Context passed through recursive compression calls.
FloatStats
Array of floating-point numbers and relevant stats for compression.
GenerateStatsOptions
Configures how stats are generated.
IntegerStats
Array of integers and relevant stats for compression.
SchemeId
Unique identifier for a compression scheme.
StringStats
Array of variable-length byte arrays, and relevant stats for compression.

Constants§

ALL_SCHEMES
All available compression schemes.
MAX_CASCADE
Maximum cascade depth for compression.

Traits§

Scheme
A single compression encoding that the CascadingCompressor can select from.
SchemeExt
Extension trait providing id for all Scheme implementors.

Functions§

compress_patches
Compresses the given patches by downscaling integers and checking for constant values.