Crate vortex_compressor 

Encoding-agnostic compression framework for Vortex arrays.

This crate provides the core compression engine: the Scheme trait, sampling-based ratio estimation, cascaded compression, and statistics infrastructure for deciding the best encoding scheme for an array.

This crate contains no encoding dependencies. Batteries-included compressors are provided by downstream crates like vortex-btrblocks, which register different encodings to the compressor.

Example

A CascadingCompressor can be created directly with a fixed scheme list. With no schemes it still canonicalizes supported inputs and recursively handles nested structure, but no leaf compression is selected.

use vortex_array::{IntoArray, VortexSessionExecute, array_session};
use vortex_array::arrays::PrimitiveArray;
use vortex_array::validity::Validity;
use vortex_buffer::buffer;
use vortex_compressor::CascadingCompressor;

let session = array_session();
let array = PrimitiveArray::new(buffer![1i32, 2, 3], Validity::NonNullable).into_array();
let compressor = CascadingCompressor::new(Vec::new());

let result = compressor.compress(&array, &mut session.create_execution_ctx())?;
assert_eq!(result.dtype(), array.dtype());
assert_eq!(result.len(), array.len());

Observability

The compressor emits a small set of tracing spans and events on a single target so you can see what it’s doing without attaching a profiler.

For example, set RUST_LOG=vortex_compressor::encode=debug to see compression decision spans and exceptional events. The vortex_compressor::encode target carries the top-level compress span, per-scheme evaluation and winning-compression spans, the cascade_exhausted event, sample.result events for zero-byte sample outputs, and both *.compress_failed events.

The winning-compression span carries scheme_chosen, input_nbytes, compressed_nbytes, estimated_ratio (absent when the scheme returned AlwaysUse or sampled to 0 bytes), achieved_ratio (absent when the compressed output is 0 bytes), and accepted.

Failure events additionally carry cascade_path and cascade_depth, so nested compression errors can be tied back to the ancestor branch that triggered them.

From those fields you can derive per-scheme savings, rejection counts, and estimator accuracy with a short jq query.

Modules

builtins

Built-in compression schemes that use only vortex-array encodings.

ctx

Compression context for recursive compression.

estimate

Compression ratio estimation types and sampling-based estimation.

scheme

Unified compression scheme trait and exclusion rules.

stats

Compression statistics types and caching.

Structs

CascadingCompressor

The main compressor type implementing cascading adaptive compression.