Crate vortex_compressor 

Encoding-agnostic compression framework for Vortex arrays.

This crate provides the core compression engine: the Scheme trait, sampling-based ratio estimation, cascaded compression, and statistics infrastructure for deciding the best encoding scheme for an array.

This crate contains no encoding dependencies. Batteries-included compressors are provided by downstream crates like vortex-btrblocks, which register different encodings to the compressor.

Example

A CascadingCompressor can be created directly with a fixed scheme list. With no schemes it still canonicalizes supported inputs, recursively handles nested structure, and encodes constant leaves (constant detection is built into the compressor), but no other leaf compression is selected.

use vortex_array::{IntoArray, VortexSessionExecute, array_session};
use vortex_array::arrays::PrimitiveArray;
use vortex_array::validity::Validity;
use vortex_buffer::buffer;
use vortex_compressor::CascadingCompressor;

let session = array_session();
let array = PrimitiveArray::new(buffer![1i32, 2, 3], Validity::NonNullable).into_array();
let compressor = CascadingCompressor::new(Vec::new());

let result = compressor.compress(&array, &mut session.create_execution_ctx())?;
assert_eq!(result.dtype(), array.dtype());
assert_eq!(result.len(), array.len());

Observability

The compressor emits a small set of tracing spans and events on a single target so you can see what it’s doing without attaching a profiler.

For example, set RUST_LOG=vortex_compressor::encode=debug to see compression decision spans and exceptional events. The vortex_compressor::encode target carries the top-level compress span, per-scheme evaluation and winning-compression spans, the cascade_exhausted event, sample.result events for zero-byte sample outputs, and both *.compress_failed events.

The winning-compression span carries scheme_chosen, input_nbytes, compressed_nbytes, estimated_ratio (absent when the scheme returned AlwaysUse or sampled to 0 bytes), achieved_ratio (absent when the compressed output is 0 bytes), and accepted.

Failure events additionally carry cascade_path and cascade_depth, so nested compression errors can be tied back to the ancestor branch that triggered them.

From those fields you can derive per-scheme savings, rejection counts, and estimator accuracy with a short jq query.

Modules

builtins

Built-in compression schemes that use only vortex-array encodings.

ctx

Compression context for recursive compression.

estimate

Compression ratio estimation types and sampling-based estimation.

scheme

Unified compression scheme trait and exclusion rules.

stats

Compression statistics types and caching.

Structs

CascadingCompressor

The main compressor type implementing cascading adaptive compression.