Expand description
Encoding-agnostic compression framework for Vortex arrays.
This crate provides the core compression engine: the Scheme trait,
sampling-based ratio estimation, cascaded compression, and statistics infrastructure for
deciding the best encoding scheme for an array.
This crate contains no encoding dependencies. Batteries-included compressors are provided by
downstream crates like vortex-btrblocks, which register different encodings to the compressor.
§Observability
The compressor emits a small set of tracing spans and events on a single target so you can
see what it’s doing without attaching a profiler.
For example, set RUST_LOG=vortex_compressor::encode=debug to see compression decision spans
and exceptional events. The vortex_compressor::encode target carries the top-level compress
span, per-scheme evaluation and winning-compression spans, the cascade_exhausted event,
sample.result events for zero-byte sample outputs, and both *.compress_failed events.
The winning-compression span carries scheme_chosen, input_nbytes, compressed_nbytes,
estimated_ratio (absent when the scheme returned AlwaysUse or sampled to 0 bytes),
achieved_ratio (absent when the compressed output is 0 bytes), and accepted.
Failure events additionally carry cascade_path and cascade_depth, so nested compression
errors can be tied back to the ancestor branch that triggered them.
From those fields you can derive per-scheme savings, rejection counts, and estimator accuracy
with a short jq query.
Modules§
- builtins
- Built-in compression schemes that use only
vortex-arrayencodings. - ctx
- Compression context for recursive compression.
- estimate
- Compression ratio estimation types and sampling-based estimation.
- scheme
- Unified compression scheme trait and exclusion rules.
- stats
- Compression statistics types and caching.
Structs§
- Cascading
Compressor - The main compressor type implementing cascading adaptive compression.