Expand description
LZ, variable: a byte-oriented LZ77 codec that picks its token layout per value instead of fixing one for all data.
The v is the whole idea. LZ4 spends four bits of every token on the
literal length and four on the match length, once, for everything it will
ever compress. lzvolt carries two layouts and writes in the header which
one a value used – and a value may switch partway.
lzvolt compresses and decompresses small-to-medium values — the sizes a cache
or a key-value store actually holds — and it reads and writes LZ4 blocks as
well, so it can be dropped in front of data somebody else compressed.
let data = br#"{"id":1,"tenant":"tenant42","role":"member"}"#;
let mut packed = Vec::new();
if lzvolt::compress(data, &mut packed) {
let mut out = Vec::new();
lzvolt::decompress(&packed, &mut out).unwrap();
assert_eq!(out, data);
}§What is different about it
The token layout is chosen per value. LZ4 spends four bits of every
token on the literal length and four on the match length. That split is a
compromise across all data, and for data that compresses well it is the
wrong one: counting real blocks, the match length overflows four bits in 99%
of them at a six-fold ratio. lzvolt carries two layouts — four/four, and two
literal bits with five match bits and one bit meaning “the offset the last
block used” — and writes in the header which one a value used. A value may
even switch partway, which is one call per section to the decoder.
The decoder is hand-written per part line, not per architecture. There
are separate assembled bodies for Intel’s server line, AMD’s, an
unrecognised SSSE3 part, Arm’s Neoverse V2 and everything else AArch64,
selected at run time from CPUID and MIDR. They exist because the same change
has measured opposite signs on two parts often enough to stop treating “x86”
as one target: the last literal run wants rep movsb on a Xeon and a copy
ladder on an EPYC, and Neoverse V2 wants its source aligned where an Apple
part does not care.
Every number this crate claims was measured on rented hardware and is
recorded, with the refutations, in docs/MEASUREMENTS.md. The format itself
is specified in docs/FORMAT.md, which is normative where this
implementation is not.
§The four entry points
compress and decompress are the codec. decompress_lz4 reads a
block somebody else’s LZ4 wrote. backend says which body this machine
selected, which is what a benchmark result should be labelled with.
Everything else is in raw, and none of it is stable.
Re-exports§
pub use format::PackError as Error;
Modules§
- cpu
- Runtime CPU feature detection.
- ffi
- The C ABI, for callers that are not Rust. See
include/lzvolt.h. - format
- A byte-oriented LZ77 packer for entry values.
- kernel
- Hand-written SIMD kernels for KevaVolt’s hottest inner loops.
- raw
- The lower level, and not stable.
Functions§
- backend
- Which assembled body this machine selected, for labelling a measurement.
- compress
- Compress
inputintoout, appending nothing if it is not worth it. - decompress
- Decompress a stream
compressproduced. - decompress_
into - Decompress into a buffer you already have, returning the bytes written.
- decompress_
lz4 - Decompress an LZ4 block, with the length known from somewhere else.
- decompressed_
bound - How much room
decompress_intowants for a value ofdeclaredbytes. - decompressed_
size - The uncompressed length a stream declares, without decoding it.