Skip to main content

Crate lzvolt

Crate lzvolt 

Source
Expand description

LZ, variable: a byte-oriented LZ77 codec that picks its token layout per value instead of fixing one for all data.

The v is the whole idea. LZ4 spends four bits of every token on the literal length and four on the match length, once, for everything it will ever compress. lzvolt carries two layouts and writes in the header which one a value used – and a value may switch partway.

lzvolt compresses and decompresses small-to-medium values — the sizes a cache or a key-value store actually holds — and it reads and writes LZ4 blocks as well, so it can be dropped in front of data somebody else compressed.

let data = br#"{"id":1,"tenant":"tenant42","role":"member"}"#;

let mut packed = Vec::new();
if lzvolt::compress(data, &mut packed) {
    let mut out = Vec::new();
    lzvolt::decompress(&packed, &mut out).unwrap();
    assert_eq!(out, data);
}

§What is different about it

The token layout is chosen per value. LZ4 spends four bits of every token on the literal length and four on the match length. That split is a compromise across all data, and for data that compresses well it is the wrong one: counting real blocks, the match length overflows four bits in 99% of them at a six-fold ratio. lzvolt carries two layouts — four/four, and two literal bits with five match bits and one bit meaning “the offset the last block used” — and writes in the header which one a value used. A value may even switch partway, which is one call per section to the decoder.

The decoder is hand-written per part line, not per architecture. There are separate assembled bodies for Intel’s server line, AMD’s, an unrecognised SSSE3 part, Arm’s Neoverse V2 and everything else AArch64, selected at run time from CPUID and MIDR. They exist because the same change has measured opposite signs on two parts often enough to stop treating “x86” as one target: the last literal run wants rep movsb on a Xeon and a copy ladder on an EPYC, and Neoverse V2 wants its source aligned where an Apple part does not care.

Every number this crate claims was measured on rented hardware and is recorded, with the refutations, in docs/MEASUREMENTS.md. The format itself is specified in docs/FORMAT.md, which is normative where this implementation is not.

§The four entry points

compress and decompress are the codec. decompress_lz4 reads a block somebody else’s LZ4 wrote. backend says which body this machine selected, which is what a benchmark result should be labelled with.

Everything else is in raw, and none of it is stable.

Re-exports§

pub use format::PackError as Error;

Modules§

cpu
Runtime CPU feature detection.
ffi
The C ABI, for callers that are not Rust. See include/lzvolt.h.
format
A byte-oriented LZ77 packer for entry values.
kernel
Hand-written SIMD kernels for KevaVolt’s hottest inner loops.
raw
The lower level, and not stable.

Functions§

backend
Which assembled body this machine selected, for labelling a measurement.
compress
Compress input into out, appending nothing if it is not worth it.
decompress
Decompress a stream compress produced.
decompress_into
Decompress into a buffer you already have, returning the bytes written.
decompress_lz4
Decompress an LZ4 block, with the length known from somewhere else.
decompressed_bound
How much room decompress_into wants for a value of declared bytes.
decompressed_size
The uncompressed length a stream declares, without decoding it.