1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
//! LZ, variable: a byte-oriented LZ77 codec that picks its token layout per
//! value instead of fixing one for all data.
//!
//! The `v` is the whole idea. LZ4 spends four bits of every token on the
//! literal length and four on the match length, once, for everything it will
//! ever compress. `lzvolt` carries two layouts and writes in the header which
//! one a value used -- and a value may switch partway.
//!
//! `lzvolt` compresses and decompresses small-to-medium values — the sizes a cache
//! or a key-value store actually holds — and it reads and writes LZ4 blocks as
//! well, so it can be dropped in front of data somebody else compressed.
//!
//! ```
//! let data = br#"{"id":1,"tenant":"tenant42","role":"member"}"#;
//!
//! let mut packed = Vec::new();
//! if lzvolt::compress(data, &mut packed) {
//! let mut out = Vec::new();
//! lzvolt::decompress(&packed, &mut out).unwrap();
//! assert_eq!(out, data);
//! }
//! ```
//!
//! # What is different about it
//!
//! **The token layout is chosen per value.** LZ4 spends four bits of every
//! token on the literal length and four on the match length. That split is a
//! compromise across all data, and for data that compresses well it is the
//! wrong one: counting real blocks, the match length overflows four bits in 99%
//! of them at a six-fold ratio. `lzvolt` carries two layouts — four/four, and two
//! literal bits with five match bits and one bit meaning "the offset the last
//! block used" — and writes in the header which one a value used. A value may
//! even switch partway, which is one call per section to the decoder.
//!
//! **The decoder is hand-written per part line, not per architecture.** There
//! are separate assembled bodies for Intel's server line, AMD's, an
//! unrecognised SSSE3 part, Arm's Neoverse V2 and everything else AArch64,
//! selected at run time from CPUID and MIDR. They exist because the same change
//! has measured opposite signs on two parts often enough to stop treating "x86"
//! as one target: the last literal run wants `rep movsb` on a Xeon and a copy
//! ladder on an EPYC, and Neoverse V2 wants its source aligned where an Apple
//! part does not care.
//!
//! Every number this crate claims was measured on rented hardware and is
//! recorded, with the refutations, in `docs/MEASUREMENTS.md`. The format itself
//! is specified in `docs/FORMAT.md`, which is normative where this
//! implementation is not.
//!
//! # The four entry points
//!
//! [`compress`] and [`decompress`] are the codec. [`decompress_lz4`] reads a
//! block somebody else's LZ4 wrote. [`backend`] says which body this machine
//! selected, which is what a benchmark result should be labelled with.
//!
//! Everything else is in [`raw`], and none of it is stable.
/// The C ABI, for callers that are not Rust. See `include/lzvolt.h`.
///
/// Always present: the symbols cost a Rust caller nothing in an `rlib`, and
/// putting them behind a feature would let the published shared library and
/// the published crate disagree about what exists.
pub use PackError as Error;
/// Compress `input` into `out`, appending nothing if it is not worth it.
///
/// Returns `false` when the value does not compress — incompressible data, or
/// data too short for a backward reference to pay for itself. The caller is
/// expected to store the original bytes in that case, which is strictly cheaper
/// than a format that must wrap them: there is no header to write and no copy to
/// make on the way back out.
///
/// `out` is appended to, not cleared.
/// Decompress a stream [`compress`] produced.
///
/// `out` is appended to. The declared length is read from the stream's header,
/// so no capacity hint is needed; a stream that declares a length it does not
/// produce is [`Error::LengthMismatch`] rather than a truncated answer.
/// Decompress into a buffer you already have, returning the bytes written.
///
/// The same decoder [`decompress`] runs, writing straight into `out` with no
/// copy afterwards. Use it when the destination is already allocated — an
/// arena, a page, a slot in a cache — which is the case this codec was built
/// for.
///
/// `out` must be at least [`decompressed_size`] of the stream, and should be
/// [`decompressed_bound`] of that for the wide copies.
/// The uncompressed length a stream declares, without decoding it.
///
/// Validates the header, so what this rejects [`decompress`] would reject too.
/// How much room [`decompress_into`] wants for a value of `declared` bytes.
///
/// Exactly `declared` decodes the same bytes; this much lets the decoders copy
/// in wide blocks, which on incompressible 4 KiB was worth 34%.
pub const
/// Decompress an LZ4 block, with the length known from somewhere else.
///
/// This is the raw LZ4 block format — no frame, no magic, no checksum — which
/// is what `LZ4_compress_default` writes and what `LZ4_decompress_safe` reads.
/// `declared` is the uncompressed size, which the LZ4 block format does not
/// carry and the caller therefore has to know.
///
/// `out` is appended to.
/// Which assembled body this machine selected, for labelling a measurement.
///
/// A number from this crate means little without it: the bodies differ in their
/// copy widths and their thresholds, and two of them exist precisely because the
/// same source measured opposite signs on two parts.
/// The lower level, and not stable.
///
/// Section decoding, the per-body entry points, the split selector and the
/// kernel dispatch. These are what the tests and the benchmarks reach for, and
/// what a caller with its own buffer management may want. They change without a
/// major version; [`compress`], [`decompress`], [`decompress_lz4`] and
/// [`backend`] do not.