pub fn optimize_cover_dict(
samples: &[u8],
sample_sizes: &[usize],
dict_size: usize,
options: &CoverOptions,
finalize: FinalizeOptions,
) -> Result<(Vec<u8>, CoverOptions)>Available on crate feature
dict-builder only.Expand description
Train COVER dictionaries over a range of k and d and keep the one the
scoring samples compress best with (the reference’s
ZDICT_optimizeTrainFromBuffer_cover). Returns it with the options that
built it.
A zero k searches 50..=2000 in steps strides (zero steps is 40), a
zero d tries 6 and 8. The leading split_point of the samples build and
the rest score; at 1 every sample does both. CoverOptions::shrink takes
effect here, which the reference’s optimizer does not honour.
§Errors
As train_cover_dict, and when split_point exceeds 1, when k is
below d, or when the split leaves fewer than five samples to build or
none to score.
§Examples
use structured_zstd::dictionary::{CoverOptions, FinalizeOptions, optimize_cover_dict};
let mut samples = Vec::new();
let mut sizes = Vec::new();
for i in 0..200u32 {
let line = format!("tenant=demo table=orders key={i} region=eu status=shipped\n");
sizes.push(line.len());
samples.extend_from_slice(line.as_bytes());
}
let (dict, chosen) = optimize_cover_dict(
&samples,
&sizes,
4096,
&CoverOptions::default(),
FinalizeOptions::default(),
)
.unwrap();
assert!(dict.len() <= 4096 && chosen.k >= 50);