pub fn train_cover_dict(
samples: &[u8],
sample_sizes: &[usize],
dict_size: usize,
options: &CoverOptions,
finalize: FinalizeOptions,
) -> Result<Vec<u8>>Available on crate feature
dict-builder only.Expand description
Train a COVER dictionary of at most dict_size bytes with the k and d
given (the reference’s ZDICT_trainFromBuffer_cover).
samples is every sample back to back and sample_sizes their lengths.
Every sample builds, and the entropy tables are drawn from all of them. With
CoverOptions::shrink the samples also score the shrinking search;
steps and split_point are ignored.
§Errors
InvalidInput when k or d is zero or d > k, when k exceeds
dict_size, when dict_size is under MIN_TRAINED_DICT_SIZE, when there
are fewer than five samples or they do not add up to samples.len();
TrainingError::of tells these causes apart.
§Examples
use structured_zstd::dictionary::{CoverOptions, FinalizeOptions, train_cover_dict};
let mut samples = Vec::new();
let mut sizes = Vec::new();
for i in 0..200u32 {
let line = format!("tenant=demo table=orders key={i} region=eu status=shipped\n");
sizes.push(line.len());
samples.extend_from_slice(line.as_bytes());
}
let options = CoverOptions { k: 64, ..CoverOptions::default() };
let dict = train_cover_dict(&samples, &sizes, 4096, &options, FinalizeOptions::default())
.unwrap();
assert!(dict.starts_with(&[0x37, 0xA4, 0x30, 0xEC]) && dict.len() <= 4096);