Skip to main content

train_cover_dict

Function train_cover_dict 

Source
pub fn train_cover_dict(
    samples: &[u8],
    sample_sizes: &[usize],
    dict_size: usize,
    options: &CoverOptions,
    finalize: FinalizeOptions,
) -> Result<Vec<u8>>
Available on crate feature dict-builder only.
Expand description

Train a COVER dictionary of at most dict_size bytes with the k and d given (the reference’s ZDICT_trainFromBuffer_cover).

samples is every sample back to back and sample_sizes their lengths. Every sample builds, and the entropy tables are drawn from all of them. With CoverOptions::shrink the samples also score the shrinking search; steps and split_point are ignored.

§Errors

InvalidInput when k or d is zero or d > k, when k exceeds dict_size, when dict_size is under MIN_TRAINED_DICT_SIZE, when there are fewer than five samples or they do not add up to samples.len(); TrainingError::of tells these causes apart.

§Examples

use structured_zstd::dictionary::{CoverOptions, FinalizeOptions, train_cover_dict};

let mut samples = Vec::new();
let mut sizes = Vec::new();
for i in 0..200u32 {
    let line = format!("tenant=demo table=orders key={i} region=eu status=shipped\n");
    sizes.push(line.len());
    samples.extend_from_slice(line.as_bytes());
}
let options = CoverOptions { k: 64, ..CoverOptions::default() };
let dict = train_cover_dict(&samples, &sizes, 4096, &options, FinalizeOptions::default())
    .unwrap();
assert!(dict.starts_with(&[0x37, 0xA4, 0x30, 0xEC]) && dict.len() <= 4096);