Skip to main content

optimize_cover_dict

Function optimize_cover_dict 

Source
pub fn optimize_cover_dict(
    samples: &[u8],
    sample_sizes: &[usize],
    dict_size: usize,
    options: &CoverOptions,
    finalize: FinalizeOptions,
) -> Result<(Vec<u8>, CoverOptions)>
Available on crate feature dict-builder only.
Expand description

Train COVER dictionaries over a range of k and d and keep the one the scoring samples compress best with (the reference’s ZDICT_optimizeTrainFromBuffer_cover). Returns it with the options that built it.

A zero k searches 50..=2000 in steps strides (zero steps is 40), a zero d tries 6 and 8. The leading split_point of the samples build and the rest score; at 1 every sample does both. CoverOptions::shrink takes effect here, which the reference’s optimizer does not honour.

§Errors

As train_cover_dict, and when split_point exceeds 1, when k is below d, or when the split leaves fewer than five samples to build or none to score.

§Examples

use structured_zstd::dictionary::{CoverOptions, FinalizeOptions, optimize_cover_dict};

let mut samples = Vec::new();
let mut sizes = Vec::new();
for i in 0..200u32 {
    let line = format!("tenant=demo table=orders key={i} region=eu status=shipped\n");
    sizes.push(line.len());
    samples.extend_from_slice(line.as_bytes());
}
let (dict, chosen) = optimize_cover_dict(
    &samples,
    &sizes,
    4096,
    &CoverOptions::default(),
    FinalizeOptions::default(),
)
.unwrap();
assert!(dict.len() <= 4096 && chosen.k >= 50);