Skip to main content

create_raw_dict_from_dir

Function create_raw_dict_from_dir 

Source
pub fn create_raw_dict_from_dir<P: AsRef<Path>, W: Write>(
    path: P,
    output: &mut W,
    dict_size: usize,
) -> Result<(), Error>
Available on crate feature dict-builder only.
Expand description

Create a “raw content” dictionary of at most dict_size bytes, with no entropy tables, from every file in this directory and its subdirectories, and write it to output.

Each file is one sample, and the content is what optimize_fastcover_dict picks at its defaults, as zstd --train does; too few files to search is trained as create_raw_dict_from_slice trains an undivided corpus.

§Errors

Returns an error reading the directory or its files, writing to output, or allocating the trainer’s tables.

§Examples

use std::fs::File;
// Create a roughly 1mb dictionary, training off of file in `sample_files`
let input_folder = "sample_files/";
let mut output = File::create("output.dict").unwrap();
structured_zstd::dictionary::create_raw_dict_from_dir(input_folder, &mut output, 1_000_000)
    .expect("dictionary training from sample_files should succeed");