pub fn create_raw_dict_from_dir<P: AsRef<Path>, W: Write>(
path: P,
output: &mut W,
dict_size: usize,
) -> Result<(), Error>Available on crate feature
dict-builder only.Expand description
Create a “raw content” dictionary of at most dict_size bytes, with no
entropy tables, from every file in this directory and its subdirectories,
and write it to output.
Each file is one sample, and the content is what optimize_fastcover_dict
picks at its defaults, as zstd --train does; too few files to search is
trained as create_raw_dict_from_slice trains an undivided corpus.
§Errors
Returns an error reading the directory or its files, writing to output,
or allocating the trainer’s tables.
§Examples
use std::fs::File;
// Create a roughly 1mb dictionary, training off of file in `sample_files`
let input_folder = "sample_files/";
let mut output = File::create("output.dict").unwrap();
structured_zstd::dictionary::create_raw_dict_from_dir(input_folder, &mut output, 1_000_000)
.expect("dictionary training from sample_files should succeed");