Skip to main content

create_raw_dict_from_slice

Function create_raw_dict_from_slice 

Source
pub fn create_raw_dict_from_slice<W: Write>(
    all: &[u8],
    output: &mut W,
    dict_size: usize,
) -> Result<()>
Available on crate feature dict-builder only.
Expand description

Create a “raw content” dictionary of at most dict_size bytes from a corpus already in memory, writing it to output.

The same training as create_raw_dict_from_source, reading source in place: a caller that holds the samples anyway does not pay for a second copy of them.

§Errors

Returns the error output reports while the dictionary is written.

§Examples

use structured_zstd::dictionary::create_raw_dict_from_slice;

let corpus: Vec<u8> = (0..20_000u32)
    .flat_map(|i| format!("record {} value {}\n", i % 100, i % 7).into_bytes())
    .collect();
let mut dict = Vec::new();
create_raw_dict_from_slice(&corpus, &mut dict, 4096).unwrap();
assert!(!dict.is_empty() && dict.len() <= 4096);