Skip to main content

create_raw_dict_from_source

Function create_raw_dict_from_source 

Source
pub fn create_raw_dict_from_source<R: Read, W: Write>(
    source: R,
    source_size: usize,
    output: &mut W,
    dict_size: usize,
) -> Result<()>
Available on crate feature dict_builder only.
Expand description

Read from source to create a “raw content” dictionary of dict_size. The completed dictionary is written to output.

  • source will be used as training data for the entire dictionary.
  • source_size is used only as a preallocation hint before reading source and does not affect sampling once all data has been buffered.
  • output is where the completed dictionary will be written.
  • dict_size determines how large the complete dictionary should be. The completed dictionary will be this size or smaller.

This function reads the entire source into an in-memory Vec<u8> before building the dictionary. The provided reader need not be buffered, but callers should avoid sources too large to fit comfortably in memory.

§API note

This public API returns io::Result<()> and propagates source/output I/O failures.