Skip to main content

find_hash_factor_for_dictionary

bpe::byte_pair_encoding

Function find_hash_factor_for_dictionary 

Source 
pub fn find_hash_factor_for_dictionary(
    tokens: impl IntoIterator<Item = Vec<u8>>,
) -> u64
Expand description

Find a suitable hash factor for a set of given tokens that prevents collisions when constructing a BytePairEncoding from those tokens.