Function rust_tokenizers::preprocessing::tokenizer::tokenization_utils::bpe

pub fn bpe(token: &str, bpe_ranks: &BpePairVocab) -> (Vec<String>, Vec<usize>)

Default bpe function, as called by Roberta and GPT2