pub fn get_bpe(tokenizer: &str) -> CoreBPE
Expand description
Returns the appropriate BPE based on the tokenizer name.
§Arguments
tokenizer - A string slice that holds the name of the tokenizer.
§Returns
Returns a CoreBPE instance for the specified tokenizer.