Skip to main content

create_tokenizer_from_file

dynamo_tokenizers

Function create_tokenizer_from_file 

Source 
pub fn create_tokenizer_from_file(file_path: &str) -> Result<Arc<dyn Tokenizer>>
Expand description

Create a tokenizer from a file path to a tokenizer file. The file extension is used to determine the tokenizer type. Supported file types are:

  • json: HuggingFace tokenizer
  • model, tiktoken: tiktoken BPE tokenizer (requires config.json with a supported model_type in the same directory; currently: kimi, kimi_k2, kimi_k25)