Module factory 

Enums

TokenizerType

Represents the type of tokenizer being used

Functions

create_tokenizer

Factory function to create tokenizer from a model name or path (blocking version)

create_tokenizer_async

Factory function to create tokenizer from a model name or path (async version)

create_tokenizer_async_with_chat_template

Factory function to create tokenizer with optional chat template (async version)

create_tokenizer_from_file

Create a tokenizer from a file path to a tokenizer file. The file extension is used to determine the tokenizer type. Supported file types are:

create_tokenizer_with_chat_template

Create a tokenizer from a file path with an optional chat template

create_tokenizer_with_chat_template_blocking

Factory function to create tokenizer with optional chat template (blocking version)

discover_chat_template_in_dir

Helper function to discover chat template files in a directory

get_tokenizer_info

Get information about a tokenizer file