Module tiktoken 

Structs

TiktokenTokenizer

Tiktoken tokenizer wrapper — supports both built-in OpenAI encodings and hub-loaded models.

Enums

TiktokenModel

Supported Tiktoken models

Functions

has_tiktoken_file

Check whether a directory contains a tiktoken model file.

is_tiktoken_file

Check whether a single file is a tiktoken model file (by name).