Module tokenizer

This module encapsulates the logic for counting the tokens in the rendered text.

Enums

TokenFormat

TokenizerType

Tokenizer types supported by tiktoken.

Functions

count_tokens

Counts the tokens in the provided text using the specified tokenizer type.