Module tokens 

Enums

TokenizerFamily

Tokenizer families for different LLM providers.

Constants

COUNTING_FAMILY

The tokenizer family count_tokens uses for all read/savings accounting.

Functions

count_tokens

Counts BPE tokens using the default tokenizer (o200k_base).

count_tokens_for

Counts BPE tokens using the specified tokenizer family.

counting_family_label

Label of the tokenizer used for counting (e.g. "o200k_base").

detect_tokenizer

Detects the appropriate tokenizer family from a client or model name.

encode_tokens

Encodes text into BPE token IDs (o200k_base).

encode_tokens_for

Encodes text into BPE token IDs for the specified tokenizer family.