Module tokens 

Enums

TokenizerFamily

Tokenizer families for different LLM providers.

Functions

count_tokens

Counts BPE tokens using the default tokenizer (o200k_base).

count_tokens_for

Counts BPE tokens using the specified tokenizer family.

detect_tokenizer

Detects the appropriate tokenizer family from a client or model name.

encode_tokens

Encodes text into BPE token IDs (o200k_base).

encode_tokens_for

Encodes text into BPE token IDs for the specified tokenizer family.