Module tokenizer 

Accurate token counting using tiktoken.

Provides precise token counting for messages to enable better context window management.

Functions

count_tokens

Count tokens in a text string.

message_overhead

Count tokens for a role prefix (e.g., “user: “, “assistant: “). Each message has overhead for role markers and formatting.