Module token 

Token estimation using character-based heuristics.

Uses the chars / 4 rule-of-thumb to estimate token counts without requiring a tokenizer. The 20,000-token buffer in compaction absorbs estimation error, and the heuristic is consistent with industry practice.

Functions

estimate_content_part_tokens

Estimates the token count for a content part.

estimate_message_tokens

Estimates the total token count for a provider Message.

estimate_messages_tokens

Estimates the total token count for a slice of provider Messages.

estimate_tokens

Estimates the number of tokens for a string.

estimate_tool_call_tokens

Estimates the token count for a tool call.