Module tokenizer 

Token estimation utilities

Simple utilities for estimating token counts. The default implementations in Model and ModelProvider use ~4 characters per token, but you can use these utilities for custom token estimation.

Structs

CharacterTokenizer

Simple character-based token estimator Uses ~4 characters per token heuristic (common approximation)