Skip to main content

Module tokenize

Module tokenize 

Source

Structsยง

TokenCounter
Estimates token counts using the cl100k_base tokenizer (GPT-4 / Claude compatible). Accuracy is within ~5% across non-OpenAI providers; the budget is a soft guarantee.