Module tokens

Module tokens 

Source
Expand description

Token counting with cascading fallback.

Attempts token counting in order:

  1. API count_tokens (Anthropic-compatible endpoints)
  2. tiktoken (OpenAI/GPT models)
  3. Character estimate (4 chars ≈ 1 token)

Structs§

TokenCounter
Token counter with cascading fallback.

Functions§

create_token_counter
Create a TokenCounter from config values.