Skip to main content

Module tokens

Module tokens 

Source
Expand description

Token estimation — single source for session and llm (issue #52 C2).

session::estimate_tokens and llm::token_counter::chars_to_tokens_fallback were two copies of chars.div_ceil(4) with a subtle mismatch: session returned 0 for empty input while llm returned max(1). Unifying here makes the policy explicit and testable.

Functions§

estimate_tokens
~4 Unicode scalars per token (matches whycodes_llm fallback family).
estimate_tokens_at_least_one
Same as estimate_tokens but at least 1 (for llm::count_tokens where an empty prompt still costs one token in provider accounting).