Expand description
Token estimation — single source for session and llm (issue #52 C2).
session::estimate_tokens and llm::token_counter::chars_to_tokens_fallback
were two copies of chars.div_ceil(4) with a subtle mismatch: session
returned 0 for empty input while llm returned max(1). Unifying here
makes the policy explicit and testable.
Functions§
- estimate_
tokens - ~4 Unicode scalars per token (matches
whycodes_llmfallback family). - estimate_
tokens_ at_ least_ one - Same as
estimate_tokensbut at least1(forllm::count_tokenswhere an empty prompt still costs one token in provider accounting).