Module tokenizer 

Tokenizer trait + default impls.

The builder uses a Tokenizer to re-measure every crate::source::Contribution before pruning, so a source that over-estimates estimated_tokens can’t blow the budget.

Two impls ship in the crate:

• CharApproxTokenizer — default; 1 token ≈ 4 chars rule of thumb. Zero runtime dependencies, fine for development and tests, but not accurate enough for production budget enforcement against a real provider.
• TiktokenCl100k — behind the tiktoken feature. Uses the tiktoken-rs crate for ground-truth OpenAI cl100k_base counts. Suitable for any provider whose tokenizer is close to GPT-4 / GPT-3.5 (Anthropic’s Claude is close enough for budget purposes; for exact Claude counts, implement Tokenizer with a Claude-specific tokenizer and pass it to the builder).

Structs

CharApproxTokenizer

Trivial tokenizer using the well-known “≈ 4 chars per token” rule of thumb. Default for the builder so the crate has no runtime model assets.

Traits

Tokenizer

Counts tokens for a &str.