Function estimate 

pub fn estimate(text: &str) -> usize

Token estimation and context usage helpers. Estimate token count using a hybrid algorithm that combines character-based and word-based heuristics.

The estimator accounts for:

• CJK characters (1 token per character – ideographic languages tokenize nearly 1:1 with modern BPE tokenizers)
• Punctuation & symbols (~1.5 tokens per character – they tend to form short, independent tokens)
• Common ASCII (~0.25 tokens per character, i.e. ~4 chars/token)
• Whitespace overhead (~1 token per whitespace-separated word)

For typical mixed English source code and prose this gives results within ±10% of tiktoken outputs for GPT-4-class tokenizers.

Examples

use oxicode_ai::estimate_tokens;
let text = "Hello, world! This is a test.";
let tokens = estimate_tokens(text);
assert!(tokens > 0);

Arguments

• text - The text to estimate tokens for

Returns

Estimated token count