Struct TokenEstimator 

pub struct TokenEstimator;

Token Estimator for different content types.

Provides methods to estimate token counts for text, messages, and message arrays.

Implementations

impl TokenEstimator

pub fn estimate_tokens(text: &str) -> usize

Estimate the number of tokens in a text string.

Uses different character-per-token ratios based on content type:

• Asian characters: ~2 chars/token
• Code: ~3 chars/token
• English text: ~3.5 chars/token

Also adds weight for special characters and newlines.

Arguments

• text - The text to estimate tokens for

Returns

Estimated number of tokens

Example

use aster::context::token_estimator::TokenEstimator;

let english_text = "Hello, world!";
let tokens = TokenEstimator::estimate_tokens(english_text);
assert!(tokens > 0);

pub fn has_asian_chars(text: &str) -> bool

Check if text contains Asian characters (Chinese, Japanese, Korean).

Arguments

• text - The text to check

Returns

true if the text contains significant Asian characters

pub fn is_code(text: &str) -> bool

Check if text appears to be code.

Uses heuristics to detect code content:

• Presence of code-specific characters ({}, [], ;, etc.)
• Indentation patterns with code keywords
• Common code keywords

Arguments

• text - The text to check

Returns

true if the text appears to be code

pub fn estimate_message_tokens(message: &Message) -> usize

Estimate the number of tokens in a message.

Includes message overhead (role, formatting) plus content tokens.

Arguments

• message - The message to estimate tokens for

Returns

Estimated number of tokens

pub fn estimate_total_tokens(messages: &[Message]) -> usize

Estimate the total number of tokens for an array of messages.

Arguments

• messages - The messages to estimate tokens for

Returns

Total estimated tokens across all messages