Struct TokenCounter

pub struct TokenCounter { /* private fields */ }

Token counter using the GPT-4 tokenizer for accurate token counting.

Provides methods for counting tokens in text and truncating content to fit within specified token limits while maintaining text coherence.

Examples

use ai_context_gen::token_counter::TokenCounter;

let counter = TokenCounter::new().unwrap();
let text = "Hello, world!";
let token_count = counter.count_tokens(text);
println!("Text has {} tokens", token_count);

Implementations

impl TokenCounter

pub fn new() -> Result<Self>

Creates a new token counter using the GPT-4 tokenizer.

Returns

A new TokenCounter instance configured with the GPT-4 BPE tokenizer.

Errors

Returns an error if the GPT-4 tokenizer model cannot be loaded.

Examples

use ai_context_gen::token_counter::TokenCounter;

let counter = TokenCounter::new().unwrap();

pub fn count_tokens(&self, text: &str) -> usize

Counts the number of tokens in the given text.

Uses the GPT-4 tokenizer to provide accurate token counts that match what would be used by OpenAI’s models and similar systems.

Arguments

• text - The text to count tokens for

Returns

The number of tokens in the text.

Examples

use ai_context_gen::token_counter::TokenCounter;

let counter = TokenCounter::new().unwrap();
let count = counter.count_tokens("Hello, world!");
assert!(count > 0);

pub fn truncate_to_token_limit(&self, text: &str, max_tokens: usize) -> String

Truncates text to fit within a specified token limit.

Attempts to preserve text coherence by truncating at token boundaries rather than character boundaries. Falls back to character truncation if token decoding fails.

Arguments

• text - The text to truncate
• max_tokens - Maximum number of tokens to include

Returns

The truncated text that fits within the token limit.

Examples

use ai_context_gen::token_counter::TokenCounter;

let counter = TokenCounter::new().unwrap();
let long_text = "This is a very long text that exceeds the token limit...";
let truncated = counter.truncate_to_token_limit(long_text, 10);
assert!(counter.count_tokens(&truncated) <= 10);