Struct TokenCounter

Source

pub struct TokenCounter { /* private fields */ }

Expand description

A simple token counter for tracking token usage in conversations

This is a very basic implementation that provides a minimal token counting utility. In a real-world scenario, you would want to use a proper tokenizer like tiktoken-rs for accurate token counting specific to the model being used.

§Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert_eq!(counter.total(), 2); // "Hello," and "world!" as two tokens

counter.subtract("Hello,");
assert_eq!(counter.total(), 1); // "world!" as one token

assert!(counter.under_budget(10)); // 1 < 10

Implementations§

Source §

impl TokenCounter

Source

pub fn new() -> Self

Creates a new TokenCounter with zero tokens

§Examples

use language_barrier_core::TokenCounter;

let counter = TokenCounter::new();
assert_eq!(counter.total(), 0);

Source

pub fn count_tokens(text: &str) -> usize

Counts the number of tokens in a string (naive implementation)

This is a very simple whitespace-based tokenization. In a real-world implementation, you would want to use a proper tokenizer like tiktoken-rs.

§Examples

use language_barrier_core::TokenCounter;

assert_eq!(TokenCounter::count_tokens("Hello, world!"), 2);
assert_eq!(TokenCounter::count_tokens("one two three four"), 4);

Source

pub fn observe(&mut self, text: &str)

Adds the token count of the given text to the total

§Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert_eq!(counter.total(), 2);

Source

pub fn subtract(&mut self, text: &str)

Subtracts the token count of the given text from the total

Will not go below zero (saturates at zero).

§Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
counter.subtract("Hello,");
assert_eq!(counter.total(), 1);

// Won't go below zero
counter.subtract("world! and more");
assert_eq!(counter.total(), 0);

Source

pub fn total(&self) -> usize

Returns the current total token count

§Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert_eq!(counter.total(), 2);

Source

pub fn under_budget(&self, max: usize) -> bool

Checks if the total token count is under the specified budget

§Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert!(counter.under_budget(5));
assert!(!counter.under_budget(1));