Struct TokenCounter

pub struct TokenCounter { /* private fields */ }

A simple token counter for tracking token usage in conversations

This is a very basic implementation that provides a minimal token counting utility. In a real-world scenario, you would want to use a proper tokenizer like tiktoken-rs for accurate token counting specific to the model being used.

Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert_eq!(counter.total(), 2); // "Hello," and "world!" as two tokens

counter.subtract("Hello,");
assert_eq!(counter.total(), 1); // "world!" as one token

assert!(counter.under_budget(10)); // 1 < 10

Implementations

impl TokenCounter

pub fn new() -> Self

Creates a new TokenCounter with zero tokens

Examples

use language_barrier_core::TokenCounter;

let counter = TokenCounter::new();
assert_eq!(counter.total(), 0);

pub fn count_tokens(text: &str) -> usize

Counts the number of tokens in a string (naive implementation)

This is a very simple whitespace-based tokenization. In a real-world implementation, you would want to use a proper tokenizer like tiktoken-rs.

Examples

use language_barrier_core::TokenCounter;

assert_eq!(TokenCounter::count_tokens("Hello, world!"), 2);
assert_eq!(TokenCounter::count_tokens("one two three four"), 4);

pub fn observe(&mut self, text: &str)

Adds the token count of the given text to the total

Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert_eq!(counter.total(), 2);

pub fn subtract(&mut self, text: &str)

Subtracts the token count of the given text from the total

Will not go below zero (saturates at zero).

Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
counter.subtract("Hello,");
assert_eq!(counter.total(), 1);

// Won't go below zero
counter.subtract("world! and more");
assert_eq!(counter.total(), 0);

pub fn total(&self) -> usize

Returns the current total token count

Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert_eq!(counter.total(), 2);

pub fn under_budget(&self, max: usize) -> bool

Checks if the total token count is under the specified budget

Examples

use language_barrier_core::TokenCounter;

let mut counter = TokenCounter::default();
counter.observe("Hello, world!");
assert!(counter.under_budget(5));
assert!(!counter.under_budget(1));

Trait Implementations

impl Clone for TokenCounter

fn clone(&self) -> TokenCounter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenCounter

fn default() -> TokenCounter

Returns the “default value” for a type.