Trait TokenCounter 

pub trait TokenCounter: Send + Sync {
    // Required method
    fn count_tokens(&self, text: &str) -> usize;
}

Trait for counting tokens in text.

Implement this to plug in tiktoken, sentencepiece, or any model-specific tokenizer for accurate context budget management.

Examples

use semantic_memory::TokenCounter;

struct MyTokenizer;
impl TokenCounter for MyTokenizer {
    fn count_tokens(&self, text: &str) -> usize {
        text.split_whitespace().count()
    }
}

Required Methods

fn count_tokens(&self, text: &str) -> usize

Count the number of tokens in the given text.

Implementors

impl TokenCounter for EstimateTokenCounter