Trait TokenCounter 

pub trait TokenCounter: Send + Sync {
    // Required method
    fn count_tokens(&self, text: &str) -> usize;
}

Abstraction for counting tokens in a string.

Implement this trait to plug in an accurate tokenizer such as tiktoken-rs.

Required Methods

fn count_tokens(&self, text: &str) -> usize

Implementors

impl TokenCounter for SimpleTokenCounter