Trait TokenEstimator 

pub trait TokenEstimator: Send + Sync {
    // Required method
    fn count_tokens(&self, text: &str) -> usize;
}

Estimates the number of tokens in a string.

Implement this trait to replace the default len / 4 heuristic with a model-specific tokenizer (e.g. tiktoken, sentencepiece).

Example

struct TiktokenEstimator { enc: tiktoken::Encoding }
impl TokenEstimator for TiktokenEstimator {
    fn count_tokens(&self, text: &str) -> usize {
        self.enc.encode_ordinary(text).len()
    }
}

Required Methods

fn count_tokens(&self, text: &str) -> usize

Return an approximate token count for text.

Implementors

impl TokenEstimator for CharDivTokenEstimator