Trait TokenCounter 

pub trait TokenCounter {
    // Required method
    fn count_tokens(&self, text: &str) -> usize;
}

Trait for counting tokens in text. Allows swapping word count for a real tokenizer (e.g. HuggingFace tokenizers) later.

Required Methods

fn count_tokens(&self, text: &str) -> usize

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl<T> TokenCounter for &T

where T: TokenCounter + ?Sized,

Blanket implementation so &dyn TokenCounter can be used anywhere TokenCounter is required.

fn count_tokens(&self, text: &str) -> usize

impl<T> TokenCounter for Box<T>

where T: TokenCounter + ?Sized,

Blanket implementation so Box<dyn TokenCounter + Send + Sync> can be passed to functions that accept impl TokenCounter (like chunk_text).

fn count_tokens(&self, text: &str) -> usize

Implementors

impl TokenCounter for HuggingFaceTokenCounter

Available on crate feature hf-tokenizer only.

impl TokenCounter for TikTokenCounter

Available on crate feature tiktoken only.

impl TokenCounter for WordCounter