Struct TokenCount 

pub struct TokenCount {
    pub count: usize,
    pub encoding: String,
    pub char_count: Option<usize>,
}

Result of token counting for a text or file.

Fields

count: usize

Number of tokens

encoding: String

Encoding used (e.g., “cl100k_base”)

char_count: Option<usize>

Original text length in characters (if available)

Implementations

impl TokenCount

pub fn new(count: usize, encoding: &str) -> Self

Create a new TokenCount.

pub fn with_char_count(count: usize, encoding: &str, char_count: usize) -> Self

Create a new TokenCount with character count.

Trait Implementations

impl Clone for TokenCount

fn clone(&self) -> TokenCount

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenCount

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.