Struct TiktokenTokenCounter 

pub struct TiktokenTokenCounter { /* private fields */ }

Accurate BPE-based token counter using OpenAI’s o200k_base encoding.

Uses tiktoken-rs with the vocabulary bundled at compile time — no runtime downloads. This is the recommended counter for production use.

Implementations

impl TiktokenTokenCounter

pub fn new(metadata_overhead: u32) -> Self

Create with a custom metadata overhead.

Trait Implementations

impl Debug for TiktokenTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TiktokenTokenCounter

fn default() -> Self

Returns the “default value” for a type.

impl TokenCounter for TiktokenTokenCounter

fn count_message(&self, message: &Message) -> u32

Count tokens in a single message.

fn count_text(&self, text: &str) -> u32

Count tokens in a plain text string.

fn count_messages(&self, messages: &[Message]) -> u32

Count tokens in multiple messages.