Struct TiktokenTokenCounter 

pub struct TiktokenTokenCounter { /* private fields */ }

Accurate BPE-based token counter using OpenAI’s o200k_base encoding.

Uses tiktoken-rs with the vocabulary bundled at compile time — no runtime downloads. This is the recommended counter for production use.

Implementations

impl TiktokenTokenCounter

pub fn new(metadata_overhead: u32) -> TiktokenTokenCounter

Create with a custom metadata overhead.

pub fn estimate_provider_visible_tool_segments<'a>( &self, segments: impl IntoIterator<Item = &'a str>, ) -> ProviderVisibleToolEstimate

Estimate already-lowered provider-visible tool segments exactly once.

segments must contain compact serialized JSON/text in the order the selected provider exposes it to the model. Each segment is one contiguous model position and is tokenized separately with the same counter used for prompt text; provider lowering must serialize adjacent content such as an initial tools array into one segment. This prevents BPE merges across positions separated by transcript content.

Repeated fragments at different model-visible positions remain repeated by design; callers must omit definitions that are not model-visible at that position. An empty slice returns an all-zero estimate.

pub fn truncate_to_token_prefix(&self, text: &str, max_tokens: u32) -> String

Truncate text to at most max_tokens tokens, keeping the START.

Encodes the text once and decodes the first max_tokens tokens back to a string — O(N) (one encode + one decode), versus the O(N²) char-by-char re-tokenization the previous find_prefix_within_tokens performed (which called count_text(&text[..i]) on every char index).

Semantics

• max_tokens == 0 → empty string (exactly 0 tokens; never exceeds budget).
• Text already within max_tokens → returned unchanged (fast path).
• Otherwise the result is an exact prefix of text (its START preserved), is valid UTF-8, and re-counts to ≤ max_tokens.

If the o200k encoder is unavailable (the issue #25 fallback path), this degrades to a conservative char-based cut instead of panicking.

pub fn truncate_to_token_suffix(&self, text: &str, max_tokens: u32) -> String

Truncate text to at most max_tokens tokens, keeping the END.

Symmetric to truncate_to_token_prefix: encodes once and decodes the last max_tokens tokens. Same budget / fast-path / fallback semantics; the result is a valid-UTF-8 exact suffix of text (its END preserved) that re-counts to ≤ max_tokens.

Trait Implementations

impl Debug for TiktokenTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Default for TiktokenTokenCounter

fn default() -> TiktokenTokenCounter

Returns the “default value” for a type.

impl TokenCounter for TiktokenTokenCounter

fn count_message(&self, message: &Message) -> u32

Count tokens in a single message.

fn count_text(&self, text: &str) -> u32

Count tokens in a plain text string.

fn count_messages(&self, messages: &[Message]) -> u32

Count tokens in multiple messages.