Skip to main content

TokenCounter

deepstrike_core::context::token_engine

Trait TokenCounter 

Source 
pub trait TokenCounter: Send + Sync {
    // Required methods
    fn count(&self, text: &str) -> u32;
    fn truncate<'a>(&self, text: &'a str, max_tokens: u32) -> &'a str;
}
Expand description

Token counting and truncation interface. Implementations must be deterministic and must never panic on any valid UTF-8 input.

Required Methods§

Source

fn count(&self, text: &str) -> u32

Count tokens in a UTF-8 string.

Source

fn truncate<'a>(&self, text: &'a str, max_tokens: u32) -> &'a str

Return the longest prefix of text that fits within max_tokens. The returned slice is always a valid UTF-8 prefix of text.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§