Struct OpenAITokenCounter 

pub struct OpenAITokenCounter { /* private fields */ }

Token counter for OpenAI GPT models using tiktoken.

Provides exact token counts for OpenAI models. Automatically selects the correct tokenizer based on the model name.

Supported Models

Model	Tokenizer	Context Window
gpt-4-turbo	cl100k_base	128K
gpt-4	cl100k_base	8K
gpt-3.5-turbo	cl100k_base	16K
o1-*	o200k_base	200K

Example

use multi_llm::{OpenAITokenCounter, TokenCounter};

let counter = OpenAITokenCounter::new("gpt-4")?;
let tokens = counter.count_tokens("Hello, world!")?;

Implementations

impl OpenAITokenCounter

pub fn new(model: &str) -> LlmResult<Self>

Create token counter for specific OpenAI model

pub fn for_lm_studio(max_tokens: u32) -> LlmResult<Self>

Create token counter for LM Studio (uses cl100k_base as default)

Trait Implementations

impl Debug for OpenAITokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TokenCounter for OpenAITokenCounter

fn count_tokens(&self, text: &str) -> LlmResult<u32>

Count tokens in a text string.

fn count_message_tokens(&self, messages: &[Value]) -> LlmResult<u32>

Count tokens in a list of messages (includes formatting overhead).

fn max_context_tokens(&self) -> u32

Get the maximum context window size for this tokenizer.

fn validate_token_limit(&self, text: &str) -> LlmResult<()>

Validate that text doesn’t exceed the token limit.

fn truncate_to_limit(&self, text: &str, max_tokens: u32) -> LlmResult<String>

Truncate text to fit within a token limit.