Struct Tokenizer 

pub struct Tokenizer;

Token counting utility using character-based estimation. Uses ~4 characters per token as a reasonable approximation.

Stateless. Only counting is implemented; context-window budget is enforced via MAX_CONTEXT_TOKENS / CONTEXT_RESERVE_TOKENS constants and the user’s num_ctx Ollama option. The old per-model-family table (get_max_tokens / remaining_tokens) and the base_model_name field were never consulted by the budget logic and have been deleted.

Implementations

impl Tokenizer

pub fn new(_model_name: &str) -> Self

Create a new tokenizer. The model_name parameter is accepted for call-site compatibility but ignored — the count is a pure function of text length.

pub fn count_tokens(&self, text: &str) -> Result<usize>

Count tokens in a single text string (~4 chars per token)

pub fn count_chat_tokens(&self, messages: &[(String, String)]) -> Result<usize>

Count tokens in a chat message format