Struct ByteCountTokenCounter 

pub struct ByteCountTokenCounter;

Zero-dependency conservative counter — bytes.div_ceil(4).

Approximates English at the ~4-bytes-per-token rule of thumb that tiktoken’s cl100k_base is built around. Systematically inaccurate for CJK, Devanagari, Arabic, and other scripts whose UTF-8 byte cost diverges from typical token boundaries — operators shipping multilingual workloads inject a vendor-accurate counter (entelix-tokenizer-tiktoken, entelix-tokenizer-hf, locale-aware companions) at ChatModel::with_token_counter(...) time.

The bias direction is deliberate: div_ceil rounds up, so the estimate skews over the real count on average. Pre-flight RunBudget checks built on top remain conservative — a near-budget call is more likely refused than admitted, which is the correct error direction for budget enforcement.

Implementations

impl ByteCountTokenCounter

pub const fn new() -> Self

Construct the counter. Stateless — every call to Self::new returns the same logical instance.

Trait Implementations

impl Clone for ByteCountTokenCounter

fn clone(&self) -> ByteCountTokenCounter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ByteCountTokenCounter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ByteCountTokenCounter

fn default() -> ByteCountTokenCounter

Returns the “default value” for a type.

impl TokenCounter for ByteCountTokenCounter

fn count(&self, text: &str) -> u64

Count the tokens in text under this counter’s encoding.

fn encoding_name(&self) -> &'static str

Vendor-published encoding name ("cl100k_base", "o200k_base", "claude", "gemini-tokenizer", …) — surfaced on OTel gen_ai.tokenizer.name and operator diagnostics.

fn count_messages(&self, msgs: &[Message]) -> u64

Sum the token count across every text-bearing content part of a message slice. The default impl walks crate::ir::ContentPart::Text parts; non-text parts (image, tool-use, tool-result blocks) are vendor-specific in their token cost — counters that need an exact tally for those shapes override this method.

impl Copy for ByteCountTokenCounter