Struct TokenEstimation 

pub struct TokenEstimation {
    pub chars_per_token: usize,
}

How many characters of serialized JSON count as one token in the cheap no-tokenizer heuristic. Configured under [context.estimation] and threaded through estimate_tokens and friends.

The default (4) is the universal rough heuristic; the per-model calibration ratio (model-self-tuning, docs/design/model-self-tuning.md §2.3) scales the result on top. Exposed as a setting because a different model/tokenizer may pack a different number of chars per token — a smart default that adapts when an expert (human or LLM) tunes it.

Fields

chars_per_token: usize

Implementations

impl TokenEstimation

pub fn new(chars_per_token: usize) -> Self

Construct from an explicit ratio (for tests / call sites that aren’t config-driven). A zero is clamped to 1 so division never panics.

pub fn tokens_for_chars(&self, chars: usize) -> usize

Token estimate for a chars-character string (ceiling-divide, so a 1-char fragment never estimates to zero — err toward counting, 18.1).

pub fn chars_for_tokens(&self, tokens: usize) -> usize

The inverse: a character budget for tokens tokens. Used to size a summarizer input cap from a token budget.

Trait Implementations

impl Clone for TokenEstimation

fn clone(&self) -> TokenEstimation

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenEstimation

impl Debug for TokenEstimation

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenEstimation

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TokenEstimation

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Eq for TokenEstimation

impl PartialEq for TokenEstimation

fn eq(&self, other: &TokenEstimation) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for TokenEstimation

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for TokenEstimation