Struct Estimate 

pub struct Estimate;

The default token estimate: pessimistic, and deterministic.

How it counts

Three bytes per token, rounded up, plus one token per line. Real byte-level BPE averages closer to four bytes per token on prose and rather worse on dense punctuation, which is most of what a slicer handles — code, diffs, and JSON. The per-line term is there because newlines and the indentation after them tokenise poorly and a slicer’s output is unusually line-dense.

It is an estimate and says so. What matters is the direction of its error: over-estimating costs a little unused window, under-estimating overruns s_max and breaks Law II after the call has been paid for.

Trait Implementations

impl Clone for Estimate

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Estimate

impl Debug for Estimate

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Estimate

fn default() -> Self

Returns the “default value” for a type.

impl Eq for Estimate

impl PartialEq for Estimate

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Estimate

impl TokenCount for Estimate

fn count(&self, text: &str) -> usize

How many tokens text is worth.