Struct ByteCeiling 

pub struct ByteCeiling;

Counts a token per byte.

The only counter that is a genuine upper bound: byte-level BPE never emits more tokens than there are bytes, because merging is all it does. Far too conservative for ordinary use — it would spend a 400-token cap on 400 bytes — but it is the right answer when a cap must not be exceeded under any tokeniser at all.

Trait Implementations

impl Clone for ByteCeiling

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for ByteCeiling

impl Debug for ByteCeiling

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ByteCeiling

fn default() -> Self

Returns the “default value” for a type.

impl Eq for ByteCeiling

impl PartialEq for ByteCeiling

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for ByteCeiling

impl TokenCount for ByteCeiling

fn count(&self, text: &str) -> usize

How many tokens text is worth.