Enum TokenMethod 

#[non_exhaustive]
pub enum TokenMethod {
    Heuristic,
    Bpe,
}

Token-count measurement methodology.

Used to make explicit which counting method produced a number, so that downstream reports cannot silently mix the fast heuristic with an accurate BPE tokenizer.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

Heuristic

Character-count heuristic (chars_per_token). Fast, dependency-free, but self-referential when used to evaluate this crate’s own output: it bakes in the same assumptions (e.g. PUA = 1 token) that the compressor optimizes for, inflating the apparent reduction.

Bpe

Real BPE tokenizer (OpenAI cl100k_base via tiktoken-rs). Slower and requires the tiktoken feature, but independent of this crate’s compression assumptions — the honest basis for reduction claims.

Trait Implementations

impl Clone for TokenMethod

fn clone(&self) -> TokenMethod

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenMethod

impl Debug for TokenMethod

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenMethod

impl PartialEq for TokenMethod

fn eq(&self, other: &TokenMethod) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenMethod

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for TokenMethod