Enum TokenizerKind 

pub enum TokenizerKind {
    Heuristic,
    Claude,
    Gpt4,
    Gpt35,
}

Which tokenizer to use for token counting.

Variants

Heuristic

Heuristic estimation: bytes/3 for code, bytes/4 for prose (default)

Claude

Claude tokenizer (uses cl100k_base as approximation)

Gpt4

GPT-4 tokenizer (cl100k_base)

Gpt35

GPT-3.5 tokenizer (cl100k_base)

Implementations

impl TokenizerKind

pub fn parse_name(s: &str) -> Option<Self>

Parse a tokenizer name from CLI input.

pub fn valid_names() -> &'static str

List valid tokenizer names for help text.

Trait Implementations

impl Clone for TokenizerKind

fn clone(&self) -> TokenizerKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerKind

fn default() -> TokenizerKind

Returns the “default value” for a type.

impl Display for TokenizerKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for TokenizerKind

fn eq(&self, other: &TokenizerKind) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for TokenizerKind

impl StructuralPartialEq for TokenizerKind