Enum TokenizerModel 

pub enum TokenizerModel {
    Cl100kBase,
    P50kBase,
    Claude,
    Llama,
    Generic,
}

Supported tokenizer models

Variants

Cl100kBase

GPT-4 / GPT-3.5-turbo (cl100k_base)

P50kBase

GPT-3 (p50k_base)

Claude

Claude models

Llama

Llama models

Generic

Generic (heuristic-based)

Implementations

impl TokenizerModel

pub fn bytes_per_token(&self) -> f32

Get bytes per token estimate for fallback

pub fn name(&self) -> &'static str

Get model name string

Trait Implementations

impl Clone for TokenizerModel

fn clone(&self) -> TokenizerModel

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for TokenizerModel

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenizerModel

fn eq(&self, other: &TokenizerModel) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenizerModel

impl Eq for TokenizerModel

impl StructuralPartialEq for TokenizerModel