Enum TokenizerModel 

pub enum TokenizerModel {
    Cl100kBase,
    O200kBase,
    P50kBase,
}

Supported tokenizer models

Variants

Cl100kBase

GPT-4, GPT-3.5-turbo, text-embedding-ada-002

O200kBase

GPT-4o, GPT-4o-mini

P50kBase

GPT-3 models (text-davinci-003, etc.)

Trait Implementations

impl Clone for TokenizerModel

fn clone(&self) -> TokenizerModel

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerModel

fn default() -> TokenizerModel

Returns the “default value” for a type.

impl FromStr for TokenizerModel

type Err = String

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self, Self::Err>

Parses a string s to return a value of this type.

impl PartialEq for TokenizerModel

fn eq(&self, other: &TokenizerModel) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for TokenizerModel

impl Eq for TokenizerModel

impl StructuralPartialEq for TokenizerModel