Enum Tokenizer 

pub enum Tokenizer {
    O200kHarmony,
    O200kBase,
    Cl100kBase,
    P50kBase,
    R50kBase,
    P50kEdit,
    Gpt2,
}

Enum representing the available tokenizers for different OpenAI models.

This enum lists the possible tokenizer types that can be used for tokenizing text when working with various OpenAI models.

Example

use tiktoken_rs::tokenizer::Tokenizer;

let tokenizer = Tokenizer::Cl100kBase;

Variants

O200kHarmony

O200kBase

Cl100kBase

P50kBase

R50kBase

P50kEdit

Gpt2

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Tokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Hash for Tokenizer

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for Tokenizer

fn eq(&self, other: &Tokenizer) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for Tokenizer

impl Eq for Tokenizer

impl StructuralPartialEq for Tokenizer