Enum llama_cpp_2::token_type::LlamaTokenType

#[repr(u32)]
pub enum LlamaTokenType {
    Undefined = 0,
    Normal = 1,
    Unknown = 2,
    Control = 3,
    UserDefined = 4,
    Unused = 5,
    Byte = 6,
}

A rust flavored equivalent of llama_token_type.

Variants

Undefined = 0

An undefined token type.

Normal = 1

A normal token type.

Unknown = 2

An unknown token type.

Control = 3

A control token type.

UserDefined = 4

A user defined token type.

Unused = 5

An unused token type.

Byte = 6

A byte token type.

Trait Implementations

impl Clone for LlamaTokenType

fn clone(&self) -> LlamaTokenType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LlamaTokenType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for LlamaTokenType

fn eq(&self, other: &LlamaTokenType) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<u32> for LlamaTokenType

A safe wrapper for converting potentially deceptive llama_token_type values into LlamaVocabType.

The error branch returns the original value.

let llama_token_type = LlamaTokenType::try_from(0 as c_uint)?;
assert_eq!(llama_token_type, LlamaTokenType::Undefined);

let bad_llama_token_type = LlamaTokenType::try_from(100 as c_uint);
assert_eq!(Err(LlamaTokenTypeFromIntError::UnknownValue(100)), bad_llama_token_type);

type Error = LlamaTokenTypeFromIntError

The type returned in the event of a conversion error.

fn try_from(value: llama_vocab_type) -> Result<Self, Self::Error>

Performs the conversion.

impl Copy for LlamaTokenType

impl Eq for LlamaTokenType

impl StructuralEq for LlamaTokenType

impl StructuralPartialEq for LlamaTokenType