Enum VocabType 

#[repr(u32)]
pub enum VocabType {
    BPE = 2,
    SPM = 1,
}

Defines the possible types of vocabulary used by the model.

The model may use different types of vocabulary depending on the tokenization method chosen during training. This enum represents these types, specifically BPE (Byte Pair Encoding) and SPM (SentencePiece).

Variants

• BPE: Byte Pair Encoding, a common tokenization method used in NLP tasks.
• SPM: SentencePiece, another popular tokenization method for NLP models.

Example

use llama_cpp_4::model::VocabType;

let vocab_type = VocabType::BPE;
match vocab_type {
    VocabType::BPE => println!("The model uses Byte Pair Encoding (BPE)"),
    VocabType::SPM => println!("The model uses SentencePiece (SPM)"),
}

Variants

BPE = 2

Byte Pair Encoding

SPM = 1

Sentence Piece Tokenizer

Trait Implementations

impl Clone for VocabType

fn clone(&self) -> VocabType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for VocabType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for VocabType

fn eq(&self, other: &VocabType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl TryFrom<u32> for VocabType

type Error = LlamaTokenTypeFromIntError

The type returned in the event of a conversion error.

fn try_from(value: llama_vocab_type) -> Result<Self, Self::Error>

Performs the conversion.

impl Copy for VocabType

impl Eq for VocabType

impl StructuralPartialEq for VocabType