Enum SpecialTokens 

pub enum SpecialTokens {
    AsText,
    Parse,
}

Whether a literal special-token marker in the input (<s>, <|im_end|>, [SEP]) is carved out as the token it names or tokenized as the characters it is written with.

llama.cpp’s parse_special. Callers choose per site, because the right answer differs: a rendered chat prompt needs its template’s own markers parsed, while a stop string, a DRY sequence breaker or a rerank document is plain text whatever it happens to mention.

Variants

AsText

parse_special = false, llama.cpp’s library default. CONTROL and UNKNOWN entries are ordinary text; USER_DEFINED ones are still carved out (llama-vocab.cpp:3169-3174).

Parse

parse_special = true: every special entry is matched as an atomic substring before normal tokenization runs.

Trait Implementations

impl Clone for SpecialTokens

fn clone(&self) -> SpecialTokens

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for SpecialTokens

impl Debug for SpecialTokens

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for SpecialTokens

impl PartialEq for SpecialTokens

fn eq(&self, other: &SpecialTokens) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for SpecialTokens