Struct AddedTokenConfig 

pub struct AddedTokenConfig {
    pub id: u32,
    pub content: String,
    pub single_word: bool,
    pub lstrip: bool,
    pub rstrip: bool,
    pub normalized: bool,
    pub special: bool,
}

An entry in the added_tokens array of tokenizer.json.

Fields

id: u32

Token ID.

content: String

Literal text content that triggers this token.

single_word: bool

Whether the token should only match as a whole word.

lstrip: bool

Whether to strip whitespace on the left when matching.

rstrip: bool

Whether to strip whitespace on the right when matching.

normalized: bool

Whether the content should be matched against normalized text.

special: bool

Whether this is a “special” token (e.g. BOS/EOS). All added tokens are matched regardless of this flag; it only affects post-processing.

Trait Implementations

impl Clone for AddedTokenConfig

fn clone(&self) -> AddedTokenConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AddedTokenConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for AddedTokenConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.