Struct HfAddedToken 

pub struct HfAddedToken {
    pub id: u32,
    pub content: String,
    pub special: bool,
    pub single_word: bool,
    pub lstrip: bool,
    pub rstrip: bool,
    pub normalized: bool,
}

A special / added token entry in the HF JSON.

Fields

id: u32

Integer ID in the vocabulary.

content: String

Token surface string.

special: bool

Whether this token is a “special” control token.

single_word: bool

Whether the token should not be split during pre-tokenisation.

lstrip: bool

Whether this token strips leading whitespace.

rstrip: bool

Whether this token strips trailing whitespace.

normalized: bool

Whether the token is normalised.

Implementations

impl HfAddedToken

pub fn special(id: u32, content: impl Into<String>) -> Self

Construct a simple special token entry.

Trait Implementations

impl Clone for HfAddedToken

fn clone(&self) -> HfAddedToken

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HfAddedToken

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.