Struct HfModel 

pub struct HfModel {
    pub model_type: String,
    pub vocab: HashMap<String, u32>,
    pub merges: Option<Vec<String>>,
    pub unk_token: Option<String>,
    pub continuing_subword_prefix: Option<String>,
    pub max_input_chars_per_word: Option<u32>,
}

The model object inside a HuggingFace tokenizers JSON.

Fields

model_type: String

The "type" discriminant string (e.g. "WordPiece", "BPE").

vocab: HashMap<String, u32>

Token-string → integer-ID vocabulary mapping.

merges: Option<Vec<String>>

Ordered BPE merge rules as "A B" strings (BPE only).

unk_token: Option<String>

The UNK token string.

continuing_subword_prefix: Option<String>

The continuation subword prefix (WordPiece: "##").

max_input_chars_per_word: Option<u32>

Maximum number of input characters per word before falling back to UNK.

Trait Implementations

impl Clone for HfModel

fn clone(&self) -> HfModel

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HfModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.