Enum VocabSource 

pub enum VocabSource {
    VocabTxt(&'static str),
    TokenizerJson(&'static str),
}

Vocabulary/tokenizer source for a model.

Variants

VocabTxt(&'static str)

Standard BERT vocab.txt (one token per line, WordPiece).

TokenizerJson(&'static str)

HuggingFace tokenizer.json (BPE/Unigram via JSON config).

Implementations

impl VocabSource

pub fn filename(&self) -> &'static str

pub fn is_wordpiece(&self) -> bool

Trait Implementations

impl Clone for VocabSource

fn clone(&self) -> VocabSource

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for VocabSource

impl Debug for VocabSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.