Enum TokenizerSource 

pub enum TokenizerSource {
    HuggingFace(String),
    LocalFile(String),
    IndexDirectory(String),
}

Tokenizer source - where to load the tokenizer from

Variants

HuggingFace(String)

Load from HuggingFace hub (e.g., “bert-base-uncased”) - native only

LocalFile(String)

Load from local file path - native only

IndexDirectory(String)

Load from index directory (relative path within index)

Implementations

impl TokenizerSource

pub fn parse(path: &str) -> Self

Parse a tokenizer path string into a TokenizerSource

• Paths starting with index:// are relative to index directory
• On native: Paths starting with / are absolute local paths
• On native: Other paths are treated as HuggingFace hub identifiers

Trait Implementations

impl Clone for TokenizerSource

fn clone(&self) -> TokenizerSource

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.