Enum LoadedTokenizer 

pub enum LoadedTokenizer {
    Bpe(BpeTokenizer),
    WordPiece(WordPieceTokenizer),
}

A tokenizer loaded from a tokenizer.json, tagged by its underlying scheme.

Both variants implement Tokenizer; match on this to recover the concrete type, or call LoadedTokenizer::as_tokenizer for a trait object.

Variants

Bpe(BpeTokenizer)

A Byte-Pair Encoding tokenizer.

WordPiece(WordPieceTokenizer)

A WordPiece tokenizer.

Implementations

impl LoadedTokenizer

pub fn as_tokenizer(&self) -> &dyn Tokenizer

Borrow the loaded tokenizer as a Tokenizer trait object.

Trait Implementations

impl Clone for LoadedTokenizer

fn clone(&self) -> LoadedTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LoadedTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.