Enum HfModelType 

#[non_exhaustive]
pub enum HfModelType {
    WordPiece,
    Bpe,
    Unigram,
    Unknown(String),
}

The tokenizer model type encoded in a HuggingFace tokenizers JSON file.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

WordPiece

BERT WordPiece model.

Bpe

Byte-Pair Encoding model (GPT-2, RoBERTa, …).

Unigram

SentencePiece Unigram Language Model.

Unknown(String)

Unknown or unrecognised model type.

Implementations

impl HfModelType

pub fn as_str(&self) -> &str

Convert to the string used in the JSON model.type field.

pub fn parse(s: &str) -> Self

Parse from the JSON string representation.

Trait Implementations

impl Clone for HfModelType

fn clone(&self) -> HfModelType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for HfModelType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for HfModelType

fn eq(&self, other: &HfModelType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for HfModelType

impl StructuralPartialEq for HfModelType