Enum TokenizerError 

#[non_exhaustive]
pub enum TokenizerError {
    UnknownToken(String),
    Io(Error),
    MalformedFile(String),
}

Errors that can occur while tokenizing text or loading a tokenizer.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

UnknownToken(String)

A sub-word produced during tokenization is not present in the vocabulary, and no <unk> / [UNK] fallback id was configured.

Io(Error)

An I/O error while loading a vocabulary or merge file (std builds only).

MalformedFile(String)

A line in a vocabulary or merge file could not be parsed.

Trait Implementations

impl Debug for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Error for TokenizerError

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0:

use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0:

replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl From<Error> for TokenizerError

Available on crate feature std only.

fn from(e: Error) -> Self

Converts to this type from the input type.