Enum TokenizerError 

pub enum TokenizerError {
    NormalizeError(NormalizeError),
    PreTokenizeError(PreTokenizeError),
    EncodeError(EncodeError),
    DecodeError(DecodeError),
}

Error type returned when tokenizing a string.

Variants

NormalizeError(NormalizeError)

PreTokenizeError(PreTokenizeError)

An error occurred while performing pre-tokenization to split the input.

EncodeError(EncodeError)

Encoding of text pieces after pre-tokenization failed.

DecodeError(DecodeError)

Decoding token IDs into text failed.

Trait Implementations

impl Clone for TokenizerError

fn clone(&self) -> TokenizerError

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Error for TokenizerError

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0: use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0: replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl From<EncodeError> for TokenizerError

fn from(err: EncodeError) -> Self

Converts to this type from the input type.

impl From<NormalizeError> for TokenizerError

fn from(err: NormalizeError) -> Self

Converts to this type from the input type.