Enum TokenizerError 

#[non_exhaustive]
pub enum TokenizerError {
    UnknownToken(String),
    InvalidVocab(String),
    EncodeFailed(String),
    DecodeFailed(String),
    InvalidJson(String),
    HfFormat(String),
    IncompleteUtf8,
    TemplateRender(String),
    Io(String),
}

All errors that can occur during tokenization operations.

This enum is marked #[non_exhaustive] so that new variants can be added in future minor releases without a breaking semver change. Consumers must always include a catch-all arm when matching on TokenizerError.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

UnknownToken(String)

A token string was not found in the vocabulary.

InvalidVocab(String)

The vocabulary data is malformed or inconsistent.

EncodeFailed(String)

Encoding of input text failed.

DecodeFailed(String)

Decoding of token IDs failed.

InvalidJson(String)

JSON deserialization failed.

HfFormat(String)

A HuggingFace tokenizer.json file could not be parsed or interpreted.

Includes missing required fields (model, vocab, merges), unsupported BPE types, and malformed merge entries.

IncompleteUtf8

A streaming decoder received token IDs that together do not form a complete UTF-8 sequence and further bytes are required to finish.

This variant is primarily returned by crate::streaming::StreamingDecoder::finish when the stream ends mid-character.

TemplateRender(String)

Rendering a chat-template failed (missing variable, bad syntax, …).

Io(String)

An underlying I/O operation (file read, etc.) failed.

We wrap the io::Error as a String so that TokenizerError can continue to derive Clone, PartialEq, Eq — std::io::Error itself does not implement those traits.

Trait Implementations

impl Clone for TokenizerError

fn clone(&self) -> TokenizerError

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerError

fn fmt(&self, __formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerError

impl Error for TokenizerError

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0:

use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0:

replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl From<Error> for TokenizerError

fn from(err: Error) -> Self

Converts to this type from the input type.

impl PartialEq for TokenizerError

fn eq(&self, other: &TokenizerError) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for TokenizerError