Enum TokenizerError 

pub enum TokenizerError {
    Io(Error),
    Json(Error),
    Base64(DecodeError),
    Tokenizers(String),
    Audio(String),
    InvalidConfig(String),
    TokenNotFound(String),
    SpecialTokenPolicy(String),
    UnsupportedFormat(String),
}

Comprehensive error type for tokenizer operations.

This enum covers all possible error conditions that can occur during tokenizer initialization, text/audio processing, and other operations. It uses the thiserror crate to provide detailed error messages and automatic conversion from underlying error types.

Error Categories

• I/O Errors: File reading/writing operations
• Parsing Errors: JSON deserialization and Base64 decoding
• Processing Errors: Tokenization and audio processing failures
• Configuration Errors: Invalid parameters or missing tokens
• Policy Errors: Special token handling violations

Variants

Io(Error)

I/O operation failed (file reading, writing, etc.).

Json(Error)

JSON parsing or serialization failed.

Base64(DecodeError)

Base64 decoding failed.

Tokenizers(String)

Error in the underlying tokenization engine.

Audio(String)

Audio processing operation failed.

InvalidConfig(String)

Configuration parameters are invalid or inconsistent.

TokenNotFound(String)

Required token (usually special token) was not found in vocabulary.

SpecialTokenPolicy(String)

Operation violated the specified special token policy.

UnsupportedFormat(String)

File format or data format is not supported.

Trait Implementations

impl Debug for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerError

fn fmt(&self, __formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Error for TokenizerError

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0: use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0: replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl From<DecodeError> for TokenizerError

fn from(source: DecodeError) -> Self

Converts to this type from the input type.

impl From<Error> for TokenizerError

fn from(source: Error) -> Self

Converts to this type from the input type.

impl From<Error> for TokenizerError

fn from(source: Error) -> Self

Converts to this type from the input type.