Enum TokenizerError 

pub enum TokenizerError {
    InvalidConfig(String),
    DimensionMismatch {
        expected: usize,
        got: usize,
        context: String,
    },
    EncodingError {
        operation: String,
        reason: String,
    },
    DecodingError {
        operation: String,
        reason: String,
    },
    CodebookNotInitialized {
        hint: String,
    },
    ValueOutOfRange {
        value: f32,
        min: f32,
        max: f32,
        context: String,
    },
    CoreError(CoreError),
    InternalError(String),
    InvalidInput {
        operation: String,
        reason: String,
    },
    SerializationError(String),
    IoError(Error),
    TrainingError {
        epoch: usize,
        reason: String,
    },
    NumericalError {
        operation: String,
        reason: String,
    },
    ResourceLimitExceeded {
        resource: String,
        details: String,
    },
    NotImplemented(String),
}

Errors that can occur in tokenizer operations

Variants

InvalidConfig(String)

Configuration parameter is invalid

This error includes detailed context about what parameter was invalid and why, to help users fix their configuration.

DimensionMismatch

Array dimensions don’t match expected values

Provides both the expected and actual dimensions to help diagnose shape mismatches in multi-dimensional arrays.

Fields

expected: usize

got: usize

context: String

EncodingError

Signal encoding operation failed

Wraps detailed information about why encoding failed, such as invalid input ranges, NaN values, or algorithmic failures.

Fields

operation: String

reason: String

DecodingError

Signal decoding operation failed

Includes context about which decoding step failed and why.

Fields

operation: String

reason: String

CodebookNotInitialized

Codebook has not been initialized before use

This typically happens when trying to use a VQ-VAE tokenizer before training or loading a pretrained codebook.

Fields

hint: String

ValueOutOfRange

Value is outside the valid range

Provides the offending value and the valid range for debugging.

Fields

value: f32

min: f32

max: f32

context: String

CoreError(CoreError)

Error from kizzasi-core crate

InternalError(String)

Internal error that should not happen under normal circumstances

If you encounter this error, it likely indicates a bug in the library. Please report it with the full error message and context.

InvalidInput

Invalid input data provided to a function

This error provides context about what was invalid in the input, such as empty arrays, NaN values, or incorrect types.

Fields

operation: String

reason: String

SerializationError(String)

Serialization or deserialization failed

IoError(Error)

File I/O operation failed

TrainingError

Training operation failed

Fields

epoch: usize

reason: String

NumericalError

Numerical computation resulted in invalid values (NaN, Inf)

Fields

operation: String

reason: String

ResourceLimitExceeded

Resource limit exceeded

Fields

resource: String

details: String

NotImplemented(String)

Feature not yet implemented

Implementations

impl TokenizerError

pub fn dim_mismatch( expected: usize, got: usize, context: impl Into<String>, ) -> Self

Create a dimension mismatch error with context

pub fn encoding(operation: impl Into<String>, reason: impl Into<String>) -> Self

Create an encoding error with context

pub fn decoding(operation: impl Into<String>, reason: impl Into<String>) -> Self

Create a decoding error with context

pub fn out_of_range( value: f32, min: f32, max: f32, context: impl Into<String>, ) -> Self

Create a value out of range error with context

pub fn invalid_input( operation: impl Into<String>, reason: impl Into<String>, ) -> Self

Create an invalid input error with context

pub fn numerical( operation: impl Into<String>, reason: impl Into<String>, ) -> Self

Create a numerical error with context

pub fn training(epoch: usize, reason: impl Into<String>) -> Self

Create a training error with context

Trait Implementations

impl Debug for TokenizerError

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for TokenizerError

fn fmt(&self, __formatter: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Error for TokenizerError

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0: use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0: replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl From<CoreError> for TokenizerError

fn from(source: CoreError) -> Self

Converts to this type from the input type.

impl From<Error> for TokenizerError

fn from(source: Error) -> Self

Converts to this type from the input type.