Enum TokenizerType 

pub enum TokenizerType {
    Continuous {
        embed_dim: usize,
    },
    MuLaw {
        bits: u8,
    },
    Linear {
        bits: u8,
        min: f32,
        max: f32,
    },
    VectorQuantized {
        codebook_size: usize,
        embed_dim: usize,
    },
    MultiScale {
        embed_dim_per_level: usize,
        num_levels: usize,
    },
    Pyramid {
        embed_dim_per_level: usize,
        num_levels: usize,
    },
}

Configuration for tokenizer selection

Variants

Continuous

Direct continuous embedding (no discretization)

Fields

embed_dim: usize

MuLaw

μ-law companding (8-bit or 16-bit)

Fields

bits: u8

Linear

Linear uniform quantization

Fields

bits: u8

min: f32

max: f32

VectorQuantized

Vector quantized (VQ-VAE style)

Fields

codebook_size: usize

embed_dim: usize

MultiScale

Multi-scale hierarchical tokenization

Fields

embed_dim_per_level: usize

num_levels: usize

Pyramid

Pyramid tokenization with residual encoding

Fields

embed_dim_per_level: usize

num_levels: usize

Trait Implementations

impl Clone for TokenizerType

fn clone(&self) -> TokenizerType

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerType

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TokenizerType

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenizerType

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.