Enum TokenizerVersion 

pub enum TokenizerVersion {
    V3,
    V7,
    V11,
    V13,
}

Enumeration of supported tokenizer versions.

Different versions may have different vocabulary sizes, special tokens, and processing capabilities. This enum provides a type-safe way to handle version-specific behavior.

Supported Versions

• V3 - Early version with basic functionality
• V7 - Version with enhanced special tokens and audio support
• V11 - Updated version with additional features
• V13 - Latest version with full multimodal capabilities

Variants

V3

V7

V11

V13

Implementations

impl TokenizerVersion

pub fn from_string(s: &str) -> Option<Self>

Parses a version string into a TokenizerVersion.

Arguments

• s - Version string (e.g., “v7”, “v11”)

Returns

The corresponding TokenizerVersion if recognized, None otherwise.

Examples

use tekken::config::TokenizerVersion;

assert_eq!(TokenizerVersion::from_string("v7"), Some(TokenizerVersion::V7));
assert_eq!(TokenizerVersion::from_string("invalid"), None);

pub const fn as_str(&self) -> &'static str

Returns the string representation of the version.

Returns

The version string (e.g., “v7”, “v11”).

Examples

use tekken::config::TokenizerVersion;

assert_eq!(TokenizerVersion::V7.as_str(), "v7");
assert_eq!(TokenizerVersion::V13.as_str(), "v13");

Trait Implementations

impl Clone for TokenizerVersion

fn clone(&self) -> TokenizerVersion

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenizerVersion

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for TokenizerVersion

impl PartialEq for TokenizerVersion

fn eq(&self, other: &TokenizerVersion) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenizerVersion