Type Alias TokenizerModel

Source

pub type TokenizerModel = TokenModel;

Expand description

Backward-compatible alias for TokenModel

§Important: No Conversion Needed

TokenizerModel and TokenModel are the same type - this is a type alias, not a separate type. Any function expecting TokenModel can directly accept TokenizerModel without conversion.

Before (incorrect, ~30 lines of duplication):

fn to_token_model(model: TokenizerModel) -> TokenModel {
    match model {
        TokenizerModel::Claude => TokenModel::Claude,
        // ... 26 more identical mappings
    }
}

After (correct):

// Direct usage - no conversion function needed
let tokenizer = Tokenizer::new();
tokenizer.count(text, model) // Works with TokenizerModel directly

This alias exists solely for backward compatibility with legacy CLI code that used the name TokenizerModel. All new code should prefer TokenModel.

Eliminated in Phase 1 refactoring (Item 2): Removed 193 lines of duplicate conversion functions and tests from pack.rs and diff.rs.

Aliased Type§

pub enum TokenizerModel {
Show 27 variants    Gpt52,
    Gpt52Pro,
    Gpt51,
    Gpt51Mini,
    Gpt51Codex,
    Gpt5,
    Gpt5Mini,
    Gpt5Nano,
    O4Mini,
    O3,
    O3Mini,
    O1,
    O1Mini,
    O1Preview,
    Gpt4o,
    Gpt4oMini,
    Gpt4,
    Gpt35Turbo,
    Claude,
    Gemini,
    Llama,
    CodeLlama,
    Mistral,
    DeepSeek,
    Qwen,
    Cohere,
    Grok,
}