Enum TokenModel 

pub enum TokenModel {
    Gpt52,
    Gpt52Pro,
    Gpt51,
    Gpt51Mini,
    Gpt51Codex,
    Gpt5,
    Gpt5Mini,
    Gpt5Nano,
    O4Mini,
    O3,
    O3Mini,
    O1,
    O1Mini,
    O1Preview,
    Gpt4o,
    Gpt4oMini,
    Gpt4,
    Gpt35Turbo,
    Claude,
    Gemini,
    Llama,
    CodeLlama,
    Mistral,
    DeepSeek,
    Qwen,
    Cohere,
    Grok,
}

Supported LLM models for token counting

Models are grouped by their tokenizer encoding family. Use TokenModel::from_model_name to parse user-friendly model names like “gpt-5.2”, “o3”, “claude-sonnet”, etc.

Variants

Gpt52

GPT-5.2 - Latest flagship model (Dec 2025), uses o200k_base

Gpt52Pro

GPT-5.2 Pro - Enhanced GPT-5.2 variant, uses o200k_base

Gpt51

GPT-5.1 - Previous flagship (Nov 2025), uses o200k_base

Gpt51Mini

GPT-5.1 Mini - Smaller GPT-5.1 variant, uses o200k_base

Gpt51Codex

GPT-5.1 Codex - Code-specialized variant, uses o200k_base

Gpt5

GPT-5 - Original GPT-5 (Aug 2025), uses o200k_base

Gpt5Mini

GPT-5 Mini - Smaller GPT-5 variant, uses o200k_base

Gpt5Nano

GPT-5 Nano - Smallest GPT-5 variant, uses o200k_base

O4Mini

O4 Mini - Latest reasoning model, uses o200k_base

O3

O3 - Reasoning model, uses o200k_base

O3Mini

O3 Mini - Smaller O3 variant, uses o200k_base

O1

O1 - Original reasoning model, uses o200k_base

O1Mini

O1 Mini - Smaller O1 variant, uses o200k_base

O1Preview

O1 Preview - O1 preview version, uses o200k_base

Gpt4o

GPT-4o - Omni model, uses o200k_base encoding (most efficient)

Gpt4oMini

GPT-4o Mini - Smaller GPT-4o variant, uses o200k_base encoding

Gpt4

GPT-4/GPT-4 Turbo - uses cl100k_base encoding (legacy)

Gpt35Turbo

GPT-3.5-turbo - uses cl100k_base encoding (legacy)

Claude

Claude (all versions) - uses estimation based on ~3.5 chars/token

Gemini

Gemini (all versions including 3, 2.5, 1.5) - estimation ~3.8 chars/token

Llama

Llama 3/4 - estimation based on ~3.5 chars/token

CodeLlama

CodeLlama - more granular for code (~3.2 chars/token)

Mistral

Mistral (Large, Medium, Small, Codestral) - estimation ~3.5 chars/token

DeepSeek

DeepSeek (V3, R1, Coder) - estimation ~3.5 chars/token

Qwen

Qwen (Qwen3, Qwen2.5) - estimation ~3.5 chars/token

Cohere

Cohere (Command R+, Command R) - estimation ~3.6 chars/token

Grok

Grok (Grok 2, Grok 3) - estimation ~3.5 chars/token

Implementations

impl TokenModel

pub fn name(&self) -> &'static str

Get human-readable name

pub fn chars_per_token(&self) -> f32

Get average characters per token (for estimation fallback)

pub fn has_exact_tokenizer(&self) -> bool

Whether this model has an exact tokenizer available (via tiktoken)

pub fn uses_o200k(&self) -> bool

Whether this model uses the o200k_base encoding

pub fn uses_cl100k(&self) -> bool

Whether this model uses the cl100k_base encoding (legacy)

pub fn from_model_name(name: &str) -> Option<Self>

Parse a model name string into a TokenModel

Supports various formats:

• OpenAI: “gpt-5.2”, “gpt-5.2-pro”, “gpt-5.1”, “gpt-5”, “o3”, “o1”, “gpt-4o”, etc.
• Claude: “claude”, “claude-3”, “claude-4”, “claude-opus”, “claude-sonnet”, “claude-haiku”
• Gemini: “gemini”, “gemini-pro”, “gemini-flash”, “gemini-2.5”, “gemini-3”
• Llama: “llama”, “llama-3”, “llama-4”, “codellama”
• Others: “mistral”, “deepseek”, “qwen”, “cohere”, “grok”

Examples

use infiniloom_engine::tokenizer::TokenModel;

assert_eq!(TokenModel::from_model_name("gpt-5.2"), Some(TokenModel::Gpt52));
assert_eq!(TokenModel::from_model_name("o3"), Some(TokenModel::O3));
assert_eq!(TokenModel::from_model_name("claude-sonnet"), Some(TokenModel::Claude));
assert_eq!(TokenModel::from_model_name("unknown-model"), None);

pub fn all() -> &'static [Self]

Get all available models

pub fn vendor(&self) -> &'static str

Get the vendor/provider name for this model

Trait Implementations

impl Clone for TokenModel

fn clone(&self) -> TokenModel

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenModel

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenModel

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Hash for TokenModel

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl PartialEq for TokenModel

fn eq(&self, other: &TokenModel) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Serialize for TokenModel

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl Copy for TokenModel

impl Eq for TokenModel

impl StructuralPartialEq for TokenModel