Struct UnknownPretokenizer 

pub struct UnknownPretokenizer {
    pub pre: String,
    pub spm_style: bool,
}

A model declared a pre-tokenizer memra has no exact split for.

Before 2026-08-19 this was one eprintln! per process followed by a silent fall-through to the qwen35 split: the model loaded, generated fluent text, and every token id was wrong — the same fluent-and-invisible class as the GGUF chat-template mint trap. Wrong ids poison goldens, parity fixtures, acceptance counts and every quality number downstream, and nothing in the stack can detect it after the fact. So it is a hard load error now.

Fields

pre: String

The value that was rejected (tokenizer.ggml.pre, or default when an HF checkpoint’s pre-tokenizer regexes matched no known family).

spm_style: bool

True when the vocab model is SPM-style (tokenizer.ggml.model == "gemma4"); a pre/model disagreement is itself the fault, so it is worth naming.

Trait Implementations

impl Clone for UnknownPretokenizer

fn clone(&self) -> UnknownPretokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for UnknownPretokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for UnknownPretokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for UnknownPretokenizer

impl Error for UnknownPretokenizer

fn source(&self) -> Option<&(dyn Error + 'static)>

Returns the lower-level source of this error, if any.

fn description(&self) -> &str

👎Deprecated since 1.42.0:

use the Display impl or to_string()

fn cause(&self) -> Option<&dyn Error>

👎Deprecated since 1.33.0:

replaced by Error::source, which can support downcasting

fn provide<'a>(&'a self, request: &mut Request<'a>)

🔬This is a nightly-only experimental API. (error_generic_member_access)

Provides type-based access to context intended for error reports.

impl PartialEq for UnknownPretokenizer

fn eq(&self, other: &UnknownPretokenizer) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for UnknownPretokenizer