Enum PreSplit 

pub enum PreSplit {
    Qwen35,
    DeepseekV3,
    Spm,
    UnknownFallbackQwen35,
}

The pre-tokenizer split a loaded Tokenizer runs. Constructed only through PreSplit::resolve, so “we do not know how to split for this model” is not a state a live tokenizer can be in unless the operator asked for it via the env opt-out.

Variants

Qwen35

unicode::split_qwen35 — serves both qwen35 and qwen2.

DeepseekV3

unicode::split_deepseek_v3 — DeepSeek-V3 and the Step-3.5/3.7-Flash family.

Spm

gemma4 SPM-style BPE: bpe_tokenize splits whole lines itself and the pre id is never consulted. Requires the gemma4 vocab model, not just the pre string.

UnknownFallbackQwen35

MEMRA_ALLOW_UNKNOWN_PRETOKENIZER=1 was set for an unrecognized pre. Runs the qwen35 split; token ids are NOT exact and every downstream measurement is invalid.

Implementations

impl PreSplit

pub fn resolve(pre: &str, spm_style: bool) -> Result<Self, UnknownPretokenizer>

Resolve a tokenizer.ggml.pre id against the implemented splits. spm_style is tokenizer.ggml.model == "gemma4".

Trait Implementations

impl Clone for PreSplit

fn clone(&self) -> PreSplit

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for PreSplit

impl Debug for PreSplit

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for PreSplit

impl PartialEq for PreSplit

fn eq(&self, other: &PreSplit) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for PreSplit