Enum MITokenizer 

#[non_exhaustive]
pub enum MITokenizer {
    HuggingFace(Box<Tokenizer>),
    Rwkv(RwkvTokenizer),
}

Unified tokenizer supporting multiple backends.

Most models use the HuggingFace tokenizers crate. RWKV-6 models ship their own vocabulary format and require a custom trie-based tokenizer, which is available behind the rwkv-tokenizer feature.

Example

use candle_mi::MITokenizer;

let tok = MITokenizer::from_hf_path("tokenizer.json")?;
let ids = tok.encode("fn main()")?;
let text = tok.decode(&ids)?;
assert!(!ids.is_empty());

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

HuggingFace(Box<Tokenizer>)

HuggingFace tokenizers backend.

Rwkv(RwkvTokenizer)

RWKV World tokenizer (trie-based greedy longest-match).

Implementations

impl MITokenizer

pub fn from_hf_path(path: impl AsRef<Path>) -> Result<Self>

Load a HuggingFace tokenizer from a tokenizer.json file.

Errors

Returns MIError::Tokenizer if the file cannot be loaded or parsed.

pub fn from_hf(tokenizer: Tokenizer) -> Self

Wrap an already-loaded HuggingFace tokenizer.

pub fn from_rwkv_path(path: impl AsRef<Path>) -> Result<Self>

Load an RWKV World tokenizer from a vocabulary file.

Errors

Returns MIError::Tokenizer if the file cannot be loaded or parsed.

pub fn encode(&self, text: &str) -> Result<Vec<u32>>

Encode text into token IDs, adding special tokens (e.g. BOS for Gemma).

Special tokens are added according to the tokenizer’s configured post-processor, matching the HuggingFace convention for inference.

Errors

Returns MIError::Tokenizer if encoding fails.

pub fn encode_raw(&self, text: &str) -> Result<Vec<u32>>

Encode text into token IDs without adding special tokens.

Useful for MI analyses that need raw tokenization without BOS/EOS.

Errors

Returns MIError::Tokenizer if encoding fails.

pub fn encode_with_offsets(&self, text: &str) -> Result<EncodingWithOffsets>

Encode text into token IDs with character offset mapping.

Returns an EncodingWithOffsets containing token IDs, token strings, and byte-offset ranges for each token. Special tokens are added (e.g., BOS for Gemma); special tokens receive a (0, 0) offset.

Errors

Returns MIError::Tokenizer if encoding fails or if the backend does not support offset mapping (RWKV).

pub fn encode_raw_with_offsets(&self, text: &str) -> Result<EncodingWithOffsets>

Encode text into token IDs with character offset mapping, without adding special tokens.

Errors

Returns MIError::Tokenizer if encoding fails or if the backend does not support offset mapping (RWKV).

pub fn decode(&self, ids: &[u32]) -> Result<String>

Decode token IDs back to a string.

Errors

Returns MIError::Tokenizer if decoding fails.

pub fn vocab_size(&self) -> usize

Get vocabulary size.

pub fn find_token_id(&self, word: &str) -> Result<u32>

Find the token ID for a word, trying " word" (with leading space) first, then bare "word".

This handles BPE tokenizers that represent word-initial tokens with a leading space (e.g., " cat" → single token).

Errors

Returns MIError::Tokenizer if the word cannot be resolved to a single token in either form.

pub fn decode_token(&self, token_id: u32) -> Result<String>

Decode a single token ID to its string representation.

Errors

Returns MIError::Tokenizer if decoding fails.

Trait Implementations

impl Debug for MITokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.