Struct MiTokenizer 

pub struct MiTokenizer {
    pub vocab_size: usize,
    /* private fields */
}

Single-Pass Mutual Information Tokenizer (BA-37).

Instead of BPE’s iterative pair counting (O(merges × corpus_len)), computes mutual information for all character pairs in ONE pass: MI(a,b) = ln(P(ab) / (P(a) × P(b))) Pairs with MI > ln(φ) ≈ 0.481 co-occur φ× more than chance — merge them.

Total cost: O(corpus_len) for counting + O(V²) for MI + O(corpus_len) per merge round. Typically 2-3 rounds on progressively shorter corpora. ~500x faster than BPE.

The golden ratio appears as the significance threshold: ln(φ) separates coherent pairs (signal) from independent pairs (noise).

Fields

vocab_size: usize

Implementations

impl MiTokenizer

pub fn train(text: &str, target_vocab: usize) -> Self

Train the MI tokenizer from a text corpus. target_vocab: desired vocabulary size. Merges stop when reached.

pub fn encode(&self, text: &str) -> Vec<usize>

pub fn decode(&self, tokens: &[usize]) -> String

Trait Implementations

impl Clone for MiTokenizer

fn clone(&self) -> MiTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.