Struct BpeTokenizer 

pub struct BpeTokenizer {
    pub vocab_size: usize,
    /* private fields */
}

BPE (Byte Pair Encoding) tokenizer. Subword tokens learned from corpus.

Training: iteratively merge the most frequent adjacent pair until target vocab size is reached. Each merge creates a new token from two existing ones.

Encoding: greedily apply merges in learned order (longest match first). Decoding: expand each token to its character sequence.

Fields

vocab_size: usize

Vocabulary size (base chars + merges).

Implementations

impl BpeTokenizer

pub fn train(text: &str, target_vocab: usize) -> Self

Train a BPE tokenizer from a text corpus.

target_vocab: desired vocabulary size (base characters + merge tokens). Typical values: 512, 1024, 2048. Larger = more semantic content per token.

pub fn encode(&self, text: &str) -> Vec<usize>

Encode text into token indices using learned BPE merges.

pub fn decode(&self, tokens: &[usize]) -> String

Decode token indices back to text.

Trait Implementations

impl Clone for BpeTokenizer

fn clone(&self) -> BpeTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.