Struct BpeTokenizer 

pub struct BpeTokenizer { /* private fields */ }

A Byte-Pair Encoding tokenizer built from a vocabulary and an ordered list of merge rules.

The merge list is applied greedily: at each step the adjacent symbol pair with the lowest rank (earliest in the list) is merged, until no ranked pair remains.

Byte-level mode

Enable with_byte_level for GPT-2 / RoBERTa / Llama style tokenizers. In this mode the input is pre-tokenized with a GPT-2 word-splitting pass and each piece’s raw bytes are mapped into the reversible byte-level unicode alphabet before BPE is applied, so encoding never fails on arbitrary UTF-8 and decode(encode(s)) == s.

Special tokens

Register atomic markers (e.g. <|endoftext|>, <s>) with with_special_tokens. They are matched verbatim (longest-match-wins) before pre-tokenization and never split.

Implementations

impl BpeTokenizer

pub fn from_vocab_merges( vocab: BTreeMap<String, TokenId>, merges: Vec<(String, String)>, ) -> Self

Builds a tokenizer from a vocabulary (token string → id) and an ordered list of merge rules. The index of a rule in merges is its rank, so earlier rules win ties.

Examples found in repository

examples/tokenize_text.rs (line 27)

fn main() {
    // --- BPE (GPT-2 style) ---
    let mut bpe_vocab = BTreeMap::new();
    for (i, t) in ["l", "o", "w", "lo", "low", "e", "r", "er", "<unk>"]
        .into_iter()
        .enumerate()
    {
        bpe_vocab.insert(t.to_string(), i as u32);
    }
    let bpe_merges = vec![
        ("l".to_string(), "o".to_string()),
        ("lo".to_string(), "w".to_string()),
        ("e".to_string(), "r".to_string()),
    ];
    let bpe = BpeTokenizer::from_vocab_merges(bpe_vocab, bpe_merges);

    let text = "low lower";
    let bpe_ids = bpe.encode(text).unwrap();
    println!("BPE   encode({text:?}) = {bpe_ids:?}");
    println!(
        "BPE   decode({bpe_ids:?}) = {:?}",
        bpe.decode(&bpe_ids).unwrap()
    );

    // --- WordPiece (BERT style) ---
    let mut wp_vocab = BTreeMap::new();
    for (i, t) in ["[UNK]", "un", "##aff", "##able", "play", "##ing"]
        .into_iter()
        .enumerate()
    {
        wp_vocab.insert(t.to_string(), i as u32);
    }
    let wp = WordPieceTokenizer::from_vocab(wp_vocab, "[UNK]").unwrap();

    let text2 = "unaffable playing";
    let wp_ids = wp.encode(text2).unwrap();
    println!("WP    encode({text2:?}) = {wp_ids:?}");
    println!(
        "WP    decode({wp_ids:?}) = {:?}",
        wp.decode(&wp_ids).unwrap()
    );
}

pub fn with_byte_level(self) -> Self

Enables GPT-2 byte-level pre-tokenization and byte-fallback encoding.

pub fn with_special_tokens(self, specials: BTreeMap<String, TokenId>) -> Self

Registers special tokens (token string → id) that are matched atomically before pre-tokenization and never split by BPE.

pub fn vocab_size(&self) -> usize

Number of tokens in the vocabulary (excluding special tokens that are not also present in the base vocabulary).

pub fn from_files( vocab_path: &str, merges_path: &str, ) -> Result<Self, TokenizerError>

Loads a GPT-2 style tokenizer from disk: vocab_path is one token per line (line number = id), merges_path is one A B pair per line (optionally prefixed by a single count header line).

Errors

Returns TokenizerError::Io on a read failure, or TokenizerError::MalformedFile if a line cannot be parsed.

Trait Implementations

impl Clone for BpeTokenizer

fn clone(&self) -> BpeTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for BpeTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Tokenizer for BpeTokenizer

fn decode(&self, ids: &[TokenId]) -> Result<String, TokenizerError>

Decode ids back into a string.

In byte-level mode this is exact: decode(encode(s)) == s. In plain mode encode discards whitespace, so decoding is lossy for multi-word input.

fn encode(&self, text: &str) -> Result<Vec<TokenId>, TokenizerError>

Encode text into a sequence of token ids.