Struct WordPieceTokenizer 

pub struct WordPieceTokenizer { /* private fields */ }

A WordPiece tokenizer built from a vocabulary.

Words are split greedily into the longest matching sub-words; a word that cannot be split at all is replaced by the [UNK] token.

Input is pre-tokenized with a BERT-style basic tokenizer that splits on whitespace and punctuation and isolates CJK characters. Enable with_lowercase to match *-uncased models.

Implementations

impl WordPieceTokenizer

pub fn from_vocab( vocab: BTreeMap<String, TokenId>, unk_token: &str, ) -> Result<Self, TokenizerError>

Builds a tokenizer from a vocabulary (token string → id). unk_token names the fallback token (typically "[UNK]"); it must be present in vocab.

Errors

Returns TokenizerError::UnknownToken if unk_token is missing.

Examples found in repository

examples/tokenize_text.rs (line 45)

fn main() {
    // --- BPE (GPT-2 style) ---
    let mut bpe_vocab = BTreeMap::new();
    for (i, t) in ["l", "o", "w", "lo", "low", "e", "r", "er", "<unk>"]
        .into_iter()
        .enumerate()
    {
        bpe_vocab.insert(t.to_string(), i as u32);
    }
    let bpe_merges = vec![
        ("l".to_string(), "o".to_string()),
        ("lo".to_string(), "w".to_string()),
        ("e".to_string(), "r".to_string()),
    ];
    let bpe = BpeTokenizer::from_vocab_merges(bpe_vocab, bpe_merges);

    let text = "low lower";
    let bpe_ids = bpe.encode(text).unwrap();
    println!("BPE   encode({text:?}) = {bpe_ids:?}");
    println!(
        "BPE   decode({bpe_ids:?}) = {:?}",
        bpe.decode(&bpe_ids).unwrap()
    );

    // --- WordPiece (BERT style) ---
    let mut wp_vocab = BTreeMap::new();
    for (i, t) in ["[UNK]", "un", "##aff", "##able", "play", "##ing"]
        .into_iter()
        .enumerate()
    {
        wp_vocab.insert(t.to_string(), i as u32);
    }
    let wp = WordPieceTokenizer::from_vocab(wp_vocab, "[UNK]").unwrap();

    let text2 = "unaffable playing";
    let wp_ids = wp.encode(text2).unwrap();
    println!("WP    encode({text2:?}) = {wp_ids:?}");
    println!(
        "WP    decode({wp_ids:?}) = {:?}",
        wp.decode(&wp_ids).unwrap()
    );
}

pub fn with_lowercase(self) -> Self

Enables lowercasing during pre-tokenization, matching *-uncased BERT models.

pub fn vocab_size(&self) -> usize

Number of tokens in the vocabulary.

pub fn from_file( vocab_path: &str, unk_token: &str, ) -> Result<Self, TokenizerError>

Loads a BERT style vocabulary from disk: one token per line (line number = id).

Errors

Returns TokenizerError::Io on a read failure or TokenizerError::UnknownToken if unk_token is absent.

Trait Implementations

impl Clone for WordPieceTokenizer

fn clone(&self) -> WordPieceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WordPieceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Tokenizer for WordPieceTokenizer

fn decode(&self, ids: &[TokenId]) -> Result<String, TokenizerError>

Decode ids back into a string, mirroring BERT’s convert_tokens_to_string: continuation tokens (those prefixed with ##) are concatenated directly, while each new word is preceded by a space. This preserves inter-word spacing that encode discards.

Errors

Returns a TokenizerError if an id is missing from the inverse vocabulary.

fn encode(&self, text: &str) -> Result<Vec<TokenId>, TokenizerError>

Encode text into a sequence of token ids.