Struct WordPieceTokenizer 

pub struct WordPieceTokenizer { /* private fields */ }

WordPiece tokenizer (used by BERT).

WordPiece is similar to BPE but uses a different scoring criterion: it maximizes the likelihood of the training data rather than frequency. Subwords (except the first) are prefixed with “##”.

Algorithm

1. Initialize vocabulary with all characters
2. Score pairs by: freq(ab) / (freq(a) * freq(b))
3. Merge pair with highest score
4. Repeat until vocabulary size reached

Examples

use aprender::text::tokenize::WordPieceTokenizer;

let corpus = vec!["playing", "played", "player", "plays"];
let tokenizer = WordPieceTokenizer::train(&corpus, 50).expect("train");

let tokens = tokenizer.encode("playing").expect("encode");
assert!(!tokens.is_empty());

References

• Wu et al. (2016): Google’s Neural Machine Translation System
• Devlin et al. (2019): BERT: Pre-training of Deep Bidirectional Transformers

Implementations

impl WordPieceTokenizer

pub fn train(corpus: &[&str], vocab_size: usize) -> Result<Self, AprenderError>

Train a WordPiece tokenizer on the given corpus.

Arguments

• corpus - Slice of text documents to train on
• vocab_size - Target vocabulary size

Examples

use aprender::text::tokenize::WordPieceTokenizer;

let corpus = vec!["unbelievable", "believable", "believe"];
let tokenizer = WordPieceTokenizer::train(&corpus, 100).expect("train");

pub fn from_vocab(vocab: HashMap<String, u32>) -> Self

Create from pre-built vocabulary.

pub fn encode(&self, text: &str) -> Result<Vec<u32>, AprenderError>

Encode text to token IDs using greedy longest-match-first.

pub fn decode(&self, ids: &[u32]) -> Result<String, AprenderError>

Decode token IDs back to text.

pub fn vocab_size(&self) -> usize

Get vocabulary size.

pub fn vocab(&self) -> &HashMap<String, u32>

Get reference to vocabulary.

Trait Implementations

impl Clone for WordPieceTokenizer

fn clone(&self) -> WordPieceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WordPieceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Tokenizer for WordPieceTokenizer

fn tokenize(&self, text: &str) -> Result<Vec<String>, AprenderError>

Tokenize the input text into a vector of string tokens.