Struct WordPieceTokenizer 

pub struct WordPieceTokenizer { /* private fields */ }

A WordPiece tokenizer (BERT-style).

WordPiece is a subword tokenization algorithm that greedily matches the longest prefix of a word from the vocabulary, using a continuation prefix (typically "##") for non-initial subwords.

Example

use scirs2_text::tokenizer::WordPieceTokenizer;
use std::collections::HashMap;

let mut vocab = HashMap::new();
vocab.insert("[UNK]".to_string(), 0);
vocab.insert("hello".to_string(), 1);
vocab.insert("world".to_string(), 2);
vocab.insert("hel".to_string(), 3);
vocab.insert("##lo".to_string(), 4);

let tokenizer = WordPieceTokenizer::new(vocab);
let tokens = tokenizer.tokenize("hello world");
assert!(tokens.contains(&"hello".to_string()) || tokens.contains(&"hel".to_string()));

Implementations

impl WordPieceTokenizer

pub fn new(vocab: HashMap<String, u32>) -> Self

Create a new WordPiece tokenizer from a vocabulary map.

The vocabulary must contain at least [UNK]. The continuation prefix defaults to "##" and max word length defaults to 200.

pub fn with_max_word_len(self, max_len: usize) -> Self

Set the maximum word length.

pub fn with_unk_token(self, unk: &str) -> Self

Set the unknown token string.

pub fn with_continuing_prefix(self, prefix: &str) -> Self

Set the continuation prefix (default "##").

pub fn from_vocab_file(path: &Path) -> Result<Self>

Load a WordPiece vocabulary from a text file (one token per line).

Token IDs are assigned sequentially starting from 0.

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text into a list of token strings (subword pieces).

Each word is greedily segmented into the longest matching vocab entries. Non-initial pieces are prefixed with ##.

Trait Implementations

impl Clone for WordPieceTokenizer

fn clone(&self) -> WordPieceTokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for WordPieceTokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TransformerTokenizer for WordPieceTokenizer

fn encode(&self, text: &str) -> Vec<u32>

Encode text into a sequence of token IDs.

fn decode(&self, ids: &[u32]) -> String

Decode a sequence of token IDs back into text.

fn vocab_size(&self) -> usize

Return the vocabulary size (number of distinct token IDs).