Struct BertTokenizer

pub struct BertTokenizer { /* private fields */ }

BERT word piece tokenizer.

This tokenizer splits CoNLL-X tokens into word pieces. For example, a sentence such as:

Veruntreute die AWO Spendengeld ?

Could be split (depending on the vocabulary) into the following word pieces:

Ver ##unt ##reute die A ##W ##O Spenden ##geld [UNK]

Then vocabulary index of each such piece is returned.

The unknown token (here [UNK]) can be specified while constructing a tokenizer.

Implementations

impl BertTokenizer

pub fn new(word_pieces: WordPieces, unknown_piece: impl Into<String>) -> Self

Construct a tokenizer from wordpieces and the unknown piece.

pub fn open<P>( model_path: P, unknown_piece: impl Into<String>, ) -> Result<Self, TokenizerError>

where P: AsRef<Path>,

pub fn read<R>( buf_read: R, unknown_piece: impl Into<String>, ) -> Result<BertTokenizer, TokenizerError>

where R: BufRead,

Trait Implementations

impl Tokenize for BertTokenizer

fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces

Tokenize the tokens in a sentence into word pieces.