Struct syntaxdot_tokenizers::BertTokenizer

pub struct BertTokenizer { /* fields omitted */ }

BERT word piece tokenizer.

This tokenizer splits CoNLL-X tokens into word pieces. For example, a sentence such as:

Veruntreute die AWO Spendengeld ?

Could be split (depending on the vocabulary) into the following word pieces:

Ver ##unt ##reute die A ##W ##O Spenden ##geld [UNK]

Then vocabulary index of each such piece is returned.

The unknown token (here [UNK]) can be specified while constructing a tokenizer.

Implementations

impl BertTokenizer

pub fn new(word_pieces: WordPieces, unknown_piece: impl Into<String>) -> Self

Construct a tokenizer from wordpieces and the unknown piece.

pub fn open<P>(
    model_path: P,
    unknown_piece: impl Into<String>
) -> Result<Self, TokenizerError> where
    P: AsRef<Path>, 

pub fn read<R>(
    buf_read: R,
    unknown_piece: impl Into<String>
) -> Result<BertTokenizer, TokenizerError> where
    R: BufRead, 

Trait Implementations

impl Tokenize for BertTokenizer

pub fn tokenize(&self, sentence: Sentence) -> SentenceWithPieces

Tokenize the tokens in a sentence into word pieces.