Module mako::tokenization[−][src]

Expand description

Tokenization module handles all tokenization and untokenization

Structs

Tokenizer

Functions

batch_tokenize_alphabet

Tokenize the string into a vector of each letter

batch_tokenize_bpe

Tokenizes strings using BPE tokenization

batch_tokenize_spaces

Tokenizes strings by splitting at whitespace

batch_tokenize_wordpiece

Tokenizes strings using WordPiece tokenization

batch_untokenize_alphabet

Untokenize alphabet tokens

batch_untokenize_bpe

Untokenize BPE tokens

batch_untokenize_spaces

Untokenize space seperated tokens

batch_untokenize_wordpiece

Untokenize wordpiece tokens

load_bpe_tokenizer

Loads the BPE tokenizer

load_wordpiece_tokenizer

Loads the wordpiece tokenizer

tokenize_alphabet

Tokenize the string into a vector of each letter

tokenize_bpe

Tokenizes string using BPE tokenization

tokenize_spaces

Tokenizes strings by splitting at whitespace

tokenize_wordpiece

Tokenizes string using BPE tokenization

untokenize_alphabet

Untokenize alphabet tokens

untokenize_bpe

Untokenize BPE tokens

untokenize_spaces

Untokenize space seperated tokens

untokenize_wordpiece

Untokenize wordpiece tokens