Module tokenizer

Module tokenizer 

Source

Structs§

REGEX_ARRAY
Tokenizer13a
Same implementation with huggingface/sacrebleu
TokenizerRegex
Same implementation with huggingface/sacrebleu

Traits§

Tokenizer
tokenize function is used to tokenize the strings