Module tokenizers

summa_core::components

Module tokenizers 

Source

Structsยง

DictTokenizer
Tokenize the text by splitting on whitespaces and punctuation.
HtmlTokenizer
Tokenize the text by splitting on whitespaces and punctuation.
MmdTokenizer
Tokenize the text by splitting on whitespaces and punctuation.
Tokenizer
Tokenize the text by splitting on whitespaces and punctuation.