Module summa_core::components::tokenizers

source ·

Structs

  • DictTokenizer
    Tokenize the text by splitting on whitespaces and punctuation.
  • HtmlTokenizer
    Tokenize the text by splitting on whitespaces and punctuation.
  • MmdTokenizer
    Tokenize the text by splitting on whitespaces and punctuation.
  • Tokenizer
    Tokenize the text by splitting on whitespaces and punctuation.