Skip to main content

tokenize_without_stopwords

Function tokenize_without_stopwords 

Source
pub fn tokenize_without_stopwords(text: &str) -> Vec<String>
Expand description

Tokenizes text without filtering stopwords.

This is used for query text where stopwords are handled at a later stage.

ยงReturns

A vector of token strings.