Struct NGramTokenizer 

pub struct NGramTokenizer {
    pub min_gram: usize,
    pub max_gram: usize,
    pub token_chars: Vec<TokenChar>,
}

Produces n-grams of specified sizes from the input text.

When token_chars is empty, treats the entire input as one token and generates n-grams from it. When token_chars is non-empty, splits on characters NOT in the set, then generates n-grams from each word.

Matches ES ngram tokenizer.

See [[analyzers#Tokenizer]].

Fields

min_gram: usize

max_gram: usize

token_chars: Vec<TokenChar>

Implementations

impl NGramTokenizer

pub fn new( min_gram: usize, max_gram: usize, token_chars: Vec<TokenChar>, ) -> Self

Trait Implementations

impl Tokenizer for NGramTokenizer

fn tokenize(&self, text: &str, output: &mut Vec<Token>)

Tokenize text, appending tokens to output.