Struct ShingleFilter 

pub struct ShingleFilter {
    pub min_size: usize,
    pub max_size: usize,
    pub output_unigrams: bool,
    pub separator: String,
    pub filler_token: String,
}

Produces word-level n-grams (shingles) from the token stream.

For example, with min_size=2, max_size=2, tokens [the, quick, brown] produce shingles [the quick, quick brown] (and optionally the original unigrams).

Matches ES shingle token filter.

See [[analyzers#Token Filters]].

Fields

min_size: usize

max_size: usize

output_unigrams: bool

separator: String

filler_token: String

Implementations

impl ShingleFilter

pub fn new(min_size: usize, max_size: usize, output_unigrams: bool) -> Self

Trait Implementations

impl TokenFilter for ShingleFilter

fn apply(&self, tokens: &mut Vec<Token>)

Apply the filter to tokens, modifying in place.