Struct TextAnalyzer

pub struct TextAnalyzer { /* private fields */ }

TextAnalyzer tokenizes an input text into tokens and modifies the resulting TokenStream.

It simply wraps a Tokenizer and a list of TokenFilter that are applied sequentially.

Implementations

impl TextAnalyzer

pub fn new<T: Tokenizer>( tokenizer: T, token_filters: Vec<BoxTokenFilter>, ) -> TextAnalyzer

Creates a new TextAnalyzer given a tokenizer and a vector of BoxTokenFilter.

When creating a TextAnalyzer from a Tokenizer alone, prefer using TextAnalyzer::from(tokenizer).

pub fn filter<F: Into<BoxTokenFilter>>(self, token_filter: F) -> Self

Appends a token filter to the current tokenizer.

The method consumes the current TokenStream and returns a new one.

Example

use tantivy::tokenizer::*;

let en_stem = TextAnalyzer::from(SimpleTokenizer)
    .filter(RemoveLongFilter::limit(40))
    .filter(LowerCaser)
    .filter(Stemmer::default());

pub fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

Creates a token stream for a given str.

Trait Implementations

impl Clone for TextAnalyzer

fn clone(&self) -> Self

Returns a duplicate of the value.

const fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for TextAnalyzer

fn default() -> TextAnalyzer

Returns the “default value” for a type.

impl<T: Tokenizer> From<T> for TextAnalyzer

fn from(tokenizer: T) -> Self

Converts to this type from the input type.