Struct tantivy::tokenizer::TextAnalyzer

pub struct TextAnalyzer { /* fields omitted */ }

TextAnalyzer tokenizes an input text into tokens and modifies the resulting TokenStream.

It simply wraps a Tokenizer and a list of TokenFilter that are applied sequentially.

Methods

impl TextAnalyzer

pub fn new<T: Tokenizer>(
    tokenizer: T,
    token_filters: Vec<BoxTokenFilter>
) -> TextAnalyzer

Creates a new TextAnalyzer given a tokenizer and a vector of BoxTokenFilter.

When creating a TextAnalyzer from a Tokenizer alone, prefer using TextAnalyzer::from(tokenizer).

pub fn filter<F: Into<BoxTokenFilter>>(self, token_filter: F) -> Self

Appends a token filter to the current tokenizer.

The method consumes the current TokenStream and returns a new one.

Example

use tantivy::tokenizer::*;

let en_stem = TextAnalyzer::from(SimpleTokenizer)
    .filter(RemoveLongFilter::limit(40))
    .filter(LowerCaser)
    .filter(Stemmer::default());

pub fn token_stream_texts<'a>(&self, texts: &'a [&'a str]) -> BoxTokenStream<'a>

Tokenize an array&str

The resulting BoxTokenStream is equivalent to what would be obtained if the &str were one concatenated &str, with an artificial position gap of 2 between the different fields to prevent accidental PhraseQuery to match accross two terms.

pub fn token_stream<'a>(&self, text: &'a str) -> BoxTokenStream<'a>

Creates a token stream for a given str.

Trait Implementations

impl Clone for TextAnalyzer

fn clone(&self) -> Self

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<T: Tokenizer> From<T> for TextAnalyzer

fn from(tokenizer: T) -> Self

Performs the conversion.