Struct tantivy_analysis_contrib::icu::ICUNormalizer2TokenFilter

pub struct ICUNormalizer2TokenFilter { /* private fields */ }

Available on crate feature icu only.

TokenFilter that converts text into a normal form. It supports all Google’s unicode normalization using Mode:

• NFC
• NFD
• NFKC
• NFKD
• NFKC casefold

See Wikipedia’s unicode normalization or Unicode documentation for more information.

Building an ICUNormalizer2TokenFilter is straightforward :

use tantivy_analysis_contrib::icu::ICUNormalizer2TokenFilter;
use tantivy_analysis_contrib::icu::Mode;

let normalizer = ICUNormalizer2TokenFilter::new(Mode::NFD)?;

Example

Here is an example showing which tokens are produce

use tantivy::tokenizer::{RawTokenizer, TextAnalyzer, Token};
use tantivy_analysis_contrib::icu::{ICUNormalizer2TokenFilter, Mode};

let mut tmp = TextAnalyzer::builder(RawTokenizer::default())
   .filter(ICUNormalizer2TokenFilter::new(Mode::NFKCCasefold)?)
   .build();
let mut token_stream = tmp.token_stream("Ruß");

let token = token_stream.next().expect("A token should be present.");

assert_eq!(token.text, "russ".to_string());

assert_eq!(None, token_stream.next());

Implementations

impl ICUNormalizer2TokenFilter

pub fn new(mode: Mode) -> Result<Self, Error>

Construct a new normalizer 2 token filter.

Parameters :

• mode : Normalization algorithm.

Trait Implementations

impl Clone for ICUNormalizer2TokenFilter

fn clone(&self) -> ICUNormalizer2TokenFilter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ICUNormalizer2TokenFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<Mode> for ICUNormalizer2TokenFilter

fn from(mode: Mode) -> Self

Converts to this type from the input type.

impl TokenFilter for ICUNormalizer2TokenFilter

type Tokenizer<T: Tokenizer> = ICUNormalizer2FilterWrapper<T>

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

fn transform<T: Tokenizer>(self, token_stream: T) -> Self::Tokenizer<T>

Wraps a Tokenizer and returns a new one.

impl Copy for ICUNormalizer2TokenFilter