Struct tantivy_analysis_contrib::icu::ICUTransformTokenFilter

pub struct ICUTransformTokenFilter { /* private fields */ }

Available on crate feature icu only.

This TokenFilter allow to transform text into another, for example, to performe transliteration. See ICU documentation

use tantivy_analysis_contrib::icu::{Direction, ICUTransformTokenFilter};

let token_filter = ICUTransformTokenFilter::new(
    "Any-Latin; NFD; [:Nonspacing Mark:] Remove; Lower;  NFC".to_string(),
    None,
    Direction::Forward
)?;

Example

Here is an example of transform that converts greek letters into latin letters

use tantivy::tokenizer::{RawTokenizer, TextAnalyzer, Token};
use tantivy_analysis_contrib::icu::{Direction, ICUTransformTokenFilter};

let mut tmp = TextAnalyzer::builder(RawTokenizer::default())
   .filter(ICUTransformTokenFilter::new(
      "Greek-Latin".to_string(),
      None,
      Direction::Forward
   )?)
   .build();
let mut token_stream = tmp.token_stream("Αλφαβητικός Κατάλογος");

let token = token_stream.next().expect("A token should be present.");
assert_eq!(token.text, "Alphabētikós Katálogos".to_string());

assert_eq!(None, token_stream.next());

Implementations

impl ICUTransformTokenFilter

pub fn new( compound_id: String, rules: Option<String>, direction: Direction ) -> Result<Self, Error>

Construct a new transform filter.

# Parameters :

• compound_id : Compound transform
• rules : Custom transform rules
• direction : Direction

Trait Implementations

impl Clone for ICUTransformTokenFilter

fn clone(&self) -> ICUTransformTokenFilter

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ICUTransformTokenFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl TokenFilter for ICUTransformTokenFilter

type Tokenizer<T: Tokenizer> = ICUTransformFilterWrapper<T>

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

fn transform<T: Tokenizer>(self, token_stream: T) -> Self::Tokenizer<T>

Wraps a Tokenizer and returns a new one.