Module token_filter 

Modules

japanese_base_form

This module defines various token filters and provides functionality to load them.

japanese_compound_word

japanese_kana

japanese_katakana_stem

japanese_keep_tags

japanese_number

japanese_reading_form

japanese_stop_tags

keep_words

korean_keep_tags

korean_reading_form

korean_stop_tags

length

lowercase

mapping

remove_diacritical_mark

stop_words

uppercase

Structs

BoxTokenFilter

A BoxTokenFilter is a wrapper around a boxed trait object that implements the TokenFilter trait. This allows for dynamic dispatch of different TokenFilter implementations at runtime. The BoxTokenFilter ensures that the contained TokenFilter is thread-safe (Send and Sync) and has a static lifetime.

TokenFilterLoader

Traits

TokenFilter

A trait for token filters that can be applied to a vector of tokens.

TokenFilterClone

A trait for cloning token filters.