Struct NormalizationRustStemmers 

pub struct NormalizationRustStemmers {
    pub anyway_above_confidence: f64,
    pub process_already_normalized: bool,
}

Will run stemming with the language tagged onto the token if an algorithm is available.

This uses the rust_stemmers crate under the hood.

This is recommended to be run after an AugmentationDetectLanguage has been used, it will not do anything if no language metadata is available!

If you need lowercase normalization, do that before this normalizer and set process_already_normalized to true. This is because some normalizers can’t handle SCREAMING CASE.

Tokens will be ignored if:

• They are known to not be an SegmentedTokenKind::AlphaNumeric
• They already have normalized_text set when process_already_normalized is false (default)

If the tokens normalization_language is already set to a Some value that one wil be used and the detected language ignored.

Fields

anyway_above_confidence: f64

Thereshold above which the flag about the lnguage detection flagging itself as reliable is ignored and the detected lnguage used for normalization anyway. Setting this can help with shorter texts.

1.0 which translates to never ignore the flag. 0.0 would mean to always ignore it.

Default is 0.4 as that is usually “good enough” for correct stemming.

process_already_normalized: bool

Wheter to process tokens that are already normalized. You want to enable this, if your pipeline does some generic preprocessing like lowercasing.

Default is false for backwards compatibility.

Implementations

impl NormalizationRustStemmers

pub fn new() -> Self

Create a new NormalizationRustStemmers instance with the default settings.

pub fn set_anyway_above_confidence(self, anyway_above_confidence: f64) -> Self

Adjust the value of anyway_above_confidence builder style.

pub fn set_process_already_normalized( self, process_already_normalized: bool, ) -> Self

Adjust the value of process_already_normalized builder style.

Trait Implementations

impl Augmenter for NormalizationRustStemmers

fn augment<'a>(&self, token: SegmentedToken<'a>) -> SegmentedToken<'a>

Apply augmentation function to the given token and return it.

impl Clone for NormalizationRustStemmers

fn clone(&self) -> NormalizationRustStemmers

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NormalizationRustStemmers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NormalizationRustStemmers

fn default() -> Self

Returns the “default value” for a type.