Struct NormalizationRustStemmers

pub struct NormalizationRustStemmers {
    pub anyway_above_confidence: f64,
}

Will run stemming with the language tagged onto the token if an algorithm is available.

This uses the rust_stemmers crate under the hood.

This is recommended to be run after an AugmentationDetectLanguage has been used, it will not do anything if no language metadata is available!

Tokens will be ignored if:

• They are known to not be an SegmentedTokenKind::AlphaNumeric
• They already have normalized_text set. Apply things like lowercasing after this.

Fields

anyway_above_confidence: f64

Thereshold above which the flag about the lnguage detection flagging itself as reliable is ignored and the detected lnguage used for normalization anyway. Setting this can help with shorter texts.

1.0 which translates to never ignore the flag. 0.0 would mean to always ignore it.

Default is 0.4 as that is usually “good enough” for correct stemming.

Implementations

impl NormalizationRustStemmers

pub fn new() -> Self

Create a new NormalizationRustStemmers instance with the default settings.

pub fn set_anyway_above_confidence(self, anyway_above_confidence: f64) -> Self

Adjust the value of anyway_above_confidence builder style.

Trait Implementations

impl Augmenter for NormalizationRustStemmers

fn augment<'a>(&self, token: SegmentedToken<'a>) -> SegmentedToken<'a>

Apply augmentation function to the given token and return it.

impl Clone for NormalizationRustStemmers

fn clone(&self) -> NormalizationRustStemmers

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NormalizationRustStemmers

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for NormalizationRustStemmers

fn default() -> Self

Returns the “default value” for a type.