Normalizer

charabia::normalizer

Trait Normalizer 

Source 
pub trait Normalizer: Sync + Send {
    // Required methods
    fn normalize<'o>(
        &self,
        token: Token<'o>,
        options: &NormalizerOption<'_>,
    ) -> Token<'o>;
    fn should_normalize(&self, token: &Token<'_>) -> bool;
}
Expand description

Trait defining a normalizer.

Required Methods§

Source

fn normalize<'o>( &self, token: Token<'o>, options: &NormalizerOption<'_>, ) -> Token<'o>

Normalize the provided Token. Options can be set using the provided NormalizerOption.

Source

fn should_normalize(&self, token: &Token<'_>) -> bool

Return true if the normalizer can process Token of a specific [Script] and [Language].

Some normalizer are specialized for a Script or/and a Language and shouldn’t be called on every Tokens.

Implementors§