Normalizer

tokenizers::tokenizer

Trait Normalizer 

Source 
pub trait Normalizer {
    // Required method
    fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>;
}
Expand description

Takes care of pre-processing strings.

Required Methods§

Source

fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>

Implementors§