Trait mako::tokenization::hf_tokenizers::tokenizer::Normalizer
source · [−]pub trait Normalizer {
fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>;
}Expand description
Takes care of pre-processing strings.