tokenizers::normalizers

Module utils

Source

Structsยง

  • Lowercase
    Lowercases the input
  • Sequence
    Allows concatenating multiple other Normalizer as a Sequence. All the normalizers run in sequence in the given order against the same NormalizedString.