Enum NormalizerWrapper 

#[non_exhaustive]
pub enum NormalizerWrapper {
    BertNormalizer(BertNormalizer),
    Strip(Strip),
    StripAccents(StripAccents),
    Nfc(Nfc),
    Nfd(Nfd),
    Nfkc(Nfkc),
    Nfkd(Nfkd),
    Sequence(Sequence),
    Lowercase(Lowercase),
    Nmt(Nmt),
    Precompiled(Precompiled),
    Replace(Replace),
    Prepend(Prepend),
    ByteLevel(ByteLevel),
}

Any built-in normalizer. This is what tokenizer.json’s "normalizer" field (de)serializes to; the "type" key selects the variant.

Loading also accepts the legacy untagged forms (no "type" key) that Hugging Face tokenizers still reads: the variant is inferred from the fields, exactly as HF’s untagged fallback does. Saving always writes the tagged form. Unknown keys are ignored, as everywhere else.

Variants (Non-exhaustive)

Non-exhaustive enums could have additional variants added in future. Therefore, when matching against variants of non-exhaustive enums, an extra wildcard arm must be added to account for any future variants.

BertNormalizer(BertNormalizer)

BERT’s normalizer.

Strip(Strip)

Strip whitespace on the left and/or right.

StripAccents(StripAccents)

Remove combining marks.

Nfc(Nfc)

Unicode NFC.

Nfd(Nfd)

Unicode NFD.

Nfkc(Nfkc)

Unicode NFKC.

Nfkd(Nfkd)

Unicode NFKD.

Sequence(Sequence)

Apply several normalizers in order.

Lowercase(Lowercase)

Unicode lowercase.

Nmt(Nmt)

NMT-style control character cleanup.

Precompiled(Precompiled)

SentencePiece precompiled character map.

Replace(Replace)

Replace a pattern with a string.

Prepend(Prepend)

Prepend a string.

ByteLevel(ByteLevel)

Map each byte to its GPT-2 byte-level char.

Trait Implementations

impl Clone for NormalizerWrapper

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NormalizerWrapper

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for NormalizerWrapper

fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error>

Deserialize this value from the given Serde deserializer.

impl From<BertNormalizer> for NormalizerWrapper

fn from(n: BertNormalizer) -> Self

Converts to this type from the input type.

impl From<ByteLevel> for NormalizerWrapper

fn from(n: ByteLevel) -> Self

Converts to this type from the input type.

impl From<Lowercase> for NormalizerWrapper

fn from(n: Lowercase) -> Self

Converts to this type from the input type.

impl From<Nfc> for NormalizerWrapper

fn from(n: Nfc) -> Self

Converts to this type from the input type.

impl From<Nfd> for NormalizerWrapper

fn from(n: Nfd) -> Self

Converts to this type from the input type.

impl From<Nfkc> for NormalizerWrapper

fn from(n: Nfkc) -> Self

Converts to this type from the input type.

impl From<Nfkd> for NormalizerWrapper

fn from(n: Nfkd) -> Self

Converts to this type from the input type.

impl From<Nmt> for NormalizerWrapper

fn from(n: Nmt) -> Self

Converts to this type from the input type.

impl From<Precompiled> for NormalizerWrapper

fn from(n: Precompiled) -> Self

Converts to this type from the input type.

impl From<Prepend> for NormalizerWrapper

fn from(n: Prepend) -> Self

Converts to this type from the input type.

impl From<Replace> for NormalizerWrapper

fn from(n: Replace) -> Self

Converts to this type from the input type.

impl From<Sequence> for NormalizerWrapper

fn from(n: Sequence) -> Self

Converts to this type from the input type.

impl From<Strip> for NormalizerWrapper

fn from(n: Strip) -> Self

Converts to this type from the input type.

impl From<StripAccents> for NormalizerWrapper

fn from(n: StripAccents) -> Self

Converts to this type from the input type.

impl Normalizer for NormalizerWrapper

fn normalize(&self, normalized: &mut NormalizedString) -> Result<()>

Normalize normalized in place.

impl PartialEq for NormalizerWrapper

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl Serialize for NormalizerWrapper

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl StructuralPartialEq for NormalizerWrapper