Struct Fold 

pub struct Fold;

The default marker removal and Unicode comparison form.

The fixed control tokens [CLS], [SEP], [PAD], [UNK], [MASK], <s>, </s>, <pad>, <unk>, <mask>, <cls>, <sep>, <|endoftext|>, and <|startoftext|> produce an empty comparison form.

One leading ## and one trailing </w> are removed. Every ▁, Ġ, and Ċ is dropped. Ġ and Ċ are also Maltese capital letters, so literal Maltese text containing either character loses it under this comparison form. Use a custom Normalizer when those letters carry text.

Each remaining character receives recursive compatibility decomposition from the Unicode Character Database 17.0.0 data. The decomposition is canonically ordered within that character, then mapped with the simple lowercase field. Final sigma ς maps to sigma σ. Characters are folded independently, so marks are not reordered across input characters.

Trait Implementations

impl Clone for Fold

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for Fold

impl Debug for Fold

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Fold

fn default() -> Self

Returns the “default value” for a type.

impl Eq for Fold

impl Hash for Fold

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Normalizer for Fold

fn normalize(&self, token: &str, out: &mut String)

Appends the comparison form of token to out.

impl Ord for Fold

fn cmp(&self, other: &Self) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

fn clamp_to<R>(self, range: R) -> Self

where Self: Sized, R: ClampBounds<Self>,

🔬This is a nightly-only experimental API. (clamp_to)

Restrict a value to a certain range.

impl PartialEq for Fold

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl PartialOrd for Fold

fn partial_cmp(&self, other: &Self) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl StructuralPartialEq for Fold