Struct ScrubConfig 

pub struct ScrubConfig {
    pub normalize_newlines: bool,
    pub remove_zero_width: bool,
    pub remove_bidi_controls: bool,
    pub collapse_whitespace: bool,
    pub normalization: ScrubNormalization,
    pub case: ScrubCase,
    pub strip_diacritics: bool,
}

Policy/config for constructing normalized keys / comparison forms.

The intent is to make the pipeline explicit: most real bugs here are from implicitly normalizing and accidentally destroying semantics (ZWJ/ZWNJ, bidi marks, punctuation, newlines).

Fields

normalize_newlines: bool

Normalize newlines (\r\n/\r → \n) before any other whitespace policy.

remove_zero_width: bool

Remove common zero-width characters (ZWSP/ZWNJ/ZWJ/WORD JOINER/BOM).

remove_bidi_controls: bool

Remove Unicode bidirectional control characters (Trojan Source-style).

collapse_whitespace: bool

Collapse all Unicode whitespace to single ASCII spaces (and trim).

normalization: ScrubNormalization

Which normalization form to apply before case/diacritics.

case: ScrubCase

Case handling strategy.

strip_diacritics: bool

Strip combining marks (diacritics) after normalization + case mapping.

Implementations

impl ScrubConfig

pub fn search_key() -> Self

See search_key() (fallback when casefold is disabled).

pub fn search_key_strict_invisibles() -> Self

Like search_key(), but also removes common zero-width characters (ZWSP/ZWNJ/ZWJ/WJ/BOM).

This is a deliberate trade-off:

• Pro: avoids “ghost mismatches” in mostly-Latin corpora where ZWJ/ZWNJ are usually artifacts (copy/paste, rich text) rather than orthographic intent.
• Con: ZWJ/ZWNJ are semantically meaningful in multiple scripts (and for emoji ZWJ sequences). Stripping can create false positives/negatives depending on the task.

Trait Implementations

impl Clone for ScrubConfig

fn clone(&self) -> ScrubConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ScrubConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ScrubConfig

fn default() -> Self

Returns the “default value” for a type.