Struct TokenizerPolicy 

pub struct TokenizerPolicy {
    pub fold_latin_diacritics: bool,
    pub fold_russian_yo: bool,
}

Controls search-specific folding without changing the scanner or its allocation behavior.

The default preserves the tokenizer’s existing index contract: Latin diacritics are folded for recall and Russian ё is treated as е. Use TokenizerPolicy::unicode when callers need Unicode lowercase and normalization without either language/search-specific equivalence.

Fields

fold_latin_diacritics: bool

Fold Latin precomposed diacritics to their ASCII base.

fold_russian_yo: bool

Treat Russian small letter ё as е after lowercase.

Implementations

impl TokenizerPolicy

pub const fn unicode() -> Self

A language-neutral policy: Unicode lowercase and normalization only.

pub const fn search() -> Self

The search policy used by crate::tokenizer::Tokenizer::new and existing indexes.

Trait Implementations

impl Clone for TokenizerPolicy

fn clone(&self) -> TokenizerPolicy

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for TokenizerPolicy

impl Debug for TokenizerPolicy

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TokenizerPolicy

fn default() -> Self

Returns the “default value” for a type.

impl Eq for TokenizerPolicy

impl PartialEq for TokenizerPolicy

fn eq(&self, other: &TokenizerPolicy) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for TokenizerPolicy