Struct LanguageAwareTokenizer 

pub struct LanguageAwareTokenizer<F>
where
    F: Fn(&str) -> Language + Clone + Send + Sync + 'static,
{ /* private fields */ }

Language-aware tokenizer that can be configured per-field

This allows selecting the stemmer language based on document metadata, such as a “language” field in the document.

Implementations

impl<F> LanguageAwareTokenizer<F>

where F: Fn(&str) -> Language + Clone + Send + Sync + 'static,

pub fn new(language_selector: F) -> Self

Create a new language-aware tokenizer with a custom language selector

The selector function receives a language hint (e.g., from a document field) and returns the appropriate Language to use for stemming.

Example

let tokenizer = LanguageAwareTokenizer::new(|hint| {
    match hint {
        "en" | "english" => Language::English,
        "de" | "german" => Language::German,
        "ru" | "russian" => Language::Russian,
        _ => Language::English,
    }
});

pub fn tokenize_with_hint(&self, text: &str, language_hint: &str) -> Vec<Token>

Tokenize text with a language hint

The hint is passed to the language selector to determine which stemmer to use.

Trait Implementations

impl<F> Clone for LanguageAwareTokenizer<F>

where F: Fn(&str) -> Language + Clone + Send + Sync + 'static + Clone,

fn clone(&self) -> LanguageAwareTokenizer<F>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<F> Tokenizer for LanguageAwareTokenizer<F>

where F: Fn(&str) -> Language + Clone + Send + Sync + 'static,

fn tokenize(&self, text: &str) -> Vec<Token>

Tokenize the input text into a vector of tokens