Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

Unicode-aware tokenizer with configurable pipeline: split → lowercase → clean → length filter → stopword filter → suffix strip.

Implementations

impl Tokenizer

pub fn new(config: TokenizerConfig) -> Self

Create a tokenizer with the given config.

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize document text — applies all filters including stopword removal.

pub fn tokenize_query(&self, text: &str) -> Vec<String>

Tokenize query text — stopwords are KEPT (critical for recall).

pub fn compute_term_frequencies(tokens: &[String]) -> HashMap<String, usize>

Compute raw term → count frequencies from a token slice.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer

fn default() -> Self

Returns the “default value” for a type.