Struct Tokenizer 

pub struct Tokenizer { /* private fields */ }

A configured tokenizer that normalizes and filters text into terms.

Implementations

impl Tokenizer

pub fn new(config: &TokenizerConfig) -> Self

Create a new tokenizer with the given configuration.

Loads the default English stopword list from stop-words crate unless config.custom_only is true. Merges extra_stopwords.

pub fn tokenize(&self, text: &str) -> Vec<String>

Tokenize text into a list of normalized terms.

1. Lowercase the entire input (ASCII only)
2. Split on ASCII whitespace
3. Strip non-alphanumeric characters from each token (keep only a-z, 0-9)
4. Remove empty tokens
5. Remove stopwords

Note: Input is treated as ASCII; non-ASCII characters are silently dropped.

pub fn is_stopword(&self, term: &str) -> bool

Return true if the given term is a stopword.

Trait Implementations

impl Clone for Tokenizer

fn clone(&self) -> Tokenizer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Tokenizer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.