Function for_each_token 

pub fn for_each_token<F>(text: &str, f: F)
where
    F: FnMut(&str),

Tokenize and iterate over processed tokens (default English pipeline: word tokenizer, lowercase, English stopwords, English stemming).

Compatibility shim over crate::bm25_text::Bm25Pipeline: allocates one Vec per call (the old stack-buffered zero-alloc form is gone — hot paths should use the pipeline directly). For other languages and options see crate::bm25_text::Bm25Pipeline.