pub fn tokenize(text: &str) -> impl Iterator<Item = String>Expand description
Split text into index/query tokens: Unicode-alphanumeric runs, lowercased,
length ≥ MIN_TOKEN_LEN. The build and query sides MUST use this same
function so a query word matches how it was indexed.