pub fn terms(text: &str) -> BTreeSet<String>Expand description
Deterministic term set of text.
- ASCII word runs (
[a-z0-9_\-/.:], lowercased) of more than one char are one term each — call ids, paths, and URLs stay whole. - Non-ASCII alphanumeric runs of more than one char are one term each; a lone non-ASCII char is kept as its own term.
- Runs containing Han chars additionally emit every adjacent-char bigram, the standard whitespace-free segmentation proxy.
Punctuation (ASCII and fullwidth alike) and whitespace never become terms.