pub fn extract_ngrams(token_str: &str, n: usize) -> HashSet<String>Expand description
Build the set of n-grams of a space-separated token string.
Mirrors the Python reference: a stream shorter than n collapses to the
single full-stream tuple (or the empty set when the stream is empty).
N-grams are joined back into a String with '\u{1f}' (unit separator) as
a delimiter — chosen because it cannot appear inside a normalised token
class — so hashing / equality stay cheap.