Expand description
The default performance-focused tokenizer
Structs§
Enums§
Functions§
- tokenize_
bytes - Like tokenize_str but for bytes
- tokenize_
rt_ bytes - Like tokenize_rt_str but for bytes
- tokenize_
rt_ str - Like tokenize_str but includes whitespace and comment tokens
- tokenize_
str - Turn str into Tokens.