Module tokenizer
rnix
The tokenizer: turns a string into tokens, such as numbers, strings, and keywords
The tokenizer. You may want to use the tokenize convenience function from this module instead.
tokenize