Module rnix::tokenizer

source · [−]
Expand description

The tokenizer: turns a string into tokens, such as numbers, strings, and keywords

Structs

Tokenizer

The tokenizer. You may want to use the tokenize convenience function from this module instead.