Module rnix::tokenizer

source ·
Expand description

The tokenizer: turns a string into tokens, such as numbers, strings, and keywords

Structs

The tokenizer. You may want to use the tokenize convenience function from this module instead.

Functions

A convenience function for tokenizing the given input

Type Definitions