[−][src]Module full_moon::tokenizer
Used for tokenizing, the process of converting the code to individual tokens. Useful for getting symbols and manually tokenizing without going using an AST.
Structs
Position | Used to represent exact positions of tokens in code |
Token | |
TokenizerError | Information about an error that occurs while tokenizing |
Enums
StringLiteralQuoteType | The types of quotes used in a Lua string |
Symbol | A literal symbol, used for both words important to syntax (like while) and operators (like +) |
TokenType | The type of tokens in parsed code |
TokenizerErrorType | The possible errors that can happen while tokenizing. |
Functions
tokens | Returns a list of |