[−][src]Module full_moon::tokenizer
Used for tokenizing, the process of converting the code to individual tokens. Useful for getting symbols and manually tokenizing without going using an AST.
Structs
Position | Used to represent exact positions of tokens in code |
Token | |
TokenReference | A reference to a token used by Ast's.
Dereferences to a |
TokenizerError | Information about an error that occurs while tokenizing |
Enums
StringLiteralQuoteType | The types of quotes used in a Lua string |
Symbol | A literal symbol, used for both words important to syntax (like while) and operators (like +) |
TokenKind | The kind of token. Contains no additional data. |
TokenType | The type of tokens in parsed code |
TokenizerErrorType | The possible errors that can happen while tokenizing. |
Functions
tokens | Returns a list of tokens.
You probably want |