Module full_moon::tokenizer

Used for tokenizing, the process of converting the code to individual tokens. Useful for getting symbols and manually tokenizing without going using an AST.

Structs

Position

Used to represent exact positions of tokens in code

Token

A token such consisting of its Position and a TokenType

TokenReference

A reference to a token used by Ast’s. Dereferences to a Token

TokenizerError

Information about an error that occurs while tokenizing

Enums

StringLiteralQuoteType

The types of quotes used in a Lua string

Symbol

A literal symbol, used for both words important to syntax (like while) and operators (like +)

TokenKind

The kind of token. Contains no additional data.

TokenType

The type of tokens in parsed code

TokenizerErrorType

The possible errors that can happen while tokenizing.

Functions

tokens

Returns a list of tokens. You probably want parse instead.