Module full_moon::tokenizer

Used for tokenizing, the process of converting the code to individual tokens. Useful for getting symbols and manually tokenizing without going using an AST.

Structs

• Position
  Used to represent exact positions of tokens in code
• Token
  A token such consisting of its Position and a TokenType
• TokenReference
  A reference to a token used by Ast’s. Dereferences to a Token
• TokenizerError
  Information about an error that occurs while tokenizing

Enums

• InterpolatedStringKindroblox
  Whether or not this section is the beginning, middle, end, or if this is a standalone string.
• StringLiteralQuoteType
  The types of quotes used in a Lua string
• Symbol
  A literal symbol, used for both words important to syntax (like while) and operators (like +)
• TokenKind
  The kind of token. Contains no additional data.
• TokenType
  The type of tokens in parsed code
• TokenizerErrorType
  The possible errors that can happen while tokenizing.

Functions

• tokens
  Returns a list of tokens. You probably want parse instead.