Expand description
Token types emitted by the lexer.
A Token can be a:
TerminalToken: an atomic symbol or reserved identifier.LiteralToken: an integer, float, char or string literal.Identifier: an identifier, for naming things.
Tokens are also fully source-preserving. Each token maintains its range in the input string, as well as the comments and newlines after the token.
Structs
Enums
A comment.
The base of an
Int literal.A literal.
A string literal.
An atomic symbol or reserved identifier.