Module token 

Token types emitted by the lexer.

A Token can be a:

• TerminalToken: an atomic symbol or reserved identifier.
• LiteralToken: an integer, float, char or string literal.
• Identifier: an identifier, for naming things.

Tokens are also fully source-preserving. Each token maintains its range in the input string, as well as the comments and newlines after the token.

Structs

Token

A source-preserving token.

TokenLine

A line of source input in a Token.

Enums

Comment

A comment.

LiteralBase

The base of an Int literal.

LiteralToken

A literal.

StringToken

A string literal.

TerminalToken

An atomic symbol or reserved identifier.

TokenType

The type of a Token.