Module clue_core::scanner

The scanner is the second step of the compilation process, it takes the preprocessed source code and turns it into a list of tokens

It exposes a single function, scan_code, which takes a Code and returns a Vec of Token

Structs

• BorrowedToken
  A token that has a raw pointer to a Token.
• Token
  Represents a token with its type, its literal string and the location in the file. The type is represented by a TokenType

Enums

• TokenType
  The token’s type, used to represent keywords, literals, symbols, etc…

Functions

• scan_code
  Scans the code and returns a Vec of Tokens It takes a preprocessed code and a filename as arguments