Expand description
Tokenizer for the extended lambda calculus surface syntax.
Spike 2 adds four tokens beyond spike 1: Bang (!), Assign (:=),
Semicolon (;), and the KwRef keyword. The lexer is otherwise the
same recursive functional pipeline.
Source must be ASCII; any non-ASCII byte is reported as
Error::UnexpectedChar.
Structs§
- Token
- A token paired with its source position.
Enums§
- Token
Kind - The syntactic kind of a token.
Functions§
- lex
- Lex the entire source string into a vector of tokens.