Module pattern_lexer::token

Module containing the tokens

Structs

• Token
  A lexical token.
• Tokenizer
  Produce Token that match a Pattern

Enums

• TokenKind
  Premade Token kinds for semantic (examples are not mandatory): | rust TokenKind | Explanation | Examples | |————————|————————————|–––––––––––| | rust KEYWORD | Reserved words | if, return, … | | rust DELIMITER | Paired delimiter symbols | (), [], {}, … | | rust PUNCTUATION | Punctuation symbols | ;, ., … | | rust OPERATOR | Symbols that operates on arguments | +, -, =, … | | rust COMMENT | Line or block comments | //, /* … */, … | | rust WHITESPACE | Non-printable characters | - | | rust LITERAL | Numerical, logical, textual values | 1, true, “true”, … | | rust IDENTIFIER | Names assigned in a program | x, temp, PRINT | These token kinds (except rust IDENTIFIER) should be constructed with a name that can be used to differentiate tokens with same kind.