Expand description
Contains the Token and TokenType enums used internally.
Defines the Token and TokenType enums.
These are used as an intermediate representation between the
Tokenizer (lexer) and the StreamingParser (parser).
This module is part of the library’s internal API.
Structs§
- Token
- A single token produced by the
Tokenizer.
Enums§
- Token
Type - The specific type of a
Token.