Skip to main content

Module token

Module token 

Source

Structs§

Point
Cursor position within the source text.
Site
One location in the source text: the index reached, and the row and column that index sits at. It is the same data TypeScript carries as the loose sI/rI/cI fields on both Point and Token (ts/src/lexer.ts), and as the ScanOut record its scan driver writes back — naming it once is what keeps those uses in step.
Token
A single lexical token produced by the lexer.
TokenCode
A token’s diagnostic code, held behind a pointer.
TokenText
A token’s name or matched source text.
TokenValFunc
Lazy token value callback, evaluated only when a parser action asks for the token’s semantic value. This is the Rust counterpart of the canonical TokenValFunc (rule, context) => value extension point.

Constants§

TIN_AA
TIN_BD
TIN_CA
TIN_CB
TIN_CL
TIN_CM
TIN_CS
TIN_LN
TIN_MAX
TIN_NR
TIN_OB
TIN_OS
TIN_SP
TIN_ST
TIN_TX
TIN_UK
TIN_VL
TIN_ZZ

Functions§

name_to_tin
tin_name

Type Aliases§

Tin
Tin is a token identification number.