Skip to main content

Module lex

praxis_parser

Module lex 

Source
Expand description

The Praxis lexer.

Turns source text into a stream of Tokens (each carrying a SyntaxKind and a Span) plus diagnostics. It is lossless — trivia (whitespace and comments) is kept as real tokens so the parser can fold them into the rowan tree verbatim (§13.1, ADR-003).

Design notes:

  • Longest match for operators: ->, =>, ==, !=, ..=, +=, … are recognized before their single-character prefixes.
  • Keywords are split out of the identifier run via SyntaxKind::from_keyword; out, panic, type names, etc. stay plain identifiers (they are builtins, not keywords).
  • Identifiers are Unicode (§4.1): classification is per scalar, using the workspace-wide praxis_syntax::ident predicates, so a leading Unicode letter starts a name and a UTF-8 continuation byte cannot extend one.
  • A bad scalar does not abort lexing: it emits a T003 diagnostic and the lexer advances one whole scalar so the rest of the file is still reported (§17.1, “multiple diagnostics from one malformed file”).

Diagnostic codes (T0xx, [DiagnosticCategory::Lex]):

  • T001 — unterminated block comment.
  • T002 — unterminated backtick template.
  • T003 — unexpected character in source.
  • T004 — unterminated text literal.
  • T005 — invalid escape in a text or character literal.
  • T006 — unterminated character literal.
  • T007 — a character literal that is not exactly one character.

Structs§

LexOutput
The result of lexing one source file: the token stream and any diagnostics.

Functions§

lex
Lex text belonging to file, returning tokens and diagnostics.