Expand description
The blue lexer.
Blue’s surface is Ruby/Elixir-shaped, so the lexer’s job is different
from an s-expression reader’s: it must distinguish foo the send from
foo(x) the call, keep :sym distinct from a ? b : c, and record
enough position to point a diagnostic at the byte the human typed.
Two decisions here are load-bearing downstream and are made once:
- Every token carries a byte span.
theory/BLUE.md§0 requires total provenance — every node in an expanded program traceable to the source that caused it — and provenance cannot be recovered later if the lexer drops it. - Trivia is a token, not a skip. Comments and newlines are emitted rather than discarded, because a canonical formatter and an LSP both need a lossless stream. The measured failure this avoids is tatara-lisp’s own reader, which discards trivia at tokenize time and thereby makes a comment-preserving formatter unbuildable on top of it. Callers that do not want trivia filter it; callers that need it cannot conjure it back.
Structs§
Enums§
Functions§
- lex
- Tokenize
src, including trivia.