Module lexer 

Lexer that tokenizes .cabal source into CST nodes. Hand-written lexer for .cabal files.

The lexer operates line-by-line, classifying each line and producing tokens that capture every byte of the input (via spans and trivia). The parser consumes these tokens to build the CST.

Key properties:

• Zero-copy: tokens are Span references into the source string.
• Full coverage: every byte is accounted for by a token span or trivia piece.
• Indentation tracking: each token records its column position (tabs = 8).

Structs

Token

A single token produced by the lexer.

TriviaPiece

A piece of trivia attached to a token.

Enums

TokenKind

The kind of a token.

TriviaKind

The kind of a trivia piece.

Functions

tokenize

Tokenize a .cabal source string into a flat list of Tokens.