Module lexer 

Lexer for kaish source code.

Converts source text into a stream of tokens using the logos lexer generator. The lexer is designed to be unambiguous: every valid input produces exactly one token sequence, and invalid input produces clear errors.

Pipeline (GH #95)

1. Scan — one composed source-order pass with explicit quote/escape/comment state extracts heredoc bodies and $((expr)) arithmetic, producing a rewritten buffer plus a complete replacement table (both coordinate systems).
2. logos — the regex vocabulary below classifies the rewritten buffer into tokens.
3. Marker resolution — scanner markers become Arithmetic / HereDoc tokens, matched POSITIONALLY against the replacement table (never by fishing identifier text); a word glued onto a marker is split so the parser can reject it loudly.
4. Span correction — every token span maps back to exact original-source byte ranges via the replacement table.
5. Fusion — flag-metachar, colon, and glob merges join span-adjacent runs, with fused text sliced VERBATIM from the source; compute_value_context (an explicit frame stack plus a statement-head DFA) decides where fusion is suppressed.

Token Categories

• Keywords: set, if, then, else, fi, for, in, do, done
• Literals: strings, integers, floats, booleans (true/false)
• Operators: =, |, &, >, >>, <, 2>, &>, &&, ||
• Punctuation: ;, :, ,, ., {, }, [, ]
• Variable references: ${...} with nested path access
• Identifiers: command names, variable names, parameter names

Structs

HereDocData

Tokens produced by the kaish lexer.

Spanned

A token with its span in the source text.

Enums

LexerError

Lexer error types.

Token

A word is anything that is not whitespace and not an operator, so the bareword and path rules below admit \u{80}-\u{10FFFF} — this file’s spelling of “any non-ASCII scalar value” — alongside their ASCII classes. bash never inspects a word’s bytes for alphabetic-ness, and café, 日本語, and ~/文書 lex the same shape as their ASCII equivalents.

TokenCategory

Semantic category for syntax highlighting.

Functions

parse_float

Parse a float literal.

parse_int

Parse an integer literal.

parse_string_literal

Extract the string content from a string token (removes quotes, processes escapes).

parse_var_ref

Parse a variable reference, extracting the path segments. Input: "${VAR.field[0].nested}" → ["VAR", "field", "[0]", "nested"]

tokenize

Tokenize kaish source into spanned tokens.

tokenize_with_comments

Tokenize, preserving Comment and LineContinuation tokens.