Module lexer 

Lexer for kaish source code.

Converts source text into a stream of tokens using the logos lexer generator. The lexer is designed to be unambiguous: every valid input produces exactly one token sequence, and invalid input produces clear errors.

Token Categories

• Keywords: set, tool, if, then, else, fi, for, in, do, done
• Literals: strings, integers, floats, booleans (true/false)
• Operators: =, |, &, >, >>, <, 2>, &>, &&, ||
• Punctuation: ;, :, ,, ., {, }, [, ]
• Variable references: ${...} with nested path access
• Identifiers: command names, variable names, parameter names

Structs

HereDocData

Tokens produced by the kaish lexer.

Spanned

A token with its span in the source text.

Enums

LexerError

Lexer error types.

Token

TokenCategory

Semantic category for syntax highlighting.

Functions

parse_float

Parse a float literal.

parse_int

Parse an integer literal.

parse_string_literal

Extract the string content from a string token (removes quotes, processes escapes).

parse_var_ref

Parse a variable reference, extracting the path segments. Input: “${VAR.field[0].nested}” → [“VAR”, “field”, “[0]”, “nested”]

tokenize

Tokenize source code into a vector of spanned tokens.

tokenize_with_comments

Tokenize source code, preserving comments.